eBPF use cases

What is eBPF and how can it be used within the Kubernetes environment?

In the dynamic world of container orchestration, where speed and adaptability are a must, eBPF, short for Extended Berkeley Packet Filter, has changed how developers interact with kernels within Kubernetes environments. At its core, eBPF crosses traditional boundaries, offering a programmable and secure in-kernel execution environment that empowers developers to use custom code without the need for modifications to the kernel itself.

In essence, eBPF represents a paradigm shift in the landscape of network technologies, ushering in a new era where agility meets security at the kernel level. 

In this article, we dive deep into the intricacies of eBPF, unraveling its capabilities and exploring its seamless integration within the Kubernetes landscape. 

How does eBPF work?

eBPF operates by executing event-driven programs at specific hook points, offering diverse data structures, and ensuring safety through rigorous verification and hardening processes.

Event-Driven capabilities

eBPF programs are event-driven, executing at hook points such as system calls and network events. This functionality empowers developers to construct efficient and adaptable programs that can be linked to various hook points within the Linux kernel, unlocking numerous event-driven capabilities. Helper calls facilitate access to kernel functions and enable manipulation of network packets. Additionally, eBPF programs can execute tail and function calls to enhance composability and replace execution contexts.

Safety measures

eBPF ensures safety through mandatory privileges, verification, hardening, and restricted memory access, establishing a secure environment for the execution of eBPF programs within critical software infrastructure components.

Loading and verification

eBPF programs are loaded as bytecode using libraries and undergo verification and just-in-time (JIT) compilation steps. eBPF maps are indispensable for information sharing and state storage, offering a wide array of data structures for data retrieval and manipulation by both eBPF programs and user space applications. The verification step ensures program safety by checking for compliance with various conditions, preventing crashes, infinite loops, uninitialized variables, and exceeding system size limits. eBPF programs leverage eBPF maps to store and retrieve data in diverse data structures.

Abstractions for development

eBPF programs can be authored directly or through higher-level abstractions like Cilium or bcc, which provide more accessible methods for working with eBPF. These abstractions abstract the complexity of direct program authorship, allowing developers to focus on intent-based definitions. 

How eBPF can help improve observability, security, and networking

With eBPF, it is possible to implement hooks on various functions in kernel space and user space, such as memory, networking, and storage.

Networking 

Network security and monitoring can be challenging with traditional tools due to significant overhead and lack of flexibility. However, eBPF filters packet traffic at various layers of the network stack, allowing for custom filters based on the IP, protocol, or event. One of the key features of eBPF is the ability to implement XDP (eXpress Data Path), which provides a high-performance, programmable network data path. XDP allows for packet processing at the earliest possible point in the software stack, enabling efficient packet filtering, forwarding, and manipulation.

It is also worth mentioning that eBPF reduces the impact on system performance by efficiently managing interactions between the kernel and physical layers, minimizing potential overhead. 

Security

eBPF enables organizations to detect suspicious activity in real-time and report it to alert systems. For instance, you can analyze the network and file activity of your pods to prevent potential harm to your system. Building runtime-based threat detection tools often requires collecting a vast amount of data from various sources, such as file activity, processes, system calls, network traffic, and more. This data is then analyzed to identify potential threats or anomalies in real-time.

For example, an eBPF program can be attached to the kernel’s file system layer to monitor file read, write, and execute operations. This allows for the detection of suspicious file activities, such as the execution of unauthorized binaries or the modification of critical system files. Similarly, eBPF programs can be attached to the network stack to inspect and filter network packets, enabling the detection of malicious network traffic or unauthorized connections.

Observability

One of the key advantages of using eBPF for observability is its versatility. eBPF programs can be attached to various kernel subsystems, such as the network stack, file system, process scheduler, and more. This means that eBPF can be used to observe and analyze different aspects of the system, depending on the specific domain of interest. For example, developers can use eBPF to monitor network traffic and protocol behavior, file system operations, system call patterns, or even application-level metrics and events.

By leveraging eBPF’s capabilities, organizations can build comprehensive observability solutions tailored to their specific needs whether it’s monitoring network performance, analyzing application behavior, or tracking system resource utilization.

Difference between kernel modules and eBPF applications

The most valuable advantage of kernel modules is that they have unlimited possibilities. You’re developing code that can be loaded and unloaded within the kernel, allowing for flexible integration and removal as needed. These modules extend the functionality of the kernel by adding new features, drivers, or file systems. 

However, kernel modules do have their disadvantages. They can lack stability and compatibility, posing risks when not regularly maintained or updated to match the latest kernel versions; also, the fact that their APIs change between versions can lead to compatibility issues.

On the other hand, eBPF does not modify the kernel itself but instead attaches to various hooks provided by the kernel to perform specific actions like packet filtering and tracing. And eBPF applications can be loaded only if they’re safe to run and won’t crash your system.

One of the key components of the eBPF ecosystem is the eBPF verifier, which plays a crucial role in ensuring the safety and correctness of eBPF programs. The verifier performs a series of checks on the eBPF bytecode before allowing it to be loaded into the kernel. These checks include verifying that the program terminates, does not violate memory safety rules, and does not attempt to access unauthorized kernel data structures or functions.

The CO-RE (Compile Once-Run Everywhere) framework in eBPF enables the development of portable applications capable of running on multiple kernel versions and configurations without requiring modifications or runtime source code compilation on the target machine. With CO-RE, a single eBPF binary object can be loaded and executed on different kernel versions without the need for recompilation or complex workarounds. This framework leverages BPF Type Format (BTF) to enable BPF program portability, allowing the same code to be executed across various kernel versions, thereby enhancing developer experience and operational efficiency

eBPF offers a powerful and flexible way to extend the functionality of the Linux kernel without the need for kernel modifications or loadable kernel modules. By adhering to the rules enforced by the verifier and operating within the CO-RE framework, eBPF applications can be loaded only if they’re safe to run and won’t crash or compromise the system.

Advantages of using eBPF

As we previously mentioned, eBPF’s primary benefits are safety, security, and easy maintenance. An eBPF application can be loaded and removed from the kernel dynamically without system interruption. Once the application is attached to the specific event type, it will trigger it once the event occurs; this flexibility is especially significant when compared to a kernel upgrade, where system interruption is typically required and the process can be more disruptive to normal operations.

eBPF applications operate as native machine instructions on the CPU because of just-in-time (JIT) compilation while loading. Executing directly within the kernel allows them to bypass the overhead associated with context switching between the user level and the kernel level, resulting in reduced latency and enhanced throughput. 

Lastly, eBPF applications are usually small, which helps to reduce memory usage and minimize cache misses; this ultimately results in better overall performance.

Various tools are available to make use of eBPF. Below, we cover some interesting ones and discuss some use cases.

Networking use case with Cilium

Let’s start our review with Cilium, a cloud-native solution that is open-source, ensures secure network connectivity between workloads, and features monitoring capabilities for this purpose. 

Cilium implements host-based routing using eBPF to bypass iptables and the upper host stack, resulting in a faster switch of your network namespace. 

This solution can be highly effective for large clusters that have many nodes, pods, and services. Generally, these clusters have a large number of iptable filters and forwarding rules, which can cause significant delays in proper functioning at scale. By bypassing iptables, you can achieve significant performance improvements.

Cilium can also replace traditional Kubernetes network policies and provide extended, lighter, and faster ways to improve network security. Traditional Kubernetes network policies rely on iptable filters, which cannot handle high volumes of network traffic efficiently. Using workload identities based on labels and metadata, Cilium decouples security from network addresses. This enables more flexible and efficient scaling without the need for constant updates to security rules.

Lastly, Cilium provides support for L7 policies, offering fine-grained API-level security for common protocols like HTTP, Kafka, gRPC, and others. 

Observability use case with OpenTelemetry

OpenTelemetry is an open-source project commonly associated with tracing and metrics in the cloud-native community. It also leverages eBPF (extended Berkeley Packet Filter) to provide deeper insights into system behavior.

The OpenTelemetry project uses eBPF in components that collect and analyze telemetry from the operating system, cloud, and container orchestrators. Its initial focus is on collecting network data to enable users to gain insight into their distributed applications.

OpenTelemetry’s has three components that use eBPF:

Kernel Collector: OpenTelemetry’s kernel collector uses eBPF to capture network data, process information, and other relevant events. By tapping into the kernel, it minimizes overhead while providing valuable insights. 

Kubernetes Collector: For Kubernetes environments, OpenTelemetry extends its reach by collecting workload metadata. This information, combined with eBPF data, paints a comprehensive picture of your applications’ behavior. Namespace changes, network interactions, and resource utilization—all become part of the observability landscape.

Cloud Collector: When running workloads in the cloud, OpenTelemetry’s cloud collector collaborates with eBPF to gather telemetry.

Choose your own eBPF adventure with Inspektor Gadget

Inspektor Gadget provides a wide range of “gadgets” out of the box. This tool is responsible for managing, packaging, deploying, and executing eBPF programs within the Kubernetes cluster.

You can start with the advice gadgets for network policy. These let you monitor network activity within the specific namespaces, analyze network traffic between your workloads, and generate network policies. 

Snapshot gadgets capture and print the state of a system at a specific point in time. These help you quickly get an overview of what’s happening in your Kubernetes cluster, which is useful for debugging and troubleshooting.

Two main Snapshot Gadgets are process and socket. The process gathers information about running processes, including their PID, PPID, state, threads, and open files; the socket gathers information about TCP and UDP sockets, including their local and remote addresses, state, and protocol.

By using these gadgets, you can simplify the building of new security policies and gain a clearer understanding of how and where traffic is going within namespaces. 

ARMO Platform uses Inspektor Gadget, by way of Kubescape, to analyze runtime behavior. This information is used in two ways. During the software development lifecycle, it is used to give additional context and insights to inform hardening activities. In runtime, it is used to identify and flag anomalous behavior of applications and malicious behavior.  

eBPF inception with bpftop

bpftop, by Netflix, is an open-source command-line utility designed to enhance the performance optimization and monitoring of eBPF applications. Much like the familiar top utility, bpftop provides a dynamic real-time view of running eBPF programs. 

bpftop leverages the BPF_ENABLE_STATS BPF syscall command to enable global eBPF runtime statistics gathering. By default, this feature is disabled to reduce overhead. It collects statistics every second, calculating average runtime, EPS, and CPU utilization for each eBPF program within that sample period. The information is presented in a top-like tabular format.

Conclusion

eBPF is emerging as a key to many cloud-native use cases. It can be instrumental to organizations in unlocking new levels of observability, security, and networking capabilities within their Kubernetes environments. 

Leveraging eBPF, enables ARMO Platform to enhance recommendations for hardening Kubernetes as well as bring users runtime threat detection. Thus, creating a virtuous cycle of security improvement across development and operations. If you’re curious about how eBPF will improve your Kubernetes security from left to right, try ARMO Platform today.