Sven Erik Matzen

Software Architect | Cloud & Security Expert | AI-enabled Solutions

The Programmable Kernel: eBPF and the Sandbox at the Heart of the Operating System

🎧 Listen to this article

Cloud Computing · 2026-08-31

EU label: fully AI-generated content Fully AI-generated article (no prior review).

The Hook: A Stranger's Code in the Holy of Holies

Imagine you were allowed to bolt your own small program into the gears of a moving train — while the train runs at full speed, with hundreds of passengers aboard, and a single mistake would halt or derail not merely your code but the entire locomotive. That, in a different guise, has always been the forbidden fruit of systems programming: running code directly in the operating-system kernel, that privileged core that sits above everything, controls all the hardware, and offers no safety nets. A program in userspace can crash, and the operating system cleans it up. A program in the kernel that crashes takes the whole machine down with it — the infamous kernel panic, the Unix world's blue screen.

For decades this was a hard, immovable boundary. Anyone who wanted to extend the kernel — to filter network packets differently, to observe system calls, to enforce a new security rule — had exactly two options, both unsatisfying. Either you wrote a kernel module: native machine code running with full privileges inside the kernel, blazingly fast but life-threatening, because a single stray pointer brings the whole system down or opens a security hole in the most powerful part of the software. Or you moved the work into userspace: safely encapsulated, but slow, because every interaction with the kernel is an expensive context switch and the data has to be laboriously copied back and forth.

It seemed like another one of those iron trade-offs that run through computer science: you can have the speed and deep visibility of the kernel — or the safety of userspace, not both. It is precisely this apparent inevitability that a technology called eBPF has broken open over the past decade, and so thoroughly that it is now regarded as one of the most influential innovations in the history of the Linux kernel. The idea is as elegant as it is subversive: you do let a stranger's code into the kernel — but only code that a mathematical inspection authority has provably deemed harmless beforehand. Not trust, but verification.

For someone like Sven, who has one foot in cloud architecture and one in IT security, eBPF is a prime example of how a clever renegotiation of the trust boundary shifts a seemingly unresolvable compromise — much as Betting on a Stranger's Code: Firecracker microVMs and the End of the Container-versus-VM Dilemma did for virtualization. This article takes you along the whole route: from the surprising origin in a packet filter of the early 1990s, through the three pillars that make eBPF safe and the building blocks it is made of, to the three great fields it is reshaping today — networking, observability, and security — and finally to the honest limits of a technique that has itself become a powerful new attack surface.

Part 1: The Surprising Origin — from a Packet Filter to a Universal Machine

The story begins not with a grand vision but with a very concrete, very sober problem: how do you efficiently fish specific network packets out of the data stream without copying yourself to death? In 1992 Steven McCanne and Van Jacobson at Lawrence Berkeley Laboratory designed an answer, which they presented at the 1993 Winter USENIX conference in San Diego: the Berkeley Packet Filter (BPF). Anyone who has ever used the tool tcpdump and typed an expression like tcp port 443 has unknowingly used BPF — that expression is translated into a small program that runs in the kernel.

The ingenious trick of McCanne and Jacobson was not the filter itself but the architecture behind it. Instead of processing rules in a rigid table, they defined a tiny, abstract virtual machine inside the kernel: a simple register machine with a clearly delineated instruction set, in which filter logic could be expressed as a small program. The filter thus became, from a data structure, code. Packets no longer had to be copied into userspace to be discarded there; the decision was made in the kernel, right at the source. That was fast, frugal, and — revolutionary for its time — programmable within a safe, tightly bounded abstraction.

For over two decades this "classic BPF" (retrospectively called cBPF) remained a useful but limited niche tool for packet filters and, later, for restricting the system calls a process is allowed to make via seccomp-bpf — the mechanism that also plays a role in container isolation. Then, in 2014, Alexei Starovoitov did something that fundamentally changed things. He took the underlying idea — a safe virtual machine in the kernel — and rebuilt it from the ground up, conceived on a far larger scale. The result, "extended BPF" or eBPF, appeared in December 2014 in Linux kernel 3.18.

The leap was qualitative, not merely quantitative. The new virtual machine got ten registers instead of two, a width of 64 bits to match modern hardware, the ability to hold persistent data structures, and — the decisive point — it was decoupled from the network stack. eBPF was no longer a packet filter that happened to be programmable, but a general-purpose execution engine inside the kernel, to which one could attach at dozens of points. The abbreviation BPF stuck for historical reasons, but it is misleading today: modern eBPF shares only its ancestry with "Berkeley" and "Packet Filter." It has become something that kernel developers describe, half in jest and half in earnest, as a "scripting language for the kernel" or the "JavaScript of the operating system": a way to safely extend the kernel at runtime, without recompiling it, without loading a module, and without rebooting the machine.

Part 2: The Dilemma — Kernel Module versus Userspace

To appreciate what eBPF accomplishes, one must dissect cleanly the problem it solves. The core of an operating system runs in the CPU's privileged kernel mode: it may do anything, sees everything, controls all hardware. Application programs run in user mode: locked inside their own memory region, without direct hardware access, supervised by the kernel. This separation, enforced by the hardware itself, is the foundation of all system security. It is the reason a crashing browser does not drag the whole machine down with it.

Anyone who wants to extend the kernel's capabilities thus faces a real dilemma. The path via a kernel module means bringing your own code into kernel mode. It is maximally powerful and fast, because it runs with full privileges right where the events happen. But it is also maximally dangerous: there is no isolation. A null-pointer access, a buffer overflow, an infinite loop — and the whole machine halts or becomes vulnerable. Every loaded module enlarges the Trusted Computing Base, the set of code one must blindly trust, and every bug in it is a potential total loss. To make matters worse, kernel modules are tightly bound to the exact kernel version and can break with every update.

The path via userspace reverses the trade-off exactly. Here the code is safely encapsulated; a crash affects only its own process. But the price is speed and visibility. Every question to the kernel — "which system calls is this process making right now?", "what is happening to this packet?" — requires a context switch, the expensive toggle between user and kernel mode, and often the copying of large amounts of data across that boundary. For a packet filter meant to sift through millions of packets per second, or for an observability tool that wants to record every system call, this overhead is ruinous. You see the events, so to speak, only from afar, through a narrow, costly window.

The common wisdom was therefore: deep visibility and speed or safety. eBPF's answer is a third position that dissolves the dilemma by drawing a new trust boundary — not between user and kernel mode, but inside the kernel, around each individual injected program. The code runs at the speed and visibility of the kernel, yet locked into a sandbox whose walls are made not of hardware but of a mathematical proof.

Part 3: The Three Pillars of Safety

How can you let a stranger's, potentially buggy code into the kernel without endangering the machine? eBPF rests on three load-bearing pillars, whose interplay is the whole trick.

The first and most important pillar is the verifier. Before an eBPF program is allowed to run at all, the kernel subjects it to static analysis — an inspection of the code without executing it. The verifier models the program as a directed graph of all possible execution paths and walks through them systematically. It demands and proves a series of hard properties. The program must terminate. Infinite loops are forbidden, because a program that never returns in the kernel would be a perfect denial-of-service attack against the machine itself. Early kernels forbade loops entirely; since kernel 5.3 the verifier allows bounded loops by mentally unrolling the loop and proving that the induction variable moves monotonically and reaches a bound. The program may touch only valid memory. Every pointer access is checked against known bounds; an out-of-bounds access, a read from uninitialized memory, a dereference of a possibly invalid pointer — the verifier simply does not permit any of it. The program may not leak sensitive kernel data and may perform only a narrowly circumscribed set of allowed operations. Only when all these proofs succeed is the program accepted; if even one fails, it is rejected outright. The verifier is thus not a guard that intervenes at runtime, but a gatekeeper that decides before the start — and, in case of doubt, against admission.

The second pillar is JIT compilation (just-in-time). A once-verified eBPF program initially exists as platform-independent bytecode for the virtual machine. Instead of interpreting this bytecode slowly, instruction by instruction, the kernel translates it at load time into native machine code for the respective CPU architecture. The result runs at the full speed of hand-written kernel code. Safety and speed therefore do not exclude each other after all: the safety lies in the inspection before execution, the speed in the native execution afterward. You pay for the safety once, at load time, not on every single instruction.

The third pillar is the restricted environment itself. An eBPF program is not a free C program. It may not call arbitrary kernel functions but only a curated, stable set of helper functions provided by the kernel — say, to read the current time, to access a data structure, or to rewrite a packet. It has a bounded stack, it cannot allocate memory arbitrarily, and it cannot block the kernel. This deliberate austerity is not harassment but part of the safety architecture: the smaller and clearer the interface between the sandbox program and the kernel, the less can go wrong and the more completely the verifier can conduct its proofs.

Together the three pillars produce a remarkable inversion of the usual logic. With a kernel module, the question is: "Do I trust the author enough to let their code run with full privileges?" With eBPF, it is: "Can a machine prove that this code, no matter who wrote it, can do no harm?" It is the transition from the social category of trust to the mathematical category of proof.

Part 4: The Building Blocks — Programs, Hooks, Maps, and Helpers

A single, isolated program in the kernel would be worth little. The power of eBPF arises from the interplay of a few basic building blocks that can be assembled into astonishingly complex systems.

First there are the hooks, the attachment points. An eBPF program does not run on its own but is pinned to an event in the kernel and executed whenever that event occurs. The range of these attachment points is the real reason for eBPF's universality. At the XDP hook (eXpress Data Path), a program attaches within the network driver itself, at the earliest possible point, before the kernel has even created a data structure for the packet — this is the fastest point for packet processing, where attacks can be repelled or packets redirected before they cause any effort. At the tc hook (traffic control), you sit a bit later in the network path, with more context. At kprobes and tracepoints, you hook into nearly arbitrary functions and defined events in the kernel to observe them. At uprobes, you observe even functions in userspace programs. And at the LSM hook (Linux Security Module), you place yourself at the kernel's security-relevant decision points and can allow or deny operations. One and the same technology thus covers networking, observation, and security — only the attachment point differs.

The second building block is the maps. An eBPF program is stateless and short-lived; it fires when its event occurs and is then done again. To hold state and pass data along, there are maps: key-value stores in the kernel that persist. Several eBPF programs can share the same map, and — crucially — userspace can also read and write a map. This creates the bridge between the sandbox in the kernel and the application world above it: an eBPF program in the kernel, for instance, counts packets per connection into a hash map, and a userspace dashboard reads out these numbers without ever having to touch a packet itself. Maps come in many forms — hash tables, arrays, ring buffers for event streams, specialized structures for load balancing — and they are the backbone of any nontrivial eBPF application.

The third building block is the aforementioned helper functions and, as a more recent generalization, the kfuncs, through which programs may use certain kernel functions in a controlled way. And a fourth, subtle building block is the tail calls: the ability of an eBPF program to hand off to another, allowing larger logic to be decomposed into chained, individually verifiable morsels — an elegant way to work around the deliberate size limits of the sandbox without giving up safety.

There remains a stubborn practical problem: portability. An eBPF program that reads internal kernel data structures is bound to their exact memory layout — and this layout changes from kernel version to kernel version. Previously you therefore had to recompile eBPF programs on the target machine against the kernel headers running there, which was a torment in practice. The solution is called BTF (BPF Type Format) and CO-RE (Compile Once – Run Everywhere). BTF is a compact description of a kernel's types and data structures; CO-RE uses it so that a once-compiled eBPF program is automatically adapted at load time to the actual memory layout of the running kernel. Together with the userspace library libbpf, CO-RE turned eBPF from a fragile piece of tinkering into a robust, shippable technology — an often underestimated but decisive step toward broad practical usability.

Part 5: The First Field — Networking, Where It All Began

It is no coincidence that BPF's rebirth found its strongest echo in networking: this is where the technique is rooted, and where the pain was greatest. The classic tool for packet and rule processing in the Linux kernel was, for decades, iptables (and its substructure netfilter). It works, but its fundamental problem is linear processing: rules are checked in chains, one after another. With a handful of rules that does not matter. In a modern Kubernetes cluster with thousands of constantly changing services, however, the rule set grows enormously, and processing time grows with it — an O(n) behavior that becomes a noticeable bottleneck at large scale.

This is exactly where Cilium comes in, the most prominent eBPF project in networking, originally developed by the company Isovalent and today a graduated project of the Cloud Native Computing Foundation. Cilium's core idea is to move Kubernetes networking logic entirely into eBPF programs that attach to XDP and tc hooks and process packets before they traverse the full network stack. The most spectacular effect is the replacement of kube-proxy, the standard component that in Kubernetes maps service IP addresses to concrete backend containers. kube-proxy traditionally did this via iptables with that very linear rule processing. Cilium replaces it with hash-table lookups in eBPF maps, thereby turning the mapping from an O(n) search into an O(1) operation: a constant lookup time independent of the number of services. The kinship with the ideas from The Ring That Shares the Load: Consistent Hashing and the Art of Moving Gracefully is obvious — here, too, it is about distributing load efficiently and stably across many targets.

Cilium also brings an observability tool called Hubble, which makes a cluster's network flows visible in real time, because the eBPF programs that steer the traffic anyway can log it at the same time. Network policies can be enforced no longer only at the IP level but all the way up to the application layer (individual HTTP paths, for instance), and directly in the kernel. What iptables dragged along as conceptual burden, eBPF dissolves through programmability at the right place.

Part 6: The Second Field — Observability Without Instrumentation

Perhaps the most magical application field is observability. Anyone who previously wanted to know how an application behaves in detail — which functions take how long, which system calls it makes, where latency arises — usually had to instrument the code: build in measurement points, link in libraries, recompile the application, and roll it out. That is laborious, invasive, and often simply impossible, for instance with third-party software or in production systems one is not allowed to touch.

eBPF inverts this relationship. Because you can attach programs to kprobes, uprobes, and tracepoints, a running system can be X-rayed from the outside, without any change to the application code — the marketing phrase for this is "zero-instrumentation." The granular bedrock of this world is bpftrace, a scripting language modeled on the classic Unix tool awk, with which a question to the running kernel can be formulated in a few lines: "Show me the distribution of response times of all read system calls" or "which process is opening how many files right now." Beneath it lies the older library bcc (BPF Compiler Collection) with a collection of ready-made diagnostic tools. This ergonomics was shaped significantly by Brendan Gregg, whose work made eBPF tracing accessible to a broad engineering community.

On this foundation whole platforms have emerged. Pixie, for instance, automatically collects telemetry about latency, throughput, and resource usage in Kubernetes clusters without a developer having to write even a line of instrumentation. Parca uses eBPF for continuous profiling in live operation, to show where compute time is really being spent. The common denominator is always the same: instead of equipping each application individually with measurement technology, you measure once at the central point where all events converge anyway — in the kernel. For a cloud in which hundreds of heterogeneous services interact, this is a profound shift: observability turns from a property each application must laboriously provide itself into a property of the platform.

Part 7: The Third Field — Runtime Security in the Kernel

The third great field closes the circle back to IT security. If, with eBPF, you can observe every system call, every process creation, every file access, and every network connection in the kernel anyway, then you can use that same capability to detect malicious behavior in real time — and, at the right hooks, even prevent it.

The most mature tool of this kind is Falco, originally built by the company Sysdig, donated to the Cloud Native Computing Foundation in 2018, and now a graduated project there. Falco observes, via eBPF, the stream of system calls and raises an alarm when a process does things that suggest a breach: a shell started from within a web-server container; an access to sensitive files such as /etc/shadow; an unexpected outbound network connection. It is, in a sense, an intrusion-detection system that sits right at the source where the suspicious actions actually happen. Tetragon, from the Cilium ecosystem, goes a step further and can not only report suspicious operations but also enforce synchronously — that is, block a call or kill a process before the damage occurs — because it attaches to hooks at which the kernel has not yet made the decision. Tracee from Aqua Security is another prominent tool in this category.

The deepest integration is offered by BPF-LSM (also KRSI, Kernel Runtime Security Instrumentation), which attaches eBPF programs directly to the decision points of the Linux Security Module framework — those places where the kernel asks anyway: "Is this operation allowed to be carried out?" Here, programmable, fine-grained security policy can be anchored directly in the kernel, with the authority to bindingly allow or deny operations. The security-policy appeal is obvious: you no longer detect attacks only after the fact from log files, but intervene at the moment of their occurrence.

At the same time, this very field counsels sobriety. eBPF is a tool, not a miracle weapon. Side-channel attacks like Spectre and Meltdown (see When the Processor Guesses Too Much: Spectre, Meltdown, and the Sin of Speculative Execution), physical attacks like Rowhammer (see Bits That Flip on Their Own: Rowhammer and the Physical Weakness of Computer Memory), or sophisticated supply-chain attacks (see The Backdoor at the Heart of Linux: The XZ Attack and the Anatomy of a Supply-Chain Compromise) operate on layers that eBPF does not cover or can even affect itself. Security remains a layered affair.

Part 8: The New Frontiers — from the Scheduler to Windows

In recent years eBPF has begun to conquer even the deepest bastions of the kernel. The most spectacular recent step is sched_ext, admitted in kernel 6.12 (November 2024): a framework that allows the CPU scheduler — that highly critical core piece which decides which process may compute when and on which core — to be written as an eBPF program. What was previously unthinkable, because a buggy scheduler would bring the machine to a standstill instantly, becomes manageable through the verifier guarantees and a safety mechanism that, if needed, automatically replaces a hung eBPF scheduler with the built-in one. Suddenly, scheduling strategies for concrete workloads — games, servers, latency-critical services — can be tried out experimentally without rebuilding the kernel or rebooting the machine. It is perhaps the most impressive demonstration of the eBPF thesis: even the holy of holies of the kernel becomes safely programmable.

In parallel, the infrastructure keeps maturing. Newer kernels brought BPF tokens, which allow safer use of eBPF by less-privileged instances, as well as BPF arenas, which provide large, shared memory between programs. Institutionally, eBPF was placed under the umbrella of the Linux Foundation in 2021: the eBPF Foundation has since bundled the interests of companies such as Meta, Google, Microsoft, Isovalent, and Netflix and steers the further development as a communal good. And the idea has even switched operating systems: with eBPF for Windows, Microsoft is developing a port that brings the same programming interface to the Windows kernel — a sign that eBPF is turning from a Linux peculiarity into a cross-platform standard for safe kernel extension.

Part 9: The Honest Limits — When the Sandbox Itself Becomes an Attack Surface

An article that only praises a technique does it no favor. Sven's preference for scientifically grounded, verifiable statements demands that the downsides be named clearly too — and eBPF has them.

The sharpest lies in the core piece itself: the entire safety of eBPF rests on the correctness of the verifier. The verifier is a complex piece of software, and a bug in it means a program is declared safe even though it is not — with full kernel privileges. In the past, verifier vulnerabilities enabling privilege escalation have been found repeatedly. Especially delicate is the class of speculative type confusion: attacks that — related to Spectre — use the processor's speculative execution to undermine the verifier's guarantees, because the verifier checks the logical control flow while the CPU speculatively also enters paths that would logically never be reached. The kernel therefore has to weave additional hardening measures against speculative execution into eBPF programs, and this interplay remains an active field of research and security.

The second limit is a question of the Trusted Computing Base. eBPF shrinks the trust base in one respect — you no longer have to trust the individual loaded program, because it is verified. But it enlarges it in another: the verifier, the JIT compilation, and the growing set of helpers and hooks are all code in the kernel that one must trust. The more powerful eBPF becomes, the larger this new, critical attack surface grows. Because eBPF also grants deep insight into the whole system, it is a double-edged sword: the same capability with which Falco discovers attackers can be used by an attacker to build a particularly hard-to-detect rootkit that eavesdrops on traffic and erases its own traces in the kernel. eBPF-based malware of this kind has already been observed.

The third, practical limit is complexity. Writing eBPF programs that pass the verifier is an art of its own; the verifier's error messages are notoriously cryptic, and the sandbox's restrictions demand unusual programming patterns. And despite CO-RE, the binding to internal, unstable kernel structures remains a persistent source of fragility, as recent studies on the stability of eBPF-based kernel extensions show.

I am of the opinion that these limits do not reverse the balance but sharpen it: eBPF is not a magic trick that delivers safety for free, but a considered trade — you shift the trust from many unknown module authors to a single, intensely scrutinized verifier, and pay for it with the responsibility of keeping precisely that verifier flawless.

The Central Takeaway

The central lesson of eBPF reaches far beyond the Linux kernel. It runs: A seemingly iron trade-off can sometimes be dissolved by shifting the basis of trust — from the person who writes the code to a proof about the code itself. The old question "May I trust this author enough to let their code run with full privileges?" is a social, unreliable, non-scalable question. The eBPF question "Can a machine prove that this code, no matter from whom, can do no harm?" is a mathematical, verifiable, automatable question. Wherever, in your architecture, you face a hard choice between power and safety, the eBPF question is worth asking: is there a verifiable property I can enforce, instead of hoping for good behavior?

For daily practice this means, quite concretely: if you are wrestling in a Kubernetes cluster with network latency, opaque behavior, or security requirements, it is worth looking at the eBPF tools before reaching for more invasive means. A bpftrace one-liner can answer a question about the running system that previously would have required a debugger, a reboot, or rebuilt code — without touching production. And the next time you hear that a cloud provider has "completely rethought the network" or offers "observability without agents," the odds are good that in the engine room a verified little program at the heart of the kernel is doing the work.

A Closing Question for Reflection

eBPF has shown that a stranger's code can be safely admitted into the holy of holies of a system, provided a machine proves its harmlessness. If this idea — execution only after a machine-conducted safety proof — is so viable: at which other places in your systems do you still rely today on trust in the author of a piece of code, where tomorrow you could demand a proof about the code — and what would change in your architecture if verification became the norm and trust the exception?

Cross-References in the Vault

Sources

← All articles