eBPF

The service mesh page ended by hinting that you can do some of the network’s work deeper than a sidecar — in the kernel itself. This page is that deeper layer. To see why it matters, recall the process model: your applications run in user space, and the kernel runs in a separate, privileged kernel space. Every interesting thing a program does — open a file, send a packet, spawn a process — is a request that crosses into the kernel via a system call. The kernel sees everything. User space sees only what the kernel hands back.

That asymmetry is the root problem eBPF solves. For decades, if you wanted to observe or change what the kernel does — trace every network connection, enforce a security policy on every syscall, route packets a new way — you had two bad options: write a kernel module (powerful, but a bug crashes the whole machine), or run a userspace agent that constantly asks the kernel for data (safe, but slow, and always a step removed from the action). eBPF is a third option that has neither flaw.

What eBPF actually is

eBPF lets you load small programs into the running kernel and attach them to events — a packet arriving, a syscall firing, a function being entered, a tracepoint being hit. When the event happens, your program runs inside the kernel, at that exact moment, with direct access to the relevant data. Then it returns. You didn’t recompile the kernel, you didn’t load a risky module, and you didn’t reboot.

   THREE WAYS TO EXTEND THE KERNEL
   ───────────────────────────────
   kernel module        userspace agent          eBPF program
   ─────────────        ───────────────          ────────────
   runs in kernel       runs in user space       runs in kernel
   full power           safe, but slow            safe AND fast
   a bug = kernel       polls/copies data         verified before load
   panic (whole box)    across the boundary       sandboxed, event-driven
   reboot to update     always a step behind      load/unload live

The magic word in that last column is verified. Before the kernel accepts an eBPF program, a component called the verifier statically analyzes it and refuses to load it unless it can prove the program is safe: it must terminate (no unbounded loops), it can’t access memory it shouldn’t, and it can’t crash the kernel. Only a program that passes this proof is allowed to run. That’s how you get a kernel module’s power with a userspace program’s safety — the safety is enforced before the code ever runs, not hoped for.

The verifier is the whole trick

Letting arbitrary code run in the kernel sounds insane — one bad pointer panics the machine. eBPF is only viable because the verifier exists. It is a mini proof-checker that rejects any program it can’t prove safe. The trade is real: eBPF programs are restricted (bounded loops, limited size, a fixed set of allowed operations) precisely so the verifier can reason about them. You give up generality to gain a safety guarantee strong enough to run in ring 0.

Why it beats a userspace agent

The traditional way to observe a system is an agent in user space: it reads /proc, parses logs, or polls the kernel for stats, then ships them off. This has three structural weaknesses that eBPF doesn’t.

Property	Userspace agent	eBPF program
Where it runs	User space, outside the action	In the kernel, at the event
How it gets data	Polls/copies across the boundary	Sees the event directly, no copy
Overhead	High — context switches, copies, sampling	Low — runs in-place, event-driven
Coverage	Misses what happens between polls	Sees every event, no sampling gap
Blind spots	Can’t see inside the kernel’s decisions	Sits exactly where the decision is made

The deepest difference is the observability gap. A userspace agent that samples every second is blind for 999ms out of every 1000 — a short-lived process or a burst of dropped packets between samples simply never existed as far as the agent is concerned. An eBPF program attached to the relevant event fires on every occurrence. It doesn’t sample reality; it witnesses it. For diagnosing the rare, fast, hard-to-reproduce problems — the ones that matter most — that difference is everything.

There’s also a cost difference. Crossing the user/kernel boundary is expensive (context switches, copying buffers). An eBPF program runs where the data already is, so it can summarize or filter in-kernel and hand user space a tiny aggregate instead of a firehose. You get more visibility for less overhead — the opposite of the usual trade.

What it’s used for

eBPF is a mechanism, not a product. The interesting thing is what people build on it, and they cluster into three domains:

   ONE MECHANISM, THREE DOMAINS
   ────────────────────────────
                ┌──────────── eBPF (programs attached to kernel events) ───────────┐
                │                                                                   │
        OBSERVABILITY                    NETWORKING                    SECURITY
        ─────────────                    ──────────                    ────────
   trace every syscall,            route/filter packets          watch every process
   connection, function;           in-kernel; replace            spawn, file open, and
   build a map of what             iptables; load-balance;       syscall; enforce policy;
   talks to what, with             enforce network policy        detect anomalies as
   no app instrumentation          by identity                   they happen

• Observability. Because an eBPF program can attach to syscalls and network events, it can build a complete picture of system behavior — which service talks to which, latency per connection, errors — without touching application code or adding a sidecar. This is the natural counterpart to the three pillars: it generates the raw signals at the source. Pixie uses eBPF to auto-instrument applications this way, capturing requests and metrics with no code changes.
• Networking. eBPF can make packet-routing decisions in the kernel, which is faster and more flexible than the old iptables chains. Cilium uses eBPF to provide Pod networking, load balancing, and identity-aware network policy — and is part of why newer service meshes can move work out of per-Pod sidecars and into the kernel, cutting the latency cost the mesh page warned about.
• Security. Because eBPF sees every syscall as it happens, it’s ideal for runtime security: detecting that a container just spawned a shell, opened a sensitive file, or made an unexpected outbound connection — and acting on it. Falco uses kernel-level instrumentation to flag exactly these runtime anomalies, complementing the build-time defenses from image scanning.

Build-time scanning and runtime detection are complementary

Image scanning asks “is there a known-bad thing in this image?” before it runs. eBPF-based tools ask “is this running container doing something it shouldn’t?” right now. The first is a snapshot; the second is a live feed. A container can pass every scan and still be exploited at runtime via something the scan couldn’t predict — which is exactly the blind spot kernel-level observation covers.

Under the hood — why eBPF beats the iptables data path

The page keeps saying eBPF “replaces iptables,” but the why is a data-structure story worth seeing. For years, in-cluster traffic routing (Kubernetes’ default kube-proxy in iptables mode) was built from iptables rules: to steer a Service’s traffic to one of its backing Pods, the kernel walks a chain of rules, roughly top to bottom, until one matches.

   iptables mode                       eBPF mode (Cilium)
   ─────────────                       ──────────────────
   rules in a CHAIN, walked            a HASH MAP, looked up
   roughly top-to-bottom               by key in one step
   match cost grows with the           match cost is ~constant
   number of rules                     regardless of table size
   1000s of Services ⇒ a long          1000s of Services ⇒ the
   list to traverse per packet         same fast lookup per packet
   O(n)                                O(1)

That linearity is the problem. The rule set grows with the number of Services and endpoints, so on a large cluster the kernel may traverse a long chain for every packet, and even updating the rules means rewriting big tables. An eBPF program instead keeps the Service-to-endpoint mapping in an eBPF map — a kernel hash table — and does a constant-time lookup. Same job, but the cost stops scaling with cluster size. That is the concrete mechanism behind “newer meshes move work into the kernel”: they swap an O(n) chain walk for an O(1) map lookup, on the hottest path in the cluster.

The thread

The manual, error-prone step eBPF removes is instrumenting systems by hand, one app and one agent at a time — adding tracing libraries to every service, deploying polling agents that miss what happens between samples, and either crashing the box with a kernel module or settling for a blurry userspace view. eBPF replaces all of that with programs that attach to the source of truth (the kernel) and see every event with near-zero overhead. Production gets safer because the things you most need to catch — the rare dropped packet, the short-lived process, the unexpected syscall a compromised container makes — are exactly the things sampling agents miss and eBPF catches. You go from a blurry, sampled, instrument-everything-yourself picture to a complete, low-overhead, witnessed one.

The honest cost: eBPF programs are constrained (the verifier’s price), the tooling is sophisticated and kernel-version-sensitive, and “a program running in your kernel” is a powerful capability that itself must be governed. But the trade — full visibility for almost no overhead, with safety proven before the code runs — is why eBPF is the foundation under the newest generation of observability, networking, and security tools. The next page returns to the deployment side and asks how to ship changes so gradually and reversibly that a bad one is caught before most users ever see it: progressive delivery.

The architect’s lens

Five questions for reaching into the kernel:

• Why does it exist? Because the kernel witnesses every syscall, packet, and process spawn, but historically you could only get in there via a risky kernel module (a bug panics the box) or a slow, blind userspace agent.
• What problem does it solve? It loads verified, sandboxed programs that attach to kernel events and run in-place — so you observe, route, or enforce at the source with near-zero overhead and no sampling gap, powering observability (Pixie), networking (Cilium), and runtime security (Falco).
• What are the trade-offs? Programs are deliberately constrained (the verifier’s price — bounded loops, limited operations), the tooling is sophisticated and kernel-version-sensitive, and “code running in your kernel” is itself a powerful capability that must be governed.
• When should I avoid it? When a simpler userspace tool or sidecar already meets the need — eBPF earns its complexity where you need every-event coverage or in-kernel speed (the O(1) map vs the O(n) iptables chain).
• What breaks if I remove it? You fall back to per-app instrumentation and sampling agents that miss what happens between polls — exactly the rare dropped packet, short-lived process, or unexpected syscall you most need to catch.

Check your understanding

1. Using the process model, explain why “the kernel sees everything” makes it the ideal place to observe and enforce — and why that was historically hard to exploit safely.
2. What are the two traditional ways to extend the kernel, and what’s wrong with each? How does eBPF avoid both flaws?
3. What does the verifier do, and why are eBPF programs deliberately restricted (bounded loops, limited operations)?
4. Explain the “observability gap” of a sampling userspace agent, and why an eBPF program attached to an event doesn’t have it.
5. Name the three domains eBPF is used for and give one tool per domain. Why are build-time image scanning and eBPF-based runtime security complementary rather than redundant?

Show answers

1. Every meaningful action (file open, packet, process spawn) crosses into the kernel via a syscall, so the kernel is the one place that witnesses every event — perfect for observing and enforcing. Historically you could only get in there via a kernel module (a bug panics the machine), so it was powerful but dangerous to exploit.
2. A kernel module (full power, but a bug crashes the whole box and you reboot to update) or a userspace agent (safe, but slow and always a step removed, polling across the boundary). eBPF runs in the kernel like a module but is verified and sandboxed like safe userspace code, and can be loaded/unloaded live.
3. The verifier statically proves a program is safe before loading it — it must terminate, can’t touch forbidden memory, can’t crash the kernel. Programs are restricted (bounded loops, limited size and operations) precisely so the verifier can prove those properties; generality is traded for a safety guarantee strong enough to run in the kernel.
4. A sampling agent is blind between samples — a process or packet burst that lives and dies between polls is invisible to it. An eBPF program fires on every occurrence of its event, so it witnesses reality instead of sampling it.
5. Observability (Pixie), networking (Cilium), security (Falco). They’re complementary because scanning is a build-time snapshot (“is a known-bad thing in the image?”) while eBPF runtime security is a live feed (“is the running container doing something it shouldn’t?”) — a container can pass every scan and still be exploited at runtime.