How eBPF runs programs inside the Linux kernel
How eBPF safely injects custom programs into the kernel using a verifier, JIT compiler, and hook points for networking, security, and observability without kernel module risks.
The Interview Question
Interviewer: "Your team uses Cilium for Kubernetes networking and Falco for runtime security. Both rely on eBPF. Walk me through what eBPF actually is, how a program gets loaded into the kernel safely, and why this replaced kernel modules for observability and networking."
This question tests whether you understand the eBPF execution model beyond "it runs programs in the kernel." The interviewer wants to hear about the verifier (how safety is guaranteed), hook points (where programs attach), maps (how kernel and userspace share data), and JIT compilation (how performance stays close to native). If you just say "eBPF is like a virtual machine in the kernel," you get partial credit. If you walk through the verification pipeline and explain why bounded loops are required, you demonstrate real depth.
What to Clarify Before Answering
You: "Before I dive in, let me scope the answer..."
- "Should I focus on the eBPF runtime itself (verifier, JIT, maps) or on specific use cases (Cilium, Falco, Pixie)?"
- "Are we talking about modern eBPF (kernel 5.x+) with CO-RE and BTF, or legacy BPF (classic packet filtering)?"
- "Is the interviewer interested in the security implications of loading code into the kernel, or more the performance characteristics?"
- "Should I compare eBPF with kernel modules and user-space alternatives like ptrace?"
Why this matters: eBPF spans networking, security, tracing, and profiling. A candidate who narrows the scope shows they understand the breadth of the technology and can focus on what the interviewer actually cares about.
The 30-Second Answer
eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel without modifying kernel source code or loading kernel modules. You write a program in restricted C, compile it to eBPF bytecode, and load it via the bpf() syscall. The kernel's verifier statically analyzes every instruction to guarantee the program terminates, does not access invalid memory, and cannot crash the kernel. After verification, the JIT compiler translates bytecode to native machine code (x86, ARM) for near-zero overhead. Programs attach to hook points (kprobes, tracepoints, XDP, TC, cgroup hooks, socket filters) and execute whenever the hook fires. Programs communicate with userspace through maps (hash tables, arrays, ring buffers) that both sides can read and write. This architecture gives you kernel-level visibility and control with the safety of a sandbox. Cilium uses it for networking, Falco for security monitoring, and Pixie for application observability.
The Architecture Overview
Looking at the diagram above, the lifecycle starts in user space. You write a program in restricted C (or Rust with the aya framework), compile it with Clang/LLVM to eBPF bytecode, and load it via the bpf() syscall using a loader like libbpf. The kernel's verifier inspects every instruction path. If verification passes, the JIT compiler converts bytecode to native machine code and attaches it to a hook point. When the hook fires (a packet arrives, a syscall executes, a function is called), the eBPF program runs and writes data to maps. User-space applications read those maps to display metrics, enforce policies, or trigger alerts.
The beauty of this model is that no kernel recompilation is needed, no kernel module loading is required, and a buggy program cannot crash the kernel because the verifier prevents it at load time.
The Verifier: How Safety is Guaranteed
The verifier is the most important component of eBPF. It is what makes the entire model safe. Without it, loading arbitrary code into the kernel would be as dangerous as loading a kernel module. I find this the most fascinating part of eBPF's design.
What the Verifier Checks
The verifier performs static analysis on the eBPF bytecode before the program is allowed to run. It walks every possible execution path and enforces strict safety rules.
The Key Safety Guarantees
1. Guaranteed termination. The verifier ensures every loop has a provably bounded iteration count. In kernels before 5.3, loops were forbidden entirely. Modern kernels allow bounded loops where the verifier can statically determine the maximum iteration count. This prevents a program from spinning in an infinite loop and hanging the kernel.
2. Memory safety. Every memory access is tracked. The verifier knows whether a register holds a pointer to the stack, a map value, a packet buffer, or a scalar. It checks bounds on every dereference. If you try to read past the end of a packet buffer without a bounds check, the verifier rejects the program.
3. No invalid pointer arithmetic. You cannot cast an integer to a pointer. You cannot add an unbounded value to a pointer. The verifier tracks pointer provenance through every instruction.
4. Restricted kernel access. eBPF programs cannot call arbitrary kernel functions. They can only call a set of pre-approved "helper functions" (e.g., bpf_map_lookup_elem(), bpf_probe_read(), bpf_get_current_pid_tgid()). Each helper has defined argument types that the verifier enforces.
// Pseudocode: simplified verifier logic for memory access
verify_memory_access(instruction, register_state):
reg = register_state[instruction.src_register]
if reg.type == SCALAR:
REJECT("cannot dereference scalar as pointer")
if reg.type == PTR_TO_PACKET:
// Check that offset + size <= packet_end
if reg.offset + instruction.size > packet_end_register.value:
REJECT("packet access out of bounds, add bounds check")
if reg.type == PTR_TO_MAP_VALUE:
// Check that offset + size <= map_value_size
if reg.offset + instruction.size > map.value_size:
REJECT("map value access out of bounds")
if reg.type == PTR_TO_STACK:
// Check that offset is within stack frame [-512, 0)
if reg.offset < -512 or reg.offset + instruction.size > 0:
REJECT("stack access out of bounds")
ALLOW()
Why the verifier complexity limit matters
The verifier has a limit of 1 million verified instructions (the number of instructions it analyzes across all paths, not the program size). Complex programs with many branches can hit this limit because the verifier explores every path. This is the most common reason production eBPF programs fail to load, and it forces you to write simple, straight-line code. This is a feature, not a bug.
JIT Compilation: From Bytecode to Native Code
After the verifier approves a program, the JIT compiler translates eBPF bytecode to native machine instructions. This step is what makes eBPF fast enough to run in the kernel's hot path (packet processing, syscall handling) without noticeable overhead.
The eBPF Virtual Machine
The eBPF VM has 11 registers (R0-R10), a 512-byte stack, and a simple instruction set. R0 is the return value. R1-R5 are function arguments. R6-R9 are callee-saved. R10 is the read-only frame pointer.
| Register | Purpose |
|---|---|
| R0 | Return value, helper function return |
| R1 | First argument (context pointer) |
| R2-R5 | Arguments 2-5 for helper calls |
| R6-R9 | Callee-saved (preserved across calls) |
| R10 | Frame pointer (read-only, stack base) |
The instruction set includes: arithmetic (add, sub, mul, div, mod), bitwise (and, or, xor, shift), memory load/store (1/2/4/8 byte), branches (conditional, unconditional), function calls (to BPF helpers), and atomic operations (add, xchg, cmpxchg).
JIT Compilation Process
The JIT compiler maps each eBPF instruction to one or more native instructions. On x86_64, most eBPF instructions map 1:1 to native instructions because the eBPF ISA was designed to match modern CPU architectures.
// eBPF bytecode (one instruction)
BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1) // R0 += R1
// JIT output: single x86_64 instruction
add rax, rdi // rax = R0, rdi = R1
The JIT output runs at native speed. There is no interpretation overhead. Benchmarks show JIT-compiled eBPF programs running within 1-5% of equivalent hand-written kernel C code for packet processing workloads.
The key performance insight
The JIT compiler is why eBPF can sit in the XDP (eXpress Data Path) hook and process millions of packets per second. At the XDP level, the program runs before the kernel allocates an sk_buff structure, so packet filtering with eBPF/XDP is faster than iptables by 3-10x because it avoids the entire network stack overhead.
Hook Points: Where eBPF Programs Attach
eBPF programs do not run continuously. They attach to specific kernel events (hook points) and execute only when those events fire. The hook type determines what data the program receives and what actions it can take.
Key Hook Types
XDP (eXpress Data Path): Runs at the network driver level, before the kernel allocates an sk_buff. Can DROP, PASS, TX (bounce back), or REDIRECT packets. This is the fastest hook point, capable of processing 10+ million packets per second. Cilium uses XDP for load balancing and DDoS mitigation.
TC (Traffic Control): Runs after sk_buff allocation but before the packet reaches the socket layer. Can classify, shape, and redirect traffic. Cilium uses TC hooks for pod-to-pod networking in Kubernetes.
Kprobes: Attach to any kernel function's entry or return point. The program receives the function arguments (entry) or return value (return). This is the most flexible hook but has no stable ABI, so programs may break across kernel versions.
Tracepoints: Attach to predefined, stable kernel events (e.g., sched:sched_switch, syscalls:sys_enter_read). The kernel defines ~2000+ tracepoints. These are preferred over kprobes when a matching tracepoint exists because the interface is stable across kernel versions.
Cgroup hooks: Attach to cgroup events (device access, memory allocation, network connection). Kubernetes uses cgroup eBPF programs to enforce per-pod network policies.
LSM hooks: Attach to Linux Security Module decision points. eBPF LSM programs can approve or deny security-sensitive operations (file open, mmap, socket connect) without writing a full LSM kernel module.
Kprobes vs tracepoints: stability matters
Kprobes can attach to any kernel function, but function signatures change between kernel
versions. If you probe tcp_v4_connect() and its arguments change in a kernel update,
your program breaks silently. Always prefer tracepoints when available. Use kprobes only
for functions with no matching tracepoint, and use CO-RE (Compile Once, Run Everywhere)
with BTF to handle struct layout changes.
Maps: Kernel-Userspace Communication
Maps are the shared data structures through which eBPF programs communicate with each other and with user-space applications. A map is created via the bpf() syscall and is accessible from both the kernel side (eBPF programs) and user space (via file descriptors).
Map Types
| Map Type | Use Case | Lookup Time | Notes |
|---|---|---|---|
| Hash Map | General key-value storage | O(1) average | Most commonly used |
| Array | Fixed-size indexed data | O(1) | Index must be < max_entries |
| Ring Buffer | High-throughput event streaming | N/A (push/pull) | Replaces perf_event_array in modern kernels |
| LRU Hash | Cache with automatic eviction | O(1) average | Evicts least-recently-used entries when full |
| Per-CPU Hash | Lock-free per-CPU counters | O(1) average | Each CPU gets its own copy, no contention |
| Per-CPU Array | Lock-free per-CPU indexed data | O(1) | Fast counters and accumulators |
| Stack Trace | Capture call stacks | O(1) | For profiling, stores kernel + user stacks |
| Perf Event Array | Per-CPU event streaming (legacy) | N/A | Use ring buffer in kernel 5.8+ |
How Maps Work
When an eBPF program attached to a kprobe fires, it can look up data in a hash map, increment a counter in a per-CPU array, or push an event to a ring buffer. User-space tools then read those maps to display the data.
// Pseudocode: eBPF program counting syscalls by PID
MAP syscall_count: HASH(key=pid_t, value=u64, max_entries=10000)
SEC("tracepoint/raw_syscalls/sys_enter")
int count_syscalls(struct trace_event_raw_sys_enter *ctx) {
u32 pid = bpf_get_current_pid_tgid() >> 32;
u64 *count = bpf_map_lookup_elem(&syscall_count, &pid);
if (count) {
__sync_fetch_and_add(count, 1); // Atomic increment
} else {
u64 initial = 1;
bpf_map_update_elem(&syscall_count, &pid, &initial, BPF_ANY);
}
return 0;
}
// User-space reads the map to see syscall count per PID
// bpf_map_get_next_key() + bpf_map_lookup_elem() iteration
Tail Calls and Function Calls: Program Composition
eBPF programs have size limits (1 million instructions for verification). For complex logic, you split programs using tail calls and BPF-to-BPF function calls.
Tail Calls
A tail call is a jump from one eBPF program to another. The current program's stack frame is replaced (like exec(), not like a function call). This lets you chain up to 33 programs together, each with its own verifier analysis.
// Pseudocode: XDP firewall using tail calls
// Program 0: Parse Ethernet header, tail call to protocol handler
// Program 1: Handle IPv4 packets
// Program 2: Handle IPv6 packets
// Program 3: Handle ARP packets
SEC("xdp")
int xdp_main(struct xdp_md *ctx) {
// Parse Ethernet header
struct ethhdr *eth = ...;
// Tail call to protocol-specific handler
bpf_tail_call(ctx, &jmp_table, eth->h_proto);
// If tail call fails (no program for this proto), pass the packet
return XDP_PASS;
}
BPF-to-BPF Function Calls
Since kernel 4.16, eBPF supports calling other BPF functions within the same program. Unlike tail calls, these are real function calls with their own stack frame (up to 8 nesting levels). The verifier analyzes the callee function inline.
When to use tail calls vs function calls
Use tail calls when you need to exceed the instruction limit or when different programs need different permissions (e.g., one program uses kprobe, another uses XDP). Use BPF function calls for code reuse within a single program. Tail calls have a nesting limit of 33 and replace the stack. Function calls have a nesting limit of 8 and preserve the caller's stack.
Real-World Use Cases
Cilium: Kubernetes Networking
Cilium replaces kube-proxy and iptables with eBPF programs attached to TC and XDP hooks. Instead of traversing iptables chains (which grow linearly with the number of services), Cilium uses eBPF hash maps for O(1) service lookup. This gives Cilium consistent latency regardless of cluster size, while iptables degrades as the number of services grows.
Falco: Runtime Security
Falco attaches eBPF programs to syscall tracepoints to monitor container behavior. When a process inside a container opens /etc/shadow, executes a shell, or makes a suspicious network connection, the eBPF program captures the event and pushes it to a ring buffer. Falco's user-space daemon reads the ring buffer and evaluates rules to generate alerts.
Pixie: Application Observability
Pixie uses uprobes to trace application-level protocols (HTTP, gRPC, MySQL, Postgres) without any code changes or sidecars. eBPF programs attached to OpenSSL's SSL_read() and SSL_write() functions capture decrypted request/response data. This gives you application-level observability with zero instrumentation overhead.
XDP-based DDoS Mitigation
Cloudflare and Facebook use XDP programs to filter malicious packets at line rate. Because XDP runs before the kernel allocates sk_buff, a single server can drop 10+ million packets per second of DDoS traffic with minimal CPU overhead. This is 3-10x faster than iptables-based filtering.
What Happens When Things Break
| Failure | What Happens | How to Detect | How to Fix |
|---|---|---|---|
| Verifier rejects program | Program never loads, error returned | bpf() returns -EACCES with log | Fix the code to satisfy verifier (add bounds checks, bound loops) |
| Map memory exhaustion | bpf_map_update_elem() returns -ENOMEM | Map operation failures in tracing | Increase max_entries or use LRU map for auto-eviction |
| Ring buffer overflow | Events dropped, bpf_ringbuf_reserve() returns NULL | Track drop count in a separate map | Increase buffer size or reduce event rate |
| Kprobe on renamed function | Program loads but never fires | No events in maps, zero counters | Use tracepoints or update function name, use CO-RE |
| JIT disabled (rare) | Program runs in interpreter (10x slower) | Check /proc/sys/net/core/bpf_jit_enable | Set bpf_jit_enable = 1 in sysctl |
| Kernel version mismatch | Program uses unavailable features | Load fails with version-specific error | Use CO-RE + BTF for cross-version portability |
| Stack overflow (512B limit) | Verifier rejects program | Verifier log shows stack size exceeded | Move large variables to maps, reduce nesting |
The safety guarantee
The most important property of eBPF is that a verified program cannot crash the kernel. The worst case is that it consumes CPU time (bounded by the scheduler) or fills a map (bounded by max_entries). Compare this with kernel modules, where a single bug can cause a kernel panic, data corruption, or a security vulnerability. This safety guarantee is why eBPF is replacing kernel modules for observability and networking.
Performance Characteristics
| Operation | Overhead | Notes |
|---|---|---|
| XDP packet processing | 10-50ns per packet | Before sk_buff allocation, near hardware speed |
| TC packet processing | 50-200ns per packet | After sk_buff, more context available |
| Kprobe/tracepoint | 50-100ns per event | Includes context switch to eBPF program |
| Map hash lookup | 20-50ns | O(1) average, kernel-optimized hash |
| Map per-CPU update | 5-15ns | No locking, cache-line local |
| Ring buffer submit | 30-80ns | Lock-free, single producer per CPU |
| Uprobe | 1-5ΞΌs per event | User/kernel boundary crossing, higher overhead |
| Tail call | ~10ns | Register save + indirect jump |
| JIT-compiled vs interpreted | 1-5% vs 10x overhead | JIT is always preferred, enabled by default |
How This Compares to Alternatives
| Feature | eBPF | Kernel Modules | SystemTap | ptrace | User-space Proxies |
|---|---|---|---|---|---|
| Safety | Verifier-guaranteed | None (can crash kernel) | Safe (uses kprobes) | Safe | Safe |
| Performance | Near-native (JIT) | Native | Good (kprobes) | Poor (50-100x overhead) | Network path overhead |
| Kernel recompile | No | No | No | No | No |
| Reboot required | No | No | No | No | No |
| Language | Restricted C/Rust | Full C | SystemTap script | N/A | Any |
| Root required | Yes (CAP_BPF) | Yes (root) | Yes (root) | Yes (CAP_SYS_PTRACE) | No |
| Networking hooks | XDP + TC (line rate) | Netfilter hooks | Limited | N/A | User-space proxy |
| Production use | Cilium, Falco, Pixie | Legacy drivers | Debugging only | Debugging only | Envoy, Nginx |
| Cross-version portability | CO-RE + BTF | Compile per kernel | Per kernel | Stable | N/A |
I reach for eBPF when I need kernel-level visibility or networking performance without the risks of kernel modules. I use SystemTap or bpftrace for quick one-off debugging sessions where I need a high-level scripting language. I fall back to user-space proxies (Envoy) when the team does not have eBPF expertise and the performance requirements do not justify kernel-level development.
The Tooling Ecosystem
Understanding the eBPF tooling layers helps you pick the right abstraction level for your use case.
| Tool | Layer | Use Case |
|---|---|---|
| bpftool | CLI utility | Inspect loaded programs, dump maps, view JIT output |
| bpftrace | High-level scripting | One-liner tracing scripts, quick debugging |
| BCC (BPF Compiler Collection) | Python/C library | Pre-built tracing tools (execsnoop, tcpconnect, etc.) |
| libbpf | Low-level C library | Production-grade program loading with CO-RE |
| Aya | Rust framework | Type-safe eBPF development in Rust |
| Cilium | Kubernetes CNI | Pod networking, service mesh, network policy |
| Falco | Security runtime | Container security monitoring and alerting |
| Pixie | Observability platform | Auto-instrumented application metrics |
Start with bpftrace, ship with libbpf
For development and debugging, use bpftrace. Its awk-like syntax lets you write one-liner
tracing programs: bpftrace -e 'tracepoint:syscalls:sys_enter_read { @[comm] = count(); }'.
For production deployment, compile with libbpf and CO-RE so your program runs on any
kernel version without recompilation. BCC was the standard but has higher runtime overhead
because it compiles programs at load time on the target machine.
Interview Cheat Sheet
- When asked what eBPF is: "eBPF lets you run sandboxed programs inside the Linux kernel without kernel modules. Programs are verified for safety at load time, JIT-compiled to native code, and attach to kernel hook points for networking, tracing, and security."
- When asked about safety: "The verifier statically analyzes every instruction path. It enforces bounded loops, valid memory access, and restricted helper function calls. A verified program cannot crash the kernel."
- When asked about performance: "JIT-compiled eBPF runs within 1-5% of native kernel code. XDP programs can process 10+ million packets per second because they run before the kernel allocates sk_buff structures."
- When asked about Cilium: "Cilium replaces iptables with eBPF hash maps for O(1) service lookup. This gives consistent latency regardless of cluster size, while iptables scales linearly with the number of rules."
- When asked about maps: "Maps are the shared data structures between kernel and user space. Hash maps for key-value lookup, per-CPU arrays for lock-free counters, ring buffers for high-throughput event streaming."
- When asked about kprobes vs tracepoints: "Tracepoints are stable kernel interfaces, safe across versions. Kprobes attach to any function but break when function signatures change. Always prefer tracepoints. Use CO-RE with BTF for kprobe portability."
- When asked about CO-RE: "Compile Once, Run Everywhere. Programs compiled with BTF (BPF Type Format) adapt to different kernel struct layouts at load time. libbpf rewrites field offsets so the same binary works across kernel versions."
- When asked about limits: "Programs have a 512-byte stack, 1 million verified instructions, and can nest up to 33 tail calls or 8 function calls deep. The verifier complexity limit is the most common production blocker."
- When asked why not kernel modules: "Kernel modules have no safety guarantees. A bug causes a kernel panic. eBPF programs are verified before loading, cannot access arbitrary memory, and can be loaded/unloaded without rebooting. The tradeoff is restricted expressiveness."
Test Your Understanding
Quick Recap
- eBPF runs sandboxed programs inside the Linux kernel by compiling restricted C to bytecode, verifying it for safety, and JIT-compiling to native machine code.
- The verifier guarantees program termination, memory safety, and restricted kernel access by statically analyzing every possible execution path.
- Programs attach to hook points (XDP, TC, kprobes, tracepoints, cgroup, LSM) and execute only when the corresponding kernel event fires.
- Maps (hash, array, ring buffer, per-CPU) provide shared storage between eBPF programs and user-space applications.
- XDP programs process packets before sk_buff allocation, achieving 10M+ packets/sec, which is 3-10x faster than iptables.
- CO-RE with BTF enables portable eBPF binaries that work across kernel versions without recompilation.
- Cilium (networking), Falco (security), and Pixie (observability) are the major production eBPF platforms in the Kubernetes ecosystem.
- The stack is limited to 512 bytes, the verifier analyzes up to 1M instructions, and tail calls nest up to 33 levels, which forces simple, efficient program design.
Related Concepts
- How Linux Containers Work - Understand namespaces and cgroups, which eBPF hooks into for per-container networking and security policy enforcement.
- How Prometheus Works - eBPF-based tools like Pixie provide metrics without the pull-based scraping model, complementing or replacing Prometheus for certain use cases.
- How Nginx Works - Compare user-space proxy networking (Nginx, Envoy) with kernel-space eBPF networking (Cilium) to understand when each approach is appropriate.