48.9k
eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that runs sandboxed, event-driven programs inside the kernel, attached to hooks such as system calls, network events, and function entry points, with no kernel modules and no application code changes. Merged into Linux 3.18 in 2014, it powers zero-code observability, networking, and security tooling.
The name is historical. Classic BPF, described by Steven McCanne and Van Jacobson in their 1993 USENIX paper on the BSD Packet Filter, was the small packet-filtering virtual machine behind tcpdump. The extended version, led by Alexei Starovoitov and Daniel Borkmann, generalised it into a programmable runtime for the whole kernel. Today eBPF is stewarded by the eBPF Foundation, formed under the Linux Foundation in 2021 by Google, Meta, Microsoft, Netflix, and Isovalent. Production systems built on it range from Cilium, the CNCF-graduated Kubernetes networking project, to Meta's Katran load balancer and Android's network usage accounting.
How does eBPF work?
An eBPF program is compiled to bytecode, checked by the kernel's verifier for memory safety and guaranteed termination, JIT-compiled to native machine code, and attached to a hook. When the hooked event fires (a system call, a packet arrival, a function entry), the program runs and writes results into shared data structures called maps.
The verifier is what makes this production-safe where kernel modules are not. Before a program loads, the kernel statically analyses every execution path: pointer arithmetic is bounds-checked, loops must provably terminate (bounded loops were allowed from kernel 5.3), and programs are capped at 1 million verified instructions since kernel 5.2. A program that could crash or hang the kernel is rejected at load time; a buggy kernel module, by contrast, can panic the whole machine.
Maps and ring buffers carry data from kernel to user space, so a program can aggregate in the kernel (counting, bucketing into histograms, filtering) and export summaries instead of shipping every raw event across the boundary. BTF (BPF Type Format) and CO-RE ("compile once, run everywhere") let one compiled program run across kernel versions without recompilation.
What can eBPF attach to?
eBPF programs attach to hook points spanning the kernel and user space: dynamic probes on kernel and application functions, stable tracepoints, network-layer hooks, and timed sampling events. Each hook determines what the program observes and when it runs. Observability agents combine several hook types on one host:
- kprobes / kretprobes: dynamic probes on the entry and exit of nearly any kernel function.
- uprobes / uretprobes: probes on user-space functions, such as OpenSSL's
SSL_readandSSL_write, which is how agents see plaintext either side of TLS encryption. - Tracepoints: stable, versioned instrumentation points the kernel maintains for events like system calls and scheduler activity.
- XDP (eXpress Data Path): packet processing at the network driver, before the kernel networking stack even runs.
- tc hooks: traffic-control layer programs for packet classification and shaping.
- Socket and cgroup hooks: per-connection and per-container-group network events.
- perf events: timed sampling of stack traces, the basis of continuous profiling.
The flow is the same in every case: event fires → verified program runs → result lands in a map or ring buffer → a user-space agent reads it and exports telemetry.
How does eBPF enable zero-code observability?
eBPF observes applications from outside the process. Because probes sit on kernel functions, socket operations, and shared-library calls, an agent can reconstruct HTTP and gRPC requests, TCP flows, and CPU profiles for every process on a host, without SDKs, in-process agents, recompilation, or redeployment. One agent at the infrastructure layer covers every codebase running on the host.
The standards path for this is OpenTelemetry eBPF Instrumentation (OBI). Grafana Labs donated its Beyla auto-instrumentation agent to OpenTelemetry in May 2025, and the renamed project shipped its first release later that year. OBI emits standard OTLP traces and metrics, so anything downstream of an OpenTelemetry Collector consumes eBPF telemetry like any other source.
| Use case | What eBPF sees | What it can't see |
|---|---|---|
| Zero-code HTTP/gRPC tracing (OBI) | Method, route, status, latency per request, for any language including compiled Go/Rust binaries | Business context: user IDs, feature flags, custom attributes inside the app |
| Network flow monitoring | Every TCP/UDP connection, DNS query, retransmit, and queue depth per container | Application intent (why the connection was made) |
| Continuous profiling | CPU stack traces sampled across every process on the host | Wall-clock waits that never consume CPU (unless paired with off-CPU probes) |
| Security/runtime events | Process execs, file opens, privilege changes at the syscall layer | Logic-level abuse that looks like normal syscalls |
eBPF vs SDK instrumentation: which should you use?
Both. eBPF and OpenTelemetry SDKs are complementary. eBPF provides an immediate coverage floor: every service on a host is traced from day one, regardless of language or team. SDKs provide depth: custom spans, business attributes, and context that only exists inside the process. Mature setups run eBPF for breadth and add SDKs to the services that matter most.
Use eBPF-based instrumentation first when you need visibility across a large, polyglot estate quickly, when you can't modify vendor or legacy binaries, or when the runtime (Go, Rust, C++) makes agent injection awkward. Reach for SDKs when spans need domain attributes, when you want distributed traces with reliable cross-service context propagation, or when sampling decisions depend on application state. The pipeline ends identically either way (OTLP into a collector), which is why the two mix cleanly.
What are the limits of eBPF observability?
eBPF sees the kernel boundary, not your business logic. It observes what crosses syscalls, sockets, and probed library calls; it cannot read in-process variables, domain context, or intent. It is Linux-specific in practice, requires elevated privileges to load, and its overhead is low but never zero.
Common misconceptions:
- "eBPF is free." Every probe has a per-event cost: negligible on low-frequency hooks, measurable on hot paths like per-packet XDP or high-rate kprobes. Kernel-side aggregation via maps is what keeps well-built agents cheap.
- "eBPF replaces instrumentation." It replaces the transport-level portion. Custom spans and business attributes still require SDKs.
- "eBPF can't see encrypted traffic." Uprobes on TLS libraries capture plaintext before encryption, though statically linked binaries and Go's built-in TLS need runtime-specific probes.
- "It works everywhere." Windows support exists as a separate Microsoft project, but production observability tooling assumes recent Linux kernels with BTF support.
Storing and querying eBPF telemetry at scale with ClickStack
eBPF's defining property, telemetry for everything on the host at kernel-event granularity, makes it one of the highest-volume sources in an observability architecture. Shopify's eBPF-based network monitoring pipeline, built on ClickHouse, handles over 30 million flow, DNS, and networking events per second.
ClickStack, the ClickHouse observability stack, ingests OBI's output through its OpenTelemetry Collector distribution and stores it in ClickHouse tables; HyperDX, the ClickStack UI, sits on top for search and visualisation. At rest, an eBPF-derived span is just a wide row, and questions stay SQL-shaped:
-- p99 latency by route from zero-code eBPF spans, last hour
SELECT SpanAttributes['http.route'] AS route,
quantile(0.99)(Duration) AS p99
FROM otel_traces
WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY route
ORDER BY p99 DESCColumnar storage and compression make it economical to keep every kernel-level event, the same property that underpins eBPF-based network monitoring pipelines.
Frequently asked questions
What is the overhead of eBPF-based observability?
Published agent measurements typically land in the low single-digit percent CPU range, because eBPF programs run as JIT-compiled native code and aggregate inside the kernel, avoiding per-event context switches. Actual cost scales with hook frequency: sampling profilers are cheapest, per-request tracing moderate, per-packet processing the most sensitive.
Can eBPF capture distributed traces?
Partially. eBPF reconstructs spans for each service it observes, and OpenTelemetry eBPF Instrumentation can propagate W3C Trace Context across services in supported protocols. Cross-service propagation is the hard part, since encrypted or unusual transports can break stitching, so teams needing guaranteed end-to-end traces still add SDK instrumentation on critical paths.
How do I get started with eBPF profiling?
Start with bpftrace one-liners or the bcc toolkit to sample stack traces on a single host, then move to a continuous profiler (the OpenTelemetry eBPF profiler or Parca) for fleet-wide, always-on coverage. Both sample perf events at fixed frequency, so profiling every process typically costs around one percent CPU.
How does eBPF enable zero-code instrumentation?
Probes attach to kernel functions, socket operations, and shared-library calls while the target process is already running, so an agent captures its requests, connections, and latencies without an SDK, a rebuild, or a restart. This works even for compiled Go and Rust binaries that resist in-process agent injection, and OpenTelemetry eBPF Instrumentation exports the result as standard OTLP traces and metrics.
Try ClickStack to see what kernel-level telemetry looks like at query time: point an OpenTelemetry Collector at it and the eBPF data you already generate becomes SQL-queryable.
