Agent overhead — methodology and measured numbers

What this is

Every observability vendor calls their agent "lightweight." This doc replaces the adjective with numbers, a reproducible method, and a CI tripwire so the claim is checkable and can't silently rot. It's the evidence behind the eBPF agent's overhead story.

What is measured, and where

The eBPF agent's cost splits into two halves, measured differently because they live in different worlds:

1. Userspace pipeline — Observe → ServiceMap → Drain → protobuf → bus publish: everything the agent does per flow event once the kernel has handed the event up. (Userspace is where ordinary programs run, outside the privileged kernel core of the OS.) This is pure Go and runs everywhere, so it's measured everywhere, CI included, by internal/ebpf/bench_test.go under a defined synthetic profile (50 destination peers × 8 ports, mixed ingress/egress, varied sizes — the same shape the recorded fixtures replay). It reports CPU time (via Getrusage — the OS's own per-process accounting of CPU seconds consumed), wall-clock throughput, and heap/RSS (heap is the memory the Go runtime manages for the program's data; RSS, resident set size, is the process's total physical-RAM footprint as the OS accounts it).

2. Kernel + ring buffer — the BPF programs and the drain of the ring buffer (the fixed-size kernel→userspace event queue; see ebpf-agent.md). This half needs a live kernel with real traffic, so it can't be priced in ordinary CI. Instead, CI proves the load/attach path works on real LTS kernels (the ebpf-kernel-matrix job), and the live overhead numbers are taken on reference hosts with the script below while driving a defined iperf3 / wrk profile (the standard load generators: bulk TCP streams and HTTP requests, respectively). (The reference-host rows are pending — see the table.)

The split matters because the two halves have very different cost profiles, and conflating them would let a cheap userspace number hide an expensive kernel one (or vice versa). They're measured separately and labeled separately.

Run the userspace half anywhere:

scripts/bench/agent_overhead.sh results.txt   # host context + benches + report

The regression tripwire (CI, every run)

TestAgentOverheadReport runs twice inside make test. The normal unit sweep runs it with Go's race detector, which preserves the full 200,000-event correctness exercise but does not treat instrumented wall time as product latency. After the concurrent package sweep finishes, make test-performance runs the same test alone without -race and fails the build if userspace pipeline throughput drops below 20,000 events/s.

That floor is deliberately loose — roughly 20–40× below the real numbers — because CI runners are shared and noisy. The separation is important: -race instruments every memory access and concurrent packages compete for the same host, so enforcing a wall clock there can reject an unchanged healthy tree. The point is not to measure performance precisely in CI; it is to catch a regression. The floor is a smoke alarm, not a thermometer: if the isolated plain pipeline suddenly does less than 20k/s, something got at least ~20× slower. A commit that makes the agent meaningfully heavier still cannot land unnoticed.

Measured numbers

These are a recorded run on one machine (Go 1.26, arm64 dev container, 4 vCPU, plain build). They are reproducible with the script above, but they are hardware-specific — treat them as a representative data point, not a guaranteed spec. The dated table below is the record; rerun on your own hardware to get your own row.

How to read this: at the flow rates a host actually sees (hundreds to a few thousand flows/s), the agent's userspace cost is a low single-digit percentage of one core and tens of MiB of memory. For comparison, the shipped Helm chart's default limits are 500m CPU and 256Mi memory — an order of magnitude of headroom above these measured figures.

The reference-host row is intentionally left for a human to fill: run the script on real hardware with live traffic and paste the row. That's also the only way to populate the live ring-buffer column the synthetic table marks n/a — which brings us to the live test.

Audit status: until a supported Linux host with BTF plus CAP_BPF and CAP_PERFMON produces an OVERHEAD ROW, the live ring-buffer reference result is blocked, not estimated. Do not convert the synthetic userspace row, a Docker Desktop run, or a skipped kernel test into a live-overhead claim.

Measuring the live ring-buffer path

TestLiveOverheadReport (internal/ebpf/live_smoke_ebpf_test.go, built with the linux and ebpf tags) measures the real kernel path the table above can't — the one marked n/a. It loads and attaches the BPF programs via newLiveSource, generates loopback TCP connects through the tracepoints for a configurable window (PROBECTL_OVERHEAD_SECONDS, default 10), drains the ring buffer, and prints an OVERHEAD ROW with CPU% (rusage user+sys over the window), heap, and max RSS. It skips cleanly when there's no kernel privilege, so it runs wherever the kernel-matrix smoke runs:

# on a reference Linux host (root, or CAP_BPF+CAP_PERFMON):
make ebpf-agent   # generates vmlinux.h, bpf2go bindings, and object digests

PROBECTL_OVERHEAD_SECONDS=60 go test -tags ebpf -count=1 -v \
  -run '^TestLiveOverheadReport$' ./internal/ebpf/

The make ebpf-agent step matters on a source checkout: the live Go files link against generated bpf2go bindings (l4flow_bpfel.go, sslsniff_*.go) and the digest manifest, and those generated files are intentionally not committed. Rows from Docker Desktop, CI-scale build checks, or a run that skipped because the host lacked BTF/capabilities do not populate the reference-host row. Only paste a row from a supported Linux host running the defined traffic mix.

Paste the logged row into the table above; the "Live ring-buffer events/s" column stops being n/a the first time this runs on real hardware.