The probectl eBPF agent — security whitepaper

The standalone document a buyer's security team reviews before privileged code runs on their hosts. Every claim here is enforced by code or a CI gate, cited inline so you can check it yourself; the deeper, system-wide view is threat-model.md.

1. What it is, in one paragraph

A single static Go binary that loads CO-RE eBPF programs to observe L3/L4 flows (and, only with explicit consent, TLS-plaintext L7 metadata) on a host, aggregates them in userspace, and publishes tenant-tagged batches to the operator's own Kafka over TLS. ("CO-RE" — Compile Once, Run Everywhere — means the programs relocate themselves against whatever kernel they land on, so there is no per-kernel build to trust.) The agent enforces nothing, captures payloads nowhere by default, never fetches code, and runs with two Linux capabilities.

Five "nevers", each proven somewhere below:

• Never enforces — physically cannot block, drop, or redirect traffic (§3).
• Never captures payloads by default — bodies are zeroed even under consented L7 capture (§4).
• Never fetches or executes code — no self-update channel exists (§8).
• Never speaks in the clear — plaintext transport is refused, not merely discouraged (§7).
• Never emits as another tenant — identity is bound to exactly one tenant (§7).

2. Privilege posture — exact, declared, enforced

A capability is a fine-grained slice of root's power. The agent takes the two it needs to load and attach observation programs, and nothing else.

3. Observe-only — a proof, not a promise

The strongest claim in this document is that the agent physically cannot block, drop, or redirect traffic — it can only watch. Two independent mechanisms back that up:

• A static gate on program type. eBPF programs come in types. Some can only observe (tracepoint, uprobe, ring buffer); others can act on packets (XDP, tc/qdisc, cgroup enforcement, kprobe-writes). A CI gate (internal/ebpf/observeonly_test.go) statically refuses any policy-capable type — only observation types may exist in the tree. An enforcement program cannot be merged.
• Load-and-attach on real kernels. The ebpf-kernel-matrix CI job loads and attaches every program on real LTS kernels (5.15 and 6.6) under QEMU on every pass — so the shipped bytecode is exactly what was reviewed, proven loadable, and proven incapable of enforcement.

This is a hard product guardrail: detection is a signal, never an inline IPS, and nothing probectl ships takes autonomous action on a network.

Object integrity. A SHA-256 manifest of the compiled eBPF objects is baked into the binary at build time. Before the kernel ever sees a program, the loader verifies the embedded bytes against that manifest and refuses on any mismatch or missing entry (internal/ebpf/integrity.go). A tampered object never loads.

4. Data categories — what is captured, and what never is

The Go-TLS limitation is disclosed, not hidden: crypto/tls does not use libssl, so Go processes are outside L7 capture today (docs/ebpf-feasibility.md §7).

5. Kernel compatibility

Memory locking: BPF maps/ring buffer need RLIMIT_MEMLOCK (the unit and chart set it); kernels ≥ 5.11 account via memcg.

6. Resource bounds and measured overhead

Defaults (Helm chart): requests 50m CPU / 64Mi, limits 500m / 256Mi. When the kernel's ring buffer fills under load, the dropped flows are counted and exported, never silently discarded — a dropped flow is a correctness gap in an observability tool, so it must be visible.

Measured numbers (methodology and the CI tripwire are in agent-overhead.md) — userspace pipeline, on the defined 50-peer × 8-port profile:

A throughput floor runs in every make test, so a regression against the "lightweight" claim fails CI. One row is still open: the live, on-host ring-buffer overhead on reference hardware is measured in the userspace pipeline but awaits a human-scheduled reference-hardware run (tracked in the same doc).

7. Identity, transport, and tenancy

The agent's identity is bound to exactly one tenant, and it cannot speak in the clear. Two transports are involved:

• The canary / enterprise agent speaks gRPC over mTLS with a SPIFFE-style, tenant-bound identity and a mandatory trust-domain pin — a certificate from the wrong trust domain is rejected at the handshake.
• The eBPF agent publishes to Kafka with TLS required: plaintext is refused unless a dev-only override is set explicitly, enforced both at runtime (internal/bus/security.go) and at Helm chart render time.

Every emitted record carries the agent's single bound tenant, so an agent physically cannot emit data as another tenant.

How the mTLS agent gets that identity (first-boot bootstrap). Enrollment is deliberately boring and fail-closed (internal/enroll, internal/agent/identity.go):

1. The operator generates the agent CA hierarchy once (probectl-control agent-ca init) and distributes only the public trust bundle to hosts (probectl-control agent-ca export) — never a key.
2. The operator mints a single-use, tenant-scoped join token (probectl-control enroll-token; short-lived, stored server-side only as a hash). The token names the tenant — an agent can never choose its own.
3. The agent boots with PROBECTL_AGENT_JOIN_TOKEN (or enroll.token_file, e.g. a mounted Secret), generates its keypair locally, and submits a CSR — the private key never leaves the host. The server dictates every certificate field (SAN/EKU/TTL; CSR-requested extensions are ignored) and issues a short-lived (24 h) SPIFFE identity; renewal happens by rotation against proof of the current identity and can never change who the agent is.

The flow is idempotent — an existing identity is never re-enrolled or overwritten — and fail-closed: an expired, replayed, or unknown token is a fatal startup error (all invalid tokens are deliberately indistinguishable to the caller), and with no token and no identity the agent simply cannot authenticate — there is no unauthenticated fallback at any point.

When the control plane is unreachable, the canary agent spools results into a disk-backed, bounded, FIFO store-and-forward buffer (internal/agent/buffer.go). At capacity the newest result is rejected with a counted error rather than growing without bound — and nothing is ever sent in the clear to compensate.

8. Updates and signing — deliberately boring

There is no self-update channel. The agent never fetches or executes code — removing the single most dangerous capability a fleet agent can have. Upgrades are operator actions through an external orchestrator (Helm / install.sh / config management), rolled out in staged waves with registry verification and halt-on-error (see fleet-rollout.md). The rollout planner refuses any artifact without a recorded cosign signature verification (verify commands in verify-artifacts.md). Releases are cosign-signed (keyless) with SPDX SBOMs and will not cut from a red CI run.

9. Install artifacts

• Kubernetes: the deploy/helm/probectl-agent DaemonSet chart. The full privilege contract is declared in the artifact itself and CI-gated (lint + hardening assertions + kubeconform), with fail-closed rendering — no tenant configured → refuse; plaintext bus without the explicit dev flag → refuse.
• VM / bare metal: deploy/agent/install.sh — installs the local binary, a dedicated non-root system user, the hardened systemd unit, and a fail-closed sample config.

Both paths are air-gap friendly: neither downloads anything at install time.

10. Review pointers

The eBPF programs live in internal/ebpf/ (the C is in bpf/, about 270 lines), the CI gates named throughout this doc are in .github/workflows/ci.yml, and the drills and benchmarks are listed under make help. Report vulnerabilities via SECURITY.md.