eBPF host agent

What it is

The probectl-ebpf-agent watches network activity from inside the host's kernel — no sidecars, no app changes, no SDK to import. It gives you two things with zero instrumentation:

• L3/L4 flow capture — every TCP connection a host makes or accepts, with the process and container behind it; and
• a live service map — the directed graph of "who talks to whom" built from those flows.

This is the shared host/L4 substrate the security, segmentation, and cost planes later build on. The single most important thing to understand about it: it is observe-only. It loads only observation programs and never enforcement — it watches, it does not block, redirect, or modify a single packet. Think of a court stenographer: admitted into the room to record every word, never permitted to speak. probectl's eBPF layer is not a CNI (the Kubernetes networking layer that routes pod traffic) and not an inline IPS (an intrusion-prevention system that drops traffic it dislikes). (This is a hard guardrail, and a build-failing test enforces it — see below.)

New to eBPF? The kernel is the privileged core of the operating system — every packet, socket, and process on the host passes through it; userspace is everything outside it, where ordinary programs run. eBPF lets you load tiny, sandboxed programs into the running Linux kernel that fire on kernel events (a socket changing state, a function being called) and report what they saw back to userspace. It's how modern tools see all network activity on a box without touching the applications — the kernel already sees everything, so a program admitted into the kernel sees it too. The kernel verifies these programs can't crash or hang it before letting them run.

How it works — the path of a flow

Here's the whole pipeline, from a packet event in the kernel to a record on the bus:

%%{init: {'theme':'base','themeVariables':{'background':'#0d1117','primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#3b82f6','lineColor':'#8b949e','secondaryColor':'#21262d','tertiaryColor':'#0d1117','clusterBkg':'#161b22','clusterBorder':'#30363d','fontFamily':'ui-monospace, SFMono-Regular, Menlo, monospace'},'flowchart':{'curve':'basis','nodeSpacing':55,'rankSpacing':55,'padding':12}}}%%
flowchart LR
  subgraph kernel["kernel (only with -tags ebpf)"]
    TP["raw tracepoint sock/inet_sock_set_state"] --> PROG["CO-RE eBPF prog"] --> RB["ring buffer"]
  end
  RB --> LIVE["liveSource (cilium/ebpf)"]
  FIX["FixtureSource (recorded JSON)"] --> AGG
  LIVE --> AGG["Aggregator"]
  AGG --> ENR["enrich: process / cgroup / container"]
  ENR --> MAP["ServiceMap (directed edges)"]
  MAP --> EMIT["BusEmitter -> probectl.ebpf.flows (FlowBatch, OTel-shaped, tenant-keyed)"]
  AGG -. drops .-> DROPS["dropped_total metric"]

1. Kernel hook. A CO-RE eBPF program (CO-RE, "Compile Once – Run Everywhere", is the portability mechanism — see Kernel compatibility) attaches to the stable sock:inet_sock_set_state tracepoint — a named hook point the kernel itself exposes, with a documented argument layout the kernel promises not to break. It attaches as a raw tracepoint: by name, through the bpf() system call itself, without reading tracefs (ING-23, see Privileges). Every time a TCP socket changes state, the program runs. It keeps the ones entering ESTABLISHED and writes the 5-tuple (source IP + port, destination IP + port, protocol — the values that uniquely name a connection) plus the PID and command name into a ring buffer — a fixed-size queue shared between kernel and userspace, a conveyor belt of bounded length: the kernel places events on one end, userspace lifts them off the other, and when userspace falls behind, new events fall off and are counted, never silently lost (internal/ebpf/bpf/l4flow.bpf.c). A raw tracepoint hands the program the tracepoint's arguments (the socket, its old and new state) rather than the formatted record, so the program reads the tuple from the socket itself — the same fields the tracepoint builds its record from — and CO-RE re-aims those reads at each kernel's struct layout. The stable hook point is what stays fixed; the event format the agent publishes is unchanged.

   Which end is the client, and whose process is it (ING-24). The kernel always reports a socket's own address first, so a server's socket reads "server → client" — and the server-side transitions run in softirq context (the kernel handling a packet on whatever CPU it landed, on behalf of no particular process), where "the current process" is a bystander, often swapper. So the program learns the role and the owner where they are knowable and caches them per socket: connect() (→ SYN_SENT) marks a client and records its caller; listen() (→ LISTEN) records which process serves a port, and a socket born from it (LISTEN → SYN_RECV) is a server owned by that process. A BPF iterator (a read-only walk over kernel objects) runs once at start and records the listeners that already existed, so a server started before the agent is attributed too. It records MPTCP listeners as well as TCP ones: Go 1.24+ listens with Multipath TCP by default wherever the kernel enables it (the 5.15/6.6 matrix kernels and most distro kernels do), and so does any binary run under mptcpize. Such a server holds an MPTCP socket (sk_protocol = IPPROTO_MPTCP, 262) while the TCP socket that accepts is an in-kernel subflow nobody holds a descriptor for; both bind the same port in the same network namespace, which is all the listener record is keyed on. A TCP-only filter dropped every such server, which is why pre-existing Go servers went unattributed on 5.15 and 6.6 while a kernel without CONFIG_MPTCP passed (ING-24). The agent's start log says plainly how listeners were seeded: ebpf pre-existing listeners seeded with method="bpf task_file iterator" and the per-stage counts (files_visited, socket_files, tcp_listeners, owners_recorded), or, at WARN, ebpf pre-existing listeners NOT seeded with method=none and the reason — a kernel that refuses the iterator (it needs bpf_sock_from_file, kernel 5.11+), an iterator that failed, or one that read to the end without ever being handed a file. Only an iterator that actually walked open files counts as seeded. When nothing was seeded, a server started before the agent is attributed only when one of its connections closes (close() runs in the owner's context); servers started after the agent are unaffected. Userspace then orients every flow client → server: an inbound connection becomes one edge keyed on the listening port with the server's process on the destination (network.io.direction = ingress), never one edge per ephemeral client port. An owner the agent could not learn is reported as unknown, not guessed from the softirq context. When both ends of a connection run on the same host the agent sees both sockets and joins them into a single client-process → server-process edge (both raw flows are still published).

2. Userspace read. The Go liveSource (built on cilium/ebpf) drains the ring buffer.

3. Aggregate. The Aggregator folds raw connection events into directed service edges ("host A → host B:443, N connections").

4. Enrich. Each flow is tagged with its process, cgroup, container, and local Kubernetes pod identity when the cgroup path carries a pod UID — the read-only /proc lookups that turn a bare PID into "the nginx container in pod 12345678-1234-…".

5. Emit. The BusEmitter marshals a batch to protobuf and publishes it to probectl.ebpf.flows, keyed by tenant.

Userspace core + a gated kernel loader

The agent is split in two on purpose, so the bulk of it runs and is tested anywhere — kernel or not:

• A pure-Go userspace core does the flow/service-edge model, the aggregator, process/cgroup enrichment, the capability probe, the OTel mapping, and the bus emitter. It drives a pluggable flow Source.
• The live Source — the CO-RE eBPF program loaded into a ring buffer — is compiled in only under the -tags ebpf build tag. Every other build uses the FixtureSource (recorded flows replayed from JSON), which is also the no-kernel CI path.

Why split it this way? The build host needs clang (the C compiler that produces BPF bytecode); the target host needs only a BTF-capable kernel (BTF, the BPF Type Format, is the kernel's machine-readable index of its own data-structure layouts — what CO-RE relocates against) and CAP_BPF (a Linux capability: one entry in the kernel's fine-grained permission list, granting exactly the right to load BPF programs rather than all of root). And most CI runners and macOS laptops can't load eBPF at all. By making the -tags ebpf files a separate, off-by- default compilation unit, the default make build and ordinary CI need no eBPF toolchain and no extra dependency — yet the shipped agent image is still the live build (see Building).

L7 visibility — application calls, including over TLS

Beyond raw connections, the agent can parse application-protocol calls — HTTP/1.1, HTTP/2, gRPC, DNS, Kafka, PostgreSQL, and MySQL — and roll per-call method / resource / status / latency onto each service edge. (L7 is layer 7 of the network stack, the application layer: not "host A talked to host B" but what they said — the HTTP request, the DNS query, the Kafka produce.) Each call is emitted as an L7Call plus an l7_* rollup on the ServiceEdge. Parsing is pure Go and kernel- independent (internal/ebpf/l7), driven by the live capture layer in production and by an L7 fixture (PROBECTL_EBPF_L7_FIXTURE_PATH) in CI and demos.

probectl gets the plaintext two ways:

• Cleartext traffic: parsed straight from socket reads/writes.
• TLS traffic: the TLS plaintext — the readable bytes that exist inside the application just before encryption and just after decryption — is captured via uprobes (user-space probes: breakpoint-like hooks placed on a named function in an ordinary program or library) on supported TLS library read/write functions (SSL_write / SSL_read for OpenSSL-compatible stacks and gnutls_record_send / gnutls_record_recv for GnuTLS). This reads the letter over the writer's shoulder before it's sealed, and over the reader's after it's opened — it never steams open the envelope in transit, so there is no CA (no certificate authority to forge the server's identity with) and no man-in-the-middle (no interception point inserted into the network path). Read calls are captured at the return uprobe, because the destination buffer is only filled when the call returns.

The OTel mapping (internal/otel.L7CallAttributes) emits http.* / rpc.* / dns.* / messaging.* / db.* attributes per protocol. SQL query text is normalized before emission: string literals, quoted identifiers, and numeric literals become ?, so the default L7 path carries operation shape (SELECT, INSERT, table/query skeleton, status, latency) rather than raw customer data. Calls are attributed to the connection's client→server edge regardless of which direction completed them.

TLS posture is a separate privacy-minimized projection. L7Call can carry version, cipher, server name, peer-certificate DER, verification state, capture source, timestamp, confidence, and an explicit visibility state. The posture consumer copies only those handshake/certificate fields—never method, resource, header, or body—and validates the complete tenant-bound batch before mutating the inventory. Live uprobes that only prove encrypted plaintext visibility emit encrypted_unknown; sidecars emit sidecar_unknown; neither is guessed into a certificate conclusion. See TLS observability and the derived-only raw-history decision.

Kubernetes identity without API custody

The default Kubernetes chart deliberately sets automountServiceAccountToken: false; the eBPF agent does not watch the Kubernetes API by default and does not need cluster-reader credentials to build a service map. Instead, the proc enricher reads /proc/<pid>/cgroup and extracts:

• the CRI container id (docker-, cri-containerd-, crio-, libpod- shapes);
• the Kubernetes pod UID from common kubepods...pod<uid>... cgroup paths; and
• the process name from /proc/<pid>/comm.

The emitted workload identity is therefore stable and local, for example k8s-pod:123456781234/checkout@cccccccccccc. That is enough to correlate flows to pod-level Kubernetes identity without adding a new control-plane data source. Namespace, workload owner, and label enrichment require an explicit future Kubernetes metadata source or operator-supplied mapping; they are not claimed by the default chart today.

Reading TLS plaintext is off by default and triple-gated

Reading application plaintext on a customer's host is PII-class, so live TLS-plaintext capture (the "sslsniff" path) is off by default and requires three explicit, independent statements before a single byte is captured (see internal/ebpf/l7policy.go):

1. l7_capture_enabled: true — the master switch.
2. l7_capture_consent_tenant must equal the agent's bound tenant exactly — an explicit, per-tenant consent. (The agent is tenant-bound at registration, so this is a deliberate statement in this tenant's deployment config; absent or mismatched, capture stays off.)
3. l7_capture_scope — a non-empty workload allowlist: entries of the form pid:<n>, exe:/abs/path, or cgroup:/abs/cgroup-dir. Container/pod scoping is the cgroup: form (a container is a cgroup). An empty scope means capture refuses to start — host-wide capture is not expressible.

The allowlist is enforced in the kernel. Uprobes on shared TLS libraries fire for every process that maps them, so the BPF program checks the in-kernel scope_tgids / scope_cgroups maps and drops a non-allowlisted process before copying a byte — that process's plaintext never enters the ring buffer at all. exe: entries are re-resolved against /proc every 10 seconds, so restarts and new workers of an opted-in binary stay in scope. They need CAP_SYS_PTRACE (root, or the Helm consented path); an agent without it refuses L7 when the scope has an exe: entry (D-63).

Redaction is layered, and defaults closed

Even for an allowlisted workload, payload bodies are redacted by default (internal/ebpf/l7policy.go). Redaction happens at two boundaries:

• The kernel capture window (l7_capture_kernel_window, default 1024 bytes) bounds how much plaintext per chunk may transit the ring buffer at all — a mail slot cut to a fixed height: only the first N bytes fit through, and what doesn't fit isn't shredded after delivery — body bytes past the window never leave kernel space. The BPF policy map's zero default is length-only, so an unprogrammed kernel ships no plaintext: it fails closed.
• At the ring-buffer → userspace boundary, on the only surviving copy, payload bodies are zeroed in place. Under l7_capture_redaction: metadata (the default, D-43) the request line and the two framing header values (Content-Length, Transfer-Encoding) survive for the in-memory parse; every other header value and the body are killed. Under headers the header values that are not credential- or identity-like survive too. The boundary remembers, per direction of each HTTP/1 connection, whether a message's headers have ended (DPR-340): a chunk that continues a message past its headers is body and is zeroed whole, even when it holds bytes that look like a header terminator. A chunk on a stream the agent is not tracking (a non-HTTP/1 protocol, or more than 8192 live HTTP/1 streams) is judged on its own: the metadata up to the header terminator, or the first 128 bytes, survives. (A consequence: HTTP/2 / gRPC call extraction is degraded under headers redaction, by design — the HPACK frames, HTTP/2's compressed header encoding, live in the zeroed region.)

The four redaction modes:

Parser byte-counts reflect the captured window; DataEvent.Size always carries the true chunk size, so loss-to-redaction is visible rather than silent.

Header secrecy vs. header metadata (EBPF-006). Under the headers opt-in (and under the metadata default, which zeroes every non-framing header value), the values of credential-bearing headers (Authorization, Cookie, Set-Cookie, Proxy-Authorization) and common non-standard secret families (X-API-Key, Api-Key, X-Amz-Security-Token, custom *Token*, *Secret*, and *Credential* headers) are zeroed in place. Non-secret identity values are denied too: built-ins cover headers like X-User-ID, X-Email, X-Subject, X-Employee-ID, X-Account-ID, X-Customer-ID, X-Session-User, and X-Person, and operators can add local fragments with PROBECTL_EBPF_L7_IDENTITY_HEADER_FRAGMENTS (or l7_capture_identity_header_fragments). Only the header names and line framing survive, so bearer tokens, API keys, session cookies, user ids, email addresses, and account identifiers never reach the control plane in default mode. PROBECTL_EBPF_L7_HASH_ALL_HEADER_VALUES=true can hash every remaining non-denied header value for correlation; denied secret and identity values are still zeroed, not hashed. If you need all header secrecy, choose length mode, which captures no payload bytes at all. These defaults are enforced by TestRedactPayloadZeroesSensitiveHeaderValues, TestRedactPayloadZeroesNonStandardSecretHeaders, TestRedactPayloadZeroesIdentityHeaderValues, and TestRedactPayloadHashAllHeaderValues.

Capture limitations (measured, not hidden)

Two real blind spots exist today. They are documented and counted or bounded, not papered over:

• Unsupported L3 families: l4flow captures IPv4 and IPv6 TCP sockets. Other address families are filtered in the kernel and counted through the legacy filtered_non_ipv4_total flush field, so a host with unsupported families shows a rising counter instead of silent gaps.
• Go's crypto/tls: the L7 capture uprobes attach to the system TLS libraries (OpenSSL / BoringSSL / GnuTLS). Go programs don't use libssl — they ship their own TLS — so a Go process's L7 plaintext is not captured. This is a known gap (detailed in ebpf-feasibility.md): the established workaround disassembles the Go binary for RET offsets and tracks the goroutine ABI, a meaningfully more brittle path. That module is explicitly post-GA / out of scope for GA; do not count Go crypto/tls plaintext as delivered L7 coverage. L4 flow and the service map still see Go processes fine — only their L7 plaintext is out of scope.

TLS-library uprobe coverage

Two limits carry over from the feasibility study: stripped or statically-linked binaries break uprobe symbol resolution (the agent falls back to socket cleartext for those), and Go-encrypted traffic needs its own capture path.

Where the agent looks for those libraries (l7_capture_host_root)

A uprobe attaches to an inode, so the agent has to open the very library file its target uses. Discovery walks the ldconfig cache and then the per-arch/per-distro candidate paths (libssl.so.3/.1.1, libgnutls.so.30/.28) — all of it inside the agent's own mount namespace.

That is fine for a host-installed agent and useless for the DaemonSet, whose image is distroless: it carries no libssl and no ldconfig at all, so discovery found nothing and consented capture came up with l7:false on every Kubernetes node (DPR-124). l7_capture_host_root names an in-pod mount point for the node's library tree, and every candidate path is resolved under it:

The chart mounts l7Capture.hostLibraryPaths (default /usr/lib, /lib) read-only under that root, and only when l7Capture.enabled — an agent without capture consent gets no view of the node filesystem. Trim the list on a node that lacks one of those directories: each is mounted type: Directory, so a missing path leaves the pod unscheduled rather than attaching to nothing. PROBECTL_EBPF_LIBSSL still overrides discovery and is taken literally — under a host root, name the path as the pod sees it (/host/...).

Because the probe attaches to the inode it resolved, this instruments processes that use that library file: the node's own binaries, and containers that share the node's library tree. A container carrying its own libssl in its image is a different inode and is not covered by the node-rooted attach; scope such a workload with cgroup:/pid: and give the agent that image's library path. A failed resolution is always a loud WARN plus l7_attach_failures — never a silent gap.

The execute-bit limit (DPR-125)

The uprobe loader (cilium/ebpf link.OpenExecutable) refuses a target with no execute bit, and the two library families split on exactly that:

Discovery therefore checks attachability, not just presence: an unattachable candidate is skipped and the search continues, so a node carrying both an unattachable and an attachable copy uses the attachable one. When nothing attachable is found, the agent names the mode, the path and this limit instead of claiming the library is missing. Everything downstream of the attach — consent, scoping, redaction, the kernel window, l7_calls — is verified working against an attachable library; on a Debian-family node treat L7/TLS plaintext as unavailable until the loader accepts a 0644 target. Do not chmod a package-managed library to work around it: the mode belongs to the inode, so that edits the distro's file for every process on the node. L4 flow, the service map and TLS metadata are unaffected.

Privileges and the observe-only guarantee

Loading the programs needs CAP_BPF + CAP_PERFMON (Linux ≥ 5.8) — the load right and the attach right respectively: attaching tracepoints and uprobes goes through the kernel's performance-monitoring interface, which CAP_PERFMON governs. Generic older kernels are unsupported because the agent requires BTF and the BPF ring buffer; do not grant broad CAP_SYS_ADMIN as a workaround for a kernel that the runtime probe marks unavailable. A CAP_SYS_ADMIN legacy exception is only for hosts where the runtime probe can still confirm BTF + ring buffer support but the platform cannot grant the split capability pair. The capability probe and the enrichment lookups are read-only and need no privileges. The L4 flow plane runs non-root and needs no tracefs (ING-23). The shipped systemd units run the agent as a dedicated non-root user with exactly AmbientCapabilities=CAP_BPF CAP_PERFMON. A classic tracepoint link would read the event id from tracefs (/sys/kernel/tracing), which systemd distros mount root-only (0700), so that user could not attach. The L4 program therefore attaches as a raw tracepoint, which needs only bpf() and the two capabilities, and the listener-seeding iterator needs nothing more either. The capability probe still reports tracefs= but no longer refuses a host where it is invisible. The kernel-matrix job proves this on every row: it runs the L4 attach as uid 65534 with only those two capabilities and no reachable tracefs, and observes a real TCP connection (TestLiveNonRootL4Attach).

L7 non-root is not covered by that change. The TLS uprobes and their tracefs fallback are a separate question (ING-23's L7 half, decision D-63). Two parts are settled. First, the non-root unit's L7 scope is pid: and cgroup: entries only: resolving an exe: entry reads /proc/<pid>/exe, which is ptrace-gated, so an agent without CAP_SYS_PTRACE refuses L7 at start when the scope contains one, and says to use pid: or cgroup: instead. The rest of the agent keeps running. There is no CAP_SYS_PTRACE drop-in for the unit; root and the Helm consented path (D-02a, which adds SYS_PTRACE) keep exe:. Second, on kernels from 6.16 the uprobe perf PMU requires CAP_SYS_ADMIN, so non-root L7 attach there depends on uprobe_multi links (6.6+). That attach path is not built yet, so treat consented L7 capture under the non-root unit as unsupported until it is. The Helm chart mounts the node's tracefs (tracefs.hostPath) only when l7Capture.enabled, writable, for the TLS uprobe registration (DPR-125); an agent with capture off gets no tracefs mount (DPR-345).

The observe-only guarantee is enforced in code, not just by convention. The agent attaches only tracepoints / kprobes / uprobes and calls no traffic-altering helper. A guard test (observeonly_test.go) parses the eBPF C sources and fails the build if anyone introduces an enforcing program type or a mutating helper (bpf_redirect, bpf_override_return, bpf_probe_write_user, packet-rewrite helpers, …). The test runs in the default build — no kernel, no clang needed — so the invariant is checked on every commit.

Emission, OTel, and tenancy

Flow and service-edge batches are published to probectl.ebpf.flows as an ebpfv1.FlowBatch protobuf, keyed by tenant (pooled tagging). Field names follow OpenTelemetry source.* / destination.* / network.* / process.* / container.* conventions from the first emission, so the OTLP layer exposes them rather than remapping. internal/otel.FlowAttributes is the canonical mapping and is held to the same "no invented attribute names" conformance bar as results.

One flush, bounded records (DPR-071). The service map is a cumulative snapshot: every flush carries every live edge, so on a busy host one record grows with the map. Kafka rejects a record above its 1,000,012-byte client batch limit (the broker's message.max.bytes defaults to 1 MiB) — and it does so asynchronously, after Publish has already returned nil. The design-partner lab hit exactly this: the first 22 batches landed (285 KB → 981 KB), the 23rd crossed the limit, and for the next three hours the agent logged ebpf flows emitted with dropped_total=0 while nothing reached the bus and every downstream view (topology, compliance, NDR) went quietly blind. Two changes close that: a flush larger than max_batch_bytes (default 768 KiB, PROBECTL_EBPF_MAX_BATCH_BYTES, chart maxBatchBytes) is split into several FlowBatch records that together carry exactly the flush — edges packed first, flows dealt across those records so none is edges-only (the topology view cannot tenant-verify an edges-only batch) — and after every emit the agent flushes the producer and reads its failure counters before it says "emitted".

Self-observation (drops are never silent)

A dropped flow is a correctness gap in an observability tool, so it is never hidden. Ring-buffer backpressure is counted at the kernel branch that loses the record: the L4 and TLS programs increment per-CPU drop_counters when bpf_ringbuf_reserve fails, and TLS capture also counts active_reads stash failures that would lose SSL_read correlation before userspace can see a chunk. The live sources sum those counters on flush and fold them into dropped_total.

The flush log carries both the roll-up and the reason labels: drop_decode_failures_total, drop_l4_ring_buffer_full_total, drop_l7_ring_buffer_full_total, drop_l7_active_read_failures_total, and drop_other_total, plus observed_total, l7_total, l7_scope_sync_failures_total, l7_attach_failures, l7_evicted_total, service_map_evicted_total, l7_manager_evicted_total, and the legacy filtered_non_ipv4_total unsupported-family counter. If only counters changed and there is no flow batch to emit, the agent still logs ebpf counters updated so an all-dropped window is visible — probectl observes probectl.

Delivery is observed the same way (DPR-071): every ebpf flows emitted line carries publish_batches_total (flushes the bus acknowledged), publish_failures_total (records the bus reported undelivered, plus flushes it never acknowledged within the bounded wait — half the flush interval, at most 5 s) and publish_degraded (the most recent flush did not deliver). A failure is logged at ERROR with the bus's own reason — ebpf publish FAILED … MESSAGE_TOO_LARGE, ebpf publish NOT acknowledged … context deadline exceeded — and the readiness probe reports not-ready until a later flush delivers cleanly, so kubectl get pods shows 0/1 READY on a host whose flows are not arriving instead of a healthy agent publishing into the void.

The metrics endpoint

metrics.enabled serves a tenant-labeled Prometheus endpoint. These gauges say whether the agent is delivering rather than merely running:

Alert on probectl_ebpf_l7_capture_requested == 1 and probectl_ebpf_l7_capture_active == 0. That is a consented, scoped capture delivering nothing — the state every attach limit above lands in — and until DPR-128 it was visible only as one WARN at startup, with the pod Ready and L4 flows emitting normally. A fixture replay never sets _active: recorded L7 is not encrypted-traffic visibility.

The tenant's redaction floor (D-43)

Redaction is enforced at the source, with the control plane's ingest strip as the second layer. Agents publish to the bus before anything reaches a store, and Kafka and NATS JetStream retain messages, so the agent itself must never emit looser data than its tenant allows.

• The floor. Each tenant has an L7 capture redaction floor — the loosest mode its agents may capture under. With no statement it is metadata. A tenant admin (governance.write) can opt the tenant into headers or full through the core tenant setting PUT /v1/ebpf/l7-capture-policy ({"redaction":"headers"}; CLI probectl ebpf set-l7-policy), audited OLD/NEW on the tenant's chain as ebpf.l7_capture_policy_set. It is a core feature on every edition and works with no licence (D-64); the ee/ governance policy and the provider plane never write it. redaction is required, so an update never resets an opt-in by omission. The regulated deployment profile refuses both opt-ins and never delivers a floor looser than metadata; GET shows the stored statement and the effective floor.
• Delivery. The agent fetches the floor over the existing agent gRPC service (FetchL7Policy) on the control plane's agent listener, over mTLS: set control_plane.grpc_addr, cert_file, key_file and ca_file (the identity files — cert.pem, key.pem, server-ca.pem; the certificate is the collector's own eBPF-plane SVID, spiffe://probectl/tenant/<t>/ebpf/<agent_id>, issued by POST /v1/collectors/register with plane=ebpf and a csr_pem — the listener admits it to the two L7 policy calls only, never as a synthetic agent, D-43 S9). The tenant the control plane answers for is the one in the certificate, never anything the agent sends — and the agent refuses to use a channel whose certificate names another tenant or another agent than the tenant_id and agent_id it publishes as (D-43 S3; agent_id is required with the channel). The floor is re-read every control_plane.policy_interval (5 minutes).
• Enforcement. The agent refuses to start capture when its local l7_capture_redaction is looser than the floor (ERROR log, posture refused_floor, probectl_ebpf_l7_capture_floor_refused = 1). It never silently tightens a stated mode under a received floor — an operator who asked for headers must see that the tenant does not allow it.
• The channel is required (D-43 S1). Consented live capture runs only with the control-plane channel configured, and starts only after the control plane has recorded the posture it is about to run under — so no capture runs that the tenant's audit chain does not show. Without a channel the agent refuses (ERROR log, probectl_ebpf_l7_capture_requested = 1 with _active = 0, probectl_ebpf_l7_capture_channel_refused = 1), and the chart refuses to render l7Capture.enabled without controlPlane.grpcAddr. A channel identity the agent cannot load (a key that does not match the certificate, an empty key, an unreadable CA) refuses consented capture the same way and is logged at ERROR; it never stops the agent, so the L4 flow plane keeps running (R3). A control plane that cannot record the posture at start defers capture; the next policy check retries. When the channel is configured and recorded the posture but returned no floor (the floor call failed), the agent applies the stricter of its local mode and metadata. The recorded fixture replay is not live capture and needs no channel.
• Floor changes. A floor that tightens while the agent runs stops capture at once, and records already buffered are re-minimized to the new floor before they are published (nothing at all under length); a loosened floor never widens a running agent.
• A stale floor (D-43 S2). When the floor cannot be read for 3 consecutive policy checks (15 minutes at the default 5-minute interval, 90 seconds at the 30-second minimum), a running mode looser than metadata falls back to metadata until a fresh floor arrives. The fresh floor restores the decided mode only after the control plane recorded that posture.
• Certificate rotation (D-43 R2). The channel certificate is the enrollment service's short-lived leaf (24 h). The agent renews it at two thirds of its lifetime through control_plane.rotate_url (the API's POST /enroll/agent/rotate, verified against control_plane.ca_file), proving possession of the current key over a CSR for a new key generated on the host, and replaces cert_file/key_file as one unit (R3): the new certificate, key and the reply's agent-CA bundle (ca.pem, as the probe agent keeps it) are written to a fresh version directory and fsynced, then one current symlink is swapped atomically — cert_file and key_file become fixed links into it, so cert_file and key_file must sit in one directory. A kill, power loss or full disk leaves the previous pair or the new one, never a certificate beside a mismatched or empty key; the previous version is kept until the next rotation. ca_file, the listener trust the operator provides, is never overwritten. The next handshake presents the new pair, and a certificate naming any other identity is refused. rotate_url is required with grpc_addr, and the identity files must be writable: the shipped systemd units keep them under the probectl-ebpf-agent state directory, and the chart seeds a node-local hostPath (controlPlane.identity.stateHostPath//) from the read-only registration Secret (control_plane.identity_seed_dir), so a restarted pod presents the certificate it rotated, not the registration's. An identity whose certificate and key do not load as one pair is repaired from the seed. A failed renewal is logged at ERROR, counted in probectl_ebpf_control_plane_cert_rotation_failures_total, and retried every minute while the current certificate is valid. The certificate's expiry is exported as probectl_ebpf_control_plane_cert_not_after_unix_seconds (0 without a channel), and the chart's opt-in alert pack (metrics.prometheusRule.enabled) fires ProbectlEBPFChannelCertExpiring when less than 6 h remain; once it expires the channel fails closed (consented capture is refused). Limitation: a node that joins after the registration Secret's certificate has expired has nothing valid to seed from — re-register (POST /v1/collectors/register) and update the Secret.
• Uninstalling leaves the node identity behind. helm uninstall (or removing the systemd unit) does not delete the node identity directory: controlPlane.identity.stateHostPath// on every node for the chart, /var/lib/probectl-ebpf-agent/identity for systemd. It holds the collector's private key and its last certificate, which stays usable until it expires (at most 24 h after the last rotation); the key file stays until it is deleted. To decommission, revoke the collector first (POST /v1/agents/{id}/revoke, which feeds the listener's deny-list at once), then delete the directory on each node. There is no cleanup hook: a Helm hook runs one Job, not one per node.
• Posture. At start and on every policy tick the agent reports its capture posture (ReportL7Posture: enabled, consent tenant, scope, local and effective mode, the floor it applied, whether capture is attached, and a state: off, refused_consent, refused_floor, attach_failed, capturing, replaying). The control plane stores it on the agent's registry row — the agent detail view shows it — and appends agent.l7_capture.posture to the tenant's hash-chained audit stream when an agent first reports consented capture and whenever its mode, scope, floor or state changes. A consent tenant naming another tenant is never stored, only the mismatch.
• Ingest strip (second layer). Whatever an agent sent, the control plane strips every L7 record at ingest, before any consumer projects or persists it, down to the tenant's floor with the same function the agent uses (internal/l7redact): query strings always; under metadata (and the regulated profile) identifier route segments, non-token methods and statuses, SQL statement text and unknown protocols' free text too, plus any field the schema does not know. Under a length floor the door drops the tenant's L7 calls whole — no method, route or status persists (counted as call_dropped). The other fields a consumer persists are validated too (D-43 S4): protocols, transports, directions and TCP states are clamped to their vocabulary (other otherwise) — the service edge's l7_protocol lands in the eBPF store and the topology graph — and workload, process, container, host and address fields must have an identifier's shape (≤ 253 bytes of letters, digits, space and . _ : / @ + - ( ) [ ] # ~ ,) or become _invalid. An identifier-shaped value cannot be content-checked; this bounds shape, not meaning. The clamp fails closed at a cost: distinct workloads whose names carry other bytes (=, %, !, non-ASCII) all become _invalid and merge into one topology node within the tenant. On the pooled bus lane a tenant's floor is read only after the record's (tenant, agent) pair verifies against the registry (a namespaced lane names its tenant); anything unverified is stripped to metadata without a lookup, and the per-tenant floor cache is bounded (4096 tenants, 30-second TTL). Each strip is counted on probectl_ebpf_l7_ingest_stripped_total{field}.

What templating turns into {id} (one shared function, l7redact.TemplateRoute): numbers; UUIDs; any token mixing letters and digits (user123, acct42, ABCD123, contract numbers such as W56HZV-23-C-0123) except a short list of protocol and product names (oauth2, ec2, s3, k8s, http2, md5, sha256, x509, ipv6, …) and API versions (v1, v2beta1); digit-free hex of 12+ characters; base64-like case churn (MzEyMw); tokens of 32+ characters; anything percent-encoded or containing @ = + ~; and segments starting like a credential (sk-, ghp_, xoxb-, glpat-, AKIA, AIza, eyJ, …). Under metadata an HTTP status is three digits and a gRPC status one or two; anything else becomes _OTHER.

DNS question names (D-65). Under metadata a DNS name is templated left of its registered domain with the same rules (48213.patients.example.com → {id}.patients.example.com); under the regulated profile only the registered domain (eTLD+1, example.com) is kept. The registered domain comes from the ICANN section of the public-suffix list compiled into golang.org/x/net/publicsuffix — offline, nothing fetched. The list's private section is deliberately not used: there a cloud customer's label counts as "registered", so bucket-patient-48213.s3.amazonaws.com becomes {id}.s3.amazonaws.com (amazonaws.com under regulated) and patient-48213.github.io becomes {id}.github.io. In a TLD the ICANN list does not know (.internal, .corp, .local, .lan) the second-level label is judged too: w56hzv-23-c-0123.internal becomes {id}.internal on both profiles. An identifier-shaped label that is itself the ICANN registered label is kept on the default profile (48213.com, patient48213.net) and templated under the regulated profile ({id}.com, {id}.net) with the same identifier check — decision D-68. Under the audited headers opt-in (never available under regulated) DNS question names are kept raw, as they always were — decision D-69. Both the agent and the ingest strip apply it; the NDR detection summaries, incidents and the SIEM export quote only that stored form. The DGA and tunnelling detectors keep their signal from features the agent computes from the raw name while it is only in agent memory — the first label's entropy and length, the name's length, and 8-byte keyed tokens (the key is random per agent process) that say whether two names or two subdomains are the same and nothing else. A record from a producer that sends no features is scored on the stored form. With no floor received the agent keeps only the registered domain (the strictest rule). The word-shaped gap applies here too: alice.users.corp.internal keeps alice.

Upgrading. The floor, the channel and DNS minimization act from the release that carries them. During a rolling upgrade, agents of the previous release (inside the N/N-1 skew window) keep publishing what they always did — literal routes and DNS names under headers — to the retained bus until they are upgraded; the control plane's ingest strip still keeps it out of every store. Upgrade the eBPF agents first when that bus retention matters, or drain the topic.

• The chart fails closed on upgrade. helm upgrade of a release that has l7Capture.enabled but no controlPlane (address, identity Secret and, from D-43 R2, controlPlane.apiURL) refuses to render — and that blocks the whole agent DaemonSet, flow plane included, not only capture. Before upgrading, register the eBPF collector with a CSR (POST /v1/collectors/register, plane=ebpf), create the identity Secret, and set controlPlane.grpcAddr, controlPlane.apiURL and controlPlane.identity.existingSecret (or disable l7Capture for the upgrade).
• DGA and tunnelling are blind for older agents during the skew. Agents of the previous release (and any producer that sends no dns_features) publish raw DNS names without the in-agent features; the ingest strip templates those names, so the detectors score them on the stored form, whose random first label is now {id}. Their DGA bursts and tunnelling go undetected until the agents are upgraded — another reason to upgrade the eBPF agents first.

What metadata cannot see: a route segment that is an identifier but is shaped like a word (a username such as /users/alice or /users/JDOE). Templating works on shape, so routes of that kind need the application to template them, or the length mode.

Tuning and kernel lockdown

ring_buffer_bytes (config, or PROBECTL_EBPF_RING_BUFFER_BYTES) sizes the L4 flow kernel ring buffer for the live source; it's rounded at load to a valid power-of-two page multiple (default 16 MiB). Raise it on high-flow hosts to reduce drop_l4_ring_buffer_full_total.

l7_ring_buffer_bytes (config, or PROBECTL_EBPF_L7_RING_BUFFER_BYTES) sizes the separate TLS/plaintext L7 chunk ring (tls_chunks) used by live L7 uprobes. It has the same rounding behavior and 256 MiB validation cap as the L4 ring, but it is tuned independently because a host can have small flow churn and large HTTP/TLS bursts, or the reverse. Raise it when drop_l7_ring_buffer_full_total climbs after confirming L7 scope and redaction are intentional.

Kernel lockdown is a hardening mode (commonly enabled alongside Secure Boot) that restricts what even a privileged process may do to the running kernel: if the kernel runs in lockdown confidentiality mode, the bpf() syscall — the one door every BPF load goes through — is blocked even with CAP_BPF. The capability probe reports this explicitly (lockdown="confidentiality", mode unavailable) and a load attempt returns a clear message instead of a bare EPERM. Boot without lockdown=confidentiality (integrity mode is fine) to run the agent.

Kernel compatibility

CO-RE ("Compile Once – Run Everywhere" — the program is compiled once, and the loader re-aims its memory reads against the running kernel's BTF at load time, so one shipped object fits every supported kernel) needs a BTF-exposing kernel and the BPF ring buffer, both mainstream from Linux 5.8 — that pair is the hard floor. On a BTF-less kernel the capability probe reports eBPF unavailable (the reason string points at BTFHub — a public archive of pre-generated BTF files for older kernels that didn't ship their own — as a manual avenue; no automatic external-BTF fallback ships today). The full matrix and distro coverage live in ebpf-feasibility.md. eBPF is Linux-only; on macOS/Windows, run the agent inside a Linux VM.

Windows replacement path. There is no Windows eBPF agent in the delivered product, and Windows kernel telemetry must not be counted as served eBPF coverage. The supported replacement is to run the normal endpoint/DEM agent and OTLP/Windows-native telemetry collectors, then correlate those signals in the control plane beside Linux eBPF flows. A native Windows host/L7 agent would be a separate roadmap item, not a minor port of this Linux CO-RE implementation.

On startup the agent logs a capability probe (BTF / ring buffer / CAP_BPF / compiled-in) and the mode it chose, so an unsupported host is a decided, visible state — never a silent failure.

Kernel-matrix CI

Static checks aren't enough for kernel code, so the ebpf-kernel-matrix CI job actually loads and attaches every BPF program on real LTS kernels (LTS — long-term support — the kernel lines distros actually ship; the images are digest-pinned, i.e. referenced by cryptographic hash rather than a floating tag) under QEMU — a machine emulator that boots a complete kernel inside an ephemeral VM — via vimto, a small tool that runs a Go test suite inside such a booted kernel. It runs the live smoke: l4flow raw-tracepoint attach (as root and again as the systemd unit's non-root identity, TestLiveNonRootL4Attach), sslsniff uprobe attach (consented + scoped), one full agent flush cycle, with object-digest verification on the load path. The images are ghcr.io/cilium/ci-kernels.

The matrix is 5.15 and 6.6 on x86_64, plus a hardened entry on x86_64 that raises kernel lockdown to integrity inside the ephemeral VM (TestLiveHardenedLockdownIntegrity, gated on PROBECTL_TEST_SET_LOCKDOWN=integrity) and proves load+attach still works there while the probe reports the mode truthfully (confidentiality is the blocking mode; the test skips loudly if the CI kernel lacks the lockdown LSM — a secure-boot distro-kernel image is the remaining infrastructure gap).

Reproducing a matrix row locally, without KVM. make ebpf-kernel-lane KERNEL=5.15 RUN=TestLiveInboundEdgeOrientation (scripts/ebpf-kernel-lane.sh) boots the row's kernel with the same vimto, under QEMU's software emulation (TCG, VIMTO_DISABLE_KVM=1) inside a Docker container, so a matrix failure can be debugged on a laptop — macOS Docker Desktop included. On an amd64 host it uses the row's digest-pinned image from ci.yml; on an arm64 host it uses the same ci-kernels series' arm64 build, pinned in the script, because the CI digests are amd64-only manifests. The guest gets 256M, the size the CI matrix boots (vimto's default 128M leaves too little free for the ring buffer and for the non-root attach's second test process; PROBECTL_EBPF_LANE_MEMORY overrides it). A boot takes seconds and the full TestLive set about two minutes on an arm64 Mac. It is a reproduction lane, not evidence: the CI job still requires KVM, and it does not change the arm64 claim below.

One arch nuance worth knowing: the live QEMU boot needs KVM (the kernel's hardware-assisted virtualization, which lets the VM run at near-native speed instead of instruction-by-instruction emulation) for usable speed, and vimto does not emulate cross-arch — so an arm64 row cannot ride a hosted x86_64 runner. If /dev/kvm is missing the job fails rather than falling back to compile-only coverage. Bump the matrix when adopting a new LTS.

arm64 eBPF is compile-verified, not live-verified (decision D-13, 2026-09-19). The BPF objects are built for arm64 by the same pinned clang/llvm/bpftool toolchain, the probectl-ebpf-agent_linux_arm64 binary is built with -tags ebpf and the release gate refuses a fixture-only build — so the arm64 agent is proven to COMPILE and to carry the live CO-RE loader. What is NOT proven on arm64 is the kernel verifier accepting the programs and the attach succeeding: that needs a self-hosted Linux/ARM64 runner with a kvm label, which does not exist for this repository. The row was removed rather than left queued forever, because a job that never starts means the whole CI run never reaches a conclusion and every release tag times out waiting for it. Do not describe arm64 eBPF as live-load-proven until that runner exists and the row is back.

Building

Release builds do not rely on the GitHub runner's live apt repository state for the compiler path. deploy/docker/Dockerfile.ebpf exposes a named ebpf-toolchain stage with a digest-pinned Go/Debian base, Debian snapshot 20260702T000000Z, and exact clang-14, llvm-14, and bpftool package versions. CI and release run BPF generation through scripts/run-ebpf-toolchain.sh, and downloadable releases include a signed probectl_<version>_ebpf-toolchain.txt receipt beside the binaries and SBOM.

The shipped artifacts are live builds. Fixture mode is dev/test-only. The shipped agent image is the live -tags ebpf build: deploy/docker/Dockerfile.ebpf runs the same bpf2go + digest-generation path (bpf2go is the cilium/ebpf generator that compiles the C with clang and embeds the resulting BPF object into Go source). The downloadable probectl-ebpf-agent_linux_{amd64,arm64} binaries use the same generator path through scripts/build-release-binaries.sh instead of the plain cross-compile loop, and the deb/rpm package job checks that same binary before wrapping it. The CI/release gates fail unless Go build metadata records -tags=ebpf — so a fixture-only image, binary, or package cannot ship unnoticed.

Trust boundary (decided): operator-supplied BPF objects are deliberately not supported. The chain is: source → bpf2go (pinned clang) → objects embedded in the binary → a SHA-256 manifest baked at the same build → VerifyObjectDigest before any kernel load → the binary/image cosign-signed at release (cosign is the Sigstore artifact-signing tool — the signature proves the artifact came from this repo's release workflow). The release signature covers the objects + manifest together, so a swapped object can't ride a signed binary. A static gate (TestNoOperatorSuppliedBPFObjectPath) trips if anyone adds a filesystem/env object-load path.

The live build regenerates vmlinux.h (a generated header containing the kernel's type definitions, dumped from its BTF) from the running kernel, runs bpf2go, and writes the SHA-256 manifest (gendigests → bpf_digests_ebpf.go) that the loaders verify before any kernel load — an object that does not match its same-build manifest (swapped, corrupted, or left behind by an un-run generator) refuses to load. Be precise about what each control proves: the digest proves generator freshness and binary self-consistency (the manifest comes from the same build, so a rebuild produces a matching one); tamper protection for the shipped artifact is the cosign release signature above, which covers objects + manifest + binary together. The cilium/ebpf loader dependency is already pinned in go.mod; only the -tags ebpf files import it, so the default build never compiles or links it. On the build host:

make ebpf-agent                    # bpftool + bpf2go + go build -tags ebpf

A single wrapper, internal/ebpf/gen_bpf.sh, is the one place the bpf2go flags and the arch-compat shim live — make ebpf-agent, the Dockerfile, the CI jobs, and go generate all route through it, so a build change touches one file instead of drifting across many. Release downloads add one more wrapper around the same path:

VERSION=v0.1.0 COMMIT="$(git rev-parse HEAD)" bash scripts/build-release-binaries.sh
go version -m dist/probectl-ebpf-agent_v0.1.0_linux_amd64 | grep -E 'build[[:space:]]+-tags=.*ebpf'

The generated bindings (l4flow_bpfel.go, bpf/vmlinux.h) carry //go:build ebpf, are git-ignored, and are regenerated per build.

Installing

If this is your first probectl producer, start with getting-started.md (the control-plane + bus bring-up) and deploying-agents.md (the per-producer deployment journeys, including this agent) — you're done when flow batches show up downstream, not when the unit reports active. The sections below are the agent-specific contract.

Kubernetes

The deploy/helm/probectl-agent chart deploys the agent as a DaemonSet (one pod per node — host-level capture needs an agent on every host) with the privilege contract declared in the manifest: drop all capabilities and add back only CAP_BPF / CAP_PERFMON. capabilityMode: legacy renders CAP_SYS_ADMIN only with the explicit legacyKernelRingBufferAck acknowledgment after the runtime probe has confirmed BTF + BPF ring-buffer support; it is not a generic <5.8 escape hatch. The chart also declares a seccomp profile (a kernel syscall filter — the process may invoke only the listed system calls; RuntimeDefault, or point seccomp.type: Localhost at the installed default-deny profile for tighter filtering), read-only root, the BTF host mount, the tracefs mount (tracefs.hostPath, consented L7 only, DPR-345), the read-only node library mounts that consented L7 capture needs to attach at all (l7Capture.hostRoot / l7Capture.hostLibraryPaths, DPR-124 — present only when l7Capture.enabled), and resource limits. Rendering fails closed: no tenantID, L7 capture without l7Capture.scope, or plaintext kafka without the explicit bus.allowPlaintext, refuses to template. CI helm-lints, hardening-asserts, and kubeconform-validates the chart on every run.

Health probes are listenerless by default. The agent writes live.json and ready.json under health.stateDir, and the DaemonSet uses exec probes that run /usr/local/bin/app healthcheck --live|--ready inside the container. The compatibility HTTP mode (/healthz and /readyz on health.port) is fenced behind health.mode=http plus health.allowPlaintextHTTP=true; the default chart does not open port 9090.

The chart also renders the cluster-side Kyverno image-integrity policy by default. Think of it as the second lock on this privileged pod: Helm requires a digest-pinned image reference, and admission then requires that digest plus the keyless cosign signature from the release.yml tag workflow. Kyverno must already be installed, and the policy goes in before the agent (a two-step install the chart enforces; a mirrored image.repository is verified too): Image-integrity admission. Because Kyverno deletes its own webhooks when it is scaled to zero, the chart also ships a read-only CronJob that runs /usr/local/bin/app admission-check from the same agent image and fails when the policy or the webhook serving it is gone (admission.imageIntegrity.webhookCheck). If a dev cluster uses a replacement admission controller, set admission.imageIntegrity.enabled=false only with a non-empty admission.imageIntegrity.acceptedRisk note.

The chart also renders probectl-agent-capability-posture (EBPF-007), a Kyverno background/audit policy for capability drift. In ELI5 terms: the DaemonSet is allowed to carry the modern BPF + PERFMON keys, and the policy reports when a pod drops the ALL baseline, runs the explicit legacy SYS_ADMIN break-glass path, or gains any extra key such as NET_ADMIN. Those policy reports are the operational alert surface for capability posture; the default is Audit so the documented legacy exception remains human-gated instead of silently blocked.

Identity, lane and bus auth (DPR-051). The DaemonSet publishes straight to the bus, so it carries a registered identity rather than an SVID: register one collector for the release (register-collector -plane ebpf, POST /v1/collectors/register, or Admin & Settings > Agents > Register collector) and pass its id as agentID. Every record carries it as agent_id — the node name stays each record's host — and the control plane verifies the (tenant, agent_id) pair against the tenant's registry before it accepts a batch (TENANT-101). The chart refuses to render without it, because an unregistered identity means every batch is rejected. bus.namespace is the tenant's lane printed by the registration (t-<tenant-slug>), required wherever the control plane refuses the shared lane (the multi-tenant and regulated profiles). Bus client auth comes from a Secret: bus.sasl.mechanism

• bus.sasl.existingSecret (keys username/password, or set bus.sasl.userKey/passwordKey), or bus.tls.clientAuth=true to present tls.crt/tls.key from bus.tls.existingSecret.

The seccomp profile is declared on the agent container, and the install-seccomp-profile initContainer runs under RuntimeDefault: a pod-level Localhost profile would have governed the installer too, and a fresh node that did not have the profile yet could never start the pod (DPR-052). The installer runs the agent image itself (/usr/local/bin/app install-seccomp-profile, which writes the profile atomically and refuses anything that is not a seccomp profile), not a separate busybox: busybox publishes no cosign signature, so the image-integrity policy could never verify it, and an unverified image does not belong in a privileged DaemonSet (RTO-12). Every container the chart renders is the one signed, digest-pinned agent image; seccomp.installerImage is gone and fails the render if set.

helm install probectl-agent deploy/helm/probectl-agent \
  --set tenantID=acme \
  --set agentID=<collector-id> \
  --set bus.namespace=t-acme \
  --set 'bus.brokers={kafka.probectl.svc:9093}' \
  --set bus.tls.existingSecret=probectl-bus-tls \
  --set bus.sasl.mechanism=scram-sha-512 --set bus.sasl.existingSecret=probectl-bus-sasl \
  --set-string image.tag='0.6.6@sha256:<digest>'

(In Kubernetes the container runs as uid 0 with everything dropped except the minimal pair — Kubernetes grants added capabilities to root only, with no ambient- capability support. The VM unit below instead runs fully non-root via ambient capabilities — the Linux mechanism that lets a systemd service hand specific capabilities to a non-root process.)

VM / bare metal

deploy/agent/install.sh installs a local binary (air-gap friendly — downloads nothing, never self-updates), creates the probectl-agent system user, installs the hardened systemd unit, and writes a fail-closed sample config:

sudo deploy/agent/install.sh ./bin/probectl-ebpf-agent
$EDITOR /etc/probectl/ebpf-agent.yaml   # set tenant_id + brokers
sudo systemctl start probectl-ebpf-agent

Hardened runtime profile

Run the agent with the minimal capability set and the shipped seccomp profile — see deploy/agent/: CAP_BPF + CAP_PERFMON on kernels ≥ 5.8 (CAP_SYS_ADMIN only as an explicit legacy exception after the runtime probe confirms BTF + ring-buffer support), LimitMEMLOCK, no root, default-deny seccomp (deploy/agent/seccomp.json), plus a hardened systemd unit and container/K8s securityContext examples.

Running

# No-kernel / CI / macOS: replay recorded flows.
PROBECTL_EBPF_TENANT_ID=<uuid> PROBECTL_EBPF_FIXTURE_PATH=flows.json probectl-ebpf-agent

# Live (Linux, built with -tags ebpf, as root or with CAP_BPF+CAP_PERFMON):
probectl-ebpf-agent -config /etc/probectl/ebpf-agent.yaml

Example config: deploy/agent/probectl-ebpf-agent.example.yml.

Configuration keys

See configuration.md for the full PROBECTL_EBPF_* table.

Scope and follow-ups

In scope today: the agent, IPv4/IPv6 L3/L4 capture with byte/packet counters, the service map, L7 parsing (HTTP/1.1+2, gRPC, DNS, Kafka) with TLS-uprobe plaintext capture, OTel emit, and the kernel/uprobe matrix. On the consuming side, the control plane already drains probectl.ebpf.flows on three independent, tenant-verified consumer groups: the topology view (service edges feed the graph), segmentation validation (declared policy vs observed traffic), and NDR detection. Natural follow-ups (out of scope here): the 5-tuple↔SSL correlation and the Go-TLS capture path; and raw-flow retention in ClickHouse with a flow-level query API. Detection, segmentation validation, TLS posture, and cost all build on this layer.