Deploying agents & collectors

probectl's control plane never goes out and measures the network itself. It consumes. The measuring is done by a small family of agents and collectors that you deploy out where the truth lives — on probe hosts, next to your routers, on the switches' management network, inside the kernel of your servers, and on end-user laptops. Each one is a single static Go binary; each one sends what it sees back to the control plane; each one is bound to exactly one tenant so its data can never land in another tenant's view.

This page is the index: what producers exist, and how do I deploy each one? Pick the producers for the planes you actually care about — you do not need all of them. If you just want to see any data flow end-to-end first, start with the hands-on getting-started guide (it walks you from zero to a first probe result), then come back here to add the rest.

The deep "how each plane works" explanations live in the per-plane docs linked from each section below. This page stays at the "which binary, where does it run, how does it ship data, what's the one command" level on purpose.

The two ways a producer ships its data

Before the catalog, the single most important distinction — because it decides what backend infrastructure each producer needs. Think courier versus postal system: one producer hands its parcel directly to the recipient (nothing else has to exist), the others drop parcels at a depot the recipient empties on its own schedule — which means the depot has to be standing.

• Streams to the control plane over gRPC/mTLS. The agent holds an authenticated, tenant-bound connection straight to the control plane (gRPC is the HTTP/2-based remote-procedure-call protocol the stream rides on; mTLS is mutual TLS — both ends present certificates, so the server proves itself to the agent and the agent proves itself to the server). It needs nothing but the control plane reachable. This is the canary/synthetic agent (and, optionally in future, roaming endpoints). This is the courier.
• Publishes to the message bus. The collector writes records onto a bus topic (probectl.<type>.results / probectl.<type>.events), and a control-plane consumer drains the topic into storage. In a real fleet that bus is Kafka (the open-source distributed message log: producers append, consumers read at their own pace, and the log absorbs bursts neither side could handle alone), and the high-cardinality planes land in ClickHouse (the column store built for billions of small event rows) — so these producers imply "you are running Kafka + ClickHouse" (or the lightweight in-memory bus, for a single-node dev box only). This is the BGP, flow, device, eBPF, and endpoint producers — the postal model, and the depot is Kafka.

So: a synthetic-only deployment can run with just the control plane and its Postgres. The moment you want BGP router feeds, flow, device telemetry, eBPF, or endpoint data at fleet scale, you are also standing up Kafka and ClickHouse. Each section below says which camp its producer is in.

Enroll once: how an agent gets its identity

Every agent that talks to the control plane proves who it is with a short-lived mTLS client certificate — an SVID — whose SPIFFE identity names its tenant and agent id (spiffe://probectl/tenant/<t>/agent/<a>; SPIFFE is the standard URI naming scheme for workload identities, and an SVID is the certificate that carries one). Until an agent has one, the mTLS transport refuses its connection and nothing it sends lands anywhere. You do not hand-distribute certificates; the trust root is managed by the control plane, and the runtime rotates the certificate automatically forever after.

The lifecycle is: set up the CA once per deployment, mint a one-time join token per agent, and redeem it on the agent host.

1. One-time, on the control host — create the certificate hierarchy:

probectl-control agent-ca init
probectl-control agent-ca export /etc/probectl/agent-ca.crt

init prints the root private key once for offline custody and never stores it; runtime operation never needs it. It prints only to a terminal: run through a Job, a Helm hook or a CI step — anywhere stdout is a pipe or a log — and it refuses, because those bytes would be stored wherever that output goes (DPR-121). Automate it with agent-ca init -key-out /path/key.pem (written 0600, move it to custody and delete the file), or -print-key when the destination really is safe custody, such as a pipe into your vault. Re-running refuses to overwrite the trust root. export writes the CA's public trust bundle (root + intermediate certificates — never a key) to a file; point the control plane's PROBECTL_AGENT_TLS_CA_FILE at it so the gRPC listener can verify enrolling agents. (Helm: kubectl exec deploy/<release> -- /usr/local/bin/app agent-ca init, then --set control.agentListener.enabled=true; the chart's init container runs the export for you and the listener is the <release>-agents Service — Helm README.)

2. Mint a join token (one per agent; the token names the tenant):

From the web UI, go to Admin & Settings → Agents → Enroll agent. The modal mints the tenant-scoped token, shows it once, and prints the probectl-agent enroll command with either the current server certificate pin or the --ca-file trust path.

probectl-control enroll-token -tenant <tenant-uuid> [-agent <id>] [-name "edge-probe-1"] [-ttl 1h]

The CLI path prints the same token (pjt_…) once; it is single-use, expires (default 1h), and is tenant-scoped. The command also prints the control plane's certificate pin for first contact when PROBECTL_TLS_CERT_FILE is configured.

3. Redeem it on the agent host:

probectl-agent enroll \
  --server https://<control-host>:8443 \
  --token pjt_... \
  --dir /var/lib/probectl-agent/identity \
  --ca-pin <hex-sha256>          # for self-signed quickstarts
  # ...or, with a CA-issued control-plane cert:  --ca-file ca.crt

In the shipped container images the agent runs as the distroless non-root user, and only /var/lib/probectl and /var/lib/probectl-agent are pre-created for it — mount the identity volume under one of those (--dir /var/lib/probectl-agent/identity); any other fresh volume is root-owned and the enrollment refuses it before touching the token.

The agent generates its private key locally (it never leaves the host), sends a CSR — a certificate signing request: the public half of the keypair, packaged for the CA to sign — and writes the issued cert, the intermediate, and the trust bundle (mode 0600) into --dir. A --ca-pin that mismatches refuses the connection — there is no trust-on-first-use fallback. enroll then prints the exact tls: / identity: config snippet to paste into the agent's config so it rotates automatically.

For fleets where a separate enroll step is awkward (containers, cloud-init), the canary agent also enrolls itself on first boot: hand it the token via the PROBECTL_AGENT_JOIN_TOKEN environment variable or the enroll.token_file config key, and it redeems the token before starting. This is idempotent — an agent that already has an identity skips it — and fail-closed: a consumed or invalid token stops the agent with a clear error instead of running unauthenticated.

For SVID rotation (every ~24h, automatic), revocation, and the full bootstrap threat model, see agent enrollment & rotation.

Register bus collectors

BGP, flow, device, eBPF, and endpoint collectors publish to the bus instead of streaming over the mTLS agent channel, so they do not redeem an SVID. They still need a tenant-bound registry identity: every published record carries tenant_id and agent_id, and the control plane checks that pair against the tenant's registry before accepting the batch.

From the web UI, go to Admin & Settings > Agents > Register collector. Pick the plane, enter an optional source label, and optionally pin a collector UUID. The UI mints a short-lived one-time token for the current tenant, immediately consumes it through /v1/collectors/register, and shows the env/YAML keys to put in the collector config. The browser never sends tenant_id; the server derives it from the authenticated tenant before consuming the token.

CLI/API equivalent:

# 1. Mint a tenant-scoped one-time token.
probectl agent enroll-token --body '{"name":"edge-flow-1","ttl_seconds":300}'

# 2. Register the collector with that token.
probectl collector register --body '{"token":"pjt_...","plane":"flow","hostname":"edge-flow-1"}'
probectl bgp setup --body '{"token":"pjt_...","plane":"bgp","hostname":"rrc00"}'

Registering a second collector under a name the tenant already uses answers 409 conflict naming the collision: reuse the existing agent_id or pick another name (DPR-048).

The response includes tenant_id, agent_id, capabilities, and concrete config hints — including the tenant's bus lane (PROBECTL_<PLANE>_BUS_NAMESPACE=t-<tenant-slug>). Every active tenant, pooled or siloed, owns a namespaced lane; in the multi-tenant and regulated profiles the shared lane is refused for agent-published planes (PROBECTL_INGEST_STRICT_TENANT_LANES), so the collector must carry that setting or every batch is rejected at the control plane (DPR-049). Flow, device, endpoint and eBPF all read the identity as agent_id (PROBECTL_<PLANE>_AGENT_ID) and the lane as bus.namespace (PROBECTL_<PLANE>_BUS_NAMESPACE); the YAML hints are the collectors' real keys, nested where the collector nests them, so they paste straight into the config file (DPR-051/DPR-053). BGP returns BMP listener hints, including source_type: bmp, PROBECTL_BMP_COLLECTOR=<collector-id>, and the startup command probectl-bmp-listener.

Bus topics

The control plane creates the topics it publishes and consumes (PROBECTL_BUS_CREATE_TOPICS=true, the default; DPR-047): the shared lanes at startup and a siloed tenant's namespaced lanes as that tenant appears. If your broker user may not create topics, set it to false and pre-create these — startup then verifies them and fails closed naming any that is missing:

Partitions and replication for created topics: PROBECTL_BUS_TOPIC_PARTITIONS (3) and PROBECTL_BUS_TOPIC_REPLICATION (-1 = the broker default).

The producers

Canary / synthetic — probectl-agent

What it observes. Active, on-purpose probes you schedule: it sends traffic and times the answer. Compiled-in probe types are icmp (loss / latency / jitter — jitter being the variation between round-trips, what voice and video feel), tcp (connect latency + reachability), udp (echo round-trip), dns (resolution time, answer, DNSSEC validation, plus iterative delegation traces — walking root → TLD → authority itself instead of trusting a resolver's cache), http (availability + DNS/connect/TLS/TTFB/total breakdown — TTFB is time-to-first-byte — and on HTTPS the TLS handshake details that feed the TLS-posture view), browser (either a non-rendering HTTP transaction or a rendered Playwright transaction, selected explicitly by both test and agent), and voice (RTP media probes scored as MOS — the 1–5 mean-opinion-score call-quality scale — plus jitter / loss) — plus a noop heartbeat. It can also do agent-to-agent (a2a) two-way measurement (TWAMP-lite style: both ends timestamp, so each direction's latency is measured separately) when you enable the a2a: block, turning a pair of agents into a synthetic mesh. A2A is opt-in (enabled: false by default), and every brokered probe/reply frame is authenticated with a per-session HMAC key delivered over the existing agent mTLS channel. Unlike the single-agent canaries above, A2A is not placed in canaries: and does not run on a local interval: an audited API/CLI request creates the pair or mesh, and the control-plane broker assigns each agent its role. See the A2A coordination ADR.

Where it runs. Any OS, unprivileged by default — ICMP uses unprivileged datagram sockets, so no CAP_NET_RAW and no root for the common case. Rendered browser probes use the separate non-root probectl-browser-agent container because Chromium and its OS libraries are intentionally not embedded in the portable Go binary.

How it ships data. It streams straight to the control plane over gRPC/mTLS — it does not use the bus. Point it at the control plane's gRPC listener via control_plane.grpc_addr (required — there is no built-in default; the shipped examples and stacks use port 9443), using the mTLS identity from enrollment. Results buffer to a local disk queue while the control plane is unreachable and drain on reconnect, so a network blip never loses data. Infra it needs: just the control plane (no Kafka, no ClickHouse).

Deploy it.

probectl-agent -config /etc/probectl/agent.yml

Config template: deploy/agent/probectl-agent.example.yml. Every key (and the per-probe parameters) is documented in configuration.md; voice specifics are in voice.md, and the browser/transaction synthetic in browser-synthetic.md.

Flow collector — probectl-flow-agent

What it observes. The passive, after-the-fact view of real traffic: routers and switches summarize the packets they forwarded into flow records (the 5-tuple — source/destination address, source/destination port, protocol — plus byte/packet counts) and export them. The collector decodes NetFlow v5/v9, IPFIX, and sFlow v5 — three export dialects of that same idea — into one normalized, tenant-bound record.

Where it runs. As a service on a host on (or adjacent to) your management network. Flow export is plaintext UDP with no authentication — by protocol design — so deploy the collector next to its exporters and never let those datagrams cross an untrusted segment. It binds three UDP listeners: :2055 (NetFlow v5 and v9, version-sniffed on one socket), :4739 (IPFIX), :6343 (sFlow). Disable any listener you don't run.

How it ships data. It publishes to the bus on probectl.flow.events; the control plane's flow consumer verifies the tenant, optionally enriches ASN/geo (ASN — the autonomous-system number, i.e. which network on the internet owns the address — and its geolocation), and writes rows into ClickHouse. Infra it needs: the message bus (bus.mode: kafka → Kafka) and ClickHouse. A single-node dev box can set bus.mode: memory instead.

Deploy it.

probectl-flow-agent -config /etc/probectl/flow-agent.yml

Config template: deploy/agent/probectl-flow-agent.example.yml. Full plane walkthrough, the OTel field mapping, sampling correction, and the query API: flow analytics.

Device / streaming telemetry — probectl-device-agent

What it observes. The health of the switches and routers themselves — interface counters and oper-status, CPU, memory, and (opt-in) temperatures — straight from the gear. It speaks two protocols and normalizes both into one DeviceMetric with identical metric names: SNMP (the Simple Network Management Protocol, v2c or v3 — the decades-old standard where the agent polls on an interval) and gNMI / OpenConfig (the gRPC Network Management Interface — the modern alternative where the device streams updates over a TLS gRPC channel). Same metric names either way is the point: dashboards and alerts don't care which protocol a given switch happened to speak.

Where it runs. As a service with line-of-sight to the devices' management addresses. gNMI dials TLS with certificate verification on by default (system roots, or pin a private CA with ca_file); there is no skip-verify, and the lab-only plaintext opt-in is loudly logged. Credentials (SNMP communities, SNMPv3 passphrases, gNMI passwords) are referenced by name only and resolved at runtime — never written to config or git, never logged. A credential that resolves to nothing fails closed at startup.

How it ships data. It publishes to the bus on probectl.device.metrics; the control plane's device consumer lands the series in the time-series database. Infra it needs: the message bus (bus.mode: kafka → Kafka) (plus your TSDB, which the control plane already owns). Single-node dev: bus.mode: memory.

Deploy it.

probectl-device-agent -config /etc/probectl/device-agent.yml

Config template: deploy/agent/probectl-device-agent.example.yml. Metric catalog, the SNMP↔gNMI unification, FIPS notes, and how device interfaces correlate path and flow data: device telemetry.

eBPF host agent — probectl-ebpf-agent

What it observes. Network activity from inside the host's kernel, with zero instrumentation — no sidecars, no app changes, no SDK. (eBPF lets the kernel run small, sandboxed, pre-verified observation programs — the agent attaches them to the kernel's own networking events instead of capturing packets.) It captures every TCP connection a host makes or accepts (the L3/L4 flow — addresses and ports — plus the process and container behind it) and builds a live service map of who-talks-to-whom. It can additionally parse L7 (application-layer) calls (HTTP/1.1+2, gRPC, DNS, Kafka), including over TLS via library uprobes — kernel hooks attached to a user-space library's functions, here the TLS library, so data is visible before it is encrypted — and reading application plaintext is off by default and triple-gated (an enable flag plus a per-tenant consent that must name this agent's tenant plus a non-empty workload allowlist; host-wide capture is not expressible). It is observe-only — it loads no enforcing program and blocks no packet, a guarantee enforced by a build-failing test. It is not a CNI and not an inline IPS.

Where it runs. Linux only, on each host you want to see. It needs CAP_BPF + CAP_PERFMON on supported kernels — the two Linux capabilities that permit loading BPF programs and attaching to perf events, granted without root — and a BTF-exposing kernel (/sys/kernel/btf/vmlinux, mainstream from 5.8 — BTF is the kernel's embedded type catalog, which lets one compiled agent adapt to any kernel) with its tracefs visible (/sys/kernel/tracing; a container must have the node's tracefs mounted in read-only — the Helm chart does, DPR-054). Generic kernels older than 5.8 are unsupported by default; CAP_SYS_ADMIN is only an explicit legacy break-glass when the runtime probe confirms BTF plus BPF ring-buffer support but split caps cannot be granted. On macOS/Windows, run it inside a Linux VM. The shipped image is the live build; fixture-replay mode exists only for CI / no-kernel boxes.

How it ships data. It publishes to the bus on probectl.ebpf.flows. The shipped artifacts default to bus.mode: kafka. Infra it needs: Kafka. Control-plane consumers (topology, segmentation, NDR) read these flows live; raw flow-by-flow retention in ClickHouse is not wired yet, so there is no queryable per-flow history — only what the consumers derive. That retention edge is tracked in the canonical limitations table. The agent refuses plaintext Kafka unless you explicitly opt in for a lab.

Deploy it (VM / bare metal).

sudo deploy/agent/install.sh ./bin/probectl-ebpf-agent
$EDITOR /etc/probectl/ebpf-agent.yaml      # set tenant_id + bus brokers
sudo systemctl start probectl-ebpf-agent

The installer is air-gap friendly (it downloads nothing and never self-updates), creates a dedicated non-root probectl-agent system user, and installs the hardened systemd unit (ambient CAP_BPF+CAP_PERFMON, a default-deny syscall filter, namespace lockdown) shipped alongside it. The unit and its matching container seccomp profile (seccomp — the Linux syscall filter; the profile is the allowlist of kernel calls the process may make) live in deploy/agent/ (probectl-ebpf-agent.service, seccomp.json). Config template: deploy/agent/probectl-ebpf-agent.example.yml. The full plane — capture pipeline, the triple-gate, redaction modes, the kernel/uprobe compatibility matrix, and the build: eBPF host agent.

Endpoint / last-mile DEM — probectl-endpoint

What it observes. The last mile that server-side canaries physically cannot see: a remote user's Wi-Fi link health, their local gateway, the ISP / last-mile path, and browser-session timings — and then the headline trick, it attributes a slowdown to the closest impaired layer (Wi-Fi → local LAN → ISP → wider network), so you can answer "is it us, or the user's Wi-Fi / ISP?". It runs on the user's own device, so data minimization is a hard rule (geolocatable identifiers — like the AP MAC, the Wi-Fi access point's hardware address, enough to place a household on a map — and public hop IPs are dropped before a sample is ever emitted), and it discloses exactly what it collects at startup.

Where it runs. Cross-OS (Linux / macOS / Windows), on end-user devices, no elevated privileges — it uses the OS's own traceroute/tracert and read-only Wi-Fi queries. Ship the single static binary via your MDM (mobile device management — the tool IT already uses to push software to managed laptops: Intune, Jamf).

How it ships data. It publishes to the bus on probectl.endpoint.results, tenant-keyed, flowing through the same pipeline as every other canary; it never phones home. Infra it needs: the message bus — Kafka in a fleet, or the lightweight in-memory bus on a single-node dev box.

Deploy it.

probectl-endpoint -config /etc/probectl/endpoint.yml

The privacy model, the attribution engine, the per-OS collection matrix, and the /v1/endpoints fleet surface: endpoint DEM.

A producer-by-producer cheat sheet

At scale

Running one agent by hand is the learning path. A fleet is run by your orchestrator — probectl deliberately has no agent self-update channel (that would be a fleet-wide remote-code-execution primitive), so update authority stays with your tooling.

• Kubernetes — the eBPF host agent as a DaemonSet (Kubernetes's run-one-copy-on-every-node primitive — exactly the shape a per-host kernel observer needs). The supported chart is deploy/helm/probectl-agent. It declares the privilege contract in the manifest (drop all capabilities, add back only CAP_BPF/CAP_PERFMON, a seccomp profile, read-only root, the BTF and tracefs host mounts, resource limits) and renders fail-closed — no tenantID, or plaintext Kafka without an explicit opt-in, refuses to template. It also renders the Kyverno image-integrity policy by default so admission requires the digest and the keyless cosign signature from the tagged release.yml workflow before the privileged pod can start. Disabling that policy requires an explicit admission.imageIntegrity.acceptedRisk note.

  helm install probectl-agent deploy/helm/probectl-agent \
    --set tenantID=<tenant> \
    --set agentID=<collector-id> \
    --set bus.namespace=t-<tenant-slug> \
    --set 'bus.brokers={kafka.probectl.svc:9093}' \
    --set bus.tls.existingSecret=probectl-bus-tls \
    --set bus.sasl.mechanism=scram-sha-512 --set bus.sasl.existingSecret=probectl-bus-sasl \
    --set-string image.tag='0.6.5@sha256:<digest>'
  

  agentID is the collector identity the tenant registered for these nodes (previous section) and bus.namespace the lane the registration printed; the chart refuses to render without the identity, because the control plane would reject every batch (DPR-051). Bus client auth is a Secret: bus.sasl.* (keys username/password by default) or bus.tls.clientAuth=true to present tls.crt/tls.key from bus.tls.existingSecret.

  Naming heads-up: the chart is named probectl-agent but it deploys the eBPF host agent (probectl-ebpf-agent image) — it is the per-node flow agent, not the synthetic probectl-agent binary. The eBPF agent is the one with a shipped Helm chart, systemd unit, and installer; the canary, flow, and device agents ship example configs and run as ordinary services or containers in your platform.

• VMs / bare metal — the installer. deploy/agent/install.sh installs a local binary (air-gap friendly: downloads nothing, never self-updates), creates the non-root system user, installs the hardened systemd unit, and writes a fail-closed sample config. Re-run it with a new binary to update.

• Staged fleet rollout. Move a fleet to a new version in waves (canary → early → main) from signed artifacts, with the agent registry verifying each wave and any straggler halting the train — so a bad version stops at a small canary instead of taking out the whole fleet. The model, the operator flow, and halt/resume semantics are in staged fleet rollout.

See also

• Getting started — zero to your first probe result, the "just try one" path.
• Installation — standing up the control plane the agents connect to.
• Configuration reference — every PROBECTL_* key and config field for every binary.
• Agent enrollment & rotation — the identity lifecycle in depth.