Deploying agents & collectors

probectl's control plane never goes out and measures the network itself. It consumes. The measuring is done by a small family of agents and collectors that you deploy out where the truth lives — on probe hosts, next to your routers, on the switches' management network, inside the kernel of your servers, and on end-user laptops. Each one is a single static Go binary; each one sends what it sees back to the control plane; each one is bound to exactly one tenant so its data can never land in another tenant's view.

This page is the index: what producers exist, and how do I deploy each one? Pick the producers for the planes you actually care about — you do not need all of them. If you just want to see any data flow end-to-end first, start with the hands-on getting-started guide (it walks you from zero to a first probe result), then come back here to add the rest.

The deep "how each plane works" explanations live in the per-plane docs linked from each section below. This page stays at the "which binary, where does it run, how does it ship data, what's the one command" level on purpose.

The two ways a producer ships its data

Before the catalog, the single most important distinction — because it decides what backend infrastructure each producer needs. Think courier versus postal system: one producer hands its parcel directly to the recipient (nothing else has to exist), the others drop parcels at a depot the recipient empties on its own schedule — which means the depot has to be standing.

• Streams to the control plane over gRPC/mTLS. The agent holds an authenticated, tenant-bound connection straight to the control plane (gRPC is the HTTP/2-based remote-procedure-call protocol the stream rides on; mTLS is mutual TLS — both ends present certificates, so the server proves itself to the agent and the agent proves itself to the server). It needs nothing but the control plane reachable. This is the canary/synthetic agent (and, optionally in future, roaming endpoints). This is the courier.
• Publishes to the message bus. The collector writes records onto a bus topic (probectl.<type>.results / probectl.<type>.events), and a control-plane consumer drains the topic into storage. In a real fleet that bus is Kafka (the open-source distributed message log: producers append, consumers read at their own pace, and the log absorbs bursts neither side could handle alone), and the high-cardinality planes land in ClickHouse (the column store built for billions of small event rows) — so these producers imply "you are running Kafka + ClickHouse" (or the lightweight in-memory bus, for a single-node dev box only). This is the BGP, flow, device, eBPF, and endpoint producers — the postal model, and the depot is Kafka.

So: a synthetic-only deployment can run with just the control plane and its Postgres. The moment you want BGP router feeds, flow, device telemetry, eBPF, or endpoint data at fleet scale, you are also standing up Kafka and ClickHouse. Each section below says which camp its producer is in.

Enroll once: how an agent gets its identity

Every agent that talks to the control plane proves who it is with a short-lived mTLS client certificate — an SVID — whose SPIFFE identity names its tenant and agent id (spiffe://probectl/tenant/<t>/agent/<a>; SPIFFE is the standard URI naming scheme for workload identities, and an SVID is the certificate that carries one). Until an agent has one, the mTLS transport refuses its connection and nothing it sends lands anywhere. You do not hand-distribute certificates; the trust root is managed by the control plane, and the runtime rotates the certificate automatically forever after.

The lifecycle is: set up the CA once per deployment, mint a one-time join token per agent, and redeem it on the agent host.

1. One-time, on the control host — create the certificate hierarchy:

probectl-control agent-ca init
probectl-control agent-ca export /etc/probectl/agent-ca.crt

init prints the root private key once for offline custody and never stores it; runtime operation never needs it. It prints only to a terminal: run through a Job, a Helm hook or a CI step — anywhere stdout is a pipe or a log — and it refuses, because those bytes would be stored wherever that output goes (DPR-121). Automate it with agent-ca init -key-out /path/key.pem (written 0600, move it to custody and delete the file), or -print-key when the destination really is safe custody, such as a pipe into your vault. Re-running refuses to overwrite the trust root. export writes the CA's public trust bundle (root + intermediate certificates — never a key) to a file; point the control plane's PROBECTL_AGENT_TLS_CA_FILE at it so the gRPC listener can verify enrolling agents. (Helm: kubectl exec deploy/<release> -- /usr/local/bin/app agent-ca init, then --set control.agentListener.enabled=true; the chart's init container runs the export for you and the listener is the <release>-agents Service — Helm README.)

2. Mint a join token (one per agent; the token names the tenant):

From the web UI, go to Admin & Settings → Agents → Enroll agent. The modal mints the tenant-scoped token, shows it once, and prints the probectl-agent enroll command with either the current server certificate pin or the --ca-file trust path.

probectl-control enroll-token -tenant <tenant-uuid> [-agent <id>] [-name "edge-probe-1"] [-ttl 1h]

The CLI path prints the same token (pjt_…) once; it is single-use, expires (default 1h), and is tenant-scoped. The command also prints the control plane's certificate pin for first contact when PROBECTL_TLS_CERT_FILE is configured.

3. Redeem it on the agent host:

probectl-agent enroll \
  --server https://<control-host>:8443 \
  --token pjt_... \
  --dir /var/lib/probectl-agent/identity \
  --ca-pin <hex-sha256>          # for self-signed quickstarts
  # ...or, with a CA-issued control-plane cert:  --ca-file ca.crt

In the shipped container images the agent runs as the distroless non-root user, and only /var/lib/probectl and /var/lib/probectl-agent are pre-created for it — mount the identity volume under one of those (--dir /var/lib/probectl-agent/identity); any other fresh volume is root-owned and the enrollment refuses it before touching the token.

The agent generates its private key locally (it never leaves the host), sends a CSR — a certificate signing request: the public half of the keypair, packaged for the CA to sign — and writes the issued cert, the intermediate, and the trust bundle (mode 0600) into --dir. A --ca-pin that mismatches refuses the connection — there is no trust-on-first-use fallback. enroll then prints the exact tls: / identity: config snippet to paste into the agent's config so it rotates automatically.

For fleets where a separate enroll step is awkward (containers, cloud-init), the canary agent also enrolls itself on first boot: hand it the token via the PROBECTL_AGENT_JOIN_TOKEN environment variable or the enroll.token_file config key, and it redeems the token before starting. This is idempotent — an agent that already has an identity skips it — and fail-closed: a consumed or invalid token stops the agent with a clear error instead of running unauthenticated.

For SVID rotation (every ~24h, automatic), revocation, and the full bootstrap threat model, see agent enrollment & rotation.

Register bus collectors

BGP, flow, device, eBPF, and endpoint collectors publish to the bus instead of streaming over the mTLS agent channel, so they do not redeem an SVID. They still need a tenant-bound registry identity: every published record carries tenant_id and agent_id, and the control plane checks that pair against the tenant's registry before accepting the batch.

From the web UI, go to Admin & Settings > Agents > Register collector. Pick the plane, enter an optional source label, and optionally pin a collector UUID. The UI mints a short-lived one-time token for the current tenant, immediately consumes it through /v1/collectors/register, and shows the env/YAML keys to put in the collector config. The browser never sends tenant_id; the server derives it from the authenticated tenant before consuming the token.

CLI/API equivalent:

# 1. Mint a tenant-scoped one-time token.
probectl agent enroll-token --body '{"name":"edge-flow-1","ttl_seconds":300}'

# 2. Register the collector with that token.
probectl collector register --body '{"token":"pjt_...","plane":"flow","hostname":"edge-flow-1"}'
probectl bgp setup --body '{"token":"pjt_...","plane":"bgp","hostname":"rrc00"}'

Registering a second collector under a name the tenant already uses answers 409 conflict naming the collision: reuse the existing agent_id or pick another name (DPR-048).

The response includes tenant_id, agent_id, capabilities, and concrete config hints — including the tenant's bus lane (PROBECTL_<PLANE>_BUS_NAMESPACE=t-<tenant-slug>). Every active tenant, pooled or siloed, owns a namespaced lane; in the multi-tenant and regulated profiles the shared lane is refused for agent-published planes (PROBECTL_INGEST_STRICT_TENANT_LANES), so the collector must carry that setting or every batch is rejected at the control plane (DPR-049). Flow, device, endpoint and eBPF all read the identity as agent_id (PROBECTL_<PLANE>_AGENT_ID) and the lane as bus.namespace (PROBECTL_<PLANE>_BUS_NAMESPACE); the YAML hints are the collectors' real keys, nested where the collector nests them, so they paste straight into the config file (DPR-051/DPR-053). BGP returns BMP listener hints, including source_type: bmp, PROBECTL_BMP_COLLECTOR=<collector-id>, and the startup command probectl-bmp-listener.

Bus topics

The control plane creates the topics it publishes and consumes (PROBECTL_BUS_CREATE_TOPICS=true, the default; DPR-047): the shared lanes at startup and a siloed tenant's namespaced lanes as that tenant appears. If your broker user may not create topics, set it to false and pre-create these — startup then verifies them and fails closed naming any that is missing:

Partitions and replication for created topics: PROBECTL_BUS_TOPIC_PARTITIONS (3) and PROBECTL_BUS_TOPIC_REPLICATION (-1 = the broker default).

Startup reads the brokers' topic metadata before it creates anything. If that read fails, startup refuses with "cannot read topic metadata from the brokers" and the cause, for example x509: certificate signed by unknown authority. That is a reachability, TLS or SASL fault, not a topic permission: check PROBECTL_BUS_BROKERS, PROBECTL_BUS_TLS_CA_FILE and the PROBECTL_BUS_SASL_* settings. Only a create the broker refuses asks you to grant the bus user CREATE (DPR-335).

The producers

Canary / synthetic — probectl-agent

What it observes. Active, on-purpose probes you schedule: it sends traffic and times the answer. Compiled-in probe types are icmp (loss / latency / jitter — jitter being the variation between round-trips, what voice and video feel), tcp (connect latency + reachability), udp (echo round-trip), dns (resolution time, answer, DNSSEC validation, plus iterative delegation traces — walking root → TLD → authority itself instead of trusting a resolver's cache), http (availability + DNS/connect/TLS/TTFB/total breakdown — TTFB is time-to-first-byte — and on HTTPS the TLS handshake details that feed the TLS-posture view), browser (either a non-rendering HTTP transaction or a rendered Playwright transaction, selected explicitly by both test and agent), and voice (RTP media probes scored as MOS — the 1–5 mean-opinion-score call-quality scale — plus jitter / loss) — plus a noop heartbeat. It can also do agent-to-agent (a2a) two-way measurement (TWAMP-lite style: both ends timestamp, so each direction's latency is measured separately) when you enable the a2a: block, turning a pair of agents into a synthetic mesh. A2A is opt-in (enabled: false by default), and every brokered probe/reply frame is authenticated with a per-session HMAC key delivered over the existing agent mTLS channel. Unlike the single-agent canaries above, A2A is not placed in canaries: and does not run on a local interval: an audited API/CLI request creates the pair or mesh, and the control-plane broker assigns each agent its role. See the A2A coordination ADR.

Where it runs. Any OS, unprivileged by default — ICMP uses unprivileged datagram sockets, so no CAP_NET_RAW and no root for the common case. Rendered browser probes use the separate non-root probectl-browser-agent container because Chromium and its OS libraries are intentionally not embedded in the portable Go binary.

How it ships data. It streams straight to the control plane over gRPC/mTLS — it does not use the bus. Point it at the control plane's gRPC listener via control_plane.grpc_addr (required — there is no built-in default; the shipped examples and stacks use port 9443), using the mTLS identity from enrollment. Results buffer to a local disk queue while the control plane is unreachable and drain on reconnect, so a network blip never loses data. Infra it needs: just the control plane (no Kafka, no ClickHouse).

Deploy it.

probectl-agent -config /etc/probectl/agent.yml

Config template: deploy/agent/probectl-agent.example.yml. Every key (and the per-probe parameters) is documented in configuration.md; voice specifics are in voice.md, and the browser/transaction synthetic in browser-synthetic.md.

Flow collector — probectl-flow-agent

What it observes. The passive, after-the-fact view of real traffic: routers and switches summarize the packets they forwarded into flow records (the 5-tuple — source/destination address, source/destination port, protocol — plus byte/packet counts) and export them. The collector decodes NetFlow v5/v9, IPFIX, and sFlow v5 — three export dialects of that same idea — into one normalized, tenant-bound record.

Where it runs. As a service on a host on (or adjacent to) your management network. Flow export is plaintext UDP with no authentication — by protocol design — so deploy the collector next to its exporters and never let those datagrams cross an untrusted segment. It binds three UDP listeners: :2055 (NetFlow v5 and v9, version-sniffed on one socket), :4739 (IPFIX), :6343 (sFlow). Disable any listener you don't run.

How it ships data. It publishes to the bus on probectl.flow.events; the control plane's flow consumer verifies the tenant, optionally enriches ASN/geo (ASN — the autonomous-system number, i.e. which network on the internet owns the address — and its geolocation), and writes rows into ClickHouse. Infra it needs: the message bus (bus.mode: kafka → Kafka) and ClickHouse. A single-node dev box can set bus.mode: memory instead.

Deploy it.

probectl-flow-agent -config /etc/probectl/flow-agent.yml

Config template: deploy/agent/probectl-flow-agent.example.yml. Full plane walkthrough, the OTel field mapping, sampling correction, and the query API: flow analytics.

Device / streaming telemetry — probectl-device-agent

What it observes. The health of the switches and routers themselves — interface counters and oper-status, CPU, memory, and (opt-in) temperatures — straight from the gear. It speaks two protocols and normalizes both into one DeviceMetric with identical metric names: SNMP (the Simple Network Management Protocol, v2c or v3 — the decades-old standard where the agent polls on an interval) and gNMI / OpenConfig (the gRPC Network Management Interface — the modern alternative where the device streams updates over a TLS gRPC channel). Same metric names either way is the point: dashboards and alerts don't care which protocol a given switch happened to speak.

Where it runs. As a service with line-of-sight to the devices' management addresses. gNMI dials TLS with certificate verification on by default (system roots, or pin a private CA with ca_file); there is no skip-verify, and the lab-only plaintext opt-in is loudly logged. Credentials (SNMP communities, SNMPv3 passphrases, gNMI passwords) are referenced by name only and resolved at runtime — never written to config or git, never logged. A credential that resolves to nothing fails closed at startup.

How it ships data. It publishes to the bus on probectl.device.metrics; the control plane's device consumer lands the series in the time-series database. Infra it needs: the message bus (bus.mode: kafka → Kafka) (plus your TSDB, which the control plane already owns). Single-node dev: bus.mode: memory.

Deploy it.

probectl-device-agent -config /etc/probectl/device-agent.yml

Config template: deploy/agent/probectl-device-agent.example.yml. Metric catalog, the SNMP↔gNMI unification, FIPS notes, and how device interfaces correlate path and flow data: device telemetry.

eBPF host agent — probectl-ebpf-agent

What it observes. Network activity from inside the host's kernel, with zero instrumentation — no sidecars, no app changes, no SDK. (eBPF lets the kernel run small, sandboxed, pre-verified observation programs — the agent attaches them to the kernel's own networking events instead of capturing packets.) It captures every TCP connection a host makes or accepts (the L3/L4 flow — addresses and ports — plus the process and container behind it) and builds a live service map of who-talks-to-whom. It can additionally parse L7 (application-layer) calls (HTTP/1.1+2, gRPC, DNS, Kafka), including over TLS via library uprobes — kernel hooks attached to a user-space library's functions, here the TLS library, so data is visible before it is encrypted — and reading application plaintext is off by default and triple-gated (an enable flag plus a per-tenant consent that must name this agent's tenant plus a non-empty workload allowlist; host-wide capture is not expressible). It is observe-only — it loads no enforcing program and blocks no packet, a guarantee enforced by a build-failing test. It is not a CNI and not an inline IPS.

Where it runs. Linux only, on each host you want to see. It needs CAP_BPF + CAP_PERFMON on supported kernels — the two Linux capabilities that permit loading BPF programs and attaching to perf events, granted without root — and a BTF-exposing kernel (/sys/kernel/btf/vmlinux, mainstream from 5.8 — BTF is the kernel's embedded type catalog, which lets one compiled agent adapt to any kernel). The L4 flow plane needs no tracefs and no root: it attaches as a raw tracepoint, and the kernel-matrix job runs it as a non-root user with only that capability pair (ING-23). The Helm chart mounts tracefs only when l7Capture.enabled, for the L7 TLS uprobe path (DPR-345). Generic kernels older than 5.8 are unsupported by default; CAP_SYS_ADMIN is only an explicit legacy break-glass when the runtime probe confirms BTF plus BPF ring-buffer support but split caps cannot be granted. On macOS/Windows, run it inside a Linux VM. The shipped image is the live build; fixture-replay mode exists only for CI / no-kernel boxes.

How it ships data. It publishes to the bus on probectl.ebpf.flows. The shipped artifacts default to bus.mode: kafka. Infra it needs: Kafka. Control-plane consumers (topology, segmentation, NDR) read these flows live; raw flow-by-flow retention in ClickHouse is not wired yet, so there is no queryable per-flow history — only what the consumers derive. That retention edge is tracked in the canonical limitations table. The agent refuses plaintext Kafka unless you explicitly opt in for a lab.

Deploy it (VM / bare metal).

sudo deploy/agent/install.sh ./bin/probectl-ebpf-agent
$EDITOR /etc/probectl/ebpf-agent.yaml      # set tenant_id + bus brokers
sudo systemctl start probectl-ebpf-agent

The installer is air-gap friendly (it downloads nothing and never self-updates), creates a dedicated non-root probectl-agent system user, and installs the hardened systemd unit (ambient CAP_BPF+CAP_PERFMON, a default-deny syscall filter, namespace lockdown) shipped alongside it. The unit and its matching container seccomp profile (seccomp — the Linux syscall filter; the profile is the allowlist of kernel calls the process may make) live in deploy/agent/ (probectl-ebpf-agent.service, seccomp.json). Config template: deploy/agent/probectl-ebpf-agent.example.yml. The full plane — capture pipeline, the triple-gate, redaction modes, the kernel/uprobe compatibility matrix, and the build: eBPF host agent.

Removing it. Uninstalling leaves the policy channel's node identity directory (its private key and last certificate, usable until it expires, at most 24 h) on each node: revoke the collector (POST /v1/agents/{id}/revoke), then delete the directory — see eBPF host agent ("Uninstalling leaves the node identity behind").

Endpoint / last-mile DEM — probectl-endpoint

What it observes. The last mile that server-side canaries physically cannot see: a remote user's Wi-Fi link health, their local gateway, the ISP / last-mile path, and browser-session timings — and then the headline trick, it attributes a slowdown to the closest impaired layer (Wi-Fi → local LAN → ISP → wider network), so you can answer "is it us, or the user's Wi-Fi / ISP?". It runs on the user's own device, so data minimization is a hard rule (geolocatable identifiers — like the AP MAC, the Wi-Fi access point's hardware address, enough to place a household on a map — and public hop IPs are dropped before a sample is ever emitted), and it discloses exactly what it collects at startup.

Where it runs. Linux in the beta, on end-user devices (macOS and Windows builds compile but are not yet supported), no elevated privileges — it uses the OS's own traceroute/tracert and read-only Wi-Fi queries. Ship the single static binary via your MDM (mobile device management — the tool IT already uses to push software to managed laptops: Intune, Jamf).

How it ships data. It publishes to the bus on probectl.endpoint.results, tenant-keyed, flowing through the same pipeline as every other canary; it never phones home. Infra it needs: the message bus — Kafka in a fleet, or durable lightweight NATS JetStream. Register its collector identity and use the returned tenant bus namespace as described above. memory is process-local and cannot deliver from this standalone agent to the control plane, even on the same host. See endpoint DEM for the NATS configuration.

Deploy it.

probectl-endpoint -config /etc/probectl/endpoint.yml

The privacy model, the attribution engine, the per-OS collection matrix, and the /v1/endpoints fleet surface: endpoint DEM.

A producer-by-producer cheat sheet

At scale

Running one agent by hand is the learning path. A fleet is run by your orchestrator — probectl deliberately has no agent self-update channel (that would be a fleet-wide remote-code-execution primitive), so update authority stays with your tooling.

• Kubernetes — the eBPF host agent as a DaemonSet (Kubernetes's run-one-copy-on-every-node primitive — exactly the shape a per-host kernel observer needs). The supported chart is deploy/helm/probectl-agent. It declares the privilege contract in the manifest (drop all capabilities, add back only CAP_BPF/CAP_PERFMON, a seccomp profile, read-only root, the BTF and tracefs host mounts, resource limits) and renders fail-closed — no tenantID, or plaintext Kafka without an explicit opt-in, refuses to template. It also renders the Kyverno image-integrity policy by default so admission requires the digest and the keyless cosign signature from the tagged release.yml workflow before the privileged pod can start — and the install is two steps, policy first, so the very first pod is verified too (Image-integrity admission below). Disabling that policy requires an explicit admission.imageIntegrity.acceptedRisk note.

  cat > agent-values.yaml <<'EOF'
  tenantID: <tenant>
  agentID: <collector-id>
  bus:
    namespace: t-<tenant-slug>
    brokers: ["kafka.probectl.svc:9093"]
    tls: {existingSecret: probectl-bus-tls}
    sasl: {mechanism: scram-sha-512, existingSecret: probectl-bus-sasl}
  image:
    tag: "0.6.6@sha256:<digest>"
  EOF
  # 1. the image-integrity policy, alone; wait until Kyverno enforces it
  helm template probectl-agent deploy/helm/probectl-agent -n probectl -f agent-values.yaml \
    --show-only templates/image-integrity-policy.yaml \
    | kubectl apply --server-side --force-conflicts --field-manager=helm -f -
  kubectl wait --for=jsonpath='{.status.conditionStatus.ready}'=true \
    imagevalidatingpolicy/probectl-agent-image-integrity --timeout=2m
  # 2. the agent (the release adopts the policy from step 1)
  helm install probectl-agent deploy/helm/probectl-agent -n probectl -f agent-values.yaml
  

  agentID is the collector identity the tenant registered for these nodes (previous section) and bus.namespace the lane the registration printed; the chart refuses to render without the identity, because the control plane would reject every batch (DPR-051). Bus client auth is a Secret: bus.sasl.* (keys username/password by default) or bus.tls.clientAuth=true to present tls.crt/tls.key from bus.tls.existingSecret.

  Naming heads-up: the chart is named probectl-agent but it deploys the eBPF host agent (probectl-ebpf-agent image) — it is the per-node flow agent, not the synthetic probectl-agent binary. The eBPF agent is the one with a shipped Helm chart, systemd unit, and installer; the canary, flow, and device agents ship example configs and run as ordinary services or containers in your platform.

• VMs / bare metal — the installer. deploy/agent/install.sh installs a local binary (air-gap friendly: downloads nothing, never self-updates), creates the non-root system user, installs the hardened systemd unit, and writes a fail-closed sample config. Re-run it with a new binary to update.

• Staged fleet rollout. Move a fleet to a new version in waves (canary → early → main) from signed artifacts, with the agent registry verifying each wave and any straggler halting the train — so a bad version stops at a small canary instead of taking out the whole fleet. The model, the operator flow, and halt/resume semantics are in staged fleet rollout.

Image-integrity admission (Kubernetes)

The eBPF agent is privileged, so the agent chart does not stop at pinning a digest: it renders a Kyverno ImageValidatingPolicy, probectl-agent-image-integrity, that checks the image again at admission time, in the cluster, every time a pod is created (RTO-12). Kyverno 1.17 or later must already be installed: the policy is policies.kyverno.io/v1, which Kyverno serves from 1.17, and Kyverno 1.19 deprecates the kyverno.io/v1 ClusterPolicy (verifyImages) form the chart used to render. Against an older Kyverno the chart refuses to install rather than render a kind nothing enforces. In ELI5 terms: the chart writes down which photo is allowed in, and a bouncer (Kyverno) checks every pod's photo against it at the door.

What it enforces. For every Pod in every namespace Kyverno watches (and, through Kyverno's autogen rules, every DaemonSet, Deployment, Job, … template) whose container, initContainer or ephemeral-container image matches <image.repository>*, or one of admission.imageIntegrity.additionalImageReferences (default */ctlplne/probectl-ebpf-agent*) — each pattern compared with the raw image string and with its Docker-normalized form (ctlplne/x is docker.io/ctlplne/x), as the ClusterPolicy did:

• the reference must carry a digest — a tag-only reference is refused (does not have a digest);
• that digest must carry a cosign signature from the configured attestor — unsigned, or signed by any other key or identity, is refused (image signature not verified by the configured attestor, naming each image; Kyverno's ImageValidatingPolicy does not tell the two apart in its answer). The default attestor is keyless: the tagged ctlplne/probectl release.yml workflow's GitHub OIDC identity, verified through the Rekor transparency log;
• admission.imageIntegrity.failureAction: Enforce renders validationActions: [Deny]. Audit reports without blocking and is accepted only with an admission.imageIntegrity.acceptedRisk note;
• failurePolicy: Fail: a Kyverno that cannot answer (hung, crashed, partitioned) refuses the pod — even a correctly signed one.

Why two steps. Helm creates object kinds it does not know (an ImageValidatingPolicy) after the DaemonSet and never waits for Kyverno to load a policy, so a one-step install admits the first agent pods before their verifier exists. Helm hooks are not the fix: Helm deletes and re-creates a hook object on every upgrade — a window with no enforcing policy — and does not wait for a hook policy to become Ready either. So the policy is applied first (step 1 above), and the chart refuses to install or upgrade (readinessGate: lookup, the default) unless the exact policy that release renders — matched by its probectl.dev/policy-sha256 annotation — is already in the cluster, Kyverno reports it ready (status.conditionStatus.ready and its WebhookConfigured condition), and Kyverno's validating resource webhook serves that policy (/ivpol/validate/<policy>) with failurePolicy: Fail. Step 1 is idempotent: re-run it before any upgrade that changes image.repository or the attestor (the chart tells you when it has to). A plain image-version upgrade does not change the policy.

Upgrading from a chart that rendered the ClusterPolicy. Run step 1 as usual: it creates the ImageValidatingPolicy beside the old ClusterPolicy (different kinds, same name), so both verify until helm upgrade removes the ClusterPolicy the release no longer renders. There is no moment with neither.

A mirrored or air-gapped registry. The verified reference is derived from image.repository, so pointing the chart at your mirror also points the policy at it — a mirror can no longer bypass the check by having a different path. image.repository must name its registry host explicitly (registry.corp.example:5443/mirror/probectl-ebpf-agent, not mirror/probectl-ebpf-agent), because the reference a bare path normalizes to is docker.io/…, which the derived pattern would never match. If the mirror keeps the upstream signatures (copy them with cosign copy), nothing else changes. If it cannot reach Rekor, or re-signs what it mirrors after verifying upstream, give the chart the mirror's public key instead of the keyless identity:

image:
  repository: registry.corp.example:5443/mirror/probectl-ebpf-agent
  tag: "0.6.6@sha256:<digest>"
admission:
  imageIntegrity:
    publicKey: |            # or: --set-file admission.imageIntegrity.publicKey=mirror-cosign.pub
      -----BEGIN PUBLIC KEY-----
      ...
      -----END PUBLIC KEY-----
    ignoreTlog: true        # only for a signer that cannot upload to a transparency log

GitOps. Argo CD renders offline but advertises the cluster's API versions, so the lookup gate cannot see the live policy there: set admission.imageIntegrity.readinessGate=external and name the mechanism in externalOrdering. The policy carries argocd.argoproj.io/sync-wave: "-1", and Argo CD needs a health check for the policy kind, or it calls the policy Healthy the moment it exists and the DaemonSet's wave goes ahead. deploy/gitops/argocd/argocd-cm-kyverno-health.yaml ships one for policies.kyverno.io/ImageValidatingPolicy (and the capability-posture ValidatingPolicy). It is Healthy only when Kyverno reports status.conditionStatus.ready: true and a WebhookConfigured condition that is True, meaning Kyverno has registered the policy in its admission webhook. Anything else is Progressing, so the wave waits. Merge it into argocd-cm:

kubectl -n argocd patch configmap argocd-cm --type merge \
  --patch-file deploy/gitops/argocd/argocd-cm-kyverno-health.yaml

With the argo-cd Helm chart, put the same two keys under configs.cm. A health check sees only its own object, so it cannot notice Kyverno being scaled to zero later: the status keeps its last value. The admission-check CronJob below covers that. Flux's helm-controller performs a real install, so the default gate works once the policy is applied first (a separate Kustomization the HelmRelease dependsOn, with a health check).

Know your Kyverno. Kyverno removes its own webhooks when its admission controller is scaled to zero; from then on nothing in the cluster is verified, while the policy still reads Ready. The chart's gate refuses to roll out in that state, and between rollouts the chart watches for it too:

• The admission-check CronJob (admission.imageIntegrity.webhookCheck, on by default; every 5 minutes) runs probectl-ebpf-agent admission-check from the agent image itself — the same signed, digest-pinned image, so the policy verifies it and no other image is pulled. It reads the ImageValidatingPolicy and Kyverno's kyverno-resource-validating-webhook-cfg, and fails when the policy is missing, is not the one this release renders (policy-sha256), is not ready, or no failurePolicy: Fail webhook serves it. A failure leaves a Failed Job (the last three are kept) and one ERROR log line naming every problem:

  kubectl -n probectl get jobs -l app.kubernetes.io/component=admission-check
  kubectl -n probectl logs job/<failed-job>
  

  Alert on it with kube-state-metrics: kube_job_status_failed{namespace="probectl", job_name=~"probectl-agent-probectl-agent-admission-check-.*"} > 0. The check is observe-only: its ClusterRole is get on exactly those two objects, it uses a 10-minute projected token (no automounted one), and it never repairs anything — restoring Kyverno stays a human decision. An unreadable state (API error, RBAC removed) fails the Job too.

• Run Kyverno with several replicas, so a single eviction does not remove the webhooks in the first place.

Kyverno accepts the shipped policies (ImageValidatingPolicy, ValidatingPolicy) without a deprecation warning.

Proof. scripts/kyverno_admission_smoke.sh (CI job kyverno-admission) runs all of this on a real kind cluster with a pinned Kyverno: Kyverno accepts the shipped policies with no deprecation warning; the chart refuses to install before its policy exists; the two-step install is admitted; an unsigned digest, a digest signed by another key, a tag-only reference, an unsigned pod outside the chart, and an unsigned image in a mirror with a different path are each denied with the policy's message; the signed mirror image is admitted; a Kyverno that cannot answer refuses a signed image; the admission-check CronJob passes while Kyverno verifies and fails, naming the missing webhook, once Kyverno is scaled to zero; and the chart refuses to roll out while Kyverno's webhooks are gone. It signs with a cosign key pair generated for the run (the publicKey + ignoreTlog configuration above); admitting a real keyless-signed release digest pulled from GHCR waits on the release packages being publicly readable.

See also

• Getting started — zero to your first probe result, the "just try one" path.
• Installation — standing up the control plane the agents connect to.
• Configuration reference — every PROBECTL_* key and config field for every binary.
• Agent enrollment & rotation — the identity lifecycle in depth.