Configuration

This is the full reference for every knob probectl reads at startup. The short version: the control plane and every server-side feature read environment variables (all prefixed PROBECTL_); the agents read a YAML file (with the same env vars as overrides). An environment variable is a named string the operating system hands a process at launch — and because the control plane is stateless, these variables are the only place its behavior comes from. This page lists each variable, its default, and what it does — and it is the contract, so every row here is checked against the code.

How to read this page:

• A variable's default is what you get if you set nothing. The defaults are chosen so a fresh install boots in a safe, sovereign posture (no outbound calls, fail-closed on missing TLS/secrets) — probectl's standing security guardrails (see security/threat-model.md).
• A default of (none) means the value is empty/unset; the feature usually stays off until you give it one.
• Where it's read: the control plane resolves its config in internal/config/config.go (one Load function that reports every bad value at once and exits non-zero — a preflight checklist that reads all the gauges before refusing takeoff, so you never chase config errors one at a time). Each agent has its own loader (internal/agent, internal/ebpf, internal/flow, internal/device, internal/endpoint).

Conventions

• Control plane (probectl-control): environment variables, PROBECTL_ prefix. Listed in the next section.
• Agents: a YAML config file is the source of truth; the matching PROBECTL_* env vars override individual fields (handy for containers). Each agent's keys are in its own section below.
• Agent YAML schema: every agent config file starts with a top-level apiVersion (probectl.io/agent/v1, probectl.io/ebpf-agent/v1, probectl.io/flow-agent/v1, probectl.io/device-agent/v1, or probectl.io/endpoint/v1). schema_version: 1 is accepted as a compatibility alias. Unknown YAML keys fail startup, so a misspelled or retired safety knob is loud instead of silently ignored.
• Secrets are never hardcoded, logged, or placed in URLs/query strings. Sensitive values at rest are sealed with envelope encryption — each value is encrypted with its own data key, and all those data keys are encrypted by one master key (the envelope key): individual lockboxes whose keys live in a single safe. Any credential in this document may be a secret reference (e.g. vault:…) instead of the raw value — see Secrets integration.

Control plane (probectl-control)

The control plane is the brain: it serves the API/UI, accepts agent connections, runs the alerting/incident/correlation engines, and talks to the datastores. It is stateless — all durable state lives in Postgres/ClickHouse/the TSDB — so every behavioral choice it makes comes from these environment variables, read once at boot. The table below is the base set every deployment uses; the feature-specific sections that follow add more.

Subcommands: probectl-control [serve] (default), probectl-control migrate (apply database migrations and exit), probectl-control version, and probectl-control gen-cert [--if-missing] [dir] — a convenience that writes a self-signed tls.crt/tls.key/ca.crt for an HTTPS quickstart (PROBECTL_CERT_HOSTS, default localhost,127.0.0.1, sets the certificate's host names; --if-missing preserves a complete existing bundle and rejects a partial one rather than rotating it; production brings its own CA-issued cert). probectl-control stage-binary <destination> is the internal, shell-free distroless init-container helper used by the Helm backup/restore Jobs. The other subcommands are covered with their features: agent-ca init|renew|export, enroll-token, and revoke-agent (agent transport and enrollment — agent/enrollment.md), scim-token (SCIM, below), mcp-stdio and mcp-token (MCP server, below), preflight (the storage-encryption preflight — hardening.md), support-bundle (supportability, below), backup-seal / backup-open (sealed backups — Tenant lifecycle, below), backup-rewrap and envelope-rewrap (deployment-envelope key rotation — hardening.md), bgp-analyzer (the optional Python analyzer bridge — BGP routing intelligence, below), and replay-deadletter (re-ingest dead-lettered records — ops/dead-letter-replay.md).

A note on the defaults: the listen address is :8080, but the database URL — the one postgres:// or postgresql:// connection string carrying host, user, password, database, and TLS mode — is required and has no default credential. PostgreSQL keyword/value strings are rejected so credentials can never bypass config-log and support-bundle redaction. Production URLs should verify the server certificate with sslmode=verify-full and an sslrootcert trust anchor; the multi-tenant/regulated profiles require it. HSTS — the response header that tells browsers to only ever reach this host over HTTPS — is on. Shipped Helm and Compose deployments set the TLS cert/key pair below so the process serves HTTPS directly, including behind an ingress.

Invalid values fail fast: probectl-control reports all configuration problems at once and exits non-zero. The database password is redacted from logs.

Tenant-owned tables are protected by Postgres Row-Level Security (RLS): the database itself filters every row by the requesting tenant, so even a buggy or hostile SQL query cannot read another tenant's rows — the isolation does not depend on application code getting a WHERE clause right. The PROBECTL_DATABASE_URL role must be able to assume the least-privilege probectl_app role (a superuser always can; otherwise run GRANT probectl_app TO <login_role>), which internal/tenancy assumes per transaction so isolation holds regardless of how the control plane authenticated. It must also be able to assume — never inherit — probectl_provider, the role the provider plane, the audit and WORM streams, agent enrollment and tenant lifecycle run as: GRANT probectl_provider TO <login_role> WITH INHERIT FALSE, SET TRUE. Boot refuses a login that cannot assume it, or that inherits it (its provider-only policies would then apply outside a provider transaction). See architecture.md.

HTTP endpoints

Every response carries an X-Request-Id (honoring an inbound one) and the security headers Strict-Transport-Security (when enabled) and X-Content-Type-Options: nosniff. The versioned resource routes under /v1 are documented in Resource API & CLI below.

Error envelope

All errors share one JSON shape and a stable domain-error → HTTP mapping:

{ "error": { "code": "not_found", "message": "…", "request_id": "…" } }

error.code is the client branching contract: it is enumerated as ErrorCode in /openapi.json. message is for humans and may be localized. request_id is optional, but when present the CLI prints it next to the code so operators can join a user-facing failure to server logs.

Specialized stable codes include writer_unavailable, quota_exceeded, tenant_suspended, tenant_offboarded, license_read_only, approvals_disabled, blast_radius_exceeded, blast_radius_unknown, and not_proposed; the OpenAPI ErrorCode enum is the authoritative list.

Transport security

probectl never wants a plaintext channel exposed to the network. The shipped deployment path uses app-terminated TLS:

• App-terminated TLS — set PROBECTL_TLS_CERT_FILE + PROBECTL_TLS_KEY_FILE, and the control plane serves HTTPS only (TLS 1.2+, prefer 1.3; plaintext is refused).
• Ingress compatibility mode — a legacy/operator-owned deployment can leave them unset only with the explicit PROBECTL_ALLOW_PLAINTEXT_HTTP=true acknowledgment behind its TLS ingress. Shipped Helm and Compose do not use this mode because it leaves the ingress-to-process hop unencrypted.

All TLS and crypto policy lives in internal/crypto; a CI guard (scripts/check_crypto_imports.sh) forbids crypto-primitive imports elsewhere so a FIPS 140-3 validated module can be swapped in. At-rest secrets use the envelope helper (a per-record data key wrapped by a KMS/HSM-pluggable KEK; the dev StaticKeyProvider reads PROBECTL_ENVELOPE_KEY).

Agent transport

This is how agents talk to the control plane, and it is locked down by design. The transport is gRPC — a binary remote-procedure-call protocol over HTTP/2, carrying Protobuf (compact, typed) messages — and it runs only when PROBECTL_AGENT_GRPC_ADDR and all three PROBECTL_AGENT_TLS_* files are set (address + server cert + server key + the CA that signs client certs). The service is probectl.agent.v1.AgentService, and it is mutual-TLS only (RequireAndVerifyClientCert): in mutual TLS both sides prove their identity with certificates — the server to the agent, and the agent back to the server — so the agent must present a client certificate, and its tenant and id are read out of that certificate's identity (spiffe://probectl/tenant/<t>/agent/<a>), never from the request body. So even a misbehaving or malicious agent can only ever write to its own tenant — the identity is cryptographic, not self-asserted. Populate PROBECTL_AGENT_TLS_CA_FILE (the client-cert CA pool) with probectl-control agent-ca export <file>, which writes the public agent-CA bundle (root + intermediate, no key). Generate dev mTLS material with the internal/crypto CA helpers. The .proto lives under proto/probectl/agent/v1/; regenerate Go with make proto (tools via make proto-tools).

Version-skew policy. At registration the control plane rejects agents outside the supported version window, so a rolling upgrade never admits an incompatible agent. See lifecycle.md.

A rejected agent gets a gRPC FailedPrecondition ("upgrade required"); a dev/unpinned build (0.0.0-dev) on either side skips the check.

Agent self-metrics

Every shipped collector process exposes Prometheus text at GET /metrics: probectl-agent, probectl-flow-agent, probectl-device-agent, probectl-ebpf-agent, probectl-endpoint, and probectl-bmp-listener. These are process-wide RED/USE signals only—counts, errors, bounded queue depth, and publish time. They never contain tenant ids, probe targets, device addresses, or telemetry payload fields.

Each process uses a distinct loopback-only default so the collectors can share one host. Replace <PREFIX> below with PROBECTL_AGENT, PROBECTL_FLOW, PROBECTL_DEVICE, PROBECTL_EBPF, PROBECTL_ENDPOINT, or PROBECTL_BMP:

Core series are probectl_agent_collections_total (probe or collector batches), probectl_agent_published_total (results/batches accepted by the output transport), probectl_agent_errors_total, probectl_agent_buffer_depth, and probectl_agent_publish_latency_seconds. Build, uptime, Go-runtime, and process series come from the same dependency-free registry.

published_total is acceptance, not delivery. An asynchronous producer returns from Publish long before the broker answers, so on the agents that publish onto a bus that counter means "the transport took it". Where the transport can report what it then failed to deliver, two more series appear and an operator should alert on them:

All three describe records that published_total has already counted, which is why a climbing published_total alongside a climbing failed_total or no_subscriber_total is data loss rather than throughput. A transport that cannot report undelivered records exposes neither series, rather than a zero that would read as healthy. The canary agent needs neither: it removes a result from its buffer only after the control plane's accepted count, so its published_total is verified delivery. The eBPF agent additionally ties readiness to it (probectl_ebpf_publish_degraded). Prometheus's job and instance target labels identify the binary/host; tenant identity is intentionally absent.

probectl-agent

The canary agent is the worker that actually runs the probes (ping, TCP, DNS, HTTP, …). Unlike the control plane, its primary config is a YAML file (-config, or the path in PROBECTL_AGENT_CONFIG), bounded to 1 MiB before YAML decoding — see deploy/agent/probectl-agent.example.yml. Crucially, the agent does not configure its own tenant or id: those come from its mTLS client certificate (above), so you can't accidentally point an agent at the wrong tenant by editing a file.

agent.labels is a bounded map of operator-declared local placement metadata. Set at least region and site when you want the native Targets coverage cockpit to group the vantage. Labels travel only on the authenticated mTLS registration RPC and are stored inside the agent's forced-RLS tenant row; they never select a tenant. Keys are normalized to lower case (maximum 32 labels, 63-byte keys, 128-byte values). probectl never infers these values from an IP address and never calls a geolocation service.

agent:
  capabilities: ["icmp", "tcp", "dns", "http"]
  labels:
    region: eu-west
    site: dub-1

A handful of env vars override individual YAML fields — useful in containers where mounting a full file is awkward:

Results buffer to disk (buffer.dir, bounded by max_records, default 10000) while the control plane is unreachable and drain on reconnect (at-least-once delivery). Probing keeps running regardless of connectivity, so a control-plane outage never blocks measurement — the agent just queues and catches up.

The buffer is also bounded by on-disk bytes (buffer.max_bytes, RESIL-009) to pre-empt ENOSPC: 0/unset uses the built-in default of 256 MiB; a negative value disables the byte bound (records-only). Each frame costs 4 header bytes plus the payload, so the footprint of a full record-capped buffer is roughly max_records × (4 + avg_payload_bytes). Enqueue fails closed (sheds the newest result, counted as dropped) when either the record or byte cap would be exceeded, and the agent logs a WARN when the buffer crosses 90% of either bound.

Reconnect drains are also chunked (buffer.drain_max_records, default 500; buffer.drain_max_bytes, default 8388608; buffer.drain_pace, default 150ms). In plain English: after an outage, the agent does not pick up a full 10,000-record / 256 MiB queue and shove it through one long stream. It lifts a bounded FIFO prefix, streams that prefix, removes only the accepted prefix, logs backlog records/bytes plus oldest result age, then jitters the next catch-up chunk so a fleet does not stampede the control plane in lockstep.

Scheduled canary identity and cadence evidence

Every entry under canaries: accepts these common fields:

test_id is evidence attribution, not remote control. When present, the agent stamps probectl.test.id and its effective probectl.test.interval_seconds onto every result after the probe returns. The control plane can then compare exact observed rounds with the server definition on Targets → Owned-vantage coverage. Duplicate or malformed IDs fail agent startup; an absent ID remains valid for older/local-only configurations but its cadence receipt stays unknown rather than guessing from a shared type/target.

The control plane does not read, push, rebalance, or verify the current local YAML. Even after an exact result, the receipt therefore says current assignment is unverified. Server-driven config remains deliberately unimplemented under adr/config-push.md.

Result pipeline

This is the path every measurement takes from an agent to a queryable metric, and two env vars decide how heavy that pipeline is: PROBECTL_BUS_MODE (the message bus — the queue that decouples ingest from storage, so a slow consumer never stalls an agent) and PROBECTL_TSDB_MODE (the time-series writer; a TSDB is a database specialized for timestamped measurements and the range queries dashboards make over them). The memory defaults make a single binary work with zero external dependencies; switch them to kafka / prometheus when you outgrow that.

A streamed result flows agent → gRPC StreamResults → control-plane ingest → result bus (probectl.network.results, Protobuf) → consumer → time-series writer. The agent sends the canonical OTel-aligned result (proto/probectl/result/v1); the control plane re-stamps the tenant and agent id from the verified mTLS certificate before publishing, so a result is always attributed to the sending agent's tenant regardless of payload contents — the tenant boundary is cryptographic, never self-asserted. The bus key is the tenant_id.

PROBECTL_BUS_MODE selects the bus, and there are three of them. Two of them used to be described by one word, "lightweight", which is how a deployment could end up on a transport that loses everything it holds when the process exits (DPR-119). They are named separately now:

In memory mode, Flush waits for current subscriber handlers to finish before the agent stream is ACKed. That makes the volatile path at-least-once with respect to the in-process store/DLQ handlers, but it is still not a broker log: a crash loses what it held, which is why it is not a production transport.

The memory bus is also a live, in-process-only pub/sub: it delivers only to a consumer running in the same process and keeps no backlog. The standalone collector binaries — probectl-flow-agent, probectl-device-agent, probectl-ebpf-agent, probectl-endpoint and probectl-bmp-listener — have no such consumer, so a collector left on memory mode (the shipped default) would discard every batch it publishes. Each therefore logs a loud WARN at boot when its PROBECTL_*_BUS_MODE is memory/unset, and any batch dropped for want of a subscriber is counted at probectl_agent_bus_no_subscriber_total (and, on the control plane's own in-process bus, probectl_bus_memory_no_subscriber) rather than reported as a silent publish success (ING-19). Memory mode stays valid for the embedded single-process/test case; point a standalone collector at a durable nats/kafka bus (with PROBECTL_*_BUS_BROKERS).

In nats mode each topic is a JetStream stream of its own (so a siloed tenant's lane stays its own lane), each consumer group is a durable consumer whose position survives restarts and rolling upgrades, a message is acknowledged only after its handler returns nil, and Flush waits for the server's acknowledgment — the same delivery contract Kafka carries. Set PROBECTL_BUS_BROKERS to one or more nats:///tls:// URLs. The transport policy is identical to Kafka's: TLS and credentials are required, and a bus without TLS is refused unless the explicit dev-only PROBECTL_BUS_ALLOW_PLAINTEXT=true flag is set. "Lightweight" never meant "plaintext". PROBECTL_TSDB_MODE selects the writer: memory (default; in-process) or prometheus remote-write to PROBECTL_TSDB_URL (Prometheus with --web.enable-remote-write-receiver, or VictoriaMetrics; use an https:// URL for TLS in transit). Each probe emits probectl_probe_success, probectl_probe_duration_seconds, and one probectl_probe_<metric> per custom metric, labeled tenant_id, agent_id, canary_type, server_address and, when the canary carries a server definition id, test_id (DPR-066: the address is the host, so two definitions against one host need the id to stay separate series). The canonical signal→OTel mapping is in otel-mapping.md.

Volatile stores lose all telemetry on restart (PLAT-01/RTO-04)

The memory modes above — the shipped defaults for the bus, the TSDB and every telemetry store (PROBECTL_PATHSTORE_MODE, PROBECTL_FLOWSTORE_MODE, PROBECTL_OTELSTORE_MODE, PROBECTL_EBPFSTORE_MODE, PROBECTL_ENDPOINTSTORE_MODE) — keep their data in process memory only. Every synthetic result, metric series, path, flow, OTLP trace/log, eBPF edge and endpoint event is erased on any restart, upgrade, OOM or node drain, and alerts then evaluate only data since the last restart. The default single-binary install runs this way so a cold start needs zero external dependencies; it is a development/test posture, not a durable one.

The control plane never runs volatile silently:

• it logs a WARN at boot naming every memory-backed plane;
• /readyz lists them under volatile_stores for an authenticated caller (the node stays ready — the request path is up; an anonymous probe gets status only, AUTHZ-25);
• /v1/diagnostics reports degraded with the named finding readiness.volatile_stores;
• probectl-control preflight --strict exits non-zero.

For any deployment whose history must survive, configure durable modes: a nats (JetStream) or kafka bus, prometheus TSDB, and ClickHouse-backed stores. Production-like profiles (PROBECTL_DEPLOYMENT_PROFILE other than single) refuse memory modes outright. To run volatile deliberately on a dev/test single-binary deployment and mark the data-loss behavior as understood — which downgrades the diagnostics finding and lets preflight --strict pass — set:

ICMP test

The icmp canary measures echo loss, latency, and jitter to a target (IPv4 or IPv6). Configure it per-canary under canaries: (see probectl-agent.example.yml). The schedule interval and reply timeout are canary fields; the rest are params:

dscp (here and on every canary that takes it) sets the Differentiated Services Code Point — the priority field in the IP header that asks routers to treat a packet as a given traffic class — so you can measure the same path your real traffic class rides, e.g. probing with the voice marking to see what voice sees.

It emits probectl_probe_loss_ratio, probectl_probe_rtt_{min,avg,max,stddev}_ms, probectl_probe_jitter_ms, and probectl_probe_packets_{sent,received}. A probe with 100% loss reports success=false (target unreachable); partial loss is a success with a non-zero loss ratio. Continuous mode records a per-second drop-timing record as result attributes (icmp.dropped_seqs, icmp.drop_send_offsets_ms) — carried as OTel attributes, not TSDB labels, so they don't widen cardinality.

Privileges. By default the agent uses unprivileged datagram ICMP (IPPROTO_ICMP), which on Linux requires the agent's group to be within net.ipv4.ping_group_range (e.g. sysctl -w net.ipv4.ping_group_range="0 2147483647"). Alternatively grant raw-socket capability (setcap cap_net_raw+ep /usr/bin/probectl-agent, or run with CAP_NET_RAW) and set privileged: "true". The canary tries the preferred socket and falls back to the other; if neither can be opened it returns an internal error (the probe is not silently reported as loss).

TCP & UDP tests

The tcp and udp canaries are agent-to-server probes. Configure a target of host:port (or a host with params.port). Both accept count and dscp.

The tcp canary measures connect latency + reachability (a connect-based, unprivileged equivalent of a TCP-SYN test): it establishes a connection and times the handshake, emitting probectl_probe_connect_{min,avg,max,stddev}_ms, probectl_probe_jitter_ms, and probectl_probe_loss_ratio (failed connects = loss; all-failed = success=false).

The udp canary is an echo round-trip probe: it sends token-tagged datagrams and matches the echoes, emitting probectl_probe_rtt_* + loss. It needs a target that echoes (a UDP echo service, or a probectl agent-to-agent responder); a non-echoing target reports as 100% loss. params.payload_bytes (≥10) sets the datagram size.

Voice/RTP tests

The voice canary streams real RTP packets (RTP is the protocol that carries live audio in VoIP calls) at codec cadence to an echoing target and scores the path: MOS + R-factor (simplified ITU-T G.107 E-model), RFC 3550 jitter, loss, and a one-way delay estimate. MOS is the 1–5 "how did the call sound" scale; the R-factor is the 0–100 engineering score it is derived from — both computed from measured loss/delay/jitter by the E-model, not from a human listener. target is host:port. Parameters: codec (g711 default, g729), duration_seconds (1–10, default 3), dscp (default 46/EF). The model variant and the one-way-estimate method ride the result attributes — a computed MOS is never presented as a measured listening score. See docs/voice.md.

DNS tests

The dns canary queries DNS and reports resolution time, the answer, and an optional DNSSEC verdict. The target is the query name. Parameters:

server defaults by transport: the first nameserver in /etc/resolv.conf (or 1.1.1.1:53) for udp/tcp, 1.1.1.1:853 for DoT (DNS over TLS — the classic query inside an encrypted TLS session), and https://cloudflare-dns.com/dns-query for DoH (DNS over HTTPS — the query as an HTTPS request). DoT verifies the resolver's TLS certificate (TLS 1.2+); DoH posts an RFC 8484 application/dns-message query over HTTPS.

In resolver mode the canary emits probectl_probe_dns_query_ms (round-trip) and probectl_probe_dns_answers (answer count), with dns.rcode and a compact dns.answer summary as attributes. The probe is success=false on a non-NOERROR rcode or an empty answer.

DNSSEC adds cryptographic signatures to DNS, so an answer can be proven to come from the zone's owner, untampered. With dnssec: "true" the canary requests DNSSEC records (the DO bit) and validates the zone's RRSIG over the answer against the zone DNSKEY, then chains that DNSKEY to a trust anchor through a DS — it does not trust the resolver's AD bit (the resolver's own "I checked" flag, which a misbehaving resolver could simply set), and it never trusts a DNSKEY carried in the response's own Answer/Additional section (attacker-controlled: a forged answer would ship a matching self-made key beside it). The DNSKEY is fetched with a separate query to the signer zone. The verdict lands in the dns.dnssec attribute:

• secure — every answer RRset carries a valid, in-window signature and the signing DNSKEY RRset chains to a configured trust anchor: a key in the set is pinned by an anchor DS and self-signs the keyset (RFC 4035). The anchor is the baked IANA root DS by default, or an operator-pinned DS (trust_anchor).
• indeterminate — every answer RRset carries a valid, in-window signature from the zone's DNSKEY, but that keyset does not chain to a trust anchor. This is an honest "signed, but not provably authentic" signal: it does not claim secure, because a forged answer signed by an attacker's own key (served by a malicious resolver) would verify identically. The probe still succeeds.
• insecure — the zone is unsigned (no RRSIG).
• bogus — RRSIGs are present but some answer RRset does not verify (tampered, expired, or wrong/absent key). A bogus result fails the probe.

probectl_probe_dns_dnssec_secure (1/0) is 1 only for secure; indeterminate reports 0. Every answer RRset must verify — a single valid RRSIG over one RRset does not vouch for the rest of the answer.

In trace mode the canary performs an iterative delegation walk from the root hints, following NS/glue referrals down to the authoritative server (UDP, capped iterations, with a recursive fallback when a referral ships no glue). The walk starts from the baked IANA roots unless root_hints names your own — point it at a private or internal root so the trace runs on an isolated or air-gapped network with no route to the public roots. It emits probectl_probe_dns_query_ms (total walk time) and probectl_probe_dns_trace_hops, with the delegation chain in the dns.trace attribute. DNS-exfiltration detection and open-data baselines are out of scope for this probe (they live in the NDR and open-data features).

HTTP server tests

The http canary measures HTTP(S) availability with a full response-time breakdown and captures TLS handshake details for the TLS-posture plane (see TLS / certificate observability below). The target is the URL. Parameters:

A word on the last row: SSRF (server-side request forgery) is the attack where someone defines a "test" that makes your agent fetch an internal-only address — the cloud metadata service, a loopback admin port — and report back what it found. That is why every canary ships the private-target deny-list, why it checks the resolved IP at dial time (a DNS name that later re-resolves to something internal is caught too), and why lifting it is a per-test, admin-only, audited act rather than a global switch.

HTTP canaries ignore ambient HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment settings. A proxy is another network target that could resolve or fetch private space on the agent's behalf, so the guarded HTTP transport goes direct unless probectl later adds an explicit tenant-scoped proxy setting that is checked by the same SSRF guard.

expect_status is a comma list of exact codes (200), classes (2xx), and inclusive ranges (200-204); a response outside the set is success=false (the status is still reported). The probe emits the timing breakdown as metrics — probectl_probe_http_dns_ms (resolution), probectl_probe_http_connect_ms (TCP connect), probectl_probe_http_tls_ms (TLS handshake), probectl_probe_http_ttfb_ms (time to first byte), and probectl_probe_http_total_ms — plus probectl_probe_http_status, probectl_probe_http_content_bytes, and probectl_probe_http_throughput_kbps. A phase that does not occur (no DNS for an IP target, no TLS for http://) is omitted rather than reported as zero. The resolved server IP is captured as the network.peer.address attribute, which correlates the result to path/traceroute data for the same destination.

TLS capture. On HTTPS the canary records the negotiated tls.protocol.version and tls.cipher, the leaf certificate's tls.server.{subject,issuer,not_before,not_after,san}, the chain shape (tls.server.chain), and a probectl_probe_http_tls_cert_expiry_days metric (negative once expired). It verifies the chain itself (hostname + trust, honoring ca_file) after capturing the certificate, so the handshake details are recorded even when the certificate is invalid or expired — an invalid cert fails the probe but its details are still attached. This means operators do not need an insecure mode to observe certificate failures. The historical insecure_skip_verify=true probe parameter is always rejected, including for callers holding the legacy test.insecure_tls permission; likewise, security.allow_insecure_skip_verify: true is rejected when an agent configuration loads. Use ca_file when the endpoint is valid under a private CA. probectl performs no TLS posture analysis here (issuer trust, weak-cipher/expiry policy, CT) — that is the TLS / certificate observability feature below, which consumes these captured fields.

Browser / transaction tests

The browser canary has two deliberately different meanings. An ordinary probectl-agent defaults to browser.driver: http: the script runs as real HTTP requests with a cookie jar, status/text assertions, per-step timings, and a request waterfall, but no DOM is rendered. The probectl-browser-agent image sets browser.driver: browser and executes the packaged Playwright worker for DOM/paint timings and visual screenshots. A test states which semantics it requires; an agent configured for the other driver rejects it, so a rendered test can never quietly become a cheaper HTTP check.

The target is the default start_url. Parameters:

The matching agent YAML is:

# ordinary probectl-agent
browser:
  driver: http

# dedicated probectl-browser-agent image
browser:
  driver: browser
  worker:
    command: node
    path: /worker/worker.mjs
    step_timeout: 15s

Rendered mode is fail-closed: configuration loading verifies the command and worker file before the agent connects. The worker opens no listener; the Go agent sends one script over the child process's stdin and reads one result from stdout. Its navigation, redirects, and page subresources pass through the same resolved-address private-target policy as the HTTP driver. The browser image is shipped in releases and air-gap bundles, with an opt-in Compose profile and an opt-in Helm DaemonSet described in browser-synthetic.md.

The spawned worker receives two internal protocol variables. Operators do not set these on the agent process: the Go driver derives and overwrites them for each child from the validated YAML/test definition above. They are listed because this page inventories every startup variable, including private agent-to-worker contracts:

Example CLI creation:

probectl test create \
  --name login-browser \
  --type browser \
  --target https://app.example/login \
  --interval 60 \
  --param browser_driver=browser \
  --param 'script={"name":"login","start_url":"https://app.example/login","steps":[{"action":"goto"},{"action":"assert_status","status":200}]}'

Browser results emit probectl_probe_transaction_total_ms, probectl_probe_transaction_steps, probectl_probe_transaction_resources, probectl_probe_transaction_failed_steps, and one probectl_probe_transaction_step_<n>_duration_ms metric per step. Step names, actions, success flags, and failure details ride as attributes (browser.step.<n>.*) so they are visible in /v1/results/latest and the web result modal without becoming high-cardinality TSDB labels. See browser-synthetic.md.

Agent-to-agent tests

An agent-to-agent (A2A) test measures between two registered agents, brokered by the control plane. The control plane assigns roles (one agent responds, opening a short-lived listener; the other initiates), rendezvouses the responder's endpoint to the initiator, and hands each agent its task when it polls (PollCoordination / ReportEndpoint). The measurement is TWAMP-lite — TWAMP is the standard two-way active measurement protocol; "lite" here means its four-timestamp echo idea without the full session protocol: the initiator timestamps each probe (T1), the responder stamps receive/send (T2/T3) and echoes, and the initiator stamps receive (T4), yielding round-trip (probectl_probe_rtt_*) plus forward and reverse one-way delay (probectl_probe_forward_avg_ms, probectl_probe_reverse_avg_ms). The responder also reports forward-direction delivery (probectl_probe_packets_received, probectl_probe_loss_ratio), so both agents and both directions are observed.

Enable participation in the agent's a2a block: enabled: true, advertise_host (the address peers use to reach this agent's responder), poll_interval (default 2s), responder_ttl (default 15s), and allow_private_targets (default false; when false the initiator refuses a responder endpoint on a private/loopback/metadata address so a compromised peer cannot aim it at internal hosts — set true only when agents legitimately measure each other across a private network). The default is enabled: false: an agent opens no A2A UDP/TCP listener unless the operator explicitly opts it into this deployment policy. For each brokered session, the control plane gives both agents a random session id over the existing agent mTLS channel; A2A probe and reply frames are HMAC-SHA256 signed through internal/crypto with that per-session key. A responder ignores short, unsigned, tampered, or wrong-session frames and only counts authenticated probes.

a2a.enabled is permission to participate, not a schedule. Do not add type: a2a under canaries:: A2A is not registered in the single-agent interval scheduler. Start a pair or mesh through the tenant-scoped A2A API/CLI; the control-plane broker then sequences responder → endpoint report → initiator. See the A2A coordination ADR for why this is a separate execution path.

Caveats (document for production):

• NAT/firewall. The responder advertises advertise_host; behind NAT this must be a reachable address and the responder's ephemeral port must be reachable from the initiator. Auto-detection picks a non-loopback IPv4 — set advertise_host explicitly when that is wrong.
• Clocks. Forward/reverse one-way delays assume the two agents' clocks are synchronized (exact within one host; use NTP across hosts). Round-trip is clock-independent.

Pair sessions and full site meshes are brokered in-memory and started through the tenant/RBAC-scoped A2A APIs (POST /v1/a2a/sessions and POST /v1/a2a/mesh). The mesh API accepts site-labeled agents, binds tenant identity from the authenticated caller, schedules every directed site pair, and contributes a site-to-site topology overlay.

Path discovery

The path engine (internal/path) is the traceroute brain — it runs Paris-style traceroutes (ICMP and TCP) and merges per-flow traces into one multi-path picture; see architecture.md. "Paris-style" matters because of ECMP — routers spreading traffic across several equal-cost paths by hashing packet header fields. A naive traceroute varies those very fields probe to probe, so its hops come from different paths stitched into a nonsense route; Paris holds the hashed fields constant so each trace coherently follows one path, and varies them deliberately to enumerate the others. It also captures MPLS labels (the tags carriers use to switch packets through label-tunnels) where routers expose them. A full per-hop trace needs raw sockets: grant CAP_NET_RAW (setcap cap_net_raw+ep, or run privileged) to capture the intermediate hops + MPLS labels. Without it, only the destination is discovered. CI exercises the full raw-socket path in the path-raw-live lane with a multi-hop Linux namespace fixture, so this path is not merely documented as a privileged local option.

Where the discovered hops/links are stored is a control-plane choice:

Each stored discovery is an immutable history round. The Path UI reads a maximum of 50 rounds by default (the API hard ceiling is 100) and can request up to two exact opaque round_id values for a stable comparison link. Those IDs never select a tenant: the authenticated session's tenant_id and the selected test's target are mandatory storage predicates, so a copied cross-tenant or cross-target ID returns an empty history.

Each new round stores a closed measurement_fidelity receipt with the actual socket acquisition mode and hop-visibility capability. Today its timing_source is application_monotonic, with kernel and hardware timestamping explicitly false. A raw ICMP path reports full-hop acquisition; an unprivileged datagram fallback reports destination-only. The same receipt is returned by latest/history API calls and generic CLI JSON:

probectl test path <test-id>
probectl test path-history <test-id>

Rounds written before this receipt existed omit it. The native Path surface labels those rounds Unknown · legacy snapshot and does not infer precision. No external service, packet-capture process, or hardware database participates in the receipt.

BGP routing intelligence

BGP is the protocol networks use to tell each other which IP blocks (prefixes) they can reach — the internet's routing gossip — so a wrong announcement elsewhere can silently hijack or blackhole your traffic without touching your equipment. The BGP plane watches for exactly that in two ways: a Python analyzer (analyzer/) plus a Go bridge (internal/bgp) for public collectors, and a direct BMP listener for routers that can stream their own BGP view; see architecture.md. The analyzer ingests public collector data — MRT files (the archive format for routing-table dumps, e.g. RouteViews/RIPE RIS) and RIS Live (RIPE's websocket firehose of route updates as they happen) — and emits probectl.bgp.events:

python -m probectl_analyzer --config config.json --mrt rib.mrt        # RouteViews/RIS dump
python -m probectl_analyzer --config config.json --replay cap.jsonl   # recorded RIS Live
python -m probectl_analyzer --config config.json --ris-live           # live RIS Live websocket

The JSON config is per tenant (tenant_id is required — every event carries it, and the bridge rejects any event without one):

RPKI is the cryptographic registry recording which network is allowed to originate which prefix; a validator (rpki-client, Routinator) distills its signed objects into VRPs — validated prefix→origin records — and RFC 6811 validation compares each seen announcement against them, labeling it valid, invalid, or not_found. With no VRP source configured, everything is honestly unknown.

The shipped probectl-control bgp-analyzer sidecar starts that Python command, tails its JSON Lines, checks every payload tenant against the trusted tenant_id in the mounted config, and republishes each as the canonical probectl.bgp.v1.BGPEvent protobuf onto the bus (topic probectl.bgp.events, keyed by tenant). Event types: origin_change (old/new origin + AS path), possible_hijack, possible_leak, rpki_invalid; each carries an RPKI status (valid / invalid / not_found / unknown), a severity, and a confidence — they are signals, not actions — probectl never acts on routing. MRT dumps are stream-processed (no full RIB in memory); a down RPKI/collector source degrades gracefully rather than breaking the plane. RouteViews/RIS are open data — their AUP/provenance matters for MSP/commercial resale, not for private development or single-tenant OSS use.

The sidecar is off by default and requires Kafka because a separate process cannot share the control plane's in-memory bus. It has no listener. Live mode restarts after crashes with exponential backoff capped at 30 seconds; finite MRT/replay jobs exit after one successful run. The child Python process does not receive Kafka/DB credentials — the Go bridge alone owns the bus connection.

Helm renders this with no Service and an explicit egress NetworkPolicy: a one-replica Deployment for ris-live, and — because an MRT dump or a recorded replay is a finite artifact that must not be re-processed on every container restart — a one-shot Job per analyzer configuration for mrt/replay — its name carries a digest of the image, source, config and bus settings, so an unrelated upgrade does not re-run the artifact and a new one gets a new Job (bgpAnalyzer.job.*, DPR-059). Compose's bgp-analyzer profile replays a recorded RIS fixture end to end for evaluation.

For operators who run routers that export BMP (BGP Monitoring Protocol), run probectl-bmp-listener next to the routing fabric. It serves mTLS only: every router/collector peer presents a SPIFFE-style client certificate (spiffe://probectl/tenant/<tenant>/bmp/<router-id>) issued through the existing collector/agent registry. The listener derives the tenant from that verified certificate and checks its exact tenant/router/SPIFFE/serial tuple against the local registry instead of trusting any BMP payload field. A CA-valid self-issued or agent-plane certificate is refused. Route-monitoring updates are decoded, the peer is recorded in an in-process tenant-scoped peer inventory, and each observed prefix is published to the same probectl.bgp.events topic as the public-collector analyzer. The setup on-ramp is the same tenant-bound collector registration used by the other bus publishers: Admin & Settings > Agents > Register collector > BGP or probectl bgp setup --body '{"token":"pjt_...","plane":"bgp","hostname":"rrc00"}'. The response includes source_type: bmp, PROBECTL_BMP_COLLECTOR, and the startup command probectl-bmp-listener.

Each router is registered through the same /v1/collectors/register endpoint with plane=bmp and a router-owned csr_pem. The returned SVID is stored in the same identity tables and uses the existing rotation and agent-revocation API. This is entirely operator-local: no external identity service or phone-home path is introduced.

For both BMP PostgreSQL URLs, the URL authority is the canonical database host. A query-string host= override is rejected before either pool opens: PostgreSQL clients may otherwise replace the checked authority, and Unix-socket overrides do not carry TLS even when the URL also says sslmode=verify-full.

Example:

PROBECTL_BMP_LISTEN_ADDR=:1179 \
PROBECTL_BMP_TLS_CERT_FILE=/etc/probectl/bmp/tls.crt \
PROBECTL_BMP_TLS_KEY_FILE=/etc/probectl/bmp/tls.key \
PROBECTL_BMP_TLS_CA_FILE=/etc/probectl/agent-ca.crt \
PROBECTL_BMP_DATABASE_URL='postgres://bmp_registry@postgres:5432/probectl?sslmode=verify-full' \
PROBECTL_BMP_REVOCATION_DATABASE_URL='postgres://bmp_revocation@postgres:5432/probectl?sslmode=verify-full' \
PROBECTL_BMP_BUS_MODE=kafka \
PROBECTL_BMP_BUS_BROKERS=kafka-1:9093 \
PROBECTL_BMP_BUS_TLS_ENABLED=true \
  probectl-bmp-listener

Create both logins in the operator's own PostgreSQL credential/key domain (DPR-058). The registry login verifies a router's issued identity through the same tenant-scoped lookup the control plane uses — the query runs after SET LOCAL ROLE probectl_app under forced row-level security — so it must be a member of the migration-created probectl_app role. The revocation login is granted only the execute-only role:

CREATE ROLE bmp_registry LOGIN NOSUPERUSER NOBYPASSRLS;
GRANT probectl_app TO bmp_registry;
CREATE ROLE bmp_revocation LOGIN NOSUPERUSER NOBYPASSRLS;
GRANT probectl_bmp_revocation_reader TO bmp_revocation;
GRANT CONNECT ON DATABASE probectl TO bmp_registry, bmp_revocation;

Both URLs take the usual libpq TLS parameters, e.g. ?sslmode=verify-full&sslrootcert=/etc/probectl/trust/ca-bundle.crt for a private PostgreSQL CA.

Set its password or client-certificate authentication outside probectl. The listener assumes probectl_bmp_revocation_reader, which can execute only provider_list_revoked_agent_identities() and cannot select agent_identities or tenant telemetry. Startup fails closed before binding if the initial complete snapshot is unavailable. Later bounded refresh failures retain the last valid snapshot; they never replace it with an empty or partial list. This is an intra-deployment PostgreSQL connection only—no Internet service, vendor dependency, or phone-home path.

Open-data enrichment

internal/opendata annotates IPs with context from public datasets: the ASN (the numbered autonomous system — i.e. which network organization an IP belongs to), geolocation, IXP presence (internet exchange points, the shared facilities where networks interconnect), and registry allocation data. A bare address like 203.0.113.7 becomes "AS64500, ExampleNet, DE, allocated 2014" — the difference between a number and a story. See architecture.md and the source provenance/AUP matrix in opendata-aup.md. The framework is a library (the flow and test pipelines consume it where enrichment is enabled); each source is pluggable and individually enable-able:

The Enricher runs every enabled source over an IP and merges the results, caching per IP and degrading gracefully: a disabled / failing / slow / panicking source is logged, marked degraded or disabled in Enricher.Status(), and skipped — a partial enrichment is returned and a down dataset never breaks a core path. A source whose configured local data cannot be loaded (a missing .mmdb, an empty stats directory) is registered unavailable with the load error and never invoked, so a broken path is visible in GET /v1/threat/intel/status instead of silently absent. Sources run in registration order — local files first (geo, then RIR), the Cymru DNS fallback after them, PeeringDB last because it keys on the ASN Cymru resolves. GET /v1/opendata/enrichment?ip=<addr> (CLI: probectl opendata enrich --query ip=<addr>) serves the merged context for one IP with per-source provenance. Each contribution records Provenance (source + license + attribution

• fields); a source's AUP (license, commercial-use permission, attribution) is on its Descriptor — the matrix that gates MSP/commercial resale (not private or single-tenant OSS use). All fetches are over TLS with certificate validation and treated as untrusted — external content never gets implicit trust. Open data is ingested once and shared; enrichment is scoped per tenant by the consuming record.

Alerting

The alerting engine (internal/alert) evaluates rules over the TSDB and notifies channels; see architecture.md. Rules are CRUD'd via /v1/alerts (tenant-scoped) and the engine runs in the control plane, ticking every PROBECTL_ALERT_EVAL_INTERVAL (default 30s).

A rule targets a metric series and is either a threshold or a baseline rule:

A channels entry is {"type":"webhook","url":...,"secret":...} or {"type":"email","recipients":[...]}. The webhook secret is the HMAC key; it is redacted (***) from API responses and never returned. SMTP for email is configured at the deployment level (a follow-up exposes it as config). The Alerts page exposes these rule channels, shows the redacted/signed state, and can send a tenant-scoped test delivery through POST /v1/alerts/test-channel before a rule is saved.

Webhook payload (probectl.alert.v1). On fire/resolve the webhook channel POSTs:

{
  "version": "probectl.alert.v1",
  "state": "firing",
  "rule": { "id": "…", "name": "loss-high" },
  "tenant_id": "…",
  "severity": "critical",
  "metric": "probectl_probe_loss_ratio",
  "labels": { "server_address": "1.1.1.1" },
  "value": 0.9,
  "threshold": 0.5,
  "comparison": "gt",
  "reason": "probectl_probe_loss_ratio=0.9 gt 0.5",
  "fired_at": "2026-01-02T15:04:05Z"
}

When the channel has a secret, the request carries X-Probectl-Signature: sha256=<hex> — the HMAC-SHA256 of the exact body. An HMAC is a keyed checksum: only a holder of the shared secret can compute it, so the receiver can verify both who sent this and that nothing was altered before acting on the alert. Each channel delivers independently: a failing channel is logged and skipped, never blocking the others. Alerts are signals; probectl notifies and does not act on the network (on-call/ITSM routing and detection-as-code are their own features below).

Incidents

The incident correlator (internal/incident) groups related signals across planes into one Incident with a unified timeline; see architecture.md. It runs in the control plane, fed by the alert engine (network plane) and a probectl.bgp.events consumer (BGP plane), and is exposed at /v1/incidents (tenant-scoped):

• GET /v1/incidents — the tenant's incidents, most-recently-active first.
• GET /v1/incidents/{id} — an incident with its time-ordered signal timeline.
• PATCH /v1/incidents/{id} with {"status":"resolved"} — resolve an incident.
• POST /v1/incidents/{id}/shares — create an audited, redacted cited snapshot (also requires ai.query). The default seven-day expiry is capped by the tenant's object_retention_days policy.
• GET /v1/incident-shares/{id} — replay a live snapshot through the current authenticated tenant's RLS scope. Missing, expired, revoked, and foreign IDs deliberately return the same not-found response.

Signals correlate into one incident when they are close in time (within PROBECTL_INCIDENT_WINDOW, default 10m) and related in target — the same target, an IP inside the other's prefix (either direction), or overlapping prefixes (so a network alert on 192.0.2.10 and a BGP event on 192.0.2.0/24 land together). An incident's severity is the max of its signals; a signal without a tenant is rejected (fail closed).

The model is extensible without schema churn: a Signal carries a free-form plane/kind and an arbitrary attributes map, so the change, threat, cost, and SLO planes attach as additional signal types onto the same Incident/timeline without schema changes. AI root-cause analysis runs over the timeline.

SSO & RBAC

probectl authenticates users with OIDC SSO — OIDC (OpenID Connect) is the standard login handshake built on OAuth 2.0, and SSO (single sign-on) means users sign in at their organization's identity provider (the IdP: Okta, Entra, Keycloak, …) rather than holding a probectl-local password — and authorizes them with role-based access control (RBAC): every action requires a permission, and permissions come from roles. The security order is the two-level boundary: a request resolves to exactly one tenant first, then RBAC decides whether the caller may perform the route's action within that tenant.

Login flow. GET /auth/login (optionally ?tenant=<uuid>) starts the OIDC authorization-code flow: it sets a short-lived, HttpOnly CSRF state cookie and separate nonce and PKCE-verifier cookies, then redirects to the tenant's identity provider with the PKCE S256 code_challenge. The IdP redirects back to GET /auth/callback, which verifies the state, supplies the matching code_verifier directly to the token endpoint, verifies the ID token and nonce, just-in-time provisions the user within the tenant (a brand-new user gets no roles — a secure default; an admin grants access), mints a server-side session, and sets the session cookie. Missing or mismatched transient values fail closed. POST /auth/logout revokes the session. GET /v1/me returns the caller's tenant, identity, and effective permissions.

Sessions. A session is a random, high-entropy opaque token. Only its server-keyed HMAC hash is stored (table sessions), so a database read cannot mint or cheaply test a session without PROBECTL_SESSION_HMAC_KEY. Production session-cookie deployments fail closed without that key: all multi-tenant / regulated session auth, and single profile session auth once OIDC is configured. The cookie is HttpOnly + SameSite=Lax, and Secure whenever the API serves HTTPS. PROBECTL_SESSION_TTL (default 12h) is the absolute lifetime; activity never extends it. PROBECTL_SESSION_IDLE_TIMEOUT (default 30m) is a second, database-enforced inactivity wall. A successful lookup atomically advances last_activity_at; an idle or absolutely expired token returns no session. Setting the idle value to 0 keeps the safe default rather than creating an unlimited session.

Every successful login replaces the browser's previous session ID. Each session also stores a deterministic digest of its effective permission keys. When a role grant or revoke changes that digest, the next request atomically consumes the old token, sets a newly random cookie, and only then serves the request. Two racing requests cannot create two successors. The session table also has a composite (tenant_id, user_id) foreign key, so the database rejects a session that pairs one tenant with another tenant's user.

The cluster-wide daily lifecycle leader is the cleanup owner for session identity detail. Under each tenant's storage scope it deletes absolutely expired rows on the next sweep and deletes replaced predecessor detail after one PROBECTL_SESSION_TTL replay horizon (there is no separate retention setting). The global credential_locators row contains only the keyed token hash, credential UUID, tenant UUID, and state timestamps; that hash-only tombstone is intentionally retained after detail cleanup. It remains the database lock for authenticated callbacks, so a cleanup/callback race still commits at most one successor. In practice, identity detail therefore ages out within the session TTL plus the daily sweep interval. Tenant erasure removes the locator by its tenant key; subject erasure removes one while its detail row still supplies the credential ID. Once detail cleanup has intentionally unlinked that hash-only tombstone from a subject, it remains only as replay-protection metadata until tenant erasure.

Per-tenant IdP. Providers are resolved per tenant through a provider factory. The environment configuration (PROBECTL_OIDC_*) is the deployment fallback; an enabled row saved through Admin & Settings → Identity administration or PUT /v1/identity/settings overrides it for that tenant. The two are reached differently on purpose. The deployment IdP is the operator's own configuration, so it may live on the operator's network — loopback, RFC1918 or in-cluster DNS — behind the CA in PROBECTL_OIDC_CA_FILE. A tenant-configured issuer is tenant input, so its discovery, keys and code exchange go through the SSRF-guarded client (INJ-04): it must resolve to a public address, and one that resolves into private, loopback or link-local space is refused. GET /v1/identity/settings returns public metadata plus client_secret_configured, never plaintext or ciphertext. Sending an empty client_secret on an update preserves the existing secret.

The tenant_idp row is protected by forced Postgres RLS from its first migration. Its client secret is envelope-encrypted through internal/crypto using tenant-bound authenticated data, so even copied ciphertext cannot be opened under another tenant. This requires PROBECTL_ENVELOPE_KEY (or the configured enterprise key provider); writes fail closed if encryption is not available. Missing or disabled tenant settings use the environment fallback, but a present enabled configuration that is malformed, cannot be read, or cannot be decrypted fails login closed instead of falling through. A login always resolves to one tenant before provider selection. Provider/MSP operators authenticate into the provider domain (the management plane), not into tenant data.

RBAC. Every /v1 route declares a required permission key; the wrapped handler returns 401 when unauthenticated and 403 when the principal lacks the permission — checked before the handler runs. Effective permissions are loaded per request from the user's role bindings (RLS-scoped to the tenant), so a role grant or revoke takes effect immediately. The permission catalog:

The seeded system roles for the default tenant are admin (all permissions), editor (read everything + manage tests/alerts/incidents), and viewer (read-only). GET /v1/me requires only authentication (no specific permission).

Programmatic bearer tokens. Browser users authenticate with the session cookie; scripts and the probectl CLI can instead send Authorization: Bearer <token>. Mint that token with probectl-control mcp-token --tenant <tenant-uuid> --user <user-uuid> --name <label> after the user exists through OIDC login or SCIM. The token is stored only as a hash in mcp_tokens, resolves to exactly that user's tenant and effective RBAC permissions, and can be sent with X-Probectl-Tenant: <tenant-uuid> as a tenant assertion. A mismatched tenant assertion fails closed; the header does not let a token switch tenants. Do not use SCIM, OTLP, or agent-enrollment tokens for normal /v1 API calls: those tokens authenticate only their own subsystem.

Dev mode. PROBECTL_AUTH_MODE=dev bypasses SSO and synthesizes an all-permissions principal for the default tenant, with the X-Probectl-Tenant: <uuid> override for multi-tenant dev. It is triple-gated: (1) the code path exists only in binaries built with -tags devauth (make build-devauth) — a release binary refuses to start in this mode; (2) PROBECTL_DEV_AUTH_ACK=i-understand must be set; (3) the listener must bind loopback (PROBECTL_HTTP_ADDR=127.0.0.1:…). When active it logs at error level and writes an auth.dev_mode_active audit event. The CI gate no-devauth-in-release proves release binaries contain neither the symbols nor the dev-principal literal. The test suite installs its own hook in _test.go files, which never ship.

Resource API & CLI

The versioned resource API lives under /v1 (full schema at /openapi.json):

• GET/POST /v1/tests, GET/PUT/DELETE /v1/tests/{id} — synthetic-test CRUD.
• GET /v1/agents, GET/PATCH/DELETE /v1/agents/{id} — agents register over mTLS; the API lists, renames, and deregisters them.
• GET/POST /v1/tests/{id}/path — the latest discovered network path for a test, and a trigger to discover it now. The Path & Topology UI consumes this.

API lifecycle

/v1 is the LTS compatibility line. A compatible field or route can be added at any time, but a removal or incompatible semantic change must first be marked in OpenAPI with deprecated: true and x-probectl-lifecycle metadata: deprecated_at, sunset, lts_until, replacement, and policy. Deprecated operations stay served for at least 12 months after deprecated_at.

Every deprecated operation also emits machine-readable response headers on live requests: Deprecation (structured-field date), Sunset (HTTP date), X-Probectl-API-Replacement, X-Probectl-API-LTS-Until, and Link entries for the successor operation plus /openapi.json. This lets clients discover both the retirement date and the replacement path without scraping release notes.

Current deprecated operation: DELETE /v1/agents/{id} is retained for metadata deregistration compatibility through 2027-06-19; prefer POST /v1/agents/{id}/revoke for security offboarding because it persists identity revocation and feeds the live mTLS deny-list.

Every /v1 route is tenant-scoped through internal/tenancy + Postgres RLS, so a request can never read or write across tenants. Authentication and RBAC are real (see SSO & RBAC below): the caller's tenant and effective permissions come from an authenticated session, and each route requires a permission. The "no undocumented routes" rule is enforced by a test that matches the route table against openapi.json.

The probectl CLI is the web-parity client. Configure it with flags or environment: PROBECTL_API_URL (default https://localhost:8443), PROBECTL_CA_FILE (a PEM bundle that verifies the control plane's certificate when it is issued by a private CA — --ca-file; Go on macOS ignores SSL_CERT_FILE, and verification is never switched off), PROBECTL_API_TOKEN (sent as Bearer), PROBECTL_TENANT (sent as X-Probectl-Tenant), PROBECTL_SESSION_COOKIE_FILE (owner-only file containing an MFA-bearing OIDC session for audit reveal and the tenant-admin break-glass consent commands provider consent|decide-consent|revoke-consent; it takes precedence over the bearer token for those requests), and PROBECTL_LOCALE (default en; accepts shipped language tags such as es/es-MX for CLI help and API error messages). API error code values stay stable and machine-readable; only the human message is localized. In en the CLI prints the server's own message — what to fix, as the console shows it — and in other locales the localized message for the code. When an API error includes request_id, the CLI includes it in the terminal error output for support/debugging.

A CLI request gives up after 15 seconds, except where the server itself runs longer and lifts its write timeout for the route: export and other downloads get 15 minutes, a tenant or subject erasure 15 minutes, a tenant provisioning (provider create-tenant, tenant create) 5 minutes, a path discovery (test path <id> --body) 150 seconds, and ai ask 90 seconds. PROBECTL_HTTP_WRITE_TIMEOUT sets the server's ordinary write timeout; the long routes keep their own budgets above it.

The terminal-native product surface is the CLI. There is no separate committed TUI mode today; if one is added later it must be declared in the surface catalog and backed by its own parity tests. For automation, use --json. The OpenAPI gate checks both tenant /v1 and provider /provider/v1 JSON APIs; every operation requires either a named CLI command or a documented none-by-design exception. Provider bootstrap, enrollment, and login payloads are credential-bearing and therefore require --body-file <0600-file|->; inline --body is rejected before any request. Use PROBECTL_API_TOKEN rather than --token for the resulting provider bearer token so it stays out of process arguments.

Journey-critical parity is currently served by the CLI:

probectl test list
probectl test create --name edge-dns --type icmp --target 1.1.1.1 --interval 30
probectl test delete <id>
probectl agent list
probectl incident list --query status=open
probectl --json incident journal <id>
probectl incident journal-append <id> --body '{"kind":"note","body":"Operator hypothesis"}'
probectl topology show
probectl alert active
probectl ai ask --body '{"question":"Why is WAN loss high?","subject":{"incident_id":"inc_123"}}'
probectl ai ask --handoff --body '{"question":"Why is WAN loss high?","subject":{"incident_id":"inc_123"}}' > ask-handoff.md
probectl remediation list
probectl slo list
probectl lifecycle export --redact > tenant-export.tar.gz
probectl --json cost summary      # machine-readable output
probectl api GET /v1/fairness     # raw escape hatch for JSON APIs

eBPF host agent

The eBPF agent watches a host's network from inside the Linux kernel — it sees which processes talk to which services without you instrumenting anything. It is observe-only: it never blocks or modifies traffic. Like the canary agent, its real config is a YAML file (-config / PROBECTL_EBPF_CONFIG), bounded to 1 MiB before YAML decoding; see deploy/agent/probectl-ebpf-agent.example.yml and ebpf-agent.md, with PROBECTL_EBPF_* env vars overriding individual fields. The in-kernel loader is compiled in only with the ebpf build tag; without it (or for tests), point fixture_path at a recording to replay.

The big idea in the keys below: layer-7 plaintext capture is off, and stays off until you prove three separate intents — turn it on (L7_CAPTURE), name the tenant that consents (L7_CONSENT_TENANT), and list the exact workloads (L7_SCOPE). Miss any one and the kernel copies no payload. That is the fail-closed posture for the most sensitive thing this agent can do.

Flows + service edges are published to probectl.ebpf.flows (ebpfv1.FlowBatch, tenant-keyed). The live loader needs a BTF Linux kernel (≥5.8; BTF is the type metadata modern kernels embed, which lets one compiled eBPF program adapt itself to many kernel versions instead of being rebuilt per machine) and CAP_BPF/CAP_PERFMON; see ebpf-agent.md.

Agent→bus TLS/SASL (eBPF, endpoint, flow, and device agents)

When a telemetry agent publishes straight to Kafka, its broker connection takes the same hardening keys as the control plane's PROBECTL_BUS_* set, under the agent's own prefix: PROBECTL_EBPF_BUS_* here, and likewise PROBECTL_ENDPOINT_BUS_*, PROBECTL_FLOW_BUS_*, and PROBECTL_DEVICE_BUS_* for the agents below. TLS encrypts the broker hop; SASL is Kafka's authentication layer on top of it — a username/password proven either directly (plain, safe only inside TLS) or via a challenge that never sends the password (scram-sha-256/-512). The policy is the same fail-closed one: kafka mode without TLS refuses to start unless the explicit dev-only plaintext flag is set. (The canary agent has no bus keys — it talks gRPC/mTLS to the control plane, which publishes on its behalf.)

Endpoint / DEM agent (probectl-endpoint)

"DEM" is digital experience monitoring: this agent runs on an end-user's laptop (Linux/macOS/Windows), measures their actual last-mile experience, and figures out whether a slowdown is the WiFi, the ISP, or the network. Because it sits on a personal device, its defaults are privacy-first — it collects the WiFi name and gateway (useful, low-risk) but not the AP MAC or public hop IPs (which can geolocate a person), and it discloses exactly what it collects on startup. It reads a YAML config (default path PROBECTL_ENDPOINT_CONFIG), bounded to 1 MiB before YAML decoding; PROBECTL_ENDPOINT_* env vars override it. See endpoint-dem.md.

Results (WiFi / gateway / last-mile / session signals + the attribution verdict) are published to probectl.endpoint.results (resultv1.Result, tenant-keyed), flowing through the same pipeline as every other canary. The agent discloses exactly what it collects at startup and never phones home.

Flow collector (probectl-flow-agent)

A flow record is a router's summary of one conversation — who talked to whom, on which ports/protocol, when, and how many bytes/packets — with no payload: the phone bill, not the phone call. NetFlow, IPFIX, and sFlow are the three export dialects devices speak; AWS VPC Flow Logs, Azure NSG Flow Logs, and GCP VPC Flow Logs are the equivalent cloud-export path. The flow collector listens for NetFlow v5/v9, IPFIX, and sFlow v5 datagrams from network devices, decodes them (template + sampling handling), and publishes normalized batches to probectl.flow.events (flowv1.FlowBatch, tenant-keyed). Cloud flow logs and metrics are imported from local files already exported by the operator's cloud pipeline. probectl does not poll cloud-provider APIs. It reads a YAML config (default path PROBECTL_FLOW_CONFIG), refusing files larger than 1 MiB before YAML decoding; PROBECTL_FLOW_* env vars override the file. The defaults serve all three protocols on their standard ports (NetFlow :2055, IPFIX :4739, sFlow :6343). See flow.md for the security posture: flow export is plaintext UDP by design, so every datagram is treated as untrusted and the collector should sit adjacent to its exporters (not exposed to the wider network).

YAML equivalent for one-shot cloud import:

cloud_import:
  provider: aws_vpc_flow_logs
  path: /var/lib/probectl/cloud-flow/aws-vpc-flow.log

If all UDP listeners are disabled, cloud_import must be complete; otherwise the flow agent refuses to start. Cloud import still publishes to the configured bus as probectl.flow.events, tenant-keyed by tenant_id.

Cloud metrics use the live probectl-cloud-metrics importer documented below. It reads operator-exported JSONL locally and sends tenant-bound remote write to the self-hosted control plane. The previous cloud-pull prototype was never wired into a command, so there is no hidden cloud-API polling path.

The control plane consumes that flow topic, optionally enriches each record with ASN/geo, and persists to the flow store behind /v1/flows/* (top-talkers / capacity / anomalies). These are control-plane keys (not flow-agent keys):

PROBECTL_PATH_RETENTION_DAYS applies the same pattern to raw path/traceroute ClickHouse rows: raw hops/links age out, while hourly tenant-scoped hop/link rollups stay available for lower-resolution trend and RCA context until tenant deletion removes them.

PROBECTL_DEPLOYMENT_PROFILE=multi-tenant and regulated are production-like profiles, so startup refuses volatile raw-ingest/serving defaults. Set PROBECTL_BUS_MODE=kafka, PROBECTL_TSDB_MODE=prometheus, and PROBECTL_PATHSTORE_MODE, PROBECTL_FLOWSTORE_MODE, PROBECTL_OTELSTORE_MODE, PROBECTL_EBPFSTORE_MODE, and PROBECTL_ENDPOINTSTORE_MODE to clickhouse with their required https:// URLs before using those profiles. Postgres writer/read-replica URLs must verify the server certificate: sslmode=verify-ca or verify-full with sslrootcert set; the server-unverified sslmode=require is single-profile dev only (CRY-05). Each ClickHouse lane also needs the corresponding scoped reader user (PROBECTL_PATHSTORE_READER_USER, PROBECTL_FLOWSTORE_READER_USER, PROBECTL_OTELSTORE_READER_USER, PROBECTL_EBPFSTORE_READER_USER, and PROBECTL_ENDPOINTSTORE_READER_USER) so boot can install the database row policy. The default single profile may still use memory modes for a lightweight sovereign/lab install.

Device telemetry agent (probectl-device-agent)

This agent reads metrics straight off network gear (routers, switches). It polls the old way (SNMP v2c/v3), listens for authenticated SNMP traps, and subscribes the modern streaming way (gNMI/OpenConfig). Polling/subscription samples normalize into one DeviceMetric shape and publish to probectl.device.metrics (tenant-keyed). An explicitly configured SNMP target with neighbors: true also publishes a bounded LLDP/CDP snapshot to probectl.device.neighbors; it never scans for devices or opens a follow-up session. Each LLDP/CDP attempt separately publishes a stable, non-secret readiness receipt to probectl.device.collection-outcomes, so failure, successful-empty, unsupported, and never-observed cannot be confused. Accepted traps become tenant-scoped event and alert rows. The full device list and optional trap listener live in a YAML config, bounded to 1 MiB before YAML decoding (see deploy/agent/probectl-device-agent.example.yml); the env vars below override it and give a single-device quick start for trying one device fast. See device-telemetry.md.

Set top-level collection_profile to minimal, standard, or topology-rich to expand one compiled evidence budget across the configured targets. The vocabulary uses only the agent's existing SNMP and gNMI collectors. minimal uses 5m SNMP / 2m gNMI cadence with no optional walks; standard uses 1m / 30s with base interface and host evidence; and topology-rich additionally enables SNMP temperature and bounded LLDP/CDP walks. gNMI stays on the two supported OpenConfig interface paths because the agent does not own a gNMI topology collector.

Profile changes belong under each target's collection_overrides block: interval, sensors, and neighbors for SNMP, or gnmi_sample_interval and gnmi_paths for gNMI. Mixing a profile with the old top-level collection knobs fails closed. SNMP overrides are bounded to 15s–24h; gNMI cadence is bounded to 5s–1h; and profile-mode paths are restricted to the compiled counters/status paths. A config without collection_profile keeps the legacy explicit behavior. Run probectl-device-agent config-check -config <path> or probectl device config-preview --config <path> for the credential- and tenant-free effective plan; neither command opens a network connection.

Credentials are referenced by NAME, never inlined — no secrets in the device list. The default credential source resolves those names from the environment (the PROBECTL_DEVICE_CRED_<NAME>_* vars below); the secrets backends plug Vault/CyberArk into the same seam. An unresolvable name fails closed at startup. <NAME> is the upper-cased credential name with -/. → _:

In legacy YAML without a profile, neighbors: true is a per-device opt-in. In profile mode, use collection_overrides.neighbors: true. Both forms are valid only for SNMP targets. One snapshot contains at most 256 normalized neighbors. The control plane retains at most 16,384 current/stale rows per tenant, returns at most 500 rows per read, and prunes evidence more than 24 hours old. Unsupported LLDP/CDP MIBs remain explicit. The independent receipt store retains at most 4,096 configured target/protocol receipts per tenant for 30 days and returns at most 500 per read. Receipt payloads contain allowlisted states/reasons/actions, not raw SNMP values, credentials, discovered targets, or free-form errors. probectl does not infer a link from a name or management address alone.

(USM is SNMPv3's user security model: a named user authenticates with the AUTH_* pair and optionally encrypts with the PRIV_* pair — leaving a passphrase empty steps the security level down, as the table notes.)

SNMP trap ingestion is opt-in YAML, not an unauthenticated open UDP port. A trap-only agent is allowed, but traps.sources must name at least one sender and credential:

traps:
  enabled: true
  listen: ":9162" # default; bind 127.0.0.1:9162 or a private interface if preferred
  sources:
    - name: core-switches
      address: 192.0.2.10       # optional source-IP allow-list
      transport: snmpv2c        # snmpv2c | snmpv3
      credential: core-traps    # resolves PROBECTL_DEVICE_CRED_CORE_TRAPS_*

For SNMPv3 trap sources, USERNAME and AUTH_PASS are required; NoAuthNoPriv traps are rejected. Accepted rows store the source name / v3 username and never store the community or passphrase.

gNMI connections always use TLS with certificate verification (system roots or a per-device ca_file); there is no skip-verify or plaintext mode. A legacy plaintext: true value is rejected during validation before credentials or a dial attempt can occur.

Discovery jobs

probectl-device-agent discover -job discovery.json -out review.json runs a bounded, tenant-scoped SNMP discovery pass and writes review candidates. It does not mutate the YAML config and does not activate monitoring; accepted devices must be explicitly reviewed into normal devices: targets.

The job file is JSON:

The command uses the same PROBECTL_DEVICE_CRED_<NAME>_* variables and secrets resolver described above. The optional -fixture fixture.json flag supplies canned device inventories for offline demos/tests and avoids network probes. Both the required job and optional fixture JSON files have an 8 MiB maximum; larger files fail closed before JSON decoding or any discovery probe.

OTLP receiver

OTLP is the OpenTelemetry protocol — the vendor-neutral standard wire format for metrics, traces, and logs. This receiver lets other systems push their OTLP data into probectl: anything that already speaks OpenTelemetry can send here without changing its exporter. It is off by default and, when on, is locked to the same posture as everything else: TLS-only, token-authenticated, tenant-scoped, on its own listeners separate from the /v1 REST API. There is no anonymous-plaintext mode: setting a listen address without a TLS cert/key pair fails config validation. Bearer tokens can be DB-backed and hot-revoked through /v1/otlp-tokens; static PROBECTL_OTLP_TOKENS entries are legacy/bootstrap only. Because those static tokens never expire, each one must clear a strength floor at config load — at least 32 characters of a real random secret (e.g. openssl rand -hex 32); a short or low-entropy token fails validation and the process refuses to start (AUTHZ-27, fail closed). See otlp.md.

Setting an address without the TLS files fails config validation — the receiver is never anonymous plaintext. Missing, unknown, revoked, or freshness-invalid credentials fail per request. Ingested metrics are tenant-tagged and published to the probectl.otlp.metrics bus topic.

Syslog receiver

An authenticated, TLS-only syslog listener for network devices and appliances that forward RFC 3164 / RFC 5424 syslog over TLS (e.g. rsyslog omfwd with StreamDriverMode=1). It is off by default and holds the same posture as the rest of probectl: TLS-only, authenticated, tenant-scoped, fail closed. Each sender is an explicit source authenticated per message by a TLS client-certificate subject (verified against PROBECTL_SYSLOG_TLS_CA_FILE) or an HMAC-SHA256 shared secret; the tenant is bound to the source here and never read from the payload, so one listener can serve many tenants. A sender that matches no source is rejected and nothing is stored. Accepted lines are normalized and land on the GET /v1/device/syslog read path. Setting a listen address without a server cert/key pair, a source without a credential, or a source without a tenant fails config validation — the listener is never anonymous plaintext.

OTLP export

probectl can export OTLP to an upstream collector. All three signals are forwarded — metrics, traces, and logs (ARCH-003): ingested traces/logs are re-exported to a customer's own trace/log backend, not just queryable inside probectl. For OTLP/HTTP the per-signal paths are derived from the configured endpoint (…/v1/metrics → …/v1/traces, …/v1/logs). This egresses confidential customer telemetry (plus the bearer token), so a remote collector must be encrypted — guardrail 12. Config validation fails closed otherwise: for http a remote endpoint must be https://; for grpc the INSECURE flag is refused for a remote endpoint. A loopback collector (a co-located sidecar) may use plain http:// / INSECURE for development.

Before egress the re-export runs the SAME redaction the storage path applies (RTP-06): span names, log bodies and every attribute value pass through the governance telemetry masking, so an Authorization header, a URL token or a known secret shape that storage masks never leaves the operator's network in the clear.

Tenant isolation (RTP-06). PROBECTL_OTLP_EXPORT_ENDPOINT is a shared endpoint — every tenant's telemetry is re-exported to it. That is safe only in the single/sovereign profile (one tenant). Under the multi-tenant/regulated profiles a shared endpoint would cross tenants (one tenant's spans reaching another tenant's or the provider's collector) and is refused at startup; configure a per-tenant, tenant-scoped map with PROBECTL_OTLP_EXPORT_TENANT_ENDPOINTS instead. Each tenant's telemetry then reaches only that tenant's own collector, and a tenant with no configured endpoint is dropped closed (never forwarded). The export configuration is recorded on the provider audit stream at startup (tokens excluded).

Ecosystem integrations

The Grafana datasource API (/v1/grafana/api/v1/*), the federation endpoint (/v1/prometheus/federate), and the remote-write receiver (/v1/prometheus/write) ride the existing TSDB config (PROBECTL_TSDB_MODE / PROBECTL_TSDB_URL) and the /v1 API listener — no extra keys. Reads need metrics.read, remote-write metrics.write (migration 0022). See ecosystem-integrations.md.

A CMDB (configuration management database) is an organization's IT inventory — a registry of configuration items (CIs: servers, applications, services) and who owns them. The ServiceNow or NetBox CMDB correlation tags probectl incidents and agents with the CI they belong to, so an alert arrives already knowing "this is the payments edge proxy, owned by team X". It is off unless configured:

Cloud metric import

probectl-cloud-metrics imports already-exported AWS CloudWatch, Azure Monitor, or Google Cloud Monitoring JSONL rows into /v1/prometheus/write. It never polls cloud APIs; the file is local input from the operator's own export pipeline. The API URL must be HTTPS except for loopback test instances.

AI assistant

Worked per-provider setups (Ollama, vLLM, OpenAI, Anthropic, Azure) are in ai-rca.md → Copy-paste recipes; the remote-egress enablement chain (operator ack + per-tenant consent) is in ai-egress.md.

The assistant (root-cause analysis + natural-language query) works out of the box with zero network access — the default builtin provider is an in-process synthesizer that writes its answers locally. You only point it at a real language model if you want nicer prose, and doing so is treated as data egress: a remote endpoint must be https, and you have to explicitly acknowledge that tenant data will leave (PROBECTL_AI_EGRESS_ACK). A loopback endpoint may be http (for a local model on the same box). The redaction keys below mask sensitive values before anything reaches an external model. Redaction tokens are tenant-scoped keyed-HMAC labels; PROBECTL_SESSION_HMAC_KEY makes those labels stable across restarts, while an unset key falls back to process-local random labels that rotate on restart. See ai-rca.md.

A non-builtin provider without an endpoint fails config validation. Whatever the backend, every answer is tenant- and RBAC-scoped by the query layer and every claim is citation-checked before it reaches the user — a model can never see out-of-scope data or inject an ungrounded claim.

MCP server

MCP — the Model Context Protocol — is the standard by which AI applications (Claude Desktop, IDE assistants, agent frameworks) call external tools. The MCP server exposes read-only, tenant- + RBAC-scoped probectl tools to such clients, which means an MCP caller is an external AI reading tenant data — so tool results ride the same egress gate as the RCA model (ai-egress.md). The HTTP transport is off by default and is TLS-only + bearer-authenticated; the stdio transport is local (the client launches probectl-control mcp-stdio as a subprocess and talks over stdin/stdout — no listener at all; the token comes from PROBECTL_MCP_TOKEN). Mint a token with probectl-control mcp-token --user <user-uuid> [--tenant <uuid>] [--name <label>] — the token prints once and only its hash is stored, so a database read can never recover it. See mcp.md.

Setting PROBECTL_MCP_HTTP_ADDR without the TLS files fails config validation — the MCP endpoint is never anonymous plaintext.

TLS / certificate observability

The control plane analyzes TLS/cert posture from TLS handshakes the HTTP and eBPF-L7 probes already captured — it never re-handshakes a target itself — and correlates the findings into threat-plane incidents. See tls-observability.md.

CT logs are Certificate Transparency's public, append-only ledgers of every certificate publicly issued — correlation against them answers "was a cert for my name issued that I don't know about?" (a rogue or mis-issued certificate). It is off by default (an external fetch — sovereignty / AUP / rate limits) and degrades gracefully when the CT source is down. Responses are read through a bounded 1 MiB sentinel: a larger or trailing-garbage body is rejected before JSON decoding and degrades closed — never accepted as a silently truncated prefix.

Threat-intel enrichment

Threat-intel feeds are public lists of IOCs — indicators of compromise: known-bad IPs, certificates, URLs. A JA3 is a fingerprint of how a client performs its TLS handshake, which identifies malware families even when their server IPs rotate. The control plane can match peer IPs / hostnames / certs / JA3 seen in your own telemetry against these feeds, surfacing confidence-scored, source-attributed threat-plane signals (a signal, not an IPS — never blocks). See threat-intel.md for the feed/AUP matrix and caveats.

Off by default (an outbound fetch — sovereignty / no-phone-home). The refresher keeps each source's last-good indicators, so a feed outage degrades gracefully and never breaks a core path.

Enterprise identity: SCIM + ABAC

SCIM is the standard API identity providers use to push user lifecycle into applications — a joiner appears, a leaver is deactivated, without anyone touching probectl. ABAC (attribute-based access control) layers policy conditions on top of RBAC roles — "editors may write tests only in project X". Both have no environment keys — the SCIM bearer token an IdP presents is minted in Admin & Settings → Identity administration or with the control-plane CLI, and ABAC policies are managed from the same Admin surface or over the API. See scim-abac.md.

# mint a per-tenant SCIM token for an IdP (shown once)
probectl-control scim-token --tenant <tenant-uuid> --name okta

The /scim/v2/* surface is gated by a valid SCIM token (no token ⇒ 401), and the directory-admin APIs (/v1/directory/scim-tokens, /v1/abac/policies) require directory.read/directory.write.

SCIM provisioning is bounded per tenant even though it has no environment knobs: each tenant may hold up to 10,000 SCIM users and 2,000 SCIM groups, list calls are SQL-paged with a maximum count of 200, and each SCIM bearer token has its own 600-request/minute bucket. A noisy or compromised IdP token therefore burns only its own local budget; it cannot force an unbounded directory scan or grow a tenant directory forever.

Change intelligence

Most outages follow a change. This feature ingests change webhooks — deploys, config pushes, route changes, IaC (infrastructure-as-code) applies, commits — each cryptographically signed by its provider, into a change timeline, and correlates them with incidents ("this incident started 4 minutes after that deploy"), feeding the AI RCA. See change-intel.md for the webhook contract + provider/signature table.

Each inbound delivery is TLS + signature-verified (HMAC/token, constant-time) + tenant-bound to the credential + replay-deduplicated by provider delivery ID; an unsigned, stale, forged, or duplicate event is rejected before a second mutation, and one tenant cannot inject another's changes. Generic webhooks sign <unix-timestamp>.<body> and include X-Probectl-Delivery, X-Probectl-Timestamp, and X-Probectl-Signature; GitHub/GitLab use their provider delivery UUID headers. Webhook secrets are runtime config — inject them from a secret manager, never commit them.

SIEM export

A SIEM (security information and event management system — Splunk, Microsoft Sentinel, Elastic, Google Chronicle, …) is the SOC's central platform where an organization's security events converge for detection and investigation. This feature forwards the audit stream and threat-plane signals there over hardened TLS. probectl is the forwarder, not a SIEM — events are rendered into a standard format and pushed; nothing is auto-blocked. See siem.md for formats, delivery guarantees, and per-SIEM setup.

The wire formats are the SIEM world's lingua francas: syslog (RFC 5424 structured syslog), cef (ArcSight's Common Event Format), ecs (the Elastic Common Schema), and otlp (OpenTelemetry logs). Off by default (an outbound connection — sovereignty / no-phone-home). Audit forwarding resumes from a durable per-tenant cursor — a saved bookmark in the audit stream, so a restart or SIEM outage picks up exactly where delivery stopped — and delivery retries without dropping under a SIEM outage. Exported audit events are PII/secret redacted (built-in denylist + PROBECTL_SIEM_REDACT_KEYS, followed by the core governance PII scanner over actor/action/target/outcome and non-denylisted values).

Inbound syslog collection is a separate input path, not SIEM export. A deployment profile that enables it registers tenant-bound sources with a source name, optional address allow-list, either an HMAC secret or TLS client-certificate subject, a per-source rate limit, and a max line size. The receiver accepts RFC 5424 and RFC 3164 over TLS only, rejects malformed or oversized lines, stamps the tenant from the registered source instead of trusting the payload, and records event provenance (source.name, auth.method, wire format, source address) for incident correlation.

On-call + ITSM integration

ITSM (IT service management) tooling is where operations work is tracked — ticketing systems like ServiceNow and Jira; on-call platforms like PagerDuty and Opsgenie are what wake the right human. This feature mirrors incidents into that tooling: page on-call, post to chat (Slack/Teams), and open + bidirectionally sync tickets (ServiceNow/Jira) — resolve it there and probectl hears about it, resolve it here and the ticket closes. probectl is the forwarder, not the system of record — it never auto-blocks anything. See oncall-itsm.md for the connector matrix, mapping, and the inbound webhook contract. Use GET /v1/oncall/status or probectl oncall status to inspect the tenant-scoped posture; the response is redacted and never returns connector secrets or endpoint path/query values.

Off by default (each connector is an outbound connection to the operator's tooling). Paging + ticket creation are idempotent (an incident opens at most once per connector — a retry/restart never double-pages), status sync is bidirectional with loop protection (an inbound resolve from one system is never echoed back to it), and routing is per-tenant (a connector only fires for its own tenant). Endpoint specifics: a Slack/Teams endpoint is the incoming-webhook URL; a Jira endpoint carries the project (and optional resolve transition) as query params, e.g. …/rest/api/2/issue?project=OPS&resolve_transition=31; a ServiceNow endpoint is the …/api/now/table/incident URL. Inbound deliveries must include X-Probectl-Signature: sha256=<hmac> or X-Probectl-Token: <secret> over TLS; an unsigned or forged delivery is rejected (401). Secrets are runtime config — inject them from a secret manager, never commit them.

The Alerts page renders the same tenant-scoped connector posture: provider choices (pagerduty, opsgenie, slack, teams, servicenow, jira, psa), sanitized endpoint host, TLS posture, credential presence, inbound webhook path, and redaction state. POST /v1/oncall/test sends an operator-triggered test incident through an already-configured connector id for the caller's tenant; the browser never supplies ad-hoc endpoints or secrets.

Topology graph + what-if

The topology graph is the live map probectl assembles from everything it observes — nodes (hosts, services, devices, prefixes) and the edges between them — and what-if is simulation over that map: "if this link or node disappeared, what would be cut off?".

The graph feeds from eBPF/BGP/device streams + path discoveries; served at GET /v1/topology with what-if simulation at POST /v1/topology/whatif. See docs/topology.md.

FinOps / egress cost

Cloud providers bill for egress — bytes leaving a zone, region, or cloud — and those line items are notoriously hard to attribute. This engine prices the flow volume probectl already observes (bytes × the provider's public list rates), entirely locally: it never calls a billing API, so the sovereignty posture holds and the numbers are explicitly estimates with stated provenance.

PROBECTL_COST_ZONES, PROBECTL_COST_SERVICES, and PROBECTL_COST_BUDGETS are deployment-wide and single-tenant-only (AI-09, guardrail 7.1): one map is applied to every tenant, so in a pooled deployment two tenants that share a prefix would be cross-attributed (one tenant's bytes booked to the other's service/team) and cross-alerted (a budget keyed on a shared owner fires for every tenant). Under PROBECTL_DEPLOYMENT_PROFILE=multi-tenant/regulated these keys are therefore refused at startup (fail closed); leave them unset and cost runs per-tenant in volume-only mode. Pricing (PROBECTL_COST_PRICES_FILE / PROBECTL_COST_PRICED) is list rates that are the same for every tenant and stays allowed in all profiles.

Summary at GET /v1/cost/summary and the Cost page; deeper cross-plane and ad-hoc views are native at /dashboards and /explore. The Prometheus-compatible API is optional interoperability, not a dashboard dependency. See docs/finops.md.

SLO engine

An SLO (service-level objective) is a reliability target — "99.9% of probes to this service succeed over 30 days"; the SLI is the measured indicator behind it, and the error budget is the failure allowance the target leaves (0.1% here): spend it slowly and you're fine, burn it fast and you get paged before the month is lost. OpenSLO is the vendor-neutral YAML standard for declaring these.

Statuses at GET /v1/slos, OpenSLO export at GET /v1/slos/openslo, and the SLOs page. See docs/slo.md.

Compliance / segmentation validation

Segmentation is the declared rule set of who may talk to whom on your network ("only the app tier may reach the database tier"). Firewalls enforce such rules; this validator checks them against the traffic probectl actually observed, producing audit-grade evidence that the declared boundaries hold — or a verdict naming the flow that crossed one. It never blocks anything.

A segmentation policy's zones are CIDR→zone maps applied to every tenant, so the policy directory is deployment-wide and single-tenant-only (AI-09, guardrail 7.1): in a pooled deployment two tenants that share a prefix would be cross-validated and cross-alerted off one tenant's zone definitions. Under PROBECTL_DEPLOYMENT_PROFILE=multi-tenant/regulated the directory is therefore refused at startup (fail closed); leave it unset in those profiles.

Verdicts at GET /v1/compliance, hash-chained audit evidence at GET /v1/compliance/evidence, and the Compliance page. See docs/compliance.md.

Collective internet-outage view

"Is it us, or is it the internet?" This view answers that: a local engine detects outages from your own vantage points (many tests failing toward the same provider at once), and — only if you opt in — correlates them with public internet-outage feeds (IODA, Cloudflare Radar) reporting trouble in the same networks or countries.

The collective view at GET /v1/outages (events + the caller-tenant's affected tests + vantage detections + feed AUP/health + coverage notes) and the Internet outages page. Scope resolution (IP→ASN/country) rides the open-data enricher (PROBECTL_FLOW_ENRICH_ASN); without it the response reports the degradation honestly. See docs/outage.md.

RUM convergence

RUM (real-user monitoring) is timing data real browsers report from real visits — page-load vitals sent home as small beacons — the complement to synthetic probes: synthetics tell you a path is broken before users arrive; RUM tells you what users actually experienced. Convergence is laying the two over each other for the same service.

Beacons ingest at POST /ingest/rum (public-key routed, optional origin allow-listed, consent-gated, URL-redacted, no IP stored — privacy is enforced server-side, fail closed); the convergence view serves at GET /v1/rum and folds into the Endpoints surface; rum.* vitals flow to the TSDB for dashboards. The SDK is web/public/probectl-rum.js. See docs/rum.md.

Carbon / power observability

Moving bytes costs energy; energy has a carbon intensity. This engine multiplies observed traffic volume by published energy coefficients and your grid's intensity to produce estimates — always labeled as such, with the methodology attached to every response — entirely locally.

Attribution reuses PROBECTL_COST_ZONES / PROBECTL_COST_SERVICES. The estimate serves at GET /v1/carbon and folds into the Cost page. See docs/carbon.md. The chaos injector and the large/extra-large scale gate are test-harness tools — see docs/chaos.md and docs/scale-gate.md.

Editions / license

A license file is a small signed document stating what was bought: the tier, the feature list, the expiry, the customer. Ed25519 is the signature scheme — verifying needs only the public key, which is baked into the binary, which is precisely why verification works with the network cable unplugged.

Verification is offline — local signature math against public keys baked into the binary at build time (never an env var; never phone-home). Expiry runs the 30-day-grace → read-only ladder and never breaks running telemetry. License state + the feature→tier map serve at GET /v1/editions and render on Admin → Editions — the one place tiers appear when unlicensed. See docs/editions.md for the file format, the signing CLI (probectl-license), and the gating pattern.

Provider / management plane (ee/)

The provider plane is the MSP operators' own console — tenant lifecycle, fleet view, metering — deliberately a separate privilege domain from tenant users. Break-glass (named for the fire-alarm cover) is its only path into tenant data: an explicit, tenant-consented, time-bounded, separately-audited emergency grant. Active only when the license grants provider_plane; otherwise /provider/* is a plain 404 (hidden, not locked).

The provider plane additionally requires PROBECTL_ENVELOPE_KEY (operator TOTP secrets are envelope-sealed at rest) and a database. Operator MFA is mandatory; operators are a privilege domain distinct from tenant users with no implicit access to tenant telemetry — see docs/provider-plane.md for the model, the break-glass consent flow, and the storage-layer confinement (probectl_provider role). That page also records the engineering eval smoke: community builds must keep /provider/* as a plain 404, while provider evals must attach the licensed ee/ plane with an envelope key and bootstrap token. Suspending a tenant rejects its users at the API (tenant_suspended) without touching data or ingestion.

Siloed / hybrid isolation (ee/)

Two ways to keep tenants apart: pooled — shared tables where every row is tagged and filtered by tenant_id (one building, locked apartments) — and siloed — physically separate stores per tenant (separate buildings); hybrid mixes the two per tenant. Pooled stays the default and needs no configuration. Siloed and hybrid tenants (per-tenant Postgres schema / ClickHouse database / bus topic namespace / object key namespace) require a license granting siloed_isolation and are selected per tenant at provisioning (isolation_model + optional residency).

Residency pins the tenant's ClickHouse flow data in this release; Postgres control state, the TSDB, object storage, and bus brokers are NOT region-pinned yet — docs/isolation.md states the exact contract, the catch-up/migration story for silo schemas, and the offboard-teardown boundary: Offboard is status-only, while retained-store reclamation requires a separate tombstone-aware maintenance flow.

Deployment theme and product identity

PROBECTL_THEME_OVERRIDES configures one token map for the whole deployment. It is a JSON object because CSS color functions and font lists can contain commas. Startup validates token names, value grammar, the 64-entry size bound, and WCAG contrast against both shipped themes; malformed or unsafe config fails closed. The light and dark operator themes remain built in; light is the default.

Public GET /branding is intentionally available before authentication so the login shell and signed-in UI receive the same theme. It always returns product_name: "probectl", never varies by host/session/tenant, and may include the validated deployment token_overrides. It is cacheable for 60 seconds. There are no provider-plane or per-tenant branding APIs, custom-domain tenant mapping, logo override, login-message override, or notification-email identity override. The product decision and schema compatibility window are recorded in white-label.md.

Advanced data governance (governance, ee/)

Per-tenant data classification + redaction, composed with retention, residency, and BYOK. No new config keys: the classification + redaction MECHANISM is core (the ?redact=true export toggle works anywhere, masking PII with a partial strategy); the governance feature adds per-tenant POLICY (stored in tenant_governance, migration 0033) set from the provider plane (GET/PUT /provider/v1/tenants/{id}/governance). IPs are PII by default. Full model: docs/governance.md. Redacted export: GET /v1/lifecycle/export?redact=true.

Tenant lifecycle: export, retention, erasure (core)

Export + verifiable deletion are a compliance right — core in every edition. GET /v1/lifecycle/export (permission lifecycle.export) streams the portability bundle; GET/PUT /v1/lifecycle/retention + POST /v1/lifecycle/erase (permission lifecycle.erase, slug-confirmed, irreversible) manage retention and run the attested cross-store erasure — the attestation is the signed receipt enumerating what was deleted from which store, the document an auditor or a departing tenant actually gets to keep. The provider console adds the operator-side erase trigger. See docs/runbooks/tenant-offboarding.md for the full procedure and the per-store verification table. The data-class and purpose matrix for those clocks is data-retention.md.

Subject lifecycle uses the same tenant-first boundary for a narrower privacy request: a data subject is a person or identifier named by the tenant admin, for example an email address, user name, host-owned endpoint IP, or trace/log attribute. Use POST /v1/lifecycle/subjects/export with {"subject":"alice@example.com","redact":true} for a subject portability bundle, and POST /v1/lifecycle/subjects/erase with {"subject":"alice@example.com","confirm":"alice@example.com","reason":"dsar"} to erase that subject. Both routes require the lifecycle permissions (lifecycle.export and lifecycle.erase) and intentionally use POST bodies so the subject never appears in URLs, proxy logs, or browser history. The receipt stores only a tenant-scoped subject hash, deleted/remaining counts per plane, and the report hash. The subject manifest is complete across privacy-relevant surfaces: flow and OTLP planes report exported/deleted row counts; immutable audit reports projected; TSDB metrics, topology labels, eBPF workload aggregates, endpoint latest views, and topology device nodes report their own export/delete receipts when the wired backend is subject-capable. RUM reports covered_by_parent because its host/path samples live in tsdb_metrics. The encrypted IR attribution sidecar reports retained_encrypted_evidence: ordinary subject export and erasure cannot read, decrypt, or delete it, while the separately authorized and audited IR investigation path remains available. Aggregate backends that cannot locally delete one subject report not_capable with the delete-series or retention age-out basis instead of disappearing from the receipt.

Audit subject erasure is layered on the append-only audit chain. A privacy.subject_erase marker and its tenant-routed audit_subject_erasures projection row store only a tenant-scoped subject hash; later audit reads/exports replace matching structured actor/target/data values with an erased token while preserving the raw chain links. The projection and every stable-alias marker commit atomically with all classified Postgres subject deletions; any alias-marker failure rolls the whole tenant transaction back. Audit retention also captures marker-only events from rolling old writers before pruning their exported prefix, so a marker never becomes a permanent retention barrier and redaction never expires with that event row. Subject portability exports match this hash directly, and both subject and full-tenant bundles serialize audit rows through the canonical projection while preserving tenant, sequence, timestamp, and chain fields. WORM provider exports are minimized before signing, so write-once audit evidence does not become a second forever-copy of personal actor/data values.

The daily lifecycle sweeper enforces most tenant policy fields from GET/PUT /v1/lifecycle/retention: flow_retention_days, otel_retention_days, ebpf_retention_days, path_retention_days, audit_retention_days, ai_answer_retention_days, object_retention_days, and derived_identity_retention_days. NULL means the deployment default remains in force; a tenant value can only tighten the clock. An audit_retention_days value above a positive PROBECTL_AUDIT_RETENTION maximum is rejected with HTTP 422 before the policy or its audit event is written. Stores with a safe tenant+cutoff delete hook prune and append a lifecycle.retention_sweep receipt. Aggregate or externally governed stores append delegated/not-capable receipts that name the owning clock (object-store lifecycle or backend TTL) instead of over-claiming. Audit is handled by its hourly runner, which emits the tamper-evident audit.retention_prune receipt only with an actual exported-prefix prune and records that tenant's effective window. Prometheus-mode TSDB series deletion remains a documented manual step.

Per-tenant metering & quotas (ee/)

No configuration keys: metering activates with a license granting metering (provider/MSP tier). Counters flush every minute; gauge snapshots run every 15 minutes; usage and quotas live in Postgres (migration 0026). The usage API, the CSV/JSONL billing-export feed, per-tenant quotas (creation-gating only — telemetry is never quota-dropped), and the console showback card are documented in docs/metering.md.

Per-tenant key isolation / BYOK (ee/)

BYOK is bring your own key: a tenant supplies (and can revoke) the encryption key its data is sealed under, instead of trusting the operator's. Unlocked by the byok feature (Enterprise). The keyring wraps managed tenant KEKs under PROBECTL_ENVELOPE_KEY (required when byok is licensed — startup fails loudly without it) and resolves BYOK references through the secret backends. Surfaces: GET/POST /v1/security/keys[...] (permission security.keys) + the Admin → Encryption keys card. The full model — sealing formats, rotation, the BYOK lockout warning, crypto-offboarding — is in docs/byok.md.

A BYOK reference submitted through POST /v1/security/keys/rotate is untrusted tenant input that the control plane resolves with its own secret-store credentials. PROBECTL_BYOK_REF_PREFIX fences which references a tenant may bind so a tenant admin cannot aim the resolver at any secret it can read (confused deputy) or store literal key material in the DB (AUTHZ-10; docs/guardrails.md G7-1, G7-6):

Tenant fairness (core)

These are the per-tenant bounds that protect a pooled (shared) deployment, so one noisy tenant can't starve the others — and they are core in every edition. The ingest-rate and query bounds are on by default with conservative numbers; you opt out of a bound by setting it to an explicit 0 (unlimited). Unset keeps the default, and a negative value is a startup error — config validation rejects it. Per-tenant overrides are set from the provider console into tenant_fairness. Full model: docs/fairness.md.

These are token-bucket rate limits: the steady rate is the value below, and the bucket can hold a burst of rate × PROBECTL_FAIRNESS_BURST_SECONDS. Telemetry over a bound is admission-controlled (shed + counted), never silently corrupted.

Multi-region / active-active HA (core)

Inert unless PROBECTL_REGION is set (single-region deployments need none of these). The control plane stays stateless and active in every region; the split-brain fence pauses API writes during a failover while reads + telemetry keep flowing — split-brain being the failure where two regions each believe they hold the writable database and diverge, which a write-pause makes impossible. Two vocabulary keys below: RPO (recovery point objective) is how much recent data a disaster may cost you, RTO (recovery time objective) how long recovery may take — both recorded here as declared targets for humans to sign off, not values the software can promise. Full model + the failover runbook: docs/multi-region.md, docs/runbooks/region-failover.md.

The writer must be reachable for API writes; cluster_state (migration 0032) holds the promotion epoch the fence reads. Promotion is cluster_promote() in the failover runbook.

Supportability (core)

Deep health + a secret-stripped support bundle for triage (CORE; the support org/SLA is contract). No new config keys; diagnostics.read (migration 0034, admin-seeded) gates GET /v1/diagnostics and GET /v1/diagnostics/bundle. An offline bundle: probectl-control support-bundle [-o file]. Self-monitoring series probectl_self_* + probectl_build_info feed the native Dashboards surface. deploy/grafana/dashboards/probectl-self.json is an optional operator-supplied-client example, not a bundled runtime. The bundle NEVER contains secrets/credentials/PII (allowlist config + anonymized topology + a final scrub). Full model: docs/supportability.md.

Guarded agentic remediation (remediation, ee/)

The assistant PROPOSES remediations; a human APPROVES; probectl NEVER executes — there is no executor in the codebase (remediation is human-gated by design). Approve is a recorded, audited, blast-radius-limited, human-only sign-off that an operator carries out in their own change process; ingested data (e.g. a prompt-injection routed through the propose_remediation MCP tool) can at most create a proposed proposal a human must approve via the authenticated UI. The feature is hidden (404) when the remediation Enterprise feature is unlicensed.

Permissions remediation.propose and remediation.approve (migration 0035, admin-seeded) gate the /v1/remediation/* routes; the dry-run blast radius is a read-only topology simulation. Full policy + architecture: docs/remediation.md.

NDR-lite detection

NDR (network detection and response) spots attacker behavior in traffic patterns rather than matching known-bad lists. The detectors here: DGA — the algorithmically-generated throwaway domains malware uses to find its command server; exfiltration — data leaving in unusual volume or via covert channels (e.g. DNS); beaconing — the metronome-regular heartbeat of an implant calling home; plus unusual egress and lateral movement (an internal host suddenly exploring its neighbors). It runs entirely over locally-collected streams — no outbound calls — and, per the guardrails, only ever signals.

Detections are confidence-scored threat-plane signals (ndr.*) exported to incidents, the Security triage surface, and the SIEM (see SIEM export above). See docs/ndr.md for the detector and tuning reference.

Secrets integration

This is the feature that lets you keep raw passwords out of your config entirely. Anywhere this document asks for a credential, you can instead hand it a pointer to where the real secret lives — a Vault path, a CyberArk query, an AWS/Azure/GCP secret id — and the control plane fetches it at boot (or per poll, for device creds). The settings below just tell probectl how to reach each backend; the references themselves go in the credential keys documented throughout this page.

Any credential value in this document may be a secret reference instead of the literal material — env:NAME, vault:<mount>/<path>#<field>, cyberark:<query>, aws:<id>[#<json-field>], azure:<vault>/<name>, gcp:<project>/<secret>[/<version>], or literal:<value> as the escape hatch. The control plane resolves PROBECTL_OIDC_CLIENT_SECRET, PROBECTL_CMDB_SECRET, PROBECTL_AI_MODEL_TOKEN, PROBECTL_SIEM_TOKEN, PROBECTL_BUS_SASL_PASSWORD, PROBECTL_OUTAGE_RADAR_TOKEN, and the secret parts of PROBECTL_CHANGE_WEBHOOKS / PROBECTL_NOTIFY_CONNECTORS / PROBECTL_NOTIFY_INBOUND at startup (fail closed); the device agent resolves every PROBECTL_DEVICE_CRED_<NAME>_* value per poll cycle. Resolved values are cached only encrypted, for a short lease (5 m). See docs/secrets.md.

Backend access settings (environment only; all remote endpoints over verified TLS). Vault and CyberArk base URLs must use https://; plaintext http:// is accepted only for explicit loopback dev/test endpoints such as 127.0.0.1, ::1, or localhost. Two Vault terms used below: AppRole is Vault's login method for machines — a role id + secret id pair playing username/password for a service — and a lease is the expiry Vault stamps on what it issues, which probectl renews before it runs out:

Backend health (counters + redacted last error, never secret material) is served at GET /v1/secrets/health and on the Admin page.

Shipped all-in-one stack (deploy/compose/probectl.yml)

The production-shaped stack reads deploy/compose/.env (copy it from .env.example). Alongside the required secrets and the image/TLS/license keys documented there, these optional variables control host exposure and container resource ceilings. They are Compose-file interpolation variables (consumed by docker compose), not control-plane config read by the binary.

Every service in this stack ships hardened (RTO-03): the control plane and the certgen/migrate one-shots run the distroless control image with a read-only root filesystem (writable paths limited to an in-memory /tmp and the controldata volume at /var/lib/probectl), all Linux capabilities dropped, and no-new-privileges. The upstream Postgres image keeps its own root filesystem and capabilities (its entrypoint chowns the data dir and drops to the postgres user) but still runs with no-new-privileges. See deploy/compose/README.md.

Local dev stack (deploy/compose/dev.yml)

Started with make compose-up. Local, non-production defaults — plaintext listeners and dev credentials for convenience. Production deploys are TLS/HTTPS-by-default — TLS on every listener.

The disposable ClickHouse probectl user has access management enabled so make test-isolation can create and remove temporary tenant reader users and row policies. That is a local test/admin convenience, not the production credential model: production uses a separate bootstrap administrator to install policies and a least-privilege runtime service credential afterward.

Kafka listeners: host clients use localhost:9092; in-network containers use kafka:19092; the KRaft controller uses 9093 (internal — KRaft is Kafka's built-in consensus mode, so there is no separate ZooKeeper to run). Prometheus runs with --web.enable-remote-write-receiver so the result pipeline can remote-write into it.

These names and ports are a contract — the integration test harness depends on them, so don't rename them casually.

Tear-down

make compose-down removes the containers and volumes (pgdata, chdata, promdata).