Configuration

This is the full reference for every knob probectl reads at startup. The short version: the control plane and every server-side feature read environment variables (all prefixed PROBECTL_); the agents read a YAML file (with the same env vars as overrides). This page lists each variable, its default, and what it does — and it is the contract, so every row here is checked against the code.

How to read this page:

• A variable's default is what you get if you set nothing. The defaults are chosen so a fresh install boots in a safe, sovereign posture (no outbound calls, fail-closed on missing TLS/secrets) — probectl's standing security guardrails (see security/threat-model.md).
• A default of (none) means the value is empty/unset; the feature usually stays off until you give it one.
• Where it's read: the control plane resolves its config in internal/config/config.go (one Load function that reports every bad value at once and exits non-zero — you never chase config errors one at a time). Each agent has its own loader (internal/agent, internal/ebpf, internal/flow, internal/device, internal/endpoint).

Conventions

• Control plane (probectl-control): environment variables, PROBECTL_ prefix. Listed in the next section.
• Agents: a YAML config file is the source of truth; the matching PROBECTL_* env vars override individual fields (handy for containers). Each agent's keys are in its own section below.
• Secrets are never hardcoded, logged, or placed in URLs/query strings. Sensitive values at rest are sealed with envelope encryption, and any credential in this document may be a secret reference (e.g. vault:…) instead of the raw value — see Secrets integration.

Control plane (probectl-control)

The control plane is the brain: it serves the API/UI, accepts agent connections, runs the alerting/incident/correlation engines, and talks to the datastores. It is stateless — all durable state lives in Postgres/ClickHouse/the TSDB — so every behavioral choice it makes comes from these environment variables, read once at boot. The table below is the base set every deployment uses; the feature-specific sections that follow add more.

Subcommands: probectl-control [serve] (default), probectl-control migrate (apply database migrations and exit), probectl-control version, and probectl-control gen-cert [dir] — a convenience that writes a self-signed tls.crt/tls.key/ca.crt for an HTTPS quickstart (PROBECTL_CERT_HOSTS, default localhost,127.0.0.1, sets the certificate's host names; production brings its own CA-issued cert). The other subcommands are covered with their features: agent-ca init|export, enroll-token, and revoke-agent (agent transport and enrollment — agent/enrollment.md), scim-token (SCIM, below), mcp-stdio and mcp-token (MCP server, below), preflight (the storage-encryption preflight — hardening.md), support-bundle (supportability, below), and backup-seal / backup-open (sealed backups — Tenant lifecycle, below).

A note on the defaults: the listen address is :8080, the database DSN points at a local Postgres with sslmode=require (TLS to the database is the default, not an afterthought), and HSTS is on. These defaults assume you front the process with a TLS-terminating ingress (the shipped Helm/compose posture); set the TLS cert/key pair below to have the process serve HTTPS itself instead.

Invalid values fail fast: probectl-control reports all configuration problems at once and exits non-zero. The database password is redacted from logs.

Tenant-owned tables are protected by Postgres Row-Level Security. The PROBECTL_DATABASE_URL role must be able to assume the least-privilege probectl_app role (a superuser always can; otherwise run GRANT probectl_app TO <login_role>), which internal/tenancy assumes per transaction so isolation holds regardless of how the control plane authenticated. See architecture.md.

HTTP endpoints

Every response carries an X-Request-Id (honoring an inbound one) and the security headers Strict-Transport-Security (when enabled) and X-Content-Type-Options: nosniff. The versioned resource routes under /v1 are documented in Resource API & CLI below.

Error envelope

All errors share one JSON shape and a stable domain-error → HTTP mapping:

{ "error": { "code": "not_found", "message": "…", "request_id": "…" } }

Transport security

probectl never wants a plaintext channel exposed to the network. There are two correct ways to get TLS in front of the API, and the config lets you pick:

The API listens over TLS in two interchangeable ways:

• App-terminated TLS — set PROBECTL_TLS_CERT_FILE + PROBECTL_TLS_KEY_FILE, and the control plane serves HTTPS only (TLS 1.2+, prefer 1.3; plaintext is refused).
• Ingress-terminated TLS — leave them unset and serve HTTP behind a TLS-terminating ingress (the shipped Helm/compose default). HSTS is set either way, so the posture is correct end to end.

All TLS and crypto policy lives in internal/crypto; a CI guard (scripts/check_crypto_imports.sh) forbids crypto-primitive imports elsewhere so a FIPS 140-3 validated module can be swapped in. At-rest secrets use the envelope helper (a per-record data key wrapped by a KMS/HSM-pluggable KEK; the dev StaticKeyProvider reads PROBECTL_ENVELOPE_KEY).

Agent transport

This is how agents talk to the control plane, and it is locked down by design. The agent gRPC transport (probectl.agent.v1.AgentService) runs only when PROBECTL_AGENT_GRPC_ADDR and all three PROBECTL_AGENT_TLS_* files are set (address + server cert + server key + the CA that signs client certs). It is mutual-TLS only (RequireAndVerifyClientCert): the agent must present a client certificate, and its tenant and id are read out of that certificate's identity (spiffe://probectl/tenant/<t>/agent/<a>), never from the request body. So even a misbehaving or malicious agent can only ever write to its own tenant — the identity is cryptographic, not self-asserted. Populate PROBECTL_AGENT_TLS_CA_FILE (the client-cert CA pool) with probectl-control agent-ca export <file>, which writes the public agent-CA bundle (root + intermediate, no key). Generate dev mTLS material with the internal/crypto CA helpers. The .proto lives under proto/probectl/agent/v1/; regenerate Go with make proto (tools via make proto-tools).

Version-skew policy. At registration the control plane rejects agents outside the supported version window, so a rolling upgrade never admits an incompatible agent. See lifecycle.md.

A rejected agent gets a gRPC FailedPrecondition ("upgrade required"); a dev/unpinned build (0.0.0-dev) on either side skips the check.

probectl-agent

The canary agent is the worker that actually runs the probes (ping, TCP, DNS, HTTP, …). Unlike the control plane, its primary config is a YAML file (-config, or the path in PROBECTL_AGENT_CONFIG) — see deploy/agent/probectl-agent.example.yml. Crucially, the agent does not configure its own tenant or id: those come from its mTLS client certificate (above), so you can't accidentally point an agent at the wrong tenant by editing a file.

A handful of env vars override individual YAML fields — useful in containers where mounting a full file is awkward:

Results buffer to disk (buffer.dir, bounded by max_records, default 10000) while the control plane is unreachable and drain on reconnect (at-least-once delivery). Probing keeps running regardless of connectivity, so a control-plane outage never blocks measurement — the agent just queues and catches up.

Result pipeline

This is the path every measurement takes from an agent to a queryable metric, and two env vars decide how heavy that pipeline is: PROBECTL_BUS_MODE (the message bus) and PROBECTL_TSDB_MODE (the time-series writer). The memory defaults make a single binary work with zero external dependencies; switch them to kafka / prometheus when you outgrow that.

A streamed result flows agent → gRPC StreamResults → control-plane ingest → result bus (probectl.network.results, Protobuf) → consumer → time-series writer. The agent sends the canonical OTel-aligned result (proto/probectl/result/v1); the control plane re-stamps the tenant and agent id from the verified mTLS certificate before publishing, so a result is always attributed to the sending agent's tenant regardless of payload contents — the tenant boundary is cryptographic, never self-asserted. The bus key is the tenant_id.

PROBECTL_BUS_MODE selects the bus: memory (default; in-process, for the lightweight <5-agent deployment and single-binary runs) or kafka (set PROBECTL_BUS_BROKERS). PROBECTL_TSDB_MODE selects the writer: memory (default; in-process) or prometheus remote-write to PROBECTL_TSDB_URL (Prometheus with --web.enable-remote-write-receiver, or VictoriaMetrics; use an https:// URL for TLS in transit). Each probe emits probectl_probe_success, probectl_probe_duration_seconds, and one probectl_probe_<metric> per custom metric, labeled tenant_id, agent_id, canary_type, and server_address. The canonical signal→OTel mapping is in otel-mapping.md.

ICMP test

The icmp canary measures echo loss, latency, and jitter to a target (IPv4 or IPv6). Configure it per-canary under canaries: (see probectl-agent.example.yml). The schedule interval and reply timeout are canary fields; the rest are params:

It emits probectl_probe_loss_ratio, probectl_probe_rtt_{min,avg,max,stddev}_ms, probectl_probe_jitter_ms, and probectl_probe_packets_{sent,received}. A probe with 100% loss reports success=false (target unreachable); partial loss is a success with a non-zero loss ratio. Continuous mode records a per-second drop-timing record as result attributes (icmp.dropped_seqs, icmp.drop_send_offsets_ms) — carried as OTel attributes, not TSDB labels, so they don't widen cardinality.

Privileges. By default the agent uses unprivileged datagram ICMP (IPPROTO_ICMP), which on Linux requires the agent's group to be within net.ipv4.ping_group_range (e.g. sysctl -w net.ipv4.ping_group_range="0 2147483647"). Alternatively grant raw-socket capability (setcap cap_net_raw+ep /usr/bin/probectl-agent, or run with CAP_NET_RAW) and set privileged: "true". The canary tries the preferred socket and falls back to the other; if neither can be opened it returns an internal error (the probe is not silently reported as loss).

TCP & UDP tests

The tcp and udp canaries are agent-to-server probes. Configure a target of host:port (or a host with params.port). Both accept count and dscp.

The tcp canary measures connect latency + reachability (a connect-based, unprivileged equivalent of a TCP-SYN test): it establishes a connection and times the handshake, emitting probectl_probe_connect_{min,avg,max,stddev}_ms, probectl_probe_jitter_ms, and probectl_probe_loss_ratio (failed connects = loss; all-failed = success=false).

The udp canary is an echo round-trip probe: it sends token-tagged datagrams and matches the echoes, emitting probectl_probe_rtt_* + loss. It needs a target that echoes (a UDP echo service, or a probectl agent-to-agent responder); a non-echoing target reports as 100% loss. params.payload_bytes (≥10) sets the datagram size.

Voice/RTP tests

The voice canary streams real RTP packets at codec cadence to an echoing target and scores the path: MOS + R-factor (simplified ITU-T G.107 E-model), RFC 3550 jitter, loss, and a one-way delay estimate. target is host:port. Parameters: codec (g711 default, g729), duration_seconds (1–10, default 3), dscp (default 46/EF). The model variant and the one-way-estimate method ride the result attributes — a computed MOS is never presented as a measured listening score. See docs/voice.md.

DNS tests

The dns canary queries DNS and reports resolution time, the answer, and an optional DNSSEC verdict. The target is the query name. Parameters:

server defaults by transport: the first nameserver in /etc/resolv.conf (or 1.1.1.1:53) for udp/tcp, 1.1.1.1:853 for DoT, and https://cloudflare-dns.com/dns-query for DoH. DoT verifies the resolver's TLS certificate (TLS 1.2+); DoH posts an RFC 8484 application/dns-message query over HTTPS.

In resolver mode the canary emits probectl_probe_dns_query_ms (round-trip) and probectl_probe_dns_answers (answer count), with dns.rcode and a compact dns.answer summary as attributes. The probe is success=false on a non-NOERROR rcode or an empty answer.

With dnssec: "true" the canary requests DNSSEC records (the DO bit) and validates the zone's RRSIG over the answer against the zone DNSKEY — it does not trust the resolver's AD bit. The verdict lands in the dns.dnssec attribute (secure, insecure for an unsigned zone, or bogus) and probectl_probe_dns_dnssec_secure (1/0); a bogus result (tampered, expired, or wrong-key signature) fails the probe. Validation verifies the signature on the answer RRset; full chain-to-root anchoring is a later refinement.

In trace mode the canary performs an iterative delegation walk from the root hints, following NS/glue referrals down to the authoritative server (UDP, capped iterations, with a recursive fallback when a referral ships no glue). It emits probectl_probe_dns_query_ms (total walk time) and probectl_probe_dns_trace_hops, with the delegation chain in the dns.trace attribute. DNS-exfiltration detection and open-data baselines are out of scope for this probe (they live in the NDR and open-data features).

HTTP server tests

The http canary measures HTTP(S) availability with a full response-time breakdown and captures TLS handshake details for the TLS-posture plane (see TLS / certificate observability below). The target is the URL. Parameters:

expect_status is a comma list of exact codes (200), classes (2xx), and inclusive ranges (200-204); a response outside the set is success=false (the status is still reported). The probe emits the timing breakdown as metrics — probectl_probe_http_dns_ms (resolution), probectl_probe_http_connect_ms (TCP connect), probectl_probe_http_tls_ms (TLS handshake), probectl_probe_http_ttfb_ms (time to first byte), and probectl_probe_http_total_ms — plus probectl_probe_http_status, probectl_probe_http_content_bytes, and probectl_probe_http_throughput_kbps. A phase that does not occur (no DNS for an IP target, no TLS for http://) is omitted rather than reported as zero. The resolved server IP is captured as the network.peer.address attribute, which correlates the result to path/traceroute data for the same destination.

TLS capture. On HTTPS the canary records the negotiated tls.protocol.version and tls.cipher, the leaf certificate's tls.server.{subject,issuer,not_before,not_after,san}, the chain shape (tls.server.chain), and a probectl_probe_http_tls_cert_expiry_days metric (negative once expired). It verifies the chain itself (hostname + trust, honoring ca_file) after capturing the certificate, so the handshake details are recorded even when the certificate is invalid or expired — an invalid cert fails the probe but its details are still attached. Set insecure_skip_verify: "true" to capture posture without failing the availability check. probectl performs no TLS posture analysis here (issuer trust, weak-cipher/expiry policy, CT) — that is the TLS / certificate observability feature below, which consumes these captured fields.

Agent-to-agent tests

An agent-to-agent (A2A) test measures between two registered agents, brokered by the control plane. The control plane assigns roles (one agent responds, opening a short-lived listener; the other initiates), rendezvouses the responder's endpoint to the initiator, and hands each agent its task when it polls (PollCoordination / ReportEndpoint). The measurement is TWAMP-lite: the initiator timestamps each probe (T1), the responder stamps receive/send (T2/T3) and echoes, and the initiator stamps receive (T4), yielding round-trip (probectl_probe_rtt_*) plus forward and reverse one-way delay (probectl_probe_forward_avg_ms, probectl_probe_reverse_avg_ms). The responder also reports forward-direction delivery (probectl_probe_packets_received, probectl_probe_loss_ratio), so both agents and both directions are observed.

Enable participation in the agent's a2a block: enabled: true, advertise_host (the address peers use to reach this agent's responder), poll_interval (default 2s), and responder_ttl (default 15s). Caveats (document for production):

• NAT/firewall. The responder advertises advertise_host; behind NAT this must be a reachable address and the responder's ephemeral port must be reachable from the initiator. Auto-detection picks a non-loopback IPv4 — set advertise_host explicitly when that is wrong.
• Clocks. Forward/reverse one-way delays assume the two agents' clocks are synchronized (exact within one host; use NTP across hosts). Round-trip is clock-independent.

Sessions are brokered in-memory; triggering them from the test API is a later addition.

Path discovery

The path engine (internal/path) is the traceroute brain — it runs Paris-style traceroutes (ICMP and TCP), which handle equal-cost multipath (ECMP) and MPLS, and merges per-flow traces into one multi-path picture; see architecture.md. A full per-hop trace needs raw sockets: grant CAP_NET_RAW (setcap cap_net_raw+ep, or run privileged) to capture the intermediate hops + MPLS labels. Without it, only the destination is discovered.

Where the discovered hops/links are stored is a control-plane choice:

BGP routing intelligence

The BGP plane is a Python analyzer (analyzer/) plus a Go bridge (internal/bgp); see architecture.md. The analyzer ingests public collector data and emits probectl.bgp.events:

python -m probectl_analyzer --config config.json --mrt rib.mrt        # RouteViews/RIS dump
python -m probectl_analyzer --config config.json --replay cap.jsonl   # recorded RIS Live
python -m probectl_analyzer --config config.json --ris-live           # live RIS Live websocket

The JSON config is per tenant (tenant_id is required — every event carries it, and the bridge rejects any event without one):

The analyzer emits probectl.bgp.events as JSON Lines; the Go bridge tails that stream, validates the tenant, and republishes each as the canonical probectl.bgp.v1.BGPEvent protobuf onto the bus (topic probectl.bgp.events, keyed by tenant). Event types: origin_change (old/new origin + AS path), possible_hijack, possible_leak, rpki_invalid; each carries an RPKI status (valid / invalid / not_found / unknown), a severity, and a confidence — they are signals, not actions — probectl never acts on routing. MRT dumps are stream-processed (no full RIB in memory); a down RPKI/collector source degrades gracefully rather than breaking the plane. RouteViews/RIS are open data — their AUP/provenance matters for MSP/commercial resale, not for private development or single-tenant OSS use.

Open-data enrichment

internal/opendata annotates IPs with ASN / geo / IXP / allocation context from public datasets; see architecture.md and the source provenance/AUP matrix in opendata-aup.md. The framework is a library (the flow and test pipelines consume it where enrichment is enabled); each source is pluggable and individually enable-able:

The Enricher runs every enabled source over an IP and merges the results, caching per IP and degrading gracefully: a disabled / failing / slow / panicking source is logged, marked degraded or disabled in Enricher.Status(), and skipped — a partial enrichment is returned and a down dataset never breaks a core path. Sources run in registration order (register the ASN source before PeeringDB). Each contribution records Provenance (source + license + attribution

• fields); a source's AUP (license, commercial-use permission, attribution) is on its Descriptor — the matrix that gates MSP/commercial resale (not private or single-tenant OSS use). All fetches are over TLS with certificate validation and treated as untrusted — external content never gets implicit trust. Open data is ingested once and shared; enrichment is scoped per tenant by the consuming record.

Alerting

The alerting engine (internal/alert) evaluates rules over the TSDB and notifies channels; see architecture.md. Rules are CRUD'd via /v1/alerts (tenant-scoped) and the engine runs in the control plane, ticking every PROBECTL_ALERT_EVAL_INTERVAL (default 30s).

A rule targets a metric series and is either a threshold or a baseline rule:

A channels entry is {"type":"webhook","url":...,"secret":...} or {"type":"email","recipients":[...]}. The webhook secret is the HMAC key; it is redacted (***) from API responses and never returned. SMTP for email is configured at the deployment level (a follow-up exposes it as config).

Webhook payload (probectl.alert.v1). On fire/resolve the webhook channel POSTs:

{
  "version": "probectl.alert.v1",
  "state": "firing",
  "rule": { "id": "…", "name": "loss-high" },
  "tenant_id": "…",
  "severity": "critical",
  "metric": "probectl_probe_loss_ratio",
  "labels": { "server_address": "1.1.1.1" },
  "value": 0.9,
  "threshold": 0.5,
  "comparison": "gt",
  "reason": "probectl_probe_loss_ratio=0.9 gt 0.5",
  "fired_at": "2026-01-02T15:04:05Z"
}

When the channel has a secret, the request carries X-Probectl-Signature: sha256=<hex> — the HMAC-SHA256 of the exact body — so the receiver can verify the sender. Each channel delivers independently: a failing channel is logged and skipped, never blocking the others. Alerts are signals; probectl notifies and does not act on the network (on-call/ITSM routing and detection-as-code are their own features below).

Incidents

The incident correlator (internal/incident) groups related signals across planes into one Incident with a unified timeline; see architecture.md. It runs in the control plane, fed by the alert engine (network plane) and a probectl.bgp.events consumer (BGP plane), and is exposed at /v1/incidents (tenant-scoped):

• GET /v1/incidents — the tenant's incidents, most-recently-active first.
• GET /v1/incidents/{id} — an incident with its time-ordered signal timeline.
• PATCH /v1/incidents/{id} with {"status":"resolved"} — resolve an incident.

Signals correlate into one incident when they are close in time (within PROBECTL_INCIDENT_WINDOW, default 10m) and related in target — the same target, an IP inside the other's prefix (either direction), or overlapping prefixes (so a network alert on 192.0.2.10 and a BGP event on 192.0.2.0/24 land together). An incident's severity is the max of its signals; a signal without a tenant is rejected (fail closed).

The model is extensible without schema churn: a Signal carries a free-form plane/kind and an arbitrary attributes map, so the change, threat, cost, and SLO planes attach as additional signal types onto the same Incident/timeline without schema changes. AI root-cause analysis runs over the timeline.

SSO & RBAC

probectl authenticates users with OIDC SSO and authorizes them with role-based access control (RBAC). The security order is the two-level boundary: a request resolves to exactly one tenant first, then RBAC decides whether the caller may perform the route's action within that tenant.

Login flow. GET /auth/login (optionally ?tenant=<uuid>) starts the OIDC authorization-code flow: it sets a short-lived, HttpOnly CSRF state cookie and redirects to the tenant's identity provider. The IdP redirects back to GET /auth/callback, which verifies the state, exchanges the code, verifies the ID token, just-in-time provisions the user within the tenant (a brand-new user gets no roles — a secure default; an admin grants access), mints a server-side session, and sets the session cookie. POST /auth/logout revokes the session. GET /v1/me returns the caller's tenant, identity, and effective permissions.

Sessions. A session is a random, high-entropy opaque token. Only its hash is stored (table sessions), so a database read cannot mint a session. The session cookie is HttpOnly + SameSite=Lax, and Secure whenever the API serves HTTPS. PROBECTL_SESSION_TTL (default 12h) bounds its lifetime.

Per-tenant IdP. Providers are resolved per tenant through a provider factory — the seam for a tenant bringing its own SSO. The shipped default is the env-configured one (PROBECTL_OIDC_*); database-backed per-tenant IdP config is a later addition. A login always resolves to a single tenant. Provider/MSP operators authenticate into the provider domain (the management plane), not into tenant data.

RBAC. Every /v1 route declares a required permission key; the wrapped handler returns 401 when unauthenticated and 403 when the principal lacks the permission — checked before the handler runs. Effective permissions are loaded per request from the user's role bindings (RLS-scoped to the tenant), so a role grant or revoke takes effect immediately. The permission catalog:

The seeded system roles for the default tenant are admin (all permissions), editor (read everything + manage tests/alerts/incidents), and viewer (read-only). GET /v1/me requires only authentication (no specific permission).

Dev mode. PROBECTL_AUTH_MODE=dev bypasses SSO and synthesizes an all-permissions principal for the default tenant, with the X-Probectl-Tenant: <uuid> override for multi-tenant dev. It is triple-gated: (1) the code path exists only in binaries built with -tags devauth (make build-devauth) — a release binary refuses to start in this mode; (2) PROBECTL_DEV_AUTH_ACK=i-understand must be set; (3) the listener must bind loopback (PROBECTL_HTTP_ADDR=127.0.0.1:…). When active it logs at error level and writes an auth.dev_mode_active audit event. The CI gate no-devauth-in-release proves release binaries contain neither the symbols nor the dev-principal literal. The test suite installs its own hook in _test.go files, which never ship.

Resource API & CLI

The versioned resource API lives under /v1 (full schema at /openapi.json):

• GET/POST /v1/tests, GET/PUT/DELETE /v1/tests/{id} — synthetic-test CRUD.
• GET /v1/agents, GET/PATCH/DELETE /v1/agents/{id} — agents register over mTLS; the API lists, renames, and deregisters them.
• GET/POST /v1/tests/{id}/path — the latest discovered network path for a test, and a trigger to discover it now. The Path & Topology UI consumes this.

Every /v1 route is tenant-scoped through internal/tenancy + Postgres RLS, so a request can never read or write across tenants. Authentication and RBAC are real (see SSO & RBAC below): the caller's tenant and effective permissions come from an authenticated session, and each route requires a permission. The "no undocumented routes" rule is enforced by a test that matches the route table against openapi.json.

The probectl CLI is the web-parity client. Configure it with flags or environment: PROBECTL_API_URL (default http://localhost:8080), PROBECTL_API_TOKEN (sent as Bearer), PROBECTL_TENANT (sent as X-Probectl-Tenant).

probectl test list
probectl test create --name edge-dns --type icmp --target 1.1.1.1 --interval 30
probectl test delete <id>
probectl agent list
probectl --json test list      # machine-readable output

eBPF host agent

The eBPF agent watches a host's network from inside the Linux kernel — it sees which processes talk to which services without you instrumenting anything. It is observe-only: it never blocks or modifies traffic. Like the canary agent, its real config is a YAML file (-config / PROBECTL_EBPF_CONFIG); see deploy/agent/probectl-ebpf-agent.example.yml and ebpf-agent.md, with PROBECTL_EBPF_* env vars overriding individual fields. The in-kernel loader is compiled in only with the ebpf build tag; without it (or for tests), point fixture_path at a recording to replay.

The big idea in the keys below: layer-7 plaintext capture is off, and stays off until you prove three separate intents — turn it on (L7_CAPTURE), name the tenant that consents (L7_CONSENT_TENANT), and list the exact workloads (L7_SCOPE). Miss any one and the kernel copies no payload. That is the fail-closed posture for the most sensitive thing this agent can do.

Flows + service edges are published to probectl.ebpf.flows (ebpfv1.FlowBatch, tenant-keyed). The live loader needs a BTF Linux kernel (≥5.8) and CAP_BPF/CAP_PERFMON; see ebpf-agent.md.

Agent→bus TLS/SASL (eBPF, endpoint, flow, and device agents)

When a telemetry agent publishes straight to Kafka, its broker connection takes the same hardening keys as the control plane's PROBECTL_BUS_* set, under the agent's own prefix: PROBECTL_EBPF_BUS_* here, and likewise PROBECTL_ENDPOINT_BUS_*, PROBECTL_FLOW_BUS_*, and PROBECTL_DEVICE_BUS_* for the agents below. The policy is the same fail-closed one: kafka mode without TLS refuses to start unless the explicit dev-only plaintext flag is set. (The canary agent has no bus keys — it talks gRPC/mTLS to the control plane, which publishes on its behalf.)

Endpoint / DEM agent (probectl-endpoint)

"DEM" is digital experience monitoring: this agent runs on an end-user's laptop (Linux/macOS/Windows), measures their actual last-mile experience, and figures out whether a slowdown is the WiFi, the ISP, or the network. Because it sits on a personal device, its defaults are privacy-first — it collects the WiFi name and gateway (useful, low-risk) but not the AP MAC or public hop IPs (which can geolocate a person), and it discloses exactly what it collects on startup. It reads a YAML config (default path PROBECTL_ENDPOINT_CONFIG); PROBECTL_ENDPOINT_* env vars override it. See endpoint-dem.md.

Results (WiFi / gateway / last-mile / session signals + the attribution verdict) are published to probectl.endpoint.results (resultv1.Result, tenant-keyed), flowing through the same pipeline as every other canary. The agent discloses exactly what it collects at startup and never phones home.

Flow collector (probectl-flow-agent)

The flow collector listens for NetFlow v5/v9, IPFIX, and sFlow v5 datagrams from network devices, decodes them (template + sampling handling), and publishes normalized batches to probectl.flow.events (flowv1.FlowBatch, tenant-keyed). It reads a YAML config (default path PROBECTL_FLOW_CONFIG); PROBECTL_FLOW_* env vars override the file. The defaults serve all three protocols on their standard ports (NetFlow :2055, IPFIX :4739, sFlow :6343). See flow.md for the security posture: flow export is plaintext UDP by design, so every datagram is treated as untrusted and the collector should sit adjacent to its exporters (not exposed to the wider network).

The control plane consumes that flow topic, optionally enriches each record with ASN/geo, and persists to the flow store behind /v1/flows/* (top-talkers / capacity / anomalies). These are control-plane keys (not flow-agent keys):

Device telemetry agent (probectl-device-agent)

This agent reads metrics straight off network gear (routers, switches). It polls the old way (SNMP v2c/v3) and subscribes the modern streaming way (gNMI/OpenConfig), normalizes both into one DeviceMetric shape, and publishes to probectl.device.metrics (tenant-keyed); the control plane lands them in the TSDB as probectl_device_* series. The full device list lives in a YAML config (see deploy/agent/probectl-device-agent.example.yml); the env vars below override it and give a single-device quick start for trying one device fast. See device-telemetry.md.

Credentials are referenced by NAME, never inlined — no secrets in the device list. The default credential source resolves those names from the environment (the PROBECTL_DEVICE_CRED_<NAME>_* vars below); the secrets backends plug Vault/CyberArk into the same seam. An unresolvable name fails closed at startup. <NAME> is the upper-cased credential name with -/. → _:

gNMI connections are TLS with certificate verification (system roots or a per-device ca_file); there is no skip-verify option. plaintext: true is an explicit lab-only YAML opt-in and is loudly logged — never a silent plaintext default.

OTLP receiver

This lets other systems push their OpenTelemetry data (metrics, traces, logs) into probectl. It is off by default and, when on, is locked to the same posture as everything else: TLS-only, token-authenticated, tenant-scoped, on its own listeners separate from the /v1 REST API. There is no anonymous-plaintext mode — setting a listen address without both a TLS cert/key pair and at least one bearer token fails config validation. See otlp.md.

Setting an address without the TLS files and at least one token fails config validation — the receiver is never anonymous plaintext. Ingested metrics are tenant-tagged and published to the probectl.otlp.metrics bus topic.

Ecosystem integrations

The Grafana datasource API (/v1/grafana/api/v1/*), the federation endpoint (/v1/prometheus/federate), and the remote-write receiver (/v1/prometheus/write) ride the existing TSDB config (PROBECTL_TSDB_MODE / PROBECTL_TSDB_URL) and the /v1 API listener — no extra keys. Reads need metrics.read, remote-write metrics.write (migration 0022). See ecosystem-integrations.md.

The ServiceNow CMDB correlation is off unless configured:

AI assistant

Worked per-provider setups (Ollama, vLLM, OpenAI, Anthropic, Azure) are in ai-rca.md → Copy-paste recipes; the remote-egress enablement chain (operator ack + per-tenant consent) is in ai-egress.md.

The assistant (root-cause analysis + natural-language query) works out of the box with zero network access — the default builtin provider is an in-process synthesizer that writes its answers locally. You only point it at a real language model if you want nicer prose, and doing so is treated as data egress: a remote endpoint must be https, and you have to explicitly acknowledge that tenant data will leave (PROBECTL_AI_EGRESS_ACK). A loopback endpoint may be http (for a local model on the same box). The redaction keys below mask sensitive values before anything reaches an external model. See ai-rca.md.

A non-builtin provider without an endpoint fails config validation. Whatever the backend, every answer is tenant- and RBAC-scoped by the query layer and every claim is citation-checked before it reaches the user — a model can never see out-of-scope data or inject an ungrounded claim.

MCP server

The MCP server exposes read-only, tenant- + RBAC-scoped tools to AI clients. The HTTP transport is off by default and is TLS-only + bearer-authenticated; the stdio transport is local (probectl-control mcp-stdio, token from PROBECTL_MCP_TOKEN). Mint a token with probectl-control mcp-token --user <user-uuid> [--tenant <uuid>] [--name <label>] — the token prints once and only its hash is stored, so a database read can never recover it. See mcp.md.

Setting PROBECTL_MCP_HTTP_ADDR without the TLS files fails config validation — the MCP endpoint is never anonymous plaintext.

TLS / certificate observability

The control plane analyzes TLS/cert posture from TLS handshakes the HTTP and eBPF-L7 probes already captured — it never re-handshakes a target itself — and correlates the findings into threat-plane incidents. See tls-observability.md.

CT correlation is off by default (an external fetch — sovereignty / AUP / rate limits) and degrades gracefully when the CT source is down.

Threat-intel enrichment

The control plane can match peer IPs / hostnames / certs / JA3 against public threat-intel feeds, surfacing confidence-scored, source-attributed threat-plane signals (a signal, not an IPS — never blocks). See threat-intel.md for the feed/AUP matrix and caveats.

Off by default (an outbound fetch — sovereignty / no-phone-home). The refresher keeps each source's last-good indicators, so a feed outage degrades gracefully and never breaks a core path.

Enterprise identity: SCIM + ABAC

SCIM 2.0 provisioning and ABAC have no environment keys — the SCIM bearer token an IdP presents is minted with the control-plane CLI, and ABAC policies are managed over the API. See scim-abac.md.

# mint a per-tenant SCIM token for an IdP (shown once)
probectl-control scim-token --tenant <tenant-uuid> --name okta

The /scim/v2/* surface is gated by a valid SCIM token (no token ⇒ 401), and the directory-admin API (/v1/abac/policies) requires directory.read/directory.write.

Change intelligence

Ingest per-provider-signed change webhooks (deploys/config/route/IaC/commits) into a change timeline + change-to-incident correlation, feeding the AI RCA. See change-intel.md for the webhook contract + provider/signature table.

Each inbound delivery is TLS + signature-verified (HMAC/token, constant-time) + tenant-bound to the credential; an unsigned or forged event is rejected before storage, and one tenant cannot inject another's changes. Webhook secrets are runtime config — inject them from a secret manager, never commit them.

SIEM export

Forward the audit stream and threat-plane signals to a SOC's SIEM over hardened TLS. probectl is the forwarder, not a SIEM — events are rendered into a standard format and pushed; nothing is auto-blocked. See siem.md for formats, delivery guarantees, and per-SIEM setup.

Off by default (an outbound connection — sovereignty / no-phone-home). Audit forwarding resumes from a durable per-tenant cursor, and delivery retries without dropping under a SIEM outage. Exported audit events are PII/secret redacted (built-in denylist + PROBECTL_SIEM_REDACT_KEYS).

On-call + ITSM integration

Mirror incidents into operational tooling: page on-call (PagerDuty/Opsgenie), post to chat (Slack/Teams), and open + bidirectionally sync tickets (ServiceNow/Jira). probectl is the forwarder, not the system of record — it never auto-blocks anything. See oncall-itsm.md for the connector matrix, mapping, and the inbound webhook contract.

Off by default (each connector is an outbound connection to the operator's tooling). Paging + ticket creation are idempotent (an incident opens at most once per connector — a retry/restart never double-pages), status sync is bidirectional with loop protection (an inbound resolve from one system is never echoed back to it), and routing is per-tenant (a connector only fires for its own tenant). Endpoint specifics: a Slack/Teams endpoint is the incoming-webhook URL; a Jira endpoint carries the project (and optional resolve transition) as query params, e.g. …/rest/api/2/issue?project=OPS&resolve_transition=31; a ServiceNow endpoint is the …/api/now/table/incident URL. Inbound deliveries must include X-Probectl-Signature: sha256=<hmac> or X-Probectl-Token: <secret> over TLS; an unsigned or forged delivery is rejected (401). Secrets are runtime config — inject them from a secret manager, never commit them.

Topology graph + what-if

The graph feeds from eBPF/BGP/device streams + path discoveries; served at GET /v1/topology with what-if simulation at POST /v1/topology/whatif. See docs/topology.md.

FinOps / egress cost

Summary at GET /v1/cost/summary and the Cost page; deep dashboards are federated to Grafana (see Ecosystem integrations above). See docs/finops.md.

SLO engine

Statuses at GET /v1/slos, OpenSLO export at GET /v1/slos/openslo, and the SLOs page. See docs/slo.md.

Compliance / segmentation validation

Verdicts at GET /v1/compliance, hash-chained audit evidence at GET /v1/compliance/evidence, and the Compliance page. See docs/compliance.md.

Collective internet-outage view

The collective view at GET /v1/outages (events + the caller-tenant's affected tests + vantage detections + feed AUP/health + coverage notes) and the Internet outages page. Scope resolution (IP→ASN/country) rides the open-data enricher (PROBECTL_FLOW_ENRICH_ASN); without it the response reports the degradation honestly. See docs/outage.md.

RUM convergence

Beacons ingest at POST /ingest/rum (app-key authenticated, consent-gated, URL-redacted, no IP stored — privacy is enforced server-side, fail closed); the convergence view serves at GET /v1/rum and folds into the Endpoints surface; rum.* vitals flow to the TSDB for dashboards. The SDK is web/public/probectl-rum.js. See docs/rum.md.

Carbon / power observability

Attribution reuses PROBECTL_COST_ZONES / PROBECTL_COST_SERVICES. The estimate serves at GET /v1/carbon and folds into the Cost page. See docs/carbon.md. The chaos injector and the large/extra-large scale gate are test-harness tools — see docs/chaos.md and docs/scale-gate.md.

Editions / license

Verification is offline — local signature math against public keys baked into the binary at build time (never an env var; never phone-home). Expiry runs the 30-day-grace → read-only ladder and never breaks running telemetry. License state + the feature→tier map serve at GET /v1/editions and render on Admin → Editions — the one place tiers appear when unlicensed. See docs/editions.md for the file format, the signing CLI (probectl-license), and the gating pattern.

Provider / management plane (ee/)

Active only when the license grants provider_plane; otherwise /provider/* is a plain 404 (hidden, not locked).

The provider plane additionally requires PROBECTL_ENVELOPE_KEY (operator TOTP secrets are envelope-sealed at rest) and a database. Operator MFA is mandatory; operators are a privilege domain distinct from tenant users with no implicit access to tenant telemetry — see docs/provider-plane.md for the model, the break-glass consent flow, and the storage-layer confinement (probectl_provider role). Suspending a tenant rejects its users at the API (tenant_suspended) without touching data or ingestion.

Siloed / hybrid isolation (ee/)

Pooled isolation stays the default and needs no configuration. Siloed and hybrid tenants (per-tenant Postgres schema / ClickHouse database / bus topic namespace / object key namespace) require a license granting siloed_isolation and are selected per tenant at provisioning (isolation_model + optional residency).

Residency pins the tenant's ClickHouse flow data in this release; Postgres control state, the TSDB, object storage, and bus brokers are NOT region-pinned yet — docs/isolation.md states the exact contract, the catch-up/migration story for silo schemas, and the offboard-teardown semantics.

White-label branding (ee/)

No configuration keys: branding activates with a license granting white_label and is configured per tenant (or as the provider master) from the provider console. The public GET /branding endpoint serves the resolved brand pre-auth (Host-resolved for custom domains; the probectl default when unlicensed); custom-domain login resolves the tenant from the serving host. Custom domains need a certificate at the TLS-terminating ingress (or via trustctl) — see docs/white-label.md for the token-override contract, the no-bleed rules, and the email-template contract.

Advanced data governance (governance, ee/)

Per-tenant data classification + redaction, composed with retention, residency, and BYOK. No new config keys: the classification + redaction MECHANISM is core (the ?redact=true export toggle works anywhere, masking PII with a partial strategy); the governance feature adds per-tenant POLICY (stored in tenant_governance, migration 0033) set from the provider plane (GET/PUT /provider/v1/tenants/{id}/governance). IPs are PII by default. Full model: docs/governance.md. Redacted export: GET /v1/lifecycle/export?redact=true.

Tenant lifecycle: export, retention, erasure (core)

Export + verifiable deletion are a compliance right — core in every edition. GET /v1/lifecycle/export (permission lifecycle.export) streams the portability bundle; GET/PUT /v1/lifecycle/retention + POST /v1/lifecycle/erase (permission lifecycle.erase, slug-confirmed, irreversible) manage retention and run the attested cross-store erasure. The provider console adds the operator-side erase trigger. See docs/runbooks/tenant-offboarding.md for the full procedure and the per-store verification table.

The daily retention sweeper enforces per-tenant flow_retention_days (tighter than the deployment TTL). Prometheus-mode TSDB series deletion is a documented manual step (the attestation says so honestly).

Per-tenant metering & quotas (ee/)

No configuration keys: metering activates with a license granting metering (provider/MSP tier). Counters flush every minute; gauge snapshots run every 15 minutes; usage and quotas live in Postgres (migration 0026). The usage API, the CSV/JSONL billing-export feed, per-tenant quotas (creation-gating only — telemetry is never quota-dropped), and the console showback card are documented in docs/metering.md.

Per-tenant key isolation / BYOK (ee/)

Unlocked by the byok feature (Enterprise). No new config keys: the keyring wraps managed tenant KEKs under PROBECTL_ENVELOPE_KEY (required when byok is licensed — startup fails loudly without it) and resolves BYOK references through the secret backends. Surfaces: GET/POST /v1/security/keys[...] (permission security.keys) + the Admin → Encryption keys card. The full model — sealing formats, rotation, the BYOK lockout warning, crypto-offboarding — is in docs/byok.md.

Tenant fairness (core)

These are the per-tenant bounds that protect a pooled (shared) deployment, so one noisy tenant can't starve the others — and they are core in every edition. The ingest-rate bounds are on by default with conservative numbers; you opt out of a bound by setting it to an explicit 0 (unlimited). Unset keeps the default, and a negative value is a startup error — config validation rejects it. The two query bounds already default to 0, i.e. unlimited until you set them. Per-tenant overrides are set from the provider console into tenant_fairness. Full model: docs/fairness.md.

These are token-bucket rate limits: the steady rate is the value below, and the bucket can hold a burst of rate × PROBECTL_FAIRNESS_BURST_SECONDS. Telemetry over a bound is admission-controlled (shed + counted), never silently corrupted.

Multi-region / active-active HA (core)

Inert unless PROBECTL_REGION is set (single-region deployments need none of these). The control plane stays stateless and active in every region; the split-brain fence pauses API writes during a failover while reads + telemetry keep flowing. Full model + the failover runbook: docs/multi-region.md, docs/runbooks/region-failover.md.

The writer must be reachable for API writes; cluster_state (migration 0032) holds the promotion epoch the fence reads. Promotion is cluster_promote() in the failover runbook.

Supportability (core)

Deep health + a secret-stripped support bundle for triage (CORE; the support org/SLA is contract). No new config keys; diagnostics.read (migration 0034, admin-seeded) gates GET /v1/diagnostics and GET /v1/diagnostics/bundle. An offline bundle: probectl-control support-bundle [-o file]. Self-monitoring series probectl_self_* + probectl_build_info feed deploy/grafana/dashboards/probectl-self.json. The bundle NEVER contains secrets/credentials/PII (allowlist config + anonymized topology + a final scrub). Full model: docs/supportability.md.

Guarded agentic remediation (remediation, ee/)

The assistant PROPOSES remediations; a human APPROVES; probectl NEVER executes — there is no executor in the codebase (remediation is human-gated by design). Approve is a recorded, audited, blast-radius-limited, human-only sign-off that an operator carries out in their own change process; ingested data (e.g. a prompt-injection routed through the propose_remediation MCP tool) can at most create a proposed proposal a human must approve via the authenticated UI. The feature is hidden (404) when the remediation Enterprise feature is unlicensed.

Permissions remediation.propose and remediation.approve (migration 0035, admin-seeded) gate the /v1/remediation/* routes; the dry-run blast radius is a read-only topology simulation. Full policy + architecture: docs/remediation.md.

NDR-lite detection

Detections are confidence-scored threat-plane signals (ndr.*) exported to incidents, the Security triage surface, and the SIEM (see SIEM export above). See docs/ndr.md for the detector and tuning reference.

Secrets integration

This is the feature that lets you keep raw passwords out of your config entirely. Anywhere this document asks for a credential, you can instead hand it a pointer to where the real secret lives — a Vault path, a CyberArk query, an AWS/Azure/GCP secret id — and the control plane fetches it at boot (or per poll, for device creds). The settings below just tell probectl how to reach each backend; the references themselves go in the credential keys documented throughout this page.

Any credential value in this document may be a secret reference instead of the literal material — env:NAME, vault:<mount>/<path>#<field>, cyberark:<query>, aws:<id>[#<json-field>], azure:<vault>/<name>, gcp:<project>/<secret>[/<version>], or literal:<value> as the escape hatch. The control plane resolves PROBECTL_OIDC_CLIENT_SECRET, PROBECTL_CMDB_SECRET, PROBECTL_AI_MODEL_TOKEN, PROBECTL_SIEM_TOKEN, PROBECTL_BUS_SASL_PASSWORD, PROBECTL_OUTAGE_RADAR_TOKEN, and the secret parts of PROBECTL_CHANGE_WEBHOOKS / PROBECTL_NOTIFY_CONNECTORS / PROBECTL_NOTIFY_INBOUND at startup (fail closed); the device agent resolves every PROBECTL_DEVICE_CRED_<NAME>_* value per poll cycle. Resolved values are cached only encrypted, for a short lease (5 m). See docs/secrets.md.

Backend access settings (environment only; all over verified TLS):

Backend health (counters + redacted last error, never secret material) is served at GET /v1/secrets/health and on the Admin page.

Local dev stack (deploy/compose/dev.yml)

Started with make compose-up. Local, non-production defaults — plaintext listeners and dev credentials for convenience. Production deploys are TLS/HTTPS-by-default — TLS on every listener.

Kafka listeners: host clients use localhost:9092; in-network containers use kafka:19092; the KRaft controller uses 9093 (internal). Prometheus runs with --web.enable-remote-write-receiver so the result pipeline can remote-write into it.

These names and ports are a contract — the integration test harness depends on them, so don't rename them casually.

Tear-down

make compose-down removes the containers and volumes (pgdata, chdata, promdata).