ADR: Agent enrollment & SVID issuance

Status: Accepted (2026-06-07). The token-mint surface is both the admin API and the operator CLI.

This is the decision record. An ADR — architecture decision record — is a dated note capturing a decision and its why, kept even after the code moves on: it records what was decided then, and gains status addenda when anything changes since. The operator how-to (mint a token, enroll an agent, rotate, revoke) lives in docs/agent/enrollment.md. If you want to run enrollment, read that; if you want to know why it works this way, read this.

The plain version

An agent only gets to talk to the control plane if it presents a valid mTLS client certificate — mutual TLS, where both ends of the connection prove themselves with certificates — whose SPIFFE identity (a standard URI scheme for naming workloads) names a tenant and an agent. The server reads the tenant/agent from that verified certificate — never from the request body — so a result can never lie about which tenant it belongs to.

There was a hole: the code verifies those certificates, but nothing in the repo issued them. The trust root was an operator running gen-cert by hand and copying files around. So the single strongest isolation mechanism in the product rested on an undocumented manual step. This ADR closes that hole: a repo-managed certificate authority (a CA — the signer whose signature makes a certificate trusted) issues short-lived agent identities (SVIDs, SPIFFE Verifiable Identity Documents), bootstrapped by a one-time join token.

What already existed (this ADR builds on, not around)

• mTLS transport with a SPIFFE URI identity, verified on every connection (internal/agenttransport, crypto.ServerMTLSConfigRevocable), with a handshake-checked revocation list.
• crypto.AgentSPIFFEID(tenant, agent) — the identity scheme spiffe://probectl/tenant/<tenant>/agent/<agent>, pinned to one trust domain.
• crypto.CA — an ECDSA P-256 CA with SPIFFE-SAN leaf issuance (dev/test only, until this ADR).
• crypto.ClientMTLSConfigRotating — the agent client already hot-reloads its certificate material, so rotation needs an issuer, not a transport change.
• Registration: the tenant/agent are read from the VERIFIED certificate, never the request.

Decision 1 — Bootstrap: single-use, tenant-scoped join tokens

An operator — through the agent.write-gated, audited admin API or the database-direct CLI — mints an enrollment token scoped to a tenant (and optionally pinned to a fixed agent id): 32 random bytes via internal/crypto, shown once, stored only as a hash — the same pattern as session tokens, so the database can confirm a presented token but can never reproduce one. The agent presents it exactly once; the row is consumed atomically (UPDATE ... WHERE used_at IS NULL, so a replay finds no row and is refused). That pre-tenant consume is exposed only through an exact-hash SECURITY DEFINER function. Direct application-role access with no tenant GUC sees and changes zero enrollment rows. Tokens expire (default 1h), and an operator can void an unused token before use with probectl-control revoke-enroll-token. Cancellation enters the separate provider role and invokes a provider-only database function; it does not reopen the token table to the application role.

Why not cloud-IID/OIDC attestation now: attestation here means proving facts about the machine an agent runs on — e.g. presenting a cloud-signed instance-identity document — instead of presenting a shared secret. probectl's primary deployments are sovereign/air-gapped, where there is no cloud identity document to attest against — a join token works everywhere. The enrollment endpoint is left as a seam: an attestor field on the request leaves room for aws-iid / gcp / oidc attestors later without changing the shape of the issued identity. (Today only join-token, or empty, is accepted; internal/enroll/enroll.go rejects anything else.)

Decision 2 — CA hierarchy: repo-managed root → intermediate → leaf

The shape is a passport system: master printing plates in a vault (the root), a passport office that prints daily (the intermediate), and passports that expire in a day (the leaves) — losing the office never forces re-cutting the plates.

• Agent root CA (10y, P-256): generated once by probectl-control agent-ca init; signs ONLY intermediates (MaxPathLen=1). The root key is printed at init for offline custody and is not stored — runtime operation never needs it.
• Issuing intermediate (1y default): held by the control plane and sealed at rest through internal/tenantcrypto (the deployment envelope / BYOK — the same posture as every other secret). Signs leaves only.
• Leaf SVIDs (default TTL — time-to-live — 24h): issued from a CSR — a certificate signing request, which carries only the public key, so the private key is generated on the agent and never leaves it. The leaf carries the SPIFFE URI SAN (subject alternative name — the certificate field holding the identity) binding tenant + agent, with client-auth EKU only (extended key usage — the field restricting what a certificate may be used for; client-auth-only means an SVID can never pose as a server).
• The CA bundle (root + intermediate) is what transports trust: agents get it at enrollment and on every rotation, so an intermediate roll-over is picked up automatically on the next rotation.
• trustctl (the sibling certificate-lifecycle product) can later replace the issuing intermediate; the enrollment/rotation API is the integration seam.

Decision 3 — Issuance flows

Enroll (pre-identity, HTTPS): POST /enroll/agent on the control API. This route is off /v1 on purpose — /v1 is the RBAC-gated session API, whereas this is a bootstrap surface like /auth. The agent has no certificate yet, so the channel is server-auth TLS only (HTTPS-by-default recipes make that safe); the request is authenticated by the join token. The server: consumes the token → derives tenant from the TOKEN (never the request) → assigns or verifies the agent id → signs a leaf → records the issued identity (serial, SPIFFE id, expiry) in the registry → returns the leaf plus the CA bundle. The registry now vouches for the tenant/agent pair, but the row remains registered with no last_seen_at: operational readiness becomes true only after the holder opens the authenticated mTLS transport and registers or heartbeats.

Rotate (identified, HTTPS): before expiry (at roughly 2/3 of TTL) the agent calls POST /enroll/agent/rotate on the same HTTPS bootstrap surface — not the mTLS data channel — carrying its CURRENT leaf in the request. Authentication is cryptographic rather than channel-level: the presented cert must chain to our hierarchy and be time-valid, its serial must be one we recorded at issuance, and the request must prove possession of the current key (an ECDSA signature over the new CSR — producible only by the key's holder). The server checks the revocation list, then issues a fresh leaf for the PROVEN identity — the SAN is set server-side and CSR-requested names are ignored, so identity can never change on rotation — and records it. ClientMTLSConfigRotating hot-swaps the files and connections re-handshake naturally.

Token revocation: a leaked-but-unredeemed join token can be voided early with probectl-control revoke-enroll-token -id <token-id> (the id is printed at mint time). The command assumes probectl_provider and can execute only the narrow unused-token cancellation function. Single-use semantics and the ~1h expiry remain the defense-in-depth behind it — even an unrevoked leak self-destructs on first use or at expiry. Issued certificates are a separate surface, revoked via revoke-agent.

Agent CLI: probectl-agent enroll --server https://control:8443 --token <jt> --dir /var/lib/probectl-agent/identity [--ca-pin <sha256>] writes key/cert/bundle (0600) and exits; the runtime config points at those paths and rotates them automatically. --ca-pin (printed at token mint when the CLI can read the serving certificate) authenticates the SERVER on first contact in self-signed / quickstart deployments — trust-on-first-use is refused when a pin is provided and mismatched.

Threat-model delta (what changes, honestly)

Each row is a concrete attack; before/after is the honest delta — several threats are bounded by this design, not eliminated.

Stated residuals

1. There is no workload attestation beyond possession of the join token at first boot — cloud/OIDC attestors are the documented extension seam (Decision 1).
2. The bus credential itself (e.g. Kafka ACLs) is unchanged — the consumer-side tenant verification plus cryptographic issuance is the compensating pair until per-tenant siloed lanes exist.

Out of scope (deliberate)

External CA integration (the trustctl seam is documented above), cloud attestors, CRL/OCSP distribution (the standard protocols for publishing revocations to third parties — here the in-process revocation list is the mechanism), and per-probe identities.