Secrets integration

What this is. probectl never needs you to paste a database password, an SNMP credential, or an API token into a config file. Anywhere it accepts a secret, you can instead hand it a reference — a short string like vault:kv/netops/snmp#auth — and probectl resolves the real material from your enterprise secret store at the moment it is needed. A reference works like a library call number: the config file holds the catalog card, not the book — stealing the card tells a thief which shelf to look at, but the library still checks probectl's credentials at the desk before handing anything over. Supported backends: HashiCorp Vault, CyberArk CCP, AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager.

The same machinery closes the loop with trustctl (the sibling certificate/identity product): agents present trustctl-issued machine identities for mTLS — mutual TLS, where both ends of a connection prove their identity with certificates — and pick up in-place certificate renewals without restarting.

This serves three of the project's security non-negotiables: crypto only through internal/crypto, no hardcoded or logged secrets, and TLS on every channel.

Three guarantees

1. No plaintext at rest. References resolve in memory, at use time. The resolver's short-lived lease cache holds values only AES-256-GCM-sealed (via the internal/crypto provider) under an ephemeral per-process key — one minted at process start and never written anywhere — so even a memory dump of the cache yields ciphertext, and a restart re-resolves everything fresh. Runtime consumers that can accept bytes use the resolver's cleanup-returning byte path; the string path exists only for legacy config seams that require immutable strings.
2. Short-lived leases. A lease is permission to keep using a resolved value for a bounded time — a borrow with an expiry, not a copy to keep. A resolved value is served from cache for the lease TTL (default 5 minutes), then re-resolved. Rotate a secret upstream and the new value applies without restarting probectl. Device credentials re-resolve even more often — on every poll cycle or stream reconnect.
3. Fail closed. An unreachable backend or an unresolvable reference is an error — never an empty, partial, or stale credential silently substituted. A secret that has been rotated away stops being used at lease expiry.

Secret references

Anywhere probectl accepts a credential value, that value may be a reference:

Anything that does not match a scheme is treated as a literal and passes through unchanged — so existing plaintext configurations keep working while you migrate.

Backend access configuration (environment only)

How probectl reaches each backend is configured through the environment only — never probectl config files, so the access credentials themselves never sit in a file probectl reads. Every remote backend call rides TLS with certificate verification — never disabled. Vault and CyberArk base URLs must use https://; plaintext http:// is allowed only for loopback dev/test endpoints (127.0.0.1, ::1, or localhost). No cloud SDKs are linked in: it is stdlib HTTP plus SigV4 / OAuth2 / JWT signing through internal/crypto (SigV4 is AWS's request-signing scheme; OAuth2 client-credentials is the machine-to-machine login grant; the GCP path signs a JWT — a self-contained signed token — to obtain its access token).

A misconfigured backend (remote plaintext Vault/CyberArk URL, a CyberArk client cert that will not load, an unreadable GCP key file) fails startup — fail closed, not a silent skip. A backend you simply did not configure leaves its scheme unavailable, which is fine.

What resolves where

Control plane — resolved at startup, before anything consumes the config, and any failure aborts startup: PROBECTL_OIDC_CLIENT_SECRET, PROBECTL_CMDB_SECRET, PROBECTL_AI_MODEL_TOKEN, PROBECTL_SIEM_TOKEN, and the secret parts of PROBECTL_CHANGE_WEBHOOKS, PROBECTL_NOTIFY_CONNECTORS, and PROBECTL_NOTIFY_INBOUND. (OTLP ingest tokens are probectl-issued inbound tokens, not external credentials, so they are configured directly.)

Device agent — resolved per poll cycle / per gNMI reconnect: every PROBECTL_DEVICE_CRED_<NAME>_* field value. For example:

export PROBECTL_SECRETS_VAULT_ADDR=https://vault.acme.example:8200
export PROBECTL_SECRETS_VAULT_ROLE_ID=...  PROBECTL_SECRETS_VAULT_SECRET_ID=...
# SNMPv3 credential "core-sw" — references, not material:
export PROBECTL_DEVICE_CRED_CORE_SW_USERNAME=monitor
export PROBECTL_DEVICE_CRED_CORE_SW_AUTH_PROTO=sha256
export PROBECTL_DEVICE_CRED_CORE_SW_AUTH_PASS='vault:kv/netops/snmp#auth'
export PROBECTL_DEVICE_CRED_CORE_SW_PRIV_PROTO=aes
export PROBECTL_DEVICE_CRED_CORE_SW_PRIV_PASS='vault:kv/netops/snmp#priv'

A failed re-resolution skips the poll cycle (incrementing a cred_errors counter and logging a warning) rather than polling a device with stale material.

Observability

GET /v1/secrets/health returns per-backend counters, live lease counts, last success time, and the last error — redacted: never any secret material, and reference fragments are masked (vault:kv/x#…). The Admin UI renders this as the Secret backends card; a resolver_running=false flag distinguishes an unwired resolver from one that is simply idle.

trustctl machine identities (agent mTLS)

The agent's client certificate is loaded through crypto.RotatingIdentity. On each handshake it checks the cert file's mtime and size (at most every 10 seconds), so a trustctl renewal written in place is presented on the next connection — including gRPC reconnects — with no agent restart. The identity is always pinned to a SPIFFE URI prefix (SPIFFE is a standard for naming workloads inside certificates as spiffe:// URIs): the deployment's trust domain by default, a narrower operator-configured prefix if set — there is no unpinned mode (crypto.NewRotatingIdentity). A renewal carrying the wrong identity is refused (the last attested key pair keeps serving; a half-written renewal caught mid-write is also skipped). On the server side, ServerMTLSConfigRotating gives the agent-transport listener the same hot-rotation behavior.

%%{init: {'theme':'base','themeVariables':{'background':'#0d1117','primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#3b82f6','lineColor':'#8b949e','secondaryColor':'#21262d','tertiaryColor':'#0d1117','clusterBkg':'#161b22','clusterBorder':'#30363d','fontFamily':'ui-monospace, SFMono-Regular, Menlo, monospace'},'flowchart':{'curve':'basis','nodeSpacing':55,'rankSpacing':55,'padding':12}}}%%
flowchart LR
  subgraph backends [Secret backends]
    V[Vault KV2] & CA[CyberArk CCP] & K[AWS / Azure / GCP]
  end
  R[secrets.Resolver<br/>sealed lease cache, fail closed]
  V & CA & K --> R
  R -->|startup, fail closed| CP[control plane config]
  R -->|per poll cycle| DA[device agent credentials]
  T[trustctl] -->|renews cert/key in place| RI[crypto.RotatingIdentity]
  RI -->|per handshake| MTLS[agent mTLS]
  R -.->|"/v1/secrets/health (redacted)"| UI[Admin · Secret backends]

Operational notes

• Lease TTL is secrets.DefaultLease (5 minutes). Health counters are process-local.
• Errors and health snapshots never contain secret material; backend HTTP response bodies are never echoed into errors (status codes only).
• The cache key is per-process and ephemeral: a restart re-resolves everything.
• Per-tenant key management / BYOK builds on this same resolver, past its lease cache: a tenant's BYOK key resolves on every use, so the customer's revocation applies at once rather than at lease expiry — see byok.md.