Supportability

What this is

When a probectl deployment misbehaves, you want to hand a diagnostician a single file that says what's wrong — without that file leaking any secrets. That's what this layer provides:

• a one-command support bundle (triage-grade diagnostics, packaged — a flight data recorder for the deployment: it captures the instruments, never the passengers' conversations),
• deep health checks (per-component status, plus one "is it healthy?" answer),
• actionable readiness findings (a stable local task for every unhealthy component, with bounded redacted evidence and a safe in-product next step),
• self-monitoring (probectl emits metrics about itself — the monitoring platform is also a service someone has to operate).

All three are core (free, every deployment gets them) — better bug reports help everyone. The paid part is the support organization and SLA (the Enterprise entitlement); that's a contract, not code.

The non-negotiable property: a support bundle never contains secrets, credentials, or PII (it falls under the project's secrets-handling non-negotiables). Everything below is built to keep that true even if someone slips up.

Support bundle

A .tar.gz of JSON files. The code lives in internal/support/bundle.go.

How it stays secret-free (defense in depth)

Three independent layers, so no single mistake leaks a secret:

1. Allowlist config, not blocklist. An allowlist names what may enter — a guest list; a blocklist names what may not — a bouncer's memory of past troublemakers, which fails precisely when a stranger walks up. Secrets added in next year's release are strangers. So config.Redacted() builds the config snapshot from a fixed list of known-non-secret keys. Database URLs have their passwords stripped; the envelope encryption key shows up only as the boolean envelope_key_configured (true/false), never the key itself. The safety is structural: a secret field someone adds later can't leak, because it simply isn't on the allowlist.
2. Tenant-scoped, anonymized operational identity. The topology-shape file is reduced at the database layer to the authenticated tenant before it is counted: one tenant, that tenant's agents, and that tenant's isolation model. Device collection and flow-ingest receipts preserve stable, allowlisted state/reason/counter evidence but replace agent, configured-target, and exporter identities with bundle-local ordinal references. None carries a tenant ID, hostname, IP, credential, raw datagram/varbind, decoded flow field, discovered neighbor, free-form dependency error, or another tenant's counts.
3. A final scrub. Before the bundle is written, it's swept once more for the specific sensitive values this deployment actually holds — the envelope key, the OIDC / CMDB / SIEM / AI-model secrets, the provider-bootstrap and OTLP tokens, and the database password. Any of those found anywhere in the bundle bytes is replaced with ***REDACTED***. A test asserts these values never appear in the output. So even an accidental inclusion is caught.

Each file is bounded (4 MiB max) and the whole bundle is gzip'd.

Getting a bundle

Deep health checks

GET /v1/diagnostics (admin diagnostics.read) returns each component's status — ok / degraded / down — plus an aggregate that equals the worst component, so one field tells you whether the deployment is healthy. Worst, not average, because a deployment is only as healthy as its sickest dependency — one amber bulb makes the whole panel amber. The checks are wired up in internal/control/diagnostics.go:

bus, tsdb, event_store and object_store (PLAT-09) make each configured external backend observable: an outage of the result bus, TSDB, ClickHouse event store or object store used to be invisible (diagnostics reported ok). A memory-backed bus/TSDB is a named degraded warning (dev/test only — data is lost on restart), and /readyz surfaces volatile planes while staying 200. Reachability is probed on this deep-health surface, not on the hot /readyz path, because these backends are shared infrastructure — draining a replica over a shared-backend blip removes serving capacity without routing around the fault.

Every degraded or down check carries one finding; an ok check never fabricates a task. A finding contains:

The native Admin → Support & diagnostics card shows the local process snapshot and build identity alongside findings and the underlying component table. The same contract is available to the CLI and generated Go/TypeScript SDKs. During a rolling upgrade an older replica may omit self-metrics, build identity, or finding details. Each missing section is displayed as unavailable/incomplete; the UI does not invent a value or pretend that state is healthy.

This is separate from the liveness/readiness probes (/healthz, /readyz) — liveness asks "is the process alive at all?" and readiness asks "should the load balancer send it traffic right now?". Those answer a blunt up/down for machines making routing decisions. The deep report is richer — it's for a human doing triage (deciding what is broken and what to look at first) and for the support bundle.

The endpoint is a tenant-authenticated, diagnostics.read-authorized sensitive read and is audited by the central route policy. Health findings are deployment-local and redacted; topology metadata is reduced to the caller's tenant at the storage layer before counting. No tenant identifier, hostname, IP address, credential, telemetry, or other-tenant count is returned. There is no remote advisor, cloud model, plugin catalog, phone-home, or outbound request in this path. Any future tenant-data section must preserve that storage-first scope before RBAC and add a cross-tenant isolation test.

Self-monitoring (probectl observes probectl)

The control plane collects and emits probectl_self_* metrics every 30 seconds — goroutines, mem_alloc_bytes, mem_sys_bytes, num_gc, uptime_seconds, max_procs — plus probectl_build_info{version,commit,go} (value 1, the standard Prometheus build-info trick: the labels carry the info, the value is just a constant). The authoritative, dependency-free native view is Admin → Support & diagnostics, backed by GET /v1/diagnostics; it shows goroutines, memory, garbage collections, uptime, process capacity, and build identity. It is intentionally administrator-only because these are deployment-global process facts. The response contains no tenant identity, labels, or telemetry. Agent scrape targets add tenant-agnostic probectl_agent_* RED/USE series: collection and publish rates, errors, bounded-buffer depth, and latest publish latency.

The product does not bundle or require an external dashboard runtime. The /metrics output and any example dashboard JSON are optional, operator-supplied protocol interoperability only; they are not the authoritative probectl surface, a runtime dependency, or part of the shipped application path.

Configuration

No new config keys. The diagnostics endpoints and the offline CLI read the existing config. The diagnostics.read permission that gates the endpoints is seeded for admins by migration 0034_diagnostics.sql.

Completeness gate

The bundle's section set is pinned by a CI test (internal/support/completeness_test.go, EXC-ORG-03): version, redacted config, health, self-metrics, topology summary, device and flow readiness receipts, runtime, and the manifest. Dropping a section — the bundle silently shrinking so it no longer carries what an F500 support contract needs to triage — reds the build, as does the manifest index drifting from the actual contents or a secret slipping through (the companion TestBundleHasNoSecrets).

Out of scope

The support organization and SLAs (the Enterprise/acquirer-provided contract — not code). In MSP mode, tier-1 support to end customers is the MSP's job. The DR drill on real multi-region infrastructure is likewise an operator action: the runbook + the CI failover drill exist (docs/ops/dr.md), but the live cross-region exercise needs real DR infra and is run by the operator, not in CI.