Hardening and FIPS 140-3 guide

Hardening is the work of removing slack from a deployment — fewer defaults trusted, fewer ports open, every security claim checked rather than assumed. This guide covers running probectl in a hardened, regulated, or air-gapped posture: the FIPS 140-3 build, a STIG/CIS-style hardening checklist (STIG and CIS are the two widely-used catalogs of security configuration requirements — the checklists auditors walk), and a secure-defaults review. It is written for operators of sovereign single-tenant and MSP/provider deployments alike.

probectl is sovereign by design — it never phones home, all crypto routes through one validated-swappable module, and every listener is TLS with authenticated, untrusted-by-default ingestion (the project's security non-negotiables). The defaults are already hardened; this guide makes the posture explicit and auditable.

0. Prometheus-mode deployment restriction

In tsdb=prometheus mode the upstream Prometheus/VictoriaMetrics (the TSDB — the time-series database holding metrics) has no server-side tenancy of its own — probectl's query proxy is the boundary. Two layers enforce that in code: every parsed selector is tenant-forced (promapi.ForceTenant strips any caller-supplied tenant_id matcher and pins the authenticated tenant), and the upstream forwarder itself refuses any selector not pinned to exactly one tenant (ErrUnscopedUpstreamQuery in internal/promapi/upstream.go).

Hard deployment restriction: the upstream TSDB must be reachable ONLY by the probectl control plane (network policy / private listener / mTLS). Any user, dashboard, or service with direct network access to the upstream can read ALL tenants' series. Grafana and federation must go through probectl's /prom endpoints, never the upstream directly.

0b. Audit WORM export

What: an off-database, tamper-evident copy of the provider audit chain that survives a database owner deleting rows.

Why it is needed. The audit chains are already tamper-evident inside Postgres — each record hash-chains to the previous one (its hash covers the previous record's hash, so editing, deleting, or reordering any record breaks every later link; internal/audit/audit.go), and the app role has no UPDATE/DELETE on them. Tamper-evident means tampering cannot be hidden — it does not mean the rows cannot be destroyed: a database owner can still truncate a table. WORM ("Write Once, Read Many") export defends against that: the record exists somewhere the database owner cannot reach.

These segments are not the tenant artifact store and do not share its volume: an operator can keep exports and evidence on S3/MinIO and still write signed WORM segments to an object-lock claim (DPR-116).

How. Turn it on in the chart with audit.worm.enabled=true and point audit.worm.existingClaim at a mount backed by an object-lock bucket (S3 Object Lock or MinIO in compliance mode — the actual immutability guarantee lives in the bucket, not in probectl). The provider audit chain then exports hourly as Ed25519-signed segments (Ed25519 is a compact, fast signature algorithm; the files are worm/audit/provider/segment-*.json plus a .sig and the public key). Each cycle catches up as many as eight 1,000-event pages, uses a read-only probe to signal any remaining lag immediately, and re-verifies signatures, sequence continuity, and the cross-segment hash chain (internal/audit/worm.go). A lagged cycle never advances the successful-cycle timestamp. A purge or gap logs an unmissable error. Because the public key is published next to the segments, any third party can verify the export with nothing but that key — no access to probectl required.

Verification treats those durable objects as untrusted input: the public-key object is capped at 4 KiB, the detached Ed25519 signature at its exact 64-byte size, and each segment at 4 MiB / 1,000 events. The filesystem and memory object adapters enforce the byte ceiling while reading or copying, so a one-byte-oversize object fails closed before its full body is buffered. Export applies the same segment ceiling, preventing probectl from writing an object it cannot later verify.

The segment is a privacy projection, not a raw table dump. It preserves seq, action, prev_hash, and hash so sequence and chain continuity still verify, but minimizes raw actor, target, and data before the object is signed. ELI5: WORM keeps the numbered, signed receipt that proves a log entry existed, while leaving the person's name out of the write-once copy. Raw-row recomputation remains an in-database/legal-hold operation; object-locked exports are safe to retain for the long evidence window.

The signing key is persisted, not ephemeral. For a single control-plane replica, set PROBECTL_WORM_SIGNING_KEY_FILE to a PEM path on the same persistent claim (generated and persisted 0600 on first boot, reused thereafter), or inject PROBECTL_WORM_SIGNING_KEY (base64 PEM) from your secret manager. Back this key up like the envelope key — it is the identity the whole exported history is signed under; lose it and you forfeit cross-restart verification of every segment signed before the loss. Enabling WORM export with no key configured fails closed: the control plane refuses to start rather than mint a fresh key each boot (which would silently invalidate every prior segment's signature).

For Helm HA, the contract is stricter. Pre-create one encrypted ReadWriteMany PVC backed by object-lock/compliance-mode storage and set it as objectStore.existingClaim; set PROBECTL_AUDIT_WORM_DIR to an absolute, canonical directory at or below objectStore.mountPath. Pre-create secrets.existingSecret with one shared PROBECTL_WORM_SIGNING_KEY for every replica. The chart rejects WORM on emptyDir, a path outside the mounted claim, any ./.. traversal spelling, or PROBECTL_WORM_SIGNING_KEY_FILE when replicaCount > 1 or the HPA can scale above one. ELI5: the pods may take turns exporting, but they must all write the same locked filing cabinet and sign with the same pen.

0c. At-rest encryption — who encrypts what

probectl is self-hosted, so some at-rest encryption is the product's job and some is necessarily the operator's. This section is the contract that draws the line; probectl-control preflight is the check that keeps it honest.

What probectl encrypts (on by default). Sealed tenant values (alert-channel secrets, integration credentials, ...) are envelope-encrypted through internal/tenantcrypto before they ever reach Postgres. Envelope encryption means the value is encrypted under a data key, and data keys are themselves encrypted ("wrapped") under one master key-encryption key (KEK) — like a hotel key cabinet: every room has its own key, and the one master key opens only the cabinet the room keys hang in, so stealing a single room key opens one room, never the building. The shipped recipes turn this on:

• compose sets PROBECTL_ENVELOPE_KEY_FILE=/var/lib/probectl/envelope.key on the controldata volume — on first boot the control plane generates a master key there (0600) and logs it loudly. Back that volume up like key material: lose the key and sealed values become unreadable.
• Helm refuses to template without secrets.envelopeKey, secrets.sessionHMACKey, or existingSecret.
• Both set PROBECTL_REQUIRE_AT_REST_ENCRYPTION=true, and the direct serve path also refuses keyless startup by default, so a keyless misconfiguration is a fatal startup error — never silent plaintext.
• Production should supply its own key: PROBECTL_ENVELOPE_KEY (which always wins over the file), injected from a KMS / secret manager; or per-tenant BYOK on the licensed tier (byok.md).
• Throwaway local-only runs may set PROBECTL_ALLOW_KEYLESS_DEV=true; that is the only supported plaintext passthrough mode and must not be used for production, provider, or regulated profiles.

Rotating the deployment envelope key. Rotation has two phases: overlap, then proof of retirement. Bring up the control plane with the new PROBECTL_ENVELOPE_KEY / PROBECTL_ENVELOPE_KEY_ID and put the old key in PROBECTL_ENVELOPE_OPENER_KEYS=old-id=<old base64 KEK>. New writes now use the new key while old rows still open. Then run:

probectl-control envelope-rewrap --dry-run --from-key-id=old-id
probectl-control envelope-rewrap --from-key-id=old-id

The execute step walks the registered deployment-envelope stores in tenant-scoped batches (alert_rules.channels, the deployment-global agent_ca.key_sealed, and — on BYOK-licensed deployments — the managed per-tenant KEKs in tenant_keys.wrapped_kek), re-seals values sealed under the retired key under the active key, and appends a no-secret provider-audit receipt (envelope.rewrap). The managed per-tenant KEKs are sealed under the deployment master, so the master is built with the same PROBECTL_ENVELOPE_OPENER_KEYS ring: during the overlap phase they keep opening under the retired key, and this step re-seals them under the active key so the old key can be fully retired (CRY-02). After removing old-id from PROBECTL_ENVELOPE_OPENER_KEYS and restarting, prove the old key is retired:

probectl-control envelope-rewrap --verify-retired-key-id=old-id

For encrypted Postgres backup containers (.pbk) that must outlive the old key, stream them through the same rotation before removing/destroying the old KEK:

probectl-control backup-rewrap < old.dump.pbk > new.dump.pbk

backup-rewrap opens the old container through the configured opener ring and seals the output under the active key; plaintext exists only in the process pipe, never as an intermediate file. Backups that are past retention may expire instead of being rewrapped.

What the operator encrypts (a documented duty, not an assumption). probectl does not re-encrypt the bulk telemetry stores' data files — at that scale it is the storage layer's job. You MUST provide at-rest encryption for the volumes backing:

The preflight check.

probectl-control preflight [--strict] [--paths /var/lib/postgresql,/var/lib/clickhouse,/var/lib/probectl]

Per data path it reports whether the backing mount is detectably encrypted: /dev/mapper/* (dm-crypt/LUKS; plain LVM also matches — confirm) and ZFS/eCryptFS pass; a plain block device warns, and --strict exits non-zero so regulated profiles and CI can gate on it. Cloud-volume encryption is invisible from inside a container — if your volumes are encrypted below the host, set PROBECTL_STORAGE_ENCRYPTION_ATTESTED=true: the finding downgrades to informational and the attestation goes on the record. The check also reports probectl's own envelope-key posture.

1. FIPS 140-3 mode

What the FIPS build is

FIPS 140-3 is the U.S./Canada government standard for validating cryptographic modules; CMVP — the Cryptographic Module Validation Program — is the body that tests them and issues the certificate numbers an auditor accepts as proof. probectl routes every cryptographic primitive through one package, internal/crypto, and a CI guard (scripts/check_crypto_imports.sh) blocks any handler or service from calling a crypto primitive directly. That single choke point is what makes a FIPS build possible: a FIPS 140-3 validated module can be compiled in transparently, swapping the underlying implementations while the Provider API and all of its outputs stay byte-for-byte identical — the same dashboard and controls, a certified engine under the hood. A test asserts that the standardized outputs are the same with or without FIPS compiled in, so "swap the module" is provably not "change the behavior."

The FIPS artifact embeds the Go Cryptographic Module v1.0.0 — validated under FIPS 140-3 as CMVP certificate #5247 (CAVP algorithm certificate A6650 — CAVP is CMVP's sibling program that validates the individual algorithms; included in Go 1.24+) — selected at build time with GOFIPS140 and marked with the probectl_fips build tag. The dated evidence snapshot for that claim lives in compliance/fips-evidence.md.

Exactly what is and is not certified — read this before quoting FIPS to an auditor. The module holds the CMVP certificate; probectl as a product holds no CMVP certificate of its own. The accurate claim is: "probectl's FIPS artifact builds against and operates the FIPS 140-3-validated Go Cryptographic Module v1.0.0 (CMVP #5247), with a power-on self-test asserting the validated module is live." The authoritative sources are the Go FIPS 140-3 documentation and the NIST CMVP listing for certificate #5247 — verify the certificate number there yourself rather than taking this doc's word for it. Certification path: if a procurement requires a product-level validation (probectl itself listed with CMVP), that is a separate vendor engagement with an accredited lab — planned only on concrete regulated-buyer demand. Until then, no probectl-level certificate is claimed anywhere.

make build-fips                 # GOFIPS140=v1.0.0 -tags probectl_fips -> bin/<FIPS_BINARIES>-fips
make fips-gate                  # build + power-on self-test with the module active

The FIPS build is gated by the artifact, not by a runtime license check — there is no lic.Has(fips) gate anywhere in the code. The fips entry in the tier table documents the entitlement (the validated distribution is an Enterprise deliverable), but the running binary enforces nothing license-side for FIPS. The build you run is the gate.

Power-on self-test (POST)

Every command in Makefile FIPS_BINARIES runs crypto.RunPowerOnSelfTest() before it can touch cryptographic material: probectl-control, probectl-agent, probectl-ebpf-agent, probectl-endpoint, probectl-flow-agent, probectl-device-agent, probectl, probectl-license, and probectl-bmp-listener. That means the POST runs before listeners, bus clients, first-boot enrollment, secret resolution, backup sealing/opening, certificate generation, license signing, and BMP mTLS setup. It fails closed if it errors (an errored self-test means the process refuses to start — never "warn and serve"). The POST:

• Known-answer tests (a KAT feeds an algorithm a fixed input and demands the standard's exact published output — catching a swapped or miscompiled primitive): SHA-256 (FIPS 180-4), HMAC-SHA-256 (RFC 4231), PBKDF2-HMAC-SHA-256 (SP 800-132).
• Operational tests: AES-256-GCM seal/open with authenticity (tampered AAD rejected); Ed25519 sign/verify through the full PEM round-trip (tampered message and foreign key rejected); DRBG draw (DRBG — deterministic random bit generator, the approved construction behind crypto/rand).
• In a probectl_fips build: asserts the validated module is actually active (crypto/fips140.Enabled()), catching an artifact tagged FIPS but built without GOFIPS140.

The Go module additionally runs its own CAST / integrity self-tests at init; the POST proves the probectl integration end-to-end.

Activating the module

/v1/editions reports the live posture under fips: build_tag, module_active, enforced, module_version, self_test_passed. The Admin → Editions card shows a FIPS badge when the build or module is present. This is a status indicator only — FIPS is a hardening mode, not a feature surface.

FIPS coverage / boundary

The validated cryptographic boundary is the Go Cryptographic Module. probectl uses only algorithms inside that boundary for security functions. (In the table, AEAD means authenticated encryption with associated data — ciphertext that detects tampering rather than merely hiding content; a KDF is a key-derivation function, the deliberately-slow transform between a password and a key.)

Documented non-approved uses (outside the security boundary, FIPS-defensible):

• TOTP uses HMAC-SHA-1 (RFC 6238 interop — authenticator apps fix the algorithm). HMAC-SHA-1 is permitted in FIPS in HMAC mode; this is not a bare SHA-1 digest.
• Certificate fingerprints (CertSHA1) use SHA-1 only as a non-secret content identifier (the abuse.ch SSLBL / CT-log scheme), never as a security primitive or signature.
• TLS negotiation offers both approved (AES-GCM, P-256) and non-approved (ChaCha20-Poly1305, X25519) options for broad interoperability. In FIPS mode the module negotiates only the approved subset — the approved options are always present in the hardened config, so handshakes succeed without ChaCha or X25519.

For fips140=only (enforced) deployments, confirm clients support an AES-GCM suite and P-256, and that any TOTP/SHA-1 fingerprint paths are acceptable in your accreditation scope (both are HMAC- or identifier-only uses).

2. STIG / CIS hardening checklist

A condensed, auditable checklist mapped to the project's security non-negotiables. probectl ships these as defaults except where noted "operator action".

Transport & network

• Every listener serves TLS 1.3 (the enforced minimum floor on probectl's own listeners); AEAD-only suites.
• Outbound probe/integration clients (canary HTTP/DNS, gNMI devices) keep a TLS 1.2+ floor for the third-party endpoints they monitor, where a 1.3-only floor would break legitimate targets; certificate validation is always on.
• Agent ↔ control-plane is mTLS with SPIFFE-style tenant-bound identity; no plaintext agent transport.
• REST API, web UI, OTLP, MCP are HTTPS; shipped Compose + Helm serve TLS on the application listener and public edge, with HSTS.
• UI sets a CSP (Content-Security-Policy — the page's allowlist of what may load or execute) and Secure + HttpOnly + SameSite session cookies.
• Inbound webhooks verify the sender's HMAC signature (a keyed hash proving the sender held the shared secret); all ingestion is authenticated, tenant-scoped, and treated as untrusted input.
• Outbound fetches validate certificates (never disabled); fetched content is untrusted.
• Operator action: terminate TLS at a hardened ingress; restrict the management/provider plane to an admin network (NetworkPolicy / firewall).

Identity, access & tenancy

• Tenant isolation enforced at the storage + query layer (RLS / partitions / physical silo), not application code alone; AI/MCP enforce tenant then RBAC.
• Provider/MSP operators get no implicit read of tenant telemetry; access is time-bounded, consented, separately-audited break-glass.
• Passwords: PBKDF2-HMAC-SHA-256, 600k iterations. TOTP MFA available (TOTP — the six-digit time-based authenticator-app code).
• Dev auth is physically absent from release builds: a release binary refuses PROBECTL_AUTH_MODE=dev at boot with a fatal error — never a warning. Even the local-evaluation build (make build-devauth, -tags devauth) additionally requires PROBECTL_DEV_AUTH_ACK=i-understand AND a loopback-only bind. The no-devauth-in-release CI job proves both the symbol absence and the boot refusal on every pass.
• Operator action: wire per-tenant SSO/SCIM (SCIM — automated user provisioning and deprovisioning from your identity provider); require MFA for admin and all provider operators; set least-privilege RBAC roles.

Crypto & secrets

• All primitives via internal/crypto; FIPS-swappable (this guide).
• Sensitive config uses envelope encryption at rest; the control plane stores no plaintext private keys for managed-host flows.
• Secrets resolve from references (Vault / CyberArk / cloud KMS) — never logged, never in URLs or git.
• At-rest sealing on by default in the shipped recipes (generated key file + PROBECTL_REQUIRE_AT_REST_ENCRYPTION=true, §0c); keyless = fatal.
• Operator action: supply PROBECTL_ENVELOPE_KEY from a secret manager in production; enable per-tenant BYOK (Bring Your Own Key — byok.md) for regulated tenants; encrypt the bulk telemetry volumes (§0c — preflight --strict).

Audit and data lifecycle

• Config changes and data-access actions write to an immutable, tamper-evident audit chain; provider/break-glass actions go to a separate provider chain.
• Per-tenant export + verifiable deletion with a recomputable attestation; crypto-offboarding destroys per-tenant keys (byok.md).
• Operator action: ship audit streams to your SIEM; set the backup-TTL statement (PROBECTL_BACKUP_RETENTION_NOTE) and retention policy.

Sovereignty

• No phone-home — no outbound telemetry/analytics on by default.
• Threat detection is a signal, not an inline IPS; never auto-blocks.
• Open-data/threat-intel is read-only, cached, ingested once, degrades gracefully; a down feed never breaks core function.
• The web UI is usable without third-party calls (no CDN fonts/beacons).
• Remediation is observe-only / human-gated by default — never autonomous.
• Operator action: for air-gapped installs, use the air-gapped bundle; point AI at a local model (Ollama / vLLM); disable external feeds if policy requires.

Container / host (CIS Docker / Kubernetes)

• Operator action: run as non-root, read-only root filesystem, no-new-privileges, all Linux capabilities dropped (the eBPF agent needs only CAP_BPF/CAP_PERFMON where used).
• Operator action: apply NetworkPolicies (default-deny egress; allow only the datastores, bus, and explicitly-configured feeds).
• Operator action: enable TLS in transit to Postgres / ClickHouse / Kafka. The multi-tenant/regulated profiles fail closed unless Postgres DSNs use sslmode=require/verify-ca/verify-full, ClickHouse URLs are https://, and Kafka TLS is enabled.
• Shipped production Compose and Helm control-plane paths require immutable image digests; tag-only references fail before workload start/render.
• Operator action: verify the release signature or approved mirror provenance and scan the pinned digest with your supply-chain tooling.

OpenShift (restricted-v2 SCC)

The chart's hardened default pins explicit numeric runAsUser/runAsGroup/ fsGroup on every pod and container, because vanilla Kubernetes restricted PSS and most policy engines want an explicit non-root uid, not just runAsNonRoot: true. OpenShift's default restricted-v2 SecurityContextConstraint is the opposite: it assigns each namespace its own uid/gid range and rejects a pod that pins an id outside that range (must be in the ranges: [...]).

Install with the OpenShift profile, which flips a single knob (assignNumericUIDs: false) so the chart drops every numeric id — from the control-plane and BGP-analyzer pods and the image-specific browser-agent and backup/restore job pods — and lets the SCC assign them. runAsNonRoot: true, the RuntimeDefault seccomp profile, the read-only root filesystem and the dropped capabilities all stay, so the pods remain non-root and hardened:

helm install probectl deploy/helm/probectl \
  -f deploy/helm/probectl/values-multitenant.yaml \
  -f deploy/helm/probectl/values-openshift.yaml \
  --set <your required secrets/stores…>

CI renders both profiles and asserts the OpenShift render carries no explicit uid while staying non-root (scripts/check_helm_openshift_scc.sh, SUP-11), so the profile cannot silently regress.

3. Secure-defaults review

The shipped default vs the hardened-deployment recommendation, per component. "Shipped" is what probectl does out of the box; "Hardened" is the regulated posture. A green default means no action needed.

CI asserts the code-level defaults in this table (TLS minimum version, HSTS, secure-cookie attributes, no-phone-home, the FIPS self-test). The operator-action rows are deployment policy and are validated by the Helm hardening gate and your own controls.

3a. Day-2 ops and the strict NetworkPolicy profile

A NetworkPolicy is Kubernetes's pod-level firewall rule — it declares which peers may talk to a pod (ingress) and which the pod may reach (egress). The default Helm profile ships NetworkPolicy on, but with two deliberate holes: an empty ingressFrom (any pod may reach the API port) and an empty egressTo (allow-all egress). That is on purpose — a default install must not lock itself out of an unknown ingress controller. For regulated or air-gapped deployments, apply the strict profile, which closes both holes:

helm install probectl deploy/helm/probectl \
  -f deploy/helm/probectl/values-strict.yaml \
  --set-string image.digest='sha256:<release-digest>' \
  --set control.tls.existingSecret=probectl-metrics-tls \
  --set ingress.host=probectl.example.com \
  --set ingress.backendTLS.trustSecret=probectl-backend-ca \
  --set ingress.backendTLS.serverName=probectl.example.com \
  --set 'control.trustedProxies={10.244.0.0/16}' \
  --set secrets.existingSecret=probectl-runtime \
  --set audit.worm.enabled=true \
  --set-string audit.worm.existingClaim=probectl-audit-worm \
  --set-string control.extraEnv.PROBECTL_SIEM_ENABLED=true \
  --set-string control.extraEnv.PROBECTL_SIEM_ENDPOINT=https://siem.example/ingest \
  --set-string control.extraEnv.PROBECTL_BUS_MODE=kafka \
  --set-string control.extraEnv.PROBECTL_BUS_BROKERS=kafka.probectl.svc:9093 \
  --set-string control.extraEnv.PROBECTL_BUS_TLS_ENABLED=true \
  --set-string control.extraEnv.PROBECTL_TSDB_MODE=prometheus \
  --set-string control.extraEnv.PROBECTL_TSDB_URL=https://prometheus.probectl.svc:9090 \
  --set-string control.extraEnv.PROBECTL_PATHSTORE_MODE=clickhouse \
  --set-string control.extraEnv.PROBECTL_PATHSTORE_URL=https://clickhouse.probectl.svc:8443 \
  --set-string control.extraEnv.PROBECTL_FLOWSTORE_MODE=clickhouse \
  --set-string control.extraEnv.PROBECTL_FLOWSTORE_URL=https://clickhouse.probectl.svc:8443 \
  --set-string control.extraEnv.PROBECTL_OTELSTORE_MODE=clickhouse \
  --set-string control.extraEnv.PROBECTL_OTELSTORE_URL=https://clickhouse.probectl.svc:8443 \
  --set-string control.extraEnv.PROBECTL_EBPFSTORE_MODE=clickhouse \
  --set-string control.extraEnv.PROBECTL_EBPFSTORE_URL=https://clickhouse.probectl.svc:8443 \
  --set-string control.extraEnv.PROBECTL_ENDPOINTSTORE_MODE=clickhouse \
  --set-string control.extraEnv.PROBECTL_ENDPOINTSTORE_URL=https://clickhouse.probectl.svc:8443

Required operator-supplied values (RTO-14). The strict overlay sets PROBECTL_DEPLOYMENT_PROFILE=regulated, which makes the control plane validate a full production surface at startup: a verified-TLS Postgres DSN (sslmode=verify-ca/verify-full — plain require is single-profile dev only), durable bus/TSDB/telemetry stores (an in-memory bus or store is refused), and WORM + SIEM audit-retention watermarks. Those values are environment-specific, so the overlay cannot ship defaults — and the chart therefore refuses to render until all of them are supplied, printing the complete list in one message instead of letting a pod boot and then crash-loop. secrets.existingSecret must carry PROBECTL_DATABASE_URL (verify-ca/verify-full), PROBECTL_ENVELOPE_KEY, PROBECTL_SESSION_HMAC_KEY, and a base64 PROBECTL_WORM_SIGNING_KEY; the ClickHouse scoped-reader users (RED-001) are shipped by the overlay. The full annotated command lives at the top of values-strict.yaml.

control.trustedProxies is mandatory whenever the bundled ingress is enabled (the default): behind it every client reaches the control plane as the ingress controller's pod IP, so the authentication throttle must key on the forwarded client address instead. Set it to the controller's pod CIDR (and any L4 load balancer in front of it); the chart refuses to render without it so a deployment can never silently collapse every client onto one throttle bucket (AUTHZ-04). If you front probectl with your own ingress, disable the bundled one (--set ingress.enabled=false) and set control.trustedProxies to that proxy.

values-strict.yaml is full default-deny: a named ingress-controller selector (plus the monitoring namespace for /metrics scraping) and an explicit datastore / bus / IdP egress allow-list — no allow-all rule survives. Match the selectors and CIDRs to your cluster before applying. A wrong selector fails closed (the API becomes unreachable), which is the safe failure direction. The strict profile also turns on the ServiceMonitor (the Prometheus operator's scrape-config object), the PrometheusRule self-alert pack, and the backup CronJobs. All profiles now mount the operator-provided control.tls.existingSecret so the process serves HTTPS directly; the Service, probes, ingress backend, and ServiceMonitor target that same named https listener. The strict overlay supplies the regulated monitoring CA reference.

Other day-2 surfaces, all chart-managed:

• Probes: the control Deployment keeps /healthz and /readyz process probes under the chart's ingress and NetworkPolicy posture. The eBPF agent DaemonSet uses exec probes by default: the agent writes health state files and probectl-ebpf-agent healthcheck verifies freshness, so no plaintext pod listener is opened. Agent readiness reflects flow-source attachment, so a stuck bpf() call or a kernel lockdown surfaces as not ready rather than a silently dead pod.
• /metrics: the control plane serves Prometheus self-metrics (process and aggregate only — no tenant data) at /metrics, scraped by the ServiceMonitor (metrics.serviceMonitor.enabled) over the same HTTPS listener.
• Self-alerts: the PrometheusRule pack (metrics.prometheusRule.enabled) watches scrape presence, process pressure, write fencing, and replica lag with runbook annotations.
• Backups: Postgres and ClickHouse backup CronJobs are folded into the chart behind backup.enabled (off by default; supply the credentials secret).

4. References

• FIPS module behavior: https://go.dev/doc/security/fips140
• FIPS module evidence snapshot: compliance/fips-evidence.md
• Editions / licensing: editions.md
• Per-tenant keys / BYOK: byok.md
• Tenant isolation models: isolation.md
• Storage-layer isolation threat model: security/tenant-isolation.md
• Lifecycle / verifiable deletion: runbooks/tenant-offboarding.md
• Security non-negotiables: ../CONTRIBUTING.md