Zero-downtime lifecycle

The goal of this page: upgrade probectl at scale without an outage, and be able to roll back at every step. Four things make that possible, and the sections below are each one of them — the control plane is stateless replicas behind a load balancer (the traffic distributor in front of them), migrations are additive (expand/contract), the agent fleet rolls out version by version, and nothing is irreversible until you choose to make it so.

If you remember one rule, remember this one: during any rolling upgrade — replacing replicas one at a time while the rest keep serving — the old release (N) and the new release (N+1) are running side by side, so everything they share — the database schema, the agent protocol — must work for both at once. It is resurfacing a bridge one lane at a time: traffic in both the old and new lanes keeps flowing throughout, so every intermediate state of the bridge must carry both. Every constraint below falls out of that rule.

Rolling control-plane upgrade

The control plane holds no durable state of its own — all state lives in Postgres, ClickHouse, and the TSDB. That's what makes replicas interchangeable, and it's why they upgrade by replace, not restart-in-place:

%%{init: {'theme':'base','themeVariables':{'background':'#0d1117','primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#3b82f6','lineColor':'#8b949e','secondaryColor':'#21262d','tertiaryColor':'#0d1117','clusterBkg':'#161b22','clusterBorder':'#30363d','fontFamily':'ui-monospace, SFMono-Regular, Menlo, monospace'},'flowchart':{'curve':'basis','nodeSpacing':55,'rankSpacing':55,'padding':12}}}%%
flowchart LR
  LB[load balancer] --> A[replica N]
  LB --> B[replica N]
  subgraph upgrade
    direction LR
    B -. SIGTERM .-> D[draining: /readyz 503]
    D -. in-flight drains .-> X[exit]
    N1[replica N+1] -- /readyz 200 --> LB
  end

1. Apply migrations first (they are additive, so they're safe for the still-running release N — see the next section).
2. Roll replicas one at a time. On SIGTERM/SIGINT (the polite please-shut-down signals) a replica flips /readyz to 503 draining immediately — so the load balancer stops sending it new requests — then drains its in-flight requests (finishes what it already accepted, accepts nothing new) within PROBECTL_SHUTDOWN_TIMEOUT before exiting. (/healthz stays 200 the whole time: the process is still serving, it's just no longer accepting new traffic. The two probes mean different things on purpose — the lights are on, but the door sign says Closed.)
3. Bring up the replacement; it starts serving once /readyz is 200 — meaning the database is reachable and it's not draining.

Because N and N+1 run side by side during the roll, both must work against the same schema, which the migration policy below guarantees.

Migrations: expand/contract (the rollback contract)

This is the heart of zero-downtime. A migration is a versioned SQL script that moves the database schema from one shape to the next; probectl's are sequential numbered files in migrations/. A safe upgrade and a safe rollback both require one thing: release N's schema has to work with both N's code and N−1's code. You get that by never making a destructive change in a single step. Instead you split it across two releases:

• Expand (release N): add the new column/table/index (nullable, or with a default), backfill it, and start writing both the old and new shapes.
• Contract (release N+1, once nothing reads the old shape anymore): drop the old column/constraint.

Anything that would break N−1 mid-roll is forbidden in a single migration, and the migration gate rejects it automatically (make migration-gate, a standing CI job: the checker in internal/store/migrate walks every embedded *.sql in migrations/):

Allowed (and used throughout): CREATE TABLE/INDEX IF NOT EXISTS, ADD COLUMN IF NOT EXISTS … (nullable or defaulted), DROP POLICY/DROP INDEX, ADD CONSTRAINT … NOT VALID, RLS enable/force. Migrations remain idempotent (safe re-run) and each applies in its own transaction under an advisory lock — a named, application-level Postgres lock, used here to elect exactly one migration applier when several replicas start at once (one applier wins; the rest find the schema already current).

Rollback is then trivial: roll the replicas back to N. N's code already works against N+1's additive schema, because the schema only added — it didn't remove anything N depends on. (The contract migration that finally removes the old shape is deliberately deferred to a later release precisely so this stays true.)

Executed v0.5.0 rollback boundary

The supported upgrade boundary for this release is v0.5.0 → the current 0.6.0 release. make upgrade-rollback-drill-isolated is the executable proof, not a mock: it extracts and builds the exact v0.5.0 tag, starts a disposable real PostgreSQL service, and then runs this sequence:

1. apply the 45 historical migrations and start the v0.5.0 HTTPS server;
2. plant two tenants with overlapping targets plus a stable test/data digest, wrapped-key bytes, and audit-chain sentinels;
3. prove a failed pre-migration connection changes nothing;
4. apply the current additive migrations and start the current HTTPS server;
5. put the v0.5.0 binary back on the current schema and start it over HTTPS;
6. roll forward to current again; and
7. after every boundary, require the same digest, forced-RLS one-tenant view, wrapped-key bytes, and audit-head continuity.

The command refuses a dirty worktree for release evidence. Developers may set PROBECTL_UPGRADE_ALLOW_DIRTY=1 only for an explicitly non-release rehearsal. It is destructive only to its unique disposable Compose project. Backup/restore remains a separate mechanism and receipt (make backup-restore-drill-isolated), so a successful binary rollback is never misrepresented as disaster recovery.

This proves binary/schema compatibility on real PostgreSQL. It does not by itself prove Kubernetes rolling-scheduler behavior, regional WAN/DNS timing, or a non-author's runbook usability; those belong in the selected two-region live receipt described in docs/ops/two-region-receipt.md.

Agent version skew

Skew is the version distance between two sides that must interoperate — here, the control plane and an agent, which inevitably differ mid-rollout. The control plane is the authority on whether an agent is compatible. At registration it compares the agent's version to its own using an N/N−1 window (internal/lifecycle):

• Same major version, with a minor skew ≤ PROBECTL_AGENT_SKEW_WINDOW (default 1) → compatible, in both directions: an N agent talking to an N+1 control plane works, and so does an N+1 agent talking to an N control plane.
• A wider skew, a major-version mismatch, or an agent below PROBECTL_AGENT_MIN_VERSION → rejected with gRPC FailedPrecondition and an "upgrade required" message. (That status is deliberately distinct from a transient error, so the agent surfaces it to an operator instead of hot-looping forever trying to reconnect.)
• A dev/unpinned build (0.0.0-dev) on either side skips the check entirely.

The payoff: a fleet can run a mix of N and N+1 agents during a rollout, and upgrading the control plane never strands an agent that's one version behind.

Upgrading into central tests

Central test distribution (PROBECTL_TESTSYNC_SIGNING_KEY_FILE or PROBECTL_TESTSYNC_SIGNING_KEY, ADR) needs agents of this release or later: an older agent never calls FetchTestBundle and keeps running its local canaries, and the fleet view shows its tests as "none applied". Agents that were enrolled before the key existed receive it at their next identity rotation (written beside cert.pem as testsync.pub) and start pulling within 30 seconds; run probectl-agent rotate to do it now, or, for an agent that never rotates, copy the output of probectl-control testsync public-key into its identity directory.

Staged fleet rollout (cohorts + pace)

You don't push a new agent version to the whole fleet at once — you promote it ring by ring, watching health between rings. internal/lifecycle assigns each agent to a cohort by a stable hash of its id (stable so an agent never flaps between rings from one evaluation to the next — the way students assigned to homerooms by surname always land in the same room): a small canary ring first (named for the coal-mine canary: it meets the new version first, so a bad release hurts a few percent of the fleet, not all of it), then early, then the main fleet. A Rollout carries the target version and a Stage that the operator advances one ring at a time:

%%{init: {'theme':'base','themeVariables':{'background':'#0d1117','primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#3b82f6','lineColor':'#8b949e','secondaryColor':'#21262d','tertiaryColor':'#0d1117','clusterBkg':'#161b22','clusterBorder':'#30363d','fontFamily':'ui-monospace, SFMono-Regular, Menlo, monospace'},'flowchart':{'curve':'basis','nodeSpacing':55,'rankSpacing':55,'padding':12}}}%%
flowchart LR
  S0[Stage 0: none] --> S1[Stage 1: canary] --> S2[Stage 2: + early] --> S3[Stage 3: all]

DesiredVersion(agentID, current) returns the target version once that agent's cohort has been released, and the agent's current version otherwise. The cohort split is configurable (default 5% canary / 20% early, leaving the remaining ~75% as the main fleet). Note the boundary: this is the cohort + pace model plus the version-skew enforcement that makes a staged roll safe to attempt. Actually delivering a self-update to the agent (advertising the desired version down to it) is a separate follow-up.

Release upgrade notes

Most releases upgrade by the rules above and nothing else. A release that needs an operator step before its pods can start is listed here, newest first.

Historical audit records during upgrade

The verifier preserves hashes written by v0.6.5 rather than rewriting audit history. Records without durable canonical bytes can verify using the exact historical format, provided their headers contain neither newlines nor backslashes. Ambiguous historical headers fail closed and require operator investigation; the upgrade does not silently reinterpret them. Current records always use the timestamp-bound, escaped-header format, including in mixed old/new chains. Removing a current record's canonical-data marker does not make a changed timestamp valid.

Historical timestamps were not hash-bound by v0.6.5. Compatibility does not retroactively attest them. Payload, stream identity, sequence and chain links still verify against the original hashes, with the existing signed head/WORM anchors checked independently. Preserve the original database and signed artifacts when rehearsing an upgrade.

0.6.6: the ClickHouse reader credential (DPR-285)

Who this affects: any install where a ClickHouse lane has PROBECTL_*STORE_TENANT_SCOPING=true — the default under PROBECTL_DEPLOYMENT_PROFILE=multi-tenant or regulated, and therefore under the Helm values-multitenant.yaml and values-strict.yaml profiles. A single profile install without tenant scoping is unaffected.

What changed: before 0.6.6 every ClickHouse read authenticated as the control login, which no row policy filtered, so only the application's WHERE tenant_id = … kept tenants apart. From 0.6.6 every tenant-scoped read runs as the SELECT-only reader login the row policy binds, so the database itself refuses another tenant's rows. Think of it as the difference between a librarian promising to fetch only your books and a reading room whose door only opens onto your shelf.

What happens if you skip it: the 0.6.6 control plane fails config validation at start and never becomes ready; it does not fall back to reading as the control login. Under the Helm chart's default rolling update the 0.6.5 pods keep serving until the credential is supplied.

Before upgrading:

1. Create the reader in ClickHouse and grant it SELECT on the control login's default database only: CREATE USER probectl_reader IDENTIFIED WITH sha256_password BY '…', then GRANT SELECT ON <database>.* TO probectl_reader.
2. Stage its owner-only (0600) credential file, {"username":"probectl_reader","password":"…"} — in Helm, add it to the control.credentialFiles Secret.
3. Set PROBECTL_CLICKHOUSE_READER_BASIC_AUTH_FILE to that file's path. Its username must equal every scoped lane's PROBECTL_*STORE_READER_USER.

At boot the control plane refuses to start unless that login is not the control login, resolves the same default database, is denied INSERT and CREATE, holds no grant beyond SELECT on that database and on individual tables (no SELECT ON *.*, no role, no grant option; DPR-328), and sees no row outside the tenant it is scoped to. The Helm commands are in deploy/helm/README.md ("Upgrading a tenant-scoped install"); the key is documented in configuration.md. The durable Compose profile (deploy/compose/probectl.prod.yml, new in 0.6.6) provisions the reader from CLICKHOUSE_READER_PASSWORD in .env, so a fresh Compose install needs no extra step.

0.6.6: per-tenant ClickHouse databases are read as the reader too (DPR-323)

Who this affects: an install with the siloed_isolation license, tenant scoping on (as above) and at least one siloed or hybrid tenant, or that will provision one. A pooled-only install is unaffected.

What changed: a siloed or hybrid tenant's own ClickHouse database used to be read as the control login, so the database name the router chose was the only boundary. From 0.6.6 those reads run as the reader login with the tenant's setting, and every per-tenant table carries the same row policy as the pooled tables, so a read the router sends to the wrong tenant's database finds nothing. Back to the reading room: each shelf now checks your card too, not just the room's door.

What happens if you skip it:

• If the control login cannot grant the reader its per-tenant tables, startup refuses at the silo reader-scope catch-up (the error names SELECT ON *.* WITH GRANT OPTION), and new siloed/hybrid provisioning is refused the same way. It never falls back to reading as the control login.
• If a siloed or hybrid tenant lives on a residency plane off the pooled ClickHouse origin and PROBECTL_DATAPLANE_READER_BASIC_AUTH_FILES has no entry for that plane, startup refuses at the reader-scope catch-up, naming the entry, and provisioning onto that plane is refused the same way. A plane that hosts no siloed or hybrid tenant needs nothing new.

Under the Helm chart's default rolling update the 0.6.5 pods keep serving until the steps below are done.

Before upgrading:

1. On every ClickHouse server that hosts per-tenant databases (the default server and each residency plane), let the control login grant SELECT: GRANT SELECT ON *.* TO <control login> WITH GRANT OPTION. The durable Compose profile's clickhouse-init does this on the next up.
2. On each residency plane, create the same reader user as on the default server (same name; SQL_probectl_tenant defaulting to '', as in deploy/compose/prod/clickhouse/users.xml). It needs no grant of its own: the control plane grants it each per-tenant table. The reader must be a SQL-created user (CREATE USER), not one defined in users.xml, whose read-only storage refuses grants.
3. For each plane that hosts a siloed or hybrid tenant, stage that plane reader's owner-only credential file and list it in PROBECTL_DATAPLANE_READER_BASIC_AUTH_FILES (eu=/etc/probectl/credentials/ch-eu-reader.json). A plane on the pooled ClickHouse origin, or one with no such tenant, needs no entry.

At boot, before the silo router is published, the control plane installs the reader scope on every existing siloed and hybrid tenant's database and proves it per plane: the reader authenticates as the configured reader user, is not the control login, is denied INSERT and CREATE, and sees no row of a per-tenant table under another tenant's setting. A scope that fails the proof is withdrawn and startup is refused. The key is documented in configuration.md; the mechanism in security/tenant-isolation.md.

Unreleased: per-tenant quarantine instead of stop-everything (D-62)

Who this affects: an install with the siloed_isolation license and at least one siloed or hybrid tenant. Nothing changes until you opt in.

What changed: the startup preconditions of the notes above (residency plane configured, reader credential, reader-scope proof, Postgres silo catch-up) refuse startup for every tenant when one tenant fails them. That is still the default (PROBECTL_SILO_FAILURE_MODE=stop). Setting PROBECTL_SILO_FAILURE_MODE=quarantine makes a failure quarantine only the affected tenant: its reads and writes are refused with HTTP 503 tenant_quarantined, every other tenant starts and runs, the quarantine is audited, alerted (ProbectlTenantQuarantined) and listed in the provider console, and the tenant is released by itself once the check passes. Migration 0130_tenant_quarantine adds the provider-domain table behind that list (expand-only; release N-1 never reads it). The intended beta default is quarantine; the switch is the one DefaultSiloFailureMode constant in internal/config.

Before opting in:

1. Enable metrics.prometheusRule (or add the ProbectlTenantQuarantined rule to your own Prometheus): in quarantine mode a broken plane no longer stops the rollout, so the alert is how you learn of it.
2. Keep Kafka's offsets.retention.minutes (7 days by default) well above how long a quarantine may last: a quarantined tenant's lanes wait unconsumed, and their committed offsets expire after that window.
3. A configuration cause (a plane removed from PROBECTL_DATAPLANES, a missing PROBECTL_DATAPLANE_READER_BASIC_AUTH_FILES entry) clears when the fixed configuration is rolled out; a runtime cause (an unreachable plane, a missing GRANT, a schema catch-up precondition) clears without a restart within one PROBECTL_SILO_QUARANTINE_RECHECK_INTERVAL of the fix.

Configuration

See configuration.md for the PROBECTL_AGENT_SKEW_WINDOW and PROBECTL_AGENT_MIN_VERSION keys (agent-transport section) and PROBECTL_SHUTDOWN_TIMEOUT (the drain window).

Out of scope

This page is about single-region rolling upgrades: migration safety, version skew, and the staged-rollout model. Two related-but-separate concerns live elsewhere: multi-region active-active HA and disaster-recovery failover (RPO/RTO) are in multi-region.md, and per-tenant backup/restore plus verifiable deletion are a separate compliance capability.