Backup & restore runbook

What this is

probectl keeps durable state in two databases plus an object store. A backup is a copy of that state placed where a failure can't reach it; a restore is putting the copy back and getting a working deployment out of it. This runbook is how you take the copy, how you put it back, how long each takes, and the automated drill — a scheduled rehearsal that performs a real restore and checks the result — that proves the restore path actually works. The drill matters because an untested backup is a fire escape you've never walked down: it looks fine on the wall map right up until the night you need it. Read this page before you need it — a restore is the kind of thing you do under stress.

What is stateful (and what is deliberately not backed up)

Stateful means the data lives nowhere else — lose the store and it's gone, which is what makes it worth copying. Anything rebuildable from another source is deliberately left out: backing it up would only slow down a restore you'll one day be running under pressure.

Backups contain tenant data, so they are encrypted at rest by default. The Postgres backup paths pipe the dump through probectl-control backup-seal, so plaintext never touches the backups volume — the artifact is a .dump.pbk envelope-encrypted container. The ClickHouse script, compose overlay, and chart CronJob first create the native server-side .zip in the encrypted ClickHouse backup staging path, then stream that zip through the same backup-seal filter, delete the raw staging zip, and keep only .zip.pbk plus its checksum for off-box custody. Filesystem object-store backup streams tar directly through backup-seal to .tar.pbk, so no plaintext archive lands on disk. All three local artifacts are sealed with the deployment's at-rest key (the same PROBECTL_ENVELOPE_KEY as live storage). Envelope encryption means the data is encrypted under a one-off data key, and that data key is itself wrapped by the deployment's master key (the KEK, key-encryption key) — so an attacker holding the artifact but not the KEK holds nothing readable, and a restore on a fresh machine needs exactly one secret.

Container integrity (format PBK2)

The .pbk container is tamper-evident end to end, not just encrypted. Each 1 MiB chunk is sealed with its position bound in, a DEK-keyed running MAC chains the header and every ordered chunk, and a final sealed trailer fixes the exact chunk count and that chain. On restore, backup-open verifies the trailer before it reports success, so a container that was truncated (at or between chunk boundaries), had its chunks reordered, had its header substituted, or had bytes appended fails closed with a clear error — an incomplete or altered backup can never open as if it were whole (PLAT-14, guardrail G7-6). This is in addition to, not a replacement for, the .sha256 sidecar operators verify before backup-open.

Format bump — PBK2 supersedes PBK1. The authenticated trailer is a deliberate on-disk format change. Legacy PBK1 containers predate it and cannot be made truncation-evident after the fact, so backup-open, backup-rewrap, and control-state restore on this build reject them with an actionable error rather than silently accepting an unverifiable backup. New backups are written as PBK2. If you still hold pre-release PBK1 artifacts, take a fresh PBK2 backup from the live data; where that is not possible, open the PBK1 container with a pre-PBK2 build (with the original KEK) and pipe it straight into backup-seal on this build, then retire the PBK1 copy under retention.

When rotating the deployment envelope key, long-lived .pbk files must either expire under retention or be rewrapped before the old opener key is removed:

PROBECTL_ENVELOPE_KEY=<new base64 KEK> \
PROBECTL_ENVELOPE_KEY_ID=new-id \
PROBECTL_ENVELOPE_OPENER_KEYS=old-id=<old base64 KEK> \
  probectl-control backup-rewrap < old.dump.pbk > new.dump.pbk

The old container is opened through the opener ring and the new container is sealed under the active key; no plaintext dump is written between those steps.

ClickHouse's BACKUP TO File still runs inside the ClickHouse server, so the server-visible staging path must be an encrypted operator-controlled volume (the encrypted-volume operator duty in hardening.md §0c, which probectl-control preflight --strict checks). Because the staging zip contains tenant telemetry before it is sealed, the script and Helm CronJob fail closed unless either a KEK is available for .zip.pbk output or you explicitly acknowledge a raw encrypted-target fallback: PROBECTL_CLICKHOUSE_BACKUP_ACK=encrypted-clickhouse-backup-target for the script/compose paths, or backup.clickhouse.encryptedTargetAck=encrypted-clickhouse-backup-target for Helm. The fallback is for operator-owned encrypted backup targets only. Keep all backup paths inside the operator's network — telemetry never leaves it (one of the project's non-negotiables).

Taking backups

One-shot (any time — e.g. right before an upgrade):

./scripts/backup_postgres.sh   /srv/probectl-backups   # → postgres-<db>-<ts>.dump.pbk + .sha256
PROBECTL_ENVELOPE_KEY=<base64 KEK> \
  ./scripts/backup_clickhouse.sh /srv/probectl-backups # → clickhouse-<db>-<ts>.zip.pbk + .sha256

PROBECTL_OBJECTSTORE_DIR=/var/lib/probectl/objects \
PROBECTL_ENVELOPE_KEY=<base64 KEK> \
  ./scripts/backup_objectstore.sh /srv/probectl-backups # → objectstore-<ts>.tar.pbk + .sha256

# Control-state: the evidence-signing key, the KEK id, and the agent-CA public
# bundle that live on the controldata volume beside the stores. One sealed
# artifact; run it against a live control plane (it reads the agent CA from the
# database). Keep it off-box with the others.
  probectl-control backup-control-state > /srv/probectl-backups/controlstate-<ts>.pcs
(cd /srv/probectl-backups && sha256sum controlstate-<ts>.pcs > controlstate-<ts>.pcs.sha256)

# Raw fallback only when both the ClickHouse staging path and off-box target are encrypted:
PROBECTL_CLICKHOUSE_BACKUP_ACK=encrypted-clickhouse-backup-target \
  ./scripts/backup_clickhouse.sh /srv/probectl-backups # → clickhouse-<db>-<ts>.zip + .sha256

backup-control-state reads the same PROBECTL_* config the control plane serves with (it is a one-shot of the same binary and image), so it finds the deployment KEK, the evidence-signing key file, and the database automatically. The .pcs artifact is envelope-encrypted under the deployment KEK — the evidence PRIVATE key is never written in the clear — so it is opened on restore with the same single secret the database and object backups need. A deployment that injects PROBECTL_EVIDENCE_SIGNING_KEY from a secret manager (rather than the file) re-injects it on the restored host and does not need this artifact; backup-control-state says so and exits.

The database scripts run the dump inside the running compose container (so you need no Postgres/ClickHouse client on the host), write a SHA-256 manifest next to the artifact (a checksum — a fingerprint that changes if even one byte of the artifact does, so corruption is caught before a restore starts), and copy the artifact off-box into the output directory you pass — that off-box file is exactly what the restore scripts consume. Postgres also needs probectl-control on the host so backup-seal can encrypt the stream: PROBECTL_CONTROL_BIN defaults to probectl-control, and the key comes from PROBECTL_ENVELOPE_KEY or PROBECTL_BACKUP_KEY_FILE/PROBECTL_ENVELOPE_KEY_FILE. Off-box is the point: a backup that lives on the same disk as the database shares the database's fate. For a non-dev deployment, override the env vars the scripts read: COMPOSE_FILE (default deploy/compose/dev.yml), PG_SERVICE/PGUSER/PGDATABASE, and CH_SERVICE/CH_USER/CH_PASSWORD/CH_DB.

The object-store script has two modes. filesystem is the default shown above: it refuses symlinks/special files and a destination nested under the source, then writes only the authenticated .tar.pbk. s3 copies PROBECTL_OBJECTSTORE_S3_URI to a timestamped destination prefix using the AWS CLI credential chain/workload identity. Custom MinIO endpoints must use verified HTTPS (provide an internal CA with AWS_CA_BUNDLE), SSE defaults to AES256, and the destination bucket must report default Object Lock retention mode COMPLIANCE. This is defense in depth: Ed25519 detects changed audit evidence; Object Lock prevents changing or deleting it during retention.

Plaintext Postgres dumps are break-glass only. The backup script and shipped cron examples refuse to write .dump unless PROBECTL_PLAINTEXT_BACKUP_ACK=allow-plaintext-tenant-backup is set exactly. The Helm chart uses the same exact acknowledgment: backup.plaintextAck=allow-plaintext-tenant-backup when backup.encryption.enabled=false. That path exists for disaster debugging, not normal operation; handle the file as raw tenant data and seal or destroy it immediately.

Scheduled backups: deploy/backup/ has a compose overlay for host cron and k8s CronJob examples (digest-pinned images, credentials sourced from a secret). Their Postgres jobs also produce .dump.pbk by default; the standalone Kubernetes manifest reads the envelope key from the probectl-envelope-key secret and expects a probectl-backup-tools PVC with the probectl-control sealing binary. A reasonable cadence: nightly, retain 7 daily + 4 weekly, and stagger all three so they don't contend (the shipped chart schedules Postgres at 02:00, ClickHouse at 02:30, and the object store at 02:45).

A failed backup fails loudly. Every chart CronJob and standalone manifest runs its script under bash -euo pipefail, so a producer that fails — a refused pg_dump, a ClickHouse BACKUP error, a tar error — fails the job, and backup-seal refuses to seal empty input. An empty artifact is never published as a successful backup; a failed job is the signal to act on.

When the chart creates the backups claim (backup.persistence.create=true), a one-shot <release>-backups-bind-<revision> Job mounts it in the same release so it binds immediately. Most storage classes (GKE, EKS, AKS, kind) bind a volume on its first consumer, and a CronJob is no consumer until it first fires, so helm install/upgrade --wait would otherwise wait on a Pending claim and fail the release. Set backup.persistence.bind=false on a class that binds immediately.

Telemetry regional DR profile: off-region ClickHouse backups

The shipped, default telemetry DR profile is off-region ClickHouse backups, not cross-region ClickHouse replication. The baseline profile is deliberately plain:

Treat the ClickHouse server's /backups path as a staging area, not the DR destination. A backup that remains only on the primary-region ClickHouse disk has the same regional fate as the database it protects. For a tighter RPO than 24 h, set backup.clickhouse.schedule to the required interval and record that numeric telemetry RPO separately from the Postgres metadata RPO in multi-region.md.

ClickHouse prerequisites. The server must (1) allow writing backups to its /backups path — deploy/compose/clickhouse-backups.xml (mounted by the dev stack) is the <backups><allowed_path> drop-in; mount the equivalent plus a /backups volume in production — and (2) be able to write that volume as the clickhouse user (uid 101). A fresh volume mounts root-owned, so the dev/compose scripts chmod 1777 /backups via a best-effort root exec; in Kubernetes set the ClickHouse server pod's securityContext.fsGroup: 101 (or pre-chown the PVC) instead.

At larger scale, move Postgres to pgBackRest / WAL archiving for point-in-time recovery — the WAL (write-ahead log) is Postgres's journal of every change, so where a nightly dump is a midnight photograph of the house, an archived WAL is the diary that lets you replay the day and stop at any minute you choose — and move ClickHouse to incremental BACKUP … SETTINGS base_backup = …. The scripts here are the supported baseline and the contract the drill verifies.

Restoring

Both restore scripts are destructive: they drop and recreate the database. Stop the control plane first — agents store-and-forward (buffer results on their own disk and replay them once the control plane returns), so no probe results are lost while it is down.

# 1. Stop probectl-control. Agents buffer; the UI is down from here.

# 2a. If the Postgres backup is ENCRYPTED (.dump.pbk), verify the sealed
#     artifact first, then decrypt it. This needs the ORIGINAL envelope key —
#     a fresh node only needs that one key:
(cd /srv/probectl-backups && sha256sum -c postgres-probectl-<ts>.dump.pbk.sha256)
PROBECTL_ENVELOPE_KEY=<base64 KEK> \
  probectl-control backup-open < postgres-probectl-<ts>.dump.pbk > postgres-probectl-<ts>.dump
(cd /srv/probectl-backups && sha256sum postgres-probectl-<ts>.dump > postgres-probectl-<ts>.dump.sha256)

# 2b. Verify + restore Postgres (drops + recreates, pg_restore from stdin):
./scripts/restore_postgres.sh   /srv/probectl-backups/postgres-probectl-<ts>.dump

# 3. Verify + restore ClickHouse (opens .zip.pbk, copies the zip back into the server, drops, RESTORE):
PROBECTL_ENVELOPE_KEY=<base64 KEK> \
  ./scripts/restore_clickhouse.sh /srv/probectl-backups/clickhouse-probectl-<ts>.zip.pbk

# 4. Restore the object tree. This verifies the checksum, opens directly into
#    a staging directory, preserves the old target, then atomically publishes:
PROBECTL_OBJECTSTORE_RESTORE_ACK=replace-objectstore \
PROBECTL_ENVELOPE_KEY=<base64 KEK> \
  ./scripts/restore_objectstore.sh \
    /srv/probectl-backups/objectstore-<ts>.tar.pbk \
    /var/lib/probectl/objects

# 5. Restore the control-state onto the FRESH controldata volume BEFORE boot.
#    This verifies the checksum, opens the sealed artifact with the KEK, and
#    writes the evidence-signing key, the KEK-id sidecar, and the agent-CA
#    trust bundle into place. The paths default to the deployment's own
#    PROBECTL_EVIDENCE_SIGNING_KEY_FILE / PROBECTL_AGENT_TLS_CA_FILE /
#    PROBECTL_ENVELOPE_KEY_FILE env, so inside the shipped compose image you
#    pass nothing but the KEK. It writes NO database — only files — so it runs
#    before the control plane does, and it refuses to overwrite a live key.
(cd /srv/probectl-backups && sha256sum -c controlstate-<ts>.pcs.sha256)
PROBECTL_ENVELOPE_KEY=<base64 KEK> \
PROBECTL_EVIDENCE_SIGNING_KEY_FILE=/var/lib/probectl/evidence-signing-ed25519.pem \
PROBECTL_AGENT_TLS_CA_FILE=/var/lib/probectl/agent-ca.crt \
PROBECTL_ENVELOPE_KEY_FILE=/var/lib/probectl/envelope.key \
  probectl-control restore-control-state < /srv/probectl-backups/controlstate-<ts>.pcs

# 6. Start probectl-control. On boot it re-runs the Postgres migrations
#    idempotently; the restored probectl_ch_migrations ledger keeps the
#    ClickHouse schema state consistent with the restored data. It RELOADS the
#    control-state restored in step 5 — the same evidence-signing key (unchanged
#    public fingerprint), the same KEK id (restored dv1 values keep opening),
#    and the agent-CA bundle — instead of generating fresh. No manual step.

# 7. Sanity-check: /readyz is green; a tenant-scoped query returns pre-incident
#    data; the audit chain verifies (the WORM verify job also re-checks the
#    exported provider chain against object storage); an incident evidence
#    export verifies against the SAME public fingerprint as before the restore.

The KEK-id sidecar (<PROBECTL_ENVELOPE_KEY_FILE>.id) is why a restore cannot silently renumber the key id: the id that sealed the at-rest data is pinned beside the key and is authoritative on every boot, so even a restore that omits PROBECTL_ENVELOPE_KEY_ID opens the restored database's dv1 values. Rotate the id through envelope-rewrap, never by editing env in place. If the original deployment used a custom id and you restore the KEK via PROBECTL_ENVELOPE_KEY, pass that id to restore-control-state with --key-id (the same contract as backup-open); the shipped default id is file and needs no flag.

The backup-open step is the normal restore path because shipped Postgres and ClickHouse backups are .pbk by default. Checksum sidecars are required restore inputs: the artifact and its .sha256 file travel together, and restore refuses to decrypt, copy, drop, or restore until the checksum passes. restore_clickhouse.sh accepts a sealed .zip.pbk, verifies its checksum sidecar, opens it into an ephemeral .zip, copies that zip back to the ClickHouse server backup path, and then runs the destructive restore. Plain .dump or .zip artifacts should exist only from explicit raw-backup acknowledgments; treat them as exposed tenant data.

One database user, all the way through

Run every migration as the SAME database user that bootstrapped the schema. Migration 0007 grants the application role DML on future tables through ALTER DEFAULT PRIVILEGES, and Postgres records that for the role that executed it: tables created later by a DIFFERENT user carry their row-level policies without the grant behind them. Nothing complains at migration time — the control plane refuses to start afterward and names the tables (DPR-122). This bites most easily in a restore into a scratch database under another login, or after a migration-user rotation.

How the Jobs reach the datastores (TLS)

The backup CronJobs and the restore Jobs carry the entire tenant metadata database and all tenant telemetry across the pod network, so they connect at least as strictly as the control plane does (§7.12):

• Postgres — backup.postgres.sslmode and restore.sslmode. Left empty they resolve to verify-full when control.trustBundle.existingConfigMap is set (that bundle is mounted into the Job and used as PGSSLROOTCERT), and to require otherwise — encrypted, but with no CA there is nothing to verify against. The connection host must match a name in the server certificate.
• ClickHouse — backup.clickhouse.secure / restore.clickhouse.secure (default on) use the native TLS port 9440 with a verified certificate. clickhouse-client takes TLS settings from a config file rather than flags, so the chart mounts one that sets strict verification against the same trust bundle. A hardened ClickHouse does not listen on plaintext 9000 at all; set secure=false and port=9000 only for a deliberately plaintext server.

Restoring on Kubernetes (chart-managed restore Jobs)

The compose scripts above are the host/dev path. On Kubernetes the chart ships one-shot restore Jobs so the restore is reproducible and audited, NOT a manual kubectl exec (OPS-001 for Postgres, OPS-007 for ClickHouse). Postgres reads its sealed .pbk from the chart backups PVC inside the restore Job. ClickHouse is different: RESTORE ... FROM File(...) runs inside the ClickHouse server, so the .zip must already be visible on the ClickHouse server's configured backup path (restore.clickhouse.serverBackupPath, default /backups). A PVC mounted only into the restore Job pod is the wrong filesystem.

# Postgres — verifies .pbk.sha256, decrypts into an ephemeral work volume, then pg_restore:
helm upgrade probectl deploy/helm/probectl --reuse-values \
  --set restore.enabled=true \
  --set restore.backupFile=postgres-probectl-<ts>.dump.pbk

# ClickHouse — sealed .zip.pbk is opened into the same server-visible CH backup
# path, then RESTORE DATABASE ... FROM File(...) runs server-side:
helm upgrade probectl deploy/helm/probectl --reuse-values \
  --set restore.clickhouse.enabled=true \
  --set restore.clickhouse.backupFile=clickhouse-probectl-<ts>.zip.pbk \
  --set restore.clickhouse.serverBackupPath=/backups

# Rehearse without touching the live database: restore into a scratch database
# on the same server (create it first). The Job restores AND migrates that
# database — restore.host/restore.port/restore.database are the single target
# for both steps, so a drill can never migrate production (DPR-107):
helm upgrade probectl deploy/helm/probectl --reuse-values \
  --set restore.enabled=true \
  --set restore.backupFile=postgres-probectl-<ts>.dump.pbk \
  --set restore.database=probectl_restore_drill

# Each is a Job with backoffLimit 0 (fail loud, never silently retry-and-clobber).
# Watch it to completion, then DISABLE it again so a later upgrade doesn't re-run it:
kubectl logs -f job/probectl-clickhouse-restore
helm upgrade probectl deploy/helm/probectl --reuse-values \
  --set restore.clickhouse.enabled=false

RPO / RTO expectations

• RPO (recovery point objective — how much data you can lose): Postgres dump RPO follows the Postgres backup cadence unless you enable WAL archiving; ClickHouse telemetry RPO follows the off-region ClickHouse backup cadence. With the shipped nightly chart schedules that is ≤ 24 h plus off-region copy lag. Tighten the schedules, add WAL archiving for Postgres, and use ClickHouse incrementals for less. The WORM audit exports run on their own interval, but their regional RPO follows the object-store snapshot/copy cadence. The default object CronJob is nightly at 02:45.
• RTO (recovery time objective — how long a restore takes):
  • Small / dev-sized: minutes — usually single-digit seconds at drill size. The CI drill measures the real number on every run and prints backup Ns, restore Ms.
  • Medium / large: dominated by the ClickHouse volume — budget roughly artifact size ÷ disk throughput plus ~2 min of orchestration. Run a production-shaped drill and record the number below.

The committed transcript is a release artifact seed; CI's backup-drill job publishes the same CSV shape as a downloadable backup-restore-results artifact on every run. For stricter hardware sign-off, run the same command on the loaded reference stack and keep the emitted BACKUP_RESTORE_RESULT row with the release evidence.

The drill (executed, not aspirational)

make backup-restore-drill

This seeds nonce-marked rows in both databases (137 rows in Postgres, 251 tenant-scoped rows in ClickHouse plus a second-tenant control) and generates a cryptographically valid three-event Ed25519-signed provider WORM segment plus a tenant object. It backs up all three stores, drops both databases and removes the object tree, restores only from the off-box artifacts, then asserts every database marker survived and the WORM segment, signature, public key, and tenant object are byte-for-byte identical. It prints the measured backup/restore times. The nonce (a one-time random marker) is what makes the check honest: a row count alone could pass on leftovers from an earlier run, but only this run's rows carry this run's nonce, so a pass proves the restore really round-tripped today's data. The ClickHouse leg is also a regional-loss proof for the default telemetry profile: after restore, it queries the marker table by tenant_id, so the drill proves tenant-scoped telemetry comes back from the off-box artifact within the documented RPO window. It runs against the dev compose stack, exits non-zero on any divergence, and runs in CI on every run (the backup-drill job), so the restore path cannot silently rot. Run it against staging after any storage-layer change.

Production-shaped restore drill

The CI drill proves the restore mechanism, but it is intentionally tiny. The M/L evidence row above must come from a loaded stack whose backup artifacts are large enough to represent the environment under review. Use the large target:

PROBECTL_DRILL_MIN_ARTIFACT_BYTES=<minimum bytes for the loaded dataset> \
PROBECTL_DRILL_RTO_BUDGET_SECONDS=<restore budget seconds> \
PROBECTL_DRILL_RPO_SECONDS=<numeric backup-cadence RPO seconds> \
PROBECTL_DRILL_RESULT_FILE=backup-restore-results.csv \
  make backup-restore-drill-large

backup-restore-drill-large runs the same destructive backup → wipe → restore → verify loop, but refuses to pass when the summed Postgres + ClickHouse + object-store backup artifacts are below PROBECTL_DRILL_MIN_ARTIFACT_BYTES or when restore time exceeds PROBECTL_DRILL_RTO_BUDGET_SECONDS. That keeps the production-shaped row honest: a marker-sized dev database cannot accidentally satisfy the reference-hardware acceptance line. PROBECTL_DRILL_RPO_SECONDS records the numeric backup-cadence RPO in the transcript; the shipped nightly profile uses 86400 seconds.

Point-in-time recovery (PITR) — WAL archiving (OPS-008)

The nightly logical dump above gives you a daily restore point. For tighter RPO — recovering to any moment, not just the last dump — enable Postgres continuous WAL archiving. This is a tested recipe, not just a pointer.

What PITR buys you: with a base backup plus the WAL stream, you can replay to a chosen timestamp (e.g. "the instant before the bad migration"), so your recovery point objective drops from "up to 24h of loss" to "seconds".

Postgres server settings (set, then restart):

wal_level = replica
archive_mode = on
# Archive each completed WAL segment to durable, OFF-host storage. The archive
# MUST be encrypted at rest (it contains tenant data) and MUST NOT live on the
# same volume as the data directory — a disk loss would take both.
# backup-seal is a stdin→stdout encryption filter (it has NO --in/--out flags):
# feed the WAL segment %p on stdin via shell redirection, pipe the sealed bytes
# to durable storage. The KEK comes from PROBECTL_ENVELOPE_KEY (or --key-file).
archive_command = 'probectl-control backup-seal < %p | aws s3 cp - s3://YOUR-BUCKET/wal/%f'
archive_timeout = 60   # force a segment at least every 60s, bounding RPO

Take a base backup (also sealed) on a schedule:

pg_basebackup -D - -Ft -z -Xnone | probectl-control backup-seal \
  | aws s3 cp - s3://YOUR-BUCKET/base/$(date -u +%Y%m%dT%H%M%SZ).tar.gz.sealed

Restore to a point in time:

1. Provision a fresh Postgres data directory.
2. backup-open the most recent base backup taken before your target time and unpack it into the data directory.
3. Create recovery.signal and set the recovery target:

   restore_command = 'aws s3 cp s3://YOUR-BUCKET/wal/%f - | probectl-control backup-open > %p'
   recovery_target_time = '2026-06-12 09:14:00+00'
   recovery_target_action = 'promote'
   

4. Start Postgres; it replays WAL up to the target and promotes.
5. Run probectl-control migrate (idempotent) and verify /readyz, then run the backup-restore drill's verify pass against the recovered instance.

Strict profile: in regulated deployments WAL archiving is required, not optional — archive_mode = on with a sealed, off-host, encrypted archive and an archive_timeout that meets your RPO. ClickHouse PITR uses its native BACKUP ... TO incrementals to the same off-host, encrypted store (see OPS-011).

Restoring onto a brand-new cluster (roles)

A logical pg_dump does not carry cluster roles. backup_postgres.sh therefore writes a <dump>.roles.sql companion (pg_dumpall --roles-only --no-role-passwords, probectl* roles only — no password hashes leave the source). When restoring onto a fresh Postgres cluster, keep that companion next to the dump and set PGAPPPASSWORD to the application role's password: restore_postgres.sh recreates probectl + probectl_app + the NOLOGIN RLS/auth roles (as the bootstrap PGSUPERUSER, which defaults to PGUSER — probectl's own compose/Helm Postgres is initdb'd with probectl as the cluster superuser and has no separate postgres role, so the default is correct there; set PGSUPERUSER to the real bootstrap role, e.g. postgres, only when the DR target cluster's superuser differs) and sets the login role's password from PGAPPPASSWORD before restoring the data, so the owners / GRANTs / RLS policies in the dump apply and the control plane can boot. Each carried CREATE ROLE is duplicate-tolerant, so re-running the restore (or restoring in place, where the roles already exist) is safe (RTO-15).