Runbook: tenant offboarding (export → erase → attest)

What this is

The clean way to remove a tenant: hand them their data, irreversibly delete it from every store, and produce a signed attestation that it's gone. An attestation is a recorded statement of fact you can hand to an auditor — here, per-store deletion counts and verified-zero checks, hash-anchored to a tamper-evident log so it can't be edited after the fact. Verifiable is the load-bearing word throughout: the engine never just issues deletes and hopes — it re-reads every store afterward and records the zero. Export and verifiable deletion are core — a compliance right available in every edition, not a paid add-on. The provider-console trigger is just the MSP convenience layer over the same engine.

0. Before you start

• Confirm the request's authority (a tenant admin or the contract owner).
• Know your backup TTL (time-to-live — the age at which a snapshot is automatically deleted). Deletion of the live stores is attested immediately, but snapshots expire on your schedule. Set PROBECTL_BACKUP_RETENTION_NOTE (e.g. "nightly snapshots, 14-day TTL, region X") so every attestation states it verbatim. An attestation that says nothing about backups is incomplete in spirit — be explicit.

1. Export (portability)

Tenant self-service: Admin → Data lifecycle → Export my data, or GET /v1/lifecycle/export (permission lifecycle.export). The bundle is a tar.gz containing:

• manifest.json — row counts, object inventory, and format notes (format_version: 1);
• postgres/<table>.jsonl — ordinary tenant-owned rows, one JSON object per line;
• flows.jsonl — every flow record, streamed from the flow store.
• endpoint_events.jsonl — every raw endpoint/DEM event, streamed through the tenant-scoped event-store boundary.
• otel_spans.jsonl and otel_logs.jsonl — every OTLP span and log record.
• ebpf_edges.jsonl — every eBPF workload aggregate (the service map).
• path_hops.jsonl and path_links.jsonl — the hop and link rows of every path discovery round.
• topology.jsonl — the topology graph's nodes and edges, as it stands.

Every store the erase in step 3 clears is either in the bundle or named in the manifest's notes, with the reason. The manifest counts each file's rows. Hourly flow and path rollups are derived aggregates and are not bundled; the raw rows they summarize are, for as long as those rows are retained.

Ordinary portability never reads, unseals, or exports the provider-only encrypted incident-response attribution sidecar. That evidence is accessible only through the audited ir-investigator separation-of-duty path. Every manifest carries the same fixed policy note, whether or not evidence exists, so the bundle exposes neither sidecar contents nor a presence/absence signal.

Time-series metrics are not in the bundle: export them via the Prometheus-compatible API (federation / PromQL) — the manifest says so. Hand the bundle to the customer before you erase anything.

1a. Subject export / erasure (data-subject request)

A subject request is the smaller privacy workflow: remove or return the rows that mention one person or identifier inside one tenant, without deleting the whole tenant. Think "empty Alice's folder in tenant A," not "search every tenant for Alice." The tenant id is still the outer wall: probectl chooses the caller's tenant first, then applies the subject filter inside that tenant.

Export uses POST /v1/lifecycle/subjects/export (permission lifecycle.export):

{"subject":"alice@example.com","redact":true}

The response is probectl-subject-export.tar.gz. It contains manifest.json with only a tenant-scoped subject hash, plus matching JSONL files such as postgres/users.jsonl, flows.jsonl, otel_spans.jsonl, and otel_logs.jsonl, tsdb_metrics.jsonl, topology_subject.jsonl, ebpf_edges.jsonl, and endpoint_subject.jsonl when those planes contain matching rows. RUM host/path labels are covered by tsdb_metrics; device labels are counted from topology device nodes. The raw subject is in the exported data because this is the tenant's portability bundle; it is not stored in the manifest or provider audit receipt.

CLI equivalent:

probectl lifecycle subject-export --subject alice@example.com --redact > subject-export.tar.gz

Erasure uses POST /v1/lifecycle/subjects/erase (permission lifecycle.erase):

{"subject":"alice@example.com","confirm":"alice@example.com","reason":"dsar"}

The exact confirmation is deliberate friction for an irreversible action. The engine removes matching identity rows, persisted AI answers, flow rows, OTLP trace/log rows, in-memory TSDB metric labels, topology/device graph labels, eBPF workload aggregates, and endpoint latest-view labels for the caller's tenant only when those stores are wired. Audit rows are append-only, so the engine records a privacy.subject_erase marker instead of rewriting history; the same transaction records its tenant-scoped hash in audit_subject_erasures. Future audit reads/exports project matching structured actor/target/data values as [erased-subject] while the hash chain stays verifiable. Classified Postgres deletions and every captured stable-alias marker/projection share one tenant transaction, so a failed marker cannot leave identity rows deleted without their privacy projection. Subject and full-tenant bundles use the same projected audit serializer and retain the tenant, sequence, timestamp, and chain fields. Exported marker events can later age out without losing that projection; retention captures marker-only events from a rolling older binary before it deletes them. Aggregate backends that cannot locally delete a single subject are not hidden: the returned report lists each plane's deleted and remaining counts, not_capable/age-out basis where needed, and includes report_sha256.

CLI equivalent:

probectl lifecycle subject-erase --subject alice@example.com --confirm alice@example.com --reason dsar

2. Suspend, then offboard (provider console)

Suspend stops the tenant's users from logging in. Offboard then frees the licensed tenant-band slot and blocks tenant access.

Stop the tenant's agents before the erase. A browser agent writes its synthetic artifacts (screenshots, page captures) straight to the object store under the tenant's prefix. The control plane cannot fence a write it does not make, so an agent still running after the erase puts new artifacts back where the erase verified zero. Stop (decommission) the tenant's agents first, or revoke their object-store credentials. Flow, eBPF and endpoint collectors publish through the bus, where the erase's fence refuses them, but they run on the tenant's hosts: stop them too. Offboard is deliberately non-destructive for every isolation model: it does not reclaim the silo, so the encrypted incident-response sidecar and its signed chain remain available to the verified erase plan.

3. Erase (irreversible, verifiable)

Provider console (admin): tenant row → Erase, or POST /provider/v1/tenants/{id}/erase with body {"confirm":"<tenant-slug>"}. Tenant-side, the equivalent is POST /v1/lifecycle/erase (permission lifecycle.erase). The engine walks every store, deletes the tenant's data, and verifies each store reads zero afterward. Note the Postgres deletes run under the same row-level security (RLS) scope as live queries — the eraser is incapable of reaching another tenant's rows, even buggy:

Before the first delete, the engine changes the tenant to offboarding, sets the write-once audit_write_fenced_at marker, and records that transition while holding the tenant-wide writer lock and the tenant's audit-stream lock. PostgreSQL rejects clearing the marker or reopening a fenced tenant. Every ordinary tenant-owned pooled or physical-silo table has an ALWAYS INSERT/UPDATE trigger that takes the matching shared writer lock and re-reads the durable registry row. An in-flight writer therefore finishes before the fence and is then erased; a later or rolling-old writer waits for the committed transition and is rejected. The append-only audit_events and audit_subject_erasures tables keep their stronger stream-lock triggers and are deleted and count-verified in one routed transaction under that lock. The production flow-store decorator applies the same contract to every ClickHouse or in-memory flow Insert: it takes shared leases in canonical tenant order, re-reads every tenant's durable lifecycle row, and holds the leases through the actual backend write. A mixed-tenant batch is rejected before any row reaches the backend if one tenant is fenced. Transient database or lease errors fail the whole flow insert and are safe to retry; once the offboarding fence commits, retries for that tenant remain rejected. Endpoint/DEM event Insert uses the same durable lease at its store boundary, including the lightweight memory backend and every pooled or silo-routed ClickHouse target. The eBPF aggregate store applies that identical contract before any service edge batch enters its memory or pooled/silo-routed ClickHouse backend. OTLP span and log writes, path Save/SaveBatch, and both tenant TSDB entry points (result-bus ingest and Prometheus remote write) hold the same lease through their concrete backend mutation. The raw TSDB handle remains available only for queries, lifecycle deletion, and provider-global self-metrics; tenant writes never receive it directly. The in-memory or indexed topology store is decorated before it is handed to the API or any consumer, so every tenant-bound graph mutation is suppressed after fencing while deletion and retention capabilities remain available to the lifecycle engine.

Agent-local browser object storage does not perform a new control-plane lease round trip. Instead, the existing mTLS StreamResults boundary recognizes an artifact-bearing result by its browser.screenshot.key attribute and holds the control-plane PostgreSQL writer lease across publication. An active tenant's artifact reference is accepted normally; an erasing/erased tenant's reference is refused before it reaches the result bus, while another tenant remains writable. Bus and pipeline buffers need no independent fence: every durable downstream writer above rechecks the authoritative lease before mutation, so a delayed record cannot refill a store after it was verified empty.

Provider-global lifecycle evidence and tenant-key destruction remain on their separate provider-maintenance paths, so retrying an incomplete erase can still record its bounded receipt or complete crypto-shred without reopening ordinary tenant writes.

Provider-row deletion, the deleted registry tombstone, and the successful attestation append commit together only after every data store and key domain has completed. If establishing and auditing the fence fails, both changes roll back. After the fence commits, any later or finalization failure leaves the tenant offboarding, fenced, and retryable; correct the reported cause and rerun the idempotent erase.

After Offboard, the isolated container remains until this erase completes. Physical container reclamation requires a separate tombstone-aware maintenance procedure; never use a raw schema drop, because that would erase the retained encrypted IR sidecar.

The engine also erases the other tenant-scoped planes the same way (path, topology, endpoint/DEM event history, and externally-ingested OTLP traces/logs) — they appear in the attestation's store list too. The table above is the representative subset most often asked about.

Crypto-shred, in one image: a tenant's data sitting inside your still-live backups is a locked safe. Crypto-shred works only when you destroy every copy of the key — so the engine can honestly attest deletion of data inside backups it never touches only when probectl held no copy of the key to begin with. That is BYOK: the key lived in your secret manager, you destroy it there, and every such safe becomes scrap metal (ciphertext without a key is not data). In managed mode the deployment master is a surviving copy of the key — it wraps the per-tenant key — so a pre-offboard backup is not scrap metal: it is covered by verifiable deletion plus your backup TTL, not by crypto-shred.

For IR attribution, overwrite/unlink covers only the exact local artifacts mounted to probectl. Destroy operator backups, snapshots, escrow copies, or an external KMS handle separately. A signed probectl tombstone permanently refuses a restored local artifact, but it cannot claim physical zeroization of storage outside the operator-controlled mount.

The tenant registry row is then tombstoned (status=deleted): the row remains as a referent for the attestation, but it holds no telemetry. The rest of what stays is provider-global integrity metadata, by design: the tenant's audit stream head (audit_stream_heads: sequence numbers and hashes, no audit payload), the IR key-shred receipts (ir_key_shred_*), and the provider audit chain carrying the attestation's digest. None of it holds the tenant's telemetry or identities.

4. The attestation

The engine returns — and the tamper-evident provider audit chain records, along with the report's SHA-256 — a deletion report: per-store deleted counts, verified-zero flags, your backup-TTL statement, and a complete flag. If complete is false, the notes name exactly what remains (e.g. the Prometheus manual step) — finish it and re-run erase (it is idempotent). Hand the attestation JSON to the customer; the SHA-256 on the provider chain is their proof it was not edited after the fact. The erase removes the tenant's users and sessions with its data, so a tenant admin who erases from Admin → Data lifecycle is signed out by it: the console keeps the receipt on screen with a Download receipt (JSON) link, and Done goes to the login page. The provider chain stores only the digest, so keep the receipt before leaving.

5. After

• Backups: the attested deletion covers the live stores. Your snapshots expire on the stated TTL — do not restore an erased tenant's backup except under legal hold.
• Bus records: the bus is a transport buffer, not a store, and the erase does not rewrite it. Records the tenant's agents and collectors published stay on their Kafka topics or NATS streams until the broker's retention drops them (the Kafka broker's log retention; PROBECTL_BUS_STREAM_MAX_AGE for NATS, 168h by default). They cannot refill a store: every writer behind the bus rechecks the tenant's lifecycle fence first. Size broker retention with that window in mind, as you do the backup TTL.
• Custom-domain TLS: if the tenant had a custom domain, remove its ingress certificate and DNS.
• Agents: the agents' mTLS identities die with the registry rows; decommission the tenant's agent hosts.