Data Retention Matrix

What this is. One privacy map for the data probectl keeps: what kind of data it is, why it exists, where it lives, and what clock deletes it. ELI5: every data set is a labeled box. The label says what is inside, why we are allowed to keep it, which shelf it sits on, and which timer removes it.

This page is a governance/control map, not a new retention engine. Today, retention is still enforced by the owning stores and lifecycle paths: ClickHouse TTLs plus ClickHouse hourly rollups, the in-memory TSDB window, derived identity-cache pruning, audit pruning after durable export, AI-answer pruning on write, and tenant/subject erasure. The audit-retention runner wakes hourly in every profile. The single deployment profile defaults to 0 (provider rows keep forever), while a finite tenant audit_retention_days override can still tighten its own local rows. multi-tenant and regulated default to a finite 8760h (365-day) deployment maximum and require WORM/SIEM watermark configuration before startup. /v1/lifecycle/retention lets a tenant set stricter per-plane clocks for flow, OTLP, eBPF, path, audit, persisted AI answers, object artifacts, and derived identity caches; NULL means the deployment default remains in force. Where a backing store cannot safely delete one tenant's older rows locally (for example a remote aggregate TSDB or object store lifecycle), the sweeper emits an honest delegated/not-capable receipt instead of pretending it pruned.

The machine-checkable data inventory lives in internal/govern.DataInventory(). The privacy gate fails if a row lacks owner, data classes, retention, processors, export behavior, tenant deletion semantics, or subject deletion semantics. Current inventory IDs: audit-evidence, ai-artifacts, backups-snapshots, device-endpoint-state, ebpf-telemetry, flow-telemetry, identity-rbac-scim, object-artifacts, otlp-traces-logs, path-topology, probe-results, shared-open-data, siem-cursors.

Enforcement Model

• Tenant first, then purpose. Every retained row, object key, metric, and export cursor is tenant-scoped before RBAC or governance policy is considered. If the tenant cannot be proven, the read/delete path returns nothing.
• Age retention is store-owned. High-volume ClickHouse stores use delete TTLs for raw rows and hourly rollup tables for lower-resolution history; the in-memory TSDB uses an arrival-time window; audit logs prune only after WORM/SIEM export proves the evidence survived elsewhere.
• Lifecycle deletion is wider than age retention. Tenant erase and subject erase are the privacy "big brooms": they remove or project data across live stores even when a store has no age TTL.
• Policy and proof commit together. A tenant retention-policy update and its explicit lifecycle.retention_set audit event use the same tenant-scoped PostgreSQL transaction. If either write fails, neither becomes visible.
• Engine-owned prunes are audit-gated. Before the lifecycle engine deletes session detail, flow/OTLP/eBPF/path rows, persisted AI answers, endpoint history, or derived topology/endpoint cache entries, a bounded provider-audit intent must commit. Its non-secret attempt_id pairs it with an enforced/failed count-only terminal receipt. Failed intent means no prune; failed terminal append is returned to the scheduler instead of being hidden in a warning.
• Delegated clocks are receipts, not triggers. The tenant objects clock describes external object lifecycle, so the lifecycle engine emits one delegated receipt and does not claim an intent or deletion it did not invoke. Audit is different: its dedicated hourly runner resolves the tenant's effective bounded window and emits audit.retention_prune only when a SIEM-watermarked prefix is actually removed.
• Copies keep their own clock. Backups, WORM exports, and customer SIEM destinations are not live stores. probectl records the configured backup window and export cursor, but the destination system owns its own retention.
• No object-store age TTL is enforced by FSStore. The filesystem object store keeps objects until tenant erase/subject erase, explicit prefix delete, or the operator's bucket/filesystem lifecycle policy removes them.

Data-Class And Purpose Matrix

Config Clocks

Practical Reading

• If an auditor asks "how long do flow rows live?", start at the flow row and check PROBECTL_FLOW_RETENTION_DAYS, then check whether the tenant has a tighter flow_retention_days override.
• If they ask about topology or endpoint labels, check PROBECTL_DERIVED_IDENTITY_RETENTION_DAYS; a tenant's tighter flow_retention_days shortens that derived-cache window too.
• If they ask about a person, use subject export/erase. Age TTLs are coarse timers; subject lifecycle is the tenant-scoped privacy control. Its manifest lists every privacy-relevant plane with exported/deleted counts, projected audit status, not-deployed status, covered_by_parent for RUM labels covered by tsdb_metrics, retained_encrypted_evidence for the provider-only IR sidecar that ordinary subject reads cannot access, or not_capable plus the age-out/delete-series basis for aggregate backends that cannot safely delete one subject locally. A subject erasure receipt is complete:true only when every deployed plane was erased, count-verified clean, or explicitly preserved as separately encrypted evidence; any deployed not_capable plane keeps it complete:false while the report is still hashed and written to provider audit.
• If they ask about backups or SIEM, give the probectl attestation/cursor plus the operator's SIEM or backup policy. probectl cannot truthfully promise the destination deleted its copy.