Tenant isolation models

"Isolation model" answers a simple question: how physically separate is one tenant's data from another's? (A tenant is one isolated customer/organization sharing the deployment — the outermost security boundary in probectl.) probectl offers three answers, and you can pick per deployment and per tenant — a single install can run most tenants pooled and a few high-compliance ones siloed. This page covers the models; the storage-layer enforcement mechanics they all share (forced RLS, partition keys, the cross-tenant test suite) live in security/tenant-isolation.md.

The mental model is a spectrum:

• Pooled — everyone shares the same tables/databases/topics, told apart by tenant_id. Cheapest, densest, and the default. Stays core (free).
• Siloed — each tenant gets its own Postgres schema, ClickHouse database, bus topics, and object-store namespace. The strongest separation.
• Hybrid — pooled control plane (the cheap, shared part) but a per-tenant ClickHouse database for the high-volume flow telemetry. A middle ground.

As housing: pooled is an apartment building — one structure, a hard lock on every unit; siloed is a street of detached houses; hybrid is an apartment with a detached private garage for the bulky stuff (the flow telemetry). The spectrum trades cost and density against physical distance.

Siloed and hybrid are ee/ features, unlocked by the siloed_isolation license feature (an MSP-tier capability); pooled always works regardless of license.

Reading the table needs three terms. RLS is Postgres Row-Level Security: the database itself appends an invisible tenant filter to every statement, so even an unscoped SELECT returns only the current tenant's rows — or nothing, when no tenant is bound. A partition key is the column ClickHouse uses to decide which physical chunk of disk a row lands in — partitioning by tenant_id keeps each tenant's rows physically grouped. The bus is the publish/subscribe message pipe between producers and the control plane; its named channels are topics.

The key idea: physical separation is layered on top of the pooled scoping, never instead of it. A siloed table has both the caller-scoped tenant_isolation policy and a RESTRICTIVE policy that fixes rows to the tenant UUID encoded by its physical schema. Every transaction still binds the tenant setting, bus messages stay tenant-keyed, and every read is still tenant-scoped at the query layer. So even if routing sent a query to the wrong silo—or a restored permissive policy survived—the query would return nothing rather than another tenant's rows, and writes would be rejected. The boot posture check refuses any canonical silo table without this exact guard. The defenses stack; they don't replace each other.

Every probectl-control one-shot command that touches tenant rows — bootstrap-admin, mcp-token, scim-token, enroll-token, register-collector, revoke-agent — installs the same tenancy router the serving process uses, so a siloed tenant's first administrator, tokens and registrations land in its schema and authenticate immediately (DPR-045).

Fail closed on routing — everywhere, including the bus. ("Fail closed": when the system cannot prove the safe answer, it refuses the operation rather than guessing.) The isolation router (ee/silo.Router, installed at the editions attach seam) resolves each tenant's storage targets from the tenant registry. A routing error fails the operation — a siloed tenant is never silently downgraded to the pooled stores, and a pooled query can never reach a siloed tenant's stores. (tenancy.InTenant resolves targets before opening the transaction; the flow store splits a batch per target and fails the whole batch on a routing error.) Bus lanes are no exception: if a siloed tenant's lane cannot be resolved when a result arrives, the control plane drops that result with a loud error rather than publishing it onto the shared topic (internal/agenttransport/service.go) — a siloed tenant's telemetry must never silently ride the shared lane. Think of a courier with an unreadable address label: the parcel goes back to the depot, never onto the public loading dock. Availability comes from the agent, not from a fallback: the agent's store-and-forward buffer (the depot) retries delivery, so a transient routing blip delays the data instead of mis-routing it.

How each leg works

• Postgres (siloed): each tenant gets a schema named t_<uuid> (the tenant UUID lowercased, dashes stripped — silo.SchemaName) containing every tenant-owned table. The table set is derived live from information_schema — any public table with a tenant_id column, minus a provider-owned deny list — so the silo automatically tracks whatever tables the schema actually has. Each table is created LIKE public.<t> INCLUDING ALL; forced RLS, the caller policy, the schema-bound restrictive policy, and app-role grants are then applied. tenancy.InTenant routes a siloed tenant by running SET LOCAL search_path TO <schema>, public, so global tables (permissions, tenants) still resolve in public.
• ClickHouse: a per-tenant database probectl_t_<uuid> holding every telemetry plane's tables — flow, path (hops/links), eBPF L7 edges, and OTLP traces+logs (TENANT-001). Inserts are split per target and reads route by the query's tenant. With a residency pin, all of them run against that data plane's ClickHouse URL. (Earlier releases physically separated flow only; the other three planes have been brought up to the same contract.)
• Bus: topics gain a namespace segment, e.g. probectl.t-<slug>.network.results. The control plane publishes a siloed tenant's results/RUM onto its own lane. The runtime lane supervisor polls the tenant registry and restarts only the bus subscribers when that namespace set changes, so a tenant siloed after boot is picked up without a process restart; the shared lanes stay subscribed throughout, so pooled tenants keep flowing. On a shared lane the record's own tenant_id is not trusted: the bus key — set by the control plane from the agent certificate or the RUM app key — is the authority, and every consumer of the shared result and RUM topics (the storage pipeline, the latest-results view, SLO, outage, IOC, NDR, TLS posture, RUM and the OTLP export) drops a record whose payload tenant disagrees with its key, so a bus credential for one tenant can never write into another's views or signals. A record keyed to a tenant the isolation router sends to its own lane (siloed or hybrid) is refused on the shared lane outright, by every lane consumer, whatever its payload says: its producers publish on its lane, so a shared-lane copy can only come from a misrouting control plane, a stale producer or a forger. Refusals are logged and counted in probectl_pipeline_tenant_rejected_total. Records already in flight on the shared lane when a tenant moves to its own lane are refused the same way, not rehomed. Pooled tenants keep using the shared lane even though they also own a collector lane for strict-lane mode.
• Object store: tenant-owned artifact callers use a bound object-store handle, so they pass relative paths like browser/shot.png and the storage adapter prepends the namespace. Siloed/hybrid keys move under silo/<tenant-id>/… (the pooled layout is tenant/<id>/…), and list/delete operations stay inside that bound root. The artifacts written today come from agents (browser failure artifacts), and an agent cannot learn its tenant's model from the control plane — there is no config push (adr/config-push.md) — so a siloed or hybrid tenant's agents declare it locally, artifact_store.isolation: hybrid (or siloed), the way they are given their bus lane. An agent that declares nothing keeps writing tenant/<id>/…; lifecycle export and erase cover both namespaces. Note the honesty caveat below: in this release that is a key namespace on the same backend, not a separate storage system.
• Topology graph: API, AI, NDR, what-if, and topology consumers bind the store with ForTenant(tenant_id) before reading or writing graph data. The resulting tenant handle has no tenant argument on Latest, SnapshotAt, Neighbors, Traverse, or Observe*, so a caller cannot accidentally swap tenant strings mid-operation. Empty/invalid tenants fail closed, and reads of never-seen tenants return empty snapshots without creating a graph.

Residency: exactly what is and is not pinned

Residency is a legal or contractual constraint on where — which jurisdiction's hardware — data may be stored. This section is deliberately precise because a residency claim you cannot back up is a compliance liability. PROBECTL_DATAPLANES names the available planes (e.g. eu=https://ch-eu:8123;us=https://ch-us:8123); a siloed or hybrid tenant provisioned with residency: eu gets its ClickHouse database created on and routed to that plane. Removing a plane from PROBECTL_DATAPLANES while a siloed or hybrid tenant (not deleted) is still pinned to it refuses startup, naming the tenant and the plane (DPR-324): the tenant is never quietly routed to the default plane, where its database does not exist and which may sit in another jurisdiction. With PROBECTL_SILO_FAILURE_MODE=quarantine that tenant is quarantined instead and every other tenant starts (see "Per-tenant quarantine" below).

Per-tenant quarantine (D-62)

Startup proves every siloed and hybrid tenant's isolation targets before the silo router is published: its residency plane is configured, its Postgres schema is caught up, the reader login has a credential for its plane, and the reader scope on its per-tenant ClickHouse database is installed and proven. PROBECTL_SILO_FAILURE_MODE decides what a failure does:

• stop (the alpha default; the v0.7.0 beta tag flips the default to quarantine, D-66): startup refuses for every tenant, naming each failing tenant and the fix.
• quarantine: only the failing tenant is quarantined; the control plane starts for everyone else.

A quarantined tenant is refused at the routing layer, not in handlers: the silo router returns no target for it (no Postgres schema, ClickHouse plane, object prefix or bus lane) and tenancy.InTenant refuses its transactions, so every path — API, CLI, UI, MCP/AI, exports, bus consumers, background jobs — fails closed for that tenant alone. Its users get HTTP 503 tenant_quarantined naming the failed check and the plane (never the operator-side error text or configuration keys). Its bus lanes are not consumed, so its records wait on the bus and resume from the committed offset on release; a record of it that reaches a shared lane is refused at every consumer's lane gate, acknowledged unprocessed and counted (probectl_pipeline_quarantine_dropped_total). A quarantine entered at runtime drops the tenant's lanes from the subscription at once, and a record already delivered on its own lane is left unacknowledged on the bus (DPR-389). Its erasure and break-glass requests and reads are refused until release (409 tenant_quarantined). One-shot commands (mcp-stdio, the token commands, bootstrap-admin) honor the quarantines the serving replicas hold.

Independently of quarantine, tenancy.InTenant refuses any siloed tenant whose routed Postgres schema does not exist (DPR-368), or that lacks any of the tenant-owned tables every silo must carry (DPR-385): Postgres accepts a missing schema on search_path and resolves a missing table to the next entry, so the tenant's rows would otherwise land in the pooled public tables. The set is the one the provisioner and the catch-up use (the public tables with tenant_id, minus the provider-owned ones), installed at startup; the check rides the same catalog query as the schema check, one per transaction. The check runs at the start of each transaction; long-running, multi-statement transactions — export, erasure, subject export and erasure, audit-retention maintenance — also take ACCESS SHARE on every tenant-owned table of the silo at their start (D-73, DPR-401), so a concurrent DROP TABLE or DROP SCHEMA blocks until they end and their later statements can never resolve to public. ACCESS SHARE conflicts only with ACCESS EXCLUSIVE, so the tenant's reads and writes are not slowed. The startup catch-up covers active and suspended silos in full and offboarding silos schema-only (D-71: empty tables and columns, no row changed), so an offboarding tenant's export and erasure keep working after an upgrade adds a tenant-owned table. Startup verifies every caught-up silo is complete after the catch-up — stop mode refuses to start naming the tenant and table — and in quarantine mode the supervisor re-verifies every silo each recheck interval and quarantines one that lost a table as silo_schema_incomplete. It never recreates the table itself: restore it (for example from backup), or roll the control plane, whose startup catch-up recreates missing tables empty; the tenant is released once its schema is complete.

Each tenant check — the silo schema catch-up and the reader-scope proof — runs on its own budget, PROBECTL_SILO_QUARANTINE_CHECK_TIMEOUT (default 5m). A check that overruns it is that tenant's own fault (a slow catch-up, a reader-scope lock another replica holds during a rollout) and is retried every recheck; only a refused connection or an open breaker marks a plane down for the other tenants on it (DPR-380).

The audit trail is per tenant transition (DPR-381). Each control-plane process keeps a row in tenant_quarantine under its own identity, the host name plus a per-process nonce (DPR-386), so two live processes on one host never take over each other's rows; a process takes over only the rows of a predecessor on its host that has stopped refreshing them for one recheck interval. tenant.quarantine_entered is recorded on the provider stream (with the cause, fix and replica) and the tenant's own chain (reason and plane; no host or pod name, DPR-388) only when the first live process starts holding the tenant, and tenant.quarantine_released only when the last one lets go — or when the replicas that held it are gone. Restarts, rollouts and several replicas do not repeat the events. A tenant-chain receipt that cannot land (the silo schema is missing, the tenant was erased, or 20 retries ran out) is recorded on the provider stream only (tenant.quarantine_notice_provider_only) and the given-up ones are counted (probectl_tenant_quarantine_notices_dropped_total). A changed cause is recorded on the provider stream (tenant.quarantine_cause_changed).

The gauge probectl_tenant_quarantined{reason} drives the ProbectlTenantQuarantined alert, and the provider console lists each quarantined tenant with its cause and fix (GET /provider/v1/tenants/quarantine). A supervisor on each replica re-runs the failed checks every PROBECTL_SILO_QUARANTINE_RECHECK_INTERVAL and releases the tenant once they pass — no restart for a plane that comes back, a grant that is applied or a schema that is repaired. A configuration fault needs a rolling restart (D-66): a plane missing from PROBECTL_DATAPLANES or a missing PROBECTL_DATAPLANE_READER_BASIC_AUTH_FILES entry is process-static configuration with no live reload, so it clears on each replica that starts with the corrected configuration.

Quarantine is per replica: each replica proves the tenant for itself, so a replica that can reach the plane serves it while one that cannot refuses it.

Accepted residuals (D-66):

• The metrics TSDB has no storage-layer quarantine check (it has no tenant-routed interface; queries are tenant-labeled PromQL). Every reader of it is gated — the API edge, MCP authentication, the alert supervisor — and a quarantined tenant's series do not arrive, because its lanes are not consumed and its shared-lane records are refused.
• A quarantined tenant's records that reach a SHARED lane are dropped and counted, not dead-lettered. Its own lanes are not consumed, so their records wait on the bus instead.
• Kafka keeps a consumer group's committed offsets for the broker's offsets.retention.minutes (7 days by default), and the topic's retention.ms bounds the records themselves: fix a quarantine well inside that window, or its lanes resume from auto.offset.reset.

Pinned today: the tenant's ClickHouse telemetry across all five planes — flow, path, eBPF L7 edges, endpoint/DEM events, and OTLP traces+logs (the PII-heaviest plane). Each plane's per-tenant database is created on and routed to the residency data plane (TENANT-001).

Not pinned today (and you should not claim otherwise): the Postgres control/config state (it is a shared control plane), the metrics TSDB (metrics stay tenant-labeled in the deployment's TSDB), the object store (namespace-isolated on a single backend), and the bus brokers. Multi-region control-plane and HA mechanics are covered separately in multi-region.md; per-tenant encryption keys are a separate capability (byok.md).

ClickHouse database-level tenancy

Code-level scoping is the first line of defense — every flow/path query pins a tenant_id, and an unscoped call refuses with ErrNoTenant. A WHERE clause is still handler-adjacent code, so with tenant scoping on (required in the multi-tenant and regulated profiles) the database enforces it too: a row policy (ClickHouse's analog of Postgres RLS — a server-side filter attached to the table itself) on every shared telemetry table admits only rows whose tenant_id equals the per-request SQL_probectl_tenant setting, and it binds the SELECT-only reader login that every tenant read is sent as (PROBECTL_CLICKHOUSE_READER_BASIC_AUTH_FILE, DPR-285). A read that loses its WHERE returns only the scoped tenant's rows. The control login, which writes, migrates and erases, serves no tenant read. Until DPR-285 every read ran as the control login, which no policy filtered, so only the WHERE scoped reads.

A siloed or hybrid tenant's own database is read the same way (DPR-323). Its database name used to be the whole boundary: those reads ran as the control login, so a routing bug that pointed tenant X's read at tenant Y's database would have returned Y's rows. Now the provisioner puts every table of the per-tenant database under the same row policy and only then grants the reader SELECT on it (per table, policy first), the read runs as the reader with X's tenant setting, and a misrouted read finds nothing. Think of a hotel where each room used to be protected only by the receptionist handing out the right key; now every room's lock also checks the guest's name. The proof runs at provisioning and, for every siloed and hybrid tenant, at every boot before the silo router is published; a silo that fails it refuses startup. On a residency plane the reader uses its own credential for that plane (PROBECTL_DATAPLANE_READER_BASIC_AUTH_FILES). Operator prerequisite on every ClickHouse server that hosts silos: the control login holds SELECT ON *.* WITH GRANT OPTION so it can grant the reader each per-tenant table, and the reader is a SQL-created user (not a users.xml one). The mechanism, its boot-time proof and the operator prerequisites are in docs/security/tenant-isolation.md. The cross-tenant CI gate exercises every ClickHouse plane against a real ClickHouse with the WHERE removed from the read to prove it.

Lifecycle

Provisioning a tenant (POST /provider/v1/tenants with isolation_model + residency, or the console's Isolation selector) creates the isolated stores before the call returns — a siloed tenant never exists without its silo. A provisioning failure is loud, and because the DDL — the CREATE/ALTER/DROP class of SQL statements — is idempotent (safe to run twice; a second run finds the work already done), the call is simply re-runnable. A siloed tenant's Postgres schema is migration-class DDL, so it (and the startup catch-up below) runs as the migration login, PROBECTL_MIGRATE_DATABASE_URL: the serve login is least-privilege and cannot create a schema. Without that login a siloed provisioning is refused with this instruction; pooled and hybrid tenants need no Postgres DDL. Offboarding is deliberately status-only for every isolation model: API access stops and the tenant-band slot is freed, but its isolated stores remain available to the separate verifiable-deletion flow. A raw DROP SCHEMA … CASCADE is not an offboarding operation because it would erase the retained encrypted IR sidecar.

Migrations across silos (the operational cost of siloed)

This is the price you pay for siloing: migrations are written against public, so every silo schema has to be brought up to the current shape separately. probectl does this by catch-up — it re-derives the tenant-owned table set, creates any missing tables (the same LIKE + RLS recipe), and adds any missing columns (an information_schema diff). This works because the migration gate only admits expand-only changes (see lifecycle.md): create-missing + add-missing-columns covers every migration the gate allows. The rarer destructive "contract" phases (drops/renames) are run by the operator across silos. Catch-up runs automatically at startup for every siloed tenant, is idempotent, and must succeed before the silo router is published. If any tenant's catch-up fails, the control plane fails closed instead of routing that tenant to a stale schema. Per-tenant drift — the gap between a silo's schema and the current public shape — is computable (DriftFor) so the lag is always visible, never silent.

Catch-up also rebuilds the deliberately small global pre-tenant indexes from existing silo rows: hash-only credential locators and agent-certificate revocation metadata. This lets a bearer hash choose the silo before any detailed row is read. Session identity/MFA/preferences, token labels/users, and full certificate issuance history remain only in the tenant schema. The backfill is idempotent and conflict-safe: it never redirects an existing hash to another tenant.

Configuration

Everything else rides existing keys (PROBECTL_FLOWSTORE_*, and PROBECTL_FLOW_RETENTION_DAYS applies to per-tenant databases too). Siloed and hybrid provisioning requires a license granting siloed_isolation; without it those models are refused (pooled always works). See configuration.md.

Tests

Unit tests cover the planner/catch-up/teardown DDL recipes, naming, drift diffing, router fail-closed semantics, flow-store routing (per-target inserts, pinned planes, malformed-name refusal), topic naming, and the provider-API lifecycle (license gating, residency validation, non-destructive offboard, and pooled↔siloed handler parity). The headline integration test (live Postgres) is TestSiloedPhysicalSeparation, which asserts: schema creation; physical separation (a siloed tenant's rows exist only in its schema — zero in public, and vice versa); pooled↔siloed parity of the same tenant-scoped operation; in-silo RLS defense-in-depth; router correctness; catch-up after a simulated later migration; and the lower-level, idempotent teardown primitive. Shipping Offboard never calls that primitive. TestPreTenantCredentialsRouteIntoSilos adds pooled-A/silo-B coverage for sessions (including concurrent authenticated replacement), MCP, SCIM, OTLP, agent enrollment, hot revocation, certificate deny-list refresh, and restored silo locator/revocation catch-up. TestSiloReaderLaneEnforcesTenantIsolationAtTheDB (live Postgres and ClickHouse; it also runs in the cross-tenant isolation gate) proves the per-tenant reader scope (DPR-323): two siloed tenants, one on a residency plane; a predicate-free read as the reader under one tenant's setting against the other's database counts zero rows, a misrouted read through the store returns nothing, a table added later is refused to the reader until scoped, and teardown revokes every grant. With the row policy skipped it goes red.

Browser sign-in during quarantine

A quarantined tenant cannot start sign-in. Browser requests accepting HTML at /auth/login receive HTTP 503 with an accessible, script-free explanation, the tenant_quarantined code, and Retry sign-in / Return to console links. The links retain only a valid tenant UUID routing hint; retry never releases quarantine or bypasses SSO. The deployment operator must repair the failed precondition first. Operator-only diagnostic details are not shown. API clients and all /v1 routes retain the structured JSON error and HTTP 503.