Capacity Model

This page is the operator math for right-sizing probectl. The data-plane sizing page tells you what a starter stack looks like; this page tells you why: how many tenants and agents a tier is meant to hold, how many telemetry events it should sustain, how fast retention turns rows into disk, and which signal says "split the shard now."

The simple mental model is a water tank. Agents and collectors are faucets, Kafka is the pipe, ClickHouse/Postgres/TSDB are tanks, and retention is the drain. If faucets add water faster than the pipe or drain can handle, the tank rises. Scale-out keeps the tank below the line before it spills.

Capacity By Tier

The tenant and host shapes are the full-scale profiles in internal/perf and docs/scale-gate.md. "Hosts" means tenant-bound enrolled agents or collectors. "Result floor" is the minimum end-to-end synthetic result ingest rate that the scale gate must sustain at full scale. "Event budget" is the day-2 planning budget across high-volume planes, especially flow/eBPF. SLO numbers stay provisional until the scale-fullstack RESULT ROW receipts are recorded.

Do not sell or capacity-plan L/XL/XXL from the provisional numbers alone. The scale-fullstack receipt is the proof that the real Kafka, Prometheus, and ClickHouse path held the target percentiles on reference hardware. Until those rows are committed in docs/scale-gate.md, treat the table as a sizing target, not a guarantee.

Planning Constants

These constants are deliberately conservative. They are not wire schema limits; they are the bytes to reserve before buying disks. Re-measure them from your own retained data once the stack has run for a week, then update the local plan.

If a tenant emits unusually wide labels, many custom metrics, or long event metadata, multiply the relevant constant by 2 until the first measured compression report is available.

Cardinality Memory Worksheet

The cardinality limiter is an in-memory guest list of active metric identities. It protects the TSDB path from an agent or tenant minting infinite new series. The limiter is intentionally per control-plane replica: the ingest hot path does not pause on a shared cross-replica counter. That keeps ingestion simple and available, but capacity planning must reserve the worst case as replicas x cap.

The limiter defaults come from internal/pipeline/cardinality.go:

Use this worksheet for control-plane heap, not disk retention:

worst_case_active_series = replicas x active_tenants x per_tenant_cap
memory_budget = worst_case_active_series x bytes_per_identity

Plan with 512 B/identity until a heap profile says otherwise. That reserve covers the identity string, tenant and agent maps, timestamps, and Go map overhead. It is deliberately separate from the 1,536 B/result retention row because this memory sits in each control-plane process before data reaches the TSDB.

Keep cardinality-index memory below 10% of control-plane RSS in steady state. Scale when cardinality-index memory exceeds 10% of control-plane RSS for 15 minutes or two load waves; page when it exceeds 20%, when drops rise for quiet tenants, or when RSS keeps climbing after the idle TTL sweep window. Adding replicas increases the replicas x cap worst case, so split high-cardinality tenants, move noisy tenants to a silo, lower per-tenant caps, or shorten the idle TTL only after checking that normal churn will not be counted as abuse.

Retention Growth

Use this formula for each stored class:

GiB = rows_per_second * 86,400 * retention_days * bytes_per_row * replicas / 1,073,741,824

Examples with one replica factor:

For replicated stores, multiply by the replica count. For object-store or backup copies, add that copy separately. Retention is the strongest cost lever: cutting flow retention from 90 days to 30 days cuts the high-volume ClickHouse footprint by two-thirds.

Scale-Out Triggers

Scale when a trigger is sustained for 15 minutes or appears in two consecutive load waves. A single spike is a page candidate, not an immediate reshard.

The ELI5 rule: if queues grow, scale the slow consumer; if disks grow, shorten retention or add storage; if tenant query tails grow, add shards or isolate the noisy tenant.

Shard-Split Rules

Split by tenant before splitting by time. Tenant-first splits preserve the outer security boundary and keep deletion/export math simple.

Every split keeps the same rule: tenant scope first, then RBAC. A shard split that makes cross-tenant reads easier is not a capacity fix; it is a security regression.

Receipt Discipline

Capacity rows become verified only when the emitted RESULT ROW lines are copied into docs/scale-gate.md:

make scale-fullstack TIER=L
make scale-fullstack TIER=XL
make scale-fullstack TIER=XXL

Until then, use this page to size an initial deployment, reserve disk, and set alerts, but keep public claims tied to the current receipt state.