SLO + business-impact engine

What this is

An SLO (service-level objective) is a promise about reliability, written as a number: "checkout should succeed 99% of the time." The thing you actually measure to check the promise — the ratio of good probes to total probes — is the SLI (service-level indicator). This engine turns that promise into something probectl watches continuously, and translates it into the language an executive cares about: are we keeping the promise, how much room do we have left before we break it, and is something burning that room down right now?

It sits in the control plane (internal/slo). It reads the same synthetic probe results probectl already collects, evaluates each SLO per tenant, and raises an alert into the incident pipeline when reliability is being spent too fast. Definitions are written in OpenSLO (an open, vendor-neutral SLO format), so they move freely between probectl and any other OpenSLO tooling.

Four pieces fit together:

• SLI/SLO definitions — what you measure (the good/total ratio) and the target you hold it to.
• Error budgets — the small amount of failure the target allows, treated as a spendable balance.
• Multi-window, multi-burn-rate alerts — the Google SRE method (SRE — site reliability engineering, Google's discipline for running services) for paging on real problems without paging on noise.
• Service/team mapping — every SLO carries a business owner, so reliability rolls up to a team, not just a probe target.

OpenSLO conformance — conform, do not diverge

A definition is a standard OpenSLO v1 document (apiVersion: openslo/v1, kind: SLO). probectl evaluates a deliberate subset of OpenSLO, and is strict about it:

• ratioMetric indicators (a good count over a total count) where metricSource.type is probectl
• budgetingMethod: Occurrences (count successes vs. attempts — not a time-slice method)
• exactly one rolling timeWindow (30d, 7d, 12h, … — rolling means the window always ends now and slides forward, never a fixed calendar month)
• exactly one objective target, a ratio strictly between 0 and 1

Anything outside that subset is rejected loudly when the file loads — the YAML parser runs in strict-fields mode, so an unknown or unsupported field stops startup rather than being silently ignored. Export emits the original document back unchanged (a lossless round-trip, enforced by a test). The reasoning: an SLO you think probectl is tracking must actually be tracked. Silently dropping one is worse than refusing to boot.

apiVersion: openslo/v1
kind: SLO
metadata:
  name: checkout-availability
  displayName: Checkout availability
  labels:
    team: payments            # the business-unit mapping (showback)
    tenant: 88929fbe-28e5-4f0a-898e-6c4302c55e57   # the tenant this SLO belongs to (id)
spec:
  service: checkout
  indicator:
    metadata: { name: checkout-probe-success }
    spec:
      ratioMetric:
        good:
          metricSource:
            type: probectl
            spec: { canary_type: http, target: checkout.acme.example, outcome: success }
        total:
          metricSource:
            type: probectl
            spec: { canary_type: http, target: checkout.acme.example }
  timeWindow: [{ duration: 30d, isRolling: true }]
  budgetingMethod: Occurrences
  objectives: [{ target: 0.99 }]

A few matching rules worth knowing:

• target (the probe target) accepts a trailing * as a prefix wildcard (api.* matches api.acme.example, api-internal.example, …).
• metadata.labels.tenant binds the definition to one tenant (its id): only that tenant's results feed it and only that tenant lists or exports it. In the multi-tenant and regulated profiles every definition must carry it — loading refuses one that does not — because a deployment-wide definition would feed on every tenant's results and appear, with its service, team, target and test id, in every tenant's /v1/slos (DPR-068). Single-tenant deployments may omit it.
• canary_type left empty matches any probe type.
• target is the probe's server.address — for HTTP canaries that is the host (checkout.acme.example), not the URL, so two HTTP definitions against one host (/ and /checkout) are indistinguishable by target. Name the server test definition instead: test_id: <id> (the id the test receipt returns; results carry it as probectl.test.id) selects exactly that definition, with or without a target (DPR-066).
• The good metric must declare outcome: success; good and total must share the same target/test_id (at least one non-empty) and the same canary_type — otherwise the ratio would compare two different things.

Definitions load from the directory named by PROBECTL_SLO_DIR (each file may hold multiple YAML documents separated by ---). A malformed file, an invalid duration, two SLOs with the same name, or a definition file larger than 1 MiB fails startup. The size check happens before YAML decoding.

Error budgets + multi-window burn-rate alerts

Start with the intuition. If your target is 99%, you are allowed to fail 1% of the time. That 1% is your error budget — a balance you spend as failures happen. Burn rate is how fast you are spending it:

burn rate = errorRate(window) / (1 − target)

The budget behaves exactly like a monthly mobile-data allowance: burn rate 1 is the pace at which you use precisely your whole allowance by the end of the month — fine, by definition. At burn rate 1 you spend exactly the whole budget over the SLO window and land on empty right at the end — sustainable. At burn rate 14.4, you spend the entire month's budget in about two days — an emergency.

The hard part of alerting is telling a real outage apart from a blip. probectl uses the Google SRE answer: require two windows — a long one and a short one — to both exceed the threshold before it fires (a logical AND). The long window proves the problem is sustained (kills noisy, flappy alerts); the short window proves it is still happening right now (kills slow, stale alerts that fire long after recovery).

(A long window is clamped to the SLO window itself — for a 12h SLO the slow tier's 3-day window evaluates over 12 hours, since no window can be longer than the budget it measures.)

Worked example, against the 99% checkout SLO above (so 1 − target = 0.01). Say over the last hour 14.4% of probes failed: 0.144 / 0.01 = burn rate 14.4. If the last 5 minutes are also failing at ~14.4% or worse, both the fast long and fast short windows clear 14.4, and the engine pages. If the 5-minute window has already recovered, nothing fires — the incident is over.

When a tier first crosses, the engine raises a slo.burn_rate signal (plane slo) into the incident pipeline. Signals are latched per window per episode — latched meaning the signal fires once and then stays armed-off: one signal when a tier starts firing, and it re-arms only after the long window drops back under the threshold. Clearing on the long window (not the short one) is deliberate hysteresis — a gap between the turn-on and turn-off conditions, the way a thermostat that heats to 21° refuses to restart until the room falls below 19° — and it stops a single episode from flapping out a stream of alerts on short-window jitter.

Cold start — an empty baseline is not an outage

A brand-new SLO, or one whose probes barely run, has almost no data. A single failure out of three probes is a 33% error rate, which would trivially trip every burn threshold — a false alarm. So the engine stays quiet until an SLO has seen at least 50 events in its full window, and reports cold_start: true until then.

The threshold is checked against the full SLO window, not against each alert window. That distinction matters: a low-cadence probe (say one every few minutes) might never accumulate 50 events inside the 1-hour fast window, so gating on the alert window would make fast alerts permanently dead. Gating on the full window instead lets slow probes still get fast alerts once they have enough history overall.

Surfaces

• GET /v1/slos (permission metrics.read) — the caller's tenant's SLO statuses: attainment, error budget remaining, total events, the cold-start flag, and per-window burn rates with their firing state. slo_running: false means the engine is not wired in.
• GET /v1/slos/openslo (permission metrics.read) — the loaded definitions as an OpenSLO v1 YAML stream. Definitions are deployment-level configuration; statuses are per tenant, so this endpoint returns the shared definitions and /v1/slos returns the tenant-scoped numbers.
• SLOs page (/slos) — the executive dashboard: attainment vs. objective, an error-budget bar, burn-rate badges, service/team labels, and honest cold-start and not-wired states.
• What-if integration — the topology what-if (a read-only simulation that fails an element in a copy of the graph) reports impacted_slos, the SLOs whose service or probe target sits inside the simulated blast radius, so "what breaks if this link dies?" answers in SLO terms.

%%{init: {'theme':'base','themeVariables':{'background':'#0d1117','primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#3b82f6','lineColor':'#8b949e','secondaryColor':'#21262d','tertiaryColor':'#0d1117','clusterBkg':'#161b22','clusterBorder':'#30363d','fontFamily':'ui-monospace, SFMono-Regular, Menlo, monospace'},'flowchart':{'curve':'basis','nodeSpacing':55,'rankSpacing':55,'padding':12}}}%%
flowchart LR
  R[synthetic results] --> C[SLOConsumer]
  C --> E[slo.Engine<br/>per-tenant minute buckets]
  D[OpenSLO YAML<br/>PROBECTL_SLO_DIR] -->|strict load| E
  E -->|statuses| API[GET /v1/slos]
  E -->|OpenSLO out| EXP[GET /v1/slos/openslo]
  E -->|"slo.burn_rate signals"| INC[incident pipeline]
  E -->|impacted SLOs| WI[what-if simulation]
  API --> UI[SLOs page]

Under the hood the engine keeps per-tenant, per-minute buckets of good/total counts, pruned to the SLO window. Burn over any window is computed by summing the buckets in that window — so the same data answers both the live status and the alert evaluation.

Configuration

On Kubernetes the definitions are a ConfigMap (one key per OpenSLO file) named in control.slo.existingConfigMap; the chart mounts it read-only at control.slo.mountPath (/etc/probectl/slo) and sets PROBECTL_SLO_DIR for you (DPR-064) — nothing else in the chart could carry an operator file to the control plane, so the engine was unreachable on that deploy path.

kubectl -n probectl create configmap probectl-slo --from-file=slos.yaml
helm upgrade probectl deploy/helm/probectl --reuse-values \
  --set control.slo.existingConfigMap=probectl-slo

Out of scope by design: application-level SLOs. probectl correlates the network planes; it does not own application instrumentation.