CI pipeline overview

A deeply-technical-ELI5 map of what happens every time you push — what runs, why each gate exists, and what "green" actually proves. The source of truth is .github/workflows/ci.yml; this doc explains it.

This page describes the contract of a green pipeline. It is not itself a claim that today's branch is green. For buyer-facing or release-facing evidence, use the actual workflow artifacts for the exact commit; after the July 1, 2026 audit harness run, coverage, integration, isolation, and e2e proof remain current only when their named gates have rerun green on that commit.

When it runs

ci.yml triggers on push to main and on every pull request. A concurrency group keyed on the ref cancels any in-progress run when you push again, so a rapid second push abandons the first (no wasted runners). The workflow starts with only contents: read permission — it can read the repo but not write to it, so CI can never push to main.

The repo's other three workflows do not run on a normal push:

The shape: fan-out, then one umbrella

A push is like sending the change through a checkpoint with 42 specialist inspectors, each examining one thing, all in parallel. A 43rd job — verify-all — is the supervisor: it needs: 39 of them, runs with if: always(), writes a receipt artifact, and fails loudly listing any non-green gate. It is the one status you actually watch: green verify-all = the whole pipeline passed.

Three jobs sit outside the roll-up and report as their own checks: commitlint and dco run only on pull requests (on a push to main they'd register as "skipped" and falsely redden the umbrella), and coverage-comment is a write-capable best-effort PR reporting job. If you turn on branch protection, docs/ops/branch-protection.md documents the exact required-check set.

The guiding principle, visible throughout the repo: every rule in the Non-negotiables has a matching CI gate, and the gates execute the thing rather than just inspecting it — they boot real kernels, real databases, real Kafka, and drill real failovers. Verification you ran, not verification you described.

The jobs, by stage

1. Cheap gatekeeping — "is this change even well-formed?"

• action-pins — every GitHub Action is pinned to a commit SHA, not a moving tag, so a hijacked upstream action can't slip in; a second pass (scripts/check_supply_pins.sh) rejects :latest image tags and unpinned tool installs, and requires literal digests for scalar/nested job containers, service images, and matrix image values in the workflows (supply-chain).
• secret-scan — gitleaks, a scanner that pattern-matches every reachable git commit for anything shaped like a credential (tokens, private keys, key=value passwords): nothing secret-shaped can be committed and then hidden by deleting it later. The deliberately fake secrets inside redaction tests and the exact historical OIDC fixture are allowlisted in .gitleaks.toml.
• commitlint — commit messages follow Conventional Commits (feat(...), fix(ci): ...).
• dco — every commit carries a Signed-off-by (the Developer Certificate of Origin; what git commit -s adds).
• no-devauth-in-release — proves the dev-auth bypass is physically absent from a release binary (symbol + literal absence + a boot-time refusal of PROBECTL_AUTH_MODE=dev), and compiles a tagged build to confirm the check isn't vacuous.

2. Static checks — compile, lint, spec; no services yet

• lint-go / lint-python — gofmt + go vet + golangci-lint on Go, plus the repo's guard scripts (crypto primitives only via internal/crypto, no swallowed errors, hardened HTTP clients, no string-built SQL, unified TLS config); ruff/black on the BGP analyzer.
• editions-gate — enforces the editions boundary via make editions-gate: scripts/check_editions_imports.sh (self-tested) proves core never imports ee/, then the core-only build (-tags probectl_core) compiles and passes its tests with the ee/ tree linked out entirely.
• fips-gate — the FIPS artifact (-tags probectl_fips, validated Go Cryptographic Module via GOFIPS140) builds, and its power-on self-test passes with the validated module active — the build is real, not just tagged.
• proto — buf lint, then buf breaking against main (the schemas are the agent/bus wire contract, so incompatible changes block the merge — see CONTRIBUTING.md), then regenerates and asserts the committed *.pb.go match the .proto files (no drift, no codegen poisoning).
• openapi-gate — every registered core /v1 and provider /provider/v1 route exactly matches its OpenAPI 3.1 spec, and every operation has a named CLI path or a reasoned none-by-design exception; no undocumented route or silent provider-CLI gap ships.
• completeness-gate — validates that every capability in the release catalog has concrete engine, binary, API, CLI, UI, docs, configuration, telemetry, real-stack, and migration evidence (or an explicit reasoned none_by_design cell, or an explicit not-done gap on a partial-evidence row). Gaps remain outside the reported spine-coverage count. Its planted negative self-test proves a missing cell turns the same production validator red, and CI retains deterministic HTML and JSON ledgers. See the capability completeness contract.
• delivery-audit-gate — exercises the signed delivery-receipt policy, exact-source/trusted-signer promotion rules, repository-only auditor boundary, and a planted fixture-only negative. The Docker-backed human-path run remains the explicit make delivery-audit workflow; this fast CI job proves its acceptance policy cannot silently accept test-only evidence.
• docs-claims gate — scripts/check_docs_claims.sh SELFTEST && scripts/check_docs_claims.sh (also make docs-claims-gate) enforces the claim register: every capability claim in a governed surface is declared in docs/claims/register.json and bound to the code path that implements it and the gate/test that proves it (scripts/claims_register_check.py, both directions — an undeclared claim-shaped line fails, and a stale code/proof/surface binding fails), plus the migrated DOCS-S honest-claim properties. SELFTEST first proves the good fixture passes, then runs one negative fixture per DOCS-S label AND per REG-* register failure shape, so the gate cannot go vacuously green in either direction.
• migration-gate — DB migrations are additive / expand-only: it rejects destructive or blocking changes (drop column, column-type change, rename, adding NOT NULL) so release N's schema still works with N-1's code during a rolling upgrade.

3. Tests + coverage

• test-go — the Go unit suite across every workspace module, run with -race (Go's race detector: it catches two goroutines touching the same memory unsynchronized — the class of bug that stays invisible until production load), plus a fuzz smoke over the untrusted-input parsers (fuzzing = feeding a parser machine-mutated random inputs to find the crashes no human would think to type). The fuzz smoke discovers every Go Fuzz* target under internal/, and make fuzz-policy proves the scheduled nightly job is matrix-sharded so each target gets its own long budget instead of all targets sharing one timeout. This job also runs the backup-encryption/erasure gate, an agent-overhead bench smoke, and cross-compile checks (linux amd64+arm64; the endpoint agent additionally builds for macOS and Windows).
• test-python — the BGP analyzer's tests, installed from a hash-locked requirements file (which is itself checked against pyproject.toml for drift), with an 85% coverage floor.
• coverage — per-package statement-coverage floors on the service-free logic/parser/probe packages (scripts/check_coverage.sh); retains coverage.out + a summary (the per-function tail of go tool cover -func) as a downloadable receipt artifact (see docs/frontend-coverage.md).
• web — the frontend: typecheck, lint, production build, surface coverage, jsdom axe, theme-token, and no-hardcoded-token gates.
• web-rendered-a11y — a real Chromium accessibility and interaction-performance gate in the pinned Playwright container. It renders every native route under light and dark at desktop and mobile reference widths, checks browser-computed contrast, keyboard focus, focus-obscured, and target size, then records five fresh-context LCP/INP runs for every J1–J6 landing route. CI recomputes p75 and retains the route/theme/viewport and raw-sample JSON receipts.
• browser-worker — the Playwright browser-synthetic worker, run inside the official Playwright image (a real scripted login, timings + failure screenshot).

4. eBPF — building and loading real BPF objects

• ebpf-kernel-matrix — boots real LTS kernels (5.15 and 6.6) in QEMU — a machine emulator that runs a complete virtual machine, real kernel and all, inside the CI runner (driven by vimto) — and actually loads + attaches the BPF programs (tracepoint + uprobes, one flush cycle). Static analysis cannot prove a BPF program loads; only a kernel's verifier can. One matrix entry raises kernel lockdown to INTEGRITY inside the VM and proves load+attach still works on a hardened kernel. The matrix is x86_64 only and runs under KVM (hardware-accelerated virtualization) on the hosted Linux runners; missing /dev/kvm is a hard CI error, not a green skip. arm64 eBPF is compile-verified, not live-verified (decision D-13): vimto does not emulate cross-arch, so the arm64 row needed a self-hosted Linux/ARM64 runner with a kvm label. Without one the job never left queued, and a job that never starts means the RUN never reaches completed — which is what release.yml polls, so every tag timed out with all other jobs green. See docs/ebpf-agent.md for what arm64 does and does not prove, and how to restore the row.
• ebpf-image-live — the shipped probectl-ebpf-agent image must carry the live CO-RE loader (CO-RE = Compile Once – Run Everywhere: the BPF object is relocated at load time to fit the running kernel's struct layouts), built from Dockerfile.ebpf and asserting -tags=ebpf — not the fixture replayer.

5. Real-infrastructure suites — spin up actual Postgres / Kafka / ClickHouse

• cross-tenant-isolation — the tenant-isolation non-negotiable, executed (make test-isolation): row-level-security posture (RLS is the Postgres feature where the database itself filters every query by tenant, so even buggy handler code cannot return another tenant's rows) + cross-tenant injection against a real Postgres (over TLS, sslmode=verify-full — the client checks the server's certificate and hostname, like production) and a real ClickHouse (including its row-policy DDL).
• integration — boots the control plane against real Postgres (TLS, verify-full) plus a remote-write Prometheus; migrations are idempotent, /readyz passes, the result pipeline round-trips — and internal/store must hold its 60% integration-coverage floor. It builds web/ and embeds the real bundle (as the release does) and installs browser-worker's pinned Chromium, because real-stack receipts render the live control plane's UI (internal/testsupport.RenderUI); the embedded placeholder or a missing browser fails the job rather than skipping a rendered leg. It also starts Dex, a real OIDC IdP, under a per-run test CA (scripts/ci_dex.sh), so the SSO receipt signs people in through a real login.
• perf-smoke — a cheap, repeatable latency/throughput baseline; the first place a pooled-cardinality or RLS-cost regression would surface.
• backup-drill — backup → wipe → restore actually runs; nonce-marked rows must survive a full database drop.
• failover-drill — kills the primary, promotes the streaming standby, and times RTO (first write on the new primary) / RPO (acked rows lost).
• load-smoke — synthetic agents → real Kafka → the production consumer (retry/DLQ — a dead-letter queue, where a message that repeatedly fails processing is parked for inspection instead of retried forever or silently dropped — plus cardinality caps) → Prometheus remote-write → tenant-scoped PromQL, end to end.

6. AI quality

• rca-eval — a blocking evaluation of the AI root-cause analysis against fixtures, with quality floors (0.85 / 0.85). If RCA quality regresses, the build fails — the AI is held to a measured bar, not just "it compiled."

7. Deploy / packaging config

• helm-gate — the Helm charts lint for every reference profile and uphold the secure-by-default invariants (non-root/read-only pod, NetworkPolicy/PDB/HPA, HTTPS Service/probes/ingress backend, required serving certificate Secret, HSTS, and no default credentials); the manifests are schema-validated with kubeconform, the GitOps (ArgoCD/Flux) manifests are checked, and the shipped compose file must pass docker compose config.
• terraform-gate — the Terraform module is fmt-clean and terraform validates via an example root that consumes it.

8. Supply chain & artifacts

• dependency-scan — vulnerability scan of the dependency graph.
• build-images — builds the multi-arch container images.
• image-scan — scans those built images for known vulnerabilities.
• sbom — emits a software bill of materials for the build.

9. The umbrella

• verify-all — depends on the gates above, writes verify-all-summary.json as a receipt, and goes red listing any gate that wasn't green. This is the single signal of overall pipeline health.

What a green pipeline proves

When verify-all is green for a commit, you've demonstrated — on that exact commit — that the code:

• compiles and passes lint in every mode (normal, FIPS, eBPF);
• passes unit, real-Postgres, real-ClickHouse, and real-Kafka tests;
• keeps tenants isolated at the storage layer (RLS, enforced and tested);
• ships container images that scan clean, with an SBOM and SHA-pinned actions;
• didn't drift its API spec, protobufs, or migrations, and keeps zero-downtime upgrade compatibility;
• holds the AI's root-cause quality above its floor;
• survives a backup/restore and a primary failover.

That is a heavier pipeline than most projects run — closer to what a regulated enterprise expects — which matches where probectl is aimed (sovereign, multi-tenant, source-available).

What the pipeline cannot self-apply

Branch-protection enforcement (making GitHub refuse to merge a red PR) is a server-side repository setting, not something a workflow in the tree can turn on. Until an admin configures it, every check above is advisory at the merge button — the recommended ruleset (require verify-all plus the two additional required checks outside its roll-up, commitlint and dco) and the one-time console steps are in docs/ops/branch-protection.md. The release path has its own independent backstop either way: release.yml refuses to publish a tag whose commit lacks a green ci run or whose strict capability ledger is below 100% spine coverage.