Development

This is the orientation page for building, testing, and contributing to probectl from source. It tells you what toolchain to install, how the repo is laid out as a Go workspace, which make target does what, and which CI gates your change has to pass. If you just want to deploy probectl, see install.md instead.

Toolchain

Install these once. Exact pinned versions for the tools probectl installs for you (golangci-lint, buf, the protobuf plugins) live at the top of the Makefile and are fetched by make tools / make proto-tools — you do not pin them yourself.

The Go version is pinned to the exact patch (1.26.9), not a loose 1.26. That is intentional — it keeps govulncheck's standard-library scan honest and keeps the FIPS build working under GOTOOLCHAIN=local. Details: build/toolchain.md.

Two terms from that table that recur on this page: a linter is a program that reads source code and flags suspicious-but-compiling constructs — the compiler answers "does it build?", the linter answers "will it bite?" — golangci-lint being Go's standard meta-linter (it runs many individual linters as one). Protobuf is the typed binary wire format probectl's agents and bus speak (schemas live in proto/), and buf is the tool that lints those schemas and drives the Go code generation from them.

Go workspace & modules

A Go module is a versioned unit of Go code with its own dependency manifest (go.mod); a workspace (the go.work file) tells the Go tool to build several modules side-by-side from local source, as if they were one project. Picture two workshops sharing one driveway: tools and work-in-progress roll freely between them while each keeps its own inventory list — and when a product ships, only that workshop's own list counts. The repo is a workspace tying together two modules:

• . — the primary module github.com/ctlplne/probectl (cmd/, internal/, pkg/). Production code and unit tests live here.
• ./test — the black-box integration harness. Kept as a separate module on purpose: its heavy, test-only dependencies stay out of the main module's go.mod/go.sum, so production builds never pull them. These tests talk to services over the wire, not through internal/.

make commands that span modules iterate both (the GO_MODULE_DIRS loop in the Makefile). Production builds use each module's own go.mod; the workspace is a local/CI convenience.

Make targets

Run make help for the authoritative, self-documenting list. The ones you'll reach for most:

The central browser API client and generated TypeScript SDK buffer JSON only through web/src/api/response.ts: successful responses are capped at 32 MiB and error envelopes at 1 MiB, with the stream canceled on the first byte past the limit. Branding and the API-docs JSON/executor use the same reader. Intentional artifact downloads remain streaming and do not use this helper.

make ci runs the core gates fast and locally. It is not the full CI suite — the integration, isolation-against-real-DBs, eBPF-kernel-matrix, coverage, and supply-chain gates run in GitHub Actions (next section). Most per-surface gates have an identically-named make target you can run yourself: editions-gate, fips-gate, openapi-gate, completeness-gate, sdk-gate, migration-gate, helm-gate, gitops-gate, terraform-gate, cover-gate, interop-offline, perf-smoke, e2e, journey-e2e, test-integration-isolated, evidence-receipt-gate, release-claim-gate (the last four are release/nightly or make lint bundles run locally, not per-PR CI jobs).

CI jobs (.github/workflows/ci.yml)

The CI job names are a contract — every pull request runs them, and they are how a change earns its way to main. A gate is a job that must conclude green before the change may proceed: each one defends a specific invariant, so the table below doubles as a list of the things probectl refuses to break. This is the full list; ci.yml is the source of truth.

Frontend design loop (no backend required)

cd web && npm run dev:fixtures starts the Vite dev server with a dev-only middleware that answers /v1 and /branding from the same read-only fixture catalog the unit suite renders (web/src/test/fixtureApi.ts) — hot reload for design/UI iteration with no control plane, database, or IdP on the laptop. Open http://localhost:5173/ui/ and every screen renders signed in as the obviously-fake fixture operator; edits to components, tokens, or the fixture catalog itself reload live. Every response carries x-probectl-fixture: 1. Flow fixture responses honor the workspace grouping, filters, and limits, so contributor pivots exercise the same request/response behavior as the real API.

This changes the laptop, not the product: the plugin (web/dev/) is apply: 'serve' so a production build never evaluates it, it is inert without PROBECTL_WEB_FIXTURES=1, and the shipped SPA still hard-requires the real backend session (plain npm run dev keeps proxying /v1 to localhost:8080).

npm run dev:fixtures:cold serves the same loop on the install-day profile: the data endpoints answer as a freshly deployed control plane before any agent enrolls (empty lists, truthful producer flags). Use it to design and regression-check the first thirty minutes; the cold states are pinned by src/test/first-touch-cold.test.tsx, and cold empty surfaces offer the isolated sample tour as a secondary next step. It is the interactive sibling of the CI review proxies (scripts/web_dashboard_fixture.mjs, scripts/web_rollout_fixture.mjs). For an interactive login against a real control plane instead, use the Dex demo overlay in ../deploy/compose/README.md.

For the rendered Chromium accessibility/performance gate, cd web && npm run a11y:browser uses a local Chrome/Chromium binary or the Playwright browser cache. On a fresh machine, use make web-rendered-a11y from the repo root instead; it runs the same digest-pinned Playwright container as CI's web-rendered-a11y job and does not require a local browser install. PROBECTL_PLAYWRIGHT_IMAGE overrides the image for mirrored or air-gapped environments. PROBECTL_A11Y_THEMES overrides the theme matrix (default light,dark, which is the whole shipped matrix; e.g. dark alone to narrow a local run). The container wrapper validates and forwards the same value to the browser process; unknown or empty theme names fail closed before Docker starts.

Trivy is vulnerability-only here, by design. Secret scanning is the separate secret-scan job (gitleaks), which owns the .gitleaks.toml allow-list for the deliberate fake secrets inside redaction tests. Trivy can't see that allow-list, so pointing it at secrets would re-flag those intentional fixtures — hence scanners: vuln in both Trivy jobs.

Two more workflows run outside the pull-request loop: nightly.yml (the black-box full-stack e2e via make e2e, ingest benchmarks, and the M-profile scale-gate regression guard — too slow to run per-PR) and security-scan.yml (weekly govulncheck / npm audit / Trivy with retained evidence artifacts; Trivy fails on High/Critical findings, so a newly-disclosed serious CVE in an unchanged pin still goes red on its own).

CI itself cannot make merges blocking — that is branch protection, a GitHub repository setting documented in ops/branch-protection.md. Independently of it, the release pipeline refuses to publish any tag whose CI run isn't green (see releasing.md).

Testing layers

probectl tests in layers, fast-to-slow, each catching a different class of bug — sieves of decreasing mesh, where each pass is slower and catches what the previous mesh let through. The vocabulary: a unit test runs a function in-process with no real services — fast, deterministic, repeatable anywhere. An integration test talks to the real thing (a live Postgres, a real Kafka broker) over the wire, catching the bugs that only exist where two systems meet. The race detector (-race) instruments the test binary so that two goroutines touching the same memory without synchronization fail the run on the spot, instead of corrupting state in production some Tuesday. And a build tag (-tags=integration, -tags=isolation) is a compile-time label on a file: tagged tests exist only when the tag is passed, which is how the slow, service-needing suites stay out of the default fast path.

Local test DSNs (documented dev-only plaintext)

Integration and isolation tests fall back to sslmode=disable DSNs (data source names — database connection strings) only when PROBECTL_DATABASE_URL is unset — the local-dev convenience path against the test/ compose stack. CI never uses those fallbacks: every DB-backed job starts Postgres with TLS under a per-run test CA and connects sslmode=verify-full (scripts/ci_pg_tls.sh), the production posture. The shipped deploy recipes are sslmode=require or stricter.

• Unit (make test, -race) — hermetic (no real services, everything in-process), table-driven (one test body run over a table of cases); the default fast path.
• Integration (make test-integration, -tags=integration) — against real Kafka (in-process kfake), Postgres, ClickHouse, and Prometheus, plus in-process HTTPS/DNS servers and loopback sockets for the probes. The DNS/HTTP/TLS canary behavior (success / 5xx / slow / expired-cert / DNSSEC-bogus) lives here. Real-stack receipts with a rendered-UI leg render the live control plane in Chromium (internal/testsupport.RenderUI), so they need the real web bundle embedded — npm --prefix web run build && cp -R web/dist/. internal/webui/dist/ before the run (CI does this) — and browser-worker's Playwright with a Chromium (npx playwright install chromium there, or point PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH at a local one). The SSO receipt also needs a real IdP: scripts/ci_dex.sh <dir> starts Dex on https://localhost:5556/dex under a throwaway CA and writes its PROBECTL_TEST_DEX_* settings to <dir>/env (set -a; . <dir>/env; set +a). Receipts that must exercise the production wiring itself (the bus lane supervisor, the silo router and provisioner, the OTLP listeners, the least-privilege serve login) boot the shipped binaries with internal/testsupport/shipped: it builds probectl-control, with a throwaway license key baked in the way a release bakes the real one, and probectl-agent; migrates a fresh database with the privileged login and serves it with a least-privilege one; and needs the same IdP, Chromium and web bundle plus the dev stack's Kafka, ClickHouse and Prometheus (PROBECTL_TEST_KAFKA, PROBECTL_TEST_CLICKHOUSE_URL, PROBECTL_PROM_URL).
• Fuzz (make fuzz-smoke) — fuzzing feeds a parser thousands of mutated, adversarial inputs hunting for the one that crashes it. The Go fuzz targets cover the untrusted-input parsers and tenant-bound ingest invariants. scripts/list_fuzz_targets.sh discovers every Fuzz* target under internal/ and, when present, ee/, so adding a fuzz function automatically puts it in PR smoke and the nightly matrix. The invariant is "never panic", and tenant-sensitive paths must additionally never publish a tenant-less event under fuzzing — the fail-closed tenancy rule (see the non-negotiables). CI runs a short smoke; nightly runs each target in its own long-budget shard.
• Property (Hypothesis, in the analyzer suite) — where fuzzing asks "does it crash?", a property test asserts a stated invariant holds for every generated input: the MRT parser and RPKI validator are checked over thousands of generated inputs (robustness + round-trip + soundness), the Python counterpart to the Go fuzzers.
• Coverage gate (make cover-gate → the coverage CI job) — statement coverage is the fraction of code statements the tests actually executed; a floor fails CI when a package drops below its number. This gate floors the service-free logic / parser / probe packages (scripts/check_coverage.sh). The stateful DB/transport packages are gated for correctness by the integration and cross-tenant-isolation jobs instead — a stronger guarantee than a coverage percentage — so they are not floored here.

Commits

Use Conventional Commits (e.g. feat(canary): add ICMP network test) — a machine-parseable subject format, type(scope): summary, which is what lets release notes be generated from history instead of written by hand — and sign off every commit (git commit -s, which adds the Signed-off-by trailer the dco CI job requires; the DCO, Developer Certificate of Origin, is your recorded assertion that you have the right to contribute the change). See ../CONTRIBUTING.md. You can pre-load the message template with:

git config commit.template .gitmessage