A ctlplne studio product

Getting started: from zero to first real data

Most "install" guides stop the moment the control plane answers /readyz. That feels like success — but a control plane on its own observes nothing. This guide takes you the rest of the way: to actual data on the screen.

The one idea that explains everything: consumer vs. producer

probectl has two kinds of moving part, and almost every "why do I see nothing?" question comes from blurring them:

  • The control plane is a consumer. It receives data, stores it, correlates it, and serves the API/UI. It produces no observations of its own. (Its tagline — see everything, send nothing — is about telemetry never leaving your network, not about the box magically seeing traffic.)
  • Agents and collectors are the producers. They are what actually watch the network: a synthetic probe sends traffic and times the answer; an eBPF agent reads the kernel's view of real connections; a flow collector receives NetFlow from your routers. Each one ships what it sees to the control plane.

No producers = no data. A freshly installed control plane is a perfectly healthy database with nothing writing to it. /readyz is green and the dashboards are empty — that is expected, not a bug. The fix is always the same: attach a producer.

This guide brings up one control plane, then walks three independent ways to put first data behind it, and finally combines them into the payoff probectl is built for — a single correlated incident drawn from more than one plane.

%%{init: {'theme':'base','themeVariables':{'background':'#0d1117','primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#3b82f6','lineColor':'#8b949e','secondaryColor':'#21262d','tertiaryColor':'#0d1117','fontFamily':'ui-monospace, SFMono-Regular, Menlo, monospace'},'flowchart':{'curve':'basis','nodeSpacing':50,'rankSpacing':45,'padding':12}}}%%
flowchart LR
  subgraph P["producers (what you add in this guide)"]
    A["canary agent<br/>(synthetic probes)"]
    E["eBPF host agent<br/>(real flows)"]
  end
  A -- "gRPC / mTLS" --> CP["control plane<br/>(consumer)"]
  E -- "bus (Kafka)" --> CP
  CP --> UI["API / UI<br/>(data on the screen)"]

One command, if you just want to see it work

make first-data

That drives the evaluation stack plus its synthetic overlay end to end: Postgres, Kafka, the control plane, the agent CA, an enrolled canary, and a probe whose result travels the same path production uses — agent, mutual TLS over gRPC, the bus, the consumer, the tenant-scoped API. It waits on readiness rather than a clock, prints the first measurement it receives, and names the commands for reading more. It generates the one secret this path needs into deploy/compose/.env.eval and reuses it on later runs.

It is evaluation only, for the reasons the next section explains, and as written it ends at the API rather than the web UI: dev auth is refused on anything but a loopback bind, so nothing on this stack is reachable from a browser.

To evaluate the interface, add one overlay — deploy/compose/eval-sso.yml. It turns dev auth off and puts the same demo identity provider in front, so the stack runs real session auth; the loopback rule governs dev auth alone, so with it off the UI is simply reachable:

export DEX_CLIENT_SECRET="$(openssl rand -hex 32)"
export DEX_DEMO_PASSWORD_HASH="$(htpasswd -bnBC 12 demo 'choose-a-password' | cut -d: -f2)"
export PROBECTL_SESSION_HMAC_KEY="$(openssl rand -hex 32)"
docker compose -f deploy/compose/eval.yml -f deploy/compose/eval-sso.yml up --build -d
open https://localhost:8443/ui/     # sign in as demo@probectl.local

That is real OIDC, real sessions and real RBAC on a stack whose bus and certificate are still the evaluation ones. For the full production shape, use the production-shaped stack (install.md plus deploy/compose/dex-demo.yml).

The rest of this section is the same path by hand, which is worth reading once.

The fastest way to first data: the evaluation stack

If you just want to watch probectl collect data with one command and no Go toolchain, use the evaluation stack (deploy/compose/eval.yml). It is a local-only, clearly-fenced build that removes the three things stopping the shipped all-in-one from showing data on a laptop (it publishes no agent port, ships no evaluation auth, and expects an external identity provider). Everything here runs entirely inside Docker.

Eval-only, on purpose. The eval stack runs the API in dev auth — every request is an unauthenticated, all-powerful admin — so the control plane binds loopback only and is never published to your network (loopback is 127.0.0.1, the interface whose packets never leave their host — an API bound there is physically unreachable from outside); you read it through an in-container helper. It also uses a plaintext bus and a self-signed certificate. Perfect for a laptop, never for production. The production-shaped stack is deploy/compose/probectl.yml (see install.md).

See first data — no enrollment, any OS

This brings up Postgres + Kafka + the control plane, plus an eBPF agent in fixture mode that replays a recorded SAMPLE flow file onto the bus. The control plane consumes it and folds the flows into the service-map (topology) graph:

# Builds the local eval images on first run; duration depends on host and cache.
docker compose -f deploy/compose/eval.yml up --build -d

# The viewer waits for control-plane readiness and sample topology data:
docker compose -f deploy/compose/eval.yml --profile tools run --rm --no-deps viewer

The viewer is a tiny curl helper that shares the control plane's network namespace — the only way to reach the loopback-only API. It prints the /v1/topology graph: nodes and flow edges built from the sample connections (e.g. 10.0.1.5 → 10.0.2.9:443). That JSON is real data through the real pipeline — agent → Kafka → consumer → topology — proving the loop end to end. It is clearly sample input, not traffic on your machine. The viewer retries readiness and the sample topology for a bounded window, then fails loudly instead of printing an empty graph as success. If it exhausts that window, watch docker compose -f deploy/compose/eval.yml logs control for tenant isolation posture verified followed by starting probectl-control. The --no-deps flag is intentional: the preceding up command already started the stack, and this prevents Compose from rerunning the one-shot certificate generator underneath the live control plane.

Tear it all down with docker compose -f deploy/compose/eval.yml down -v. If you also layered the synthetic overlay below, name it and its profile too, or down leaves the canary container and its identity volume behind (DPR-138) — and the next run reuses an agent certificate signed by a CA that no longer exists, which surfaces much later as tls: unknown certificate authority:

docker compose -f deploy/compose/eval.yml -f deploy/compose/eval-synthetic.yml \
  --profile synthetic --profile browser-synthetic down -v

Add synthetic probes — the canary, with enrollment

Active probes need an enrolled, mutually-authenticated agent, so they layer on deploy/compose/eval-synthetic.yml, which provisions the agent CA and turns on the control plane's agent gRPC listener. Mint a join token, hand it to the canary (which self-enrolls on boot), then read its results:

docker compose -f deploy/compose/eval.yml -f deploy/compose/eval-synthetic.yml up --build -d

# 1. mint a one-time join token (prints pjt_…):
docker compose -f deploy/compose/eval.yml -f deploy/compose/eval-synthetic.yml \
  exec control /usr/local/bin/app enroll-token \
  -tenant 00000000-0000-0000-0000-000000000001 -name laptop

# 2. hand the token to the canary — it enrolls itself on boot, then probes:
PROBECTL_JOIN_TOKEN=pjt_xxx docker compose -f deploy/compose/eval.yml \
  -f deploy/compose/eval-synthetic.yml --profile synthetic up --build -d canary

# 3. read its synthetic results:
docker compose -f deploy/compose/eval.yml --profile tools run --rm \
  -e URL=/v1/results/latest viewer

The web First-run setup page measures the same end-to-end path. It keeps credential creation outside the readiness count, then waits for four server-observed milestones: authenticated agent connection, fresh producer heartbeat, first tenant-scoped result, and a named first-finding receipt. The default first test is an ICMP check of 127.0.0.1, so the proof is a real local observation and does not depend on outbound Internet access. From a healthy reference Compose stack, this path is budgeted at no more than seven UI interactions and 15 minutes; the deterministic journey test records four UI interactions plus the shell enrollment handoff.

The synthetic overlay is the most intricate part of the eval stack and the least exercised. If a step misbehaves, don't fight it — the from-source walkthrough below is the fully verified canary path and needs none of this overlay.

You already have data — you can jump to where to go next. The rest of this guide builds the same thing by hand from source, which is worth reading if you want to understand every moving part, or to drive the real production-shaped control plane rather than the eval build.

Building it yourself from source (the ground truth under the eval stack)

This matters, because it changes the commands below — so read it once.

The shipped all-in-one (deploy/compose/probectl.yml) is deliberately minimal: it runs only the control plane (HTTPS on 8443) plus a Postgres. It does not publish the agent gRPC port, does not wire the agent-transport TLS files, and runs a release image — which (by design) has no built-in evaluation auth. It is the right thing for a production-shaped control plane, and install.md is its home. But you cannot attach a canary to it, or drive its API without an identity provider, without extra wiring — so it is not the smoothest "see data today" path.

The dev dependency stack (deploy/compose/dev.yml) is the opposite shape: it brings up the backing services a fuller control plane talks to — Postgres + Kafka + ClickHouse + Prometheus — but no control plane (you run that from source against it).

The eval stack above is the one-command shortcut over exactly this shape. To do it by hand — to see every moving part, or to drive the production-shaped control plane rather than the eval build — build the binaries from source once, run the dev dependency stack, and run the control plane from source wired to it. That single control plane serves all three producer paths below.

Everything below assumes one tenant, the built-in default tenant 00000000-0000-0000-0000-000000000001. (A tenant is one isolated customer or organization in a deployment — the outermost boundary every record, agent, and query is scoped by.) In a real multi-tenant or provider deployment you create tenants first (see admin.md); the single-tenant case is just the one-tenant version of the same code.

Shared setup — one control plane for all three paths

1. Build the binaries (Go 1.26+). make build produces every component in ./bin; make build-devauth produces a control plane with evaluation auth compiled in (so you can drive the API without an IdP — see the warning below).

make build           # ./bin/probectl-control, probectl-agent, probectl-ebpf-agent, ...
make build-devauth   # ./bin/probectl-control-devauth  (LOCAL EVALUATION ONLY)

2. Start the backing services (Postgres — the relational store for durable state; Kafka — the message bus; ClickHouse — the high-volume event store; Prometheus — metrics):

make compose-up      # docker compose -f deploy/compose/dev.yml up -d --wait

3. Run the control plane from source, wired to those services. This single environment turns on what every path below needs — the agent gRPC/mTLS listener (gRPC is the HTTP/2-based remote-procedure-call protocol the agents stream over; mTLS is mutual TLS — both ends present certificates, not just the server) for the canary, and the Kafka bus + ClickHouse stores for the bus-based producers:

# Apply the schema once.
PROBECTL_DATABASE_URL='postgres://probectl:probectl@localhost:5432/probectl?sslmode=disable' \
  ./bin/probectl-control migrate

# Create the agent certificate hierarchy once (Path 1 uses it; harmless otherwise).
# The root key prints ONCE — copy it somewhere safe; runtime never needs it again.
PROBECTL_DATABASE_URL='postgres://probectl:probectl@localhost:5432/probectl?sslmode=disable' \
  ./bin/probectl-control agent-ca init

# A self-signed server cert (tls.crt/tls.key/ca.crt). The control plane serves
# HTTPS with it, and Path 1's agent enrollment reuses it. SANs: localhost,127.0.0.1.
./bin/probectl-control gen-cert ./certs

# Export the agent CA's PUBLIC trust bundle (root + intermediate certificates,
# never a key) into that directory. This is the file the gRPC listener checks
# agent certificates against.
PROBECTL_DATABASE_URL='postgres://probectl:probectl@localhost:5432/probectl?sslmode=disable' \
  ./bin/probectl-control agent-ca export ./certs/agent-ca.crt

# Run the control plane. (Run this in its own terminal; it stays in the foreground.)
PROBECTL_AUTH_MODE=dev \
PROBECTL_DEV_AUTH_ACK=i-understand \
PROBECTL_HTTP_ADDR=127.0.0.1:8443 \
PROBECTL_TLS_CERT_FILE=./certs/tls.crt \
PROBECTL_TLS_KEY_FILE=./certs/tls.key \
PROBECTL_DATABASE_URL='postgres://probectl:probectl@localhost:5432/probectl?sslmode=disable' \
PROBECTL_REQUIRE_AT_REST_ENCRYPTION=false \
PROBECTL_ALLOW_KEYLESS_DEV=true \
PROBECTL_BUS_MODE=kafka \
PROBECTL_BUS_BROKERS=localhost:9092 \
PROBECTL_BUS_ALLOW_PLAINTEXT=true \
PROBECTL_FLOWSTORE_MODE=clickhouse \
PROBECTL_FLOWSTORE_URL=http://localhost:8123 \
PROBECTL_AGENT_GRPC_ADDR=127.0.0.1:9443 \
PROBECTL_AGENT_TLS_CERT_FILE=./certs/tls.crt \
PROBECTL_AGENT_TLS_KEY_FILE=./certs/tls.key \
PROBECTL_AGENT_TLS_CA_FILE=./certs/agent-ca.crt \
  ./bin/probectl-control-devauth

This control plane serves HTTPS on loopback (https://127.0.0.1:8443), the agent gRPC/mTLS listener on 127.0.0.1:9443 (Path 1 needs it), and has the Kafka bus and ClickHouse wired (Path 2 needs them). The gRPC listener is all-or-nothing: all four PROBECTL_AGENT_* values must be set, and it reads the cert files at startup — which is exactly why gen-cert and agent-ca export ran first. One process now serves all three producer paths; if you only want Path 2, the agent lines are harmless.

⚠️ PROBECTL_AUTH_MODE=dev is evaluation-only, and it is strict

Dev auth gives every request an all-permissions principal with no authentication. It exists so you can poke the API on a laptop without standing up SSO. Because it is dangerous, the code makes it hard to enable by accident — all three of these are required, or the control plane refuses to start:

  1. the binary must be built with dev auth (make build-devauth — the shipped release image does not contain it, so PROBECTL_AUTH_MODE=dev on the shipped compose just crashes with a clear error);
  2. PROBECTL_DEV_AUTH_ACK=i-understand must be set; and
  3. the listener must bind loopback only (127.0.0.1:...) — never a network interface.

This is why the shared setup binds 127.0.0.1. For anything beyond a laptop, use the real path: PROBECTL_AUTH_MODE=session with an OIDC IdP — OIDC is OpenID Connect, the standard web-login protocol, and the IdP is your identity provider: Okta, Entra ID, Keycloak, … (see install.md and admin.md). The control plane is HTTPS/TLS everywhere by design — in production there is no plaintext or no-auth mode to lean on.

Dev auth and TLS are independent: the listener binds loopback (required for dev auth) and serves HTTPS (from the cert you just generated). Confirm it is up — pass the generated CA so curl trusts the self-signed cert:

curl --cacert ./certs/ca.crt https://127.0.0.1:8443/readyz   # -> {"status":"ready"}

That is your consumer, ready and empty. Now pick a path.

First authenticated production API call

The commands above use dev auth so a laptop can prove the data path without an identity provider. In a production-shaped deployment you should also prove the same API path with session auth (PROBECTL_AUTH_MODE=session) and a tenant-scoped bearer token. Think of the browser session as your interactive login, and the bearer token as a script badge for one already-provisioned user: the token inherits that user's tenant and RBAC permissions.

A brand-new deployment has no administrator yet: every SSO login lands with no roles. Grant the first admin from the control host (idempotent, audited; it creates the user row if the person has not logged in yet):

PROBECTL_DATABASE_URL='postgres://probectl:probectl@localhost:5432/probectl?sslmode=disable' \
  ./bin/probectl-control bootstrap-admin -tenant 00000000-0000-0000-0000-000000000001 -email operator@example.com

After the user has logged in once through OIDC, or has been provisioned by SCIM, mint a one-time-visible control-plane bearer token on the control host. The token uses the same hash-only mcp_tokens store as the MCP server, but it also works for normal /v1 API and CLI calls:

export PROBECTL_DATABASE_URL='postgres://probectl:probectl@localhost:5432/probectl?sslmode=disable'
export TENANT_ID=00000000-0000-0000-0000-000000000001
export USER_EMAIL='operator@example.com'

# Find the user row that the session-mode login or SCIM sync created.
export USER_ID="$(
  psql "$PROBECTL_DATABASE_URL" -At \
    -v tenant="$TENANT_ID" \
    -v email="$USER_EMAIL" \
    -c "SELECT id FROM users WHERE tenant_id = :'tenant'::uuid AND email = :'email' LIMIT 1;"
)"
test -n "$USER_ID"

# Print the bearer secret once. PROBECTL_LOG_LEVEL=error keeps stdout token-only.
PROBECTL_LOG_LEVEL=error \
  ./bin/probectl-control mcp-token \
  -tenant "$TENANT_ID" \
  -user "$USER_ID" \
  -name first-api-smoke > .probectl-api-token

export PROBECTL_API_TOKEN="$(cat .probectl-api-token)"

Now make the first authenticated API read. Authorization proves the user, and X-Probectl-Tenant is a tenant assertion: it must match the token's tenant or the request fails closed.

curl --cacert ./certs/ca.crt \
  -H "Authorization: Bearer $PROBECTL_API_TOKEN" \
  -H "X-Probectl-Tenant: $TENANT_ID" \
  https://127.0.0.1:8443/v1/me

curl --cacert ./certs/ca.crt \
  -H "Authorization: Bearer $PROBECTL_API_TOKEN" \
  -H "X-Probectl-Tenant: $TENANT_ID" \
  https://127.0.0.1:8443/v1/results/latest

The same tenant-scoped read through the CLI, in JSON form:

PROBECTL_API_URL=https://probectl.example.com \
PROBECTL_API_TOKEN="$PROBECTL_API_TOKEN" \
PROBECTL_TENANT="$TENANT_ID" \
  probectl --json result latest

Use the CLI form against a deployment whose TLS certificate is trusted by the host running the CLI. For the self-signed laptop certificate in this guide, the curl --cacert ./certs/ca.crt form is the exact smoke test.

The remaining curl examples keep the production-shaped auth headers visible. If you are only running the local dev auth control plane from the source walkthrough, you may omit those two headers; dev auth synthesizes the admin principal for loopback evaluation only.

Path 1 — Synthetic probes (the canary agent)

Any OS, unprivileged, no Kafka. The canary agent sends traffic on a schedule — an HTTP GET, an ICMP ping, a DNS lookup — and times the answer. It streams results straight to the control plane over gRPC/mTLS, so it needs nothing on the bus: just the control plane reachable. This is the lightest possible "first data."

Why this needs the agent CA (the mTLS part)

The agent connection is mutually authenticated: the control plane verifies the agent, and the agent verifies the control plane. The agent proves who it is with a short-lived certificate — an SVID — whose identity names its tenant (spiffe://probectl/tenant/<t>/agent/<a>). Those certificates are issued by the agent CA you created once in the shared setup (agent-ca init), and agent-ca export wrote that CA's public trust bundle to ./certs/agent-ca.crt — the file the shared setup pointed PROBECTL_AGENT_TLS_CA_FILE at, so the gRPC listener checks every agent certificate against it. The trust wiring is already done; all that's left is to give one agent an identity.

1. Mint a one-time join token

The token, not the agent, names the tenant — that is why -tenant is required. This is an operator CLI that works directly against the control plane's database (hence PROBECTL_DATABASE_URL); pointing PROBECTL_TLS_CERT_FILE at the serving cert additionally makes it print the server-certificate pin:

PROBECTL_DATABASE_URL='postgres://probectl:probectl@localhost:5432/probectl?sslmode=disable' \
PROBECTL_TLS_CERT_FILE=./certs/tls.crt \
  ./bin/probectl-control enroll-token \
  -tenant 00000000-0000-0000-0000-000000000001 \
  -name laptop

It prints a pjt_… token once (single-use, expires in ~1h) and — because we pointed it at the server cert — the server certificate pin for first contact. Copy the token.

2. Enroll the agent

enroll generates the agent's private key locally (it never leaves the host), redeems the token over HTTPS, and writes four files into --dir: cert.pem (the SVID), key.pem, ca.pem (the agent CA trust bundle — the same public bundle agent-ca export wrote, delivered to the agent side), and server-ca.pem (the trust it verified the control plane with — here a copy of ./certs/ca.crt — so the agent can verify the server the same way).

./bin/probectl-agent enroll \
  --server https://localhost:8443 \
  --token pjt_REPLACE_WITH_THE_TOKEN_YOU_MINTED \
  --dir ./identity \
  --ca-file ./certs/ca.crt

The --ca-file ./certs/ca.crt tells the agent how to trust the control plane's self-signed certificate on first contact; alternatively pass --ca-pin <hex-sha256> using the pin the mint step printed. No control-plane restart is needed — the gRPC listener has been up since the shared setup, already trusting the agent CA.

Two CAs, two jobs. Two independent trust checks happen on the gRPC handshake. The agent verifies the control plane's server cert against its tls.ca_file (here, the quickstart ca.crt). The control plane verifies the agent's client cert against PROBECTL_AGENT_TLS_CA_FILE (the agent CA bundle from agent-ca export). They are different CAs doing different jobs. In a real deployment you would mint a proper gRPC server certificate and distribute the agent CA bundle through your config management; on a laptop, reusing the quickstart server cert is the shortest honest path.

3. Write a minimal agent config

Point the agent at the gRPC listener and the identity files, and give it one inline probe — an http check against a public URL. Save as agent.yml:

control_plane:
  grpc_addr: "localhost:9443"      # the control plane's PROBECTL_AGENT_GRPC_ADDR

tls:
  cert_file: ./identity/cert.pem   # the agent's SVID (proves the agent to the server)
  key_file:  ./identity/key.pem
  ca_file:   ./identity/server-ca.pem  # verifies the SERVER to the agent: the trust
                                        # enroll captured (a copy of ./certs/ca.crt), NOT ca.pem
  server_name: "localhost"         # must match a SAN on the server cert

agent:
  capabilities: ["http"]
  heartbeat_interval: 30s

canaries:
  - type: http
    target: "https://example.com/"
    interval: 30s
    timeout: 10s
    params:
      method: "GET"
      expect_status: "2xx,3xx"

The full set of probe types and per-probe parameters is in deploy/agent/probectl-agent.example.yml and configuration.md. Probes run straight from this config — you do not have to create anything server-side first.

Why server-ca.pem and not ca.pem. The gRPC listener presents the control plane's server certificate — here the quickstart cert signed by ./certs/ca.crt — so the agent must verify it with that CA. enroll captured exactly that trust (the --ca-file you passed, or the certificate a --ca-pin matched) into server-ca.pem, which is why the snippet it prints works as-is. ca.pem is the agent CA bundle: it is what the control plane trusts (PROBECTL_AGENT_TLS_CA_FILE) to verify agents, and it verifies the server only in deployments that issue the gRPC server certificate from the agent CA itself. Same handshake, two CAs, two jobs.

4. Run the agent

./bin/probectl-agent -config agent.yml

Within one interval (30s) it connects, runs the http probe, and streams the result.

5. See the result

curl --cacert ./certs/ca.crt \
  -H "Authorization: Bearer $PROBECTL_API_TOKEN" \
  -H "X-Probectl-Tenant: $TENANT_ID" \
  https://127.0.0.1:8443/v1/results/latest

You will see the latest synthetic result per target — your https://example.com/ probe with its DNS / connect / TLS / time-to-first-byte breakdown. In the UI, the same data is on the Targets / synthetic results screen.

Why curl/UI and not the probectl CLI here? The probectl CLI speaks the same /v1 API and supports --json, but it relies on the host trust store for TLS. Against this guide's self-signed HTTPS endpoint, curl --cacert ca.crt is the exact tool. Against a production/session-mode endpoint with a trusted cert, use the CLI form above.

Where is the UI? The control plane serves the web UI itself, embedded in the binary (ARCH-004): open the control plane's base URL — / redirects to /ui/ — behind the same TLS + CSP as the API. Release binaries bundle the built SPA; a from-source build that hasn't run npm --prefix web run build serves a placeholder page that says so (the API at /v1 works regardless).

For the full enrollment lifecycle (rotation, revocation, the bootstrap threat model), see agent/enrollment.md.

Path 2 — Real host flows (the eBPF agent)

Linux with CAP_BPF (the Linux capability that permits loading eBPF programs without root) for live capture; any OS (including macOS) for the recorded-sample path. The eBPF agent loads tiny, sandboxed observation programs into the Linux kernel and watches every real TCP connection the host makes or accepts — with the process and container behind it — and rolls those into a live service map (the directed "who talks to whom" graph). It is strictly observe-only: it never blocks or modifies a packet.

Why this needs the bus (and where the data lands)

Unlike the canary, the eBPF agent does not stream over gRPC. It is a separate process from the control plane, so the two cannot share an in-process channel — they meet on the message bus. The agent publishes flow + service-edge batches to the topic probectl.ebpf.flows; a control-plane consumer drains that topic and folds the edges into the topology graph. That bus is Kafka — the open-source distributed message log: producers append, consumers read at their own pace, and neither has to be up for the other to keep working — which is why the shared setup runs dev.yml and wired the control plane to PROBECTL_BUS_MODE=kafka.

Where eBPF data shows up — read this carefully. The eBPF host agent feeds the topology / service-map plane (GET /v1/topology). The /v1/flows/* analytics views (top talkers, capacity, anomalies) are a different producer — the NetFlow/IPFIX/sFlow flow collector (probectl-flow-agent) or the local cloud-flow import path, which land in ClickHouse and are covered in flow.md. Both are "flow data" in plain English, but they are distinct planes with distinct producers. This path is the eBPF service map.

Option A — recorded sample flows (no kernel; works on a Mac)

The agent ships with a recorded flow fixture (internal/ebpf/testdata/flows.json) so you can light up the service map with clearly-labeled SAMPLE data on any OS, no privileges. Point the agent at that file and at the dev Kafka:

PROBECTL_EBPF_TENANT_ID=00000000-0000-0000-0000-000000000001 \
PROBECTL_EBPF_FIXTURE_PATH=internal/ebpf/testdata/flows.json \
PROBECTL_EBPF_BUS_MODE=kafka \
PROBECTL_EBPF_BUS_BROKERS=localhost:9092 \
PROBECTL_EBPF_BUS_ALLOW_PLAINTEXT=true \
  ./bin/probectl-ebpf-agent

The sample contains a few synthetic connections (10.0.1.5 → 10.0.2.9:443, …→ 10.0.3.3:5432) — not real traffic on your machine. It is exactly enough to see the pipeline work end-to-end.

PROBECTL_EBPF_BUS_ALLOW_PLAINTEXT=true is required only because the dev Kafka broker is plaintext. Like the control plane, the agent refuses a plaintext broker unless you opt in explicitly — production Kafka is TLS (*_BUS_TLS_ENABLED).

Option B — live flows (Linux + CAP_BPF)

The live kernel loader is compiled in only with the -tags ebpf build, which needs a Linux host, a BTF-enabled kernel (BTF — the kernel's embedded type catalog, which is what lets one compiled agent adapt to any kernel), and clang (the C compiler that builds the BPF objects; the shipped probectl-ebpf-agent container image is this live build). Build it and run with a config:

make ebpf-agent      # builds ./bin/probectl-ebpf-agent WITH the live CO-RE loader
# needs CAP_BPF (+CAP_PERFMON); the plaintext-bus opt-in is env-only:
sudo PROBECTL_EBPF_BUS_ALLOW_PLAINTEXT=true ./bin/probectl-ebpf-agent -config ebpf.yml

(CO-RE — compile once, run everywhere — is the eBPF build model where the compiled program carries relocation info and adapts itself to the running kernel at load time, instead of being recompiled per kernel.)

A minimal ebpf.yml for the dev stack (see deploy/agent/probectl-ebpf-agent.example.yml):

tenant_id: "00000000-0000-0000-0000-000000000001"
bus:
  mode: kafka
  brokers: ["localhost:9092"]

(The plaintext-bus escape hatch is not a config-file field — pass it as the PROBECTL_EBPF_BUS_ALLOW_PLAINTEXT=true environment variable, as above. With a real TLS broker you would instead set PROBECTL_EBPF_BUS_TLS_ENABLED=true.)

Either option emits batches every flush_interval (default 10s). The kernel, capabilities, and hardening details are in ebpf-agent.md.

See the service map

curl --cacert ./certs/ca.crt https://127.0.0.1:8443/v1/topology

The response is the dependency graph — nodes for the hosts/services seen and flow edges between them, built from the agent's batches. In the UI this is the Topology view; the same edges power the change-aware what-if analysis.

Path 3 — Combine both (the actual payoff)

Each path on its own is one plane. probectl's reason to exist is cross-plane correlation: when something breaks, you want one incident that pulls evidence from every plane at once, rather than five dashboards you have to mentally join.

The shared-setup control plane already runs the gRPC listener and the Kafka bus — nothing to reconfigure; just run both producers at once:

# terminal 1 — the control plane from the shared setup (gRPC listener + Kafka bus on)
# terminal 2 — the canary
./bin/probectl-agent -config agent.yml
# terminal 3 — the eBPF agent (sample flows, any OS)
PROBECTL_EBPF_TENANT_ID=00000000-0000-0000-0000-000000000001 \
PROBECTL_EBPF_FIXTURE_PATH=internal/ebpf/testdata/flows.json \
PROBECTL_EBPF_BUS_MODE=kafka PROBECTL_EBPF_BUS_BROKERS=localhost:9092 \
PROBECTL_EBPF_BUS_ALLOW_PLAINTEXT=true \
  ./bin/probectl-ebpf-agent

Now you have synthetic results (Path 1) and a service map (Path 2) behind one tenant, on one control plane. Confirm both planes are populated:

curl --cacert ./certs/ca.crt \
  -H "Authorization: Bearer $PROBECTL_API_TOKEN" \
  -H "X-Probectl-Tenant: $TENANT_ID" \
  https://127.0.0.1:8443/v1/results/latest   # synthetic plane

curl --cacert ./certs/ca.crt \
  -H "Authorization: Bearer $PROBECTL_API_TOKEN" \
  -H "X-Probectl-Tenant: $TENANT_ID" \
  https://127.0.0.1:8443/v1/topology          # eBPF service map

See a correlated incident

The control plane continuously groups related signals across planes into a single incident, tenant-scoped. List what it has correlated:

curl --cacert ./certs/ca.crt \
  -H "Authorization: Bearer $PROBECTL_API_TOKEN" \
  -H "X-Probectl-Tenant: $TENANT_ID" \
  https://127.0.0.1:8443/v1/incidents
# then drill into one:
curl --cacert ./certs/ca.crt \
  -H "Authorization: Bearer $PROBECTL_API_TOKEN" \
  -H "X-Probectl-Tenant: $TENANT_ID" \
  https://127.0.0.1:8443/v1/incidents/<id>

When a synthetic probe to a service starts failing and that service is a node in the eBPF map, those signals land in one incident with evidence from both planes — the canary failure and the affected service edge — instead of two unrelated alerts. (The deepest demonstration of this is the cross-plane correlation gate that the test suite runs against an injected multi-plane fault; the same machinery is what you are watching here in miniature.) In the UI this is the Incidents view, and each incident shows its cross-plane evidence and the related change/topology context.

You have data — where to go next

You have proven the loop end to end: producers observe, the control plane consumes and correlates, the API and UI show it. From here:

  • Add more producers. deploying-agents.md is the catalog of every producer and which infrastructure each needs (gRPC-to-control vs. bus). The depth docs per plane:
  • Make it production-shaped. Swap dev auth for OIDC SSO and run behind real TLS — start with install.md (the shipped all-in-one and the Helm chart) and admin.md (tenants, roles, audit, SSO).
  • Tune every knob. Every PROBECTL_* control-plane variable and every agent config key is documented in configuration.md.
Rendered live from github.com/ctlplne/probectl — found a mistake? edit this page.