Endpoint agent — last-mile / WiFi DEM

What it is

probectl-endpoint is a lightweight, cross-OS (Linux / macOS / Windows) agent that runs on a user's own device and measures the last mile — the final stretch between that device and the wider internet (its Wi-Fi link, the home router, the ISP's access line): the part of the path that probectl's server-side canaries physically cannot see. Your data-center probes can tell you the service is up; they can't tell you that a particular remote employee's Wi-Fi is weak or their home ISP is dropping packets. This agent fills that gap.

It captures four things from where the user actually sits — Wi-Fi link health, the local gateway, the ISP / last-mile path, and browser-session timings — and then does the part that makes it useful: it attributes a slowdown to the closest impaired layer, so an operator can finally answer the hybrid-work question, "is it us, or the user's Wi-Fi / ISP?"

It emits like every other probectl agent — DEM results to the operator's own bus, tenant-tagged, and it never phones home — but it sits in a deliberately lower-trust tier (threat model B9), not as an identity peer of the mTLS canary agents. Unlike those, it has no certificate or SPIFFE identity: the tenant_id it stamps is a local assertion, never authoritative. The control plane re-verifies every endpoint result against the agent registry before anything is cached or stored and refuses to consume the lane at all without that binding (INV-08), so a device claiming another tenant reaches no tenant's storage. This is "Digital Experience Monitoring" (DEM) done on-device and sovereignty-respecting.

%%{init: {'theme':'base','themeVariables':{'background':'#0d1117','primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#3b82f6','lineColor':'#8b949e','secondaryColor':'#21262d','tertiaryColor':'#0d1117','clusterBkg':'#161b22','clusterBorder':'#30363d','fontFamily':'ui-monospace, SFMono-Regular, Menlo, monospace'},'flowchart':{'curve':'basis','nodeSpacing':55,'rankSpacing':55,'padding':12}}}%%
flowchart LR
  subgraph device[User device]
    W[WiFi reader\nnmcli/airport/netsh] --> C[Collector]
    T[traceroute / tracert] --> C
    H[HTTP session\nDNS/connect/TLS/TTFB] --> C
    C --> A[Attribution engine]
    A --> P[Privacy minimize]
  end
  P -->|resultv1.Result, tenant-keyed| B[(probectl.endpoint.results)]
  B --> PIPE[pipeline → TSDB / incidents]
  B --> CH[(tenant-partitioned ClickHouse events)]
  CH --> API[GET /v1/endpoints read-through cache]

What it measures

(RSSI is received signal strength in dBm — decibels relative to one milliwatt; the values are negative and closer to zero is stronger, so −50 is excellent and −80 is weak. RSRP / RSRQ / SINR are the cellular equivalents. RTT is round-trip time. traceroute / tracert is the OS's own path-mapping tool — it elicits a reply from each router along the way, hop by hop. httptrace is Go's hook set that timestamps each phase — DNS, connect, TLS, first byte — of a real HTTP request.)

Every metric is best-effort. A device with no Wi-Fi, or an OS that doesn't expose a given field, degrades that field to "unavailable" rather than reporting a false reading. A Have.* flag travels with each signal so the rest of the system can tell "we didn't measure this" apart from "we measured this and it was zero" — an important distinction when the absence of a value is itself the diagnosis.

Attribution — "is it Wi-Fi / ISP or the network?"

This is the headline feature. The attribution engine (internal/endpoint/attribution.go, pure logic and exhaustively tested) assesses each layer, then blames the closest impaired layer, walking outward from the device:

wifi  →  local (gateway/LAN)  →  isp (access edge)  →  network (beyond)

Why closest-first is the whole point: a weak Wi-Fi link inflates the gateway, ISP, and session numbers that are all measured through that link. If you naively looked at the slow session and blamed "the network," you'd be wrong — the real fault is the user's Wi-Fi, and the downstream numbers are just collateral. It's checking your own tap before calling the water company: when the pressure is low at the tap, every reading downstream of it is low too, and the nearest fault owns the verdict. Walking outward and stopping at the first impaired layer avoids that trap.

The verdicts (endpoint.cause):

• wifi — weak signal (RSSI ≤ −75 dBm, or signal ≤ 35% when only a percentage is available) or noise.
• local — the default gateway is unreachable, lossy, or high-RTT.
• isp — the first public hop (the ISP access edge) is high-RTT or lossy.
• network — the whole local path is healthy but a session is still slow, so the fault is beyond the last mile (the service or wider network — explicitly not the user's Wi-Fi or ISP).
• none — nothing impaired.
• unknown — a slow session, but no path visibility to localize it.

Each verdict carries a confidence (0–1) and a human-readable summary. A named fault domain is emitted only at confidence 0.80 or higher. Below that floor, the result is unknown, while the candidate layer scores and reason remain visible for investigation. ELI5: the engine may say “I see a clue near the gateway,” but it does not say “the gateway did it” until the clue is strong enough. The result also carries endpoint.calibration_status and confidence_floor; the status is provisional-synthetic-corpus until a representative design-partner corpus replaces the synthetic positive, negative, ambiguous, and multi-cause fixtures. The synthetic gate permits no more than 1% high-confidence misattribution, but it is not a field-accuracy claim.

The measurement cutoffs are configurable (thresholds: in config, defaulting to DefaultThresholds — where the −75 dBm and 35% values above come from).

Coverage is part of the verdict (DPR-061). A "no impairment detected" verdict is only as good as the layers it saw. When the local gateway, the ISP edge or the session probes could not be measured — the trace never left the LAN, traceroute is missing, the device is wired — the verdict lists them (attribution.unmeasured, e.g. isp, sessions), its confidence drops to 0.5, the summary reads "no impairment detected in the measured layers; unmeasured: …", and the sample carries the reasons (unavailable, e.g. last_mile: traceroute: exit status 2). The agent logs a warning whenever the set of unavailable signals changes and an info line when every layer is measured again. The last-mile trace targets the host of the first configured target (e.g. a configured https://portal.internal → portal.internal), and a hop that never answered (* * *) is silence, not a 100%-loss ISP edge: the ISP segment is the first answering public hop, or unmeasured.

No probe targets ship by default (no-phone-home, guardrails.md G7-2). Targets are operator-chosen input, never a built-in default — shipping an external anycast host (Cloudflare/Google) would make a freshly installed agent beacon out every interval with no operator intent, which contradicts the sovereign / air-gap posture. An agent with no configured targets still measures local Wi-Fi and gateway health; it simply runs no last-mile trace or session probe until you point it at your own key destinations (targets: in config, or PROBECTL_ENDPOINT_TARGETS).

Privacy — it runs on someone's personal device

Because the agent lives on an end user's device, data minimization is a hard requirement, not a setting. The principle: keep the measurements that diagnose experience (signal, RTT, loss, timings — none of which identify a person) and gate the identifiers.

Two properties make this trustworthy:

• Drop-on-collect. A gated-off field is cleared before the sample is ever mapped, emitted, or logged — so it never leaves the device, not even into a log line. There's also a StrictPrivacy() preset that collects no identifiers at all, for high-governance fleets.
• Disclosure at startup. The agent prints exactly what it collects every time it starts (the disclosure banner) — transparency is part of running software on someone's machine. Tune it with PROBECTL_ENDPOINT_COLLECT_* (see configuration.md).

Cross-OS collection matrix

The per-OS readers are build-tag gated — build tags are Go's compile-time switches: each file carries an OS constraint and compiles only on that OS — (wifi_linux.go, wifi_darwin.go, wifi_windows.go, wifi_unsupported.go, netprobe_unix.go, netprobe_windows.go), each defining newPlatformWiFiCollector / newPlatformLastMileCollector so the package compiles on every OS (the endpoint-cross CI build proves it). The pattern mirrors the eBPF and browser layers: a fully-tested portable core with thin, gated platform edges — the fragile per-OS bit is small, and the output parsers are portable and fixture-tested on every platform.

Result schema → pipeline

A sample maps onto the canonical canary.Result envelope (ToResults), one result per signal, typed endpoint.attribution / .wifi / .gateway / .lastmile / .session. Numeric fields become Metrics (TSDB series); identifiers and labels become Attributes (OTel attributes — kept out of the time-series keyspace, so identifiers can't mint unbounded series: no cardinality blow-up). The attribution result is the headline: its endpoint.cause attribute is the Wi-Fi / ISP / network verdict.

Results are published to probectl.endpoint.results as a resultv1.Result, tenant-keyed, so they flow through the same pipeline → TSDB / incident path as every other canary — the control-plane result consumer drains them on its own consumer group alongside probectl.network.results. A separate durable consumer writes the event-shaped fields (SSID/gateway/session labels and attribution) to ClickHouse. It verifies the bus lane or registered agent-to-tenant binding before either cache or durable persistence, retries a bounded number of times, and sends the original protobuf to probectl.deadletter.results.endpoint if ClickHouse remains unavailable. A store failure therefore cannot silently acknowledge and lose an endpoint event.

Deploy

Ship the single static binary to managed devices via your MDM (mobile-device management — the fleet tool that pushes software and policy to managed laptops; Intune, Jamf). It needs no elevated privileges: it uses the OS's own traceroute / tracert and read-only Wi-Fi queries. Point it at the bus (Kafka in a fleet, or the lightweight in-memory bus for a single-node dev deploy) with a tenant id.

First deployment? The cross-producer journey — control plane, bus, and each agent's enrollment-or-bus path — lives in deploying-agents.md (start at getting-started.md if nothing is running yet). You're done when the device shows up in GET /v1/endpoints with an attribution verdict — not when the process starts.

Notes and deferrals

• Gateway-health derivation (flagged): gateway health is derived from the first hop of the last-mile trace rather than a separate privileged ping — it's simpler and needs no raw sockets. A dedicated low-overhead gateway probe is a possible future refinement.
• Out of scope (by design): full real-user monitoring (RUM) and deep packet capture. For roaming devices behind NAT (network address translation — the router trick that hides many devices behind one shared public address, which also blocks connections inbound to them), a future option is to forward results over the tenant-bound mTLS agent gRPC stream instead of the bus.

The fleet surface

Endpoint results are served from a tenant-scoped read-through snapshot: the per-replica cache keeps the latest result per signal type per endpoint, remains bounded to 2,000 endpoints per tenant and 10 session targets per agent, and evicts the stalest entries. The durable event store is ClickHouse in production (probectl_endpoint_events, partitioned and ordered with tenant_id first) and memory only in the lightweight profile. After a control-plane restart, the first GET /v1/endpoints for tenant A loads only tenant A's latest rows into the cache; it never scans or hydrates another tenant. The endpoint is guarded by RBAC agent.read — endpoints are DEM agents — and collector_running=false distinguishes an unwired consumer from a genuinely empty fleet.

The web surface lives at /endpoints: a fleet list (attribution verdict first — "slow: WiFi / ISP / network" — with Wi-Fi strength, gateway and ISP-edge RTT, server-side cause/text filters, saved views, and keyboard-submit controls) and a per-endpoint detail (Wi-Fi link, gateway / local network, last-mile segments, browser sessions, and the per-layer attribution scores). Saved views are tenant-owned filter sets under GET/POST /v1/inventory/views; they store only the filters, not endpoint rows, so opening one always re-queries the caller's current tenant-scoped inventory. Tenant B cannot list or open tenant A's saved view ID.

Privacy display contract: identifiers the agent withheld (SSID / BSSID / gateway IP / public hops) are absent from the results, and the UI renders that absence honestly — "withheld (privacy)" — never a re-derived or fabricated value. Numeric history lives in the TSDB series; event-shaped history lives in ClickHouse and reconstructs the latest view after restart. The latest-view snapshot is also an age-pruned derived cache: PROBECTL_DERIVED_IDENTITY_RETENTION_DAYS removes stale SSIDs, gateway IPs, session targets, and attribution labels from tenant query surfaces and durable endpoint events, with lifecycle.retention_sweep receipts when entries are deleted. The ClickHouse table also has the deployment TTL configured by PROBECTL_ENDPOINT_RETENTION_DAYS (default 90 days; 0 disables the table TTL).