Incident response plan

How a probectl security incident is detected, triaged, handled, evidenced, and disclosed. Companion to SECURITY.md (reporting intake + coordinated disclosure) and threat-model.md. Two scopes, kept deliberately distinct:

• Product/vendor incidents — a vulnerability or compromise in probectl code, releases, or build infrastructure. Vendor-led; affects all deployments.
• Deployment incidents — a compromise inside one operator's installation. Operator-led under their own IR (incident-response process); probectl's job is to give them the evidence tooling and this playbook's deployment sections.

The shape to keep in mind: a product incident is a manufacturer's recall — one defect, every car on the road; a deployment incident is a break-in at one owner's garage — their site, their locks, their police report, with probectl supplying the forensic toolkit.

1. Severity matrix

Severity is the dial that sets response speed: judge the impact, read off the clock. Cross-tenant exposure sits alone at the top because tenant isolation is the product's outermost security boundary.

Cross-tenant anything starts at SEV-1 until proven otherwise. When in doubt, classify up.

2. Roles

Solo-founder reality, stated plainly: the maintainer is Incident Commander, investigator, and fixer. The compensating structure is this written plan; predefined external deputies (counsel for notification-duty questions; the affected operator's security team for deployment incidents — co-responders by design, since they hold the infrastructure); and an immutable trail (the signed WORM audit export — WORM, Write Once, Read Many: an off-database copy a database owner cannot rewrite or quietly purge; see ../hardening.md §0b) so every action is reviewable after the fact. The role table will be re-cut at the first security hire.

3. Response flow

1. Intake. A report through the SECURITY.md channels (private advisory / security.txt contact), a CI security gate (security-scan.yml), the WORM chain-verification alarm, or an operator report.
2. Declare + log. Assign a severity; open a private timeline doc; record UTC timestamps for every action from this point on.
3. Contain.
   • Product scope: pull or revoke the affected artifacts (every release is cosign-signed, so its signing identity makes "which artifact" provable) and gate the release lane.
   • Deployment scope: the operator isolates per the runbooks — agents halt-on-error via the rollout machinery, credentials rotate, and any insecure override is audited.
4. Preserve evidence — before remediation mutates state.
   • Export the audit chains: the provider stream's signed WORM segments are already off-database and chain-verified (internal/audit/worm.go); snapshot the tenant streams.
   • Run scripts/backup_postgres.sh and scripts/backup_clickhouse.sh to capture store state with SHA-256 manifests.
   • Generate a support bundle (internal/support — the redacted config-and-version snapshot) for posture evidence.
   • Hash and retain the agent binaries/objects in question, comparing against the baked-in integrity manifest and the release signatures.
5. Eradicate + recover. Fix forward; release through the normal lane (the red-CI refusal stays on); upgrade the fleet as staged waves of signed artifacts with halt-on-error — an emergency changes the wave sizes, never the verification.
6. Notify (§4) and disclose (§5).
7. Post-incident review within 7 days: timeline, root cause, a tracked follow-up for every systemic gap found, and a re-review of the threat model if a trust boundary was involved.

4. Customer notification

The self-hosted nuance: the vendor cannot see deployments, so notification means telling operators what to check and what to ship, fast.

• Channels: GitHub Security Advisory (the canonical record) plus release notes; direct contact for known enterprise/MSP operators.
• SEV-1: an initial notice within 72h of confirmation, even if incomplete — affected versions, severity, interim mitigations, and indicators of compromise (the concrete fingerprints of the attack: how an operator checks their own audit chains and registry for the pattern). Update cadence at least every 72h until resolved.
• SEV-2: notice with the fix release.
• Content discipline: facts only — affected versions, the upgrade path (../ops/fleet-rollout.md), and verification steps (../ops/verify-artifacts.md). MSP operators re-notify their tenants under their own obligations; the per-tenant audit trail gives them the evidence to do it.

5. Disclosure

Coordinated disclosure per SECURITY.md: reporter credit, advisory published when a fix is available, CVE (the public vulnerability-catalog entry) requested via GitHub for qualifying issues. Embargo — the agreed quiet period before details go public — target ≤ 90 days, shorter when exploitation is observed.

6. Drill

Tabletop this plan twice yearly — a tabletop is a paper walk-through of the plan, no production systems touched — one product-scope, one deployment-scope (using the backup/failover drills as the recovery legs), and after any SEV-1/2. Record outcomes in the review log below.