IaC, GitOps & Helm hardening

What this is

Infrastructure as code (IaC) means your deployment is a reviewed, versioned file instead of a sequence of console clicks; GitOps runs that idea on a loop — a controller inside the cluster watches a Git repository and continually reconciles the cluster to match the file. Like a thermostat: you set the temperature in Git, the controller keeps nudging the room until it matches, and a hand-edit on the live cluster is an opened window the controller quietly closes again. probectl ships that whole stack, declarative (you state the desired end state; the tooling works out the steps) and secure-by-default: a hardened Helm chart, Terraform modules that wrap it, and ArgoCD/Flux manifests so that a git push is the only deploy action you take. Everything is HTTPS-by-default and refuses to run with default credentials.

%%{init: {'theme':'base','themeVariables':{'background':'#0d1117','primaryColor':'#161b22','primaryTextColor':'#e6edf3','primaryBorderColor':'#3b82f6','lineColor':'#8b949e','secondaryColor':'#21262d','tertiaryColor':'#0d1117','clusterBkg':'#161b22','clusterBorder':'#30363d','fontFamily':'ui-monospace, SFMono-Regular, Menlo, monospace'},'flowchart':{'curve':'basis','nodeSpacing':55,'rankSpacing':55,'padding':12}}}%%
flowchart LR
  subgraph Git
    V[values overlay\nsmall/medium/large/multitenant]
  end
  V --> TF[Terraform module] --> H[Helm chart]
  V --> GO[ArgoCD / Flux] --> H
  H --> K[Kubernetes:\nDeployment · Service · Ingress(HTTPS)\nNetworkPolicy · PDB · HPA]

Three ways in (one chart)

All deployment paths deploy the same hardened chart — Terraform and GitOps just wrap it. After deployment, the native terraform-provider-probectl manages served /v1 and /provider/v1 resources directly; the modules/probectl-resources module is the no-plugin fallback that shells out to the checked-in probectl api command.

Hardened Helm chart

values.schema.json types every key (Helm validates input against it on install/upgrade). Three Kubernetes objects in the table deserve a one-line introduction: a NetworkPolicy is the pod's firewall (which traffic may come in, where traffic may go out); a PodDisruptionBudget (PDB) is the floor on how many replicas voluntary maintenance — a node drain, an upgrade — may take down at once; a HorizontalPodAutoscaler (HPA) adds and removes replicas to follow load. The secure defaults below are enforced by the CI hardening gate (make helm-gate), which renders the chart and greps for each invariant — a regression here fails the build:

About the default NetworkPolicy. It is on in every profile. API ingress is already limited to the named ingress-controller namespace, so ordinary in-cluster pods cannot bypass the TLS ingress and talk to the plaintext API listener. Adjust networkPolicy.ingressFrom to your ingress controller's labels. The remaining deliberate hole is egress: empty egressTo allows all non-DNS egress until you name your datastore, bus, IdP, and feed destinations. values-large.yaml ships the filled-in reference egress shape; copy that pattern and narrow the CIDRs. The hardening gate checks the default ingress selector and the strict profile's fully narrowed egress policy.

Reference sizing

The size overlays differ only in runtime sizing — every one of them gets the same HTTPS-by-default, hardened-pod, NetworkPolicy-on posture above.

Two more overlays ship for specialized profiles: values-strict.yaml (the regulated/hardened profile the gate renders) and values-multiregion.yaml (active-active HA — see multi-region.md).

The selected two-region reference is executable configuration, not prose: deploy/terraform/examples/two-region instantiates the module against two Kubernetes providers, while applicationset-multiregion.yaml declares the same two releases for continuing ArgoCD reconciliation. PostgreSQL writer and local-read DSNs remain Secret-only. Synchronous metadata replication, RPO 0, and RTO <=60s are the acceptance thresholds; ClickHouse and object recovery remain distinct backup/replication measurements and never inherit the metadata number by implication.

helm install probectl deploy/helm/probectl -f deploy/helm/probectl/values-medium.yaml \
  --set ingress.host=probectl.example.com --set ingress.tlsSecretName=probectl-tls \
  --set control.tls.existingSecret=probectl-tls \
  --set-string image.digest='sha256:<release-digest>' \
  --set secrets.envelopeKey="$(openssl rand -base64 32)"

Private PKI. control.trustBundle.existingConfigMap names a ConfigMap holding the operator's CA bundle; the chart mounts it read-only into both containers and sets SSL_CERT_FILE, so an IdP, SIEM, CMDB or sslmode=verify-full database behind an enterprise CA verifies without any skip-verify switch (DPR-026). See the chart README.

Config-as-code

The declarative config is the Helm values: control.*, oidc.*, database.url, and control.extraEnv map to PROBECTL_* environment variables via the chart's ConfigMap; the size overlays are the reference config. Commit your overlay, point Terraform or Argo/Flux at it, and the cluster converges to it.

Terraform

deploy/terraform/modules/probectl deploys the chart plus a Kubernetes Secret for the sensitive config — so credentials never land in the ConfigMap or release values. It requires image_digest; the deprecated image_tag input is accepted only in <version>@sha256:<digest> form and discards the mutable tag. Pre-create the module's ingress_backend_tls_trust_secret with the complete ingress-nginx proxy-ssl Secret contract: tls.crt, tls.key, and the control listener issuer in ca.crt. The module enables ingress-nginx certificate and name verification and derives the expected name from ingress_host unless overridden. This is ingress-nginx's outbound identity/trust Secret, not the public ingress or control-listener TLS Secret. It's cloud-agnostic: point the providers at any kubeconfig. The module interface (inputs / outputs / secret handling) is documented in deploy/terraform/README.md. The native terraform-provider-probectl then manages tests, alert routes, Provider/MSP tenants, and advanced API resources against the self-hosted control plane. Its read-only data sources look up or page through tenants, tests, and agents, so modules can compose with existing probectl objects without importing them as managed resources. Tenant and agent/test reads remain tenant-scoped; provider tenant inventory uses the separate operator privilege domain. make terraform-gate runs terraform fmt -check and terraform validate against the single-region and two-region roots; provider data-source examples live in deploy/terraform/examples/provider-data-sources.

GitOps

deploy/gitops/ has an ArgoCD Application (argocd/application.yaml), a two-region ArgoCD ApplicationSet (argocd/applicationset-multiregion.yaml), and a Flux OCIRepository + HelmRelease (flux/) that pin and cosign-verify the signed OCI chart. All reference secrets.existingSecret rather than inlining credentials — Git history is forever, so a secret must never enter it. Manage that Secret with Sealed Secrets or the External Secrets Operator (both keep only an encrypted or referenced form in Git; a cluster-side controller materializes the real value). The examples also reference a probectl-backend-ca Secret; manage its tls.crt, tls.key, and ca.crt through the same controller so ingress-nginx can authenticate and verify the HTTPS control listener. Replace each manifest's invalid image.digest placeholder with the signed release or approved-mirror digest before syncing; leaving it unchanged fails Helm validation. ArgoCD automated sync (prune deletes resources removed from Git; selfHeal re-applies the desired state over hand-edits) and Flux's install/upgrade remediation.retries together give a self-correcting, auto-rolling-back deployment. See deploy/gitops/README.md. make gitops-gate structurally validates the manifests (every doc has an apiVersion + kind).

CI gates

• helm-gate — lints every profile, asserts the hardening invariants above, and validates the GitOps manifests and the compose config. (This is the CI job name to require in branch protection; make gitops-gate runs inside it.)
• terraform-gate — terraform fmt -check + validate of the module through both deployment roots.
• gitops-gate — a make target: the ArgoCD/Flux manifests are well-formed (apiVersion + kind).

Scope

This stack includes single-cluster and exact two-region IaC/GitOps roots over a secure-by-default chart. Active-active multi-region topology and DR is documented separately (multi-region.md, ops/dr.md, runbooks/region-failover.md) and is an Enterprise entitlement (the validated failover runbooks and support, not the fence itself). The FIPS build is likewise the Enterprise-distributed artifact; the STIG/CIS-style hardening checklist itself is public — see hardening.md.