magnus919_agent-skills/release-engineering/references/supply-chain-security.md

Supply-Chain Security in Release Engineering

Supply-chain security is release engineering — not a separate security discipline bolted on afterwards. The artifact your pipeline builds is a bundle of provenance questions: what source produced it, what dependencies it contains, who signed it, and how a consumer verifies any of that. This reference covers SLSA levels, SBOM generation, software signing, dependency-update automation, registries as a security boundary, regulatory drivers, provenance attestations, and the incident playbook for a compromised dependency in a shipped release. For the audit-evidence side of these same artifacts, see change-governance-and-compliance.md.

SLSA Levels

SLSA (Supply-chain Levels for Software Artifacts, v1.0) is a graduated framework that answers: how much can I trust that this artifact was built from this source by this process? It has a build track (levels 1–3) and a source track, with provenance attestations as the core mechanism — machine-readable statements (in-toto format) recording source repo, commit, build workflow, and builder identity.

Level	What it proves	Practical meaning
L1	Provenance exists	Documentation-only; the statement is recorded but not strongly protected
L2	Provenance is generated by a hosted build platform and signed	An attestation exists that the artifact came from a specific build service; verifiable without shared secrets (sigstore keyless)
L3	The build platform prevents tampering; artifact traceable to a specific source commit and build environment	The builder isolates the build and signs provenance that cannot be forged by the build's own steps

SLSA is a ratchet: you cannot skip to L3, and consumers can demand a minimum level from dependencies. For most teams, L2 is achievable with default GitHub Actions/GitLab CI (hosted, signed provenance via Sigstore and slsa-github-generator); L3 requires stronger build isolation. Higher levels (beyond L3) cover more of the supply chain (source integrity, dependency integrity) and are the direction of the framework's continued work.

Reaching Each Level in Practice

Target	Minimum work	Check that proves it
L1	Generate a provenance statement in the pipeline	Provenance file exists for the artifact
L2	Use a hosted builder's signed provenance (slsa-github-generator, OIDC)	cosign verify-attestation succeeds against the expected workflow identity
L3	Isolate the build (ephemeral, no ambient creds), enforce source pins	Provenance builder ID + hermetic build inputs match policy; no untrusted steps can alter the attestation

An OpenSSF Scorecard check on each direct dependency (CI presence, code review, pinned deps, fuzzing) is a cheap complement: it grades the source-level hygiene of the projects you depend on, which SLSA build-track provenance does not cover.

Gotcha — Provenance is only as good as its builder: Provenance signed by a self-hosted runner you do not control, or by a workflow that could be modified by untrusted PRs, proves little. This is why SLSA levels and the "pipeline definitions protected from the code they process" principle in change-governance-and-compliance.md go together.

SBOM Generation

A Software Bill of Materials is an inventory of the components in an artifact — names, versions, licenses, and (ideally) dependency relationships — in a standardized format (CycloneDX or SPDX). It is the foundation for vulnerability queries, license compliance, and incident response.

Tooling

Tool	Strength	Notes
Syft (Anchore)	Deepest cataloger coverage; best-in-class binary analysis (Go, Rust)	Generates SPDX + CycloneDX; does not scan for vulnerabilities — pair with Grype
Trivy (Aqua)	All-in-one: SBOM + vulnerability + misconfig + secrets scanning	Broadest integration (GitHub Action, Kubernetes operator, SARIF); depth slightly lower than Syft in edge cases
CycloneDX CLI	Build-time generation, SBOM merge/diff/validate, native VEX	Most accurate build-time resolution; requires per-language plugin setup; good for merging per-package SBOMs in monorepos

CycloneDX vs SPDX

Dimension	CycloneDX	SPDX
Orientation	Security-focused (vulnerability, VEX, attestation data models)	License/legal-focused; Linux ecosystem heritage
Current	1.6	3.0
Regulatory pick	Explicitly accepted for EU CRA (alongside SPDX)	The other explicitly accepted format (SPDX 2.3 per BSI TR-03183-2)
Best when	You need vulnerability-matching and security tooling	You need license compliance and legal review

Build-Time vs Runtime Accuracy

Generate the SBOM at build time from the actual dependency resolution (lockfiles, package manifests, resolved module graphs), not by scanning a deployed container. Build-time SBOMs reflect what was linked; runtime scans can misattribute layers, miss stripped binaries, or include build-only tooling. CycloneDX CLI's build-time resolution is the most accurate of the three tools for this reason. Attach the SBOM as an artifact or attestation beside the image, and record its reference in the deployment record — this is the artifact that makes change-governance-and-compliance.md's evidence chain supply-chain-aware.

SBOM Quality

An SBOM is only as good as its depth and fidelity. Minimum expectations for an SBOM that will be queried in an incident:

• Component identity that survives across ecosystems: purl (package URL) or similar canonical identifiers, not display names.
• Full transitive coverage — the EU CRA explicitly requires "all components, including dependencies"; a first-level-only SBOM will miss the vulnerable transitive package.
• Relationship data (component A depends on B), not just a flat list — blast-radius analysis needs the graph.
• Regeneration on every release — an SBOM generated once and reused across versions is stale by definition. Make it a pipeline step that fails the build if missing.

A minimal CycloneDX component entry shows the fidelity expected — canonical identity, version, and purl are the fields an incident query keys on:

{
  "type": "library",
  "name": "requests",
  "version": "2.31.0",
  "purl": "pkg:pypi/requests@2.31.0",
  "licenses": [ { "license": { "id": "Apache-2.0" } } ]
}

Where a known vulnerability exists in a component with no fix available, attach a VEX (Vulnerability Exploitability eXchange) statement recording the exploitability assessment — it is the difference between "this component is present" and "this component is known-vulnerable and we have decided how to treat it."

Software Signing

Signing answers "who vouches for this artifact and its metadata?" and enables verification at deploy time.

Mechanism	Status	Model
Cosign / sigstore	Current standard for OCI signing	Keyless signing via OIDC (no long-lived private keys); signatures + attestations published to a transparency log (Rekor); verifiable without shared secrets
Notary / Docker Content Trust	Legacy, declining	Key-based image signing; being superseded by sigstore/Cosign; Notary v2 (notation) retains some traction
Code signing	Complementary	Platform trust (macOS notarization, Windows Authenticode, mobile provisioning); proves publisher identity to end-user OSes
Verification at deploy time	Practice	Gate deployments on signature + attestation verification: reject unsigned images, mismatched digests, or provenance from unapproved builders (admission controllers like Kyverno, cosign verify in the pipeline)

A minimal signing workflow: (1) build the artifact and generate the SBOM; (2) sign the artifact (image/package) with Cosign; (3) sign the SBOM and the provenance attestation; (4) push artifact + signatures + attestations to the registry; (5) at deploy time, verify the signature, the attestation policy (workflow identity, source repo), and the digest before the artifact can run. Steps 2–3 are where "keyless" matters most: the build platform's OIDC identity is the signer, so there is no human-held key to leak and no key rotation treadmill.

Gotcha — Signing without verifying: A pipeline that signs images but never verifies signatures at deploy time has produced ceremony, not security. The verification step (admission controller or pipeline gate) is what converts signing into a control; it belongs in the same pipeline stage as digest pinning.

Dependency Update Automation

Keeping dependencies current is both a vulnerability mitigation and a release-process requirement (a stale dependency matrix makes backports and emergency respins expensive):

Dimension	Renovate (Mend)	Dependabot (GitHub)
Platforms	GitHub, GitLab, Bitbucket, Azure DevOps	GitHub only
Package managers	90+	Broad but narrower, GitHub-first
Monorepo awareness	Grouping, scheduling, auto-merge, dependency dashboard	Basic per-repo PRs
Complexity	Higher configuration surface	Zero-config
Fit	Complex monorepos, multi-platform orgs	Simple GitHub-hosted repos

Both generate per-repo PRs. In a polyrepo with 50 services on a shared library, a breaking change triggers 50 PRs — automation generates them, but humans still review and merge (see monorepo-polyrepo-release.md for the coordination angle). Set update policy deliberately: security-patch auto-merge (fast), minor bumps in batches (reviewed), majors as scheduled upgrades (planned breaking-change handling). Pin everything that can be pinned — base images by digest, actions by SHA, dependencies via lockfiles.

Registries as a Security Boundary

The registry is where the supply chain meets the release pipeline — treat it as a security control, not a file server:

• Digest-pinned base images: reference base images by digest, not mutable latest tags, so a republished tag cannot silently change what your builds run on. This mirrors the artifact-immutability rules in versioning-and-artifacts.md.
• Registry scanning: scan images at push time (Harbor+Trivy, ECR scanning, JFrog Xray) and block or quarantine images failing policy (critical CVEs, unknown provenance, unsigned).
• Proxy/mirror for hermetic builds: a registry proxy/mirror (or an internal mirror with allowlisting) keeps builds hermetic — they resolve from a controlled source rather than the public internet, which is both a reproducibility and a poisoning-defense measure (a compromised upstream package can be blocked centrally).
• RBAC and audit: registry access is scoped per team/service account; every push is logged with its pipeline identity so a malicious artifact can be traced to its build (which is exactly the trace change-governance-and-compliance.md demands).

Regulatory Drivers

• US EO 14028 (2021): SBOM required for software sold to the US federal government; drove SBOM generation into the mainstream of build pipelines.
• EU Cyber Resilience Act (CRA): SBOM mandatory for digital products sold in the EU — vulnerability reporting by 11 September 2026, full compliance (including SBOM) by 11 December 2027; fines up to €15M or 2.5% of global annual turnover. Requires SBOMs covering all components including transitive dependencies, plus continuous vulnerability monitoring.
• Direction of travel: SBOMs move from optional to a build-step requirement; provenance attestations (SLSA + in-toto) connect source → build → artifact; dependency pinning (SHA-pinned actions, lockfiles) becomes standard practice; OpenSSF Scorecard provides a health score for OSS dependencies.

Provenance Attestations and Build Integrity

Provenance is the connective tissue of the supply chain: in-toto attestations (signed statements, typically SLSA provenance) link the source commit → build workflow → artifact digest → SBOM. Build integrity underneath them requires hermetic, reproducible builds: same source revision and inputs yield identical outputs, independent of build-host state (this is also what makes rollback trustworthy — see rollback-and-recovery.md). Practical build-time checks: pin the toolchain, disable network access during the build where possible, run builds in isolated ephemeral environments, and sign the provenance from the build platform, not from a step the build itself controls.

When a Dependency in a Shipped Release Is Compromised

The incident playbook when a dependency in a released artifact is found compromised:

1. Query the SBOM: find every artifact that contains the affected component (name + version range). This is the moment an accurate build-time SBOM pays for itself — scanning containers post-hoc is slower and less reliable. Concretely: query the SBOM store by purl or component name, list every artifact digest that embeds the vulnerable version, and map each digest to the releases/environments it reached via the deployment records.
2. Assess blast radius: which releases, environments, and consumers are affected; did the vulnerable version reach production; does the compromise require an active exploit path or mere presence? Distinguish shipped (in production) from published (in the registry) — the registry may hold vulnerable versions that never ran anywhere.
3. Determine remediation per artifact: upgrade the dependency (roll-forward with a new build) vs. emergency respin of the affected release vs. flag-gate the vulnerable code path (see feature-flag-lifecycle.md — but remember flags cannot undo state).
4. Respin through the pipeline: publish a fixed build with a new digest, update promotion pointers, and redeploy — through the normal pipeline, not a side channel, so the evidence chain stays intact (see change-governance-and-compliance.md).
5. Enforce forward: add the compromised version to a denylist/policy (block at the registry or dependency-update gate), verify signatures on everything republished, and run a postmortem on why the vulnerable version shipped and whether the SBOM matched reality.

Gotcha — The legacy-release blind spot: Old releases are where compromised dependencies bite hardest: their SBOMs were never generated, their builds are unreproducible, and nobody remembers the promotion path. For anything long-lived (mobile apps with store-review lag, embedded/OT devices, LTS products), retroactively generate SBOMs and keep old builds reproducible — the CRA's December 2027 deadline will make this mandatory for EU-market software.

Where the Controls Live

Supply-chain controls belong at a specific layer; putting them in the wrong layer is the usual failure. Map each control to where it can actually fire:

Layer	Controls	Enforced when
Source	Branch protection, CODEOWNERS, signed commits, OpenSSF Scorecard checks on deps	Commit/PR time
Build	Lockfiles, digest-pinned base images, hermetic build, SBOM generation, signing	Pipeline time
Registry	Push-time scanning, allowlist/denylist policy, provenance verification on push, RBAC, audit	Push time
Deploy	Signature + attestation verification (admission controller), digest-pinned manifests, image age policy	Deploy time
Runtime	Continuous vulnerability monitoring, drift detection, quarantine of running vulnerable images	Post-deploy

A control that only exists at one layer (e.g., scanning only at the registry) leaves the other layers unguarded — the respin and the admission gate both need to verify, not just the scanner.

Gotchas

• SBOM without verification is inventory, not security. A list of components is useful for querying; the control comes from scanning those components continuously and gating on findings.
• Keyless is not key-free. Sigstore keyless signing still has a trust model — verify that the OIDC identity (workflow, repo, environment) in the signature is actually the one you approve, via policy rules in the verifier.
• Scanning the final image misses the build. Build-time SBOMs capture the resolved dependency graph; post-push scans may include or exclude layers differently. Publish the build-time SBOM as the canonical artifact and treat registry scans as a second opinion.
• Dependency bots are not a policy. Auto-merge ranges are how a compromised patch propagates fastest. Gate automated merges behind scan results and allowlist behavior.
• Supply-chain incidents are release incidents. The respin is an emergency release; run it through the break-glass/emergency path with the same evidence discipline, not as an unrecorded hotfix. See change-governance-and-compliance.md.

Sources and Further Reading

• SLSA Specification v1.0 — the levels framework and provenance model
• SBOM Tools Compared: Syft vs Trivy vs CycloneDX CLI — tool-by-tool comparison with build-time guidance
• EU CRA SBOM requirements (Anchore) — CRA deadlines, formats, and enforcement
• Renovate vs Dependabot — dependency-update automation comparison
• CycloneDX and SPDX — the two SBOM formats
• sigstore / Cosign — keyless signing and verification
• Google SRE Book — Release Engineering (Chapter 8) — hermetic builds and signed artifacts (CC BY-NC-ND: cite, do not copy)