ci: sign release artifacts with cosign and attach SPDX SBOMs - #824
Open
Yetkin Timocin (ytimocin) wants to merge 2 commits into
Open
ci: sign release artifacts with cosign and attach SPDX SBOMs#824Yetkin Timocin (ytimocin) wants to merge 2 commits into
Yetkin Timocin (ytimocin) wants to merge 2 commits into
Conversation
Images and the CRD bundle now carry Sigstore signatures, made keyless with the release workflow's GitHub OIDC identity, so there is no long-lived signing key to hold or rotate. Each image also carries a per-platform SPDX SBOM attached to its index, readable without pulling the image. Images are signed by digest, not by tag. buildx records the digest it pushed via --metadata-file and the signing script reads it from there: signing a tag would sign whatever that tag resolves to when cosign runs, which is not necessarily what the run built. --recursive covers the per-platform manifests inside the index as well, so a verifier that has already resolved to a platform-specific digest still finds a signature. The signing script derives its work list from the build metadata directory rather than from a list of image names. A hand-maintained list can drift from what is actually built and publish an unsigned image while the job still exits 0 - the same failure shape as a hardcoded chart version. The complementary invariant, that everything a release should contain was in fact built, stays with the verification step, which now also asserts the SBOM is really attached and runs before anything is signed. Both signing steps verify what they just produced and fail the release if it does not check out: a signature nobody can verify is worse than no signature, because RELEASING.md tells users to rely on it. The identity those checks pin is bound to this workflow file rather than to the repository, so another workflow that later gains id-token: write cannot mint signatures that pass, and it carries no ref constraint so both tag pushes and workflow_dispatch runs verify. RELEASING.md documents the byte-identical pattern for consumers. The CRD bundle's checksum is signed as a blob and its signature bundle uploaded as a third release asset, which publish-release now requires before it will publish. Part of #693. Signed-off-by: Yetkin Timocin <ytimocin@microsoft.com>
Yetkin Timocin (ytimocin)
added
process-improvement
Contributor
There was a problem hiding this comment.
Pull request overview
Adds keyless Cosign signing and SPDX SBOMs to the release pipeline.
Changes:
- Signs image digests and CRD checksums using GitHub OIDC.
- Generates and validates per-platform image SBOMs.
- Documents artifact verification and extends release-script tests.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/release.yml |
Generates SBOMs and signs release artifacts. |
.github/workflows/workflow-lint.yml |
Shellchecks all release test stubs. |
.gitignore |
Ignores image metadata output. |
Makefile |
Adds SBOM and build-metadata flags. |
RELEASING.md |
Documents signed-artifact verification. |
hack/release/publish-release.sh |
Requires the CRD signature bundle. |
hack/release/sign-crd-bundle.sh |
Signs and verifies the CRD checksum. |
hack/release/sign-images.sh |
Signs image digests recursively. |
hack/release/test-release-scripts.sh |
Tests signing and release behavior. |
hack/release/testdata/cosign |
Adds a Cosign test stub. |
Suppressed comments (1)
RELEASING.md:134
- The CRD verification command repeats the unconstrained-ref identity pattern, so it trusts
release.ymlsignatures from arbitrary branch-selected workflow runs. Restrict it to release tags and the trusted dispatch branch in lockstep with the production checks.
--certificate-identity-regexp '^https://github\.com/kubefleet-dev/kubefleet/\.github/workflows/release\.yml@' \
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| # that ever gains id-token: write could otherwise mint signatures that pass. No | ||
| # ref constraint - a workflow_dispatch release signs as @refs/heads/<branch> | ||
| # while a tag push signs as @refs/tags/<tag>. | ||
| IDENTITY_PATTERN="^https://github\.com/${repo_pattern}/\.github/workflows/release\.yml@" |
| # See sign-images.sh for why this is bound to the workflow file and why the | ||
| # repository name has its dots escaped before going into a regexp. | ||
| repo_pattern="${GITHUB_REPOSITORY//./\\.}" | ||
| IDENTITY_PATTERN="^https://github\.com/${repo_pattern}/\.github/workflows/release\.yml@" |
| ```bash | ||
| cosign verify \ | ||
| --certificate-oidc-issuer https://token.actions.githubusercontent.com \ | ||
| --certificate-identity-regexp '^https://github\.com/kubefleet-dev/kubefleet/\.github/workflows/release\.yml@' \ |
Comment on lines
+151
to
+156
| sbom="$(docker buildx imagetools inspect "${REGISTRY}/${IMAGE}:${TAG}" \ | ||
| --format '{{ json (index .SBOM "linux/amd64").SPDX }}' 2>/dev/null || true)" | ||
| if [ -z "${sbom}" ] || [ "${sbom}" = "null" ]; then | ||
| echo "::error::${REGISTRY}/${IMAGE}:${TAG} has no SPDX SBOM attached" | ||
| exit 1 | ||
| fi |
Comment on lines
+146
to
+149
| `release.yml` is the only workflow holding `id-token: write`. Any OIDC trust | ||
| policy added later (cloud role assumption, trusted publishing) must be scoped to | ||
| that workflow's `job_workflow_ref`, never to `repo:kubefleet-dev/kubefleet:*`, | ||
| or it would be assumable from any workflow in the repository. |
…form The cosign identity pattern ended immediately after "@", so it accepted release.yml run from any ref. A workflow_dispatch runs the workflow definition from the ref it was started on, which made "any branch in the repository" part of the trusted set. Constrain it to release tags and the main / release-X.Y branches. Move the pattern and the OIDC issuer into hack/release/identity.sh, read by both signing scripts and by their tests. The tests previously reimplemented the pattern, so they asserted a copy and would have kept passing after the scripts drifted. Be explicit about what this does not buy: it bounds which workflow definitions can sign, not who can sign. Someone with write access can still push a v* tag at a commit carrying a modified release.yml, and the tag arm accepts it. Closing that needs a ruleset restricting who may create v* tags, which the repository does not have today. Signing runs after the images are pushed, so a run from an untrusted ref would have published three images, signed them, written a Rekor entry, and only then failed verification. Add check-signing-ref.sh as the first step of create-draft-release, which gates every publishing job: the same identity is now tested before anything exists to clean up. The SBOM gate inspected only linux/amd64 while the release promises one per platform, so an image missing its arm64 SBOM would have published behind a green check. Iterate the platforms instead, cross-checked against the Makefile as a set so a narrowed RELEASE_PLATFORMS fails loudly rather than narrowing the check to match. Capture buildx's stderr too - a registry error and a missing attestation need different fixes and were reported identically. RELEASING.md claimed release.yml was the only workflow holding id-token: write; squad-docs.yml has it as well. Tests: 49 -> 76. Mutating the issuer, dropping the pattern's anchor, widening setup-release.yml's tag grammar, or leaving one of the two documented patterns stale each now fail the suite; before, all four passed. Signed-off-by: Yetkin Timocin <ytimocin@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Images and the CRD bundle now carry Sigstore signatures, keyless via the release workflow’s GitHub OIDC identity — no long-lived signing key to hold or rotate. Each image also carries a per-platform SPDX SBOM attached to its index, readable without pulling the image.
Three decisions worth reviewing:
--metadata-fileand the signing script reads it from there. Signing a tag would sign whatever that tag resolves to when cosign runs, which is not necessarily what the run built.--recursivealso covers the per-platform manifests inside the index, so a verifier that already resolved to an arch-specific digest still finds a signature.release.yml’sjob_workflow_ref, not to the repository, so another workflow that later gainsid-token: writecannot mint signatures that pass. No ref constraint, so both tag pushes andworkflow_dispatchverify. Any OIDC trust policy added later must be scoped the same way, neverrepo:...:*.Both signing steps verify what they just produced and fail the release if it does not check out — a signature nobody can verify is worse than none, because
RELEASING.mdtells users to rely on it. The CRD bundle’s checksum is signed as a blob and its bundle uploaded as a third release asset, whichpublish-releasenow requires. Consumers that enumerate release assets should know about that addition.SBOM and signing are one PR because signing by digest needs the
--metadata-fileplumbing the SBOM change adds; splitting them would land plumbing with no consumer.Not yet executed end to end, and this one cannot be locally. Keyless signing needs real Actions OIDC, so the test suite covers the logic around cosign — which digest is signed, what happens when a step fails — against a stub, not cosign itself. A dry run in a scratch repo is planned before merge; the certificate SAN is the specific thing to confirm against a real Fulcio cert, since the identity pattern is strict enough that a wrong guess fails every release.
Scoped out deliberately:
cosign attest --type=spdxjsonandrelease-metadata.json, both still open in #693. The attestation question was raised for maintainer input in this comment and has not been answered; this PR proceeds without it rather than settling it.Part of #693.