Skip to content

Commit baa0f76

Browse files
fix(build-push-docker-manifest): idempotent manifest + cosign reruns (RANE-4683) (#1577)
* fix(build-push-docker-manifest): idempotent manifest + cosign reruns (RANE-4683) - Skip imagetools create when the tag already references the expected platform digests; fail if the tag exists with different digests - Skip cosign sign when verify already succeeds (idempotent rerun) - Retry cosign verify after sign for Sigstore propagation flakes Together these changes make build-publish manifest jobs safe to rerun without digest drift or redundant signing, while still failing loudly on real conflicts. * fix(build-push-docker-manifest): address PR review feedback (RANE-4683) - Ensure jq is installed before digest comparison - Validate docker-image-name-digests is non-empty - Only skip cosign sign when OIDC identity constraints verify - Fail fast on unexpected imagetools inspect errors (not just missing tag) * chore(build-push-docker-manifest): address additional Copilot nits Keep MANIFEST_CREATE_SKIPPED expression on one line and only sleep between cosign verify retries, not after the final failed attempt. * chore(build-push-docker-manifest): address latest Copilot nits - Fail fast when jq is missing and apt-get/sudo are unavailable - Shorten step/output ids so summary env expressions stay on one line under Prettier (avoids split ${{ }} expressions) * fix(build-push-docker-manifest): de-duplicate platform digests for idempotency Treat digest lists as sets: sort -u in normalization and existing-manifest inspection, and reuse normalized digests when building imagetools create args. * fix(build-push-docker-manifest): address follow-up Copilot nits - Only sleep between manifest digest inspect retries, not after the final fail - Fail fast when an existing tag has no extractable platform digests instead of falling through to imagetools create (digest drift on rerun) * fix(build-push-docker-manifest): validate sha256 digest format early Reject non-sha256 tokens in normalize_digest_csv so bad docker-image-name-digests input fails fast with a clear error instead of deferring to imagetools create. * chore(build-push-docker-manifest): replace jq install with availability guard jq is preinstalled on GitHub-hosted ubuntu-24.04 runners; fail fast with a clear error when missing instead of apt-get/sudo install logic. * fix(build-push-docker-manifest): ECR propagation guard + move verify to workflow Retry manifest inspect until platform digests match expected inputs (ECR tag propagation lag). Move the cosign verify gate with 5×10s retry from the composite action into reusable-docker-build-publish; keep idempotent create, sign-skip internal verify, and sign in the action (RANE-4683). * fix(build-push-docker-manifest): read additional tags from action input in summary The step summary referenced a manifest-additional-tags output that inspect-docker-manifest never sets; use the manifest-additional-tags input instead so additional tags display correctly. * refactor(build-push-docker-manifest): verify signature in its own job (RANE-4683) Per review (Erik/Chad): move cosign signature verification out of the build-push-docker-manifest action into a standalone verify-manifest-signature job in reusable-docker-build-publish, so a Sigstore propagation-lag flake can be retried in isolation without re-running manifest create/sign. That rerun was the root cause of index digest drift stranding the tag on an unsigned wrapper (hit again on v2.55.0-rc.0 GA). - reusable-docker-build-publish: new verify-manifest-signature job (needs docker-manifest); expose docker-manifest name-with-digest output; remove the inline verify step. - build-push-docker-manifest: always cosign sign (safe now that verify is decoupled and imagetools create is idempotent); drop sign-skip verification shell; drop the jq availability guard (preinstalled on ubuntu-24.04); tidy summary. - regenerate synced .github/workflows copy; update changeset. * refactor(verify): extract cosign verify+retry into verify-docker-signature action Address "too much bash in the job" (Chad): move the standalone verify job's inline cosign install + retry loop into a dedicated composite action `verify-docker-signature`, with the retry logic in scripts/verify-with-retry.sh. The verify-manifest-signature job in reusable-docker-build-publish is now declarative (aws login -> ecr login -> uses: verify-docker-signature). - new action actions/verify-docker-signature (action.yml, scripts, README, package.json, project.json); cosign verify with configurable retry for Sigstore propagation lag. - reusable-docker-build-publish: verify job uses the new action (pinned @main pending first release; TODO to pin to /v1). - changeset + regenerated .github/workflows copy. * refactor(verify): keep verification inline in the workflow job Revert the verify-docker-signature composite action / external script; per follow-up review we keep the logic directly in the workflow (matching the existing inline style in reusable-docker-build-publish). Verification stays in its own verify-manifest-signature job so a Sigstore propagation flake retries without re-running create/sign, but the cosign install + retry loop live inline. * refactor(verify): single isolated verify step, no in-step retry loop Per review (Chad: the retry loop isn't good; Erik: keep verify as its own isolated step): drop the 5x10s hand-rolled retry loop and verify once. The verify-manifest-signature job is already isolated and independently re-runnable, so it is the retry unit for the Sigstore propagation-lag case — re-running it never touches manifest create/sign, so no index digest drift (RANE-4683). * feat(verify): intelligent verify retry — exponential backoff, 10 min cap Per Chad: replace the fixed retry with an exponential-backoff retry (10s, 20s, 40s ... capped at 120s/wait) bounded to 10 minutes total, in the isolated verify-manifest-signature job. Absorbs Sigstore propagation lag without ever re-running manifest create/sign (RANE-4683). * fix(verify): bump verify-manifest-signature timeout to 12m Job timeout was 5m while the in-step retry budget is 600s, so GitHub could kill the job before Sigstore propagation retries finished. * docs(action): comment idempotent manifest create for reviewers Explain why we compare platform digests (not index digests), skip vs fail-hard paths, and how create idempotency relates to the isolated verify job (RANE-4683). * fix(verify): include last cosign verify output on timeout Makes auth/identity mismatches distinguishable from Sigstore propagation lag when the 10-minute retry budget is exhausted. * fix(action): require platform os+arch in digest jq filter Descriptors without .platform (e.g. SBOMs/attachments) no longer pass the idempotency/propagation digest comparison. * Debloat: reduce to a minimal idempotency guard (RANE-4683) - Skip imagetools create when the tag already exists (lightweight sanity check that it references the digests being published) - Revert inspect step to original retry loop - Move cosign verify back inline with a small backoff retry; drop the separate verify-manifest-signature job (redundant once create is idempotent; it also hardcoded the OIDC issuer/identity) - Delete normalize_digest_csv / get_existing_platform_digests / wait_for_manifest_tag_propagation helpers * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(build-push-docker-manifest): repair idempotency guard The autofix rewrote the create-step guard into Python heredocs that produced invalid YAML (unindented heredoc body terminated the run block scalar) and captured inspect stderr into the JSON payload (2>&1). Restore a plain-bash guard that: - validates/normalizes docker-image-name-digests up front (fail fast on empty/invalid input, before both the idempotency check and create) - captures inspect stdout and stderr separately (no warnings in payload) - distinguishes 'tag missing' from real inspect errors (no silent fall-through to create on transient/auth failures) * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent 1f49f1a commit baa0f76

2 files changed

Lines changed: 121 additions & 12 deletions

File tree

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
"build-push-docker-manifest": minor
3+
---
4+
5+
Make manifest create idempotent for build-publish reruns (RANE-4683):
6+
7+
- Skip `imagetools create` when the manifest tag already exists, so a rerun doesn't mint a new index digest and move the tag onto an unsigned wrapper. Fail if the existing tag doesn't reference the digests being published.
8+
- Retry `cosign verify` to absorb Sigstore propagation lag ("no signatures found" right after a good sign). Because create is idempotent, retrying verify never re-runs create/sign.

‎actions/build-push-docker-manifest/action.yml‎

Lines changed: 113 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -314,8 +314,13 @@ runs:
314314
sleep 30
315315
fi
316316
317+
# Idempotency (RANE-4683): re-running `imagetools create` mints a new
318+
# multi-arch index digest even when the children are identical, which moves
319+
# the tag onto an unsigned wrapper and breaks downstream cosign verify /
320+
# promote. So if the tag already exists we skip create and keep the existing
321+
# index digest.
317322
- name: Create and push Docker manifest
318-
id: create-push-docker-manifest
323+
id: manifest-create
319324
shell: bash
320325
env:
321326
DOCKER_MANIFEST_NAME: ${{ steps.manifest-name.outputs.name }}
@@ -327,12 +332,69 @@ runs:
327332
${{ steps.generate-annotations.outputs.annotation-flags }}
328333
TAG_FLAGS: ${{ steps.process-additional-tags.outputs.tag-flags }}
329334
run: |
335+
set -euo pipefail
336+
330337
DOCKER_MANIFEST_NAME_WITH_TAG="${DOCKER_MANIFEST_NAME}:${DOCKER_MANIFEST_TAG}"
331-
# Convert comma-separated list into array and pass as separate arguments
332-
IFS=',' read -ra DIGESTS <<< "$DOCKER_IMAGE_NAME_DIGESTS"
333-
# Map each digest to include the manifest name
338+
339+
# Normalize + validate the expected digests up front (before both the
340+
# idempotency check and create) so bad/empty input fails fast here
341+
# rather than deep inside imagetools with a less actionable error.
342+
EXPECTED_DIGESTS=()
343+
IFS=',' read -ra RAW_DIGESTS <<< "$DOCKER_IMAGE_NAME_DIGESTS"
344+
for digest in "${RAW_DIGESTS[@]}"; do
345+
digest=$(echo "$digest" | xargs)
346+
[[ -z "$digest" ]] && continue
347+
if [[ ! "$digest" =~ ^sha256:[a-f0-9]{64}$ ]]; then
348+
echo "::error::docker-image-name-digests contains an invalid entry '${digest}'; expected sha256:<64-hex>"
349+
exit 1
350+
fi
351+
EXPECTED_DIGESTS+=("$digest")
352+
done
353+
if [[ ${#EXPECTED_DIGESTS[@]} -eq 0 ]]; then
354+
echo "::error::docker-image-name-digests must contain at least one sha256:<64-hex> digest"
355+
exit 1
356+
fi
357+
358+
# Idempotency guard: if the tag already exists, don't recreate it
359+
# (re-creating mints a new index digest even when children are identical,
360+
# moving the tag onto an unsigned wrapper). Capture stdout (the raw
361+
# manifest) and stderr separately so registry warnings never pollute the
362+
# payload, and distinguish "tag missing" from real inspect failures so a
363+
# transient/auth error doesn't silently fall through to create.
364+
INSPECT_ERR=$(mktemp)
365+
INSPECT_RC=0
366+
EXISTING_MANIFEST=$(docker buildx imagetools inspect "${DOCKER_MANIFEST_NAME_WITH_TAG}" --raw 2>"${INSPECT_ERR}") || INSPECT_RC=$?
367+
368+
if [[ "${INSPECT_RC}" -eq 0 ]]; then
369+
rm -f "${INSPECT_ERR}"
370+
# Tag exists — sanity check it references the digests we're about to
371+
# publish; if it points at something else, fail rather than trust it.
372+
MISSING_DIGESTS=()
373+
for digest in "${EXPECTED_DIGESTS[@]}"; do
374+
grep -Fq -- "${digest}" <<< "${EXISTING_MANIFEST}" || MISSING_DIGESTS+=("$digest")
375+
done
376+
if [[ ${#MISSING_DIGESTS[@]} -gt 0 ]]; then
377+
echo "::error::Manifest tag ${DOCKER_MANIFEST_NAME_WITH_TAG} already exists but does not reference expected digest(s): ${MISSING_DIGESTS[*]}"
378+
exit 1
379+
fi
380+
echo "✅ Manifest tag ${DOCKER_MANIFEST_NAME_WITH_TAG} already exists and references the expected digests; skipping create (idempotent rerun)"
381+
echo "skipped=true" | tee -a "${GITHUB_OUTPUT}"
382+
exit 0
383+
fi
384+
385+
if ! grep -qiE 'not found|no such manifest|manifest unknown|MANIFEST_UNKNOWN|404' "${INSPECT_ERR}"; then
386+
echo "::error::Failed to inspect manifest tag ${DOCKER_MANIFEST_NAME_WITH_TAG}"
387+
cat "${INSPECT_ERR}" >&2
388+
rm -f "${INSPECT_ERR}"
389+
exit 1
390+
fi
391+
rm -f "${INSPECT_ERR}"
392+
393+
echo "skipped=false" | tee -a "${GITHUB_OUTPUT}"
394+
395+
# Map each validated digest to include the manifest name
334396
PREFIXED_DIGESTS=()
335-
for digest in "${DIGESTS[@]}"; do
397+
for digest in "${EXPECTED_DIGESTS[@]}"; do
336398
PREFIXED_DIGESTS+=("${DOCKER_MANIFEST_NAME}@${digest}")
337399
done
338400
@@ -422,7 +484,7 @@ runs:
422484
if:
423485
inputs.docker-manifest-sign == 'true' &&
424486
inputs.cosign-oidc-identity-regexp != ''
425-
shell: sh
487+
shell: bash
426488
env:
427489
MANIFEST_NAME_WITH_DIGEST: >-
428490
${{
@@ -432,18 +494,53 @@ runs:
432494
OIDC_ISSUER: ${{ inputs.cosign-oidc-issuer }}
433495
OIDC_IDENTITY_REGEXP: ${{ inputs.cosign-oidc-identity-regexp }}
434496
run: |
435-
cosign verify "${MANIFEST_NAME_WITH_DIGEST}" \
436-
--certificate-oidc-issuer "${OIDC_ISSUER}" \
437-
--certificate-identity-regexp "${OIDC_IDENTITY_REGEXP}" \
438-
--certificate-github-workflow-repository "${GITHUB_WORKFLOW_REPOSITORY}"
497+
set -euo pipefail
498+
499+
# Retry to absorb Sigstore/Rekor propagation lag ("no signatures found"
500+
# right after a good sign). Keep the retry budget large enough to absorb
501+
# longer propagation delays.
502+
last_verify_output=""
503+
start_ts=$(date +%s)
504+
max_elapsed_seconds=600
505+
delay=5
506+
507+
while true; do
508+
if last_verify_output=$(cosign verify "${MANIFEST_NAME_WITH_DIGEST}" \
509+
--certificate-oidc-issuer "${OIDC_ISSUER}" \
510+
--certificate-identity-regexp "${OIDC_IDENTITY_REGEXP}" \
511+
--certificate-github-workflow-repository "${GITHUB_WORKFLOW_REPOSITORY}" 2>&1); then
512+
echo "${last_verify_output}"
513+
exit 0
514+
fi
515+
516+
now=$(date +%s)
517+
elapsed=$(( now - start_ts ))
518+
if (( elapsed >= max_elapsed_seconds )); then
519+
break
520+
fi
521+
522+
sleep_for=$delay
523+
if (( elapsed + sleep_for > max_elapsed_seconds )); then
524+
sleep_for=$(( max_elapsed_seconds - elapsed ))
525+
fi
526+
echo "Signature not yet verifiable; retrying in ${sleep_for}s (elapsed ${elapsed}s/${max_elapsed_seconds}s)..."
527+
sleep "${sleep_for}"
528+
529+
delay=$(( delay * 2 ))
530+
if (( delay > 60 )); then delay=60; fi
531+
done
532+
533+
echo "::error::Signature not verified for ${MANIFEST_NAME_WITH_DIGEST}"
534+
echo "Last cosign verify output:" >&2
535+
echo "${last_verify_output}" >&2
536+
exit 1
439537
440538
- name: Summary output
441539
shell: bash
442540
env:
443541
DOCKER_MANIFEST_SIGNED: ${{ inputs.docker-manifest-sign }}
444542
GITHUB_WORKFLOW_REPOSITORY: ${{ inputs.github-workflow-repository }}
445-
MANIFEST_ADDITIONAL_TAGS:
446-
${{ steps.inspect-docker-manifest.outputs.manifest-additional-tags }}
543+
MANIFEST_ADDITIONAL_TAGS: ${{ inputs.manifest-additional-tags }}
447544
MANIFEST_DIGEST:
448545
${{ steps.inspect-docker-manifest.outputs.manifest-digest }}
449546
MANIFEST_NAME: ${{ steps.inspect-docker-manifest.outputs.manifest-name}}
@@ -456,6 +553,7 @@ runs:
456553
steps.inspect-docker-manifest.outputs.manifest-name-with-tag
457554
}}
458555
MANIFEST_TAG: ${{ inputs.docker-manifest-tag }}
556+
CREATE_SKIPPED: ${{ steps.manifest-create.outputs.skipped }}
459557
OIDC_ISSUER: ${{ inputs.cosign-oidc-issuer }}
460558
OIDC_IDENTITY_REGEXP: ${{ inputs.cosign-oidc-identity-regexp }}
461559
run: |
@@ -467,6 +565,9 @@ runs:
467565
echo "Manifest additional tags: \`${MANIFEST_ADDITIONAL_TAGS:-None}\`" | tee -a "${GITHUB_STEP_SUMMARY}"
468566
echo "Manifest name with tag: \`${MANIFEST_NAME_WITH_TAG}\`" | tee -a "${GITHUB_STEP_SUMMARY}"
469567
echo "Manifest name with digest: \`${MANIFEST_NAME_WITH_DIGEST}\`" | tee -a "${GITHUB_STEP_SUMMARY}"
568+
if [[ "${CREATE_SKIPPED}" == "true" ]]; then
569+
echo "Manifest create: skipped (tag already exists and references the expected digests)" | tee -a "${GITHUB_STEP_SUMMARY}"
570+
fi
470571
if [[ "${DOCKER_MANIFEST_SIGNED}" == 'true' ]]; then
471572
echo >> "${GITHUB_STEP_SUMMARY}"
472573
echo "#### Docker Manifest signed 📝" | tee -a "${GITHUB_STEP_SUMMARY}"

0 commit comments

Comments
 (0)