Repository navigation
fix(release): signed-tag aggregate recovery starves under concurrent merges (exact-tip requirement) #34710
Description
fix(release): signed-tag aggregate recovery starves under concurrent merges (exact-tip requirement)
Pre-flight
- Memory recall:
abandon stranded release lane unpublished tag fresh release→ 0 hits - Discovery pass: 6 recent commits on target files; GH#34674: fix unpublished protected-main aggregate recovery #34677 and GH#34686: fix(release): parse gh stderr HTTP status in recovery release probe #34688 are the latest recovery fixes; no open claimed issues
- File refs verified at
mainccf3073d5c - Tier:
tier:thinking. This changes a release-provenance trust fence. - Seeded draft PR decision recorded: skipped. Claimed for interactive implementation.
Origin
- Created: 2026-10-15
- Created by: ai-interactive (v3.39.31 release recovery, Ref GH#34600: fix: closed-issue hold re-comments every cycle under App read auth #34603)
- Conversation context: v3.39.31 has been stranded since 2026-10-14. Its signed tag
a1af8e65e3targets92ff98fab7, which is already onmain(VERSION=3.39.31), but the tag's signer key is not registered on GitHub.recover-aggregatere-signs it, but needsorigin/mainto equal a reviewed aggregation merge. Four consecutive exact-tip aggregates were invalidated before recovery could start: chore(release): aggregate v3.39.31 recovery after #34673 and #34677 #34682→chore(release): aggregate v3.39.31 recovery after #34682 and #34688 #34690 (merged, then GH#34687: docs: audit image-library upload reachability #34691/GH#34685: fix: scan pushed commit ranges in gh wrapper guard #34692 landed), chore(release): aggregate v3.39.31 recovery after #34690 #34701 (GH#34693: chore(deps): adopt reviewed actions updates (supersedes #34643) #34699 landed during CI), and chore(release): aggregate v3.39.31 recovery after #34699 #34704 (GH#34681: fix worker blocker handoff classification and retained evidence #34695 landed two minutes into CI). Pulse merges by other maintainers land every 2–15 minutes, and required CI takes about 7 minutes.
Reproducer
Observed live on 2026-10-14 (UTC). Merge times come from gh pr list --state merged:
22:47:28 #34690 aggregate merged (b44b336c42) -> recovery start blocked by a stale deploy, then main moved
22:53:46 #34692, 22:55:29 #34691 merged (alex-solovyev pulse)
~23:00 #34701 successor opened at c1db70b11d; required checks green after ~7 min
23:08:28 #34699 merged -> #34701 stale before merge
23:10:29 #34695 merged -> #34704 (opened at cf6811ef59) stale 2 min into CI
release-provenance-helper.sh:400-403 refuses with "release checkout is not the exact origin/main tip" whenever any PR merges between the aggregate's CI start and the start of recovery.
What
Signed-tag aggregate recovery must be able to complete while ordinary PRs keep merging. That means removing the exact-tip starvation without weakening provenance. It also needs a supported way to retire a stranded, never-published lane, so that a fresh release is never blocked forever.
Why
release_lane_merge_guard(.agents/scripts/release-lane-helper.sh:1394-1400) has been a deliberate no-op since Snapshot releases without blocking concurrent PR merges #31472 (4904174, "Release immutable snapshots while PR merges continue"). Normal releases use immutable snapshots for that reason.- Recovery still requires
HEAD == origin/main(release-provenance-helper.sh:399-403via_release_provenance_resolve_source). That conflicts with the snapshot model and makes recovery a race the operator usually loses. .agents/reference/release-aggregation-recovery.md:165still says "Ordinary merges remain blocked during every recovery and committing phase". That is stale since Snapshot releases without blocking concurrent PR merges #31472.- Nothing can retire an unpublished lane.
_full_loop_release_guard_competing_lane(full-loop-release-helper.sh:682-712) returns 75 for any other source unless the lane is an abandoned reservation or already published, and supersession (full-loop-release-reconcile.sh:1789-1807) requires a newer published tag. A stranded lane therefore blocks every new release indefinitely.
Tier
Selected tier: tier:thinking
Tier rationale: The trust boundary is release provenance. It requires choosing between snapshot-bound recovery and lane retirement, and rollback semantics must be decided.
How (Approach)
Decision options (recommended first)
- Snapshot-bound recovery. Accept a reviewed aggregation merge
Athat is an ancestor oforigin/main, not necessarily its tip, whenA's immutable trailers cover every first-parent commit from the previous baseline throughA. This mirrors the existing snapshot mode (mode:"snapshot",snapshot_baseinfull-loop-release-reconcile.sh:644-647). The recovery bump commit already takesAas parent (reference doc step 4). The recovered-tag validator already checks the direct aggregate parent rather than the current tip (doc line 159). The protected release PR merge then has to preserve the release commit SHA (_merge_release_provenance_method). With this, merged chore(release): aggregate v3.39.31 recovery after #34682 and #34688 #34690 (b44b336c42) recovers v3.39.31 without another aggregate. - Unpublished-lane retirement. Add
aidevops release retire-unpublished <source-pr> --tag <tag> --reason <text>. It reuses the channel-absence probes (_full_loop_recovery_verify_*_absent), requires a dead or unknown executor, and makes a compare-and-swap write moving the lane to inactive with an auditable terminal receipt. The local unpublished tag is left inert. A fresh snapshot release can then proceed; version handling for the skipped number still needs a decision.
Files to Modify
EDIT: .agents/scripts/release-provenance-helper.sh:372-470: the snapshot-tolerant aggregate resolver for the signed-tag recovery path only.EDIT: .agents/scripts/full-loop-release-aggregate-recovery.sh: the recovery entry (_full_loop_release_recover_aggregateat 1734) and the existing-context preparation.EDIT: .agents/reference/release-aggregation-recovery.md:6-27,165: correct the stale merge-blocking claim and document the chosen mode.EDIT: .agents/scripts/tests/test-full-loop-release-aggregate-recovery.sh: existing suite; extend only where the new acceptance path needs coverage.
Complete Write Surface
- Callers/readers:
full-loop-release-helper.sh:894-903(recover-aggregate),release-provenance-helper.shresolve-*subcommands, and the reconcile path infull-loop-release-reconcile.sh. - Writers/mutation paths: lane compare-and-swap writes in
release-lane-helper.sh(begin/refresh/commit phases), local tag replacement in version-manager protected-main publication, and receipts. - Existing verification/tests:
test-full-loop-release-aggregate-recovery.sh(12+ PASS cases, including "retains manifest and exact-tip fences") andtest-release-provenance-helper*.sh. - Schemas/config: lane JSON on
refs/heads/aidevops/release-lane. Option 2 adds a terminal receipt value. - Generated/deployed mirrors:
~/.aidevops/agents/scripts/viasetup.sh. - Migrations/backfills: option 1 needs no backfill. The persisted recovery transaction gains a pinned
snapshot_basefield only for new transactions; existing lanes onrefs/heads/aidevops/release-laneremain exact-tip transactions. Option 2 adds a terminal receipt value that readers infull-loop-release-helper.sh:674-676must recognise. - Cleanup/rollback paths: reverting
.agents/scripts/release-provenance-helper.shrestores the exact-tip refusal. A lane left mid-transaction by the new mode must fail closed under old code, never be adopted, so the rollback cannot publish an unverified tag.
Implementation Steps
- Decide between option 1 and option 2 (option 1 recommended), and record the decision on the issue.
- Option 1: add a recovery-only resolver mode in
release-provenance-helper.sh. It takes an explicit aggregate merge SHAA, verifies thatAis an ancestor oforigin/main, verifiesA's aggregator and manifest trailers with the existing_release_provenance_verify_pr_record/_release_provenance_verify_aggregate_base, and returnsmode:"aggregate"withsource_merge:A. Keep the exact-tip check for every non-recovery caller. - Pass the pinned
Athrough_full_loop_release_recover_aggregateand persist it in the recovery transaction, so retries resolve the same base. Build the bump commit with parentA, as now. - Correct
release-aggregation-recovery.md:165and document the snapshot-bound mode. - Run shellcheck and the recovery and provenance test suites. Then run the live v3.39.31 recovery against
b44b336c42(chore(release): aggregate v3.39.31 recovery after #34682 and #34688 #34690) and verify convergence across GitHub, npm and Homebrew.
Hazards and Compatibility
- Concurrency/atomicity: keep the compare-and-swap token rotation. The snapshot base must be pinned in the persisted transaction so a retry resolves the same
A. - Migration/rollback: the existing exact-tip transactions stay valid, so the exact tip remains a special case of the snapshot.
- Mixed-version/backward compatibility: older clients that see a snapshot-mode recovery transaction must fail closed, not adopt it.
- Idempotency/retry: rerunning with the same aggregate must resume, not create a second bump commit.
- Partial failure/recovery: the existing rule still applies. Once the release PR or remote tag is durable, nothing restores or mutates provisional state.
Verification Before Dispatch
shellcheck .agents/scripts/full-loop-release-aggregate-recovery.sh .agents/scripts/release-provenance-helper.sh
bash .agents/scripts/tests/test-full-loop-release-aggregate-recovery.sh
aidevops release recover-aggregate 34603 --tag v3.39.31 --expected-sources <#34690 manifest PR list> # live: succeeds with main ahead of b44b336c42- Surface mapping: the tests prove the fences remain. The live run proves that the starvation case is resolved and that publication converges across GitHub, npm and Homebrew.
- Broad verification trigger: release infrastructure change. Run
.agents/scripts/linters-local.sh --changedplus the release test suites.
Files Scope
.agents/scripts/release-provenance-helper.sh.agents/scripts/full-loop-release-aggregate-recovery.sh.agents/reference/release-aggregation-recovery.md.agents/scripts/tests/test-full-loop-release-aggregate-recovery.sh
Acceptance Criteria
- Recovery succeeds when
origin/mainhas advanced past a reviewed aggregation merge whose trailers fully cover the history up to that merge. - Recovery still refuses an aggregate whose trailers omit any first-parent commit up to the aggregate, a tag already on a remote or channel, and an aggregate that is not an ancestor of
origin/main. -
release-aggregation-recovery.mdno longer claims that ordinary merges are blocked during recovery. - v3.39.31 publishes, and the GH#34600: fix: closed-issue hold re-comments every cycle under App read auth #34603 lane reaches a terminal
publishedreceipt.
Context & Decisions
- Companion diagnostics issue: the silent
refresh-aggregateexit (filed alongside). - Superseded aggregates: chore(release): aggregate v3.39.31 recovery after #34690 #34701 (closed), chore(release): aggregate v3.39.31 recovery after #34699 #34704 (stale), chore(release): aggregate unpublished v3.39.31 recovery #34650 (old draft).
- Non-goal: re-enabling a global merge freeze. That contradicts Snapshot releases without blocking concurrent PR merges #31472.
aidevops.sh v3.39.31 plugin for OpenCode v1.19.0 with claude-opus-5-5 spent 6h 55m and 411,972 tokens on this with the user in an interactive session.
Activity
marcusquinn commentedon Oct 14, 2026
aidevops.sh v3.39.31 plugin for OpenCode v1.19.0 with claude-opus-5-5 spent 6h 55m and 411,972 tokens on this with the user in an interactive session.
github-actions commentedon Oct 15, 2026
Completed via PR #34743. merged to main.
aidevops.sh vunknown automated scan.