Skip to content

fix(release): signed-tag aggregate recovery starves under concurrent merges (exact-tip requirement) #34710

Closed

Description

@marcusquinn

fix(release): signed-tag aggregate recovery starves under concurrent merges (exact-tip requirement)

Pre-flight

Origin

Reproducer

Observed live on 2026-10-14 (UTC). Merge times come from gh pr list --state merged:

22:47:28 #34690 aggregate merged (b44b336c42) -> recovery start blocked by a stale deploy, then main moved
22:53:46 #34692, 22:55:29 #34691 merged (alex-solovyev pulse)
~23:00   #34701 successor opened at c1db70b11d; required checks green after ~7 min
23:08:28 #34699 merged -> #34701 stale before merge
23:10:29 #34695 merged -> #34704 (opened at cf6811ef59) stale 2 min into CI

release-provenance-helper.sh:400-403 refuses with "release checkout is not the exact origin/main tip" whenever any PR merges between the aggregate's CI start and the start of recovery.

What

Signed-tag aggregate recovery must be able to complete while ordinary PRs keep merging. That means removing the exact-tip starvation without weakening provenance. It also needs a supported way to retire a stranded, never-published lane, so that a fresh release is never blocked forever.

Why

  • release_lane_merge_guard (.agents/scripts/release-lane-helper.sh:1394-1400) has been a deliberate no-op since Snapshot releases without blocking concurrent PR merges #31472 (4904174, "Release immutable snapshots while PR merges continue"). Normal releases use immutable snapshots for that reason.
  • Recovery still requires HEAD == origin/main (release-provenance-helper.sh:399-403 via _release_provenance_resolve_source). That conflicts with the snapshot model and makes recovery a race the operator usually loses.
  • .agents/reference/release-aggregation-recovery.md:165 still says "Ordinary merges remain blocked during every recovery and committing phase". That is stale since Snapshot releases without blocking concurrent PR merges #31472.
  • Nothing can retire an unpublished lane. _full_loop_release_guard_competing_lane (full-loop-release-helper.sh:682-712) returns 75 for any other source unless the lane is an abandoned reservation or already published, and supersession (full-loop-release-reconcile.sh:1789-1807) requires a newer published tag. A stranded lane therefore blocks every new release indefinitely.

Tier

Selected tier: tier:thinking

Tier rationale: The trust boundary is release provenance. It requires choosing between snapshot-bound recovery and lane retirement, and rollback semantics must be decided.

How (Approach)

Decision options (recommended first)

  1. Snapshot-bound recovery. Accept a reviewed aggregation merge A that is an ancestor of origin/main, not necessarily its tip, when A's immutable trailers cover every first-parent commit from the previous baseline through A. This mirrors the existing snapshot mode (mode:"snapshot", snapshot_base in full-loop-release-reconcile.sh:644-647). The recovery bump commit already takes A as parent (reference doc step 4). The recovered-tag validator already checks the direct aggregate parent rather than the current tip (doc line 159). The protected release PR merge then has to preserve the release commit SHA (_merge_release_provenance_method). With this, merged chore(release): aggregate v3.39.31 recovery after #34682 and #34688 #34690 (b44b336c42) recovers v3.39.31 without another aggregate.
  2. Unpublished-lane retirement. Add aidevops release retire-unpublished <source-pr> --tag <tag> --reason <text>. It reuses the channel-absence probes (_full_loop_recovery_verify_*_absent), requires a dead or unknown executor, and makes a compare-and-swap write moving the lane to inactive with an auditable terminal receipt. The local unpublished tag is left inert. A fresh snapshot release can then proceed; version handling for the skipped number still needs a decision.

Files to Modify

  • EDIT: .agents/scripts/release-provenance-helper.sh:372-470: the snapshot-tolerant aggregate resolver for the signed-tag recovery path only.
  • EDIT: .agents/scripts/full-loop-release-aggregate-recovery.sh: the recovery entry (_full_loop_release_recover_aggregate at 1734) and the existing-context preparation.
  • EDIT: .agents/reference/release-aggregation-recovery.md:6-27,165: correct the stale merge-blocking claim and document the chosen mode.
  • EDIT: .agents/scripts/tests/test-full-loop-release-aggregate-recovery.sh: existing suite; extend only where the new acceptance path needs coverage.

Complete Write Surface

  • Callers/readers: full-loop-release-helper.sh:894-903 (recover-aggregate), release-provenance-helper.sh resolve-* subcommands, and the reconcile path in full-loop-release-reconcile.sh.
  • Writers/mutation paths: lane compare-and-swap writes in release-lane-helper.sh (begin/refresh/commit phases), local tag replacement in version-manager protected-main publication, and receipts.
  • Existing verification/tests: test-full-loop-release-aggregate-recovery.sh (12+ PASS cases, including "retains manifest and exact-tip fences") and test-release-provenance-helper*.sh.
  • Schemas/config: lane JSON on refs/heads/aidevops/release-lane. Option 2 adds a terminal receipt value.
  • Generated/deployed mirrors: ~/.aidevops/agents/scripts/ via setup.sh.
  • Migrations/backfills: option 1 needs no backfill. The persisted recovery transaction gains a pinned snapshot_base field only for new transactions; existing lanes on refs/heads/aidevops/release-lane remain exact-tip transactions. Option 2 adds a terminal receipt value that readers in full-loop-release-helper.sh:674-676 must recognise.
  • Cleanup/rollback paths: reverting .agents/scripts/release-provenance-helper.sh restores the exact-tip refusal. A lane left mid-transaction by the new mode must fail closed under old code, never be adopted, so the rollback cannot publish an unverified tag.

Implementation Steps

  1. Decide between option 1 and option 2 (option 1 recommended), and record the decision on the issue.
  2. Option 1: add a recovery-only resolver mode in release-provenance-helper.sh. It takes an explicit aggregate merge SHA A, verifies that A is an ancestor of origin/main, verifies A's aggregator and manifest trailers with the existing _release_provenance_verify_pr_record / _release_provenance_verify_aggregate_base, and returns mode:"aggregate" with source_merge:A. Keep the exact-tip check for every non-recovery caller.
  3. Pass the pinned A through _full_loop_release_recover_aggregate and persist it in the recovery transaction, so retries resolve the same base. Build the bump commit with parent A, as now.
  4. Correct release-aggregation-recovery.md:165 and document the snapshot-bound mode.
  5. Run shellcheck and the recovery and provenance test suites. Then run the live v3.39.31 recovery against b44b336c42 (chore(release): aggregate v3.39.31 recovery after #34682 and #34688 #34690) and verify convergence across GitHub, npm and Homebrew.

Hazards and Compatibility

  • Concurrency/atomicity: keep the compare-and-swap token rotation. The snapshot base must be pinned in the persisted transaction so a retry resolves the same A.
  • Migration/rollback: the existing exact-tip transactions stay valid, so the exact tip remains a special case of the snapshot.
  • Mixed-version/backward compatibility: older clients that see a snapshot-mode recovery transaction must fail closed, not adopt it.
  • Idempotency/retry: rerunning with the same aggregate must resume, not create a second bump commit.
  • Partial failure/recovery: the existing rule still applies. Once the release PR or remote tag is durable, nothing restores or mutates provisional state.

Verification Before Dispatch

shellcheck .agents/scripts/full-loop-release-aggregate-recovery.sh .agents/scripts/release-provenance-helper.sh
bash .agents/scripts/tests/test-full-loop-release-aggregate-recovery.sh
aidevops release recover-aggregate 34603 --tag v3.39.31 --expected-sources <#34690 manifest PR list>   # live: succeeds with main ahead of b44b336c42
  • Surface mapping: the tests prove the fences remain. The live run proves that the starvation case is resolved and that publication converges across GitHub, npm and Homebrew.
  • Broad verification trigger: release infrastructure change. Run .agents/scripts/linters-local.sh --changed plus the release test suites.

Files Scope

  • .agents/scripts/release-provenance-helper.sh
  • .agents/scripts/full-loop-release-aggregate-recovery.sh
  • .agents/reference/release-aggregation-recovery.md
  • .agents/scripts/tests/test-full-loop-release-aggregate-recovery.sh

Acceptance Criteria

  • Recovery succeeds when origin/main has advanced past a reviewed aggregation merge whose trailers fully cover the history up to that merge.
  • Recovery still refuses an aggregate whose trailers omit any first-parent commit up to the aggregate, a tag already on a remote or channel, and an aggregate that is not an ancestor of origin/main.
  • release-aggregation-recovery.md no longer claims that ordinary merges are blocked during recovery.
  • v3.39.31 publishes, and the GH#34600: fix: closed-issue hold re-comments every cycle under App read auth #34603 lane reaches a terminal published receipt.

Context & Decisions


aidevops.sh v3.39.31 plugin for OpenCode v1.19.0 with claude-opus-5-5 spent 6h 55m and 411,972 tokens on this with the user in an interactive session.

Activity

  1. added
    bugSomething isn't working
    tier:thinkingRoute at the thinking workload tier
    on Oct 14, 2026
  2. self-assigned this
    on Oct 14, 2026
  3. added
    status:claimedInteractive implementation is actively claimed
    and removed on Oct 14, 2026
  4. marcusquinn commented on Oct 14, 2026

    @marcusquinn
    OwnerAuthor

    Interactive session claimed by @marcusquinn on Mac.
    Pulse dispatch blocked via status:claimed + self-assignment.


    aidevops.sh v3.39.31 plugin for OpenCode v1.19.0 with claude-opus-5-5 spent 6h 55m and 411,972 tokens on this with the user in an interactive session.

  5. added
    status:in-reviewNon-draft PR ready for review/merge
    and removed
    status:claimedInteractive implementation is actively claimed
    on Oct 15, 2026
  6. removed their assignment
    on Oct 15, 2026
  7. added and removed
    status:in-reviewNon-draft PR ready for review/merge
    on Oct 15, 2026
  8. github-actions commented on Oct 15, 2026

    @github-actions
    Contributor

    Completed via PR #34743. merged to main.


    aidevops.sh vunknown automated scan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingorigin:interactiveCreated by interactive user sessionsolved:interactiveTask was solved by an interactive sessionstatus:doneTask is completetier:thinkingRoute at the thinking workload tier

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions