diff --git a/.agents/skills/agent-tuning/SKILL.md b/.agents/skills/agent-tuning/SKILL.md new file mode 100644 index 000000000..adc20bfd2 --- /dev/null +++ b/.agents/skills/agent-tuning/SKILL.md @@ -0,0 +1,15 @@ +--- +name: agent-tuning +description: Use when changing agent model or effort configuration, adapter mappings, or eval candidate profiles. +user-invocable: false +metadata: + internal: true +--- + +**Unified Agent Tuning (`internal/agentcfg`)** + +- `agentcfg` is the single owner of the harness-neutral model/effort surface and of the mapping down to each harness's native mechanism (claude/copilot `--effort`, codex `-m` + `-c model_reasoning_effort`, grok `--reasoning-effort`, pi `--thinking`, opencode's session-message `model`/`variant`, acpx `--model` for `cursor`/`acp:`). Add a harness there, not in an adapter or in eval. `rovodev` and `antigravity` are deliberately declared unmappable, so a request for them is a config error rather than a flag that is silently ignored. +- `agent.NewWithOptions` is the one funnel: it validates `Options.Profile` and splices the mapped args after the operator's raw `agent_args_override` args, so both the pipeline (`cfg.AgentProfileFor`) and eval replay (`Candidate.Profile()`) reach every harness by the same path. Never re-derive a model or effort flag at a call site. +- Precedence is fixed: a raw `agent_args_override` flag that already pins a knob natively wins and the mapped value is not emitted, which is what keeps every pre-`agent_config` configuration byte-identical and stops a harness receiving one knob twice. `agent_config` is global-only for the same reason as `agent_args_override`. +- Eval candidates are `agent,model=[,effort=]` (the previous `agent+model` spelling is refused with a migration message), effort is part of the persisted candidate identity, and `agentNeutralGlobalConfig` strips `agent`, `agent_args_override`, and `agent_config` so a replay never inherits the capturing machine's pins. +- Regressions: `internal/agentcfg`, `internal/agent/profile_test.go`, `internal/config/config_agent_config_test.go`, `internal/daemon/pipeline_agent_profile_test.go`, `TestParseCandidate*`, `TestReplayPinsCandidateModelAndEffortOnTheHarness`, `TestCaptureStripsEveryHarnessPinFromThePinnedConfig`. diff --git a/.agents/skills/branch-sync-and-push-safety/SKILL.md b/.agents/skills/branch-sync-and-push-safety/SKILL.md new file mode 100644 index 000000000..1460ed3f3 --- /dev/null +++ b/.agents/skills/branch-sync-and-push-safety/SKILL.md @@ -0,0 +1,38 @@ +--- +name: branch-sync-and-push-safety +description: Use when changing local branch synchronization, custody recovery, post-review head binding, rebasing, or force-push safety. +user-invocable: false +metadata: + internal: true +--- + +**Guarded Local Branch Synchronization (`internal/branchsync`)** + +- `sync`, `axi sync`, and the TUI `u` action share one service whose only ordinary worktree mutation is a clean guarded move to an exact freshly verified pipeline push binding: strict fast-forward for behind branches, or an anchored reset to an equivalent diverged pipeline head when local unique work is already represented there. Under `--recover`, the worktree can only strict-fast-forward to the gate-preserved head, or adopt a diverged preserved head that `preservedContainsLocalWork` proves carries every local change. Passive status never fetches, and blocked states never reset, stash, merge, rebase, force, switch, delete, or update an external remote. +- Give each network remote operation its own bounded child context derived from the caller: `Refresh` must not share one deadline across sequential `git.LsRemote` and `git.FetchRemoteBranchToPrivateRef` calls, and `Apply` uses the same per-operation budget for its final live check. The per-operation budget is `Service.RemoteTimeout`, sourced only from the operator's global `branch_sync_remote_timeout` setting (default `config.DefaultBranchSyncRemoteTimeout`, 60s); `RepoConfig` deliberately has no matching field. `Recover`'s local-gate fetch is outside this network deadline contract. Regressions: `TestRefreshSlowSuccessfulLsRemoteDoesNotStealFetchBudget`, `TestRefreshSlowButSuccessfulLsRemoteAloneExceedsItsOwnBudgetReportsOffline`, `TestRefreshRaisedRemoteTimeoutAcceptsTheSameLegitimateSlowLsRemote`, `TestRefreshParentCancellationStopsFetchAfterLsRemoteSucceeds`, `TestServiceRemoteTimeoutDefaultsToConfigDefault`, `TestLoadGlobal_InvalidBranchSyncRemoteTimeout`, `TestLoadRepo_BranchSyncRemoteTimeoutIsNotARepoSetting`. +- Successful pipeline pushes persist the exact SHA, credential-free target fingerprint/ref, and generation; legacy rows remain nullable and must never infer provenance from mutable `head_sha`. Structured PR lifecycle retires merged/closed branches. The service rechecks the invoking worktree, target, live remote equality, ancestry or equivalent-divergence proof, generation, and all mutable assumptions immediately before apply. +- A TERMINAL run with unpublished pipeline commits (moved head) is recoverable only from verified, non-conflicting evidence: inspection and `Recover` share one eligibility model. Equal/ahead local ancestry can create the local anchor without requiring gate access, but available gate evidence must agree; importing a missing preserved head requires exact or safely anchorable gate evidence, a clean worktree, and either ancestry or the content-preservation proof below. Only then does inspection report `blocked_pipeline_owned_recoverable` + `next_action recover_custody` with the exact submitted/current-head and relation facts (active runs keep the plain block). Missing, non-commit, symbolic, or conflicting evidence, and import cases that are dirty or genuinely divergent, never advertise `recover_custody`; where the record is self-inconsistent (missing, non-commit, or conflicting evidence) inspection names the keep-local settlement below, and every other shape falls back to manual reconciliation. `sync --recover` anchors the preserved head at `refs/no-mistakes/recover/` before stamping `runs.custody_returned_at`. + Cancellation RELEASES a terminal run that never changed the submitted head (`head_sha == submitted_head_sha`, no push, no custody stamp): selection keeps it visible so it never misreports as `blocked_wrong_branch`, and it classifies `user_owned` - no `next_action`, non-blocking exit, never represented as recoverable custody, `--recover` there is an idempotent no-op that mutates nothing, and a fresh `axi run` or separately authorized direct push is never blocked. + Equal/ahead worktrees anchor locally without requiring gate access, but an available gate's existing recovery ref must agree with the recorded head; behind/diverged worktrees verify and fetch the preserved head from the run-specific recovery ref, fast-forwarding only a clean behind worktree. + A cancelled validation routinely leaves a preserved head that is a REBASE of the local branch, which equality and ancestry read as plain divergence, so a clean diverged worktree is adopted when `preservedContainsLocalWork` proves containment. That proof is an executable `merge-tree` three-way merge whose result must equal the preserved head's tree, anchored on the merge-base - never `runs.base_sha`, the previous gate head. It deliberately does NOT use patch identity: patch IDs discard hunk locations and whitespace, so they cannot tell a genuine replay from a same-shaped edit to another identical block, and a containment claim built on them is not a proof. Everything undecidable escalates, including a rebase whose fix rounds also rewrote operator lines, where nothing separates a deliberate fix from a dropped change. + Adoption anchors the pre-recovery local head at `refs/no-mistakes/recover-local/`, then moves the branch with Git operations that fail closed on their own rather than after an observation - an atomic `update-ref` CAS plus `read-tree -m -u`, never check-then-act followed by `reset --hard`, which destroys anything landing in the gap. `recoverAdoptPreserved` owns the reasoning. + Terminalization pins every verified unpublished head at `refs/no-mistakes/recover/` before the managed worktree can be removed. Recovery reads that run-specific ref rather than requiring the gate branch to match, so aborts, rebases, and pre-push failures remain recoverable while an independently moved gate branch is preserved. Legacy recorded heads that still exist as dangling gate objects are anchored on recovery; a truly missing recorded head never advertises an impossible `recover_custody` command and instead reports the keep-local settlement below, falling back to manual reconciliation whenever that settlement is itself unreachable. + When the operator keeps a behind or diverged local head instead of taking the preserved head, `--keep-local` never touches the worktree and CAS-moves the gate branch to the kept head, staging objects via gate-side fetch - never a push, which would fire the receive hook and start a run. The CAS is reached only where the gate branch still names a different head; equal/ahead, `user_owned`, already-recovered, and absent-gate-branch paths return custody without moving any branch ref (equal/ahead still writes the private recovery anchor, which is what anchoring a locally reachable preserved head means), and the flag help must keep saying so. + `--keep-local` is also the ONLY settlement for a SELF-INCONSISTENT custody record (#824): a terminal run whose recorded pipeline head is in no reachable object store, or whose own recovery evidence names something else, has nothing verifiable to import, so every default recovery refused, `abort` of the terminal run was a no-op, and the branch stayed `pipeline_owned` forever. Inspection names that exit (`next_action.code: return_custody_keep_local`) instead of #814's dead-end manual-reconciliation pointer, terminal-run `abort` responses carry the same command, and the TUI `u` action reaches the same settlement - keyed on that advertised next action, never on a safety code, so it cannot drift from the predicate that decides where the settlement can complete. #814's polarity (never advertise `recover_custody` for an unverifiable record) is preserved. Terminal-run no-op abort help is separately allowlisted to custody-settlement codes only (`custodySettlementHelp`) AND gated on `StatePipelineOwned`, so an abort that cancelled nothing never answers by prescribing a fresh run against a released branch, nor by repeating ordinary-divergence `git log` advice; a SUCCESSFUL cancellation still echoes the branch's own next action. + Three invariants make that safe, and the `recoverSettleInconsistent`, `recoverKeepLocal`, and `selfInconsistentCustodyRecord` doc comments own the mechanics and the per-shape rationale. First, the settlement is fail-closed and never a shortcut past unique content: every reachable copy of the recorded head is pinned before anything moves, a head that still exists but cannot be pinned refuses, the gate moves only by CAS, and the settlement's own refusals plus every refusal raised while keep-local moves the gate branch name `inspect_and_reconcile_manually` (`blockedPlan` nils `NextAction`, so this is per-site via `recoverBlocked`, never a global guarantee). The one failure that is not a refusal gets the same treatment: `finishRecover`'s stamp failure runs AFTER the Git side already succeeded, so it reports that the changes are applied and names the same recovery command again (`recoveryRetryAction`), which completes because every Git step it repeats is idempotent once applied. Second, absence must be PROVEN, not inferred: `git.CommitPresence` treats ONLY git's exit 1 (the store was read and the object is not there) as an absence, because the settlement's whole safety argument is "nothing still has this head, so nothing can be lost". An unreadable store and a present-but-wrong-type object (`cat-file -e` exits 0 for a tree, blob, or tag) are both undetermined and refuse; collapsing either into "absent" let the settlement complete while the object was still there. `settlementAnchorsFree` mirrors the same probe so the advertisement agrees with the write. Third, the predicate may name the settlement only where `Recover(keepLocal)` reaches it AND it can complete, so unverified (#707's scope), uninspectable, symbolic, and mid-adoption evidence, plus an unreadable gate branch or an occupied stranded/gate anchor, all fall back to manual reconciliation. Advertising any of those recreates the very wedge this change removes. + Write ordering inside `recoverKeepLocal` is load-bearing: `refs/no-mistakes/recover-gate/` guards exactly the CAS that moves the gate branch off the displaced head, and nothing before that swap can strand it, so the anchor is written immediately before the CAS and every refusal that can precede it leaves no ref of ITS OWN behind - which is what makes that half of the claim true by construction rather than by a cleanup that could itself fail. A refusal only speaks for the whole attempt if it also carries what its CALLER wrote, so `recoverKeepLocal` takes an anchor note (`keepLocalNoChangeClause`): a delegation that anchored the preserved head at `refs/no-mistakes/recover/`, or a settlement that pinned it at `refs/no-mistakes/recover-stranded/`, makes every PRE-SWAP refusal report "no branch, worktree, or file changes were made" plus where that anchor now is, and only a delegation that wrote nothing makes the blanket "no files or refs were changed" claim. The lost CAS is excluded from that substitution on purpose: it has already written the gate anchor, so it keeps its own narrower "no LOCAL files or refs were changed" claim and APPENDS the note. The anchor CONFLICT check stays first because it is read-only. Exactly one post-write refusal remains, the lost CAS, and it deliberately keeps the anchor: the swap failed because the gate moved, so that pin may be the only ref still naming the displaced head, and its message says "no LOCAL files or refs were changed" and names the ref to reconcile. + The full relation matrix and fail-safe rules live in the `Recover` doc comment in `internal/branchsync/sync.go`. +- Public guidance is owned by `internal/skill/skill.go` plus live AXI strings, then regenerated with `make skill`. Core regressions live in `internal/branchsync` (incl. `recover_test.go`, whose `wedgedCustodyFixture` is the #824 shape), `internal/cli/sync_test.go`, `internal/cli/axi_abort_custody_test.go`, `internal/cli/axi_settle_custody_test.go`, `internal/tui/branch_sync_test.go`, and e2e `TestAxiBranchSyncJourney` / `TestAxiCustodyRecoveryJourney` / `TestAxiCustodyRecoveryAfterRebaseJourney` / `TestAxiPrePushAbortUnmovedHeadCustodyJourney`. + +**Post-Review Head Continuity and Push Binding** + +- Every step after Review in the fixed pipeline order (Test, Document, Lint, Push, PR, CI) calls `assertPipelineHeadContinuity` at entry. The helper is the single semantic owner: equal or descendant live heads continue; backward, sibling, and unverifiable heads fail before the step performs work. Regression: `TestPostReviewStepsRefuseHeadClobberAtEntry`. +- A successfully completed full review atomically records `runs.review_approved_head_sha`; parked, failed, skipped, and legacy reviews carry no inferred authority. Push reads that durable binding, permits only the exact commit or a descendant, and pushes the verified immutable SHA rather than mutable `HEAD`. Never infer approval from `runs.head_sha`, a worktree, gate ref, or remote branch. Regressions: `TestPushStep_RefusesPostReviewClobberWithoutLaterPipelineCommit`, `TestPushStep_BindsRemoteAndDatabaseToVerifiedCommitWhenHEADMovesDuringPush`, `TestExecutor_FullRereviewReplacesApprovalWithoutAuthorizingParkedRound`. + +**Rebase Base & Force-Push Safety (data-loss prevention)** + +- The whole job of this tool is to not lose people's code; favor refusing the push and surfacing a finding over any clever recovery. The comments in `internal/pipeline/steps/forcepush.go` own the full reasoning; the invariants are the next three bullets. +- Rebase bases come from the freshly fetched authoritative remote refs, never local or stale state; and a branch built on unpushed local-default-branch commits parks with `NeedsApproval` + `AutoFixable=false` instead of silently widening the PR (`detectBundledLocalDefaultCommits`, #283). +- Every force-push routes through `resolveForcePushDecision`, which re-reads the live remote head and allows the push only for a new branch, an already-equal remote, an unchanged `lastSeenSHA`, or remote commits already incorporated by patch-id (excluding `^baseSHA` history the run knowingly rewrites). Anything else refuses, and a failed ls-remote/fetch fails closed; never degrade to a bare `--force`/`--force-with-lease` without an explicit anchor. +- `lastSeenSHA` must stay the head the run last **observed** (from run/prior-run push provenance or the remote-tracking ref), never the live remote tip: the rebase step refreshes `origin/` only on a normal push, NOT on a force push. CI repairs commit locally and restart validation at Review; the later Push step owns their remote update and force-push safety. Anchoring a lease to a SHA read immediately before pushing is the original #281 bug (it always passes and protects nothing); always-fetching the branch on force push recreates it. Never reintroduce either. +- Regressions: `TestPushStep_RefusesToClobberAdvancedUpstreamBranch` (#305), `TestForcePushRun_RefusesToClobberOutOfBandBranchCommit`, `TestRebaseStep_DetectsUnpushedLocalDefaultBranchCommits` (#283), `TestResolveForcePushDecision_*`, `TestExecutor_CIRestartRevalidatesBeforePush`, `TestPushStep_AllowsForcePushAfterMidRunRebaseOverPriorPushedGeneration` (#837), `TestPushStep_AllowsForcePushOnRerunOverPriorRunPushedGeneration` (#837). diff --git a/.agents/skills/ci-monitor/SKILL.md b/.agents/skills/ci-monitor/SKILL.md new file mode 100644 index 000000000..f2dace818 --- /dev/null +++ b/.agents/skills/ci-monitor/SKILL.md @@ -0,0 +1,20 @@ +--- +name: ci-monitor +description: Use when changing CI readiness, forge check collection, reruns, CI timeouts, or PR lifecycle monitoring. +user-invocable: false +metadata: + internal: true +--- + +**CI Monitor Lifecycle** + +- `ci_timeout` is an idle timeout, not an absolute deadline: only `timeoutAnchor` re-arms when the upstream default-branch tip advances, `started` stays fixed for poll pacing, and re-arm only ever extends the deadline (fail-safe on transient base-tip failures). Value semantics (`0` unset, negative unlimited sentinel, keyword parsing) live in `config.go`; keep `config.DefaultCITimeout` and `defaultConfigYAML` in sync (`TestDefaultConfigYAML_MatchesGoDefaults`). User-facing semantics are owned by `docs/src/content/docs/reference/global-config.md`. +- GitHub readiness is the union of the exact current PR head commit's check rollup and every Actions workflow run returned by the Actions API for that same SHA. A workflow rejected before creating jobs/check-runs is absent from the commit rollup but still present in that API; run discovery errors and unknown run states fail closed instead of certifying a green rollup. Regressions: `TestGetChecksIncludesFailedWorkflowRunMissingFromPRRollup`, `TestGetChecksBindsRollupAcrossABAHeadMovement`, `TestCIStep_FailedHeadWorkflowRunPreventsChecksPassed`. +- GitHub's raw commit `statusCheckRollup` returns every check run a commit ever had, including a same-named run a later run at that same head has already superseded (e.g. a required-check job re-triggered by `synchronize` after a pipeline auto-fix push). `Host.GetChecks` (`internal/scm/github/github.go`) collapses same-name reruns of one workflow to the newest `startedAt` (using Actions run identity to break timestamp ties, comparing against every retained duplicate, and preserving unordered records, including ambiguous pending replacements, before a `completedAt` fallback) while preserving independent workflows, unidentified external checks, and same-named commit status contexts, restoring the semantics `gh pr checks` already applies and this repo's own gate `verify.py` already assumes (last-wins). Collapse must run AFTER `appendUnrepresentedWorkflowRuns`, never before: that call dedupes the Actions-run union against checks by run ID over the FULL uncollapsed rollup, so collapsing first drops a superseded run's ID out of the "represented" set and the union re-adds the same stale run under its own workflow run name. Without this collapse, a monitor that keeps `synchronize` in its gate trigger set can loop forever: each auto-fix push legitimately fails the gate at the old head, a later same-head rebind turns it green, but the stale FAILURE stays visible and re-triggers another auto-fix round. Regressions: `TestGetChecksCollapsesSupersededSameNameCheckToLatestAtOneHead`, `TestGetChecksCollapseOrderingDoesNotLetWorkflowRunUnionResurrectSupersededCheck`, `TestGetChecksPreservesIndependentSameNameWorkflows`, `TestGetChecksPreservesIndependentSameNameExternalCheckRuns`, `TestGetChecksCollapseComparesNewestRunWithEverySameNameCandidate`, `TestGetChecksKeepsQueuedReplacementWithEqualStartTime`, `TestGetChecksPreservesUnorderedExternalPendingReplacement`, `TestGetChecksPreservesSameNameStatusContextAndCheckRun`. +- CI readiness never treats an unproven empty forge check list as green. Ready requires observed all-green checks, or trusted default-branch `no_ci: true` with zero registered checks (`internal/pipeline/steps/ci.go` decides whether the declaration applies; `internal/cimonitor` owns the agent-facing log vocabulary and Ready/DeclaredNoCI parse). Delayed registration, pending checks, failures, errors, unknowns, and stale-head evidence stay not-ready; registered checks on a declared no-CI repo are still honored. Regressions: `TestChecksPassed_PR607RealLogSequence`, `TestCIStep_EmptyChecksWithoutNoCIStaysNotReadyPastOldGracePeriod`, `TestCIStep_EmptyChecksWithTrustedNoCIBecomesReady`, `TestCIStep_DelayedCheckRegistrationStaysNotReadyUntilGreen`, `TestCIStep_DeclaredNoCIWithUnexpectedChecksHonorsThem`, `TestEffectiveRepoConfig_NoCITrustedOnly`. +- Persistent provider check-read failures are not an invisible spin: 6 consecutive `GetChecks` errors while the PR is still open park the CI step at an ask-user gate whose provider-neutral finding names the provider CLI/credentials support and includes the underlying error (with a GitHub-specific clause requiring `gh` >= 2.50 for the `gh pr checks --json` call), and the streak resets on any successful read. Regression: `TestCIStep_PersistentCheckReadFailureParksAtAskUser`. +- Reap an orphaned monitor from outside its worktree with `no-mistakes axi abort --run `; it needs only `NM_HOME` and never starts a stopped daemon. A known run succeeds only with durable terminal truth, a recorded nonterminal run fails unconfirmed, and only an unknown id is an idempotent no-op. Bare `axi abort` stays worktree/branch-scoped. +- A merged or closed PR observation transactionally completes an active run and its CI step; PR lifecycle state is monotonic, so duplicate or delayed observations cannot reactivate or regress a terminal run. Startup reconciles legacy `pending` or `running` rows that already hold terminal PR state before parked-run planning and generic crash recovery. Regressions: `TestUpdateRunPRStateFinalizesActiveTerminalOutcomes`, `TestUpdateRunPRStateIgnoresDuplicateAndDelayedRegressions`, `TestReconcileTerminalPRRunsFinalizesLegacyActiveRows`, `TestRecoverOnStartup_FinalizesLegacyTerminalPRRun`, e2e `TestTerminalPRRunDisappearsFromActiveListing`. +- A provider-reported `cancelled` check is never a job verdict, so the deterministic rerun runs strictly before any CI fix round: cancellation and a GitHub pre-run infrastructure failure (a job that failed in setup/action resolution before any repository step ran, flagged `Check.PreRunFailure` by the opt-in `scm.PreRunFailureDetector` and re-bucketed to cancel so it never masks a real test/lint failure, which cleared setup and failed a later step) are the outcomes that earn a rerun, either outcome after its budget parks as `ask-user` instead of entering the `auto_fix.ci` loop, and any genuine or unrecognized failure or merge conflict in the same poll suppresses reruns so real failures still escalate on their first observation. The budget is per check name per run and spent on request rather than on success, and a rerun is never issued once the published branch head no longer equals `runs.head_sha`, because it would certify a commit this run never delivered. Each outstanding rerun records its verified pipeline head and the same-name provider links visible when it was requested; it retires durably when the run head advances or a new conclusive non-cancel link appears. Retirement keeps the spent budget and never changes check buckets. A delayed same-named green sibling can satisfy the link trigger on the same head, matching the default branch's existing name-keyed masking; removing that limitation requires provider truth outside this policy. Classification, the deliberate `TIMED_OUT`/`STALE` exclusions, rollup-lag grace, and retirement live in `internal/pipeline/steps/ci_transient.go`; provider support is the optional `scm.CheckRerunner` (GitHub only), pre-run infrastructure detection is the optional `scm.PreRunFailureDetector` (GitHub only, folded in by `markPreRunInfraFailures`), and user-facing semantics are owned by `docs/src/content/docs/reference/repo-config.md`. Regressions: `TestCIStep_CancelledCheckIsRerunBeforeEscalating`, `TestCIStep_CancelledCheckStaysUnresolvedAfterItsBudget`, `TestCIStep_LaggingRerunRollupKeepsWaitingForTheRepublishedCheck`, `TestCIStep_SameHeadGreenRerunEmitsChecksPassed`, `TestCIStep_DelayedSameNameCheckRetainsLegacyNameBehavior`, `TestCIStep_ResolvedRerunDoesNotParkALaterGreenHead`, `TestRetireResolvedReruns`, `TestRetireResolvedRerunsRetriesAfterPersistenceFailure`, `TestCIStep_MovedPublishedHeadTerminatesInsteadOfRerunning`, `TestCIStep_MovedPublishedHeadClearsCIReadiness`, `TestClassifyCheckFailure`, `TestMarkPreRunInfraFailures_RetriesInfraButNotGenuine`, `TestMarkPreRunInfraFailures_OptInGated`, `TestPreRunFailures_FlagsSetupFailureNotGenuine`, `TestPreRunFailures_FailsClosedOnUnreadableRun`. +- **Terminal is not pending.** Readiness must reject every non pass/fail/skip bucket (`hasUnresolvedChecks`), but only checks that can still finish on their own (`hasPendingChecks`) may keep the monitor polling. A `cancel` bucket - GitHub `CANCELLED`, GitLab `canceled`, Bitbucket `STOPPED`, and how GitHub reports a job killed by its own `timeout-minutes` - is a published conclusion that nothing will replace, so with no rerun outstanding it parks at `ciUnresolvedCancelledOutcome` (`cancelledWithoutRerun`) instead of waiting. Conflating the two is the #628 regression that hung real runs for their whole `ci_timeout`; an unrecognized bucket is deliberately still treated as waiting, because unknown is not evidence of terminal. Regressions: `TestCIStep_CancelledCheckAmongPassingChecksEscalatesInsteadOfPollingForever`, `TestCIStep_ZeroRerunBudgetEscalatesCancelledCheckWithoutMakingItReady`, `TestCIStep_BitbucketStoppedCheckParksForADecision`. +- CI readiness is read from the provider's live PR head check rollup on every poll, so it always describes the head the forge currently has for that PR; no recorded SHA gates it, and a run whose row still names a pre-advance commit must still recognize green at the head the pipeline last pushed. Regression: `TestCIStep_GreenChecksAtAdvancedHeadAreRecognizedWhileRunTracksOlderHead`. diff --git a/.agents/skills/daemon-runtime/SKILL.md b/.agents/skills/daemon-runtime/SKILL.md new file mode 100644 index 000000000..fef4c3cdc --- /dev/null +++ b/.agents/skills/daemon-runtime/SKILL.md @@ -0,0 +1,36 @@ +--- +name: daemon-runtime +description: Use when changing daemon startup, singleton ownership, shutdown, logging, event subscriptions, or lifecycle commands. +user-invocable: false +metadata: + internal: true +--- + +**Daemon Singleton Lock (`internal/daemon/lock.go`)** + +- Only one live daemon may own an `NM_HOME`: an exclusive OS file lock on `/daemon.lock` is acquired as the very first action in `RunWithOptions`, strictly before stale-run recovery and socket bind, and held for the process lifetime. The kernel releases it on any process death, so a held lock always means a live holder and no staleness heuristic is needed. Without it, a second daemon stole the socket and ran global crash recovery against the live daemon's runs and worktrees. +- Process launch is not readiness: the PID record is published after the singleton lock and before exclusive recovery, while startup succeeds only after a real IPC health response. The 45s production budget covers cold environment setup and recovery; early exits fail promptly, timeout cleanup reaps detached children before fallback or rollback, and managed plus detached failures retain both causes. Regressions: `TestStartDetachedDaemonDetectsChildExitPromptly`, `TestStartDetachedDaemonTimeoutKillsAndReapsChild`, `TestStartPreservesManagedAndDetachedFallbackErrors`, `TestColdDetachedStartupProductionGateCardinality`. +- A successful stop means the daemon process is gone, not merely that IPC health has disappeared, because only process exit releases the singleton lock. Capture the daemon instance before requesting shutdown, and close the shutdown client before waiting because the daemon drains in-flight handlers during exit. See `waitForDaemonStop` and `stopDetachedDaemon`; regressions: e2e `TestDaemonStopLeavesNoDaemonProcessOwningTheRoot`, `TestDaemonRestartReplacesTheDaemonWithExactlyOneOwner`. +- Independent layers: `internal/ipc` `listen()` dials the socket before unlinking it and refuses to steal a live one; client probes bound the dial with `daemon_connect_timeout` and fail fast on a dead or wedged socket instead of starting a replacement daemon (`EnsureDaemon` surfaces the error with a `daemon start` recovery hint; the health RPC itself is bounded separately by `ipc.DefaultDialTimeout`). +- Daemon execution is explicit-only (`no-mistakes daemon run --root`); never let inherited environment reinterpret probes like `--version` or `status` as daemon workers. +- Startup worktree cleanup is DB-aware: never remove a worktree whose run row is `pending` or `running`; `startRun` inserts the run row before creating the worktree, so a no-row directory is safe to remove immediately. That no-row rule holds only inside `/worktrees`, which is discovered by walking because no-mistakes owns it; a configured worktree root is the operator's directory, so cleanup and eject there act on exactly the recorded run worktrees and never enumerate anything else. +- The user-facing model lives in `docs/src/content/docs/concepts/daemon.md`; the lock rationale lives in the `internal/daemon/lock.go` and `daemon.go` comments. Regressions: `TestAcquireSingletonLock_*`, `TestRunWithResources_SecondDaemonForSameRootFailsWithoutStealingSocket`, `TestRunWithOptions_RequiresSingletonLockBeforeRecovery`, `TestRecoverOnStartup_DoesNotDeleteActiveRunWorktree`, `TestServe_SecondListenerForLiveSocketDoesNotStealIt`, `TestDialConnectTimeoutFailsFastAndNamesSocket`, `TestIsRunningFailsFastWhenSocketAcceptsButDoesNotRespond`, `TestIsRunningSurfacesExistingDeadSocket`, `TestDaemonRunRootFromArgs_EnvDoesNotForceDaemonModeForProbes`, `TestValidateDaemonPIDFallback_RefusesToKillOwnProcess`. + +**Bounded Daemon Logging and Event-Driven AXI Runs** + +- `internal/logstore` owns all daemon-process byte and retention bounds. Lifecycle output uses `logs/daemon.log`, managed Rovo Dev/OpenCode stdout and stderr use `logs/managed-server.log`, and service bootstrap/direct crash output uses `logs/daemon-bootstrap.log`. Rotation snapshots backups and truncates the current inode in place so held service and child descriptors keep writing to the bounded current file. Regressions: `internal/logstore/rotate_test.go`, `TestDetachedDaemonUsesBoundedDedicatedLogSinks`, `TestManagedServerOutputIsSeparatedFromLifecycleFailureSummary`. +- Successful read-only IPC methods are DEBUG; mutations and stream starts are INFO; every request failure is WARN. AXI run driving is subscribe-first and `internal/cli/run_reconciler.go` is the sole owner of event reconciliation, reconnect, duplicate-event coalescing, and the slow lost-event heartbeat. Do not reintroduce fixed-interval `get_run` polling. Regressions: `TestSuccessfulReadRequestsDoNotLogAtInfo`, `TestRequestLoggingKeepsMutationsAndFailuresVisible`, `TestDriveRun_HealthyWaitStaysWithinRequestBudget`, `TestRunReconciler_*`. + +**Bounded Loss-Aware Event Subscriptions** + +- `internal/ipc/events.go` (`ClassOf`) is the single event taxonomy: activity is droppable, state is not, control is broker-generated, and an unrecognized type fails safe to state. Brokers and consumers must read loss tolerance from it rather than re-listing event names. +- `internal/daemon/eventmailbox.go` is the single overflow owner: a per-subscriber ring bounded by 64 events and 1 MiB, non-blocking publish (the executor is never stalled), activity as the only evictable class, and everything else folded into one sticky coalescing `stream_gap` that drains ahead of queued payload. A reserved slot is not enough - it fails at the second simultaneous transition - and producer-side channel receives race the reader, which is why the queue is a ring under a mutex. +- Every state event and every `get_run` snapshot carries a monotonic `StateRev`; `runSnapshot` samples it **before** the DB read, which is sound only because every producer writes state and then emits. Consumers apply a delta only when its revision is newer, so a delta queued before a snapshot cannot regress state after it. Every subscription opens gapped, so attach and reconnect always reconcile first. +- The fix-review working-tree diff is the only gate context that is never persisted, so it is served on demand by `ipc.MethodGetStepDiff` (`RunManager.StepDiff`, bounded at 512 KiB) instead of riding the stream: it was the only unbounded payload, and one frame past the 1 MiB transport line limit ends the subscription and hides every later event. +- Regressions: `internal/daemon/eventmailbox_test.go` (A1-A13 plus the byte/count ceilings), `TestRunSnapshot_*`, `TestStepDiff_*`, `TestExecutor_StateEventsAreEmittedAfterTheirDatabaseWrite`, `TestClassOfUnknownEventFailsSafeToState`, `TestRunReconciler_StreamGapForcesOneAuthoritativeRead`, `TestSubscribeOversizedFrameEndsTheStreamAndHidesLaterEvents`, `internal/tui/overflow_contract_test.go`. + +**Destructive Daemon Lifecycle Guard (`internal/lifecycle/guard.go`)** + +- `daemon stop`, `daemon restart`, and `update` refuse by default while pending/running runs exist (the daemon is machine-wide, so stopping it can fail every active pipeline), list the runs via the shared `lifecycle.ActiveRuns`/`lifecycle.RunList` helpers, and require an explicit `--force`. `update -y` answers only the different-executable prompt and deliberately does not bypass this guard. +- Every invocation of the three commands is logged with caller attribution (PID, PPID, parent command line) via `logLifecycleInvocation` to `/logs/cli.log`; this is the incident forensic trail, do not remove or weaken it. +- Regressions: `TestDaemonStopRefusesWithActiveRunsAndListsThem`, `TestDaemonStopForceOverridesActiveRunGuard`, `TestDaemonRestartRefusesWithActiveRuns`, `TestLifecycleCommandsWriteCallerAttributionToCLILog` (`internal/cli/daemon_lifecycle_test.go`), `TestUpdaterRunRefusesWithActiveRunsAndListsThem`, `TestUpdaterActiveRunGuardAllowsForce` (`internal/update`). diff --git a/.agents/skills/documentation-guidance/SKILL.md b/.agents/skills/documentation-guidance/SKILL.md new file mode 100644 index 000000000..c5dbcf4b6 --- /dev/null +++ b/.agents/skills/documentation-guidance/SKILL.md @@ -0,0 +1,21 @@ +--- +name: documentation-guidance +description: Use when changing documentation ownership, generated agent guidance, or review auto-fix guidance. +user-invocable: false +metadata: + internal: true +--- + +**Documentation** + +- Keep `README.md` concise and high-level; the bar needs to be extremely high for what shows up there. +- Most documentation lives in `docs/`, the published docs site. +- One owner per fact: `docs/src/content/docs/reference/global-config.md` and `docs/src/content/docs/reference/repo-config.md` own configuration keys, `docs/src/content/docs/reference/environment.md` owns environment variables and the telemetry local/remote split, `docs/src/content/docs/concepts/daemon.md` owns the daemon lifecycle model, and guides pages explain purpose and link to those owners instead of restating tables and examples. +- The `document.instructions` block in `.no-mistakes.yaml` states this ownership map for the pipeline's document step; update it when ownership moves. + +**Agent-Guidance Surfaces** + +- `skills/no-mistakes/SKILL.md` is **generated**: the source of truth is the `body` constant in `internal/skill/skill.go`. Edit the body, then `make skill`; `make lint` fails CI on drift. Never edit `SKILL.md` directly. `no-mistakes init` ships this rendering to agents at user level. +- Agent-driving guidance is owned by the skill body and the live `axi` output strings (`internal/cli/axi*.go`); `docs/src/content/docs/guides/agents.md` carries only the canonical invariant sentences pinned by `internal/cli/axi_guidance_test.go` plus a pointer to the skill. When you change driving guidance, change the skill body and the point-of-use `axi` strings together; that drift test is the sync check. +- The shared default test-quality rule lives in `internal/testguidance`; render it only into the task-first skill and pipeline roles that can author, repair, or review tests. Its fake-agent prompt tests are the intentional generated-interface contract, not source-text checks. +- Review auto-fix is disabled by default (`auto_fix.review: 0` in `config.go` `autoFixDefaults`), so blocking and ask-user review findings park for an agent decision; keep the skill, the live `axi` gate `note`, and docs qualified if you touch review auto-fix. diff --git a/.agents/skills/eval-corpus/SKILL.md b/.agents/skills/eval-corpus/SKILL.md new file mode 100644 index 000000000..b26b94db3 --- /dev/null +++ b/.agents/skills/eval-corpus/SKILL.md @@ -0,0 +1,19 @@ +--- +name: eval-corpus +description: Use when changing local eval capture, gold labels, diversified and tune sets, matching, replay storage, or eval CLI behavior. +user-invocable: false +metadata: + internal: true +--- + +**Local Eval Corpus Collection (`internal/eval`)** + +- Collection is automatic and default-on through `eval.capture_provenance` / `eval.auto_capture` / `eval.max_cases` / `eval.diversified_size` in `config.yaml`, never an environment variable: the daemon's launchd/systemd unit is re-rendered on install and update and preserves only proxy variables (`internal/daemon/service.go` `proxyEnvKeys`), so an env-gated corpus silently stops collecting after an update. The keys are global-only - `Merge` copies them straight from `GlobalConfig`, and an `eval` block in a repo's `.no-mistakes.yaml` is ignored. +- Provenance is unrecoverable: `executor.go` writes it with the review round or never. A round recorded with `capture_provenance` off can never be captured, so the rejection names the setting rather than the round's age. +- The trigger is `RunManager.autoCaptureEvalCase`, called last in the run goroutine after the outcome is already reported: it recovers its own panic (the enclosing recover would otherwise mark a finished run failed), bounds itself with `evalAutoCaptureTimeout` off the run context, serializes runs on `evalCaptureMu` (shared pool + registry), and logs rather than propagates. `ErrNoCapturableReview` separates "nothing to freeze" (DEBUG) from a real fault (WARN). Automatic and manual capture call the same `eval.Capture`. A merged PR also best-effort relabels already-captured cases via `RunManager.relabelEvalRun` (same mutex/timeout); `eval relabel` is the CLI path. +- The unit of truth is finding-level gold, not park/pass, and it is keyed on the round's **recorded fix-vs-skip decision** plus merge state, never on whether a later round still raises the finding (a fix and a ship both make it disappear): a user-selected Fix is true-positive gold (no merge required); an auto-fix selection on a merged run is true-positive gold even if a later round re-raised or rewrote it; a raised `auto-fix`/`ask-user` finding the human did NOT select, on a merged run, is false-positive gold - deliberately reversing the older "never auto-FP from a skip" stance, because in this operator's corpus an approved-and-shipped finding IS a false positive; a human-added finding is false-negative gold; skip/approve/abort without a merge and any round with no recorded decision stay unlabeled / pending; `no-op` findings are never labeled; unmatched candidate findings stay queued - never inferred as false positives - and a confirmed post-PR miss ingested via `eval miss ingest` is also false-negative gold (`recorded-post-pr-miss`). Owner: `internal/eval` (`goldFromRound`, `hasRecordedDecision`, `IngestPostPRMiss`, `ScoreCandidate`); user-facing language is `docs/src/content/docs/reference/eval.md`. +- `diversified` is gold-only and pinned (empty gold -> empty set + `eval sets` warning, never unlabeled fill). Those pins are the held-out official set; leftover labeled cases are `tune`. ListCases trims pins to the live `eval.diversified_size` cap (at most one per stratum when reconciling to 0 or a lower cap); `RefreshDiversified` is only for an explicit rebuild. Never fit matcher thresholds or review prompts on `diversified`. Report F1 as the headline metric only when false-positive gold exists; otherwise recall + precision bounds. RelabelRun recomputes derived merge labels and drops the obsolete ones. Matcher assignment is ONE globally optimal bipartite matching over all gold and candidate findings, weighted so an exact match outweighs any number of fuzzy ones; per-strength-tier greedy assignment understated recall and must not come back. Regressions: `TestListCasesDiversified_*`, `TestGoldFromRoundLabelsByRecordedDecision`, `TestCaptureWritesAutoFixMergedAsTruePositive`, `TestCaptureWritesShippedUnfixedAsFalsePositive`, `TestCaptureWritesShippedUnfixedEvenWhenTheFinalRoundNoLongerRaisesIt`, `TestCaptureLabelsSelectedAutoFixAsTruePositiveEvenWhenLaterRoundReRaisesIt`, `TestRelabelReplacesShippedUnfixedWhenTheRoundLaterRecordsAFixDecision`, `TestMergeGoldClearsStoredShippedUnfixedWhenRecomputedUnlabeled`, `TestRelabelClearsStoredShippedUnfixedFPWhenRecomputedUnlabeled`, `TestScoreCandidateDoesNotLetFuzzyEarlierGoldStealExactLaterMatch`, `TestScoreCandidateRecoversMatchTheTieredMatcherLost`, `TestMaxWeightAssignmentMatchesBruteForceOptimum`, `TestEvaluationSummaryWithholdsHeadlineF1WithoutFalsePositiveGold`, `TestCaptureDoesNotLabelSkipOrApproveAsPass`, `TestCaptureWritesFalseNegativeGoldForUserAddedFinding`, `TestCaptureSkipsIncompleteReviewRoundAndKeepsCompletedSibling`, `TestIngestPostPRMissWritesFalseNegativeGoldOnGreenReview`, `TestCaptureAndReport*`, CLI `TestEvalCaptureAndSetsSpeakInFindingGoldTerms`, `TestEvalMissIngestLabelsFalseNegativeGold`. +- A case stores no Git bundle. Bundles were a full history copy per review pass (~8 MB each here) and cannot be trimmed, because a bundle built with negative refs records prerequisites an empty restore gate lacks. Cases of one repository instead share `/eval/pools/.git`, pinned by `refs/no-mistakes/eval//{head,source-head,base,trusted-config}`; the marginal case costs ~8 KB. `Store.Prune` applies `max_cases` oldest-first but protects active replay reservations and cases with recorded evaluations, so the cap is a retention target rather than a hard bound. +- Capture stays read-only against the gate, so objects reach the pool through a throwaway bare clone plus a refspec fetch - never a bare-object-id fetch, whose want policy is off by default and version-dependent. +- Every eval subcommand is idempotent and tested so (`internal/eval/idempotency_test.go`, CLI `TestEvalCaptureSetsReportAndRelabelAreIdempotentAtTheCLI`): capture/relabel converge in place, sets reads self-stabilize their pins, and replay is additive-by-cohort but never rewrites case labels or manifests - queued unmatched-finding counts derive from the evaluations table (`Store.pendingFindingCounts`), never from a stored counter. The `eval sets` and `eval run` dashboards render in `internal/cli/eval_render.go`, sharing the stats box idioms (`renderTitledBox`); the diversified headline's instant self-score is `SelfScoreRecordedReviews` scoring each case's recorded review against its own gold. +- Regressions: `TestCaptureDoesNotCopyRepositoryHistoryPerCase`, `TestPruneBoundsTheCorpusOldestFirstAndKeepsEvaluatedCases`, `TestDropCaseObjectsReleasesOnlyItsOwnPins`, `TestAutoCaptureEvalCase*` (`internal/daemon`), `TestEvalDefaultsCollectWithoutSetup`, `TestRepoConfigCannotChangeEvalCollection`, e2e `TestEvalAutoCaptureJourney`. diff --git a/.agents/skills/gate-worktree-git-safety/SKILL.md b/.agents/skills/gate-worktree-git-safety/SKILL.md new file mode 100644 index 000000000..5a66f2ec8 --- /dev/null +++ b/.agents/skills/gate-worktree-git-safety/SKILL.md @@ -0,0 +1,35 @@ +--- +name: gate-worktree-git-safety +description: Use when changing recursive gate containment, run worktree placement, bare-gate Git calls, GitHub PR targeting, or post-receive hooks. +user-invocable: false +metadata: + internal: true +--- + +**Recursive Gate-Execution Containment** + +- `internal/gatecontext` is the single classifier for recursive pipeline control. It combines canonical registered gate common-directory identity with OS-authenticated IPC peer ancestry; `NO_MISTAKES_GATE` is diagnostic only. CLI preflight, daemon mutation ingress, gate init/eject, branch-sync mutation, and the managed pre-receive hook must all keep using that owner so marker removal, cwd changes, and direct pushes cannot bypass refusal. Read-only AXI status/logs, help, and doctor remain available. Regressions: `internal/gatecontext`, `TestGateStepCannotStartRecursivePipeline`. +- Every pipeline agent prompt receives the phase boundary from `internal/gateguidance`, and the generated user-level skill reuses the same owner. Step agents return only their assigned phase; the outer executor alone controls other validation, push, PR, and CI phases. Edit `internal/skill/skill.go`, then run `make skill`; never edit the generated skill directly. + +**Filesystem and Paths** + +- Use `filepath.Join`; respect `NM_HOME` for app state; directories are `0o755` and files `0o644` by convention. +- On macOS, path comparisons may need symlink resolution (`/var` vs `/private/var`); use `worktrees.Canonical`/`worktrees.Contains` wherever run worktree paths are compared, so one spelling matches everywhere. +- Run worktree placement (`worktree_roots`) is owned by `internal/worktrees`. Configuration decides it exactly once, at run creation (`Layout.Dir` in `RunManager.startRunWithIntentSource`), and the result is persisted in `runs.worktree_dir`; every later consumer - resume, step diff, startup cleanup, `procreap`, eject, gatecontext attribution - must read it back through `worktrees.RecordedDir` and never re-derive it from config, so a mid-flight edit can neither strand a parked run nor point a removal at a directory the run never used. An empty column means the default `/worktrees//`. +- `worktrees.CheckPlacement` is the single policy for an unusable root (inside `NM_HOME`, inside any registered checkout); `config.ValidateWorktreeRoots` owns what the config can judge alone. The daemon refuses to start on an unusable placement, so `init --worktree-root` must refuse exactly the same set or it prints a paste that takes the operator's CLI down, and EVERY `init` refuses to register a checkout that contains a configured root - the same state reached from the other direction. User-facing semantics live in `docs/src/content/docs/reference/global-config.md`. Regressions: `internal/worktrees`, `internal/config/config_worktree_roots_test.go`, `internal/daemon/worktree_roots_test.go`, `internal/gate/eject_sweep_test.go`, `internal/cli/init_test.go`. + +**Git on Bare Gate Repos (`safe.bareRepository`)** + +- Agent harnesses and hardened CI inject `safe.bareRepository=explicit`, which forbids cwd-based discovery of bare repositories. Route every gate git call through `git.Run`, which detects a bare git dir and prepends `--git-dir=`; never shell out to git in a bare gate repo relying on `cmd.Dir` or `-C` discovery (issue #362). +- Startup gate migration is DB-authoritative with a strict validated `.git` legacy fallback; it must reject non-gates before hook or Git mutation and use `git.RunBare` so a malformed directory cannot discover an ancestor worktree. Completed migrations carry the content-versioned gate-config stamp and normal restarts must stay filesystem-only for current gates. Regressions: `TestMigrateGateConfigsRejectsInvalidDirectoriesAndSkipsCurrentGates`, `TestColdDetachedStartupProductionGateCardinality`. +- Regressions: `TestRunOnBareRepoUnderSafeBareRepositoryExplicit`, `TestWorktreeAddRemoveOnBareRepoUnderSafeBareRepositoryExplicit`, `TestInitUnderSafeBareRepositoryExplicit`. + +**`gh` PR-Targeting From the Bare Gate Repo (`internal/scm/github`)** + +- The daemon runs `gh` from the detached bare gate repo whose HEAD is the default branch, so every PR-targeting command must name the exact PR explicitly: an empty positional makes `gh pr ` infer the cwd branch (`main`) and return `no pull requests found for branch main` even when the feature PR's checks are green. `GetChecks`, `GetPRState`, `GetMergeableState`, and `UpdatePR` route through the shared `prSelector` (number, else URL, else fail closed) - never append a bare `pr.Number`/`pr.URL` that can be empty. This is the `gh` analogue of the git bare-gate-repo trap above. +- Regressions: `TestGetChecksTargetsKnownPRByURLWhenNumberMissing`, `TestPRTargetingReadsFailClosedWithoutIdentity`, `TestPRStateAndMergeableTargetKnownPRByURL`, `TestUpdatePRTargetsKnownPRByURLWhenNumberMissing`, `TestUpdatePRFailsClosedWithoutIdentity`. + +**Post-Receive Hook Gate Path Resolution (`internal/git/hook.go`)** + +- The hook's `--gate` value must never come from a bare `$(pwd)`: Git can invoke `post-receive` from a cwd that collapses to `.` (issue #269), which the daemon rejects and the pipeline silently never starts. The hook script resolves an absolute gate dir (git first, hook location fallback), and `normalizeNotifyGatePath` in `internal/cli/daemon_cmd.go` is an independent second layer that absolutizes whatever an already-installed older hook sends. +- Regressions: `TestPostReceiveHook_ResolvesAbsoluteGateDir`, `TestPostReceiveHook_FallsBackToHookLocationForGateDir`, `TestNormalizeNotifyGatePathResolvesLegacyDotGate`. diff --git a/.agents/skills/pipeline-review-and-agents/SKILL.md b/.agents/skills/pipeline-review-and-agents/SKILL.md new file mode 100644 index 000000000..7e278ddbd --- /dev/null +++ b/.agents/skills/pipeline-review-and-agents/SKILL.md @@ -0,0 +1,63 @@ +--- +name: pipeline-review-and-agents +description: Use when changing review sessions, finding decisions, agent timeouts, local Test behavior, or intent conformance. +user-invocable: false +metadata: + internal: true +--- + +**Review-Loop Agent Sessions (`internal/pipeline/sessions.go`)** + +- Per run, the review loop keeps ONE durable fixer session across review-fix turns, and EVERY review turn (initial review and every full rereview) runs session-free. A rereview certifies fixes implementing the previous review turn's findings, so resuming any review session seats the prescriber as certifier - the mechanism that let one fix round ship wrong code plus the test blessing it with zero findings. Cross-round review context travels only in the explicit sanitized round history; the fixer session is never lent to review turns, no other step uses sessions, and sessions are keyed strictly by run. The rereview prompt reframes fix-round changes as pipeline-authored code under the author-grade adversarial standard (`fixRoundProvenanceClause`); the same clause is emitted on a later run's initial review when a persisted uncertified range is bound. Prior findings, fix summaries, and same-round tests are claims, not evidence. +- Fail-safe rules: unsupported adapter runs cold; a failed fixer resume drops the identity and re-runs the same turn in a fresh fixer session, never skipping the turn; a cancelled ctx gets no fallback retry; `session_reuse: false` forces everything cold. Persistence is minimum metadata only, never prompts or transcripts; `SessionRoleReviewer` remains only so crash recovery accepts legacy persisted rows, which are never resumed. +- `codex exec resume` has a narrower flag surface than `codex exec`, so an unsupported override fails the resume and falls back; the e2e fakeagent must keep parsing both codex argv shapes (`extractCodexPrompt`). +- Regressions: `internal/pipeline/sessions_test.go`, `internal/pipeline/steps/review_session_test.go` (incl. `TestReviewLoop_RereviewNeverResumesTheSessionThatPrescribedItsFixes`), `TestReviewStep_RereviewTreatsFixRoundsAsPipelineAuthoredCode`, `internal/agent/session_test.go`. + +**Recorded Human Decisions on Findings** + +- Approve, skip, and abort each record `selected_finding_ids = "[]"` plus `selection_source = user_declined` on a gated round with findings (`executor.go` `recordDeclinedRound`, `db.SetStepRoundDeclined`); a round with no findings records no decision. The conditional write must never erase an existing selection. User-facing semantics are owned by `docs/src/content/docs/reference/pipeline-steps.md`. +- A decline is stored as the COMPLEMENT of the selection, never as its own list; `declinedFindingLines` derives it and deliberately excludes `auto_fix` selections, whose complement is findings still awaiting a decision (rendered under `auto_fix_left_unselected`, which carries no do-not-re-report instruction). +- `roundHistoryPromptSection` (`internal/pipeline/steps/round_history.go`) now carries three parts: this step's rounds, this run's OTHER steps' decisions, and earlier runs' decisions on this branch (bound per step by `pipeline.BindBranchDecisions`, unlike review-only `BindUncertifiedPipelineRange`). Nothing clears branch decisions - a completed review deletes the uncertified range, which is why that channel could not carry a decision forward, but approving a gate IS the decision. The prompt states that a recorded decision SUPERSEDES the user-intent wording. +- Deliberately ADVISORY and fail-open: no step is blocked and no commit is gated, so an agent may still re-raise a declined finding when the code genuinely changed. There is no reversion detector; `assertPipelineHeadContinuity` and `assertReviewApprovedPushHead` remain lineage-only. `ci_fix.go` and `rebase.go` build prompts without `roundHistoryPromptSection`, so they do not receive decisions. +- Regressions: `TestExecutor_GateResolutionsWithoutASelectionRecordTheDecline`, `TestExecutor_GateResolutionWithNoFindingsRecordsNoDecision`, `TestExecutor_FixResolutionStillRecordsAUserSelection`, `internal/db/round_decisions_test.go`, `TestDeclinedFindingReachesALaterStepInTheSameRun`, `TestDeclinedFindingReachesALaterRunOnTheSameBranch`, `TestCompletedReviewDoesNotClearBranchDecisions`, `TestAutoFixComplementIsNeverPresentedAsAUserDecision`. + +**Uncertified Review Provenance (`internal/pipeline/uncertified.go`)** + +- When a review-step fixer round commits and its re-review does not complete, persist the per-branch uncertified range (`from_sha`, `to_sha`). Persist on review-step fixer commits only, not lint or document. On the next run's initial review, bind that range and emit `fixRoundProvenanceClause` even when `Fixing==false`, so the replacement reviewer is not cold. Rerun proceeds; there is no refusal or `--ack-uncertified-review` gate. +- Missing git objects warn and continue, never block. Clear the range only after a completed review whose approved head equals or is a descendant of `to_sha`; parked, failed, skipped, and aborted reviews must not clear it. Rebase remaps the persisted SHAs onto the rewritten head so the next review can still bind. +- Regressions: `internal/pipeline/uncertified_test.go`, `TestCommitAgentFixes_PersistsUncertifiedRangeForReview`, `TestCommitAgentFixes_LintDoesNotPersistUncertifiedRange`, `TestCommitAgentFixes_DocumentDoesNotPersistUncertifiedRange`, `TestFixRoundProvenanceClause_EmitsForUncertifiedRangeWhenNotFixing`, `TestUncertifiedRange_PersistsThenFeedsNextInitialReview`, `TestRebaseStep_RemapsUncertifiedRangeWhenHeadRewritten`. + +**Review Fixer Verification Discipline (`internal/pipeline/steps/review.go`)** + +- The review-fix prompt requires all fixes before one focused verification limited to the changed area and forbids the whole repository test/lint suite during the fix round. + The dedicated Test and Lint steps are the authoritative gates, although their coverage may be focused when commands are unconfigured. + This is a prompt contract, not an enforced sandbox. + Regression: `TestReviewStep_FixMode_FocusedVerificationContract`. + +**Agent-Invocation Timeouts Report Measured Silence, Never the Budget** + +- A timeout diagnostic may only state what was observed, never restate the configured budget as measured silence. `agentActivity` in `agent_run.go` is the single owner of the measurement and resets per-attempt evidence whenever a retry or fallback starts a replacement attempt, including provider, session-resume, and OpenCode prompt-format fallbacks. A substantive adapter error (a native agent's exit status plus captured stderr) is URL-redacted, length-bounded, and appended as `agent reported: ...`. +- Observed output is streamed assistant text plus throttled `agent.LifecyclePhaseActivity`, sourced from every non-empty read of a native subprocess's stdout or stderr. Prose alone cannot prove liveness: verified against pi 0.84.3, a tool-using turn emits only `tool_execution_*`/`toolcall_*` and no `text_delta` until the very end, and no adapter forwards those to `OnChunk`. Subprocess start and exit are deliberately NOT output - start proves launch, not work, and exit is the deadline's own consequence, so counting either would recreate the fabricated evidence. +- The executor consumes `LifecyclePhaseActivity` into step activity only, never the step log: `axi status` needs the liveness, and a half-hour turn would otherwise emit hundreds of log lines. +- A CI auto-fix agent that exhausts its budget parks at an ask-user gate (`ciFixAgentTimeoutOutcome`) instead of being logged as a warning and re-issued on the next poll. That old path spent up to `auto_fix.ci` full budgets invisibly until `ci_timeout`. Only `pipeline.ErrAgentTimeout` parks; other fix failures keep warn-and-retry. Review deliberately still fails the run rather than parking - Push commits leftover worktree changes, so an approved park would ship a half-finished, unreviewed fix. +- Docs owners: `docs/src/content/docs/reference/global-config.md` (`agent_timeout`) for the diagnostic vocabulary, `docs/src/content/docs/reference/pipeline-steps.md` (CI) for the park. Regressions: `TestRunAgent_Timeout*`, `TestRunAgent_SubprocessStartAloneIsNotObservedOutput`, `TestRunAgent_OperatorCancellationIsNotDressedUpAsAnAgentFault`, `TestPiAgent_ToolOnlyStreamStillReportsSubprocessLiveness`, `TestPiAgent_SilentSubprocessReportsNoLiveness`, `TestExecutor_SubprocessLivenessUpdatesActivityWithoutFloodingTheStepLog`, `TestCIStep_FixAgentBudgetExhaustionParksForADecisionInsteadOfRetrying`, `TestCIStep_NonTimeoutFixFailureKeepsRetrying`, `TestReviewStep_RoundBudgetTimeoutPreservesTheAgentReport`, e2e `TestSilentAgentTimeoutReportsMeasuredEvidence`. + +**Local Test Is Targeted Validation (`internal/pipeline/steps/test.go`)** + +- Local Test (normal evidence agent and Test-repair agent) validates the requested intent with the smallest relevant checks and end-user-aligned evidence; it is never a repository-wide regression-suite walk. + Broad regression belongs to remote CI (`go test -race ./...` in `.github/workflows/ci.yml`) and remains mandatory before a PR is ready. + `commands.test` is the same contract when set: targeted baseline, not CI-parity complete-suite configuration; docs owner is `docs/src/content/docs/reference/repo-config.md` (`commands.test`), step behavior owner is `docs/src/content/docs/reference/pipeline-steps.md` (Test). + This repository dogfoods an empty `commands.test` so the agent-driven targeted path is the default; do not reintroduce `go test -race ./...` as a local Test override. + Process-group reaping on clean/error exit (#357) and Unix WaitDelay remain the lifecycle safety net when agents spawn test workers - restoring the agent-driven path must not revive the daemon OOM leak. + Those agent turns are bounded by `test_agent_timeout` (default 30m, global-only): a stalled evidence or repair agent is cancelled and the run fails instead of waiting forever. Native adapters already honor that deadline through `CommandContext`; the missing piece was the Test step never setting one. Docs owner is `docs/src/content/docs/reference/global-config.md`. + Every other pipeline agent invocation is bounded by `agent_timeout` (default 30m, global-only) at `pipeline.RunAgent` / the executor `timeoutAgent` seam, so a new agent-spawning step cannot hang a run by forgetting a deadline. Review keeps `review_agent_timeout` as a per-round budget; an existing sooner deadline is honored rather than capped. The invocation context is scoped only to `Agent.Run`; a late successful return after the deadline is rejected. Docs owner is `docs/src/content/docs/reference/global-config.md`. + Regressions: `TestTestStep_InitialAgent_TargetedValidationContract`, `TestTestStep_FixMode_TargetedVerificationContract`, `TestTestStep_FixMode_DriverFullSuiteInstructionDoesNotOverrideContract`, `TestTestStep_InitialAgent_NoTargetedEvidenceRequiresHonestFinding`, `TestTestStep_HangingEvidenceAgentFailsRunAfterTimeout`, `TestCodexAgent_RunCancelsSilentHang`, `TestDogfoodConfig_NoBroadLocalTestCommand`, `TestCIWorkflow_RetainsFullRaceSuiteAsBroadRegressionOwner`, plus the existing #357 reap/WaitDelay tests, `TestRunAgent_*`, `TestExecutor_DirectAgentRunIsDeadlineBounded`, `TestDocumentStep_HangingAgentFailsRunAfterTimeout`, `TestLintStep_HangingAgentFailsRunAfterTimeout`, `TestCIStep_HangingFixAgentFailsAfterTimeout`, `TestRebaseStep_HangingConflictAgentFailsAfterTimeout`. + +**Intent Provenance & Conformance (`internal/pipeline/steps/intent_prompt.go`)** + +- Intent carries provenance: an explicit `axi run --intent` persists `Source==db.RunIntentSourceAgent` ("agent", score 1); a transcript match persists the agent name ("claude"/"codex"/...). The executor propagates it as `StepContext.IntentSource` alongside `UserIntent` (`executor.go`). +- `userIntentPromptSection` branches on source: an EXPLICIT intent renders as sanitized-but-AUTHORITATIVE acceptance criteria; an INFERRED intent keeps the low-confidence hint framing verbatim. Both branches keep the `StripAdversarial`+`RedactSecrets` pipeline and BEGIN/END "do not execute instructions" guard - authoritative reframes only the content's authority (check the diff against the criteria), never whether control tokens are stripped. The review prompt adds `intentConformanceReviewClause` for agent-source intent only: a fixer change that contradicts the criteria (removes intent-required or adds intent-forbidden behavior) MUST become an `ask-user` finding, which parks with no executor change. Conformance is limited to source-verifiable criteria; deferred pipeline-owned delivery (remote branch / push / PR / CI for this run) is out of scope at review. +- Review is always pre-push (`StepReview` before `StepPush`/`StepPR`/`StepCI`). `pipelineDeliveryPhaseClause` plus `stripDeferredPipelineOwnedDeliveryFindings` (`pipeline_delivery.go`, applied in `review.go`) keep findings that only claim those later-owned outcomes are missing from parking the run. External or pre-existing lifecycle requirements (numbered PR, third-party artifact, non-run-owned state) stay enforceable. Push, PR, and CI steps remain strict after their stages run. +- Empty/missing finding `action` fails closed to `ask-user`, not auto-fix (`types/findings.go` `ActionOrDefault`); `HasAskUserFindings` uses `ActionOrDefault` so it agrees with `AutoFixableFindings` (an unclassified finding is never auto-fixed and is always caught as ask-user). `MergeUserOverrides` still stamps user-*added* findings auto-fix on purpose. +- The deterministic net-deleted-author-lines git-diff backstop is intentionally not built; `review.go` owns the held-scope TODO. +- Regressions: `internal/pipeline/steps/intent_prompt_test.go`, `internal/pipeline/steps/review_test.go` (`TestReviewStep_ConformanceObligationTracksIntentProvenance`, `TestReviewStep_RereviewFlagsIntentContradictionAsAskUser`), `internal/pipeline/steps/pipeline_delivery_test.go`, `internal/pipeline/steps/review_pipeline_delivery_test.go`, `internal/pipeline/executor_intent_conformance_test.go`, `internal/types/findings_test.go`, e2e `TestIntentJourney` (inferred-source framing), e2e `TestReviewPipelineOwnedPRCriterionDoesNotPark` / `TestReviewExternalPRLifecycleStillParks`. diff --git a/.agents/skills/pr-enforcement-action/SKILL.md b/.agents/skills/pr-enforcement-action/SKILL.md new file mode 100644 index 000000000..2cb6027f2 --- /dev/null +++ b/.agents/skills/pr-enforcement-action/SKILL.md @@ -0,0 +1,17 @@ +--- +name: pr-enforcement-action +description: Use when changing or migrating the shared require-no-mistakes PR-enforcement action or its workflow caller. +user-invocable: false +metadata: + internal: true +--- + +**Shared PR-Enforcement Action (`.github/actions/require-no-mistakes`)** + +- The shared implementation of the `PR must be raised via no-mistakes` gate is a composite action that lets enforcing repositories replace copied, drift-prone scripts. It verifies the signature line, parses the v1 pipeline-step attestation, binds `head_sha` to the PR head, and requires `review`, `test`, and `document` to be `completed`. Callers pin a release tag or commit SHA, never `@main`, which the judged PR can edit. Per-repo configuration is exemptions only (`exempt-authors`, `exempt-bot-authors`, `exempt-head-branches`); which steps are required is deliberately not an input, so no caller can weaken the gate while still reporting the same check name. The action README owns usage; `CONTRIBUTING.md` owns the contributor-facing contract. +- This repository's own gate (`.github/workflows/no-mistakes-required.yml`) is a thin caller of the action, pinned at an already-published commit SHA. GitHub downloads `uses:` at job setup, so the pin must always name a ref that already carries the action. That pin IS the self-certification guard: a PR editing the action is fully tested on its own head (the Go tests execute the working-tree `verify.py`) while the required check judging it runs the published pinned copy, so the change cannot rewrite its own judge. Bumping the pin is a separate deliberate PR. +- This repo's automation exemptions stay in the job-level `if:`, not in `exempt-authors`. An in-job exemption still needs the run to start, and a GITHUB_TOKEN PR's run is created in `action_required` and never starts; the `paths-ignore` entries exist for the same reason. Repos without that constraint should prefer the action's inputs. +- Duplicate step records are LAST-WINS by design (`check_required_steps` in `verify.py`), and a skip-shaped sibling field on a `completed` record is deliberately not inspected. Some pre-migration inline gates were stricter (requiring every record of a name to be `completed`); that strictness is explicitly NOT the standard, and relaxing to last-wins on migration is the intended outcome, not a regression. Do not "harden" this without an owner decision. +- A caller that gains `head_sha` binding must also drop `synchronize` from `on.pull_request.types`, matching this repository's own post-#773 trigger set. Binding plus `synchronize` pins a stale FAILURE check run to a head the pipeline is about to fix, and `gh pr checks` collapses same-named runs by `startedAt`, so the CI monitor can park the run red forever. That change is only safe where no ruleset or branch protection REQUIRES the check - otherwise a pushed head gets no run and the requirement blocks the merge forever. Verify per repo with `gh api repos///rulesets` plus `.../branches//protection`; at the time of the fleet migration `treehouse`, `sshhip`, and `wheelhouse` required it and therefore keep `synchronize`. +- Migrating a repository is rarely a one-file swap. Repos whose tests extract and execute the inline `run:` block (an `extractGateScript()` helper and its gate test) break at import once the block is gone, and repo-level `AGENTS.md` notes that tell agents to hand-copy the gate from a sibling repository must be rewritten - that copying is the drift the shared action exists to remove. +- Regressions: `require_no_mistakes_action_test.go` executes `verify.py` the way a runner does (verdicts, exemption surface, event-payload binding); `workflow_no_mistakes_required_test.go` owns the CALLER - immutable-SHA pin, single delegating step, exemptions, triggers, concurrency identity, fork boundary - and drives the real action through the event payload. diff --git a/.agents/skills/pr-publication-safety/SKILL.md b/.agents/skills/pr-publication-safety/SKILL.md new file mode 100644 index 000000000..2dd0b6c39 --- /dev/null +++ b/.agents/skills/pr-publication-safety/SKILL.md @@ -0,0 +1,17 @@ +--- +name: pr-publication-safety +description: Use when changing PR body rendering, home-path redaction, artifact path publication, or pipeline-attestation markers. +user-invocable: false +metadata: + internal: true +--- + +**Home-Path Redaction in Published PR Content (security)** + +- `internal/safepath` is the one owner of home-directory redaction, the path analogue of `internal/safeurl`. `RedactText` rewrites the process's own home plus `/home/`, `/Users/`, and `C:\Users\` to `~`, unconditionally and for every occurrence. Add new shapes there rather than scrubbing paths at a call site. Candidate resolution must stay free of `filepath.IsAbs`/`VolumeName` and of any reliance on `filepath.Clean`'s separator normalisation: those answer for the build platform, and on Windows `IsAbs` discards the POSIX-rooted `HOME` that Git Bash, MSYS2, and Cygwin set - silently disabling redaction instead of failing. Regression: `TestUsableHomeCandidate_AcceptsBothPlatformSpellings`, `TestHomeCandidates_AreSeparatorSpellingIndependent`. +- `PRStep.buildPRContent` is the single render boundary: it drafts through `draftPRContent` and returns `redactPRContent(content)`, and `Execute` publishes exactly that. Every source that can reach a PR body - agent prose, extracted intent, findings, fix summaries, step errors, artifact `path`, artifact captions, and captured output embedded from evidence files - is covered there, so a new rendering path cannot reintroduce the leak. Redaction runs after every length cap, which is only safe because the placeholder is never longer than the path it replaces. +- The `artifacts[].path` description in `testFindingsSchema` (`common.go`) must not solicit absolute paths, and must not forbid them either. The renderer's allowlist is the worktree or the run's evidence directory and a path under neither is dropped, while the evidence directory defaults under the operator's home - so soliciting more just re-supplies what the boundary has to strip, and a blanket "never report a home directory path" clause makes an obedient agent drop its own evidence. Publication safety is the `pr.go` boundary's job; the schema only stops soliciting paths from elsewhere on the machine. Regressions: `TestTestFindingsSchema_DoesNotSolicitAbsolutePaths`, `TestTestFindingsSchema_KeepsEvidenceDirectoryPathsReportable`. +- Two other public surfaces deliberately do NOT share this rendering and are not covered: agent-authored commit subjects (`commitAgentFixes` -> `Commit.RenderFixMessage`), which reach the remote through Push, and the opt-in evidence branch (`test.evidence.store_in_repo`), which copies artifact files verbatim. Keep the `internal/safepath` package doc honest about that scope. +- The PR body must contain exactly ONE live pipeline-attestation marker, the run's own. `require-no-mistakes` (`.github/actions/require-no-mistakes/verify.py`) binds the FIRST marker in the RAW body to the PR head, so a foreign copy placed earlier fails a PR the pipeline did produce - and a code fence is no defense, because that scan is raw text. Step agents embed foreign markers routinely, by capturing a generated PR body as evidence. +- Neutralize at the assembly choke point (`appendGeneratedSectionsToCleanBodyWithinLimit` plus the two intent paths), never per render path. `pipelineMD` alone carries the real marker and is left intact; `BuildPipelineSummaryFor` neutralizes its own step-detail blocks, which quote agent text. A first attempt put this in `escapePipelineFoldMarkers` - per-render-path - and shipped three live foreign markers to #831 anyway. Regressions: `TestPRStep_ForeignAttestationsInEveryComponentDoNotShadowTheRealOne` (all components at once), plus the per-component guards in `pr_test.go`. +- Regressions: `internal/safepath/redact_test.go`, `internal/pipeline/steps/pr_homepath_test.go`. diff --git a/.agents/skills/process-lifecycle/SKILL.md b/.agents/skills/process-lifecycle/SKILL.md new file mode 100644 index 000000000..962bedf83 --- /dev/null +++ b/.agents/skills/process-lifecycle/SKILL.md @@ -0,0 +1,16 @@ +--- +name: process-lifecycle +description: Use when changing subprocess execution, cancellation, process-tree cleanup, orphan reaping, or Windows child hardening. +user-invocable: false +metadata: + internal: true +--- + +**Context, Concurrency, and Processes** + +- Thread `context.Context` through long-running, subprocess, and networked work; prefer `exec.CommandContext`; use derived contexts and timeouts for cleanup and HTTP calls. +- Route every long-lived subprocess spawned for a cancellable step or agent invocation through `shellenv.ConfigureShellCommand(cmd)`: it creates a process-tree boundary and installs `cmd.Cancel` to kill the whole tree, so grandchildren (test workers, build watchers) cannot outlive cancellation and hold the next run's worktree locked. +- `cmd.Cancel` covers only cancellation; on clean exit or error the group is not reaped, and leaked grandchildren accumulate until the OS OOM-kills the daemon (surfacing as `daemon crashed during execution` with no stack trace). Use `shellenv.RunShellCommand` / `OutputShellCommand` / `CombinedOutputShellCommand` for one-shot commands, or `StartShellCommand` plus `TerminateShellCommandGroup` when handling pipes manually; the helper doc comments in `internal/shellenv` own the details. `ConfigureShellCommand` also installs a 5s `cmd.WaitDelay` backstop so a grandchild holding an inherited pipe cannot wedge `cmd.Wait` forever. Regressions: `TestCodexAgent_Run_ReapsLeakedGrandchildOnCleanExit`, `TestRunShellCommandWithEnv_ReapsGrandchildOnCleanExit`, `TestTerminateShellCommandGroup_*`. +- A process group is a lineage container, not a sandbox: a descendant that calls `setsid(2)`/`setpgid(2)` (agent CLIs sandboxing their tool runners, any daemonizing worker script) leaves the group, and after its parent exits nothing lineage-based can name it again - it burns CPU and holds a deleted worktree's cwd forever. `internal/procreap` is the identity-based backstop: it matches a process by the run worktree its **cwd** resolves under (deliberately never argv, which a legitimate `git worktree remove` also carries), never touches pid<=1/itself/its ancestors, spares worktrees whose run is still pending or running, and escalates SIGTERM to SIGKILL only after a grace period. Reach is `/worktrees` by path shape plus exactly the run worktrees a caller names from run records (`Options.Worktrees`), never a configured worktree root by shape - an operator's own directory is unmatchable unless a run row names it. Every site that removes a run worktree sweeps it first through `procreap.SweepRunWorktree(s)` (run cleanup and setup failure via `RunManager.removeRunWorktree`, startup cleanup, eject), scoped and therefore without age floor or run-active check; the unscoped startup sweep in `recoverOnStartup` keeps the `orphanProcessMinAge` floor. All best effort. Windows needs none of this - job objects contain the whole tree - so the platform layer reports an empty table. Regressions: `internal/procreap`, `TestSweepOrphanRunProcessesReapsFinishedRunAndSparesActiveOne`, `TestSweepRunWorktreeProcessesReapsLeakedChildAtRunCleanup`, `TestTerminateShellCommandGroup_AsksBeforeKilling`, `TestTerminateShellCommandGroup_EscalatesWhenSIGTERMIsIgnored`. +- On Windows the daemon runs console-less, so route every console child through `winproc.Harden(cmd)` (no-op elsewhere, idempotent, preserves existing creation flags) or a console window flashes per child (#287). `shellenv.ConfigureShellCommand` already calls it; one-shot commands built directly must call it themselves. Regressions: `TestHarden*` in `internal/winproc`. +- Protect shared mutable state with the standard sync/atomic tools, and be explicit about ownership and cleanup of goroutines, worktrees, temp dirs, and channels. diff --git a/.agents/skills/release-signing/SKILL.md b/.agents/skills/release-signing/SKILL.md new file mode 100644 index 000000000..c98c000d3 --- /dev/null +++ b/.agents/skills/release-signing/SKILL.md @@ -0,0 +1,16 @@ +--- +name: release-signing +description: Use when changing macOS release signing, release artifact verification, or the release workflow. +user-invocable: false +metadata: + internal: true +--- + +**macOS Release Signing (permanent identity)** + +- Every official macOS release artifact - both `darwin/arm64` and `darwin/amd64` - is Developer ID Application signed on a macOS runner with a fixed identifier, hardened runtime, secure timestamp, and no entitlements, then strictly verified before it is archived or checksummed; the Linux and Windows release paths are unchanged. +- The executable identifier `com.kunchenguid.no-mistakes` and Team ID `9T2J7MNUP9` are the permanent Developer ID identity and MUST NEVER change: they are the invariant of the identity-based designated requirement that lets macOS permission grants survive `no-mistakes update`, so changing either resets every grant once. +- Signing runs only in the darwin build job gated behind the `release-signing` GitHub environment; the certificate is the base64 `CSC_LINK` secret unlocked with `CSC_KEY_PASSWORD`, imported into an ephemeral keychain with a runtime-generated password that is deleted on success and failure, and no other job may reference those secrets. +- Signing happens before tarball creation and checksum generation, and the verify gate fails the release closed on any missing or ambiguous signature, wrong Team ID, non-permanent identifier, content-based (`cdhash`) requirement, missing hardened runtime or timestamp, or wrong architecture. +- Mechanics live in `.github/workflows/release.yml`; the contract is pinned by the root `TestReleaseWorkflow*` static tests in `workflow_release_signing_test.go`, and secret values are never recorded here or in any test fixture. +- Notarization, stapling, a PKG, Homebrew, and universal binaries are intentionally out of scope for this phase. diff --git a/.agents/skills/repository-routing-security/SKILL.md b/.agents/skills/repository-routing-security/SKILL.md new file mode 100644 index 000000000..d87c1d4b4 --- /dev/null +++ b/.agents/skills/repository-routing-security/SKILL.md @@ -0,0 +1,28 @@ +--- +name: repository-routing-security +description: Use when changing fork routing, forge-profile identity, repository URL persistence, or credential redaction. +user-invocable: false +metadata: + internal: true +--- + +**Fork Routing** + +- `repos.upstream_url` is the parent repository used for PR base routing; `repos.fork_url` is an optional GitHub fork push target. +- `no-mistakes init --fork-url ` expects `origin` to point at the GitHub parent repository and `` at the contributor fork; plain `no-mistakes init` preserves an existing fork URL on idempotent refresh. +- Push code must resolve the push URL via `resolvePushURL` (`internal/pipeline/steps/common_git.go`) so configured forks still receive branch updates, including after a CI repair restarts validation; the non-fork path recovers the credentialled upstream from the worktree's `origin` remote at run time because the DB `upstream_url` is stored redacted (see Credential Redaction below). `Repo.PushURL()` remains correct only for fork-only callers (e.g. `rebase.go`), since fork URLs carry no embedded credentials. +- GitHub PR code must keep `--repo` pointed at the parent and use `--head :` when `fork_url` is set; existing-PR lookup must list by the bare branch and filter head-owner fields, never pass `:` to `gh pr list --head`. +- Non-GitHub fork MR/PR routing is intentionally out of scope until implemented end to end; if a legacy row has `fork_url` for another provider, PR creation must skip instead of opening a self PR. +- Every new run best-effort refreshes registered upstream/fork URLs from the working clone through `gate.RefreshRepoURLs`: origin is the upstream authority, an existing fork requires one uniquely matching clone remote, both DB fields replace atomically, and every discovery/validation/write failure logs only a bounded reason and continues with the exact old registration. The refresh never rewrites clone or gate remotes; `Repo.URLsVerified` is run-scoped evidence that trusted fetch/push may use the refreshed DB URL instead of an inherited stale gate origin. + +**Repository Forge Identity (`internal/forgecontext`)** + +- Optional global `forge_profiles` map raw remote host tokens/SSH aliases to one isolated `gh` or `glab` config directory, plus an optional `expected_login` pin. The resolver owns profile selection, validation, parent/fork ambiguity, provider-specific fail-closed activation, and the immutable run environment; do not add ambient account switching or per-step routing. Profile identity for the parent/fork same-profile check is the config directory AND the pin, so conflicting pins fail as ambiguous instead of silently picking one account (`sameProfile`/`expectLogin` own the rationale). +- A resolved context must reach built-in provider commands, configured shell commands, native agents, managed agent servers, and recovered approval reconciliation. Never mutate the daemon environment or persist credentials/profile selection in the DB; recovery re-resolves from current global config. +- No configured profiles means exact legacy ambient behavior. Online auth failures keep provider steps' existing skip behavior; deterministic config/routing errors fail before the pipeline. The public contract lives in `docs/src/content/docs/reference/global-config.md`. + +**Credential Redaction in Stored URLs and Errors (security)** + +- `gate.InitWithFork` runs the upstream URL through `safeurl.Redact` before every DB persist (`UpdateRepoMetadata*`, `InsertRepoWithIDAndFork`) and the "gate initialized" log line; the bare gate's `origin` remote still carries the full credentialled URL (via `provisionGate`) so carved worktrees authenticate. Because the DB copy is redacted, push and branch-sync code must recover the credential from the worktree's `origin` remote at run time (`resolvePushURL`/`resolveUpstreamURL`), never from `Repo.UpstreamURL`/`Repo.PushURL()`. +- Step-failure errors (`executor.go` `FailStep`/log/IPC emit) and the Bitbucket resolve-repo error are redacted via `safeurl.RedactText`/`safeurl.Redact` so a credentialled URL wrapped into an error can never reach a step log or `runs.error`. Reuse `internal/safeurl` for new redaction sites rather than adding a git-local helper; it is already wired into `git.Run`/step git-run error formatting. +- Regressions: `TestInitRedactsCredentialURL`, `TestResolveUpstreamURL_PreservesCredential`, `TestResolveUpstreamURL_FallsBackToRecordedURL`, `TestResolvePushURL_ForkWinsOverCredential`. diff --git a/.agents/skills/test-evidence-storage/SKILL.md b/.agents/skills/test-evidence-storage/SKILL.md new file mode 100644 index 000000000..20e870832 --- /dev/null +++ b/.agents/skills/test-evidence-storage/SKILL.md @@ -0,0 +1,23 @@ +--- +name: test-evidence-storage +description: Use when changing test evidence collection, publication, paths, retention, or scratch-directory cleanup. +user-invocable: false +metadata: + internal: true +--- + +**Test Evidence Lives on an Orphan Branch, Never in the Code Branch** + +- The test step always collects evidence OUTSIDE the worktree, in the directory the executor resolved once as `StepContext.EvidenceDir`; nothing stages or commits it into the pushed branch, so evidence can never reach the default branch's history. With `test.evidence.store_in_repo` and a derivable GitHub link base, the PR step calls `publishRunEvidence` (`internal/pipeline/steps/evidence_publish.go`), which copies the directory onto the push-target repo's orphan evidence branch through `internal/evidence` and hands the PR body its links. A provider without derivable links does not push the branch. +- `internal/evidence` owns the mechanism and its fail-closed rules: plumbing only (scratch `GIT_INDEX_FILE` + `hash-object`/`write-tree`/`commit-tree`), so HEAD, the index, and the worktree are untouched and a detached or shallow clone works; the parent is the just-fetched remote tip so the push is a plain fast-forward and never a force; an existing branch without the `.no-mistakes-evidence` marker at its tip is refused, which is what makes a wrong branch name (`main`) harmless. Every failure returns an error and the PR body falls back to local-path references rather than links that would not resolve. +- PR links are pinned to the evidence COMMIT, not the branch, so a later run overwriting the same paths cannot change what an old PR shows. Link bases come from `Repo.UpstreamURL`/`ForkURL`, never the push URL, which can carry a credential. +- `test.evidence.branch` is trusted-only in `EffectiveRepoConfig` (it names a ref the daemon pushes to); `local_root`/`retention`/`max_runs` are global-only (`applyEvidenceStorageOverrides` is called from `Merge` with `GlobalConfig` alone); the rest of `test.evidence` stays pushed-readable. Invalid branch names, relative `local_root`, unparseable `retention`, and negative `max_runs` all fail the config at parse time (`validateTestRaw`). +- Regressions: `internal/evidence/publish_test.go`, `internal/evidence/branch_test.go`, `internal/pipeline/steps/evidence_publish_test.go`, `TestPushStep_DoesNotPublishTestEvidenceIntoThePushedBranch`, `TestEffectiveRepoConfig_EvidenceBranchTrustedOnly`, `TestLoadGlobalConfig_InvalidEvidenceBranchFailsClosed`, `internal/config/evidence_storage_test.go`. + +**no-mistakes Owns Its Own Scratch (never the shared system temp dir)** + +- Evidence lives at `/evidence/` (`paths.EvidenceDir`/`EvidenceRoot`/`RunEvidenceDir`), never `os.TempDir()`. The daemon's service unit exports only HOME, PATH, and proxy vars, so `TMPDIR` is unset and `os.TempDir()` resolved to the shared `/tmp` - a systemd tmpfs on Ubuntu 24.10+, so evidence consumed RAM. The app root is disk-backed on all three platforms, so there is deliberately NO `runtime.GOOS` branch; do not add one. +- One owner for the path: the executor resolves it (`Executor.runEvidenceDir`) into `StepContext.EvidenceDir`, and `agent.WithSteering(a, evidenceRoot)` takes it as an argument. Steps and the steering preamble must never rebuild it - two independent `os.TempDir()` copies is exactly the drift this replaced. +- Cleanup is ours, in three layers: `RunManager.cleanupRunEvidence` removes a finished run's dir when empty (`os.Remove`, never `RemoveAll` - the test step creates the dir before the agent decides it has anything to write, and that litter was 94% of observed accumulation), `reapEvidence` bounds the directory by age and count oldest-first, and `reapLegacyEvidence` drains the pre-relocation temp directory under the same policy. All three reuse `skipWorktreeCleanup`'s pending/running guard and are best effort. No OS temp timer is load-bearing. +- HELD SCOPE: `internal/eval/replay.go` sandboxes stay in the system temp directory. They are the largest scratch this program creates, but a replay materializes its own nested NM_HOME and worktree while `Store.Prune`, the case records, and the object pools all live under `/eval` - so relocating the sandbox inside the app root nests it in the state it is replaying, which e2e `TestEvalJourney` refuses on purpose. Moving it needs a disk-backed root outside NM_HOME, which does not exist yet; do not "fix" it by weakening that assertion. Every remaining `os.MkdirTemp("", ...)` caller is auto-named and self-cleaning with `defer`; keep it that way. +- Regressions: `internal/paths/evidence_test.go`, `internal/config/evidence_storage_test.go`, `internal/daemon/evidence_reap_test.go`, `TestSteeringNamesTheConfiguredEvidenceRoot`, `TestTestEvidenceDir_DefaultResolutionStaysUnderTheAppRoot`, e2e `TestTestEvidenceLivesUnderAppRootNotSharedTemp` / `TestRunCleanupLeavesNoEmptyEvidenceDirectory`. diff --git a/.agents/skills/testing-conventions/SKILL.md b/.agents/skills/testing-conventions/SKILL.md new file mode 100644 index 000000000..830f277a0 --- /dev/null +++ b/.agents/skills/testing-conventions/SKILL.md @@ -0,0 +1,20 @@ +--- +name: testing-conventions +description: Use when adding or changing tests, the e2e harness, test process isolation, or CI test sharding. +user-invocable: false +metadata: + internal: true +--- + +**Testing Conventions** + +- Prefer e2e tests for behavior that crosses a process or I/O boundary (CLI flags, config loading, git operations, agent spawning, daemon coordination, stdout/stderr, recorded fixtures); unit-test pure helpers where speed and failure localization matter. Prefer creating real git repos in temp dirs over heavy mocking. +- The e2e suite is behind the `e2e` build tag; `make e2e` runs `scripts/e2e.sh`, which sweeps `./internal/e2e/...` and `./internal/pipeline/steps/...`, so keep new step-local e2e tests behind the tag too. +- Temporary e2e daemons (`NM_TEST_START_DAEMON=1` / harness) are owned by `internal/e2edaemon`: exact inventory, concurrency cap (`NM_E2E_DAEMON_MAX`, default 2), bounded argv checks, and reapers in harness Cleanup, package `TestMain`, and `scripts/e2e.sh` EXIT/INT/TERM. A SIGKILL of the wrapper shell does not run its trap; next-run inventory recovery covers that. External sleep-loop keepalives are out of scope. Never point inventory reaping at the shared `~/.no-mistakes` service. Regressions: `internal/e2edaemon/*_test.go`. +- Packages whose tests shell out to git unset `GIT_CONFIG_COUNT` in `TestMain` so ambient `GIT_CONFIG_*` injection from agent harnesses cannot leak in; a test exercising injected config re-sets it with `t.Setenv` (see `internal/git`, `internal/gate`, `internal/daemon`, `internal/pipeline/steps`). +- Packages whose tests can start a daemon or touch ambient state (`cmd/no-mistakes`, `internal/cli`, `internal/update`) use a package-wide `TestMain` that points `NM_HOME` and `HOME` at fresh temp dirs and disables telemetry/update-check env vars, so a full test run never touches a real `~/.no-mistakes`. Follow the same pattern in new such packages. +- `paths.New()` refuses the default `~/.no-mistakes` root under `go test`; tests that touch app state must set `NM_HOME` to a temp dir, and only the production-default path test may opt in with `NO_MISTAKES_ALLOW_DEFAULT_ROOT_IN_TESTS=1`. +- Isolate filesystem and environment state with `t.TempDir()` and `t.Setenv()`. +- The Windows CI leg is process-spawn bound, not compute bound: git-backed packages cost roughly 10x their Linux time (`internal/git` 5.7s -> 53s, `internal/branchsync` 31s -> 415s). The Windows matrix is split into a git-heavy shard and a core remainder so each job's wall stays inside `timeout-minutes: 40` and a hang still surfaces as `go test -timeout` (15m) rather than an evidence-free job cancel. Keep long git-heavy packages off the serial critical path (`internal/branchsync` runs `t.Parallel()` for exactly that reason) and keep the Defender scan-exclusion step in `ci.yml`, whose comment owns the rationale. Regressions: `TestCIWorkflow_WindowsTestsRunWithScanExclusions`, `TestCIWorkflow_WindowsHangSurfacesAsGoTimeoutNotJobCancellation`. +- Go applies an implicit GOOS constraint from a filename suffix, so a test file named `*_windows_test.go` (or `_linux`, `_darwin`) silently compiles only on that platform. Name platform-agnostic tests about Windows something else. +- On macOS a git-heavy package under `-race` intermittently reports `git : signal: segmentation fault`. That is not a git or repo bug: `~/Library/Logs/DiagnosticReports/*.ips` records the crash as `procName: .test, parentProc: .test, asi: "crashed on child side of fork pre-exec"` - the forked child dies before `execve`. Confirm there before chasing it in Go code; the CI legs are Linux and Windows. The same fork mechanic explains a stray `.test -test.timeout=...` process at high CPU that appears to ignore its own deadline: a pre-exec child inherits the parent's name, argv, and cwd, so it is not a running test binary and no test-side timeout applies to it. `internal/procreap` reaps those by cwd. diff --git a/.claude/skills b/.claude/skills new file mode 120000 index 000000000..2b7a412b8 --- /dev/null +++ b/.claude/skills @@ -0,0 +1 @@ +../.agents/skills \ No newline at end of file diff --git a/.release-please-manifest.json b/.release-please-manifest.json index 1aa934c21..6133d13ca 100644 --- a/.release-please-manifest.json +++ b/.release-please-manifest.json @@ -1,3 +1,3 @@ { - ".": "1.59.3" + ".": "1.60.0" } diff --git a/AGENTS.md b/AGENTS.md index 105b04ee9..460bef294 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,37 +14,6 @@ Safest local verification sequence after non-trivial changes: - `make e2e` when touching agent integrations, the e2e harness, or recorded fixtures - `go build -o ./bin/no-mistakes ./cmd/no-mistakes` -**Fork Routing** - -- `repos.upstream_url` is the parent repository used for PR base routing; `repos.fork_url` is an optional GitHub fork push target. -- `no-mistakes init --fork-url ` expects `origin` to point at the GitHub parent repository and `` at the contributor fork; plain `no-mistakes init` preserves an existing fork URL on idempotent refresh. -- Push code must resolve the push URL via `resolvePushURL` (`internal/pipeline/steps/common_git.go`) so configured forks still receive branch updates, including after a CI repair restarts validation; the non-fork path recovers the credentialled upstream from the worktree's `origin` remote at run time because the DB `upstream_url` is stored redacted (see Credential Redaction below). `Repo.PushURL()` remains correct only for fork-only callers (e.g. `rebase.go`), since fork URLs carry no embedded credentials. -- GitHub PR code must keep `--repo` pointed at the parent and use `--head :` when `fork_url` is set; existing-PR lookup must list by the bare branch and filter head-owner fields, never pass `:` to `gh pr list --head`. -- Non-GitHub fork MR/PR routing is intentionally out of scope until implemented end to end; if a legacy row has `fork_url` for another provider, PR creation must skip instead of opening a self PR. -- Every new run best-effort refreshes registered upstream/fork URLs from the working clone through `gate.RefreshRepoURLs`: origin is the upstream authority, an existing fork requires one uniquely matching clone remote, both DB fields replace atomically, and every discovery/validation/write failure logs only a bounded reason and continues with the exact old registration. The refresh never rewrites clone or gate remotes; `Repo.URLsVerified` is run-scoped evidence that trusted fetch/push may use the refreshed DB URL instead of an inherited stale gate origin. - -**Repository Forge Identity (`internal/forgecontext`)** - -- Optional global `forge_profiles` map raw remote host tokens/SSH aliases to one isolated `gh` or `glab` config directory, plus an optional `expected_login` pin. The resolver owns profile selection, validation, parent/fork ambiguity, provider-specific fail-closed activation, and the immutable run environment; do not add ambient account switching or per-step routing. Profile identity for the parent/fork same-profile check is the config directory AND the pin, so conflicting pins fail as ambiguous instead of silently picking one account (`sameProfile`/`expectLogin` own the rationale). -- A resolved context must reach built-in provider commands, configured shell commands, native agents, managed agent servers, and recovered approval reconciliation. Never mutate the daemon environment or persist credentials/profile selection in the DB; recovery re-resolves from current global config. -- No configured profiles means exact legacy ambient behavior. Online auth failures keep provider steps' existing skip behavior; deterministic config/routing errors fail before the pipeline. The public contract lives in `docs/src/content/docs/reference/global-config.md`. - -**Credential Redaction in Stored URLs and Errors (security)** - -- `gate.InitWithFork` runs the upstream URL through `safeurl.Redact` before every DB persist (`UpdateRepoMetadata*`, `InsertRepoWithIDAndFork`) and the "gate initialized" log line; the bare gate's `origin` remote still carries the full credentialled URL (via `provisionGate`) so carved worktrees authenticate. Because the DB copy is redacted, push and branch-sync code must recover the credential from the worktree's `origin` remote at run time (`resolvePushURL`/`resolveUpstreamURL`), never from `Repo.UpstreamURL`/`Repo.PushURL()`. -- Step-failure errors (`executor.go` `FailStep`/log/IPC emit) and the Bitbucket resolve-repo error are redacted via `safeurl.RedactText`/`safeurl.Redact` so a credentialled URL wrapped into an error can never reach a step log or `runs.error`. Reuse `internal/safeurl` for new redaction sites rather than adding a git-local helper; it is already wired into `git.Run`/step git-run error formatting. -- Regressions: `TestInitRedactsCredentialURL`, `TestResolveUpstreamURL_PreservesCredential`, `TestResolveUpstreamURL_FallsBackToRecordedURL`, `TestResolvePushURL_ForkWinsOverCredential`. - -**Home-Path Redaction in Published PR Content (security)** - -- `internal/safepath` is the one owner of home-directory redaction, the path analogue of `internal/safeurl`. `RedactText` rewrites the process's own home plus `/home/`, `/Users/`, and `C:\Users\` to `~`, unconditionally and for every occurrence. Add new shapes there rather than scrubbing paths at a call site. Candidate resolution must stay free of `filepath.IsAbs`/`VolumeName` and of any reliance on `filepath.Clean`'s separator normalisation: those answer for the build platform, and on Windows `IsAbs` discards the POSIX-rooted `HOME` that Git Bash, MSYS2, and Cygwin set - silently disabling redaction instead of failing. Regression: `TestUsableHomeCandidate_AcceptsBothPlatformSpellings`, `TestHomeCandidates_AreSeparatorSpellingIndependent`. -- `PRStep.buildPRContent` is the single render boundary: it drafts through `draftPRContent` and returns `redactPRContent(content)`, and `Execute` publishes exactly that. Every source that can reach a PR body - agent prose, extracted intent, findings, fix summaries, step errors, artifact `path`, artifact captions, and captured output embedded from evidence files - is covered there, so a new rendering path cannot reintroduce the leak. Redaction runs after every length cap, which is only safe because the placeholder is never longer than the path it replaces. -- The `artifacts[].path` description in `testFindingsSchema` (`common.go`) must not solicit absolute paths, and must not forbid them either. The renderer's allowlist is the worktree or the run's evidence directory and a path under neither is dropped, while the evidence directory defaults under the operator's home - so soliciting more just re-supplies what the boundary has to strip, and a blanket "never report a home directory path" clause makes an obedient agent drop its own evidence. Publication safety is the `pr.go` boundary's job; the schema only stops soliciting paths from elsewhere on the machine. Regressions: `TestTestFindingsSchema_DoesNotSolicitAbsolutePaths`, `TestTestFindingsSchema_KeepsEvidenceDirectoryPathsReportable`. -- Two other public surfaces deliberately do NOT share this rendering and are not covered: agent-authored commit subjects (`commitAgentFixes` -> `Commit.RenderFixMessage`), which reach the remote through Push, and the opt-in evidence branch (`test.evidence.store_in_repo`), which copies artifact files verbatim. Keep the `internal/safepath` package doc honest about that scope. -- The PR body must contain exactly ONE live pipeline-attestation marker, the run's own. `require-no-mistakes` (`.github/actions/require-no-mistakes/verify.py`) binds the FIRST marker in the RAW body to the PR head, so a foreign copy placed earlier fails a PR the pipeline did produce - and a code fence is no defense, because that scan is raw text. Step agents embed foreign markers routinely, by capturing a generated PR body as evidence. -- Neutralize at the assembly choke point (`appendGeneratedSectionsToCleanBodyWithinLimit` plus the two intent paths), never per render path. `pipelineMD` alone carries the real marker and is left intact; `BuildPipelineSummaryFor` neutralizes its own step-detail blocks, which quote agent text. A first attempt put this in `escapePipelineFoldMarkers` - per-render-path - and shipped three live foreign markers to #831 anyway. Regressions: `TestPRStep_ForeignAttestationsInEveryComponentDoNotShadowTheRealOne` (all components at once), plus the per-component guards in `pr_test.go`. -- Regressions: `internal/safepath/redact_test.go`, `internal/pipeline/steps/pr_homepath_test.go`. - **GitLab Backend (`internal/scm/gitlab`)** - The backend is pinned against `glab v1.5x`, whose flag surface drifts between versions: the auth check must be host-scoped (`--hostname `, falling back to unscoped only when the host is unknown), `glab mr list` no longer accepts `--state opened`, `glab mr update` has no `-y`/`--yes` flag at all (unlike `mr create`, which does, so `UpdatePR` must not pass it), and the daemon's detached-HEAD worktree breaks `glab ci get`, so pipeline jobs are read via the branch-independent `glab api .../pipelines//jobs` REST endpoint. @@ -61,16 +30,6 @@ Safest local verification sequence after non-trivial changes: - `tea actions runs list`'s array order is not documented as newest-first, and a branch can have more than one run sharing the same head SHA (e.g. a manual UI re-run), so `GetChecks`/`FetchFailedCheckLogs` select the run via `mostRecentRun` (highest numeric run ID) rather than trusting list order or index `[0]`. - The comments in `internal/scm/gitea/gitea.go` own the full rationale for each trap. -**Shared PR-Enforcement Action (`.github/actions/require-no-mistakes`)** - -- The shared implementation of the `PR must be raised via no-mistakes` gate is a composite action that lets enforcing repositories replace copied, drift-prone scripts. It verifies the signature line, parses the v1 pipeline-step attestation, binds `head_sha` to the PR head, and requires `review`, `test`, and `document` to be `completed`. Callers pin a release tag or commit SHA, never `@main`, which the judged PR can edit. Per-repo configuration is exemptions only (`exempt-authors`, `exempt-bot-authors`, `exempt-head-branches`); which steps are required is deliberately not an input, so no caller can weaken the gate while still reporting the same check name. The action README owns usage; `CONTRIBUTING.md` owns the contributor-facing contract. -- This repository's own gate (`.github/workflows/no-mistakes-required.yml`) is a thin caller of the action, pinned at an already-published commit SHA. GitHub downloads `uses:` at job setup, so the pin must always name a ref that already carries the action. That pin IS the self-certification guard: a PR editing the action is fully tested on its own head (the Go tests execute the working-tree `verify.py`) while the required check judging it runs the published pinned copy, so the change cannot rewrite its own judge. Bumping the pin is a separate deliberate PR. -- This repo's automation exemptions stay in the job-level `if:`, not in `exempt-authors`. An in-job exemption still needs the run to start, and a GITHUB_TOKEN PR's run is created in `action_required` and never starts; the `paths-ignore` entries exist for the same reason. Repos without that constraint should prefer the action's inputs. -- Duplicate step records are LAST-WINS by design (`check_required_steps` in `verify.py`), and a skip-shaped sibling field on a `completed` record is deliberately not inspected. Some pre-migration inline gates were stricter (requiring every record of a name to be `completed`); that strictness is explicitly NOT the standard, and relaxing to last-wins on migration is the intended outcome, not a regression. Do not "harden" this without an owner decision. -- A caller that gains `head_sha` binding must also drop `synchronize` from `on.pull_request.types`, matching this repository's own post-#773 trigger set. Binding plus `synchronize` pins a stale FAILURE check run to a head the pipeline is about to fix, and `gh pr checks` collapses same-named runs by `startedAt`, so the CI monitor can park the run red forever. That change is only safe where no ruleset or branch protection REQUIRES the check - otherwise a pushed head gets no run and the requirement blocks the merge forever. Verify per repo with `gh api repos///rulesets` plus `.../branches//protection`; at the time of the fleet migration `treehouse`, `sshhip`, and `wheelhouse` required it and therefore keep `synchronize`. -- Migrating a repository is rarely a one-file swap. Repos whose tests extract and execute the inline `run:` block (an `extractGateScript()` helper and its gate test) break at import once the block is gone, and repo-level `AGENTS.md` notes that tell agents to hand-copy the gate from a sibling repository must be rewritten - that copying is the drift the shared action exists to remove. -- Regressions: `require_no_mistakes_action_test.go` executes `verify.py` the way a runner does (verdicts, exemption surface, event-payload binding); `workflow_no_mistakes_required_test.go` owns the CALLER - immutable-SHA pin, single delegating step, exemptions, triggers, concurrency identity, fork boundary - and drives the real action through the event payload. - **OpenCode Adapter Failure Reporting (`internal/agent/opencode*.go`)** - opencode reports a failed turn on `info.error` with an HTTP 200 and no parts, and serializes every named error as `{"name": ..., "data": {...}}` - the payload fields are nested under `data`, never at the top level. Decoding only the flat shape silently blanks the message, and ignoring non-`StructuredOutputError` variants drops the cause entirely so the run reports the undiagnosable `opencode returned no text output`. `opencodeMessageError` owns the wire shape and `opencodeMessageFailure` owns the surfaced error. @@ -80,107 +39,6 @@ Safest local verification sequence after non-trivial changes: - That fallback is a second attempt in a fresh session too, and a `session.error` can arrive at any point in a turn, so it carries the same gate: `thinkingConflict` annotates the trigger with `errOpencodeToolsAlreadyRan` when the turn ran a tool and `runOnce` then reports the conflict instead of falling back. Regression: `TestOpencodeAgent_ThinkingConflictAfterToolActivityDoesNotFallBack`. - `opencodeMessageError.rawData` keeps the verbatim `data` payload beside the typed `Data`, because thinking/tool_choice detection scans the payload as text (`providerText`) and a decode into the typed struct drops every provider-specific extra it has no field for. -**Documentation** - -- Keep `README.md` concise and high-level; the bar needs to be extremely high for what shows up there. -- Most documentation lives in `docs/`, the published docs site. -- One owner per fact: `docs/src/content/docs/reference/global-config.md` and `docs/src/content/docs/reference/repo-config.md` own configuration keys, `docs/src/content/docs/reference/environment.md` owns environment variables and the telemetry local/remote split, `docs/src/content/docs/concepts/daemon.md` owns the daemon lifecycle model, and guides pages explain purpose and link to those owners instead of restating tables and examples. -- The `document.instructions` block in `.no-mistakes.yaml` states this ownership map for the pipeline's document step; update it when ownership moves. - -**Agent-Guidance Surfaces** - -- `skills/no-mistakes/SKILL.md` is **generated**: the source of truth is the `body` constant in `internal/skill/skill.go`. Edit the body, then `make skill`; `make lint` fails CI on drift. Never edit `SKILL.md` directly. `no-mistakes init` ships this rendering to agents at user level. -- Agent-driving guidance is owned by the skill body and the live `axi` output strings (`internal/cli/axi*.go`); `docs/src/content/docs/guides/agents.md` carries only the canonical invariant sentences pinned by `internal/cli/axi_guidance_test.go` plus a pointer to the skill. When you change driving guidance, change the skill body and the point-of-use `axi` strings together; that drift test is the sync check. -- The shared default test-quality rule lives in `internal/testguidance`; render it only into the task-first skill and pipeline roles that can author, repair, or review tests. Its fake-agent prompt tests are the intentional generated-interface contract, not source-text checks. -- Review auto-fix is disabled by default (`auto_fix.review: 0` in `config.go` `autoFixDefaults`), so blocking and ask-user review findings park for an agent decision; keep the skill, the live `axi` gate `note`, and docs qualified if you touch review auto-fix. - -**Unified Agent Tuning (`internal/agentcfg`)** - -- `agentcfg` is the single owner of the harness-neutral model/effort surface and of the mapping down to each harness's native mechanism (claude/copilot `--effort`, codex `-m` + `-c model_reasoning_effort`, grok `--reasoning-effort`, pi `--thinking`, opencode's session-message `model`/`variant`, acpx `--model` for `cursor`/`acp:`). Add a harness there, not in an adapter or in eval. `rovodev` and `antigravity` are deliberately declared unmappable, so a request for them is a config error rather than a flag that is silently ignored. -- `agent.NewWithOptions` is the one funnel: it validates `Options.Profile` and splices the mapped args after the operator's raw `agent_args_override` args, so both the pipeline (`cfg.AgentProfileFor`) and eval replay (`Candidate.Profile()`) reach every harness by the same path. Never re-derive a model or effort flag at a call site. -- Precedence is fixed: a raw `agent_args_override` flag that already pins a knob natively wins and the mapped value is not emitted, which is what keeps every pre-`agent_config` configuration byte-identical and stops a harness receiving one knob twice. `agent_config` is global-only for the same reason as `agent_args_override`. -- Eval candidates are `agent,model=[,effort=]` (the previous `agent+model` spelling is refused with a migration message), effort is part of the persisted candidate identity, and `agentNeutralGlobalConfig` strips `agent`, `agent_args_override`, and `agent_config` so a replay never inherits the capturing machine's pins. -- Regressions: `internal/agentcfg`, `internal/agent/profile_test.go`, `internal/config/config_agent_config_test.go`, `internal/daemon/pipeline_agent_profile_test.go`, `TestParseCandidate*`, `TestReplayPinsCandidateModelAndEffortOnTheHarness`, `TestCaptureStripsEveryHarnessPinFromThePinnedConfig`. - -**Context, Concurrency, and Processes** - -- Thread `context.Context` through long-running, subprocess, and networked work; prefer `exec.CommandContext`; use derived contexts and timeouts for cleanup and HTTP calls. -- Route every long-lived subprocess spawned for a cancellable step or agent invocation through `shellenv.ConfigureShellCommand(cmd)`: it creates a process-tree boundary and installs `cmd.Cancel` to kill the whole tree, so grandchildren (test workers, build watchers) cannot outlive cancellation and hold the next run's worktree locked. -- `cmd.Cancel` covers only cancellation; on clean exit or error the group is not reaped, and leaked grandchildren accumulate until the OS OOM-kills the daemon (surfacing as `daemon crashed during execution` with no stack trace). Use `shellenv.RunShellCommand` / `OutputShellCommand` / `CombinedOutputShellCommand` for one-shot commands, or `StartShellCommand` plus `TerminateShellCommandGroup` when handling pipes manually; the helper doc comments in `internal/shellenv` own the details. `ConfigureShellCommand` also installs a 5s `cmd.WaitDelay` backstop so a grandchild holding an inherited pipe cannot wedge `cmd.Wait` forever. Regressions: `TestCodexAgent_Run_ReapsLeakedGrandchildOnCleanExit`, `TestRunShellCommandWithEnv_ReapsGrandchildOnCleanExit`, `TestTerminateShellCommandGroup_*`. -- A process group is a lineage container, not a sandbox: a descendant that calls `setsid(2)`/`setpgid(2)` (agent CLIs sandboxing their tool runners, any daemonizing worker script) leaves the group, and after its parent exits nothing lineage-based can name it again - it burns CPU and holds a deleted worktree's cwd forever. `internal/procreap` is the identity-based backstop: it matches a process by the run worktree its **cwd** resolves under (deliberately never argv, which a legitimate `git worktree remove` also carries), never touches pid<=1/itself/its ancestors, spares worktrees whose run is still pending or running, and escalates SIGTERM to SIGKILL only after a grace period. Reach is `/worktrees` by path shape plus exactly the run worktrees a caller names from run records (`Options.Worktrees`), never a configured worktree root by shape - an operator's own directory is unmatchable unless a run row names it. Every site that removes a run worktree sweeps it first through `procreap.SweepRunWorktree(s)` (run cleanup and setup failure via `RunManager.removeRunWorktree`, startup cleanup, eject), scoped and therefore without age floor or run-active check; the unscoped startup sweep in `recoverOnStartup` keeps the `orphanProcessMinAge` floor. All best effort. Windows needs none of this - job objects contain the whole tree - so the platform layer reports an empty table. Regressions: `internal/procreap`, `TestSweepOrphanRunProcessesReapsFinishedRunAndSparesActiveOne`, `TestSweepRunWorktreeProcessesReapsLeakedChildAtRunCleanup`, `TestTerminateShellCommandGroup_AsksBeforeKilling`, `TestTerminateShellCommandGroup_EscalatesWhenSIGTERMIsIgnored`. -- On Windows the daemon runs console-less, so route every console child through `winproc.Harden(cmd)` (no-op elsewhere, idempotent, preserves existing creation flags) or a console window flashes per child (#287). `shellenv.ConfigureShellCommand` already calls it; one-shot commands built directly must call it themselves. Regressions: `TestHarden*` in `internal/winproc`. -- Protect shared mutable state with the standard sync/atomic tools, and be explicit about ownership and cleanup of goroutines, worktrees, temp dirs, and channels. - -**Recursive Gate-Execution Containment** - -- `internal/gatecontext` is the single classifier for recursive pipeline control. It combines canonical registered gate common-directory identity with OS-authenticated IPC peer ancestry; `NO_MISTAKES_GATE` is diagnostic only. CLI preflight, daemon mutation ingress, gate init/eject, branch-sync mutation, and the managed pre-receive hook must all keep using that owner so marker removal, cwd changes, and direct pushes cannot bypass refusal. Read-only AXI status/logs, help, and doctor remain available. Regressions: `internal/gatecontext`, `TestGateStepCannotStartRecursivePipeline`. -- Every pipeline agent prompt receives the phase boundary from `internal/gateguidance`, and the generated user-level skill reuses the same owner. Step agents return only their assigned phase; the outer executor alone controls other validation, push, PR, and CI phases. Edit `internal/skill/skill.go`, then run `make skill`; never edit the generated skill directly. - -**Filesystem and Paths** - -- Use `filepath.Join`; respect `NM_HOME` for app state; directories are `0o755` and files `0o644` by convention. -- On macOS, path comparisons may need symlink resolution (`/var` vs `/private/var`); use `worktrees.Canonical`/`worktrees.Contains` wherever run worktree paths are compared, so one spelling matches everywhere. -- Run worktree placement (`worktree_roots`) is owned by `internal/worktrees`. Configuration decides it exactly once, at run creation (`Layout.Dir` in `RunManager.startRunWithIntentSource`), and the result is persisted in `runs.worktree_dir`; every later consumer - resume, step diff, startup cleanup, `procreap`, eject, gatecontext attribution - must read it back through `worktrees.RecordedDir` and never re-derive it from config, so a mid-flight edit can neither strand a parked run nor point a removal at a directory the run never used. An empty column means the default `/worktrees//`. -- `worktrees.CheckPlacement` is the single policy for an unusable root (inside `NM_HOME`, inside any registered checkout); `config.ValidateWorktreeRoots` owns what the config can judge alone. The daemon refuses to start on an unusable placement, so `init --worktree-root` must refuse exactly the same set or it prints a paste that takes the operator's CLI down, and EVERY `init` refuses to register a checkout that contains a configured root - the same state reached from the other direction. User-facing semantics live in `docs/src/content/docs/reference/global-config.md`. Regressions: `internal/worktrees`, `internal/config/config_worktree_roots_test.go`, `internal/daemon/worktree_roots_test.go`, `internal/gate/eject_sweep_test.go`, `internal/cli/init_test.go`. - -**Git on Bare Gate Repos (`safe.bareRepository`)** - -- Agent harnesses and hardened CI inject `safe.bareRepository=explicit`, which forbids cwd-based discovery of bare repositories. Route every gate git call through `git.Run`, which detects a bare git dir and prepends `--git-dir=`; never shell out to git in a bare gate repo relying on `cmd.Dir` or `-C` discovery (issue #362). -- Startup gate migration is DB-authoritative with a strict validated `.git` legacy fallback; it must reject non-gates before hook or Git mutation and use `git.RunBare` so a malformed directory cannot discover an ancestor worktree. Completed migrations carry the content-versioned gate-config stamp and normal restarts must stay filesystem-only for current gates. Regressions: `TestMigrateGateConfigsRejectsInvalidDirectoriesAndSkipsCurrentGates`, `TestColdDetachedStartupProductionGateCardinality`. -- Regressions: `TestRunOnBareRepoUnderSafeBareRepositoryExplicit`, `TestWorktreeAddRemoveOnBareRepoUnderSafeBareRepositoryExplicit`, `TestInitUnderSafeBareRepositoryExplicit`. - -**`gh` PR-Targeting From the Bare Gate Repo (`internal/scm/github`)** - -- The daemon runs `gh` from the detached bare gate repo whose HEAD is the default branch, so every PR-targeting command must name the exact PR explicitly: an empty positional makes `gh pr ` infer the cwd branch (`main`) and return `no pull requests found for branch main` even when the feature PR's checks are green. `GetChecks`, `GetPRState`, `GetMergeableState`, and `UpdatePR` route through the shared `prSelector` (number, else URL, else fail closed) — never append a bare `pr.Number`/`pr.URL` that can be empty. This is the `gh` analogue of the git bare-gate-repo trap above. -- Regressions: `TestGetChecksTargetsKnownPRByURLWhenNumberMissing`, `TestPRTargetingReadsFailClosedWithoutIdentity`, `TestPRStateAndMergeableTargetKnownPRByURL`, `TestUpdatePRTargetsKnownPRByURLWhenNumberMissing`, `TestUpdatePRFailsClosedWithoutIdentity`. - -**Post-Receive Hook Gate Path Resolution (`internal/git/hook.go`)** - -- The hook's `--gate` value must never come from a bare `$(pwd)`: Git can invoke `post-receive` from a cwd that collapses to `.` (issue #269), which the daemon rejects and the pipeline silently never starts. The hook script resolves an absolute gate dir (git first, hook location fallback), and `normalizeNotifyGatePath` in `internal/cli/daemon_cmd.go` is an independent second layer that absolutizes whatever an already-installed older hook sends. -- Regressions: `TestPostReceiveHook_ResolvesAbsoluteGateDir`, `TestPostReceiveHook_FallsBackToHookLocationForGateDir`, `TestNormalizeNotifyGatePathResolvesLegacyDotGate`. - -**Daemon Singleton Lock (`internal/daemon/lock.go`)** - -- Only one live daemon may own an `NM_HOME`: an exclusive OS file lock on `/daemon.lock` is acquired as the very first action in `RunWithOptions`, strictly before stale-run recovery and socket bind, and held for the process lifetime. The kernel releases it on any process death, so a held lock always means a live holder and no staleness heuristic is needed. Without it, a second daemon stole the socket and ran global crash recovery against the live daemon's runs and worktrees. -- Process launch is not readiness: the PID record is published after the singleton lock and before exclusive recovery, while startup succeeds only after a real IPC health response. The 45s production budget covers cold environment setup and recovery; early exits fail promptly, timeout cleanup reaps detached children before fallback or rollback, and managed plus detached failures retain both causes. Regressions: `TestStartDetachedDaemonDetectsChildExitPromptly`, `TestStartDetachedDaemonTimeoutKillsAndReapsChild`, `TestStartPreservesManagedAndDetachedFallbackErrors`, `TestColdDetachedStartupProductionGateCardinality`. -- A successful stop means the daemon process is gone, not merely that IPC health has disappeared, because only process exit releases the singleton lock. Capture the daemon instance before requesting shutdown, and close the shutdown client before waiting because the daemon drains in-flight handlers during exit. See `waitForDaemonStop` and `stopDetachedDaemon`; regressions: e2e `TestDaemonStopLeavesNoDaemonProcessOwningTheRoot`, `TestDaemonRestartReplacesTheDaemonWithExactlyOneOwner`. -- Independent layers: `internal/ipc` `listen()` dials the socket before unlinking it and refuses to steal a live one; client probes bound the dial with `daemon_connect_timeout` and fail fast on a dead or wedged socket instead of starting a replacement daemon (`EnsureDaemon` surfaces the error with a `daemon start` recovery hint; the health RPC itself is bounded separately by `ipc.DefaultDialTimeout`). -- Daemon execution is explicit-only (`no-mistakes daemon run --root`); never let inherited environment reinterpret probes like `--version` or `status` as daemon workers. -- Startup worktree cleanup is DB-aware: never remove a worktree whose run row is `pending` or `running`; `startRun` inserts the run row before creating the worktree, so a no-row directory is safe to remove immediately. That no-row rule holds only inside `/worktrees`, which is discovered by walking because no-mistakes owns it; a configured worktree root is the operator's directory, so cleanup and eject there act on exactly the recorded run worktrees and never enumerate anything else. -- The user-facing model lives in `docs/src/content/docs/concepts/daemon.md`; the lock rationale lives in the `internal/daemon/lock.go` and `daemon.go` comments. Regressions: `TestAcquireSingletonLock_*`, `TestRunWithResources_SecondDaemonForSameRootFailsWithoutStealingSocket`, `TestRunWithOptions_RequiresSingletonLockBeforeRecovery`, `TestRecoverOnStartup_DoesNotDeleteActiveRunWorktree`, `TestServe_SecondListenerForLiveSocketDoesNotStealIt`, `TestDialConnectTimeoutFailsFastAndNamesSocket`, `TestIsRunningFailsFastWhenSocketAcceptsButDoesNotRespond`, `TestIsRunningSurfacesExistingDeadSocket`, `TestDaemonRunRootFromArgs_EnvDoesNotForceDaemonModeForProbes`, `TestValidateDaemonPIDFallback_RefusesToKillOwnProcess`. - -**Bounded Daemon Logging and Event-Driven AXI Runs** - -- `internal/logstore` owns all daemon-process byte and retention bounds. Lifecycle output uses `logs/daemon.log`, managed Rovo Dev/OpenCode stdout and stderr use `logs/managed-server.log`, and service bootstrap/direct crash output uses `logs/daemon-bootstrap.log`. Rotation snapshots backups and truncates the current inode in place so held service and child descriptors keep writing to the bounded current file. Regressions: `internal/logstore/rotate_test.go`, `TestDetachedDaemonUsesBoundedDedicatedLogSinks`, `TestManagedServerOutputIsSeparatedFromLifecycleFailureSummary`. -- Successful read-only IPC methods are DEBUG; mutations and stream starts are INFO; every request failure is WARN. AXI run driving is subscribe-first and `internal/cli/run_reconciler.go` is the sole owner of event reconciliation, reconnect, duplicate-event coalescing, and the slow lost-event heartbeat. Do not reintroduce fixed-interval `get_run` polling. Regressions: `TestSuccessfulReadRequestsDoNotLogAtInfo`, `TestRequestLoggingKeepsMutationsAndFailuresVisible`, `TestDriveRun_HealthyWaitStaysWithinRequestBudget`, `TestRunReconciler_*`. - -**Bounded Loss-Aware Event Subscriptions** - -- `internal/ipc/events.go` (`ClassOf`) is the single event taxonomy: activity is droppable, state is not, control is broker-generated, and an unrecognized type fails safe to state. Brokers and consumers must read loss tolerance from it rather than re-listing event names. -- `internal/daemon/eventmailbox.go` is the single overflow owner: a per-subscriber ring bounded by 64 events and 1 MiB, non-blocking publish (the executor is never stalled), activity as the only evictable class, and everything else folded into one sticky coalescing `stream_gap` that drains ahead of queued payload. A reserved slot is not enough - it fails at the second simultaneous transition - and producer-side channel receives race the reader, which is why the queue is a ring under a mutex. -- Every state event and every `get_run` snapshot carries a monotonic `StateRev`; `runSnapshot` samples it **before** the DB read, which is sound only because every producer writes state and then emits. Consumers apply a delta only when its revision is newer, so a delta queued before a snapshot cannot regress state after it. Every subscription opens gapped, so attach and reconnect always reconcile first. -- The fix-review working-tree diff is the only gate context that is never persisted, so it is served on demand by `ipc.MethodGetStepDiff` (`RunManager.StepDiff`, bounded at 512 KiB) instead of riding the stream: it was the only unbounded payload, and one frame past the 1 MiB transport line limit ends the subscription and hides every later event. -- Regressions: `internal/daemon/eventmailbox_test.go` (A1-A13 plus the byte/count ceilings), `TestRunSnapshot_*`, `TestStepDiff_*`, `TestExecutor_StateEventsAreEmittedAfterTheirDatabaseWrite`, `TestClassOfUnknownEventFailsSafeToState`, `TestRunReconciler_StreamGapForcesOneAuthoritativeRead`, `TestSubscribeOversizedFrameEndsTheStreamAndHidesLaterEvents`, `internal/tui/overflow_contract_test.go`. - -**Destructive Daemon Lifecycle Guard (`internal/lifecycle/guard.go`)** - -- `daemon stop`, `daemon restart`, and `update` refuse by default while pending/running runs exist (the daemon is machine-wide, so stopping it can fail every active pipeline), list the runs via the shared `lifecycle.ActiveRuns`/`lifecycle.RunList` helpers, and require an explicit `--force`. `update -y` answers only the different-executable prompt and deliberately does not bypass this guard. -- Every invocation of the three commands is logged with caller attribution (PID, PPID, parent command line) via `logLifecycleInvocation` to `/logs/cli.log`; this is the incident forensic trail, do not remove or weaken it. -- Regressions: `TestDaemonStopRefusesWithActiveRunsAndListsThem`, `TestDaemonStopForceOverridesActiveRunGuard`, `TestDaemonRestartRefusesWithActiveRuns`, `TestLifecycleCommandsWriteCallerAttributionToCLILog` (`internal/cli/daemon_lifecycle_test.go`), `TestUpdaterRunRefusesWithActiveRunsAndListsThem`, `TestUpdaterActiveRunGuardAllowsForce` (`internal/update`). - -**Testing Conventions** - -- Prefer e2e tests for behavior that crosses a process or I/O boundary (CLI flags, config loading, git operations, agent spawning, daemon coordination, stdout/stderr, recorded fixtures); unit-test pure helpers where speed and failure localization matter. Prefer creating real git repos in temp dirs over heavy mocking. -- The e2e suite is behind the `e2e` build tag; `make e2e` runs `scripts/e2e.sh`, which sweeps `./internal/e2e/...` and `./internal/pipeline/steps/...`, so keep new step-local e2e tests behind the tag too. -- Temporary e2e daemons (`NM_TEST_START_DAEMON=1` / harness) are owned by `internal/e2edaemon`: exact inventory, concurrency cap (`NM_E2E_DAEMON_MAX`, default 2), bounded argv checks, and reapers in harness Cleanup, package `TestMain`, and `scripts/e2e.sh` EXIT/INT/TERM. A SIGKILL of the wrapper shell does not run its trap; next-run inventory recovery covers that. External sleep-loop keepalives are out of scope. Never point inventory reaping at the shared `~/.no-mistakes` service. Regressions: `internal/e2edaemon/*_test.go`. -- Packages whose tests shell out to git unset `GIT_CONFIG_COUNT` in `TestMain` so ambient `GIT_CONFIG_*` injection from agent harnesses cannot leak in; a test exercising injected config re-sets it with `t.Setenv` (see `internal/git`, `internal/gate`, `internal/daemon`, `internal/pipeline/steps`). -- Packages whose tests can start a daemon or touch ambient state (`cmd/no-mistakes`, `internal/cli`, `internal/update`) use a package-wide `TestMain` that points `NM_HOME` and `HOME` at fresh temp dirs and disables telemetry/update-check env vars, so a full test run never touches a real `~/.no-mistakes`. Follow the same pattern in new such packages. -- `paths.New()` refuses the default `~/.no-mistakes` root under `go test`; tests that touch app state must set `NM_HOME` to a temp dir, and only the production-default path test may opt in with `NO_MISTAKES_ALLOW_DEFAULT_ROOT_IN_TESTS=1`. -- Isolate filesystem and environment state with `t.TempDir()` and `t.Setenv()`. -- The Windows CI leg is process-spawn bound, not compute bound: git-backed packages cost roughly 10x their Linux time (`internal/git` 5.7s -> 53s, `internal/branchsync` 31s -> 415s). The Windows matrix is split into a git-heavy shard and a core remainder so each job's wall stays inside `timeout-minutes: 40` and a hang still surfaces as `go test -timeout` (15m) rather than an evidence-free job cancel. Keep long git-heavy packages off the serial critical path (`internal/branchsync` runs `t.Parallel()` for exactly that reason) and keep the Defender scan-exclusion step in `ci.yml`, whose comment owns the rationale. Regressions: `TestCIWorkflow_WindowsTestsRunWithScanExclusions`, `TestCIWorkflow_WindowsHangSurfacesAsGoTimeoutNotJobCancellation`. -- Go applies an implicit GOOS constraint from a filename suffix, so a test file named `*_windows_test.go` (or `_linux`, `_darwin`) silently compiles only on that platform. Name platform-agnostic tests about Windows something else. -- On macOS a git-heavy package under `-race` intermittently reports `git : signal: segmentation fault`. That is not a git or repo bug: `~/Library/Logs/DiagnosticReports/*.ips` records the crash as `procName: .test, parentProc: .test, asi: "crashed on child side of fork pre-exec"` - the forked child dies before `execve`. Confirm there before chasing it in Go code; the CI legs are Linux and Windows. The same fork mechanic explains a stray `.test -test.timeout=...` process at high CPU that appears to ignore its own deadline: a pre-exec child inherits the parent's name, argv, and cwd, so it is not a running test binary and no test-side timeout applies to it. `internal/procreap` reaps those by cwd. - **Repo Config Trust Boundary (security)** - The daemon runs `commands.*` from `.no-mistakes.yaml` verbatim via `sh -c`, and `agent` selects which process launches with the maintainer's credentials. The code-executing selection fields (`commands.{test,lint,format}` and `agent`) are therefore loaded from the trusted default branch at a **pinned SHA** resolved by a fresh fetch, never from the pushed SHA. The run aborts when the trusted commit or its present config cannot be read and parsed; a readable tree with no config is valid. See `internal/daemon/manager.go` `startRun`, `loadTrustedRepoConfig`, and `assertGateTrustedConfigReadable`. @@ -190,19 +48,6 @@ Safest local verification sequence after non-trivial changes: - `allow_repo_commands` is per-repo, read only from the trusted default-branch copy, and defaults `false`; a contributor cannot self-enable it from a pushed branch. The e2e harness models a trusted single-developer environment and commits `allow_repo_commands: true` via `SetupOpts.AllowRepoCommands`; security tests pass `false`. - Regressions: `TestLoadTrustedRepoConfig_FailClosedOnFetchFailure`, `TestLoadTrustedRepoConfig_PinnedSHAReadsFreshDefaultBranch`, `TestEffectiveRepoConfig_DocumentPolicyTrustedOnly`, `TestEffectiveRepoConfig_ReviewPathInstructionsTrustedOnly`, `TestMatchPathInstructions_PushedIgnorePatternsCannotSuppressTrustedRule`, `TestReviewStep_PushedIgnorePatternsCannotSuppressPathInstructions`, `TestEffectiveRepoConfig_DisableProjectSettingsTrustedOnly`, `TestEffectiveRepoConfig_CIRerunTransientTrustedOnly`, `TestAssertGateTrustedConfigReadable_*`, `TestNewPipelineAgent_OptOut_*`, `TestLoadRecoveredConfig_BoundsFetchAndFailsClosed`, e2e `TestRepoConfigCommandsFromDefaultBranch` (incl. `pushed_branch_cannot_self_enable`), e2e `TestReviewPathInstructionsJourney`. -**CI Monitor Lifecycle** - -- `ci_timeout` is an idle timeout, not an absolute deadline: only `timeoutAnchor` re-arms when the upstream default-branch tip advances, `started` stays fixed for poll pacing, and re-arm only ever extends the deadline (fail-safe on transient base-tip failures). Value semantics (`0` unset, negative unlimited sentinel, keyword parsing) live in `config.go`; keep `config.DefaultCITimeout` and `defaultConfigYAML` in sync (`TestDefaultConfigYAML_MatchesGoDefaults`). User-facing semantics are owned by `docs/src/content/docs/reference/global-config.md`. -- GitHub readiness is the union of the exact current PR head commit's check rollup and every Actions workflow run returned by the Actions API for that same SHA. A workflow rejected before creating jobs/check-runs is absent from the commit rollup but still present in that API; run discovery errors and unknown run states fail closed instead of certifying a green rollup. Regressions: `TestGetChecksIncludesFailedWorkflowRunMissingFromPRRollup`, `TestGetChecksBindsRollupAcrossABAHeadMovement`, `TestCIStep_FailedHeadWorkflowRunPreventsChecksPassed`. -- GitHub's raw commit `statusCheckRollup` returns every check run a commit ever had, including a same-named run a later run at that same head has already superseded (e.g. a required-check job re-triggered by `synchronize` after a pipeline auto-fix push). `Host.GetChecks` (`internal/scm/github/github.go`) collapses same-name reruns of one workflow to the newest `startedAt` (using Actions run identity to break timestamp ties, comparing against every retained duplicate, and preserving unordered records, including ambiguous pending replacements, before a `completedAt` fallback) while preserving independent workflows, unidentified external checks, and same-named commit status contexts, restoring the semantics `gh pr checks` already applies and this repo's own gate `verify.py` already assumes (last-wins). Collapse must run AFTER `appendUnrepresentedWorkflowRuns`, never before: that call dedupes the Actions-run union against checks by run ID over the FULL uncollapsed rollup, so collapsing first drops a superseded run's ID out of the "represented" set and the union re-adds the same stale run under its own workflow run name. Without this collapse, a monitor that keeps `synchronize` in its gate trigger set can loop forever: each auto-fix push legitimately fails the gate at the old head, a later same-head rebind turns it green, but the stale FAILURE stays visible and re-triggers another auto-fix round. Regressions: `TestGetChecksCollapsesSupersededSameNameCheckToLatestAtOneHead`, `TestGetChecksCollapseOrderingDoesNotLetWorkflowRunUnionResurrectSupersededCheck`, `TestGetChecksPreservesIndependentSameNameWorkflows`, `TestGetChecksPreservesIndependentSameNameExternalCheckRuns`, `TestGetChecksCollapseComparesNewestRunWithEverySameNameCandidate`, `TestGetChecksKeepsQueuedReplacementWithEqualStartTime`, `TestGetChecksPreservesUnorderedExternalPendingReplacement`, `TestGetChecksPreservesSameNameStatusContextAndCheckRun`. -- CI readiness never treats an unproven empty forge check list as green. Ready requires observed all-green checks, or trusted default-branch `no_ci: true` with zero registered checks (`internal/pipeline/steps/ci.go` decides whether the declaration applies; `internal/cimonitor` owns the agent-facing log vocabulary and Ready/DeclaredNoCI parse). Delayed registration, pending checks, failures, errors, unknowns, and stale-head evidence stay not-ready; registered checks on a declared no-CI repo are still honored. Regressions: `TestChecksPassed_PR607RealLogSequence`, `TestCIStep_EmptyChecksWithoutNoCIStaysNotReadyPastOldGracePeriod`, `TestCIStep_EmptyChecksWithTrustedNoCIBecomesReady`, `TestCIStep_DelayedCheckRegistrationStaysNotReadyUntilGreen`, `TestCIStep_DeclaredNoCIWithUnexpectedChecksHonorsThem`, `TestEffectiveRepoConfig_NoCITrustedOnly`. -- Persistent provider check-read failures are not an invisible spin: 6 consecutive `GetChecks` errors while the PR is still open park the CI step at an ask-user gate whose provider-neutral finding names the provider CLI/credentials support and includes the underlying error (with a GitHub-specific clause requiring `gh` >= 2.50 for the `gh pr checks --json` call), and the streak resets on any successful read. Regression: `TestCIStep_PersistentCheckReadFailureParksAtAskUser`. -- Reap an orphaned monitor from outside its worktree with `no-mistakes axi abort --run `; it needs only `NM_HOME` and never starts a stopped daemon. A known run succeeds only with durable terminal truth, a recorded nonterminal run fails unconfirmed, and only an unknown id is an idempotent no-op. Bare `axi abort` stays worktree/branch-scoped. -- A merged or closed PR observation transactionally completes an active run and its CI step; PR lifecycle state is monotonic, so duplicate or delayed observations cannot reactivate or regress a terminal run. Startup reconciles legacy `pending` or `running` rows that already hold terminal PR state before parked-run planning and generic crash recovery. Regressions: `TestUpdateRunPRStateFinalizesActiveTerminalOutcomes`, `TestUpdateRunPRStateIgnoresDuplicateAndDelayedRegressions`, `TestReconcileTerminalPRRunsFinalizesLegacyActiveRows`, `TestRecoverOnStartup_FinalizesLegacyTerminalPRRun`, e2e `TestTerminalPRRunDisappearsFromActiveListing`. -- A provider-reported `cancelled` check is never a job verdict, so the deterministic rerun runs strictly before any CI fix round: cancellation and a GitHub pre-run infrastructure failure (a job that failed in setup/action resolution before any repository step ran, flagged `Check.PreRunFailure` by the opt-in `scm.PreRunFailureDetector` and re-bucketed to cancel so it never masks a real test/lint failure, which cleared setup and failed a later step) are the outcomes that earn a rerun, either outcome after its budget parks as `ask-user` instead of entering the `auto_fix.ci` loop, and any genuine or unrecognized failure or merge conflict in the same poll suppresses reruns so real failures still escalate on their first observation. The budget is per check name per run and spent on request rather than on success, and a rerun is never issued once the published branch head no longer equals `runs.head_sha`, because it would certify a commit this run never delivered. Each outstanding rerun records its verified pipeline head and the same-name provider links visible when it was requested; it retires durably when the run head advances or a new conclusive non-cancel link appears. Retirement keeps the spent budget and never changes check buckets. A delayed same-named green sibling can satisfy the link trigger on the same head, matching the default branch's existing name-keyed masking; removing that limitation requires provider truth outside this policy. Classification, the deliberate `TIMED_OUT`/`STALE` exclusions, rollup-lag grace, and retirement live in `internal/pipeline/steps/ci_transient.go`; provider support is the optional `scm.CheckRerunner` (GitHub only), pre-run infrastructure detection is the optional `scm.PreRunFailureDetector` (GitHub only, folded in by `markPreRunInfraFailures`), and user-facing semantics are owned by `docs/src/content/docs/reference/repo-config.md`. Regressions: `TestCIStep_CancelledCheckIsRerunBeforeEscalating`, `TestCIStep_CancelledCheckStaysUnresolvedAfterItsBudget`, `TestCIStep_LaggingRerunRollupKeepsWaitingForTheRepublishedCheck`, `TestCIStep_SameHeadGreenRerunEmitsChecksPassed`, `TestCIStep_DelayedSameNameCheckRetainsLegacyNameBehavior`, `TestCIStep_ResolvedRerunDoesNotParkALaterGreenHead`, `TestRetireResolvedReruns`, `TestRetireResolvedRerunsRetriesAfterPersistenceFailure`, `TestCIStep_MovedPublishedHeadTerminatesInsteadOfRerunning`, `TestCIStep_MovedPublishedHeadClearsCIReadiness`, `TestClassifyCheckFailure`, `TestMarkPreRunInfraFailures_RetriesInfraButNotGenuine`, `TestMarkPreRunInfraFailures_OptInGated`, `TestPreRunFailures_FlagsSetupFailureNotGenuine`, `TestPreRunFailures_FailsClosedOnUnreadableRun`. -- **Terminal is not pending.** Readiness must reject every non pass/fail/skip bucket (`hasUnresolvedChecks`), but only checks that can still finish on their own (`hasPendingChecks`) may keep the monitor polling. A `cancel` bucket - GitHub `CANCELLED`, GitLab `canceled`, Bitbucket `STOPPED`, and how GitHub reports a job killed by its own `timeout-minutes` - is a published conclusion that nothing will replace, so with no rerun outstanding it parks at `ciUnresolvedCancelledOutcome` (`cancelledWithoutRerun`) instead of waiting. Conflating the two is the #628 regression that hung real runs for their whole `ci_timeout`; an unrecognized bucket is deliberately still treated as waiting, because unknown is not evidence of terminal. Regressions: `TestCIStep_CancelledCheckAmongPassingChecksEscalatesInsteadOfPollingForever`, `TestCIStep_ZeroRerunBudgetEscalatesCancelledCheckWithoutMakingItReady`, `TestCIStep_BitbucketStoppedCheckParksForADecision`. -- CI readiness is read from the provider's live PR head check rollup on every poll, so it always describes the head the forge currently has for that PR; no recorded SHA gates it, and a run whose row still names a pre-advance commit must still recognize green at the head the pipeline last pushed. Regression: `TestCIStep_GreenChecksAtAdvancedHeadAreRecognizedWhileRunTracksOlderHead`. - **Parked / Awaiting-Agent Signal** - `runs.awaiting_agent_since` is non-nil **iff** a step is actually parked at an `awaiting_approval`/`fix_review` gate: the executor sets it on gate entry, clears it when `waitForApproval` returns, and `RecoverStaleRuns` clears it on crash recovery. It is observability only (rendered as `awaiting_agent: parked ` in `axi status`) and never changes gate resolution, auto-resume, or the `--yes` default. @@ -218,135 +63,17 @@ A detached HEAD owns no branch, so it resolves nothing either. The rationale lives in the `resolveRun` doc comment and the status-rendering comments (`internal/cli/axi_query.go`). - Regressions: `internal/cli/axi_status_branch_test.go` covers branch-scoped resolution, positive-evidence labels, observation-only gate help, branch lookup failures, detached-HEAD help, snapshot consistency, explicit-ID errors, and run-scoped log continuations; `TestAxiStatusUnknownBranchRunRelationshipGuidance_InInstalledSkill` pins the published relationship wording. -**Review-Loop Agent Sessions (`internal/pipeline/sessions.go`)** - -- Per run, the review loop keeps ONE durable fixer session across review-fix turns, and EVERY review turn (initial review and every full rereview) runs session-free. A rereview certifies fixes implementing the previous review turn's findings, so resuming any review session seats the prescriber as certifier - the mechanism that let one fix round ship wrong code plus the test blessing it with zero findings. Cross-round review context travels only in the explicit sanitized round history; the fixer session is never lent to review turns, no other step uses sessions, and sessions are keyed strictly by run. The rereview prompt reframes fix-round changes as pipeline-authored code under the author-grade adversarial standard (`fixRoundProvenanceClause`); the same clause is emitted on a later run's initial review when a persisted uncertified range is bound. Prior findings, fix summaries, and same-round tests are claims, not evidence. -- Fail-safe rules: unsupported adapter runs cold; a failed fixer resume drops the identity and re-runs the same turn in a fresh fixer session, never skipping the turn; a cancelled ctx gets no fallback retry; `session_reuse: false` forces everything cold. Persistence is minimum metadata only, never prompts or transcripts; `SessionRoleReviewer` remains only so crash recovery accepts legacy persisted rows, which are never resumed. -- `codex exec resume` has a narrower flag surface than `codex exec`, so an unsupported override fails the resume and falls back; the e2e fakeagent must keep parsing both codex argv shapes (`extractCodexPrompt`). -- Regressions: `internal/pipeline/sessions_test.go`, `internal/pipeline/steps/review_session_test.go` (incl. `TestReviewLoop_RereviewNeverResumesTheSessionThatPrescribedItsFixes`), `TestReviewStep_RereviewTreatsFixRoundsAsPipelineAuthoredCode`, `internal/agent/session_test.go`. - -**Recorded Human Decisions on Findings** - -- Approve, skip, and abort each record `selected_finding_ids = "[]"` plus `selection_source = user_declined` on a gated round with findings (`executor.go` `recordDeclinedRound`, `db.SetStepRoundDeclined`); a round with no findings records no decision. The conditional write must never erase an existing selection. User-facing semantics are owned by `docs/src/content/docs/reference/pipeline-steps.md`. -- A decline is stored as the COMPLEMENT of the selection, never as its own list; `declinedFindingLines` derives it and deliberately excludes `auto_fix` selections, whose complement is findings still awaiting a decision (rendered under `auto_fix_left_unselected`, which carries no do-not-re-report instruction). -- `roundHistoryPromptSection` (`internal/pipeline/steps/round_history.go`) now carries three parts: this step's rounds, this run's OTHER steps' decisions, and earlier runs' decisions on this branch (bound per step by `pipeline.BindBranchDecisions`, unlike review-only `BindUncertifiedPipelineRange`). Nothing clears branch decisions - a completed review deletes the uncertified range, which is why that channel could not carry a decision forward, but approving a gate IS the decision. The prompt states that a recorded decision SUPERSEDES the user-intent wording. -- Deliberately ADVISORY and fail-open: no step is blocked and no commit is gated, so an agent may still re-raise a declined finding when the code genuinely changed. There is no reversion detector; `assertPipelineHeadContinuity` and `assertReviewApprovedPushHead` remain lineage-only. `ci_fix.go` and `rebase.go` build prompts without `roundHistoryPromptSection`, so they do not receive decisions. -- Regressions: `TestExecutor_GateResolutionsWithoutASelectionRecordTheDecline`, `TestExecutor_GateResolutionWithNoFindingsRecordsNoDecision`, `TestExecutor_FixResolutionStillRecordsAUserSelection`, `internal/db/round_decisions_test.go`, `TestDeclinedFindingReachesALaterStepInTheSameRun`, `TestDeclinedFindingReachesALaterRunOnTheSameBranch`, `TestCompletedReviewDoesNotClearBranchDecisions`, `TestAutoFixComplementIsNeverPresentedAsAUserDecision`. - -**Uncertified Review Provenance (`internal/pipeline/uncertified.go`)** - -- When a review-step fixer round commits and its re-review does not complete, persist the per-branch uncertified range (`from_sha`, `to_sha`). Persist on review-step fixer commits only, not lint or document. On the next run's initial review, bind that range and emit `fixRoundProvenanceClause` even when `Fixing==false`, so the replacement reviewer is not cold. Rerun proceeds; there is no refusal or `--ack-uncertified-review` gate. -- Missing git objects warn and continue, never block. Clear the range only after a completed review whose approved head equals or is a descendant of `to_sha`; parked, failed, skipped, and aborted reviews must not clear it. Rebase remaps the persisted SHAs onto the rewritten head so the next review can still bind. -- Regressions: `internal/pipeline/uncertified_test.go`, `TestCommitAgentFixes_PersistsUncertifiedRangeForReview`, `TestCommitAgentFixes_LintDoesNotPersistUncertifiedRange`, `TestCommitAgentFixes_DocumentDoesNotPersistUncertifiedRange`, `TestFixRoundProvenanceClause_EmitsForUncertifiedRangeWhenNotFixing`, `TestUncertifiedRange_PersistsThenFeedsNextInitialReview`, `TestRebaseStep_RemapsUncertifiedRangeWhenHeadRewritten`. - -**Review Fixer Verification Discipline (`internal/pipeline/steps/review.go`)** - -- The review-fix prompt requires all fixes before one focused verification limited to the changed area and forbids the whole repository test/lint suite during the fix round. - The dedicated Test and Lint steps are the authoritative gates, although their coverage may be focused when commands are unconfigured. - This is a prompt contract, not an enforced sandbox. - Regression: `TestReviewStep_FixMode_FocusedVerificationContract`. - -**Local Test Is Targeted Validation (`internal/pipeline/steps/test.go`)** - -- Local Test (normal evidence agent and Test-repair agent) validates the requested intent with the smallest relevant checks and end-user-aligned evidence; it is never a repository-wide regression-suite walk. - Broad regression belongs to remote CI (`go test -race ./...` in `.github/workflows/ci.yml`) and remains mandatory before a PR is ready. - `commands.test` is the same contract when set: targeted baseline, not CI-parity complete-suite configuration; docs owner is `docs/src/content/docs/reference/repo-config.md` (`commands.test`), step behavior owner is `docs/src/content/docs/reference/pipeline-steps.md` (Test). - This repository dogfoods an empty `commands.test` so the agent-driven targeted path is the default; do not reintroduce `go test -race ./...` as a local Test override. - Process-group reaping on clean/error exit (#357) and Unix WaitDelay remain the lifecycle safety net when agents spawn test workers - restoring the agent-driven path must not revive the daemon OOM leak. - Those agent turns are bounded by `test_agent_timeout` (default 30m, global-only): a stalled evidence or repair agent is cancelled and the run fails instead of waiting forever. Native adapters already honor that deadline through `CommandContext`; the missing piece was the Test step never setting one. Docs owner is `docs/src/content/docs/reference/global-config.md`. - Every other pipeline agent invocation is bounded by `agent_timeout` (default 30m, global-only) at `pipeline.RunAgent` / the executor `timeoutAgent` seam, so a new agent-spawning step cannot hang a run by forgetting a deadline. Review keeps `review_agent_timeout` as a per-round budget; an existing sooner deadline is honored rather than capped. The invocation context is scoped only to `Agent.Run`; a late successful return after the deadline is rejected. Docs owner is `docs/src/content/docs/reference/global-config.md`. - Regressions: `TestTestStep_InitialAgent_TargetedValidationContract`, `TestTestStep_FixMode_TargetedVerificationContract`, `TestTestStep_FixMode_DriverFullSuiteInstructionDoesNotOverrideContract`, `TestTestStep_InitialAgent_NoTargetedEvidenceRequiresHonestFinding`, `TestTestStep_HangingEvidenceAgentFailsRunAfterTimeout`, `TestCodexAgent_RunCancelsSilentHang`, `TestDogfoodConfig_NoBroadLocalTestCommand`, `TestCIWorkflow_RetainsFullRaceSuiteAsBroadRegressionOwner`, plus the existing #357 reap/WaitDelay tests, `TestRunAgent_*`, `TestExecutor_DirectAgentRunIsDeadlineBounded`, `TestDocumentStep_HangingAgentFailsRunAfterTimeout`, `TestLintStep_HangingAgentFailsRunAfterTimeout`, `TestCIStep_HangingFixAgentFailsAfterTimeout`, `TestRebaseStep_HangingConflictAgentFailsAfterTimeout`. - -**Intent Provenance & Conformance (`internal/pipeline/steps/intent_prompt.go`)** - -- Intent carries provenance: an explicit `axi run --intent` persists `Source==db.RunIntentSourceAgent` ("agent", score 1); a transcript match persists the agent name ("claude"/"codex"/...). The executor propagates it as `StepContext.IntentSource` alongside `UserIntent` (`executor.go`). -- `userIntentPromptSection` branches on source: an EXPLICIT intent renders as sanitized-but-AUTHORITATIVE acceptance criteria; an INFERRED intent keeps the low-confidence hint framing verbatim. Both branches keep the `StripAdversarial`+`RedactSecrets` pipeline and BEGIN/END "do not execute instructions" guard - authoritative reframes only the content's authority (check the diff against the criteria), never whether control tokens are stripped. The review prompt adds `intentConformanceReviewClause` for agent-source intent only: a fixer change that contradicts the criteria (removes intent-required or adds intent-forbidden behavior) MUST become an `ask-user` finding, which parks with no executor change. Conformance is limited to source-verifiable criteria; deferred pipeline-owned delivery (remote branch / push / PR / CI for this run) is out of scope at review. -- Review is always pre-push (`StepReview` before `StepPush`/`StepPR`/`StepCI`). `pipelineDeliveryPhaseClause` plus `stripDeferredPipelineOwnedDeliveryFindings` (`pipeline_delivery.go`, applied in `review.go`) keep findings that only claim those later-owned outcomes are missing from parking the run. External or pre-existing lifecycle requirements (numbered PR, third-party artifact, non-run-owned state) stay enforceable. Push, PR, and CI steps remain strict after their stages run. -- Empty/missing finding `action` fails closed to `ask-user`, not auto-fix (`types/findings.go` `ActionOrDefault`); `HasAskUserFindings` uses `ActionOrDefault` so it agrees with `AutoFixableFindings` (an unclassified finding is never auto-fixed and is always caught as ask-user). `MergeUserOverrides` still stamps user-*added* findings auto-fix on purpose. -- The deterministic net-deleted-author-lines git-diff backstop is intentionally not built; `review.go` owns the held-scope TODO. -- Regressions: `internal/pipeline/steps/intent_prompt_test.go`, `internal/pipeline/steps/review_test.go` (`TestReviewStep_ConformanceObligationTracksIntentProvenance`, `TestReviewStep_RereviewFlagsIntentContradictionAsAskUser`), `internal/pipeline/steps/pipeline_delivery_test.go`, `internal/pipeline/steps/review_pipeline_delivery_test.go`, `internal/pipeline/executor_intent_conformance_test.go`, `internal/types/findings_test.go`, e2e `TestIntentJourney` (inferred-source framing), e2e `TestReviewPipelineOwnedPRCriterionDoesNotPark` / `TestReviewExternalPRLifecycleStillParks`. - -**Test Evidence Lives on an Orphan Branch, Never in the Code Branch** - -- The test step always collects evidence OUTSIDE the worktree, in the directory the executor resolved once as `StepContext.EvidenceDir`; nothing stages or commits it into the pushed branch, so evidence can never reach the default branch's history. With `test.evidence.store_in_repo` and a derivable GitHub link base, the PR step calls `publishRunEvidence` (`internal/pipeline/steps/evidence_publish.go`), which copies the directory onto the push-target repo's orphan evidence branch through `internal/evidence` and hands the PR body its links. A provider without derivable links does not push the branch. -- `internal/evidence` owns the mechanism and its fail-closed rules: plumbing only (scratch `GIT_INDEX_FILE` + `hash-object`/`write-tree`/`commit-tree`), so HEAD, the index, and the worktree are untouched and a detached or shallow clone works; the parent is the just-fetched remote tip so the push is a plain fast-forward and never a force; an existing branch without the `.no-mistakes-evidence` marker at its tip is refused, which is what makes a wrong branch name (`main`) harmless. Every failure returns an error and the PR body falls back to local-path references rather than links that would not resolve. -- PR links are pinned to the evidence COMMIT, not the branch, so a later run overwriting the same paths cannot change what an old PR shows. Link bases come from `Repo.UpstreamURL`/`ForkURL`, never the push URL, which can carry a credential. -- `test.evidence.branch` is trusted-only in `EffectiveRepoConfig` (it names a ref the daemon pushes to); `local_root`/`retention`/`max_runs` are global-only (`applyEvidenceStorageOverrides` is called from `Merge` with `GlobalConfig` alone); the rest of `test.evidence` stays pushed-readable. Invalid branch names, relative `local_root`, unparseable `retention`, and negative `max_runs` all fail the config at parse time (`validateTestRaw`). -- Regressions: `internal/evidence/publish_test.go`, `internal/evidence/branch_test.go`, `internal/pipeline/steps/evidence_publish_test.go`, `TestPushStep_DoesNotPublishTestEvidenceIntoThePushedBranch`, `TestEffectiveRepoConfig_EvidenceBranchTrustedOnly`, `TestLoadGlobalConfig_InvalidEvidenceBranchFailsClosed`, `internal/config/evidence_storage_test.go`. - -**no-mistakes Owns Its Own Scratch (never the shared system temp dir)** - -- Evidence lives at `/evidence/` (`paths.EvidenceDir`/`EvidenceRoot`/`RunEvidenceDir`), never `os.TempDir()`. The daemon's service unit exports only HOME, PATH, and proxy vars, so `TMPDIR` is unset and `os.TempDir()` resolved to the shared `/tmp` - a systemd tmpfs on Ubuntu 24.10+, so evidence consumed RAM. The app root is disk-backed on all three platforms, so there is deliberately NO `runtime.GOOS` branch; do not add one. -- One owner for the path: the executor resolves it (`Executor.runEvidenceDir`) into `StepContext.EvidenceDir`, and `agent.WithSteering(a, evidenceRoot)` takes it as an argument. Steps and the steering preamble must never rebuild it - two independent `os.TempDir()` copies is exactly the drift this replaced. -- Cleanup is ours, in three layers: `RunManager.cleanupRunEvidence` removes a finished run's dir when empty (`os.Remove`, never `RemoveAll` - the test step creates the dir before the agent decides it has anything to write, and that litter was 94% of observed accumulation), `reapEvidence` bounds the directory by age and count oldest-first, and `reapLegacyEvidence` drains the pre-relocation temp directory under the same policy. All three reuse `skipWorktreeCleanup`'s pending/running guard and are best effort. No OS temp timer is load-bearing. -- HELD SCOPE: `internal/eval/replay.go` sandboxes stay in the system temp directory. They are the largest scratch this program creates, but a replay materializes its own nested NM_HOME and worktree while `Store.Prune`, the case records, and the object pools all live under `/eval` - so relocating the sandbox inside the app root nests it in the state it is replaying, which e2e `TestEvalJourney` refuses on purpose. Moving it needs a disk-backed root outside NM_HOME, which does not exist yet; do not "fix" it by weakening that assertion. Every remaining `os.MkdirTemp("", ...)` caller is auto-named and self-cleaning with `defer`; keep it that way. -- Regressions: `internal/paths/evidence_test.go`, `internal/config/evidence_storage_test.go`, `internal/daemon/evidence_reap_test.go`, `TestSteeringNamesTheConfiguredEvidenceRoot`, `TestTestEvidenceDir_DefaultResolutionStaysUnderTheAppRoot`, e2e `TestTestEvidenceLivesUnderAppRootNotSharedTemp` / `TestRunCleanupLeavesNoEmptyEvidenceDirectory`. - **Combined Document+Lint Housekeeping Pass** - When `commands.lint` is empty, the document step performs both duties in one agent invocation and stashes the lint half on `RunShared` (consume-once); the lint step consumes it instead of paying a second cold pass. Neither duty is ever silently dropped: a skipped pass, untrusted structured output, or a lint fix round falls back to lint's own agent pass. Configured `commands.lint` stays a first-class deterministic gate. Uncategorized findings fail safe to the stricter documentation gate. - The document prompt enforces the placement policy (one owner per fact, stale duplicates become pointers, no AGENTS.md postmortems, scope limited to docs the change made stale). Do not reintroduce exhaustive-corpus-sweep language; it caused doc commits in 90 of 121 audited PRs. Contract test: `TestDocumentStep_PromptAppliesPlacementPolicy`; behavior tests: `internal/pipeline/steps/housekeeping_test.go`. -**Local Eval Corpus Collection (`internal/eval`)** - -- Collection is automatic and default-on through `eval.capture_provenance` / `eval.auto_capture` / `eval.max_cases` / `eval.diversified_size` in `config.yaml`, never an environment variable: the daemon's launchd/systemd unit is re-rendered on install and update and preserves only proxy variables (`internal/daemon/service.go` `proxyEnvKeys`), so an env-gated corpus silently stops collecting after an update. The keys are global-only - `Merge` copies them straight from `GlobalConfig`, and an `eval` block in a repo's `.no-mistakes.yaml` is ignored. -- Provenance is unrecoverable: `executor.go` writes it with the review round or never. A round recorded with `capture_provenance` off can never be captured, so the rejection names the setting rather than the round's age. -- The trigger is `RunManager.autoCaptureEvalCase`, called last in the run goroutine after the outcome is already reported: it recovers its own panic (the enclosing recover would otherwise mark a finished run failed), bounds itself with `evalAutoCaptureTimeout` off the run context, serializes runs on `evalCaptureMu` (shared pool + registry), and logs rather than propagates. `ErrNoCapturableReview` separates "nothing to freeze" (DEBUG) from a real fault (WARN). Automatic and manual capture call the same `eval.Capture`. A merged PR also best-effort relabels already-captured cases via `RunManager.relabelEvalRun` (same mutex/timeout); `eval relabel` is the CLI path. -- The unit of truth is finding-level gold, not park/pass, and it is keyed on the round's **recorded fix-vs-skip decision** plus merge state, never on whether a later round still raises the finding (a fix and a ship both make it disappear): a user-selected Fix is true-positive gold (no merge required); an auto-fix selection on a merged run is true-positive gold even if a later round re-raised or rewrote it; a raised `auto-fix`/`ask-user` finding the human did NOT select, on a merged run, is false-positive gold - deliberately reversing the older "never auto-FP from a skip" stance, because in this operator's corpus an approved-and-shipped finding IS a false positive; a human-added finding is false-negative gold; skip/approve/abort without a merge and any round with no recorded decision stay unlabeled / pending; `no-op` findings are never labeled; unmatched candidate findings stay queued - never inferred as false positives - and a confirmed post-PR miss ingested via `eval miss ingest` is also false-negative gold (`recorded-post-pr-miss`). Owner: `internal/eval` (`goldFromRound`, `hasRecordedDecision`, `IngestPostPRMiss`, `ScoreCandidate`); user-facing language is `docs/src/content/docs/reference/eval.md`. -- `diversified` is gold-only and pinned (empty gold -> empty set + `eval sets` warning, never unlabeled fill). Those pins are the held-out official set; leftover labeled cases are `tune`. ListCases trims pins to the live `eval.diversified_size` cap (at most one per stratum when reconciling to 0 or a lower cap); `RefreshDiversified` is only for an explicit rebuild. Never fit matcher thresholds or review prompts on `diversified`. Report F1 as the headline metric only when false-positive gold exists; otherwise recall + precision bounds. RelabelRun recomputes derived merge labels and drops the obsolete ones. Matcher assignment is ONE globally optimal bipartite matching over all gold and candidate findings, weighted so an exact match outweighs any number of fuzzy ones; per-strength-tier greedy assignment understated recall and must not come back. Regressions: `TestListCasesDiversified_*`, `TestGoldFromRoundLabelsByRecordedDecision`, `TestCaptureWritesAutoFixMergedAsTruePositive`, `TestCaptureWritesShippedUnfixedAsFalsePositive`, `TestCaptureWritesShippedUnfixedEvenWhenTheFinalRoundNoLongerRaisesIt`, `TestCaptureLabelsSelectedAutoFixAsTruePositiveEvenWhenLaterRoundReRaisesIt`, `TestRelabelReplacesShippedUnfixedWhenTheRoundLaterRecordsAFixDecision`, `TestMergeGoldClearsStoredShippedUnfixedWhenRecomputedUnlabeled`, `TestRelabelClearsStoredShippedUnfixedFPWhenRecomputedUnlabeled`, `TestScoreCandidateDoesNotLetFuzzyEarlierGoldStealExactLaterMatch`, `TestScoreCandidateRecoversMatchTheTieredMatcherLost`, `TestMaxWeightAssignmentMatchesBruteForceOptimum`, `TestEvaluationSummaryWithholdsHeadlineF1WithoutFalsePositiveGold`, `TestCaptureDoesNotLabelSkipOrApproveAsPass`, `TestCaptureWritesFalseNegativeGoldForUserAddedFinding`, `TestCaptureSkipsIncompleteReviewRoundAndKeepsCompletedSibling`, `TestIngestPostPRMissWritesFalseNegativeGoldOnGreenReview`, `TestCaptureAndReport*`, CLI `TestEvalCaptureAndSetsSpeakInFindingGoldTerms`, `TestEvalMissIngestLabelsFalseNegativeGold`. -- A case stores no Git bundle. Bundles were a full history copy per review pass (~8 MB each here) and cannot be trimmed, because a bundle built with negative refs records prerequisites an empty restore gate lacks. Cases of one repository instead share `/eval/pools/.git`, pinned by `refs/no-mistakes/eval//{head,source-head,base,trusted-config}`; the marginal case costs ~8 KB. `Store.Prune` applies `max_cases` oldest-first but protects active replay reservations and cases with recorded evaluations, so the cap is a retention target rather than a hard bound. -- Capture stays read-only against the gate, so objects reach the pool through a throwaway bare clone plus a refspec fetch - never a bare-object-id fetch, whose want policy is off by default and version-dependent. -- Every eval subcommand is idempotent and tested so (`internal/eval/idempotency_test.go`, CLI `TestEvalCaptureSetsReportAndRelabelAreIdempotentAtTheCLI`): capture/relabel converge in place, sets reads self-stabilize their pins, and replay is additive-by-cohort but never rewrites case labels or manifests - queued unmatched-finding counts derive from the evaluations table (`Store.pendingFindingCounts`), never from a stored counter. The `eval sets` and `eval run` dashboards render in `internal/cli/eval_render.go`, sharing the stats box idioms (`renderTitledBox`); the diversified headline's instant self-score is `SelfScoreRecordedReviews` scoring each case's recorded review against its own gold. -- Regressions: `TestCaptureDoesNotCopyRepositoryHistoryPerCase`, `TestPruneBoundsTheCorpusOldestFirstAndKeepsEvaluatedCases`, `TestDropCaseObjectsReleasesOnlyItsOwnPins`, `TestAutoCaptureEvalCase*` (`internal/daemon`), `TestEvalDefaultsCollectWithoutSetup`, `TestRepoConfigCannotChangeEvalCollection`, e2e `TestEvalAutoCaptureJourney`. - **Telemetry Shape** - Read-only surfaces (`axi` home/status/logs, `status`, `runs`) emit NO pageview and gate their command event through `telemetry.ReadSurfaceGate` (emit on state-fingerprint change, else at most once per 10 min, persisted at `/telemetry-gate.json`). Never reintroduce the pageview+command double emit for read surfaces - `axi-status` alone was 42% of all remote event rows. Mutation surfaces stay full-fidelity via `trackAxiSurface`/`trackCommand`. - Detailed performance evidence is LOCAL-ONLY (`agent_invocations` rows plus `runs.parked_ms`); never store prompts, outputs, diffs, or raw command arguments there (shape-guard test `TestAgentInvocations_PrivacySafeShape`) and never send run IDs, paths, session identities, or per-invocation records to Umami - the only remote perf data is three bounded counts on the terminal `run finished` event. The local/remote split is documented in `docs/src/content/docs/reference/environment.md`; read locally with `no-mistakes stats`. - Session-fidelity metric counts and timing boundaries have ONE authoritative home, `internal/agent/invocationmetrics.go` (tool-category classifier, `InvocationMetrics`, `FreshInputTokens`, `PerRoundTokens`, `ModelTimeMS`); the codex adapter fills them from its live `exec --json` event stream (`codex_metrics.go`) and the additive fidelity fields plus cache-creation usage are nullable so a not-reported datum is stored as NULL, never a fabricated zero. Codex's live stream exposes neither the model (resolved best-effort from the `~/.codex/sessions` rollout) nor internal model-request counts (it batches one exec into a single `turn.completed`, so round-trips are counted from completed items and subprocess wait is the reader-timed tool-item interval); codex usage is cumulative across a resumed session, so per-round deltas subtract the same session's prior cumulative (`Result.SessionUsageCumulative`). Regressions: `internal/agent/invocationmetrics_test.go`, `internal/agent/codex_metrics_test.go`, `internal/pipeline/instrument_fidelity_test.go`, `internal/db/agent_invocation_test.go` (`TestOpenMigratesSessionFidelityColumns`). -**Guarded Local Branch Synchronization (`internal/branchsync`)** - -- `sync`, `axi sync`, and the TUI `u` action share one service whose only ordinary worktree mutation is a clean guarded move to an exact freshly verified pipeline push binding: strict fast-forward for behind branches, or an anchored reset to an equivalent diverged pipeline head when local unique work is already represented there. Under `--recover`, the worktree can only strict-fast-forward to the gate-preserved head, or adopt a diverged preserved head that `preservedContainsLocalWork` proves carries every local change. Passive status never fetches, and blocked states never reset, stash, merge, rebase, force, switch, delete, or update an external remote. -- Give each network remote operation its own bounded child context derived from the caller: `Refresh` must not share one deadline across sequential `git.LsRemote` and `git.FetchRemoteBranchToPrivateRef` calls, and `Apply` uses the same per-operation budget for its final live check. The per-operation budget is `Service.RemoteTimeout`, sourced only from the operator's global `branch_sync_remote_timeout` setting (default `config.DefaultBranchSyncRemoteTimeout`, 60s); `RepoConfig` deliberately has no matching field. `Recover`'s local-gate fetch is outside this network deadline contract. Regressions: `TestRefreshSlowSuccessfulLsRemoteDoesNotStealFetchBudget`, `TestRefreshSlowButSuccessfulLsRemoteAloneExceedsItsOwnBudgetReportsOffline`, `TestRefreshRaisedRemoteTimeoutAcceptsTheSameLegitimateSlowLsRemote`, `TestRefreshParentCancellationStopsFetchAfterLsRemoteSucceeds`, `TestServiceRemoteTimeoutDefaultsToConfigDefault`, `TestLoadGlobal_InvalidBranchSyncRemoteTimeout`, `TestLoadRepo_BranchSyncRemoteTimeoutIsNotARepoSetting`. -- Successful pipeline pushes persist the exact SHA, credential-free target fingerprint/ref, and generation; legacy rows remain nullable and must never infer provenance from mutable `head_sha`. Structured PR lifecycle retires merged/closed branches. The service rechecks the invoking worktree, target, live remote equality, ancestry or equivalent-divergence proof, generation, and all mutable assumptions immediately before apply. -- A TERMINAL run with unpublished pipeline commits (moved head) is recoverable only from verified, non-conflicting evidence: inspection and `Recover` share one eligibility model. Equal/ahead local ancestry can create the local anchor without requiring gate access, but available gate evidence must agree; importing a missing preserved head requires exact or safely anchorable gate evidence, a clean worktree, and either ancestry or the content-preservation proof below. Only then does inspection report `blocked_pipeline_owned_recoverable` + `next_action recover_custody` with the exact submitted/current-head and relation facts (active runs keep the plain block). Missing, non-commit, symbolic, or conflicting evidence, and import cases that are dirty or genuinely divergent, fail closed with manual reconciliation instead. `sync --recover` anchors the preserved head at `refs/no-mistakes/recover/` before stamping `runs.custody_returned_at`. - Cancellation RELEASES a terminal run that never changed the submitted head (`head_sha == submitted_head_sha`, no push, no custody stamp): selection keeps it visible so it never misreports as `blocked_wrong_branch`, and it classifies `user_owned` - no `next_action`, non-blocking exit, never represented as recoverable custody, `--recover` there is an idempotent no-op that mutates nothing, and a fresh `axi run` or separately authorized direct push is never blocked. - Equal/ahead worktrees anchor locally without requiring gate access, but an available gate's existing recovery ref must agree with the recorded head; behind/diverged worktrees verify and fetch the preserved head from the run-specific recovery ref, fast-forwarding only a clean behind worktree. - A cancelled validation routinely leaves a preserved head that is a REBASE of the local branch, which equality and ancestry read as plain divergence, so a clean diverged worktree is adopted when `preservedContainsLocalWork` proves containment. That proof is an executable `merge-tree` three-way merge whose result must equal the preserved head's tree, anchored on the merge-base - never `runs.base_sha`, the previous gate head. It deliberately does NOT use patch identity: patch IDs discard hunk locations and whitespace, so they cannot tell a genuine replay from a same-shaped edit to another identical block, and a containment claim built on them is not a proof. Everything undecidable escalates, including a rebase whose fix rounds also rewrote operator lines, where nothing separates a deliberate fix from a dropped change. - Adoption anchors the pre-recovery local head at `refs/no-mistakes/recover-local/`, then moves the branch with Git operations that fail closed on their own rather than after an observation - an atomic `update-ref` CAS plus `read-tree -m -u`, never check-then-act followed by `reset --hard`, which destroys anything landing in the gap. `recoverAdoptPreserved` owns the reasoning. - Terminalization pins every verified unpublished head at `refs/no-mistakes/recover/` before the managed worktree can be removed. Recovery reads that run-specific ref rather than requiring the gate branch to match, so aborts, rebases, and pre-push failures remain recoverable while an independently moved gate branch is preserved. Legacy recorded heads that still exist as dangling gate objects are anchored on recovery; a truly missing recorded head reports a distinct manual-reconciliation action instead of advertising an impossible `recover_custody` command. - When the operator keeps a behind or diverged local head instead of taking the preserved head, `--keep-local` never touches the worktree and CAS-moves the gate branch to the kept head, staging objects via gate-side fetch - never a push, which would fire the receive hook and start a run. - The full relation matrix and fail-safe rules live in the `Recover` doc comment in `internal/branchsync/sync.go`. -- Public guidance is owned by `internal/skill/skill.go` plus live AXI strings, then regenerated with `make skill`. Core regressions live in `internal/branchsync` (incl. `recover_test.go`), `internal/cli/sync_test.go`, `internal/tui/branch_sync_test.go`, and e2e `TestAxiBranchSyncJourney` / `TestAxiCustodyRecoveryJourney` / `TestAxiCustodyRecoveryAfterRebaseJourney` / `TestAxiPrePushAbortUnmovedHeadCustodyJourney`. - -**Post-Review Head Continuity and Push Binding** - -- Every step after Review in the fixed pipeline order (Test, Document, Lint, Push, PR, CI) calls `assertPipelineHeadContinuity` at entry. The helper is the single semantic owner: equal or descendant live heads continue; backward, sibling, and unverifiable heads fail before the step performs work. Regression: `TestPostReviewStepsRefuseHeadClobberAtEntry`. -- A successfully completed full review atomically records `runs.review_approved_head_sha`; parked, failed, skipped, and legacy reviews carry no inferred authority. Push reads that durable binding, permits only the exact commit or a descendant, and pushes the verified immutable SHA rather than mutable `HEAD`. Never infer approval from `runs.head_sha`, a worktree, gate ref, or remote branch. Regressions: `TestPushStep_RefusesPostReviewClobberWithoutLaterPipelineCommit`, `TestPushStep_BindsRemoteAndDatabaseToVerifiedCommitWhenHEADMovesDuringPush`, `TestExecutor_FullRereviewReplacesApprovalWithoutAuthorizingParkedRound`. - -**Rebase Base & Force-Push Safety (data-loss prevention)** - -- The whole job of this tool is to not lose people's code; favor refusing the push and surfacing a finding over any clever recovery. The comments in `internal/pipeline/steps/forcepush.go` own the full reasoning; the invariants are the next three bullets. -- Rebase bases come from the freshly fetched authoritative remote refs, never local or stale state; and a branch built on unpushed local-default-branch commits parks with `NeedsApproval` + `AutoFixable=false` instead of silently widening the PR (`detectBundledLocalDefaultCommits`, #283). -- Every force-push routes through `resolveForcePushDecision`, which re-reads the live remote head and allows the push only for a new branch, an already-equal remote, an unchanged `lastSeenSHA`, or remote commits already incorporated by patch-id (excluding `^baseSHA` history the run knowingly rewrites). Anything else refuses, and a failed ls-remote/fetch fails closed; never degrade to a bare `--force`/`--force-with-lease` without an explicit anchor. -- `lastSeenSHA` must stay the head the run last **observed** (from run/prior-run push provenance or the remote-tracking ref), never the live remote tip: the rebase step refreshes `origin/` only on a normal push, NOT on a force push. CI repairs commit locally and restart validation at Review; the later Push step owns their remote update and force-push safety. Anchoring a lease to a SHA read immediately before pushing is the original #281 bug (it always passes and protects nothing); always-fetching the branch on force push recreates it. Never reintroduce either. -- Regressions: `TestPushStep_RefusesToClobberAdvancedUpstreamBranch` (#305), `TestForcePushRun_RefusesToClobberOutOfBandBranchCommit`, `TestRebaseStep_DetectsUnpushedLocalDefaultBranchCommits` (#283), `TestResolveForcePushDecision_*`, `TestExecutor_CIRestartRevalidatesBeforePush`, `TestPushStep_AllowsForcePushAfterMidRunRebaseOverPriorPushedGeneration` (#837), `TestPushStep_AllowsForcePushOnRerunOverPriorRunPushedGeneration` (#837). - -**macOS Release Signing (permanent identity)** - -- Every official macOS release artifact - both `darwin/arm64` and `darwin/amd64` - is Developer ID Application signed on a macOS runner with a fixed identifier, hardened runtime, secure timestamp, and no entitlements, then strictly verified before it is archived or checksummed; the Linux and Windows release paths are unchanged. -- The executable identifier `com.kunchenguid.no-mistakes` and Team ID `9T2J7MNUP9` are the permanent Developer ID identity and MUST NEVER change: they are the invariant of the identity-based designated requirement that lets macOS permission grants survive `no-mistakes update`, so changing either resets every grant once. -- Signing runs only in the darwin build job gated behind the `release-signing` GitHub environment; the certificate is the base64 `CSC_LINK` secret unlocked with `CSC_KEY_PASSWORD`, imported into an ephemeral keychain with a runtime-generated password that is deleted on success and failure, and no other job may reference those secrets. -- Signing happens before tarball creation and checksum generation, and the verify gate fails the release closed on any missing or ambiguous signature, wrong Team ID, non-permanent identifier, content-based (`cdhash`) requirement, missing hardened runtime or timestamp, or wrong architecture. -- Mechanics live in `.github/workflows/release.yml`; the contract is pinned by the root `TestReleaseWorkflow*` static tests in `workflow_release_signing_test.go`, and secret values are never recorded here or in any test fixture. -- Notarization, stapling, a PKG, Homebrew, and universal binaries are intentionally out of scope for this phase. - -**When Making Changes** - -- Whenever you must bring in new dependencies, check latest documentation for knowledge, and discuss with the user. -- Always use test driven development for bug fixes and feature development. - ## Maintaining this file Keep this file for knowledge useful to almost every future agent session in this project. diff --git a/CHANGELOG.md b/CHANGELOG.md index 2945bc3a7..050dee0a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +## [1.60.0](https://github.com/kunchenguid/no-mistakes/compare/v1.59.3...v1.60.0) (2026-08-28) + + +### Features + +* **pipeline:** ingest review comments and synchronize gate refs ([#873](https://github.com/kunchenguid/no-mistakes/issues/873)) ([605481b](https://github.com/kunchenguid/no-mistakes/commit/605481b9e3523fb7c11acf671826b22b8076b2fa)) + + +### Bug Fixes + +* **pipeline:** make native agent timeouts observable and bounded ([#877](https://github.com/kunchenguid/no-mistakes/issues/877)) ([dc45531](https://github.com/kunchenguid/no-mistakes/commit/dc4553128e4e29c78b7f5223daa9d9aabcca5471)) + ## [1.59.3](https://github.com/kunchenguid/no-mistakes/compare/v1.59.2...v1.59.3) (2026-08-27) diff --git a/docs/src/content/docs/guides/agents.md b/docs/src/content/docs/guides/agents.md index 4942af8e6..37c300ec4 100644 --- a/docs/src/content/docs/guides/agents.md +++ b/docs/src/content/docs/guides/agents.md @@ -178,6 +178,9 @@ no-mistakes axi abort --run Before any post-pipeline local commit or fresh run, read `branch_sync`. Only when its structured `next_action.code` is `sync`, run `no-mistakes axi sync` first. When `next_action.code` is `recover_custody` - a terminal run left unpublished pipeline commits preserved in the local gate - run `no-mistakes axi sync --recover` to return custody, or `no-mistakes rerun` to resume validating the preserved head. +When `next_action.code` is `return_custody_keep_local` - a terminal run whose recorded pipeline head cannot be verified, either because no reachable object store still has it or because the run's own recovery evidence names a different commit - run `no-mistakes axi sync --recover --keep-local` to return custody at your current head and point the gate branch at it. +If any copy of that head still exists, the settlement anchors it at `refs/no-mistakes/recover-stranded/` before moving the gate branch off it - but only in whichever repository still holds the object, your worktree or the local gate under `NM_HOME` or both, so look in both rather than assuming your own; if no store still has it, nothing is anchored because nothing was left to anchor and the settlement took nothing from you. +That anchor is not `refs/no-mistakes/recover/`, which in this state is the conflicting evidence and can name an entirely different commit. A `branch_sync.state` of `user_owned` means the run went terminal before changing the submitted head and cancellation released the branch: it is immediately usable and needs no sync action. When `next_action.code` is `continue_active_run`, run the reported command and keep driving the active run. If synchronization is blocked, process that state instead of improvising reset, stash, merge, rebase, force, or branch replacement. @@ -216,7 +219,7 @@ All agents implement the same interface. Each invocation receives: - **Environment** - the daemon environment plus non-interactive Git overrides (`GIT_EDITOR=true`, `GIT_SEQUENCE_EDITOR=true`, and `GIT_TERMINAL_PROMPT=0`) so agent-invoked Git commands do not hang on editors or credential prompts - **JSONSchema** - optional structured output schema for typed responses - **OnChunk** - callback for streaming text output to the TUI -- **OnLifecycle** - callback for native subprocess start, exit, and retry activity that is recorded in step logs and AXI active-step status +- **OnLifecycle** - callback for native subprocess start, exit, retry, fallback, and output-liveness activity; control events reach step logs and AXI active-step status, while throttled output liveness updates status without flooding the log - **Session** - optional no-mistakes-owned native session identity for review-fixer reuse - **Purpose** - local performance label for the pipeline duty served diff --git a/docs/src/content/docs/guides/troubleshooting.md b/docs/src/content/docs/guides/troubleshooting.md index 16a55fefc..98e35d0da 100644 --- a/docs/src/content/docs/guides/troubleshooting.md +++ b/docs/src/content/docs/guides/troubleshooting.md @@ -263,7 +263,7 @@ It does not cancel the step, fail the run, or mean the pipeline is safe to bypas A quiet Review step still ends on its own: its agent turns are bounded by [`review_agent_timeout`](/no-mistakes/reference/global-config/#review_agent_timeout), after which the run fails with a timeout diagnostic in the step log. A quiet Test step is bounded the same way by [`test_agent_timeout`](/no-mistakes/reference/global-config/#test_agent_timeout), covering the post-test evidence-gathering agent and a Test-repair turn. -Every other agent-spawning step (Document, Lint, Rebase conflict repair, PR drafting, CI auto-fix) is bounded by [`agent_timeout`](/no-mistakes/reference/global-config/#agent_timeout), so a stall reaches the step's normal agent-error handling instead of remaining active until you abort. Those errors fail agent-driven mutation steps; PR drafting instead continues with its deterministic fallback content. +Every other agent-spawning step (Document, Lint, Rebase conflict repair, PR drafting, CI auto-fix) is bounded by [`agent_timeout`](/no-mistakes/reference/global-config/#agent_timeout), so a stall reaches the step's normal agent-error handling instead of remaining active until you abort. Most mutation steps fail, PR drafting continues with deterministic fallback content, and CI auto-fix parks for a user decision as described in the [CI step reference](/no-mistakes/reference/pipeline-steps/#ci). Start by reading the active run and the step log: diff --git a/docs/src/content/docs/guides/tui.md b/docs/src/content/docs/guides/tui.md index 3f223ffb4..96d966616 100644 --- a/docs/src/content/docs/guides/tui.md +++ b/docs/src/content/docs/guides/tui.md @@ -144,6 +144,7 @@ Pressing `u` explicitly refreshes the configured upstream or fork target, then o Confirm with `u` or Enter, or cancel with Escape. The apply path rechecks every mutable assumption and can only perform the same exact strict fast-forward or anchored equivalent-diverged advance as `no-mistakes sync`; blocked states never trigger destructive Git recovery. When the owning run ended without publishing its pipeline commits, the same box offers `u recover custody` instead: `u` opens a confirmation naming the terminal status, the local head, and the preserved head, and applying routes through the guarded recovery documented in [`no-mistakes axi sync`](/no-mistakes/reference/cli/#no-mistakes-axi-sync). +When the state instead carries `next_action.code: return_custody_keep_local` - the terminal run's recorded pipeline head can no longer be verified, so there is nothing to recover - the box offers `u settle custody at local head`, and its own separate confirmation names the kept local head and the unverifiable recorded head before applying. Settlement keeps your head where recovery takes the preserved one, so the two never share a confirmation; applying routes through the same `--recover --keep-local` settlement documented in [`no-mistakes axi sync`](/no-mistakes/reference/cli/#no-mistakes-axi-sync). ### Footer @@ -193,7 +194,7 @@ When the instruction editor is open, press `Ctrl+s` or `Ctrl+enter` to save, or | `?` | Toggle help overlay | | `y` | Toggle yolo mode, which auto-resolves paused steps | | `r` | Retry a failed fix-review state or diff load; otherwise start a rerun after a failed or cancelled run | -| `u` | Refresh and confirm local branch synchronization, or confirm custody recovery, when offered | +| `u` | Refresh and confirm local branch synchronization, or confirm custody recovery or custody settlement at the local head, when offered | | `q` | Detach from TUI (or quit if run is done) | In diff view, `n`/`p` jumps the viewport to the file and line of the next/previous finding. diff --git a/docs/src/content/docs/reference/cli.md b/docs/src/content/docs/reference/cli.md index f7621dc17..d6f16c72f 100644 --- a/docs/src/content/docs/reference/cli.md +++ b/docs/src/content/docs/reference/cli.md @@ -115,7 +115,7 @@ When starting a new run, `axi run` refuses the default branch and uncommitted wo Reattaching to an in-flight run does not require `--intent`. Reattachment accepts either the run's immutable submitted head or its current pipeline head, so pipeline-created fix commits do not detach an unchanged submitting worktree. When neither identity matches, `axi run` keeps the fresh-run path but refuses a gate push while `branch_sync` says the pipeline still owns the branch. -That refusal returns the complete structured state and its `continue_active_run` or `recover_custody` next action instead of a raw Git non-fast-forward. +That refusal returns the complete structured state and whichever next action that state carries - `continue_active_run`, `recover_custody`, or the `return_custody_keep_local` settlement described under [`no-mistakes axi sync`](#no-mistakes-axi-sync) - instead of a raw Git non-fast-forward. Reattaching to an in-flight run can proceed while the daemon is already running even if the global config file has become invalid, but starting a fresh run still requires valid global config. Starting a fresh run also requires a runnable effective pipeline agent. If the configured native agent or ACP runner is unavailable, the run fails before any pipeline step starts instead of reporting command-only validation as a passed gate. @@ -205,7 +205,7 @@ no-mistakes axi sync --recover --keep-local | -------------- | ------ | ------- | ---------------------------------------------------------------------------- | | `--check` | `bool` | `false` | Verify the live target and exact plan without changing `HEAD` | | `--recover` | `bool` | `false` | Return custody of a branch stranded by a terminal run with unpublished pipeline commits (a no-op when cancellation already released the branch) | -| `--keep-local` | `bool` | `false` | With `--recover`: keep the current local head; never touches the worktree | +| `--keep-local` | `bool` | `false` | With `--recover`: keep the current local head; never touches the worktree, and where the gate branch still names a different head, points it at the kept head | The default command is an explicit non-interactive apply request and never prompts. All modes return the complete `branch_sync` object as TOON. @@ -223,7 +223,7 @@ Run `axi sync` only when structured output offers `next_action.code: sync`; proc A run that goes terminal (cancelled, failed, or completed without a push stage) after moving the pipeline head leaves the branch `pipeline_owned`. Status offers `next_action.code: recover_custody` only when recovery can establish the same eligibility it will enforce: an equal or ahead local head proves the source locally and can create the local anchor when the gate is unavailable, but any existing gate recovery ref must still match the recorded head; importing a missing preserved head requires an exact run-specific gate anchor (or legacy commit evidence that can be anchored), a clean worktree, and either local ancestry or the content-preservation proof described below. The eligible state reports `safety: blocked_pipeline_owned_recoverable`, the run's terminal `pipeline.status`, and the exact `submitted_head`/`current_head`/`relation` ownership facts. A run whose terminalization verifies that the managed worktree head never changed from the submitted head releases the branch instead: the terminal outcome, including cancellation, ends ownership; status reports `state: user_owned` with the same exact ownership facts and no `next_action`; the branch and head are immediately usable for any separately authorized delivery; and nothing blocks a direct push or PR. -Without positive evidence that the submitted head stayed unchanged, custody is not guessed away. Missing or conflicting evidence, and import cases with a dirty worktree or genuinely divergent history, require manual reconciliation instead of advertising a recovery that will refuse. +Without positive evidence that the submitted head stayed unchanged, custody is not guessed away. Missing or conflicting evidence, and import cases with a dirty worktree or genuinely divergent history, never advertise a recovery that will refuse: where the recorded head can no longer be verified at all, status names the `--keep-local` settlement described below, and every remaining shape reports manual reconciliation. While a run is still active, it reports `state: pipeline_owned`, the exact submitted/current heads and their relation, and `next_action.code: continue_active_run` with `no-mistakes axi status`, even when its head has not moved yet. `--recover` verifies the run is terminal, anchors the preserved head under `refs/no-mistakes/recover/` in the invoking repository, and stamps custody returned so a fresh run can start. For equal or ahead worktrees where the preserved head is already locally reachable, recovery writes that anchor locally without requiring gate access. If the gate is available, an existing symbolic, non-commit, or mismatched recovery ref is conflicting evidence and recovery refuses without overwriting it. @@ -231,12 +231,19 @@ For behind or diverged worktrees, recovery verifies the preserved head at the ru A clean behind worktree fast-forwards. A diverged worktree is adopted only when the preserved head provably carries every local change, proven by an executable three-way merge whose result is exactly the preserved head's tree. This covers a pipeline rebase onto a newer base without requiring the gate branch to advance to the preserved head. -Terminalization pins a verified unpublished pipeline head under a run-specific recovery ref, so recovery does not require the gate branch itself to have advanced. If the recorded head is genuinely missing, status reports manual reconciliation instead of advertising `recover_custody`. +Terminalization pins a verified unpublished pipeline head under a run-specific recovery ref, so recovery does not require the gate branch itself to have advanced. If the recorded head can no longer be verified - it is missing from every reachable object store, or the run's own recovery ref names a different object - status reports `next_action.code: return_custody_keep_local` instead of advertising `recover_custody`, and `--recover --keep-local` settles that self-inconsistent record. That adoption anchors the pre-recovery local head under `refs/no-mistakes/recover-local/`, then moves the branch with Git operations that refuse on their own rather than after a preceding check: an atomic compare-and-swap on the branch ref, and a working-tree update that aborts instead of overwriting a modified or untracked file. The proof is deliberately narrow and never uses patch identity, which discards hunk locations and whitespace and so cannot tell a genuine replay from a same-shaped edit elsewhere. Anything it cannot decide - unlanded local commits, or a rebase whose fix rounds also rewrote your own lines - still refuses with the anchor named, because only escalation can tell a deliberate pipeline fix apart from a dropped change. A dirty worktree refuses with explicit choices. -When you explicitly keep a behind or diverged local head instead of taking the preserved head, `--keep-local` returns custody at the current head without touching the worktree and atomically points the gate branch at it. If the gate branch moved independently, recovery first preserves that head under `refs/no-mistakes/recover-gate/`; a conflicting pre-existing anchor makes recovery refuse, and a concurrent gate push wins the compare-and-swap and also makes recovery refuse. +When you explicitly keep a behind or diverged local head instead of taking the preserved head, `--keep-local` returns custody at the current head without touching the worktree and atomically points the gate branch at it. Where the gate branch already names the kept head - and on the paths that return before the gate is reached at all, such as a released `user_owned` branch, an already-returned custody record, and a local head that already contains the preserved one - custody returns without moving any branch ref. Those paths can still create the private recovery anchor, which is what anchoring a locally reachable preserved head means. If the gate branch moved independently, recovery first preserves that head under `refs/no-mistakes/recover-gate/`; a conflicting pre-existing anchor makes recovery refuse, and a concurrent gate push wins the compare-and-swap and also makes recovery refuse. The anchor is written immediately before the compare-and-swap, so every refusal that can precede it leaves no anchor of its own behind. Their no-change report stays exact for what the recovery as a whole wrote: where the attempt had already anchored the preserved head - under `refs/no-mistakes/recover/` on this path, or under `refs/no-mistakes/recover-stranded/` when it arrived through the settlement below - each of those refusals reports that no branch, worktree, or file changes were made and names the anchor and the repository holding it, instead of claiming nothing was written. A lost compare-and-swap does leave that anchor pinned at the head the attempt observed, because the gate has moved and the anchor may be the only ref still naming the displaced head; nothing retires it, so once the gate has moved on, later attempts refuse on the anchor conflict until you reconcile that ref in the gate by hand. That refusal names the ref and the gate directory, and reports that no *local* files or refs were changed - it keeps that narrower claim precisely because it has just written a ref in the gate, and where the attempt had already anchored the preserved head it appends the same anchor note rather than dropping the `local` qualifier. + +`--keep-local` is also the settlement for a self-inconsistent custody record: a terminal run whose recorded pipeline head cannot be verified has no preserved head to import, so the default `--recover` refuses - with `safety: blocked_recover_preserved_head_missing`, `blocked_recover_anchor_mismatch`, or `blocked_recover_preserve_failed` when the conflicting recovery ref is the invoking worktree's own and the preserved head is already reachable from the local branch - and nothing else could settle the branch. Those refusals are settleable with `--keep-local` only where recovery can actually reach the settlement and the settlement can complete: a record whose own recovery evidence cannot be inspected at all, is symbolic, or sits mid-adoption is refused earlier, and one whose local gate branch cannot be read or whose recovery anchors are already occupied by another commit is refused inside the settlement, so status keeps reporting manual reconciliation for all of those rather than advertising a settlement that would also refuse. +Settlement pins every reachable copy of the recorded head under `refs/no-mistakes/recover-stranded/` first, so a head that still exists survives as inspectable evidence; if such a head exists and cannot be pinned, the settlement refuses with `safety: blocked_recover_preserve_failed` rather than stranding it. +The gate branch then moves by the same compare-and-swap, so a concurrent gate push still wins and the settlement refuses; only a gate branch proven absent settles without that swap, since an unreadable gate branch is not evidence of absence. +The settlement's own refusals, and every refusal raised while `--keep-local` moves the gate branch, name `next_action.code: inspect_and_reconcile_manually`. Other `--recover` refusals carry whatever next action their state already defines, and some carry none - including a missing recorded head under the default `--recover`. A failure to record custody AFTER the move itself succeeded is not a refusal and is reported as `safety: blocked_recover_stamp_failed`: the Git changes are already applied, so it says so and names the same recovery command again (`recover_custody`, or `return_custody_keep_local` for `--keep-local`), which completes the record because every Git step it repeats is idempotent once applied. +A terminal run whose head was never verified is excluded: recovery refuses it earlier, so status keeps reporting manual reconciliation rather than advertising a settlement that would refuse. +`no-mistakes axi abort` on an already-terminal run stays an idempotent no-op - there is nothing left to cancel - but its response names that settlement command when the invoking worktree's branch is still held by that run. `no-mistakes rerun` is the alternative exit that resumes validating the preserved head instead of taking the branch back. A recovered never-pushed run reports `state: custody_returned`; a recovered pushed run reports its ordinary classification against the last push binding, typically `local_ahead`. On a `user_owned` branch, `--recover` is an idempotent no-op success: nothing pipeline-created exists to recover, and no file, ref, or database row changes. @@ -363,7 +370,7 @@ no-mistakes sync --recover --keep-local | `--check` | `bool` | `false` | Verify and print the fresh plan without changing `HEAD` | | `-y`, `--yes` | `bool` | `false` | Apply an eligible guarded synchronization without an interactive prompt | | `--recover` | `bool` | `false` | Return custody of a branch stranded by a terminal run with unpublished pipeline commits (a no-op when cancellation already released the branch) | -| `--keep-local` | `bool` | `false` | With `--recover`: keep the current local head; never touches the worktree | +| `--keep-local` | `bool` | `false` | With `--recover`: keep the current local head; never touches the worktree, and where the gate branch still names a different head, points it at the kept head | Without `--yes`, apply prints the exact full-SHA plan and requires TTY confirmation; `--recover` prompts the same way before returning custody. A non-TTY apply or recovery refuses with a direct `--yes` hint. diff --git a/docs/src/content/docs/reference/global-config.md b/docs/src/content/docs/reference/global-config.md index 0e687c9b3..642a824d0 100644 --- a/docs/src/content/docs/reference/global-config.md +++ b/docs/src/content/docs/reference/global-config.md @@ -394,9 +394,18 @@ For older active runs that do not yet have activity rows, AXI falls back to the Maximum wall-clock time for one pipeline agent invocation that does not already have a more specific deadline. This is the default-by-construction budget: Document, Lint, Rebase conflict repair, PR drafting, CI auto-fix, and any future agent-spawning step are bounded even if they forget to install their own timer. Review still uses [`review_agent_timeout`](#review_agent_timeout) as a per-round budget, Test still uses [`test_agent_timeout`](#test_agent_timeout) per invocation, and Intent keeps its five-minute extraction cap; any existing deadline is honored rather than capped. -When this deadline expires, the agent is cancelled and the invocation returns a timeout diagnostic instead of remaining active indefinitely. Agent-driven mutation steps fail the run, while PR drafting follows its existing agent-error fallback and continues with deterministic content. +When this deadline expires, the agent is cancelled and the invocation returns a timeout diagnostic instead of remaining active indefinitely. Most agent-driven mutation steps fail the run, CI auto-fix parks for a user decision, and PR drafting follows its existing agent-error fallback and continues with deterministic content. The [CI step reference](/no-mistakes/reference/pipeline-steps/#ci) owns the approval behavior. A late successful return after the deadline is rejected, so post-agent commits and PR content cannot use work from a timed-out turn. +The diagnostic reports what was actually measured, not the budget restated. Evidence resets whenever a retry or fallback starts a replacement attempt, including provider fallback, failed session resume, and OpenCode's prompt-only structured-output fallback, so the diagnostic describes only the attempt that reached the deadline: + +- `agent produced no output at all in 30m0s after its subprocess started (pid=1234)` - the current attempt launched and then emitted nothing. Check that the agent CLI is authenticated and responsive. +- `agent last produced output 4s ago (312 observed)` - the current attempt was working right up to the deadline. The turn needs a larger budget, or the request is too large for one turn. +- `agent produced no output at all in 30m0s and never reported a subprocess start` - the current attempt never reached a running agent process. + +Output means anything observable: streamed assistant text, or raw bytes on the agent subprocess's stdout or stderr. Subprocess bytes matter because an agent spends most of a long turn running tools rather than writing prose, so prose alone cannot tell a working agent from a wedged one. +Any substantive report from the agent adapter - for a native agent, its exit status and captured stderr - is appended to the diagnostic as `agent reported: ...`; credential-bearing URLs are redacted and the report is length-bounded before it can reach logs or findings. A bare context cancellation is omitted because it adds no evidence. + | | | | ------- | ---------------------- | | Type | `string` (Go duration) | @@ -412,6 +421,7 @@ It is global-only: repository config and environment variables cannot override i Maximum wall-clock time for the Review step's agent turns in one review round. The budget starts at that round's first agent turn and covers its optional review-fix turn plus the rereview turn together; every later auto-fix round starts a fresh budget. When the deadline expires, the review agent is cancelled and the run fails with a diagnostic naming the timeout instead of remaining active indefinitely. +That diagnostic carries the same measured evidence and adapter report described under [`agent_timeout`](#agent_timeout). | | | | ------- | ---------------------- | @@ -427,6 +437,7 @@ Raise it for repositories whose reviews legitimately run long; it bounds only th Maximum wall-clock time for one Test-step agent invocation. The budget covers the post-test evidence-gathering turn, and a Test-repair turn gets its own budget of the same length. When the deadline expires, the test agent is cancelled and the run fails with a diagnostic naming the timeout instead of remaining active indefinitely. +That diagnostic carries the same measured evidence and adapter report described under [`agent_timeout`](#agent_timeout). | | | | ------- | ---------------------- | diff --git a/docs/src/content/docs/reference/pipeline-steps.md b/docs/src/content/docs/reference/pipeline-steps.md index d3350026b..f84e068aa 100644 --- a/docs/src/content/docs/reference/pipeline-steps.md +++ b/docs/src/content/docs/reference/pipeline-steps.md @@ -301,6 +301,7 @@ Monitors PR health after creation and auto-fixes CI failures. Mergeability polli - On GitHub, includes unresolved review-thread comments from supported review bots (currently Greptile) in CI repair prompts when an auto-fix attempt starts; the comments are framed as untrusted external data and the rendered section is capped at 32 KiB - Preserves steps already skipped for the run when restarting validation, including after recovery from a daemon restart - Bounds that CI-fix agent with [`agent_timeout`](/no-mistakes/reference/global-config/#agent_timeout): an expired budget cancels the agent and fails the attempt with a timeout diagnostic rather than leaving the run active indefinitely, and a late successful return after the deadline is not committed +- If the CI-fix agent exhausts that budget, pauses for user approval instead of re-issuing the same request on the next poll. A budget burn is not transient - repeating it costs another full budget - so the remaining auto-fix attempts are left for the user to spend deliberately with a fix response. The finding carries the measured timeout diagnostic and, when the timed-out agent left uncommitted work in the run worktree, that worktree's path. Ordinary (non-timeout) fix failures keep retrying as before - On GitHub, GitLab, Forgejo, or Azure DevOps merge conflict: asks the agent to rebase onto the latest PR base branch tip and make the smallest correct root-cause fix for the conflicts, using user intent when available - If both CI failures and a GitHub, GitLab, Forgejo, or Azure DevOps merge conflict are present: fixes both in the same attempt - If a fix attempt produces no changes: automatic mode leaves the failure undeduplicated so it can retry until the auto-fix limit, while manual fix mode returns immediately for manual intervention diff --git a/internal/agent/acpx.go b/internal/agent/acpx.go index e5c820fe0..d323e76ca 100644 --- a/internal/agent/acpx.go +++ b/internal/agent/acpx.go @@ -53,7 +53,7 @@ func (a *acpxAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error) if err != nil { return nil, fmt.Errorf("acpx stdin pipe: %w", err) } - started, err := startNativeAgentCommand(cmd) + started, err := startNativeAgentCommand(cmd, nativeAgentActivityObserver(opts, a.Name())) if err != nil { _ = stdin.Close() return nil, fmt.Errorf("acpx start: %w", err) diff --git a/internal/agent/antigravity.go b/internal/agent/antigravity.go index 0dbee2d6e..d215194a6 100644 --- a/internal/agent/antigravity.go +++ b/internal/agent/antigravity.go @@ -98,7 +98,7 @@ func (a *antigravityAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, cmd.Env = a.gitSafeEnv(opts.CWD) shellenv.ConfigureShellCommand(cmd) - started, err := startNativeAgentCommand(cmd) + started, err := startNativeAgentCommand(cmd, nativeAgentActivityObserver(opts, "antigravity")) if err != nil { return nil, fmt.Errorf("antigravity start: %w", err) } diff --git a/internal/agent/claude.go b/internal/agent/claude.go index 82598173f..f157ef3a3 100644 --- a/internal/agent/claude.go +++ b/internal/agent/claude.go @@ -83,7 +83,7 @@ func (a *claudeAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error var stderrBuf []byte var stderrWG sync.WaitGroup - started, err := startNativeAgentCommand(cmd) + started, err := startNativeAgentCommand(cmd, nativeAgentActivityObserver(opts, "claude")) if err != nil { return nil, fmt.Errorf("claude start: %w", err) } diff --git a/internal/agent/codex.go b/internal/agent/codex.go index 9c83b3d1b..bc9f66c86 100644 --- a/internal/agent/codex.go +++ b/internal/agent/codex.go @@ -99,7 +99,7 @@ func (a *codexAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error) var stderrBuf []byte var stderrWG sync.WaitGroup - started, err := startNativeAgentCommand(cmd) + started, err := startNativeAgentCommand(cmd, nativeAgentActivityObserver(opts, "codex")) if err != nil { return nil, fmt.Errorf("codex start: %w", err) } diff --git a/internal/agent/copilot.go b/internal/agent/copilot.go index 86271511c..b27829d07 100644 --- a/internal/agent/copilot.go +++ b/internal/agent/copilot.go @@ -46,7 +46,7 @@ func (a *copilotAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, erro var stderrBuf []byte var stderrWG sync.WaitGroup - started, err := startNativeAgentCommand(cmd) + started, err := startNativeAgentCommand(cmd, nativeAgentActivityObserver(opts, "copilot")) if err != nil { return nil, fmt.Errorf("copilot start: %w", err) } diff --git a/internal/agent/fallback.go b/internal/agent/fallback.go index 423436e96..78d1d0d06 100644 --- a/internal/agent/fallback.go +++ b/internal/agent/fallback.go @@ -107,9 +107,7 @@ func (a *fallbackAgent) Run(ctx context.Context, opts RunOpts) (*Result, error) return nil, err } next := candidates[i+1] - if opts.OnChunk != nil { - opts.OnChunk(fmt.Sprintf("\nagent %s failed (%s); falling back to %s\n", current.Name(), fallbackReason(err), next.Name())) - } + emitAgentFallback(opts, current.Name(), next.Name(), err) } return nil, lastErr } diff --git a/internal/agent/grok.go b/internal/agent/grok.go index f712edafa..9a10edd7c 100644 --- a/internal/agent/grok.go +++ b/internal/agent/grok.go @@ -103,7 +103,7 @@ func (a *grokAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error) var stderrBuf []byte var stderrWG sync.WaitGroup - started, err := startNativeAgentCommand(cmd) + started, err := startNativeAgentCommand(cmd, nativeAgentActivityObserver(opts, "grok")) if err != nil { return nil, fmt.Errorf("grok start: %w", err) } diff --git a/internal/agent/lifecycle.go b/internal/agent/lifecycle.go index 3f0fdbda4..87a095306 100644 --- a/internal/agent/lifecycle.go +++ b/internal/agent/lifecycle.go @@ -1,6 +1,10 @@ package agent -import "fmt" +import ( + "fmt" + "sync" + "time" +) const ( // LifecyclePhaseStart marks native subprocess startup. @@ -9,8 +13,27 @@ const ( LifecyclePhaseExit = "exit" // LifecyclePhaseRetry marks a transient retry before the next subprocess attempt. LifecyclePhaseRetry = "retry" + // LifecyclePhaseFallback marks any fallback before a fresh agent attempt, + // including provider, session-resume, and structured-output fallbacks. + LifecyclePhaseFallback = "fallback" + // LifecyclePhaseActivity marks observed liveness of a running native + // subprocess: bytes arrived on its stdout or stderr. + // + // Start and exit alone cannot tell a wedged agent from a working one. Every + // adapter forwards only assistant prose to OnChunk, and an agent spends most + // of a long turn emitting tool events instead, so a healthy multi-minute fix + // round is indistinguishable from a process that is blocked before its first + // byte. Together with streamed assistant text, this phase supplies the + // measured output evidence used by invocation-timeout diagnostics. + LifecyclePhaseActivity = "activity" ) +// nativeAgentActivityInterval throttles LifecyclePhaseActivity so a chatty +// subprocess cannot flood the observer. It is a liveness signal, not a log: the +// first byte of a quiet period is reported immediately and further bytes are +// coalesced until the interval elapses. +const nativeAgentActivityInterval = 5 * time.Second + func emitAgentStarted(opts RunOpts, name string, pid int) { emitLifecycle(opts, LifecycleEvent{ Agent: name, @@ -33,18 +56,59 @@ func emitAgentExited(opts RunOpts, name string, pid int, err error) { }) } -func emitAgentRetry(opts RunOpts, name string, label string, attempt, max int) { - message := fmt.Sprintf("%s retrying after transient error %q (attempt %d/%d)", name, label, attempt, max) - if opts.OnLifecycle != nil { +// nativeAgentActivityObserver returns the throttled liveness callback handed to +// startNativeAgentCommand, or nil when nobody is observing this invocation. +// Returning nil keeps the read path allocation-free for callers that do not +// care (tests, eval replay). +func nativeAgentActivityObserver(opts RunOpts, name string) func() { + if opts.OnLifecycle == nil { + return nil + } + var ( + mu sync.Mutex + lastEmit time.Time + ) + return func() { + now := time.Now() + mu.Lock() + if !lastEmit.IsZero() && now.Sub(lastEmit) < nativeAgentActivityInterval { + mu.Unlock() + return + } + lastEmit = now + mu.Unlock() emitLifecycle(opts, LifecycleEvent{ Agent: name, - Phase: LifecyclePhaseRetry, - Message: message, + Phase: LifecyclePhaseActivity, + Message: fmt.Sprintf("%s producing output", name), }) + } +} + +func emitAgentRetry(opts RunOpts, name string, label string, attempt, max int) { + message := fmt.Sprintf("%s retrying after transient error %q (attempt %d/%d)", name, label, attempt, max) + emitAgentControl(opts, LifecycleEvent{ + Agent: name, + Phase: LifecyclePhaseRetry, + Message: message, + }) +} + +func emitAgentFallback(opts RunOpts, current, next string, err error) { + emitAgentControl(opts, LifecycleEvent{ + Agent: current, + Phase: LifecyclePhaseFallback, + Message: fmt.Sprintf("agent %s failed (%s); falling back to %s", current, fallbackReason(err), next), + }) +} + +func emitAgentControl(opts RunOpts, event LifecycleEvent) { + if opts.OnLifecycle != nil { + emitLifecycle(opts, event) return } if opts.OnChunk != nil { - opts.OnChunk(message) + opts.OnChunk(event.Message) } } diff --git a/internal/agent/native_command.go b/internal/agent/native_command.go index 2faf035f9..e78d7f29d 100644 --- a/internal/agent/native_command.go +++ b/internal/agent/native_command.go @@ -38,10 +38,17 @@ type nativeAgentPipe struct { file *os.File done func() doneOnce sync.Once + // activity, when set, is called for every non-empty read. It is the only + // evidence no-mistakes has that a native agent is still alive during the + // long tool-using stretches that produce no assistant prose. + activity func() } func (p *nativeAgentPipe) Read(b []byte) (int, error) { n, err := p.file.Read(b) + if n > 0 && p.activity != nil { + p.activity() + } if err != nil { p.markDone() } @@ -58,7 +65,11 @@ func (p *nativeAgentPipe) markDone() { p.doneOnce.Do(p.done) } -func startNativeAgentCommand(cmd *exec.Cmd) (*nativeAgentCommand, error) { +// startNativeAgentCommand starts cmd with dedicated stdout/stderr pipes. +// activity, when non-nil, is invoked on every non-empty read from either pipe; +// see LifecyclePhaseActivity for why subprocess byte liveness - not assistant +// prose - is the signal that distinguishes a working agent from a wedged one. +func startNativeAgentCommand(cmd *exec.Cmd, activity func()) (*nativeAgentCommand, error) { stdoutR, stdoutW, err := os.Pipe() if err != nil { return nil, fmt.Errorf("stdout pipe: %w", err) @@ -88,8 +99,8 @@ func startNativeAgentCommand(cmd *exec.Cmd) (*nativeAgentCommand, error) { remainingPipes: 2, pipesDone: make(chan struct{}), } - started.stdout = &nativeAgentPipe{file: stdoutR, done: started.markPipeDone} - started.stderr = &nativeAgentPipe{file: stderrR, done: started.markPipeDone} + started.stdout = &nativeAgentPipe{file: stdoutR, done: started.markPipeDone, activity: activity} + started.stderr = &nativeAgentPipe{file: stderrR, done: started.markPipeDone, activity: activity} go func() { err := cmd.Wait() started.terminate() diff --git a/internal/agent/opencode.go b/internal/agent/opencode.go index 2e9a9c7c4..4ece020cd 100644 --- a/internal/agent/opencode.go +++ b/internal/agent/opencode.go @@ -103,6 +103,11 @@ func (a *opencodeAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, err // tool call. Some thinking-enabled models reject that combination. Retry // once without the native format, while keeping the schema in the prompt // and validating the returned JSON against it in finalizeTextResult. + emitAgentControl(opts, LifecycleEvent{ + Agent: a.Name(), + Phase: LifecyclePhaseFallback, + Message: "opencode starting a fresh prompt-only structured output session", + }) result, fallbackErr := a.runOnceWithFormat(ctx, opts, false) if fallbackErr != nil { return nil, fmt.Errorf("opencode prompt-only structured output fallback: %w", fallbackErr) diff --git a/internal/agent/opencode_test.go b/internal/agent/opencode_test.go index dc92e2f89..7ac1f4bf8 100644 --- a/internal/agent/opencode_test.go +++ b/internal/agent/opencode_test.go @@ -524,6 +524,7 @@ func TestOpencodeAgent_StructuredOutputError(t *testing.T) { func TestOpencodeAgent_ThinkingToolChoiceConflictFallsBackToValidatedText(t *testing.T) { var sessions atomic.Int32 + var eventStreams atomic.Int32 var nativeFormatSeen atomic.Bool var fallbackFormatSeen atomic.Bool @@ -534,6 +535,10 @@ func TestOpencodeAgent_ThinkingToolChoiceConflictFallsBackToValidatedText(t *tes fmt.Fprintf(w, `{"id":"s%d"}`, id) case r.URL.Path == "/global/event" && r.Method == http.MethodGet: + if eventStreams.Add(1) == 1 { + fmt.Fprint(w, "data: {\"payload\":{\"type\":\"message.part.updated\",\"properties\":{\"sessionID\":\"s1\",\"part\":{\"id\":\"p1\",\"messageID\":\"msg1\",\"type\":\"text\",\"text\":\"thinking before conflict\"}}}}\n\n") + fmt.Fprint(w, "data: {\"payload\":{\"type\":\"message.updated\",\"properties\":{\"sessionID\":\"s1\",\"info\":{\"id\":\"msg1\",\"role\":\"assistant\"}}}}\n\n") + } fmt.Fprint(w, "data: {\"payload\":{\"type\":\"session.idle\"}}\n\n") case r.URL.Path == "/session/s1/message" && r.Method == http.MethodPost: @@ -567,10 +572,18 @@ func TestOpencodeAgent_ThinkingToolChoiceConflictFallsBackToValidatedText(t *tes bin: "opencode", server: &managedServer{port: mustParsePort(server.URL)}, } + var chunks []string + var fallbackEvents int result, err := a.Run(context.Background(), RunOpts{ Prompt: "review the changes", CWD: t.TempDir(), JSONSchema: json.RawMessage(`{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"],"additionalProperties":false}`), + OnChunk: func(text string) { chunks = append(chunks, text) }, + OnLifecycle: func(event LifecycleEvent) { + if event.Phase == LifecyclePhaseFallback { + fallbackEvents++ + } + }, }) if err != nil { t.Fatalf("Run: %v", err) @@ -587,6 +600,12 @@ func TestOpencodeAgent_ThinkingToolChoiceConflictFallsBackToValidatedText(t *tes if got := sessions.Load(); got != 2 { t.Fatalf("sessions = %d, want 2", got) } + if !strings.Contains(strings.Join(chunks, ""), "thinking before conflict") { + t.Fatalf("chunks = %q, want output from the failed native-format attempt", chunks) + } + if fallbackEvents != 1 { + t.Fatalf("fallback lifecycle events = %d, want one fresh prompt-only attempt boundary", fallbackEvents) + } t.Logf("native format=%v; fallback format=%v; validated output=%s", nativeFormatSeen.Load(), fallbackFormatSeen.Load(), result.Output) } diff --git a/internal/agent/pi.go b/internal/agent/pi.go index 479ba7f20..2786bf6de 100644 --- a/internal/agent/pi.go +++ b/internal/agent/pi.go @@ -72,7 +72,7 @@ func (a *piAgent) runOnce(ctx context.Context, opts RunOpts) (*Result, error) { return nil, fmt.Errorf("pi stdin pipe: %w", err) } - started, err := startNativeAgentCommand(cmd) + started, err := startNativeAgentCommand(cmd, nativeAgentActivityObserver(opts, "pi")) if err != nil { _ = stdin.Close() return nil, fmt.Errorf("pi start: %w", err) diff --git a/internal/agent/pi_test.go b/internal/agent/pi_test.go index 630f6f383..b24a95cd2 100644 --- a/internal/agent/pi_test.go +++ b/internal/agent/pi_test.go @@ -611,3 +611,96 @@ sleep 30 t.Logf("got error: %v", err) } } + +// TestPiAgent_ToolOnlyStreamStillReportsSubprocessLiveness pins the proof of +// life that separates a working native agent from a wedged one. +// +// Verified against pi 0.84.3: a turn that uses tools emits tool_execution_start +// / tool_execution_update / tool_execution_end and toolcall_* assistant events, +// and no text_delta at all until the very end. Every adapter forwards only +// assistant prose to OnChunk, so for the whole tool-using stretch - which is +// most of a fix round - the pipeline saw nothing and could not tell the agent +// was alive. Subprocess byte liveness is that missing signal. +func TestPiAgent_ToolOnlyStreamStillReportsSubprocessLiveness(t *testing.T) { + dir := t.TempDir() + bin := writeFakePi(t, dir, `#!/bin/sh +cat > /dev/null +printf '%s\n' '{"type":"tool_execution_start","tool":"bash"}' +printf '%s\n' '{"type":"tool_execution_update","tool":"bash"}' +printf '%s\n' '{"type":"tool_execution_end","tool":"bash"}' +printf '%s\n' '{"type":"agent_end","messages":[{"role":"assistant","content":[{"type":"text","text":"done"}]}]}' +`, strings.Join([]string{ + "@echo off", + "more > nul", + "echo {\"type\":\"tool_execution_start\",\"tool\":\"bash\"}", + "echo {\"type\":\"tool_execution_update\",\"tool\":\"bash\"}", + "echo {\"type\":\"tool_execution_end\",\"tool\":\"bash\"}", + "echo {\"type\":\"agent_end\",\"messages\":[{\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"done\"}]}]}", + }, "\r\n")) + + var chunks []string + var phases []string + pa := &piAgent{bin: bin} + if _, err := pa.Run(context.Background(), RunOpts{ + Prompt: "fix ci", + CWD: t.TempDir(), + OnChunk: func(s string) { chunks = append(chunks, s) }, + OnLifecycle: func(e LifecycleEvent) { phases = append(phases, e.Phase) }, + }); err != nil { + t.Fatalf("run pi: %v", err) + } + + if len(chunks) != 0 { + t.Fatalf("OnChunk = %q, want a tool-only turn to stream no assistant prose", chunks) + } + activityAt := -1 + exitAt := -1 + for i, phase := range phases { + if phase == LifecyclePhaseActivity && activityAt < 0 { + activityAt = i + } + if phase == LifecyclePhaseExit { + exitAt = i + } + } + if activityAt < 0 { + t.Fatalf("lifecycle phases = %v, want subprocess liveness reported for a tool-only turn", phases) + } + if exitAt < 0 || activityAt > exitAt { + t.Fatalf("lifecycle phases = %v, want liveness reported while the agent was still running", phases) + } +} + +// TestPiAgent_SilentSubprocessReportsNoLiveness is the counter-test: an agent +// that produces no bytes must produce no liveness, or the signal would say +// every agent is fine and the wedge would stay invisible. +func TestPiAgent_SilentSubprocessReportsNoLiveness(t *testing.T) { + dir := t.TempDir() + bin := writeFakePi(t, dir, `#!/bin/sh +cat > /dev/null +exit 0 +`, strings.Join([]string{ + "@echo off", + "more > nul", + "exit 0", + }, "\r\n")) + + var phases []string + pa := &piAgent{bin: bin} + // A pi run with no events yields no text; the error is expected and is not + // what this test is about. + _, _ = pa.Run(context.Background(), RunOpts{ + Prompt: "fix ci", + CWD: t.TempDir(), + OnLifecycle: func(e LifecycleEvent) { phases = append(phases, e.Phase) }, + }) + + for _, phase := range phases { + if phase == LifecyclePhaseActivity { + t.Fatalf("lifecycle phases = %v, want no liveness from a subprocess that emitted nothing", phases) + } + } + if len(phases) == 0 { + t.Fatal("expected start and exit lifecycle events even for a silent subprocess") + } +} diff --git a/internal/agent/reap_unix_test.go b/internal/agent/reap_unix_test.go index 68de75a4f..6d538eca5 100644 --- a/internal/agent/reap_unix_test.go +++ b/internal/agent/reap_unix_test.go @@ -33,7 +33,7 @@ func TestNativeAgentCommand_WaitDelayClosesEscapedPipeHolder(t *testing.T) { shellenv.ConfigureShellCommand(cmd) cmd.WaitDelay = 100 * time.Millisecond - started, err := startNativeAgentCommand(cmd) + started, err := startNativeAgentCommand(cmd, nil) if err != nil { t.Fatalf("startNativeAgentCommand: %v", err) } diff --git a/internal/branchsync/recover_test.go b/internal/branchsync/recover_test.go index 99481251e..b57ef0c40 100644 --- a/internal/branchsync/recover_test.go +++ b/internal/branchsync/recover_test.go @@ -842,6 +842,10 @@ func TestRecoverUsesTerminalAnchorWhenGateBranchLags(t *testing.T) { } } +// TestInspectDoesNotAdvertiseRecoveryWhenRecordedHeadIsMissing keeps the #814 +// polarity - an unverifiable record never advertises recover_custody - while +// #824 replaces the dead-end manual-reconciliation pointer with the keep-local +// settlement, which is a command that can actually complete. func TestInspectDoesNotAdvertiseRecoveryWhenRecordedHeadIsMissing(t *testing.T) { t.Parallel() @@ -855,7 +859,7 @@ func TestInspectDoesNotAdvertiseRecoveryWhenRecordedHeadIsMissing(t *testing.T) if state.Safety != "blocked_recover_preserved_head_missing" { t.Fatalf("missing-head safety = %q, want blocked_recover_preserved_head_missing: %#v", state.Safety, state) } - if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + if state.NextAction == nil || state.NextAction.Code != "return_custody_keep_local" { t.Fatalf("missing-head next action = %#v", state.NextAction) } if state.NextAction.Code == "recover_custody" { @@ -869,7 +873,8 @@ func TestInspectDoesNotAdvertiseRecoveryWhenTerminalAnchorConflicts(t *testing.T f := newRecoverFixture(t, types.RunCancelled) // The recorded preserved commit remains available in the gate, but the // run-specific evidence points elsewhere. Status must honor that conflict - // instead of advertising a recovery command that Recover will refuse. + // instead of advertising a recovery command that Recover will refuse, and + // must point at the keep-local settlement that can complete instead. mustRun(t, f.local, "fetch", f.gate, f.preserved) mustRun(t, f.gate, "update-ref", f.anchorRef(), f.submitted) @@ -877,7 +882,7 @@ func TestInspectDoesNotAdvertiseRecoveryWhenTerminalAnchorConflicts(t *testing.T if state.Safety != "blocked_recover_preserved_head_missing" { t.Fatalf("conflicting-anchor safety = %q, want blocked_recover_preserved_head_missing: %#v", state.Safety, state) } - if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + if state.NextAction == nil || state.NextAction.Code != "return_custody_keep_local" { t.Fatalf("conflicting-anchor next action = %#v", state.NextAction) } if state.NextAction.Code == "recover_custody" { @@ -896,7 +901,7 @@ func TestInspectDoesNotAdvertiseRecoveryWhenTerminalAnchorIsNotACommit(t *testin mustRun(t, f.gate, "update-ref", f.anchorRef(), blob) state := f.service.InspectCached(f.ctx) - if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + if state.NextAction == nil || state.NextAction.Code != "return_custody_keep_local" { t.Fatalf("non-commit-anchor next action = %#v", state.NextAction) } if state.NextAction.Code == "recover_custody" { @@ -1341,7 +1346,12 @@ func TestRecoverConcurrentGatePushLosesCleanly(t *testing.T) { mustWrite(t, filepath.Join(f.local, "rescope.txt"), "rescope\n") mustRun(t, f.local, "add", "rescope.txt") mustRun(t, f.local, "commit", "-m", "diverging rescope") + raced := false f.service.beforeGateReset = func() { + if raced { + return + } + raced = true writer := filepath.Join(t.TempDir(), "racer") mustRun(t, filepath.Dir(writer), "-c", "core.autocrlf=false", "clone", f.gate, writer) configureIdentity(t, writer) @@ -1358,6 +1368,25 @@ func TestRecoverConcurrentGatePushLosesCleanly(t *testing.T) { if f.custodyReturned() { t.Fatal("racing recover stamped custody") } + // The gate sat exactly at the recorded head, so no displaced-gate-head + // anchor was written and a retry genuinely succeeds. The refusal must say + // so rather than sending the operator to reconcile a ref that is absent. + if _, exists, err := gitpkg.ExactRefTarget(f.ctx, f.gate, custody.RecoveryGateRef(f.run.ID)); err != nil || exists { + t.Fatalf("fixture invariant broken: gate anchor exists=%v err=%v", exists, err) + } + if strings.Contains(state.Error, custody.RecoveryGateRef(f.run.ID)) { + t.Fatalf("refusal pointed at an anchor that was never written: %q", state.Error) + } + if !strings.Contains(state.Error, "re-run the recovery") { + t.Fatalf("refusal withheld the retry guidance that is correct here: %q", state.Error) + } + if state.NextAction == nil { + t.Fatalf("racing keep-local recover named no exit = %#v", state) + } + retry := f.service.Recover(f.ctx, true) + if !retry.Recovered { + t.Fatalf("the retry the refusal prescribes did not succeed = %#v", retry) + } } func TestRecoverRetryDoesNotOverwriteIndependentGateAnchor(t *testing.T) { @@ -1396,6 +1425,17 @@ func TestRecoverRetryDoesNotOverwriteIndependentGateAnchor(t *testing.T) { if got := mustRun(t, f.gate, "rev-parse", anchor); got != firstGate { t.Fatalf("independent gate anchor = %s, want original %s", got, firstGate) } + // Nothing retires this anchor, so hand-reconciling it in the bare gate is + // the operator's only exit; the refusal has to say which ref and where. + if !strings.Contains(second.Error, anchor) { + t.Fatalf("refusal did not name the conflicting anchor: %q", second.Error) + } + if !strings.Contains(second.Error, f.gate) { + t.Fatalf("refusal did not name the gate holding that anchor: %q", second.Error) + } + if !strings.Contains(second.Error, firstGate) { + t.Fatalf("refusal did not name what the anchor holds: %q", second.Error) + } if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != secondGate { t.Fatalf("retry moved gate branch = %s, want %s", got, secondGate) } @@ -1884,3 +1924,1297 @@ func TestRecoverSquashedPreservedHeadStillEscalatesForDroppedLocalWork(t *testin t.Fatal("dropped-work escalation stamped custody") } } + +// wedgedCustodyFixture reproduces the exact self-inconsistent custody record +// reported in issue #824: the bound run is already TERMINAL with no push +// binding (pushed_head empty), its recorded pipeline head is not present in +// the invoking worktree or the local gate, and the gate branch sits at a +// different, later run's head. Nothing in that record can be verified, so +// every guarded recovery refused and abort of the terminal run was a no-op - +// the record had no supported settlement at all. +func wedgedCustodyFixture(t *testing.T, status types.RunStatus) (*recoverFixture, string, string) { + t.Helper() + f := newRecoverFixture(t, status) + + // A later run pushed its own head onto the gate branch and was then + // cancelled, so the gate branch no longer names this run's head. + writer := filepath.Join(t.TempDir(), "later-run") + mustRun(t, filepath.Dir(writer), "-c", "core.autocrlf=false", "clone", f.gate, writer) + configureIdentity(t, writer) + mustRun(t, writer, "checkout", "feature/recover") + mustWrite(t, filepath.Join(writer, "later.txt"), "later cancelled run\n") + mustRun(t, writer, "add", "later.txt") + mustRun(t, writer, "commit", "-m", "no-mistakes(review): later run fix") + mustRun(t, writer, "push", "origin", "HEAD:refs/heads/feature/recover") + staleGate := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover") + + // The recorded pipeline head is gone from every reachable object store. + missing := strings.Repeat("a", 40) + if err := f.db.UpdateRunStatusWithVerifiedHead(f.run.ID, status, missing); err != nil { + t.Fatal(err) + } + run, err := f.db.GetRun(f.run.ID) + if err != nil || run == nil { + t.Fatalf("reload run: %#v, %v", run, err) + } + f.run = run + return f, staleGate, missing +} + +// TestRecoverKeepLocalSettlesWedgedCustodyRecord is the issue #824 regression: +// a terminal run with an empty pushed head, a recorded pipeline head that no +// object store still has, and a gate branch parked at another run's head must +// be settleable by the explicit human choice `--recover --keep-local`. Custody +// returns at the kept local head and the gate branch compare-and-swaps onto +// it, so the record stops advertising a recovery that always refuses. +func TestRecoverKeepLocalSettlesWedgedCustodyRecord(t *testing.T) { + t.Parallel() + + f, staleGate, missing := wedgedCustodyFixture(t, types.RunFailed) + + state := f.service.Recover(f.ctx, true) + if !state.Recovered { + t.Fatalf("keep-local settlement did not return custody = %#v", state) + } + if state.Changed { + t.Fatalf("keep-local settlement moved the worktree = %#v", state) + } + if state.NextAction != nil && state.NextAction.Code == "recover_custody" { + t.Fatalf("settled record still advertises recover_custody = %#v", state.NextAction) + } + if !f.custodyReturned() { + t.Fatal("keep-local settlement did not stamp custody returned") + } + if got := mustRun(t, f.local, "rev-parse", "HEAD"); got != f.submitted { + t.Fatalf("keep-local settlement moved local HEAD = %s, want %s", got, f.submitted) + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != f.submitted { + t.Fatalf("gate branch = %s, want the kept local head %s", got, f.submitted) + } + if got := mustRun(t, f.gate, "rev-parse", custody.RecoveryGateRef(f.run.ID)); got != staleGate { + t.Fatalf("displaced gate head anchor = %s, want %s", got, staleGate) + } + if objectExists(f.ctx, f.gate, missing) { + t.Fatalf("fixture invariant broken: recorded head %s exists in the gate", missing) + } +} + +// TestRecoverMissingPreservedHeadStillRefusesWithoutKeepLocal keeps the +// fail-closed default: only the explicit human choice settles a record whose +// preserved head cannot be verified. +func TestRecoverMissingPreservedHeadStillRefusesWithoutKeepLocal(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + + state := f.service.Recover(f.ctx, false) + if state.Recovered || state.Changed { + t.Fatalf("default recovery settled an unverifiable record = %#v", state) + } + if state.Safety != "blocked_recover_preserved_head_missing" { + t.Fatalf("default recovery safety = %q, want blocked_recover_preserved_head_missing", state.Safety) + } + if f.custodyReturned() { + t.Fatal("default recovery stamped custody for an unverifiable record") + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != staleGate { + t.Fatalf("default recovery moved the gate branch = %s, want %s", got, staleGate) + } +} + +// TestRecoverKeepLocalSettlesConflictingGateAnchorAndPinsPreservedHead covers +// the second self-inconsistent shape: the recorded pipeline head still exists, +// but the run's own recovery evidence in the gate names a different commit, so +// nothing about the record can be verified. Keep-local settles it, and the +// still-present preserved head is pinned first so the settlement can never +// strand it. +func TestRecoverKeepLocalSettlesConflictingGateAnchorAndPinsPreservedHead(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + mustRun(t, f.gate, "update-ref", f.anchorRef(), f.submitted) + + state := f.service.Recover(f.ctx, true) + if !state.Recovered || state.Changed { + t.Fatalf("keep-local settlement of a conflicting anchor = %#v", state) + } + if !f.custodyReturned() { + t.Fatal("keep-local settlement did not stamp custody returned") + } + if got := mustRun(t, f.local, "rev-parse", "HEAD"); got != f.submitted { + t.Fatalf("keep-local settlement moved local HEAD = %s, want %s", got, f.submitted) + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != f.submitted { + t.Fatalf("gate branch = %s, want the kept local head %s", got, f.submitted) + } + if got := mustRun(t, f.gate, "rev-parse", f.anchorRef()); got != f.submitted { + t.Fatalf("settlement overwrote the conflicting evidence = %s, want %s", got, f.submitted) + } + if got := mustRun(t, f.gate, "rev-parse", custody.RecoveryStrandedRef(f.run.ID)); got != f.preserved { + t.Fatalf("stranded preserved head anchor = %s, want %s", got, f.preserved) + } +} + +// TestRecoverKeepLocalRefusesWhenStillPresentPreservedHeadCannotBeAnchored is +// the fail-closed half of the settlement: unique unpublished pipeline commits +// that still exist but cannot be anchored keep refusing, with a safety code +// that names the reason. Here an earlier partial settlement left the stranded +// anchor at a different commit, so pinning the preserved head fails. +func TestRecoverKeepLocalRefusesWhenStillPresentPreservedHeadCannotBeAnchored(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + mustRun(t, f.gate, "update-ref", f.anchorRef(), f.submitted) + mustRun(t, f.gate, "update-ref", custody.RecoveryStrandedRef(f.run.ID), f.submitted) + + // The settlement can only refuse here, so inspection must not prescribe it. + inspected := f.service.InspectCached(f.ctx) + if inspected.NextAction == nil || inspected.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("occupied-stranded-anchor next action = %#v", inspected.NextAction) + } + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Changed { + t.Fatalf("keep-local settled a record whose preserved head could not be anchored = %#v", state) + } + if state.Safety != "blocked_recover_preserve_failed" { + t.Fatalf("safety = %q, want blocked_recover_preserve_failed", state.Safety) + } + // A refusal must still name an exit that completes rather than one that + // loops back into the same refusal. + if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("refused settlement next action = %#v", state.NextAction) + } + if !strings.Contains(state.Error, custody.RecoveryStrandedRef(f.run.ID)) { + t.Fatalf("refusal did not name the conflicting anchor: %q", state.Error) + } + if f.custodyReturned() { + t.Fatal("unanchorable preserved head stamped custody") + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != f.preserved { + t.Fatalf("refused settlement moved the gate branch = %s, want %s", got, f.preserved) + } + if got := mustRun(t, f.gate, "rev-parse", custody.RecoveryStrandedRef(f.run.ID)); got != f.submitted { + t.Fatalf("refused settlement overwrote existing evidence = %s, want %s", got, f.submitted) + } +} + +// TestRecoverKeepLocalSettlementLosesConcurrentGatePushCleanly pins the +// compare-and-swap: the settlement never force-moves the gate branch. A push +// landing after the stale head was observed wins, and the settlement refuses +// without stamping custody. +func TestRecoverKeepLocalSettlementLosesConcurrentGatePushCleanly(t *testing.T) { + t.Parallel() + + f, _, _ := wedgedCustodyFixture(t, types.RunFailed) + var raced string + f.service.beforeGateReset = func() { + if raced != "" { + return + } + writer := filepath.Join(t.TempDir(), "racer") + mustRun(t, filepath.Dir(writer), "-c", "core.autocrlf=false", "clone", f.gate, writer) + configureIdentity(t, writer) + mustRun(t, writer, "checkout", "feature/recover") + mustWrite(t, filepath.Join(writer, "raced.txt"), "raced\n") + mustRun(t, writer, "add", "raced.txt") + mustRun(t, writer, "commit", "-m", "concurrent gate push") + mustRun(t, writer, "push", "origin", "HEAD:refs/heads/feature/recover") + raced = mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover") + } + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Safety != "blocked_recover_gate_race" { + t.Fatalf("racing settlement = %#v", state) + } + if f.custodyReturned() { + t.Fatal("lost compare-and-swap stamped custody") + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != raced { + t.Fatalf("gate branch = %s, want the concurrent push %s", got, raced) + } + // The race refusal itself has to name an exit, not just the retry after it. + if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("lost compare-and-swap next action = %#v", state.NextAction) + } + // The pre-CAS pin makes the retry this message used to prescribe refuse + // forever, so the refusal must point at the anchor instead of a re-run. + if strings.Contains(state.Error, "re-run the recovery") { + t.Fatalf("refusal still prescribes a retry that cannot succeed: %q", state.Error) + } + if !strings.Contains(state.Error, custody.RecoveryGateRef(f.run.ID)) { + t.Fatalf("refusal did not name the anchor to reconcile: %q", state.Error) + } + + // The lost swap leaves refs/no-mistakes/recover-gate/ pinned at the + // head observed before the race, and nothing retires it. The retry the + // refusal invites therefore hits the anchor conflict permanently, so it + // must name a completable exit and status must stop prescribing it. + retry := f.service.Recover(f.ctx, true) + if retry.Recovered { + t.Fatalf("retry after a lost compare-and-swap settled the record = %#v", retry) + } + if retry.NextAction == nil { + t.Fatalf("permanently refusing retry named no exit at all = %#v", retry) + } + if retry.NextAction.Code == "return_custody_keep_local" { + t.Fatalf("retry prescribed the settlement that just refused = %#v", retry.NextAction) + } + if f.custodyReturned() { + t.Fatal("refused retry stamped custody") + } + inspected := f.service.InspectCached(f.ctx) + if inspected.NextAction == nil || inspected.NextAction.Code == "return_custody_keep_local" { + t.Fatalf("wedged-anchor next action = %#v", inspected.NextAction) + } +} + +// TestSettlementLostSwapKeepsItsLocalScopedClaimBesideTheAnchorNote is the +// review regression for the one refusal that must NOT take the anchor note as +// a replacement for its own claim. The lost compare-and-swap has just written +// refs/no-mistakes/recover-gate/ into the gate, which is exactly why its +// message is scoped to LOCAL files and refs; substituting the settlement's +// broader "no branch, worktree, or file changes were made" there would deny a +// gate-side ref file this very attempt created. +func TestSettlementLostSwapKeepsItsLocalScopedClaimBesideTheAnchorNote(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + // A later run parked the gate branch past the recorded head, so the swap + // has a displaced gate head to anchor, while conflicting gate evidence + // routes the recovery through the settlement and its pin. + later := filepath.Join(t.TempDir(), "later-run") + mustRun(t, filepath.Dir(later), "-c", "core.autocrlf=false", "clone", f.gate, later) + configureIdentity(t, later) + mustRun(t, later, "checkout", "feature/recover") + mustWrite(t, filepath.Join(later, "later.txt"), "later cancelled run\n") + mustRun(t, later, "add", "later.txt") + mustRun(t, later, "commit", "-m", "no-mistakes(review): later run fix") + mustRun(t, later, "push", "origin", "HEAD:refs/heads/feature/recover") + staleGate := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover") + mustRun(t, f.gate, "update-ref", f.anchorRef(), f.submitted) + + var raced string + f.service.beforeGateReset = func() { + if raced != "" { + return + } + mustWrite(t, filepath.Join(later, "raced.txt"), "raced\n") + mustRun(t, later, "add", "raced.txt") + mustRun(t, later, "commit", "-m", "concurrent gate push") + mustRun(t, later, "push", "origin", "HEAD:refs/heads/feature/recover") + raced = mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover") + } + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Safety != "blocked_recover_gate_race" { + t.Fatalf("racing settlement = %#v", state) + } + if f.custodyReturned() { + t.Fatal("lost compare-and-swap stamped custody") + } + // The fixture has to have produced the exact state under test: a pinned + // recorded head, a written gate anchor, and a gate branch that moved on. + if got := mustRun(t, f.gate, "rev-parse", custody.RecoveryStrandedRef(f.run.ID)); got != f.preserved { + t.Fatalf("gate stranded anchor = %s, want the recorded head %s", got, f.preserved) + } + if got := mustRun(t, f.gate, "rev-parse", custody.RecoveryGateRef(f.run.ID)); got != staleGate { + t.Fatalf("displaced gate head anchor = %s, want %s", got, staleGate) + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != raced { + t.Fatalf("gate branch = %s, want the concurrent push %s", got, raced) + } + if !strings.Contains(state.Error, "no local files or refs were changed") { + t.Fatalf("refusal dropped the local qualifier it earned by writing a gate ref: %q", state.Error) + } + if !strings.Contains(state.Error, custody.RecoveryStrandedRef(f.run.ID)) || !strings.Contains(state.Error, "the local gate") { + t.Fatalf("refusal did not name where the recorded head is now anchored: %q", state.Error) + } + if !strings.Contains(state.Error, custody.RecoveryGateRef(f.run.ID)) { + t.Fatalf("refusal did not name the anchor to reconcile: %q", state.Error) + } + if state.NextAction == nil { + t.Fatalf("lost compare-and-swap named no exit at all = %#v", state) + } +} + +// TestInspectNamesKeepLocalSettlementForWedgedCustodyRecord is the R5 half of +// issue #824: a refused recovery must still name a supported exit. Inspection +// must not advertise recover_custody for a record it cannot verify, but the +// next action it does advertise has to be one that can actually complete. +func TestInspectNamesKeepLocalSettlementForWedgedCustodyRecord(t *testing.T) { + t.Parallel() + + f, _, _ := wedgedCustodyFixture(t, types.RunFailed) + + state := f.service.InspectCached(f.ctx) + if state.Safety != "blocked_recover_preserved_head_missing" { + t.Fatalf("wedged-record safety = %q, want blocked_recover_preserved_head_missing: %#v", state.Safety, state) + } + if state.NextAction == nil || state.NextAction.Code != "return_custody_keep_local" { + t.Fatalf("wedged-record next action = %#v", state.NextAction) + } + if state.NextAction.Code == "recover_custody" { + t.Fatal("unverifiable record advertised an impossible recovery") + } + if !strings.Contains(state.NextAction.Command, "--keep-local") { + t.Fatalf("next action command = %q, want the keep-local settlement", state.NextAction.Command) + } +} + +// TestInspectDoesNotAdvertiseSettlementForUnverifiedTerminalHead is the review +// regression for the sibling shape that would have recreated the #824 wedge: +// a terminal run whose head was never verified (a daemon crash after the +// managed worktree was already gone) refuses at Recover's unverified-head +// guard, strictly before any keep-local interception. Inspection must not +// advertise a settlement that guard will always refuse. +func TestInspectDoesNotAdvertiseSettlementForUnverifiedTerminalHead(t *testing.T) { + t.Parallel() + + // A non-terminal fixture status records the run without a verified + // terminal head; the head is then moved off every reachable object and + // terminalized the way a crash-recovered run is. + f := newRecoverFixture(t, types.RunRunning) + if err := f.db.UpdateRunHeadSHA(f.run.ID, strings.Repeat("b", 40)); err != nil { + t.Fatal(err) + } + if err := f.db.UpdateRunStatus(f.run.ID, types.RunFailed); err != nil { + t.Fatal(err) + } + run, err := f.db.GetRun(f.run.ID) + if err != nil || run == nil { + t.Fatalf("reload run: %#v, %v", run, err) + } + f.run = run + if run.TerminalHeadVerifiedAt != nil { + t.Fatal("fixture invariant broken: the terminal head is verified") + } + + state := f.service.InspectCached(f.ctx) + if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("unverified-head next action = %#v", state.NextAction) + } + + // The advertisement must match what recovery actually does. + recovered := f.service.Recover(f.ctx, true) + if recovered.Recovered || recovered.Safety != "blocked_recover_unverified_head" { + t.Fatalf("keep-local on an unverified terminal head = %#v", recovered) + } + if f.custodyReturned() { + t.Fatal("an unverified terminal head stamped custody") + } +} + +// TestInspectDoesNotAdvertiseSettlementWhenGateRecoveryRefIsUninspectable is +// the review regression for the fail-closed advertisement: a gate recovery ref +// that cannot be inspected at all (corrupt packed-refs while the gate +// directory itself is intact) is not evidence of inconsistency, and Recover +// refuses that probe error strictly before any keep-local interception, so +// inspection must keep the honest manual-reconciliation pointer instead of +// advertising a settlement that always refuses. +func TestInspectDoesNotAdvertiseSettlementWhenGateRecoveryRefIsUninspectable(t *testing.T) { + t.Parallel() + + f, _, _ := wedgedCustodyFixture(t, types.RunFailed) + mustWrite(t, filepath.Join(f.gate, "packed-refs"), "garbage line\n") + if _, _, refErr := gitpkg.ExactRefTarget(f.ctx, f.gate, f.anchorRef()); refErr == nil { + t.Fatal("fixture invariant broken: the gate recovery ref is still inspectable") + } + + state := f.service.InspectCached(f.ctx) + if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("uninspectable-gate next action = %#v", state.NextAction) + } + + // The advertisement must match what recovery actually does. + recovered := f.service.Recover(f.ctx, true) + if recovered.Recovered || recovered.Safety != "blocked_recover_anchor_mismatch" { + t.Fatalf("keep-local with an uninspectable gate recovery ref = %#v", recovered) + } + if f.custodyReturned() { + t.Fatal("an uninspectable gate recovery ref stamped custody") + } +} + +// TestInspectDoesNotAdvertiseSettlementForIncompleteAdoption is the review +// regression for the sibling shape: a conflicting recovery anchor on a record +// whose branch already reached the preserved head with a stale mid-adoption +// worktree is refused by Recover's incomplete-adoption guard, strictly before +// any keep-local interception, so inspection must not advertise the +// settlement there either. +func TestInspectDoesNotAdvertiseSettlementForIncompleteAdoption(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + mustRun(t, f.local, "fetch", "--no-tags", f.gate, "+refs/heads/feature/recover:"+f.anchorRef()) + mustRun(t, f.local, "update-ref", f.localAnchorRef(), f.submitted, "") + mustRun(t, f.local, "update-ref", "refs/heads/feature/recover", f.preserved, f.submitted) + mustRun(t, f.gate, "update-ref", f.anchorRef(), f.submitted) + + state := f.service.InspectCached(f.ctx) + if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("incomplete-adoption next action = %#v", state.NextAction) + } + + // The advertisement must match what recovery actually does. + recovered := f.service.Recover(f.ctx, true) + if recovered.Recovered || recovered.Safety != "blocked_recover_incomplete_adoption" { + t.Fatalf("keep-local on an incomplete adoption = %#v", recovered) + } + if f.custodyReturned() { + t.Fatal("an incomplete adoption stamped custody") + } +} + +// TestRecoverKeepLocalSettlesConflictingWorktreeAnchorOnReachableHead covers +// the review regression on the locally reachable path: when the preserved head +// is already reachable from the local branch but the invoking worktree's own +// recovery ref names something else, the record is just as self-inconsistent as +// the gate-side shapes - and keeping a head that already contains the preserved +// commits can lose nothing at all. +func TestRecoverKeepLocalSettlesConflictingWorktreeAnchorOnReachableHead(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + mustRun(t, f.local, "fetch", f.gate, f.preserved) + mustRun(t, f.local, "reset", "--hard", f.preserved) + mustRun(t, f.local, "update-ref", f.anchorRef(), f.submitted) + + // The default recovery stays fail-closed on the same record. + refused := f.service.Recover(f.ctx, false) + if refused.Recovered || refused.Safety != "blocked_recover_preserve_failed" { + t.Fatalf("default recovery with a conflicting worktree anchor = %#v", refused) + } + if f.custodyReturned() { + t.Fatal("default recovery stamped custody for a conflicting worktree anchor") + } + + state := f.service.Recover(f.ctx, true) + if !state.Recovered || state.Changed { + t.Fatalf("keep-local settlement of a conflicting worktree anchor = %#v", state) + } + if !f.custodyReturned() { + t.Fatal("keep-local settlement did not stamp custody returned") + } + if got := mustRun(t, f.local, "rev-parse", "HEAD"); got != f.preserved { + t.Fatalf("settlement moved local HEAD = %s, want %s", got, f.preserved) + } + if got := mustRun(t, f.local, "rev-parse", f.anchorRef()); got != f.submitted { + t.Fatalf("settlement overwrote the conflicting worktree evidence = %s, want %s", got, f.submitted) + } + if got := mustRun(t, f.local, "rev-parse", custody.RecoveryStrandedRef(f.run.ID)); got != f.preserved { + t.Fatalf("stranded anchor in the worktree = %s, want %s", got, f.preserved) + } +} + +// TestRecoverKeepLocalSettlementTreatsAnUnreadableGateBranchAsUnknown is the +// review regression for the fail-open conflation: only a gate branch PROVEN +// absent settles without the compare-and-swap. A gate branch that exists but +// cannot be resolved to a commit is not evidence of absence, so the settlement +// refuses rather than stamping custody against a head it never observed. +func TestRecoverKeepLocalSettlementTreatsAnUnreadableGateBranchAsUnknown(t *testing.T) { + t.Parallel() + + t.Run("absent gate branch settles without a gate move", func(t *testing.T) { + f, _, _ := wedgedCustodyFixture(t, types.RunFailed) + mustRun(t, f.gate, "update-ref", "-d", "refs/heads/feature/recover") + + // A gate branch proven absent keeps the advertisement, because the + // settlement completes there with no compare-and-swap at all. + inspected := f.service.InspectCached(f.ctx) + if inspected.NextAction == nil || inspected.NextAction.Code != "return_custody_keep_local" { + t.Fatalf("absent-gate-branch next action = %#v", inspected.NextAction) + } + + state := f.service.Recover(f.ctx, true) + if !state.Recovered || state.Changed { + t.Fatalf("settlement with an absent gate branch = %#v", state) + } + if !f.custodyReturned() { + t.Fatal("absent gate branch did not stamp custody returned") + } + }) + + t.Run("unreadable gate branch refuses", func(t *testing.T) { + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + // A branch ref that exists but does not name a commit: git refuses to + // write one through update-ref, so the loose ref is written directly. + mustRun(t, f.gate, "pack-refs", "--all") + blob := mustRun(t, f.gate, "hash-object", "-w", filepath.Join(f.local, "file.txt")) + mustWrite(t, filepath.Join(f.gate, "refs", "heads", "feature", "recover"), blob+"\n") + + // The settlement refuses this shape, so inspection must not prescribe + // it: an advertisement that always refuses is the exact wedge #824 + // removes. + inspected := f.service.InspectCached(f.ctx) + if inspected.NextAction == nil || inspected.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("unreadable-gate-branch next action = %#v", inspected.NextAction) + } + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Safety != "blocked_recover_gate_unavailable" { + t.Fatalf("settlement with an unreadable gate branch = %#v", state) + } + if state.NextAction == nil { + t.Fatalf("refused settlement named no exit at all = %#v", state) + } + if f.custodyReturned() { + t.Fatal("an unobserved gate head stamped custody") + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got == staleGate { + t.Fatalf("fixture invariant broken: the gate branch still resolves to %s", staleGate) + } + }) +} + +// TestRecoverKeepLocalRefusalNamesTheAnchorItAlreadyWrote is the review +// regression for the honesty claim: the stranded anchor is the one ref the +// settlement can write before failing, so a refusal must report where it now +// exists instead of claiming nothing changed. +func TestRecoverKeepLocalRefusalNamesTheAnchorItAlreadyWrote(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + mustRun(t, f.local, "fetch", f.gate, f.preserved) + mustRun(t, f.gate, "update-ref", f.anchorRef(), f.submitted) + // The worktree pin succeeds; the gate pin then fails on prior evidence. + mustRun(t, f.gate, "update-ref", custody.RecoveryStrandedRef(f.run.ID), f.submitted) + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Safety != "blocked_recover_preserve_failed" { + t.Fatalf("refused settlement = %#v", state) + } + if got := mustRun(t, f.local, "rev-parse", custody.RecoveryStrandedRef(f.run.ID)); got != f.preserved { + t.Fatalf("worktree stranded anchor = %s, want %s", got, f.preserved) + } + if strings.Contains(state.Error, "no files or refs were changed") { + t.Fatalf("refusal claimed nothing changed after writing the worktree anchor: %q", state.Error) + } + if !strings.Contains(state.Error, "the invoking worktree") { + t.Fatalf("refusal did not name where the anchor now exists: %q", state.Error) + } + if f.custodyReturned() { + t.Fatal("refused settlement stamped custody") + } +} + +// TestKeepLocalRefusalAfterDelegationNamesTheAnchorTheAttemptWrote carries the +// same honesty claim across the DELEGATION seam, where it used to be dropped. +// recoverKeepLocal writes nothing of its own before the compare-and-swap, but +// both of its callers can already have anchored a surviving recorded head - +// the settlement pins every copy at the stranded ref, and the ordinary +// keep-local path anchors the preserved head at the run recovery ref - so a +// blanket "no files or refs were changed" there reports on an attempt that +// did write a ref, and hides the very anchor the operator needs to find their +// preserved commits. +func TestKeepLocalRefusalAfterDelegationNamesTheAnchorTheAttemptWrote(t *testing.T) { + t.Parallel() + + t.Run("settlement pin in the local gate", func(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + // Conflicting gate evidence routes keep-local through the settlement, + // whose pin loop finds the recorded head in the gate alone. + mustRun(t, f.gate, "update-ref", f.anchorRef(), f.submitted) + f.service.absPathFn = func(string) (string, error) { + return "", errors.New("working directory has been removed") + } + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Safety != "blocked_recover_assumptions_changed" { + t.Fatalf("delegated refusal = %#v", state) + } + if got := mustRun(t, f.gate, "rev-parse", custody.RecoveryStrandedRef(f.run.ID)); got != f.preserved { + t.Fatalf("gate stranded anchor = %s, want %s", got, f.preserved) + } + if strings.Contains(state.Error, "no files or refs were changed") { + t.Fatalf("refusal claimed nothing changed after the settlement pinned the recorded head: %q", state.Error) + } + if !strings.Contains(state.Error, custody.RecoveryStrandedRef(f.run.ID)) || !strings.Contains(state.Error, "the local gate") { + t.Fatalf("refusal did not name where the recorded head is now anchored: %q", state.Error) + } + if f.custodyReturned() { + t.Fatal("delegated refusal stamped custody") + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != f.preserved { + t.Fatalf("delegated refusal moved the gate branch = %s, want %s", got, f.preserved) + } + f.service.absPathFn = nil + retry := f.service.Recover(f.ctx, true) + if !retry.Recovered { + t.Fatalf("the settlement stayed wedged after an honest refusal = %#v", retry) + } + }) + + t.Run("recovery anchor in the invoking worktree", func(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + mustWrite(t, filepath.Join(f.local, "rescope.txt"), "rescope\n") + mustRun(t, f.local, "add", "rescope.txt") + mustRun(t, f.local, "commit", "-m", "diverging rescope") + f.service.absPathFn = func(string) (string, error) { + return "", errors.New("working directory has been removed") + } + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Safety != "blocked_recover_assumptions_changed" { + t.Fatalf("delegated refusal = %#v", state) + } + if got := mustRun(t, f.local, "rev-parse", f.anchorRef()); got != f.preserved { + t.Fatalf("worktree recovery anchor = %s, want %s", got, f.preserved) + } + if strings.Contains(state.Error, "no files or refs were changed") { + t.Fatalf("refusal claimed nothing changed after anchoring the preserved head: %q", state.Error) + } + if !strings.Contains(state.Error, f.anchorRef()) || !strings.Contains(state.Error, "the invoking worktree") { + t.Fatalf("refusal did not name the anchor this attempt wrote: %q", state.Error) + } + if f.custodyReturned() { + t.Fatal("delegated refusal stamped custody") + } + f.service.absPathFn = nil + retry := f.service.Recover(f.ctx, true) + if !retry.Recovered { + t.Fatalf("keep-local stayed wedged after an honest refusal = %#v", retry) + } + }) +} + +// TestInspectDoesNotAdvertiseSettlementForSymbolicGateAnchor is the review +// regression for the residual advertise-then-refuse corner. A DANGLING +// symbolic gate recovery ref is invisible to `for-each-ref` while +// `symbolic-ref -q` still succeeds, so recoveryAnchorCompatible reports +// "incompatible" with no error while Recover's own ExactRefTarget probe sees +// no ref at all. Recovery then fails inside PreserveRecoveryHead's symbolic +// check, at a refusal site that deliberately has no keep-local interception. +// Symbolic gate-anchor evidence must therefore disqualify the record outright. +func TestInspectDoesNotAdvertiseSettlementForSymbolicGateAnchor(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + // Dangling: the target never exists, so for-each-ref omits the ref. + mustRun(t, f.gate, "symbolic-ref", f.anchorRef(), "refs/no-mistakes/evidence/"+f.run.ID) + if _, exists, err := gitpkg.ExactRefTarget(f.ctx, f.gate, f.anchorRef()); err != nil || exists { + t.Fatalf("fixture invariant broken: exact ref target exists=%v err=%v", exists, err) + } + + state := f.service.InspectCached(f.ctx) + if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("symbolic-anchor next action = %#v", state.NextAction) + } + + // The advertisement must match what recovery actually does. + recovered := f.service.Recover(f.ctx, true) + if recovered.Recovered || recovered.Safety != "blocked_recover_preserve_failed" { + t.Fatalf("keep-local with a symbolic gate anchor = %#v", recovered) + } + // This is the acceptance criterion's own shape: the preserved commits + // still exist in the gate and cannot be anchored, so the refusal has to + // name an exit rather than strand the operator. + if recovered.NextAction == nil || recovered.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("unanchorable preserved head named no completable exit = %#v", recovered) + } + if f.custodyReturned() { + t.Fatal("a symbolic gate anchor stamped custody") + } + if got := mustRun(t, f.gate, "symbolic-ref", f.anchorRef()); got != "refs/no-mistakes/evidence/"+f.run.ID { + t.Fatalf("symbolic evidence was rewritten = %s", got) + } +} + +// TestInspectDoesNotAdvertiseSettlementForResolvingSymbolicGateAnchor covers +// the softer sibling named in the same finding: a symbolic gate anchor that +// DOES resolve is listed by for-each-ref, so recovery gets all the way through +// and keep-local actually completes. The predicate disqualifies symbolic +// evidence CONSERVATIVELY, without separating the dangling variant (genuinely +// unreachable) from this resolving one, so this is a deliberate +// UNDER-advertisement: status names a manual exit that can complete instead of +// promising a settlement whose reachability it did not prove. Under-advertising +// is fail-safe; over-advertising is the #824 wedge. The Recover half pins what +// the command really does here, so a later change that starts advertising this +// shape is a deliberate choice rather than an accident. +func TestInspectDoesNotAdvertiseSettlementForResolvingSymbolicGateAnchor(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + mustRun(t, f.gate, "symbolic-ref", f.anchorRef(), "refs/heads/feature/recover") + + state := f.service.InspectCached(f.ctx) + if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("resolving-symbolic-anchor next action = %#v", state.NextAction) + } + if f.custodyReturned() { + t.Fatal("inspection stamped custody") + } + + recovered := f.service.Recover(f.ctx, true) + if !recovered.Recovered { + t.Fatalf("keep-local with a resolving symbolic gate anchor = %#v", recovered) + } + if recovered.Changed { + t.Fatalf("keep-local moved the worktree = %#v", recovered) + } + if !f.custodyReturned() { + t.Fatal("completed settlement did not stamp custody returned") + } + if got := mustRun(t, f.local, "rev-parse", "HEAD"); got != f.submitted { + t.Fatalf("local HEAD = %s, want the kept head %s", got, f.submitted) + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != f.submitted { + t.Fatalf("gate branch = %s, want the kept local head %s", got, f.submitted) + } +} + +// TestRecoverGateAndRunStateRefusalsNameAnExit pins the next action on exactly +// four refusal shapes: an absent gate, the two unverified-head guards, and a +// still-active run. It is deliberately NOT a general guarantee - blockedPlan +// nils NextAction, and several other refusals (a missing preserved head under +// the default --recover, the anchorReachablePreserved sites, finishRecover's +// stamp failure) still return none. Naming this test for the broad invariant +// would restate the claim that kept outrunning the code, so it names its +// shapes; extending the table is what proving a wider claim would require. +func TestRecoverGateAndRunStateRefusalsNameAnExit(t *testing.T) { + t.Parallel() + + cases := []struct { + name string + wantSafety string + setup func(t *testing.T) *recoverFixture + }{ + { + name: "no gate configured", + wantSafety: "blocked_recover_gate_unavailable", + setup: func(t *testing.T) *recoverFixture { + f := newRecoverFixture(t, types.RunCancelled) + f.service.GateDir = "" + return f + }, + }, + { + name: "unverified head with no gate", + wantSafety: "blocked_recover_unverified_head", + setup: func(t *testing.T) *recoverFixture { + f := newRecoverFixture(t, types.RunCancelled) + if err := f.db.UpdateRunStatus(f.run.ID, types.RunCancelled); err != nil { + t.Fatal(err) + } + run, err := f.db.GetRun(f.run.ID) + if err != nil || run == nil { + t.Fatalf("reload run: %#v, %v", run, err) + } + f.run = run + f.service.GateDir = "" + return f + }, + }, + { + name: "unverified head with an unreadable gate branch", + wantSafety: "blocked_recover_unverified_head", + setup: func(t *testing.T) *recoverFixture { + f := newRecoverFixture(t, types.RunCancelled) + if err := f.db.UpdateRunStatus(f.run.ID, types.RunCancelled); err != nil { + t.Fatal(err) + } + run, err := f.db.GetRun(f.run.ID) + if err != nil || run == nil { + t.Fatalf("reload run: %#v, %v", run, err) + } + f.run = run + mustRun(t, f.gate, "update-ref", "-d", "refs/heads/feature/recover") + return f + }, + }, + { + name: "the owning run is still active", + wantSafety: "blocked_recover_run_active", + setup: func(t *testing.T) *recoverFixture { + return newRecoverFixture(t, types.RunRunning) + }, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + for _, keepLocal := range []bool{false, true} { + f := tc.setup(t) + state := f.service.Recover(f.ctx, keepLocal) + if state.Recovered { + t.Fatalf("keepLocal=%v recovered instead of refusing = %#v", keepLocal, state) + } + if state.Safety != tc.wantSafety { + t.Fatalf("keepLocal=%v safety = %q, want %q: %#v", keepLocal, state.Safety, tc.wantSafety, state) + } + if state.NextAction == nil || strings.TrimSpace(state.NextAction.Command) == "" { + t.Fatalf("keepLocal=%v refusal named no runnable exit = %#v", keepLocal, state) + } + if f.custodyReturned() { + t.Fatalf("keepLocal=%v refusal stamped custody", keepLocal) + } + } + }) + } +} + +// TestInspectDoesNotAdvertiseSettlementForSymbolicStrandedAnchor is the review +// regression for the probe/write disagreement. settlementAnchorsFree predicts +// whether custody.PreserveRecoveryAnchor can write an anchor, but that write +// refuses ANY symbolic ref as its first check, while the probe used +// git.ExactRefTarget alone - and a DANGLING symref is invisible to +// for-each-ref while symbolic-ref still succeeds. So the probe reported "free" +// for a ref the settlement can never write, and inspection advertised a +// settlement that refuses on every attempt. +func TestInspectDoesNotAdvertiseSettlementForSymbolicStrandedAnchor(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + // The conflicting-gate-anchor shape: the recorded head is still in the + // gate, so the settlement's pin loop will try to write the stranded anchor + // there. + mustRun(t, f.gate, "update-ref", f.anchorRef(), f.submitted) + strandedRef := custody.RecoveryStrandedRef(f.run.ID) + mustRun(t, f.gate, "symbolic-ref", strandedRef, "refs/no-mistakes/nonexistent/"+f.run.ID) + if _, exists, err := gitpkg.ExactRefTarget(f.ctx, f.gate, strandedRef); err != nil || exists { + t.Fatalf("fixture invariant broken: dangling symref is visible to for-each-ref (exists=%v err=%v)", exists, err) + } + + state := f.service.InspectCached(f.ctx) + if state.NextAction != nil && state.NextAction.Code == "return_custody_keep_local" { + t.Fatalf("advertised a settlement whose stranded anchor can never be written: %#v", state.NextAction) + } + + // The advertisement must match what recovery actually does. + recovered := f.service.Recover(f.ctx, true) + if recovered.Recovered || recovered.Safety != "blocked_recover_preserve_failed" { + t.Fatalf("keep-local with a symbolic stranded anchor = %#v", recovered) + } + if f.custodyReturned() { + t.Fatal("a symbolic stranded anchor stamped custody") + } + if got := mustRun(t, f.gate, "symbolic-ref", strandedRef); got != "refs/no-mistakes/nonexistent/"+f.run.ID { + t.Fatalf("symbolic evidence was rewritten = %s", got) + } +} + +// assertKeepLocalRefusalLeftNothing is the shared check behind the four +// pre-swap refusals in recoverKeepLocal. Each of them tells the operator that +// no files or refs were changed, and the displaced-gate-head anchor is the one +// ref they could previously have left behind - the same ref whose staleness +// then makes settlementAnchorsFree stop advertising the settlement and +// PreserveRecoveryAnchor refuse every later attempt, which is the #824 wedge +// this settlement exists to clear. A refusal that recreates it while claiming +// to have changed nothing is the defect, so every such path is pinned here. +func assertKeepLocalRefusalLeftNothing(t *testing.T, f *recoverFixture, state State, staleGate string) { + t.Helper() + if state.Recovered || state.Changed { + t.Fatalf("pre-swap refusal did not refuse = %#v", state) + } + if f.custodyReturned() { + t.Fatal("pre-swap refusal stamped custody returned") + } + if !strings.Contains(state.Error, "no files or refs were changed") { + t.Fatalf("refusal did not make the no-change claim under test: %q", state.Error) + } + anchor := custody.RecoveryGateRef(f.run.ID) + target, exists, err := gitpkg.ExactRefTarget(f.ctx, f.gate, anchor) + if err != nil { + t.Fatalf("read %s: %v", anchor, err) + } + if exists { + t.Fatalf("refusal claiming no refs changed left %s behind at %s", anchor, target) + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != staleGate { + t.Fatalf("refusal moved the gate branch = %s, want %s", got, staleGate) + } + if _, exists, err := gitpkg.ExactRefTarget(f.ctx, f.gate, "refs/no-mistakes/custody-return/"+f.run.ID); err != nil || exists { + t.Fatalf("refusal left the staging ref behind: exists=%v err=%v", exists, err) + } + // The wedge is only proven cleared if the settlement can still complete, + // so every case re-runs the recovery with its inducer removed. + f.service.beforeGateReset = nil + f.service.beforeGateStage = nil + f.service.absPathFn = nil + retry := f.service.Recover(f.ctx, true) + if !retry.Recovered { + t.Fatalf("settlement stayed wedged after a refusal that changed nothing = %#v", retry) + } +} + +// TestRecoverKeepLocalMovedLocalHeadRefusalLeavesNoGateAnchor covers the +// local-head re-read refusal. +func TestRecoverKeepLocalMovedLocalHeadRefusalLeavesNoGateAnchor(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + moved := false + f.service.beforeGateReset = func() { + if moved { + return + } + moved = true + mustWrite(t, filepath.Join(f.local, "raced.txt"), "local moved mid-recovery\n") + mustRun(t, f.local, "add", "raced.txt") + mustRun(t, f.local, "commit", "-m", "operator commit during recovery") + } + + state := f.service.Recover(f.ctx, true) + if state.Safety != "blocked_recover_assumptions_changed" { + t.Fatalf("moved-local-head refusal = %#v", state) + } + assertKeepLocalRefusalLeftNothing(t, f, state, staleGate) +} + +// TestRecoverKeepLocalUnresolvableWorktreeRefusalLeavesNoGateAnchor covers the +// worktree-path resolution refusal. +func TestRecoverKeepLocalUnresolvableWorktreeRefusalLeavesNoGateAnchor(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + f.service.absPathFn = func(string) (string, error) { + return "", errors.New("working directory has been removed") + } + + state := f.service.Recover(f.ctx, true) + if state.Safety != "blocked_recover_assumptions_changed" { + t.Fatalf("unresolvable-worktree refusal = %#v", state) + } + if !strings.Contains(state.Error, "could not be resolved") { + t.Fatalf("refusal did not name the resolution failure: %q", state.Error) + } + assertKeepLocalRefusalLeftNothing(t, f, state, staleGate) +} + +// TestRecoverKeepLocalFailedStagingRefusalLeavesNoGateAnchor covers the +// staging-fetch refusal. The fetch is failed by occupying the staging ref's +// name with a deeper ref, which Git refuses to turn into a directory. +func TestRecoverKeepLocalFailedStagingRefusalLeavesNoGateAnchor(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + blocker := "refs/no-mistakes/custody-return/" + f.run.ID + "/blocked" + f.service.beforeGateStage = func() { + mustRun(t, f.gate, "update-ref", blocker, staleGate) + } + + state := f.service.Recover(f.ctx, true) + if state.Safety != "blocked_recover_assumptions_changed" { + t.Fatalf("failed-staging refusal = %#v", state) + } + if !strings.Contains(state.Error, "could not be staged") { + t.Fatalf("refusal did not name the staging failure: %q", state.Error) + } + // The blocker is the operator's own ref, so the retry cannot succeed while + // it stands; clear it exactly as an operator would before re-running. + defer func() { mustRun(t, f.gate, "update-ref", "-d", blocker) }() + f.service.beforeGateStage = nil + if state.Recovered || state.Changed { + t.Fatalf("failed-staging refusal did not refuse = %#v", state) + } + if f.custodyReturned() { + t.Fatal("failed-staging refusal stamped custody returned") + } + if !strings.Contains(state.Error, "no files or refs were changed") { + t.Fatalf("refusal did not make the no-change claim under test: %q", state.Error) + } + anchor := custody.RecoveryGateRef(f.run.ID) + if target, exists, err := gitpkg.ExactRefTarget(f.ctx, f.gate, anchor); err != nil || exists { + t.Fatalf("refusal claiming no refs changed left %s behind at %s (err=%v)", anchor, target, err) + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != staleGate { + t.Fatalf("refusal moved the gate branch = %s, want %s", got, staleGate) + } + mustRun(t, f.gate, "update-ref", "-d", blocker) + retry := f.service.Recover(f.ctx, true) + if !retry.Recovered { + t.Fatalf("settlement stayed wedged after a refusal that changed nothing = %#v", retry) + } +} + +// TestRecoverKeepLocalStagedHeadMismatchRefusalLeavesNoGateAnchor covers the +// staged-head verification refusal: the branch moves after the local head was +// re-read, so the gate stages a commit the operator never chose to keep. +func TestRecoverKeepLocalStagedHeadMismatchRefusalLeavesNoGateAnchor(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + moved := false + f.service.beforeGateStage = func() { + if moved { + return + } + moved = true + mustWrite(t, filepath.Join(f.local, "raced.txt"), "local moved after the head re-read\n") + mustRun(t, f.local, "add", "raced.txt") + mustRun(t, f.local, "commit", "-m", "operator commit after the head re-read") + } + + state := f.service.Recover(f.ctx, true) + if state.Safety != "blocked_recover_assumptions_changed" { + t.Fatalf("staged-head-mismatch refusal = %#v", state) + } + assertKeepLocalRefusalLeftNothing(t, f, state, staleGate) +} + +// TestRecoverKeepLocalLostAnchorWriteRefusalLeavesNoGateAnchor covers the last +// refusal that can precede the compare-and-swap: the anchor CONFLICT check is +// read-only and runs first, so the write itself happens immediately before the +// swap and can still lose a race to whoever occupied the ref in between. +// PreserveRecoveryAnchor refuses to retarget an anchor naming another commit, +// so nothing of ours is written - and the refusal has to leave the staging ref +// behind no more than it leaves an anchor of its own. +func TestRecoverKeepLocalLostAnchorWriteRefusalLeavesNoGateAnchor(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + anchor := custody.RecoveryGateRef(f.run.ID) + // Some other commit entirely, so PreserveRecoveryAnchor sees a conflict + // rather than an idempotent re-write. + intruder := f.base + if intruder == staleGate || intruder == "" { + t.Fatalf("fixture invariant broken: need a third commit distinct from the gate head %s", staleGate) + } + occupied := false + f.service.beforeGateStage = func() { + if occupied { + return + } + occupied = true + mustRun(t, f.gate, "update-ref", anchor, intruder) + } + + state := f.service.Recover(f.ctx, true) + if state.Safety != "blocked_recover_preserve_failed" { + t.Fatalf("lost-anchor-write refusal = %#v", state) + } + if state.Recovered || state.Changed { + t.Fatalf("lost-anchor-write refusal did not refuse = %#v", state) + } + if f.custodyReturned() { + t.Fatal("lost-anchor-write refusal stamped custody returned") + } + if !strings.Contains(state.Error, "no files or branch refs were changed") { + t.Fatalf("refusal did not make the no-change claim under test: %q", state.Error) + } + // The intruder's anchor is not ours to retire, so it must survive exactly + // as found - but nothing of ours may be left beside it. + if got := mustRun(t, f.gate, "rev-parse", anchor); got != intruder { + t.Fatalf("refusal rewrote an anchor it did not create = %s, want %s", got, intruder) + } + if _, exists, err := gitpkg.ExactRefTarget(f.ctx, f.gate, "refs/no-mistakes/custody-return/"+f.run.ID); err != nil || exists { + t.Fatalf("refusal left the staging ref behind: exists=%v err=%v", exists, err) + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != staleGate { + t.Fatalf("refusal moved the gate branch = %s, want %s", got, staleGate) + } + // Clearing the intruding anchor is the exit the refusal prescribes, and it + // has to actually work. + mustRun(t, f.gate, "update-ref", "-d", anchor) + f.service.beforeGateStage = nil + retry := f.service.Recover(f.ctx, true) + if !retry.Recovered { + t.Fatalf("settlement stayed wedged after reconciling the anchor = %#v", retry) + } +} + +// TestSettlementRefusesWhenRecordedHeadAbsenceCannotBeProven pins the +// settlement's data-safety argument to actual evidence. The argument is "no +// reachable store still has this head, so settling cannot lose it" - which is +// a claim about PROVEN absence. A bare presence probe cannot make that claim: +// `git cat-file -e` exits 1 only when the store was read and the object really +// is not there, and exits 128 for every store or name it could not resolve, so +// reading any non-zero exit as "absent" skips the pin on evidence that was +// never gathered and then settles anyway. Undetermined must refuse, and the +// advertisement has to agree so the settlement is never offered where it can +// only refuse. +func TestSettlementRefusesWhenRecordedHeadAbsenceCannotBeProven(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + // A recorded head Git cannot resolve to an object name at all: every store + // answers 128, so absence is undetermined rather than proven. + unresolvable := "not-a-resolvable-object-name" + if err := f.db.UpdateRunStatusWithVerifiedHead(f.run.ID, types.RunFailed, unresolvable); err != nil { + t.Fatal(err) + } + run, err := f.db.GetRun(f.run.ID) + if err != nil || run == nil { + t.Fatalf("reload run: %#v, %v", run, err) + } + f.run = run + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Changed { + t.Fatalf("settlement completed without proving the recorded head absent = %#v", state) + } + if state.Safety != "blocked_recover_preserve_failed" { + t.Fatalf("undetermined-absence safety = %q: %#v", state.Safety, state) + } + if f.custodyReturned() { + t.Fatal("settlement stamped custody without proving the recorded head absent") + } + if !strings.Contains(state.Error, "could not be determined") { + t.Fatalf("refusal did not name the undetermined probe: %q", state.Error) + } + if state.NextAction == nil { + t.Fatalf("undetermined-absence refusal named no exit = %#v", state) + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != staleGate { + t.Fatalf("refusal moved the gate branch = %s, want %s", got, staleGate) + } + // The advertisement must agree with the write, or the record is offered a + // settlement that can only refuse - the exact #824 shape. + inspected := f.service.InspectCached(f.ctx) + if inspected.NextAction != nil && inspected.NextAction.Code == "return_custody_keep_local" { + t.Fatalf("inspection advertised a settlement that always refuses = %#v", inspected.NextAction) + } +} + +// TestSettlementRefusesWhenRecordedHeadIsPresentButNotACommit closes the other +// half of the proven-absence rule. `git cat-file -e` exits 0 for ANY object, so +// a recorded head that resolves to a tree, blob, or tag is present in the store +// while not being a commit. Reporting that as a plain "not a commit" sends it +// back through the same door as a proven absence, and the settlement would then +// move the gate on the strength of "nothing still has this head" while the +// object is right there. Present-but-wrong-type is undetermined, not absent. +func TestSettlementRefusesWhenRecordedHeadIsPresentButNotACommit(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + tree := mustRun(t, f.local, "rev-parse", "HEAD^{tree}") + if tree == "" { + t.Fatal("fixture invariant broken: no tree to record") + } + // Present in the invoking worktree's store, and not a commit. + if !objectPresentInStore(t, f.local, tree) { + t.Fatalf("fixture invariant broken: tree %s is not present in %s", tree, f.local) + } + if err := f.db.UpdateRunStatusWithVerifiedHead(f.run.ID, types.RunFailed, tree); err != nil { + t.Fatal(err) + } + run, err := f.db.GetRun(f.run.ID) + if err != nil || run == nil { + t.Fatalf("reload run: %#v, %v", run, err) + } + f.run = run + + state := f.service.Recover(f.ctx, true) + if state.Recovered || state.Changed { + t.Fatalf("settlement treated a present non-commit object as an absent head = %#v", state) + } + if f.custodyReturned() { + t.Fatal("settlement stamped custody while the recorded object was still present") + } + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != staleGate { + t.Fatalf("refusal moved the gate branch = %s, want %s", got, staleGate) + } + // And the advertisement has to agree, or the record is offered a + // settlement that can only refuse. + inspected := f.service.InspectCached(f.ctx) + if inspected.NextAction != nil && inspected.NextAction.Code == "return_custody_keep_local" { + t.Fatalf("inspection advertised a settlement that always refuses = %#v", inspected.NextAction) + } +} + +// objectPresentInStore is deliberately the raw presence question - any object +// type - so the test asserts its own fixture rather than trusting the predicate +// under test. +func objectPresentInStore(t *testing.T, dir, sha string) bool { + t.Helper() + _, err := gitpkg.Run(context.Background(), dir, "cat-file", "-e", sha) + return err == nil +} + +// TestKeepLocalStampFailureAfterTheGateMovedNamesACompletableRetry is the +// post-SUCCESS failure window, which a search for refusals does not surface. +// Once the gate compare-and-swap lands, the settlement's Git work is done and +// only the database stamp remains; if that write fails, the run is left with a +// moved gate branch, a written recovery anchor, and no custody record. Nilling +// NextAction there reproduced the #824 shape one layer down - refs changed and +// nothing named that could end it. The retry has to be named structurally, and +// it has to actually work. +func TestKeepLocalStampFailureAfterTheGateMovedNamesACompletableRetry(t *testing.T) { + t.Parallel() + + f, staleGate, _ := wedgedCustodyFixture(t, types.RunFailed) + stampCalls := 0 + f.service.stampCustodyReturnedFn = func(string) error { + stampCalls++ + return errors.New("database is locked") + } + + state := f.service.Recover(f.ctx, true) + if state.Recovered { + t.Fatalf("failed stamp reported custody returned = %#v", state) + } + if stampCalls != 1 { + t.Fatalf("stamp calls = %d, want 1", stampCalls) + } + if state.Safety != "blocked_recover_stamp_failed" { + t.Fatalf("stamp-failure safety = %q: %#v", state.Safety, state) + } + if f.custodyReturned() { + t.Fatal("failed stamp recorded custody anyway") + } + // The Git side really did complete, which is exactly why the state must + // not claim otherwise and must not dead-end. + if got := mustRun(t, f.gate, "rev-parse", "refs/heads/feature/recover"); got != f.submitted { + t.Fatalf("gate branch = %s, want the kept local head %s", got, f.submitted) + } + if got := mustRun(t, f.gate, "rev-parse", custody.RecoveryGateRef(f.run.ID)); got != staleGate { + t.Fatalf("displaced gate head anchor = %s, want %s", got, staleGate) + } + if strings.Contains(state.Error, "no files or refs were changed") { + t.Fatalf("stamp failure claimed nothing changed after moving the gate: %q", state.Error) + } + if state.NextAction == nil { + t.Fatalf("stamp failure named no exit at all = %#v", state) + } + if state.NextAction.Code != "return_custody_keep_local" { + t.Fatalf("stamp failure named %q, want the settlement it was performing", state.NextAction.Code) + } + if !strings.Contains(state.NextAction.Command, "--keep-local") { + t.Fatalf("stamp-failure retry command = %q", state.NextAction.Command) + } + + // The prescribed retry must complete, not refuse: the gate now already + // equals the kept head, so the settlement's whole move is skipped. + f.service.stampCustodyReturnedFn = nil + retry := f.service.Recover(f.ctx, true) + if !retry.Recovered { + t.Fatalf("the retry the stamp failure prescribes did not complete = %#v", retry) + } + if !f.custodyReturned() { + t.Fatal("the prescribed retry did not record custody") + } +} + +// TestDefaultRecoveryStampFailureNamesItsOwnRetry keeps the same guarantee on +// the default --recover path, whose worktree has already moved by then. +func TestDefaultRecoveryStampFailureNamesItsOwnRetry(t *testing.T) { + t.Parallel() + + f := newRecoverFixture(t, types.RunCancelled) + f.service.stampCustodyReturnedFn = func(string) error { + return errors.New("database is locked") + } + + state := f.service.Recover(f.ctx, false) + if state.Safety != "blocked_recover_stamp_failed" { + t.Fatalf("default-recovery stamp-failure safety = %q: %#v", state.Safety, state) + } + if state.NextAction == nil || state.NextAction.Code != "recover_custody" { + t.Fatalf("default-recovery stamp failure named %#v, want recover_custody", state.NextAction) + } + if strings.Contains(state.NextAction.Command, "--keep-local") { + t.Fatalf("default recovery prescribed the keep-local settlement: %q", state.NextAction.Command) + } + + f.service.stampCustodyReturnedFn = nil + retry := f.service.Recover(f.ctx, false) + if !retry.Recovered { + t.Fatalf("the retry the stamp failure prescribes did not complete = %#v", retry) + } +} diff --git a/internal/branchsync/sync.go b/internal/branchsync/sync.go index 52e040a13..0a7e9fe65 100644 --- a/internal/branchsync/sync.go +++ b/internal/branchsync/sync.go @@ -154,8 +154,20 @@ type Service struct { lsRemote func(context.Context, string, string, string) (string, error) fetchRemote func(context.Context, string, string, string, string) error + // absPath resolves the invoking worktree to an absolute path. It is a + // seam because filepath.Abs fails only when the process working directory + // has been removed, which cannot be induced in-process without breaking + // every parallel test - and that refusal path has to prove it leaves no + // recovery anchor behind. + absPathFn func(string) (string, error) + + // stampCustodyReturnedFn is the recovery's database write. Nil uses the + // production DB. + stampCustodyReturnedFn func(string) error + beforeApply func() beforeGateReset func() + beforeGateStage func() beforeRecoverWorktreeMove func() beforeRecoverBranchMove func() afterRecoverBranchMove func() @@ -522,7 +534,16 @@ func (s *Service) Apply(ctx context.Context) State { // work // diverged any refuse (anchor named, manual custody at local head; // reconcile / rerun offered) gate reset to it (CAS) -// P missing any refuse refuse +// P missing any refuse settle: custody at local +// or its head, gate reset to it +// evidence (CAS), any surviving +// conflicts copy of P pinned first +// +// The last row is the self-inconsistent record of issue #824: a record that +// contradicts itself has no verifiable head to protect, and refusing there +// left the branch pipeline_owned with no command that could ever settle it. +// recoverSettleInconsistent owns that reasoning and keeps it fail-closed for a +// surviving-but-unanchorable head. // // The containment row exists because a cancelled validation routinely leaves P // as a REBASE of the local branch onto a newer base: the same logical commits @@ -583,23 +604,25 @@ func (s *Service) Recover(ctx context.Context, keepLocal bool) State { return blockedPlan(state, state.State, "blocked_recover_not_applicable", "nothing to recover: the branch is not held by a terminal run with unpublished pipeline commits; no files or refs were changed") } if !terminalRunStatus(run.Status) { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_run_active", "the run that owns this branch is still active; drive it to completion or abort it first; no files or refs were changed") + blocked := blockedPlan(state, StatePipelineOwned, "blocked_recover_run_active", "the run that owns this branch is still active; drive it to completion or abort it first; no files or refs were changed") + blocked.NextAction = &NextAction{Code: "continue_active_run", Command: "no-mistakes axi status"} + return blocked } if run.TerminalHeadVerifiedAt == nil { branch := state.Local.Branch if strings.TrimSpace(s.GateDir) == "" { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_unverified_head", "the terminal run has no verified head and no gate is available to prove preserved custody; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_unverified_head", "the terminal run has no verified head and no gate is available to prove preserved custody; no files or refs were changed") } gateHead, err := git.Run(ctx, s.GateDir, "rev-parse", "refs/heads/"+branch+"^{commit}") if err != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_unverified_head", "the terminal run has no verified head and the preserved gate head could not be read; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_unverified_head", "the terminal run has no verified head and the preserved gate head could not be read; no files or refs were changed") } if gateHead != run.HeadSHA { if !isAncestor(ctx, s.GateDir, run.HeadSHA, gateHead) { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_unverified_head", "the terminal run has no verified head and the gate head does not descend from the recorded head; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_unverified_head", "the terminal run has no verified head and the gate head does not descend from the recorded head; no files or refs were changed") } if err := s.DB.UpdateRunHeadSHA(run.ID, gateHead); err != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_unverified_head", "the verified gate head could not be preserved; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_unverified_head", "the verified gate head could not be preserved; no files or refs were changed") } run.HeadSHA = gateHead state.Pipeline.CurrentHead = gateHead @@ -628,70 +651,96 @@ func (s *Service) Recover(ctx context.Context, keepLocal bool) State { if objectExists(ctx, wd, preserved) && (local == preserved || isAncestor(ctx, wd, preserved, local)) { if blocked, ok := s.anchorReachablePreserved(ctx, state, run.ID, preserved); !ok { + // The recovery ref in the invoking worktree is unusable, which is + // the same self-inconsistency the gate-side sites settle - and here + // the preserved head is already reachable from the local branch, so + // keeping that head can lose nothing at all. + if keepLocal { + return s.recoverSettleInconsistent(ctx, run, state, gateDir, preserved) + } return blocked } if gateAvailable { compatible, err := recoveryAnchorCompatible(ctx, gateDir, run.ID, preserved) if err != nil || !compatible { + if keepLocal { + return s.recoverSettleInconsistent(ctx, run, state, gateDir, preserved) + } return blockedPlan(state, StatePipelineOwned, "blocked_recover_anchor_mismatch", "the run recovery ref in the local gate conflicts with the recorded pipeline head; inspect both objects before returning custody; no files or refs were changed") } } - return s.finishRecover(ctx, run, false) + return s.finishRecover(ctx, run, false, keepLocal) } if !gateAvailable { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_gate_unavailable", "no local gate is configured for this repository, so the preserved pipeline head cannot be imported; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_gate_unavailable", "no local gate is configured for this repository, so the preserved pipeline head cannot be imported; no files or refs were changed") } gateAnchor := custody.RecoveryRef(run.ID) gateAnchorAvailable := false gateAnchorTarget, gateAnchorExists, targetErr := git.ExactRefTarget(ctx, gateDir, gateAnchor) if targetErr != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_anchor_mismatch", "the run recovery ref could not be inspected; inspect the recorded and live heads before returning custody; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_anchor_mismatch", "the run recovery ref could not be inspected; inspect the recorded and live heads before returning custody; no files or refs were changed") } if gateAnchorExists { gateAnchored, err := git.Run(ctx, gateDir, "rev-parse", gateAnchor+"^{commit}") if err != nil { + if keepLocal { + return s.recoverSettleInconsistent(ctx, run, state, gateDir, preserved) + } return blockedPlan(state, StatePipelineOwned, "blocked_recover_anchor_mismatch", fmt.Sprintf("the run recovery ref points at non-commit object %s instead of the recorded pipeline head %s; inspect both objects before returning custody; no files or refs were changed", gateAnchorTarget, preserved)) } if gateAnchored != preserved { + if keepLocal { + return s.recoverSettleInconsistent(ctx, run, state, gateDir, preserved) + } return blockedPlan(state, StatePipelineOwned, "blocked_recover_anchor_mismatch", fmt.Sprintf("the run recovery ref points at %s instead of the recorded pipeline head %s; inspect both heads before returning custody; no files or refs were changed", gateAnchored, preserved)) } gateAnchorAvailable = true } if !gateAnchorAvailable { if !objectExists(ctx, gateDir, preserved) { + if keepLocal { + return s.recoverSettleInconsistent(ctx, run, state, gateDir, preserved) + } return blockedPlan(state, StatePipelineOwned, "blocked_recover_preserved_head_missing", fmt.Sprintf("the recorded pipeline head %s is missing from the local gate; inspect the recorded and live heads before returning custody; no files or refs were changed", preserved)) } if err := custody.PreserveRecoveryHead(ctx, gateDir, run.ID, preserved); err != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_preserve_failed", "the recorded pipeline head exists but could not be anchored in the local gate; no files or worktree refs were changed") + return recoverBlocked(state, "blocked_recover_preserve_failed", "the recorded pipeline head exists but could not be anchored in the local gate; no files or worktree refs were changed") } } anchored := false + // What this attempt itself writes, so a later keep-local refusal reports + // the anchor rather than claiming nothing was written. + anchoredNote := "" if existing, anchorErr := git.Run(ctx, wd, "rev-parse", anchorRef+"^{commit}"); anchorErr == nil && existing == preserved { anchored = true } if !anchored { if fetchErr := git.FetchRemoteRef(ctx, wd, gateDir, gateAnchor, preserved); fetchErr != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_preserve_failed", "the preserved pipeline commits could not be fetched from the local gate; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_preserve_failed", "the preserved pipeline commits could not be fetched from the local gate; no files or refs were changed") } if preserveErr := custody.PreserveRecoveryAnchor(ctx, wd, anchorRef, preserved); preserveErr != nil { + if keepLocal { + return s.recoverSettleInconsistent(ctx, run, state, gateDir, preserved) + } return blockedPlan(state, StatePipelineOwned, "blocked_recover_anchor_mismatch", "the invoking worktree recovery ref conflicts with the recorded pipeline head; inspect both objects before returning custody; no files or refs were changed") } + anchoredNote = anchoredElsewhere([]string{"the invoking worktree"}, anchorRef) } switch { case local == preserved, isAncestor(ctx, wd, preserved, local): // Equal or ahead, discovered only after anchoring made the preserved // head comparable locally. - return s.finishRecover(ctx, run, false) + return s.finishRecover(ctx, run, false, keepLocal) case isAncestor(ctx, wd, local, preserved): if keepLocal { gateHead, err := git.Run(ctx, gateDir, "rev-parse", "refs/heads/"+branch+"^{commit}") if err != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_gate_unavailable", fmt.Sprintf("the local gate no longer has branch %s, so it cannot be updated with the kept local head; no files or refs were changed", branch)) + return recoverBlocked(state, "blocked_recover_gate_unavailable", fmt.Sprintf("the local gate no longer has branch %s, so it cannot be updated with the kept local head; no files or refs were changed", branch)) } - return s.recoverKeepLocal(ctx, run, state, gateHead) + return s.recoverKeepLocal(ctx, run, state, gateHead, anchoredNote) } if !state.Local.Clean { state.Relation = RelationBehind @@ -704,9 +753,9 @@ func (s *Service) Recover(ctx context.Context, keepLocal bool) State { if keepLocal { gateHead, err := git.Run(ctx, gateDir, "rev-parse", "refs/heads/"+branch+"^{commit}") if err != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_gate_unavailable", fmt.Sprintf("the local gate no longer has branch %s, so it cannot be updated with the kept local head; no files or refs were changed", branch)) + return recoverBlocked(state, "blocked_recover_gate_unavailable", fmt.Sprintf("the local gate no longer has branch %s, so it cannot be updated with the kept local head; no files or refs were changed", branch)) } - return s.recoverKeepLocal(ctx, run, state, gateHead) + return s.recoverKeepLocal(ctx, run, state, gateHead, anchoredNote) } if preservedContainsLocalWork(ctx, wd, local, preserved) { if !state.Local.Clean { @@ -724,57 +773,256 @@ func (s *Service) Recover(ctx context.Context, keepLocal bool) State { } } +// recoverSettleInconsistent is the explicit keep-local settlement of a +// SELF-INCONSISTENT custody record: a terminal run whose recorded pipeline +// head cannot be verified at all, because no reachable object store still has +// it or because the run's own recovery evidence names a different object. Such +// a record used to have no supported exit - every guarded recovery refused on +// the same unverifiable head, abort of the already-terminal run was an +// idempotent no-op, and the branch stayed pipeline_owned forever (issue #824). +// +// The settlement is deliberately narrow. It is reached only with --keep-local, +// which is the operator stating that the head they want is the one already in +// their worktree, and it is a shortcut for a record that contradicts itself, +// never for a record protecting unique unpublished content: +// +// - Every reachable copy of the recorded pipeline head is pinned first, at +// RecoveryStrandedRef, so a head that still exists survives the settlement +// as inspectable evidence. If such a head exists and CANNOT be pinned, the +// settlement refuses: that is exactly the "unique unpublished pipeline +// commits that cannot be anchored" case, and it stays fail-closed. +// - A head that no store still has cannot be pinned and cannot be lost. It +// is already unrecoverable, so refusing protects nothing and only strands +// the branch. +// - The gate branch then moves through the ordinary keep-local path, which +// anchors an independently moved gate head before an atomic +// compare-and-swap. The gate is never force-moved, so a concurrent gate +// push still wins and the settlement refuses. +// +// Only a gate branch PROVEN absent settles without that compare-and-swap. An +// unreadable gate branch is not evidence of absence, so it refuses exactly like +// the sibling keep-local sites in Recover rather than stamping custody against +// a gate head that was never observed. +// +// Every refusal here names a completable exit. blockedPlan clears NextAction by +// default, and this function is the terminal settlement, so a refusal with no +// next action would be the dead end the whole issue exists to remove - even for +// a shape selfInconsistentCustodyRecord failed to disqualify. +func (s *Service) recoverSettleInconsistent(ctx context.Context, run *db.Run, state State, gateDir, preserved string) State { + strandedRef := custody.RecoveryStrandedRef(run.ID) + gateDir = strings.TrimSpace(gateDir) + // The stranded anchor is the ONLY ref this function can write before the + // gate move, so a refusal reports precisely where it now exists rather + // than claiming nothing changed. + pinned := []string{} + for _, store := range []struct{ name, dir string }{ + {"the invoking worktree", s.workDir()}, + {"the local gate", gateDir}, + } { + if store.dir == "" { + continue + } + // The settlement's whole data-safety argument is "no store still has + // this head, so settling cannot lose it". That is a claim about proven + // absence, and a bare presence probe cannot make it: it reads an + // unreadable store exactly like an empty one and would skip the pin on + // evidence it never actually gathered. Only git.CommitPresence's exit-1 + // answer proves absence; anything else is undetermined and refuses. + present, err := git.CommitPresence(ctx, store.dir, preserved) + if err != nil { + blocked := blockedPlan(state, StatePipelineOwned, "blocked_recover_preserve_failed", fmt.Sprintf("whether the recorded pipeline head %s still exists could not be determined in %s, so its absence cannot be proven and custody cannot be settled without risking it; inspect that object store before retrying; no branch, worktree, or file changes were made%s", preserved, store.name, anchoredElsewhere(pinned, strandedRef))) + blocked.NextAction = &NextAction{Code: "inspect_and_reconcile_manually", Command: "no-mistakes axi status"} + return blocked + } + if !present { + continue + } + if err := custody.PreserveRecoveryAnchor(ctx, store.dir, strandedRef, preserved); err != nil { + blocked := blockedPlan(state, StatePipelineOwned, "blocked_recover_preserve_failed", fmt.Sprintf("the recorded pipeline head %s still exists but could not be anchored at %s in %s, so custody cannot be settled without stranding it; inspect that object before retrying; no branch, worktree, or file changes were made%s", preserved, strandedRef, store.name, anchoredElsewhere(pinned, strandedRef))) + blocked.NextAction = &NextAction{Code: "inspect_and_reconcile_manually", Command: "no-mistakes axi status"} + return blocked + } + pinned = append(pinned, store.name) + } + if gateDir == "" { + return s.finishRecover(ctx, run, false, true) + } + gateBranchRef := "refs/heads/" + state.Local.Branch + _, gateBranchExists, err := git.ExactRefTarget(ctx, gateDir, gateBranchRef) + if err != nil { + blocked := blockedPlan(state, StatePipelineOwned, "blocked_recover_gate_unavailable", fmt.Sprintf("the local gate branch %s could not be read, so the kept local head cannot be compared and swapped onto it; no branch refs were changed%s", state.Local.Branch, anchoredElsewhere(pinned, strandedRef))) + blocked.NextAction = &NextAction{Code: "inspect_and_reconcile_manually", Command: "no-mistakes axi status"} + return blocked + } + if !gateBranchExists { + // Proven absent: the gate holds no branch ref for this run, so nothing + // there holds custody and there is no ref to compare-and-swap. + return s.finishRecover(ctx, run, false, true) + } + gateHead, err := git.Run(ctx, gateDir, "rev-parse", gateBranchRef+"^{commit}") + if err != nil { + blocked := blockedPlan(state, StatePipelineOwned, "blocked_recover_gate_unavailable", fmt.Sprintf("the local gate branch %s does not resolve to a commit, so the kept local head cannot be compared and swapped onto it; no branch refs were changed%s", state.Local.Branch, anchoredElsewhere(pinned, strandedRef))) + blocked.NextAction = &NextAction{Code: "inspect_and_reconcile_manually", Command: "no-mistakes axi status"} + return blocked + } + return s.recoverKeepLocal(ctx, run, state, gateHead, anchoredElsewhere(pinned, strandedRef)) +} + +// anchoredElsewhere keeps a refusal honest about a recorded head its own +// attempt has already anchored before failing. +func anchoredElsewhere(pinned []string, ref string) string { + if len(pinned) == 0 { + return "" + } + return fmt.Sprintf("; the recorded head is now anchored at %s in %s", ref, strings.Join(pinned, " and ")) +} + +// keepLocalNoChangeClause closes a PRE-SWAP recoverKeepLocal refusal with a +// claim that stays true for its caller too. Those refusals write nothing of +// their own by construction, but their callers can already have anchored a +// surviving recorded head, so the blanket claim belongs only to a delegation +// that carries no such anchor; otherwise the refusal reports exactly what it +// did not change and hands over the anchor note naming where that head now is. +// +// The lost compare-and-swap is deliberately NOT built here. That refusal has +// just written the displaced-gate-head anchor into the gate, so its narrower +// "no LOCAL files or refs were changed" claim is the accurate one and the +// anchor note is appended to it rather than replacing it. +func keepLocalNoChangeClause(blanket, anchored string) string { + if anchored == "" { + return blanket + } + return "no branch, worktree, or file changes were made" + anchored +} + +// recoverBlocked is the one constructor for a recovery refusal that has no +// more specific exit to offer, because blockedPlan clears NextAction and a +// refusal naming no exit is itself the R1/R5 dead end this subsystem exists to +// remove. That principle does not depend on --keep-local: a default --recover +// that refuses strands the operator exactly as badly. Manual reconciliation is +// the honest exit here - recovery is already the recorded head's last resort, +// so a refusal has nothing further to prescribe. Building it in one place is +// what keeps the guarantee true as new refusal sites appear. +func recoverBlocked(state State, safety, message string) State { + blocked := blockedPlan(state, StatePipelineOwned, safety, message) + blocked.NextAction = &NextAction{Code: "inspect_and_reconcile_manually", Command: "no-mistakes axi status"} + return blocked +} + // recoverKeepLocal performs the explicit keep-local custody return: the // worktree is never touched; the gate branch moves to the kept local head with // an atomic compare-and-swap so a concurrent gate push refuses instead of // being clobbered. The kept head's objects reach the gate through a gate-side // fetch - never a push, which would fire the gate's receive hooks and start a // pipeline run. The preserved head stays reachable through the anchor ref. -func (s *Service) recoverKeepLocal(ctx context.Context, run *db.Run, state State, gateHead string) State { +// +// Write ordering is load-bearing, not incidental. The displaced-gate-head +// anchor (RecoveryGateRef) is the ref whose staleness wedges every later +// attempt: settlementAnchorsFree stops advertising the settlement once it +// names a commit the gate has since moved off, PreserveRecoveryAnchor then +// refuses to retarget it, and nothing in the product retires it. So a refusal +// that leaves it behind recreates exactly the #824 dead end this settlement +// exists to clear - while telling the operator that nothing changed. +// +// The anchor therefore protects exactly one operation: the compare-and-swap +// that moves the gate branch off gateHead. Nothing before that swap can strand +// gateHead, because refs/heads/ still names it. So every check that can +// refuse - the local head re-read, the worktree path resolution, the staging +// fetch, and the staged-head verification - runs BEFORE the anchor is written, +// and the write happens immediately before the swap it guards. That keeps +// "no files or refs were changed" true of THIS function on every one of those +// paths by construction rather than by a cleanup that could itself fail. +// +// A refusal can only speak for itself, though, and both callers reach here +// after possibly anchoring a surviving recorded head of their own - Recover +// pins the preserved head at the run recovery ref, and +// recoverSettleInconsistent pins every surviving copy at the stranded ref. +// anchoredNote carries that fact in, so the blanket claim is made only by a +// delegation that wrote nothing and every other refusal names where the +// anchor now is instead of reporting that nothing was written. The lost +// compare-and-swap keeps its own narrower local-scoped claim and appends the +// note, because by then this function has written the gate anchor itself. +// +// The anchor CONFLICT check is deliberately still read-only and still runs +// first: it is the cheapest refusal and it must not be reached only after a +// staging ref exists. Once the swap has been attempted, the anchor is load +// bearing and is never retired - a failed compare-and-swap means the gate +// moved, so gateHead may now be reachable through the anchor alone. +func (s *Service) recoverKeepLocal(ctx context.Context, run *db.Run, state State, gateHead, anchoredNote string) State { if s.beforeGateReset != nil { s.beforeGateReset() } + gateHeadAnchored := false if gateHead != state.Local.Head { + gateAnchor := "" + writeGateAnchor := false if gateHead != run.HeadSHA { - gateAnchor := custody.RecoveryGateRef(run.ID) + gateAnchor = custody.RecoveryGateRef(run.ID) existing, exists, err := git.ExactRefTarget(ctx, s.GateDir, gateAnchor) if err != nil || (exists && existing != gateHead) { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_preserve_failed", "the independently moved gate head conflicts with the existing run recovery anchor; inspect both refs before returning custody; no files or branch refs were changed") - } - if !exists { - err = custody.PreserveRecoveryAnchor(ctx, s.GateDir, gateAnchor, gateHead) - } - if err != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_preserve_failed", "the independently moved gate head could not be anchored before returning custody; no files or branch refs were changed") + conflict := "could not be read" + if err == nil { + conflict = "names " + existing + } + return recoverBlocked(state, "blocked_recover_preserve_failed", fmt.Sprintf("the independently moved gate head %s conflicts with the existing run recovery anchor %s in the local gate %s, which %s; nothing retires that anchor, so reconcile it there before returning custody; %s", gateHead, gateAnchor, s.GateDir, conflict, keepLocalNoChangeClause("no files or branch refs were changed", anchoredNote))) } + // An anchor that already names this head needs no write, but it + // still guards the swap, so the race refusal below must know it + // is there. + gateHeadAnchored = true + writeGateAnchor = !exists } head, err := git.HeadSHA(ctx, s.workDir()) if err != nil || head != state.Local.Head { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_assumptions_changed", "the local branch head changed while custody was being returned; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_assumptions_changed", "the local branch head changed while custody was being returned; "+keepLocalNoChangeClause("no files or refs were changed", anchoredNote)) } // The fetch source must be absolute: the command runs inside the gate // directory, where a relative invoking-worktree path would resolve to // the gate itself. - source, err := filepath.Abs(s.workDir()) + source, err := s.absPath(s.workDir()) if err != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_assumptions_changed", "the invoking worktree path could not be resolved; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_assumptions_changed", "the invoking worktree path could not be resolved; "+keepLocalNoChangeClause("no files or refs were changed", anchoredNote)) + } + if s.beforeGateStage != nil { + s.beforeGateStage() } stagingRef := "refs/no-mistakes/custody-return/" + run.ID if _, err := git.Run(ctx, s.GateDir, "fetch", "--no-tags", "--no-write-fetch-head", source, "+refs/heads/"+state.Local.Branch+":"+stagingRef); err != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_assumptions_changed", "the kept local head could not be staged into the gate; no files or refs were changed") + // A partly completed fetch can still have created the staging ref, + // and the refusal claims nothing was left behind. + _, _ = git.Run(ctx, s.GateDir, "update-ref", "-d", stagingRef) + return recoverBlocked(state, "blocked_recover_assumptions_changed", "the kept local head could not be staged into the gate; "+keepLocalNoChangeClause("no files or refs were changed", anchoredNote)) } staged, err := git.Run(ctx, s.GateDir, "rev-parse", stagingRef+"^{commit}") if err != nil || staged != state.Local.Head { _, _ = git.Run(ctx, s.GateDir, "update-ref", "-d", stagingRef) - return blockedPlan(state, StatePipelineOwned, "blocked_recover_assumptions_changed", "the local branch head changed while custody was being returned; no files or refs were changed") + return recoverBlocked(state, "blocked_recover_assumptions_changed", "the local branch head changed while custody was being returned; "+keepLocalNoChangeClause("no files or refs were changed", anchoredNote)) + } + // Point of no return: from here the gate branch is about to leave + // gateHead, so the anchor has to exist first. + if writeGateAnchor { + if err := custody.PreserveRecoveryAnchor(ctx, s.GateDir, gateAnchor, gateHead); err != nil { + _, _ = git.Run(ctx, s.GateDir, "update-ref", "-d", stagingRef) + return recoverBlocked(state, "blocked_recover_preserve_failed", "the independently moved gate head could not be anchored before returning custody; "+keepLocalNoChangeClause("no files or branch refs were changed", anchoredNote)) + } } _, casErr := git.Run(ctx, s.GateDir, "update-ref", "refs/heads/"+state.Local.Branch, state.Local.Head, gateHead) _, _ = git.Run(ctx, s.GateDir, "update-ref", "-d", stagingRef) if casErr != nil { - return blockedPlan(state, StatePipelineOwned, "blocked_recover_gate_race", "the gate branch changed while custody was being returned; re-run the recovery; no local files or refs were changed") + // The displaced-gate-head anchor is written only above, when the + // gate had already moved off the recorded head. Without it a retry + // simply observes the new head and succeeds; with it, the retry + // hits the anchor conflict instead, so only then is reconciling + // the anchor the honest advice. It is never retired here: the swap + // failed because the gate moved, so gateHead may now be reachable + // through this anchor alone. + if gateHeadAnchored { + return recoverBlocked(state, "blocked_recover_gate_race", fmt.Sprintf("the gate branch changed while custody was being returned, so the compare-and-swap refused instead of clobbering it; the run recovery anchor %s still names the gate head this attempt observed, so a further attempt refuses on that conflict - reconcile that anchor against the live gate head before returning custody; no local files or refs were changed%s", custody.RecoveryGateRef(run.ID), anchoredNote)) + } + return recoverBlocked(state, "blocked_recover_gate_race", "the gate branch changed while custody was being returned, so the compare-and-swap refused instead of clobbering it; no displaced-gate-head anchor was written, so re-run the recovery to return custody against the new gate head; no local files or refs were changed"+anchoredNote) } } - return s.finishRecover(ctx, run, false) + return s.finishRecover(ctx, run, false, true) } // recoverFastForward advances the clean checked-out branch to the preserved @@ -808,7 +1056,7 @@ func (s *Service) recoverFastForward(ctx context.Context, run *db.Run, state Sta state.NextAction = &NextAction{Code: "inspect_worktree", Command: "git status"} return state } - return s.finishRecover(ctx, run, true) + return s.finishRecover(ctx, run, true, false) } // preservedContainsLocalWork proves the preserved pipeline head already carries @@ -970,7 +1218,7 @@ func (s *Service) recoverAdoptPreserved(ctx context.Context, run *db.Run, state state.NextAction = &NextAction{Code: "inspect_worktree", Command: "git status"} return state } - return s.finishRecover(ctx, run, true) + return s.finishRecover(ctx, run, true, false) } func (s *Service) anchorReachablePreserved(ctx context.Context, state State, runID, preserved string) (State, bool) { @@ -986,13 +1234,24 @@ func (s *Service) anchorReachablePreserved(ctx context.Context, state State, run // finishRecover stamps custody returned and reports the fresh post-recovery // truth. changed reports whether this call moved the worktree HEAD. -func (s *Service) finishRecover(ctx context.Context, run *db.Run, changed bool) State { - if err := s.DB.SetRunCustodyReturned(run.ID); err != nil { +// finishRecover records the custody return. Its failure branch is the one +// place in recovery where the Git side has ALREADY succeeded - a keep-local +// settlement has moved the gate branch, a fast-forward or adoption has moved +// the worktree - and only the database write failed. It therefore must never +// be a dead end: leaving NextAction nil there reproduces the #824 shape one +// layer down, a state that changed refs and then names nothing that can end +// it. Re-running the SAME recovery command is genuinely completable, because +// every Git step it repeats is idempotent once applied: the gate branch now +// equals the kept head so recoverKeepLocal skips its whole move, and an +// already-advanced worktree takes the equal/ahead path. That is why the retry +// is named structurally here and not only in prose. +func (s *Service) finishRecover(ctx context.Context, run *db.Run, changed, keepLocal bool) State { + if err := s.stampCustodyReturned(run.ID); err != nil { state, _, _ := s.inspect(ctx) state.Changed = changed state.Safety = "blocked_recover_stamp_failed" - state.Error = "the custody return could not be recorded; re-run the recovery" - state.NextAction = nil + state.Error = "the recovery's Git changes are already applied, but the custody return could not be recorded; re-run the same recovery to complete the record" + state.NextAction = recoveryRetryAction(keepLocal) return state } state, _, _ := s.inspect(ctx) @@ -1001,6 +1260,25 @@ func (s *Service) finishRecover(ctx context.Context, run *db.Run, changed bool) return state } +// recoveryRetryAction names the exact command that finishes an interrupted +// recovery: the one the operator already ran. +func recoveryRetryAction(keepLocal bool) *NextAction { + if keepLocal { + return &NextAction{Code: "return_custody_keep_local", Command: "no-mistakes axi sync --recover --keep-local"} + } + return &NextAction{Code: "recover_custody", Command: "no-mistakes axi sync --recover"} +} + +// stampCustodyReturned is the recovery's only database write, behind a seam so +// its failure - the one post-success failure recovery has - is reachable in a +// test without corrupting a shared database. +func (s *Service) stampCustodyReturned(runID string) error { + if s.stampCustodyReturnedFn != nil { + return s.stampCustodyReturnedFn(runID) + } + return s.DB.SetRunCustodyReturned(runID) +} + func recoverAnchorRef(runID string) string { return custody.RecoveryRef(runID) } @@ -1313,6 +1591,15 @@ func (s *Service) remoteName(ctx context.Context) string { return "origin" } +// absPath resolves a path through the service's seam so the keep-local +// custody return can prove what it does when that resolution fails. +func (s *Service) absPath(path string) (string, error) { + if s.absPathFn != nil { + return s.absPathFn(path) + } + return filepath.Abs(path) +} + func (s *Service) workDir() string { if strings.TrimSpace(s.WorkDir) == "" { return "." @@ -1436,6 +1723,16 @@ func (s *Service) classifyPipelineOwned(ctx context.Context, state *State, run * if terminalRunStatus(run.Status) { if !s.recoverySourceAvailable(ctx, state, run) { state.Safety = "blocked_recover_preserved_head_missing" + // A record that contradicts itself must still name an exit that + // can complete: `--keep-local` settles it at the head the operator + // already has (issue #824). Only a record that keeps its own + // evidence intact falls back to manual reconciliation, because + // there the refusal is protecting something real. + if s.selfInconsistentCustodyRecord(ctx, state, run) { + state.Error = "the run finished " + string(run.Status) + " but its recorded pipeline head cannot be verified in the invoking worktree or local gate; return custody at the current local head, which also points the gate branch at it" + state.NextAction = &NextAction{Code: "return_custody_keep_local", Command: "no-mistakes axi sync --recover --keep-local"} + return + } state.Error = "the run finished " + string(run.Status) + " but its recorded pipeline head is not available in the invoking worktree or local gate; inspect and reconcile the recorded and live heads manually" state.NextAction = &NextAction{Code: "inspect_and_reconcile_manually", Command: "no-mistakes axi status"} return @@ -1450,6 +1747,194 @@ func (s *Service) classifyPipelineOwned(ctx context.Context, state *State, run * state.NextAction = &NextAction{Code: "continue_active_run", Command: "no-mistakes axi status"} } +// selfInconsistentCustodyRecord reports the exact class of stranded record +// that `--recover --keep-local` can settle: the recorded pipeline head is not +// in any reachable object store, or the run's own recovery evidence names +// something other than that head. Both mean nothing about the record can be +// verified, so a refusal protects nothing and only strands the branch. +// +// It is deliberately conservative. A record whose evidence is intact but whose +// recovery is merely blocked (a genuinely diverged head, an absent gate that +// might just be unmounted) is NOT self-inconsistent: settlement there would +// abandon a preserved head that recovery can still import, so those keep +// pointing at manual reconciliation. +// +// The advertisement carries the same fail-closed polarity as +// recoverSettleInconsistent: it may name the settlement only when +// Recover(keepLocal) actually reaches it AND the settlement can complete +// there. A gate recovery ref that cannot be inspected at all is not evidence +// of inconsistency - Recover refuses that probe error strictly before any +// keep-local interception - and an incomplete adoption (the branch reached the +// preserved head while the worktree still differs) is refused by its own guard +// even earlier, so both disqualify the record and keep the honest +// manual-reconciliation pointer. +// +// The three refs the settlement itself writes or swaps are probed for the same +// reason, exactly the way it probes them, because these refuse INSIDE the +// settlement rather than at a guard Recover reaches first: the gate BRANCH it +// compare-and-swaps (a ref that cannot be read or does not name a commit +// refuses, while one PROVEN absent is the deliberate exception, since the +// settlement completes there with no swap at all), and the stranded and gate +// anchor refs of settlementAnchorsFree. +// +// An UNVERIFIED terminal head is excluded for a different reason: Recover +// refuses it at the unverified-head guard, strictly before any keep-local +// interception, so advertising the settlement there would recreate the very +// "advertised action that always refuses" wedge this change exists to remove. +// Making that shape recoverable is issue #707's scope, not this one; until +// then the honest pointer is manual reconciliation. +func (s *Service) selfInconsistentCustodyRecord(ctx context.Context, state *State, run *db.Run) bool { + if run == nil || run.TerminalHeadVerifiedAt == nil { + return false + } + preserved := strings.TrimSpace(run.HeadSHA) + gateDir := strings.TrimSpace(s.GateDir) + if gateDir == "" { + return false + } + if _, err := os.Stat(gateDir); err != nil { + return false + } + wd := s.workDir() + // Symbolic gate-anchor evidence disqualifies the record outright. A + // DANGLING symref is invisible to for-each-ref while symbolic-ref still + // succeeds, so recoveryAnchorCompatible reports "incompatible" with no + // error while Recover's own probe sees no ref at all and then fails inside + // PreserveRecoveryHead's symbolic check - a refusal site that deliberately + // carries no keep-local interception. A RESOLVING symref gets further and + // still refuses on the ordinary keep-local path. Either way the settlement + // is unreachable, so it must not be advertised. + if symbolic, symErr := git.Run(ctx, gateDir, "symbolic-ref", "-q", custody.RecoveryRef(run.ID)); symErr == nil && symbolic != "" { + return false + } + if !s.settlementGateBranchUsable(ctx, state, gateDir) { + return false + } + if !s.settlementAnchorsFree(ctx, state, run, gateDir, preserved) { + return false + } + gateCompatible, err := recoveryAnchorCompatible(ctx, gateDir, run.ID, preserved) + if err != nil { + return false + } + if state != nil && state.Local.Head == preserved && !state.Local.Clean { + if anchoredLocal, err := git.Run(ctx, wd, "rev-parse", "--verify", custody.RecoveryLocalRef(run.ID)+"^{commit}"); err == nil && anchoredLocal != preserved { + return false + } + } + if preserved == "" { + return true + } + if !objectExists(ctx, wd, preserved) && !objectExists(ctx, gateDir, preserved) { + return true + } + if !gateCompatible { + return true + } + if compatible, err := recoveryAnchorCompatible(ctx, wd, run.ID, preserved); err == nil && !compatible { + return true + } + return false +} + +// settlementGateBranchUsable answers the one question recoverSettleInconsistent +// asks of the gate branch before its compare-and-swap: can the kept local head +// be swapped onto it, or is the branch proven absent so no swap is needed. An +// unreadable ref and a ref that does not name a commit are neither, so the +// settlement refuses there and must not be advertised. +func (s *Service) settlementGateBranchUsable(ctx context.Context, state *State, gateDir string) bool { + if state == nil { + return false + } + branch := strings.TrimSpace(state.Local.Branch) + if branch == "" { + return false + } + ref := "refs/heads/" + branch + _, exists, err := git.ExactRefTarget(ctx, gateDir, ref) + if err != nil { + return false + } + if !exists { + return true + } + _, err = git.Run(ctx, gateDir, "rev-parse", ref+"^{commit}") + return err == nil +} + +// settlementAnchorsFree answers the other two questions the settlement asks +// before it can complete, both of which refuse INSIDE it rather than at a +// guard Recover reaches first. +// +// The stranded ref is where the pin loop puts every still-reachable copy of +// the recorded head; PreserveRecoveryAnchor refuses to overwrite a ref that +// already names something else, so a leftover anchor at another commit makes +// the settlement refuse permanently. +// +// The gate ref is where recoverKeepLocal pins an independently moved gate head +// immediately before its compare-and-swap. Every refusal that can precede that +// write leaves the ref absent, so only a settlement that lost the swap race +// itself leaves it pinned at the head observed then - and there it is load +// bearing, because the gate has moved and that pin may be the only thing still +// naming the displaced head. Once the gate moves on again, that pin conflicts +// with every later attempt and nothing retires it. Probing both keeps the +// advertisement honest: a record that can only refuse falls back to manual +// reconciliation. +func (s *Service) settlementAnchorsFree(ctx context.Context, state *State, run *db.Run, gateDir, preserved string) bool { + anchorFreeAt := func(dir, ref, want string) bool { + // PreserveRecoveryAnchor refuses ANY symbolic ref as its first check, + // and a dangling symref is invisible to for-each-ref while + // symbolic-ref still resolves it - so probing with ExactRefTarget + // alone reports "free" for a ref the settlement can never write. + // Mirror recoveryAnchorCompatible so the probe agrees with the write. + if symbolic, symErr := git.Run(ctx, dir, "symbolic-ref", "-q", ref); symErr == nil && symbolic != "" { + return false + } + target, exists, err := git.ExactRefTarget(ctx, dir, ref) + if err != nil { + return false + } + if !exists { + return true + } + if target == want { + return true + } + resolved, err := git.Run(ctx, dir, "rev-parse", ref+"^{commit}") + return err == nil && resolved == want + } + stranded := custody.RecoveryStrandedRef(run.ID) + if preserved != "" { + for _, dir := range []string{s.workDir(), gateDir} { + if dir == "" { + continue + } + // Mirror the pin loop's probe, not a weaker one: a store that + // cannot answer makes the settlement refuse, so advertising it + // here would restore the advertised-action-that-always-refuses + // shape this whole change exists to remove. + present, err := git.CommitPresence(ctx, dir, preserved) + if err != nil { + return false + } + if !present { + continue + } + if !anchorFreeAt(dir, stranded, preserved) { + return false + } + } + } + if state == nil || strings.TrimSpace(state.Local.Branch) == "" { + return false + } + gateHead, err := git.Run(ctx, gateDir, "rev-parse", "refs/heads/"+state.Local.Branch+"^{commit}") + if err != nil || gateHead == state.Local.Head || gateHead == preserved { + return true + } + return anchorFreeAt(gateDir, custody.RecoveryGateRef(run.ID), gateHead) +} + func (s *Service) recoverySourceAvailable(ctx context.Context, state *State, run *db.Run) bool { if state == nil || run == nil || strings.TrimSpace(run.HeadSHA) == "" { return false diff --git a/internal/cli/axi_abort_custody_test.go b/internal/cli/axi_abort_custody_test.go new file mode 100644 index 000000000..7bb83c5f7 --- /dev/null +++ b/internal/cli/axi_abort_custody_test.go @@ -0,0 +1,489 @@ +package cli + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/kunchenguid/no-mistakes/internal/branchsync" + "github.com/kunchenguid/no-mistakes/internal/db" + "github.com/kunchenguid/no-mistakes/internal/git" + "github.com/kunchenguid/no-mistakes/internal/ipc" + "github.com/kunchenguid/no-mistakes/internal/paths" + "github.com/kunchenguid/no-mistakes/internal/types" +) + +// wedgedCustodyAbortFixture builds the issue #824 record on real state: a +// registered operator worktree, a real local gate whose branch sits at a +// LATER run's head, and a terminal run row with no push binding whose recorded +// pipeline head is in no object store at all. Aborting that run cannot cancel +// anything, so its response is the only place left to name a settlement. +func wedgedCustodyAbortFixture(t *testing.T) (string, *paths.Paths, string) { + t.Helper() + nmHome := makeSocketSafeTempDir(t) + t.Setenv("NM_HOME", nmHome) + + root := t.TempDir() + local := filepath.Join(root, "operator") + cliGit(t, root, "init", "-b", "main", local) + cliGit(t, local, "config", "user.name", "Test") + cliGit(t, local, "config", "user.email", "test@example.com") + if err := os.WriteFile(filepath.Join(local, "file.txt"), []byte("base\n"), 0o644); err != nil { + t.Fatal(err) + } + cliGit(t, local, "add", "file.txt") + cliGit(t, local, "commit", "-m", "base") + cliGit(t, local, "checkout", "-b", "feature/wedged") + if err := os.WriteFile(filepath.Join(local, "file.txt"), []byte("feature\n"), 0o644); err != nil { + t.Fatal(err) + } + cliGit(t, local, "commit", "-am", "feature") + submitted := cliGit(t, local, "rev-parse", "HEAD") + + p, err := paths.New() + if err != nil { + t.Fatal(err) + } + if err := p.EnsureDirs(); err != nil { + t.Fatal(err) + } + database, err := db.Open(p.DB()) + if err != nil { + t.Fatal(err) + } + registeredRoot, err := git.FindGitRoot(local) + if err != nil { + t.Fatal(err) + } + repo, err := database.InsertRepo(registeredRoot, filepath.Join(root, "remote.git"), "main") + if err != nil { + t.Fatal(err) + } + + gate := p.RepoDir(repo.ID) + cliGit(t, root, "init", "--bare", gate) + cliGit(t, local, "push", gate, "refs/heads/feature/wedged:refs/heads/feature/wedged") + // A later run pushed its own head onto the gate branch and was cancelled, + // so the gate no longer names this run's head either. + pipelineClone := filepath.Join(root, "later-run") + cliGit(t, root, "-c", "core.autocrlf=false", "clone", gate, pipelineClone) + cliGit(t, pipelineClone, "config", "user.name", "Test") + cliGit(t, pipelineClone, "config", "user.email", "test@example.com") + cliGit(t, pipelineClone, "checkout", "feature/wedged") + if err := os.WriteFile(filepath.Join(pipelineClone, "later.txt"), []byte("later run\n"), 0o644); err != nil { + t.Fatal(err) + } + cliGit(t, pipelineClone, "add", "later.txt") + cliGit(t, pipelineClone, "commit", "-m", "no-mistakes(review): later run fix") + cliGit(t, pipelineClone, "push", "origin", "HEAD:refs/heads/feature/wedged") + + run, err := database.InsertRun(repo.ID, "feature/wedged", submitted, submitted) + if err != nil { + t.Fatal(err) + } + // The recorded pipeline head is gone from every reachable object store. + if err := database.UpdateRunStatusWithVerifiedHead(run.ID, types.RunFailed, strings.Repeat("a", 40)); err != nil { + t.Fatal(err) + } + if err := database.Close(); err != nil { + t.Fatal(err) + } + + chdir(t, local) + return run.ID, p, local +} + +func assertNamesKeepLocalSettlement(t *testing.T, out string) { + t.Helper() + for _, want := range []string{"aborted: false", "run_status: failed", "already terminal"} { + if !strings.Contains(out, want) { + t.Errorf("terminal-run abort output missing %q:\n%s", want, out) + } + } + // The prescribed action is the FIRST help entry; the standing branch-sync + // guidance that follows it names `--recover` in general prose, so only the + // prescribed entry can be asserted. On this record the plain recovery is + // the command that always refuses, and it must never be prescribed here. + prescribed := "" + for _, line := range strings.Split(out, "\n") { + if trimmed := strings.TrimSpace(line); strings.HasPrefix(trimmed, "help[") { + _, rest, _ := strings.Cut(trimmed, ": ") + prescribed, _, _ = strings.Cut(rest, ",") + break + } + } + if prescribed != "Run `no-mistakes axi sync --recover --keep-local`" { + t.Errorf("terminal-run abort prescribed %q, want the keep-local settlement:\n%s", prescribed, out) + } +} + +// TestAxiAbortOfTerminalRunNamesTheSupportedCustodySettlement is the issue #824 +// regression on the abort surface. Aborting an already-terminal run is an +// idempotent no-op by design - there is nothing left to cancel - but the +// reporter was left with that no-op plus a `sync --check` that kept offering a +// recovery which always refused. The no-op must therefore name the command +// that can actually settle the record. +func TestAxiAbortOfTerminalRunNamesTheSupportedCustodySettlement(t *testing.T) { + t.Run("daemon unavailable", func(t *testing.T) { + runID, _, _ := wedgedCustodyAbortFixture(t) + + out, err := executeCmd("axi", "abort", "--run", runID) + t.Logf("daemon-down terminal abort output:\n%s", out) + if err != nil { + t.Fatalf("terminal run must resolve idempotently: %v\n%s", err, out) + } + assertNamesKeepLocalSettlement(t, out) + }) + + t.Run("daemon reports no active run", func(t *testing.T) { + runID, p, _ := wedgedCustodyAbortFixture(t) + startInactiveAbortDaemon(t, p, runID) + + out, err := executeCmd("axi", "abort", "--run", runID) + t.Logf("daemon-up terminal abort output:\n%s", out) + if err != nil { + t.Fatalf("terminal run must resolve idempotently: %v\n%s", err, out) + } + assertNamesKeepLocalSettlement(t, out) + }) +} + +// startInactiveAbortDaemon serves the exact daemon responses a terminal run +// produces: cancel_run has nothing active to cancel, and get_run reports the +// durable terminal record. +func startInactiveAbortDaemon(t *testing.T, p *paths.Paths, runID string) { + t.Helper() + srv := ipc.NewServer() + srv.Handle(ipc.MethodHealth, func(context.Context, json.RawMessage) (interface{}, error) { + return &ipc.HealthResult{Status: "ok"}, nil + }) + srv.Handle(ipc.MethodGateContext, func(context.Context, json.RawMessage) (interface{}, error) { + return &ipc.GateContextResult{Nested: false}, nil + }) + srv.Handle(ipc.MethodCancelRun, func(context.Context, json.RawMessage) (interface{}, error) { + return nil, noActiveRunErr(runID) + }) + srv.Handle(ipc.MethodGetRun, func(context.Context, json.RawMessage) (interface{}, error) { + return &ipc.GetRunResult{Run: &ipc.RunInfo{ + ID: runID, Branch: "feature/wedged", Status: types.RunFailed, + }}, nil + }) + errCh := make(chan error, 1) + go func() { errCh <- srv.Serve(p.Socket()) }() + t.Cleanup(func() { + srv.Close() + select { + case <-errCh: + case <-time.After(time.Second): + t.Error("fake daemon did not stop") + } + }) + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if client, dialErr := ipc.Dial(p.Socket()); dialErr == nil { + client.Close() + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatal("fake daemon did not become reachable") +} + +// TestBareAbortNoOpNeverPrescribesLaunchingAPipeline is the review regression +// for the branch-scoped abort no-op. The `--run` sites already guard their +// help behind an exact run match, but the bare-abort site emitted the branch's +// next action unconditionally - so on a branch whose custody was already +// returned it answered an abort by prescribing `axi run`, telling the operator +// to LAUNCH a pipeline. Abort help exists to name a custody settlement (issue +// #824 constraint 2); it must never prescribe starting a run. +func TestBareAbortNoOpNeverPrescribesLaunchingAPipeline(t *testing.T) { + runID, p, _ := wedgedCustodyAbortFixture(t) + // Custody already returned: the branch's own next action becomes + // run_pipeline, which an abort response must not hand back. + database, err := db.Open(p.DB()) + if err != nil { + t.Fatal(err) + } + if err := database.SetRunCustodyReturned(runID); err != nil { + t.Fatal(err) + } + if err := database.Close(); err != nil { + t.Fatal(err) + } + startNoActiveRunDaemon(t, p) + + out, err := executeCmd("axi", "abort") + t.Logf("bare abort on a released branch:\n%s", out) + if err != nil { + t.Fatalf("bare abort with no active run must be a no-op success: %v\n%s", err, out) + } + if !strings.Contains(out, "aborted: false") { + t.Errorf("bare abort no-op output missing %q:\n%s", "aborted: false", out) + } + // The structured branch_sync object still REPORTS the branch's own + // next_action (run_pipeline here); that is ownership state, not a + // prescription. Only the abort's own help prescribes a command, and a + // non-settlement action must yield no help at all. + if strings.Contains(out, "Run `no-mistakes axi run") { + t.Errorf("abort prescribed launching a pipeline:\n%s", out) + } + for _, line := range strings.Split(out, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "help[") { + t.Errorf("abort emitted help for a non-settlement next action:\n%s", line) + } + } +} + +// startNoActiveRunDaemon serves a daemon with no active run for the branch, so +// the bare abort takes its documented idempotent no-op path. +func startNoActiveRunDaemon(t *testing.T, p *paths.Paths) { + t.Helper() + srv := ipc.NewServer() + srv.Handle(ipc.MethodHealth, func(context.Context, json.RawMessage) (interface{}, error) { + return &ipc.HealthResult{Status: "ok"}, nil + }) + srv.Handle(ipc.MethodGateContext, func(context.Context, json.RawMessage) (interface{}, error) { + return &ipc.GateContextResult{Nested: false}, nil + }) + srv.Handle(ipc.MethodGetActiveRun, func(context.Context, json.RawMessage) (interface{}, error) { + return &ipc.GetActiveRunResult{Run: nil}, nil + }) + errCh := make(chan error, 1) + go func() { errCh <- srv.Serve(p.Socket()) }() + t.Cleanup(func() { + srv.Close() + select { + case <-errCh: + case <-time.After(time.Second): + t.Error("fake daemon did not stop") + } + }) + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if client, dialErr := ipc.Dial(p.Socket()); dialErr == nil { + client.Close() + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatal("fake daemon did not become reachable") +} + +// TestBareAbortNoOpEmitsNoHelpForOrdinaryDivergence pins the custody scope of +// the abort help. inspect_and_reconcile_manually is not custody-specific - +// classifyRelation emits it for ordinary divergence with a `git log` command - +// so a bare abort on a branch no run holds must stay silent rather than answer +// with unrelated reconciliation advice. +func TestBareAbortNoOpEmitsNoHelpForOrdinaryDivergence(t *testing.T) { + runID, p, local := wedgedCustodyAbortFixture(t) + root := filepath.Dir(local) + + // A real pipeline push binding, then local work that conflicts with it, so + // classification lands on ordinary divergence rather than pipeline custody. + cliGit(t, local, "checkout", "-b", "pipeline-pushed") + if err := os.WriteFile(filepath.Join(local, "file.txt"), []byte("pipeline rewrite\n"), 0o644); err != nil { + t.Fatal(err) + } + cliGit(t, local, "commit", "-am", "pipeline rewrite") + pushed := cliGit(t, local, "rev-parse", "HEAD") + cliGit(t, local, "checkout", "feature/wedged") + if err := os.WriteFile(filepath.Join(local, "file.txt"), []byte("local rewrite\n"), 0o644); err != nil { + t.Fatal(err) + } + cliGit(t, local, "commit", "-am", "local rewrite") + + database, err := db.Open(p.DB()) + if err != nil { + t.Fatal(err) + } + if err := database.UpdateRunPushBinding(runID, db.PushBinding{ + HeadSHA: pushed, + TargetKind: "upstream", + TargetFingerprint: branchsync.TargetFingerprint(filepath.Join(root, "remote.git")), + Ref: "refs/heads/feature/wedged", + }); err != nil { + t.Fatal(err) + } + if err := database.SetRunCustodyReturned(runID); err != nil { + t.Fatal(err) + } + if err := database.Close(); err != nil { + t.Fatal(err) + } + startNoActiveRunDaemon(t, p) + + out, err := executeCmd("axi", "abort") + t.Logf("bare abort on a diverged, unheld branch:\n%s", out) + if err != nil { + t.Fatalf("bare abort with no active run must be a no-op success: %v\n%s", err, out) + } + if !strings.Contains(out, "aborted: false") { + t.Errorf("bare abort no-op output missing %q:\n%s", "aborted: false", out) + } + if !strings.Contains(out, "safety: blocked_diverged") { + t.Fatalf("fixture did not reach ordinary divergence:\n%s", out) + } + // The structured branch_sync object still REPORTS the branch's own + // next_action; that is ownership state, not a prescription. Only the + // abort's own help prescribes, and here it must prescribe nothing. + if !strings.Contains(out, "code: inspect_and_reconcile_manually") { + t.Errorf("branch_sync stopped reporting the branch's own next action:\n%s", out) + } + for _, line := range strings.Split(out, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "help[") { + t.Errorf("abort emitted help for a branch no run holds:\n%s", line) + } + if strings.HasPrefix(strings.TrimSpace(line), "Run `git log") { + t.Errorf("abort prescribed unrelated divergence advice:\n%s", line) + } + } +} + +// TestRunScopedAbortNoOpEmitsNoHelpForOrdinaryDivergence is the sibling of +// TestBareAbortNoOpEmitsNoHelpForOrdinaryDivergence for the two `--run ` +// abort sites. Their only guard is state.Pipeline.RunID != runID, which proves +// the branch RESOLVES to that run - not that a run still holds it - so a +// diverged, already-released branch still reached custodySettlementHelp and +// answered an abort with `git log` reconciliation advice the bare site had +// already been taught not to emit. +func TestRunScopedAbortNoOpEmitsNoHelpForOrdinaryDivergence(t *testing.T) { + runID, p, _ := divergedReleasedBranchFixture(t) + assertDivergedReleasedFixtureInvariants(t, p, runID) + + t.Run("daemon unavailable", func(t *testing.T) { + out, err := executeCmd("axi", "abort", "--run", runID) + t.Logf("daemon-down --run abort on a diverged, unheld branch:\n%s", out) + if err != nil { + t.Fatalf("terminal run must resolve idempotently: %v\n%s", err, out) + } + assertNoAbortHelpEmitted(t, out) + }) +} + +// TestRunScopedAbortNoOpEmitsNoHelpForOrdinaryDivergenceWithDaemon covers the +// same guard on the daemon-up resolveInactiveAbortTruth path. +func TestRunScopedAbortNoOpEmitsNoHelpForOrdinaryDivergenceWithDaemon(t *testing.T) { + runID, p, _ := divergedReleasedBranchFixture(t) + assertDivergedReleasedFixtureInvariants(t, p, runID) + startInactiveAbortDaemon(t, p, runID) + + out, err := executeCmd("axi", "abort", "--run", runID) + t.Logf("daemon-up --run abort on a diverged, unheld branch:\n%s", out) + if err != nil { + t.Fatalf("terminal run must resolve idempotently: %v\n%s", err, out) + } + assertNoAbortHelpEmitted(t, out) +} + +// divergedReleasedBranchFixture builds a branch whose run pushed successfully +// and whose custody was already returned, then diverges it locally, so +// classification lands on ordinary divergence rather than pipeline custody +// while the branch still resolves to that run. +func divergedReleasedBranchFixture(t *testing.T) (string, *paths.Paths, string) { + t.Helper() + runID, p, local := wedgedCustodyAbortFixture(t) + root := filepath.Dir(local) + + cliGit(t, local, "checkout", "-b", "pipeline-pushed") + if err := os.WriteFile(filepath.Join(local, "file.txt"), []byte("pipeline rewrite\n"), 0o644); err != nil { + t.Fatal(err) + } + cliGit(t, local, "commit", "-am", "pipeline rewrite") + pushed := cliGit(t, local, "rev-parse", "HEAD") + cliGit(t, local, "checkout", "feature/wedged") + if err := os.WriteFile(filepath.Join(local, "file.txt"), []byte("local rewrite\n"), 0o644); err != nil { + t.Fatal(err) + } + cliGit(t, local, "commit", "-am", "local rewrite") + + database, err := db.Open(p.DB()) + if err != nil { + t.Fatal(err) + } + if err := database.UpdateRunPushBinding(runID, db.PushBinding{ + HeadSHA: pushed, + TargetKind: "upstream", + TargetFingerprint: branchsync.TargetFingerprint(filepath.Join(root, "remote.git")), + Ref: "refs/heads/feature/wedged", + }); err != nil { + t.Fatal(err) + } + if err := database.SetRunCustodyReturned(runID); err != nil { + t.Fatal(err) + } + if err := database.Close(); err != nil { + t.Fatal(err) + } + return runID, p, local +} + +// assertDivergedReleasedFixtureInvariants proves what the `--run` abort sites +// cannot show from their own output. Their guard is +// `custodySettlementHelp(state.NextAction) != empty AND state == +// StatePipelineOwned`, but a `--run` response carries no branch_sync object at +// all, so "no help was printed" is equally true when the fixture never reached +// divergence, or when the branch carries no next action there was anything to +// suppress. Those are exactly the two invariants the bare-abort sibling +// asserts inline and these two sites dropped, leaving them unable to fail if +// the StatePipelineOwned clause were deleted. +// +// It reads the classification the abort sites themselves consult - the same +// branchsync.InspectCached over the same database, repository, and gate that +// terminalRunCustodyHelpWithDB builds - rather than going through a CLI +// surface. The bare `axi abort` renders it too, but that path ensures a live +// daemon, which this package cannot start, so reading the service directly is +// what keeps the invariant observable with or without a fake daemon. +func assertDivergedReleasedFixtureInvariants(t *testing.T, p *paths.Paths, runID string) { + t.Helper() + database, err := db.Open(p.DB()) + if err != nil { + t.Fatalf("reading the fixture classification must not fail: %v", err) + } + defer database.Close() + repo, err := findRepo(database) + if err != nil || repo == nil { + t.Fatalf("reading the fixture classification must not fail: repo=%#v err=%v", repo, err) + } + service := &branchsync.Service{ + DB: database, + Repo: repo, + WorkDir: ".", + GateDir: p.RepoDir(repo.ID), + Paths: p, + } + state := service.InspectCached(context.Background()) + if state.State != branchsync.StateDiverged || state.Safety != "blocked_diverged" { + t.Fatalf("fixture did not reach ordinary divergence, so suppression proves nothing: state=%q safety=%q", state.State, state.Safety) + } + // The run guard is the other half of the gate under test; if the branch + // stopped resolving to this run, removing the pipeline_owned clause would + // suppress the help anyway and the tests below could not fail. + if state.Pipeline.RunID != runID { + t.Fatalf("fixture branch resolves to run %q, want %q, so the pipeline_owned clause is not what suppresses the help", state.Pipeline.RunID, runID) + } + // The branch's own next action is the value custodySettlementHelp would + // have turned into help, so its presence is what proves the + // pipeline_owned half of the guard is doing the suppressing. + if state.NextAction == nil || state.NextAction.Code != "inspect_and_reconcile_manually" { + t.Fatalf("fixture branch carries no next action for the guard to suppress: %#v", state.NextAction) + } +} + +func assertNoAbortHelpEmitted(t *testing.T, out string) { + t.Helper() + if !strings.Contains(out, "aborted: false") { + t.Errorf("abort no-op output missing %q:\n%s", "aborted: false", out) + } + for _, line := range strings.Split(out, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "help[") { + t.Errorf("abort emitted help for a branch no run holds:\n%s", line) + } + if strings.HasPrefix(strings.TrimSpace(line), "Run `git log") { + t.Errorf("abort prescribed unrelated divergence advice:\n%s", line) + } + } +} diff --git a/internal/cli/axi_drive.go b/internal/cli/axi_drive.go index 0805b0708..c2cdabfc1 100644 --- a/internal/cli/axi_drive.go +++ b/internal/cli/axi_drive.go @@ -264,6 +264,80 @@ func inspectAxiBranchSync(ctx context.Context, env *axiEnv) branchsync.State { return service.InspectCached(ctx) } +// terminalRunCustodyHelp names the supported settlement for a branch that an +// already-terminal run still holds. Cancelling a terminal run is necessarily +// an idempotent no-op - there is nothing left to cancel - but that no-op used +// to be the whole answer, leaving the operator with a branch whose only +// advertised next action was a recovery that always refused (issue #824). The +// abort response therefore carries whatever settlement the branch's own +// ownership state names, so every terminal-run abort hands back a command that +// can complete. +// +// Best effort by construction: an `--run ` abort may be issued from +// outside the run's worktree entirely, so any failure to resolve the invoking +// repo, branch, or ownership simply adds no guidance. +func terminalRunCustodyHelp(ctx context.Context, p *paths.Paths, runID string) []string { + if p == nil { + return nil + } + d, err := db.Open(p.DB()) + if err != nil { + return nil + } + defer d.Close() + return terminalRunCustodyHelpWithDB(ctx, p, d, runID) +} + +// custodySettlementHelp renders abort help for a branch-ownership next action, +// but ONLY for the custody-settlement actions an abort response is entitled to +// prescribe. Abort help exists to name how to settle custody the pipeline +// still holds (issue #824); a released branch's ordinary next action is +// run_pipeline, and answering an abort by telling the operator to LAUNCH a +// fresh run exceeds that scope entirely. The allowlist is deliberate: an +// unrecognized or pipeline-launching action yields no help rather than being +// passed through. +func custodySettlementHelp(action *branchsync.NextAction) []string { + if action == nil { + return nil + } + switch action.Code { + case "recover_custody", "return_custody_keep_local", "inspect_and_reconcile_manually": + return []string{ + "Run `" + action.Command + "`", + branchSyncAgentGuidance, + } + default: + return nil + } +} + +func terminalRunCustodyHelpWithDB(ctx context.Context, p *paths.Paths, d *db.DB, runID string) []string { + if p == nil || d == nil || strings.TrimSpace(runID) == "" { + return nil + } + repo, err := findRepo(d) + if err != nil || repo == nil { + return nil + } + service := &branchsync.Service{ + DB: d, + Repo: repo, + WorkDir: ".", + GateDir: p.RepoDir(repo.ID), + Paths: p, + } + state := service.InspectCached(ctx) + // Resolving to this run is not the same as being HELD by it: a released + // branch still resolves to its last run, and inspect_and_reconcile_manually + // is not custody-specific (classifyRelation emits it for ordinary + // divergence with a `git log` command). Both conditions are required, which + // is the same gate the bare-abort site applies. + if state.Pipeline.RunID != runID || state.State != branchsync.StatePipelineOwned { + return nil + } + return custodySettlementHelp(state.NextAction) +} + func freshRunBranchOwnershipState(ctx context.Context, env *axiEnv) *branchsync.State { state := inspectAxiBranchSync(ctx, env) switch state.State { @@ -893,9 +967,19 @@ func runAxiAbort(cmd *cobra.Command, runID string) error { {Key: "aborted", Value: false}, {Key: "detail", Value: "no active run (no-op)"}, } - if state := inspectAxiBranchSync(ctx, env); relevantCachedSyncState(state) { + state := inspectAxiBranchSync(ctx, env) + if relevantCachedSyncState(state) { fields = append(fields, branchSyncField(state)) } + // The branch a terminal run still holds must leave with a command that + // can settle it, not just the no-op (issue #824) - but only a custody + // settlement, never an instruction to launch a fresh pipeline. The + // pipeline_owned gate is what makes that true: inspect_and_reconcile_manually + // is also emitted for ordinary divergence, where it is git-log advice + // rather than a custody settlement and belongs on no abort response. + if help := custodySettlementHelp(state.NextAction); len(help) > 0 && state.State == branchsync.StatePipelineOwned { + fields = append(fields, toon.Field{Key: "help", Value: help}) + } emitDoc(cmd, fields...) return nil } @@ -1070,7 +1154,7 @@ func runAxiAbortByRunID(cmd *cobra.Command, runID string) error { // terminal no-op, the documented unknown-id no-op, and the nonzero // terminal-unconfirmed contract. if strings.Contains(err.Error(), "no active run") { - return resolveInactiveAbortTruth(cmd, client, runID) + return resolveInactiveAbortTruth(cmd, p, client, runID) } return emitError(cmd, 1, fmt.Sprintf("abort run: %v", err)) } @@ -1106,7 +1190,7 @@ func runViewPtrFromIPC(run *ipc.RunInfo) *runView { // fabricated), a positively proven unknown id keeps the documented no-op, and // a still-nonterminal or unreadable run is the nonzero terminal-unconfirmed // contract. -func resolveInactiveAbortTruth(cmd *cobra.Command, client *ipc.Client, runID string) error { +func resolveInactiveAbortTruth(cmd *cobra.Command, p *paths.Paths, client *ipc.Client, runID string) error { ctx, cancel := context.WithTimeout(cmd.Context(), abortStateWaitTimeout) defer cancel() var result ipc.GetRunResult @@ -1132,12 +1216,16 @@ func resolveInactiveAbortTruth(cmd *cobra.Command, client *ipc.Client, runID str return emitUnconfirmedAbort(cmd, runID, "", fmt.Sprintf("the daemon returned durable state for run %s instead of the requested run %s", run.ID, runID), nil, true) } if terminalStatus(string(run.Status)) { - emitDoc(cmd, - toon.Field{Key: "aborted", Value: false}, - toon.Field{Key: "run", Value: runID}, - toon.Field{Key: "run_status", Value: string(run.Status)}, - toon.Field{Key: "detail", Value: "run is already terminal (idempotent no-op)"}, - ) + fields := []toon.Field{ + {Key: "aborted", Value: false}, + {Key: "run", Value: runID}, + {Key: "run_status", Value: string(run.Status)}, + {Key: "detail", Value: "run is already terminal (idempotent no-op)"}, + } + if help := terminalRunCustodyHelp(ctx, p, runID); len(help) > 0 { + fields = append(fields, toon.Field{Key: "help", Value: help}) + } + emitDoc(cmd, fields...) return nil } return emitUnconfirmedAbort(cmd, runID, run.Branch, fmt.Sprintf("the daemon reported no active run, but the exact run's durable state is still %s", run.Status), runViewPtrFromIPC(run), true) @@ -1171,12 +1259,16 @@ func resolveDaemonDownAbortTruth(cmd *cobra.Command, p *paths.Paths, runID strin return emitUnconfirmedAbort(cmd, runID, "", fmt.Sprintf("the durable record identified run %s instead of the requested run %s", run.ID, runID), nil, false) } if terminalStatus(string(run.Status)) { - emitDoc(cmd, - toon.Field{Key: "aborted", Value: false}, - toon.Field{Key: "run", Value: runID}, - toon.Field{Key: "run_status", Value: string(run.Status)}, - toon.Field{Key: "detail", Value: "daemon not running; run is already terminal (idempotent no-op)"}, - ) + fields := []toon.Field{ + {Key: "aborted", Value: false}, + {Key: "run", Value: runID}, + {Key: "run_status", Value: string(run.Status)}, + {Key: "detail", Value: "daemon not running; run is already terminal (idempotent no-op)"}, + } + if help := terminalRunCustodyHelpWithDB(cmd.Context(), p, database, runID); len(help) > 0 { + fields = append(fields, toon.Field{Key: "help", Value: help}) + } + emitDoc(cmd, fields...) return nil } return emitUnconfirmedAbort(cmd, runID, run.Branch, fmt.Sprintf("the daemon is not running, so cancellation cannot be requested, and the durable run record is still %s", run.Status), nil, false) diff --git a/internal/cli/axi_guidance.go b/internal/cli/axi_guidance.go index 178b913ba..b361894ef 100644 --- a/internal/cli/axi_guidance.go +++ b/internal/cli/axi_guidance.go @@ -26,4 +26,4 @@ const preserveGateFixCommitsGuidance = "Commit post-pipeline follow-up work on t // branchSyncAgentGuidance is emitted only when a relevant branch_sync object // is present. Keeping it conditional avoids flooding ordinary runs whose local // and pipeline heads never differed. -const branchSyncAgentGuidance = "Before a post-pipeline local commit or fresh run, follow the structured `branch_sync.next_action`. Run `no-mistakes axi sync` only when its code is `sync`; that guarded sync may be a strict fast-forward or a content-equivalent diverged advance that anchors the pre-sync head before moving the branch with reset semantics. Run `no-mistakes axi sync --recover` only when its code is `recover_custody` (a terminal run left unpublished pipeline commits preserved in the local gate). A `user_owned` state means cancellation released the branch before changing the submitted head: the exact branch and head are yours, immediately usable, and no sync action is needed. Process blocked or pipeline-owned states instead of improvising reset, stash, merge, rebase, force, or branch replacement." +const branchSyncAgentGuidance = "Before a post-pipeline local commit or fresh run, follow the structured `branch_sync.next_action`. Run `no-mistakes axi sync` only when its code is `sync`; that guarded sync may be a strict fast-forward or a content-equivalent diverged advance that anchors the pre-sync head before moving the branch with reset semantics. Run `no-mistakes axi sync --recover` only when its code is `recover_custody` (a terminal run left unpublished pipeline commits preserved in the local gate). Run `no-mistakes axi sync --recover --keep-local` only when its code is `return_custody_keep_local` (a terminal run whose recorded pipeline head can no longer be verified): it returns custody at your current head and points the gate branch at it. A `user_owned` state means cancellation released the branch before changing the submitted head: the exact branch and head are yours, immediately usable, and no sync action is needed. Process blocked or pipeline-owned states instead of improvising reset, stash, merge, rebase, force, or branch replacement." diff --git a/internal/cli/axi_guidance_test.go b/internal/cli/axi_guidance_test.go index 212e3be58..bb642c038 100644 --- a/internal/cli/axi_guidance_test.go +++ b/internal/cli/axi_guidance_test.go @@ -49,6 +49,13 @@ var canonicalBranchSyncPhrases = []string{ // that it needs no recovery. "user_owned", "before changing the submitted head", + // Settlement of a self-inconsistent custody record (issue #824): a record + // whose recorded pipeline head cannot be verified - absent from every + // object store, or contradicted by the run's own recovery evidence - must + // name the keep-local settlement that can actually complete instead of a + // recovery that always refuses. + "return_custody_keep_local", + "no-mistakes axi sync --recover --keep-local", } const canonicalPipelineAgentPrerequisite = "a supported native agent binary, the `agent: cursor` ACP alias, or an explicit `acp:` through `acpx`" diff --git a/internal/cli/axi_settle_custody_test.go b/internal/cli/axi_settle_custody_test.go new file mode 100644 index 000000000..c694e057c --- /dev/null +++ b/internal/cli/axi_settle_custody_test.go @@ -0,0 +1,128 @@ +package cli + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/kunchenguid/no-mistakes/internal/custody" + "github.com/kunchenguid/no-mistakes/internal/db" + "github.com/kunchenguid/no-mistakes/internal/paths" +) + +// TestWedgedCustodyRecordSettlesThroughTheCLI walks the whole issue #824 +// operator journey on the surface the operator actually types, rather than on +// the branchsync service the unit tests drive directly: a terminal run whose +// recorded pipeline head is in no object store leaves the branch +// pipeline_owned, and before this change every command the surfaces offered +// refused on that same unverifiable head, so the branch could never be taken +// back. +// +// The journey has to hold end to end, not just report a next action: the +// advertised command must be the one that completes, it must leave the gate +// branch at the head the operator kept, and a second read must show the branch +// released - which is the only proof the dead end is actually gone. +func TestWedgedCustodyRecordSettlesThroughTheCLI(t *testing.T) { + runID, p, local := wedgedCustodyAbortFixture(t) + gateDir, keptHead := wedgedFixtureGateAndHead(t, p, runID, local) + displacedGateHead := cliGit(t, gateDir, "rev-parse", "refs/heads/feature/wedged^{commit}") + if displacedGateHead == keptHead { + t.Fatalf("fixture must start with the gate branch off the kept head: %s", keptHead) + } + + blocked, err := executeCmd("axi", "sync", "--check") + t.Logf("STEP 1 - `no-mistakes axi sync --check` on the wedged record:\n%s", blocked) + if err == nil { + t.Fatalf("a wedged custody record must refuse the ordinary plan:\n%s", blocked) + } + for _, want := range []string{ + "state: pipeline_owned", + "code: return_custody_keep_local", + "command: no-mistakes axi sync --recover --keep-local", + "Run `no-mistakes axi sync --recover --keep-local`", + } { + if !strings.Contains(blocked, want) { + t.Errorf("wedged plan missing %q:\n%s", want, blocked) + } + } + + // The plain recovery is the command that can only refuse on this record, + // so the surface must not be quietly settling it under --recover alone. + refused, err := executeCmd("axi", "sync", "--recover") + t.Logf("STEP 2 - `no-mistakes axi sync --recover` (no --keep-local) still refuses:\n%s", refused) + if err == nil { + t.Fatalf("the plain recovery must refuse a record with no verifiable head:\n%s", refused) + } + if strings.Contains(refused, "state: custody_returned") { + t.Errorf("the plain recovery settled a wedged record:\n%s", refused) + } + if cliGit(t, gateDir, "rev-parse", "refs/heads/feature/wedged^{commit}") != displacedGateHead { + t.Error("a refused recovery moved the gate branch") + } + + settled, err := executeCmd("axi", "sync", "--recover", "--keep-local") + t.Logf("STEP 3 - the advertised `no-mistakes axi sync --recover --keep-local` completes:\n%s", settled) + if err != nil { + t.Fatalf("the advertised settlement must complete: %v\n%s", err, settled) + } + for _, want := range []string{"state: custody_returned", "recovered: true", "safety: custody_returned"} { + if !strings.Contains(settled, want) { + t.Errorf("settlement output missing %q:\n%s", want, settled) + } + } + + // The worktree is never touched, and the gate branch follows the head the + // operator kept - both are the settlement's stated contract. + if head := cliGit(t, local, "rev-parse", "HEAD"); head != keptHead { + t.Errorf("settlement moved the worktree HEAD: %s -> %s", keptHead, head) + } + if gateHead := cliGit(t, gateDir, "rev-parse", "refs/heads/feature/wedged^{commit}"); gateHead != keptHead { + t.Errorf("gate branch = %s, want the kept local head %s", gateHead, keptHead) + } + // The displaced gate head is never dropped on the floor: it stays + // reachable through the run's gate anchor. + if anchored := cliGit(t, gateDir, "rev-parse", custody.RecoveryGateRef(runID)+"^{commit}"); anchored != displacedGateHead { + t.Errorf("displaced gate head anchor = %s, want %s", anchored, displacedGateHead) + } + + after, err := executeCmd("axi", "sync", "--check") + t.Logf("STEP 4 - `no-mistakes axi sync --check` after settling:\n%s", after) + if strings.Contains(after, "state: pipeline_owned") { + t.Errorf("the branch is still held by the terminal run after settling:\n%s", after) + } + + database, err := db.Open(p.DB()) + if err != nil { + t.Fatal(err) + } + defer database.Close() + run, err := database.GetRun(runID) + if err != nil { + t.Fatal(err) + } + if run.CustodyReturnedAt == nil { + t.Error("the custody return was not recorded on the run row") + } +} + +// wedgedFixtureGateAndHead resolves the fixture's real gate directory and the +// head the operator is keeping, so the journey asserts against live Git state +// rather than restating what the fixture built. +func wedgedFixtureGateAndHead(t *testing.T, p *paths.Paths, runID, local string) (string, string) { + t.Helper() + database, err := db.Open(p.DB()) + if err != nil { + t.Fatal(err) + } + defer database.Close() + run, err := database.GetRun(runID) + if err != nil { + t.Fatal(err) + } + gateDir := p.RepoDir(run.RepoID) + if _, err := os.Stat(filepath.Join(gateDir, "HEAD")); err != nil { + t.Fatalf("fixture gate is not a repository: %v", err) + } + return gateDir, cliGit(t, local, "rev-parse", "HEAD") +} diff --git a/internal/cli/sync.go b/internal/cli/sync.go index 2d4d1bc68..b495d1374 100644 --- a/internal/cli/sync.go +++ b/internal/cli/sync.go @@ -34,7 +34,10 @@ func newSyncCmd() *cobra.Command { "carry every local change. Unproven divergence refuses. A run cancelled before\n" + "the pipeline changed anything releases the branch by itself (user_owned) and\n" + "makes --recover a no-op. --recover --keep-local keeps the current local head\n" + - "instead and never touches the worktree.", + "instead and never touches the worktree; where the gate branch still names a\n" + + "different head it is compare-and-swapped onto the kept head, and where it does\n" + + "not, custody returns without moving any branch ref. That is also the settlement\n" + + "for a record whose preserved head can no longer be verified.", Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, args []string) error { if check && yes { @@ -55,7 +58,7 @@ func newSyncCmd() *cobra.Command { cmd.Flags().BoolVar(&check, "check", false, "freshly verify and show the synchronization plan without changing HEAD") cmd.Flags().BoolVarP(&yes, "yes", "y", false, "apply an eligible guarded synchronization without prompting") cmd.Flags().BoolVar(&recover, "recover", false, "return custody of a branch stranded by a terminal run with unpublished pipeline commits (a no-op when cancellation already released the branch)") - cmd.Flags().BoolVar(&keepLocal, "keep-local", false, "with --recover: keep the current local head; the preserved commits stay anchored and the gate follows the kept head") + cmd.Flags().BoolVar(&keepLocal, "keep-local", false, "with --recover: keep the current local head; the worktree is never touched, surviving preserved commits stay anchored, and where the gate branch still names a different head it compare-and-swaps onto the kept head. Also settles a record whose preserved head can no longer be verified") return cmd } @@ -72,7 +75,10 @@ func newAxiSyncCmd() *cobra.Command { "verified pipeline head with reset semantics.\n" + "--check performs the same fresh read-only plan. Blocked states change nothing.\n" + "--recover performs the guarded custody return offered by\n" + - "next_action.code: recover_custody; --keep-local keeps the current local head.", + "next_action.code: recover_custody; --keep-local keeps the current local head\n" + + "and, where the gate branch still names a different head, moves the gate branch\n" + + "to it, which is also the settlement offered by\n" + + "next_action.code: return_custody_keep_local.", Args: cobra.NoArgs, SilenceErrors: true, SilenceUsage: true, @@ -88,7 +94,7 @@ func newAxiSyncCmd() *cobra.Command { } cmd.Flags().BoolVar(&check, "check", false, "freshly verify and return the plan without changing HEAD") cmd.Flags().BoolVar(&recover, "recover", false, "return custody of a branch stranded by a terminal run with unpublished pipeline commits (a no-op when cancellation already released the branch)") - cmd.Flags().BoolVar(&keepLocal, "keep-local", false, "with --recover: keep the current local head; the preserved commits stay anchored and the gate follows the kept head") + cmd.Flags().BoolVar(&keepLocal, "keep-local", false, "with --recover: keep the current local head; the worktree is never touched, surviving preserved commits stay anchored, and where the gate branch still names a different head it compare-and-swaps onto the kept head; otherwise custody returns without moving any branch ref. Also settles a record whose preserved head can no longer be verified") return cmd } diff --git a/internal/custody/refs.go b/internal/custody/refs.go index 90db8bea6..4820335a5 100644 --- a/internal/custody/refs.go +++ b/internal/custody/refs.go @@ -53,3 +53,13 @@ func RecoveryLocalRef(runID string) string { func RecoveryGateRef(runID string) string { return "refs/no-mistakes/recover-gate/" + runID } + +// RecoveryStrandedRef keeps a terminal run's recorded pipeline head reachable +// when custody is settled at the operator's kept local head instead of at that +// head. It is deliberately separate from RecoveryRef: the settlement exists +// precisely because the run's own recovery evidence is unusable (missing, or +// naming another commit), so the surviving object needs an anchor that cannot +// collide with the evidence being preserved for inspection. +func RecoveryStrandedRef(runID string) string { + return "refs/no-mistakes/recover-stranded/" + runID +} diff --git a/internal/e2e/harness.go b/internal/e2e/harness.go index b26d0ddfa..d774b5a67 100644 --- a/internal/e2e/harness.go +++ b/internal/e2e/harness.go @@ -44,6 +44,7 @@ type Harness struct { agentName string // claude / codex / grok / opencode / antigravity allowRepoCommands *bool // mirrors SetupOpts.AllowRepoCommands + globalConfigExtra string // mirrors SetupOpts.GlobalConfigExtra daemonOwn *e2edaemon.Ownership } @@ -68,6 +69,12 @@ type SetupOpts struct { // (commands must come from the trusted default branch) pass a pointer // to false to exercise the secure default. AllowRepoCommands *bool + + // GlobalConfigExtra is appended verbatim to the generated global + // config.yaml. It exists for operator-only settings a test must exercise + // through the real loader - agent_timeout and review_agent_timeout, whose + // production defaults are half an hour, are the reason it was added. + GlobalConfigExtra string } const e2eDaemonStartTimeout = "45s" @@ -102,6 +109,7 @@ func NewHarness(t *testing.T, opts SetupOpts) *Harness { Scenario: opts.Scenario, agentName: opts.Agent, allowRepoCommands: opts.AllowRepoCommands, + globalConfigExtra: opts.GlobalConfigExtra, } for _, dir := range []string{h.BinDir, h.NMHome, h.HomeDir, h.WorkDir} { @@ -213,6 +221,9 @@ auto_fix: document: 0 ci: 0 `, h.agentName, h.agentName, binLink) + if extra := strings.TrimSpace(h.globalConfigExtra); extra != "" { + cfg += extra + "\n" + } if err := os.WriteFile(configPath, []byte(cfg), 0o644); err != nil { h.t.Fatalf("write config: %v", err) } diff --git a/internal/e2e/silent_agent_timeout_test.go b/internal/e2e/silent_agent_timeout_test.go new file mode 100644 index 000000000..737440ca8 --- /dev/null +++ b/internal/e2e/silent_agent_timeout_test.go @@ -0,0 +1,110 @@ +//go:build e2e + +package e2e + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// silentAgentScenario makes the review agent go completely silent: it produces +// no stdout, no stderr, and never responds, exactly like the native agent that +// burned two 30-minute budgets on one task without emitting a single byte. The +// hang is far longer than the budget the test configures, so the pipeline - not +// the fake - is what ends the invocation. +func silentAgentScenario(t *testing.T) string { + t.Helper() + path := filepath.Join(t.TempDir(), "silent-agent-scenario.yaml") + content := `actions: + - match: "Review the code changes and return structured findings" + delay_ms: 60000 + text: "never reached" + structured: + findings: [] + summary: "never reached" + risk_level: low + risk_rationale: "never reached" + - text: "no issues found" + structured: + findings: [] + summary: "no issues found" + risk_level: low + risk_rationale: "no risks detected in the diff" + tested: ["fakeagent: simulated test run"] + testing_summary: "simulated tests passed" + title: "feat: fakeagent change" + body: "## Summary\nfakeagent canned PR body" +` + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatalf("write silent agent scenario: %v", err) + } + return path +} + +// TestSilentAgentTimeoutReportsMeasuredEvidence reproduces the end-user failure +// through the stock `no-mistakes axi` surface: a pipeline agent that starts and +// then emits nothing at all until its invocation budget expires. +// +// What the operator used to get was "agent timed out after 30m0s (agent silent +// for 30m0s)" - the configured budget printed twice, with whatever the adapter +// reported discarded. Nothing in that line was measured, so a wedged agent, a +// busy one, and a crashed one were indistinguishable, and two silent 30-minute +// timeouts on one task produced no evidence to act on. +// +// The budgets here are seconds rather than the production 30 minutes; the +// timeout path is identical, and the test must never be "fixed" by raising them. +func TestSilentAgentTimeoutReportsMeasuredEvidence(t *testing.T) { + h := NewHarness(t, SetupOpts{ + Agent: "claude", + Scenario: silentAgentScenario(t), + GlobalConfigExtra: strings.Join([]string{ + `agent_timeout: "3s"`, + `review_agent_timeout: "3s"`, + }, "\n"), + }) + + h.CommitChange("init-silent", "seed.txt", "seed\n", "seed silent init") + initWorktree := h.AddWorktree("init-silent") + if out, err := h.RunInDir(initWorktree, "init"); err != nil { + t.Fatalf("init: %v\n%s", err, out) + } + + h.CommitChange("feature/silent-agent", "feature.txt", "value\n", "add feature") + operator := h.AddWorktree("feature/silent-agent") + + runOut, runErr := h.RunInDir(operator, "axi", "run", "--intent", "validate the feature while the agent is wedged") + if runErr == nil { + t.Fatalf("expected the wedged review agent to fail the run:\n%s", runOut) + } + + statusOut, _ := h.RunInDir(operator, "axi", "status") + // Keep the real operator surfaces visible under `go test -v` so this + // regression can also produce reviewer-visible evidence of the executable + // journey rather than only a pass/fail result. + t.Logf("stock axi run surface:\n%s", runOut) + t.Logf("stock axi status surface:\n%s", statusOut) + surfaces := runOut + "\n" + statusOut + + // The budget that expired must be named... + if !strings.Contains(surfaces, "timed out after 3s") { + t.Fatalf("axi surfaces did not name the expired budget:\n--- run ---\n%s\n--- status ---\n%s", runOut, statusOut) + } + // ...and the silence must be a measurement, not the budget restated. + if !strings.Contains(surfaces, "produced no output at all") { + t.Fatalf("axi surfaces did not report measured silence:\n--- run ---\n%s\n--- status ---\n%s", runOut, statusOut) + } + if strings.Contains(surfaces, "silent for 3s") { + t.Fatalf("axi surfaces restated the budget as if it were a measurement:\n--- run ---\n%s\n--- status ---\n%s", runOut, statusOut) + } + // The launched-but-mute subprocess is named, which is the fact that + // separates "the agent never started" from "the agent started and wedged". + if !strings.Contains(surfaces, "after its subprocess started") { + t.Fatalf("axi surfaces did not distinguish a launched agent from one that never ran:\n--- run ---\n%s\n--- status ---\n%s", runOut, statusOut) + } + // The adapter's own account of the killed process survives to the operator. + if !strings.Contains(surfaces, "agent reported:") { + t.Fatalf("axi surfaces discarded the adapter's report:\n--- run ---\n%s\n--- status ---\n%s", runOut, statusOut) + } +} diff --git a/internal/git/git.go b/internal/git/git.go index 91468dbe1..818bf3008 100644 --- a/internal/git/git.go +++ b/internal/git/git.go @@ -432,6 +432,46 @@ func IsDetachedHEAD(ctx context.Context, dir string) (bool, error) { return false, nil } +// CommitPresence reports whether dir's object store holds sha as a commit, +// and - unlike a bare boolean probe - separates a PROVEN absence from a store +// that could not answer. `git cat-file -e ` exits 1 only when the store +// was read and the object genuinely is not there; every other failure (an +// unreadable repository, a name Git cannot parse) exits 128. Peeling the +// argument with ^{commit} collapses both onto 128, so it must not be used +// where the difference is load bearing. +// +// A caller that only needs "can I use this object" should keep using the +// simple probe; this exists for the callers whose safety argument is "nothing +// still has this object, so nothing can be lost", which is a claim only exit 1 +// supports. For the same reason an object that is present but is not a commit +// is reported as an error rather than as false: the store still has it, so it +// is not an absence, and collapsing it into one would restore the hole. +func CommitPresence(ctx context.Context, dir, sha string) (bool, error) { + if strings.TrimSpace(sha) == "" { + return false, nil + } + if _, err := Run(ctx, dir, "cat-file", "-e", sha); err != nil { + var exitErr *exec.ExitError + if errors.As(err, &exitErr) && exitErr.ExitCode() == 1 { + return false, nil + } + return false, err + } + kind, err := Run(ctx, dir, "cat-file", "-t", sha) + if err != nil { + return false, err + } + if kind != "commit" { + // Present but the wrong type is NOT an absence - the store still has + // the object. Returning it as a plain false would send it back through + // the same door as a proven absence and re-open the very hole this + // function exists to close, so it is an error like every other answer + // that is not "the store was read and it is not there". + return false, fmt.Errorf("object %s is a %s, not a commit", sha, kind) + } + return true, nil +} + // DefaultBranch queries a remote to determine its default branch name. // Uses git ls-remote --symref to read the remote's HEAD symref. // Falls back to "main" if detection fails (e.g. empty remote, unreachable). diff --git a/internal/pipeline/agent_run.go b/internal/pipeline/agent_run.go index da31ec1de..d944bc178 100644 --- a/internal/pipeline/agent_run.go +++ b/internal/pipeline/agent_run.go @@ -4,10 +4,13 @@ import ( "context" "errors" "fmt" + "strings" + "sync" "time" "github.com/kunchenguid/no-mistakes/internal/agent" "github.com/kunchenguid/no-mistakes/internal/config" + "github.com/kunchenguid/no-mistakes/internal/safeurl" ) // ErrAgentTimeout is the context cause used when the default per-invocation @@ -59,7 +62,8 @@ func (sctx *StepContext) runAgent(parent context.Context, opts agent.RunOpts, se ag = sctx.Agent timeout = AgentTimeout(sctx.Config) } - return invokeAgent(parent, timeout, func(ctx context.Context) (*agent.Result, error) { + activity := observeAgentActivity(&opts) + return invokeAgent(parent, timeout, activity, func(ctx context.Context) (*agent.Result, error) { if sessionRole != "" && sctx != nil && sctx.Sessions != nil { return sctx.Sessions.Run(ctx, ag, sessionRole, opts, sctx.Log) } @@ -70,10 +74,10 @@ func (sctx *StepContext) runAgent(parent context.Context, opts agent.RunOpts, se }) } -func invokeAgent(parent context.Context, timeout time.Duration, run func(context.Context) (*agent.Result, error)) (*agent.Result, error) { +func invokeAgent(parent context.Context, timeout time.Duration, activity *agentActivity, run func(context.Context) (*agent.Result, error)) (*agent.Result, error) { ctx, cancel, applied := bindAgentDeadline(parent, timeout) result, err := run(ctx) - runErr := classifyAgentRun(ctx, applied, err) + runErr := classifyAgentRun(ctx, applied, activity, err) cancel() if runErr != nil { return nil, runErr @@ -81,6 +85,139 @@ func invokeAgent(parent context.Context, timeout time.Duration, run func(context return result, nil } +// agentActivity records when an in-flight invocation last produced anything +// observable: streamed assistant text or raw subprocess bytes +// (agent.LifecyclePhaseActivity). Lifecycle control metadata is not output. +// +// It exists because the timeout diagnostics used to assert that the agent had +// been "silent for " without ever measuring silence - the budget was +// simply printed twice. An operator reading that line cannot tell a wedged +// process from one that streamed until the last second, which is exactly the +// distinction that decides whether to re-run, raise the budget, or go look at +// the agent CLI. Everything reported now is measured. +type agentActivity struct { + mu sync.Mutex + // begun is when the current attempt was handed to the agent. + begun time.Time + // last is when output was most recently observed; zero when none ever was. + last time.Time + // observed counts output events. A subprocess launch is deliberately not + // one of them: launching proves the binary ran, not that it is doing + // anything, and counting it would erase the difference this whole + // measurement exists to expose. + observed int + // launchedPID is the native subprocess PID, when one was reported. + launchedPID int + launchedAt time.Time + launched bool +} + +func newAgentActivity() *agentActivity { + return &agentActivity{begun: time.Now()} +} + +func (a *agentActivity) observe() { + if a == nil { + return + } + a.mu.Lock() + a.observed++ + a.last = time.Now() + a.mu.Unlock() +} + +func (a *agentActivity) beginAttempt() { + if a == nil { + return + } + a.mu.Lock() + a.begun = time.Now() + a.last = time.Time{} + a.observed = 0 + a.launchedPID = 0 + a.launchedAt = time.Time{} + a.launched = false + a.mu.Unlock() +} + +func (a *agentActivity) observeLaunch(pid int) { + if a == nil { + return + } + a.mu.Lock() + a.launched = true + a.launchedPID = pid + a.launchedAt = time.Now() + a.mu.Unlock() +} + +// evidence renders what was actually observed, for the timeout message. +func (a *agentActivity) evidence() string { + if a == nil { + return "agent activity was not observed for this invocation" + } + a.mu.Lock() + observed, begun, last := a.observed, a.begun, a.last + launched, launchedAt, pid := a.launched, a.launchedAt, a.launchedPID + a.mu.Unlock() + if observed > 0 { + return fmt.Sprintf("agent last produced output %s ago (%d observed)", + roundActivity(time.Since(last)), observed) + } + if launched { + return fmt.Sprintf("agent produced no output at all in %s after its subprocess started (pid=%d)", + roundActivity(time.Since(launchedAt)), pid) + } + return fmt.Sprintf("agent produced no output at all in %s and never reported a subprocess start", + roundActivity(time.Since(begun))) +} + +func roundActivity(d time.Duration) time.Duration { + if d < time.Second { + return d.Round(time.Millisecond) + } + return d.Round(time.Second) +} + +// observeAgentActivity instruments opts so every streamed chunk and every +// native lifecycle event is recorded before it reaches the caller's callbacks. +// The wrappers are pure observers: they always forward. +func observeAgentActivity(opts *agent.RunOpts) *agentActivity { + activity := newAgentActivity() + onChunk := opts.OnChunk + opts.OnChunk = func(text string) { + activity.observe() + if onChunk != nil { + onChunk(text) + } + } + onLifecycle := opts.OnLifecycle + opts.OnLifecycle = func(event agent.LifecycleEvent) { + switch event.Phase { + case agent.LifecyclePhaseStart: + // Launching proves the binary ran, not that it is doing anything. + activity.observeLaunch(event.PID) + case agent.LifecyclePhaseActivity: + activity.observe() + case agent.LifecyclePhaseRetry, agent.LifecyclePhaseFallback: + activity.beginAttempt() + case agent.LifecyclePhaseExit: + // Exit is the deadline's own consequence: cancelling the context + // kills the subprocess and the adapter reports it. Counting that as + // agent output would make every timeout claim the agent was busy + // until the last instant, which is the fabricated-evidence problem + // this measurement replaces. + default: + // Unknown lifecycle phases are adapter control metadata, not evidence + // of assistant text or subprocess output. + } + if onLifecycle != nil { + onLifecycle(event) + } + } + return activity +} + func bindAgentDeadline(parent context.Context, timeout time.Duration) (context.Context, context.CancelFunc, time.Duration) { if parent == nil { parent = context.Background() @@ -95,17 +232,91 @@ func bindAgentDeadline(parent context.Context, timeout time.Duration) (context.C return ctx, cancel, timeout } -func classifyAgentRun(ctx context.Context, applied time.Duration, err error) error { - if applied > 0 && errors.Is(context.Cause(ctx), ErrAgentTimeout) { - return fmt.Errorf("agent timed out after %s (agent silent for %s): %w", applied, applied, ErrAgentTimeout) +func classifyAgentRun(ctx context.Context, applied time.Duration, activity *agentActivity, err error) error { + cause := context.Cause(ctx) + if cause == nil { + return err } - if cause := context.Cause(ctx); cause != nil { - return cause + // Only a deadline earns a diagnosis. A plain cancellation (operator abort, + // daemon shutdown) is already self-explanatory and must not be dressed up + // as an agent fault. + if errors.Is(ctx.Err(), context.DeadlineExceeded) { + if applied > 0 && errors.Is(cause, ErrAgentTimeout) { + return diagnoseAgentTimeout( + fmt.Sprintf("agent timed out after %s", applied), activity, err, cause) + } + // The budget belongs to the caller (a review round, a Test invocation). + // Keep its cause identity so the caller's own classifier still matches, + // and hand it the measurement plus whatever the adapter managed to say. + return diagnoseAgentTimeout("", activity, err, cause) } - if err != nil { - return err + return cause +} + +// diagnoseAgentTimeout builds the one error a timed-out invocation returns. It +// always carries the measured activity evidence and, crucially, whatever the +// adapter reported - a killed native agent's stderr and exit status is the only +// account of what the process was doing, and dropping it is what made this +// failure mode undiagnosable in the first place. +func diagnoseAgentTimeout(prefix string, activity *agentActivity, adapterErr, cause error) error { + parts := make([]string, 0, 3) + if prefix != "" { + parts = append(parts, prefix) + } + parts = append(parts, activity.evidence()) + if clause := agentReportClause(adapterErr); clause != "" { + parts = append(parts, clause) + } + return &agentInvocationError{ + message: strings.Join(parts, "; "), + cause: cause, + adapter: adapterErr, } - return nil +} + +// agentReportClause renders the adapter's own error for the timeout message. +// A nil error, or one that only restates the cancellation the deadline caused, +// adds nothing and is dropped. +func agentReportClause(err error) string { + if err == nil { + return "" + } + // A bare context error is the deadline we are already reporting, echoed back + // by the adapter. It adds no account of what the process was doing. + if err.Error() == context.DeadlineExceeded.Error() || err.Error() == context.Canceled.Error() { + return "" + } + text := safeurl.RedactText(strings.Join(strings.Fields(err.Error()), " ")) + if text == "" { + return "" + } + const max = 400 + if len([]rune(text)) > max { + text = string([]rune(text)[:max]) + "..." + } + return "agent reported: " + text +} + +// agentInvocationError carries both the deadline cause (so a step's own +// sentinel keeps matching) and the adapter's error (so the concrete failure +// stays matchable, not just quoted in the message). +type agentInvocationError struct { + message string + cause error + adapter error +} + +func (e *agentInvocationError) Error() string { return e.message } + +func (e *agentInvocationError) Unwrap() []error { + errs := make([]error, 0, 2) + if e.cause != nil { + errs = append(errs, e.cause) + } + if e.adapter != nil { + errs = append(errs, e.adapter) + } + return errs } // timeoutAgent is the executor backstop: every sctx.Agent.Run is bounded even @@ -121,7 +332,26 @@ func (a *timeoutAgent) Name() string { return a.inner.Name() } func (a *timeoutAgent) Close() error { return a.inner.Close() } func (a *timeoutAgent) Run(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { - return invokeAgent(ctx, a.timeout, func(runCtx context.Context) (*agent.Result, error) { + if _, bounded := ctx.Deadline(); bounded { + // An outer seam (RunAgent, a review round, a Test wrap) already owns the + // budget and the diagnosis for this invocation; re-diagnosing here would + // nest the same measurement inside itself. The backstop this wrapper + // exists for still holds: a result produced after the deadline is + // refused, so work from an expired turn can never reach a commit. + result, err := a.inner.Run(ctx, opts) + cause := context.Cause(ctx) + switch { + case cause == nil: + return result, err + case err != nil: + // The adapter's own account beats restating the cause. + return nil, err + default: + return nil, cause + } + } + activity := observeAgentActivity(&opts) + return invokeAgent(ctx, a.timeout, activity, func(runCtx context.Context) (*agent.Result, error) { return a.inner.Run(runCtx, opts) }) } diff --git a/internal/pipeline/agent_run_test.go b/internal/pipeline/agent_run_test.go index a5d122d4d..40f4dae83 100644 --- a/internal/pipeline/agent_run_test.go +++ b/internal/pipeline/agent_run_test.go @@ -184,3 +184,472 @@ func TestExecutor_DirectAgentRunIsDeadlineBounded(t *testing.T) { t.Fatalf("run error = %q, want timeout diagnostic", msg) } } + +// The next three tests pin the diagnostic contract that a silent-agent timeout +// has to satisfy for an operator to act on it. Before this contract the +// invocation-timeout error printed the configured budget twice ("agent timed +// out after 30m0s (agent silent for 30m0s)") and discarded whatever the adapter +// had reported, so a wedged agent, a busy one, and a crashed one all produced +// the same undiagnosable line. + +func TestRunAgent_TimeoutReportsMeasuredSilenceWhenTheAgentNeverEmits(t *testing.T) { + t.Parallel() + ag := &hangingAgent{ + name: "mute", + runFn: func(ctx context.Context, _ agent.RunOpts) (*agent.Result, error) { + <-ctx.Done() + return nil, ctx.Err() + }, + } + sctx := &StepContext{ + Ctx: context.Background(), + Agent: ag, + Config: &config.Config{AgentTimeout: 30 * time.Millisecond}, + } + + _, err := sctx.RunAgent(agent.RunOpts{Prompt: "work"}) + if err == nil || !errors.Is(err, ErrAgentTimeout) { + t.Fatalf("error = %v, want ErrAgentTimeout", err) + } + if !strings.Contains(err.Error(), "produced no output at all") { + t.Fatalf("error = %q, want the measured absence of output", err) + } + if strings.Contains(err.Error(), "silent for 30ms") { + t.Fatalf("error = %q, must not restate the budget as if it were a measurement", err) + } +} + +func TestRunAgent_TimeoutReportsRecentOutputWhenTheAgentWasStreaming(t *testing.T) { + t.Parallel() + ag := &hangingAgent{ + name: "busy", + runFn: func(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { + // A working agent: it streams right up to the deadline. This must + // never be described the same way as an agent that emitted nothing. + for { + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-time.After(2 * time.Millisecond): + opts.OnChunk("thinking\n") + } + } + }, + } + var logged strings.Builder + sctx := &StepContext{ + Ctx: context.Background(), + Agent: ag, + Config: &config.Config{AgentTimeout: 60 * time.Millisecond}, + } + + _, err := sctx.RunAgent(agent.RunOpts{ + Prompt: "work", + OnChunk: func(text string) { logged.WriteString(text) }, + }) + if err == nil || !errors.Is(err, ErrAgentTimeout) { + t.Fatalf("error = %v, want ErrAgentTimeout", err) + } + if !strings.Contains(err.Error(), "last produced output") { + t.Fatalf("error = %q, want the measured recency of the agent's last output", err) + } + if strings.Contains(err.Error(), "no output at all") { + t.Fatalf("error = %q, must not report silence for an agent that was streaming", err) + } + // Observation must stay an observation: the caller's own callback still runs. + if logged.Len() == 0 { + t.Fatal("streamed chunks did not reach the caller's OnChunk") + } +} + +func TestRunAgent_TimeoutPreservesWhatTheAdapterReported(t *testing.T) { + t.Parallel() + // A killed native agent's error is the only account of what its process was + // doing - it carries the exit status and the subprocess stderr. Dropping it + // is what left a real 30-minute failure with nothing to diagnose. + adapterErr := errors.New("pi exited: signal: killed: pi: provider authentication required") + ag := &hangingAgent{ + name: "native", + runFn: func(ctx context.Context, _ agent.RunOpts) (*agent.Result, error) { + <-ctx.Done() + return nil, adapterErr + }, + } + sctx := &StepContext{ + Ctx: context.Background(), + Agent: ag, + Config: &config.Config{AgentTimeout: 20 * time.Millisecond}, + } + + _, err := sctx.RunAgent(agent.RunOpts{Prompt: "work"}) + if err == nil { + t.Fatal("expected a timeout error") + } + if !strings.Contains(err.Error(), "provider authentication required") { + t.Fatalf("error = %q, want the adapter's own report preserved", err) + } + if !errors.Is(err, ErrAgentTimeout) { + t.Fatalf("error = %v, want ErrAgentTimeout to stay matchable", err) + } + if !errors.Is(err, adapterErr) { + t.Fatalf("error = %v, want the adapter error to stay matchable", err) + } +} + +func TestRunAgent_NativeSubprocessLivenessCountsAsObservedOutput(t *testing.T) { + t.Parallel() + // Adapters forward only assistant prose to OnChunk, so a long tool-using + // turn is prose-silent while the subprocess streams events. Subprocess + // liveness is what separates "working" from "wedged", so it has to count. + ag := &hangingAgent{ + name: "tooling", + runFn: func(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { + opts.OnLifecycle(agent.LifecycleEvent{Agent: "tooling", Phase: agent.LifecyclePhaseStart, PID: 4242}) + for { + select { + case <-ctx.Done(): + return nil, ctx.Err() + case <-time.After(2 * time.Millisecond): + opts.OnLifecycle(agent.LifecycleEvent{Agent: "tooling", Phase: agent.LifecyclePhaseActivity}) + } + } + }, + } + sctx := &StepContext{ + Ctx: context.Background(), + Agent: ag, + Config: &config.Config{AgentTimeout: 60 * time.Millisecond}, + } + + _, err := sctx.RunAgent(agent.RunOpts{ + Prompt: "work", + OnLifecycle: func(agent.LifecycleEvent) {}, + }) + if err == nil { + t.Fatal("expected a timeout error") + } + if !strings.Contains(err.Error(), "last produced output") { + t.Fatalf("error = %q, want subprocess liveness counted as observed output", err) + } +} + +func TestRunAgent_RetryMetadataIsNotObservedOutput(t *testing.T) { + t.Parallel() + // Retry is adapter control metadata, not evidence that the assistant or its + // subprocess produced output. A silent invocation that retries must retain + // the same diagnosis as any other launched-but-mute invocation. + ag := &hangingAgent{ + name: "retrying", + runFn: func(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { + opts.OnLifecycle(agent.LifecycleEvent{Agent: "retrying", Phase: agent.LifecyclePhaseStart, PID: 5150}) + opts.OnLifecycle(agent.LifecycleEvent{Agent: "retrying", Phase: agent.LifecyclePhaseRetry}) + <-ctx.Done() + return nil, ctx.Err() + }, + } + var retries int + sctx := &StepContext{ + Ctx: context.Background(), + Agent: ag, + Config: &config.Config{AgentTimeout: 20 * time.Millisecond}, + } + + _, err := sctx.RunAgent(agent.RunOpts{ + Prompt: "work", + OnLifecycle: func(event agent.LifecycleEvent) { + if event.Phase == agent.LifecyclePhaseRetry { + retries++ + } + }, + }) + if err == nil { + t.Fatal("expected a timeout error") + } + if !strings.Contains(err.Error(), "produced no output at all") { + t.Fatalf("error = %q, want retry-only invocation reported as silent", err) + } + if strings.Contains(err.Error(), "last produced output") { + t.Fatalf("error = %q, retry metadata must not count as output", err) + } + if retries != 1 { + t.Fatalf("forwarded retry events = %d, want 1", retries) + } +} + +func TestRunAgent_FallbackResetsPriorAttemptActivity(t *testing.T) { + t.Parallel() + first := &hangingAgent{ + name: "active", + runFn: func(_ context.Context, opts agent.RunOpts) (*agent.Result, error) { + opts.OnLifecycle(agent.LifecycleEvent{Agent: "active", Phase: agent.LifecyclePhaseStart, PID: 5151}) + opts.OnLifecycle(agent.LifecycleEvent{Agent: "active", Phase: agent.LifecyclePhaseActivity}) + return nil, errors.New("active exited: unavailable") + }, + } + second := &hangingAgent{ + name: "silent", + runFn: func(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { + opts.OnLifecycle(agent.LifecycleEvent{Agent: "silent", Phase: agent.LifecyclePhaseStart, PID: 6161}) + <-ctx.Done() + return nil, ctx.Err() + }, + } + sctx := &StepContext{ + Ctx: context.Background(), + Agent: agent.NewFallback([]agent.Agent{first, second}), + Config: &config.Config{AgentTimeout: 20 * time.Millisecond}, + } + + var lifecycleLog strings.Builder + _, err := sctx.RunAgent(agent.RunOpts{ + Prompt: "work", + OnLifecycle: func(event agent.LifecycleEvent) { + if event.Message != "" { + lifecycleLog.WriteString(event.Message) + } + }, + }) + if err == nil || !errors.Is(err, ErrAgentTimeout) { + t.Fatalf("error = %v, want ErrAgentTimeout", err) + } + if !strings.Contains(lifecycleLog.String(), "falling back to silent") { + t.Fatalf("lifecycle log = %q, want operator-visible fallback notice", lifecycleLog.String()) + } + if !strings.Contains(err.Error(), "produced no output at all") { + t.Fatalf("error = %q, want silent fallback agent reported as producing no output", err) + } + if strings.Contains(err.Error(), "last produced output") { + t.Fatalf("error = %q, prior attempt activity must not count as replacement output", err) + } + if !strings.Contains(err.Error(), "pid=6161") { + t.Fatalf("error = %q, want silence attributed to the replacement subprocess", err) + } +} + +func TestRunAgent_PromptFormatFallbackResetsPriorAttemptActivity(t *testing.T) { + t.Parallel() + ag := &hangingAgent{ + name: "format-fallback", + runFn: func(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { + opts.OnLifecycle(agent.LifecycleEvent{Agent: "format-fallback", Phase: agent.LifecyclePhaseStart, PID: 8181}) + opts.OnChunk("thinking before conflict") + opts.OnLifecycle(agent.LifecycleEvent{Agent: "format-fallback", Phase: agent.LifecyclePhaseFallback}) + opts.OnLifecycle(agent.LifecycleEvent{Agent: "format-fallback", Phase: agent.LifecyclePhaseStart, PID: 8282}) + <-ctx.Done() + return nil, ctx.Err() + }, + } + sctx := &StepContext{ + Ctx: context.Background(), + Agent: ag, + Config: &config.Config{AgentTimeout: 20 * time.Millisecond}, + } + + _, err := sctx.RunAgent(agent.RunOpts{Prompt: "work"}) + if err == nil { + t.Fatal("expected the prompt-only fallback to time out") + } + if !strings.Contains(err.Error(), "produced no output at all") { + t.Fatalf("error = %q, want prompt-only fallback silence", err) + } + if strings.Contains(err.Error(), "last produced output") { + t.Fatalf("error = %q, native-format output must not describe the prompt-only fallback", err) + } + if !strings.Contains(err.Error(), "pid=8282") { + t.Fatalf("error = %q, want silence attributed to the prompt-only subprocess", err) + } +} + +func TestRunAgent_FallbackWithoutLaunchMeasuresSilenceFromAttempt(t *testing.T) { + t.Parallel() + const firstAttemptDuration = 100 * time.Millisecond + var fallbackStarted time.Time + first := &hangingAgent{ + name: "unavailable", + runFn: func(_ context.Context, _ agent.RunOpts) (*agent.Result, error) { + time.Sleep(firstAttemptDuration) + return nil, errors.New("unavailable exited: before replacement") + }, + } + second := &hangingAgent{ + name: "never-launched", + runFn: func(ctx context.Context, _ agent.RunOpts) (*agent.Result, error) { + fallbackStarted = time.Now() + <-ctx.Done() + return nil, ctx.Err() + }, + } + sctx := &StepContext{ + Ctx: context.Background(), + Agent: agent.NewFallback([]agent.Agent{first, second}), + Config: &config.Config{AgentTimeout: 250 * time.Millisecond}, + } + + invocationStarted := time.Now() + _, err := sctx.RunAgent(agent.RunOpts{Prompt: "work"}) + if err == nil { + t.Fatal("expected a timeout error") + } + prefix := "produced no output at all in " + text := err.Error() + start := strings.Index(text, prefix) + if start < 0 || !strings.Contains(text, "never reported a subprocess start") { + t.Fatalf("error = %q, want measured pre-launch fallback silence", text) + } + durationText := strings.Fields(text[start+len(prefix):])[0] + reported, parseErr := time.ParseDuration(durationText) + if parseErr != nil { + t.Fatalf("parse reported duration %q: %v", durationText, parseErr) + } + fallbackElapsed := time.Since(fallbackStarted) + invocationElapsed := time.Since(invocationStarted) + if delta := fallbackElapsed - reported; delta < -20*time.Millisecond || delta > 20*time.Millisecond { + t.Fatalf("reported silence = %s, fallback elapsed = %s", reported, fallbackElapsed) + } + if reported >= invocationElapsed-firstAttemptDuration/2 { + t.Fatalf("reported silence = %s, invocation elapsed = %s; want first attempt excluded", reported, invocationElapsed) + } +} + +func TestRunAgent_SubprocessStartAloneIsNotObservedOutput(t *testing.T) { + t.Parallel() + // Launching proves the binary ran, not that it is doing anything. Counting + // the start event as output would erase the exact distinction this + // measurement exists to expose. + ag := &hangingAgent{ + name: "launched", + runFn: func(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { + opts.OnLifecycle(agent.LifecycleEvent{Agent: "launched", Phase: agent.LifecyclePhaseStart, PID: 777}) + <-ctx.Done() + return nil, ctx.Err() + }, + } + sctx := &StepContext{ + Ctx: context.Background(), + Agent: ag, + Config: &config.Config{AgentTimeout: 20 * time.Millisecond}, + } + + _, err := sctx.RunAgent(agent.RunOpts{ + Prompt: "work", + OnLifecycle: func(agent.LifecycleEvent) {}, + }) + if err == nil { + t.Fatal("expected a timeout error") + } + if !strings.Contains(err.Error(), "produced no output at all") { + t.Fatalf("error = %q, want a launched-but-mute agent still reported as silent", err) + } + if !strings.Contains(err.Error(), "pid=777") { + t.Fatalf("error = %q, want the launched subprocess identified", err) + } +} + +func TestRunAgent_LateSubprocessLaunchMeasuresSilenceFromLaunch(t *testing.T) { + t.Parallel() + var launchedAt time.Time + ag := &hangingAgent{ + name: "late-launch", + runFn: func(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { + time.Sleep(100 * time.Millisecond) + launchedAt = time.Now() + opts.OnLifecycle(agent.LifecycleEvent{Agent: "late-launch", Phase: agent.LifecyclePhaseStart, PID: 888}) + <-ctx.Done() + return nil, ctx.Err() + }, + } + sctx := &StepContext{ + Ctx: context.Background(), + Agent: ag, + Config: &config.Config{AgentTimeout: 250 * time.Millisecond}, + } + + invocationStarted := time.Now() + _, err := sctx.RunAgent(agent.RunOpts{ + Prompt: "work", + OnLifecycle: func(agent.LifecycleEvent) {}, + }) + if err == nil { + t.Fatal("expected a timeout error") + } + prefix := "produced no output at all in " + text := err.Error() + start := strings.Index(text, prefix) + if start < 0 { + t.Fatalf("error = %q, want measured subprocess silence", text) + } + durationText := strings.Fields(text[start+len(prefix):])[0] + reported, parseErr := time.ParseDuration(durationText) + if parseErr != nil { + t.Fatalf("parse reported duration %q: %v", durationText, parseErr) + } + processElapsed := time.Since(launchedAt) + invocationElapsed := time.Since(invocationStarted) + if delta := processElapsed - reported; delta < -20*time.Millisecond || delta > 20*time.Millisecond { + t.Fatalf("reported silence = %s, subprocess elapsed = %s", reported, processElapsed) + } + if reported >= invocationElapsed-50*time.Millisecond { + t.Fatalf("reported silence = %s, invocation elapsed = %s; want late launch excluded", reported, invocationElapsed) + } +} + +func TestRunAgent_OperatorCancellationIsNotDressedUpAsAnAgentFault(t *testing.T) { + t.Parallel() + parent, cancel := context.WithCancel(context.Background()) + ag := &hangingAgent{ + name: "cancelled", + runFn: func(ctx context.Context, _ agent.RunOpts) (*agent.Result, error) { + <-ctx.Done() + return nil, ctx.Err() + }, + } + sctx := &StepContext{ + Ctx: parent, + Agent: ag, + Config: &config.Config{AgentTimeout: time.Minute}, + } + go func() { + time.Sleep(20 * time.Millisecond) + cancel() + }() + + _, err := sctx.RunAgent(agent.RunOpts{Prompt: "work"}) + if !errors.Is(err, context.Canceled) { + t.Fatalf("error = %v, want context.Canceled", err) + } + if strings.Contains(err.Error(), "produced no output") { + t.Fatalf("error = %q, an abort must not be reported as an agent silence diagnosis", err) + } +} + +// TestExecutor_DirectAgentRunUnderACallerDeadlineRefusesLateWork pins the +// executor backstop for the one shape it exists to cover: a step that calls +// Agent.Run itself, under a context whose deadline someone else owns. The +// invocation must still not hand back work produced after that deadline. +func TestExecutor_DirectAgentRunUnderACallerDeadlineRefusesLateWork(t *testing.T) { + database, p, run, repo := setupTest(t) + ag := &hangingAgent{ + name: "late", + runFn: func(ctx context.Context, _ agent.RunOpts) (*agent.Result, error) { + <-ctx.Done() + return &agent.Result{Text: "produced after the deadline"}, nil + }, + } + step := &adaptiveCallStep{ + name: types.StepDocument, + fn: func(sctx *StepContext) (*StepOutcome, error) { + bounded, cancel := context.WithTimeout(sctx.Ctx, 20*time.Millisecond) + defer cancel() + result, err := sctx.Agent.Run(bounded, agent.RunOpts{Prompt: "work"}) + if err == nil { + t.Fatalf("agent result %#v accepted after its caller's deadline expired", result) + } + return nil, err + }, + } + cfg := &config.Config{AgentTimeout: time.Hour} + exec := NewExecutor(database, p, cfg, ag, []Step{step}, nil) + if err := exec.Execute(context.Background(), run, repo, t.TempDir()); err == nil { + t.Fatal("expected the expired caller deadline to fail the run") + } +} diff --git a/internal/pipeline/executor.go b/internal/pipeline/executor.go index 9bc46243d..1315beeb2 100644 --- a/internal/pipeline/executor.go +++ b/internal/pipeline/executor.go @@ -774,6 +774,15 @@ func (e *Executor) executeStep(ctx context.Context, step Step, sr *db.StepResult if dbErr := e.db.SetStepAgentActivity(sr.ID, text, nil); dbErr != nil { slog.Warn("failed to set step agent activity in db", "step", stepName, "error", dbErr) } + case agent.LifecyclePhaseActivity: + // Subprocess liveness, not narrative: record that the agent is still + // producing bytes so `axi status` can distinguish a working fix round + // from a wedged one, but never write it to the step log. A long turn + // emits these every few seconds and the log is what an operator reads. + if dbErr := e.db.TouchStepActivity(sr.ID, text); dbErr != nil { + slog.Warn("failed to touch step activity in db", "step", stepName, "error", dbErr) + } + return default: if dbErr := e.db.TouchStepActivity(sr.ID, text); dbErr != nil { slog.Warn("failed to touch step activity in db", "step", stepName, "error", dbErr) diff --git a/internal/pipeline/executor_logging_test.go b/internal/pipeline/executor_logging_test.go index aa98c7a5c..814aee8d7 100644 --- a/internal/pipeline/executor_logging_test.go +++ b/internal/pipeline/executor_logging_test.go @@ -457,3 +457,89 @@ func TestExecutor_LogFileMultipleSteps(t *testing.T) { t.Error("review log should not contain test message") } } + +// TestExecutor_SubprocessLivenessUpdatesActivityWithoutFloodingTheStepLog pins +// how the executor consumes an agent's subprocess-liveness signal. +// +// The signal has to reach step activity, because that is the only thing that +// tells `axi status` a long tool-using fix round is alive rather than wedged. +// It must NOT reach the step log: a real turn emits it every few seconds for +// half an hour, and the step log is what an operator actually reads. +func TestExecutor_SubprocessLivenessUpdatesActivityWithoutFloodingTheStepLog(t *testing.T) { + database, p, run, repo := setupTest(t) + workDir := t.TempDir() + + livenessAgent := &lifecycleEmittingAgent{ + events: []agent.LifecycleEvent{ + {Agent: "pi", Phase: agent.LifecyclePhaseStart, PID: 1234, Message: "pi started pid=1234"}, + {Agent: "pi", Phase: agent.LifecyclePhaseActivity, Message: "pi producing output"}, + {Agent: "pi", Phase: agent.LifecyclePhaseActivity, Message: "pi producing output"}, + }, + } + + var stepResultID string + step := &adaptiveCallStep{ + name: types.StepReview, + fn: func(sctx *StepContext) (*StepOutcome, error) { + stepResultID = sctx.StepResultID + if _, err := sctx.RunAgent(agent.RunOpts{Prompt: "review"}); err != nil { + t.Fatalf("run agent: %v", err) + } + got, err := database.GetStepResult(sctx.StepResultID) + if err != nil { + t.Fatalf("get step result: %v", err) + } + if got.LastActivity == nil || !strings.Contains(*got.LastActivity, "producing output") { + var activity string + if got.LastActivity != nil { + activity = *got.LastActivity + } + t.Fatalf("last_activity = %q, want the subprocess liveness signal recorded", activity) + } + return &StepOutcome{ExitCode: 0}, nil + }, + } + + exec := NewExecutor(database, p, nil, livenessAgent, []Step{step}, nil) + if err := exec.Execute(context.Background(), run, repo, workDir); err != nil { + t.Fatalf("execute: %v", err) + } + + sr, err := database.GetStepResult(stepResultID) + if err != nil { + t.Fatalf("get step result: %v", err) + } + if sr.LogPath == nil { + t.Fatal("expected a step log path") + } + logBytes, err := os.ReadFile(*sr.LogPath) + if err != nil { + t.Fatalf("read step log: %v", err) + } + log := string(logBytes) + if !strings.Contains(log, "pi started pid=1234") { + t.Fatalf("step log = %q, want the subprocess start still logged", log) + } + if strings.Contains(log, "producing output") { + t.Fatalf("step log = %q, liveness ticks must not be written to the step log", log) + } +} + +// lifecycleEmittingAgent replays a fixed lifecycle sequence, standing in for a +// native adapter without spawning a subprocess. +type lifecycleEmittingAgent struct { + events []agent.LifecycleEvent +} + +func (a *lifecycleEmittingAgent) Name() string { return "pi" } + +func (a *lifecycleEmittingAgent) Close() error { return nil } + +func (a *lifecycleEmittingAgent) Run(_ context.Context, opts agent.RunOpts) (*agent.Result, error) { + for _, event := range a.events { + if opts.OnLifecycle != nil { + opts.OnLifecycle(event) + } + } + return &agent.Result{Text: "done"}, nil +} diff --git a/internal/pipeline/sessions.go b/internal/pipeline/sessions.go index 6e50513f6..db4e63a1b 100644 --- a/internal/pipeline/sessions.go +++ b/internal/pipeline/sessions.go @@ -104,6 +104,13 @@ func (rs *RunSessions) Run(ctx context.Context, a agent.Agent, role SessionRole, opts.Session = &agent.SessionRef{} opts.SessionFallback = true opts.SessionFallbackReason = classifyFallbackReason(err) + if opts.OnLifecycle != nil { + opts.OnLifecycle(agent.LifecycleEvent{ + Agent: a.Name(), + Phase: agent.LifecyclePhaseFallback, + Message: fmt.Sprintf("%s session resume failed; starting a fresh %s session", a.Name(), role), + }) + } result, err = a.Run(ctx, opts) if err != nil { return nil, err diff --git a/internal/pipeline/steps/ci.go b/internal/pipeline/steps/ci.go index fb870ee19..a8b5da04e 100644 --- a/internal/pipeline/steps/ci.go +++ b/internal/pipeline/steps/ci.go @@ -515,6 +515,9 @@ func (s *CIStep) Execute(sctx *pipeline.StepContext) (*pipeline.StepOutcome, err sctx.Log(fmt.Sprintf("issues detected: %s - manual fix requested...", issueDesc)) previousHeadSHA := sctx.Run.HeadSHA changed, err := s.autoFixCI(sctx, host, pr, fixTargets, mergeConflict) + if outcome := ciFixAgentBudgetOutcome(sctx, issueDesc, err); outcome != nil { + return outcome, nil + } if err != nil { sctx.Log(fmt.Sprintf("warning: CI manual fix failed: %v", err)) } else if changed || sctx.Run.HeadSHA != previousHeadSHA { @@ -546,6 +549,9 @@ func (s *CIStep) Execute(sctx *pipeline.StepContext) (*pipeline.StepOutcome, err sctx.Log(fmt.Sprintf("issues detected: %s - auto-fixing (attempt %d/%d)...", issueDesc, s.ciFixAttempts, ciFixLimit)) previousHeadSHA := sctx.Run.HeadSHA changed, err := s.autoFixCI(sctx, host, pr, fixTargets, mergeConflict) + if outcome := ciFixAgentBudgetOutcome(sctx, issueDesc, err); outcome != nil { + return outcome, nil + } if err != nil { sctx.Log(fmt.Sprintf("warning: CI auto-fix failed: %v", err)) } else if changed || sctx.Run.HeadSHA != previousHeadSHA { diff --git a/internal/pipeline/steps/ci_autofix_test.go b/internal/pipeline/steps/ci_autofix_test.go index b6c9a16ce..00ba6e8e3 100644 --- a/internal/pipeline/steps/ci_autofix_test.go +++ b/internal/pipeline/steps/ci_autofix_test.go @@ -3,6 +3,7 @@ package steps import ( "context" "encoding/json" + "errors" "fmt" "os" "path/filepath" @@ -1234,3 +1235,153 @@ func TestFormatReviewComments_FramesAndBoundsUntrustedText(t *testing.T) { t.Fatalf("review comment prompt lacks truncation marker") } } + +// TestCIStep_FixAgentBudgetExhaustionParksForADecisionInsteadOfRetrying pins the +// bounded outcome for a CI auto-fix agent that burns its whole invocation +// budget without finishing. +// +// The failure this replaces: the timeout was downgraded to a step-log warning +// and the poll loop re-issued the identical request on the next tick, up to +// auto_fix.ci attempts. Each retry cost another full agent budget, produced no +// operator-visible signal outside the CI step log, and ended the run at +// ci_timeout hours later with nothing to act on. +func TestCIStep_FixAgentBudgetExhaustionParksForADecisionInsteadOfRetrying(t *testing.T) { + t.Parallel() + dir, baseSHA, headSHA := setupGitRepo(t) + + checksJSON := `[{"name":"greptile","state":"FAILURE","bucket":"fail"}]` + env := fakeCIGH(t, "OPEN", checksJSON) + + var invocations int + ag := &mockAgent{ + name: "wedged", + runFn: func(ctx context.Context, _ agent.RunOpts) (*agent.Result, error) { + invocations++ + <-ctx.Done() + return nil, errors.New("pi exited: unable to access 'https://operator:secret@example.com/owner/repo.git': denied") + }, + } + + prURL := "https://github.com/test/repo/pull/3195" + sctx := newTestContext(t, ag, dir, baseSHA, headSHA, config.Commands{}) + sctx.Env = env + sctx.Run.PRURL = &prURL + sctx.Config.CITimeout = 30 * time.Second + sctx.Config.AutoFix = config.AutoFix{CI: 10} + sctx.Config.AgentTimeout = 50 * time.Millisecond + + var logs []string + sctx.Log = func(s string) { logs = append(logs, s) } + + polls := 0 + step := &CIStep{ + waitForNextPoll: func(ctx context.Context, interval time.Duration) error { + polls++ + if polls > 3 { + t.Fatal("CI monitor kept polling after the fix agent exhausted its budget") + } + return nil + }, + } + + outcome, err := step.Execute(sctx) + if err != nil { + t.Fatalf("CI step returned error %v, want a parked decision that keeps the run alive", err) + } + if outcome == nil || !outcome.NeedsApproval { + t.Fatalf("outcome = %#v, want the step parked for a decision", outcome) + } + if invocations != 1 { + t.Fatalf("agent invocations = %d, want exactly one budget spent before asking", invocations) + } + + var findings Findings + if jsonErr := json.Unmarshal([]byte(outcome.Findings), &findings); jsonErr != nil { + t.Fatalf("parse findings %q: %v", outcome.Findings, jsonErr) + } + if len(findings.Items) != 1 { + t.Fatalf("findings = %#v, want one gate finding", findings.Items) + } + item := findings.Items[0] + if item.Action != types.ActionAskUser { + t.Fatalf("finding action = %q, want %q so the gate parks for a human decision", item.Action, types.ActionAskUser) + } + if !strings.Contains(item.Description, "greptile") { + t.Fatalf("finding %q, want the check it was repairing named", item.Description) + } + if !strings.Contains(item.Description, "produced no output at all") { + t.Fatalf("finding %q, want the measured silence carried into the gate", item.Description) + } + if strings.Contains(item.Description, "operator:secret") { + t.Fatalf("finding %q leaked adapter URL credentials", item.Description) + } + if !strings.Contains(item.Description, "https://redacted@example.com/owner/repo.git") { + t.Fatalf("finding %q, want the adapter URL preserved with credentials redacted", item.Description) + } +} + +// TestCIStep_NonTimeoutFixFailureKeepsRetrying is the counter-test: only a +// proven full-budget burn parks. An ordinary transient fix failure keeps its +// existing warn-and-retry behaviour, because repeating it is cheap and often +// works. +func TestCIStep_NonTimeoutFixFailureKeepsRetrying(t *testing.T) { + t.Parallel() + dir, baseSHA, headSHA := setupGitRepo(t) + + checksJSON := `[{"name":"greptile","state":"FAILURE","bucket":"fail"}]` + env := fakeCIGH(t, "OPEN", checksJSON) + + var invocations int + ag := &mockAgent{ + name: "flaky", + runFn: func(context.Context, agent.RunOpts) (*agent.Result, error) { + invocations++ + return nil, errors.New("transient provider hiccup") + }, + } + + prURL := "https://github.com/test/repo/pull/3195" + sctx := newTestContext(t, ag, dir, baseSHA, headSHA, config.Commands{}) + sctx.Env = env + sctx.Run.PRURL = &prURL + sctx.Config.CITimeout = 30 * time.Second + sctx.Config.AutoFix = config.AutoFix{CI: 10} + + var logs []string + sctx.Log = func(s string) { logs = append(logs, s) } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + sctx.Ctx = ctx + + polls := 0 + step := &CIStep{ + waitForNextPoll: func(ctx context.Context, interval time.Duration) error { + polls++ + if polls >= 2 { + cancel() + } + return ctx.Err() + }, + } + + outcome, _ := step.Execute(sctx) + if outcome != nil && outcome.NeedsApproval { + t.Fatalf("outcome = %#v, want a transient fix failure to keep retrying rather than park", outcome) + } + if invocations == 0 { + t.Fatal("expected the fix agent to be invoked") + } + warned := false + for _, l := range logs { + if strings.Contains(l, "CI auto-fix failed") { + warned = true + } + if strings.Contains(l, "exceeded its invocation budget") { + t.Fatalf("logs = %v, a transient failure must not be reported as a budget burn", logs) + } + } + if !warned { + t.Fatalf("logs = %v, want the transient failure still warned about", logs) + } +} diff --git a/internal/pipeline/steps/ci_checks.go b/internal/pipeline/steps/ci_checks.go index d62526d40..d9cf62302 100644 --- a/internal/pipeline/steps/ci_checks.go +++ b/internal/pipeline/steps/ci_checks.go @@ -224,6 +224,45 @@ func ciCheckReadFailureOutcome(err error) *pipeline.StepOutcome { } } +// ciFixAgentTimeoutOutcome parks the CI step for a decision after the auto-fix +// agent burned its whole invocation budget without finishing. +// +// The previous behaviour downgraded this to a log warning and let the poll loop +// issue the same request again on the next tick, up to auto_fix.ci attempts - +// each one another full agent budget, all of it invisible except for warning +// lines inside the CI step log, until ci_timeout ended the run hours later with +// nothing to act on. That is the same invisible spin consecutiveCheckErrorLimit +// exists to prevent, and repeating an invocation that has already proven it +// cannot finish is a blind retry, not a recovery. +// +// Parking instead is bounded (one budget, then a decision), keeps the run and +// its worktree alive rather than tearing them down, and leaves any further +// attempt to the operator, who can respond with a fix selection to spend +// another budget deliberately. +func ciFixAgentTimeoutOutcome(issueDesc string, dirtyWorktree string, err error) *pipeline.StepOutcome { + description := fmt.Sprintf( + "The CI auto-fix agent did not finish within its invocation budget while repairing: %s. "+ + "Reported: %v. Re-running the same request costs another full budget, so no further attempt is made automatically. "+ + "Check that the configured agent CLI is authenticated and responsive, then respond with a fix selection to spend another budget, or resolve the CI failure outside the pipeline.", + issueDesc, err) + if dirtyWorktree != "" { + description += fmt.Sprintf(" The timed-out agent left uncommitted changes in the run worktree at %s; they are not committed or pushed.", dirtyWorktree) + } + findings := Findings{ + Summary: "CI auto-fix agent exceeded its invocation budget", + Items: []Finding{{ + Severity: "warning", + Description: description, + Action: types.ActionAskUser, + }}, + } + findingsJSON, _ := json.Marshal(findings) + return &pipeline.StepOutcome{ + NeedsApproval: true, + Findings: string(findingsJSON), + } +} + func ciMergeabilityOutcome(summary, description string) *pipeline.StepOutcome { findings := Findings{ Summary: summary, diff --git a/internal/pipeline/steps/ci_fix.go b/internal/pipeline/steps/ci_fix.go index 0239d0647..1a97a738d 100644 --- a/internal/pipeline/steps/ci_fix.go +++ b/internal/pipeline/steps/ci_fix.go @@ -2,6 +2,7 @@ package steps import ( "encoding/json" + "errors" "fmt" "log/slog" "strings" @@ -141,6 +142,31 @@ CI logs: return s.commitRepair(sctx, summary) } +// ciFixAgentBudgetOutcome converts an auto-fix invocation that exhausted its +// agent budget into a bounded ask-user gate, and returns nil for every other +// result so ordinary transient fix failures keep their existing warn-and-retry +// behaviour. Only a proven full-budget burn parks: it is the one failure that +// is guaranteed to cost the same again on the next poll. +func ciFixAgentBudgetOutcome(sctx *pipeline.StepContext, issueDesc string, err error) *pipeline.StepOutcome { + if err == nil || !errors.Is(err, pipeline.ErrAgentTimeout) { + return nil + } + sctx.Log(fmt.Sprintf("CI auto-fix agent exceeded its invocation budget: %v", err)) + return ciFixAgentTimeoutOutcome(issueDesc, dirtyRunWorktree(sctx), err) +} + +// dirtyRunWorktree reports the run worktree path when the timed-out agent left +// uncommitted work there, so the gate can say where it is instead of letting it +// disappear with the worktree at cleanup. Best effort: an unreadable status +// simply omits the detail. +func dirtyRunWorktree(sctx *pipeline.StepContext) string { + status, err := stepGitRun(sctx, "status", "--porcelain") + if err != nil || strings.TrimSpace(status) == "" { + return "" + } + return sctx.WorkDir +} + const maxReviewCommentsPromptBytes = 32 * 1024 type promptReviewComment struct { diff --git a/internal/pipeline/steps/review.go b/internal/pipeline/steps/review.go index 9eb631a11..c915de85d 100644 --- a/internal/pipeline/steps/review.go +++ b/internal/pipeline/steps/review.go @@ -414,9 +414,13 @@ func reviewAgentContext(sctx *pipeline.StepContext) (context.Context, context.Ca var errReviewAgentTimeout = errors.New("review agent timeout") +// reviewAgentError renders a review-round budget expiry. The measured activity +// evidence comes from the shared agent-run seam; the budget is never restated +// as if it were the silence, because the two are different facts and only one +// of them was observed. func reviewAgentError(ctx context.Context, timeout time.Duration, prefix string, err error) error { if timeout > 0 && errors.Is(context.Cause(ctx), errReviewAgentTimeout) { - return fmt.Errorf("%s timed out after %s (review agent silent for %s): %w", prefix, timeout, timeout, err) + return fmt.Errorf("%s timed out after %s: %w", prefix, timeout, err) } return fmt.Errorf("%s: %w", prefix, err) } diff --git a/internal/pipeline/steps/review_session_test.go b/internal/pipeline/steps/review_session_test.go index e7aa9e970..84a5b6aae 100644 --- a/internal/pipeline/steps/review_session_test.go +++ b/internal/pipeline/steps/review_session_test.go @@ -58,6 +58,28 @@ func (m *sessionMockAgent) snapshot() []agent.RunOpts { return append([]agent.RunOpts(nil), m.calls...) } +type sessionFallbackTimeoutAgent struct { + calls int +} + +func (a *sessionFallbackTimeoutAgent) Name() string { return "session-timeout" } + +func (a *sessionFallbackTimeoutAgent) SupportsSessionResume() bool { return true } + +func (a *sessionFallbackTimeoutAgent) Close() error { return nil } + +func (a *sessionFallbackTimeoutAgent) Run(ctx context.Context, opts agent.RunOpts) (*agent.Result, error) { + a.calls++ + if opts.Session != nil && opts.Session.ID != "" { + opts.OnLifecycle(agent.LifecycleEvent{Agent: a.Name(), Phase: agent.LifecyclePhaseStart, PID: 7171}) + opts.OnLifecycle(agent.LifecycleEvent{Agent: a.Name(), Phase: agent.LifecyclePhaseActivity}) + return nil, fmt.Errorf("resume failed") + } + opts.OnLifecycle(agent.LifecycleEvent{Agent: a.Name(), Phase: agent.LifecyclePhaseStart, PID: 7272}) + <-ctx.Done() + return nil, ctx.Err() +} + // reviewSessionHarness wires a real executor around real steps with a // session-capable mock agent and real git worktree. func reviewSessionHarness(t *testing.T, mock *sessionMockAgent, steps []pipeline.Step) (*pipeline.Executor, *db.DB, *db.Run, *db.Repo, string) { @@ -302,6 +324,37 @@ func TestReviewLoop_ParkRespondFixKeepsRoleSessions(t *testing.T) { } } +func TestReviewFixerSession_FreshFallbackTimeoutExcludesResumeActivity(t *testing.T) { + dir, baseSHA, headSHA := setupGitRepo(t) + ag := &sessionFallbackTimeoutAgent{} + sctx := newTestContextWithDBRecords(t, ag, dir, baseSHA, headSHA, config.Commands{}) + sctx.Config.ReviewAgentTimeout = 50 * time.Millisecond + sctx.Config.SessionReuse = true + sctx.Fixing = true + sctx.PreviousFindings = `{"findings":[{"id":"f-1","severity":"error","description":"fix this","action":"auto-fix"}]}` + if err := sctx.DB.UpsertRunAgentSession(sctx.Run.ID, string(pipeline.SessionRoleFixer), ag.Name(), "stale-session"); err != nil { + t.Fatalf("store fixer session: %v", err) + } + sctx.Sessions = pipeline.NewRunSessions(sctx.DB, sctx.Run.ID, ag, true) + + _, err := (&ReviewStep{}).Execute(sctx) + if err == nil { + t.Fatal("expected the fresh fixer session to time out") + } + if ag.calls != 2 { + t.Fatalf("agent calls = %d, want failed resume and fresh fallback", ag.calls) + } + if !strings.Contains(err.Error(), "produced no output at all") { + t.Fatalf("error = %q, want fresh fallback silence", err) + } + if strings.Contains(err.Error(), "last produced output") { + t.Fatalf("error = %q, dead resume activity must not describe the fresh fallback", err) + } + if !strings.Contains(err.Error(), "pid=7272") { + t.Fatalf("error = %q, want silence attributed to the fresh fixer subprocess", err) + } +} + // TestReviewLoop_OtherStepsStaySessionIsolated proves the reviewer/fixer // sessions are never lent to other pipeline steps: agent-driven document and // lint work runs with no session at all. diff --git a/internal/pipeline/steps/review_test.go b/internal/pipeline/steps/review_test.go index 46153d5a9..417aa5834 100644 --- a/internal/pipeline/steps/review_test.go +++ b/internal/pipeline/steps/review_test.go @@ -3,6 +3,7 @@ package steps import ( "context" "encoding/json" + "errors" "fmt" "os" "path/filepath" @@ -42,12 +43,61 @@ func TestReviewStep_HangingAgentFailsRunAfterTimeout(t *testing.T) { if run.Status != types.RunFailed { t.Fatalf("run status = %s, want %s", run.Status, types.RunFailed) } - if run.Error == nil || !strings.Contains(*run.Error, "review agent silent for 20ms") { - var got string - if run.Error != nil { - got = *run.Error - } - t.Fatalf("run error = %q, want timeout diagnostic", got) + var got string + if run.Error != nil { + got = *run.Error + } + // The diagnostic must name the budget that expired AND report what was + // actually observed. An agent that never emitted anything is a different + // operator problem from one that streamed until the deadline, and the run + // error is the only place that distinction survives. + if !strings.Contains(got, "timed out after 20ms") { + t.Fatalf("run error = %q, want the expired review budget named", got) + } + if !strings.Contains(got, "produced no output at all") { + t.Fatalf("run error = %q, want the measured silence of a never-emitting agent", got) + } + if strings.Contains(got, "silent for 20ms") { + t.Fatalf("run error = %q, must not restate the budget as if it were a measurement", got) + } +} + +// TestReviewStep_RoundBudgetTimeoutPreservesTheAgentReport pins the other half +// of the diagnostic contract at the review round budget: whatever the adapter +// managed to report reaches the operator. For a native agent that error is the +// killed subprocess's exit status and stderr - the only account of what the +// process was actually doing - and it is what makes a silent 30-minute review +// timeout diagnosable instead of a dead end. +func TestReviewStep_RoundBudgetTimeoutPreservesTheAgentReport(t *testing.T) { + dir, baseSHA, headSHA := setupGitRepo(t) + ag := &mockAgent{ + name: "reporting-review-agent", + runFn: func(ctx context.Context, _ agent.RunOpts) (*agent.Result, error) { + <-ctx.Done() + return nil, errors.New("pi exited: signal: killed: pi: provider authentication required") + }, + } + sctx := newTestContextWithDBRecords(t, ag, dir, baseSHA, headSHA, config.Commands{}) + sctx.Config.ReviewAgentTimeout = 20 * time.Millisecond + + exec := pipeline.NewExecutor(sctx.DB, paths.WithRoot(t.TempDir()), sctx.Config, ag, []pipeline.Step{&ReviewStep{}}, nil) + if err := exec.Execute(context.Background(), sctx.Run, sctx.Repo, dir); err == nil { + t.Fatal("expected the review round budget to fail the run") + } + + run, err := sctx.DB.GetRun(sctx.Run.ID) + if err != nil { + t.Fatalf("get run: %v", err) + } + var got string + if run.Error != nil { + got = *run.Error + } + if !strings.Contains(got, "provider authentication required") { + t.Fatalf("run error = %q, want the agent's own report preserved", got) + } + if !strings.Contains(got, "timed out after 20ms") { + t.Fatalf("run error = %q, want the expired review budget named", got) } } diff --git a/internal/pipeline/steps/test.go b/internal/pipeline/steps/test.go index dfb20fd70..318996d04 100644 --- a/internal/pipeline/steps/test.go +++ b/internal/pipeline/steps/test.go @@ -290,9 +290,16 @@ func testAgentContext(sctx *pipeline.StepContext) (context.Context, context.Canc var errTestAgentTimeout = errors.New("test agent timeout") +// testAgentError renders a Test-invocation budget expiry. It keeps the agent's +// own error rather than replacing it with the bare context cause: for a native +// agent that error carries the killed subprocess's exit status and stderr, and +// is the only account of what the process was doing when the budget ran out. func testAgentError(ctx context.Context, timeout time.Duration, prefix string, err error) error { if timeout > 0 && errors.Is(context.Cause(ctx), errTestAgentTimeout) { - return fmt.Errorf("%s timed out after %s (test agent silent for %s): %w", prefix, timeout, timeout, context.Cause(ctx)) + if err == nil { + err = context.Cause(ctx) + } + return fmt.Errorf("%s timed out after %s: %w", prefix, timeout, err) } if err != nil { return fmt.Errorf("%s: %w", prefix, err) diff --git a/internal/pipeline/steps/test_test.go b/internal/pipeline/steps/test_test.go index cfb52dc76..b36e59717 100644 --- a/internal/pipeline/steps/test_test.go +++ b/internal/pipeline/steps/test_test.go @@ -41,12 +41,18 @@ func TestTestStep_HangingEvidenceAgentFailsRunAfterTimeout(t *testing.T) { if run.Status != types.RunFailed { t.Fatalf("run status = %s, want %s", run.Status, types.RunFailed) } - if run.Error == nil || !strings.Contains(*run.Error, "test agent silent for 20ms") { - var got string - if run.Error != nil { - got = *run.Error - } - t.Fatalf("run error = %q, want timeout diagnostic", got) + var got string + if run.Error != nil { + got = *run.Error + } + if !strings.Contains(got, "timed out after 20ms") { + t.Fatalf("run error = %q, want the expired test budget named", got) + } + if !strings.Contains(got, "produced no output at all") { + t.Fatalf("run error = %q, want the measured silence of a never-emitting agent", got) + } + if strings.Contains(got, "silent for 20ms") { + t.Fatalf("run error = %q, must not restate the budget as if it were a measurement", got) } } diff --git a/internal/skill/skill.go b/internal/skill/skill.go index f3fda1497..26ad52a6f 100644 --- a/internal/skill/skill.go +++ b/internal/skill/skill.go @@ -242,6 +242,9 @@ Recovery takes that head by fast-forward, or by adopting a diverged preserved he That proof is deliberately narrow, so a rebase whose fix rounds also rewrote your own lines refuses instead of being adopted: when nothing can tell a deliberate pipeline fix from a dropped change, the decision is yours. A ` + "`branch_sync.state`" + ` of ` + "`user_owned`" + ` means the run went terminal before changing the submitted head and cancellation released the branch: the exact branch and head are yours and immediately usable for whichever delivery path is authorized - no sync action is needed, and a repeated ` + "`--recover`" + ` there is a harmless no-op. A dirty worktree, or divergence that cannot be proven contained, makes the recovery refuse with explicit choices; ` + "`--keep-local`" + ` keeps your current head while the preserved commits stay anchored under ` + "`refs/no-mistakes/recover/`" + `. +When ` + "`next_action.code`" + ` is ` + "`return_custody_keep_local`" + `, the terminal run's recorded pipeline head cannot be verified - it is in no reachable object store, or the run's own recovery evidence names a different commit: run ` + "`no-mistakes axi sync --recover --keep-local`" + ` to return custody at your current head and point the gate branch at it. +If any copy of that head still exists, the settlement anchors it at ` + "`refs/no-mistakes/recover-stranded/`" + ` before moving the gate branch off it - but only in whichever repository still holds the object, your worktree or the local gate under ` + "`NM_HOME`" + ` or both, so look in both rather than assuming your own; if no store still has it, nothing is anchored because nothing was left to anchor and the settlement took nothing from you. +That anchor is NOT ` + "`refs/no-mistakes/recover/`" + `, which in this state is the conflicting evidence and can name an entirely different commit. If synchronization is blocked, process that structured state instead of improvising reset, stash, merge, rebase, force, or branch replacement. After synchronization, commit the follow-up on top and re-run ` + "`no-mistakes axi run --intent \"...\"`" + ` with the original user intent. This preserves every prior gate-fix commit regardless of its configured subject. diff --git a/internal/tui/app.go b/internal/tui/app.go index 0e3b0534b..c08a6c5fd 100644 --- a/internal/tui/app.go +++ b/internal/tui/app.go @@ -90,8 +90,10 @@ type Model struct { syncRefresh func() branchsync.State syncApply func() branchsync.State syncRecover func() branchsync.State + syncSettle func() branchsync.State syncConfirm bool recoverConfirm bool + settleConfirm bool syncRefreshing bool } @@ -382,6 +384,7 @@ func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) { m.syncRefreshing = false m.syncConfirm = false m.recoverConfirm = false + m.settleConfirm = false m.branchSync = &msg.state if msg.state.Error != "" { m.err = fmt.Errorf("branch sync: %s", msg.state.Error) @@ -498,6 +501,10 @@ func Run(socketPath string, client *ipc.Client, run *ipc.RunInfo, latestVersion model.syncRefresh = func() branchsync.State { return service.Refresh(context.Background()) } model.syncApply = func() branchsync.State { return service.Apply(context.Background()) } model.syncRecover = func() branchsync.State { return service.Recover(context.Background(), false) } + // keepLocal=true is the settlement exit for a self-inconsistent + // record; it is a separate seam so the plain recovery can never + // silently become one. + model.syncSettle = func() branchsync.State { return service.Recover(context.Background(), true) } model.refreshCachedSync() } p := tea.NewProgram(model, tea.WithAltScreen()) diff --git a/internal/tui/branch_sync.go b/internal/tui/branch_sync.go index d96fc8707..b6c2bf224 100644 --- a/internal/tui/branch_sync.go +++ b/internal/tui/branch_sync.go @@ -23,6 +23,9 @@ func renderLocalBranchStatus(state *branchsync.State, refreshing bool, width int if recoverableBranchSync(state) { message = "Run ended without publishing its pipeline commits; they are preserved in the local gate. Recover custody to take the branch back, or rerun to resume validation." footer = "u recover custody" + } else if settleableBranchSync(state) { + message = "Run ended terminally and its recorded pipeline head can no longer be verified, so there is nothing to recover. Settle custody at your current head to take the branch back." + footer = "u settle custody at local head" } else { message = "Local branch unchanged; the pipeline fix is not pushed yet. Do not make follow-up commits." } @@ -106,6 +109,23 @@ func recoverableBranchSync(state *branchsync.State) bool { return state != nil && state.State == branchsync.StatePipelineOwned && state.Safety == "blocked_pipeline_owned_recoverable" } +// settleableBranchSync reports whether the state is the SELF-INCONSISTENT +// terminal custody record whose only exit is the keep-local settlement (#824). +// +// It keys on the advertised next action rather than on a safety code, and that +// is deliberate. A self-inconsistent record reaches the TUI under whichever +// safety code described how it failed to verify - blocked_recover_preserved_ +// head_missing is only the commonest - so a safety-code list here would drift +// out of agreement with the branchsync predicate that decides whether the +// settlement can actually complete. Keying on the action the state machine +// itself advertises is what makes this exactly the road the CLI offers: the +// service never names return_custody_keep_local for a record the settlement +// would only refuse, and the TUI must never offer one it does not. +func settleableBranchSync(state *branchsync.State) bool { + return state != nil && state.State == branchsync.StatePipelineOwned && + state.NextAction != nil && state.NextAction.Code == "return_custody_keep_local" +} + func renderRecoverConfirmation(state branchsync.State, width int) string { if width < 40 { width = 80 @@ -121,6 +141,26 @@ func renderRecoverConfirmation(state branchsync.State, width int) string { return renderBoxWithFooter("Confirm custody recovery", b.String(), width, "u/enter recover · esc cancel") } +// renderSettleConfirmation is deliberately NOT renderRecoverConfirmation with +// different words. Recovery takes the preserved pipeline head; settlement +// keeps the local head and moves the gate to it, abandoning a recorded head +// that can no longer be verified. The CLI makes that an explicit --keep-local +// choice, so the TUI has to state the same consequence before asking for it. +func renderSettleConfirmation(state branchsync.State, width int) string { + if width < 40 { + width = 80 + } + var b strings.Builder + fmt.Fprintf(&b, "The run ended %s and its recorded pipeline head can no longer be verified,\n", state.Pipeline.Status) + fmt.Fprintf(&b, "so there is nothing to recover. Settling returns custody at the head you\n") + fmt.Fprintf(&b, "already have and points the gate branch at it.\n\n") + fmt.Fprintf(&b, "Local branch: %s\n", state.Local.Branch) + fmt.Fprintf(&b, "Kept HEAD: %s\n", state.Local.Head) + fmt.Fprintf(&b, "Recorded HEAD: %s (unverifiable)\n\n", state.Pipeline.CurrentHead) + b.WriteString("Your worktree is never touched. Any still-reachable copy of the recorded head is anchored first, and the settlement refuses rather than proceeding if one exists and cannot be anchored. The gate moves only by compare-and-swap, so a concurrent gate push wins. This is `no-mistakes sync --recover --keep-local`.") + return renderBoxWithFooter("Confirm custody settlement at local head", b.String(), width, "u/enter settle · esc cancel") +} + func renderSyncConfirmation(state branchsync.State, width int) string { if width < 40 { width = 80 diff --git a/internal/tui/branch_sync_test.go b/internal/tui/branch_sync_test.go index f5608a82c..afc7bba71 100644 --- a/internal/tui/branch_sync_test.go +++ b/internal/tui/branch_sync_test.go @@ -252,3 +252,122 @@ func TestActivePipelineOwnedStateOffersNoRecoveryAction(t *testing.T) { t.Fatalf("u acted on an active pipeline_owned state: %#v", m) } } + +// TestWedgedCustodyRecordReachesTheSameSettlementExitAsTheCLI is the TUI half +// of issue #824. A self-inconsistent custody record - terminal run, recorded +// pipeline head no longer verifiable - carries the settlement next action +// rather than blocked_pipeline_owned_recoverable, so keying the u affordance on +// the recoverable safety code alone left the TUI as the one operator surface +// with no exit at all: the CLI could settle the record and the TUI could only +// describe it. The exit must be the same one, and it must be an explicit +// choice, because settling KEEPS the local head instead of taking the +// preserved one. +func TestWedgedCustodyRecordReachesTheSameSettlementExitAsTheCLI(t *testing.T) { + run := &ipc.RunInfo{ID: "run-1", Branch: "feature", Status: types.RunFailed} + m := NewModel("socket", nil, run) + wedged := branchsync.State{ + State: branchsync.StatePipelineOwned, Relation: branchsync.RelationUnknown, + Safety: "blocked_recover_preserved_head_missing", + Local: branchsync.LocalState{Branch: "feature", Head: strings.Repeat("a", 40), Clean: true}, + Pipeline: branchsync.PipelineState{RunID: "run-1", Status: "failed", Phase: "pre_push", CurrentHead: strings.Repeat("c", 40)}, + NextAction: &branchsync.NextAction{Code: "return_custody_keep_local", Command: "no-mistakes axi sync --recover --keep-local"}, + } + m.branchSync = &wedged + + // The dead end was visible here first: the status line described the block + // and offered no key at all. + view := stripANSI(renderLocalBranchStatus(m.branchSync, false, 80)) + for _, want := range []string{"can no longer be verified", "u settle custody at local head"} { + if !strings.Contains(view, want) { + t.Errorf("wedged status missing %q:\n%s", want, view) + } + } + + settleCalls := 0 + recoverCalls := 0 + m.syncRecover = func() branchsync.State { + recoverCalls++ + t.Error("a wedged record must not take the plain recovery path") + return branchsync.State{} + } + m.syncSettle = func() branchsync.State { + settleCalls++ + settled := wedged + settled.State = branchsync.StateCustodyReturned + settled.Safety = "custody_returned" + settled.Relation = branchsync.RelationEqual + settled.Recovered = true + settled.NextAction = nil + return settled + } + + nextModel, cmd := m.handleKey(keyMsg("u")) + m = nextModel.(Model) + if cmd != nil || !m.settleConfirm || settleCalls != 0 { + t.Fatalf("u must open the settlement confirmation without acting: confirm=%v calls=%d", m.settleConfirm, settleCalls) + } + if m.recoverConfirm { + t.Fatal("wedged record opened the recovery confirmation instead of the settlement one") + } + // Settling keeps the local head and abandons an unverifiable recorded one, + // so the confirmation has to say which head survives before asking. + plain := stripANSI(m.View()) + for _, want := range []string{"custody", strings.Repeat("a", 40), strings.Repeat("c", 40), "u/enter settle", "--keep-local"} { + if !strings.Contains(plain, want) { + t.Errorf("settlement confirmation missing %q:\n%s", want, plain) + } + } + + // esc must back out without touching anything. + escModel, escCmd := m.handleKey(keyMsg("esc")) + escaped := escModel.(Model) + if escCmd != nil || escaped.settleConfirm || settleCalls != 0 { + t.Fatalf("esc did not cancel the settlement cleanly: confirm=%v calls=%d", escaped.settleConfirm, settleCalls) + } + + nextModel, cmd = m.handleKey(keyMsg("enter")) + m = nextModel.(Model) + if cmd == nil || settleCalls != 0 { + t.Fatal("settlement did not wait for its async command") + } + next, _ := m.Update(cmd()) + m = next.(Model) + if settleCalls != 1 || recoverCalls != 0 { + t.Fatalf("settlement calls=%d recover calls=%d", settleCalls, recoverCalls) + } + if m.settleConfirm || m.branchSync.State != branchsync.StateCustodyReturned || !m.branchSync.Recovered { + t.Fatalf("settlement result = %#v", m.branchSync) + } + if m.err != nil { + t.Fatalf("successful settlement left an error: %v", m.err) + } +} + +// TestPipelineOwnedStateWithoutASettlementActionOffersNoSettlement keeps the +// TUI from inventing an exit the service did not advertise: the branchsync +// predicate decides where the settlement can actually complete, and a record it +// sent to manual reconciliation must not be offered a key that would only +// refuse - which is the #824 shape in reverse. +func TestPipelineOwnedStateWithoutASettlementActionOffersNoSettlement(t *testing.T) { + run := &ipc.RunInfo{ID: "run-1", Branch: "feature", Status: types.RunFailed} + m := NewModel("socket", nil, run) + m.branchSync = &branchsync.State{ + State: branchsync.StatePipelineOwned, Safety: "blocked_recover_preserved_head_missing", + Local: branchsync.LocalState{Branch: "feature", Head: strings.Repeat("a", 40), Clean: true}, + Pipeline: branchsync.PipelineState{RunID: "run-1", Status: "failed", Phase: "pre_push"}, + NextAction: &branchsync.NextAction{Code: "inspect_and_reconcile_manually", Command: "no-mistakes axi status"}, + } + m.syncSettle = func() branchsync.State { + t.Fatal("settlement must not be reachable for a record sent to manual reconciliation") + return branchsync.State{} + } + view := stripANSI(renderLocalBranchStatus(m.branchSync, false, 80)) + if strings.Contains(view, "u settle") { + t.Fatalf("unadvertised settlement offered a key:\n%s", view) + } + nextModel, cmd := m.handleKey(keyMsg("u")) + m = nextModel.(Model) + if cmd != nil || m.settleConfirm || m.recoverConfirm || m.syncConfirm { + t.Fatalf("u acted on a record with no advertised settlement: %#v", m) + } +} diff --git a/internal/tui/commands.go b/internal/tui/commands.go index bb75630f2..aa2d0644c 100644 --- a/internal/tui/commands.go +++ b/internal/tui/commands.go @@ -378,6 +378,26 @@ func (m Model) applyRecoverCmd() tea.Cmd { } } +// applySettleCmd runs the keep-local settlement. It is reported under its own +// telemetry action so a settlement is never counted as a recovery: they end +// the same state but keep opposite heads. +func (m Model) applySettleCmd() tea.Cmd { + settle := m.syncSettle + if settle == nil { + return nil + } + return func() tea.Msg { + started := time.Now() + state := settle() + result := "refused" + if state.Recovered { + result = "applied" + } + trackTUISyncAttempt("settle", state, result, started) + return syncAppliedMsg{state: state} + } +} + func (m Model) spinnerTickCmd() tea.Cmd { return tea.Tick(spinnerTickInterval, func(time.Time) tea.Msg { return spinnerTickMsg{} diff --git a/internal/tui/keys.go b/internal/tui/keys.go index 3693b52e5..a39c1d7f9 100644 --- a/internal/tui/keys.go +++ b/internal/tui/keys.go @@ -52,6 +52,25 @@ func (m Model) handleKey(msg tea.KeyMsg) (tea.Model, tea.Cmd) { } } + if m.settleConfirm { + switch key { + case "esc": + m.settleConfirm = false + return m, nil + case "u", "enter": + if m.syncRefreshing { + return m, nil + } + m.syncRefreshing = true + return m, m.applySettleCmd() + case "q", "ctrl+c": + m.quitting = true + return m, tea.Sequence(tea.SetWindowTitle(""), tea.Quit) + default: + return m, nil + } + } + // Reset abort confirmation on any key except 'x'. if key != "x" { m.confirmAbort = false @@ -253,6 +272,11 @@ func (m Model) handleKey(msg tea.KeyMsg) (tea.Model, tea.Cmd) { m.recoverConfirm = true return m, nil } + if settleableBranchSync(m.branchSync) && m.syncSettle != nil { + m.err = nil + m.settleConfirm = true + return m, nil + } if m.syncRefresh == nil || m.branchSync.NextAction == nil || m.branchSync.NextAction.Code != "sync" { return m, nil } diff --git a/internal/tui/view.go b/internal/tui/view.go index 1cb4e8202..90edcdb1a 100644 --- a/internal/tui/view.go +++ b/internal/tui/view.go @@ -116,6 +116,9 @@ func (m Model) View() string { if m.recoverConfirm && m.branchSync != nil { extraSections = append(extraSections, renderRecoverConfirmation(*m.branchSync, rightWidth)) } + if m.settleConfirm && m.branchSync != nil { + extraSections = append(extraSections, renderSettleConfirmation(*m.branchSync, rightWidth)) + } // Modal editor takes priority over findings/logs so it always renders // when active. Bypass the content budget so it never gets dropped on diff --git a/skills/no-mistakes/SKILL.md b/skills/no-mistakes/SKILL.md index 36f3b973f..e2d5c9a7f 100644 --- a/skills/no-mistakes/SKILL.md +++ b/skills/no-mistakes/SKILL.md @@ -242,6 +242,9 @@ Recovery takes that head by fast-forward, or by adopting a diverged preserved he That proof is deliberately narrow, so a rebase whose fix rounds also rewrote your own lines refuses instead of being adopted: when nothing can tell a deliberate pipeline fix from a dropped change, the decision is yours. A `branch_sync.state` of `user_owned` means the run went terminal before changing the submitted head and cancellation released the branch: the exact branch and head are yours and immediately usable for whichever delivery path is authorized - no sync action is needed, and a repeated `--recover` there is a harmless no-op. A dirty worktree, or divergence that cannot be proven contained, makes the recovery refuse with explicit choices; `--keep-local` keeps your current head while the preserved commits stay anchored under `refs/no-mistakes/recover/`. +When `next_action.code` is `return_custody_keep_local`, the terminal run's recorded pipeline head cannot be verified - it is in no reachable object store, or the run's own recovery evidence names a different commit: run `no-mistakes axi sync --recover --keep-local` to return custody at your current head and point the gate branch at it. +If any copy of that head still exists, the settlement anchors it at `refs/no-mistakes/recover-stranded/` before moving the gate branch off it - but only in whichever repository still holds the object, your worktree or the local gate under `NM_HOME` or both, so look in both rather than assuming your own; if no store still has it, nothing is anchored because nothing was left to anchor and the settlement took nothing from you. +That anchor is NOT `refs/no-mistakes/recover/`, which in this state is the conflicting evidence and can name an entirely different commit. If synchronization is blocked, process that structured state instead of improvising reset, stash, merge, rebase, force, or branch replacement. After synchronization, commit the follow-up on top and re-run `no-mistakes axi run --intent "..."` with the original user intent. This preserves every prior gate-fix commit regardless of its configured subject.