Skip to content

fix(teardown): skip worktree steps on rerun when the pool slot was returned or reissued - #2387

Open
Kallas95 wants to merge 3 commits into
kunchenguid:mainfrom
Kallas95:fm/fm-main-rebase-amont
Open

fix(teardown): skip worktree steps on rerun when the pool slot was returned or reissued#2387
Kallas95 wants to merge 3 commits into
kunchenguid:mainfrom
Kallas95:fm/fm-main-rebase-amont

Conversation

@Kallas95

Copy link
Copy Markdown

Intent

Port the local-only teardown ownership guard onto the current upstream head and ship it as an upstream PR against kunchenguid/firstmate:main.

Background: commit f603232 existed only on this firstmate home's local main and never reached upstream. Upstream has since heavily reworked bin/fm-teardown.sh and both teardown test suites. Verified that upstream does NOT already carry this guard, so it is not redundant. Because a cherry-pick was unavailable in this environment, the guard's intent was RE-FITTED into the current upstream structure of bin/fm-teardown.sh rather than forcing the old hunks in; f603232's commit message body is the deliberate basis for the new commit message.

Behavior contract to preserve exactly: a teardown rerun after a post-return failure must never reset or reap a pool worktree slot that (a) this task already returned - proven by a durable state/.worktree-returned marker touched right after its own successful return and removed by teardown's final cleanup - or (b) another task's meta now records, which proves reissue even when the marker is lost. When either signal shows the slot is no longer this task's, teardown skips the safety inspection, run-abort, worktree process reap, branch delete, hook removal, and the return itself, while tasktmp reaping, the pane close retry, and durable-record cleanup continue unchanged.

Deliberate decisions a reviewer reading only the diff would not know:

  • The guard is deliberately scoped to the treehouse pool-return branch and to non-secondmate tasks, exactly as in the original fix. The Orca branch removes its worktree through its own backend call rather than returning a shared pool slot, so it is intentionally out of scope.
  • The marker is touched immediately AFTER a successful return and BEFORE any later step can fail; a failed return must NOT write it, so a rerun still performs the return.
  • A failure to write the marker warns but does not fail teardown, because live fleet bindings remain as the second signal.
  • The AGENTS.md state-layout line for .worktree-returned was re-added and worded consistently with upstream's current AGENTS.md text.
  • Four ported test cases were added to tests/fm-teardown.test.sh: an end-to-end replay of the 2026-08-08 incident, a plain first-run return, a rerun with the worktree still bound and un-returned, and the lost-marker reissue caught by fleet bindings alone. All four were verified to FAIL against the unpatched script, so they pin the guard rather than passing vacuously.
  • tests/fm-teardown-endpoint-safety.test.sh is deliberately untouched and passes 7/7.
  • Known pre-existing, unrelated local-environment failure: herdr-preflight-missing-adapter reproduces identically at origin/main on this host and is NOT caused by this change. It was temporarily disabled only to let the rest of the suite run locally; that temporary edit was reverted and is not in the commit.

Constraints: this changes firstmate's shared tracked material, so .agents/skills/firstmate-coding-guidelines/SKILL.md applies - one sentence per line in Markdown, plain dash never an em dash, no agent co-author, shellcheck-clean bin scripts under the pinned version, and colocated tests extending the existing suite. Delivery route: no write access to origin, so the branch fm/fm-main-rebase-amont must go to the fork Kallas95/firstmate and the PR must open against kunchenguid/firstmate:main. Terminal state is local validation green plus the upstream PR open.

What Changed

  • bin/fm-teardown.sh adds a rerun tenant-safety guard: a durable state/<id>.worktree-returned marker is touched immediately after this task's own successful pool-worktree return (removed by teardown's final cleanup), and ownership is re-checked on rerun from that marker plus live fleet bindings — another task's meta recording the same worktree path proves reissue even when the marker is lost, while bindings whose task carries its own returned marker are ignored as past tenancies. When either signal shows the slot is no longer this task's, the safety inspection, run-abort, worktree process reap, branch delete, hook removal, and the return itself are skipped; tasktmp reaping, the pane close retry, and durable-record cleanup continue unchanged. A failed marker write warns instead of failing teardown, leaving fleet bindings as the fallback signal.
  • tests/fm-teardown.test.sh gains four cases pinning the guard: an end-to-end replay of the 2026-08-08 teardown-rerun-reissue incident, a plain first-run return, a rerun whose worktree is still bound and un-returned (the return must still happen), and a lost-marker reissue caught by fleet bindings alone, including ignoring a predecessor's stale past-tenancy binding.
  • AGENTS.md documents the new <id>.worktree-returned state entry, and docs/architecture.md qualifies the teardown worktree checks as applying only while the slot is provably still this task's.

Risk Assessment

✅ Low: The fix round implements the captain-authorized refinement exactly as prescribed (marker-bearing bindings skipped with the same presence test, header contract updated, scan semantics otherwise untouched), adds a colocated behavior-observable regression test that is non-vacuous by inspection against the pre-fix code, leaves no test disabled, and the only remaining observation is an explicitly authorized residual corner that never harms a live tenant.

Testing

Exercised the teardown ownership guard end-to-end via the colocated suites and swapped-script trees: the endpoint-safety suite passes 7/7, the full teardown suite passes 62/62 at the target commit once the one proven pre-existing local herdr-preflight environment failure (reproduced identically on the pure base commit) is excluded, the new tests were shown to fail against the unpatched and pre-review-fix scripts so they genuinely pin the guard, and an archived CLI transcript of the 2026-08-08 incident replay shows the rerun skipping the re-return while pane close and record cleanup complete and the reissued slot's tenant survives untouched; no UI surface is involved, so CLI transcripts are the product-level evidence.

Evidence: Incident replay CLI transcript (attempt 1 fails after the return, rerun skips every worktree step, tenant untouched)
# Incident replay (teardown-rerun-reissue, 2026-08-08) - CLI transcript
# Sandbox: hermetic fixture from tests/fm-teardown.test.sh::test_rerun_after_return_and_reissue_never_rereturns
# Script under test: bin/fm-teardown.sh at target commit 7213b9b

## Attempt 1: teardown returns the pool worktree, then fails on the refused pane close (the incident's exact failure point)
### stderr:
error: herdr pane default:wG:pQ for task-x1 is not confirmed gone after its close was refused, skipped, or failed; retaining every durable task record - rerun teardown once the close can run under the session lock

### treehouse invocations so far (exactly one return):
return --force /var/folders/5q/snmnm0_926ddy6xlmggzx1g80000gn/T//fm-teardown-tests.duhMyC/rerun-reissue/wt

### durable marker written right after the successful return:
state/task-x1.worktree-returned existed after attempt 1 (asserted by the test before the rerun)

## Between attempts: the pool reissues the SAME slot to newer task task-x2
task-x2.meta binds worktree=/var/folders/5q/snmnm0_926ddy6xlmggzx1g80000gn/T//fm-teardown-tests.duhMyC/rerun-reissue/wt ; its session sits on branch fm/task-x2 with uncommitted tenant.txt

## Attempt 2 (sanctioned rerun): stdout
teardown: skipping worktree return and every other worktree step for task-x1: this task's return already succeeded on a previous attempt
teardown task-x1 complete (window default:wG:pQ, worktree /var/folders/5q/snmnm0_926ddy6xlmggzx1g80000gn/T//fm-teardown-tests.duhMyC/rerun-reissue/wt)
Backlog: task-x1 just finished. Run tasks-axi done task-x1 --note "local main", then run tasks-axi ready for dependency-cleared candidates, check date gates, and dispatch only work whose blockers are gone and date is due.

## After the rerun:
### treehouse invocations in total (STILL exactly one - the reissued slot was never re-returned):
return --force /var/folders/5q/snmnm0_926ddy6xlmggzx1g80000gn/T//fm-teardown-tests.duhMyC/rerun-reissue/wt

### tenant worktree untouched:
branch: fm/task-x2
uncommitted tenant file: tenant work in flight

### state dir: task-x1 records (meta + worktree-returned marker) cleaned, tenant task-x2.meta preserved:
task-x2.meta
Evidence: Fail-before/pass-after regression-pinning matrix (base 6789876, mid 1097082, target 7213b9b)
# Regression pinning: fail-before / pass-after matrix for the teardown ownership guard
# Trees are pristine 'git archive' extracts; only bin/fm-teardown.sh is swapped per column.
# base   = 6789876 (upstream head, no guard)
# mid    = 1097082 (guard ported, before the review fix)
# target = 7213b9b (guard + review fix: ignore returned predecessors)

## New tests vs base script (no guard):
--- test_rerun_after_return_and_reissue_never_rereturns:
not ok - rerun-reissue: the completed return left no durable marker for the rerun
--- test_first_run_still_returns_worktree:
ok - a first-run teardown of an owned worktree still returns it to the pool
--- test_rerun_still_bound_unreturned_returns:
ok - a rerun with the worktree still bound to this task and un-returned performs the return
--- test_reissued_slot_without_marker_skips_return:
not ok - reissue-no-marker: the reissued slot was still returned: return --force /var/folders/5q/snmnm0_926ddy6xlmggzx1g80000gn/T//fm-teardown-tests.3jBoAz/reissue-no-marker/wt
--- test_returned_predecessor_binding_does_not_skip_current_return:
ok - a returned predecessor's stale binding does not block the current tenant's own return

## New tests vs mid script (1097082, before the review fix):
--- test_returned_predecessor_binding_does_not_skip_current_return:
not ok - predecessor-returned: the owned slot was never returned: 
--- test_rerun_after_return_and_reissue_never_rereturns:
ok - a rerun after a completed return and a reissued slot closes the pane and cleans records without re-returning

## Reading:
- The incident replay and the lost-marker reissue case FAIL without the guard: they pin the guard itself.
- The first-run and still-bound-rerun cases pass with and without the guard: they pin the non-skip side (the guard must not over-trigger); they fail only if the guard wrongly skips an owned return.
- The returned-predecessor case FAILS at 1097082 and passes at target: it pins the review fix that ignores marker-bearing stale bindings.

## Known pre-existing local failure (not caused by this change):
test_herdr_flat_teardown_preflight_refuses_before_changes (herdr-preflight-missing-adapter) fails identically on the pure base commit on this host:
not ok - herdr-preflight-missing-adapter: teardown continued without its required preflight
Evidence: Rerun stdout (the guard's one-line skip followed by clean completion)
teardown: skipping worktree return and every other worktree step for task-x1: this task's return already succeeded on a previous attempt
teardown task-x1 complete (window default:wG:pQ, worktree .../rerun-reissue/wt)
- Outcome: ⚠️ 2 infos across 1 run (9m6s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ⚠️ bin/fm-teardown.sh:2310 - Signal (b) of the tenant-ownership guard can false-positive on a PREDECESSOR's stale meta: worktree_reissued_to_other_task (loop at bin/fm-teardown.sh:2310) treats any other task's meta binding the same worktree path as proof of reissue, without excluding a task that has provably relinquished the slot via its own state/<id>.worktree-returned marker. Concrete reachable sequence, the mirror image of the pinned 2026-08-08 incident: task C returns slot S, touches C's marker, fails on the refused pane close and awaits its sanctioned rerun (C's meta and marker persist by design); the pool reissues S to task A; A finishes and its FIRST teardown runs before C's rerun converges; the scan finds C's meta binding S and sets WORKTREE_OWNED_BY_TASK=0, so A's teardown skips its own safety inspection (the unlanded/dirty-work refusal is silently bypassed), run-abort, worktree reap, branch delete, hook removal, and the pool return itself, then reports complete and erases A's durable records - leaking A's pool lease until manual intervention and leaving A's branch and hook files in the slot for the next tenant. The failure direction is fail-safe (nothing is destroyed, and the skip line names C), but the window is exactly the operational state this fix targets, so every new tenant of an incident slot that tears down before the old rerun converges hits it. A contract-compatible refinement: in the scan, skip any other task whose own <other_id>.worktree-returned marker exists, since that marker proves its binding is a stale record of an already-returned tenancy rather than a live claim; the intent's clause (b) ('another task's meta now records') still holds for marker-less bindings, and the accepted marker-lost fallback is unchanged. Classified ask-user because the two-signal semantics were deliberately ported 'exactly as in the original fix'.

🔧 Fix: ignore returned predecessors when detecting pool slot reissue
1 info still open:

  • ℹ️ bin/fm-teardown.sh:2324 - Residual authorized corner of the captain-approved refinement (documentation only, no action needed): a marker-lost rerun whose successor has ALSO already returned the slot (successor's own worktree-returned marker present while it awaits its sanctioned rerun) no longer sees that marker-bearing binding as proof of reissue, so the rerunning task re-executes the full worktree path and re-returns an idle pool slot. This requires two overlapping degraded states (this task's marker lost - a warn-only failure mode - plus the successor inside the post-return-failure window), never touches a live occupant (any marker-bearing binding provably returned the slot; any live tenant's marker-less meta is still detected), matches the pre-guard baseline behavior for idle slots, and stays behind the safety inspection which re-runs on that path. It is exactly the semantics the captain's fix instruction authorized ('only a task binding the same worktree path WITHOUT its own return marker proves the slot was reissued'), so it is an accepted tradeoff, not a defect.
⚠️ **Test** - 2 infos
  • ℹ️ tests/fm-teardown.test.sh - The intent states all four ported tests were verified to FAIL against the unpatched script, but empirically only two do against the upstream base script (the incident replay and the lost-marker reissue case). test_first_run_still_returns_worktree and test_rerun_still_bound_unreturned_returns pass both before and after the patch: they pin the non-skip side of the contract (the guard must not over-trigger), which only the patched script could violate. The guard is still non-vacuously pinned; only the narrative claim is imprecise.
  • ℹ️ tests/fm-teardown.test.sh - tests/fm-teardown.test.sh aborts locally at test_herdr_flat_teardown_preflight_refuses_before_changes (herdr-preflight-missing-adapter) before reaching the new tests. I reproduced the identical failure on the pure base commit 6789876 on this host, confirming it is a pre-existing local-environment issue not caused by this change, exactly as the intent discloses. The targeted run excluded only this case; remote CI remains the authority for it.
  • bin/fm-test-run.sh tests/fm-teardown.test.sh tests/fm-teardown-endpoint-safety.test.sh at target (endpoint-safety 7/7 ok; main suite aborted at the known herdr-preflight-missing-adapter environment failure before reaching the new tests)
  • single-test run of test_herdr_flat_teardown_preflight_refuses_before_changes on a pristine base-commit (6789876) tree, reproducing the identical failure and proving it pre-existing on this host
  • full tests/fm-teardown.test.sh at target with only the known-bad preflight case excluded: 62/62 ok including all five new guard tests
  • each of the five new tests individually against the unpatched base bin/fm-teardown.sh (6789876): incident replay and bindings-only reissue FAIL as designed; first-run, still-bound-rerun, and returned-predecessor pass (they pin the non-skip side)
  • test_returned_predecessor_binding_does_not_skip_current_return against the pre-review-fix script (1097082): FAILS, pinning the review fix; the incident replay passes there, confirming the ported guard itself
  • manual evidence capture: re-ran the incident replay with fixture cleanup disabled and archived the real fm-teardown.sh transcripts (attempt-1 stderr, rerun stdout, treehouse return log, tenant worktree state, state-dir contents)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

A teardown rerun after a post-return failure (e.g. a refused focus-unsafe
pane close) re-executed every worktree-scoped step, so a slot the pool had
meanwhile reissued to a newer task was reset under the live tenant, killing
its session (observed 2026-08-08).

Ownership is now decided from two provable signals before any worktree step:
a durable state/<id>.worktree-returned marker touched immediately after this
task's own successful return, and the current fleet bindings - another
task's meta recording the same worktree path proves reissue even when the
marker is lost. When either shows the worktree is no longer this task's,
the safety inspection, run-abort, worktree process reap, branch delete,
hook removal, and the return itself are skipped with one line, while
tasktmp reaping, the pane close retry, and durable-record cleanup continue
unchanged. The marker is removed with the rest of the volatile state.

Tests: an end-to-end replay of the incident (return ok, close refused,
slot reissued, rerun converges without re-returning), a plain first-run
return, a rerun with the worktree still bound and un-returned, and the
lost-marker reissue caught by fleet bindings alone. All four fail against
the unpatched script, so they pin the guard rather than passing vacuously.

Verification: bin/fm-lint.sh clean under pinned ShellCheck 0.11.0;
bin/fm-doc-audience-check.sh ok; tests/fm-teardown.test.sh 62/62 and
tests/fm-teardown-endpoint-safety.test.sh 7/7 pass with one pre-existing,
unrelated local-environment failure (herdr-preflight-missing-adapter)
disabled: on this host's bash, sourcing a deliberately removed
backends/herdr.sh from fm-backend.sh kills the shell before the graceful
refusal. That failure reproduces identically at origin/main and is
unrelated to this change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant