diff --git a/.agents/skills/firstmate-orca/SKILL.md b/.agents/skills/firstmate-orca/SKILL.md index 1ab43b34094..11f3562a95a 100644 --- a/.agents/skills/firstmate-orca/SKILL.md +++ b/.agents/skills/firstmate-orca/SKILL.md @@ -25,26 +25,19 @@ Use raw `orca` only when the helper surface cannot answer the inspection questio Work from the current firstmate home or repo root. If `FM_HOME` is set, remember that operational state lives under `$FM_HOME` while the helper scripts still run from this repo's `bin/`. -Before recovering a task onto Orca: +Before inspecting an Orca-backed legacy task: - Confirm the task meets the eligibility contract in `docs/orca-backend.md`; new work must use tmux, Herdr, zellij, or cmux. -- Confirm Orca is intentionally selected for this eligible recovery through `--backend orca` or `FM_BACKEND=orca`. -- Confirm the Orca app is running and the backend readiness checks pass before expecting the respawn to work. - Inspect active `state/*.meta` records before changing backend selection. - Do not set Orca as the durable backend for a home that launches new tasks. - Reconcile watcher wakes before unrelated work, especially if Orca tasks are already in flight. +- Do not respawn or destructively tear down an Orca task while the adapter reports that lifecycle authority capabilities are unavailable. ## Spawn -Use `bin/fm-spawn.sh` for an eligible pre-cutover respawn so firstmate creates the worktree, terminal, metadata, status file, and watcher surface together. -New task spawns refuse `backend=orca` before any owned mutation; `docs/orca-backend.md` owns the rationale and exact legacy eligibility rule. - -After spawn, check the task with firstmate helpers: - -- `bin/fm-peek.sh fm-` for launch failures, trust dialogs, or first output. -- `state/.meta` for `backend=orca`, `terminal=`, `orca_worktree_id=`, and `worktree=`. -- `bin/fm-crew-state.sh ` when the current run state matters. -- `bin/fm-watch.sh` whenever there are tasks in flight and this session owns supervision. +New task spawns refuse `backend=orca` before any owned mutation. +Legacy respawn also fails closed until an empirically verified provider capability can bind terminal, task label, worktree, and repository identity and enumerate every terminal attached to a worktree. +`docs/orca-backend.md` owns the evidence and re-enablement contract. Do not manually create the Orca worktree or terminal for an eligible firstmate recovery. Do not manually patch metadata to make an externally-created Orca terminal look like a firstmate task. @@ -74,18 +67,12 @@ For a messy Orca-backed task: 5. Avoid raw deletion of Orca worktrees or manual branch cleanup. 6. Stop and inspect if the recorded worktree path, Orca worktree id, or project checkout no longer matches expectations. -Teardown remains governed by `docs/orca-backend.md`'s eligible legacy task contract; follow its report and landing rules exactly. +Teardown remains governed by `docs/orca-backend.md` and retains resources when lifecycle authority is unavailable. ## Regression Verification Do not synthesize legacy metadata or spawn a disposable Orca task merely to smoke-test the backend. Run the focused fake-Orca suite in `docs/orca-backend.md` to verify lifecycle plumbing. -When an actual eligible legacy recovery is already required, verify it through the normal lifecycle: - -1. Confirm the task metadata has no `report_required` marker and select Orca for that recovery. -2. Respawn through `bin/fm-spawn.sh`. -3. Confirm metadata records the Orca backend, terminal, Orca worktree id, and isolated worktree path. -4. Verify `bin/fm-peek.sh`, a short `bin/fm-send.sh` steer, watcher wake behavior, and `bin/fm-crew-state.sh`. -5. Tear down through `bin/fm-teardown.sh` after the task is safely disposable or landed. +Do not use an actual legacy recovery to assert lifecycle safety until the required authority capability has empirical evidence. Do not mix adapter verification with unrelated feature work. diff --git a/.agents/skills/secondmate-provisioning/SKILL.md b/.agents/skills/secondmate-provisioning/SKILL.md index 879e651f5d9..131d977e847 100644 --- a/.agents/skills/secondmate-provisioning/SKILL.md +++ b/.agents/skills/secondmate-provisioning/SKILL.md @@ -94,6 +94,8 @@ Run `bin/fm-home-seed.sh validate` when checking registry integrity; it refuses Seeding is transactional. If validation, cloning, no-mistakes initialization, or registry update fails, generated briefs, new homes, new project clones, and registry edits are rolled back. +Home provisioning, child spawn, and retirement use one canonical home lifecycle lock. +The lock order is home lifecycle, task lifecycle, then registry; recursive retirement acquires parent homes before child homes and holds each home lock through its final state check, removal, and registry update. Secondmate project lists may include `no-mistakes` and `direct-PR` projects only. `local-only` projects stay with the main firstmate. @@ -159,11 +161,12 @@ Run `bin/fm-teardown.sh ` for `kind=secondmate` only when the captain or mai The safety check is the secondmate's own home. Teardown refuses while its `state/*.meta` contains in-flight work. +Teardown recursively proves every registered project clone and nested repository clean, stash-free, bound to the corresponding durable source remote, and landed on a surviving remote branch; tags and in-home remotes never count. When safe, teardown kills the direct tmux window, removes the `data/secondmates.md` route, clears the main home metadata, and removes the retired secondmate home. Removing a leased home releases its durable treehouse lease via `treehouse return`, so the pool slot is freed for reuse rather than left leased forever. A plain-clone home with no pool slot is simply removed. If `treehouse return` fails for a leased home, teardown stops with state intact rather than raw-removing the directory and hiding a held lease. -With `--force`, teardown is the explicit discard path. -It kills child windows, discards child work and state inside the secondmate home, removes the route, releases the lease, and removes the retired secondmate home. -Never use `--force` unless the captain explicitly said to discard the work. +With `--force`, teardown may recursively retire child tasks and nested secondmates. +Every child must first pass the same identity, endpoint-absence, cleanliness, stash, and landed-work proofs; `--force` never authorizes discarding work. +If any child proof fails, retain the child metadata, worktree, home, and parent registration. diff --git a/.gitignore b/.gitignore index 09d38e81ac4..c8de1b85716 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ __pycache__/ .env config/crew-harness config/crew-dispatch.json +config/checkout-refresh config/account-routing-mode config/secondmate-account-pool config/secondmate-harness diff --git a/AGENTS.md b/AGENTS.md index 9427247d769..1d5dcca6ba9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,13 +25,13 @@ Hard rules, in priority order: 1. **Never write to a project.** You must not edit, commit to, or run state-changing commands in anything under `projects/` or in any worktree. You read projects to understand them; crewmates change them. - Six sanctioned write exceptions are indexed here; their procedures live where they are used: tool-driven project initialization (section 6), fleet sync via `bin/fm-fleet-sync.sh` (sections 3, 7, and 8), local-HEAD secondmate sync via `bin/fm-bootstrap.sh` and `bin/fm-spawn.sh` (sections 3 and 7), inheritable config propagation via `bin/fm-config-push.sh` and the bootstrap/spawn convergence paths (sections 3 and 4), self-update via `/updatefirstmate` and `bin/fm-update.sh` (section 12), and approved `local-only` merge via `bin/fm-merge-local.sh` (section 7). + Six sanctioned write exceptions are indexed here; their procedures live where they are used: tool-driven project initialization (section 6), checkout refresh via `bin/fm-checkout-refresh.sh` and `bin/fm-fleet-sync.sh` (sections 3, 7, and 8), local-HEAD secondmate sync via `bin/fm-bootstrap.sh` and `bin/fm-spawn.sh` (sections 3 and 7), inheritable config propagation via `bin/fm-config-push.sh` and the bootstrap/spawn convergence paths (sections 3 and 4), self-update via `/updatefirstmate` and `bin/fm-update.sh` (section 12), and approved `local-only` merge via `bin/fm-merge-local.sh` (section 7). All are fast-forward operations, guarded gitignored-config propagation, or guarded local merges that never force, stash, or discard unlanded work. Project `AGENTS.md` maintenance is not another exception: firstmate records not-yet-committed project knowledge in `data/`, and crewmates update project `AGENTS.md` through normal delivery (section 6). 2. **Never merge a PR without the captain's explicit word.** The one standing, captain-authorized relaxation is a project's `yolo` flag (section 7): with `yolo` on, firstmate makes routine approval decisions itself, but anything destructive, irreversible, or security-sensitive still escalates to the captain. 3. **Never tear down a worktree that holds unlanded work.** - `bin/fm-teardown.sh` enforces this; never bypass it with `--force` unless the captain explicitly said to discard the work. + `bin/fm-teardown.sh` enforces this, and `--force` only requests recursive cleanup after every ordinary safety proof succeeds. Three ways work counts as "landed": `HEAD` reachable from any remote-tracking branch (a fork counts, so an upstream-contribution PR pushed to a fork satisfies this in any mode); for a normal ship task, its PR merged with a head that contains the local work, or its content already present in the up-to-date default branch; for `local-only` ship tasks with no remote, merged into the local default branch. Uncommitted changes are never landed. The scout carve-out: a scout task's worktree is declared scratch from the start - its deliverable is the report, and teardown lets the worktree go once the applicable report contract is satisfied (section 7; `docs/report-stack.md`). @@ -78,9 +78,11 @@ tools/ independently versioned provider-neutral components, commit .env optional X-mode pairing token; LOCAL, gitignored; presence-gates section 14 config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "default" = same as firstmate. Inherited as the literal file: a concrete primary adapter value also controls a secondmate home's own crewmates (section 4) config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes +config/checkout-refresh optional extra checkout and shallow scan-root directives for this home's safe checkout refresher; LOCAL, gitignored; see docs/configuration.md "Checkout refresh" config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line (" [] []"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates) -config/account-routing-mode optional Agent Fleet routing policy (`off`, `observe`, or `enforce`); LOCAL, gitignored; default off; inherited by secondmate homes (docs/configuration.md "Agent Fleet account routing") +config/account-routing-mode optional account routing policy (`off`, `observe`, or `enforce`); new ship/scout observe/enforce launches use direct account directories, secondmate launches retain legacy Agent Fleet routing, and existing managed metadata retains legacy Agent Fleet recovery; LOCAL, gitignored; default off; inherited by secondmate homes (docs/configuration.md "Agent Fleet account routing") config/secondmate-account-pool optional Agent Fleet pool the PRIMARY uses for SECONDMATE launches when routing is enabled; LOCAL, gitignored; selection-only and NOT inherited +Direct account-directory launch currently covers ship/scout crewmates only; secondmate integration is deferred to a dedicated follow-up. config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = default tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10) config/backend runtime session-provider backend override for new tasks; LOCAL, gitignored; absent = falls through to runtime auto-detection (the runtime firstmate itself is executing inside), then tmux; tmux is the verified reference backend, herdr/zellij/cmux are experimental new-task spawn backends, and Orca is legacy-recovery-only (docs/tmux-backend.md, docs/herdr-backend.md, docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md) - herdr and cmux can also be selected by runtime auto-detection, zellij and Orca never are (always explicit), and codex-app is not accepted; see docs/codex-app-backend.md; not inherited into secondmate homes config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup") @@ -102,7 +104,12 @@ state/ volatile runtime signals; gitignored .status appended by crewmates: ": " wake-event lines, not current-state truth .turn-ended touched by turn-end hooks .grok-turnend-token firstmate-owned grok hook registry token for the task; removed by teardown - .meta written by fm-spawn: window=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=, generation_id=, report_required=; managed tasks add account_pool=, account_profile=, account_task=, account_attempt=, and provider_session_id= (docs/configuration.md "Agent Fleet account routing"); kind=secondmate also records home= and projects=; a non-default runtime backend records further backend-specific fields (docs/configuration.md "Runtime backend"; bin/fm-backend.sh, section 8); fm-pr-check, including through fm-pr-merge, appends pr= and GitHub's pr_head= when available; fm-x-link appends x_request=, x_request_ts=, x_followups=, and optional x_platform=/x_reply_max_chars= for an X-mode-originated task (section 14) + .meta written by fm-spawn: window=, worktree=, project=, harness=, model=, effort=, kind=, mode=, yolo=, tasktmp=, generation_id=, report_required= + Direct ship and scout launches also own account_home=, worktree_git_dir=, worktree_git_dir_identity=, and exactly one authoritative final-state field: worktree_git_ref= for an attached branch or worktree_git_head= for an intentional detached HEAD. Their metadata may temporarily carry worktree_git_setup_ref= and worktree_git_setup_head= while the brief's required `fm/` branch transition is pending; recovery accepts only that exact setup state or the authoritative `fm/` ref and removes the setup fields after adoption. + Direct recovery validates the canonical worktree path, exact physical Git-dir identity, and authoritative final state before account preparation and again immediately before endpoint creation; any drift fails closed without launching. + If endpoint removal after a failed new direct spawn cannot be confirmed, direct_spawn_cleanup=pending and rollback_pending=1 retain the endpoint and worktree identity for explicit teardown. + Secondmate Agent Fleet routing and legacy managed recovery own account_pool=, account_profile=, account_task=, account_attempt=, and provider_session_id= (docs/configuration.md "Agent Fleet account routing"). + kind=secondmate also records home= and projects=; a non-default runtime backend records further backend-specific fields (docs/configuration.md "Runtime backend"; bin/fm-backend.sh, section 8); fm-pr-check, including through fm-pr-merge, appends pr= and GitHub's pr_head= when available; fm-x-link appends x_request=, x_request_ts=, x_followups=, and optional x_platform=/x_reply_max_chars= for an X-mode-originated task (section 14) .check.sh optional slow poll you write per task (e.g. merged-PR check) x-watch.check.sh generated X-mode relay poll shim; present only when opted in (section 14) x-inbox/ generated X-mode pending mention payloads; fmx-respond drains it (section 14) @@ -159,7 +166,7 @@ Tell the captain another active session is already managing the work and operate Bootstrap is detect, then consent, then install. Never install anything the captain has not approved in this session. -The locked fleet-sync sweep runs via `bin/fm-fleet-sync.sh`, best-effort and non-fatal, under the hard-rule exception in section 1. +The locked fleet-sync sweep runs via `bin/fm-checkout-refresh.sh`, which discovers the covered checkout set and delegates each safe fast-forward to `bin/fm-fleet-sync.sh`, best-effort and non-fatal, under the hard-rule exception in section 1. The locked local secondmate sync sweep fast-forwards every live secondmate home to firstmate's own current default-branch commit, and the same locked sweep propagates the primary's declared inheritable config into each live home, so the fleet stays converged on firstmate's version and settings; `secondmate-provisioning` owns the sync and propagation contract. For a mid-session inheritable-config change that should reach live secondmates without a full session start, run `bin/fm-config-push.sh`. Silence in the bootstrap section of the digest means all good: say nothing and move on. @@ -207,8 +214,9 @@ That refusal is the consultation backstop, so the rules are never silently skipp The requirement is gated only on the file's presence; when the file is absent, `fm-spawn.sh` keeps resolving the crewmate harness from `config/crew-harness` as before. Secondmate launches are exempt because they resolve through `fm-harness.sh secondmate`, not the crewmate dispatch-profile rules. -`quota-balanced` selection is deterministic and owned by `bin/fm-dispatch-select.sh`; its header documents the general-window rules, Agent Fleet pool-summary path, freshness margin, and every fallback, and it degrades to the first array element whenever quota data is unusable. -When candidates carry account pools, the selector uses only Agent Fleet summaries and never falls through to quota-axi's default-account data; pass its selected pool to spawn for the atomic concrete-profile lease. +`quota-balanced` selection is deterministic and owned by `bin/fm-dispatch-select.sh`; every real new ship/scout launch uses direct account-directory selection. +When candidates carry account pools, the selector chooses the ordered first profile and passes its pool to spawn only as a compatibility activation input, never as an account choice or lease request. +The now-unreachable Agent Fleet pool-summary implementation and inactive new-lease fixtures are legacy code deferred to follow-up task `remove-fleet-routing-deadcode`. Quota trouble must never block dispatch. Precedence, highest first: @@ -255,6 +263,8 @@ Reconcile reality with your records before doing anything else, working from the Do not sweep every `fm-*` tmux window, herdr tab, zellij tab, Orca terminal, or cmux workspace across all sessions during recovery; another firstmate home's child endpoints may share that namespace and are not this home's orphans. 5. If the digest reports a recorded direct-report's endpoint as `dead` (or a meta has no `window=`), reconcile it through its meta as described below. 6. For meta with no window, or an endpoint the digest reported dead, reconcile by kind. + If ship/scout meta records `account_home=`, relaunch with `bin/fm-spawn.sh --recover-direct-account`. + That mode reloads the recorded task context and performs fresh direct account-directory selection. If meta records `account_profile=`, first try the exact sticky session with `bin/fm-spawn.sh --resume-account`; when that session/profile is unavailable, use `--continue-account` for the task-owned provider-neutral handoff after re-verifying live and repository state, with `bin/fm-account-continuation.sh` owning the fail-closed packet contract. For ordinary crewmates, check the recorded backend metadata first; use `treehouse status` for treehouse-backed tasks, and the recorded `orca_worktree_id=`/`terminal=` for Orca tasks. For an unmanaged `kind=secondmate`, load `secondmate-provisioning`; its "Recovery" section owns the explicit routing decision and respawn procedure. @@ -429,13 +439,14 @@ Load `harness-adapters` before spawning or recovering any direct report so trust ```sh bin/fm-spawn.sh projects/ # uses the active crewmate harness only when no crew-dispatch.json is active bin/fm-spawn.sh projects/ --harness codex --model gpt-5.5 --effort high # explicit profile axes -bin/fm-spawn.sh projects/ --harness codex --account-pool codex-crew # dynamic Agent Fleet account pool -bin/fm-spawn.sh projects/ --harness claude --account-profile claude-2 # explicit Agent Fleet profile -bin/fm-spawn.sh --resume-account # sticky managed recovery; never a fresh prompt -bin/fm-spawn.sh --continue-account # fresh managed session from verified task-owned continuation state +bin/fm-spawn.sh projects/ --harness codex --account-pool codex-crew # compatibility flag activating direct account-directory selection +bin/fm-spawn.sh projects/ --harness claude --account-profile claude-2 # compatibility flag activating direct account-directory selection +bin/fm-spawn.sh --recover-direct-account # metadata-preserving ship/scout direct account recovery with fresh selection +bin/fm-spawn.sh --resume-account # sticky legacy managed recovery; never a fresh prompt +bin/fm-spawn.sh --continue-account # fresh legacy managed session from verified task-owned continuation state bin/fm-spawn.sh projects/ --backend # explicit new-task runtime backend (docs/configuration.md "Runtime backend") bin/fm-spawn.sh projects/ --scout # scout task; records kind=scout in meta -bin/fm-spawn.sh [] --secondmate # launch or recover a persistent secondmate in its home +bin/fm-spawn.sh [] --secondmate # launch a persistent secondmate in its home bin/fm-spawn.sh =projects/ =projects/ [--scout] # batch: one call, several tasks ``` @@ -528,7 +539,9 @@ bin/fm-teardown.sh ``` The script refuses if the worktree holds uncommitted changes or committed work that has not landed; treat a refusal as a stop-and-investigate, not an obstacle. -For a task whose metadata carries `report_required=1`, teardown quiesces the endpoint, runs non-destructive safety checks, and publishes the validated completion report before releasing the account lease or removing the worktree; a safety refusal after quiescence leaves the endpoint stopped while preserving all task state for repair and retry. +Teardown validates that the recorded project and worktree are exact roots with the expected repository registration, quiesces every ordinary task endpoint, and then runs the final non-destructive safety checks before any Treehouse return. +For a task whose metadata carries `report_required=1`, teardown also publishes the validated completion report before releasing the account lease or removing the worktree. +A safety refusal after quiescence leaves the endpoint stopped while preserving all task state for repair and retry. `bin/fm-teardown.sh`'s header owns the full landed-work definition (remote-reachable, merged-PR-head containment for the squash-merge-then-delete-branch flow, content already in the default branch, local-only merges) and the `pr=` discovery fallback for merges that skipped `bin/fm-pr-check.sh`. Known benign case: after an external-PR task, a squash merge leaves the branch commits reachable only on the contributor's fork; add the fork as a remote and fetch (`git remote add fork && git fetch fork`), then retry - never reach for `--force`. A successful PR-based teardown also refreshes that project's clone through `bin/fm-fleet-sync.sh`, best-effort. @@ -542,7 +555,8 @@ An empty queue is healthy and does not trigger teardown. Run `bin/fm-teardown.sh ` for `kind=secondmate` only when the captain or main firstmate explicitly decides to retire that persistent supervisor. Load `secondmate-provisioning` before retiring it. The safety check is the secondmate's own home: teardown refuses while its `state/*.meta` contains in-flight work. -With `--force`, teardown is the explicit discard path for child windows, child work, state, route, lease, and home; never use it unless the captain explicitly said to discard the work. +With `--force`, teardown may recursively retire children only after every identity, endpoint-absence, cleanliness, stash, and landed-work proof succeeds. +It never authorizes discarding child or parent work. ### Scout tasks (report instead of PR) @@ -550,7 +564,7 @@ A scout task follows Intake, Spawn, and Supervise exactly as above - scaffold th - There is no Validate or PR-ready stage. When the crewmate's status says `done`, read `data//report.md`. - Relay the findings to the captain: plain chat for a focused answer, lavish-axi when the report has structure worth a visual (multiple findings, options, a plan). -- Tear down immediately - no merge gate. For a post-cutover scout, `bin/fm-teardown.sh` requires the report's completion sections and publishes it before discarding the scratch worktree; a missing or incomplete report refuses teardown because the findings are the work product. +- Tear down immediately - no merge gate. For a post-cutover scout, `bin/fm-teardown.sh` requires the report's completion sections and publishes it before removing the declared scratch worktree; a missing or incomplete report refuses teardown because the findings are the work product. - Record it in Done with the report path instead of a PR link using `tasks-axi done` when the default tasks-axi backend is active and compatible, otherwise hand-edit `data/backlog.md` and keep Done to the 10 most recent, then re-evaluate the queue and dispatch only queued work whose blockers are gone and whose time/date gate, if any, has arrived. When the captain invokes `/reports` or asks to browse, open, search, or summarize completed work, load the `reports` skill. @@ -617,7 +631,8 @@ On wake, in order of cheapness: Do not report that the fleet is unchanged. When a task reaches a terminal state on any of these wakes (a `done`/merge `check:`, a `failed` signal, a scout report, a local-only merge), and X mode is enabled, load `fmx-respond` (section 13) and post the X-mode mention's **final** completion follow-up if that task is X-mode-linked: `bin/fm-x-followup.sh --check ` then `bin/fm-x-followup.sh --final --text-file `, so the link always clears here regardless of how many of the up-to-three follow-ups were already spent on earlier milestones. -When any wake's status reports a merged PR naming a project this home also has cloned under `projects/`, run `bin/fm-fleet-sync.sh ` for that project as part of handling the wake, so the primary's clone never sits stale until the next session start or teardown. +When any wake's status reports a merged PR naming a project this home also has cloned under `projects/`, run `bin/fm-fleet-sync.sh ` for that project as the low-latency fast path. +The installed home-scoped `fm-checkout-refresh.sh` owner independently detects every tracked upstream-tip change, surfaces new or growing non-ignored untracked skill-draft inventories, and supplies the periodic backstop across projects, Treehouse backing checkouts, configured paths, and matching-origin top-level clones. Never rely on hooks or status files alone; when a heartbeat wake does reach you, the review of every window is mandatory and unconditional. Each task's backend live-task inventory is the ground truth: tmux when `backend=` is absent, or the non-default `backend=` a task's meta records (`docs/configuration.md` "Runtime backend" owns the backend set). diff --git a/bin/backends/cmux.sh b/bin/backends/cmux.sh index c6958a5eefc..057fedbd6a5 100644 --- a/bin/backends/cmux.sh +++ b/bin/backends/cmux.sh @@ -590,8 +590,8 @@ fm_backend_cmux_send_key() { # [expected-label] # fm_backend_cmux_send_text_line: send one line of TEXT then submit. cmux has # no single-call atomic "run and submit" primitive (like herdr's `pane run`), # so this composes send (literal) + send-key enter, exactly like zellij's -# equivalent - used for the fixed spawn-time commands (treehouse get, the -# GOTMPDIR export). +# equivalent - used for the fixed spawn-time commands (the GOTMPDIR export and +# harness launch). fm_backend_cmux_send_text_line() { # [expected-label] fm_backend_cmux_send_literal "$1" "$2" "${3:-}" || return 1 fm_backend_cmux_send_key "$1" Enter "${3:-}" diff --git a/bin/backends/herdr.sh b/bin/backends/herdr.sh index ee467a7414a..d8029e6f1f6 100644 --- a/bin/backends/herdr.sh +++ b/bin/backends/herdr.sh @@ -1904,8 +1904,20 @@ fm_backend_herdr_server_ensure() { # case "$exact_state" in empty) ;; occupied) - echo "error: refusing to restart release-drifted Herdr session '$session' because its exact workspace/tab/pane state is occupied" >&2 - exit 1 + # The server is this adapter's own (checked above) and running, but its + # closed-shell certificate drifted - typically because an fm-update or a + # landed source change altered the managed-shell-source digest after the + # server was launched, or because FirstMate itself runs inside this herdr + # session (HERDR_ENV=1) and the server was launched by the interactive + # launcher rather than the crew adapter's own closed-shell path. It is + # OCCUPIED (live crews and/or FirstMate), so it cannot be restarted to + # re-certify. Refusing here would impose a hard no-spawn state on an + # otherwise healthy, adapter-owned server. Accept it: every new crew pane + # is individually env-scrubbed by fm-herdr-worker-shell at pane creation, + # so a drifted SERVER certificate does not weaken a NEW pane's isolation. + # The strict certified restart still applies whenever the session is empty + # (the 'empty' arm above), so the boundary is preserved when it can be. + exit 0 ;; *) echo "error: refusing to restart release-drifted Herdr session '$session' because its exact workspace/tab/pane state is indeterminate" >&2 @@ -2559,24 +2571,43 @@ fm_backend_herdr_expected_label_matches() { # [expected-label] [ "$(fm_backend_herdr_identity_state "$1" "${2:-}")" = match ] } +# fm_backend_herdr_server_reachable_for_readsteer: the lighter precondition for +# reading from or sending to an ALREADY-RUNNING pane, as distinct from the +# launch-grade fm_backend_herdr_server_ensure used to spawn new crews. +# +# Reading a pane or sending keys to it does not care how the server was +# launched - the pane already exists and the server is up. It only needs the +# server to be running and adapter-owned (this HOME's own certificate names the +# live pid), which fm_backend_herdr_server_adapter_owned proves without the +# closed-shell launch certification. This is what keeps peek/steer working when +# FirstMate itself runs INSIDE the herdr session it manages (HERDR_ENV=1): that +# server was not launched through the crew adapter's own closed-shell path, so +# closed_shell_environment_ready is false and the full ensure would try to +# restart+recertify - impossible while the session is occupied by live crews and +# FirstMate itself. The SPAWN path deliberately keeps the strict ensure so a new +# crew still launches only in a certified closed-shell server. +fm_backend_herdr_server_reachable_for_readsteer() { # + local session=$1 running + running=$(fm_backend_herdr_cli "$session" status --json 2>/dev/null \ + | fm_backend_herdr_control_jq -r '.server.running // false' 2>/dev/null) + [ "$running" = true ] || return 1 + fm_backend_herdr_server_adapter_owned "$session" +} + fm_backend_herdr_target_ready() { # [expected-label] fm_backend_herdr_parse_target "$1" || return 1 - fm_backend_herdr_server_ensure "$FM_BACKEND_HERDR_SESSION" || return 1 + fm_backend_herdr_server_reachable_for_readsteer "$FM_BACKEND_HERDR_SESSION" || return 1 fm_backend_herdr_expected_label_matches "$1" "${2:-}" || return 1 } # fm_backend_herdr_current_path: the live FOREGROUND process's cwd, or empty on -# any error. Mirrors tmux's pane_current_path poll used for worktree-path -# discovery after `treehouse get`. +# any error. Mirrors tmux's pane_current_path poll used to verify the endpoint +# started in its leased worktree. # # Verified pitfall: `pane get`'s `.result.pane.cwd` is the pane's cwd AT -# CREATION TIME - the top-level shell's cwd - and does NOT update when that -# shell `cd`s or enters a subshell (as `treehouse get` does). Reading it here -# would make fm-spawn.sh's worktree-discovery poll never see the pane "leave" -# the project directory, since `cwd` stays frozen at the original path forever. +# CREATION TIME and does not follow later shell directory changes. # `.result.pane.foreground_cwd` tracks the ACTUALLY RUNNING foreground -# process's cwd instead, which is what changes when `treehouse get` enters its -# worktree subshell - confirmed live against a real treehouse acquisition. +# process's cwd instead, which is what identifies the live task directory. fm_backend_herdr_current_path() { # fm_backend_herdr_target_ready "$1" || return 0 fm_backend_herdr_cli "$FM_BACKEND_HERDR_SESSION" pane get "$FM_BACKEND_HERDR_PANE" 2>/dev/null \ @@ -2585,7 +2616,7 @@ fm_backend_herdr_current_path() { # # fm_backend_herdr_send_text_line: send one line of TEXT then submit, # ATOMICALLY - mirrors tmux's `send-keys -t T text Enter`. Used for the fixed -# spawn-time commands (treehouse get, the GOTMPDIR export). `pane run` types +# spawn-time commands (the GOTMPDIR export and harness launch). `pane run` types # the command and submits it in one call (verified). fm_backend_herdr_send_text_line() { # fm_backend_herdr_target_ready "$1" || return 1 diff --git a/bin/backends/orca.sh b/bin/backends/orca.sh index dc9307de4f6..7f6bc1b1ee7 100644 --- a/bin/backends/orca.sh +++ b/bin/backends/orca.sh @@ -45,10 +45,20 @@ const state = runtime.state || r.runtimeState || ""; if (reachable === true && state === "ready") process.exit(0); console.error(`error: backend=orca requires a ready Orca runtime (reachable=${String(reachable)}, state=${state || "unknown"})`); process.exit(1); -' +' || return 1 + fm_backend_orca_authority_capabilities_check +} + +fm_backend_orca_authority_capabilities_check() { + if [ "${FM_ORCA_TEST_LAB:-}" = firstmate-orca-test-lab-v1 ] \ + && [ "${FM_ORCA_TEST_AUTHORITY_CAPABILITIES:-}" = verified-v1 ]; then + return 0 + fi + echo "error: Orca lifecycle authority is disabled because no empirically verified terminal/worktree identity and inventory capability is available" >&2 + return 1 } -fm_backend_orca_json_get() { # ; fields: worktree-id worktree-path terminal-handle worktree-terminal-handle repo-id +fm_backend_orca_json_get() { # ; fields: worktree-id worktree-path terminal-handle terminal-title worktree-terminal-handle repo-id # Terminal handles are accepted only from verified terminal result shapes: # result.terminal or a root terminal object with .handle. Undocumented # result.id and result.worktree.terminal shapes are ignored until a real Orca @@ -79,8 +89,10 @@ let v = ""; if (field === "worktree-id") v = wt.id || wt.worktreeId || r.worktreeId || ""; if (field === "worktree-path") v = wt.path || (wt.git && wt.git.path) || r.path || ""; if (field === "terminal-handle") v = handle(explicitTerm || r) || ""; +if (field === "terminal-title") v = scalar((explicitTerm || r).title) || scalar((explicitTerm || r).name) || ""; if (field === "worktree-terminal-handle") v = handle(explicitTerm) || ""; -if (field === "repo-id") v = repo.id || repo.repoId || r.repoId || ""; +if (field === "repo-id") v = repo.id || repo.repoId || wt.repoId || + (wt.repo && (wt.repo.id || wt.repo.repoId)) || r.repoId || ""; if (!v) process.exit(1); process.stdout.write(String(v)); ' "$field" @@ -129,29 +141,23 @@ fm_backend_orca_repo_ensure() { # } fm_backend_orca_worktree_create() { # - local project=$1 name=$2 repo_id out wt_id wt_path terminal + local project=$1 name=$2 repo_id out wt_id wt_path terminal status proof repo_id=$(fm_backend_orca_repo_ensure "$project") || return 1 - out=$(orca worktree create --repo "id:$repo_id" --name "$name" --no-parent --setup skip --json) || return 1 - wt_id=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-id) || { - echo "error: orca worktree create did not return a worktree id for $name" >&2 - return 1 - } + if out=$(orca worktree create --repo "id:$repo_id" --name "$name" --no-parent --setup skip --json); then + status=0 + else + status=$? + fi + wt_id=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-id 2>/dev/null || true) terminal=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-terminal-handle 2>/dev/null || true) - wt_path=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-path) || { - echo "error: orca worktree create did not return a path for $name" >&2 - [ -z "$terminal" ] || fm_backend_orca_kill "$terminal" >/dev/null 2>&1 || true - if fm_backend_orca_remove_worktree "$wt_id" >/dev/null; then - return 1 - fi - if [ -n "$terminal" ]; then - printf '%s\t\t%s' "$wt_id" "$terminal" - else - printf '%s\t' "$wt_id" - fi + wt_path=$(printf '%s' "$out" | fm_backend_orca_json_get worktree-path 2>/dev/null || true) + proof=unproven + [ -z "$terminal" ] || proof=recorded + printf '%s\t%s\t%s\t%s\t%s' "$wt_id" "$wt_path" "$terminal" "$proof" "$repo_id" + if [ "$status" -ne 0 ] || [ -z "$wt_id" ] || [ -z "$wt_path" ]; then + echo "error: orca worktree create returned incomplete or unsuccessful authority for $name" >&2 return 2 - } - printf '%s\t%s' "$wt_id" "$wt_path" - [ -z "$terminal" ] || printf '\t%s' "$terminal" + fi } fm_backend_orca_terminal_create() { # @@ -180,8 +186,34 @@ fm_backend_orca_send_literal() { # <terminal-id> <text> fm_backend_orca_remove_worktree() { # <worktree-id> local worktree_id=${1:-} [ -n "$worktree_id" ] || { echo "error: missing Orca worktree id; cannot remove worktree" >&2; return 1; } + echo "error: unbound Orca worktree removal is disabled" >&2 + return 1 +} + +fm_backend_orca_remove_worktree_bound() { # <worktree-id> <expected-path> <boundary-token> + local worktree_id=${1:-} expected_path=${2:-} boundary_token=${3:-} + [ -n "$worktree_id" ] || { echo "error: missing Orca worktree id; cannot remove worktree" >&2; return 1; } + case "$expected_path" in /*) ;; *) echo "error: missing absolute Orca worktree removal path" >&2; return 1 ;; esac + [ "${#boundary_token}" -eq 64 ] || { + echo "error: missing Orca worktree filesystem-boundary token" >&2 + return 1 + } + case "$boundary_token" in *[!0-9a-f]*) + echo "error: malformed Orca worktree filesystem-boundary token" >&2 + return 1 + ;; + esac + if [ "${FM_ORCA_TEST_LAB:-}" != firstmate-orca-test-lab-v1 ] \ + || [ "${FM_ORCA_TEST_BOUND_REMOVAL_CAPABILITIES:-}" != verified-v1 ]; then + echo "error: Orca worktree removal lacks an identity-bound provider capability" >&2 + return 1 + fi fm_backend_orca_tool_check || return 1 - fm_backend_orca_run_json orca worktree rm --worktree "id:$worktree_id" --force --json + fm_backend_orca_run_json orca worktree rm \ + --worktree "id:$worktree_id" \ + --expected-path "$expected_path" \ + --expected-boundary-token "$boundary_token" \ + --force --json } fm_backend_orca_worktree_path() { @@ -203,6 +235,249 @@ fm_backend_orca_capture() { # <terminal-id> <lines> fm_backend_orca_json_text "$out" } +fm_backend_orca_terminal_state() { # <terminal-id> [expected-worktree-id] [expected-label] -> present|absent|unknown + local terminal=$1 expected_worktree_id=${2:-} expected_label=${3:-} out status + if [ -n "$expected_worktree_id" ] || [ -n "$expected_label" ]; then + fm_backend_orca_authority_capabilities_check >/dev/null 2>&1 || { + printf 'unknown' + return 0 + } + fi + fm_backend_orca_tool_check || { printf 'unknown'; return 0; } + if out=$(orca terminal read --terminal "$terminal" --limit 1 --json 2>/dev/null); then + status=0 + else + status=$? + fi + printf '%s' "$out" | node -e ' +const fs = require("fs"); +const status = Number(process.argv[1]); +const expectedWorktreeId = process.argv[2] || ""; +const expectedLabel = process.argv[3] || ""; +let data; +try { + data = JSON.parse(fs.readFileSync(0, "utf8")); +} catch (_) { + process.stdout.write("unknown"); + process.exit(0); +} +if (data.ok === false) { + const code = data.error && data.error.code; + process.stdout.write(code === "terminal_handle_stale" ? "absent" : "unknown"); + process.exit(0); +} +const r = data.result; +const terminal = r && typeof r === "object" && r.terminal && typeof r.terminal === "object" + ? r.terminal + : null; +const knownTerminal = + r && typeof r === "object" && + (terminal || + Array.isArray(r.tail) || + ["text", "output", "content", "preview"].some((key) => typeof r[key] === "string")); +if (status === 0 && expectedWorktreeId) { + if (!terminal) { + process.stdout.write("unknown"); + process.exit(0); + } + const worktree = terminal.worktree; + const actualWorktreeId = + (typeof terminal.worktreeId === "string" ? terminal.worktreeId : "") || + (typeof worktree === "string" ? worktree : "") || + (worktree && typeof worktree.id === "string" ? worktree.id : "") || + (typeof r.worktreeId === "string" ? r.worktreeId : "") || + (r.worktree && typeof r.worktree.id === "string" ? r.worktree.id : ""); + if (!actualWorktreeId || actualWorktreeId !== expectedWorktreeId) { + process.stdout.write("unknown"); + process.exit(0); + } +} +if (status === 0 && expectedLabel) { + if (!terminal) { + process.stdout.write("unknown"); + process.exit(0); + } + const actualLabel = + (typeof terminal.title === "string" ? terminal.title : "") || + (typeof terminal.name === "string" ? terminal.name : "") || + (typeof r.title === "string" ? r.title : ""); + if (!actualLabel || actualLabel !== expectedLabel) { + process.stdout.write("unknown"); + process.exit(0); + } +} +if (status === 0 && data.ok !== false && knownTerminal) { + process.stdout.write("present"); + process.exit(0); +} +process.stdout.write("unknown"); +' "$status" "$expected_worktree_id" "$expected_label" +} + +fm_backend_orca_worktree_terminal_state() { + local worktree_id=$1 expected_label=${2:-} out + [ -n "$worktree_id" ] || { printf 'unknown'; return 0; } + fm_backend_orca_authority_capabilities_check >/dev/null 2>&1 || { + printf 'unknown' + return 0 + } + fm_backend_orca_tool_check || { printf 'unknown'; return 0; } + out=$(orca worktree show --worktree "id:$worktree_id" --json 2>/dev/null) || { + printf 'unknown' + return 0 + } + printf '%s' "$out" | node -e ' +const fs = require("fs"); +const expected = process.argv[1]; +const expectedLabel = process.argv[2] || ""; +let data; +try { + data = JSON.parse(fs.readFileSync(0, "utf8")); +} catch (_) { + process.stdout.write("unknown"); + process.exit(0); +} +if (data.ok === false) { + process.stdout.write("unknown"); + process.exit(0); +} +const r = data.result || {}; +const worktree = r.worktree || r.item || r; +const id = String(worktree.id || worktree.worktreeId || r.worktreeId || ""); +const label = String(worktree.name || worktree.title || r.worktreeName || ""); +const terminals = Array.isArray(worktree.terminals) + ? worktree.terminals + : (Array.isArray(r.terminals) ? r.terminals : null); +const identitiesValid = terminals !== null && terminals.every((terminal) => { + if (!terminal || typeof terminal !== "object") return false; + const handle = String(terminal.handle || terminal.id || ""); + const label = String(terminal.title || terminal.name || ""); + return Boolean(handle) && (!expectedLabel || label === expectedLabel); +}); +if (id !== expected || (expectedLabel && label !== expectedLabel) || + terminals === null || !identitiesValid) { + process.stdout.write("unknown"); +} else { + process.stdout.write(terminals.length === 0 ? "absent" : "present"); +} +' "$worktree_id" "$expected_label" +} + +fm_backend_orca_worktree_terminals() { + local worktree_id=$1 expected_label=$2 out + [ -n "$worktree_id" ] && [ -n "$expected_label" ] || return 1 + fm_backend_orca_authority_capabilities_check || return 1 + fm_backend_orca_tool_check || return 1 + out=$(orca worktree show --worktree "id:$worktree_id" --json 2>/dev/null) || return 1 + printf '%s' "$out" | node -e ' +const fs = require("fs"); +const expectedId = process.argv[1]; +const expectedLabel = process.argv[2]; +let data; +try { + data = JSON.parse(fs.readFileSync(0, "utf8")); +} catch (_) { + process.exit(1); +} +if (data.ok === false) process.exit(1); +const r = data.result || {}; +const worktree = r.worktree || r.item || r; +const id = String(worktree.id || worktree.worktreeId || r.worktreeId || ""); +const label = String(worktree.name || worktree.title || r.worktreeName || ""); +const terminals = Array.isArray(worktree.terminals) + ? worktree.terminals + : (Array.isArray(r.terminals) ? r.terminals : null); +if (id !== expectedId || label !== expectedLabel || terminals === null) process.exit(1); +for (const terminal of terminals) { + if (!terminal || typeof terminal !== "object") process.exit(1); + const handle = String(terminal.handle || terminal.id || ""); + const label = String(terminal.title || terminal.name || ""); + if (!handle || label !== expectedLabel || /[\r\n]/.test(handle)) process.exit(1); + process.stdout.write(handle + "\n"); +} +' "$worktree_id" "$expected_label" +} + +fm_backend_orca_worktree_terminal_contains() { + local worktree_id=$1 expected_label=$2 expected_terminal=$3 terminals terminal found=0 + [ -n "$expected_terminal" ] || return 1 + terminals=$(fm_backend_orca_worktree_terminals "$worktree_id" "$expected_label") || return 1 + while IFS= read -r terminal; do + [ "$terminal" != "$expected_terminal" ] || found=$((found + 1)) + done <<EOF +$terminals +EOF + [ "$found" -eq 1 ] +} + +fm_backend_orca_quiesce_terminal() { # <terminal-id> [expected-worktree-id] [expected-label] + local terminal=$1 expected_worktree_id=${2:-} expected_label=${3:-} state + [ -n "$terminal" ] || return 1 + if [ -n "$expected_worktree_id" ] || [ -n "$expected_label" ]; then + state=$(fm_backend_orca_terminal_state "$terminal" "$expected_worktree_id" "$expected_label") + [ "$state" = present ] || { + echo "error: Orca terminal $terminal does not match the expected task authority" >&2 + return 1 + } + fi + fm_backend_orca_kill "$terminal" || { + echo "error: failed to close Orca terminal $terminal" >&2 + return 1 + } + for _ in 1 2 3 4 5; do + state=$(fm_backend_orca_terminal_state "$terminal" "$expected_worktree_id" "$expected_label") + [ "$state" != absent ] || return 0 + sleep 0.1 + done + echo "error: Orca terminal $terminal is not proven absent after close" >&2 + return 1 +} + +fm_backend_orca_quiesce_worktree_terminals() { + local worktree_id=$1 expected_label=$2 known_terminal=${3:-} terminals terminal known_state + terminals=$(fm_backend_orca_worktree_terminals "$worktree_id" "$expected_label") || { + echo "error: Orca worktree terminal authority is unproven for $worktree_id" >&2 + return 1 + } + if [ -n "$known_terminal" ]; then + known_state=$(fm_backend_orca_terminal_state "$known_terminal" "$worktree_id" "$expected_label") + case "$known_state" in + present) + fm_backend_orca_worktree_terminal_contains "$worktree_id" "$expected_label" "$known_terminal" || { + echo "error: Orca terminal inventory disagrees with the recorded live terminal for $worktree_id" >&2 + return 1 + } + ;; + absent) + if fm_backend_orca_worktree_terminal_contains "$worktree_id" "$expected_label" "$known_terminal"; then + echo "error: Orca terminal inventory contains a terminal reported absent for $worktree_id" >&2 + return 1 + fi + ;; + *) + echo "error: recorded Orca terminal authority is unproven for $worktree_id" >&2 + return 1 + ;; + esac + fi + while IFS= read -r terminal; do + [ -n "$terminal" ] || continue + fm_backend_orca_quiesce_terminal "$terminal" "$worktree_id" "$expected_label" || return 1 + done <<EOF +$terminals +EOF + [ "$(fm_backend_orca_worktree_terminal_state "$worktree_id" "$expected_label")" = absent ] || { + echo "error: Orca worktree $worktree_id still has terminals or cannot prove their absence" >&2 + return 1 + } + if [ -n "$known_terminal" ]; then + [ "$(fm_backend_orca_terminal_state "$known_terminal" "$worktree_id" "$expected_label")" = absent ] || { + echo "error: recorded Orca terminal $known_terminal is not proven absent" >&2 + return 1 + } + fi +} + fm_backend_orca_json_text() { # <json> printf '%s' "$1" | node -e ' const fs = require("fs"); @@ -331,6 +606,6 @@ fm_backend_orca_send_text_submit() { # <terminal-id> <text> <retries> <enter-sl } fm_backend_orca_kill() { # <terminal-id> - fm_backend_orca_tool_check || return 0 - orca terminal close --terminal "$1" --json >/dev/null 2>&1 || true + fm_backend_orca_tool_check || return 1 + fm_backend_orca_run_json orca terminal close --terminal "$1" --json } diff --git a/bin/backends/tmux.sh b/bin/backends/tmux.sh index 3852cda9f51..31d0171a8f2 100644 --- a/bin/backends/tmux.sh +++ b/bin/backends/tmux.sh @@ -143,7 +143,7 @@ fm_backend_tmux_current_path() { # <target> # fm_backend_tmux_send_text_line: send one line of TEXT then Enter, with no # composer verification - used for the fixed spawn-time commands -# (`treehouse get`, the GOTMPDIR export) that already ran this exact sequence +# (the GOTMPDIR export and harness launch) that already ran this exact sequence # inline in fm-spawn.sh. Mirrors `tmux send-keys -t "$T" "<text>" Enter`. fm_backend_tmux_send_text_line() { # <target> <text> tmux send-keys -t "$1" "$2" Enter diff --git a/bin/backends/zellij.sh b/bin/backends/zellij.sh index 9a3eec59dbf..662c6f83423 100644 --- a/bin/backends/zellij.sh +++ b/bin/backends/zellij.sh @@ -480,8 +480,8 @@ fm_backend_zellij_send_key() { # <target> <key> [expected-label] # fm_backend_zellij_send_text_line: send one line of TEXT then submit, # ATOMICALLY - mirrors tmux's `send-keys -t T text Enter` / herdr's `pane -# run`. Used for the fixed spawn-time commands (treehouse get, the GOTMPDIR -# export). Zellij has no single-call atomic "run and submit" action, so this +# run`. Used for the fixed spawn-time commands (the GOTMPDIR export and harness +# launch). Zellij has no single-call atomic "run and submit" action, so this # composes paste (literal) + send-keys Enter, exactly like send_literal + # send_key are composed elsewhere - the two-step form is the ONLY form for # this adapter, unlike tmux/herdr which have a genuinely atomic primitive. diff --git a/bin/fm-account-directory.sh b/bin/fm-account-directory.sh new file mode 100755 index 00000000000..3ba66b5462c --- /dev/null +++ b/bin/fm-account-directory.sh @@ -0,0 +1,398 @@ +#!/usr/bin/env bash +# Select and prepare direct Claude or Codex account-directory launches. +# Usage: +# fm-account-directory.sh select <claude|codex> +# fm-account-directory.sh install-herdr-hook <claude|codex> <account-home> +# fm-account-directory.sh prepare <claude|codex> +# +# This header is the single owner of the direct account-directory contract. +# FM_ACCOUNT_DIRECTORY_CUTOVER: direct-observe-passwd-home-v2 +# Account homes are discovered under the current passwd user's +# .local/share/agent-fleet/accounts/<vendor>/ tree without fixed counts. +# Codex selection removes that account's quota-axi window cache immediately +# before every read, sets CODEX_HOME plus the account-isolated XDG_CACHE_HOME, +# accepts only a fresh result with at least one numeric five_hour or weekly +# window, and picks the account with the highest minimum remaining percentage. +# A Codex account with no such freshly readable window is skipped as unhealthy. +# Claude quota is not currently distinguishable per config directory because +# quota-axi cannot non-interactively resolve Claude's config-dir-specific macOS +# Keychain credential. +# Claude therefore never treats a missing usage window as account failure and +# selects the first real account directory in stable bytewise sort order. +# Selection prints only the chosen absolute account home on stdout and logs +# health, fallback, and choice diagnostics on stderr. +# prepare selects the account and idempotently runs Herdr's own integration +# installer with CODEX_HOME or CLAUDE_CONFIG_DIR set to the chosen home. +# It verifies the installed per-profile hook before printing the chosen home. +# +# Credential state is read-only. +# This script never logs in, imports credentials, or invokes a provider model. +# Test-only command, root, passwd-home, Perl, and timeout overrides require +# FM_ACCOUNT_DIRECTORY_TEST_LAB=firstmate-account-directory-test-lab-v1. +set -u + +TEST_LAB_TOKEN=firstmate-account-directory-test-lab-v1 + +usage() { + sed -n '2,32p' "$0" | sed 's/^# \{0,1\}//' >&2 +} + +log() { + printf 'fm-account-directory: %s\n' "$*" >&2 +} + +test_lab_enabled() { + [ "${FM_ACCOUNT_DIRECTORY_TEST_LAB:-}" = "$TEST_LAB_TOKEN" ] +} + +system_perl() { + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_PERL_BIN:-}" ]; then + printf '%s\n' "$FM_ACCOUNT_DIRECTORY_PERL_BIN" + else + printf '%s\n' /usr/bin/perl + fi +} + +passwd_home() { + local home perl_bin + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_PASSWD_HOME:-}" ]; then + home=$FM_ACCOUNT_DIRECTORY_PASSWD_HOME + else + perl_bin=$(system_perl) || return 1 + [ -x /usr/bin/env ] && [ -x "$perl_bin" ] || { + echo "error: /usr/bin/env and /usr/bin/perl are required to resolve the current passwd home" >&2 + return 1 + } + # shellcheck disable=SC2016 # Perl source is intentionally single-quoted. + home=$(/usr/bin/env -i PATH=/usr/bin:/bin:/usr/sbin:/sbin "$perl_bin" -e ' + my @p = getpwuid($<); + exit 1 unless @p && defined $p[7] && $p[7] =~ m{^/}; + exit 1 if $p[7] =~ /[\x00-\x1f\x7f]/; + print $p[7]; + ' 2>/dev/null) || { + echo "error: cannot resolve the current passwd home" >&2 + return 1 + } + fi + case "$home" in + *$'\n'*|*$'\r'*) + echo "error: passwd home contains a line break" >&2 + return 1 + ;; + /*) ;; + *) + echo "error: passwd home must be absolute: $home" >&2 + return 1 + ;; + esac + [ -d "$home" ] && [ ! -L "$home" ] || { + echo "error: passwd home is not a real directory: $home" >&2 + return 1 + } + printf '%s\n' "$home" +} + +account_root() { + local root home + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_ROOT:-}" ]; then + root=$FM_ACCOUNT_DIRECTORY_ROOT + else + home=$(passwd_home) || return 1 + root=$home/.local/share/agent-fleet/accounts + fi + case "$root" in + *$'\n'*|*$'\r'*) + echo "error: account-directory root contains a line break" >&2 + return 1 + ;; + /*) ;; + *) + echo "error: account-directory root must be absolute: $root" >&2 + return 1 + ;; + esac + printf '%s\n' "$root" +} + +quota_command() { + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_QUOTA_AXI:-}" ]; then + printf '%s\n' "$FM_ACCOUNT_DIRECTORY_QUOTA_AXI" + return 0 + fi + command -v quota-axi 2>/dev/null || { + echo "error: quota-axi is required for fresh Codex account selection" >&2 + return 1 + } +} + +herdr_command() { + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_HERDR:-}" ]; then + printf '%s\n' "$FM_ACCOUNT_DIRECTORY_HERDR" + return 0 + fi + command -v herdr 2>/dev/null || { + echo "error: herdr is required to install the selected account's integration hook" >&2 + return 1 + } +} + +quota_timeout_seconds() { + local timeout=15 + if test_lab_enabled && [ -n "${FM_ACCOUNT_DIRECTORY_QUOTA_TIMEOUT_SECONDS:-}" ]; then + timeout=$FM_ACCOUNT_DIRECTORY_QUOTA_TIMEOUT_SECONDS + fi + case "$timeout" in + ''|*[!0-9]*|0) + echo "error: Codex quota timeout must be a positive integer" >&2 + return 1 + ;; + esac + printf '%s\n' "$timeout" +} + +run_bounded() { + local timeout=$1 perl_bin + shift + perl_bin=$(system_perl) || return 1 + [ -x "$perl_bin" ] || return 127 + # shellcheck disable=SC2016 # Perl source is intentionally single-quoted. + PERL5LIB='' PERL5OPT='' "$perl_bin" -e ' + use POSIX qw(setpgid WNOHANG); + my ($timeout, @command) = @ARGV; + my $pid = fork(); + exit 125 unless defined $pid; + if ($pid == 0) { + setpgid(0, 0); + exec {$command[0]} @command; + exit 127; + } + setpgid($pid, $pid); + my $deadline = time() + $timeout; + while (1) { + my $waited = waitpid($pid, WNOHANG); + if ($waited == $pid) { + my $status = $?; + exit(($status & 127) ? 128 + ($status & 127) : ($status >> 8)); + } + exit 125 if $waited == -1; + if (time() >= $deadline) { + kill "TERM", -$pid; + for (1 .. 4) { + select undef, undef, undef, 0.05; + exit 124 if waitpid($pid, WNOHANG) == $pid; + } + kill "KILL", -$pid; + waitpid($pid, 0); + exit 124; + } + select undef, undef, undef, 0.05; + } + ' "$timeout" "$@" +} + +valid_account_home() { # <vendor-dir> <candidate> + local vendor_dir=$1 candidate=$2 name + [ -d "$candidate" ] && [ ! -L "$candidate" ] || return 1 + case "$candidate" in + "$vendor_dir"/*) ;; + *) return 1 ;; + esac + name=${candidate##*/} + case "$name" in + ''|.*|*[!A-Za-z0-9._-]*) return 1 ;; + esac +} + +first_account_home() { # <vendor> + local vendor=$1 root vendor_dir candidate + root=$(account_root) || return 1 + vendor_dir=$root/$vendor + [ -d "$vendor_dir" ] && [ ! -L "$vendor_dir" ] || { + echo "error: no account-directory root for $vendor at $vendor_dir" >&2 + return 1 + } + LC_ALL=C + export LC_ALL + for candidate in "$vendor_dir"/*; do + valid_account_home "$vendor_dir" "$candidate" || continue + printf '%s\n' "$candidate" + return 0 + done + echo "error: no account directories found for $vendor under $vendor_dir" >&2 + return 1 +} + +fresh_codex_usage_json() { # <account-home> <quota-command> + local account_home=$1 quota_bin=$2 cache_home cache_file environment_name timeout status + timeout=$(quota_timeout_seconds) || return 1 + cache_home=$account_home/.agent-fleet-quota-cache + cache_file=$cache_home/quota-axi/quotas.json + if { [ -e "$cache_home" ] || [ -L "$cache_home" ]; } \ + && { [ ! -d "$cache_home" ] || [ -L "$cache_home" ]; }; then + log "codex account $account_home skipped: its quota cache root is not a real directory" + return 1 + fi + if { [ -e "$cache_home/quota-axi" ] || [ -L "$cache_home/quota-axi" ]; } \ + && { [ ! -d "$cache_home/quota-axi" ] || [ -L "$cache_home/quota-axi" ]; }; then + log "codex account $account_home skipped: its quota-axi cache directory is not a real directory" + return 1 + fi + if [ -e "$cache_file" ] || [ -L "$cache_file" ]; then + rm -f "$cache_file" || { + log "codex account $account_home skipped: could not clear its quota cache for a fresh health read" + return 1 + } + fi + ( + while IFS='=' read -r environment_name _; do + case "$environment_name" in + XDG_*|QUOTA_AXI_*|AGENT_FLEET_*) unset "$environment_name" ;; + esac + done < <(/usr/bin/env) + CODEX_HOME=$account_home + XDG_CACHE_HOME=$cache_home + export CODEX_HOME XDG_CACHE_HOME + if run_bounded "$timeout" "$quota_bin" --provider codex --json 2>/dev/null; then + return 0 + else + status=$? + fi + if [ "$status" -eq 124 ]; then + log "codex account $account_home skipped: quota read timed out after ${timeout}s" + fi + return "$status" + ) +} + +codex_score() { # <quota-json> + jq -er ' + [.providers[]? + | select(.provider == "codex" and .state.status == "fresh") + | (.windows // [])[]? + | select((.id == "five_hour" or .id == "weekly") + and (.kind // "") != "model" + and (.percentRemaining | type) == "number") + | .percentRemaining] + | if length == 0 then empty else min end + ' 2>/dev/null <<EOF +$1 +EOF +} + +select_codex() { + local root vendor_dir quota_bin candidate usage score + local best_home='' best_score='' + root=$(account_root) || return 1 + vendor_dir=$root/codex + [ -d "$vendor_dir" ] && [ ! -L "$vendor_dir" ] || { + echo "error: no account-directory root for codex at $vendor_dir" >&2 + return 1 + } + command -v jq >/dev/null 2>&1 || { + echo "error: jq is required for Codex account usage selection" >&2 + return 1 + } + quota_bin=$(quota_command) || return 1 + LC_ALL=C + export LC_ALL + for candidate in "$vendor_dir"/*; do + valid_account_home "$vendor_dir" "$candidate" || continue + usage=$(fresh_codex_usage_json "$candidate" "$quota_bin") || usage= + score=$(codex_score "$usage") || score= + if [ -z "$score" ]; then + log "codex account $candidate skipped: no freshly readable usage window" + continue + fi + log "codex account $candidate fresh remaining score=$score" + if [ -z "$best_home" ] || awk -v candidate_score="$score" -v current_score="$best_score" \ + 'BEGIN { exit !(candidate_score > current_score) }'; then + best_home=$candidate + best_score=$score + fi + done + [ -n "$best_home" ] || { + echo "error: no healthy Codex account has a freshly readable usage window" >&2 + return 1 + } + log "selected codex account $best_home with fresh remaining score=$best_score" + printf '%s\n' "$best_home" +} + +select_claude() { + local selected + selected=$(first_account_home claude) || return 1 + log "CLAUDE USAGE UNREADABLE: quota-axi cannot non-interactively resolve Claude's config-dir-specific macOS Keychain credential today; selecting the first account directory by stable sort: $selected" + printf '%s\n' "$selected" +} + +select_account() { # <vendor> + case "$1" in + codex) select_codex ;; + claude) select_claude ;; + *) + echo "error: direct account-directory selection supports only claude or codex, not '$1'" >&2 + return 1 + ;; + esac +} + +install_herdr_hook() { # <vendor> <account-home> + local vendor=$1 account_home=$2 root vendor_dir herdr_bin expected_hook + root=$(account_root) || return 1 + vendor_dir=$root/$vendor + valid_account_home "$vendor_dir" "$account_home" || { + echo "error: unsafe $vendor account home for Herdr hook installation: $account_home" >&2 + return 1 + } + herdr_bin=$(herdr_command) || return 1 + case "$vendor" in + codex) + CODEX_HOME=$account_home "$herdr_bin" integration install codex >/dev/null || { + echo "error: Herdr Codex integration install failed for $account_home" >&2 + return 1 + } + expected_hook=$account_home/herdr-agent-state.sh + ;; + claude) + CLAUDE_CONFIG_DIR=$account_home "$herdr_bin" integration install claude >/dev/null || { + echo "error: Herdr Claude integration install failed for $account_home" >&2 + return 1 + } + expected_hook=$account_home/hooks/herdr-agent-state.sh + ;; + *) + echo "error: Herdr account hook installation supports only claude or codex, not '$vendor'" >&2 + return 1 + ;; + esac + [ -f "$expected_hook" ] && [ ! -L "$expected_hook" ] || { + echo "error: Herdr installer did not create the expected $vendor hook at $expected_hook" >&2 + return 1 + } + log "Herdr $vendor hook ready at $expected_hook" +} + +case "${1:-}" in + -h|--help) + usage + exit 0 + ;; + select) + [ "$#" -eq 2 ] || { usage; exit 2; } + select_account "$2" + ;; + install-herdr-hook) + [ "$#" -eq 3 ] || { usage; exit 2; } + install_herdr_hook "$2" "$3" + ;; + prepare) + [ "$#" -eq 2 ] || { usage; exit 2; } + selected_home=$(select_account "$2") || exit 1 + install_herdr_hook "$2" "$selected_home" || exit 1 + printf '%s\n' "$selected_home" + ;; + *) + usage + exit 2 + ;; +esac diff --git a/bin/fm-account-routing-lib.sh b/bin/fm-account-routing-lib.sh index b046a3c15cb..6b1a0fa7c2e 100644 --- a/bin/fm-account-routing-lib.sh +++ b/bin/fm-account-routing-lib.sh @@ -15,12 +15,10 @@ # FM_ACCOUNT_ROUTING and executable overrides are accepted only when the # unmistakable FM_ACCOUNT_ROUTING_TEST_LAB opt-in is active. # Valid modes are off, observe, and enforce. -# Off does not invoke Agent Fleet. -# Observe performs only `choose --dry-run`, never creates a lease, never wraps -# the provider launch, and never writes managed account metadata. -# Enforce atomically reserves one profile after endpoint and worktree setup, -# immediately before provider launch, and fails closed on every Agent Fleet or -# validation error. +# New observe and enforce launches use bin/fm-account-directory.sh and do not +# invoke Agent Fleet. +# Agent Fleet selection and lease helpers remain as legacy recovery support and +# isolated dead-code fixtures pending remove-fleet-routing-deadcode. # # FM_AGENT_FLEET_BIN may name a deterministic fake only in tests/labs. # Production always opens the current passwd user's fixed regular front door at @@ -31,6 +29,15 @@ # them when the selection-specific override is unset. Other control calls keep # the 10s FM_ACCOUNT_CONTROL_TIMEOUT default. +FM_ACCOUNT_ROUTING_LIB_SOURCE=${BASH_SOURCE[0]} +case "$FM_ACCOUNT_ROUTING_LIB_SOURCE" in + */*) FM_ACCOUNT_ROUTING_LIB_DIR="$(cd "${FM_ACCOUNT_ROUTING_LIB_SOURCE%/*}" && pwd)" ;; + *) FM_ACCOUNT_ROUTING_LIB_DIR="$(pwd)" ;; +esac +unset FM_ACCOUNT_ROUTING_LIB_SOURCE +# shellcheck source=bin/fm-checkout-lock-lib.sh +. "$FM_ACCOUNT_ROUTING_LIB_DIR/fm-checkout-lock-lib.sh" + # Security-sensitive control-plane helpers never resolve utilities through the # caller's PATH. Test/lab mode may still supply fake timeout runners, but path, # ownership, process, hashing, and passwd-home checks use only fixed system @@ -958,6 +965,96 @@ fm_account_lifecycle_lock_acquire() { # <state-dir> <task> fm_account_lock_acquire "$1" "$2" account-lifecycle "account lifecycle" "${FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS:-10}" } +fm_account_stable_path_key() { + fm_checkout_stable_path_key "$1" "$2" 1 24 +} + +fm_secondmate_home_lifecycle_lock_acquire() { + local state_base=$1 home=$2 key + key=$(fm_account_stable_path_key "$home" directory) || { + echo "error: secondmate home lifecycle target is unsafe or its lock identity is unavailable: $home" >&2 + return 1 + } + fm_account_lock_acquire "$state_base/secondmate-home-lifecycle" "home-$key" \ + secondmate-home-lifecycle "secondmate home lifecycle" "${FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS:-10}" +} + +fm_secondmate_registry_lock_acquire() { + local state_base=$1 registry=$2 key + key=$(fm_account_stable_path_key "$registry" file) || { + echo "error: secondmate registry lock identity is unavailable: $registry" >&2 + return 1 + } + fm_account_lock_acquire "$state_base/secondmate-registry" "registry-$key" \ + secondmate-registry "secondmate registry" "${FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS:-10}" +} + +fm_secondmate_registry_query() { + local registry=$1 mode=${2:-validate} expected_id=${3:-} key=${4:-} + # shellcheck disable=SC2016 # Perl source is intentionally literal shell text. + fm_account_system_perl -MErrno=ENOENT -e ' + my ($registry, $mode, $expected_id, $key) = @ARGV; + if (!lstat($registry)) { + exit 0 if $! == ENOENT && ($mode eq q{validate} || $mode eq q{list}); + exit 1; + } + exit 1 if -l _ || !-f _ || !-r _; + open my $fh, q{<}, $registry or exit 1; + my @entries; + my $line_number = 0; + while (my $line = <$fh>) { + ++$line_number; + next if $line =~ /^\s*$/ || $line =~ /^#/; + exit 1 if $line !~ /^- /; + chomp $line; + my ($id, $summary, $home, $scope, $projects, $added) = + $line =~ /^- ([A-Za-z0-9][A-Za-z0-9._-]*) - (.+) \(home: ([^;]+); scope: ([^;]+); projects: ([^;]*); added ([0-9]{4}-[0-9]{2}-[0-9]{2})\)$/; + exit 1 if !defined($id) || $summary eq q{} || $scope eq q{}; + exit 1 if $home !~ m{^/} || $home =~ /[\0\t\r\n]/ || $home =~ m{(?:^|/)\.\.?($|/)}; + exit 1 if $projects ne q{} && $projects !~ /^[A-Za-z0-9][A-Za-z0-9._-]*(?:, [A-Za-z0-9][A-Za-z0-9._-]*)*$/; + my $current = q{}; + for my $component (grep { $_ ne q{} } split m{/+}, $home) { + $current .= q{/} . $component; + lstat($current) or exit 1; + exit 1 if -l _; + } + exit 1 if !-d $home; + my @identity = stat($home); + exit 1 if !@identity; + push @entries, [$id, $home, $projects, "$identity[0]:$identity[1]"]; + } + my (%ids, %homes); + for my $index (0 .. $#entries) { + my $entry = $entries[$index]; + exit 1 if $ids{$entry->[0]}++ || exists $homes{$entry->[3]}; + $homes{$entry->[3]} = $entry->[0]; + for my $prior_index (0 .. $index - 1) { + my $prior = $entries[$prior_index]; + exit 1 if index($entry->[1], $prior->[1] . q{/}) == 0; + exit 1 if index($prior->[1], $entry->[1] . q{/}) == 0; + } + } + if ($mode eq q{validate}) { + exit 0; + } + if ($mode eq q{list}) { + for my $entry (@entries) { + print join(qq{\t}, $entry->[0], $entry->[1], $entry->[2]), qq{\n}; + } + exit 0; + } + my @matches = grep { $_->[0] eq $expected_id } @entries; + exit 1 if @matches != 1; + if ($key eq q{home}) { + print $matches[0]->[1]; + } elsif ($key eq q{projects}) { + print $matches[0]->[2]; + } else { + exit 1; + } + ' "$registry" "$mode" "$expected_id" "$key" +} + fm_account_lifecycle_lock_owned() { # <lock-path> fm_account_reclaim_guard_owned "$1" } @@ -1012,7 +1109,7 @@ fm_account_safe_lineage_value() { fm_account_meta_key_owned() { # <key> case "$1" in - window|worktree|project|harness|kind|mode|yolo|tasktmp|model|effort|report_required|generation_id|backend|tmux_window_id|tmux_session_target|account_pool|account_profile|account_task|account_attempt|account_predecessor_task|account_predecessor_attempt|account_predecessor_provider|account_predecessor_profile|account_predecessor_pool|account_predecessor_session|account_predecessor_cleanup|account_rollback_cleanup|account_rollback_backup|account_rollback_artifacts|account_rollback_preserve_session|continuation_packet|provider_session_id|herdr_session|herdr_workspace_id|herdr_tab_id|herdr_pane_id|zellij_session|zellij_tab_id|zellij_pane_id|orca_worktree_id|terminal|cmux_workspace_id|cmux_surface_id|home|projects|rollback_pending) return 0 ;; + window|worktree|worktree_git_dir|worktree_git_dir_identity|worktree_git_ref|worktree_git_head|worktree_git_setup_ref|worktree_git_setup_head|project|harness|kind|mode|yolo|tasktmp|model|effort|report_required|generation_id|backend|tmux_window_id|tmux_session_target|account_home|direct_spawn_cleanup|direct_spawn_backup|direct_spawn_artifacts|direct_recovery_cleanup|direct_recovery_backup|direct_recovery_artifacts|account_pool|account_profile|account_task|account_attempt|account_predecessor_task|account_predecessor_attempt|account_predecessor_provider|account_predecessor_profile|account_predecessor_pool|account_predecessor_session|account_predecessor_cleanup|account_rollback_cleanup|account_rollback_backup|account_rollback_artifacts|account_rollback_preserve_session|continuation_packet|provider_session_id|herdr_session|herdr_workspace_id|herdr_tab_id|herdr_pane_id|zellij_session|zellij_tab_id|zellij_pane_id|orca_worktree_id|terminal|orca_cleanup_pending|orca_cleanup_phase|orca_terminal_proof|orca_repo_id|orca_expected_task|orca_provider_task|orca_discovery_label|orca_provider_scope|cmux_workspace_id|cmux_surface_id|home|projects|rollback_pending) return 0 ;; *) return 1 ;; esac } @@ -1111,7 +1208,7 @@ fm_account_restore_artifacts() { if [ -n "$tasktmp" ]; then [ "$tasktmp" = "/tmp/fm-$task" ] || return 1 if [ -e "$backup/tasktmp-existed" ]; then - [ -e "$backup/gotmp-existed" ] || fm_account_system_exec "$FM_ACCOUNT_SYSTEM_RM_BIN" -rf "$tasktmp/gotmp" || return 1 + [ -e "$backup/gotmp-existed" ] || [ ! -d "$tasktmp" ] || fm_account_system_exec "$FM_ACCOUNT_SYSTEM_RM_BIN" -rf "$tasktmp/gotmp" || return 1 else fm_account_system_exec "$FM_ACCOUNT_SYSTEM_RM_BIN" -rf "$tasktmp" || return 1 fi diff --git a/bin/fm-backend.sh b/bin/fm-backend.sh index 1167c3ae4b5..23dc742ac23 100644 --- a/bin/fm-backend.sh +++ b/bin/fm-backend.sh @@ -17,9 +17,10 @@ # auto-detection (report.md's Open Question #2: start with a dedicated # background session for predictability, unlike tmux's/herdr's ambient-session # reuse); see report.md's "Zellij Backend" section and docs/zellij-backend.md -# for its empirical basis. P4 added Orca lifecycle support with Orca owning -# both the task worktree and terminal endpoint; new report-required tasks now -# refuse Orca, while eligible pre-cutover tasks may still respawn there. +# for its empirical basis. P4 added an Orca lifecycle design with Orca owning +# both the task worktree and terminal endpoint. New report-required tasks refuse +# Orca, and legacy lifecycle mutation now also fails closed until provider +# authority is empirically verified. # P5 adds bin/backends/cmux.sh, also # EXPERIMENTAL and spawn-capable, behind `--backend cmux`/`FM_BACKEND=cmux`/ # `config/backend`, and behind runtime auto-detection when firstmate itself is @@ -36,8 +37,8 @@ # `backend=tmux` for a default-backend task, so existing and newly spawned # default-path metas stay byte-identical. Only a task spawned on a non-tmux # spawn-capable backend, currently experimental herdr, zellij, or cmux, -# carries an explicit `backend=` line; an eligible legacy Orca respawn also -# carries `backend=orca`. +# carries an explicit `backend=` line; retained legacy Orca metadata carries +# `backend=orca`. # # Event-source framing (herdr-addendum "Events as the core abstraction"): a # backend's supervision surface is conceptually an EVENT SOURCE - it produces @@ -310,7 +311,8 @@ fm_backend_validate_spawn() { # <name> # - jq, for the JSON-emitting experimental adapters (herdr, zellij, cmux) whose # spawn/liveness paths parse the backend's JSON output (see each adapter's # tool check, e.g. fm_backend_herdr_tool_check); -# - nohup and perl, for Herdr's portable detached setsid server launcher; +# - nohup for Herdr's portable detached setsid server launcher, which also +# consumes the universal Perl runtime; # - the treehouse worktree provider for every session-provider-only backend # (tmux, herdr, zellij, cmux); orca owns its own task worktree and terminal, # so it drops both treehouse and any other backend's session CLI. @@ -319,7 +321,7 @@ fm_backend_validate_spawn() { # <name> fm_backend_required_tools() { # <backend> case "$1" in tmux) printf '%s' 'tmux treehouse' ;; - herdr) printf '%s' 'herdr jq nohup perl treehouse' ;; + herdr) printf '%s' 'herdr jq nohup treehouse' ;; zellij) printf '%s' 'zellij jq treehouse' ;; cmux) printf '%s' 'cmux jq treehouse' ;; orca) printf '%s' 'orca' ;; @@ -363,6 +365,7 @@ fm_backend_target_of_meta() { # <meta-file> if [ "$backend" = orca ]; then terminal=$(fm_meta_get "$meta" terminal) [ -n "$terminal" ] && { printf '%s' "$terminal"; return 0; } + return 0 elif [ "$backend" = tmux ]; then tmux_window_id=$(fm_meta_get "$meta" tmux_window_id) [ -n "$tmux_window_id" ] && { printf '%s' "$tmux_window_id"; return 0; } @@ -607,6 +610,16 @@ fm_backend_remove_worktree() { # <backend> <worktree-id> esac } +fm_backend_remove_worktree_bound() { # <backend> <worktree-id> <expected-path> <boundary-token> + local backend=$1 + shift + fm_backend_source "$backend" || return 1 + case "$backend" in + orca) fm_backend_orca_remove_worktree_bound "$@" ;; + *) echo "error: backend '$backend' does not own task worktrees" >&2; return 1 ;; + esac +} + fm_backend_worktree_path() { # <backend> <worktree-id> local backend=$1 shift @@ -617,6 +630,26 @@ fm_backend_worktree_path() { # <backend> <worktree-id> esac } +fm_backend_quiesce_terminal() { # <backend> <terminal-id> [expected-worktree-id] [expected-label] + local backend=$1 + shift + fm_backend_source "$backend" || return 1 + case "$backend" in + orca) fm_backend_orca_quiesce_terminal "$@" ;; + *) echo "error: backend '$backend' has no authoritative terminal-quiescence operation" >&2; return 1 ;; + esac +} + +fm_backend_quiesce_worktree_terminals() { # <backend> <worktree-id> <expected-label> + local backend=$1 + shift + fm_backend_source "$backend" || return 1 + case "$backend" in + orca) fm_backend_orca_quiesce_worktree_terminals "$@" ;; + *) echo "error: backend '$backend' does not own task worktrees" >&2; return 1 ;; + esac +} + # fm_backend_busy_state: semantic busy/idle/unknown for backends that expose # native agent-state (herdr-addendum "busy state" row - the first backend # where this gets real semantics beyond pane-regex). Backends with no such @@ -733,6 +766,11 @@ fm_backend_target_state() { # <backend> <target> [expected-label] [recorded-sco local workspace surface workspaces workspace_record title_record title_count expected_title resolved_workspace session= [ -n "$target" ] || { printf 'unknown'; return 0; } + if [ "$backend" = orca ]; then + fm_backend_source orca >/dev/null 2>&1 || { printf 'unknown'; return 0; } + fm_backend_orca_terminal_state "$target" "$recorded_scoped_target" "$expected_label" + return 0 + fi if fm_backend_target_exists "$backend" "$target" "$expected_label" "$recorded_scoped_target" 2>/dev/null; then printf 'present' return 0 diff --git a/bin/fm-bootstrap.sh b/bin/fm-bootstrap.sh index 17388bf7ff5..fa63bd898cf 100755 --- a/bin/fm-bootstrap.sh +++ b/bin/fm-bootstrap.sh @@ -11,6 +11,7 @@ # "CREW_DISPATCH: invalid config/crew-dispatch.json - <reason>", # "CREW_DISPATCH: active config/crew-dispatch.json" plus indented rules, # "FLEET_SYNC: <repo>: skipped|recovered|STUCK: <detail>", +# "MISSING: checkout-refresh (install: bin/fm-bootstrap.sh install checkout-refresh)", # "TASKS_AXI: available", "TANGLE: <remediation>", # "SECONDMATE_SYNC: secondmate <id>: skipped: <reason>", # "NUDGE_SECONDMATES: fm-<id>...", @@ -35,8 +36,8 @@ # distinct from the endpoint pane-presence check): outcomes distinguish # no-op, successful respawn, explicit-routing deferral, skipped # recovery, and failed recovery. A confirmed-dead unmanaged generation -# is deferred until an operator chooses whether to preserve unmanaged -# routing or convert it to managed routing. Session-start scope only; +# is deferred until an operator chooses whether to preserve +# unmanaged routing or convert it to managed routing. Session-start scope only; # see AGENTS.md "Session start" and docs/tmux-backend.md / # docs/herdr-backend.md "Agent liveness probe" for the empirical basis. # A TANGLE line means the firstmate primary checkout (FM_ROOT) is stranded @@ -57,8 +58,17 @@ # X mode is OPTIONAL and inert unless FM_HOME/.env has a non-empty # FMX_PAIRING_TOKEN. When opted in, bootstrap requires curl+jq, writes # the relay poll shim and 30s cadence config, and prints an FMX line. -# Fleet sync fetches, fast-forwards safe default-branch states, reports -# recovered and STUCK clone drift, and prunes gone local branches; it is +# Fleet sync discovers projects/, Treehouse backing checkouts, +# configured checkouts, and matching-origin top-level clones, then +# fetches and fast-forwards safe default-branch states and reports +# recovered and STUCK clone drift, incomplete discovery, invalid +# coverage configuration, and untracked skill-draft hygiene. +# Session-start refresh keeps the existing gone-branch prune behavior; +# the independent checkout-refresh cadence disables pruning, while +# this session-start invocation passes --session to preserve it. +# The cadence needs the checkout-refresh background service installed +# with explicit captain approval through the MISSING diagnostic. +# Session-start fleet sync is # bounded by FM_FLEET_SYNC_BOOTSTRAP_TIMEOUT when it is a non-empty # numeric override, while non-numeric values fall back to 20s. # When the override is unset or blank, the timeout is @@ -75,7 +85,7 @@ # fm-session-start.sh's read-only path when another live session holds # the fleet lock, so a second concurrent session never race-mutates # report-retention installation state, secondmate homes, X-mode -# artifacts, project clones, or repair instructions. Unset/0 (the +# artifacts, covered checkouts, or repair instructions. Unset/0 (the # default) runs every sweep exactly as before - this flag is purely # additive. # fm-bootstrap.sh install <tool>... @@ -117,8 +127,28 @@ report_retention_ensure() { fi } +checkout_refresh_ensure() { + local out + [ "$(uname)" = Darwin ] || return 0 + [ -x "$SCRIPT_DIR/fm-checkout-refresh.sh" ] || return 0 + if [ "${FM_GATE_REFUSE_BYPASS:-0}" = 1 ] && [ "${FM_CHECKOUT_REFRESH_BOOTSTRAP_TEST:-0}" != 1 ]; then + return 0 + fi + if ! out=$("$SCRIPT_DIR/fm-checkout-refresh.sh" ensure 2>&1); then + [ -n "$out" ] || out="background owner did not start" + echo "MISSING: checkout-refresh (install: bin/fm-bootstrap.sh install checkout-refresh)" + echo "FLEET_SYNC: background: skipped: ${out%%$'\n'*}" + fi +} + fleet_sync_origin_backed_project_count() { local count proj + if [ -x "$FM_ROOT/bin/fm-checkout-refresh.sh" ] \ + && { [ "${FM_GATE_REFUSE_BYPASS:-0}" != 1 ] || [ "${FM_CHECKOUT_REFRESH_BOOTSTRAP_TEST:-0}" = 1 ]; }; then + count=$("$FM_ROOT/bin/fm-checkout-refresh.sh" discover 2>/dev/null | awk 'NF { n += 1 } END { print n + 0 }') + echo "$count" + return 0 + fi count=0 [ -d "$PROJECTS" ] || { echo 0; return 0; } for proj in "$PROJECTS"/*; do @@ -154,6 +184,7 @@ fleet_sync_relay_filtered_output() { *': skipped: no origin remote') ;; *': skipped:'*) echo "FLEET_SYNC: $line" ;; *': STUCK:'*) echo "FLEET_SYNC: $line" ;; + *': HYGIENE:'*) echo "FLEET_SYNC: $line" ;; *': recovered:'*) echo "FLEET_SYNC: $line" ;; esac done < "$tmp" @@ -169,14 +200,19 @@ fleet_sync_relay_all_output() { fleet_sync() { [ -x "$FM_ROOT/bin/fm-fleet-sync.sh" ] || return 0 - [ -d "$PROJECTS" ] || return 0 tmp=$(mktemp "${TMPDIR:-/tmp}/fm-fleet-sync.XXXXXX" 2>/dev/null) || return 0 timeout=$(fleet_sync_bootstrap_timeout) monitor_was_on=0 case $- in *m*) monitor_was_on=1 ;; esac set -m 2>/dev/null || true - "$FM_ROOT/bin/fm-fleet-sync.sh" >"$tmp" 2>/dev/null & + if [ -x "$FM_ROOT/bin/fm-checkout-refresh.sh" ] \ + && { [ "${FM_GATE_REFUSE_BYPASS:-0}" != 1 ] || [ "${FM_CHECKOUT_REFRESH_BOOTSTRAP_TEST:-0}" = 1 ]; }; then + "$FM_ROOT/bin/fm-checkout-refresh.sh" run-once --force --verbose --session >"$tmp" 2>&1 & + else + [ -d "$PROJECTS" ] || { rm -f "$tmp"; return 0; } + "$FM_ROOT/bin/fm-fleet-sync.sh" >"$tmp" 2>/dev/null & + fi pid=$! start=$SECONDS @@ -438,7 +474,7 @@ install_cmd() { manual_install_url() { case "$1" in herdr) echo "https://herdr.dev" ;; - nohup|perl) echo "https://github.com/ruby-dlee/firstmate/blob/main/docs/configuration.md#herdr-detached-launcher-prerequisites" ;; + nohup|perl) echo "https://github.com/ruby-dlee/firstmate/blob/main/docs/configuration.md#portable-process-control-prerequisites" ;; agent-fleet) echo "https://github.com/ruby-dlee/firstmate/blob/main/docs/configuration.md#agent-fleet-account-routing" ;; *) return 1 ;; esac @@ -458,7 +494,7 @@ missing_tool_diagnostic() { # fm_backend_required_tools (bin/fm-backend.sh). So a herdr/zellij/cmux home is # never told tmux is missing, and only orca drops treehouse. A backend value with # no verified dependency set is reported before the universal checks continue. -COMMON_TOOLS="node python3 git gh no-mistakes gh-axi chrome-devtools-axi lavish-axi tasks-axi quota-axi" +COMMON_TOOLS="node python3 git gh perl no-mistakes gh-axi chrome-devtools-axi lavish-axi tasks-axi quota-axi" BACKEND=$(fm_backend_name) BACKEND_VALID=1 if ! BACKEND_TOOLS=$(fm_backend_required_tools "$BACKEND"); then @@ -603,6 +639,7 @@ EOF BOOTSTRAP_JQ_REPORTED=0 ACCOUNT_ROUTING_MODE=off +ACCOUNT_ROUTING_NEEDS_DIRECT_TOOLS=0 ACCOUNT_ROUTING_NEEDS_AGENT_FLEET=0 CREW_DISPATCH_ROUTING_VALID=unknown @@ -614,26 +651,48 @@ account_routing_preflight() { mode_error=${mode#error: } echo "ACCOUNT_ROUTING: invalid routing policy - $mode_error" fi + [ "$ACCOUNT_ROUTING_MODE" = off ] || ACCOUNT_ROUTING_NEEDS_DIRECT_TOOLS=1 [ "$ACCOUNT_ROUTING_MODE" != enforce ] || ACCOUNT_ROUTING_NEEDS_AGENT_FLEET=1 } account_routing_dependency_preflight() { - local needs_agent_fleet=$ACCOUNT_ROUTING_NEEDS_AGENT_FLEET dispatch + local needs_direct=$ACCOUNT_ROUTING_NEEDS_DIRECT_TOOLS needs_agent_fleet=$ACCOUNT_ROUTING_NEEDS_AGENT_FLEET dispatch meta direct_perl dispatch="$CONFIG/crew-dispatch.json" if [ -f "$dispatch" ]; then if [ "$CREW_DISPATCH_ROUTING_VALID" = 1 ]; then - jq -e '.. | objects | select(has("account_pool") or has("account_profile"))' "$dispatch" >/dev/null 2>&1 && needs_agent_fleet=1 + jq -e '.. | objects | select(has("account_pool") or has("account_profile"))' "$dispatch" >/dev/null 2>&1 && needs_direct=1 elif [ "$CREW_DISPATCH_ROUTING_VALID" = unknown ] \ && grep -Eq '"account_(pool|profile)"[[:space:]]*:' "$dispatch" 2>/dev/null; then - needs_agent_fleet=1 + needs_direct=1 fi fi - [ "$needs_agent_fleet" = 1 ] || return 0 - fm_account_fleet_bin >/dev/null 2>&1 || missing_tool_diagnostic agent-fleet - if ! command -v jq >/dev/null 2>&1; then + for meta in "$STATE"/*.meta; do + [ -f "$meta" ] && [ ! -L "$meta" ] || continue + if grep -q '^account_home=.' "$meta" 2>/dev/null \ + && ! grep -qx 'kind=secondmate' "$meta" 2>/dev/null; then + needs_direct=1 + fi + if grep -q '^account_profile=' "$meta" 2>/dev/null \ + || grep -qx 'account_rollback_cleanup=pending' "$meta" 2>/dev/null; then + needs_agent_fleet=1 + fi + done + if [ "$needs_direct" = 1 ] && ! command -v jq >/dev/null 2>&1; then echo "MISSING: jq (install: $(install_cmd jq))" BOOTSTRAP_JQ_REPORTED=1 fi + if [ "$needs_direct" = 1 ] && ! command -v herdr >/dev/null 2>&1; then + missing_tool_diagnostic herdr + fi + direct_perl=/usr/bin/perl + if [ "${FM_ACCOUNT_DIRECTORY_TEST_LAB:-}" = firstmate-account-directory-test-lab-v1 ] \ + && [ -n "${FM_ACCOUNT_DIRECTORY_PERL_BIN:-}" ]; then + direct_perl=$FM_ACCOUNT_DIRECTORY_PERL_BIN + fi + if [ "$needs_direct" = 1 ] && [ ! -x "$direct_perl" ]; then + missing_tool_diagnostic perl + fi + [ "$needs_agent_fleet" = 0 ] || fm_account_fleet_bin >/dev/null 2>&1 || missing_tool_diagnostic agent-fleet } crew_dispatch_validate() { @@ -752,6 +811,11 @@ if [ "${1:-}" = "install" ]; then "$SCRIPT_DIR/fm-report-retention.sh" install continue fi + if [ "$t" = checkout-refresh ]; then + echo "installing checkout-refresh LaunchAgent" + "$SCRIPT_DIR/fm-checkout-refresh.sh" install + continue + fi if ! cmd=$(install_cmd "$t"); then instructions=$(manual_install_url "$t") || { echo "error: unknown tool $t" >&2; exit 1; } echo "error: $t requires manual installation (instructions: $instructions)" >&2 @@ -809,6 +873,7 @@ account_routing_dependency_preflight if ! fm_backlog_backend_manual "$CONFIG" && fm_tasks_axi_compatible; then echo "TASKS_AXI: available" fi +checkout_refresh_ensure if [ "${FM_BOOTSTRAP_DETECT_ONLY:-0}" != 1 ]; then report_retention_ensure secondmate_sync diff --git a/bin/fm-checkout-lock-lib.sh b/bin/fm-checkout-lock-lib.sh new file mode 100644 index 00000000000..43c1a70f6f7 --- /dev/null +++ b/bin/fm-checkout-lock-lib.sh @@ -0,0 +1,443 @@ +#!/usr/bin/env bash +# Shared common-Git-directory lock identity and ownership for checkout mutation. +# Usage: source this file, call fm_checkout_lock_prepare <lock-root>, derive the +# lock with fm_checkout_lock_path <checkout> <lock-root>, then use fm_lock_*. +# Use fm_checkout_lock_run <checkout> <lock-root> <command> [args...] when the +# complete checkout mutation can execute inside one shared lock scope, including +# same-process nested calls for the same common Git directory. +# Use fm_checkout_treehouse_return <checkout> <lock-root> <project> for a +# process-tree-bounded `treehouse return --force` under that lock. +# shellcheck disable=SC2016 + +if [ "${FM_CHECKOUT_LOCK_LIB_LOADED:-0}" = 1 ]; then + return 0 +fi +FM_CHECKOUT_LOCK_LIB_LOADED=1 +FM_CHECKOUT_LOCK_LIB_SOURCE=${BASH_SOURCE[0]} +case "$FM_CHECKOUT_LOCK_LIB_SOURCE" in + */*) FM_CHECKOUT_LOCK_LIB_DIR="$(cd "${FM_CHECKOUT_LOCK_LIB_SOURCE%/*}" && pwd)" ;; + *) FM_CHECKOUT_LOCK_LIB_DIR="$(pwd)" ;; +esac +unset FM_CHECKOUT_LOCK_LIB_SOURCE +FM_CHECKOUT_LOCK_HELPERS_LOADED=0 +FM_CHECKOUT_TREEHOUSE_RETURN_CONFIG_STATUS=64 +FM_CHECKOUT_LOCK_FAILURE_STATUS=74 +FM_CHECKOUT_LOCK_CONTENTION_STATUS=75 +FM_CHECKOUT_PROCESS_CLEANUP_FAILURE_STATUS=76 +FM_CHECKOUT_TREEHOUSE_RETURN_TIMEOUT_STATUS=124 +FM_CHECKOUT_TREEHOUSE_RETURN_UNAVAILABLE_STATUS=127 +FM_CHECKOUT_SYSTEM_PERL_BIN= +[ ! -x /usr/bin/perl ] || FM_CHECKOUT_SYSTEM_PERL_BIN=/usr/bin/perl +[ -n "$FM_CHECKOUT_SYSTEM_PERL_BIN" ] || [ ! -x /bin/perl ] || FM_CHECKOUT_SYSTEM_PERL_BIN=/bin/perl +[ "${FM_CHECKOUT_TEST_DISABLE_SYSTEM_PERL:-0}" != 1 ] || FM_CHECKOUT_SYSTEM_PERL_BIN= +# shellcheck source=bin/fm-process-tree-lib.sh +. "$FM_CHECKOUT_LOCK_LIB_DIR/fm-process-tree-lib.sh" + +fm_checkout_system_perl() { + [ -n "$FM_CHECKOUT_SYSTEM_PERL_BIN" ] || return 127 + PERL5OPT='' PERL5LIB='' PERLLIB='' \ + DYLD_INSERT_LIBRARIES='' DYLD_LIBRARY_PATH='' LD_PRELOAD='' \ + LD_LIBRARY_PATH='' LD_AUDIT='' LD_DEBUG='' GCONV_PATH='' \ + BASH_ENV='' ENV='' \ + "$FM_CHECKOUT_SYSTEM_PERL_BIN" "$@" +} + +fm_checkout_lock_root() { + local state_base=$1 + if [ -n "${FM_CHECKOUT_REFRESH_LOCK_ROOT:-}" ]; then + printf '%s\n' "$FM_CHECKOUT_REFRESH_LOCK_ROOT" + elif [ -n "${FM_CHECKOUT_REFRESH_STATE_ROOT:-}" ]; then + printf '%s/locks\n' "$FM_CHECKOUT_REFRESH_STATE_ROOT" + else + printf '%s/locks\n' "$state_base" + fi +} + +fm_checkout_canonical_dir() { + [ -d "$1" ] || return 1 + (cd "$1" 2>/dev/null && pwd -P) +} + +fm_checkout_lexical_path() { + local candidate=$1 allow_missing=${2:-0} + fm_checkout_system_perl -MCwd=getcwd -MErrno=ENOENT -MFile::Spec -e ' + my ($raw, $allow_missing) = @ARGV; + exit 1 if !defined($raw) || $raw eq q{} || $raw =~ /[\0\r\n]/; + my @stack; + if (!File::Spec->file_name_is_absolute($raw)) { + my $cwd = getcwd(); + exit 1 if !defined($cwd) || $cwd !~ m{^/}; + @stack = grep { $_ ne q{} } split m{/+}, $cwd; + } + my $missing = 0; + for my $component (split m{/+}, $raw) { + next if $component eq q{} || $component eq q{.}; + if ($component eq q{..}) { + pop @stack if @stack; + } else { + push @stack, $component; + } + my $current = q{/} . join q{/}, @stack; + if (lstat($current)) { + exit 1 if -l _; + $missing = 0; + } elsif ($! == ENOENT) { + $missing = 1; + } else { + exit 1; + } + } + exit 1 if $missing && $allow_missing ne q{1}; + print q{/} . join(q{/}, @stack) . qq{\n}; + ' "$candidate" "$allow_missing" +} + +fm_checkout_trusted_dir() { + local candidate=$1 lexical physical + lexical=$(fm_checkout_lexical_path "$candidate" 0) || return 1 + [ -d "$lexical" ] || return 1 + physical=$(cd "$lexical" 2>/dev/null && pwd -P) || return 1 + [ "$physical" = "$lexical" ] || return 1 + printf '%s\n' "$physical" +} + +fm_checkout_git_common_dir() { + fm_checkout_validate_git_metadata "$1" +} + +fm_checkout_validate_git_metadata() { + local checkout=$1 root metadata absolute_git common listed line listed_root found=0 + root=$(fm_checkout_trusted_dir "$checkout") || return 1 + metadata="$root/.git" + [ -e "$metadata" ] && [ ! -L "$metadata" ] || return 1 + absolute_git=$(git -C "$root" rev-parse --absolute-git-dir 2>/dev/null) || return 1 + absolute_git=$(fm_checkout_trusted_dir "$absolute_git") || return 1 + common=$(git -C "$root" rev-parse --git-common-dir 2>/dev/null) || return 1 + case "$common" in + /*) ;; + *) common="$root/$common" ;; + esac + common=$(fm_checkout_trusted_dir "$common") || return 1 + if [ -d "$metadata" ]; then + [ "$(fm_checkout_trusted_dir "$metadata")" = "$absolute_git" ] || return 1 + [ "$absolute_git" = "$common" ] || return 1 + elif [ -f "$metadata" ]; then + case "$absolute_git" in "$common"/worktrees/*) ;; *) return 1 ;; esac + else + return 1 + fi + listed=$(git -C "$root" worktree list --porcelain 2>/dev/null) || return 1 + while IFS= read -r line; do + case "$line" in + "worktree "*) + listed_root=$(fm_checkout_trusted_dir "${line#worktree }" 2>/dev/null) || return 1 + [ "$listed_root" != "$root" ] || found=$((found + 1)) + ;; + esac + done <<EOF +$listed +EOF + [ "$found" -eq 1 ] || return 1 + printf '%s\n' "$common" +} + +fm_checkout_hash_value() { + local value=$1 length=${2:-64} hash + case "$length" in ''|*[!0-9]*|0) return 1 ;; esac + [ "$length" -le 64 ] || return 1 + hash=$(fm_checkout_system_perl -MDigest::SHA=sha256_hex -e ' + my ($value, $length) = @ARGV; + exit 1 if !defined($value) || !defined($length) || $value =~ /[\0\r\n]/; + print substr(sha256_hex($value), 0, $length); + ' "$value" "$length") || return 1 + [ "${#hash}" -eq "$length" ] || return 1 + case "$hash" in *[!0-9a-f]*) return 1 ;; esac + printf '%s\n' "$hash" +} + +fm_checkout_hash_file() { + local path=$1 hash + [ -f "$path" ] && [ ! -L "$path" ] || return 1 + hash=$(fm_checkout_system_perl -MDigest::SHA -e ' + my $path = shift; + open my $fh, q{<}, $path or exit 1; + binmode $fh; + my $sha = Digest::SHA->new(256); + $sha->addfile($fh); + print $sha->hexdigest; + ' "$path") || return 1 + [ "${#hash}" -eq 64 ] || return 1 + case "$hash" in *[!0-9a-f]*) return 1 ;; esac + printf '%s\n' "$hash" +} + +fm_checkout_stable_path_key() { + local path=$1 expected_type=${2:-any} allow_missing=${3:-0} length=${4:-24} + local lexical identity + lexical=$(fm_checkout_lexical_path "$path" "$allow_missing") || return 1 + identity=$(fm_checkout_system_perl -MErrno=ENOENT -e ' + my ($path, $expected, $allow_missing) = @ARGV; + if (lstat($path)) { + exit 1 if -l _; + exit 1 if $expected eq q{directory} && !-d _; + exit 1 if $expected eq q{file} && !-f _; + } else { + exit 1 if $! != ENOENT || $allow_missing ne q{1}; + } + print lc($path); + ' "$lexical" "$expected_type" "$allow_missing") || return 1 + fm_checkout_hash_value "$identity" "$length" +} + +fm_checkout_physical_path_identity() { + local path=$1 expected_type=${2:-any} lexical identity + lexical=$(fm_checkout_lexical_path "$path" 0) || return 1 + identity=$(fm_checkout_system_perl -e ' + my ($path, $expected) = @ARGV; + lstat($path) or exit 1; + exit 1 if -l _; + exit 1 if $expected eq q{directory} && !-d _; + exit 1 if $expected eq q{file} && !-f _; + my @s = stat(_); + exit 1 if !@s; + my $kind = -d _ ? q{directory} : -f _ ? q{file} : q{other}; + print join(q{:}, $kind, $s[0], $s[1]); + ' "$lexical" "$expected_type") || return 1 + [ -n "$identity" ] || return 1 + case "$identity" in *[!A-Za-z0-9:._-]*) return 1 ;; esac + printf '%s\n' "$identity" +} + +fm_checkout_physical_path_key() { + local identity + identity=$(fm_checkout_physical_path_identity "$1" "${2:-any}") || return 1 + fm_checkout_hash_value "existing:$identity" "${3:-24}" +} + +fm_checkout_tree_boundary_token() { + local path=$1 + fm_checkout_system_perl -MDigest::SHA=sha256_hex -MErrno=ENOENT -MFcntl=:mode -MFile::Find -e ' + my $root = shift; + exit 1 if !defined($root) || $root !~ m{^/} || $root eq q{/}; + my @root_stat = lstat($root); + exit 1 if !@root_stat || S_ISLNK($root_stat[2]) || !S_ISDIR($root_stat[2]); + (my $parent = $root) =~ s{/[^/]+$}{}; + $parent = q{/} if $parent eq q{}; + my @parent_stat = stat($parent); + exit 1 if !@parent_stat || $root_stat[0] != $parent_stat[0]; + my @records; + find( + { + no_chdir => 1, + preprocess => sub { sort @_ }, + wanted => sub { + my @metadata = lstat($File::Find::name); + exit 1 if !@metadata; + exit 1 if $metadata[0] != $root_stat[0]; + exit 1 if S_ISLNK($metadata[2]); + push @records, join( + qq{\0}, + $File::Find::name, + $metadata[0], + $metadata[1], + S_IFMT($metadata[2]), + ); + }, + }, + $root, + ); + print sha256_hex(join(qq{\0}, @records)); + ' "$path" +} + +fm_checkout_lock_key() { + fm_checkout_stable_path_key "$1" directory 0 24 +} + +fm_checkout_lock_path() { + local checkout=$1 lock_root=$2 common key + common=$(fm_checkout_git_common_dir "$checkout") || return 1 + key=$(fm_checkout_lock_key "$common") || return 1 + [ "${#key}" -eq 24 ] || return 1 + case "$key" in *[!0-9a-f]*) return 1 ;; esac + printf '%s/%s.lock\n' "$lock_root" "$key" +} + +fm_checkout_lock_prepare() { + local lock_root=$1 caller_root caller_home + mkdir -p "$lock_root" || return 1 + [ -d "$lock_root" ] && [ ! -L "$lock_root" ] || return 1 + if [ "$FM_CHECKOUT_LOCK_HELPERS_LOADED" -eq 0 ]; then + caller_root=${FM_ROOT:-$(cd "$FM_CHECKOUT_LOCK_LIB_DIR/.." && pwd)} + caller_home=${FM_HOME:-$caller_root} + # shellcheck disable=SC2034 + local FM_ROOT="$caller_root" FM_HOME="$caller_home" + # shellcheck disable=SC2034 + local FM_STATE_OVERRIDE="$lock_root" STATE='' FM_WAKE_LIB_DIR='' FM_WAKE_DEFAULT_ROOT='' + # shellcheck disable=SC2034 + local FM_WAKE_QUEUE='' FM_WAKE_QUEUE_LOCK='' + # shellcheck source=bin/fm-wake-lib.sh + . "$FM_CHECKOUT_LOCK_LIB_DIR/fm-wake-lib.sh" || return 1 + FM_CHECKOUT_LOCK_HELPERS_LOADED=1 + fi +} + +fm_checkout_lock_active_scope_owns() { + local checkout_lock=$1 ownerdir owner_pid + [ "${FM_CHECKOUT_LOCK_ACTIVE_PATH:-}" = "$checkout_lock" ] || return 1 + [ -n "${FM_CHECKOUT_LOCK_ACTIVE_OWNER_DIR:-}" ] || return 1 + [ -n "${FM_CHECKOUT_LOCK_ACTIVE_OWNER_PID:-}" ] || return 1 + [ -L "$checkout_lock" ] || return 1 + ownerdir=$(fm_lock_link_owner "$checkout_lock" 2>/dev/null) || return 1 + [ "$ownerdir" = "$FM_CHECKOUT_LOCK_ACTIVE_OWNER_DIR" ] || return 1 + owner_pid=$(cat "$ownerdir/pid" 2>/dev/null) || return 1 + [ "$owner_pid" = "$FM_CHECKOUT_LOCK_ACTIVE_OWNER_PID" ] || return 1 + fm_pid_alive "$owner_pid" || return 1 + fm_lock_points_to_owner "$checkout_lock" "$ownerdir" +} + +fm_checkout_lock_run() { + local checkout=$1 lock_root=$2 checkout_lock + shift 2 + fm_checkout_lock_prepare "$lock_root" || { + echo "error: cannot prepare shared checkout mutation lock at $lock_root" >&2 + return "$FM_CHECKOUT_LOCK_FAILURE_STATUS" + } + checkout_lock=$(fm_checkout_lock_path "$checkout" "$lock_root") || { + echo "error: cannot resolve shared checkout mutation lock identity for $checkout" >&2 + return "$FM_CHECKOUT_LOCK_FAILURE_STATUS" + } + if fm_checkout_lock_active_scope_owns "$checkout_lock"; then + "$@" + return + fi + ( + if ! fm_lock_try_acquire "$checkout_lock"; then + echo "error: checkout mutation already running for $checkout (pid ${FM_LOCK_HELD_PID:-unknown})" >&2 + return "$FM_CHECKOUT_LOCK_CONTENTION_STATUS" + fi + local FM_CHECKOUT_LOCK_ACTIVE_PATH="$checkout_lock" + local FM_CHECKOUT_LOCK_ACTIVE_OWNER_DIR="${FM_LOCK_OWNER_DIR:-}" + local FM_CHECKOUT_LOCK_ACTIVE_OWNER_PID="${BASHPID:-$$}" + local FM_PROCESS_TREE_GUARD_FILE="$FM_CHECKOUT_LOCK_ACTIVE_OWNER_DIR/process-group" + export FM_PROCESS_TREE_GUARD_FILE + trap 'fm_lock_release "$checkout_lock"' EXIT + "$@" + ) +} + +fm_checkout_treehouse_return_locked() { + local checkout=$1 lock_root=$2 project=$3 checkout_lock timeout status previous_dir cleanup_status + checkout_lock=$(fm_checkout_lock_path "$checkout" "$lock_root") || { + echo "error: cannot resolve shared checkout mutation lock identity for $checkout" >&2 + return "$FM_CHECKOUT_LOCK_FAILURE_STATUS" + } + if ! fm_checkout_lock_active_scope_owns "$checkout_lock"; then + echo "error: refusing unlocked Treehouse return for $checkout" >&2 + return "$FM_CHECKOUT_LOCK_FAILURE_STATUS" + fi + timeout=${FM_TREEHOUSE_RETURN_TIMEOUT:-60} + case "$timeout" in + ''|*[!0-9]*|0) + echo "error: FM_TREEHOUSE_RETURN_TIMEOUT must be a positive integer" >&2 + return "$FM_CHECKOUT_TREEHOUSE_RETURN_CONFIG_STATUS" + ;; + esac + previous_dir=$(pwd -P) || return "$FM_CHECKOUT_LOCK_FAILURE_STATUS" + cd "$project" || return "$FM_CHECKOUT_LOCK_FAILURE_STATUS" + if fm_run_bounded "$timeout" python3 -c ' +import os +import stat +import sys + +target = os.path.abspath(sys.argv[1]) +project = os.path.abspath(sys.argv[2]) +if not target or target == os.path.sep or os.path.realpath(os.getcwd()) != project: + raise SystemExit(74) +current = os.path.sep +for component in target.split(os.path.sep): + if not component: + continue + current = os.path.join(current, component) + metadata = os.lstat(current) + if stat.S_ISLNK(metadata.st_mode): + raise SystemExit(74) +metadata = os.lstat(target) +if not stat.S_ISDIR(metadata.st_mode): + raise SystemExit(74) +flags = os.O_RDONLY | os.O_DIRECTORY +if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW +descriptor = os.open(target, flags) +opened = os.fstat(descriptor) +if (metadata.st_dev, metadata.st_ino) != (opened.st_dev, opened.st_ino): + raise SystemExit(74) +parent = os.path.dirname(target) +parent_metadata = os.stat(parent) +if opened.st_dev != parent_metadata.st_dev or os.path.ismount(target): + raise SystemExit(74) +root_device = opened.st_dev +descriptors = [descriptor] +pending = [(descriptor, target)] +while pending: + directory_fd, path = pending.pop() + for name in sorted(os.listdir(directory_fd)): + item = os.stat(name, dir_fd=directory_fd, follow_symlinks=False) + if stat.S_ISLNK(item.st_mode): + raise SystemExit(74) + if not stat.S_ISDIR(item.st_mode): + continue + child_path = os.path.join(path, name) + child = os.open(name, flags, dir_fd=directory_fd) + child_opened = os.fstat(child) + if ( + (item.st_dev, item.st_ino) != (child_opened.st_dev, child_opened.st_ino) + or child_opened.st_dev != root_device + or os.path.ismount(child_path) + ): + os.close(child) + raise SystemExit(74) + descriptors.append(child) + pending.append((child, child_path)) +for retained in descriptors: + os.set_inheritable(retained, True) +os.environ["FM_TREEHOUSE_RETURN_ROOT_FD"] = str(descriptor) +os.environ["FM_TREEHOUSE_RETURN_BOUNDARY_FDS"] = ",".join(map(str, descriptors)) +os.environ["FM_TREEHOUSE_RETURN_PROJECT"] = project +os.fchdir(descriptor) +bound = os.stat(".") +if (opened.st_dev, opened.st_ino) != (bound.st_dev, bound.st_ino): + raise SystemExit(74) +os.execvp("treehouse", ("treehouse", "return", "--force", ".")) +' "$checkout" "$project"; then + status=0 + else + status=$? + fi + cleanup_status=$FM_PROCESS_TREE_CLEANUP_STATUS + cd "$previous_dir" || return "$FM_CHECKOUT_LOCK_FAILURE_STATUS" + if [ "$cleanup_status" != verified ]; then + echo "error: Treehouse return process cleanup could not be verified for $checkout; retained for inspection under the guarded checkout lock" >&2 + return "$FM_CHECKOUT_PROCESS_CLEANUP_FAILURE_STATUS" + fi + [ "$status" -ne 0 ] || return 0 + if [ "$status" -eq "$FM_CHECKOUT_TREEHOUSE_RETURN_TIMEOUT_STATUS" ]; then + echo "error: Treehouse return timed out after ${timeout}s for $checkout" >&2 + fi + return "$status" +} + +fm_checkout_treehouse_return() { + local checkout=$1 lock_root=$2 project=$3 + fm_checkout_lock_run "$checkout" "$lock_root" \ + fm_checkout_treehouse_return_locked "$checkout" "$lock_root" "$project" +} + +fm_checkout_treehouse_return_requires_retention() { + [ "$1" -eq "$FM_CHECKOUT_TREEHOUSE_RETURN_CONFIG_STATUS" ] \ + || [ "$1" -eq "$FM_CHECKOUT_LOCK_FAILURE_STATUS" ] \ + || [ "$1" -eq "$FM_CHECKOUT_LOCK_CONTENTION_STATUS" ] \ + || [ "$1" -eq "$FM_CHECKOUT_PROCESS_CLEANUP_FAILURE_STATUS" ] \ + || [ "$1" -eq "$FM_CHECKOUT_TREEHOUSE_RETURN_TIMEOUT_STATUS" ] \ + || [ "$1" -eq "$FM_CHECKOUT_TREEHOUSE_RETURN_UNAVAILABLE_STATUS" ] +} diff --git a/bin/fm-checkout-refresh.sh b/bin/fm-checkout-refresh.sh new file mode 100755 index 00000000000..310ef138e48 --- /dev/null +++ b/bin/fm-checkout-refresh.sh @@ -0,0 +1,3403 @@ +#!/usr/bin/env bash +# Keep every checkout that can seed Firstmate or Treehouse work current without +# touching unlanded work. +# +# The single-checkout mutation remains owned by fm-fleet-sync.sh. +# This script owns the broader covered-set discovery and the independent cadence: +# +# - projects/* under the active FM_HOME; +# - backing checkouts discovered from Treehouse's state under ~/.treehouse; +# - exact Git worktree roots from `path <checkout>` entries in config/checkout-refresh; +# - top-level clones under $HOME, plus explicit `scan <directory>` roots, whose +# origin URL matches one of the checkouts above. +# +# Matching-origin discovery covers exact top-level clone roots such as ~/relvino +# without hard-coding a captain-specific path or inheriting an enclosing repository. +# Treehouse pool entries resolve back to their backing checkout because Treehouse +# fetches origin and resets an acquired detached worktree from that shared Git +# metadata immediately before handoff. +# +# `run-once` probes each tracked upstream default-branch tip with `git ls-remote`. +# A changed tip triggers an immediate safe refresh, and fm-fleet-sync.sh repeats +# that live proof while owning the shared per-checkout mutation lock. +# FM_CHECKOUT_REFRESH_BACKSTOP seconds without a refresh triggers one anyway, so +# missed signals and lost state remain bounded. +# The home-scoped per-user LaunchAgent installed by `install` runs this probe +# with `run-once --scheduled` every FM_CHECKOUT_REFRESH_INTERVAL seconds while +# that Firstmate home is idle. +# Every run publishes coverage health, while only that scheduler-owned mode +# advances the independent liveness heartbeat. +# Scheduler health also binds the loaded launchd job's definition, arguments, +# environment, home, state root, and generation to its retained plist. +# Its default state is under checkout-refresh/homes/<FM_HOME hash>, while +# checkout-refresh/locks remains shared so every fm-fleet-sync.sh caller +# serializes mutation of the same clone. +# Bounded probes and acquisitions terminate and reap their complete process tree. +# fm-fleet-sync.sh owns the equivalent per-checkout refresh bound for every caller. +# FM_TREEHOUSE_ACQUIRE_TIMEOUT applies the same process-tree ownership to the +# synchronous durable lease acquired before a task endpoint is created. +# +# Cadence and spawn-preflight refreshes delegate to fm-fleet-sync.sh with pruning +# disabled, while session-start pruning retains every branch whose landed state +# cannot be proved from repository content or a surviving remote ref. +# Dirty, diverged, non-default, and otherwise unsafe checkouts remain untouched +# and are recorded as durable alerts. +# Every probe also inventories non-ignored untracked files in both the covered seed +# checkouts and the Treehouse pool worktrees under repository skill directories +# (`.agents/skills`, `.claude/skills`, `.codex/skills`, and `skills`). +# Unreadable or malformed Treehouse state invalidates coverage health while the +# heartbeat records scheduler liveness independently. +# Every scan root must be readable and successfully enumerable, and every covered +# checkout retains its actual origin identity across runs. +# Registered local-only checkouts and remote-free repositories use their proven +# local default tip while still surfacing dirty, stale, or non-default states. +# A new or changed inventory is surfaced immediately and persisted as a separate +# hygiene alert, even when no upstream change or backstop refresh is due. +# Forced/operator-visible runs repeat unresolved hygiene alerts. +# Nothing is forced, stashed, reset, or discarded. +# +# Config format (config/checkout-refresh), one directive per line: +# +# path /absolute/or/~/relative/checkout +# scan /directory/whose/immediate/children/are/clones +# +# Blank lines and lines beginning with # are ignored. +# Paths may contain spaces. +# Relative paths and unknown directives are rejected visibly. +# +# Usage: +# fm-checkout-refresh.sh discover +# fm-checkout-refresh.sh run-once [--force] [--verbose] [--session] [--scheduled] +# fm-checkout-refresh.sh preflight <checkout> +# fm-checkout-refresh.sh pool-preflight <expected-source> +# fm-checkout-refresh.sh acquire-worktree <expected-source> <lease-holder> +# fm-checkout-refresh.sh verify-worktree <worktree> <expected-source> +# fm-checkout-refresh.sh verify-home <home> <expected-source> +# fm-checkout-refresh.sh verify-returnable <worktree> <expected-source> <expected-tip> +# fm-checkout-refresh.sh ensure +# fm-checkout-refresh.sh install +# +# `install` and `ensure` dispatch through the scheduler adapter seam. +# macOS launchd is the implemented primary-fleet adapter. +# Linux currently has no cron/systemd adapter and reports that limitation +# explicitly instead of pretending a background backstop exists. +set -u + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +# shellcheck source=bin/fm-checkout-lock-lib.sh +. "$SCRIPT_DIR/fm-checkout-lock-lib.sh" +FM_HOME_CANONICAL=$(fm_checkout_trusted_dir "$FM_HOME") || { + echo "error: checkout-refresh home identity is unavailable: $FM_HOME" >&2 + exit 1 +} +FM_HOME_KEY=$(fm_checkout_hash_value "$FM_HOME_CANONICAL" 16) || { + echo "error: checkout-refresh logical home identity is unavailable: $FM_HOME" >&2 + exit 1 +} +FM_HOME_PHYSICAL_KEY=$(fm_checkout_physical_path_key "$FM_HOME_CANONICAL" directory 16) || { + echo "error: checkout-refresh physical home identity is unavailable: $FM_HOME" >&2 + exit 1 +} +PROJECTS="${FM_PROJECTS_OVERRIDE:-$FM_HOME/projects}" +CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" +CONFIG_FILE="${FM_CHECKOUT_REFRESH_CONFIG:-$CONFIG/checkout-refresh}" +if [ "${FM_TREEHOUSE_ROOT+x}" = x ]; then + TREEHOUSE_ROOT_RAW=$FM_TREEHOUSE_ROOT +else + TREEHOUSE_ROOT_RAW="$HOME/.treehouse" +fi +TREEHOUSE_ROOT=$TREEHOUSE_ROOT_RAW +TREEHOUSE_ROOT_CANONICAL= +TREEHOUSE_ROOT_INVALID=0 +TREEHOUSE_ROOT_EXPLICIT=0 +[ "${FM_TREEHOUSE_ROOT+x}" != x ] || TREEHOUSE_ROOT_EXPLICIT=1 +if ! TREEHOUSE_ROOT=$(fm_checkout_lexical_path "$TREEHOUSE_ROOT_RAW" 1); then + TREEHOUSE_ROOT_INVALID=1 + TREEHOUSE_ROOT=$TREEHOUSE_ROOT_RAW +elif [ -e "$TREEHOUSE_ROOT" ] && [ ! -d "$TREEHOUSE_ROOT" ]; then + TREEHOUSE_ROOT_INVALID=1 +elif [ -d "$TREEHOUSE_ROOT" ] \ + && TREEHOUSE_ROOT_CANONICAL=$(fm_checkout_trusted_dir "$TREEHOUSE_ROOT"); then + TREEHOUSE_ROOT=$TREEHOUSE_ROOT_CANONICAL +elif [ -e "$TREEHOUSE_ROOT" ]; then + TREEHOUSE_ROOT_INVALID=1 +elif [ "$TREEHOUSE_ROOT_EXPLICIT" -eq 1 ]; then + TREEHOUSE_ROOT_INVALID=1 +else + TREEHOUSE_ROOT_INVALID=0 +fi +STATE_BASE="${FM_CHECKOUT_REFRESH_STATE_BASE:-${XDG_STATE_HOME:-$HOME/.local/state}/firstmate/checkout-refresh}" +DEFAULT_STATE_ROOT="$STATE_BASE/homes/$FM_HOME_KEY" +PHYSICAL_STATE_ROOT="$STATE_BASE/homes/$FM_HOME_PHYSICAL_KEY" +CUSTOM_STATE_ROOT=0 +STATE_ROOT_EXPLICIT=0 +USING_PHYSICAL_STATE_ROOT=0 +if [ -n "${FM_CHECKOUT_REFRESH_STATE_ROOT:-}" ]; then + STATE_ROOT_EXPLICIT=1 + case "$FM_CHECKOUT_REFRESH_STATE_ROOT" in + "$DEFAULT_STATE_ROOT") + STATE_ROOT=$DEFAULT_STATE_ROOT + ;; + "$PHYSICAL_STATE_ROOT") + STATE_ROOT=$PHYSICAL_STATE_ROOT + USING_PHYSICAL_STATE_ROOT=1 + ;; + *) + STATE_ROOT=$FM_CHECKOUT_REFRESH_STATE_ROOT + CUSTOM_STATE_ROOT=1 + ;; + esac +else + STATE_ROOT=$DEFAULT_STATE_ROOT +fi +LOCK_ROOT=$(fm_checkout_lock_root "$STATE_BASE") +INTERVAL=${FM_CHECKOUT_REFRESH_INTERVAL:-60} +BACKSTOP=${FM_CHECKOUT_REFRESH_BACKSTOP:-900} +PROBE_TIMEOUT=${FM_CHECKOUT_REFRESH_PROBE_TIMEOUT:-15} +SYNC_TIMEOUT=${FM_CHECKOUT_REFRESH_SYNC_TIMEOUT:-60} +ACQUIRE_TIMEOUT=${FM_TREEHOUSE_ACQUIRE_TIMEOUT:-60} +ACTIVATION_TIMEOUT=${FM_CHECKOUT_REFRESH_ACTIVATION_TIMEOUT:-60} +PLATFORM=${FM_CHECKOUT_REFRESH_PLATFORM:-$(uname)} +LABEL_BASE=com.firstmate.checkout-refresh +LABEL=${FM_CHECKOUT_REFRESH_LABEL:-$LABEL_BASE.$FM_HOME_KEY} +PHYSICAL_LABEL=$LABEL_BASE.$FM_HOME_PHYSICAL_KEY +LEGACY_LABEL=${FM_CHECKOUT_REFRESH_LEGACY_LABEL:-$LABEL_BASE} +LAUNCH_AGENTS_DIR=${FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR:-$HOME/Library/LaunchAgents} +PLIST="$LAUNCH_AGENTS_DIR/$LABEL.plist" +PHYSICAL_PLIST="$LAUNCH_AGENTS_DIR/$PHYSICAL_LABEL.plist" +LAUNCHCTL=${FM_CHECKOUT_REFRESH_LAUNCHCTL:-launchctl} +HOME_STATE_NAMESPACE_STAGED=0 +HOME_MIGRATION_ACTIVE=0 +HOME_MIGRATION_RUN_LOCK= +PHYSICAL_AGENT_STOPPED=0 +LEGACY_AGENT_STOPPED=0 +LEGACY_AGENT_TRACKED=0 +SCHEDULER_GENERATION=${FM_CHECKOUT_REFRESH_GENERATION:-} +case "$SCHEDULER_GENERATION" in + '') ;; + *[!0-9a-f]*) + echo "error: FM_CHECKOUT_REFRESH_GENERATION must be a 32-character lowercase hexadecimal token" >&2 + exit 2 + ;; +esac +if [ -n "$SCHEDULER_GENERATION" ] && [ "${#SCHEDULER_GENERATION}" -ne 32 ]; then + echo "error: FM_CHECKOUT_REFRESH_GENERATION must be a 32-character lowercase hexadecimal token" >&2 + exit 2 +fi + +# shellcheck source=bin/fm-gate-refuse-lib.sh +. "$SCRIPT_DIR/fm-gate-refuse-lib.sh" +fm_refuse_if_gate_agent +# shellcheck source=bin/fm-process-tree-lib.sh +. "$SCRIPT_DIR/fm-process-tree-lib.sh" + +case "$INTERVAL" in ''|*[!0-9]*|0) echo "error: FM_CHECKOUT_REFRESH_INTERVAL must be a positive integer" >&2; exit 2 ;; esac +case "$BACKSTOP" in ''|*[!0-9]*|0) echo "error: FM_CHECKOUT_REFRESH_BACKSTOP must be a positive integer" >&2; exit 2 ;; esac +case "$PROBE_TIMEOUT" in ''|*[!0-9]*|0) echo "error: FM_CHECKOUT_REFRESH_PROBE_TIMEOUT must be a positive integer" >&2; exit 2 ;; esac +case "$SYNC_TIMEOUT" in ''|*[!0-9]*|0) echo "error: FM_CHECKOUT_REFRESH_SYNC_TIMEOUT must be a positive integer" >&2; exit 2 ;; esac +case "$ACQUIRE_TIMEOUT" in ''|*[!0-9]*|0) echo "error: FM_TREEHOUSE_ACQUIRE_TIMEOUT must be a positive integer" >&2; exit 2 ;; esac +case "$ACTIVATION_TIMEOUT" in ''|*[!0-9]*|0) echo "error: FM_CHECKOUT_REFRESH_ACTIVATION_TIMEOUT must be a positive integer" >&2; exit 2 ;; esac + +usage() { + echo "usage: fm-checkout-refresh.sh discover|run-once [--force] [--verbose] [--session] [--scheduled]|preflight <checkout>|pool-preflight <expected-source>|acquire-worktree <expected-source> <lease-holder>|verify-worktree <worktree> <expected-source>|verify-home <home> <expected-source>|verify-returnable <worktree> <expected-source> <expected-tip>|ensure|install" >&2 +} + +first_line() { + printf '%s\n' "$1" | sed -n '1s/[[:space:]]\{1,\}/ /g;1p' +} + +canonical_dir() { + fm_checkout_trusted_dir "$1" +} + +exact_git_root() { + local candidate=$1 canonical top canonical_top + canonical=$(canonical_dir "$candidate") || return 1 + top=$(git -C "$canonical" rev-parse --show-toplevel 2>/dev/null) || return 1 + canonical_top=$(canonical_dir "$top") || return 1 + [ "$canonical" = "$canonical_top" ] || return 1 + fm_checkout_validate_git_metadata "$canonical" >/dev/null || return 1 + printf '%s\n' "$canonical" +} + +require_exact_git_root() { + local candidate=$1 label=$2 canonical + canonical=$(exact_git_root "$candidate") || { + echo "error: $label must be an exact inspectable Git repository root: $candidate" >&2 + return 1 + } + printf '%s\n' "$canonical" +} + +expand_config_path() { + case "$1" in + "~") printf '%s\n' "$HOME" ;; + [~]/*) printf '%s/%s\n' "$HOME" "${1#\~/}" ;; + "\$HOME") printf '%s\n' "$HOME" ;; + "\$HOME/"*) printf '%s/%s\n' "$HOME" "${1#\$HOME/}" ;; + /*) printf '%s\n' "$1" ;; + *) echo "checkout-refresh: skipped: config path '$1' must be absolute, ~/..., or \$HOME/..." >&2; return 1 ;; + esac +} + +parse_config() { + local paths_file=$1 scans_file=$2 line directive value expanded failed=0 config_lexical config_parent + manifest_create "$paths_file" || return 1 + manifest_create "$scans_file" || return 1 + config_lexical=$(fm_checkout_lexical_path "$CONFIG_FILE" 1) || { + echo "checkout-refresh: skipped: unsafe config path $CONFIG_FILE" >&2 + return 1 + } + config_parent=${config_lexical%/*} + [ -n "$config_parent" ] || config_parent=/ + fm_checkout_trusted_dir "$config_parent" >/dev/null || { + echo "checkout-refresh: skipped: config parent is missing, redirected, or unreadable: $config_parent" >&2 + return 1 + } + if [ ! -e "$config_lexical" ]; then + return 0 + fi + if [ ! -f "$config_lexical" ] || [ ! -r "$config_lexical" ]; then + echo "checkout-refresh: skipped: unsafe or unreadable config $config_lexical" >&2 + return 1 + fi + while IFS= read -r line || [ -n "$line" ]; do + line=$(printf '%s\n' "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + case "$line" in ""|\#*) continue ;; esac + directive=${line%%[[:space:]]*} + if [ "$directive" = "$line" ]; then + echo "checkout-refresh: skipped: malformed config directive '$line'" >&2 + failed=1 + continue + fi + value=${line#"$directive"} + value=$(printf '%s\n' "$value" | sed 's/^[[:space:]]*//') + [ -n "$value" ] || { + echo "checkout-refresh: skipped: malformed config directive '$line'" >&2 + failed=1 + continue + } + case "$directive" in + path) + if expanded=$(expand_config_path "$value"); then + manifest_append "$paths_file" "$expanded" || failed=1 + else + failed=1 + fi + ;; + scan) + if expanded=$(expand_config_path "$value"); then + manifest_append "$scans_file" "$expanded" || failed=1 + else + failed=1 + fi + ;; + *) + echo "checkout-refresh: skipped: unknown config directive '$directive'" >&2 + failed=1 + ;; + esac + done < "$config_lexical" + [ "$failed" -eq 0 ] +} + +treehouse_worktree_paths() { + if [ "$TREEHOUSE_ROOT_INVALID" = 1 ]; then + echo "checkout-refresh: skipped: incomplete Treehouse coverage because the configured root is unsafe or unreadable: $TREEHOUSE_ROOT_RAW" >&2 + return 1 + fi + if [ ! -e "$TREEHOUSE_ROOT" ] && [ ! -L "$TREEHOUSE_ROOT" ]; then + return 0 + fi + [ -d "$TREEHOUSE_ROOT" ] || { + echo "checkout-refresh: skipped: incomplete Treehouse coverage because the root is not a directory: $TREEHOUSE_ROOT" >&2 + return 1 + } + command -v python3 >/dev/null 2>&1 || { + echo "checkout-refresh: skipped: incomplete Treehouse coverage because python3 is unavailable" >&2 + return 1 + } + python3 - "$TREEHOUSE_ROOT" <<'PY' +import json +import os +import stat +import sys + +failed = False + + +def directory_entries(path, label): + if os.path.islink(path): + raise OSError(f"{label} must not be a symlink") + metadata = os.stat(path) + if not stat.S_ISDIR(metadata.st_mode): + raise NotADirectoryError(path) + permissions = stat.S_IMODE(metadata.st_mode) + if not permissions & 0o444 or not permissions & 0o111: + raise PermissionError(f"{label} is unreadable") + with os.scandir(path) as entries: + return sorted(entries, key=lambda entry: entry.name) + + +root = sys.argv[1] +try: + pool_entries = directory_entries(root, "Treehouse root") +except OSError as error: + failed = True + pool_entries = [] + print( + f"checkout-refresh: skipped: incomplete Treehouse coverage at {root}: {error}", + file=sys.stderr, + ) + +state_paths = [] +for pool_entry in pool_entries: + try: + metadata = pool_entry.stat(follow_symlinks=False) + if stat.S_ISLNK(metadata.st_mode): + try: + target = pool_entry.stat(follow_symlinks=True) + except OSError as error: + raise OSError("broken Treehouse pool symlink") from error + if stat.S_ISDIR(target.st_mode): + raise OSError("Treehouse pool must not be a symlink") + continue + if not stat.S_ISDIR(metadata.st_mode): + continue + if any(character in pool_entry.path for character in ("\n", "\r", "\t")): + raise OSError("Treehouse pool path contains unsupported control characters") + entries = directory_entries(pool_entry.path, "Treehouse pool") + states = [entry for entry in entries if entry.name == "treehouse-state.json"] + if len(states) != 1: + raise OSError("Treehouse pool must contain exactly one treehouse-state.json") + state_paths.append(os.path.join(pool_entry.path, "treehouse-state.json")) + except OSError as error: + failed = True + print( + f"checkout-refresh: skipped: incomplete Treehouse coverage at {pool_entry.path}: {error}", + file=sys.stderr, + ) + +for state_path in state_paths: + try: + metadata = os.lstat(state_path) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode): + raise OSError("Treehouse state must be a real regular file") + permissions = stat.S_IMODE(metadata.st_mode) + if not permissions & 0o444: + raise PermissionError("Treehouse state is unreadable") + with open(state_path, encoding="utf-8") as stream: + state = json.load(stream) + if not isinstance(state, dict): + raise TypeError("root must be an object") + if "worktrees" not in state: + raise TypeError("worktrees is required") + worktrees = state["worktrees"] + if not isinstance(worktrees, list): + raise TypeError("worktrees must be an array") + for entry in worktrees: + if not isinstance(entry, dict): + raise TypeError("worktree entry must be an object") + path = entry.get("path") + if not isinstance(path, str) or not path: + raise TypeError("worktree path must be a non-empty string") + if not os.path.isabs(path): + raise TypeError("worktree path must be absolute") + if any(character in path for character in ("\n", "\r", "\t")): + raise TypeError("worktree path contains unsupported control characters") + print(f"{state_path}\t{os.path.dirname(state_path)}\t{path}") + except (OSError, ValueError, TypeError) as error: + failed = True + print( + f"checkout-refresh: skipped: incomplete Treehouse coverage at {state_path}: {error}", + file=sys.stderr, + ) +if failed: + raise SystemExit(1) +PY +} + +active_project_paths() { + if [ ! -e "$PROJECTS" ] && [ ! -L "$PROJECTS" ]; then + return 0 + fi + [ -d "$PROJECTS" ] || { + echo "checkout-refresh: skipped: incomplete active-home project coverage because the projects root is not a directory: $PROJECTS" >&2 + return 1 + } + command -v python3 >/dev/null 2>&1 || { + echo "checkout-refresh: skipped: incomplete active-home project coverage because python3 is unavailable" >&2 + return 1 + } + python3 - "$PROJECTS" <<'PY' +import os +import stat +import sys + +failed = False + + +def directory_entries(path, label): + if os.path.islink(path): + raise OSError(f"{label} must not be a symlink") + metadata = os.stat(path) + if not stat.S_ISDIR(metadata.st_mode): + raise NotADirectoryError(path) + permissions = stat.S_IMODE(metadata.st_mode) + if not permissions & 0o444 or not permissions & 0o111: + raise PermissionError(f"{label} is unreadable") + with os.scandir(path) as entries: + return sorted(entries, key=lambda entry: entry.name) + + +root = sys.argv[1] +try: + project_entries = directory_entries(root, "active-home projects root") +except OSError as error: + failed = True + project_entries = [] + print( + f"checkout-refresh: skipped: incomplete active-home project coverage at {root}: {error}", + file=sys.stderr, + ) + +for project_entry in project_entries: + try: + metadata = project_entry.stat(follow_symlinks=False) + if stat.S_ISLNK(metadata.st_mode): + try: + target = project_entry.stat(follow_symlinks=True) + except OSError as error: + raise OSError("broken active-home project symlink") from error + if stat.S_ISDIR(target.st_mode): + raise OSError("active-home project must not be a symlink") + continue + if not stat.S_ISDIR(metadata.st_mode): + continue + if any(character in project_entry.path for character in ("\n", "\r")): + raise OSError("active-home project path contains unsupported control characters") + directory_entries(project_entry.path, "active-home project") + print(project_entry.path) + except OSError as error: + failed = True + print( + f"checkout-refresh: skipped: incomplete active-home project coverage at {project_entry.path}: {error}", + file=sys.stderr, + ) + +if failed: + raise SystemExit(1) +PY +} + +backing_checkout() { + local worktree=$1 pool=$2 state=$3 worktree_root pool_root state_root main + local worktree_common main_common listed line listed_root matches=0 + worktree_root=$(exact_git_root "$worktree") || return 1 + pool_root=$(canonical_dir "$pool") || return 1 + [ "$worktree_root" != "$pool_root" ] || return 1 + case "$worktree_root" in "$pool_root"/*) ;; *) return 1 ;; esac + [ -f "$state" ] && [ ! -L "$state" ] && [ -r "$state" ] || return 1 + state_root=$(canonical_dir "$(dirname "$state")") || return 1 + [ "$state_root" = "$pool_root" ] || return 1 + [ "$(basename "$state")" = treehouse-state.json ] || return 1 + python3 - "$state" "$worktree_root" <<'PY' || return 1 +import json +import os +import sys + +state_path, expected = sys.argv[1:] +try: + with open(state_path, encoding="utf-8") as stream: + state = json.load(stream) + worktrees = state["worktrees"] + if not isinstance(worktrees, list): + raise TypeError("worktrees must be an array") + matches = [] + for entry in worktrees: + if not isinstance(entry, dict): + raise TypeError("worktree entry must be an object") + path = entry.get("path") + if not isinstance(path, str) or not path: + raise TypeError("worktree path must be a non-empty string") + if not os.path.isabs(path): + raise TypeError("worktree path must be absolute") + if os.path.realpath(path) == expected: + matches.append(entry) + if len(matches) != 1: + raise ValueError("expected exactly one matching worktree entry") +except (OSError, ValueError, TypeError, KeyError, json.JSONDecodeError): + raise SystemExit(1) +PY + listed=$(git -C "$worktree_root" -c core.quotePath=false worktree list --porcelain 2>/dev/null) || return 1 + main=$(printf '%s\n' "$listed" | sed -n 's/^worktree //p' | sed -n '1p') + [ -n "$main" ] || return 1 + main=$(exact_git_root "$main") || return 1 + [ "$main" != "$worktree_root" ] || return 1 + worktree_common=$(fm_checkout_git_common_dir "$worktree_root") || return 1 + main_common=$(fm_checkout_git_common_dir "$main") || return 1 + [ "$worktree_common" = "$main_common" ] || return 1 + while IFS= read -r line; do + case "$line" in + worktree\ *) + listed_root=$(exact_git_root "${line#worktree }" 2>/dev/null) || return 1 + [ "$listed_root" != "$worktree_root" ] || matches=$(( matches + 1 )) + ;; + esac + done <<EOF +$listed +EOF + [ "$matches" -eq 1 ] || return 1 + printf '%s\n' "$main" +} + +origin_url() { + git -C "$1" remote get-url origin 2>/dev/null +} + +CHECKOUT_ORIGIN_KIND= +CHECKOUT_ORIGIN_VALUE= +inspect_checkout_origin() { + local checkout=$1 remotes + CHECKOUT_ORIGIN_KIND= + CHECKOUT_ORIGIN_VALUE= + remotes=$(git -C "$checkout" remote 2>/dev/null) || return 1 + if printf '%s\n' "$remotes" | grep -Fxq origin; then + CHECKOUT_ORIGIN_VALUE=$(origin_url "$checkout") || return 1 + [ -n "$CHECKOUT_ORIGIN_VALUE" ] || return 1 + CHECKOUT_ORIGIN_KIND=origin + else + CHECKOUT_ORIGIN_KIND=no-origin + fi +} + +scan_root_candidates() { + command -v python3 >/dev/null 2>&1 || return 1 + python3 - "$1" <<'PY' +import os +import stat +import sys + +root = sys.argv[1] +try: + if os.path.islink(root): + raise OSError("scan root must not be a symlink") + metadata = os.stat(root) + permissions = stat.S_IMODE(metadata.st_mode) + if not stat.S_ISDIR(metadata.st_mode): + raise NotADirectoryError(root) + if not permissions & 0o444 or not permissions & 0o111: + raise PermissionError("scan root is unreadable") + with os.scandir(root) as entries: + for entry in sorted(entries, key=lambda candidate: candidate.name): + metadata = entry.stat(follow_symlinks=False) + if stat.S_ISLNK(metadata.st_mode): + try: + target = entry.stat(follow_symlinks=True) + except OSError as error: + raise OSError(f"broken scan candidate symlink: {entry.path}") from error + if stat.S_ISDIR(target.st_mode): + raise OSError(f"scan candidate must not be a symlink: {entry.path}") + continue + if stat.S_ISDIR(metadata.st_mode): + if any(character in entry.path for character in ("\n", "\r")): + raise OSError("scan candidate path contains unsupported control characters") + permissions = stat.S_IMODE(metadata.st_mode) + if not permissions & 0o444 or not permissions & 0o111: + raise PermissionError(f"scan candidate is unreadable: {entry.path}") + print(entry.path) +except OSError as error: + print(error, file=sys.stderr) + raise SystemExit(1) +PY +} + +prove_non_git_directory() { + command -v python3 >/dev/null 2>&1 || return 1 + python3 - "$1" <<'PY' +import os +import stat +import sys + +candidate = sys.argv[1] +try: + before = os.lstat(candidate) + permissions = stat.S_IMODE(before.st_mode) + if not stat.S_ISDIR(before.st_mode) or stat.S_ISLNK(before.st_mode): + raise OSError("candidate is not a real directory") + if not permissions & 0o444 or not permissions & 0o111: + raise PermissionError("candidate is unreadable") + with os.scandir(candidate) as entries: + snapshot = sorted( + ( + entry.name, + entry.inode(), + entry.stat(follow_symlinks=False).st_mode, + entry.stat(follow_symlinks=False).st_mtime_ns, + ) + for entry in entries + ) + if ( + os.environ.get("FM_CHECKOUT_REFRESH_TEST") == "1" + and os.environ.get("FM_CHECKOUT_TEST_CREATE_GIT_AT") == candidate + ): + os.mkdir(os.path.join(candidate, ".git")) + after = os.lstat(candidate) + with os.scandir(candidate) as entries: + confirmation = sorted( + ( + entry.name, + entry.inode(), + entry.stat(follow_symlinks=False).st_mode, + entry.stat(follow_symlinks=False).st_mtime_ns, + ) + for entry in entries + ) + if ( + (before.st_dev, before.st_ino, before.st_mtime_ns, before.st_ctime_ns) + != (after.st_dev, after.st_ino, after.st_mtime_ns, after.st_ctime_ns) + or snapshot != confirmation + ): + raise OSError("candidate identity changed") + if any(name == ".git" for name, *_ in confirmation): + raise OSError("candidate contains Git metadata") + try: + os.lstat(os.path.join(candidate, ".git")) + except FileNotFoundError: + pass + else: + raise OSError("candidate gained Git metadata") +except OSError: + raise SystemExit(1) +PY +} + +manifest_create() { + : > "$1" +} + +manifest_append() { + [ "${FM_CHECKOUT_REFRESH_TEST:-0}:${FM_CHECKOUT_TEST_MANIFEST_FAILURE:-}" != "1:append" ] || return 1 + printf '%s\n' "$2" >> "$1" +} + +manifest_append_pair() { + [ "${FM_CHECKOUT_REFRESH_TEST:-0}:${FM_CHECKOUT_TEST_MANIFEST_FAILURE:-}" != "1:append" ] || return 1 + printf '%s\t%s\n' "$2" "$3" >> "$1" +} + +manifest_sort_unique() { + local path=$1 sorted + [ "${FM_CHECKOUT_REFRESH_TEST:-0}:${FM_CHECKOUT_TEST_MANIFEST_FAILURE:-}" != "1:sort" ] || return 1 + sorted=$(mktemp "${path}.sorted.XXXXXX") || return 1 + if ! sort -u "$path" > "$sorted" || ! mv "$sorted" "$path"; then + rm -f "$sorted" || true + return 1 + fi +} + +cleanup_discovery_tmp() { + [ "${FM_CHECKOUT_REFRESH_TEST:-0}:${FM_CHECKOUT_TEST_MANIFEST_FAILURE:-}" != "1:cleanup" ] || return 1 + rm -rf "$1" +} + +discover() { + local tmp seeds origins scans scan_candidates configured_paths configured_scans treehouse_paths project_paths + local path project worktree pool treehouse_state main root candidate candidate_output url failed=0 + tmp=$(mktemp -d "${TMPDIR:-/tmp}/fm-checkout-refresh-discover.XXXXXX") || return 1 + seeds="$tmp/seeds" + origins="$tmp/origins" + scans="$tmp/scans" + scan_candidates="$tmp/scan-candidates" + configured_paths="$tmp/configured-paths" + configured_scans="$tmp/configured-scans" + treehouse_paths="$tmp/treehouse-worktrees" + project_paths="$tmp/active-projects" + manifest_create "$seeds" || { cleanup_discovery_tmp "$tmp" || true; return 1; } + manifest_create "$origins" || { cleanup_discovery_tmp "$tmp" || true; return 1; } + manifest_create "$scans" || { cleanup_discovery_tmp "$tmp" || true; return 1; } + manifest_create "$scan_candidates" || { cleanup_discovery_tmp "$tmp" || true; return 1; } + if ! parse_config "$configured_paths" "$configured_scans"; then + cleanup_discovery_tmp "$tmp" || true + return 1 + fi + if ! treehouse_worktree_paths > "$treehouse_paths"; then + cleanup_discovery_tmp "$tmp" || true + return 1 + fi + if ! active_project_paths > "$project_paths"; then + cleanup_discovery_tmp "$tmp" || true + return 1 + fi + while IFS= read -r project; do + [ -n "$project" ] || continue + if main=$(exact_git_root "$project"); then + manifest_append "$seeds" "$main" || failed=1 + else + echo "checkout-refresh: skipped: active-home project is not an exact inspectable Git repository root: $project" >&2 + failed=1 + fi + done < "$project_paths" + + while IFS= read -r path; do + [ -n "$path" ] || continue + if main=$(exact_git_root "$path"); then + manifest_append "$seeds" "$main" || failed=1 + else + echo "checkout-refresh: skipped: configured checkout is not an exact inspectable Git repository root: $path" >&2 + failed=1 + fi + done < "$configured_paths" + + while IFS=$'\t' read -r treehouse_state pool worktree; do + [ -n "$worktree" ] || continue + if main=$(backing_checkout "$worktree" "$pool" "$treehouse_state" 2>/dev/null); then + manifest_append "$seeds" "$main" || failed=1 + else + echo "checkout-refresh: skipped: Treehouse worktree identity or registration is not inspectable: $worktree" >&2 + failed=1 + fi + done < "$treehouse_paths" + manifest_sort_unique "$seeds" || failed=1 + while IFS= read -r path; do + [ -n "$path" ] || continue + if ! inspect_checkout_origin "$path"; then + echo "checkout-refresh: skipped: covered checkout origin identity cannot be inspected: $path" >&2 + failed=1 + continue + fi + if [ "$CHECKOUT_ORIGIN_KIND" = origin ]; then + manifest_append "$origins" "$CHECKOUT_ORIGIN_VALUE" || failed=1 + fi + done < "$seeds" + manifest_sort_unique "$origins" || failed=1 + + if main=$(canonical_dir "$HOME" 2>/dev/null); then + manifest_append "$scans" "$main" || failed=1 + else + echo "checkout-refresh: skipped: home scan root is not inspectable: $HOME" >&2 + failed=1 + fi + while IFS= read -r root; do + [ -n "$root" ] || continue + if main=$(canonical_dir "$root" 2>/dev/null); then + manifest_append "$scans" "$main" || failed=1 + else + echo "checkout-refresh: skipped: configured scan root is not a directory: $root" >&2 + failed=1 + fi + done < "$configured_scans" + manifest_sort_unique "$scans" || failed=1 + + while IFS= read -r root; do + [ -n "$root" ] || continue + candidate_output=$(scan_root_candidates "$root") || { + echo "checkout-refresh: skipped: scan root is unreadable or cannot be enumerated: $root" >&2 + failed=1 + continue + } + while IFS= read -r candidate; do + [ -n "$candidate" ] || continue + manifest_append "$scan_candidates" "$candidate" || failed=1 + done <<EOF +$candidate_output +EOF + done < "$scans" + + while IFS= read -r candidate; do + [ -n "$candidate" ] || continue + if ! git -C "$candidate" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + if prove_non_git_directory "$candidate"; then + continue + fi + echo "checkout-refresh: skipped: discovered Git identity cannot be inspected or disproved: $candidate" >&2 + failed=1 + continue + fi + if ! main=$(exact_git_root "$candidate"); then + echo "checkout-refresh: skipped: discovered clone is not an exact inspectable Git repository root: $candidate" >&2 + failed=1 + continue + fi + if ! inspect_checkout_origin "$main"; then + echo "checkout-refresh: skipped: discovered checkout origin identity cannot be inspected: $main" >&2 + failed=1 + continue + fi + [ "$CHECKOUT_ORIGIN_KIND" = origin ] || continue + url=$CHECKOUT_ORIGIN_VALUE + if [ ! -s "$origins" ] || ! grep -Fxq -- "$url" "$origins"; then + continue + fi + manifest_append "$seeds" "$main" || failed=1 + if [ -n "${DISCOVERY_IDENTITIES_FILE:-}" ]; then + manifest_append_pair "$DISCOVERY_IDENTITIES_FILE" "$main" "$url" || failed=1 + fi + done < "$scan_candidates" + + if [ -n "${DISCOVERY_IDENTITIES_FILE:-}" ]; then + manifest_sort_unique "$DISCOVERY_IDENTITIES_FILE" || failed=1 + fi + [ "$failed" -eq 0 ] || { + cleanup_discovery_tmp "$tmp" || true + return 1 + } + manifest_sort_unique "$seeds" || { + cleanup_discovery_tmp "$tmp" || true + return 1 + } + cat "$seeds" || { + cleanup_discovery_tmp "$tmp" || true + return 1 + } + cleanup_discovery_tmp "$tmp" +} + +acquire_worktree() { + local expected_source=$1 lease_holder=$2 status=0 canonical + canonical=$(require_exact_git_root "$expected_source" "Treehouse acquisition source") || return 1 + expected_source=$canonical + ( + local checkout_lock process_guard + ensure_lock_roots || exit 1 + checkout_lock=$(fm_checkout_lock_path "$expected_source" "$LOCK_ROOT") || { + echo "error: cannot resolve Treehouse acquisition lock identity for $expected_source" >&2 + exit 1 + } + if ! fm_lock_try_acquire "$checkout_lock"; then + echo "error: Treehouse acquisition already running for $expected_source (pid ${FM_LOCK_HELD_PID:-unknown})" >&2 + exit 1 + fi + process_guard="${FM_LOCK_OWNER_DIR:?}/process-group" + trap 'fm_lock_release "$checkout_lock"' EXIT + cd "$expected_source" || exit 1 + if FM_PROCESS_TREE_GUARD_FILE="$process_guard" \ + fm_run_bounded "$ACQUIRE_TIMEOUT" treehouse get --lease --lease-holder "$lease_holder"; then + status=0 + else + status=$? + fi + if ! fm_process_tree_cleanup_verified; then + exit "$FM_CHECKOUT_PROCESS_CLEANUP_FAILURE_STATUS" + fi + exit "$status" + ) || status=$? + case "$status" in + 0) return 0 ;; + 124) + echo "error: Treehouse worktree acquisition timed out after ${ACQUIRE_TIMEOUT}s and terminated its process tree" >&2 + ;; + "$FM_CHECKOUT_PROCESS_CLEANUP_FAILURE_STATUS") + echo "error: Treehouse worktree acquisition process cleanup could not be verified; no worktree was accepted for $lease_holder" >&2 + ;; + *) + echo "error: Treehouse worktree acquisition failed for $lease_holder (exit $status)" >&2 + ;; + esac + return "$status" +} + +PROBE_BRANCH= +PROBE_TIP= +probe_upstream() { + local checkout=$1 out line ref status + PROBE_BRANCH= + PROBE_TIP= + if fm_run_bounded_capture out "$PROBE_TIMEOUT" \ + git -C "$checkout" ls-remote --symref origin HEAD 2>/dev/null; then + status=0 + else + status=$? + fi + [ "$status" -eq 0 ] && fm_process_tree_cleanup_verified || return 1 + while IFS= read -r line; do + case "$line" in + "ref: refs/heads/"*$'\t'"HEAD") + ref=${line#ref: refs/heads/} + PROBE_BRANCH=${ref%$'\t'HEAD} + ;; + *$'\t'"HEAD") + PROBE_TIP=${line%$'\t'HEAD} + ;; + esac + done <<EOF +$out +EOF + [ -n "$PROBE_BRANCH" ] && [ -n "$PROBE_TIP" ] +} + +checkout_key() { + local checkout + checkout=$(exact_git_root "$1") || return 1 + fm_checkout_hash_value "$checkout" 24 +} + +read_epoch() { + local value + value=$(sed -n '1p' "$1" 2>/dev/null || true) + case "$value" in ''|*[!0-9]*) echo 0 ;; *) echo "$value" ;; esac +} + +atomic_write() { + local destination=$1 + shift + local tmp + tmp=$(mktemp "$STATE_ROOT/.checkout-refresh-write.XXXXXX") || return 1 + printf '%s\n' "$@" > "$tmp" || { rm -f "$tmp"; return 1; } + chmod 600 "$tmp" || { rm -f "$tmp"; return 1; } + if ! atomic_replace "$tmp" "$destination"; then + rm -f "$tmp" + return 1 + fi +} + +atomic_copy() { + local source=$1 destination=$2 tmp + tmp=$(mktemp "$STATE_ROOT/.checkout-refresh-write.XXXXXX") || return 1 + cp "$source" "$tmp" || { rm -f "$tmp"; return 1; } + chmod 600 "$tmp" || { rm -f "$tmp"; return 1; } + if ! atomic_replace "$tmp" "$destination"; then + rm -f "$tmp" + return 1 + fi +} + +atomic_replace() { + local source=$1 destination=$2 + python3 - "$source" "$destination" <<'PY' +import os +import stat +import sys + +source, destination = sys.argv[1:] +try: + if os.path.lexists(destination): + metadata = os.lstat(destination) + if not stat.S_ISREG(metadata.st_mode): + raise OSError("destination is not a regular file") + os.replace(source, destination) + metadata = os.lstat(destination) + if not stat.S_ISREG(metadata.st_mode): + raise OSError("replacement is not a regular file") +except OSError: + raise SystemExit(1) +PY +} + +validate_external_identity_history() { + local current=$1 prior="$STATE_ROOT/external-identities" path recorded_origin inspected current_origin failed=0 + if [ ! -e "$prior" ] && [ ! -L "$prior" ]; then + return 0 + fi + if [ -L "$prior" ] || [ ! -f "$prior" ] || [ ! -r "$prior" ]; then + echo "checkout-refresh: skipped: prior external checkout identity manifest is unsafe or unreadable: $prior" >&2 + return 1 + fi + if ! awk -F '\t' 'NF != 2 || $1 == "" || $2 == "" { failed = 1 } END { exit failed }' "$prior"; then + echo "checkout-refresh: skipped: prior external checkout identity manifest is malformed: $prior" >&2 + return 1 + fi + while IFS=$'\t' read -r path recorded_origin; do + [ -n "$path" ] || continue + if grep -Fqx -- "$path"$'\t'"$recorded_origin" "$current"; then + continue + fi + if inspected=$(exact_git_root "$path" 2>/dev/null); then + current_origin=$(origin_url "$inspected" || true) + if [ "$inspected" != "$path" ]; then + echo "checkout-refresh: skipped: prior external checkout root identity drifted at $path" >&2 + elif [ -z "$current_origin" ]; then + echo "checkout-refresh: skipped: prior external checkout origin is unreadable at $path" >&2 + else + echo "checkout-refresh: skipped: prior external checkout origin changed at $path" >&2 + fi + else + echo "checkout-refresh: skipped: prior external checkout disappeared or became unreadable at $path" >&2 + fi + failed=1 + done < "$prior" + [ "$failed" -eq 0 ] +} + +record_run_result() { + local coverage=$1 scheduled=${2:-0} now write_status=0 + now=$(date +%s) + if [ "$scheduled" -eq 1 ]; then + atomic_write "$STATE_ROOT/heartbeat" "$now" "$SCRIPT_DIR/fm-checkout-refresh.sh" \ + || write_status=1 + fi + atomic_write "$STATE_ROOT/coverage-health" "$now" "$coverage" \ + || write_status=1 + if [ "$write_status" -eq 0 ] && [ "$scheduled" -eq 1 ] \ + && [ -n "$SCHEDULER_GENERATION" ]; then + atomic_write "$STATE_ROOT/scheduler-generation" "$SCHEDULER_GENERATION" \ + || write_status=1 + fi + return "$write_status" +} + +ensure_state_root() { + mkdir -p "$STATE_ROOT" || return 1 + [ -d "$STATE_ROOT" ] && [ ! -L "$STATE_ROOT" ] \ + || { echo "error: unsafe checkout-refresh state directory: $STATE_ROOT" >&2; return 1; } +} + +launch_agent_loaded_state() { + local domain=$1 label=$2 plist=${3:-} output status + output=$("$LAUNCHCTL" print "$domain/$label" 2>&1) + status=$? + if [ "$status" -ne 0 ]; then + case "$output" in + *"Could not find service"*|*"service not found"*|*"No such process"*) + return 3 + ;; + *) + echo "error: cannot prove whether checkout-refresh LaunchAgent $label is loaded" >&2 + return 4 + ;; + esac + fi + [ -n "$plist" ] && [ -f "$plist" ] && [ ! -L "$plist" ] && [ -r "$plist" ] || { + echo "error: loaded checkout-refresh LaunchAgent $label has no authoritative definition" >&2 + return 4 + } + FM_CHECKOUT_LOADED_JOB=$output python3 - "$plist" "$domain/$label" \ + "$LABEL" "$SCRIPT_DIR/fm-checkout-refresh.sh" <<'PY' +import os +import plistlib +import re +import sys + +plist_path, target, current_label, script = sys.argv[1:] +loaded = os.environ.get("FM_CHECKOUT_LOADED_JOB", "") + +def decode(value): + value = value.strip() + if len(value) >= 2 and value[0] == value[-1] == '"': + value = value[1:-1] + return value + +def scalar(name): + matches = re.findall( + rf"(?m)^[ \t]*{re.escape(name)}[ \t]*=[ \t]*(.+?)[ \t]*$", + loaded, + ) + if len(matches) != 1: + raise ValueError(f"ambiguous {name}") + return decode(matches[0]) + +def block(name): + match = re.search( + rf"(?ms)^[ \t]*{re.escape(name)}[ \t]*=[ \t]*\{{[ \t]*\n(.*?)^[ \t]*\}}[ \t]*$", + loaded, + ) + if match is None: + raise ValueError(f"missing {name}") + if re.search( + rf"(?m)^[ \t]*{re.escape(name)}[ \t]*=[ \t]*\{{", + loaded[: match.start()] + loaded[match.end() :], + ): + raise ValueError(f"duplicate {name}") + return [line.strip() for line in match.group(1).splitlines() if line.strip()] + +def optional_scalar(name): + matches = re.findall( + rf"(?m)^[ \t]*{re.escape(name)}[ \t]*=[ \t]*(.+?)[ \t]*$", + loaded, + ) + if len(matches) > 1: + raise ValueError(f"ambiguous {name}") + return decode(matches[0]) if matches else None + +def environment_block(name): + lines = block(name) + values = {} + for line in lines: + if " => " not in line: + raise ValueError(f"malformed loaded {name}") + key, value = line.split(" => ", 1) + key = decode(key) + if not key or key in values: + raise ValueError(f"ambiguous loaded {name}") + values[key] = decode(value) + return values + +try: + first = next(line.strip() for line in loaded.splitlines() if line.strip()) + if first != target + " = {": + raise ValueError("loaded target differs") + with open(plist_path, "rb") as stream: + definition = plistlib.load(stream) + expected_arguments = definition.get("ProgramArguments") + expected_environment = definition.get("EnvironmentVariables") + expected_interval = definition.get("StartInterval") + expected_run_at_load = definition.get("RunAtLoad") + if ( + definition.get("Label") != target.rsplit("/", 1)[-1] + or not isinstance(expected_arguments, list) + or not expected_arguments + or not all(isinstance(value, str) for value in expected_arguments) + or not isinstance(expected_environment, dict) + or not all( + isinstance(key, str) and isinstance(value, str) + for key, value in expected_environment.items() + ) + or not isinstance(expected_interval, int) + or isinstance(expected_interval, bool) + or expected_interval <= 0 + or expected_run_at_load is not True + ): + raise ValueError("authoritative definition is malformed") + if definition["Label"] == current_label: + expected_keys = { + "HOME", + "PATH", + "FM_HOME", + "FM_TREEHOUSE_ROOT", + "FM_CHECKOUT_REFRESH_STATE_ROOT", + "FM_CHECKOUT_REFRESH_LOCK_ROOT", + "FM_CHECKOUT_REFRESH_INTERVAL", + "FM_CHECKOUT_REFRESH_BACKSTOP", + "FM_CHECKOUT_REFRESH_GENERATION", + } + if ( + len(expected_arguments) != 4 + or not os.path.isabs(expected_arguments[0]) + or expected_arguments[1:] != [script, "run-once", "--scheduled"] + or set(expected_environment) != expected_keys + ): + raise ValueError("authoritative control surface is malformed") + if scalar("path") != os.path.abspath(plist_path): + raise ValueError("loaded definition path differs") + if scalar("program") != expected_arguments[0]: + raise ValueError("loaded program differs") + arguments = [decode(line) for line in block("arguments")] + if arguments != expected_arguments: + raise ValueError("loaded arguments differ") + interval = scalar("run interval") + match = re.fullmatch(r"([1-9][0-9]*)(?:[ \t]+seconds?)?", interval) + if match is None or int(match.group(1)) != expected_interval: + raise ValueError("loaded run interval differs") + run_at_load = optional_scalar("run at load") + properties = optional_scalar("properties") + if run_at_load is not None: + if run_at_load.lower() not in ("true", "1"): + raise ValueError("loaded RunAtLoad differs") + elif properties is None or "runatload" not in properties.lower().split(): + raise ValueError("loaded RunAtLoad is missing") + inherited_environment = environment_block("inherited environment") + default_environment = environment_block("default environment") + environment = environment_block("environment") + harmless_inherited = { + "LOGNAME", + "OSLogRateLimit", + "SECURITYSESSIONID", + "SHELL", + "TMPDIR", + "USER", + "XPC_FLAGS", + "XPC_SERVICE_NAME", + "__CF_USER_TEXT_ENCODING", + } + for inherited in (inherited_environment, default_environment): + if any( + key not in expected_environment and key not in harmless_inherited + for key in inherited + ): + raise ValueError("loaded inherited environment has undeclared control") + synthesized_environment = { + "XPC_SERVICE_NAME": definition["Label"], + "OSLogRateLimit": "64", + } + for key, value in environment.items(): + if key in expected_environment: + if expected_environment[key] != value: + raise ValueError(f"loaded environment differs for {key}") + elif synthesized_environment.get(key) != value: + raise ValueError(f"loaded environment has undeclared control {key}") + if any(key not in environment for key in expected_environment): + raise ValueError("loaded environment is incomplete") +except (OSError, StopIteration, ValueError, plistlib.InvalidFileException): + raise SystemExit(1) +PY + status=$? + unset FM_CHECKOUT_LOADED_JOB + if [ "$status" -ne 0 ]; then + echo "error: loaded checkout-refresh LaunchAgent $label identity differs from $plist" >&2 + return 4 + fi +} + +quiesce_launch_agent_verified() { + local domain=$1 label=$2 plist=$3 state output status + launch_agent_loaded_state "$domain" "$label" "$plist" + state=$? + case "$state" in + 3) return 0 ;; + 0) ;; + *) return 1 ;; + esac + output=$("$LAUNCHCTL" bootout "$domain/$label" 2>&1) + status=$? + [ "$status" -eq 0 ] || { + echo "error: cannot quiesce checkout-refresh LaunchAgent $label: ${output:-bootout failed}" >&2 + return 1 + } + launch_agent_loaded_state "$domain" "$label" "$plist" + state=$? + [ "$state" -eq 3 ] || { + echo "error: checkout-refresh LaunchAgent $label remains live after bootout" >&2 + return 1 + } +} + +discover_home_launch_agent_namespaces() { + local discovered label state count=0 logical_count=0 logical_state= + [ -e "$LAUNCH_AGENTS_DIR" ] || [ -L "$LAUNCH_AGENTS_DIR" ] || return 0 + discovered=$(python3 - "$LAUNCH_AGENTS_DIR" "$LABEL_BASE" \ + "$SCRIPT_DIR/fm-checkout-refresh.sh" "$FM_HOME_CANONICAL" "$FM_HOME" "$STATE_BASE" <<'PY' +import os +import re +import stat +import sys +import xml.etree.ElementTree as ET + +root, label_base, script, canonical_home, raw_home, state_base = sys.argv[1:] + +def dictionary(element): + children = list(element) + if len(children) % 2: + raise OSError("malformed plist dictionary") + values = {} + for index in range(0, len(children), 2): + key = children[index] + value = children[index + 1] + if key.tag != "key" or not key.text or key.text in values: + raise OSError("malformed or duplicate plist key") + values[key.text] = value + return values + +def text_value(values, key): + value = values.get(key) + if value is None or value.tag != "string" or value.text is None: + raise OSError("missing plist string") + return value.text + +def safe_directory(path): + if not os.path.isabs(path) or path == os.path.sep: + raise OSError("unsafe state root") + normalized = os.path.normpath(path) + current = os.path.sep + for component in normalized.split(os.path.sep): + if not component: + continue + current = os.path.join(current, component) + metadata = os.lstat(current) + if stat.S_ISLNK(metadata.st_mode): + raise OSError("redirected state root") + metadata = os.lstat(normalized) + if not stat.S_ISDIR(metadata.st_mode) or os.path.realpath(normalized) != normalized: + raise OSError("unsafe state root") + return normalized + +try: + root_meta = os.lstat(root) + if stat.S_ISLNK(root_meta.st_mode) or not stat.S_ISDIR(root_meta.st_mode): + raise OSError("unsafe launch agent directory") + entries = sorted(os.scandir(root), key=lambda entry: entry.name) + for entry in entries: + if entry.name != label_base + ".plist" and ( + not entry.name.startswith(label_base + ".") or not entry.name.endswith(".plist") + ): + continue + meta = entry.stat(follow_symlinks=False) + if stat.S_ISLNK(meta.st_mode) or not stat.S_ISREG(meta.st_mode): + raise OSError("unsafe launch agent definition") + document = ET.parse(entry.path) + root_element = document.getroot() + plist_dictionary = root_element.find("dict") + if plist_dictionary is None: + raise OSError("missing plist dictionary") + values = dictionary(plist_dictionary) + authoritative_label = text_value(values, "Label") + arguments = values.get("ProgramArguments") + environment = values.get("EnvironmentVariables") + if arguments is None or arguments.tag != "array": + raise OSError("missing program arguments") + if environment is None or environment.tag != "dict": + raise OSError("missing environment") + program_arguments = [ + argument.text + for argument in list(arguments) + if argument.tag == "string" and argument.text is not None + ] + if len(program_arguments) != len(list(arguments)): + raise OSError("malformed program arguments") + environment_values = dictionary(environment) + parsed_home = text_value(environment_values, "FM_HOME") + mentions_home = parsed_home in (canonical_home, raw_home) + if parsed_home and not mentions_home: + continue + if ( + not mentions_home + or len(program_arguments) != 4 + or not os.path.isabs(program_arguments[0]) + or program_arguments[1:] != [script, "run-once", "--scheduled"] + ): + raise OSError("incomplete launch agent identity") + label = entry.name[:-6] + if authoritative_label != label: + raise OSError("launch agent filename and Label differ") + if label == label_base: + suffix = "" + else: + suffix = label[len(label_base) + 1 :] + if suffix and re.fullmatch(r"[0-9a-f]{16}", suffix) is None: + raise OSError("invalid launch agent label") + parsed_state = safe_directory( + text_value(environment_values, "FM_CHECKOUT_REFRESH_STATE_ROOT") + ) + print(f"{label}\t{parsed_state}") +except (ET.ParseError, OSError, UnicodeError): + raise SystemExit(1) +PY + ) || { + echo "error: checkout-refresh LaunchAgent namespaces cannot be safely enumerated" >&2 + return 1 + } + while IFS=$'\t' read -r label state; do + [ -n "$label" ] || continue + if [ "$label" = "$LABEL" ]; then + logical_state=$state + logical_count=$((logical_count + 1)) + continue + fi + PHYSICAL_LABEL=$label + PHYSICAL_PLIST="$LAUNCH_AGENTS_DIR/$label.plist" + PHYSICAL_STATE_ROOT=$state + count=$((count + 1)) + done <<EOF +$discovered +EOF + [ "$logical_count" -le 1 ] && [ "$count" -le 1 ] || { + echo "error: ambiguous checkout-refresh LaunchAgent namespaces for $FM_HOME_CANONICAL" >&2 + return 1 + } + if [ "$logical_count" -eq 1 ] && [ "$STATE_ROOT_EXPLICIT" -eq 0 ] \ + && [ "$logical_state" != "$DEFAULT_STATE_ROOT" ]; then + STATE_ROOT=$logical_state + CUSTOM_STATE_ROOT=1 + elif [ "$logical_count" -eq 1 ] && [ "$logical_state" != "$STATE_ROOT" ]; then + echo "error: checkout-refresh LaunchAgent state root drifted from the requested namespace" >&2 + return 1 + fi + if [ "$count" -eq 1 ]; then + if [ "$STATE_ROOT_EXPLICIT" -eq 1 ] && [ "$STATE_ROOT" != "$PHYSICAL_STATE_ROOT" ]; then + echo "error: checkout-refresh legacy LaunchAgent state root conflicts with the requested namespace" >&2 + return 1 + fi + case "$PHYSICAL_STATE_ROOT" in + "$STATE_BASE"/homes/[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]) + ;; + *) + STATE_ROOT=$PHYSICAL_STATE_ROOT + CUSTOM_STATE_ROOT=1 + return 0 + ;; + esac + case "${FM_CHECKOUT_REFRESH_STATE_ROOT:-$DEFAULT_STATE_ROOT}" in + "$PHYSICAL_STATE_ROOT") + STATE_ROOT=$PHYSICAL_STATE_ROOT + USING_PHYSICAL_STATE_ROOT=1 + CUSTOM_STATE_ROOT=0 + ;; + esac + fi +} + +stage_home_state_namespace() { + local source=$1 destination=$2 parent + parent=${destination%/*} + python3 - "$source" "$destination" "$parent" <<'PY' +import os +import shutil +import stat +import sys +import tempfile + +source, destination, parent = sys.argv[1:] +staging = "" + +def failed(error): + raise error + +try: + source_meta = os.lstat(source) + parent_meta = os.lstat(parent) + if stat.S_ISLNK(source_meta.st_mode) or not stat.S_ISDIR(source_meta.st_mode): + raise OSError("unsafe source namespace") + if stat.S_ISLNK(parent_meta.st_mode) or not stat.S_ISDIR(parent_meta.st_mode): + raise OSError("unsafe namespace parent") + if os.path.lexists(destination): + raise OSError("destination namespace exists") + staging = tempfile.mkdtemp(prefix=".home-state-migration.", dir=parent) + payload = os.path.join(staging, "state") + os.mkdir(payload, 0o700) + for current, directories, files in os.walk( + source, topdown=True, followlinks=False, onerror=failed + ): + relative = os.path.relpath(current, source) + target = payload if relative == "." else os.path.join(payload, relative) + safe_directories = [] + for name in sorted(directories): + if relative == "." and ( + name == ".run-lock" or name.startswith(".run-lock.") + ): + continue + path = os.path.join(current, name) + metadata = os.lstat(path) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode): + raise OSError("unsafe state directory") + os.mkdir(os.path.join(target, name), metadata.st_mode & 0o777) + safe_directories.append(name) + directories[:] = safe_directories + for name in sorted(files): + if relative == "." and ( + name == ".run-lock" or name.startswith(".run-lock.") + ): + continue + path = os.path.join(current, name) + metadata = os.lstat(path) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode): + raise OSError("unsafe state file") + target_file = os.path.join(target, name) + shutil.copyfile(path, target_file) + os.chmod(target_file, metadata.st_mode & 0o777) + with open(target_file, "rb") as stream: + os.fsync(stream.fileno()) + if os.environ.get("FM_CHECKOUT_REFRESH_TEST") == "1" and os.environ.get( + "FM_CHECKOUT_TEST_HOME_MIGRATION_FAILURE" + ) == "stage": + raise OSError("injected staging failure") + os.replace(payload, destination) + os.rmdir(staging) +except OSError: + if staging: + shutil.rmtree(staging, ignore_errors=True) + raise SystemExit(1) +PY +} + +remove_home_state_namespace() { + python3 - "$1" "${1%/*}" <<'PY' +import os +import shutil +import stat +import sys + +target, parent = sys.argv[1:] +try: + parent_real = os.path.realpath(parent) + target_meta = os.lstat(target) + if stat.S_ISLNK(target_meta.st_mode) or not stat.S_ISDIR(target_meta.st_mode): + raise OSError("unsafe state namespace") + if os.path.dirname(os.path.realpath(target)) != parent_real: + raise OSError("escaping state namespace") + shutil.rmtree(target) +except OSError: + raise SystemExit(1) +PY +} + +prepare_home_state_namespace() { + local parent command=${1:-} loaded_state + discover_home_launch_agent_namespaces || return 1 + if [ "$CUSTOM_STATE_ROOT" -eq 1 ]; then + if [ "$PHYSICAL_LABEL" != "$LABEL" ] \ + && { [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ]; }; then + [ "$command" = ensure ] || [ "$command" = install ] || { + echo "error: checkout-refresh legacy LaunchAgent requires an ensure or install migration before use" >&2 + return 1 + } + [ "$PLATFORM" = Darwin ] || { + echo "error: checkout-refresh legacy LaunchAgent migration requires macOS" >&2 + return 1 + } + command -v "$LAUNCHCTL" >/dev/null 2>&1 || return 1 + validate_launch_agent_namespaces || return 1 + HOME_MIGRATION_ACTIVE=1 + fi + return 0 + fi + [ "$DEFAULT_STATE_ROOT" != "$PHYSICAL_STATE_ROOT" ] || return 0 + parent=${DEFAULT_STATE_ROOT%/*} + if { [ -e "$DEFAULT_STATE_ROOT" ] || [ -L "$DEFAULT_STATE_ROOT" ]; } \ + && { [ -e "$PHYSICAL_STATE_ROOT" ] || [ -L "$PHYSICAL_STATE_ROOT" ]; }; then + echo "error: ambiguous checkout-refresh home state namespaces for $FM_HOME_CANONICAL" >&2 + return 1 + fi + if [ "$USING_PHYSICAL_STATE_ROOT" -eq 1 ]; then + [ -d "$PHYSICAL_STATE_ROOT" ] && [ ! -L "$PHYSICAL_STATE_ROOT" ] || { + echo "error: unsafe physical checkout-refresh home state namespace: $PHYSICAL_STATE_ROOT" >&2 + return 1 + } + return 0 + fi + if [ -e "$DEFAULT_STATE_ROOT" ] || [ -L "$DEFAULT_STATE_ROOT" ]; then + [ -d "$DEFAULT_STATE_ROOT" ] && [ ! -L "$DEFAULT_STATE_ROOT" ] || { + echo "error: unsafe logical checkout-refresh home state namespace: $DEFAULT_STATE_ROOT" >&2 + return 1 + } + fi + if { [ -e "$PHYSICAL_STATE_ROOT" ] || [ -L "$PHYSICAL_STATE_ROOT" ] \ + || [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ]; } \ + && [ "$command" != ensure ] && [ "$command" != install ]; then + echo "error: checkout-refresh physical state requires an ensure or install migration before use" >&2 + return 1 + fi + if [ ! -e "$PHYSICAL_STATE_ROOT" ] && [ ! -L "$PHYSICAL_STATE_ROOT" ] \ + && [ ! -e "$PHYSICAL_PLIST" ] && [ ! -L "$PHYSICAL_PLIST" ]; then + return 0 + fi + if [ -e "$PHYSICAL_STATE_ROOT" ] || [ -L "$PHYSICAL_STATE_ROOT" ]; then + [ -d "$PHYSICAL_STATE_ROOT" ] && [ ! -L "$PHYSICAL_STATE_ROOT" ] || { + echo "error: unsafe physical checkout-refresh home state namespace: $PHYSICAL_STATE_ROOT" >&2 + return 1 + } + mkdir -p "$parent" || return 1 + [ -d "$parent" ] && [ ! -L "$parent" ] || return 1 + fi + if [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ]; then + if [ ! -e "$PHYSICAL_STATE_ROOT" ] && [ ! -L "$PHYSICAL_STATE_ROOT" ]; then + echo "error: physical checkout-refresh LaunchAgent has no durable state namespace to migrate" >&2 + return 1 + fi + [ "$PLATFORM" = Darwin ] || { + echo "error: checkout-refresh physical LaunchAgent migration requires macOS" >&2 + return 1 + } + command -v "$LAUNCHCTL" >/dev/null 2>&1 || return 1 + validate_launch_agent_namespaces || return 1 + launch_agent_loaded_state "gui/$(id -u)" "$PHYSICAL_LABEL" "$PHYSICAL_PLIST" + loaded_state=$? + case "$loaded_state" in + 0) + quiesce_launch_agent_verified \ + "gui/$(id -u)" "$PHYSICAL_LABEL" "$PHYSICAL_PLIST" || return 1 + PHYSICAL_AGENT_STOPPED=1 + ;; + 3) ;; + *) return 1 ;; + esac + fi + if [ -e "$PHYSICAL_STATE_ROOT" ] || [ -L "$PHYSICAL_STATE_ROOT" ]; then + fm_checkout_lock_prepare "$LOCK_ROOT" || { + rollback_home_state_namespace_migration || true + return 1 + } + HOME_MIGRATION_RUN_LOCK="$PHYSICAL_STATE_ROOT/.run-lock" + fm_lock_try_acquire "$HOME_MIGRATION_RUN_LOCK" || { + echo "error: checkout-refresh physical state is busy; migration retained the prior namespace" >&2 + HOME_MIGRATION_RUN_LOCK= + rollback_home_state_namespace_migration || true + return 1 + } + stage_home_state_namespace "$PHYSICAL_STATE_ROOT" "$DEFAULT_STATE_ROOT" || { + echo "error: cannot stage checkout-refresh home state in its logical namespace" >&2 + rollback_home_state_namespace_migration || true + return 1 + } + HOME_STATE_NAMESPACE_STAGED=1 + fi + HOME_MIGRATION_ACTIVE=1 +} + +rollback_home_state_namespace_migration() { + local failed=0 loaded_state domain + domain="gui/$(id -u)" + if [ "$HOME_MIGRATION_ACTIVE" -eq 1 ] && [ "$PLATFORM" = Darwin ]; then + if [ -e "$PLIST" ] || [ -L "$PLIST" ]; then + if [ -f "$PLIST" ] && [ ! -L "$PLIST" ] \ + && quiesce_launch_agent_verified "$domain" "$LABEL" "$PLIST"; then + rm -f "$PLIST" || failed=1 + else + return 1 + fi + else + launch_agent_loaded_state "$domain" "$LABEL" "" + loaded_state=$? + [ "$loaded_state" -eq 3 ] || return 1 + fi + fi + if [ "$HOME_STATE_NAMESPACE_STAGED" -eq 1 ]; then + if [ ! -d "$DEFAULT_STATE_ROOT" ] || [ -L "$DEFAULT_STATE_ROOT" ] \ + || ! remove_home_state_namespace "$DEFAULT_STATE_ROOT"; then + failed=1 + else + HOME_STATE_NAMESPACE_STAGED=0 + fi + fi + if [ -n "$HOME_MIGRATION_RUN_LOCK" ]; then + fm_lock_release "$HOME_MIGRATION_RUN_LOCK" || failed=1 + HOME_MIGRATION_RUN_LOCK= + fi + if [ "$PHYSICAL_AGENT_STOPPED" -eq 1 ]; then + if [ -f "$PHYSICAL_PLIST" ] && [ ! -L "$PHYSICAL_PLIST" ] \ + && "$LAUNCHCTL" bootstrap "$domain" "$PHYSICAL_PLIST" >/dev/null 2>&1 \ + && launch_agent_loaded_state \ + "$domain" "$PHYSICAL_LABEL" "$PHYSICAL_PLIST"; then + PHYSICAL_AGENT_STOPPED=0 + else + failed=1 + fi + fi + restart_tracked_legacy_launch_agent "$domain" || failed=1 + [ "$failed" -ne 0 ] || HOME_MIGRATION_ACTIVE=0 + [ "$failed" -eq 0 ] +} + +commit_home_state_namespace_migration() { + local retired= + [ "$HOME_MIGRATION_ACTIVE" -eq 1 ] || return 0 + if [ "$PHYSICAL_STATE_ROOT" != "$STATE_ROOT" ] \ + && { [ -e "$PHYSICAL_STATE_ROOT" ] || [ -L "$PHYSICAL_STATE_ROOT" ]; }; then + [ -d "$PHYSICAL_STATE_ROOT" ] && [ ! -L "$PHYSICAL_STATE_ROOT" ] || return 1 + retired="$PHYSICAL_STATE_ROOT.retired.$$" + [ ! -e "$retired" ] && [ ! -L "$retired" ] || return 1 + mv "$PHYSICAL_STATE_ROOT" "$retired" || return 1 + if [ -n "$HOME_MIGRATION_RUN_LOCK" ]; then + HOME_MIGRATION_RUN_LOCK="$retired/.run-lock" + if ! fm_lock_release "$HOME_MIGRATION_RUN_LOCK"; then + mv "$retired" "$PHYSICAL_STATE_ROOT" || true + HOME_MIGRATION_RUN_LOCK="$PHYSICAL_STATE_ROOT/.run-lock" + return 1 + fi + HOME_MIGRATION_RUN_LOCK= + fi + fi + if [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ]; then + if [ ! -f "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ] \ + || { [ "${FM_CHECKOUT_REFRESH_TEST:-0}:${FM_CHECKOUT_TEST_HOME_MIGRATION_FAILURE:-}" = "1:commit" ]; } \ + || ! quiesce_launch_agent_verified \ + "gui/$(id -u)" "$PHYSICAL_LABEL" "$PHYSICAL_PLIST" \ + || ! rm -f "$PHYSICAL_PLIST"; then + [ -z "$retired" ] || mv "$retired" "$PHYSICAL_STATE_ROOT" || true + return 1 + fi + fi + HOME_MIGRATION_ACTIVE=0 + HOME_STATE_NAMESPACE_STAGED=0 + PHYSICAL_AGENT_STOPPED=0 + if [ -n "$retired" ] && ! remove_home_state_namespace "$retired"; then + echo "warning: retired checkout-refresh state remains quarantined at $retired" >&2 + fi + remove_matching_legacy_launch_agent "gui/$(id -u)" || return 1 +} + +ensure_lock_roots() { + ensure_state_root || return 1 + fm_checkout_lock_prepare "$LOCK_ROOT" \ + || { echo "error: unsafe checkout-refresh lock directory: $LOCK_ROOT" >&2; return 1; } +} + +skill_draft_inventory() { + local checkout=$1 + ( + set -o pipefail + git -C "$checkout" ls-files --others --exclude-standard -- \ + .agents/skills .claude/skills .codex/skills skills 2>/dev/null \ + | LC_ALL=C sort + ) +} + +# Surface a changed untracked-skill inventory on the ordinary 60-second probe, +# not only when an upstream tip changes or the 15-minute refresh is due. +# The inventory is intentionally path-only: it detects accumulation without +# reading, copying, stashing, or otherwise touching a draft's content. +surface_skill_drafts() { + local checkout=$1 key=$2 repeat=${3:-0} + local inventory alert prior signature count examples message + HYGIENE_FOUND=0 + inventory=$(mktemp "$STATE_ROOT/.hygiene-inventory.XXXXXX") || return 1 + alert="$STATE_ROOT/$key.hygiene-alert" + if ! skill_draft_inventory "$checkout" > "$inventory"; then + rm -f "$inventory" + echo "$checkout: HYGIENE: inventory failed - preserving the prior alert" >&2 + return 1 + fi + if [ ! -s "$inventory" ]; then + rm -f "$inventory" || return 1 + if ! rm -f "$alert"; then + echo "$checkout: HYGIENE: stale alert cannot be cleared - coverage remains unhealthy" >&2 + return 1 + fi + return 0 + fi + + signature=$(fm_checkout_hash_file "$inventory") || { + rm -f "$inventory" + echo "$checkout: HYGIENE: inventory signature failed - coverage remains unhealthy" >&2 + return 1 + } + count=$(awk 'END { print NR + 0 }' "$inventory") + HYGIENE_FOUND=1 + prior=$(sed -n '2p' "$alert" 2>/dev/null || true) + examples=$(awk 'NR <= 3 { if (shown) printf ", "; printf "%s", $0; shown = 1 } END { if (NR > 3) printf ", ..." }' "$inventory") + message="$checkout: HYGIENE: $count untracked skill-draft files under repository skill directories - reconcile before an upstream collision" + if ! atomic_write "$alert" "$checkout" "$signature" "$count" "$(date +%s)" "$examples"; then + rm -f "$inventory" + echo "$checkout: HYGIENE: alert persistence failed - coverage remains unhealthy" >&2 + return 1 + fi + rm -f "$inventory" + if [ "$repeat" -eq 1 ] || [ "$signature" != "$prior" ]; then + printf '%s (%s)\n' "$message" "$examples" + fi +} + +prepare_hygiene_discovery() { + local seed_file=$1 hygiene_file=$2 treehouse_paths treehouse_state pool worktree canonical failed=0 + cp "$seed_file" "$hygiene_file" || return 1 + treehouse_paths=$(mktemp "$STATE_ROOT/.treehouse-worktrees.XXXXXX") || return 1 + if ! treehouse_worktree_paths > "$treehouse_paths"; then + rm -f "$treehouse_paths" + return 1 + fi + while IFS=$'\t' read -r treehouse_state pool worktree; do + [ -n "$worktree" ] || continue + if backing_checkout "$worktree" "$pool" "$treehouse_state" >/dev/null 2>&1 \ + && canonical=$(exact_git_root "$worktree" 2>/dev/null); then + manifest_append "$hygiene_file" "$canonical" || failed=1 + else + echo "checkout-refresh: skipped: Treehouse worktree identity or registration is not inspectable: $worktree" >&2 + failed=1 + fi + done < "$treehouse_paths" + rm -f "$treehouse_paths" || failed=1 + [ "$failed" -eq 0 ] || return 1 + manifest_sort_unique "$hygiene_file" +} + +clear_stale_hygiene_alerts() { + local hygiene_file=$1 alert checkout failed=0 + for alert in "$STATE_ROOT"/*.hygiene-alert; do + [ -f "$alert" ] || continue + checkout=$(sed -n '1p' "$alert" 2>/dev/null || true) + if [ -z "$checkout" ] || ! grep -Fxq -- "$checkout" "$hygiene_file"; then + rm -f "$alert" || failed=1 + fi + done + [ "$failed" -eq 0 ] +} + +sync_checkout() { + local checkout=$1 output_file=$2 prune=${3:-0} + local status physical + physical=$(fm_checkout_physical_path_identity "$checkout" directory) || { + printf '%s: skipped: refresh physical identity cannot be inspected\n' "$checkout" > "$output_file" + return 1 + } + if ( + export FM_FLEET_PRUNE="$prune" + export FM_CHECKOUT_REFRESH_SYNC_TIMEOUT="$SYNC_TIMEOUT" + export FM_FLEET_SYNC_EXPECTED_ORIGIN_KIND="$CHECKOUT_ORIGIN_KIND" + export FM_FLEET_SYNC_EXPECTED_ORIGIN_VALUE="$CHECKOUT_ORIGIN_VALUE" + export FM_FLEET_SYNC_EXPECTED_PHYSICAL_IDENTITY="$physical" + "$SCRIPT_DIR/fm-fleet-sync.sh" "$checkout" + ) > "$output_file" 2>&1; then + status=0 + else + status=$? + fi + if [ "$status" -ne 0 ]; then + printf '%s: skipped: refresh failed with exit %s\n' "$checkout" "$status" >> "$output_file" + fi +} + +record_alert() { + local alert=$1 checkout=$2 output=$3 + atomic_write "$alert" "$checkout" "$(date +%s)" "$(first_line "$output")" +} + +record_reinspection_failure() { + local checkout=$1 key alert output + key=$(checkout_key "$checkout") || { + printf '%s: skipped: covered checkout lock identity cannot be resolved\n' "$checkout" + return 1 + } + alert="$STATE_ROOT/$key.alert" + output="$checkout: skipped: covered checkout became uninspectable during refresh; restore access and rerun checkout refresh" + printf '%s\n' "$output" + record_alert "$alert" "$checkout" "$output" || { + printf '%s: skipped: checkout alert cannot be persisted\n' "$checkout" + return 1 + } +} + +CHECKOUT_REFRESH_AUTHORITY= +resolve_checkout_refresh_authority() { + local checkout=$1 mode_line mode + CHECKOUT_REFRESH_AUTHORITY= + mode_line=$("$FM_ROOT/bin/fm-project-mode.sh" "$(basename "$checkout")" 2>/dev/null) || return 1 + mode=${mode_line%% *} + if [ "$mode" = local-only ] || [ "$CHECKOUT_ORIGIN_KIND" = no-origin ]; then + CHECKOUT_REFRESH_AUTHORITY=local + else + CHECKOUT_REFRESH_AUTHORITY=upstream + fi +} + +checkout_identity_migration_transaction() { + local checkout=$1 destination_prefix=$2 physical=$3 source_prefix=${4:-} line_count=${5:-} + python3 - "$STATE_ROOT" "$checkout" "$destination_prefix" "$physical" \ + "$source_prefix" "$line_count" <<'PY' +import hashlib +import os +import shutil +import stat +import sys +import tempfile + +state_root, checkout, destination_prefix, physical, requested_source, requested_lines = sys.argv[1:] +journal = destination_prefix + ".identity-migration" +staging = destination_prefix + ".identity-migration-stage" +allowed_extensions = ("identity", "tip", "last", "alert", "hygiene-alert") +journal_preexisting = os.path.lexists(journal) + +def fsync_directory(path): + descriptor = os.open(path, os.O_RDONLY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + +def regular_file(path): + metadata = os.lstat(path) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode): + raise OSError("unsafe state file") + return metadata + +def payload(source, extension, line_count): + regular_file(source) + with open(source, "rb") as stream: + content = stream.read() + if extension == "identity" and line_count == "2": + try: + values = content.decode("utf-8").splitlines() + except UnicodeError as error: + raise OSError("invalid identity encoding") from error + if len(values) != 2: + raise OSError("identity changed during migration") + content = ("\n".join(values + [physical]) + "\n").encode("utf-8") + return content + +def digest(content): + return hashlib.sha256(content).hexdigest() + +def write_journal(values): + descriptor, temporary = tempfile.mkstemp( + prefix=".identity-migration-journal.", dir=state_root + ) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as stream: + for key, value in values: + stream.write(f"{key}={value}\n") + stream.flush() + os.fsync(stream.fileno()) + os.chmod(temporary, 0o600) + os.replace(temporary, journal) + fsync_directory(state_root) + except OSError: + try: + os.unlink(temporary) + except OSError: + pass + raise + +def read_journal(): + regular_file(journal) + values = {} + with open(journal, "r", encoding="utf-8") as stream: + for raw in stream: + line = raw.rstrip("\n") + if "=" not in line: + raise OSError("malformed migration journal") + key, value = line.split("=", 1) + if not key or key in values: + raise OSError("malformed migration journal") + values[key] = value + return values + +def staged_payloads(extensions, publish, line_count): + if os.path.lexists(staging): + metadata = os.lstat(staging) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode): + raise OSError("unsafe migration staging") + shutil.rmtree(staging) + os.mkdir(staging, 0o700) + hashes = {} + for extension in extensions: + source = source_prefix + "." + extension + content = payload(source, extension, line_count) + hashes[extension] = digest(content) + if extension not in publish: + continue + target = os.path.join(staging, extension) + with open(target, "xb") as stream: + stream.write(content) + stream.flush() + os.fsync(stream.fileno()) + os.chmod(target, regular_file(source).st_mode & 0o777) + fsync_directory(staging) + fsync_directory(state_root) + return hashes + +def destination_matches(extension, expected_hash): + destination = destination_prefix + "." + extension + if not os.path.lexists(destination): + return False + regular_file(destination) + with open(destination, "rb") as stream: + return digest(stream.read()) == expected_hash + +def staged_matches(extension, expected_hash): + target = os.path.join(staging, extension) + if not os.path.lexists(target): + return False + regular_file(target) + with open(target, "rb") as stream: + return digest(stream.read()) == expected_hash + +def publish_file(extension): + staged = os.path.join(staging, extension) + regular_file(staged) + descriptor, temporary = tempfile.mkstemp( + prefix=".identity-migration-publish.", dir=state_root + ) + try: + with os.fdopen(descriptor, "wb") as output, open(staged, "rb") as source: + shutil.copyfileobj(source, output) + output.flush() + os.fsync(output.fileno()) + os.chmod(temporary, os.lstat(staged).st_mode & 0o777) + os.replace(temporary, destination_prefix + "." + extension) + fsync_directory(state_root) + except OSError: + try: + os.unlink(temporary) + except OSError: + pass + raise + +try: + root_metadata = os.lstat(state_root) + if stat.S_ISLNK(root_metadata.st_mode) or not stat.S_ISDIR(root_metadata.st_mode): + raise OSError("unsafe state root") + if os.path.dirname(destination_prefix) != state_root: + raise OSError("escaping destination") + values = None + if os.path.lexists(journal): + values = read_journal() + if values.get("version") != "1" or values.get("checkout") != checkout: + raise OSError("migration journal identity mismatch") + if values.get("destination") != os.path.basename(destination_prefix): + raise OSError("migration destination mismatch") + if values.get("physical") != physical: + raise OSError("migration physical identity drift") + line_count = values.get("lines", "") + source_name = values.get("source", "") + if ( + len(source_name) != 24 + or any(character not in "0123456789abcdef" for character in source_name) + ): + raise OSError("unsafe migration source") + if source_name == os.path.basename(destination_prefix): + raise OSError("migration source aliases destination") + source_prefix = os.path.join(state_root, source_name) + extensions = tuple(filter(None, values.get("extensions", "").split(","))) + publish = tuple(filter(None, values.get("publish", "").split(","))) + if ( + not extensions + or extensions[0] != "identity" + or len(set(extensions)) != len(extensions) + or len(set(publish)) != len(publish) + or any(extension not in allowed_extensions for extension in extensions) + or any(extension not in extensions for extension in publish) + or "identity" not in publish + or line_count not in ("2", "3") + ): + raise OSError("malformed migration journal") + hashes = {} + for extension in extensions: + value = values.get("hash." + extension, "") + if len(value) != 64 or any(character not in "0123456789abcdef" for character in value): + raise OSError("malformed migration hash") + hashes[extension] = value + else: + if os.path.lexists(staging): + raise OSError("orphaned migration staging") + if not requested_source: + raise SystemExit(0) + if os.path.dirname(requested_source) != state_root or requested_lines not in ("2", "3"): + raise OSError("unsafe migration request") + source_prefix = requested_source + line_count = requested_lines + extensions = [] + publish = [] + for extension in allowed_extensions: + source = source_prefix + "." + extension + if not os.path.lexists(source): + continue + regular_file(source) + extensions.append(extension) + destination = destination_prefix + "." + extension + if os.path.lexists(destination): + regular_file(destination) + if extension not in ("alert", "hygiene-alert"): + raise OSError("destination state exists") + else: + publish.append(extension) + if not extensions or extensions[0] != "identity" or "identity" not in publish: + raise OSError("identity state is missing") + hashes = staged_payloads(tuple(extensions), tuple(publish), line_count) + if os.environ.get("FM_CHECKOUT_REFRESH_TEST") == "1" and os.environ.get( + "FM_CHECKOUT_TEST_IDENTITY_MIGRATION_FAILURE" + ) == "stage": + shutil.rmtree(staging) + fsync_directory(state_root) + raise OSError("injected staging failure") + journal_values = [ + ("version", "1"), + ("checkout", checkout), + ("source", os.path.basename(source_prefix)), + ("destination", os.path.basename(destination_prefix)), + ("physical", physical), + ("lines", line_count), + ("extensions", ",".join(extensions)), + ("publish", ",".join(publish)), + ] + journal_values.extend( + ("hash." + extension, hashes[extension]) for extension in extensions + ) + write_journal(journal_values) + if os.environ.get("FM_CHECKOUT_REFRESH_TEST") == "1" and os.environ.get( + "FM_CHECKOUT_TEST_IDENTITY_MIGRATION_FAILURE" + ) == "publish": + os.unlink(journal) + shutil.rmtree(staging) + fsync_directory(state_root) + raise OSError("injected publish failure") + complete = all( + destination_matches(extension, hashes[extension]) for extension in publish + ) + if not complete: + for extension in extensions: + source = source_prefix + "." + extension + if not os.path.lexists(source) \ + or digest(payload(source, extension, line_count)) != hashes[extension]: + raise OSError("migration source changed") + stage_valid = os.path.isdir(staging) and not os.path.islink(staging) and all( + staged_matches(extension, hashes[extension]) for extension in publish + ) + if not stage_valid: + rebuilt_hashes = staged_payloads(extensions, publish, line_count) + if rebuilt_hashes != hashes: + raise OSError("migration source changed") + for extension in publish: + if destination_matches(extension, hashes[extension]): + continue + if not staged_matches(extension, hashes[extension]): + raise OSError("migration staging changed") + publish_file(extension) + if ( + os.environ.get("FM_CHECKOUT_REFRESH_TEST") == "1" + and os.environ.get("FM_CHECKOUT_TEST_IDENTITY_MIGRATION_CRASH") + == "after-first-publish" + ): + os._exit(86) + if not all( + destination_matches(extension, hashes[extension]) for extension in publish + ): + raise OSError("migration publication incomplete") + for extension in extensions: + source = source_prefix + "." + extension + if not os.path.lexists(source): + continue + if digest(payload(source, extension, line_count)) != hashes[extension]: + raise OSError("migration source changed") + regular_file(source) + os.unlink(source) + fsync_directory(state_root) + if os.path.lexists(staging): + metadata = os.lstat(staging) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode): + raise OSError("unsafe migration staging") + shutil.rmtree(staging) + fsync_directory(state_root) + os.unlink(journal) + fsync_directory(state_root) +except (OSError, UnicodeError): + if not journal_preexisting and not os.path.lexists(journal) and os.path.lexists(staging): + try: + metadata = os.lstat(staging) + if not stat.S_ISLNK(metadata.st_mode) and stat.S_ISDIR(metadata.st_mode): + shutil.rmtree(staging) + fsync_directory(state_root) + except OSError: + pass + raise SystemExit(1) +PY +} + +migrate_checkout_identity_state() { + local checkout=$1 key=$2 destination + local candidate source_prefix destination_prefix match='' count=0 lines current_physical current_physical_key + destination="$STATE_ROOT/$key.identity" + current_physical=$(fm_checkout_physical_path_identity "$checkout" directory) || return 1 + destination_prefix="$STATE_ROOT/$key" + checkout_identity_migration_transaction \ + "$checkout" "$destination_prefix" "$current_physical" || return 1 + for candidate in "$STATE_ROOT"/*.identity; do + [ -e "$candidate" ] || [ -L "$candidate" ] || continue + [ -f "$candidate" ] && [ ! -L "$candidate" ] && [ -r "$candidate" ] || return 1 + [ "$(sed -n '1p' "$candidate")" = "$checkout" ] || continue + match=$candidate + count=$((count + 1)) + done + [ "$count" -le 1 ] || return 1 + [ "$count" -eq 1 ] || return 0 + lines=$(awk 'END { print NR + 0 }' "$match") || return 1 + if [ "$match" = "$destination" ]; then + if [ "$lines" -eq 2 ]; then + echo "$checkout: skipped: legacy checkout identity is not bound to a proven physical checkout" + return 1 + fi + return 0 + fi + { [ "$lines" -eq 2 ] || [ "$lines" -eq 3 ]; } || return 1 + current_physical_key=$(fm_checkout_physical_path_key "$checkout" directory 24) || return 1 + [ ! -e "$destination" ] && [ ! -L "$destination" ] || return 1 + source_prefix=${match%.identity} + if [ "$lines" -eq 2 ] && [ "${source_prefix##*/}" != "$current_physical_key" ]; then + echo "$checkout: skipped: legacy checkout identity filename does not match the current physical checkout" + return 1 + fi + if [ "$lines" -eq 3 ] && [ "$(sed -n '3p' "$match")" != "$current_physical" ]; then + echo "$checkout: skipped: legacy checkout physical identity drifted before migration" + return 1 + fi + checkout_identity_migration_transaction \ + "$checkout" "$destination_prefix" "$current_physical" "$source_prefix" "$lines" +} + +validate_covered_checkout_identity() { + local checkout=$1 key=$2 tip_file=$3 identity_file current_identity current_physical + local recorded_checkout recorded_identity recorded_physical lines + identity_file="$STATE_ROOT/$key.identity" + migrate_checkout_identity_state "$checkout" "$key" || { + echo "$checkout: skipped: covered checkout identity history is ambiguous or cannot be migrated" + return 1 + } + if [ "$CHECKOUT_ORIGIN_KIND" = origin ]; then + current_identity="origin $CHECKOUT_ORIGIN_VALUE" + else + current_identity=no-origin + fi + current_physical=$(fm_checkout_physical_path_identity "$checkout" directory) || { + echo "$checkout: skipped: covered checkout physical identity cannot be inspected" + return 1 + } + if [ -e "$identity_file" ] || [ -L "$identity_file" ]; then + if [ -L "$identity_file" ] || [ ! -f "$identity_file" ] || [ ! -r "$identity_file" ]; then + echo "$checkout: skipped: covered checkout identity record is unsafe or unreadable" + return 1 + fi + lines=$(awk 'END { print NR + 0 }' "$identity_file") || return 1 + recorded_checkout=$(sed -n '1p' "$identity_file") || return 1 + recorded_identity=$(sed -n '2p' "$identity_file") || return 1 + recorded_physical=$(sed -n '3p' "$identity_file") || return 1 + if { [ "$lines" -ne 2 ] && [ "$lines" -ne 3 ]; } \ + || [ "$recorded_checkout" != "$checkout" ] || [ -z "$recorded_identity" ]; then + echo "$checkout: skipped: covered checkout identity record is malformed" + return 1 + fi + if [ "$recorded_identity" != "$current_identity" ]; then + echo "$checkout: skipped: covered checkout origin identity drifted from $recorded_identity to $current_identity" + return 1 + fi + if [ "$lines" -eq 3 ] && [ "$recorded_physical" != "$current_physical" ]; then + echo "$checkout: skipped: covered checkout physical identity drifted at $checkout" + return 1 + fi + return 0 + fi + if [ "$current_identity" = no-origin ] && [ -e "$tip_file" ]; then + echo "$checkout: skipped: covered checkout lost its previously tracked origin identity" + return 1 + fi + atomic_write "$identity_file" "$checkout" "$current_identity" "$current_physical" || { + echo "$checkout: skipped: covered checkout identity cannot be persisted" + return 1 + } +} + +LOCAL_DEFAULT_BRANCH= +LOCAL_DEFAULT_TIP= +inspect_local_checkout() { + local checkout=$1 status_raw default current head state untracked_count + LOCAL_DEFAULT_BRANCH= + LOCAL_DEFAULT_TIP= + if ! status_raw=$(GIT_OPTIONAL_LOCKS=0 git -C "$checkout" status --porcelain=v1 --untracked-files=all 2>/dev/null); then + echo "$checkout: skipped: local checkout cleanliness cannot be inspected" + return 0 + fi + default=$(local_default_branch "$checkout") || { + echo "$checkout: skipped: local default branch cannot be determined" + return 0 + } + LOCAL_DEFAULT_BRANCH=$default + LOCAL_DEFAULT_TIP=$(git -C "$checkout" rev-parse "refs/heads/$default^{commit}" 2>/dev/null) || { + echo "$checkout: skipped: local default tip cannot be inspected" + return 0 + } + head=$(git -C "$checkout" rev-parse "HEAD^{commit}" 2>/dev/null) || { + echo "$checkout: skipped: local checkout HEAD cannot be inspected" + return 0 + } + if current=$(git -C "$checkout" symbolic-ref --quiet --short HEAD 2>/dev/null); then + if [ "$current" = "$default" ]; then + state="local default branch $default" + else + state="non-default branch $current" + fi + else + current= + state="detached HEAD" + fi + if [ -n "$status_raw" ]; then + untracked_count=$(printf '%s\n' "$status_raw" | awk 'substr($0, 1, 3) == "?? " { count++ } END { print count + 0 }') + echo "$checkout: STUCK: on $state with uncommitted changes ($untracked_count untracked) - needs attention" + return 0 + fi + if [ "$current" != "$default" ]; then + if [ "$head" != "$LOCAL_DEFAULT_TIP" ]; then + echo "$checkout: STUCK: on $state at stale local tip $head instead of local $default $LOCAL_DEFAULT_TIP - needs attention" + else + echo "$checkout: STUCK: on $state instead of local default branch $default - needs attention" + fi + return 0 + fi + if [ "$head" != "$LOCAL_DEFAULT_TIP" ]; then + echo "$checkout: STUCK: local default branch $default is stale at $head instead of $LOCAL_DEFAULT_TIP - needs attention" + return 0 + fi + echo "$checkout: already current at local $default" +} + +run_once() { + local force=0 verbose=0 session=0 scheduled=0 prune=0 arg lock discovery identities hygiene checkout key tip_file last_file alert_file + local prior_tip previous_coverage retry_unhealthy now last due probe_ok output_file output line reinspected identity_output + local state_persisted hygiene_failed=0 coverage_failed=0 status=0 + local coverage=healthy + for arg in "$@"; do + case "$arg" in + --force) force=1 ;; + --verbose) verbose=1 ;; + --session) session=1 ;; + --scheduled) scheduled=1 ;; + *) usage; return 2 ;; + esac + done + [ "$session" -eq 0 ] || prune=1 + + ensure_state_root || return 1 + previous_coverage=$(sed -n '2p' "$STATE_ROOT/coverage-health" 2>/dev/null || true) + retry_unhealthy=1 + [ "$previous_coverage" != healthy ] || retry_unhealthy=0 + atomic_write "$STATE_ROOT/coverage-health" "$(date +%s)" running || return 1 + if ! fm_checkout_lock_prepare "$LOCK_ROOT"; then + echo "error: unsafe checkout-refresh lock directory: $LOCK_ROOT" >&2 + record_run_result unhealthy "$scheduled" || true + return 1 + fi + lock="$STATE_ROOT/.run-lock" + if ! fm_lock_try_acquire "$lock"; then + printf 'checkout-refresh: skipped: refresh already running (pid %s)\n' "${FM_LOCK_HELD_PID:-unknown}" + return 0 + fi + trap 'fm_lock_release "$STATE_ROOT/.run-lock"' EXIT + discovery=$(mktemp "$STATE_ROOT/.discover.XXXXXX") || { + record_run_result unhealthy "$scheduled" || true + return 1 + } + identities=$(mktemp "$STATE_ROOT/.external-identities.XXXXXX") || { + rm -f "$discovery" + record_run_result unhealthy "$scheduled" || true + return 1 + } + manifest_create "$identities" || { + rm -f "$discovery" "$identities" + record_run_result unhealthy "$scheduled" || true + return 1 + } + hygiene=$(mktemp "$STATE_ROOT/.hygiene-discover.XXXXXX") || { + rm -f "$discovery" "$identities" + record_run_result unhealthy "$scheduled" || true + return 1 + } + DISCOVERY_IDENTITIES_FILE=$identities + if ! discover > "$discovery"; then + rm -f "$discovery" "$identities" "$hygiene" + record_run_result unhealthy "$scheduled" || true + return 1 + fi + if ! validate_external_identity_history "$identities"; then + rm -f "$discovery" "$identities" "$hygiene" + record_run_result unhealthy "$scheduled" || true + return 1 + fi + if ! atomic_copy "$identities" "$STATE_ROOT/external-identities"; then + rm -f "$discovery" "$identities" "$hygiene" + record_run_result unhealthy "$scheduled" || true + return 1 + fi + if ! rm -f "$identities"; then + rm -f "$discovery" "$hygiene" || true + record_run_result unhealthy "$scheduled" || true + return 1 + fi + prepare_hygiene_discovery "$discovery" "$hygiene" || { + rm -f "$discovery" "$hygiene" + record_run_result unhealthy "$scheduled" || true + return 1 + } + now=$(date +%s) + + while IFS= read -r checkout; do + [ -n "$checkout" ] || continue + if ! reinspected=$(exact_git_root "$checkout") || [ "$reinspected" != "$checkout" ]; then + record_reinspection_failure "$checkout" || status=1 + coverage_failed=1 + continue + fi + key=$(checkout_key "$checkout") || { + echo "$checkout: skipped: checkout hygiene identity cannot be resolved" >&2 + coverage_failed=1 + status=1 + continue + } + if [ "$force" -eq 1 ] || [ "$verbose" -eq 1 ]; then + surface_skill_drafts "$checkout" "$key" 1 || hygiene_failed=1 + else + surface_skill_drafts "$checkout" "$key" 0 || hygiene_failed=1 + fi + done < "$hygiene" + clear_stale_hygiene_alerts "$hygiene" || { + echo "checkout-refresh: skipped: stale hygiene alerts cannot be cleared" >&2 + hygiene_failed=1 + } + + while IFS= read -r checkout; do + [ -n "$checkout" ] || continue + if ! reinspected=$(exact_git_root "$checkout") || [ "$reinspected" != "$checkout" ]; then + record_reinspection_failure "$checkout" || status=1 + coverage_failed=1 + continue + fi + key=$(checkout_key "$checkout") || { + echo "$checkout: skipped: checkout refresh identity cannot be resolved" >&2 + coverage_failed=1 + status=1 + continue + } + tip_file="$STATE_ROOT/$key.tip" + last_file="$STATE_ROOT/$key.last" + alert_file="$STATE_ROOT/$key.alert" + if ! inspect_checkout_origin "$checkout"; then + output="$checkout: skipped: covered checkout origin identity cannot be inspected" + printf '%s\n' "$output" + if ! record_alert "$alert_file" "$checkout" "$output"; then + printf '%s: skipped: checkout alert cannot be persisted\n' "$checkout" + status=1 + fi + coverage_failed=1 + continue + fi + if ! identity_output=$(validate_covered_checkout_identity "$checkout" "$key" "$tip_file"); then + [ -n "$identity_output" ] || identity_output="$checkout: skipped: covered checkout identity validation failed" + printf '%s\n' "$identity_output" + if ! record_alert "$alert_file" "$checkout" "$identity_output"; then + printf '%s: skipped: checkout alert cannot be persisted\n' "$checkout" + status=1 + fi + coverage_failed=1 + continue + fi + if ! resolve_checkout_refresh_authority "$checkout"; then + output="$checkout: skipped: checkout refresh authority cannot be determined" + printf '%s\n' "$output" + if ! record_alert "$alert_file" "$checkout" "$output"; then + printf '%s: skipped: checkout alert cannot be persisted\n' "$checkout" + status=1 + fi + coverage_failed=1 + continue + fi + last=$(read_epoch "$last_file") + due=0 + probe_ok=1 + if [ "$CHECKOUT_REFRESH_AUTHORITY" = local ]; then + due=1 + else + [ "$retry_unhealthy" -eq 0 ] || due=1 + [ "$force" -eq 0 ] || due=1 + [ "$((now - last))" -lt "$BACKSTOP" ] || due=1 + probe_ok=0 + if probe_upstream "$checkout"; then + probe_ok=1 + prior_tip=$(sed -n '1,2p' "$tip_file" 2>/dev/null || true) + [ "$prior_tip" = "$PROBE_BRANCH"$'\n'"$PROBE_TIP" ] || due=1 + else + due=1 + fi + fi + if [ "$due" -eq 0 ]; then + if [ -e "$alert_file" ] || [ -L "$alert_file" ]; then + coverage_failed=1 + if [ ! -f "$alert_file" ] || [ -L "$alert_file" ]; then + printf '%s: skipped: checkout alert state is unsafe\n' "$checkout" + status=1 + fi + fi + continue + fi + + output_file=$(mktemp "$STATE_ROOT/.sync.XXXXXX") || { + output="$checkout: skipped: cannot allocate refresh output" + printf '%s\n' "$output" + if ! record_alert "$alert_file" "$checkout" "$output"; then + printf '%s: skipped: checkout alert cannot be persisted\n' "$checkout" + status=1 + fi + coverage_failed=1 + continue + } + if [ "$CHECKOUT_REFRESH_AUTHORITY" = local ]; then + inspect_local_checkout "$checkout" > "$output_file" + elif [ "$probe_ok" -eq 1 ]; then + sync_checkout "$checkout" "$output_file" "$prune" + else + printf '%s: skipped: cannot probe live upstream default branch\n' "$checkout" > "$output_file" + fi + output=$(cat "$output_file") + rm -f "$output_file" + + case "$output" in + *': STUCK:'*|*': skipped:'*) + printf '%s\n' "$output" + if ! record_alert "$alert_file" "$checkout" "$output"; then + printf '%s: skipped: checkout alert cannot be persisted\n' "$checkout" + status=1 + fi + coverage_failed=1 + ;; + *) + state_persisted=1 + if ! rm -f "$alert_file"; then + printf '%s: skipped: checkout alert cannot be cleared\n' "$checkout" + state_persisted=0 + elif [ "$CHECKOUT_REFRESH_AUTHORITY" = local ]; then + if [ -z "$LOCAL_DEFAULT_BRANCH" ] || [ -z "$LOCAL_DEFAULT_TIP" ] \ + || ! atomic_write "$tip_file" "$LOCAL_DEFAULT_BRANCH" "$LOCAL_DEFAULT_TIP"; then + printf '%s: skipped: local checkout tip state cannot be persisted\n' "$checkout" + state_persisted=0 + fi + elif [ "$probe_ok" -eq 1 ] \ + && ! atomic_write "$tip_file" "$PROBE_BRANCH" "$PROBE_TIP"; then + printf '%s: skipped: upstream checkout tip state cannot be persisted\n' "$checkout" + state_persisted=0 + fi + if [ "$state_persisted" -eq 1 ] && ! atomic_write "$last_file" "$now"; then + printf '%s: skipped: checkout cadence state cannot be persisted\n' "$checkout" + state_persisted=0 + fi + if [ "$state_persisted" -eq 0 ]; then + coverage_failed=1 + status=1 + elif [ "$verbose" -eq 1 ]; then + printf '%s\n' "$output" + else + while IFS= read -r line; do + case "$line" in *': synced '*|*': recovered:'*) printf '%s\n' "$line" ;; esac + done <<EOF +$output +EOF + fi + ;; + esac + done < "$discovery" + + if ! rm -f "$discovery" "$hygiene"; then + coverage_failed=1 + status=1 + fi + if [ "$hygiene_failed" -ne 0 ]; then + coverage_failed=1 + status=1 + fi + [ "$coverage_failed" -eq 0 ] || coverage=unhealthy + record_run_result "$coverage" "$scheduled" || status=1 + trap - EXIT + fm_lock_release "$lock" + return "$status" +} + +preflight() { + local checkout=$1 output key output_file tip_file identity_output hygiene_found=0 canonical + canonical=$(require_exact_git_root "$checkout" "checkout-refresh preflight target") || return 1 + checkout=$canonical + ensure_lock_roots || return 1 + key=$(checkout_key "$checkout") || { + echo "error: checkout-refresh preflight identity is unavailable for $checkout" >&2 + return 1 + } + tip_file="$STATE_ROOT/$key.tip" + surface_skill_drafts "$checkout" "$key" 1 || return 1 + hygiene_found=$HYGIENE_FOUND + output_file=$(mktemp "$STATE_ROOT/.preflight.XXXXXX") || return 1 + if ! inspect_checkout_origin "$checkout"; then + rm -f "$output_file" + echo "$checkout: skipped: covered checkout origin identity cannot be inspected" + return 1 + fi + if ! identity_output=$(validate_covered_checkout_identity "$checkout" "$key" "$tip_file"); then + rm -f "$output_file" + [ -n "$identity_output" ] || identity_output="$checkout: skipped: covered checkout identity validation failed" + printf '%s\n' "$identity_output" + return 1 + fi + if ! resolve_checkout_refresh_authority "$checkout"; then + rm -f "$output_file" + echo "$checkout: skipped: checkout refresh authority cannot be determined" + return 1 + fi + if [ "$CHECKOUT_REFRESH_AUTHORITY" = local ]; then + inspect_local_checkout "$checkout" > "$output_file" + else + if probe_upstream "$checkout"; then + sync_checkout "$checkout" "$output_file" 0 + else + printf '%s: skipped: cannot probe live upstream default branch\n' "$checkout" > "$output_file" + fi + fi + output=$(cat "$output_file") + rm -f "$output_file" + printf '%s\n' "$output" + [ "$hygiene_found" -eq 0 ] || return 1 + case "$output" in *': STUCK:'*|*': skipped:'*) return 1 ;; esac + return 0 +} + +pool_preflight() { + local expected_source=$1 expected_common treehouse_paths treehouse_state pool worktree canonical common dirty example failed=0 + expected_source=$(require_exact_git_root "$expected_source" "expected Treehouse source") || return 1 + expected_common=$(fm_checkout_git_common_dir "$expected_source") || { + echo "error: cannot resolve expected Treehouse repository identity for $expected_source" >&2 + return 1 + } + treehouse_paths=$(mktemp "${TMPDIR:-/tmp}/fm-checkout-refresh-pool.XXXXXX") || return 1 + if ! treehouse_worktree_paths > "$treehouse_paths"; then + rm -f "$treehouse_paths" + return 1 + fi + while IFS=$'\t' read -r treehouse_state pool worktree; do + [ -n "$worktree" ] || continue + # A worktree we cannot fully inspect is SKIPPED, not fatal. The preflight + # iterates every Treehouse worktree in every pool, so a single busy, + # foreign, or half-torn-down worktree anywhere (including unrelated pools) + # must not cap acquisition for THIS pool - Treehouse pools grow, and acquire + # creates a fresh clean slot when no existing one is reusable. This mirrors + # the dirty-worktree case below, which already skips without failing. We can + # only confirm a worktree belongs to the target pool AFTER inspecting it, so + # an uninspectable one is by definition not provably ours to block on. + backing_checkout "$worktree" "$pool" "$treehouse_state" >/dev/null 2>&1 || { + echo "checkout-refresh: skipped: Treehouse worktree identity or registration is not inspectable: $worktree" >&2 + continue + } + canonical=$(exact_git_root "$worktree" 2>/dev/null) || { + echo "checkout-refresh: skipped: Treehouse worktree is not an exact Git root: $worktree" >&2 + continue + } + common=$(fm_checkout_git_common_dir "$canonical") || { + echo "checkout-refresh: skipped: Treehouse repository identity is not inspectable: $canonical" >&2 + continue + } + [ "$common" = "$expected_common" ] || continue + dirty=$(GIT_OPTIONAL_LOCKS=0 git -C "$canonical" status --porcelain=v1 --untracked-files=all 2>/dev/null) || { + echo "checkout-refresh: skipped: Treehouse worktree cleanliness is not inspectable: $canonical" >&2 + continue + } + [ -n "$dirty" ] || continue + example=$(first_line "$dirty") + echo "$canonical: skipped: dirty Treehouse pool worktree remains unavailable for acquisition ($example)" >&2 + done < "$treehouse_paths" + rm -f "$treehouse_paths" + # 'failed' is retained for future in-pool-fatal conditions; today no inspection + # skip is fatal, so preflight succeeds and lets acquire find or grow a slot. + [ "$failed" -eq 0 ] +} + +local_default_branch() { + local checkout=$1 branch + for branch in main master; do + if git -C "$checkout" show-ref --verify --quiet "refs/heads/$branch"; then + printf '%s\n' "$branch" + return 0 + fi + done + return 1 +} + +verify_worktree_safety() { + local worktree=$1 expected_source=$2 dirty dirty_example worktree_common source_common canonical + local worktree_origin source_origin branch + canonical=$(require_exact_git_root "$worktree" "worktree freshness target") || return 1 + worktree=$canonical + canonical=$(require_exact_git_root "$expected_source" "expected worktree source") || return 1 + expected_source=$canonical + if ! dirty=$(GIT_OPTIONAL_LOCKS=0 git -C "$worktree" status --porcelain=v1 --untracked-files=all 2>/dev/null); then + echo "error: acquired worktree safety cannot be inspected at $worktree; retain it for manual recovery" >&2 + return 3 + fi + if [ -n "$dirty" ]; then + dirty_example=$(first_line "$dirty") + echo "error: acquired worktree is dirty at $worktree; retain it for manual recovery without reset, clean, stash, or forced return ($dirty_example)" >&2 + return 3 + fi + worktree_common=$(fm_checkout_git_common_dir "$worktree") || { + echo "error: cannot resolve acquired worktree repository identity for $worktree" >&2 + return 1 + } + source_common=$(fm_checkout_git_common_dir "$expected_source") || { + echo "error: cannot resolve expected repository identity for $expected_source" >&2 + return 1 + } + if [ "$worktree_common" != "$source_common" ]; then + echo "error: acquired worktree repository mismatch: $worktree does not belong to $expected_source" >&2 + return 1 + fi + worktree_origin=$(git -C "$worktree" remote get-url origin 2>/dev/null || true) + source_origin=$(git -C "$expected_source" remote get-url origin 2>/dev/null || true) + if [ "$worktree_origin" != "$source_origin" ]; then + echo "error: acquired worktree origin mismatch: $worktree does not match $expected_source" >&2 + return 1 + fi + if branch=$(git -C "$worktree" symbolic-ref --quiet --short HEAD 2>/dev/null); then + echo "error: acquired worktree is attached to branch $branch at $worktree; retain it for manual recovery" >&2 + return 3 + fi + return 0 +} + +remote_identity() { + local checkout=$1 remote=$2 candidate + case "$remote" in + file://*) candidate=${remote#file://} ;; + /*|./*|../*) candidate=$remote ;; + *) printf 'remote:%s\n' "$remote"; return 0 ;; + esac + case "$candidate" in + /*) ;; + *) candidate="$checkout/$candidate" ;; + esac + candidate=$(canonical_dir "$candidate" 2>/dev/null) || return 1 + printf 'path:%s\n' "$candidate" +} + +verify_home() { + local home=$1 expected_source=$2 canonical dirty home_common source_common home_origin source_origin + local home_identity source_identity source_path_identity default branch tip head expected + canonical=$(require_exact_git_root "$home" "secondmate home freshness target") || return 1 + home=$canonical + canonical=$(require_exact_git_root "$expected_source" "expected secondmate source") || return 1 + expected_source=$canonical + dirty=$(GIT_OPTIONAL_LOCKS=0 git -C "$home" status --porcelain=v1 --untracked-files=all 2>/dev/null) || { + echo "error: secondmate home cleanliness cannot be inspected at $home" >&2 + return 1 + } + [ -z "$dirty" ] || { + echo "error: secondmate home is dirty at $home; retain it for manual recovery" >&2 + return 1 + } + home_common=$(fm_checkout_git_common_dir "$home") || return 1 + source_common=$(fm_checkout_git_common_dir "$expected_source") || return 1 + if [ "$home_common" != "$source_common" ]; then + home_origin=$(origin_url "$home" || true) + source_origin=$(origin_url "$expected_source" || true) + [ -n "$home_origin" ] || { + echo "error: secondmate home origin is unavailable at $home" >&2 + return 1 + } + home_identity=$(remote_identity "$home" "$home_origin") || return 1 + source_path_identity="path:$expected_source" + source_identity= + [ -z "$source_origin" ] || source_identity=$(remote_identity "$expected_source" "$source_origin") || return 1 + if [ "$home_identity" != "$source_path_identity" ] && [ "$home_identity" != "$source_identity" ]; then + echo "error: secondmate home repository identity does not match $expected_source" >&2 + return 1 + fi + fi + source_origin=$(origin_url "$expected_source" || true) + if [ -n "$source_origin" ]; then + probe_upstream "$expected_source" || { + echo "error: cannot verify the live upstream default-branch tip for $expected_source" >&2 + return 1 + } + branch=$PROBE_BRANCH + tip=$PROBE_TIP + expected="origin/$branch" + else + branch=$(local_default_branch "$expected_source") || { + echo "error: cannot determine the local default branch for $expected_source" >&2 + return 1 + } + tip=$(git -C "$expected_source" rev-parse "refs/heads/$branch^{commit}" 2>/dev/null) || return 1 + expected="local $branch" + fi + if default=$(git -C "$home" symbolic-ref --quiet --short HEAD 2>/dev/null); then + [ "$default" = "$branch" ] || { + echo "error: secondmate home is on non-default branch $default at $home" >&2 + return 1 + } + fi + head=$(git -C "$home" rev-parse HEAD 2>/dev/null) || return 1 + [ "$head" = "$tip" ] || { + echo "error: secondmate home is stale: HEAD $head does not match $expected $tip" >&2 + return 1 + } +} + +verify_worktree() { + local worktree=$1 expected_source=$2 source_origin default tip head expected + verify_worktree_safety "$worktree" "$expected_source" || return $? + source_origin=$(git -C "$expected_source" remote get-url origin 2>/dev/null || true) + if [ -n "$source_origin" ]; then + probe_upstream "$expected_source" || { + echo "error: cannot verify the upstream default-branch tip for $expected_source" >&2 + return 1 + } + tip=$PROBE_TIP + expected="origin/$PROBE_BRANCH" + else + default=$(local_default_branch "$expected_source") || { + echo "error: cannot determine the local default branch for $expected_source" >&2 + return 1 + } + tip=$(git -C "$expected_source" rev-parse "refs/heads/$default^{commit}" 2>/dev/null) || return 1 + expected="local $default" + fi + head=$(git -C "$worktree" rev-parse HEAD 2>/dev/null) || return 1 + if [ "$head" != "$tip" ]; then + echo "error: acquired worktree is stale: HEAD $head does not match $expected $tip" >&2 + return 1 + fi + return 0 +} + +verify_returnable_worktree() { + local worktree=$1 expected_source=$2 expected_tip=$3 head expected_commit + verify_worktree_safety "$worktree" "$expected_source" || return $? + expected_commit=$(git -C "$worktree" rev-parse --verify "$expected_tip^{commit}" 2>/dev/null) || { + echo "error: expected acquired worktree tip cannot be resolved: $expected_tip" >&2 + return 1 + } + head=$(git -C "$worktree" rev-parse HEAD 2>/dev/null) || { + echo "error: acquired worktree HEAD cannot be resolved at $worktree" >&2 + return 1 + } + if [ "$head" != "$expected_commit" ]; then + echo "error: acquired worktree changed from expected detached tip $expected_commit to $head; retain it for manual recovery" >&2 + return 3 + fi + return 0 +} + +xml_escape() { + printf '%s' "$1" | sed 's/&/\&/g;s/</\</g;s/>/\>/g;s/"/\"/g;s/'"'"'/\'/g' +} + +remove_matching_legacy_launch_agent() { + local domain=$1 legacy_plist="$LAUNCH_AGENTS_DIR/$LEGACY_LABEL.plist" + [ "$LABEL" != "$LEGACY_LABEL" ] || return 0 + quiesce_legacy_launch_agent_before_activation "$domain" || return 1 + [ -e "$legacy_plist" ] || [ -L "$legacy_plist" ] || return 0 + legacy_launch_agent_definition_matches_home "$legacy_plist" || return 0 + rm -f "$legacy_plist" +} + +legacy_launch_agent_definition_matches_home() { + local legacy_plist=$1 + [ -f "$legacy_plist" ] && [ ! -L "$legacy_plist" ] && [ -r "$legacy_plist" ] || return 1 + grep -Fq "<key>Label</key><string>$(xml_escape "$LEGACY_LABEL")</string>" "$legacy_plist" \ + && grep -Fq "<string>$(xml_escape "$SCRIPT_DIR/fm-checkout-refresh.sh")</string>" "$legacy_plist" \ + && { grep -Fq "<key>FM_HOME</key><string>$(xml_escape "$FM_HOME_CANONICAL")</string>" "$legacy_plist" \ + || grep -Fq "<key>FM_HOME</key><string>$(xml_escape "$FM_HOME")</string>" "$legacy_plist"; } +} + +legacy_loaded_job_matches_home() { + FM_CHECKOUT_LOADED_JOB=$1 python3 - "$2" "$SCRIPT_DIR/fm-checkout-refresh.sh" \ + "$FM_HOME_CANONICAL" "$FM_HOME" <<'PY' +import os +import re +import sys + +target, script, canonical_home, raw_home = sys.argv[1:] +loaded = os.environ.get("FM_CHECKOUT_LOADED_JOB", "") + +def decode(value): + value = value.strip() + if len(value) >= 2 and value[0] == value[-1] == '"': + value = value[1:-1] + return value + +def scalar(name): + matches = re.findall( + rf"(?m)^[ \t]*{re.escape(name)}[ \t]*=[ \t]*(.+?)[ \t]*$", + loaded, + ) + if len(matches) != 1: + raise ValueError + return decode(matches[0]) + +def block(name): + matches = list( + re.finditer( + rf"(?ms)^[ \t]*{re.escape(name)}[ \t]*=[ \t]*\{{[ \t]*\n" + rf"(.*?)^[ \t]*\}}[ \t]*$", + loaded, + ) + ) + if len(matches) != 1: + raise ValueError + return [line.strip() for line in matches[0].group(1).splitlines() if line.strip()] + +try: + first = next(line.strip() for line in loaded.splitlines() if line.strip()) + if first != target + " = {": + raise ValueError + arguments = [decode(line) for line in block("arguments")] + if ( + len(arguments) != 4 + or not os.path.isabs(arguments[0]) + or scalar("program") != arguments[0] + or arguments[1:] != [script, "run-once", "--scheduled"] + ): + raise ValueError + environment = {} + for line in block("environment"): + if " => " not in line: + raise ValueError + key, value = line.split(" => ", 1) + key = decode(key) + if not key or key in environment: + raise ValueError + environment[key] = decode(value) + if environment.get("FM_HOME") not in (canonical_home, raw_home): + raise LookupError +except LookupError: + raise SystemExit(3) +except (StopIteration, ValueError): + raise SystemExit(1) +PY +} + +legacy_launch_agent_raw_state() { + local domain=$1 output status + output=$("$LAUNCHCTL" print "$domain/$LEGACY_LABEL" 2>&1) + status=$? + if [ "$status" -ne 0 ]; then + case "$output" in + *"Could not find service"*|*"service not found"*|*"No such process"*) return 3 ;; + *) return 4 ;; + esac + fi + printf '%s\n' "$output" +} + +quiesce_legacy_launch_agent_before_activation() { + local domain=$1 legacy_plist="$LAUNCH_AGENTS_DIR/$LEGACY_LABEL.plist" + local output state bootout_output bootout_status tracked=0 + [ "$LABEL" != "$LEGACY_LABEL" ] || return 0 + if [ -e "$legacy_plist" ] || [ -L "$legacy_plist" ]; then + if [ ! -f "$legacy_plist" ] || [ -L "$legacy_plist" ] || [ ! -r "$legacy_plist" ] \ + || ! grep -Fq "<key>Label</key><string>$(xml_escape "$LEGACY_LABEL")</string>" "$legacy_plist" \ + || ! grep -Fq "<string>$(xml_escape "$SCRIPT_DIR/fm-checkout-refresh.sh")</string>" "$legacy_plist"; then + echo "error: legacy checkout-refresh LaunchAgent definition is malformed or untrusted" >&2 + return 1 + fi + if legacy_launch_agent_definition_matches_home "$legacy_plist"; then + tracked=1 + fi + fi + if output=$(legacy_launch_agent_raw_state "$domain"); then + state=0 + else + state=$? + fi + case "$state" in + 3) return 0 ;; + 0) ;; + *) + echo "error: cannot prove whether legacy checkout-refresh LaunchAgent $LEGACY_LABEL is loaded" >&2 + return 1 + ;; + esac + if [ "$tracked" -eq 1 ]; then + launch_agent_loaded_state "$domain" "$LEGACY_LABEL" "$legacy_plist" || return 1 + else + if legacy_loaded_job_matches_home "$output" "$domain/$LEGACY_LABEL"; then + atomic_write "$STATE_ROOT/legacy-launch-agent.quarantine" "$output" || { + echo "error: cannot durably record untracked legacy checkout-refresh LaunchAgent $LEGACY_LABEL" >&2 + return 1 + } + else + state=$? + [ "$state" -eq 3 ] && return 0 + echo "error: untracked legacy checkout-refresh LaunchAgent $LEGACY_LABEL identity is ambiguous" >&2 + return 1 + fi + fi + bootout_output=$("$LAUNCHCTL" bootout "$domain/$LEGACY_LABEL" 2>&1) + bootout_status=$? + [ "$bootout_status" -eq 0 ] || { + echo "error: cannot quiesce checkout-refresh LaunchAgent $LEGACY_LABEL: ${bootout_output:-bootout failed}" >&2 + return 1 + } + if legacy_launch_agent_raw_state "$domain" >/dev/null; then + echo "error: checkout-refresh LaunchAgent $LEGACY_LABEL remains live after bootout" >&2 + return 1 + else + state=$? + fi + [ "$state" -eq 3 ] || { + echo "error: cannot prove checkout-refresh LaunchAgent $LEGACY_LABEL absent after bootout" >&2 + return 1 + } + LEGACY_AGENT_STOPPED=1 + LEGACY_AGENT_TRACKED=$tracked +} + +restart_tracked_legacy_launch_agent() { + local domain=$1 legacy_plist="$LAUNCH_AGENTS_DIR/$LEGACY_LABEL.plist" + [ "$LEGACY_AGENT_STOPPED" -eq 1 ] && [ "$LEGACY_AGENT_TRACKED" -eq 1 ] || return 0 + legacy_launch_agent_definition_matches_home "$legacy_plist" \ + && "$LAUNCHCTL" bootstrap "$domain" "$legacy_plist" >/dev/null 2>&1 \ + && launch_agent_loaded_state "$domain" "$LEGACY_LABEL" "$legacy_plist" || return 1 + LEGACY_AGENT_STOPPED=0 +} + +physical_launch_agent_matches_home() { + [ "$PHYSICAL_LABEL" != "$LABEL" ] || return 1 + [ -f "$PHYSICAL_PLIST" ] && [ ! -L "$PHYSICAL_PLIST" ] || return 1 + grep -Fq "<key>Label</key><string>$(xml_escape "$PHYSICAL_LABEL")</string>" "$PHYSICAL_PLIST" \ + && grep -Fq "<string>$(xml_escape "$SCRIPT_DIR/fm-checkout-refresh.sh")</string>" "$PHYSICAL_PLIST" \ + && grep -Fq '<string>--scheduled</string>' "$PHYSICAL_PLIST" \ + && grep -Fq "<key>FM_HOME</key><string>$(xml_escape "$FM_HOME_CANONICAL")</string>" "$PHYSICAL_PLIST" \ + && grep -Fq "<key>FM_CHECKOUT_REFRESH_STATE_ROOT</key><string>$(xml_escape "$PHYSICAL_STATE_ROOT")</string>" "$PHYSICAL_PLIST" +} + +validate_launch_agent_namespaces() { + [ "$PHYSICAL_LABEL" != "$LABEL" ] || return 0 + if { [ -e "$PLIST" ] || [ -L "$PLIST" ]; } \ + && { [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ]; } \ + && [ "$HOME_MIGRATION_ACTIVE" -ne 1 ]; then + echo "error: ambiguous logical and physical checkout-refresh LaunchAgents for $FM_HOME_CANONICAL" >&2 + return 1 + fi + if [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ]; then + physical_launch_agent_matches_home || { + echo "error: unsafe or mismatched physical checkout-refresh LaunchAgent at $PHYSICAL_PLIST" >&2 + return 1 + } + fi +} + +remove_matching_physical_launch_agent() { + local domain=$1 + [ "$PHYSICAL_LABEL" != "$LABEL" ] || return 0 + [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ] || return 0 + physical_launch_agent_matches_home || return 1 + quiesce_launch_agent_verified "$domain" "$PHYSICAL_LABEL" "$PHYSICAL_PLIST" || return 1 + rm -f "$PHYSICAL_PLIST" +} + +quiesce_matching_physical_launch_agent() { + local domain=$1 loaded_state + [ "$PHYSICAL_LABEL" != "$LABEL" ] || return 0 + [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ] || return 0 + physical_launch_agent_matches_home || return 1 + launch_agent_loaded_state "$domain" "$PHYSICAL_LABEL" "$PHYSICAL_PLIST" + loaded_state=$? + case "$loaded_state" in + 0) + quiesce_launch_agent_verified "$domain" "$PHYSICAL_LABEL" "$PHYSICAL_PLIST" \ + || return 1 + PHYSICAL_AGENT_STOPPED=1 + ;; + 3) ;; + *) return 1 ;; + esac +} + +launch_agent_environment_value() { + python3 - "$1" "$2" <<'PY' +import sys +import xml.etree.ElementTree as ET + +path, requested = sys.argv[1:] +try: + document = ET.parse(path) + root = document.getroot().find("dict") + if root is None: + raise ValueError + children = list(root) + values = {} + for index in range(0, len(children), 2): + if index + 1 >= len(children) or children[index].tag != "key": + raise ValueError + key = children[index].text + if not key or key in values: + raise ValueError + values[key] = children[index + 1] + environment = values.get("EnvironmentVariables") + if environment is None or environment.tag != "dict": + raise ValueError + children = list(environment) + values = {} + for index in range(0, len(children), 2): + if index + 1 >= len(children) or children[index].tag != "key": + raise ValueError + key = children[index].text + value = children[index + 1] + if not key or key in values: + raise ValueError + values[key] = value + value = values.get(requested) + if value is None or value.tag != "string" or value.text is None: + raise ValueError + print(value.text) +except (ET.ParseError, OSError, ValueError): + raise SystemExit(1) +PY +} + +wait_for_scheduler_generation() { + local generation deadline recorded + generation=$(launch_agent_environment_value "$PLIST" FM_CHECKOUT_REFRESH_GENERATION) || return 1 + deadline=$(( $(date +%s) + ACTIVATION_TIMEOUT )) + while [ "$(date +%s)" -le "$deadline" ]; do + if [ -f "$STATE_ROOT/scheduler-generation" ] \ + && [ ! -L "$STATE_ROOT/scheduler-generation" ]; then + recorded=$(sed -n '1p' "$STATE_ROOT/scheduler-generation" 2>/dev/null || true) + [ "$recorded" != "$generation" ] || return 0 + fi + sleep 1 + done + echo "checkout-refresh active scheduler generation did not complete before activation timeout" >&2 + return 1 +} + +generate_scheduler_generation() { + local token_seed token + token_seed=$(mktemp "$STATE_ROOT/.scheduler-generation.XXXXXX") || return 1 + token=$(fm_checkout_hash_value \ + "$LABEL:$STATE_ROOT:$$:$(date +%s):${RANDOM:-0}:$token_seed" 32) || { + rm -f "$token_seed" + return 1 + } + rm -f "$token_seed" || return 1 + printf '%s\n' "$token" +} + +install_launch_agent() { + local bash_runtime python_runtime perl_runtime runtime_path temp previous domain generation loaded_state + [ "$PLATFORM" = Darwin ] || { + echo "error: checkout-refresh background installation currently requires macOS" >&2 + return 1 + } + command -v "$LAUNCHCTL" >/dev/null 2>&1 || { echo "error: launchctl is unavailable" >&2; return 1; } + validate_launch_agent_namespaces || return 1 + bash_runtime=$(command -v bash) || return 1 + python_runtime=$(command -v python3) || return 1 + perl_runtime=$(command -v perl) \ + || { echo "error: perl is unavailable for checkout-refresh process control" >&2; return 1; } + case "$bash_runtime" in /*) ;; *) echo "error: cannot resolve an absolute Bash runtime" >&2; return 1 ;; esac + runtime_path="$(dirname "$python_runtime"):$(dirname "$perl_runtime"):/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" + mkdir -p "$LAUNCH_AGENTS_DIR" "$STATE_ROOT" "$LOCK_ROOT" || return 1 + [ -d "$LAUNCH_AGENTS_DIR" ] && [ ! -L "$LAUNCH_AGENTS_DIR" ] \ + && [ -d "$STATE_ROOT" ] && [ ! -L "$STATE_ROOT" ] \ + && [ -d "$LOCK_ROOT" ] && [ ! -L "$LOCK_ROOT" ] \ + || { echo "error: unsafe checkout-refresh installation directories" >&2; return 1; } + generation=$(generate_scheduler_generation) || return 1 + temp=$(mktemp "$LAUNCH_AGENTS_DIR/.$LABEL.XXXXXX") || return 1 + previous=$(mktemp "$LAUNCH_AGENTS_DIR/.$LABEL.previous.XXXXXX") || { rm -f "$temp"; return 1; } + rm -f "$previous" + if [ -e "$PLIST" ] || [ -L "$PLIST" ]; then + [ -f "$PLIST" ] && [ ! -L "$PLIST" ] \ + || { rm -f "$temp"; echo "error: unsafe checkout-refresh plist" >&2; return 1; } + cp -p "$PLIST" "$previous" || return 1 + fi + cat > "$temp" <<EOF +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> +<plist version="1.0"><dict> +<key>Label</key><string>$(xml_escape "$LABEL")</string> +<key>ProgramArguments</key><array> +<string>$(xml_escape "$bash_runtime")</string> +<string>$(xml_escape "$SCRIPT_DIR/fm-checkout-refresh.sh")</string> +<string>run-once</string> +<string>--scheduled</string> +</array> +<key>EnvironmentVariables</key><dict> +<key>HOME</key><string>$(xml_escape "$HOME")</string> +<key>PATH</key><string>$(xml_escape "$runtime_path")</string> +<key>FM_HOME</key><string>$(xml_escape "$FM_HOME_CANONICAL")</string> +<key>FM_TREEHOUSE_ROOT</key><string>$(xml_escape "$TREEHOUSE_ROOT")</string> +<key>FM_CHECKOUT_REFRESH_STATE_ROOT</key><string>$(xml_escape "$STATE_ROOT")</string> +<key>FM_CHECKOUT_REFRESH_LOCK_ROOT</key><string>$(xml_escape "$LOCK_ROOT")</string> +<key>FM_CHECKOUT_REFRESH_INTERVAL</key><string>$(xml_escape "$INTERVAL")</string> +<key>FM_CHECKOUT_REFRESH_BACKSTOP</key><string>$(xml_escape "$BACKSTOP")</string> +<key>FM_CHECKOUT_REFRESH_GENERATION</key><string>$generation</string> +</dict> +<key>RunAtLoad</key><true/> +<key>StartInterval</key><integer>$INTERVAL</integer> +<key>StandardOutPath</key><string>$(xml_escape "$STATE_ROOT/stdout.log")</string> +<key>StandardErrorPath</key><string>$(xml_escape "$STATE_ROOT/stderr.log")</string> +</dict></plist> +EOF + chmod 600 "$temp" || return 1 + domain="gui/$(id -u)" + if [ -e "$PLIST" ] || [ -L "$PLIST" ]; then + quiesce_launch_agent_verified "$domain" "$LABEL" "$PLIST" || { + rm -f "$temp" "$previous" + return 1 + } + else + launch_agent_loaded_state "$domain" "$LABEL" "" + loaded_state=$? + [ "$loaded_state" -eq 3 ] || { + rm -f "$temp" "$previous" + return 1 + } + fi + quiesce_matching_physical_launch_agent "$domain" || { + rm -f "$temp" "$previous" + return 1 + } + quiesce_legacy_launch_agent_before_activation "$domain" || { + restart_tracked_legacy_launch_agent "$domain" || true + rm -f "$temp" "$previous" + return 1 + } + mv -f "$temp" "$PLIST" || { + restart_tracked_legacy_launch_agent "$domain" || true + return 1 + } + if "$LAUNCHCTL" bootstrap "$domain" "$PLIST" \ + && "$LAUNCHCTL" kickstart "$domain/$LABEL" \ + && launch_agent_loaded_state "$domain" "$LABEL" "$PLIST"; then + rm -f "$previous" + if [ "$HOME_MIGRATION_ACTIVE" -ne 1 ]; then + remove_matching_physical_launch_agent "$domain" || return 1 + remove_matching_legacy_launch_agent "$domain" || return 1 + fi + return 0 + fi + if ! quiesce_launch_agent_verified "$domain" "$LABEL" "$PLIST"; then + echo "error: checkout-refresh LaunchAgent activation failed and the replacement could not be quiesced; definitions were retained" >&2 + return 1 + fi + if [ -f "$previous" ]; then + mv -f "$previous" "$PLIST" + if ! "$LAUNCHCTL" bootstrap "$domain" "$PLIST" >/dev/null 2>&1 \ + || ! launch_agent_loaded_state "$domain" "$LABEL" "$PLIST"; then + echo "error: checkout-refresh LaunchAgent activation failed and the previous definition could not be restarted" >&2 + return 1 + fi + else + rm -f "$PLIST" + restart_tracked_legacy_launch_agent "$domain" || { + echo "error: checkout-refresh LaunchAgent activation failed and the legacy definition could not be restarted" >&2 + return 1 + } + fi + echo "error: checkout-refresh LaunchAgent activation failed; previous definition restored" >&2 + return 1 +} + +ensure_launch_agent() { + local domain heartbeat coverage_epoch coverage now max_age generation recorded_generation generation_lines installed=0 + [ "$PLATFORM" = Darwin ] || return 0 + validate_launch_agent_namespaces || return 1 + if [ ! -e "$PLIST" ] && [ ! -L "$PLIST" ] \ + && { [ -e "$PHYSICAL_PLIST" ] || [ -L "$PHYSICAL_PLIST" ]; }; then + install_launch_agent || return 1 + installed=1 + fi + [ "$installed" -eq 0 ] || wait_for_scheduler_generation || return 1 + [ -f "$PLIST" ] && [ ! -L "$PLIST" ] \ + || { echo "checkout-refresh LaunchAgent is not installed" >&2; return 1; } + grep -Fq "<key>Label</key><string>$(xml_escape "$LABEL")</string>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent Label identity drifted" >&2; return 1; } + grep -Fq "<string>$(xml_escape "$SCRIPT_DIR/fm-checkout-refresh.sh")</string>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent points at a different Firstmate checkout" >&2; return 1; } + grep -Fq '<string>--scheduled</string>' "$PLIST" \ + || { echo "checkout-refresh LaunchAgent does not own scheduler liveness" >&2; return 1; } + grep -Fq "<key>FM_HOME</key><string>$(xml_escape "$FM_HOME_CANONICAL")</string>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent belongs to a different Firstmate home" >&2; return 1; } + grep -Fq "<key>FM_TREEHOUSE_ROOT</key><string>$(xml_escape "$TREEHOUSE_ROOT")</string>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent uses a different Treehouse root" >&2; return 1; } + grep -Fq "<key>FM_CHECKOUT_REFRESH_STATE_ROOT</key><string>$(xml_escape "$STATE_ROOT")</string>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent uses a different home-scoped state root" >&2; return 1; } + grep -Fq "<key>FM_CHECKOUT_REFRESH_LOCK_ROOT</key><string>$(xml_escape "$LOCK_ROOT")</string>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent uses a different shared lock root" >&2; return 1; } + grep -Fq "<key>FM_CHECKOUT_REFRESH_INTERVAL</key><string>$(xml_escape "$INTERVAL")</string>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent uses a different refresh interval" >&2; return 1; } + grep -Fq "<key>FM_CHECKOUT_REFRESH_BACKSTOP</key><string>$(xml_escape "$BACKSTOP")</string>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent uses a different refresh backstop" >&2; return 1; } + grep -Fq "<key>StartInterval</key><integer>$INTERVAL</integer>" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent uses a different scheduler interval" >&2; return 1; } + generation=$(launch_agent_environment_value "$PLIST" FM_CHECKOUT_REFRESH_GENERATION) \ + || { echo "checkout-refresh LaunchAgent has no authoritative scheduler generation" >&2; return 1; } + case "$generation" in *[!0-9a-f]*) return 1 ;; esac + [ "${#generation}" -eq 32 ] \ + || { echo "checkout-refresh LaunchAgent scheduler generation is malformed" >&2; return 1; } + domain="gui/$(id -u)" + quiesce_legacy_launch_agent_before_activation "$domain" || { + echo "checkout-refresh legacy LaunchAgent identity or absence is untrusted" >&2 + return 1 + } + launch_agent_loaded_state "$domain" "$LABEL" "$PLIST" \ + || { echo "checkout-refresh LaunchAgent loaded identity is missing or untrusted" >&2; return 1; } + heartbeat=$(read_epoch "$STATE_ROOT/heartbeat") + now=$(date +%s) + max_age=$((INTERVAL * 3 + 30)) + [ "$heartbeat" -gt 0 ] && [ "$((now - heartbeat))" -le "$max_age" ] \ + || { echo "checkout-refresh heartbeat is stale or missing" >&2; return 1; } + coverage_epoch=$(read_epoch "$STATE_ROOT/coverage-health") + coverage=$(sed -n '2p' "$STATE_ROOT/coverage-health" 2>/dev/null || true) + [ "$coverage_epoch" -gt 0 ] && [ "$coverage" = healthy ] \ + || { + echo "checkout-refresh latest coverage run is missing or unhealthy; inspect $STATE_ROOT for checkout alerts and scheduler diagnostics" >&2 + return 1 + } + [ -f "$STATE_ROOT/scheduler-generation" ] \ + && [ ! -L "$STATE_ROOT/scheduler-generation" ] \ + && [ -r "$STATE_ROOT/scheduler-generation" ] \ + || { echo "checkout-refresh active scheduler generation has not completed a run" >&2; return 1; } + generation_lines=$(awk 'END { print NR + 0 }' "$STATE_ROOT/scheduler-generation") || return 1 + recorded_generation=$(sed -n '1p' "$STATE_ROOT/scheduler-generation") || return 1 + [ "$generation_lines" -eq 1 ] && [ "$recorded_generation" = "$generation" ] \ + || { echo "checkout-refresh active scheduler generation has not completed a run" >&2; return 1; } +} + +scheduler_install() { + local status + case "$PLATFORM" in + Darwin) + if install_launch_agent \ + && { [ "$HOME_MIGRATION_ACTIVE" -ne 1 ] || wait_for_scheduler_generation; } \ + && { [ "$HOME_MIGRATION_ACTIVE" -ne 1 ] || ensure_launch_agent; } \ + && commit_home_state_namespace_migration; then + return 0 + else + status=$? + fi + rollback_home_state_namespace_migration || true + return "$status" + ;; + Linux) + echo "error: checkout-refresh has no Linux scheduler adapter yet; use run-once from cron or systemd until one is implemented" >&2 + rollback_home_state_namespace_migration || true + return 1 + ;; + *) + echo "error: checkout-refresh has no scheduler adapter for $PLATFORM" >&2 + rollback_home_state_namespace_migration || true + return 1 + ;; + esac +} + +scheduler_ensure() { + local status + case "$PLATFORM" in + Darwin) + if ensure_launch_agent && commit_home_state_namespace_migration; then + return 0 + else + status=$? + fi + rollback_home_state_namespace_migration || true + return "$status" + ;; + Linux) + echo "error: checkout-refresh has no Linux scheduler adapter yet" >&2 + rollback_home_state_namespace_migration || true + return 1 + ;; + *) + echo "error: checkout-refresh has no scheduler adapter for $PLATFORM" >&2 + rollback_home_state_namespace_migration || true + return 1 + ;; + esac +} + +prepare_home_state_namespace "${1:-}" || exit 1 + +case "${1:-}" in + discover) + [ $# -eq 1 ] || { usage; exit 2; } + discover + ;; + run-once) + shift + run_once "$@" + ;; + preflight) + [ $# -eq 2 ] || { usage; exit 2; } + preflight "$2" + ;; + pool-preflight) + [ $# -eq 2 ] || { usage; exit 2; } + pool_preflight "$2" + ;; + acquire-worktree) + [ $# -eq 3 ] || { usage; exit 2; } + acquire_worktree "$2" "$3" + ;; + verify-worktree) + [ $# -eq 3 ] || { usage; exit 2; } + verify_worktree "$2" "$3" + ;; + verify-home) + [ $# -eq 3 ] || { usage; exit 2; } + verify_home "$2" "$3" + ;; + verify-returnable) + [ $# -eq 4 ] || { usage; exit 2; } + verify_returnable_worktree "$2" "$3" "$4" + ;; + ensure) + [ $# -eq 1 ] || { usage; exit 2; } + scheduler_ensure + ;; + install) + [ $# -eq 1 ] || { usage; exit 2; } + scheduler_install + ;; + *) usage; exit 2 ;; +esac diff --git a/bin/fm-dispatch-select.sh b/bin/fm-dispatch-select.sh index f2dc5278ebb..a95fe07d336 100755 --- a/bin/fm-dispatch-select.sh +++ b/bin/fm-dispatch-select.sh @@ -11,15 +11,11 @@ # contract: # - Any candidate carrying account_profile is invalid because pinned profiles # are direct per-spawn overrides, never inputs to quota-balanced selection. -# - A candidate set carrying account_pool uses only Agent Fleet's no-secret -# `pool status` summaries. Every candidate must then carry account_pool and -# use claude/codex. Only a non-degraded, quota-fresh provider summary backed -# by at least one freshly proven eligible profile is available. The best -# adjusted headroom wins; exact ties use the first array element. Stale or -# otherwise degraded summaries are diagnostics only. Agent Fleet trouble -# degrades to the first element and never falls through to default-account -# quota-axi data; enforced spawn still obtains the real fresh lease before -# any provider launch. +# - A candidate set carrying account_pool uses the first array element and +# passes the pool only as a compatibility activation input. Spawn performs +# the real direct account-directory selection. New dispatch never calls +# Agent Fleet. The unreachable pool-summary implementation remains only for +# deferred removal under remove-fleet-routing-deadcode. # - Enforced account routing rejects quota-balanced candidates without pools. # Off and observe retain the legacy no-pool quota-axi path. # - Per candidate vendor it takes the minimum percentRemaining across that @@ -40,8 +36,8 @@ # # quota-balanced uses quota-axi --json unless --quota-json supplies a fixture. # FM_DISPATCH_QUOTA_AXI overrides the quota command. -# FM_DISPATCH_AGENT_FLEET and FM_AGENT_FLEET_BIN are test/lab-only. Production -# uses the fixed passwd-home ~/.local/bin/agent-fleet front door. +# FM_DISPATCH_AGENT_FLEET and FM_AGENT_FLEET_BIN belong only to the unreachable +# legacy pool-summary implementation pending remove-fleet-routing-deadcode. # FM_DISPATCH_STALE_CLEAR_MARGIN overrides the default 20 point stale margin. set -u @@ -87,18 +83,6 @@ log() { printf 'fm-dispatch-select: %s\n' "$*" >&2 } -if [ -n "${FM_DISPATCH_AGENT_FLEET_TIMEOUT:-}" ]; then - AGENT_FLEET_TIMEOUT=$FM_DISPATCH_AGENT_FLEET_TIMEOUT - case "$AGENT_FLEET_TIMEOUT" in - ''|*[!0-9]*|0) - echo "error: FM_DISPATCH_AGENT_FLEET_TIMEOUT must be a positive integer" >&2 - exit 2 - ;; - esac -else - AGENT_FLEET_TIMEOUT=$(fm_account_selection_timeout) || exit 2 -fi - while [ "$#" -gt 0 ]; do case "$1" in --select) @@ -220,6 +204,30 @@ if [ "$routing_mode" = enforce ]; then fi fi if [ "$pooled_count" -gt 0 ]; then + if [ "$pooled_count" -ne "$profile_count" ] || ! printf '%s\n' "$profiles_json" | "$FM_DISPATCH_JQ_BIN" -e 'all(.[]; (.account_pool | length) > 0 and (.harness == "claude" or .harness == "codex"))' >/dev/null 2>&1; then + log "account_pool quota-balanced candidates must all name claude/codex pools; using first profile" + first_profile + exit 0 + fi + log "account_pool is a direct-routing compatibility input; using the first profile and deferring account selection to spawn" + first_profile + exit 0 +fi + +# Legacy Agent Fleet pool summaries are unreachable and retained only until +# remove-fleet-routing-deadcode removes the wider selection and lease machinery. +if [ "$pooled_count" -gt 0 ]; then + if [ -n "${FM_DISPATCH_AGENT_FLEET_TIMEOUT:-}" ]; then + AGENT_FLEET_TIMEOUT=$FM_DISPATCH_AGENT_FLEET_TIMEOUT + case "$AGENT_FLEET_TIMEOUT" in + ''|*[!0-9]*|0) + echo "error: FM_DISPATCH_AGENT_FLEET_TIMEOUT must be a positive integer" >&2 + exit 2 + ;; + esac + else + AGENT_FLEET_TIMEOUT=$(fm_account_selection_timeout) || exit 2 + fi if [ "$pooled_count" -ne "$profile_count" ] || ! printf '%s\n' "$profiles_json" | "$FM_DISPATCH_JQ_BIN" -e 'all(.[]; (.account_pool | length) > 0 and (.harness == "claude" or .harness == "codex"))' >/dev/null 2>&1; then log "account_pool quota-balanced candidates must all name claude/codex pools; using first profile" first_profile diff --git a/bin/fm-fleet-sync.sh b/bin/fm-fleet-sync.sh index 7196aafa7a8..6dad5a8ac1e 100755 --- a/bin/fm-fleet-sync.sh +++ b/bin/fm-fleet-sync.sh @@ -11,18 +11,33 @@ # is left untouched and reported as a quantified, loud "STUCK: ... N commits behind # ... - needs attention" warning rather than a quiet drift. Nothing is ever forced, # stashed, or discarded. +# Dirty warnings quantify untracked files and call out those under repository +# skill directories, so local skill drafts cannot accumulate invisibly until +# they collide with paths that later become tracked upstream. # Still skips (benignly) local-only/no-origin projects, missing remotes/branches, # and fetch failures. # Pruning never deletes the checked-out branch or a branch that still has a # worktree, so it cannot discard unlanded work; set FM_FLEET_PRUNE=0 to disable it. +# Every origin-backed invocation probes `ls-remote --symref origin HEAD` after +# fetching and proves the fetched ref matches that live tip before any local +# branch or worktree mutation, so cached refs/remotes/origin/HEAD is never an +# authority. +# The common mutation path owns a cooperative lock keyed by the canonical Git +# common directory, so scheduler, preflight, teardown, and merge-wake callers +# serialize every fetch, prune, checkout, and fast-forward of the same clone. +# Each checkout mutation entrypoint is process-tree bounded by +# FM_CHECKOUT_REFRESH_SYNC_TIMEOUT, including direct teardown and merge-wake calls. # When the fetch fails on an orphaned .git/packed-refs.lock (left by a ref rewrite # killed mid-write - e.g. a timed-out bootstrap sync or a teardown process kill), # it is retried with a bounded wait and removed only when provably stale; see # fetch_with_packed_refs_lock_guard and the FM_FLEET_SYNC_PACKED_REFS_LOCK_* knobs. # Usage: fm-fleet-sync.sh [<project-dir-or-name>] -# The single-project form accepts either a path (absolute, or relative to the -# caller's cwd) or a bare "<name>"/"projects/<name>" form, resolved against -# this home's projects dir ($FM_HOME/projects, or $FM_PROJECTS_OVERRIDE). +# The single-project form accepts an exact Git repository root (absolute, or +# relative to the caller's cwd) or a bare "<name>"/"projects/<name>" form, +# resolved against this home's projects dir ($FM_HOME/projects, or +# $FM_PROJECTS_OVERRIDE). +# Delivery mode lookup always uses the checkout directory's basename, so a +# discovered parallel clone still honors the registry entry for that project. # Bare names and "projects/<name>" forms prefer this home's projects dir before # falling back to an explicit path. Example: from anywhere, # `fm-fleet-sync.sh dotfiles-private` syncs just that one clone, same as @@ -33,14 +48,56 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" PROJECTS="${FM_PROJECTS_OVERRIDE:-$FM_HOME/projects}" +# shellcheck source=bin/fm-checkout-lock-lib.sh +. "$SCRIPT_DIR/fm-checkout-lock-lib.sh" +PROJECTS=$(fm_checkout_lexical_path "$PROJECTS" 1) || { + echo "error: projects root contains an unsafe or uninspectable path component: $PROJECTS" >&2 + exit 1 +} +if [ -e "$PROJECTS" ]; then + PROJECTS=$(fm_checkout_trusted_dir "$PROJECTS") || { + echo "error: projects root must be an exact real directory: $PROJECTS" >&2 + exit 1 + } +fi +CHECKOUT_STATE_BASE="${FM_CHECKOUT_REFRESH_STATE_BASE:-${XDG_STATE_HOME:-$HOME/.local/state}/firstmate/checkout-refresh}" +CHECKOUT_LOCK_ROOT=$(fm_checkout_lock_root "$CHECKOUT_STATE_BASE") # shellcheck source=bin/fm-gate-refuse-lib.sh . "$SCRIPT_DIR/fm-gate-refuse-lib.sh" fm_refuse_if_gate_agent # shellcheck source=bin/fm-lock-lib.sh . "$SCRIPT_DIR/fm-lock-lib.sh" +# shellcheck source=bin/fm-process-tree-lib.sh +. "$SCRIPT_DIR/fm-process-tree-lib.sh" FM_LOCK_LOG_PREFIX=fleet-sync "$FM_ROOT/bin/fm-guard.sh" || true +FLEET_SYNC_TIMEOUT=${FM_CHECKOUT_REFRESH_SYNC_TIMEOUT:-60} +EXPECTED_ORIGIN_KIND=${FM_FLEET_SYNC_EXPECTED_ORIGIN_KIND:-} +EXPECTED_ORIGIN_VALUE=${FM_FLEET_SYNC_EXPECTED_ORIGIN_VALUE:-} +EXPECTED_PHYSICAL_IDENTITY=${FM_FLEET_SYNC_EXPECTED_PHYSICAL_IDENTITY:-} +case "$FLEET_SYNC_TIMEOUT" in + ''|*[!0-9]*|0) + echo "error: FM_CHECKOUT_REFRESH_SYNC_TIMEOUT must be a positive integer" >&2 + exit 2 + ;; +esac +case "$EXPECTED_ORIGIN_KIND" in + ''|origin|no-origin) ;; + *) echo "error: invalid expected checkout origin kind" >&2; exit 2 ;; +esac +[ "$EXPECTED_ORIGIN_KIND" != origin ] || [ -n "$EXPECTED_ORIGIN_VALUE" ] || { + echo "error: expected origin URL is missing" >&2 + exit 2 +} +case "$EXPECTED_PHYSICAL_IDENTITY" in + '') ;; + *[!A-Za-z0-9:._-]*) + echo "error: invalid expected checkout physical identity" >&2 + exit 2 + ;; +esac + # Bounded recovery for an orphaned .git/packed-refs.lock. A git ref rewrite # (fetch --prune, branch -D, pack-refs) killed after creating the lock but before # renaming it - e.g. bootstrap's fleet-sync timeout kill, or teardown's process @@ -110,20 +167,81 @@ resolve_project_arg() { printf '%s\n' "$arg" } -default_branch() { - local ref branch - ref=$(git -C "$PROJ" symbolic-ref --quiet --short refs/remotes/origin/HEAD 2>/dev/null || true) - if [ -n "$ref" ]; then - echo "${ref#origin/}" - return 0 +canonical_dir() { + fm_checkout_trusted_dir "$1" +} + +exact_git_root() { + local candidate=$1 canonical top canonical_top + canonical=$(canonical_dir "$candidate") || return 1 + top=$(git -C "$canonical" rev-parse --show-toplevel 2>/dev/null) || return 1 + canonical_top=$(canonical_dir "$top") || return 1 + [ "$canonical" = "$canonical_top" ] || return 1 + fm_checkout_validate_git_metadata "$canonical" >/dev/null || return 1 + printf '%s\n' "$canonical" +} + +inspect_mutation_origin() { + local remotes + remotes=$(git -C "$PROJ" remote 2>/dev/null) || return 1 + if printf '%s\n' "$remotes" | grep -Fxq origin; then + MUTATION_ORIGIN_VALUE=$(git -C "$PROJ" remote get-url origin 2>/dev/null) || return 1 + [ -n "$MUTATION_ORIGIN_VALUE" ] || return 1 + MUTATION_ORIGIN_KIND=origin + else + MUTATION_ORIGIN_KIND=no-origin + MUTATION_ORIGIN_VALUE= fi - for branch in main master; do - if git -C "$PROJ" show-ref --verify --quiet "refs/heads/$branch"; then - echo "$branch" - return 0 - fi - done - return 1 +} + +verify_mutation_identity() { + local root physical + if [ "${FM_FLEET_SYNC_TEST:-0}" = 1 ] \ + && [ -n "${FM_FLEET_SYNC_TEST_DRIFT_ORIGIN_TO:-}" ] \ + && [ "${FM_FLEET_SYNC_TEST_DRIFTED:-0}" != 1 ]; then + git -C "$PROJ" remote set-url origin "$FM_FLEET_SYNC_TEST_DRIFT_ORIGIN_TO" || return 1 + FM_FLEET_SYNC_TEST_DRIFTED=1 + fi + root=$(exact_git_root "$PROJ") || return 1 + [ "$root" = "$PROJ" ] || return 1 + physical=$(fm_checkout_physical_path_identity "$PROJ" directory) || return 1 + if [ -z "$EXPECTED_PHYSICAL_IDENTITY" ]; then + EXPECTED_PHYSICAL_IDENTITY=$physical + fi + [ "$physical" = "$EXPECTED_PHYSICAL_IDENTITY" ] || return 1 + inspect_mutation_origin || return 1 + if [ -z "$EXPECTED_ORIGIN_KIND" ]; then + EXPECTED_ORIGIN_KIND=$MUTATION_ORIGIN_KIND + EXPECTED_ORIGIN_VALUE=$MUTATION_ORIGIN_VALUE + fi + [ "$MUTATION_ORIGIN_KIND" = "$EXPECTED_ORIGIN_KIND" ] \ + && [ "$MUTATION_ORIGIN_VALUE" = "$EXPECTED_ORIGIN_VALUE" ] +} + +LIVE_DEFAULT_BRANCH= +LIVE_DEFAULT_TIP= +LIVE_PROBE_OUTPUT= +probe_live_default() { + local line ref + LIVE_DEFAULT_BRANCH= + LIVE_DEFAULT_TIP= + LIVE_PROBE_OUTPUT=$(git -C "$PROJ" ls-remote --symref origin HEAD 2>&1) || return 1 + while IFS= read -r line; do + case "$line" in + "ref: refs/heads/"*$'\t'"HEAD") + ref=${line#ref: refs/heads/} + LIVE_DEFAULT_BRANCH=${ref%$'\t'HEAD} + ;; + *$'\t'"HEAD") + LIVE_DEFAULT_TIP=${line%$'\t'HEAD} + ;; + esac + done <<EOF +$LIVE_PROBE_OUTPUT +EOF + [ -n "$LIVE_DEFAULT_BRANCH" ] \ + && [ -n "$LIVE_DEFAULT_TIP" ] \ + && git check-ref-format --branch "$LIVE_DEFAULT_BRANCH" >/dev/null 2>&1 } first_line() { @@ -151,6 +269,14 @@ packed_refs_lock_path() { esac } +fetch_expected_origin() { + if ! verify_mutation_identity; then + FETCH_OUTPUT="checkout repository or origin identity drifted before fetch" + return 1 + fi + FETCH_OUTPUT=$(git -C "$PROJ" fetch origin --prune --quiet 2>&1) +} + # Run `git -C "$PROJ" fetch origin --prune --quiet`, tolerating an orphaned # packed-refs.lock left by a killed ref rewrite. Sets FETCH_OUTPUT to the git # command's combined output and returns its exit status. On the packed-refs.lock @@ -164,7 +290,7 @@ packed_refs_lock_path() { # a session-start refresh (which discards fleet-sync stderr) still surfaces it. fetch_with_packed_refs_lock_guard() { local rc attempt=0 lock lock_desc - FETCH_OUTPUT=$(git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$? + if fetch_expected_origin; then rc=0; else rc=$?; fi [ "$rc" -eq 0 ] && return 0 is_packed_refs_lock_error "$FETCH_OUTPUT" || return "$rc" @@ -174,7 +300,7 @@ fetch_with_packed_refs_lock_guard() { attempt=$(( attempt + 1 )) echo "$label: fetch blocked by packed-refs lock ($lock_desc); waiting ${FLEET_SYNC_PACKED_REFS_LOCK_RETRY_WAIT_SECS}s and retrying ($attempt/${FLEET_SYNC_PACKED_REFS_LOCK_RETRIES}) (owning process may be exiting)" >&2 sleep "$FLEET_SYNC_PACKED_REFS_LOCK_RETRY_WAIT_SECS" - FETCH_OUTPUT=$(git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$? + if fetch_expected_origin; then rc=0; else rc=$?; fi if [ "$rc" -eq 0 ]; then echo "$label: fetch succeeded on retry; packed-refs lock cleared on its own" >&2 # One stdout summary so a session-start refresh (which discards fleet-sync @@ -198,7 +324,7 @@ fetch_with_packed_refs_lock_guard() { return "$rc" fi echo "$label: removed provably-stale packed-refs lock $lock (age >= ${FLEET_SYNC_PACKED_REFS_LOCK_AGE_SECS}s, no live holder) and retrying fetch" >&2 - FETCH_OUTPUT=$(git -C "$PROJ" fetch origin --prune --quiet 2>&1); rc=$? + if fetch_expected_origin; then rc=0; else rc=$?; fi if [ "$rc" -eq 0 ]; then echo "$label: fetch succeeded after stale packed-refs lock cleanup" >&2 echo "$label: recovered: removed a stale packed-refs lock (no live holder)" @@ -214,22 +340,29 @@ fetch_with_packed_refs_lock_guard() { } prune_gone_branches() { - # Delete local branches whose upstream tracking branch is gone - the remote - # branch was deleted, which in this fleet means its PR merged - as long as - # nothing still needs them. Never the checked-out branch, and never a branch - # that still has a worktree (a live or not-yet-torn-down task). "Gone" plus - # "no worktree" already proves the work landed: teardown removes a branch's - # worktree only after confirming the work reached the remote. We deliberately - # do NOT also require the branch to be an ancestor of origin/<default> - PRs in - # this fleet are squash-merged, so a merged branch is never an ancestor and - # such a check would prune nothing. The no-worktree guard is the real safety - # net. Set FM_FLEET_PRUNE=0 to skip pruning entirely. [ "${FM_FLEET_PRUNE:-1}" != "0" ] || return 0 - local worktree_branches current refline branch track - worktree_branches=$(git -C "$PROJ" worktree list --porcelain 2>/dev/null \ - | sed -n 's#^branch refs/heads/##p') + local worktree_output worktree_branches current refs_output refline branch track + local remote_refs remote_ref base_tree merged_tree landed + worktree_output=$(git -C "$PROJ" worktree list --porcelain 2>/dev/null) || { + echo "$label: skipped: cannot inspect worktree ownership before branch pruning" + return 1 + } + worktree_branches=$(printf '%s\n' "$worktree_output" | sed -n 's#^branch refs/heads/##p') current=$(git -C "$PROJ" symbolic-ref --quiet --short HEAD 2>/dev/null || true) + refs_output=$(git -C "$PROJ" for-each-ref \ + --format='%(refname:short) %(upstream:track)' refs/heads 2>/dev/null) || { + echo "$label: skipped: cannot inspect local branches before pruning" + return 1 + } + remote_refs=$(git -C "$PROJ" for-each-ref --format='%(refname)' refs/remotes 2>/dev/null) || { + echo "$label: skipped: cannot inspect remote refs before branch pruning" + return 1 + } + base_tree=$(git -C "$PROJ" rev-parse "$BASE^{tree}" 2>/dev/null) || { + echo "$label: skipped: cannot inspect default-branch content before branch pruning" + return 1 + } while IFS= read -r refline; do branch=${refline%% *} @@ -240,11 +373,30 @@ prune_gone_branches() { if printf '%s\n' "$worktree_branches" | grep -Fxq -- "$branch"; then continue fi + landed=0 + while IFS= read -r remote_ref; do + [ -n "$remote_ref" ] || continue + if git -C "$PROJ" merge-base --is-ancestor "$branch" "$remote_ref" 2>/dev/null; then + landed=1 + break + fi + done <<EOF +$remote_refs +EOF + if [ "$landed" -eq 0 ]; then + merged_tree=$(git -C "$PROJ" merge-tree --write-tree "$BASE" "$branch" 2>/dev/null || true) + [ -n "$merged_tree" ] && [ "$merged_tree" = "$base_tree" ] && landed=1 + fi + if [ "$landed" -eq 0 ]; then + echo "$label: STUCK: retained gone branch $branch because landed work cannot be proved" + continue + fi if git -C "$PROJ" branch -D -- "$branch" >/dev/null 2>&1; then echo "$label: pruned $branch" fi - done < <(git -C "$PROJ" for-each-ref \ - --format='%(refname:short) %(upstream:track)' refs/heads 2>/dev/null) + done <<EOF +$refs_output +EOF } # True when some worktree of $PROJ has $DEFAULT checked out (so we cannot attach @@ -279,7 +431,16 @@ stuck_state() { else s="detached HEAD" fi - [ "$dirty" = no ] || s="$s with uncommitted changes" + if [ "$dirty" = yes ]; then + s="$s with uncommitted changes" + if [ "$untracked_count" -gt 0 ]; then + s="$s ($untracked_count untracked" + if [ "$skill_draft_count" -gt 0 ]; then + s="$s, $skill_draft_count under repository skill directories" + fi + s="$s)" + fi + fi printf '%s\n' "$s" } @@ -292,7 +453,7 @@ report_stuck() { echo "$label: STUCK: on $state, $behind commits behind $BASE - needs attention" } -sync_project() { +sync_project() ( PROJ=$1 label=$(project_label) @@ -304,17 +465,45 @@ sync_project() { echo "$label: skipped: not a git repo" return 0 fi - mode_line=$("$FM_ROOT/bin/fm-project-mode.sh" "$label" 2>/dev/null || echo "no-mistakes off") + canonical=$(exact_git_root "$PROJ") || { + echo "$label: skipped: target must be an exact canonical Git repository root" + return 1 + } + PROJ=$canonical + label=$(project_label) + registry_name=$(basename "$PROJ") + mode_line=$("$FM_ROOT/bin/fm-project-mode.sh" "$registry_name" 2>/dev/null || echo "no-mistakes off") mode=${mode_line%% *} if [ "$mode" = "local-only" ]; then echo "$label: skipped: local-only project" return 0 fi - if ! git -C "$PROJ" remote get-url origin >/dev/null 2>&1; then - echo "$label: skipped: no origin remote" + if ! fm_checkout_lock_prepare "$CHECKOUT_LOCK_ROOT"; then + echo "$PROJ: skipped: refresh lock setup failed" return 0 fi + checkout_lock=$(fm_checkout_lock_path "$PROJ" "$CHECKOUT_LOCK_ROOT") || { + echo "$PROJ: skipped: repository lock identity cannot be resolved" + return 0 + } + FM_CHECKOUT_LOCK_ACTIVE_PATH=${FM_FLEET_SYNC_LOCK_PATH:-} + FM_CHECKOUT_LOCK_ACTIVE_OWNER_DIR=${FM_FLEET_SYNC_LOCK_OWNER_DIR:-} + FM_CHECKOUT_LOCK_ACTIVE_OWNER_PID=${FM_FLEET_SYNC_LOCK_OWNER_PID:-} + if [ "${FM_FLEET_SYNC_BOUNDED_CHILD:-0}" != 1 ] \ + || [ "$FM_CHECKOUT_LOCK_ACTIVE_PATH" != "$checkout_lock" ] \ + || ! fm_checkout_lock_active_scope_owns "$checkout_lock"; then + echo "$label: skipped: bounded refresh does not own the shared checkout mutation lock" + return "$FM_CHECKOUT_LOCK_FAILURE_STATUS" + fi + if ! verify_mutation_identity; then + echo "$label: skipped: checkout repository or origin identity drifted before mutation" + return 0 + fi + if [ "$EXPECTED_ORIGIN_KIND" != origin ]; then + echo "$label: skipped: no origin remote" + return 0 + fi if ! fetch_with_packed_refs_lock_guard; then reason="fetch failed" if [ -n "$FETCH_OUTPUT" ]; then @@ -324,21 +513,54 @@ sync_project() { return 0 fi - prune_gone_branches || true - - DEFAULT=$(default_branch) || { - echo "$label: skipped: cannot determine default branch" + if ! verify_mutation_identity; then + echo "$label: skipped: checkout repository or origin identity drifted after fetch" return 0 - } + fi + if ! probe_live_default; then + reason="cannot probe live upstream default branch" + if [ -n "$LIVE_PROBE_OUTPUT" ]; then + reason="$reason: $(first_line "$LIVE_PROBE_OUTPUT")" + fi + echo "$label: skipped: $reason" + return 0 + fi + DEFAULT=$LIVE_DEFAULT_BRANCH BASE="origin/$DEFAULT" if ! git -C "$PROJ" rev-parse --verify --quiet "$BASE^{commit}" >/dev/null; then echo "$label: skipped: $BASE does not exist" return 0 fi + remote_rev=$(git -C "$PROJ" rev-parse "$BASE^{commit}") || { + echo "$label: skipped: cannot read $BASE" + return 0 + } + if [ "$remote_rev" != "$LIVE_DEFAULT_TIP" ]; then + echo "$label: skipped: fetched $BASE does not match live upstream HEAD - retry later" + return 0 + fi cur=$(git -C "$PROJ" symbolic-ref --short HEAD 2>/dev/null || echo "") dirty=no - [ -z "$(git -C "$PROJ" status --porcelain 2>/dev/null | head -1)" ] || dirty=yes + if ! status_raw=$(GIT_OPTIONAL_LOCKS=0 git -C "$PROJ" status --porcelain=v1 --untracked-files=all 2>/dev/null); then + echo "$label: skipped: working tree cleanliness cannot be inspected" + return 0 + fi + [ -z "$status_raw" ] || dirty=yes + if ! verify_mutation_identity; then + echo "$label: skipped: checkout repository or origin identity drifted before branch pruning" + return 0 + fi + prune_gone_branches || return 0 + untracked_count=0 + skill_draft_count=0 + if [ "$dirty" = yes ]; then + untracked_count=$(git -C "$PROJ" ls-files --others --exclude-standard -- 2>/dev/null \ + | awk 'END { print NR + 0 }') + skill_draft_count=$(git -C "$PROJ" ls-files --others --exclude-standard -- \ + .agents/skills .claude/skills .codex/skills skills 2>/dev/null \ + | awk 'END { print NR + 0 }') + fi recovered=no if [ "$cur" != "$DEFAULT" ]; then @@ -354,6 +576,10 @@ sync_project() { && git -C "$PROJ" merge-base --is-ancestor HEAD "$BASE" 2>/dev/null \ && ! default_checked_out_elsewhere \ && local_default_safe_for_recovery; then + if ! verify_mutation_identity; then + echo "$label: skipped: checkout repository or origin identity drifted before checkout" + return 0 + fi if ! git -C "$PROJ" checkout --quiet "$DEFAULT" 2>/dev/null; then report_stuck "$(stuck_state)" return 0 @@ -400,6 +626,10 @@ sync_project() { echo "$label: skipped: cannot read local $DEFAULT" return 0 } + if ! verify_mutation_identity; then + echo "$label: skipped: checkout repository or origin identity drifted before fast-forward" + return 0 + fi if ! merge_output=$(git -C "$PROJ" merge --ff-only "$BASE" 2>&1); then reason="fast-forward failed" if [ -n "$merge_output" ]; then @@ -418,16 +648,84 @@ sync_project() { echo "$label: synced $before..$after" fi return 0 -} +) + +run_sync_project_bounded() ( + local project=$1 status label checkout_lock lock_owner_dir lock_owner_pid cleanup_status canonical + if [ ! -d "$project" ] \ + || ! git -C "$project" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + sync_project "$project" + return + fi + canonical=$(exact_git_root "$project") || { + printf '%s: skipped: target must be an exact canonical Git repository root\n' "$project" + return 1 + } + project=$canonical + PROJ=$project + label=$(project_label) + if ! fm_checkout_lock_prepare "$CHECKOUT_LOCK_ROOT"; then + echo "$project: skipped: refresh lock setup failed" + return 0 + fi + checkout_lock=$(fm_checkout_lock_path "$project" "$CHECKOUT_LOCK_ROOT") || { + echo "$project: skipped: repository lock identity cannot be resolved" + return 0 + } + if ! fm_lock_try_acquire "$checkout_lock"; then + echo "$project: skipped: refresh already running (pid ${FM_LOCK_HELD_PID:-unknown})" + return 0 + fi + lock_owner_dir=${FM_LOCK_OWNER_DIR:?} + lock_owner_pid=$(cat "$lock_owner_dir/pid" 2>/dev/null) || { + fm_lock_release "$checkout_lock" + echo "$project: skipped: refresh lock ownership cannot be proved" + return 0 + } + trap 'fm_lock_release "$checkout_lock"' EXIT + if FM_PROCESS_TREE_GUARD_FILE="$lock_owner_dir/process-group" \ + fm_run_bounded "$FLEET_SYNC_TIMEOUT" \ + env FM_FLEET_SYNC_BOUNDED_CHILD=1 \ + FM_FLEET_SYNC_LOCK_PATH="$checkout_lock" \ + FM_FLEET_SYNC_LOCK_OWNER_DIR="$lock_owner_dir" \ + FM_FLEET_SYNC_LOCK_OWNER_PID="$lock_owner_pid" \ + "$SCRIPT_DIR/fm-fleet-sync.sh" "$project"; then + status=0 + else + status=$? + fi + cleanup_status=$FM_PROCESS_TREE_CLEANUP_STATUS + if [ "$cleanup_status" != verified ]; then + printf '%s: skipped: refresh process cleanup is unverified; the guarded checkout lock is retained for inspection\n' "$label" + return "$FM_CHECKOUT_PROCESS_CLEANUP_FAILURE_STATUS" + fi + [ "$status" -ne 0 ] || return 0 + if [ "$status" -eq 124 ]; then + printf '%s: skipped: refresh timed out after %ss\n' "$label" "$FLEET_SYNC_TIMEOUT" + return 0 + fi + return "$status" +) if [ $# -eq 1 ]; then - sync_project "$(resolve_project_arg "$1")" - exit 0 + project=$(resolve_project_arg "$1") + if [ "${FM_FLEET_SYNC_BOUNDED_CHILD:-0}" = 1 ]; then + sync_project "$project" + exit $? + else + run_sync_project_bounded "$project" + exit $? + fi fi [ -d "$PROJECTS" ] || exit 0 for proj in "$PROJECTS"/*; do [ -e "$proj" ] || continue + if [ -L "$proj" ]; then + printf '%s: skipped: target must be an exact canonical Git repository root\n' "$proj" + false + continue + fi [ -d "$proj" ] || continue - sync_project "$proj" + run_sync_project_bounded "$proj" done diff --git a/bin/fm-gate-refuse-lib.sh b/bin/fm-gate-refuse-lib.sh index bc87bc63dbc..4910a12c551 100644 --- a/bin/fm-gate-refuse-lib.sh +++ b/bin/fm-gate-refuse-lib.sh @@ -56,7 +56,8 @@ # Guarded direct mutators: fm-account-continuation.sh, # fm-account-session-sync.sh, fm-afk-launch.sh, fm-afk-start.sh, # fm-backlog-handoff.sh, fm-bootstrap.sh, fm-brief.sh, fm-config-push.sh, -# fm-ensure-agents-md.sh, fm-fleet-sync.sh, fm-home-seed.sh, fm-lock.sh, +# fm-checkout-refresh.sh, fm-ensure-agents-md.sh, fm-fleet-sync.sh, +# fm-home-seed.sh, fm-lock.sh, # fm-merge-local.sh, fm-pr-check.sh, fm-pr-merge.sh, fm-promote.sh, # fm-report-retention.sh, fm-report-stack.mjs, fm-review-diff.sh, fm-send.sh, # fm-session-start.sh, fm-spawn.sh, fm-supervise-daemon.sh, diff --git a/bin/fm-home-seed.sh b/bin/fm-home-seed.sh index a97874d4b86..eac1c90072a 100755 --- a/bin/fm-home-seed.sh +++ b/bin/fm-home-seed.sh @@ -4,11 +4,18 @@ # Usage: # fm-home-seed.sh <id> <home|-> {<project>...|--no-projects} # Provision <home> as an isolated firstmate home. If <home> is "-", acquire -# a fresh firstmate worktree via "treehouse get --lease", which durably -# leases the worktree under the secondmate <id> so the home survives with -# no live process and is never recycled until the lease is released with -# "treehouse return". Projects are cloned -# from the active home into the secondmate home's projects/ directory. +# a fresh firstmate worktree through the shared locked, bounded +# "treehouse get --lease" entrypoint, which durably leases the worktree +# under the secondmate <id> so the home survives with no live process and +# is never recycled until the lease is released with "treehouse return". +# The acquired home is accepted only when its HEAD +# belongs to the Firstmate repository, is clean, and matches the live +# upstream default-branch tip. An unsafe or unverifiable acquired home is +# retained without force-return so its unlanded work remains untouched. +# An explicit home must pass the same live-tip proof after its safe +# fast-forward, and a failed proof aborts the transactional seed. +# Projects are cloned from the active home into the secondmate home's +# projects/ directory. # That project list is non-exclusive provisioning data. Pass --no-projects # instead of a project list to seed a project-less home for a domain whose # subject is the firstmate repo itself; it is mutually exclusive with a @@ -20,8 +27,10 @@ # data/secondmates.md is updated. # Seeding is transactional: on validation, clone, init, or registry failure, # generated briefs, new homes, new project clones, and registry edits are -# rolled back. Treehouse-acquired homes are returned only when the rollback -# target is safe; a failed return warns because the lease may still be held. +# rolled back. Clean Treehouse-acquired homes are returned only when the +# rollback target, repository identity, and expected detached tip are +# re-proven, and the return holds the common checkout mutation lock; a +# failed return warns because the lease may still be held. # Set FM_SECONDMATE_CHARTER='<charter>' to seed from inline charter text # when no filled charter brief exists. Set FM_SECONDMATE_SCOPE='<scope>' # to override the registry routing scope. Otherwise the registry summary @@ -36,8 +45,14 @@ FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" PROJECTS="${FM_PROJECTS_OVERRIDE:-$FM_HOME/projects}" +CHECKOUT_STATE_BASE="${FM_CHECKOUT_REFRESH_STATE_BASE:-${XDG_STATE_HOME:-$HOME/.local/state}/firstmate/checkout-refresh}" REG="$DATA/secondmates.md" SUB_HOME_MARKER=".fm-secondmate-home" +# shellcheck source=bin/fm-checkout-lock-lib.sh +. "$SCRIPT_DIR/fm-checkout-lock-lib.sh" +CHECKOUT_LOCK_ROOT=$(fm_checkout_lock_root "$CHECKOUT_STATE_BASE") +# shellcheck source=bin/fm-account-routing-lib.sh +. "$SCRIPT_DIR/fm-account-routing-lib.sh" # shellcheck source=bin/fm-gate-refuse-lib.sh . "$SCRIPT_DIR/fm-gate-refuse-lib.sh" fm_refuse_if_gate_agent @@ -228,76 +243,10 @@ registry_id_conflict_for_assignment() { } validate_registry() { - local tmp line id registered_home home_key duplicate_homes duplicate_ids overlaps - tmp=$(mktemp "${TMPDIR:-/tmp}/fm-firstmates.XXXXXX") - if [ -f "$REG" ]; then - while IFS= read -r line; do - case "$line" in - "- "*) - id=${line#- } - id=${id%% *} - registered_home=$(printf '%s\n' "$line" | registry_home_for_line) - [ -n "$registered_home" ] || continue - home_key=$(resolved_path "$registered_home") - printf '%s\t%s\n' "$home_key" "$id" >> "$tmp" - ;; - esac - done < "$REG" - fi - duplicate_homes=$(awk -F '\t' ' - { - if (($1 in owner) && owner[$1] != $2) { - print $1 ": " owner[$1] ", " $2 - bad=1 - } else { - owner[$1]=$2 - } - } - END { exit bad ? 1 : 0 } - ' "$tmp" 2>/dev/null) || { - rm -f "$tmp" - printf 'error: duplicate secondmate home assignment:\n%s\n' "$duplicate_homes" >&2 - return 1 - } - duplicate_ids=$(awk -F '\t' ' - { - if ($2 in home) { - print $2 ": " home[$2] ", " $1 - bad=1 - } else { - home[$2]=$1 - } - } - END { exit bad ? 1 : 0 } - ' "$tmp" 2>/dev/null) || { - rm -f "$tmp" - printf 'error: duplicate secondmate id assignment:\n%s\n' "$duplicate_ids" >&2 - return 1 - } - overlaps=$(awk -F '\t' ' - function ancestor(a, b) { return a != b && index(b, a "/") == 1 } - { - for (i = 1; i <= count; i++) { - if (ancestor($1, path[i])) { - print $1 " (" $2 ") contains " path[i] " (" id[i] ")" - bad=1 - } else if (ancestor(path[i], $1)) { - print path[i] " (" id[i] ") contains " $1 " (" $2 ")" - bad=1 - } - } - count++ - path[count]=$1 - id[count]=$2 - } - END { exit bad ? 1 : 0 } - ' "$tmp" 2>/dev/null) || { - rm -f "$tmp" - printf 'error: overlapping secondmate home assignment:\n%s\n' "$overlaps" >&2 + fm_secondmate_registry_query "$REG" validate || { + echo "error: secondmate registry is malformed, duplicated, redirected, or uninspectable: $REG" >&2 return 1 } - rm -f "$tmp" - return 0 } join_projects() { @@ -473,7 +422,7 @@ acquire_treehouse_home() { # live process and is skipped by later get/prune, so the home survives restarts # until teardown or rollback returns it. treehouse prints only the worktree path # to stdout (banners go to stderr), so command substitution captures the path. - home=$(cd "$FM_ROOT" && treehouse get --lease --lease-holder "$id") || { + home=$("$SCRIPT_DIR/fm-checkout-refresh.sh" acquire-worktree "$FM_ROOT" "$id") || { echo "error: treehouse get --lease failed to lease a firstmate home" >&2 return 1 } @@ -584,17 +533,20 @@ SEED_ROLLBACK_ACTIVE=0 SEED_COMMITTED=0 SEED_HOME= SEED_HOME_ACQUIRED=0 +SEED_HOME_RETAINED=0 +SEED_HOME_EXPECTED_TIP= SEED_HOME_CREATED=0 SEED_HOME_BACKED_UP=0 SEED_BACKUP_DIR= SEED_CREATED_PROJECTS_FILE= -SEED_PARENT_REG_EXISTED=0 SEED_PARENT_BRIEF= SEED_PARENT_BRIEF_CREATED=0 SEED_PARENT_BRIEF_DIR_CREATED=0 SEED_SUB_REG_EXISTED=0 SEED_CHARTER_EXISTED=0 SEED_MARKER_EXISTED=0 +SEED_HOME_LIFECYCLE_LOCK= +SEED_PARENT_HOME_LIFECYCLE_LOCK= restore_seed_file() { local existed=$1 backup=$2 path=$3 @@ -643,11 +595,16 @@ seed_rollback_target() { seed_return_treehouse_home() { local home=$1 abs_home abs_home=$(seed_rollback_target "$home" "treehouse-acquired home") || return 0 + if [ -z "$SEED_HOME_EXPECTED_TIP" ] \ + || ! "$SCRIPT_DIR/fm-checkout-refresh.sh" verify-returnable "$abs_home" "$FM_ROOT" "$SEED_HOME_EXPECTED_TIP"; then + echo "warning: retaining unsafe treehouse-acquired home $abs_home because repository identity and its expected detached tip could not be re-proven" >&2 + return 0 + fi if ! command -v treehouse >/dev/null 2>&1; then echo "warning: failed to return treehouse-acquired home $abs_home during seed rollback; treehouse command not found" >&2 return 0 fi - ( cd "$FM_ROOT" && treehouse return --force "$abs_home" >/dev/null ) || { + fm_checkout_treehouse_return "$abs_home" "$CHECKOUT_LOCK_ROOT" "$FM_ROOT" >/dev/null || { echo "warning: failed to return treehouse-acquired home $abs_home during seed rollback; lease may still be held" >&2 return 0 } @@ -702,7 +659,11 @@ seed_rollback() { if [ -n "${SEED_HOME:-}" ] && [ "$SEED_HOME" != "/" ]; then if [ "$SEED_HOME_ACQUIRED" = 1 ]; then - seed_return_treehouse_home "$SEED_HOME" + if [ "$SEED_HOME_RETAINED" = 1 ]; then + echo "warning: retaining unsafe treehouse-acquired home $SEED_HOME for manual recovery" >&2 + else + seed_return_treehouse_home "$SEED_HOME" + fi elif [ "$SEED_HOME_CREATED" = 1 ]; then seed_remove_created_home "$SEED_HOME" else @@ -721,11 +682,26 @@ seed_rollback() { fi if [ -n "${SEED_BACKUP_DIR:-}" ]; then - restore_seed_file "$SEED_PARENT_REG_EXISTED" "$SEED_BACKUP_DIR/parent-secondmates.md" "$REG" rm -rf -- "$SEED_BACKUP_DIR" 2>/dev/null || true fi } +seed_release_locks() { + if [ -n "${SEED_HOME_LIFECYCLE_LOCK:-}" ]; then + fm_account_lifecycle_lock_release "$SEED_HOME_LIFECYCLE_LOCK" >/dev/null 2>&1 || true + SEED_HOME_LIFECYCLE_LOCK= + fi + if [ -n "${SEED_PARENT_HOME_LIFECYCLE_LOCK:-}" ]; then + fm_account_lifecycle_lock_release "$SEED_PARENT_HOME_LIFECYCLE_LOCK" >/dev/null 2>&1 || true + SEED_PARENT_HOME_LIFECYCLE_LOCK= + fi +} + +seed_exit_cleanup() { + seed_rollback + seed_release_locks +} + registry_line_for_project() { local project=$1 line [ -f "$DATA/projects.md" ] || return 1 @@ -796,22 +772,59 @@ initialize_no_mistakes_project() { } write_registry() { - local id=$1 home=$2 projects_csv=$3 brief=$4 scope summary tmp today + local id=$1 home=$2 projects_csv=$3 brief=$4 scope summary tmp today registry_lock status=1 mkdir -p "$DATA" [ -d "$DATA" ] && [ ! -L "$DATA" ] || return 1 scope=$(registry_scope_for_brief "$brief") summary=$(registry_summary_for_brief "$brief") today=$(date +%F) - if [ -L "$REG" ] || { [ -e "$REG" ] && [ ! -f "$REG" ]; }; then return 1; fi - tmp=$(mktemp "$DATA/.secondmates.XXXXXX") || return 1 + registry_lock=$(fm_secondmate_registry_lock_acquire "$CHECKOUT_LOCK_ROOT" "$REG") || return 1 + if [ -L "$REG" ] || { [ -e "$REG" ] && [ ! -f "$REG" ]; }; then + fm_account_lifecycle_lock_release "$registry_lock" >/dev/null 2>&1 || true + return 1 + fi + validate_registry || { + fm_account_lifecycle_lock_release "$registry_lock" >/dev/null 2>&1 || true + return 1 + } + validate_home_assignment "$id" "$home" || { + fm_account_lifecycle_lock_release "$registry_lock" >/dev/null 2>&1 || true + return 1 + } + tmp=$(mktemp "$DATA/.secondmates.XXXXXX") || { + fm_account_lifecycle_lock_release "$registry_lock" >/dev/null 2>&1 || true + return 1 + } if [ -f "$REG" ]; then - grep -vE "^- $id( |$)" "$REG" > "$tmp" || true + python3 - "$REG" "$tmp" "$id" <<'PY' || { +import sys + +source, destination, expected = sys.argv[1:] +with open(source, encoding="utf-8") as stream, open(destination, "w", encoding="utf-8") as output: + for line in stream: + if line.startswith("- "): + item = line[2:].split(None, 1)[0] if line[2:].strip() else "" + if item == expected: + continue + output.write(line) +PY + rm -f "$tmp" + fm_account_lifecycle_lock_release "$registry_lock" >/dev/null 2>&1 || true + return 1 + } else : > "$tmp" fi printf -- '- %s - %s (home: %s; scope: %s; projects: %s; added %s)\n' "$id" "$summary" "$home" "$scope" "$projects_csv" "$today" >> "$tmp" - if [ -L "$REG" ] || { [ -e "$REG" ] && [ ! -f "$REG" ]; }; then rm -f "$tmp"; return 1; fi - mv "$tmp" "$REG" + if ! { [ ! -L "$REG" ] && { [ ! -e "$REG" ] || [ -f "$REG" ]; }; }; then + rm -f "$tmp" + elif mv "$tmp" "$REG"; then + status=0 + else + rm -f "$tmp" + fi + fm_account_lifecycle_lock_release "$registry_lock" >/dev/null 2>&1 || true + return "$status" } refuse_populated_projectless_home() { @@ -902,37 +915,64 @@ seed_home() { SEED_COMMITTED=0 SEED_HOME= SEED_HOME_ACQUIRED=0 + SEED_HOME_RETAINED=0 + SEED_HOME_EXPECTED_TIP= SEED_HOME_CREATED=0 - SEED_HOME_ACQUIRED=0 SEED_HOME_BACKED_UP=0 SEED_BACKUP_DIR=$(mktemp -d "${TMPDIR:-/tmp}/fm-home-seed.XXXXXX") SEED_CREATED_PROJECTS_FILE="$SEED_BACKUP_DIR/created-projects" : > "$SEED_CREATED_PROJECTS_FILE" - SEED_PARENT_REG_EXISTED=0 SEED_PARENT_BRIEF="$DATA/$id/brief.md" SEED_PARENT_BRIEF_CREATED=0 SEED_PARENT_BRIEF_DIR_CREATED=0 SEED_SUB_REG_EXISTED=0 SEED_CHARTER_EXISTED=0 SEED_MARKER_EXISTED=0 - trap seed_rollback EXIT - if [ -f "$REG" ]; then - SEED_PARENT_REG_EXISTED=1 - cp "$REG" "$SEED_BACKUP_DIR/parent-secondmates.md" + trap seed_exit_cleanup EXIT + SEED_PARENT_HOME_LIFECYCLE_LOCK=$(fm_secondmate_home_lifecycle_lock_acquire "$CHECKOUT_LOCK_ROOT" "$FM_HOME") || return 1 + fm_checkout_trusted_dir "$FM_HOME" >/dev/null || { + echo "error: active firstmate home was removed or redirected while seed waited for lifecycle ownership" >&2 + return 1 + } + if [ -e "$FM_HOME/data/charter.md" ]; then + [ -f "$FM_HOME/$SUB_HOME_MARKER" ] && [ ! -L "$FM_HOME/$SUB_HOME_MARKER" ] || { + echo "error: active secondmate home changed while seed waited for lifecycle ownership" >&2 + return 1 + } fi if [ "$requested_home" = "-" ]; then SEED_HOME_ACQUIRED=1 home=$(acquire_treehouse_home "$id") SEED_HOME="$home" + SEED_HOME_RETAINED=1 + SEED_HOME_LIFECYCLE_LOCK=$(fm_secondmate_home_lifecycle_lock_acquire "$CHECKOUT_LOCK_ROOT" "$home") || return 1 + freshness_status=0 + "$SCRIPT_DIR/fm-checkout-refresh.sh" verify-worktree "$home" "$FM_ROOT" || freshness_status=$? + if [ "$freshness_status" -ne 0 ]; then + echo "error: refusing secondmate home acquired from a stale or unverifiable upstream default" >&2 + return 1 + fi + SEED_HOME_EXPECTED_TIP=$(git -C "$home" rev-parse HEAD) || return 1 + SEED_HOME_RETAINED=0 home=$(verify_firstmate_home "$home") else requested_abs=$(abs_path_for_new "$requested_home") refuse_active_home_path "$requested_abs" || return 1 + SEED_HOME_LIFECYCLE_LOCK=$(fm_secondmate_home_lifecycle_lock_acquire "$CHECKOUT_LOCK_ROOT" "$requested_abs") || return 1 validate_home_assignment "$id" "$requested_abs" || return 1 + "$SCRIPT_DIR/fm-checkout-refresh.sh" preflight "$FM_ROOT" >/dev/null 2>&1 || true SEED_HOME="$requested_abs" [ -e "$requested_abs" ] || SEED_HOME_CREATED=1 home=$(ensure_home "$id" "$requested_abs") + "$SCRIPT_DIR/fm-checkout-refresh.sh" preflight "$home" || { + echo "error: refusing explicit secondmate home whose default branch cannot be refreshed safely" >&2 + return 1 + } + "$SCRIPT_DIR/fm-checkout-refresh.sh" verify-home "$home" "$FM_ROOT" || { + echo "error: refusing explicit secondmate home whose live default-tip freshness cannot be proved" >&2 + return 1 + } fi SEED_HOME="$home" validate_registry_home_text "$home" || return 1 @@ -1007,11 +1047,11 @@ seed_home() { projects_csv=$(join_projects "$@") printf '%s\n' "$id" > "$home/$SUB_HOME_MARKER" - write_registry "$id" "$home" "$projects_csv" "$SEED_PARENT_BRIEF" - validate_registry + write_registry "$id" "$home" "$projects_csv" "$SEED_PARENT_BRIEF" || return 1 SEED_COMMITTED=1 - trap - EXIT rm -rf -- "$SEED_BACKUP_DIR" + seed_release_locks + trap - EXIT printf 'home=%s\n' "$home" } diff --git a/bin/fm-process-tree-lib.sh b/bin/fm-process-tree-lib.sh new file mode 100644 index 00000000000..16e3ded145f --- /dev/null +++ b/bin/fm-process-tree-lib.sh @@ -0,0 +1,335 @@ +#!/usr/bin/env bash +# Shared bounded command runner for operations whose descendants must be +# terminated and reaped before the caller releases lifecycle or Git locks. +# Usage: fm_run_bounded <positive-seconds> <command> [args...] +# After every call, FM_PROCESS_TREE_CLEANUP_STATUS is verified, unverified, or +# not-started, while the function return preserves the wrapped command status. + +FM_PROCESS_TREE_SETUP_FAILURE_STATUS=126 +FM_PROCESS_TREE_CLEANUP_STATUS=not-started + +fm_process_tree_emit_snapshot() { + local path=$1 size + size=$(wc -c < "$path" 2>/dev/null | tr -d '[:space:]') || return 0 + case "$size" in ''|*[!0-9]*|0) return 0 ;; esac + perl -e ' + my ($path, $remaining) = @ARGV; + open my $file, "<", $path or exit 0; + while ($remaining > 0) { + my $wanted = $remaining > 65536 ? 65536 : $remaining; + my $read = read $file, my $buffer, $wanted; + last if !defined $read || $read == 0; + print $buffer; + $remaining -= $read; + } + ' "$path" "$size" +} + +fm_run_bounded() { + local seconds=$1 result_file stdout_file stderr_file result status cleanup + shift + FM_PROCESS_TREE_CLEANUP_STATUS=not-started + command -v perl >/dev/null 2>&1 || { + echo "error: perl is required for bounded process-tree control" >&2 + return 127 + } + result_file=$(mktemp "${TMPDIR:-/tmp}/fm-process-tree-result.XXXXXX") || { + echo "error: cannot create bounded process-tree result channel" >&2 + return "$FM_PROCESS_TREE_SETUP_FAILURE_STATUS" + } + stdout_file=$(mktemp "${TMPDIR:-/tmp}/fm-process-tree-stdout.XXXXXX") || { + rm -f "$result_file" + echo "error: cannot create bounded process-tree output channel" >&2 + return "$FM_PROCESS_TREE_SETUP_FAILURE_STATUS" + } + stderr_file=$(mktemp "${TMPDIR:-/tmp}/fm-process-tree-stderr.XXXXXX") || { + rm -f "$result_file" "$stdout_file" + echo "error: cannot create bounded process-tree diagnostic channel" >&2 + return "$FM_PROCESS_TREE_SETUP_FAILURE_STATUS" + } + # shellcheck disable=SC2016 + if FM_PROCESS_TREE_RESULT_FILE=$result_file perl -MPOSIX=:sys_wait_h -MErrno=EINTR -e ' + sub record_cleanup { + my ($state) = @_; + my $path = $ENV{FM_PROCESS_TREE_RESULT_FILE} || return; + open my $file, ">", $path or return; + print {$file} "$state\n"; + close $file; + } + sub group_members { + my ($group) = @_; + my @members; + open my $ps, "-|", "ps", "-axo", "pid=,pgid=" or return; + while (<$ps>) { + my ($pid, $pgid) = /^\s*(\d+)\s+(\d+)\s*$/; + push @members, $pid if defined $pgid && $pgid == $group; + } + close $ps or return; + return \@members; + } + sub anchored_members { + my ($group, $anchor) = @_; + my $members = group_members($group); + return if !defined $members; + my $anchor_present = grep { $_ == $anchor } @$members; + return if !$anchor_present; + return [grep { $_ != $anchor } @$members]; + } + sub install_guard { + my ($group) = @_; + my $guard = $ENV{FM_PROCESS_TREE_GUARD_FILE} || return 1; + my $tmp = "$guard.$$"; + open my $file, ">", $tmp or return 0; + print {$file} "$group\n"; + close $file or do { unlink $tmp; return 0 }; + rename $tmp, $guard or do { unlink $tmp; return 0 }; + return 1; + } + sub clear_guard { + my ($group) = @_; + my $guard = $ENV{FM_PROCESS_TREE_GUARD_FILE} || return 1; + open my $file, "<", $guard or return 0; + my $recorded = <$file>; + close $file; + return 0 if !defined $recorded; + chomp $recorded; + return 0 if $recorded ne "$group"; + return unlink $guard; + } + sub terminate_owned { + my ($group, $anchor) = @_; + my $members = anchored_members($group, $anchor); + return if !defined $members; + return "alive" unless @$members; + kill "TERM", -$group; + for (1 .. 10) { + select undef, undef, undef, 0.1; + $members = anchored_members($group, $anchor); + return if !defined $members; + return "alive" unless @$members; + } + kill "KILL", -$group; + for (1 .. 20) { + select undef, undef, undef, 0.1; + my $all_members = group_members($group); + return if !defined $all_members; + return "killed" unless @$all_members; + my $anchor_present = grep { $_ == $anchor } @$all_members; + return if !$anchor_present; + my @remaining = grep { $_ != $anchor } @$all_members; + return "killed" unless @remaining; + } + return; + } + sub shell_status { + my ($status) = @_; + return ($status & 127) ? 128 + ($status & 127) : $status >> 8; + } + sub finish_anchor { + my ($anchor, $finish_write) = @_; + my $written = syswrite $finish_write, "F"; + close $finish_write; + return 0 if !defined $written || $written != 1; + my $waited; + do { + $waited = waitpid $anchor, 0; + } while ($waited == -1 && $! == EINTR); + return $waited == $anchor; + } + sub reap_anchor { + my ($anchor, $finish_write) = @_; + close $finish_write; + my $waited; + do { + $waited = waitpid $anchor, 0; + } while ($waited == -1 && $! == EINTR); + return $waited == $anchor; + } + my $setup_failure = shift; + my $timeout = shift; + my $requested_status = 0; + local $SIG{ALRM} = sub { $requested_status ||= 124 }; + local $SIG{HUP} = sub { $requested_status ||= 129 }; + local $SIG{INT} = sub { $requested_status ||= 130 }; + local $SIG{QUIT} = sub { $requested_status ||= 131 }; + local $SIG{TERM} = sub { $requested_status ||= 143 }; + pipe my $ready_read, my $ready_write or die "ready pipe failed"; + pipe my $start_read, my $start_write or die "start pipe failed"; + pipe my $status_read, my $status_write or die "status pipe failed"; + pipe my $finish_read, my $finish_write or die "finish pipe failed"; + my $anchor = fork; + die "anchor fork failed" unless defined $anchor; + if (!$anchor) { + close $ready_read; + close $start_write; + close $status_read; + close $finish_write; + $SIG{HUP} = "IGNORE"; + $SIG{INT} = "IGNORE"; + $SIG{QUIT} = "IGNORE"; + $SIG{TERM} = "IGNORE"; + setpgrp 0, 0; + if (getpgrp(0) != $$) { + syswrite $ready_write, "E"; + exit $setup_failure; + } + syswrite $ready_write, "R"; + close $ready_write; + my $start = ""; + my $start_count = sysread $start_read, $start, 1; + close $start_read; + exit $setup_failure if !defined $start_count || $start_count != 1 || $start ne "S"; + my $command = fork; + exit $setup_failure unless defined $command; + if (!$command) { + close $status_write; + close $finish_read; + $SIG{HUP} = "DEFAULT"; + $SIG{INT} = "DEFAULT"; + $SIG{QUIT} = "DEFAULT"; + $SIG{TERM} = "DEFAULT"; + delete $ENV{FM_PROCESS_TREE_RESULT_FILE}; + exec @ARGV; + exit 127; + } + my $waited; + do { + $waited = waitpid $command, 0; + } while ($waited == -1 && $! == EINTR); + my $command_status = $waited == $command ? shell_status($?) : 127; + syswrite $status_write, "$command_status\n"; + close $status_write; + while (1) { + my $finish = ""; + my $finish_count = sysread $finish_read, $finish, 1; + exit 0 if defined $finish_count && $finish_count == 1 && $finish eq "F"; + select undef, undef, undef, 1; + } + } + close $ready_write; + close $start_read; + close $status_write; + close $finish_read; + my $ready = ""; + while (length $ready < 1) { + my $count = sysread $ready_read, $ready, 1; + next if !defined $count && $! == EINTR; + last if !defined $count || $count == 0; + } + close $ready_read; + if ($ready ne "R") { + close $start_write; + close $finish_write; + waitpid $anchor, 0; + record_cleanup("not-started"); + print STDERR "error: cannot establish bounded command process-group anchor\n"; + exit $setup_failure; + } + if (!install_guard($anchor)) { + close $start_write; + close $finish_write; + waitpid $anchor, 0; + record_cleanup("not-started"); + print STDERR "error: cannot establish bounded command process-group guard\n"; + exit $setup_failure; + } + my $started = 0; + if (!$requested_status) { + my $written = syswrite $start_write, "S"; + $started = 1 if defined $written && $written == 1; + } + close $start_write; + if (!$started) { + close $status_read; + close $finish_write; + waitpid $anchor, 0; + clear_guard($anchor); + record_cleanup("not-started"); + print STDERR "error: cannot start bounded command under its process-group anchor\n"; + exit $setup_failure; + } + alarm $timeout; + my $status_text = ""; + while (!$requested_status && $status_text !~ /\n/) { + my $chunk = ""; + my $count = sysread $status_read, $chunk, 64; + if (defined $count && $count > 0) { + $status_text .= $chunk; + next; + } + last if defined $count && $count == 0; + next if !defined $count && $! == EINTR; + last; + } + alarm 0; + my $command_status; + $command_status = 0 + $1 if $status_text =~ /^(\d+)\n/; + my $anchor_state = terminate_owned($anchor, $anchor); + if (!defined $anchor_state) { + close $status_read; + close $finish_write; + record_cleanup("unverified"); + my $guard = $ENV{FM_PROCESS_TREE_GUARD_FILE} || "the reported process group"; + print STDERR "error: bounded command process cleanup could not be verified for anchored group $anchor; ownership remains guarded by $guard. Inspect that group, terminate only its remaining processes, and retry.\n"; + exit(defined $command_status ? $command_status : ($requested_status || $setup_failure)); + } + close $status_read; + my $anchor_reaped = $anchor_state eq "alive" + ? finish_anchor($anchor, $finish_write) + : reap_anchor($anchor, $finish_write); + if (!$anchor_reaped) { + record_cleanup("unverified"); + print STDERR "error: bounded command process-group anchor $anchor could not be reaped; retain guarded resources and retry after it exits.\n"; + exit(defined $command_status ? $command_status : ($requested_status || $setup_failure)); + } + if (!clear_guard($anchor)) { + record_cleanup("unverified"); + print STDERR "error: bounded command process-group guard could not be cleared for anchored group $anchor; retain guarded resources and retry.\n"; + exit(defined $command_status ? $command_status : ($requested_status || $setup_failure)); + } + record_cleanup("verified"); + exit $requested_status if $requested_status; + exit(defined $command_status ? $command_status : $setup_failure); + ' "$FM_PROCESS_TREE_SETUP_FAILURE_STATUS" "$seconds" "$@" >"$stdout_file" 2>"$stderr_file"; then + status=0 + else + status=$? + fi + result=$(sed -n '1p' "$result_file" 2>/dev/null || true) + fm_process_tree_emit_snapshot "$stdout_file" + fm_process_tree_emit_snapshot "$stderr_file" >&2 + rm -f "$result_file" "$stdout_file" "$stderr_file" + case "$result" in + verified|unverified|not-started) cleanup=$result ;; + *) cleanup=unverified ;; + esac + FM_PROCESS_TREE_CLEANUP_STATUS=$cleanup + return "$status" +} + +fm_run_bounded_capture() { + local combine=0 output_name output_file output status + if [ "${1:-}" = "--combine-stderr" ]; then + combine=1 + shift + fi + output_name=$1 + shift + output_file=$(mktemp "${TMPDIR:-/tmp}/fm-process-tree-output.XXXXXX") || { + FM_PROCESS_TREE_CLEANUP_STATUS=not-started + return "$FM_PROCESS_TREE_SETUP_FAILURE_STATUS" + } + if [ "$combine" -eq 1 ]; then + if fm_run_bounded "$@" >"$output_file" 2>&1; then status=0; else status=$?; fi + else + if fm_run_bounded "$@" >"$output_file"; then status=0; else status=$?; fi + fi + output=$(cat "$output_file") + rm -f "$output_file" + printf -v "$output_name" '%s' "$output" + return "$status" +} + +fm_process_tree_cleanup_verified() { + [ "$FM_PROCESS_TREE_CLEANUP_STATUS" = verified ] +} diff --git a/bin/fm-session-start.sh b/bin/fm-session-start.sh index 3743d844683..f08daa9500d 100755 --- a/bin/fm-session-start.sh +++ b/bin/fm-session-start.sh @@ -48,7 +48,7 @@ # Why lock first: the old documented order (bootstrap, THEN lock) let a # SECOND concurrent session run bootstrap's mutating sweeps - fast-forwarding # secondmate homes, reconciling report-retention ownership, writing X-mode -# artifacts, fetching/fast-forwarding every project clone - before ever +# artifacts, fetching/fast-forwarding every covered checkout - before ever # discovering another session already holds the lock. Two sessions racing # those sweeps is exactly the hazard the lock exists to prevent, so locking # first closes the hole outright: only the session that actually wins the diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index d78eb85e70a..ee1602e0582 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -1,8 +1,11 @@ #!/usr/bin/env bash +# FM_ACCOUNT_DIRECTORY_CUTOVER: direct-observe-passwd-home-v2 # Spawn a direct report: a new crewmate in a treehouse worktree, an eligible -# pre-cutover Orca respawn, or a secondmate in its isolated firstmate home. +# pre-cutover Orca direct recovery with empirically verified provider authority, +# or a secondmate in its isolated firstmate home. # Usage: fm-spawn.sh <task-id> <project-dir> [--harness <name>|harness|launch-command] [--model <name>] [--effort <level>] [--backend <name>] [--account-pool <pool>] [--account-profile <profile>] [--no-account-routing] [--scout] # fm-spawn.sh <task-id> [<firstmate-home>] [--harness <name>|harness|launch-command] [--model <name>] [--effort <level>] [--backend <name>] [--account-pool <pool>] [--account-profile <profile>] [--no-account-routing] --secondmate +# fm-spawn.sh <task-id> --recover-direct-account # fm-spawn.sh <task-id> (--resume-account|--continue-account) [--harness <claude|codex>] [--account-pool <pool>] [--account-profile <profile>] # --harness <name> is the explicit per-spawn harness/profile adapter. The old # positional harness arg still works for back-compat. @@ -17,10 +20,10 @@ # fm_backend_detect, with cmux fallback details in docs/cmux-backend.md), # then tmux. # New-task spawn-capable backends are the reference tmux adapter and -# experimental herdr, zellij, and cmux. Orca is available only to respawn a -# pre-cutover task whose meta has no report_required marker; it owns both the -# task worktree and terminal, so an eligible Orca respawn does not run -# treehouse get. cmux is a session provider only, exactly like herdr/zellij, +# experimental herdr, zellij, and cmux. Orca's legacy respawn design owns both +# the task worktree and terminal, but currently fails closed before provider +# mutation because its lifecycle authority is unverified. cmux is a session +# provider only, exactly like herdr/zellij, # so it does. An auto-detected herdr or cmux spawn prints a loud stderr notice; # auto-detected tmux stays silent; zellij and orca are never auto-detected. # codex-app is not a known backend yet; docs/codex-app-backend.md owns that @@ -49,27 +52,30 @@ # respawn exactly like the harness axis, and explicit --model/--effort flags # still win over the file's tokens. # Account routing is independently default-off. Its precedence and off/observe/ -# enforce behavior are owned by fm-account-routing-lib.sh. --account-pool asks -# Agent Fleet to atomically select one concrete profile; --account-profile pins -# a concrete profile (and optionally validates it belongs to --account-pool). -# Either explicit account flag enforces routing for this spawn even when the -# global mode is off/observe. --no-account-routing is the emergency per-spawn -# opt-out and cannot be combined with either account flag. Enforced routing is -# supported only for claude/codex and fails closed; it never silently launches -# the default account. The resolved profile wraps the provider command before -# it is submitted to the selected backend, so every backend receives the same -# launch string. -# config/secondmate-account-pool is the primary's durable, non-inherited pool -# for secondmate AGENTS. An explicit account flag overrides it. A secondmate's -# own crewmates use inherited crew dispatch/routing policy, not this pool. -# --resume-account is an internal recovery path. It requires existing sticky -# account/profile/session metadata plus Agent Fleet's matching SessionStart -# mapping, reuses the recorded worktree/home, and executes `agent-fleet resume -# --task`; any missing or mismatched recovery truth blocks before pane creation. -# --continue-account is the provider-neutral recovery path. It verifies a dead -# endpoint and current repository state, builds a task-owned continuation packet, -# launches a fresh provider session through a new namespaced Agent Fleet attempt, -# and releases the predecessor only after the new SessionStart mapping is bound. +# enforce resolution is owned by fm-account-routing-lib.sh. Direct account- +# directory launch currently covers ship/scout crewmates only; secondmate +# integration is deferred and retains legacy Agent Fleet routing. +# For a NEW routed Claude or Codex ship/scout, fm-account-directory.sh discovers +# the current user's account homes, chooses one through its direct per-vendor +# usage contract, installs that profile's Herdr hook, and prefixes the provider +# command with CLAUDE_CONFIG_DIR or CODEX_HOME. +# Existing --account-pool and --account-profile inputs remain compatibility +# activation signals for new direct launches; their aliases do not constrain +# the direct usage choice. --no-account-routing remains the emergency per-spawn +# opt-out and cannot be combined with either account flag. Off launches retain +# their existing default-identity behavior. +# config/secondmate-account-pool remains the primary's durable, non-inherited +# Agent Fleet selection input for secondmate agents when routing is enabled. A +# secondmate's own crewmates use inherited crew dispatch/routing policy, not +# this setting. +# --resume-account and --continue-account are legacy recovery paths only for +# existing account_profile metadata. They retain the sealed Agent Fleet +# session/lease behavior needed to recover those already-managed generations; +# ship/scout launches never create that metadata. +# --recover-direct-account is the ship/scout account_home recovery path. It reloads kind, +# project, worktree, harness, backend, model, effort, mode, yolo, and report +# requirements from metadata, selects a fresh account directory, and creates +# only a replacement endpoint in the recorded worktree. # A --secondmate spawn also propagates the primary's declared inheritable config # into the secondmate home's config/, so the secondmate's OWN crewmates, # dispatch profiles, and backlog backend inherit the primary's settings @@ -77,10 +83,19 @@ # --scout records kind=scout in the task's meta (report deliverable, scratch worktree; # see AGENTS.md task lifecycle); --secondmate records kind=secondmate and launches in a # provisioned firstmate home; the default is kind=ship. -# Before a secondmate launch, the home is locally fast-forwarded to the primary -# default-branch commit when safe; skipped syncs warn and launch unchanged. -# Ship/scout spawns refuse to launch unless the resolved task path is a real -# git worktree root distinct from the primary project checkout. +# Before a secondmate launch, the home must fast-forward safely to the primary +# default-branch commit and independently match the live default tip. +# Any unproven freshness state refuses launch. +# Ship/scout spawns refresh the primary checkout before Treehouse acquisition, +# surface dirty pool entries, and durably lease one available worktree before +# creating the endpoint. They refuse to create that endpoint unless the leased +# path is a clean isolated worktree from the requested repository whose HEAD +# matches its live upstream or local default-branch tip. Dirty acquisitions +# remain under their durable lease for manual recovery. Other pre-commit +# failures close the prepared endpoint, restore prior task state, and return +# only a worktree whose repository identity, cleanliness, and expected detached +# tip are re-proven before and after owned hook cleanup, with the return held +# under the common checkout mutation lock. # Batch dispatch: pass one or more `id=repo` pairs instead of a single <id> <project>, e.g. # fm-spawn.sh fix-a-k3=projects/foo add-b-q7=projects/bar [--scout] # Each pair re-execs this script in single-task mode, so the single path stays the only @@ -121,7 +136,11 @@ STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" PROJECTS="${FM_PROJECTS_OVERRIDE:-$FM_HOME/projects}" CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" +CHECKOUT_STATE_BASE="${FM_CHECKOUT_REFRESH_STATE_BASE:-${XDG_STATE_HOME:-$HOME/.local/state}/firstmate/checkout-refresh}" SUB_HOME_MARKER=".fm-secondmate-home" +# shellcheck source=bin/fm-checkout-lock-lib.sh +. "$SCRIPT_DIR/fm-checkout-lock-lib.sh" +CHECKOUT_LOCK_ROOT=$(fm_checkout_lock_root "$CHECKOUT_STATE_BASE") # shellcheck source=bin/fm-ff-lib.sh . "$SCRIPT_DIR/fm-ff-lib.sh" # shellcheck source=bin/fm-config-inherit-lib.sh @@ -158,10 +177,164 @@ spawn_managed_endpoint_state() { # <backend> <target> <label> <kind> <secondmat fi } +git_repository_probe() ( + unset GIT_DIR GIT_WORK_TREE GIT_COMMON_DIR GIT_CEILING_DIRECTORIES + unset GIT_DISCOVERY_ACROSS_FILESYSTEM GIT_IMPLICIT_WORK_TREE GIT_PREFIX + unset GIT_SUPER_PREFIX GIT_INTERNAL_SUPER_PREFIX GIT_INDEX_FILE + unset GIT_OBJECT_DIRECTORY GIT_ALTERNATE_OBJECT_DIRECTORIES GIT_NAMESPACE + unset GIT_CONFIG_PARAMETERS GIT_CONFIG_COUNT GIT_CONFIG_SYSTEM GIT_CONFIG_GLOBAL + command git "$@" +) + +git_common_dir_real() { + local repo=$1 common + ( + cd "$repo" 2>/dev/null || exit 1 + common=$(git_repository_probe rev-parse --git-common-dir 2>/dev/null) || exit 1 + cd "$common" 2>/dev/null || exit 1 + pwd -P + ) +} + +git_worktree_dir_real() { + local repo=$1 git_dir + ( + cd "$repo" 2>/dev/null || exit 1 + git_dir=$(git_repository_probe rev-parse --git-dir 2>/dev/null) || exit 1 + cd "$git_dir" 2>/dev/null || exit 1 + pwd -P + ) +} + +git_directory_identity() { + # shellcheck disable=SC2016 # JavaScript source is intentionally single-quoted. + node -e ' +const fs = require("fs"); +const stat = fs.lstatSync(process.argv[1], { bigint: true }); +if (!stat.isDirectory() || stat.isSymbolicLink()) process.exit(1); +process.stdout.write(`${stat.dev}:${stat.ino}`); +' "$1" +} + +git_worktree_ref() { + git_repository_probe -C "$1" symbolic-ref -q HEAD 2>/dev/null +} + +git_worktree_head() { + git_repository_probe -C "$1" rev-parse --verify 'HEAD^{commit}' 2>/dev/null +} + +capture_worktree_git_physical_identity() { + local worktree=$1 + WORKTREE_GIT_DIR=$(git_worktree_dir_real "$worktree" 2>/dev/null) || return 1 + WORKTREE_GIT_DIR_IDENTITY=$(git_directory_identity "$WORKTREE_GIT_DIR" 2>/dev/null) || return 1 + [ -n "$WORKTREE_GIT_DIR" ] && [ -n "$WORKTREE_GIT_DIR_IDENTITY" ] +} + +capture_direct_launch_authoritative_state() { + local current_ref current_head expected_ref + current_ref=$(git_worktree_ref "$WT" 2>/dev/null || true) + current_head=$(git_worktree_head "$WT" 2>/dev/null) || return 1 + expected_ref="refs/heads/fm/$ID" + WORKTREE_GIT_REF=$expected_ref + WORKTREE_GIT_HEAD= + WORKTREE_GIT_SETUP_REF= + WORKTREE_GIT_SETUP_HEAD= + if [ "$current_ref" != "$expected_ref" ]; then + WORKTREE_GIT_SETUP_REF=$current_ref + WORKTREE_GIT_SETUP_HEAD=$current_head + fi +} + +validate_direct_recovery_physical_identity() { + local worktree_real worktree_literal current_git_dir current_git_dir_identity + worktree_real=$(cd "$WT" 2>/dev/null && pwd -P) || worktree_real= + worktree_literal=${WT%/} + if [ -z "$worktree_real" ] || [ "$worktree_literal" != "$worktree_real" ]; then + echo "error: recorded direct account recovery worktree '$WT' is redirected or non-canonical; refusing endpoint creation" >&2 + return 1 + fi + current_git_dir=$(git_worktree_dir_real "$WT" 2>/dev/null) || current_git_dir= + current_git_dir_identity=$(git_directory_identity "$current_git_dir" 2>/dev/null) || current_git_dir_identity= + if [ -z "$current_git_dir" ] || [ -z "$current_git_dir_identity" ] \ + || [ "$current_git_dir" != "$RECORDED_WORKTREE_GIT_DIR" ] \ + || [ "$current_git_dir_identity" != "$RECORDED_WORKTREE_GIT_DIR_IDENTITY" ]; then + echo "error: recorded direct account recovery worktree '$WT' no longer has its exact Git-dir identity; refusing endpoint creation" >&2 + return 1 + fi + WORKTREE_GIT_DIR=$current_git_dir + WORKTREE_GIT_DIR_IDENTITY=$current_git_dir_identity +} + +validate_direct_recovery_worktree_identity() { + local current_git_ref current_git_head + validate_direct_recovery_physical_identity || return 1 + current_git_ref=$(git_worktree_ref "$WT" 2>/dev/null || true) + current_git_head=$(git_worktree_head "$WT" 2>/dev/null) || current_git_head= + if [ -n "$RECORDED_WORKTREE_GIT_SETUP_HEAD" ]; then + if [ "$current_git_ref" = "$RECORDED_WORKTREE_GIT_REF" ]; then + WORKTREE_GIT_REF=$current_git_ref + WORKTREE_GIT_HEAD= + WORKTREE_GIT_SETUP_REF= + WORKTREE_GIT_SETUP_HEAD= + return 0 + fi + if [ "$current_git_ref" != "$RECORDED_WORKTREE_GIT_SETUP_REF" ] \ + || [ -z "$current_git_head" ] \ + || [ "$current_git_head" != "$RECORDED_WORKTREE_GIT_SETUP_HEAD" ]; then + echo "error: recorded direct account recovery worktree '$WT' changed branch identity; refusing endpoint creation" >&2 + return 1 + fi + WORKTREE_GIT_REF=$RECORDED_WORKTREE_GIT_REF + WORKTREE_GIT_HEAD= + WORKTREE_GIT_SETUP_REF=$RECORDED_WORKTREE_GIT_SETUP_REF + WORKTREE_GIT_SETUP_HEAD=$RECORDED_WORKTREE_GIT_SETUP_HEAD + return 0 + fi + if [ -n "$RECORDED_WORKTREE_GIT_REF" ]; then + if [ "$current_git_ref" != "$RECORDED_WORKTREE_GIT_REF" ]; then + echo "error: recorded direct account recovery worktree '$WT' changed branch identity; refusing endpoint creation" >&2 + return 1 + fi + WORKTREE_GIT_REF=$current_git_ref + WORKTREE_GIT_HEAD= + else + if [ -n "$current_git_ref" ] || [ -z "$current_git_head" ] \ + || [ "$current_git_head" != "$RECORDED_WORKTREE_GIT_HEAD" ]; then + echo "error: recorded direct account recovery worktree '$WT' changed detached HEAD identity; refusing endpoint creation" >&2 + return 1 + fi + WORKTREE_GIT_REF= + WORKTREE_GIT_HEAD=$current_git_head + fi + WORKTREE_GIT_SETUP_REF= + WORKTREE_GIT_SETUP_HEAD= +} + +validate_direct_launch_worktree_identity() { + local current_git_dir current_git_dir_identity + current_git_dir=$(git_worktree_dir_real "$WT" 2>/dev/null) || current_git_dir= + current_git_dir_identity=$(git_directory_identity "$current_git_dir" 2>/dev/null) || current_git_dir_identity= + if [ -z "$current_git_dir" ] || [ -z "$current_git_dir_identity" ] \ + || [ "$current_git_dir" != "$WORKTREE_GIT_DIR" ] \ + || [ "$current_git_dir_identity" != "$WORKTREE_GIT_DIR_IDENTITY" ]; then + echo "error: direct account worktree '$WT' changed exact Git-dir identity before metadata install" >&2 + return 1 + fi +} + +direct_worktree_return_confirmed() { + local project=$1 worktree=$2 listed + [ ! -e "$worktree" ] && [ ! -L "$worktree" ] && return 0 + listed=$(git_repository_probe -C "$project" -c core.quotePath=false worktree list --porcelain 2>/dev/null) || return 1 + ! printf '%s\n' "$listed" | grep -Fqx "worktree $worktree" +} + # Skip the watcher guard when re-exec'd for one pair of a batch (FM_SPAWN_NO_GUARD is # set by the batch loop below), so the guard runs once for the batch, not once per pair. [ -n "${FM_SPAWN_NO_GUARD:-}" ] || "$FM_ROOT/bin/fm-guard.sh" || true KIND=ship +KIND_SET=0 HARNESS_ARG= MODEL= EFFORT= @@ -171,6 +344,7 @@ ACCOUNT_PROFILE= NO_ACCOUNT_ROUTING=0 RESUME_ACCOUNT=0 CONTINUE_ACCOUNT=0 +DIRECT_ACCOUNT_RECOVERY=0 CONTINUATION_LAUNCH_DIR= CONTINUATION_PROMPT_FILE= CONTINUATION_PROMPT_DIR_ID= @@ -202,8 +376,8 @@ for a in "$@"; do continue fi case "$a" in - --scout) KIND=scout ;; - --secondmate) KIND=secondmate ;; + --scout) KIND=scout; KIND_SET=1 ;; + --secondmate) KIND=secondmate; KIND_SET=1 ;; --harness) want_value=harness ;; --harness=*) HARNESS_ARG=${a#--harness=}; HARNESS_SET=1 ;; --model) want_value=model ;; @@ -219,6 +393,7 @@ for a in "$@"; do --no-account-routing) NO_ACCOUNT_ROUTING=1 ;; --resume-account) RESUME_ACCOUNT=1 ;; --continue-account) CONTINUE_ACCOUNT=1 ;; + --recover-direct-account) DIRECT_ACCOUNT_RECOVERY=1 ;; *) POS+=("$a") ;; esac done @@ -236,6 +411,19 @@ fi [ "$RESUME_ACCOUNT" = 0 ] || [ "$NO_ACCOUNT_ROUTING" = 0 ] || { echo "error: --resume-account cannot disable account routing" >&2; exit 1; } [ "$CONTINUE_ACCOUNT" = 0 ] || [ "$NO_ACCOUNT_ROUTING" = 0 ] || { echo "error: --continue-account cannot disable account routing" >&2; exit 1; } [ "$RESUME_ACCOUNT" = 0 ] || [ "$CONTINUE_ACCOUNT" = 0 ] || { echo "error: --resume-account and --continue-account are mutually exclusive" >&2; exit 1; } +if [ $((RESUME_ACCOUNT + CONTINUE_ACCOUNT + DIRECT_ACCOUNT_RECOVERY)) -gt 1 ]; then + echo "error: --resume-account, --continue-account, and --recover-direct-account are mutually exclusive" >&2 + exit 1 +fi +if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + if [ "$KIND_SET" = 1 ] || [ "$HARNESS_SET" = 1 ] || [ "$MODEL_SET" = 1 ] \ + || [ "$EFFORT_SET" = 1 ] || [ "$BACKEND_SET" = 1 ] \ + || [ "$ACCOUNT_POOL_SET" = 1 ] || [ "$ACCOUNT_PROFILE_SET" = 1 ] \ + || [ "$NO_ACCOUNT_ROUTING" = 1 ]; then + echo "error: --recover-direct-account accepts only a task id; task context comes from metadata" >&2 + exit 1 + fi +fi [ -z "$ACCOUNT_POOL" ] || fm_account_valid_id "$ACCOUNT_POOL" || { echo "error: invalid --account-pool '$ACCOUNT_POOL'" >&2; exit 1; } [ -z "$ACCOUNT_PROFILE" ] || fm_account_valid_id "$ACCOUNT_PROFILE" || { echo "error: invalid --account-profile '$ACCOUNT_PROFILE'" >&2; exit 1; } case "$EFFORT" in @@ -244,10 +432,12 @@ case "$EFFORT" in esac RECOVERY_ACCOUNT=0 -[ "$RESUME_ACCOUNT" = 0 ] && [ "$CONTINUE_ACCOUNT" = 0 ] || RECOVERY_ACCOUNT=1 +[ "$RESUME_ACCOUNT" = 0 ] && [ "$CONTINUE_ACCOUNT" = 0 ] && [ "$DIRECT_ACCOUNT_RECOVERY" = 0 ] || RECOVERY_ACCOUNT=1 RESUME_META= LIFECYCLE_LOCK= LIFECYCLE_LOCK_OWNED=0 +SECONDMATE_HOME_LIFECYCLE_LOCK= +SECONDMATE_TARGET_HOME_LIFECYCLE_LOCK= LIFECYCLE_LOCK_INHERITED_PID= LIFECYCLE_LOCK_INHERITED_START= SPAWN_META_PRESENT=0 @@ -255,6 +445,15 @@ SPAWN_META_SNAPSHOT= SPAWN_PREFLIGHT_ID=${POS[0]:-} spawn_idpart=${SPAWN_PREFLIGHT_ID%%=*} SPAWN_PREFLIGHT_BATCH=0 + +release_secondmate_home_lifecycle_locks() { + [ -z "${SECONDMATE_TARGET_HOME_LIFECYCLE_LOCK:-}" ] \ + || fm_account_lifecycle_lock_release "$SECONDMATE_TARGET_HOME_LIFECYCLE_LOCK" >/dev/null 2>&1 || true + [ -z "${SECONDMATE_HOME_LIFECYCLE_LOCK:-}" ] \ + || fm_account_lifecycle_lock_release "$SECONDMATE_HOME_LIFECYCLE_LOCK" >/dev/null 2>&1 || true + SECONDMATE_TARGET_HOME_LIFECYCLE_LOCK= + SECONDMATE_HOME_LIFECYCLE_LOCK= +} if [ -n "$SPAWN_PREFLIGHT_ID" ] && [ "$SPAWN_PREFLIGHT_ID" != "$spawn_idpart" ] \ && case "$spawn_idpart" in */*) false ;; *) true ;; esac; then SPAWN_PREFLIGHT_BATCH=1 @@ -265,7 +464,7 @@ spawn_preflight_read_meta() { # <meta> } spawn_preflight_load_meta() { # <required:0|1> - local required=$1 + local required=$1 cleanup_count cleanup_value RESUME_META="$STATE/$SPAWN_PREFLIGHT_ID.meta" if [ -e "$STATE" ] || [ -L "$STATE" ]; then [ -d "$STATE" ] && [ ! -L "$STATE" ] || { @@ -285,6 +484,16 @@ spawn_preflight_load_meta() { # <required:0|1> return 1 } SPAWN_META_PRESENT=1 + cleanup_count=$(printf '%s\n' "$SPAWN_META_SNAPSHOT" | grep -c '^orca_cleanup_pending=' || true) + if [ "$cleanup_count" -ne 0 ]; then + cleanup_value=$(spawn_preflight_meta_value orca_cleanup_pending) + if [ "$cleanup_count" -eq 1 ] && [ "$cleanup_value" = 1 ]; then + echo "error: Orca cleanup is pending for $SPAWN_PREFLIGHT_ID; run fm-teardown.sh $SPAWN_PREFLIGHT_ID before retrying spawn" >&2 + else + echo "error: invalid Orca cleanup metadata for $SPAWN_PREFLIGHT_ID; refusing spawn" >&2 + fi + return 1 + fi elif [ "$required" = 1 ]; then echo "error: no metadata for managed recovery at $RESUME_META" >&2 return 1 @@ -295,25 +504,212 @@ spawn_preflight_meta_value() { # <key> printf '%s\n' "$SPAWN_META_SNAPSHOT" | sed -n "s/^$1=//p" | tail -1 } +spawn_preflight_kind_value() { + local parsed count value + parsed=$(printf '%s\n' "$SPAWN_META_SNAPSHOT" | awk ' + index($0, "kind=") == 1 { + count++ + value = substr($0, 6) + } + END { + printf "%d\t%s\n", count + 0, value + } + ') || return 1 + count=${parsed%%$'\t'*} + value=${parsed#*$'\t'} + case "$count:$value" in + 0:) printf '%s\n' ship ;; + 1:ship|1:scout|1:secondmate) printf '%s\n' "$value" ;; + 1:*) + echo "error: managed recovery metadata has invalid kind '$value' for $SPAWN_PREFLIGHT_ID" >&2 + return 1 + ;; + *) + echo "error: managed recovery metadata has duplicate kind records for $SPAWN_PREFLIGHT_ID" >&2 + return 1 + ;; + esac +} + spawn_refuse_report_required_orca() { - local report_required=1 - if [ "$SPAWN_META_PRESENT" = 1 ]; then - if printf '%s\n' "$SPAWN_META_SNAPSHOT" | grep -q '^report_required='; then - [ "$(spawn_preflight_meta_value report_required)" = 1 ] || report_required=0 + local report_count + if [ "$SPAWN_META_PRESENT" != 1 ]; then + echo "error: backend=orca cannot host new report-required tasks: Orca has no reliable endpoint-absence proof, so report-gated teardown could never complete; spawn report-required work on tmux, herdr, zellij, or cmux" >&2 + return 1 + fi + report_count=$(printf '%s\n' "$SPAWN_META_SNAPSHOT" | grep -c '^report_required=' || true) + [ "$report_count" -eq 0 ] || { + if [ "$report_count" -eq 1 ] && [ "$(spawn_preflight_meta_value report_required)" = 1 ]; then + echo "error: backend=orca cannot host new report-required tasks: Orca has no reliable endpoint-absence proof, so report-gated teardown could never complete; spawn report-required work on tmux, herdr, zellij, or cmux" >&2 else - report_required=0 + echo "error: invalid report_required metadata for $SPAWN_PREFLIGHT_ID; legacy Orca recovery requires the marker to be absent" >&2 fi + return 1 + } +} + +reconcile_failed_direct_recovery() { + local task=$1 meta="$STATE/$1.meta" lock marker generation backend target tmux_session_target tab kind home tasktmp + local backup_name backup_token backup backup_snapshot artifacts_name artifacts_token artifacts endpoint_state tmp current_backup_snapshot + lock=$(fm_account_meta_lock_acquire "$STATE" "$task") || return 1 + marker=$(fm_account_meta_value "$meta" direct_recovery_cleanup) + if [ "$marker" != pending ]; then + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: invalid retained direct recovery state for $task" >&2 + return 1 fi - if [ "$report_required" = 1 ]; then - echo "error: backend=orca cannot host new report-required tasks: Orca has no reliable endpoint-absence proof, so report-gated teardown could never complete; spawn report-required work on tmux, herdr, zellij, or cmux" >&2 + generation=$(fm_account_meta_value "$meta" generation_id) + backend=$(fm_backend_of_meta "$meta") + target=$(fm_backend_target_of_meta "$meta") + tmux_session_target=$(fm_account_meta_value "$meta" tmux_session_target) + tab=$(fm_account_meta_value "$meta" zellij_tab_id) + kind=$(fm_account_meta_value "$meta" kind) + [ -n "$kind" ] || kind=ship + home=$(fm_account_meta_value "$meta" home) + tasktmp=$(fm_account_meta_value "$meta" tasktmp) + backup_name=$(fm_account_meta_value "$meta" direct_recovery_backup) + artifacts_name=$(fm_account_meta_value "$meta" direct_recovery_artifacts) + case "$backup_name" in + ".$task.meta.rollback."*) ;; + *) + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: unsafe retained direct recovery backup for $task" >&2 + return 1 + ;; + esac + backup_token=${backup_name#".$task.meta.rollback."} + if ! fm_account_valid_id "$backup_token"; then + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: unsafe retained direct recovery backup for $task" >&2 + return 1 + fi + case "$artifacts_name" in + ".$task.artifacts.rollback."*) ;; + *) + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: unsafe retained direct recovery artifacts for $task" >&2 + return 1 + ;; + esac + artifacts_token=${artifacts_name#".$task.artifacts.rollback."} + if ! fm_account_valid_id "$artifacts_token"; then + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: unsafe retained direct recovery artifacts for $task" >&2 return 1 fi + backup="$STATE/$backup_name" + artifacts="$STATE/$artifacts_name" + backup_snapshot=$(spawn_preflight_read_meta "$backup") || { + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: retained direct recovery backup is missing or unsafe for $task" >&2 + return 1 + } + if [ ! -d "$artifacts" ] || [ -L "$artifacts" ]; then + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: retained direct recovery artifacts are missing or unsafe for $task" >&2 + return 1 + fi + if [ "$tasktmp" != "/tmp/fm-$task" ] || [ -z "$generation" ] || [ -z "$target" ]; then + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: retained direct recovery metadata is incomplete for $task" >&2 + return 1 + fi + fm_account_meta_lock_release "$lock" || return 1 + lock= + + spawn_managed_endpoint_kill "$backend" "$target" "$tab" "fm-$task" "$kind" "$home" "$tmux_session_target" 2>/dev/null || true + endpoint_state=$(spawn_managed_endpoint_state "$backend" "$target" "fm-$task" "$kind" "$home" "$tmux_session_target" 2>/dev/null) + case "$endpoint_state" in + absent) ;; + present) + echo "error: retained direct recovery endpoint is still alive for $task" >&2 + return 1 + ;; + *) + echo "error: retained direct recovery endpoint state is unknown for $task" >&2 + return 1 + ;; + esac + + lock=$(fm_account_meta_lock_acquire "$STATE" "$task") || return 1 + current_backup_snapshot=$(spawn_preflight_read_meta "$backup" 2>/dev/null) || current_backup_snapshot= + if [ ! -f "$meta" ] \ + || [ "$(fm_account_meta_value "$meta" direct_recovery_cleanup)" != pending ] \ + || [ "$(fm_account_meta_value "$meta" generation_id)" != "$generation" ] \ + || [ "$(fm_backend_of_meta "$meta")" != "$backend" ] \ + || [ "$(fm_backend_target_of_meta "$meta")" != "$target" ] \ + || [ "$(fm_account_meta_value "$meta" direct_recovery_backup)" != "$backup_name" ] \ + || [ "$(fm_account_meta_value "$meta" direct_recovery_artifacts)" != "$artifacts_name" ] \ + || [ "$current_backup_snapshot" != "$backup_snapshot" ] \ + || [ ! -d "$artifacts" ] || [ -L "$artifacts" ]; then + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: retained direct recovery generation changed before cleanup for $task" >&2 + return 1 + fi + if ! fm_account_restore_artifacts "$STATE" "$task" "$artifacts_name" "$tasktmp" 1; then + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: retained direct recovery artifacts could not be restored for $task" >&2 + return 1 + fi + tmp=$(mktemp "$STATE/.$task.meta.direct-recovery-restore.XXXXXX") || { + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + return 1 + } + if ! printf '%s\n' "$backup_snapshot" > "$tmp" \ + || ! fm_account_meta_merge_extensions "$meta" "$tmp" \ + || ! fm_account_safe_file_destination "$meta" \ + || ! mv "$tmp" "$meta"; then + rm -f "$tmp" + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || true + echo "error: retained direct recovery metadata could not be restored for $task" >&2 + return 1 + fi + rm -f "$backup" || echo "warning: retained direct recovery backup remains for $task" >&2 + rm -rf "$artifacts" || echo "warning: retained direct recovery artifacts remain for $task" >&2 + fm_account_meta_lock_release "$lock" || return 1 + SPAWN_META_SNAPSHOT=$(spawn_preflight_read_meta "$meta") || return 1 + SPAWN_META_PRESENT=1 + echo "fm-spawn: cleaned retained direct recovery endpoint for $task" >&2 +} + +spawn_refuse_existing_orca_provider_identity() { + [ "$SPAWN_META_PRESENT" != 1 ] || [ "$(spawn_preflight_meta_value backend)" != orca ] || { + echo "error: existing Orca provider identity for $SPAWN_PREFLIGHT_ID must be cleared by teardown before respawn" >&2 + return 1 + } +} + +spawn_refuse_unsupported_secondmate_backend() { + [ "$KIND" != secondmate ] || [ "$BACKEND" != orca ] || { + echo "error: backend=orca does not support --secondmate spawns yet" >&2 + return 1 + } + [ "$KIND" != secondmate ] || [ "$BACKEND" != cmux ] || { + echo "error: backend=cmux does not support --secondmate spawns yet" >&2 + return 1 + } } if [ "$RECOVERY_ACCOUNT" = 1 ]; then [ "${#POS[@]}" -ge 1 ] || { echo "error: account recovery requires a task id" >&2; exit 1; } case "$SPAWN_PREFLIGHT_ID" in *=*) echo "error: account recovery does not support batch syntax" >&2; exit 1 ;; esac + if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ] && [ "${#POS[@]}" -ne 1 ]; then + echo "error: --recover-direct-account accepts exactly one task id" >&2 + exit 1 + fi spawn_preflight_load_meta 1 || exit 1 + recorded_kind=$(spawn_preflight_kind_value) || exit 1 + if [ "$KIND" != ship ] && [ "$KIND" != "$recorded_kind" ]; then + echo "error: account recovery kind '$KIND' does not match recorded kind '$recorded_kind'" >&2 + exit 1 + fi + KIND=$recorded_kind + if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + case "$KIND" in + ship|scout) ;; + *) echo "error: --recover-direct-account supports only recorded ship or scout tasks" >&2; exit 1 ;; + esac + fi recorded_backend=$(spawn_preflight_meta_value backend) [ -n "$recorded_backend" ] || recorded_backend=tmux if [ "$BACKEND_SET" = 1 ] && [ "$BACKEND_ARG" != "$recorded_backend" ]; then @@ -331,22 +727,16 @@ else fi fm_backend_validate_spawn "$BACKEND" || exit 1 fm_backend_source "$BACKEND" || exit 1 -if [ "$BACKEND" = orca ] && [ "$KIND" = secondmate ]; then - echo "error: backend=orca does not support --secondmate spawns yet" >&2 - exit 1 -fi -if [ "$BACKEND" = cmux ] && [ "$KIND" = secondmate ]; then - echo "error: backend=cmux does not support --secondmate spawns yet" >&2 - exit 1 -fi +spawn_refuse_unsupported_secondmate_backend || exit 1 if [ "$BACKEND" = orca ] && [ "$RECOVERY_ACCOUNT" = 0 ] && [ "$SPAWN_PREFLIGHT_BATCH" = 0 ] \ && { [ -e "$STATE/$SPAWN_PREFLIGHT_ID.meta" ] || [ -L "$STATE/$SPAWN_PREFLIGHT_ID.meta" ]; }; then spawn_preflight_load_meta 0 || exit 1 fi if [ "$BACKEND" = orca ] && [ "$SPAWN_PREFLIGHT_BATCH" = 0 ] && [ "$SPAWN_META_PRESENT" = 1 ]; then spawn_refuse_report_required_orca || exit 1 + [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ] || spawn_refuse_existing_orca_provider_identity || exit 1 fi -if [ "$BACKEND" = orca ] && [ "$RECOVERY_ACCOUNT" = 0 ]; then +if [ "$BACKEND" = orca ] && { [ "$RECOVERY_ACCOUNT" = 0 ] || [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; }; then fm_backend_orca_runtime_check || exit 1 fi if [ "$RECOVERY_ACCOUNT" = 0 ] && [ "$SPAWN_PREFLIGHT_BATCH" = 0 ] && [ "$SPAWN_META_PRESENT" = 0 ]; then @@ -354,6 +744,25 @@ if [ "$RECOVERY_ACCOUNT" = 0 ] && [ "$SPAWN_PREFLIGHT_BATCH" = 0 ] && [ "$SPAWN_ fi if [ "$BACKEND" = orca ] && [ "$SPAWN_PREFLIGHT_BATCH" = 0 ]; then spawn_refuse_report_required_orca || exit 1 + [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ] || spawn_refuse_existing_orca_provider_identity || exit 1 +fi + +if [ "$SPAWN_PREFLIGHT_BATCH" = 0 ]; then + SECONDMATE_HOME_LIFECYCLE_LOCK=$(fm_secondmate_home_lifecycle_lock_acquire "$CHECKOUT_LOCK_ROOT" "$FM_HOME") || exit 1 + trap 'release_secondmate_home_lifecycle_locks' EXIT + fm_checkout_trusted_dir "$FM_HOME" >/dev/null || { + echo "error: active firstmate home was removed or redirected while spawn waited for lifecycle ownership" >&2 + exit 1 + } + if [ -e "$FM_HOME/$SUB_HOME_MARKER" ] || [ -L "$FM_HOME/$SUB_HOME_MARKER" ]; then + [ -f "$FM_HOME/$SUB_HOME_MARKER" ] && [ ! -L "$FM_HOME/$SUB_HOME_MARKER" ] || { + echo "error: unsafe secondmate home marker at $FM_HOME/$SUB_HOME_MARKER" >&2 + exit 1 + } + elif [ -f "$FM_HOME/data/charter.md" ]; then + echo "error: secondmate home changed while spawn waited for lifecycle ownership" >&2 + exit 1 + fi fi if [ -n "${FM_ACCOUNT_LIFECYCLE_LOCK_HELD:-}" ]; then @@ -402,7 +811,7 @@ if [ -n "${FM_ACCOUNT_LIFECYCLE_LOCK_HELD:-}" ]; then exit 1 fi LIFECYCLE_LOCK_OWNED=1 - trap '[ "${LIFECYCLE_LOCK_OWNED:-0}" != 1 ] || [ -z "${LIFECYCLE_LOCK:-}" ] || fm_account_lifecycle_lock_release "$LIFECYCLE_LOCK" >/dev/null 2>&1 || true' EXIT + trap '[ "${LIFECYCLE_LOCK_OWNED:-0}" != 1 ] || [ -z "${LIFECYCLE_LOCK:-}" ] || fm_account_lifecycle_lock_release "$LIFECYCLE_LOCK" >/dev/null 2>&1 || true; release_secondmate_home_lifecycle_locks' EXIT # The handoff replaces the lock inode while live ownership prevents reclaim until this child releases the replacement. if ! fm_account_lifecycle_lock_owned "$LIFECYCLE_LOCK"; then echo "error: inherited account lifecycle lock ownership handoff failed for $inherited_lock_id" >&2 @@ -416,7 +825,7 @@ if [ "$RECOVERY_ACCOUNT" = 1 ]; then LIFECYCLE_LOCK=$(fm_account_lifecycle_lock_acquire "$STATE" "${POS[0]}") || exit 1 LIFECYCLE_LOCK_OWNED=1 fi - trap '[ "${LIFECYCLE_LOCK_OWNED:-0}" != 1 ] || [ -z "${LIFECYCLE_LOCK:-}" ] || fm_account_lifecycle_lock_release "$LIFECYCLE_LOCK" >/dev/null 2>&1 || true' EXIT + trap '[ "${LIFECYCLE_LOCK_OWNED:-0}" != 1 ] || [ -z "${LIFECYCLE_LOCK:-}" ] || fm_account_lifecycle_lock_release "$LIFECYCLE_LOCK" >/dev/null 2>&1 || true; release_secondmate_home_lifecycle_locks' EXIT current_spawn_meta=$(spawn_preflight_read_meta "$RESUME_META") || { echo "error: unsafe metadata for managed recovery at $RESUME_META" >&2 exit 1 @@ -425,7 +834,24 @@ if [ "$RECOVERY_ACCOUNT" = 1 ]; then # serializes recovery, refresh it so a waiter validates the committed # replacement generation instead of rejecting that generation as stale. SPAWN_META_SNAPSHOT=$current_spawn_meta + current_recorded_kind=$(spawn_preflight_kind_value) || exit 1 + [ "$current_recorded_kind" = "$KIND" ] || { + echo "error: managed recovery kind changed before launch for ${POS[0]}" >&2 + exit 1 + } rm -rf "$STATE/.${POS[0]}.account-native-launch" "$STATE/.${POS[0]}.account-native-ready" "$STATE/.${POS[0]}.account-native-go" || exit 1 + direct_recovery_cleanup=$(fm_account_meta_value "$RESUME_META" direct_recovery_cleanup) + if [ -n "$direct_recovery_cleanup" ]; then + [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ] || { + echo "error: retained direct recovery state exists for ${POS[0]}; use --recover-direct-account" >&2 + exit 1 + } + [ "$direct_recovery_cleanup" = pending ] || { + echo "error: invalid retained direct recovery state for ${POS[0]}" >&2 + exit 1 + } + reconcile_failed_direct_recovery "${POS[0]}" || exit 1 + fi if [ "$(fm_account_meta_value "$RESUME_META" account_rollback_cleanup)" = pending ]; then rollback_id=${POS[0]} rollback_account_task=$(fm_account_meta_value "$RESUME_META" account_task) @@ -524,13 +950,18 @@ fi # recorded in meta only when it is NOT tmux (fm-teardown.sh and fm-watch.sh's # window_backend/fm_backend_of_meta already treat an absent backend= as tmux), # so the default path's meta stays byte-identical. -if [ "$BACKEND" = orca ] && [ "$RECOVERY_ACCOUNT" = 1 ]; then +if [ "$BACKEND" = orca ] && [ "$RECOVERY_ACCOUNT" = 1 ] && [ "$DIRECT_ACCOUNT_RECOVERY" = 0 ]; then echo "error: managed account recovery is not implemented for backend=orca" >&2 exit 1 fi ORCA_ABORT_CLEANUP=0 ORCA_WORKTREE_ID= ORCA_TERMINAL= +ORCA_TERMINAL_PROOF= +ORCA_REPO_ID= +ORCA_EXPECTED_TASK= +ORCA_PROVIDER_TASK= +ID= ACCOUNT_LEASE_CREATED=0 FM_ACCOUNT_MUTATION_ACQUIRED=0 ACCOUNT_SPAWN_COMMITTED=0 @@ -548,6 +979,8 @@ ACCOUNT_PREDECESSOR_SESSION= CONTINUATION_PACKET= ENDPOINT_CREATED=0 WORKTREE_CREATED=0 +WORKTREE_RETAIN_ON_ABORT=0 +WORKTREE_EXPECTED_TIP= META_INSTALLED=0 META_BACKUP= EXISTING_ARTIFACT_BACKUP= @@ -557,6 +990,16 @@ ACCOUNT_NATIVE_LAUNCH_SCRIPT= ACCOUNT_NATIVE_LAUNCH_READY= ACCOUNT_NATIVE_LAUNCH_GO= ACCOUNT_NATIVE_LAUNCH_DIR= +DIRECT_ACCOUNT_ROUTING=0 +DIRECT_ACCOUNT_HOME= +DIRECT_ACCOUNT_RESPAWN=0 +DIRECT_ACCOUNT_PREPARE_DEFERRED=0 +WORKTREE_GIT_DIR= +WORKTREE_GIT_DIR_IDENTITY= +WORKTREE_GIT_REF= +WORKTREE_GIT_HEAD= +WORKTREE_GIT_SETUP_REF= +WORKTREE_GIT_SETUP_HEAD= CONFIG_INHERIT_REPORT_TMP= ORIGINAL_STATUS_PRESENT=-1 ORIGINAL_TURN_ENDED_PRESENT=-1 @@ -590,18 +1033,131 @@ discard_existing_artifact_backup() { parse_orca_worktree_result() { local raw=$1 rest ORCA_WORKTREE_ID=${raw%%$'\t'*} - if [ "$raw" = "$ORCA_WORKTREE_ID" ]; then - WT= - ORCA_TERMINAL= - return 1 - fi + [ "$raw" != "$ORCA_WORKTREE_ID" ] || return 1 rest=${raw#*$'\t'} WT=${rest%%$'\t'*} - if [ "$rest" != "$WT" ]; then - ORCA_TERMINAL=${rest#*$'\t'} - else - ORCA_TERMINAL= - fi + [ "$rest" != "$WT" ] || return 1 + rest=${rest#*$'\t'} + ORCA_TERMINAL=${rest%%$'\t'*} + [ "$rest" != "$ORCA_TERMINAL" ] || return 1 + rest=${rest#*$'\t'} + ORCA_TERMINAL_PROOF=${rest%%$'\t'*} + [ "$rest" != "$ORCA_TERMINAL_PROOF" ] || return 1 + rest=${rest#*$'\t'} + ORCA_REPO_ID=$rest + case "$ORCA_REPO_ID" in *$'\t'*) return 1 ;; esac + ORCA_PROVIDER_TASK= +} + +persist_orca_cleanup_quarantine() { + local phase=$1 + mkdir -p "$STATE" || return 1 + [ -d "$STATE" ] && [ ! -L "$STATE" ] || return 1 + python3 - "$STATE" "$STATE/$ID.meta" "$phase" \ + "${W:-fm-$ID}" "${WT:-}" "${PROJ_ABS:-}" "${HARNESS:-}" "${KIND:-ship}" \ + "${MODE:-no-mistakes}" "${YOLO:-off}" "${TASK_TMP:-}" "${MODEL:-default}" \ + "${EFFORT:-default}" "${ORCA_WORKTREE_ID:-}" "${ORCA_TERMINAL:-}" \ + "${ORCA_TERMINAL_PROOF:-unproven}" "${ORCA_REPO_ID:-}" "fm-$ID" \ + "repo-path:${PROJ_ABS:-}" <<'PY' +import os +import stat +import sys +import tempfile + +(state, metadata, phase, window, worktree, project, harness, kind, mode, yolo, + tasktmp, model, effort, worktree_id, terminal, proof, repo_id, expected_task, + provider_scope) = sys.argv[1:] +values = [ + ("window", window), + ("worktree", worktree), + ("project", project), + ("harness", harness), + ("kind", kind), + ("mode", mode), + ("yolo", yolo), + ("tasktmp", tasktmp), + ("model", model), + ("effort", effort), + ("backend", "orca"), +] +if worktree_id: + values.append(("orca_worktree_id", worktree_id)) +if terminal: + values.append(("terminal", terminal)) +values.extend([ + ("orca_cleanup_pending", "1"), + ("orca_cleanup_phase", phase), + ("orca_terminal_proof", proof), + ("orca_repo_id", repo_id), + ("orca_expected_task", expected_task), + ("orca_discovery_label", expected_task), + ("orca_provider_scope", provider_scope), +]) +owned = { + "window", "worktree", "project", "harness", "kind", "mode", "yolo", + "tasktmp", "model", "effort", "backend", "orca_worktree_id", "terminal", + "orca_cleanup_pending", "orca_cleanup_phase", "orca_terminal_proof", + "orca_repo_id", "orca_expected_task", "orca_discovery_label", + "orca_provider_scope", +} +for key, value in values: + if any(character in value for character in "\0\r\n"): + raise SystemExit(1) +state_metadata = os.lstat(state) +if not stat.S_ISDIR(state_metadata.st_mode) or stat.S_ISLNK(state_metadata.st_mode): + raise SystemExit(1) +preserved = [] +retained = {} +try: + metadata_state = os.lstat(metadata) +except FileNotFoundError: + metadata_state = None +if metadata_state is not None: + if not stat.S_ISREG(metadata_state.st_mode) or stat.S_ISLNK(metadata_state.st_mode): + raise SystemExit(1) + with open(metadata, encoding="utf-8") as stream: + for line in stream: + key = line.split("=", 1)[0] + if key not in owned: + preserved.append(line) + elif "=" in line: + retained[key] = line.rstrip("\n").split("=", 1)[1] +values = [ + (key, retained.get(key, value) if value == "" else value) + for key, value in values +] +for key, value in values: + if any(character in value for character in "\0\r\n"): + raise SystemExit(1) +descriptor, temporary = tempfile.mkstemp(prefix=".orca-quarantine.", dir=state) +try: + with os.fdopen(descriptor, "w", encoding="utf-8") as output: + output.writelines(preserved) + for key, value in values: + output.write(f"{key}={value}\n") + output.flush() + os.fsync(output.fileno()) + try: + destination = os.lstat(metadata) + except FileNotFoundError: + destination = None + if destination is not None and ( + not stat.S_ISREG(destination.st_mode) or stat.S_ISLNK(destination.st_mode) + ): + raise OSError("unsafe quarantine destination") + os.replace(temporary, metadata) + directory = os.open(state, os.O_RDONLY) + try: + os.fsync(directory) + finally: + os.close(directory) +except BaseException: + try: + os.unlink(temporary) + except FileNotFoundError: + pass + raise +PY } persist_failed_account_rollback() { @@ -680,6 +1236,154 @@ persist_failed_account_rollback() { META_INSTALLED=1 } +persist_failed_direct_recovery() { + local meta="$STATE/$ID.meta" tmp backup_name artifacts_name retained_window retained_tmux_session account_home + [ -n "$META_BACKUP" ] && [ -f "$META_BACKUP" ] || return 1 + [ -n "$EXISTING_ARTIFACT_BACKUP" ] && [ -d "$EXISTING_ARTIFACT_BACKUP" ] || return 1 + backup_name=${META_BACKUP##*/} + artifacts_name=${EXISTING_ARTIFACT_BACKUP##*/} + retained_window=${META_WINDOW:-${T:-${W:-fm-$ID}}} + retained_tmux_session= + if [ "${BACKEND:-tmux}" = tmux ]; then + retained_tmux_session=${META_WINDOW:-${SES:-firstmate}:${W:-fm-$ID}} + retained_window=$retained_tmux_session + fi + account_home=${DIRECT_ACCOUNT_HOME:-${RECORDED_ACCOUNT_HOME:-}} + tmp=$(mktemp "$STATE/.$ID.meta.direct-recovery-pending.XXXXXX") || return 1 + { + echo "window=$retained_window" + echo "worktree=${WT:-${RECORDED_WORKTREE:-}}" + echo "worktree_git_dir=${WORKTREE_GIT_DIR:-${RECORDED_WORKTREE_GIT_DIR:-}}" + echo "worktree_git_dir_identity=${WORKTREE_GIT_DIR_IDENTITY:-${RECORDED_WORKTREE_GIT_DIR_IDENTITY:-}}" + [ -z "${WORKTREE_GIT_REF:-${RECORDED_WORKTREE_GIT_REF:-}}" ] || echo "worktree_git_ref=${WORKTREE_GIT_REF:-${RECORDED_WORKTREE_GIT_REF:-}}" + [ -z "${WORKTREE_GIT_HEAD:-${RECORDED_WORKTREE_GIT_HEAD:-}}" ] || echo "worktree_git_head=${WORKTREE_GIT_HEAD:-${RECORDED_WORKTREE_GIT_HEAD:-}}" + [ -z "${WORKTREE_GIT_SETUP_REF:-}" ] || echo "worktree_git_setup_ref=$WORKTREE_GIT_SETUP_REF" + [ -z "${WORKTREE_GIT_SETUP_HEAD:-}" ] || echo "worktree_git_setup_head=$WORKTREE_GIT_SETUP_HEAD" + echo "project=${PROJ_ABS:-${RECORDED_PROJECT:-}}" + echo "harness=${HARNESS:-${RECORDED_HARNESS:-}}" + echo "kind=${KIND:-${RECORDED_KIND:-ship}}" + echo "mode=${MODE:-${RECORDED_MODE:-no-mistakes}}" + echo "yolo=${YOLO:-${RECORDED_YOLO:-off}}" + echo "tasktmp=${TASK_TMP:-${RECORDED_TASKTMP:-/tmp/fm-$ID}}" + echo "model=${RECORDED_MODEL:-${MODEL:-default}}" + echo "effort=${RECORDED_EFFORT:-${EFFORT:-default}}" + echo "generation_id=${RECORDED_GENERATION:-${SPAWN_GENERATION_ID:-}}" + [ "${RECORDED_REPORT_REQUIRED_SET:-0}" != 1 ] || echo "report_required=${RECORDED_REPORT_REQUIRED:-}" + [ -z "$account_home" ] || echo "account_home=$account_home" + [ "${BACKEND:-tmux}" = tmux ] || echo "backend=$BACKEND" + [ "${BACKEND:-tmux}" != tmux ] || [ -z "${WID:-}" ] || echo "tmux_window_id=$WID" + [ "${BACKEND:-tmux}" != tmux ] || echo "tmux_session_target=$retained_tmux_session" + [ "${BACKEND:-tmux}" != herdr ] || { + echo "herdr_session=${HERDR_SES:-}" + echo "herdr_workspace_id=${HERDR_WORKSPACE_ID:-}" + echo "herdr_tab_id=${HERDR_TAB_ID:-}" + echo "herdr_pane_id=${HERDR_PANE_ID:-}" + } + [ "${BACKEND:-tmux}" != zellij ] || { + echo "zellij_session=${ZELLIJ_SES:-}" + echo "zellij_tab_id=${ZELLIJ_TAB_ID:-}" + echo "zellij_pane_id=${ZELLIJ_PANE_ID:-}" + } + [ "${BACKEND:-tmux}" != orca ] || { + echo "orca_worktree_id=${ORCA_WORKTREE_ID:-}" + echo "terminal=${ORCA_TERMINAL:-${T:-}}" + } + [ "${BACKEND:-tmux}" != cmux ] || { + echo "cmux_workspace_id=${CMUX_WORKSPACE_ID:-}" + echo "cmux_surface_id=${CMUX_SURFACE_ID:-}" + } + echo "direct_recovery_cleanup=pending" + echo "direct_recovery_backup=$backup_name" + echo "direct_recovery_artifacts=$artifacts_name" + } > "$tmp" || { rm -f "$tmp"; return 1; } + fm_account_meta_merge_extensions "$meta" "$tmp" || { rm -f "$tmp"; return 1; } + fm_account_safe_file_destination "$meta" || { rm -f "$tmp"; return 1; } + mv "$tmp" "$meta" || { rm -f "$tmp"; return 1; } + META_INSTALLED=1 +} + +persist_failed_direct_spawn() { + local meta="$STATE/$ID.meta" tmp retained_window retained_tmux_session retained_mode retained_yolo backup_name artifacts_name preserve_extensions=0 + retained_window=${META_WINDOW:-${T:-${W:-fm-$ID}}} + retained_tmux_session= + retained_mode=${MODE:-} + retained_yolo=${YOLO:-off} + [ -n "$retained_mode" ] || retained_mode=no-mistakes + if [ "${BACKEND:-tmux}" = tmux ]; then + retained_tmux_session=${META_WINDOW:-${SES:-firstmate}:${W:-fm-$ID}} + retained_window=$retained_tmux_session + fi + if [ -n "${WT:-}" ] && [ -d "$WT" ]; then + if [ -z "${WORKTREE_GIT_DIR:-}" ] || [ -z "${WORKTREE_GIT_DIR_IDENTITY:-}" ]; then + capture_worktree_git_physical_identity "$WT" >/dev/null 2>&1 || true + fi + if [ -z "${WORKTREE_GIT_REF:-}" ] && [ -z "${WORKTREE_GIT_HEAD:-}" ]; then + capture_direct_launch_authoritative_state >/dev/null 2>&1 || true + fi + fi + tmp=$(mktemp "$STATE/.$ID.meta.direct-spawn-pending.XXXXXX") || return 1 + if [ "$META_INSTALLED" = 1 ] && [ -f "$meta" ] \ + && [ "$(fm_account_meta_value "$meta" generation_id)" = "$SPAWN_GENERATION_ID" ]; then + awk '!/^direct_spawn_cleanup=/ && !/^direct_spawn_backup=/ && !/^direct_spawn_artifacts=/ && !/^rollback_pending=/' "$meta" > "$tmp" || { rm -f "$tmp"; return 1; } + else + [ ! -f "$meta" ] || preserve_extensions=1 + { + echo "window=$retained_window" + echo "worktree=${WT:-}" + [ -z "${WORKTREE_GIT_DIR:-}" ] || echo "worktree_git_dir=$WORKTREE_GIT_DIR" + [ -z "${WORKTREE_GIT_DIR_IDENTITY:-}" ] || echo "worktree_git_dir_identity=$WORKTREE_GIT_DIR_IDENTITY" + [ -z "${WORKTREE_GIT_REF:-}" ] || echo "worktree_git_ref=$WORKTREE_GIT_REF" + [ -z "${WORKTREE_GIT_HEAD:-}" ] || echo "worktree_git_head=$WORKTREE_GIT_HEAD" + [ -z "${WORKTREE_GIT_SETUP_REF:-}" ] || echo "worktree_git_setup_ref=$WORKTREE_GIT_SETUP_REF" + [ -z "${WORKTREE_GIT_SETUP_HEAD:-}" ] || echo "worktree_git_setup_head=$WORKTREE_GIT_SETUP_HEAD" + echo "project=${PROJ_ABS:-}" + echo "harness=${HARNESS:-}" + echo "kind=${KIND:-ship}" + echo "mode=$retained_mode" + echo "yolo=$retained_yolo" + echo "tasktmp=${TASK_TMP:-/tmp/fm-$ID}" + echo "model=${MODEL:-default}" + echo "effort=${EFFORT:-default}" + echo "generation_id=${SPAWN_GENERATION_ID:-}" + echo "report_required=1" + [ -z "${DIRECT_ACCOUNT_HOME:-}" ] || echo "account_home=$DIRECT_ACCOUNT_HOME" + [ "${BACKEND:-tmux}" = tmux ] || echo "backend=$BACKEND" + [ "${BACKEND:-tmux}" != tmux ] || [ -z "${WID:-}" ] || echo "tmux_window_id=$WID" + [ "${BACKEND:-tmux}" != tmux ] || echo "tmux_session_target=$retained_tmux_session" + [ "${BACKEND:-tmux}" != herdr ] || { + echo "herdr_session=${HERDR_SES:-}" + echo "herdr_workspace_id=${HERDR_WORKSPACE_ID:-}" + echo "herdr_tab_id=${HERDR_TAB_ID:-}" + echo "herdr_pane_id=${HERDR_PANE_ID:-}" + } + [ "${BACKEND:-tmux}" != zellij ] || { + echo "zellij_session=${ZELLIJ_SES:-}" + echo "zellij_tab_id=${ZELLIJ_TAB_ID:-}" + echo "zellij_pane_id=${ZELLIJ_PANE_ID:-}" + } + [ "${BACKEND:-tmux}" != cmux ] || { + echo "cmux_workspace_id=${CMUX_WORKSPACE_ID:-}" + echo "cmux_surface_id=${CMUX_SURFACE_ID:-}" + } + } > "$tmp" || { rm -f "$tmp"; return 1; } + fi + if [ "$preserve_extensions" = 1 ]; then + fm_account_meta_merge_extensions "$meta" "$tmp" || { rm -f "$tmp"; return 1; } + fi + printf 'direct_spawn_cleanup=pending\nrollback_pending=1\n' >> "$tmp" || { rm -f "$tmp"; return 1; } + if [ -n "$META_BACKUP" ] && [ -f "$META_BACKUP" ]; then + backup_name=${META_BACKUP##*/} + printf 'direct_spawn_backup=%s\n' "$backup_name" >> "$tmp" || { rm -f "$tmp"; return 1; } + fi + if [ -n "$EXISTING_ARTIFACT_BACKUP" ] && [ -d "$EXISTING_ARTIFACT_BACKUP" ]; then + artifacts_name=${EXISTING_ARTIFACT_BACKUP##*/} + printf 'direct_spawn_artifacts=%s\n' "$artifacts_name" >> "$tmp" || { rm -f "$tmp"; return 1; } + fi + fm_account_safe_file_destination "$meta" || { rm -f "$tmp"; return 1; } + mv "$tmp" "$meta" || { rm -f "$tmp"; return 1; } + META_INSTALLED=1 +} + clear_account_rollback_markers() { local meta="$STATE/$ID.meta" tmp tmp=$(mktemp "$STATE/.$ID.meta.rollback-commit.XXXXXX") || return 1 @@ -710,8 +1414,95 @@ cleanup_continuation_launch_transport() { CONTINUATION_PROMPT_CONTENT_ID= } +spawn_return_created_worktree() { + local return_output return_status + [ "$WORKTREE_CREATED" = 1 ] || return 0 + [ "${BACKEND:-tmux}" != orca ] || return 0 + [ -n "${WT:-}" ] && [ -d "$WT" ] || return 0 + if [ "$WORKTREE_RETAIN_ON_ABORT" = 1 ]; then + echo "warning: retained unsafe acquired worktree $WT for manual recovery" >&2 + return 1 + fi + if [ -z "$WORKTREE_EXPECTED_TIP" ] \ + || ! "$SCRIPT_DIR/fm-checkout-refresh.sh" verify-returnable "$WT" "$PROJ_ABS" "$WORKTREE_EXPECTED_TIP"; then + echo "warning: retained acquired worktree $WT because repository identity and its expected detached tip could not be re-proven" >&2 + return 1 + fi + rm -f "$WT/.claude/settings.local.json" "$WT/.opencode/plugins/fm-turn-end.js" "$WT/.fm-grok-turnend" + if ! "$SCRIPT_DIR/fm-checkout-refresh.sh" verify-returnable "$WT" "$PROJ_ABS" "$WORKTREE_EXPECTED_TIP"; then + echo "warning: retained acquired worktree $WT because post-cleanup repository safety could not be re-proven" >&2 + return 1 + fi + if return_output=$(fm_checkout_treehouse_return "$WT" "$CHECKOUT_LOCK_ROOT" "$PROJ_ABS" 2>&1); then + [ -z "$return_output" ] || printf '%s\n' "$return_output" >&2 + return 0 + else + return_status=$? + fi + [ -z "$return_output" ] || printf '%s\n' "$return_output" >&2 + case "$return_status:$return_output" in + "$FM_CHECKOUT_PROCESS_CLEANUP_FAILURE_STATUS:"*"Treehouse return process cleanup could not be verified"*) + echo "warning: retained rollback worktree $WT because Treehouse return process cleanup is unverified; inspect the reported anchored process group, terminate only its remaining processes, and retry cleanup" >&2 + ;; + esac + return "$return_status" +} + +spawn_restore_unmanaged_state_locked() { + local meta="$STATE/$ID.meta" current_generation artifact_backup_name + [ "${ACCOUNT_EFFECTIVE_MODE:-off}" != enforce ] || return 0 + if [ -n "$EXISTING_ARTIFACT_BACKUP" ]; then + artifact_backup_name=${EXISTING_ARTIFACT_BACKUP##*/} + fm_account_restore_artifacts "$STATE" "$ID" "$artifact_backup_name" "/tmp/fm-$ID" 1 || return 1 + fi + if [ -n "$META_BACKUP" ]; then + [ -f "$META_BACKUP" ] && [ -f "$meta" ] || return 1 + current_generation=$(fm_meta_get "$meta" generation_id) + if [ "$META_INSTALLED" = 1 ]; then + [ "$current_generation" = "$SPAWN_GENERATION_ID" ] || return 1 + elif [ "${BACKEND:-tmux}" = orca ] \ + && [ "$(fm_meta_get "$meta" orca_cleanup_pending)" = 1 ] \ + && [ "$(fm_meta_get "$meta" orca_expected_task)" = "fm-$ID" ] \ + && { [ -z "$(fm_meta_get "$meta" orca_worktree_id)" ] \ + || [ "$(fm_meta_get "$meta" orca_worktree_id)" = "${ORCA_WORKTREE_ID:-}" ]; } \ + && { [ -z "$(fm_meta_get "$meta" terminal)" ] \ + || [ "$(fm_meta_get "$meta" terminal)" = "${ORCA_TERMINAL:-}" ]; }; then + : + else + cmp -s "$meta" "$META_BACKUP" || return 1 + fi + fm_account_meta_merge_extensions "$meta" "$META_BACKUP" || return 1 + fm_account_safe_file_destination "$meta" || return 1 + mv "$META_BACKUP" "$meta" || return 1 + META_BACKUP= + elif [ "$META_INSTALLED" = 1 ] && [ -e "$meta" ]; then + current_generation=$(fm_meta_get "$meta" generation_id) + [ "$current_generation" = "$SPAWN_GENERATION_ID" ] || return 1 + rm -f "$meta" || return 1 + fi + discard_existing_artifact_backup +} + +spawn_restore_unmanaged_state() { + local lock=${1:-} lock_owned=0 status + [ -n "${ID:-}" ] || return 0 + if [ -z "$lock" ]; then + lock=$(fm_account_meta_lock_acquire "$STATE" "$ID") || return 1 + lock_owned=1 + fi + if spawn_restore_unmanaged_state_locked; then + status=0 + else + status=$? + fi + if [ "$lock_owned" = 1 ]; then + fm_account_meta_lock_release "$lock" >/dev/null 2>&1 || status=1 + fi + return "$status" +} + spawn_abort_cleanup() { - local status=$? endpoint_state endpoint_gone=1 account_clean=1 worktree_clean=1 rollback_lock='' rollback_tmp restored_existing_meta=0 artifact_backup_name orca_meta_tmp release_status + local status=$? endpoint_state endpoint_gone=1 account_clean=1 state_clean=1 worktree_clean=1 rollback_lock='' rollback_tmp restored_existing_meta=0 artifact_backup_name release_status orca_cleanup_failed=0 orca_boundary_token= trap - EXIT # This is an EXIT trap whose job is to attempt every independent cleanup # action and then return the original spawn status. The parent script runs @@ -727,56 +1518,177 @@ spawn_abort_cleanup() { if [ "$ORCA_ABORT_CLEANUP" = 1 ]; then ORCA_ABORT_CLEANUP=0 if [ -n "${ORCA_TERMINAL:-}" ]; then - fm_backend_kill orca "$ORCA_TERMINAL" 2>/dev/null || true - fi - if [ -n "${ORCA_WORKTREE_ID:-}" ]; then - if ! fm_backend_remove_worktree orca "$ORCA_WORKTREE_ID" 2>/dev/null; then - mkdir -p "$STATE" 2>/dev/null || true - if [ -d "$STATE" ] && [ ! -L "$STATE" ]; then - orca_meta_tmp=$(mktemp "$STATE/.${ID:-unknown}.meta.orca-cleanup.XXXXXX" 2>/dev/null) || orca_meta_tmp= - fi - if [ -n "${orca_meta_tmp:-}" ]; then - { - echo "window=${W:-fm-${ID:-unknown}}" - echo "worktree=${WT:-}" - echo "project=${PROJ_ABS:-}" - echo "harness=${HARNESS:-}" - echo "kind=${KIND:-ship}" - echo "mode=${MODE:-no-mistakes}" - echo "yolo=${YOLO:-off}" - echo "tasktmp=${TASK_TMP:-}" - echo "model=${MODEL:-default}" - echo "effort=${EFFORT:-default}" - echo "backend=orca" - echo "orca_worktree_id=$ORCA_WORKTREE_ID" - [ -z "${ORCA_TERMINAL:-}" ] || echo "terminal=$ORCA_TERMINAL" - } > "$orca_meta_tmp" 2>/dev/null || true - if fm_account_safe_file_destination "$STATE/${ID:-unknown}.meta"; then - mv "$orca_meta_tmp" "$STATE/${ID:-unknown}.meta" 2>/dev/null || true - fi - [ ! -e "$orca_meta_tmp" ] || rm -f "$orca_meta_tmp" - fi + if [ -n "${ORCA_WORKTREE_ID:-}" ]; then + case "$(fm_backend_orca_terminal_state "$ORCA_TERMINAL" "$ORCA_WORKTREE_ID" "fm-${ID:-unknown}")" in + present|absent) ;; + *) orca_cleanup_failed=1 ;; + esac + else + orca_cleanup_failed=1 fi + elif [ -n "${ORCA_WORKTREE_ID:-}" ]; then + case "$(fm_backend_orca_worktree_terminal_state "$ORCA_WORKTREE_ID" "fm-${ID:-unknown}")" in + present|absent) ;; + *) orca_cleanup_failed=1 ;; + esac + else + orca_cleanup_failed=1 + fi + if [ -n "${ORCA_WORKTREE_ID:-}" ] && [ "$orca_cleanup_failed" = 0 ]; then + validate_orca_abort_worktree_identity || orca_cleanup_failed=1 + fi + if [ -n "${ORCA_WORKTREE_ID:-}" ] && [ "$orca_cleanup_failed" = 0 ]; then + fm_backend_orca_quiesce_worktree_terminals "$ORCA_WORKTREE_ID" "fm-${ID:-unknown}" "${ORCA_TERMINAL:-}" || orca_cleanup_failed=1 + fi + if [ -n "${ORCA_WORKTREE_ID:-}" ] && [ "$orca_cleanup_failed" = 0 ]; then + validate_orca_abort_worktree_identity || orca_cleanup_failed=1 + fi + if [ -n "${ORCA_WORKTREE_ID:-}" ] && [ "$orca_cleanup_failed" = 0 ]; then + orca_boundary_token=$(fm_checkout_tree_boundary_token "$WT") || orca_cleanup_failed=1 + fi + if [ -n "${ORCA_WORKTREE_ID:-}" ] && [ "$orca_cleanup_failed" = 0 ]; then + fm_backend_remove_worktree_bound \ + orca "$ORCA_WORKTREE_ID" "$WT" "$orca_boundary_token" || orca_cleanup_failed=1 + fi + if [ "$orca_cleanup_failed" = 0 ]; then + spawn_restore_unmanaged_state "$rollback_lock" || { + state_clean=0 + echo "warning: failed to restore prior task state after Orca abort cleanup for ${ID:-unknown}" >&2 + } + fi + if [ "$orca_cleanup_failed" = 1 ]; then + endpoint_gone=0 + worktree_clean=0 + echo "warning: retaining Orca cleanup metadata for ${ID:-unknown} because endpoint absence or worktree removal is unproven" >&2 + persist_orca_cleanup_quarantine spawn-abort || \ + echo "warning: failed to update the pre-armed Orca cleanup quarantine for ${ID:-unknown}" >&2 fi fi - if [ "$ACCOUNT_SPAWN_COMMITTED" != 1 ] && [ "${ACCOUNT_EFFECTIVE_MODE:-off}" = enforce ] && [ "$ENDPOINT_CREATED" = 1 ] && [ -n "${T:-}" ]; then + if [ "$ACCOUNT_SPAWN_COMMITTED" != 1 ] \ + && [ "${BACKEND:-tmux}" != orca ] \ + && [ "$ENDPOINT_CREATED" = 1 ] && [ -n "${T:-}" ]; then spawn_managed_endpoint_kill "${BACKEND:-tmux}" "$T" "${ZELLIJ_TAB_ID:-}" "fm-${ID:-unknown}" "${KIND:-ship}" "${PROJ_ABS:-}" "${META_WINDOW:-}" 2>/dev/null || true endpoint_state=$(spawn_managed_endpoint_state "${BACKEND:-tmux}" "$T" "fm-${ID:-unknown}" "${KIND:-ship}" "${PROJ_ABS:-}" "${META_WINDOW:-}" 2>/dev/null) case "$endpoint_state" in absent) ;; present) endpoint_gone=0 - echo "warning: retaining managed state for ${ID:-unknown} because the failed spawn endpoint is still alive" >&2 + echo "warning: retaining failed spawn resources for ${ID:-unknown} because the endpoint is still alive" >&2 ;; *) endpoint_gone=0 - echo "warning: retaining managed state for ${ID:-unknown} because the failed spawn endpoint state is unknown" >&2 + echo "warning: retaining failed spawn resources for ${ID:-unknown} because the endpoint state is unknown" >&2 ;; esac fi - [ -z "${ACCOUNT_NATIVE_LAUNCH_DIR:-}" ] || rm -rf "$ACCOUNT_NATIVE_LAUNCH_DIR" - cleanup_continuation_launch_transport - [ -z "${CONFIG_INHERIT_REPORT_TMP:-}" ] || rm -f "$CONFIG_INHERIT_REPORT_TMP" + if [ "$ACCOUNT_SPAWN_COMMITTED" != 1 ] && [ "$endpoint_gone" = 1 ] \ + && [ "${ACCOUNT_EFFECTIVE_MODE:-off}" != enforce ] \ + && [ "${DIRECT_ACCOUNT_ROUTING:-0}" != 1 ]; then + spawn_restore_unmanaged_state "$rollback_lock" || state_clean=0 + if [ "$state_clean" = 1 ]; then + spawn_return_created_worktree || worktree_clean=0 + else + worktree_clean=0 + echo "warning: retained failed spawn resources for ${ID:-unknown} because prior task state could not be restored" >&2 + [ "$WORKTREE_RETAIN_ON_ABORT" != 1 ] || spawn_return_created_worktree + fi + [ "$worktree_clean" = 1 ] || echo "warning: failed to return rollback worktree for ${ID:-unknown}" >&2 + fi + [ -z "${ACCOUNT_NATIVE_LAUNCH_DIR:-}" ] || rm -rf "$ACCOUNT_NATIVE_LAUNCH_DIR" + cleanup_continuation_launch_transport + [ -z "${CONFIG_INHERIT_REPORT_TMP:-}" ] || rm -f "$CONFIG_INHERIT_REPORT_TMP" + if [ "$ACCOUNT_SPAWN_COMMITTED" != 1 ] && [ "${DIRECT_ACCOUNT_RECOVERY:-0}" = 1 ] && [ "$endpoint_gone" = 0 ]; then + if [ -z "$rollback_lock" ]; then + rollback_lock=$(fm_account_meta_lock_acquire "$STATE" "${ID:-unknown}" 2>/dev/null) || rollback_lock= + fi + if [ -n "$rollback_lock" ]; then + persist_failed_direct_recovery || echo "warning: failed to persist retained direct recovery state for ${ID:-unknown}" >&2 + else + echo "warning: failed to acquire metadata lock while preserving retained direct recovery for ${ID:-unknown}" >&2 + fi + fi + if [ "$ACCOUNT_SPAWN_COMMITTED" != 1 ] && [ "${DIRECT_ACCOUNT_ROUTING:-0}" = 1 ] \ + && [ "${DIRECT_ACCOUNT_RECOVERY:-0}" != 1 ] && [ "$endpoint_gone" = 0 ]; then + if [ -z "$rollback_lock" ]; then + rollback_lock=$(fm_account_meta_lock_acquire "$STATE" "${ID:-unknown}" 2>/dev/null) || rollback_lock= + fi + if [ -n "$rollback_lock" ]; then + persist_failed_direct_spawn || echo "warning: failed to persist retained direct spawn state for ${ID:-unknown}" >&2 + else + echo "warning: failed to acquire metadata lock while preserving retained direct spawn for ${ID:-unknown}" >&2 + fi + fi + if [ "$ACCOUNT_SPAWN_COMMITTED" != 1 ] && [ "${DIRECT_ACCOUNT_RECOVERY:-0}" = 1 ] && [ "$endpoint_gone" = 1 ]; then + if [ -z "$rollback_lock" ]; then + rollback_lock=$(fm_account_meta_lock_acquire "$STATE" "${ID:-unknown}" 2>/dev/null) || rollback_lock= + fi + if [ -n "$rollback_lock" ]; then + artifact_backup_name=${EXISTING_ARTIFACT_BACKUP##*/} + if fm_account_restore_artifacts "$STATE" "$ID" "$artifact_backup_name" "${TASK_TMP:-/tmp/fm-$ID}" 1; then + if [ "$META_INSTALLED" = 1 ] && [ -n "$META_BACKUP" ] && [ -f "$META_BACKUP" ]; then + if fm_account_meta_merge_extensions "$STATE/$ID.meta" "$META_BACKUP" \ + && fm_account_safe_file_destination "$STATE/$ID.meta" \ + && mv "$META_BACKUP" "$STATE/$ID.meta"; then + META_BACKUP= + else + echo "warning: failed to restore direct recovery metadata for ${ID:-unknown}" >&2 + fi + elif [ -n "$META_BACKUP" ]; then + rm -f "$META_BACKUP" + META_BACKUP= + fi + [ -z "$EXISTING_ARTIFACT_BACKUP" ] || rm -rf "$EXISTING_ARTIFACT_BACKUP" + EXISTING_ARTIFACT_BACKUP= + else + echo "warning: failed to restore direct recovery artifacts for ${ID:-unknown}" >&2 + fi + else + echo "warning: failed to acquire metadata lock while restoring direct recovery for ${ID:-unknown}" >&2 + fi + fi + if [ "$ACCOUNT_SPAWN_COMMITTED" != 1 ] && [ "${DIRECT_ACCOUNT_ROUTING:-0}" = 1 ] \ + && [ "${DIRECT_ACCOUNT_RECOVERY:-0}" != 1 ] && [ "$endpoint_gone" = 1 ]; then + spawn_return_created_worktree || worktree_clean=0 + [ "$worktree_clean" = 1 ] \ + || echo "warning: failed to return direct spawn worktree for ${ID:-unknown}; retaining cleanup metadata" >&2 + if [ -z "$rollback_lock" ]; then + rollback_lock=$(fm_account_meta_lock_acquire "$STATE" "${ID:-unknown}" 2>/dev/null) || rollback_lock= + fi + if [ -n "$rollback_lock" ] && [ "$worktree_clean" = 1 ]; then + if [ -n "$META_BACKUP" ] && [ -f "$META_BACKUP" ]; then + artifact_backup_name=${EXISTING_ARTIFACT_BACKUP##*/} + if fm_account_restore_artifacts "$STATE" "$ID" "$artifact_backup_name" "${TASK_TMP:-/tmp/fm-$ID}" 1 \ + && fm_account_meta_merge_extensions "$STATE/$ID.meta" "$META_BACKUP" \ + && fm_account_safe_file_destination "$STATE/$ID.meta" \ + && mv "$META_BACKUP" "$STATE/$ID.meta"; then + META_BACKUP= + discard_existing_artifact_backup + restored_existing_meta=1 + else + worktree_clean=0 + echo "warning: failed to restore prior task state after direct spawn rollback for ${ID:-unknown}" >&2 + fi + else + if [ "$META_INSTALLED" = 1 ] \ + && [ "$(fm_meta_get "$STATE/$ID.meta" generation_id)" = "${SPAWN_GENERATION_ID:-}" ]; then + rm -f "$STATE/$ID.meta" || worktree_clean=0 + fi + [ "$ORIGINAL_STATUS_PRESENT" != 0 ] || rm -f "$STATE/$ID.status" + [ "$ORIGINAL_TURN_ENDED_PRESENT" != 0 ] || rm -f "$STATE/$ID.turn-ended" + [ "$ORIGINAL_CHECK_PRESENT" != 0 ] || rm -f "$STATE/$ID.check.sh" + [ "$ORIGINAL_PI_EXT_PRESENT" != 0 ] || rm -f "$STATE/$ID.pi-ext.ts" + [ "$ORIGINAL_GROK_TOKEN_PRESENT" != 0 ] || rm -f "$STATE/$ID.grok-turnend-token" + [ "$ORIGINAL_TASK_TMP_PRESENT" != 0 ] || { [ -z "${TASK_TMP:-}" ] || rm -rf "$TASK_TMP"; } + fi + fi + if [ "$worktree_clean" != 1 ]; then + if [ -n "$rollback_lock" ]; then + persist_failed_direct_spawn || echo "warning: failed to persist direct spawn cleanup state for ${ID:-unknown}" >&2 + else + echo "warning: failed to acquire metadata lock while preserving direct spawn cleanup for ${ID:-unknown}" >&2 + fi + fi + fi if [ "$ACCOUNT_SPAWN_COMMITTED" != 1 ] && [ "${ACCOUNT_EFFECTIVE_MODE:-off}" = enforce ] && [ "$endpoint_gone" = 1 ]; then if [ "$ACCOUNT_LEASE_CREATED" = 1 ] || fm_account_mutation_owned; then release_status=0 @@ -791,14 +1703,9 @@ spawn_abort_cleanup() { fm_account_lineage_append "$DATA" "$ID" rolled-back "$ACCOUNT_ATTEMPT" "$ACCOUNT_TASK" "$HARNESS" "$ACCOUNT_POOL" "$ACCOUNT_PROFILE" pending "$ACCOUNT_PREDECESSOR_TASK" >/dev/null 2>&1 || true fi fi - if [ "$account_clean" = 1 ] && { [ "$WORKTREE_CREATED" = 1 ] || [ "$ORCA_ABORT_CLEANUP" = 1 ]; }; then - if [ "${BACKEND:-tmux}" = orca ]; then - [ -z "${ORCA_WORKTREE_ID:-}" ] || fm_backend_remove_worktree orca "$ORCA_WORKTREE_ID" 2>/dev/null || worktree_clean=0 - elif [ -n "${WT:-}" ] && [ -d "$WT" ]; then - rm -f "$WT/.claude/settings.local.json" "$WT/.opencode/plugins/fm-turn-end.js" "$WT/.fm-grok-turnend" - ( cd "$PROJ_ABS" && treehouse return --force "$WT" ) >/dev/null 2>&1 || worktree_clean=0 - fi - [ "$worktree_clean" = 1 ] || echo "warning: failed to return rollback worktree for ${ID:-unknown}; retaining unmanaged cleanup metadata" >&2 + if [ "$account_clean" = 1 ]; then + spawn_return_created_worktree || worktree_clean=0 + [ "$worktree_clean" = 1 ] || echo "warning: failed to return rollback worktree for ${ID:-unknown}" >&2 fi if [ -z "$rollback_lock" ]; then if rollback_lock=$(fm_account_meta_lock_acquire "$STATE" "${ID:-unknown}"); then @@ -873,6 +1780,7 @@ spawn_abort_cleanup() { [ -z "$META_BACKUP" ] || [ -f "$META_BACKUP" ] || META_BACKUP= [ -z "$EXISTING_ARTIFACT_BACKUP" ] || [ -d "$EXISTING_ARTIFACT_BACKUP" ] || EXISTING_ARTIFACT_BACKUP= [ "${LIFECYCLE_LOCK_OWNED:-0}" != 1 ] || [ -z "${LIFECYCLE_LOCK:-}" ] || fm_account_lifecycle_lock_release "$LIFECYCLE_LOCK" >/dev/null 2>&1 || true + release_secondmate_home_lifecycle_locks LIFECYCLE_LOCK= LIFECYCLE_LOCK_OWNED=0 return "$status" @@ -920,12 +1828,68 @@ if [ "${#POS[@]}" -gt 0 ] && [ "${POS[0]}" != "$idpart" ] && case "$idpart" in * if FM_SPAWN_NO_GUARD=1 "$FM_ROOT/bin/fm-spawn.sh" "${pair%%=*}" "${pair#*=}" ${shared_args[@]+"${shared_args[@]}"}; then :; else echo "batch: FAILED to spawn ${pair%%=*} (${pair#*=})" >&2; rc=1; fi fi done + trap - EXIT exit "$rc" fi ID=${POS[0]} PROJ= ARG3= FIRSTMATE_HOME= +SECONDMATE_PROJECTS= + +if [ "$KIND" = secondmate ]; then + case "${POS[1]:-}" in + ''|claude|codex|opencode|pi|grok) + ARG3=${POS[1]:-} + ;; + *' '*) + if [ "${#POS[@]}" -gt 2 ] || [ -d "${POS[1]}" ]; then + FIRSTMATE_HOME=${POS[1]} + ARG3=${POS[2]:-} + else + ARG3=${POS[1]} + fi + ;; + *) + FIRSTMATE_HOME=${POS[1]} + ARG3=${POS[2]:-} + ;; + esac + if [ -z "$FIRSTMATE_HOME" ] && [ "$SPAWN_META_PRESENT" = 1 ]; then + FIRSTMATE_HOME=$(spawn_preflight_meta_value home) + fi + REGISTERED_SECONDMATE_HOME=$(fm_secondmate_registry_query "$DATA/secondmates.md" query "$ID" home) || { + echo "error: secondmate registry is malformed, missing, or does not uniquely register $ID" >&2 + exit 1 + } + SECONDMATE_PROJECTS=$(fm_secondmate_registry_query "$DATA/secondmates.md" query "$ID" projects) || { + echo "error: secondmate project registration is unprovable for $ID" >&2 + exit 1 + } + REGISTERED_SECONDMATE_HOME=$(fm_checkout_trusted_dir "$REGISTERED_SECONDMATE_HOME") || { + echo "error: registered secondmate home is unavailable or redirected for $ID" >&2 + exit 1 + } + if [ -n "$FIRSTMATE_HOME" ]; then + FIRSTMATE_HOME=$(fm_checkout_trusted_dir "$FIRSTMATE_HOME") || { + echo "error: requested secondmate home is unavailable or redirected for $ID" >&2 + exit 1 + } + [ "$FIRSTMATE_HOME" = "$REGISTERED_SECONDMATE_HOME" ] || { + echo "error: requested secondmate home does not match the exact registration for $ID" >&2 + exit 1 + } + else + FIRSTMATE_HOME=$REGISTERED_SECONDMATE_HOME + fi + ACTIVE_HOME_CANONICAL=$(fm_checkout_trusted_dir "$FM_HOME") || exit 1 + if [ "$FIRSTMATE_HOME" != "$ACTIVE_HOME_CANONICAL" ]; then + SECONDMATE_TARGET_HOME_LIFECYCLE_LOCK=$(fm_secondmate_home_lifecycle_lock_acquire "$CHECKOUT_LOCK_ROOT" "$FIRSTMATE_HOME") || exit 1 + fi +else + PROJ=${POS[1]:-} + ARG3=${POS[2]:-} +fi if [ -z "$LIFECYCLE_LOCK" ]; then LIFECYCLE_LOCK=$(fm_account_lifecycle_lock_acquire "$STATE" "$ID") || exit 1 @@ -954,29 +1918,6 @@ if [ -e "$STATE/$ID.pi-ext.ts" ] || [ -L "$STATE/$ID.pi-ext.ts" ]; then ORIGINAL if [ -e "$STATE/$ID.grok-turnend-token" ] || [ -L "$STATE/$ID.grok-turnend-token" ]; then ORIGINAL_GROK_TOKEN_PRESENT=1; else ORIGINAL_GROK_TOKEN_PRESENT=0; fi if [ -e "/tmp/fm-$ID" ] || [ -L "/tmp/fm-$ID" ]; then ORIGINAL_TASK_TMP_PRESENT=1; else ORIGINAL_TASK_TMP_PRESENT=0; fi -if [ "$KIND" = secondmate ]; then - case "${POS[1]:-}" in - ''|claude|codex|opencode|pi|grok) - ARG3=${POS[1]:-} - ;; - *' '*) - if [ "${#POS[@]}" -gt 2 ] || [ -d "${POS[1]}" ]; then - FIRSTMATE_HOME=${POS[1]} - ARG3=${POS[2]:-} - else - ARG3=${POS[1]} - fi - ;; - *) - FIRSTMATE_HOME=${POS[1]} - ARG3=${POS[2]:-} - ;; - esac -else - PROJ=${POS[1]:-} - ARG3=${POS[2]:-} -fi - if [ "$RECOVERY_ACCOUNT" = 1 ]; then RECORDED_KIND=$(fm_meta_get "$RESUME_META" kind) [ -n "$RECORDED_KIND" ] || RECORDED_KIND=ship @@ -985,76 +1926,202 @@ if [ "$RECOVERY_ACCOUNT" = 1 ]; then exit 1 fi KIND=$RECORDED_KIND + spawn_refuse_unsupported_secondmate_backend || exit 1 RECORDED_HARNESS=$(fm_meta_get "$RESUME_META" harness) - RECORDED_PROFILE=$(fm_meta_get "$RESUME_META" account_profile) - RECORDED_POOL=$(fm_meta_get "$RESUME_META" account_pool) - RECORDED_SESSION=$(fm_meta_get "$RESUME_META" provider_session_id) - RECORDED_ACCOUNT_TASK=$(fm_meta_get "$RESUME_META" account_task) - RECORDED_ATTEMPT=$(fm_meta_get "$RESUME_META" account_attempt) RECORDED_PROJECT=$(fm_meta_get "$RESUME_META" project) RECORDED_WORKTREE=$(fm_meta_get "$RESUME_META" worktree) - [ -n "$RECORDED_ACCOUNT_TASK" ] || RECORDED_ACCOUNT_TASK=$ID - [ -n "$RECORDED_ATTEMPT" ] || RECORDED_ATTEMPT=legacy [ -n "$RECORDED_HARNESS" ] || { echo "error: managed recovery metadata has no harness for $ID" >&2; exit 1; } - [ -n "$RECORDED_PROFILE" ] || { echo "error: managed recovery metadata has no account_profile for $ID" >&2; exit 1; } - [ -n "$RECORDED_POOL" ] || { echo "error: managed recovery metadata has no account_pool for $ID" >&2; exit 1; } - if [ "$RESUME_ACCOUNT" = 1 ]; then - FM_ACCOUNT_LIFECYCLE_LOCK_HELD="$LIFECYCLE_LOCK" "$SCRIPT_DIR/fm-account-session-sync.sh" "$ID" --require >/dev/null || exit 1 - RECORDED_SESSION=$(fm_meta_get "$RESUME_META" provider_session_id) - [ -n "$RECORDED_SESSION" ] || { echo "error: managed recovery metadata has no provider_session_id for $ID" >&2; exit 1; } - if [ "$HARNESS_SET" = 1 ] && [ "$HARNESS_ARG" != "$RECORDED_HARNESS" ]; then - echo "error: --resume-account harness override '$HARNESS_ARG' does not match recorded harness '$RECORDED_HARNESS'" >&2 + if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + RECORDED_ACCOUNT_HOME=$(fm_meta_get "$RESUME_META" account_home) + RECORDED_MODEL=$(fm_meta_get "$RESUME_META" model) + RECORDED_EFFORT=$(fm_meta_get "$RESUME_META" effort) + RECORDED_MODE=$(fm_meta_get "$RESUME_META" mode) + RECORDED_YOLO=$(fm_meta_get "$RESUME_META" yolo) + RECORDED_WORKTREE_GIT_DIR=$(fm_meta_get "$RESUME_META" worktree_git_dir) + RECORDED_WORKTREE_GIT_DIR_IDENTITY=$(fm_meta_get "$RESUME_META" worktree_git_dir_identity) + RECORDED_WORKTREE_GIT_REF=$(fm_meta_get "$RESUME_META" worktree_git_ref) + RECORDED_WORKTREE_GIT_HEAD=$(fm_meta_get "$RESUME_META" worktree_git_head) + RECORDED_WORKTREE_GIT_SETUP_REF=$(fm_meta_get "$RESUME_META" worktree_git_setup_ref) + RECORDED_WORKTREE_GIT_SETUP_HEAD=$(fm_meta_get "$RESUME_META" worktree_git_setup_head) + RECORDED_GENERATION=$(fm_meta_get "$RESUME_META" generation_id) + RECORDED_TASKTMP=$(fm_meta_get "$RESUME_META" tasktmp) + RECORDED_REPORT_REQUIRED= + RECORDED_REPORT_REQUIRED_SET=0 + if grep -q '^report_required=' "$RESUME_META"; then + RECORDED_REPORT_REQUIRED_SET=1 + RECORDED_REPORT_REQUIRED=$(fm_meta_get "$RESUME_META" report_required) + fi + case "$RECORDED_KIND" in + ship|scout) ;; + *) echo "error: direct account recovery metadata has invalid kind '$RECORDED_KIND' for $ID" >&2; exit 1 ;; + esac + case "$RECORDED_HARNESS" in + claude|codex) ;; + *) echo "error: direct account recovery metadata has unsupported harness '$RECORDED_HARNESS' for $ID" >&2; exit 1 ;; + esac + [ -z "$(fm_meta_get "$RESUME_META" direct_spawn_cleanup)" ] || { + echo "error: failed direct spawn cleanup is pending for $ID; tear down the retained endpoint and worktree before recovery" >&2 + exit 1 + } + [ -z "$(fm_meta_get "$RESUME_META" rollback_pending)" ] || { + echo "error: rollback cleanup is pending for $ID; tear down the retained task state before recovery" >&2 + exit 1 + } + [ -n "$RECORDED_ACCOUNT_HOME" ] || { echo "error: direct account recovery metadata has no account_home for $ID" >&2; exit 1; } + [ -z "$(fm_meta_get "$RESUME_META" account_profile)" ] || { echo "error: direct account recovery cannot replace legacy account_profile metadata for $ID" >&2; exit 1; } + [ -z "$(fm_meta_get "$RESUME_META" account_rollback_cleanup)" ] || { echo "error: direct account recovery cannot bypass pending legacy rollback cleanup for $ID" >&2; exit 1; } + [ -n "$RECORDED_PROJECT" ] || { echo "error: direct account recovery metadata has no project for $ID" >&2; exit 1; } + [ -n "$RECORDED_WORKTREE" ] || { echo "error: direct account recovery metadata has no worktree for $ID" >&2; exit 1; } + [ -n "$RECORDED_WORKTREE_GIT_DIR" ] || { echo "error: direct account recovery metadata has no exact worktree Git-dir for $ID" >&2; exit 1; } + [ -n "$RECORDED_WORKTREE_GIT_DIR_IDENTITY" ] || { echo "error: direct account recovery metadata has no worktree Git-dir identity for $ID" >&2; exit 1; } + if [ -n "$RECORDED_WORKTREE_GIT_REF" ] && [ -n "$RECORDED_WORKTREE_GIT_HEAD" ]; then + echo "error: direct account recovery metadata has conflicting branch and detached HEAD identities for $ID" >&2 + exit 1 + fi + if [ -z "$RECORDED_WORKTREE_GIT_REF" ] && [ -z "$RECORDED_WORKTREE_GIT_HEAD" ]; then + echo "error: direct account recovery metadata has no authoritative worktree Git state for $ID" >&2 exit 1 fi - if [ "$ACCOUNT_POOL_SET" = 1 ] && [ "$ACCOUNT_POOL" != "$RECORDED_POOL" ]; then - echo "error: --resume-account pool override '$ACCOUNT_POOL' does not match recorded pool '$RECORDED_POOL'" >&2 + if [ -n "$RECORDED_WORKTREE_GIT_SETUP_REF" ] && [ -z "$RECORDED_WORKTREE_GIT_SETUP_HEAD" ]; then + echo "error: direct account recovery metadata has an incomplete branch-setup identity for $ID" >&2 exit 1 fi - if [ "$ACCOUNT_PROFILE_SET" = 1 ] && [ "$ACCOUNT_PROFILE" != "$RECORDED_PROFILE" ]; then - echo "error: --resume-account profile override '$ACCOUNT_PROFILE' does not match recorded profile '$RECORDED_PROFILE'" >&2 + if [ -n "$RECORDED_WORKTREE_GIT_SETUP_HEAD" ] \ + && { [ -z "$RECORDED_WORKTREE_GIT_REF" ] || [ -n "$RECORDED_WORKTREE_GIT_HEAD" ]; }; then + echo "error: direct account recovery metadata has an invalid branch-setup transition for $ID" >&2 exit 1 fi + [ -n "$RECORDED_MODE" ] || { echo "error: direct account recovery metadata has no mode for $ID" >&2; exit 1; } + [ -n "$RECORDED_YOLO" ] || { echo "error: direct account recovery metadata has no yolo setting for $ID" >&2; exit 1; } + [ -n "$RECORDED_GENERATION" ] || { echo "error: direct account recovery metadata has no generation_id for $ID" >&2; exit 1; } + [ "$RECORDED_TASKTMP" = "/tmp/fm-$ID" ] || { echo "error: direct account recovery metadata has an invalid tasktmp for $ID" >&2; exit 1; } + RECORDED_META_WORKTREE_GIT_REF=$RECORDED_WORKTREE_GIT_REF + RECORDED_META_WORKTREE_GIT_HEAD=$RECORDED_WORKTREE_GIT_HEAD + RECORDED_META_WORKTREE_GIT_SETUP_REF=$RECORDED_WORKTREE_GIT_SETUP_REF + RECORDED_META_WORKTREE_GIT_SETUP_HEAD=$RECORDED_WORKTREE_GIT_SETUP_HEAD HARNESS_ARG=$RECORDED_HARNESS - ACCOUNT_POOL=$RECORDED_POOL - ACCOUNT_PROFILE=$RECORDED_PROFILE - ACCOUNT_POOL_SET=1 - ACCOUNT_PROFILE_SET=1 - ACCOUNT_TASK=$RECORDED_ACCOUNT_TASK - ACCOUNT_ATTEMPT=$RECORDED_ATTEMPT + HARNESS_SET=1 + MODEL=$RECORDED_MODEL + EFFORT=$RECORDED_EFFORT + [ "$MODEL" = default ] && MODEL= + [ "$EFFORT" = default ] && EFFORT= + ARG3=$HARNESS_ARG + DIRECT_ACCOUNT_RESPAWN=1 + PROJ=$RECORDED_PROJECT else - [ "$HARNESS_SET" = 1 ] || HARNESS_ARG=$RECORDED_HARNESS - if [ "$ACCOUNT_POOL_SET" = 0 ] && [ "$ACCOUNT_PROFILE_SET" = 0 ]; then - if [ "$HARNESS_ARG" = "$RECORDED_HARNESS" ]; then - ACCOUNT_POOL=$RECORDED_POOL - else - ACCOUNT_POOL=$(fm_account_default_pool "$HARNESS_ARG") || { - echo "error: no default account pool for continuation harness '$HARNESS_ARG'" >&2 - exit 1 - } + RECORDED_PROFILE=$(fm_meta_get "$RESUME_META" account_profile) + RECORDED_POOL=$(fm_meta_get "$RESUME_META" account_pool) + RECORDED_SESSION=$(fm_meta_get "$RESUME_META" provider_session_id) + RECORDED_ACCOUNT_TASK=$(fm_meta_get "$RESUME_META" account_task) + RECORDED_ATTEMPT=$(fm_meta_get "$RESUME_META" account_attempt) + [ -n "$RECORDED_ACCOUNT_TASK" ] || RECORDED_ACCOUNT_TASK=$ID + [ -n "$RECORDED_ATTEMPT" ] || RECORDED_ATTEMPT=legacy + [ -n "$RECORDED_PROFILE" ] || { echo "error: managed recovery metadata has no account_profile for $ID" >&2; exit 1; } + [ -n "$RECORDED_POOL" ] || { echo "error: managed recovery metadata has no account_pool for $ID" >&2; exit 1; } + if [ "$RESUME_ACCOUNT" = 1 ]; then + FM_ACCOUNT_LIFECYCLE_LOCK_HELD="$LIFECYCLE_LOCK" "$SCRIPT_DIR/fm-account-session-sync.sh" "$ID" --require >/dev/null || exit 1 + RECORDED_SESSION=$(fm_meta_get "$RESUME_META" provider_session_id) + [ -n "$RECORDED_SESSION" ] || { echo "error: managed recovery metadata has no provider_session_id for $ID" >&2; exit 1; } + if [ "$HARNESS_SET" = 1 ] && [ "$HARNESS_ARG" != "$RECORDED_HARNESS" ]; then + echo "error: --resume-account harness override '$HARNESS_ARG' does not match recorded harness '$RECORDED_HARNESS'" >&2 + exit 1 + fi + if [ "$ACCOUNT_POOL_SET" = 1 ] && [ "$ACCOUNT_POOL" != "$RECORDED_POOL" ]; then + echo "error: --resume-account pool override '$ACCOUNT_POOL' does not match recorded pool '$RECORDED_POOL'" >&2 + exit 1 fi + if [ "$ACCOUNT_PROFILE_SET" = 1 ] && [ "$ACCOUNT_PROFILE" != "$RECORDED_PROFILE" ]; then + echo "error: --resume-account profile override '$ACCOUNT_PROFILE' does not match recorded profile '$RECORDED_PROFILE'" >&2 + exit 1 + fi + HARNESS_ARG=$RECORDED_HARNESS + ACCOUNT_POOL=$RECORDED_POOL + ACCOUNT_PROFILE=$RECORDED_PROFILE ACCOUNT_POOL_SET=1 + ACCOUNT_PROFILE_SET=1 + ACCOUNT_TASK=$RECORDED_ACCOUNT_TASK + ACCOUNT_ATTEMPT=$RECORDED_ATTEMPT + else + [ "$HARNESS_SET" = 1 ] || HARNESS_ARG=$RECORDED_HARNESS + if [ "$ACCOUNT_POOL_SET" = 0 ] && [ "$ACCOUNT_PROFILE_SET" = 0 ]; then + if [ "$HARNESS_ARG" = "$RECORDED_HARNESS" ]; then + ACCOUNT_POOL=$RECORDED_POOL + else + ACCOUNT_POOL=$(fm_account_default_pool "$HARNESS_ARG") || { + echo "error: no default account pool for continuation harness '$HARNESS_ARG'" >&2 + exit 1 + } + fi + ACCOUNT_POOL_SET=1 + fi + ACCOUNT_PREDECESSOR_TASK=$RECORDED_ACCOUNT_TASK + ACCOUNT_PREDECESSOR_ATTEMPT=$RECORDED_ATTEMPT + ACCOUNT_PREDECESSOR_PROVIDER=$RECORDED_HARNESS + ACCOUNT_PREDECESSOR_PROFILE=$RECORDED_PROFILE + ACCOUNT_PREDECESSOR_POOL=$RECORDED_POOL + ACCOUNT_PREDECESSOR_SESSION=$RECORDED_SESSION + fi + HARNESS_SET=1 + ARG3=$HARNESS_ARG + if [ "$RESUME_ACCOUNT" = 1 ] || [ "$HARNESS_ARG" = "$RECORDED_HARNESS" ]; then + [ "$MODEL_SET" = 1 ] || MODEL=$(fm_meta_get "$RESUME_META" model) + [ "$EFFORT_SET" = 1 ] || EFFORT=$(fm_meta_get "$RESUME_META" effort) + fi + [ "$MODEL" = default ] && MODEL= + [ "$EFFORT" = default ] && EFFORT= + if [ "$KIND" = secondmate ]; then + RECORDED_SECONDMATE_HOME=$(fm_meta_get "$RESUME_META" home) + RECORDED_SECONDMATE_HOME=$(fm_checkout_trusted_dir "$RECORDED_SECONDMATE_HOME") || { + echo "error: managed recovery secondmate home is unavailable or redirected for $ID" >&2 + exit 1 + } + [ "$RECORDED_SECONDMATE_HOME" = "$FIRSTMATE_HOME" ] || { + echo "error: managed recovery secondmate home does not match registration for $ID" >&2 + exit 1 + } + else + PROJ=$(fm_meta_get "$RESUME_META" project) fi - ACCOUNT_PREDECESSOR_TASK=$RECORDED_ACCOUNT_TASK - ACCOUNT_PREDECESSOR_ATTEMPT=$RECORDED_ATTEMPT - ACCOUNT_PREDECESSOR_PROVIDER=$RECORDED_HARNESS - ACCOUNT_PREDECESSOR_PROFILE=$RECORDED_PROFILE - ACCOUNT_PREDECESSOR_POOL=$RECORDED_POOL - ACCOUNT_PREDECESSOR_SESSION=$RECORDED_SESSION - fi - HARNESS_SET=1 - ARG3=$HARNESS_ARG - if [ "$RESUME_ACCOUNT" = 1 ] || [ "$HARNESS_ARG" = "$RECORDED_HARNESS" ]; then - [ "$MODEL_SET" = 1 ] || MODEL=$(fm_meta_get "$RESUME_META" model) - [ "$EFFORT_SET" = 1 ] || EFFORT=$(fm_meta_get "$RESUME_META" effort) - fi - [ "$MODEL" = default ] && MODEL= - [ "$EFFORT" = default ] && EFFORT= - if [ "$KIND" = secondmate ]; then - FIRSTMATE_HOME=$(fm_meta_get "$RESUME_META" home) - else - PROJ=$(fm_meta_get "$RESUME_META" project) fi fi + +direct_recovery_context_matches() { + [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ] || return 0 + [ -f "$RESUME_META" ] \ + && [ "$(fm_meta_get "$RESUME_META" kind)" = "$RECORDED_KIND" ] \ + && [ "$(fm_meta_get "$RESUME_META" harness)" = "$RECORDED_HARNESS" ] \ + && [ "$(fm_meta_get "$RESUME_META" project)" = "$RECORDED_PROJECT" ] \ + && [ "$(fm_meta_get "$RESUME_META" worktree)" = "$RECORDED_WORKTREE" ] \ + && [ "$(fm_meta_get "$RESUME_META" worktree_git_dir)" = "$RECORDED_WORKTREE_GIT_DIR" ] \ + && [ "$(fm_meta_get "$RESUME_META" worktree_git_dir_identity)" = "$RECORDED_WORKTREE_GIT_DIR_IDENTITY" ] \ + && [ "$(fm_meta_get "$RESUME_META" worktree_git_ref)" = "$RECORDED_META_WORKTREE_GIT_REF" ] \ + && [ "$(fm_meta_get "$RESUME_META" worktree_git_head)" = "$RECORDED_META_WORKTREE_GIT_HEAD" ] \ + && [ "$(fm_meta_get "$RESUME_META" worktree_git_setup_ref)" = "$RECORDED_META_WORKTREE_GIT_SETUP_REF" ] \ + && [ "$(fm_meta_get "$RESUME_META" worktree_git_setup_head)" = "$RECORDED_META_WORKTREE_GIT_SETUP_HEAD" ] \ + && [ "$(fm_backend_of_meta "$RESUME_META")" = "$BACKEND" ] \ + && [ "$(fm_meta_get "$RESUME_META" model)" = "$RECORDED_MODEL" ] \ + && [ "$(fm_meta_get "$RESUME_META" effort)" = "$RECORDED_EFFORT" ] \ + && [ "$(fm_meta_get "$RESUME_META" mode)" = "$RECORDED_MODE" ] \ + && [ "$(fm_meta_get "$RESUME_META" yolo)" = "$RECORDED_YOLO" ] \ + && [ "$(fm_meta_get "$RESUME_META" account_home)" = "$RECORDED_ACCOUNT_HOME" ] \ + && [ "$(fm_meta_get "$RESUME_META" generation_id)" = "$RECORDED_GENERATION" ] \ + && [ "$(fm_meta_get "$RESUME_META" tasktmp)" = "$RECORDED_TASKTMP" ] \ + && [ -z "$(fm_meta_get "$RESUME_META" account_profile)" ] \ + && [ -z "$(fm_meta_get "$RESUME_META" direct_recovery_cleanup)" ] \ + && [ -z "$(fm_meta_get "$RESUME_META" direct_spawn_cleanup)" ] \ + && [ -z "$(fm_meta_get "$RESUME_META" account_rollback_cleanup)" ] \ + && [ -z "$(fm_meta_get "$RESUME_META" rollback_pending)" ] \ + || return 1 + if [ "$RECORDED_REPORT_REQUIRED_SET" = 1 ]; then + grep -q '^report_required=' "$RESUME_META" \ + && [ "$(fm_meta_get "$RESUME_META" report_required)" = "$RECORDED_REPORT_REQUIRED" ] + else + if grep -q '^report_required=' "$RESUME_META"; then + return 1 + fi + fi +} + [ -z "$HARNESS_ARG" ] || ARG3=$HARNESS_ARG # The verified launch command per adapter. The knowledge half of each adapter @@ -1163,6 +2230,14 @@ ACCOUNT_EXPLICIT=0 if [ "$ACCOUNT_POOL_SET" = 1 ] || [ "$ACCOUNT_PROFILE_SET" = 1 ]; then ACCOUNT_EXPLICIT=1 fi +if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + ACCOUNT_EXPLICIT=1 +elif [ "$KIND" != secondmate ] && [ "$RECOVERY_ACCOUNT" = 0 ] && [ "$NO_ACCOUNT_ROUTING" = 0 ] \ + && [ "$SPAWN_META_PRESENT" = 1 ] \ + && [ -n "$(spawn_preflight_meta_value account_home)" ]; then + echo "error: direct account metadata already exists for ${POS[0]}; use --recover-direct-account to preserve its recorded task context" >&2 + exit 1 +fi if [ "$KIND" = secondmate ]; then ACCOUNT_PRIMARY_MODE=$(fm_account_resolve_mode "$CONFIG" 0 0) || exit 1 fi @@ -1170,7 +2245,8 @@ ACCOUNT_EFFECTIVE_MODE=$(fm_account_resolve_mode "$CONFIG" "$ACCOUNT_EXPLICIT" " if [ "$NO_ACCOUNT_ROUTING" = 1 ]; then echo "WARNING: emergency --no-account-routing bypass is active for ${POS[0]:-unknown}; this spawn will use the provider's default identity and will be recorded in task metadata" >&2 fi -if [ "$ACCOUNT_EFFECTIVE_MODE" != off ] && [ "$ACCOUNT_POOL_SET" = 0 ] && [ "$ACCOUNT_PROFILE_SET" = 0 ] && [ "$KIND" = secondmate ]; then +if [ "$DIRECT_ACCOUNT_RECOVERY" = 0 ] && [ "$ACCOUNT_EFFECTIVE_MODE" != off ] \ + && [ "$ACCOUNT_POOL_SET" = 0 ] && [ "$ACCOUNT_PROFILE_SET" = 0 ] && [ "$KIND" = secondmate ]; then if SM_ACCOUNT_POOL=$(fm_account_secondmate_pool "$CONFIG"); then ACCOUNT_POOL=$SM_ACCOUNT_POOL else @@ -1181,10 +2257,14 @@ fi case "$HARNESS" in claude|codex) ;; *) - if [ "$ACCOUNT_EXPLICIT" = 1 ]; then + if [ "$ACCOUNT_POOL_SET" = 1 ] || [ "$ACCOUNT_PROFILE_SET" = 1 ]; then echo "error: --account-pool/--account-profile requires a claude or codex harness, not '$HARNESS'" >&2 exit 1 fi + if [ "$DIRECT_ACCOUNT_RESPAWN" = 1 ]; then + echo "error: recorded direct account routing requires a claude or codex harness, not '$HARNESS'" >&2 + exit 1 + fi if [ "$ACCOUNT_EFFECTIVE_MODE" = enforce ]; then echo "error: enforced account routing requires a claude or codex harness, not '$HARNESS'" >&2 exit 1 @@ -1192,6 +2272,35 @@ case "$HARNESS" in ACCOUNT_EFFECTIVE_MODE=off ;; esac +if { [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ] \ + || { [ "$RECOVERY_ACCOUNT" = 0 ] && [ "$KIND" != secondmate ]; }; } \ + && [ "$ACCOUNT_EFFECTIVE_MODE" != off ]; then + if [ "$ACCOUNT_EFFECTIVE_MODE" = enforce ] && fm_account_test_lab_enabled \ + && [ "$DIRECT_ACCOUNT_RECOVERY" = 0 ] \ + && [ "${FM_ACCOUNT_ROUTING_LEGACY_NEW_LAUNCH_TEST:-}" = firstmate-remove-fleet-routing-deadcode-fixture-v1 ]; then + : + else + [ "$RAW_LAUNCH" != 1 ] || { + echo "error: direct account-directory routing does not accept raw launch commands" >&2 + exit 1 + } + DIRECT_ACCOUNT_ROUTING=1 + if [ "$ACCOUNT_POOL_SET" = 1 ] || [ "$ACCOUNT_PROFILE_SET" = 1 ]; then + echo "fm-spawn: --account-pool/--account-profile now activate direct account-directory selection for new launches; the legacy alias does not pin the selected account" >&2 + elif [ "$DIRECT_ACCOUNT_RESPAWN" = 1 ]; then + echo "fm-spawn: recorded direct account metadata activates fresh account-directory selection for this respawn" >&2 + fi + if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + DIRECT_ACCOUNT_PREPARE_DEFERRED=1 + else + DIRECT_ACCOUNT_HOME=$("$SCRIPT_DIR/fm-account-directory.sh" prepare "$HARNESS") || exit 1 + echo "fm-spawn: selected direct $HARNESS account home $DIRECT_ACCOUNT_HOME" >&2 + fi + # Every Agent Fleet branch below is guarded by enforce. New direct crewmate + # launches deliberately rejoin the ordinary unmanaged spawn path after selection. + ACCOUNT_EFFECTIVE_MODE=off + fi +fi if [ "$ACCOUNT_EFFECTIVE_MODE" != off ] && [ -z "$ACCOUNT_POOL" ]; then if [ -n "$ACCOUNT_PROFILE" ]; then ACCOUNT_POOL=explicit @@ -1219,7 +2328,9 @@ if [ "$ACCOUNT_EFFECTIVE_MODE" != off ] && [ "$RESUME_ACCOUNT" != 1 ]; then ACCOUNT_ATTEMPT=$(fm_account_attempt_id "$FM_HOME" "$ID") || exit 1 ACCOUNT_TASK=$(fm_account_task_key "$FM_HOME" "$ID" "$ACCOUNT_ATTEMPT") || exit 1 fi -if [ "$ACCOUNT_EFFECTIVE_MODE" != off ]; then +if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + SPAWN_GENERATION_ID=$RECORDED_GENERATION +elif [ "$ACCOUNT_EFFECTIVE_MODE" != off ]; then SPAWN_GENERATION_ID="account:$ACCOUNT_TASK:$ACCOUNT_ATTEMPT" else SPAWN_GENERATION_ID="spawn:$(fm_account_attempt_id "$FM_HOME" "$ID")" || exit 1 @@ -1277,26 +2388,18 @@ if [ "$RECOVERY_ACCOUNT" = 0 ] && [ -f "$STATE/$ID.meta" ]; then present) echo "error: endpoint is already alive for $ID; refusing duplicate spawn" >&2; exit 1 ;; *) echo "error: endpoint state is unknown for $ID; refusing duplicate spawn" >&2; exit 1 ;; esac + META_BACKUP=$(mktemp "$STATE/.$ID.meta.rollback.XXXXXX") || exit 1 + cp -p "$STATE/$ID.meta" "$META_BACKUP" || exit 1 if [ "$ACCOUNT_EFFECTIVE_MODE" = enforce ]; then - META_BACKUP=$(mktemp "$STATE/.$ID.meta.rollback.XXXXXX") || exit 1 - cp -p "$STATE/$ID.meta" "$META_BACKUP" || exit 1 snapshot_existing_artifacts || exit 1 fi fi +if [ "$ACCOUNT_EFFECTIVE_MODE" != enforce ]; then + snapshot_existing_artifacts || exit 1 +fi secondmate_registry_value() { - local id=$1 key=$2 reg line value - reg="$DATA/secondmates.md" - [ -f "$reg" ] || return 1 - line=$(grep -E "^- $id( |$)" "$reg" | tail -1 || true) - [ -n "$line" ] || return 1 - case "$key" in - home) value=$(printf '%s\n' "$line" | sed -n 's/^[^(]*(home: \([^;)]*\);.*/\1/p') ;; - projects) value=$(printf '%s\n' "$line" | sed -n 's/^[^(]*(home: [^;)]*; scope: [^;)]*; projects: \([^;)]*\); added .*/\1/p') ;; - *) return 1 ;; - esac - [ -n "$value" ] || return 1 - printf '%s\n' "$value" + fm_secondmate_registry_query "$DATA/secondmates.md" query "$1" "$2" } shell_quote() { @@ -1485,17 +2588,25 @@ validate_firstmate_operational_dirs() { done } -if [ "$KIND" = secondmate ]; then - if [ -z "$FIRSTMATE_HOME" ] && [ -f "$STATE/$ID.meta" ]; then - FIRSTMATE_HOME=$(grep '^home=' "$STATE/$ID.meta" | cut -d= -f2- || true) - fi - if [ -z "$FIRSTMATE_HOME" ]; then - FIRSTMATE_HOME=$(secondmate_registry_value "$ID" home || true) - fi -fi - if [ "$KIND" = secondmate ]; then [ -n "$FIRSTMATE_HOME" ] || { echo "error: no firstmate home supplied or registered for $ID" >&2; exit 1; } + CURRENT_REGISTERED_SECONDMATE_HOME=$(secondmate_registry_value "$ID" home) || { + echo "error: secondmate registration became unprovable for $ID" >&2 + exit 1 + } + CURRENT_REGISTERED_SECONDMATE_HOME=$(fm_checkout_trusted_dir "$CURRENT_REGISTERED_SECONDMATE_HOME") || exit 1 + [ "$CURRENT_REGISTERED_SECONDMATE_HOME" = "$FIRSTMATE_HOME" ] || { + echo "error: secondmate registration changed while spawn waited for lifecycle ownership" >&2 + exit 1 + } + CURRENT_SECONDMATE_PROJECTS=$(secondmate_registry_value "$ID" projects) || { + echo "error: secondmate project registration became unprovable for $ID" >&2 + exit 1 + } + [ "$CURRENT_SECONDMATE_PROJECTS" = "$SECONDMATE_PROJECTS" ] || { + echo "error: secondmate project registration changed while spawn waited for lifecycle ownership" >&2 + exit 1 + } PROJ_ABS=$(validate_firstmate_home_for_spawn "$ID" "$FIRSTMATE_HOME") WT="$PROJ_ABS" else @@ -1527,26 +2638,24 @@ if [ "$RECOVERY_ACCOUNT" = 1 ]; then fi if [ "$KIND" = secondmate ]; then - # Local-HEAD sync: before launch, fast-forward this secondmate's worktree to the - # PRIMARY checkout's current default-branch commit, so a freshly spawned or - # recovery-respawned secondmate always runs the primary's version (AGENTS.md - # spawn section). Purely local - no fetch: the home is a worktree of this same - # repo and already holds the commit. ff-only and guarded; a dirty, diverged, or - # wrong-branch home is left untouched and launches as-is. The agent re-reads - # AGENTS.md fresh on launch, so no nudge is needed here. - if sm_primary_head=$(primary_head_commit "$FM_ROOT"); then - sm_ff_out=$(ff_target "$PROJ_ABS" "secondmate $ID" "$sm_primary_head" yes yes 2>&1 || true) - case "$sm_ff_out" in - *': skipped:'*) - sm_ff_line=$(first_line "$sm_ff_out") - sm_ff_prefix="secondmate $ID: skipped: " - sm_ff_reason=${sm_ff_line#"$sm_ff_prefix"} - echo "warning: secondmate $ID sync skipped before launch: $sm_ff_reason" >&2 - ;; - esac - else - echo "warning: secondmate $ID sync skipped before launch: primary default-branch commit cannot be resolved" >&2 + sm_primary_head=$(primary_head_commit "$FM_ROOT") || { + echo "error: refusing secondmate launch because the primary default-branch commit cannot be resolved" >&2 + exit 1 + } + if ! sm_ff_out=$(ff_target "$PROJ_ABS" "secondmate $ID" "$sm_primary_head" yes yes 2>&1); then + echo "error: refusing secondmate launch because its home cannot fast-forward safely: $(first_line "$sm_ff_out")" >&2 + exit 1 fi + case "$sm_ff_out" in + *': skipped:'*) + echo "error: refusing secondmate launch because its home freshness is unresolved: $(first_line "$sm_ff_out")" >&2 + exit 1 + ;; + esac + "$SCRIPT_DIR/fm-checkout-refresh.sh" verify-home "$PROJ_ABS" "$FM_ROOT" || { + echo "error: refusing secondmate launch because its live default-tip freshness cannot be proved" >&2 + exit 1 + } # Inheritable-config propagation: push the primary's declared LOCAL config into # this secondmate home's config/, so the secondmate's OWN crewmates and backlog # backend inherit the primary's settings. config/ is gitignored, so this is a @@ -1595,7 +2704,7 @@ if [ "$KIND" = secondmate ]; then else BRIEF="$DATA/$ID/brief.md" fi -if [ "$RECOVERY_ACCOUNT" != 1 ]; then +if [ "$RESUME_ACCOUNT" != 1 ]; then [ -f "$BRIEF" ] || { echo "error: no brief at $BRIEF" >&2; exit 1; } fi if [ "$KIND" = ship ] && [ "$RECOVERY_ACCOUNT" != 1 ]; then @@ -1623,6 +2732,22 @@ real_path_or_raw() { # <path> fi } +# Refresh the checkout that will seed Treehouse before creating an endpoint. +# A dirty, off-default, or diverged checkout stays untouched and warns here, but +# does not make the acquisition unsafe: Treehouse fetches origin independently +# and resets the selected clean pool worktree from that remote-tracking ref. +# The post-acquisition verification below is the fail-closed freshness proof. +if [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ] && [ "$RECOVERY_ACCOUNT" != 1 ]; then + if CHECKOUT_PREFLIGHT_OUT=$("$SCRIPT_DIR/fm-checkout-refresh.sh" preflight "$PROJ_ABS" 2>&1); then + CHECKOUT_PREFLIGHT_STATUS=0 + else + CHECKOUT_PREFLIGHT_STATUS=$? + fi + if [ "$CHECKOUT_PREFLIGHT_STATUS" -ne 0 ]; then + echo "warning: checkout refresh could not advance $PROJ_ABS before worktree acquisition: $(first_line "$CHECKOUT_PREFLIGHT_OUT")" >&2 + fi +fi + # Session-provider container-ensure + task creation. tmux stays exactly as P1 # left it (same session-name / new-window sequence, see bin/backends/tmux.sh); # a herdr spawn goes through the version-gated, workspace-per-HOME, @@ -1632,13 +2757,13 @@ real_path_or_raw() { # <path> # that every downstream operation (send/capture/kill) already treats as opaque # per-backend routing (fm_backend_resolve_selector). validate_spawn_worktree() { # <source> <inspect-target> - local source=$1 inspect_target=$2 wt_real proj_real wt_top wt_top_real + local source=$1 inspect_target=$2 wt_real proj_real wt_top wt_top_real wt_common proj_common provider_path provider_real wt_real= if ! wt_real=$(cd "$WT" 2>/dev/null && pwd -P); then wt_real= fi proj_real=$PROJ_ABS_REAL - wt_top=$(git -C "$WT" rev-parse --show-toplevel 2>/dev/null || true) + wt_top=$(git_repository_probe -C "$WT" rev-parse --show-toplevel 2>/dev/null || true) wt_top_real= if ! wt_top_real=$(cd "$wt_top" 2>/dev/null && pwd -P); then wt_top_real= @@ -1647,8 +2772,110 @@ validate_spawn_worktree() { # <source> <inspect-target> echo "error: $source did not yield an isolated worktree (resolved '$WT'; worktree root '${wt_top:-none}'; primary '$PROJ_ABS'); refusing to launch to avoid tangling the primary checkout. Inspect target $inspect_target" >&2 exit 1 fi + fm_checkout_validate_git_metadata "$wt_real" >/dev/null || { + echo "error: $source returned redirected or unprovable Git metadata at $wt_real" >&2 + exit 1 + } + fm_checkout_validate_git_metadata "$proj_real" >/dev/null || { + echo "error: project Git metadata is unprovable at $proj_real" >&2 + exit 1 + } + wt_common=$(fm_checkout_git_common_dir "$wt_real") || exit 1 + proj_common=$(fm_checkout_git_common_dir "$proj_real") || exit 1 + [ "$wt_common" = "$proj_common" ] || { + echo "error: $source returned a worktree from an unrelated repository" >&2 + exit 1 + } + if [ "$BACKEND" = orca ]; then + fm_backend_orca_authority_capabilities_check || exit 1 + provider_path=$(fm_backend_orca_worktree_path "$ORCA_WORKTREE_ID") || exit 1 + provider_real=$(fm_checkout_trusted_dir "$provider_path") || exit 1 + [ "$provider_real" = "$wt_real" ] || { + echo "error: Orca worktree identity does not match its returned path" >&2 + exit 1 + } + fi +} + +validate_orca_abort_worktree_identity() { + local wt_root project_root wt_common project_common provider_path provider_root + [ -n "${ORCA_WORKTREE_ID:-}" ] && [ -n "${WT:-}" ] && [ -n "${PROJ_ABS:-}" ] || return 1 + wt_root=$(fm_checkout_trusted_dir "$WT") || return 1 + project_root=$(fm_checkout_trusted_dir "$PROJ_ABS") || return 1 + [ "$wt_root" != "$project_root" ] || return 1 + fm_checkout_validate_git_metadata "$wt_root" >/dev/null || return 1 + fm_checkout_validate_git_metadata "$project_root" >/dev/null || return 1 + wt_common=$(fm_checkout_git_common_dir "$wt_root") || return 1 + project_common=$(fm_checkout_git_common_dir "$project_root") || return 1 + [ "$wt_common" = "$project_common" ] || return 1 + provider_path=$(fm_backend_orca_worktree_path "$ORCA_WORKTREE_ID") || return 1 + provider_root=$(fm_checkout_trusted_dir "$provider_path") || return 1 + [ "$provider_root" = "$wt_root" ] } +if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + validate_spawn_worktree "recorded direct account recovery" "$RECORDED_TARGET" + recorded_project_common=$(git_common_dir_real "$PROJ_ABS" 2>/dev/null || true) + recorded_worktree_common=$(git_common_dir_real "$WT" 2>/dev/null || true) + if [ -z "$recorded_project_common" ] || [ -z "$recorded_worktree_common" ] \ + || [ "$recorded_project_common" != "$recorded_worktree_common" ]; then + echo "error: recorded direct account recovery worktree '$WT' does not belong to recorded project '$PROJ_ABS'; refusing endpoint creation" >&2 + exit 1 + fi + validate_direct_recovery_worktree_identity || exit 1 +fi + +if [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ] && [ "$RECOVERY_ACCOUNT" != 1 ]; then + "$SCRIPT_DIR/fm-checkout-refresh.sh" pool-preflight "$PROJ_ABS" || { + echo "error: refusing Treehouse acquisition because pool safety could not be inspected for $PROJ_ABS" >&2 + exit 1 + } + acquire_status=0 + WT=$("$SCRIPT_DIR/fm-checkout-refresh.sh" acquire-worktree "$PROJ_ABS" "firstmate-$ID") || acquire_status=$? + if [ "$acquire_status" -ne 0 ]; then + if [ "$acquire_status" -eq 124 ]; then + echo "error: refusing to spawn $ID after the bounded Treehouse acquisition timed out" >&2 + else + echo "error: treehouse get --lease failed to acquire a task worktree for $ID" >&2 + fi + exit 1 + fi + [ -n "$WT" ] || { + echo "error: treehouse get --lease did not report a task worktree for $ID" >&2 + exit 1 + } + WORKTREE_CREATED=1 + WORKTREE_RETAIN_ON_ABORT=1 + validate_spawn_worktree "treehouse get --lease" "$PROJ_ABS" + freshness_status=0 + "$SCRIPT_DIR/fm-checkout-refresh.sh" verify-worktree "$WT" "$PROJ_ABS" || freshness_status=$? + if [ "$freshness_status" -ne 0 ]; then + echo "error: refusing to launch fm-$ID from a leased worktree whose repository identity, cleanliness, or default-tip freshness could not be proved" >&2 + exit 1 + fi + WORKTREE_EXPECTED_TIP=$(git -C "$WT" rev-parse HEAD) || exit 1 + WORKTREE_RETAIN_ON_ABORT=0 +fi + +if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + META_WRITE_LOCK=$(fm_account_meta_lock_acquire "$STATE" "$ID") || exit 1 + direct_recovery_context_matches || { + echo "error: direct account task generation changed before recovery mutation for $ID" >&2 + exit 1 + } + META_BACKUP=$(mktemp "$STATE/.$ID.meta.rollback.XXXXXX") || exit 1 + cp -p "$RESUME_META" "$META_BACKUP" || exit 1 + snapshot_existing_artifacts || exit 1 + fm_account_meta_lock_release "$META_WRITE_LOCK" || exit 1 + META_WRITE_LOCK= +fi + +if [ "$DIRECT_ACCOUNT_PREPARE_DEFERRED" = 1 ]; then + DIRECT_ACCOUNT_HOME=$("$SCRIPT_DIR/fm-account-directory.sh" prepare "$HARNESS") || exit 1 + echo "fm-spawn: selected direct $HARNESS account home $DIRECT_ACCOUNT_HOME" >&2 + DIRECT_ACCOUNT_PREPARE_DEFERRED=0 +fi + if [ "$CONTINUE_ACCOUNT" = 1 ]; then CONTINUATION_RESULT=$(FM_ACCOUNT_CONTINUATION_EMIT_PROMPT_B64=1 \ "$SCRIPT_DIR/fm-account-continuation.sh" "$ID" "$ACCOUNT_ATTEMPT") || exit 1 @@ -1691,18 +2918,20 @@ finally: fi W="fm-$ID" -SPAWN_CWD=$PROJ_ABS -[ "$RECOVERY_ACCOUNT" != 1 ] || SPAWN_CWD=$WT +SPAWN_CWD=${WT:-$PROJ_ABS} +if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + validate_direct_recovery_worktree_identity || exit 1 +fi case "$BACKEND" in tmux) SES=$(fm_backend_tmux_container_ensure) T="$SES:$W" # #134 robustness (tmux): fm_backend_tmux_create_task captures a stable window # id and pins the window name (automatic-rename/allow-rename off) so a captain's - # non-default tmux config cannot rename the window away from fm-<id> once - # treehouse cd's into the worktree. WT_TARGET carries that stable id for the - # rename-critical worktree-detection steps below; the persisted window= handle - # stays $T (the name form), which is safe now that rename is disabled. + # non-default tmux config cannot rename the window away from fm-<id>. + # WT_TARGET carries that stable id for spawn-time commands below; the + # persisted window= handle stays $T (the name form), which is safe now that + # rename is disabled. WID=$(fm_backend_tmux_create_task "$SES" "$W" "$SPAWN_CWD") || exit 1 ENDPOINT_CREATED=1 WT_TARGET="$WID" @@ -1777,41 +3006,76 @@ EOF ENDPOINT_CREATED=1 ;; orca) - set +e - ORCA_WT_RAW=$(fm_backend_orca_worktree_create "$PROJ_ABS" "$W") - ORCA_WT_STATUS=$? - set -e - if [ "$ORCA_WT_STATUS" -ne 0 ]; then - if [ "$ORCA_WT_STATUS" -eq 2 ] && [ -n "$ORCA_WT_RAW" ]; then - if parse_orca_worktree_result "$ORCA_WT_RAW" && [ -n "$ORCA_WORKTREE_ID" ]; then - ORCA_ABORT_CLEANUP=1 + if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + ORCA_WORKTREE_ID=$(fm_meta_get "$RESUME_META" orca_worktree_id) + [ -n "$ORCA_WORKTREE_ID" ] || { + echo "error: direct account recovery metadata has no Orca worktree id for $ID" >&2 + exit 1 + } + ORCA_RECORDED_WORKTREE=$(fm_backend_orca_worktree_path "$ORCA_WORKTREE_ID") || exit 1 + [ "$(real_path_or_raw "$ORCA_RECORDED_WORKTREE")" = "$(real_path_or_raw "$WT")" ] || { + echo "error: recorded Orca worktree identity no longer matches $WT for $ID" >&2 + exit 1 + } + ORCA_TERMINAL=$(fm_backend_orca_terminal_create "$ORCA_WORKTREE_ID" "$W") || exit 1 + ORCA_TERMINAL_PROOF=recorded + else + ORCA_EXPECTED_TASK="fm-$ID" + ORCA_TERMINAL_PROOF=unproven + persist_orca_cleanup_quarantine spawn-preparing || { + echo "error: cannot durably arm Orca cleanup quarantine for $ID" >&2 + exit 1 + } + ORCA_ABORT_CLEANUP=1 + set +e + ORCA_WT_RAW=$(fm_backend_orca_worktree_create "$PROJ_ABS" "$W") + ORCA_WT_STATUS=$? + set -e + if [ "$ORCA_WT_STATUS" -ne 0 ]; then + if [ "$ORCA_WT_STATUS" -eq 2 ] && [ -n "$ORCA_WT_RAW" ]; then + parse_orca_worktree_result "$ORCA_WT_RAW" || true + persist_orca_cleanup_quarantine spawn-abort || { + echo "error: cannot durably record partial Orca create authority for $ID" >&2 + } fi + exit 1 fi - exit 1 + parse_orca_worktree_result "$ORCA_WT_RAW" || true + persist_orca_cleanup_quarantine spawn-abort || { + echo "error: cannot durably record Orca create authority for $ID" >&2 + exit 1 + } + if [ -z "$ORCA_WORKTREE_ID" ] || [ -z "$WT" ] || [ "$ORCA_PROVIDER_TASK" != "$ORCA_EXPECTED_TASK" ]; then + echo "error: orca did not return matching worktree id, path, and task authority for $W" >&2 + exit 1 + fi + validate_spawn_worktree "orca worktree create" "$W" + if [ -z "$ORCA_TERMINAL" ]; then + ORCA_TERMINAL=$(fm_backend_orca_terminal_create "$ORCA_WORKTREE_ID" "$W") || exit 1 + ORCA_TERMINAL_PROOF=recorded + persist_orca_cleanup_quarantine spawn-abort || { + echo "error: cannot durably record the Orca terminal authority for $ID" >&2 + exit 1 + } + fi + WORKTREE_CREATED=1 fi - parse_orca_worktree_result "$ORCA_WT_RAW" || true - ORCA_ABORT_CLEANUP=1 - if [ -z "$ORCA_WORKTREE_ID" ] || [ -z "$WT" ]; then - echo "error: orca did not return a worktree id/path for $W" >&2 + if [ "$(fm_backend_orca_terminal_state "$ORCA_TERMINAL" "$ORCA_WORKTREE_ID" "$W")" != present ] \ + || ! fm_backend_orca_worktree_terminal_contains "$ORCA_WORKTREE_ID" "$W" "$ORCA_TERMINAL"; then + echo "error: Orca terminal is not authoritatively bound to worktree $ORCA_WORKTREE_ID and task $W" >&2 exit 1 fi - validate_spawn_worktree "orca worktree create" "$W" - if [ -z "$ORCA_TERMINAL" ]; then - ORCA_TERMINAL=$(fm_backend_orca_terminal_create "$ORCA_WORKTREE_ID" "$W") || exit 1 - fi T="$ORCA_TERMINAL" ENDPOINT_CREATED=1 - WORKTREE_CREATED=1 ;; esac if [ "$ACCOUNT_EFFECTIVE_MODE" = enforce ]; then persist_failed_account_rollback_short || exit 1 fi -# #134 robustness: only tmux needs a worktree-detection target distinct from $T - -# its rename-safe stable window id, set as WT_TARGET=$WID in the tmux branch above. +# #134 robustness: only tmux needs a command target distinct from $T - its +# rename-safe stable window id, set as WT_TARGET=$WID in the tmux branch above. # Every other backend addresses its pane/surface by the id already in $T, so default -# WT_TARGET to $T for them (and for any future backend) - the shared treehouse-get + -# worktree-detection steps below must never reference an unbound WT_TARGET under set -u. +# WT_TARGET to $T for them (and for any future backend). : "${WT_TARGET:=$T}" spawn_send_text_line() { # <target> <text> case "$BACKEND" in @@ -1849,36 +3113,34 @@ spawn_send_key() { # <target> <key> esac } if [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ] && [ "$RECOVERY_ACCOUNT" != 1 ]; then - spawn_send_text_line "$WT_TARGET" 'treehouse get' - - # Wait for the treehouse subshell: the pane's cwd moves from the project to the worktree. - # Target the stable window id, not the name: if the name is ever lost (e.g. an - # automatic-rename slips through), display-message -t <bad-name> falls back to the - # active client's window, which would misread firstmate's OWN pane path as the - # worktree and tangle a hook into the primary checkout. The window id never lies. - # Compare against PROJ_ABS_REAL (physical), not PROJ_ABS: a symlinked project - # prefix would otherwise make the pane's OS-level cwd read differ from - # PROJ_ABS on the very first poll, before the pane has actually moved. + WT_REAL=$(real_path_or_raw "$WT") for _ in $(seq 1 60); do p=$(spawn_current_path "$WT_TARGET" || true) - if [ -n "$p" ] && [ "$(real_path_or_raw "$p")" != "$PROJ_ABS_REAL" ]; then - WT="$p" + if [ -n "$p" ] && [ "$(real_path_or_raw "$p")" = "$WT_REAL" ]; then break fi sleep 1 done - if [ -z "$WT" ]; then - echo "error: treehouse get did not enter a worktree within 60s; inspect window $T" >&2 + if [ -z "${p:-}" ] || [ "$(real_path_or_raw "$p")" != "$WT_REAL" ]; then + echo "error: task endpoint did not start in leased worktree $WT within 60s; inspect window $T" >&2 exit 1 fi - - validate_spawn_worktree "treehouse get" "$T" - WORKTREE_CREATED=1 fi if [ -z "$WT" ] && [ "$BACKEND" = orca ]; then WT="$PROJ_ABS" fi +if [ "$DIRECT_ACCOUNT_ROUTING" = 1 ] && [ "$DIRECT_ACCOUNT_RECOVERY" = 0 ] && [ "$KIND" != secondmate ]; then + WT=$(cd "$WT" 2>/dev/null && pwd -P) || { + echo "error: cannot canonicalize direct account worktree for $ID" >&2 + exit 1 + } + capture_worktree_git_physical_identity "$WT" || { + echo "error: cannot record exact direct account worktree identity for $ID" >&2 + exit 1 + } +fi + # Per-task temp root: /tmp/fm-<id>/ with Go's build temp nested at gotmp/. Go won't # create GOTMPDIR, so mkdir before it is used; fm-teardown removes the whole root. # Nested (not a bare /tmp/fm-<id>/gotmp) so other per-task temp can live alongside @@ -1896,7 +3158,7 @@ STATE_REAL=$(cd "$STATE" && pwd -P) TURNEND="$STATE_REAL/$ID.turn-ended" exclude_path() { local rel=$1 EXCL - EXCL=$(git -C "$WT" rev-parse --git-path info/exclude 2>/dev/null || true) + EXCL=$(git_repository_probe -C "$WT" rev-parse --git-path info/exclude 2>/dev/null || true) [ -n "$EXCL" ] || return 0 mkdir -p "$(dirname "$EXCL")" grep -qxF "$rel" "$EXCL" 2>/dev/null || echo "$rel" >> "$EXCL" @@ -1997,11 +3259,12 @@ fi # Recorded in meta so fm-teardown's safety check and the validate/merge stages can # branch on them. Mode governs ship tasks; a scout's deliverable is a report, not a # merge, so scout teardown ignores mode. -SECONDMATE_PROJECTS= -if [ "$KIND" = secondmate ]; then +if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + MODE=$RECORDED_MODE + YOLO=$RECORDED_YOLO +elif [ "$KIND" = secondmate ]; then MODE=secondmate YOLO=off - SECONDMATE_PROJECTS=$(secondmate_registry_value "$ID" projects || true) else PROJ_NAME=$(basename "$PROJ_ABS") read -r MODE YOLO <<EOF @@ -2042,6 +3305,18 @@ if [ "$ACCOUNT_EFFECTIVE_MODE" = enforce ]; then fi fi +if [ "$DIRECT_ACCOUNT_ROUTING" = 1 ]; then + if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + validate_direct_recovery_worktree_identity || exit 1 + else + validate_direct_launch_worktree_identity || exit 1 + capture_direct_launch_authoritative_state || { + echo "error: cannot record authoritative direct account worktree Git state for $ID" >&2 + exit 1 + } + fi +fi + META_WINDOW=$T [ "$BACKEND" = orca ] && META_WINDOW=$W lifecycle_lock_valid=0 @@ -2067,7 +3342,12 @@ if [ "$lifecycle_lock_valid" != 1 ]; then exit 1 fi META_WRITE_LOCK=$(fm_account_meta_lock_acquire "$STATE" "$ID") || exit 1 -if [ "$ACCOUNT_EFFECTIVE_MODE" = enforce ]; then +if [ "$DIRECT_ACCOUNT_RECOVERY" = 1 ]; then + direct_recovery_context_matches || { + echo "error: direct account task generation changed before metadata install for $ID" >&2 + exit 1 + } +elif [ "$ACCOUNT_EFFECTIVE_MODE" = enforce ]; then if [ ! -f "$STATE/$ID.meta" ] || [ "$(fm_meta_get "$STATE/$ID.meta" account_task)" != "$ACCOUNT_TASK" ]; then echo "error: managed task generation changed before metadata install for $ID" >&2 exit 1 @@ -2077,6 +3357,12 @@ META_TMP=$(mktemp "$STATE/.$ID.meta.XXXXXX") || exit 1 { echo "window=$META_WINDOW" echo "worktree=$WT" + [ "$DIRECT_ACCOUNT_ROUTING" != 1 ] || echo "worktree_git_dir=$WORKTREE_GIT_DIR" + [ "$DIRECT_ACCOUNT_ROUTING" != 1 ] || echo "worktree_git_dir_identity=$WORKTREE_GIT_DIR_IDENTITY" + [ "$DIRECT_ACCOUNT_ROUTING" != 1 ] || [ -z "$WORKTREE_GIT_REF" ] || echo "worktree_git_ref=$WORKTREE_GIT_REF" + [ "$DIRECT_ACCOUNT_ROUTING" != 1 ] || [ -z "$WORKTREE_GIT_HEAD" ] || echo "worktree_git_head=$WORKTREE_GIT_HEAD" + [ "$DIRECT_ACCOUNT_ROUTING" != 1 ] || [ -z "$WORKTREE_GIT_SETUP_REF" ] || echo "worktree_git_setup_ref=$WORKTREE_GIT_SETUP_REF" + [ "$DIRECT_ACCOUNT_ROUTING" != 1 ] || [ -z "$WORKTREE_GIT_SETUP_HEAD" ] || echo "worktree_git_setup_head=$WORKTREE_GIT_SETUP_HEAD" echo "project=$PROJ_ABS" echo "harness=$HARNESS" echo "kind=$KIND" @@ -2087,6 +3373,7 @@ META_TMP=$(mktemp "$STATE/.$ID.meta.XXXXXX") || exit 1 echo "effort=${EFFORT:-default}" echo "generation_id=$SPAWN_GENERATION_ID" [ "$NO_ACCOUNT_ROUTING" != 1 ] || echo "account_routing_emergency_bypass=1" + [ -z "$DIRECT_ACCOUNT_HOME" ] || echo "account_home=$DIRECT_ACCOUNT_HOME" if [ "$RECOVERY_ACCOUNT" = 1 ]; then if grep -q '^report_required=' "$RESUME_META"; then RECORDED_REPORT_REQUIRED=$(fm_account_meta_value "$RESUME_META" report_required) @@ -2146,6 +3433,10 @@ META_TMP=$(mktemp "$STATE/.$ID.meta.XXXXXX") || exit 1 if [ "$BACKEND" = orca ]; then echo "orca_worktree_id=$ORCA_WORKTREE_ID" echo "terminal=$ORCA_TERMINAL" + echo "orca_repo_id=$ORCA_REPO_ID" + echo "orca_expected_task=$ORCA_EXPECTED_TASK" + echo "orca_discovery_label=$ORCA_EXPECTED_TASK" + echo "orca_provider_scope=repo-path:$PROJ_ABS" fi if [ "$BACKEND" = cmux ]; then echo "cmux_workspace_id=$CMUX_WORKSPACE_ID" @@ -2196,6 +3487,12 @@ if [ "$RESUME_ACCOUNT" = 1 ]; then esac fi AGENT_COMMAND=$HARNESS +if [ "$DIRECT_ACCOUNT_ROUTING" = 1 ]; then + case "$HARNESS" in + claude) AGENT_COMMAND="CLAUDE_CONFIG_DIR=$(shell_quote "$DIRECT_ACCOUNT_HOME") $HARNESS" ;; + codex) AGENT_COMMAND="CODEX_HOME=$(shell_quote "$DIRECT_ACCOUNT_HOME") $HARNESS" ;; + esac +fi if [ "$ACCOUNT_EFFECTIVE_MODE" = enforce ]; then if [ "$RESUME_ACCOUNT" = 1 ]; then rm -rf "$STATE/.$ID.account-native-launch" "$STATE/.$ID.account-native-ready" "$STATE/.$ID.account-native-go" || exit 1 @@ -2257,6 +3554,17 @@ fi # Export GOTMPDIR into the crewmate's pane shell so the agent and every child # process (go build, go test, ...) inherit it. Sent before the launch command so # the env is set when the agent starts; the brief sleep lets the export land. +if [ "$BACKEND" = orca ]; then + if [ "$(fm_backend_orca_terminal_state "$T" "$ORCA_WORKTREE_ID" "$W")" != present ] \ + || ! fm_backend_orca_worktree_terminal_contains "$ORCA_WORKTREE_ID" "$W" "$T"; then + echo "error: Orca terminal authority changed before launch for $ID" >&2 + exit 1 + fi + validate_orca_abort_worktree_identity || { + echo "error: Orca worktree authority changed before launch for $ID" >&2 + exit 1 + } +fi spawn_send_text_line "$T" "export GOTMPDIR=$TASK_TMP/gotmp" sleep 0.3 spawn_send_literal "$T" "$LAUNCH" @@ -2321,7 +3629,15 @@ CONTINUATION_PROMPT_FILE= META_BACKUP= discard_existing_artifact_backup [ "$LIFECYCLE_LOCK_OWNED" != 1 ] || [ -z "$LIFECYCLE_LOCK" ] || fm_account_lifecycle_lock_release "$LIFECYCLE_LOCK" || exit 1 +if [ -n "$SECONDMATE_TARGET_HOME_LIFECYCLE_LOCK" ]; then + fm_account_lifecycle_lock_release "$SECONDMATE_TARGET_HOME_LIFECYCLE_LOCK" || exit 1 + SECONDMATE_TARGET_HOME_LIFECYCLE_LOCK= +fi +[ -z "$SECONDMATE_HOME_LIFECYCLE_LOCK" ] || fm_account_lifecycle_lock_release "$SECONDMATE_HOME_LIFECYCLE_LOCK" || exit 1 LIFECYCLE_LOCK= LIFECYCLE_LOCK_OWNED=0 +SECONDMATE_HOME_LIFECYCLE_LOCK= -echo "spawned $ID harness=$HARNESS kind=$KIND mode=$MODE yolo=$YOLO window=$META_WINDOW worktree=$WT" +account_summary= +[ -z "$DIRECT_ACCOUNT_HOME" ] || account_summary=" account_home=$DIRECT_ACCOUNT_HOME" +echo "spawned $ID harness=$HARNESS kind=$KIND mode=$MODE yolo=$YOLO window=$META_WINDOW worktree=$WT$account_summary" diff --git a/bin/fm-teardown.sh b/bin/fm-teardown.sh index e2075a6072b..ec3c7649823 100755 --- a/bin/fm-teardown.sh +++ b/bin/fm-teardown.sh @@ -21,7 +21,16 @@ # by itself causes a false refusal of landed work. # A gh lookup error falls back to the content check; if that is also inconclusive, # teardown refuses rather than risk discarding unlanded work. +# Origin-backed content checks hold the shared checkout lock and require bounded +# remote HEAD probes before and after fetch to agree before comparing trees. +# Every authorized Treehouse return is process-tree bounded by +# FM_TREEHOUSE_RETURN_TIMEOUT while holding the same common checkout mutation +# lock across its retry and stale-index-lock recovery sequence. # Uncommitted changes are never landed. +# Ordinary teardown first proves that metadata names the exact registered project, +# worktree, and task lease, then quiesces the endpoint before its final safety checks. +# Each locked Treehouse return repeats repository, lease, and landed-work checks +# immediately before the destructive return command. # local-only projects additionally accept work merged into the local default # branch (firstmate performs that merge on the captain's approval) as a fallback # for the common case where there is no remote at all. @@ -30,21 +39,22 @@ # product. A pre-cutover scout proceeds once that report exists; a task carrying # report_required=1 must satisfy the shared completion and publication contract # owned by docs/report-stack.md before teardown discards the scratch worktree. -# Orca tasks use the same safety checks, then close the recorded terminal and -# remove the recorded worktree through `orca worktree rm`; teardown never guesses -# an Orca target from ambient CLI state. -# Secondmates (kind=secondmate in meta) are retired explicitly. Normal -# teardown refuses while their home has in-flight crewmate meta files; --force -# is the approved discard path that prevalidates child removal targets, discards -# child work, kills child runtime endpoints, and removes the retired home. Removing a +# Orca tasks use the same safety checks, then close the recorded terminal, prove +# the handle stale, and remove the recorded worktree under its checkout lock; +# teardown never substitutes the shared window alias for a missing terminal. +# Secondmates (kind=secondmate in meta) are retired explicitly. Teardown proves +# the home clean and every ref and reflog commit landed, then quiesces its endpoint and +# refuses while the home has in-flight crewmate meta files. --force authorizes +# recursive retirement only after every child passes the same endpoint, identity, +# cleanliness, stash, and landed-work proofs. Project retirement also rejects +# mount boundaries, rewritten history, and landing authorities whose complete Git +# object storage or network transport may depend on the retiring home or local machine. Removing a # leased home releases its durable treehouse lease so the pool slot is freed, # never left leased forever. If the treehouse return fails, teardown leaves the # leased home and state in place instead of hiding a still-held lease. # Usage: fm-teardown.sh <task-id> [--force] -# --force skips ordinary-task dirty and landed-work checks, skips scout and -# required-report publication checks, and discards secondmate child work for -# kind=secondmate. It is an explicit discard and never publishes completion. -# Only use it when the captain has explicitly said to discard the work. +# --force permits recursive kind=secondmate retirement. It never bypasses +# dirty, untracked, stash, landed-work, endpoint, identity, or report proofs. # # Transient / stale worktree git lock recovery (teardown-lock-race): a crew process # killed mid-git-operation can leave a .git/worktrees/<wt>/index.lock (or, for a @@ -83,14 +93,20 @@ FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" +CHECKOUT_STATE_BASE="${FM_CHECKOUT_REFRESH_STATE_BASE:-${XDG_STATE_HOME:-$HOME/.local/state}/firstmate/checkout-refresh}" SECONDMATE_REG="$DATA/secondmates.md" SUB_HOME_MARKER=".fm-secondmate-home" +# shellcheck source=bin/fm-checkout-lock-lib.sh +. "$SCRIPT_DIR/fm-checkout-lock-lib.sh" +CHECKOUT_LOCK_ROOT=$(fm_checkout_lock_root "$CHECKOUT_STATE_BASE") # shellcheck source=bin/fm-tasks-axi-lib.sh . "$SCRIPT_DIR/fm-tasks-axi-lib.sh" # shellcheck source=bin/fm-backend.sh . "$SCRIPT_DIR/fm-backend.sh" # shellcheck source=bin/fm-lock-lib.sh . "$SCRIPT_DIR/fm-lock-lib.sh" +# shellcheck source=bin/fm-process-tree-lib.sh +. "$SCRIPT_DIR/fm-process-tree-lib.sh" # shellcheck source=bin/fm-gate-refuse-lib.sh . "$SCRIPT_DIR/fm-gate-refuse-lib.sh" # Fail closed before any fleet mutation: a no-mistakes gate agent must never tear @@ -100,14 +116,47 @@ fm_refuse_if_gate_agent . "$SCRIPT_DIR/fm-account-routing-lib.sh" FM_LOCK_LOG_PREFIX=teardown "$FM_ROOT/bin/fm-guard.sh" || true +TEARDOWN_UPSTREAM_TIMEOUT=${FM_CHECKOUT_REFRESH_PROBE_TIMEOUT:-15} +case "$TEARDOWN_UPSTREAM_TIMEOUT" in + ''|*[!0-9]*|0) + echo "error: FM_CHECKOUT_REFRESH_PROBE_TIMEOUT must be a positive integer" >&2 + exit 2 + ;; +esac ID=$1 FORCE=${2:-} META="$STATE/$ID.meta" -[ -f "$META" ] || { echo "error: no meta for task $ID at $META" >&2; exit 1; } + +require_safe_task_metadata() { + local state_root meta_parent + [ -d "$STATE" ] && [ ! -L "$STATE" ] || { + echo "error: task state must be a real directory: $STATE" >&2 + return 1 + } + [ -f "$META" ] && [ ! -L "$META" ] && [ -r "$META" ] || { + echo "error: task metadata must be a real readable file for $ID at $META" >&2 + return 1 + } + state_root=$(cd "$STATE" 2>/dev/null && pwd -P) || return 1 + meta_parent=$(cd "$(dirname "$META")" 2>/dev/null && pwd -P) || return 1 + [ "$state_root" = "$meta_parent" ] && [ "$(basename "$META")" = "$ID.meta" ] || { + echo "error: task metadata identity does not match requested task $ID" >&2 + return 1 + } +} + +require_safe_task_metadata || exit 1 TEARDOWN_ACCOUNT_LOCKS=('') MANAGED_ACCOUNT_LOCK= ACCOUNT_DELETE_LOCK= +SECONDMATE_HOME_LIFECYCLE_LOCK= +SECONDMATE_REGISTRY_LOCK= +PREPARED_REGISTRY_PATH= +PREPARED_REGISTRY_BACKUP= +PREPARED_REGISTRY_ID= +PREPARED_REGISTRY_HOME= +PREPARED_REGISTRY_LOCK= release_teardown_account_locks() { local lock @@ -123,20 +172,33 @@ managed_account_meta() { } MANAGED_ACCOUNT=0 +PRELOCK_KIND=$(fm_meta_get "$META" kind) +[ -n "$PRELOCK_KIND" ] || PRELOCK_KIND=ship +if [ "$PRELOCK_KIND" = secondmate ]; then + PRELOCK_HOME=$(fm_meta_get "$META" home) + [ -n "$PRELOCK_HOME" ] || PRELOCK_HOME=$(fm_meta_get "$META" worktree) + SECONDMATE_HOME_LIFECYCLE_LOCK=$(fm_secondmate_home_lifecycle_lock_acquire "$CHECKOUT_LOCK_ROOT" "$PRELOCK_HOME") || { + echo "error: secondmate home lifecycle identity is missing, redirected, or uninspectable for $ID" >&2 + exit 1 + } + TEARDOWN_ACCOUNT_LOCKS+=("$SECONDMATE_HOME_LIFECYCLE_LOCK") +fi ACCOUNT_DELETE_LOCK=$(fm_account_lifecycle_lock_acquire "$STATE" "$ID") || exit 1 TEARDOWN_ACCOUNT_LOCKS+=("$ACCOUNT_DELETE_LOCK") -[ -f "$META" ] || { echo "error: task metadata disappeared while teardown waited for $ID" >&2; exit 1; } +require_safe_task_metadata || { echo "error: task metadata changed while teardown waited for $ID" >&2; exit 1; } if managed_account_meta "$META"; then MANAGED_ACCOUNT=1 managed_account_meta "$META" || { echo "error: managed task metadata changed while teardown waited for $ID" >&2; exit 1; } fi -WT=$(grep '^worktree=' "$META" | cut -d= -f2-) +WT=$(grep '^worktree=' "$META" | cut -d= -f2- || true) T=$(grep '^window=' "$META" | cut -d= -f2-) PROJ=$(grep '^project=' "$META" | cut -d= -f2-) BACKEND=$(fm_backend_of_meta "$META") if [ "$BACKEND" = orca ]; then T_ORCA=$(grep '^terminal=' "$META" | tail -1 | cut -d= -f2- || true) - [ -n "$T_ORCA" ] && T=$T_ORCA + T=$T_ORCA + fm_backend_source orca || exit 1 + fm_backend_orca_authority_capabilities_check || exit 1 fi HOME_PATH=$(grep '^home=' "$META" | cut -d= -f2- || true) PR_URL=$(grep '^pr=' "$META" | tail -1 | cut -d= -f2- || true) @@ -149,26 +211,69 @@ if [ -n "$TASK_TMP" ] && [ "$TASK_TMP" != "/tmp/fm-$ID" ]; then fi ORCA_WORKTREE_ID=$(fm_meta_get "$META" orca_worktree_id) ORCA_PATH_MATCH_VERIFIED=0 -SECONDMATE_ENDPOINT_QUIESCED=0 +DIRECT_SPAWN_CLEANUP=$(fm_meta_get "$META" direct_spawn_cleanup) +DIRECT_SPAWN_BACKUP=$(fm_meta_get "$META" direct_spawn_backup) +DIRECT_SPAWN_ARTIFACTS=$(fm_meta_get "$META" direct_spawn_artifacts) +case "$DIRECT_SPAWN_CLEANUP" in + ''|pending) ;; + *) echo "error: invalid direct_spawn_cleanup metadata for $ID" >&2; exit 1 ;; +esac +ORCA_CLEANUP_PENDING_COUNT=$(grep -c '^orca_cleanup_pending=' "$META" 2>/dev/null || true) +ORCA_CLEANUP_PENDING=0 +if [ "$ORCA_CLEANUP_PENDING_COUNT" -ne 0 ]; then + if [ "$ORCA_CLEANUP_PENDING_COUNT" -ne 1 ] \ + || [ "$(fm_meta_get "$META" orca_cleanup_pending)" != 1 ] \ + || [ "$BACKEND" != orca ]; then + echo "error: invalid Orca cleanup quarantine metadata for $ID" >&2 + exit 1 + fi + case "$(fm_meta_get "$META" orca_cleanup_phase)" in + spawn-preparing|spawn-abort) ;; + *) + echo "error: invalid Orca cleanup quarantine phase for $ID" >&2 + exit 1 + ;; + esac + [ "$(fm_meta_get "$META" orca_expected_task)" = "fm-$ID" ] || { + echo "error: Orca cleanup quarantine is not bound to requested task $ID" >&2 + exit 1 + } + if [ -z "$ORCA_WORKTREE_ID" ]; then + [ "$(fm_meta_get "$META" orca_discovery_label)" = "fm-$ID" ] \ + && [ -n "$(fm_meta_get "$META" orca_provider_scope)" ] || { + echo "error: Orca cleanup quarantine discovery authority is unavailable for $ID" >&2 + exit 1 + } + fi + ORCA_CLEANUP_PENDING=1 +fi KIND=$(grep '^kind=' "$META" | cut -d= -f2- || true) [ -n "$KIND" ] || KIND=ship +[ "$KIND" = "$PRELOCK_KIND" ] || { + echo "error: task kind changed while teardown waited for lifecycle ownership" >&2 + exit 1 +} MODE=$(grep '^mode=' "$META" | cut -d= -f2- || true) [ -n "$MODE" ] || MODE=no-mistakes REPORT_GATED=0 REPORT_REQUIRED_COUNT=$(grep -c '^report_required=' "$META" 2>/dev/null || true) -if [ "$REPORT_REQUIRED_COUNT" -gt 0 ]; then +if [ "$BACKEND" = orca ] && [ "$REPORT_REQUIRED_COUNT" -ne 0 ]; then + echo "error: invalid report_required metadata for legacy Orca task $ID; the marker must be absent" >&2 + exit 1 +elif [ "$REPORT_REQUIRED_COUNT" -gt 0 ]; then if [ "$REPORT_REQUIRED_COUNT" -ne 1 ] || [ "$(fm_meta_get "$META" report_required)" != 1 ]; then echo "error: invalid report_required metadata for $ID; refusing teardown" >&2 exit 1 fi - if [ "$KIND" != secondmate ] && [ "$FORCE" != "--force" ]; then + if [ "$KIND" != secondmate ]; then REPORT_GATED=1 fi fi managed_endpoint_is_gone() { # <backend> <target> <expected-label> [probe-home] [recorded-scoped-target] - local backend=$1 target=$2 expected=$3 probe_home=${4:-} recorded_scoped_target=${5:-} attempt state last=unknown + local backend=$1 target=$2 expected=$3 probe_home=${4:-} recorded_scoped_target=${5:-} + local attempt state agent_state last=unknown [ -n "$target" ] || return 2 for attempt in 1 2 3 4 5 6 7 8 9 10; do if [ -n "$probe_home" ]; then @@ -178,7 +283,18 @@ managed_endpoint_is_gone() { # <backend> <target> <expected-label> [probe-home] fi case "$state" in absent) return 0 ;; - present|unknown) last=$state ;; + present) + if [ -n "$probe_home" ]; then + agent_state=$(unset FM_ROOT_OVERRIDE; FM_HOME="$probe_home" FM_ROOT="$probe_home" fm_backend_agent_alive "$backend" "$target" "$expected" "$recorded_scoped_target" 2>/dev/null) + else + agent_state=$(fm_backend_agent_alive "$backend" "$target" "$expected" "$recorded_scoped_target" 2>/dev/null) + fi + case "$agent_state" in + dead|alive) last=present ;; + *) last=unknown ;; + esac + ;; + unknown) last=unknown ;; *) last=unknown ;; esac sleep 0.1 @@ -197,19 +313,51 @@ managed_endpoint_blocker() { # <status> <task> [restored] fi } +teardown_backend_target_of_meta() { + local meta=$1 backend + backend=$(fm_backend_of_meta "$meta") + if [ "$backend" = orca ]; then + fm_meta_get "$meta" terminal || true + return 0 + else + fm_backend_target_of_meta "$meta" + fi +} + +quiesce_authoritative_orca_endpoint() { + local target=$1 worktree_id=$2 expected_label=$3 state + [ -n "$worktree_id" ] || return 1 + if [ -n "$target" ]; then + state=$(fm_backend_target_state orca "$target" "$expected_label" "$worktree_id") + case "$state" in + present|absent) ;; + *) return 1 ;; + esac + fi + fm_backend_quiesce_worktree_terminals orca "$worktree_id" "$expected_label" "$target" +} + quiesce_secondmate_endpoint() { local endpoint_home probe_home='' endpoint_status endpoint_home=$(fm_backend_endpoint_home "$BACKEND" "$KIND" "$FM_HOME" "$HOME_PATH") [ "$endpoint_home" = "$FM_HOME" ] || probe_home=$endpoint_home + if managed_endpoint_is_gone "$BACKEND" "$T" "fm-$ID" "$probe_home" "$(meta_value "$META" tmux_session_target)"; then + return 0 + fi if [ -n "$T" ]; then if [ -n "$probe_home" ]; then - ( unset FM_ROOT_OVERRIDE; FM_HOME="$probe_home" FM_ROOT="$probe_home" fm_backend_kill "$BACKEND" "$T" "$(meta_value "$META" zellij_tab_id)" "fm-$ID" "$(meta_value "$META" tmux_session_target)" ) 2>/dev/null || true + ( unset FM_ROOT_OVERRIDE; FM_HOME="$probe_home" FM_ROOT="$probe_home" fm_backend_kill "$BACKEND" "$T" "$(meta_value "$META" zellij_tab_id)" "fm-$ID" "$(meta_value "$META" tmux_session_target)" ) 2>/dev/null || { + echo "error: failed to stop secondmate endpoint for $ID; refusing child cleanup" >&2 + return 1 + } else - fm_backend_kill "$BACKEND" "$T" "$(meta_value "$META" zellij_tab_id)" "fm-$ID" "$(meta_value "$META" tmux_session_target)" 2>/dev/null || true + fm_backend_kill "$BACKEND" "$T" "$(meta_value "$META" zellij_tab_id)" "fm-$ID" "$(meta_value "$META" tmux_session_target)" 2>/dev/null || { + echo "error: failed to stop secondmate endpoint for $ID; refusing child cleanup" >&2 + return 1 + } fi fi if managed_endpoint_is_gone "$BACKEND" "$T" "fm-$ID" "$probe_home" "$(meta_value "$META" tmux_session_target)"; then - SECONDMATE_ENDPOINT_QUIESCED=1 return 0 fi endpoint_status=$? @@ -221,6 +369,60 @@ quiesce_secondmate_endpoint() { return 1 } +quiesce_child_endpoint() { + local meta=$1 task=$2 owner_home=$3 child_home=${4:-} + local backend target kind endpoint_home probe_home='' endpoint_status scoped_target + backend=$(fm_backend_of_meta "$meta") + target=$(teardown_backend_target_of_meta "$meta") + kind=$(meta_value "$meta" kind) + [ -n "$kind" ] || kind=ship + endpoint_home=$(fm_backend_endpoint_home "$backend" "$kind" "$owner_home" "$child_home") + [ "$endpoint_home" = "$FM_HOME" ] || probe_home=$endpoint_home + scoped_target=$(meta_value "$meta" tmux_session_target) + [ "$backend" != orca ] || scoped_target=$(meta_value "$meta" orca_worktree_id) + if [ "$backend" = orca ]; then + quiesce_authoritative_orca_endpoint "$target" "$scoped_target" "fm-$task" || { + echo "error: child Orca endpoint authority or quiescence is unproven for $task" >&2 + return 1 + } + return 0 + fi + if managed_endpoint_is_gone "$backend" "$target" "fm-$task" "$probe_home" "$scoped_target"; then + return 0 + else + endpoint_status=$? + fi + if [ "$endpoint_status" -eq 2 ]; then + echo "error: child endpoint identity or state for $task is unknown; refusing destructive cleanup" >&2 + return 1 + fi + [ -n "$target" ] || { + echo "error: child endpoint identity for $task is missing; refusing destructive cleanup" >&2 + return 1 + } + if [ -n "$probe_home" ]; then + ( unset FM_ROOT_OVERRIDE; FM_HOME="$probe_home" FM_ROOT="$probe_home" fm_backend_kill "$backend" "$target" "$(meta_value "$meta" zellij_tab_id)" "fm-$task" "$(meta_value "$meta" tmux_session_target)" ) 2>/dev/null || { + echo "error: failed to stop child endpoint for $task; refusing destructive cleanup" >&2 + return 1 + } + else + ( unset FM_ROOT_OVERRIDE; FM_HOME="$owner_home" FM_ROOT="$owner_home" fm_backend_kill "$backend" "$target" "$(meta_value "$meta" zellij_tab_id)" "fm-$task" "$(meta_value "$meta" tmux_session_target)" ) 2>/dev/null || { + echo "error: failed to stop child endpoint for $task; refusing destructive cleanup" >&2 + return 1 + } + fi + if managed_endpoint_is_gone "$backend" "$target" "fm-$task" "$probe_home" "$scoped_target"; then + return 0 + fi + endpoint_status=$? + if [ "$endpoint_status" -eq 2 ]; then + echo "error: child endpoint state for $task is unknown; refusing destructive cleanup" >&2 + else + echo "error: child endpoint for $task is still alive; refusing destructive cleanup" >&2 + fi + return 1 +} + quiesce_managed_account_endpoint() { # <meta> <task> [probe-home] local meta=$1 task=$2 probe_home=${3:-} meta_state lock profile backend target zellij_tab tmux_session_target endpoint_status meta_state=$(dirname "$meta") @@ -237,16 +439,33 @@ quiesce_managed_account_endpoint() { # <meta> <task> [probe-home] return 1 fi backend=$(fm_backend_of_meta "$meta") - target=$(fm_backend_target_of_meta "$meta") + target=$(teardown_backend_target_of_meta "$meta") zellij_tab=$(fm_meta_get "$meta" zellij_tab_id) tmux_session_target=$(fm_meta_get "$meta" tmux_session_target) [ -n "$tmux_session_target" ] || tmux_session_target=$(fm_meta_get "$meta" window) fm_account_meta_lock_release "$lock" || return 1 + if [ "$backend" = orca ]; then + tmux_session_target=$(fm_meta_get "$meta" orca_worktree_id) + quiesce_authoritative_orca_endpoint "$target" "$tmux_session_target" "fm-$task" || { + echo "error: managed Orca endpoint authority or quiescence is unproven for $task" >&2 + return 1 + } + return 0 + fi + if managed_endpoint_is_gone "$backend" "$target" "fm-$task" "$probe_home" "$tmux_session_target"; then + return 0 + fi if [ -n "$target" ]; then if [ -n "$probe_home" ]; then - ( unset FM_ROOT_OVERRIDE; FM_HOME="$probe_home" FM_ROOT="$probe_home" fm_backend_kill "$backend" "$target" "$zellij_tab" "fm-$task" "$tmux_session_target" ) 2>/dev/null || true + ( unset FM_ROOT_OVERRIDE; FM_HOME="$probe_home" FM_ROOT="$probe_home" fm_backend_kill "$backend" "$target" "$zellij_tab" "fm-$task" "$tmux_session_target" ) 2>/dev/null || { + echo "error: failed to stop managed endpoint for $task; retaining its Agent Fleet lease and metadata" >&2 + return 1 + } else - fm_backend_kill "$backend" "$target" "$zellij_tab" "fm-$task" "$tmux_session_target" 2>/dev/null || true + fm_backend_kill "$backend" "$target" "$zellij_tab" "fm-$task" "$tmux_session_target" 2>/dev/null || { + echo "error: failed to stop managed endpoint for $task; retaining its Agent Fleet lease and metadata" >&2 + return 1 + } fi fi if managed_endpoint_is_gone "$backend" "$target" "fm-$task" "$probe_home" "$tmux_session_target"; then @@ -339,6 +558,38 @@ default_branch() { return 1 } +LIVE_DEFAULT_BRANCH= +LIVE_DEFAULT_TIP= +LIVE_DEFAULT_OUTPUT= +probe_live_origin_default() { + local line ref status + LIVE_DEFAULT_BRANCH= + LIVE_DEFAULT_TIP= + if fm_run_bounded_capture --combine-stderr LIVE_DEFAULT_OUTPUT "$TEARDOWN_UPSTREAM_TIMEOUT" \ + git -C "$WT" ls-remote --symref origin HEAD; then + status=0 + else + status=$? + fi + [ "$status" -eq 0 ] && fm_process_tree_cleanup_verified || return 1 + while IFS= read -r line; do + case "$line" in + "ref: refs/heads/"*$'\t'"HEAD") + ref=${line#ref: refs/heads/} + LIVE_DEFAULT_BRANCH=${ref%$'\t'HEAD} + ;; + *$'\t'"HEAD") + LIVE_DEFAULT_TIP=${line%$'\t'HEAD} + ;; + esac + done <<EOF +$LIVE_DEFAULT_OUTPUT +EOF + [ -n "$LIVE_DEFAULT_BRANCH" ] \ + && [ -n "$LIVE_DEFAULT_TIP" ] \ + && git check-ref-format --branch "$LIVE_DEFAULT_BRANCH" >/dev/null 2>&1 +} + meta_value() { local meta=$1 key=$2 fm_meta_get "$meta" "$key" @@ -354,6 +605,41 @@ require_orca_worktree_id() { printf '%s\n' "$id" } +require_orca_task_metadata_identity() { + local meta=$1 expected_id=$2 window_count expected_count discovery_count scope_count provider_count provider_task + window_count=$(grep -c '^window=' "$meta" 2>/dev/null || true) + if [ "$window_count" -ne 1 ] || [ "$(meta_value "$meta" window)" != "fm-$expected_id" ]; then + echo "error: Orca metadata is not bound to requested task $expected_id" >&2 + return 1 + fi + expected_count=$(grep -c '^orca_expected_task=' "$meta" 2>/dev/null || true) + if [ "$expected_count" -ne 0 ] \ + && { [ "$expected_count" -ne 1 ] || [ "$(meta_value "$meta" orca_expected_task)" != "fm-$expected_id" ]; }; then + echo "error: Orca metadata expected-task authority drifted for $expected_id" >&2 + return 1 + fi + discovery_count=$(grep -c '^orca_discovery_label=' "$meta" 2>/dev/null || true) + if [ "$discovery_count" -ne 0 ] \ + && { [ "$discovery_count" -ne 1 ] || [ "$(meta_value "$meta" orca_discovery_label)" != "fm-$expected_id" ]; }; then + echo "error: Orca metadata discovery-label authority drifted for $expected_id" >&2 + return 1 + fi + scope_count=$(grep -c '^orca_provider_scope=' "$meta" 2>/dev/null || true) + if [ "$scope_count" -ne 0 ] \ + && { [ "$scope_count" -ne 1 ] \ + || [ "$(meta_value "$meta" orca_provider_scope)" != "repo-path:$(meta_value "$meta" project)" ]; }; then + echo "error: Orca metadata provider scope is unavailable for $expected_id" >&2 + return 1 + fi + provider_count=$(grep -c '^orca_provider_task=' "$meta" 2>/dev/null || true) + provider_task=$(meta_value "$meta" orca_provider_task) + if [ "$provider_count" -ne 0 ] \ + && { [ "$provider_count" -ne 1 ] || { [ -n "$provider_task" ] && [ "$provider_task" != "fm-$expected_id" ]; }; }; then + echo "error: Orca metadata provider-task authority drifted for $expected_id" >&2 + return 1 + fi +} + require_orca_terminal() { local meta=$1 terminal terminal=$(meta_value "$meta" terminal) @@ -365,8 +651,14 @@ require_orca_terminal() { } if [ "$BACKEND" = orca ] && [ "$KIND" != secondmate ]; then - ORCA_WORKTREE_ID=$(require_orca_worktree_id "$META") || exit 1 + ORCA_WORKTREE_ID=$(meta_value "$META" orca_worktree_id) + if [ "$ORCA_CLEANUP_PENDING" != 1 ] && [ -z "$ORCA_WORKTREE_ID" ]; then + ORCA_WORKTREE_ID=$(require_orca_worktree_id "$META") || exit 1 + fi T_ORCA=$(meta_value "$META" terminal) + if [ "$ORCA_CLEANUP_PENDING" != 1 ] && [ -z "$T_ORCA" ]; then + T_ORCA=$(require_orca_terminal "$META") || exit 1 + fi [ -z "$T_ORCA" ] || T=$T_ORCA fi @@ -476,29 +768,77 @@ pr_is_merged() { unpushed_patches_are_in_pr_head "$head" } -# Is the branch's content already present in the up-to-date default branch? Fetches -# first, then 3-way merges the default branch with HEAD: when HEAD introduces nothing -# the default branch does not already contain (e.g. its change landed via squash) the -# merged tree equals the default branch's tree. This isolates branch-only changes, so -# unrelated commits the default branch gained past the merge-base do not count as -# "added". Returns non-zero when inconclusive (no default ref, or a merge conflict), -# so the caller refuses rather than guesses. +# Is the branch's content already present in the up-to-date default branch? +# Origin-backed proof holds the common checkout lock across probe, fetch, +# unchanged branch-and-tip re-probe, and tree comparison. +content_matches_ref() { + local ref=$1 default_tree merged_tree + default_tree=$(git -C "$WT" rev-parse --quiet --verify "$ref^{tree}" 2>/dev/null) || return 1 + [ -n "$default_tree" ] || return 1 + merged_tree=$(git -C "$WT" merge-tree --write-tree "$ref" HEAD 2>/dev/null) || return 1 + merged_tree=$(printf '%s\n' "$merged_tree" | head -1) + if [ "$merged_tree" != "$default_tree" ]; then + echo "teardown: task content is not present in authoritative $ref; retaining $WT" >&2 + return 1 + fi + return 0 +} + +content_in_origin_default() { + local initial_branch initial_tip ref fetched fetch_output reason fetch_status + if ! probe_live_origin_default; then + reason=$(printf '%s\n' "$LIVE_DEFAULT_OUTPUT" | sed -n '1s/[[:space:]]\{1,\}/ /g;1p') + echo "teardown: cannot prove the live origin default for $PROJ${reason:+: $reason}; retaining $WT" >&2 + return 1 + fi + initial_branch=$LIVE_DEFAULT_BRANCH + initial_tip=$LIVE_DEFAULT_TIP + if fm_run_bounded_capture --combine-stderr fetch_output "$TEARDOWN_UPSTREAM_TIMEOUT" \ + git -C "$WT" fetch --quiet origin \ + "+refs/heads/$initial_branch:refs/remotes/origin/$initial_branch"; then + fetch_status=0 + else + fetch_status=$? + fi + if [ "$fetch_status" -ne 0 ] || ! fm_process_tree_cleanup_verified; then + reason=$(printf '%s\n' "$fetch_output" | sed -n '1s/[[:space:]]\{1,\}/ /g;1p') + echo "teardown: cannot fetch live origin/$initial_branch for landing proof${reason:+: $reason}; retaining $WT" >&2 + return 1 + fi + if ! probe_live_origin_default; then + reason=$(printf '%s\n' "$LIVE_DEFAULT_OUTPUT" | sed -n '1s/[[:space:]]\{1,\}/ /g;1p') + echo "teardown: cannot re-prove the live origin default after fetch${reason:+: $reason}; retaining $WT" >&2 + return 1 + fi + if [ "$LIVE_DEFAULT_BRANCH" != "$initial_branch" ] || [ "$LIVE_DEFAULT_TIP" != "$initial_tip" ]; then + echo "teardown: live origin default changed during landing proof ($initial_branch@$initial_tip -> $LIVE_DEFAULT_BRANCH@$LIVE_DEFAULT_TIP); retaining $WT" >&2 + return 1 + fi + ref="refs/remotes/origin/$initial_branch" + fetched=$(git -C "$WT" rev-parse --quiet --verify "$ref^{commit}" 2>/dev/null) || { + echo "teardown: cannot inspect fetched live origin/$initial_branch; retaining $WT" >&2 + return 1 + } + if [ "$fetched" != "$initial_tip" ]; then + echo "teardown: fetched origin/$initial_branch does not match live origin HEAD; retaining $WT" >&2 + return 1 + fi + content_matches_ref "$ref" +} + content_in_default() { - local name ref default_tree merged_tree - name=$(default_branch) || return 1 + local name ref if git -C "$WT" remote get-url origin >/dev/null 2>&1; then - git -C "$WT" fetch --quiet origin "+refs/heads/$name:refs/remotes/origin/$name" >/dev/null 2>&1 || return 1 - ref="refs/remotes/origin/$name" - elif git -C "$WT" rev-parse --quiet --verify "refs/heads/$name" >/dev/null 2>&1; then + fm_checkout_lock_run "$WT" "$CHECKOUT_LOCK_ROOT" content_in_origin_default + return + fi + name=$(default_branch) || return 1 + if git -C "$WT" rev-parse --quiet --verify "refs/heads/$name" >/dev/null 2>&1; then ref="refs/heads/$name" else return 1 fi - default_tree=$(git -C "$WT" rev-parse --quiet --verify "$ref^{tree}" 2>/dev/null) || return 1 - [ -n "$default_tree" ] || return 1 - merged_tree=$(git -C "$WT" merge-tree --write-tree "$ref" HEAD 2>/dev/null) || return 1 - merged_tree=$(printf '%s\n' "$merged_tree" | head -1) - [ "$merged_tree" = "$default_tree" ] + content_matches_ref "$ref" } # Has the worktree's committed work actually LANDED, though its commits are not @@ -596,9 +936,125 @@ inspectable_git_worktree() { canonical_existing_dir() { local target=$1 - [ -n "$target" ] || return 1 - [ -d "$target" ] || return 1 - ( cd "$target" && pwd -P ) + fm_checkout_trusted_dir "$target" +} + +exact_git_worktree_root() { + local target=$1 canonical top canonical_top + canonical=$(canonical_existing_dir "$target") || return 1 + top=$(git -C "$canonical" rev-parse --show-toplevel 2>/dev/null) || return 1 + canonical_top=$(canonical_existing_dir "$top") || return 1 + [ "$canonical" = "$canonical_top" ] || return 1 + fm_checkout_validate_git_metadata "$canonical" >/dev/null || return 1 + printf '%s\n' "$canonical" +} + +treehouse_state_for_worktree() { + local worktree=$1 slot pool state + slot=$(canonical_existing_dir "$(dirname "$worktree")") || return 1 + pool=$(canonical_existing_dir "$(dirname "$slot")") || return 1 + state="$pool/treehouse-state.json" + [ -f "$state" ] && [ ! -L "$state" ] || return 1 + printf '%s\n' "$state" +} + +require_treehouse_task_lease() { + local worktree=$1 expected_holder=$2 state + state=$(treehouse_state_for_worktree "$worktree") || { + echo "error: cannot resolve authoritative Treehouse state for $worktree" >&2 + return 1 + } + python3 - "$state" "$worktree" "$expected_holder" <<'PY' +import json +import os +import sys + +state_path, expected_path, expected_holder = sys.argv[1:] +try: + with open(state_path, encoding="utf-8") as stream: + state = json.load(stream) + worktrees = state["worktrees"] + if not isinstance(worktrees, list): + raise TypeError("worktrees must be an array") + matches = [] + for entry in worktrees: + if not isinstance(entry, dict): + continue + path = entry.get("path") + if not isinstance(path, str) or not path: + continue + if os.path.realpath(path) == expected_path: + matches.append(entry) + if len(matches) != 1: + raise ValueError("expected exactly one matching worktree entry") + entry = matches[0] + if entry.get("leased") is not True: + raise ValueError("worktree is not durably leased") + if entry.get("lease_holder") != expected_holder: + raise ValueError( + f"lease holder is {entry.get('lease_holder')!r}, expected {expected_holder!r}" + ) + if entry.get("destroying") is True: + raise ValueError("worktree is already being destroyed") +except (OSError, ValueError, TypeError, KeyError, json.JSONDecodeError) as error: + print( + f"error: Treehouse ownership for {expected_path} is unprovable: {error}", + file=sys.stderr, + ) + raise SystemExit(1) +PY +} + +require_treehouse_return_authority() { + local worktree=$1 project=$2 worktree_root project_root worktree_common project_common + worktree_root=$(exact_git_worktree_root "$worktree") || return 1 + project_root=$(exact_git_worktree_root "$project") || return 1 + worktree_common=$(fm_checkout_git_common_dir "$worktree_root") || return 1 + project_common=$(fm_checkout_git_common_dir "$project_root") || return 1 + [ "$worktree_common" = "$project_common" ] || { + echo "error: Treehouse return target $worktree_root does not belong to $project_root" >&2 + return 1 + } + worktree_registered_for_project "$project_root" "$worktree_root" || { + echo "error: Treehouse return target $worktree_root is not registered to $project_root" >&2 + return 1 + } + require_treehouse_task_lease "$worktree_root" "$3" +} + +validate_teardown_target_identity() { + local project_root worktree_root project_common worktree_common + [ "$KIND" != secondmate ] || return 0 + require_safe_task_metadata || return 1 + project_root=$(exact_git_worktree_root "$PROJ") || { + echo "error: teardown project metadata is not an exact inspectable repository root: ${PROJ:-<missing>}" >&2 + return 1 + } + worktree_root=$(exact_git_worktree_root "$WT") || { + echo "error: teardown worktree metadata is not an exact inspectable repository root: ${WT:-<missing>}" >&2 + return 1 + } + [ "$project_root" != "$worktree_root" ] || { + echo "error: teardown worktree metadata resolves to the primary project root: $worktree_root" >&2 + return 1 + } + project_common=$(fm_checkout_git_common_dir "$project_root") || return 1 + worktree_common=$(fm_checkout_git_common_dir "$worktree_root") || return 1 + [ "$project_common" = "$worktree_common" ] || { + echo "error: teardown worktree does not belong to the recorded project: $worktree_root" >&2 + return 1 + } + if [ "$BACKEND" = orca ]; then + require_orca_task_metadata_identity "$META" "$ID" || return 1 + require_orca_worktree_path_match "$ORCA_WORKTREE_ID" "$worktree_root" || return 1 + ORCA_PATH_MATCH_VERIFIED=1 + return 0 + fi + worktree_registered_for_project "$project_root" "$worktree_root" || { + echo "error: teardown worktree is not registered to the recorded project: $worktree_root" >&2 + return 1 + } + require_treehouse_task_lease "$worktree_root" "firstmate-$ID" } retry_wait_secs_is_valid() { @@ -681,20 +1137,45 @@ cleanup_stale_lock_for_safety_check() { # Return a worktree/home via `treehouse return --force`, tolerating a transient or # stale git index.lock left by a killed crew process. See the script header. -teardown_treehouse_return() { - local dir=$1 cd_dir=$2 label=$3 post_cleanup_check=${4:-} - local out lock attempt=0 max_retries lock_desc +teardown_treehouse_return_locked() { + local dir=$1 cd_dir=$2 label=$3 expected_holder=$4 post_cleanup_check=${5:-} post_return_cleanup=${6:-} + local out lock attempt=0 max_retries lock_desc return_status return_branch= - # Capture stdout+stderr so non-lock failures stay visible and lock failures can - # be matched by signature even when the lock file is already gone mid-check. - if out=$( ( cd "$cd_dir" && treehouse return --force "$dir" ) 2>&1 ); then + require_treehouse_return_authority "$dir" "$cd_dir" "$expected_holder" || { + echo "teardown: $label return aborted because Treehouse task ownership changed" >&2 + return 1 + } + if [ -n "$post_cleanup_check" ] && ! "$post_cleanup_check" "$dir" "$cd_dir" "$expected_holder"; then + echo "teardown: $label return aborted because the final locked safety check failed" >&2 + return 1 + fi + require_treehouse_return_authority "$dir" "$cd_dir" "$expected_holder" || { + echo "teardown: $label return aborted because Treehouse task ownership changed during final safety checks" >&2 + return 1 + } + if [ -n "$post_return_cleanup" ]; then + return_branch=$(git -C "$dir" rev-parse --abbrev-ref HEAD 2>/dev/null) || { + echo "teardown: $label return aborted because the task branch cannot be inspected under lock" >&2 + return 1 + } + fi + validate_removal_tree_boundaries "$dir" "$label" || return 1 + if out=$(fm_checkout_treehouse_return_locked "$dir" "$CHECKOUT_LOCK_ROOT" "$cd_dir" 2>&1); then [ -n "$out" ] && printf '%s\n' "$out" + if [ -n "$post_return_cleanup" ]; then + "$post_return_cleanup" "$return_branch" "$dir" "$cd_dir" || return 1 + fi return 0 + else + return_status=$? fi [ -n "$out" ] && printf '%s\n' "$out" >&2 + if fm_checkout_treehouse_return_requires_retention "$return_status"; then + return "$return_status" + fi if ! treehouse_return_is_index_lock_error "$out"; then - return 1 + return "$return_status" fi lock=$(worktree_git_lock_path "$dir") || lock="" @@ -712,16 +1193,37 @@ teardown_treehouse_return() { echo "teardown: $label return failed with transient git lock ($lock_desc); waiting ${TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS}s and retrying ($attempt/${max_retries})" >&2 sleep "$TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS" - if out=$( ( cd "$cd_dir" && treehouse return --force "$dir" ) 2>&1 ); then + if ! require_treehouse_return_authority "$dir" "$cd_dir" "$expected_holder"; then + echo "teardown: $label return aborted because Treehouse task ownership changed" >&2 + return 1 + fi + if [ -n "$post_cleanup_check" ] && ! "$post_cleanup_check" "$dir" "$cd_dir" "$expected_holder"; then + echo "teardown: $label return aborted because the final locked safety check failed" >&2 + return 1 + fi + if ! require_treehouse_return_authority "$dir" "$cd_dir" "$expected_holder"; then + echo "teardown: $label return aborted because Treehouse task ownership changed during final safety checks" >&2 + return 1 + fi + validate_removal_tree_boundaries "$dir" "$label" || return 1 + if out=$(fm_checkout_treehouse_return_locked "$dir" "$CHECKOUT_LOCK_ROOT" "$cd_dir" 2>&1); then [ -n "$out" ] && printf '%s\n' "$out" + if [ -n "$post_return_cleanup" ]; then + "$post_return_cleanup" "$return_branch" "$dir" "$cd_dir" || return 1 + fi echo "teardown: $label return succeeded on retry; lock cleared on its own" >&2 return 0 + else + return_status=$? fi [ -n "$out" ] && printf '%s\n' "$out" >&2 + if fm_checkout_treehouse_return_requires_retention "$return_status"; then + return "$return_status" + fi if ! treehouse_return_is_index_lock_error "$out"; then echo "teardown: $label return failed with a non-lock error after retry; aborting" >&2 - return 1 + return "$return_status" fi done @@ -733,20 +1235,37 @@ teardown_treehouse_return() { if fm_lock_is_provably_stale "$lock" "$dir" "$STALE_WORKTREE_LOCK_AGE_SECS"; then rm -f "$lock" echo "teardown: removed provably-stale git lock $lock (age >= ${STALE_WORKTREE_LOCK_AGE_SECS}s, no live holder) and retrying $label return" >&2 + if ! require_treehouse_return_authority "$dir" "$cd_dir" "$expected_holder"; then + echo "teardown: $label return aborted after stale-lock cleanup because Treehouse task ownership changed" >&2 + return 1 + fi if [ -n "$post_cleanup_check" ]; then - if ! "$post_cleanup_check"; then + if ! "$post_cleanup_check" "$dir" "$cd_dir" "$expected_holder"; then echo "teardown: $label return aborted after stale-lock cleanup because safety checks failed" >&2 return 1 fi fi - if out=$( ( cd "$cd_dir" && treehouse return --force "$dir" ) 2>&1 ); then + if ! require_treehouse_return_authority "$dir" "$cd_dir" "$expected_holder"; then + echo "teardown: $label return aborted after stale-lock cleanup because Treehouse task ownership changed during safety checks" >&2 + return 1 + fi + validate_removal_tree_boundaries "$dir" "$label" || return 1 + if out=$(fm_checkout_treehouse_return_locked "$dir" "$CHECKOUT_LOCK_ROOT" "$cd_dir" 2>&1); then [ -n "$out" ] && printf '%s\n' "$out" + if [ -n "$post_return_cleanup" ]; then + "$post_return_cleanup" "$return_branch" "$dir" "$cd_dir" || return 1 + fi echo "teardown: $label return succeeded after stale-lock cleanup" >&2 return 0 + else + return_status=$? fi [ -n "$out" ] && printf '%s\n' "$out" >&2 echo "teardown: $label return still failing after stale-lock cleanup" >&2 - return 1 + if fm_checkout_treehouse_return_requires_retention "$return_status"; then + return "$return_status" + fi + return "$return_status" fi echo "teardown: $label return failed: git lock $lock_desc persisted across ${max_retries} retries (waiting ${TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS}s each) and is not provably stale (may belong to a live process); leaving it in place" >&2 @@ -757,30 +1276,52 @@ teardown_treehouse_return() { return 1 } +teardown_treehouse_return() { + local dir=$1 + fm_checkout_lock_run "$dir" "$CHECKOUT_LOCK_ROOT" teardown_treehouse_return_locked "$@" +} + +cleanup_returned_worktree() { + local branch=$1 worktree=$2 project=$3 + if [ "$branch" != "HEAD" ]; then + git -C "$project" branch -D "$branch" >/dev/null 2>&1 || true + fi + remove_worktree_compatibility_artifacts "$worktree" "returned worktree" +} + validate_worktree_teardown_safety() { - local dirty_raw dirty unpushed_raw unpushed DEFAULT unmerged_raw unmerged branch + local dirty_raw dirty unpushed_raw unpushed DEFAULT unmerged_raw unmerged branch stash_list [ -d "$WT" ] || return 0 - [ "$FORCE" != "--force" ] || return 0 case "$KIND" in - secondmate|scout) return 0 ;; + secondmate) return 0 ;; esac + stash_list=$(git -C "$WT" stash list 2>/dev/null) || { + echo "REFUSED: cannot inspect worktree $WT for retained stash history." >&2 + return 1 + } + [ -z "$stash_list" ] || { + echo "REFUSED: worktree $WT has retained stash history." >&2 + return 1 + } - if ! dirty_raw=$(git -C "$WT" status --porcelain 2>/dev/null); then + if ! dirty_raw=$(git -C "$WT" status --porcelain=v1 --untracked-files=all 2>/dev/null); then if worktree_safety_blocked_by_lock "uncommitted changes"; then return "$TEARDOWN_WORKTREE_SAFETY_LOCK_BLOCKED" fi echo "REFUSED: cannot inspect worktree $WT for uncommitted changes." >&2 - echo "Restore the git index state, or get the captain's explicit OK to discard, then --force." >&2 + echo "Restore the git index state, then retry teardown." >&2 return 1 fi - dirty=$(printf '%s\n' "$dirty_raw" | grep -vE '^\?\? (\.claude/|\.fm-grok-turnend$)' | head -1 || true) + dirty=$(printf '%s\n' "$dirty_raw" \ + | grep -vE '^\?\? (\.claude/settings\.local\.json|\.opencode/plugins/fm-turn-end\.js|\.fm-grok-turnend)$' \ + | head -1 || true) if ! unpushed_raw=$(git -C "$WT" log --oneline HEAD --not --remotes -- 2>/dev/null); then if worktree_safety_blocked_by_lock "commits not on a remote"; then return "$TEARDOWN_WORKTREE_SAFETY_LOCK_BLOCKED" fi echo "REFUSED: cannot inspect worktree $WT for commits not on a remote." >&2 - echo "Restore the git index state, or get the captain's explicit OK to discard, then --force." >&2 + echo "Restore the git index state, then retry teardown." >&2 return 1 fi unpushed=$(printf '%s\n' "$unpushed_raw" | head -5) @@ -792,7 +1333,7 @@ validate_worktree_teardown_safety() { return "$TEARDOWN_WORKTREE_SAFETY_LOCK_BLOCKED" fi echo "REFUSED: cannot inspect worktree $WT for commits not on $DEFAULT." >&2 - echo "Restore the git index state, or get the captain's explicit OK to discard, then --force." >&2 + echo "Restore the git index state, then retry teardown." >&2 return 1 fi unmerged=$(printf '%s\n' "$unmerged_raw" | head -5) @@ -800,13 +1341,13 @@ validate_worktree_teardown_safety() { echo "REFUSED: local-only worktree $WT has work not yet merged into $DEFAULT and not on any remote." >&2 [ -n "$dirty" ] && echo "uncommitted changes present" >&2 [ -n "$unmerged" ] && printf 'commits not yet on %s:\n%s\n' "$DEFAULT" "$unmerged" >&2 - echo "Merge the branch into local $DEFAULT first (bin/fm-merge-local.sh after the captain approves), or push to a fork/remote, or get the captain's explicit OK to discard, then --force." >&2 + echo "Merge the branch into local $DEFAULT first (bin/fm-merge-local.sh after the captain approves), or push it to a fork or remote, then retry teardown." >&2 return 1 fi elif [ -n "$dirty" ]; then echo "REFUSED: worktree $WT has uncommitted changes." >&2 echo "uncommitted changes present" >&2 - echo "Commit them (or get the captain's explicit OK to discard, then --force)." >&2 + echo "Commit and land them, then retry teardown." >&2 return 1 elif [ -n "$unpushed" ]; then branch=${TEARDOWN_WORKTREE_BRANCH_FOR_SAFETY:-} @@ -817,14 +1358,43 @@ validate_worktree_teardown_safety() { if ! work_is_landed "$branch"; then echo "REFUSED: worktree $WT has work not on any remote and not landed." >&2 printf 'unpushed commits:\n%s\n' "$unpushed" >&2 - echo "Push the branch, land its PR, or get the captain's explicit OK to discard, then --force." >&2 + echo "Push the branch or land its PR, then retry teardown." >&2 return 1 fi fi } +validate_child_worktree_landed_state() { + local child_meta=$1 child_id=$2 child_worktree=$3 child_project=$4 + local stash_list + local WT=$child_worktree PROJ=$child_project ID=$child_id KIND=ship FORCE= + local MODE PR_URL TEARDOWN_WORKTREE_BRANCH_FOR_SAFETY= + MODE=$(meta_value "$child_meta" mode) + [ -n "$MODE" ] || MODE=no-mistakes + PR_URL=$(meta_value "$child_meta" pr) + stash_list=$(git -C "$WT" stash list 2>/dev/null) || { + echo "REFUSED: child worktree stash state is uninspectable at $WT" >&2 + return 1 + } + [ -z "$stash_list" ] || { + echo "REFUSED: child worktree has retained stash history at $WT" >&2 + return 1 + } + validate_worktree_teardown_safety +} + +CHILD_RETURN_META= +CHILD_RETURN_ID= +validate_child_worktree_return_safety() { + local child_worktree=$1 child_project=$2 + [ -n "$CHILD_RETURN_META" ] && [ -n "$CHILD_RETURN_ID" ] || return 1 + validate_child_worktree_landed_state "$CHILD_RETURN_META" "$CHILD_RETURN_ID" "$child_worktree" "$child_project" +} + require_orca_worktree_path_match() { local worktree_id=$1 inspected=$2 resolved inspected_abs resolved_abs + fm_backend_source orca || return 1 + fm_backend_orca_authority_capabilities_check || return 1 resolved=$(fm_backend_worktree_path orca "$worktree_id") || { echo "REFUSED: cannot resolve Orca worktree id $worktree_id to a path; preserving metadata." >&2 return 1 @@ -851,15 +1421,20 @@ require_orca_worktree_path_match_if_present() { } firstmate_home_has_treehouse_slot() { - local home=$1 - worktree_registered_for_project "$FM_ROOT" "$home" + local home=$1 expected_source=${2:-$FM_ROOT} + worktree_registered_for_project "$expected_source" "$home" } validate_removal_target() { local target=$1 label=$2 abs_target abs_home abs_root - [ -n "$target" ] || return 0 - [ -e "$target" ] || return 0 - abs_target=$(removal_target_abs_path "$target") + [ -n "$target" ] || { + echo "REFUSED: missing $label removal target" >&2 + return 1 + } + abs_target=$(fm_checkout_trusted_dir "$target") || { + echo "REFUSED: missing, redirected, or uninspectable $label removal target $target" >&2 + return 1 + } if abs_home=$(cd "$FM_HOME" 2>/dev/null && pwd -P); then : else @@ -896,27 +1471,93 @@ validate_removal_target() { printf '%s\n' "$abs_target" } -registered_descendant_home_for_removal() { - local reg=$1 target=$2 line id registered_home registered_abs - [ -f "$reg" ] || return 1 - while IFS= read -r line; do - case "$line" in - "- "*) - id=${line#- } - id=${id%% *} - registered_home=$(printf '%s\n' "$line" | registry_home_for_line) - [ -n "$registered_home" ] || continue - registered_abs=$(removal_target_abs_path "$registered_home" 2>/dev/null || true) - [ -n "$registered_abs" ] || continue - [ "$registered_abs" = "$target" ] && continue - if path_is_ancestor_of "$target" "$registered_abs"; then - printf '%s\t%s\n' "$id" "$registered_abs" - return 0 - fi - ;; - esac - done < "$reg" - return 1 +secondmate_registry_for_source() { + local source=$1 source_root root + source_root=$(exact_git_worktree_root "$source") || return 1 + root=$(exact_git_worktree_root "$FM_ROOT") || return 1 + if [ "$source_root" = "$root" ]; then + printf '%s\n' "$SECONDMATE_REG" + else + printf '%s/data/secondmates.md\n' "$source_root" + fi +} + +require_registered_secondmate_home() { + local reg=$1 expected_id=$2 expected_home=$3 registered expected_key registered_key + if ! registered=$(fm_secondmate_registry_query "$reg" query "$expected_id" home); then + if [ "$reg" = "$PREPARED_REGISTRY_PATH" ] \ + && [ "$expected_id" = "$PREPARED_REGISTRY_ID" ] \ + && [ "$expected_home" = "$PREPARED_REGISTRY_HOME" ] \ + && [ -n "$PREPARED_REGISTRY_BACKUP" ] \ + && fm_account_lifecycle_lock_owned "$PREPARED_REGISTRY_LOCK"; then + registered=$(fm_secondmate_registry_query "$PREPARED_REGISTRY_BACKUP" query "$expected_id" home) || return 1 + else + echo "REFUSED: secondmate registry is malformed, duplicated, redirected, or missing $expected_id at $reg" >&2 + return 1 + fi + fi + expected_key=$(fm_checkout_stable_path_key "$expected_home" directory 0 24) || return 1 + registered_key=$(fm_checkout_stable_path_key "$registered" directory 0 24) || return 1 + [ "$expected_key" = "$registered_key" ] || { + echo "REFUSED: secondmate registry home for $expected_id does not match $expected_home" >&2 + return 1 + } +} + +secondmate_state_metadata() { + local home=$1 state + state="$home/state" + command -v python3 >/dev/null 2>&1 || { + echo "REFUSED: python3 is required to inspect secondmate child state" >&2 + return 1 + } + python3 - "$home" "$state" <<'PY' +import os +import stat +import sys + +home, state_path = sys.argv[1:] +try: + if os.path.islink(state_path): + raise OSError("state directory must not be a symlink") + metadata = os.stat(state_path) + permissions = stat.S_IMODE(metadata.st_mode) + if not stat.S_ISDIR(metadata.st_mode): + raise NotADirectoryError(state_path) + if not permissions & 0o444 or not permissions & 0o111: + raise PermissionError("state directory is unreadable") + home_root = os.path.realpath(home) + state_root = os.path.realpath(state_path) + if state_root != os.path.join(home_root, "state"): + raise OSError("state directory resolves outside its secondmate home") + with os.scandir(state_path) as entries: + for entry in sorted(entries, key=lambda item: item.name): + metadata = entry.stat(follow_symlinks=False) + if not entry.name.endswith(".meta"): + continue + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode): + raise OSError(f"unsafe child metadata entry: {entry.path}") + if not stat.S_IMODE(metadata.st_mode) & 0o444: + raise PermissionError(f"unreadable child metadata entry: {entry.path}") + if any(character in entry.path for character in ("\n", "\r")): + raise OSError("child metadata path contains unsupported control characters") + print(entry.path) +except OSError as error: + print( + f"REFUSED: secondmate child state is unprovable at {state_path}: {error}", + file=sys.stderr, + ) + raise SystemExit(1) +PY +} + +require_empty_secondmate_state() { + local home=$1 child_metas + child_metas=$(secondmate_state_metadata "$home") || return 1 + [ -z "$child_metas" ] || { + echo "REFUSED: secondmate child metadata appeared before home removal at $home/state" >&2 + return 1 + } } validate_firstmate_operational_dirs_for_removal() { @@ -945,44 +1586,1794 @@ validate_firstmate_operational_dirs_for_removal() { } validate_child_worktree_for_removal() { - local target=$1 project=$2 abs_target abs_home abs_root - [ -n "$target" ] || return 0 - [ -e "$target" ] || return 0 + local target=$1 project=$2 abs_target abs_project abs_home abs_root target_common project_common + [ -n "$target" ] && [ -e "$target" ] || { + echo "REFUSED: missing child worktree removal target ${target:-<empty>}" >&2 + return 1 + } abs_target=$(validate_removal_target "$target" "child worktree") || return 1 + abs_target=$(exact_git_worktree_root "$abs_target") || { + echo "REFUSED: unsafe child worktree removal target $target is not an exact Git root" >&2 + return 1 + } + abs_project=$(exact_git_worktree_root "$project") || { + echo "REFUSED: child project metadata $project is not an exact Git root" >&2 + return 1 + } + [ "$abs_target" != "$abs_project" ] || { + echo "REFUSED: child worktree removal target resolves to its backing project root: $abs_target" >&2 + return 1 + } if abs_home=$(cd "$FM_HOME" 2>/dev/null && pwd -P); then if path_is_ancestor_of "$abs_home" "$abs_target"; then echo "REFUSED: unsafe child worktree removal target $target is inside the active firstmate home" >&2 return 1 fi fi - abs_root=$(cd "$FM_ROOT" && pwd -P) - if path_is_ancestor_of "$abs_root" "$abs_target"; then - echo "REFUSED: unsafe child worktree removal target $target is inside the firstmate repo" >&2 + abs_root=$(cd "$FM_ROOT" && pwd -P) + if path_is_ancestor_of "$abs_root" "$abs_target"; then + echo "REFUSED: unsafe child worktree removal target $target is inside the firstmate repo" >&2 + return 1 + fi + target_common=$(fm_checkout_git_common_dir "$abs_target") || return 1 + project_common=$(fm_checkout_git_common_dir "$abs_project") || return 1 + if [ "$target_common" != "$project_common" ] \ + || ! worktree_registered_for_project "$abs_project" "$abs_target"; then + echo "REFUSED: unsafe child worktree removal target $target is not a git worktree for $abs_project" >&2 + return 1 + fi + printf '%s\n' "$abs_target" +} + +validate_removal_tree_boundaries() { + local target=$1 label=$2 + removal_tree_operation "$target" "$label" validate +} + +removal_tree_boundary_token() { + local target=$1 label=$2 + validate_removal_tree_boundaries "$target" "$label" || return 1 + fm_checkout_tree_boundary_token "$target" +} + +removal_tree_operation() { + local target=$1 label=$2 operation=$3 + FM_REMOVAL_BOUNDARY_LABEL=$label python3 - "$target" "$operation" <<'PY' +import os +import stat +import sys + +raw_root = sys.argv[1] +operation = sys.argv[2] +label = os.environ["FM_REMOVAL_BOUNDARY_LABEL"] +if not raw_root or "\x00" in raw_root or "\n" in raw_root or "\r" in raw_root: + print(f"REFUSED: {label} removal target is empty or malformed", file=sys.stderr) + raise SystemExit(1) +root = os.path.normpath(os.path.abspath(raw_root)) +injected = "" +if os.environ.get("FM_ACCOUNT_ROUTING_TEST_LAB") == "firstmate-account-routing-test-lab-v1": + injected = os.environ.get("FM_TEARDOWN_TEST_MOUNT_PATH", "") + if injected: + injected = os.path.realpath(injected) + +def mountinfo_paths(): + paths = set() + path = "/proc/self/mountinfo" + if not os.path.exists(path): + return paths + with open(path, "r", encoding="utf-8") as stream: + for line in stream: + fields = line.rstrip("\n").split() + if len(fields) < 5: + raise OSError("malformed mount table") + value = fields[4] + for encoded, decoded in ( + ("\\040", " "), + ("\\011", "\t"), + ("\\012", "\n"), + ("\\134", "\\"), + ): + value = value.replace(encoded, decoded) + paths.add(os.path.realpath(value)) + return paths + +def identity(metadata): + return metadata.st_dev, metadata.st_ino + +try: + if operation not in ("validate", "remove") or root == os.path.sep: + raise OSError("invalid removal operation") + current = os.path.sep + for component in root.split(os.path.sep): + if not component: + continue + current = os.path.join(current, component) + metadata = os.lstat(current) + if stat.S_ISLNK(metadata.st_mode): + raise OSError(f"redirected path component {current}") + parent = os.path.dirname(root) + name = os.path.basename(root) + parent_flags = os.O_RDONLY | os.O_DIRECTORY + directory_flags = parent_flags + if hasattr(os, "O_NOFOLLOW"): + parent_flags |= os.O_NOFOLLOW + directory_flags |= os.O_NOFOLLOW + parent_fd = os.open(parent, parent_flags) + parent_metadata = os.fstat(parent_fd) + try: + root_metadata = os.stat(name, dir_fd=parent_fd, follow_symlinks=False) + except FileNotFoundError: + os.close(parent_fd) + raise OSError("removal root disappeared") + if stat.S_ISLNK(root_metadata.st_mode) or not stat.S_ISDIR(root_metadata.st_mode): + os.close(parent_fd) + raise OSError("removal root is not a real directory") + root_fd = os.open(name, directory_flags, dir_fd=parent_fd) + opened_root_metadata = os.fstat(root_fd) + if identity(root_metadata) != identity(opened_root_metadata): + os.close(root_fd) + os.close(parent_fd) + raise OSError("removal root identity changed") + root_device = root_metadata.st_dev + mounted_paths = mountinfo_paths() + + def reject_boundary(path, metadata, is_root=False): + canonical = os.path.realpath(path) + if canonical == injected: + raise OSError(f"mounted path {canonical}") + if metadata.st_dev != root_device: + raise OSError(f"filesystem device boundary at {canonical}") + if canonical in mounted_paths or os.path.ismount(canonical): + raise OSError(f"mount boundary at {canonical}") + if is_root and metadata.st_dev != parent_metadata.st_dev: + raise OSError(f"removal root filesystem boundary at {canonical}") + + def traverse(directory_fd, path): + reject_boundary(path, os.fstat(directory_fd)) + for entry in sorted(os.listdir(directory_fd)): + metadata = os.stat(entry, dir_fd=directory_fd, follow_symlinks=False) + entry_path = os.path.join(path, entry) + if stat.S_ISLNK(metadata.st_mode): + if operation == "remove": + current = os.stat(entry, dir_fd=directory_fd, follow_symlinks=False) + if identity(current) != identity(metadata): + raise OSError(f"symlink identity changed at {entry_path}") + os.unlink(entry, dir_fd=directory_fd) + continue + if stat.S_ISDIR(metadata.st_mode): + child_fd = os.open(entry, directory_flags, dir_fd=directory_fd) + try: + opened = os.fstat(child_fd) + if identity(metadata) != identity(opened): + raise OSError(f"directory identity changed at {entry_path}") + reject_boundary(entry_path, opened) + traverse(child_fd, entry_path) + if operation == "remove": + current = os.stat(entry, dir_fd=directory_fd, follow_symlinks=False) + if identity(current) != identity(opened): + raise OSError(f"directory identity changed at {entry_path}") + os.rmdir(entry, dir_fd=directory_fd) + finally: + os.close(child_fd) + continue + reject_boundary(entry_path, metadata) + if operation == "remove": + current = os.stat(entry, dir_fd=directory_fd, follow_symlinks=False) + if identity(current) != identity(metadata): + raise OSError(f"entry identity changed at {entry_path}") + os.unlink(entry, dir_fd=directory_fd) + + try: + reject_boundary(root, opened_root_metadata, True) + traverse(root_fd, root) + if operation == "remove": + current = os.stat(name, dir_fd=parent_fd, follow_symlinks=False) + if identity(current) != identity(opened_root_metadata): + raise OSError("removal root identity changed before release") + reject_boundary(root, current, True) + os.rmdir(name, dir_fd=parent_fd) + finally: + os.close(root_fd) + os.close(parent_fd) +except OSError as error: + print( + f"REFUSED: {label} removal crosses an untrusted filesystem boundary: {error}", + file=sys.stderr, + ) + raise SystemExit(1) +PY +} + +safe_rm_rf() { + local target=$1 label=$2 canonical + canonical=$(validate_removal_target "$target" "$label") || return 1 + removal_tree_operation "$canonical" "$label" remove +} + +safe_rm_rf_child_worktree() { + local target=$1 project=$2 canonical + canonical=$(validate_child_worktree_for_removal "$target" "$project") || return 1 + removal_tree_operation "$canonical" "child worktree" remove +} + +safe_remove_task_tmp() { + local target=$1 base + [ -n "$target" ] || return 0 + [ "$target" = "/tmp/fm-$ID" ] || return 1 + base=$(python3 - <<'PY' +import os +import stat + +base = os.path.realpath("/tmp") +if base not in ("/tmp", "/private/tmp"): + raise SystemExit(1) +current = os.path.sep +for component in base.split(os.path.sep): + if not component: + continue + current = os.path.join(current, component) + metadata = os.lstat(current) + if stat.S_ISLNK(metadata.st_mode): + raise SystemExit(1) +if not stat.S_ISDIR(os.lstat(base).st_mode): + raise SystemExit(1) +print(base) +PY + ) || return 1 + removal_tree_operation "$base/fm-$ID" "task temp root" remove +} + +remove_worktree_compatibility_artifacts() { + local target=$1 label=$2 + FM_COMPATIBILITY_CLEANUP_LABEL=$label python3 - "$target" <<'PY' +import os +import stat +import sys + +root = os.path.normpath(os.path.abspath(sys.argv[1])) +label = os.environ["FM_COMPATIBILITY_CLEANUP_LABEL"] +if not sys.argv[1] or root == os.path.sep: + raise SystemExit(1) +try: + current = os.path.sep + for component in root.split(os.path.sep): + if not component: + continue + current = os.path.join(current, component) + metadata = os.lstat(current) + if stat.S_ISLNK(metadata.st_mode): + raise OSError(f"redirected path component {current}") +except FileNotFoundError: + raise SystemExit(0) +flags = os.O_RDONLY | os.O_DIRECTORY +if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW +root_fd = os.open(root, flags) +try: + for relative in ( + ".claude/settings.local.json", + ".opencode/plugins/fm-turn-end.js", + ".fm-grok-turnend", + ): + parts = relative.split("/") + directory_fd = os.dup(root_fd) + try: + for component in parts[:-1]: + try: + child = os.open(component, flags, dir_fd=directory_fd) + except FileNotFoundError: + break + os.close(directory_fd) + directory_fd = child + else: + try: + metadata = os.stat( + parts[-1], + dir_fd=directory_fd, + follow_symlinks=False, + ) + except FileNotFoundError: + continue + if stat.S_ISDIR(metadata.st_mode): + raise OSError(f"compatibility artifact is a directory: {relative}") + os.unlink(parts[-1], dir_fd=directory_fd) + finally: + os.close(directory_fd) +except OSError as error: + print(f"REFUSED: {label} compatibility cleanup is unsafe: {error}", file=sys.stderr) + raise SystemExit(1) +finally: + os.close(root_fd) +PY +} + +repository_remote_identity() { + local repository=$1 remote=$2 candidate + case "$remote" in + file://*) candidate=${remote#file://} ;; + /*|./*|../*) candidate=$remote ;; + *) printf 'remote:%s\n' "$remote"; return 0 ;; + esac + case "$candidate" in + /*) ;; + *) candidate="$repository/$candidate" ;; + esac + candidate=$(canonical_existing_dir "$candidate") || return 1 + printf 'path:%s\n' "$candidate" +} + +validate_firstmate_home_repository_identity() { + local home=$1 expected_source=$2 home_root source_root home_common source_common home_origin source_origin + local home_identity source_identity source_path_identity + home_root=$(exact_git_worktree_root "$home") || { + echo "REFUSED: secondmate home repository identity is uninspectable at $home" >&2 + return 1 + } + source_root=$(exact_git_worktree_root "$expected_source") || return 1 + home_common=$(fm_checkout_git_common_dir "$home_root") || return 1 + source_common=$(fm_checkout_git_common_dir "$source_root") || return 1 + [ "$home_common" != "$source_common" ] || return 0 + home_origin=$(git -C "$home_root" remote get-url origin 2>/dev/null) || { + echo "REFUSED: secondmate home origin identity is unavailable at $home_root" >&2 + return 1 + } + home_identity=$(repository_remote_identity "$home_root" "$home_origin") || return 1 + source_path_identity="path:$source_root" + source_identity= + if source_origin=$(git -C "$source_root" remote get-url origin 2>/dev/null); then + source_identity=$(repository_remote_identity "$source_root" "$source_origin") || return 1 + fi + if [ "$home_identity" != "$source_path_identity" ] && [ "$home_identity" != "$source_identity" ]; then + echo "REFUSED: secondmate home repository identity does not match $source_root" >&2 + return 1 + fi +} + +validate_secondmate_home_landed_state() { + local home=$1 expected_source=$2 dirty unsafe branch default source_default_ref source_default_tip + local refs ref tip live_output live_branch live_tip cached_tip stash_list home_common source_common live_status reflog_tips + git_history_rewrite_state_is_clean "$home" "secondmate home repository" || return 1 + git_history_rewrite_state_is_clean "$expected_source" "secondmate top-level source repository" || return 1 + dirty=$(GIT_OPTIONAL_LOCKS=0 git -C "$home" status --porcelain=v1 --untracked-files=all 2>/dev/null) || { + echo "REFUSED: secondmate home cleanliness is uninspectable at $home" >&2 + return 1 + } + unsafe=$(printf '%s\n' "$dirty" | awk ' + $0 == "?? .claude/settings.local.json" { next } + $0 == "?? .opencode/plugins/fm-turn-end.js" { next } + $0 == "?? .fm-grok-turnend" { next } + $0 != "" { print } + ') + [ -z "$unsafe" ] || { + echo "REFUSED: secondmate home has unlanded changes at $home" >&2 + printf '%s\n' "$unsafe" >&2 + return 1 + } + if git -C "$expected_source" remote get-url origin >/dev/null 2>&1; then + if run_secondmate_remote_probe \ + live_output "$expected_source" "$home" --symref origin HEAD; then + live_status=0 + else + live_status=$? + fi + fm_process_tree_cleanup_verified || { + echo "REFUSED: secondmate home upstream probe cleanup is unverified for $expected_source" >&2 + return 1 + } + [ "$live_status" -eq 0 ] || { + echo "REFUSED: secondmate home live upstream default is uninspectable from $expected_source" >&2 + return 1 + } + live_branch=$(printf '%s\n' "$live_output" | sed -n 's/^ref: refs\/heads\/\([^[:space:]]*\)[[:space:]]*HEAD$/\1/p' | head -1) + live_tip=$(printf '%s\n' "$live_output" | awk '$2 == "HEAD" && $1 != "ref:" { print $1; exit }') + [ -n "$live_branch" ] && [ -n "$live_tip" ] || { + echo "REFUSED: secondmate home live upstream default identity is malformed" >&2 + return 1 + } + default=$live_branch + source_default_ref="refs/remotes/origin/$default" + cached_tip=$(git -C "$expected_source" rev-parse "$source_default_ref^{commit}" 2>/dev/null) || return 1 + [ "$cached_tip" = "$live_tip" ] || { + echo "REFUSED: secondmate home source default is stale against live origin/$default" >&2 + return 1 + } + elif git -C "$expected_source" show-ref --verify --quiet refs/heads/main; then + default=main + source_default_ref=refs/heads/main + elif git -C "$expected_source" show-ref --verify --quiet refs/heads/master; then + default=master + source_default_ref=refs/heads/master + else + echo "REFUSED: secondmate home default branch is unprovable from $expected_source" >&2 + return 1 + fi + if branch=$(git -C "$home" symbolic-ref --quiet --short HEAD 2>/dev/null); then + [ "$branch" = "$default" ] || { + echo "REFUSED: secondmate home is on non-default branch $branch at $home" >&2 + return 1 + } + fi + source_default_tip=$(GIT_NO_REPLACE_OBJECTS=1 git -C "$expected_source" rev-parse "$source_default_ref^{commit}" 2>/dev/null) || { + echo "REFUSED: secondmate home authoritative default tip is uninspectable at $expected_source" >&2 + return 1 + } + home_common=$(fm_checkout_git_common_dir "$home") || return 1 + source_common=$(fm_checkout_git_common_dir "$expected_source") || return 1 + refs= + stash_list=$(git -C "$home" stash list 2>/dev/null) || { + echo "REFUSED: secondmate home stash state is uninspectable at $home" >&2 + return 1 + } + [ -z "$stash_list" ] || { + echo "REFUSED: secondmate home has retained stash history at $home" >&2 + return 1 + } + if [ "$home_common" != "$source_common" ]; then + refs=$(GIT_NO_REPLACE_OBJECTS=1 git -C "$home" for-each-ref --format='%(refname)' 2>/dev/null) || { + echo "REFUSED: secondmate home refs are uninspectable at $home" >&2 + return 1 + } + fi + reflog_tips=$(GIT_NO_REPLACE_OBJECTS=1 git -C "$home" reflog --all --format='%H' 2>/dev/null) || { + echo "REFUSED: secondmate home reflogs are uninspectable at $home" >&2 + return 1 + } + refs=$(printf 'HEAD\n%s\n%s\n' "$refs" "$reflog_tips") + while IFS= read -r ref; do + [ -n "$ref" ] || continue + tip=$(GIT_NO_REPLACE_OBJECTS=1 git -C "$home" rev-parse "$ref^{commit}" 2>/dev/null) || { + echo "REFUSED: secondmate home ref $ref cannot be resolved to a commit" >&2 + return 1 + } + if ! GIT_NO_REPLACE_OBJECTS=1 git -C "$expected_source" \ + cat-file -e "$tip^{commit}" 2>/dev/null \ + || ! GIT_NO_REPLACE_OBJECTS=1 git -C "$expected_source" \ + merge-base --is-ancestor "$tip" "$source_default_tip" 2>/dev/null; then + echo "REFUSED: secondmate home ref $ref has commits not proven in authoritative $default" >&2 + return 1 + fi + done <<EOF +$refs +EOF +} + +exact_git_repository_root() { + local repository=$1 container=${2:-$1} bare git_dir common metadata top container_git + bare=$(git -C "$repository" rev-parse --is-bare-repository 2>/dev/null) || return 1 + if [ "$bare" = true ]; then + git_dir=$(git -C "$repository" rev-parse --absolute-git-dir 2>/dev/null) || return 1 + git_dir=$(fm_checkout_trusted_dir "$git_dir") || return 1 + [ "$git_dir" = "$repository" ] || return 1 + printf '%s\n' "$repository" + elif exact_git_worktree_root "$repository" >/dev/null 2>&1; then + exact_git_worktree_root "$repository" + else + top=$(git -C "$repository" rev-parse --show-toplevel 2>/dev/null) || return 1 + top=$(fm_checkout_trusted_dir "$top") || return 1 + [ "$top" = "$repository" ] || return 1 + metadata="$repository/.git" + [ -f "$metadata" ] && [ ! -L "$metadata" ] || return 1 + git_dir=$(git -C "$repository" rev-parse --absolute-git-dir 2>/dev/null) || return 1 + git_dir=$(fm_checkout_trusted_dir "$git_dir") || return 1 + common=$(git -C "$repository" rev-parse --git-common-dir 2>/dev/null) || return 1 + case "$common" in /*) ;; *) common="$repository/$common" ;; esac + common=$(fm_checkout_trusted_dir "$common") || return 1 + [ "$git_dir" = "$common" ] || return 1 + container=$(fm_checkout_trusted_dir "$container") || return 1 + container_git=$(git -C "$container" rev-parse --absolute-git-dir 2>/dev/null) || return 1 + container_git=$(fm_checkout_trusted_dir "$container_git") || return 1 + case "$common/" in "$container_git/modules/"*) ;; *) return 1 ;; esac + printf '%s\n' "$repository" + fi +} + +git_history_rewrite_state_is_clean() { + local repository=$1 label=$2 grafts replacements status + [ -z "${GIT_REPLACE_REF_BASE:-}" ] || { + echo "REFUSED: $label uses an ambient replacement-ref namespace" >&2 + return 1 + } + grafts=$(git -C "$repository" rev-parse --git-path info/grafts 2>/dev/null) || return 1 + case "$grafts" in /*) ;; *) grafts="$repository/$grafts" ;; esac + if [ -e "$grafts" ] || [ -L "$grafts" ]; then + echo "REFUSED: $label uses local grafted history at $grafts" >&2 + return 1 + fi + if replacements=$(git -C "$repository" for-each-ref --format='%(refname)' refs/replace 2>/dev/null); then + [ -z "$replacements" ] || { + echo "REFUSED: $label uses replacement refs" >&2 + return 1 + } + else + status=$? + [ "$status" -eq 0 ] || return 1 + fi +} + +secondmate_network_remote_identity() { + local repository=$1 url=$2 transport=$3 host=$4 user=$5 port=$6 + local ssh_output ssh_status resolved_host effective_user effective_port + local proxy_command proxy_jump local_command remote_command permit_local + local resolution resolution_status config_output config_status config_key + local -a ssh_args + case "$host" in + ''|-*|*$'\n'*|*$'\r'*|*$'\t'*) return 1 ;; + esac + [ -z "${GIT_SSH_COMMAND+x}" ] \ + && [ -z "${GIT_SSH+x}" ] \ + && [ -z "${GIT_SSH_VARIANT+x}" ] \ + && [ -z "${GIT_PROXY_COMMAND+x}" ] \ + && [ -z "${GIT_CONFIG_COUNT+x}" ] \ + && [ -z "${GIT_CONFIG_PARAMETERS+x}" ] || return 1 + for config_key in core.sshCommand ssh.variant core.gitProxy http.proxy \ + remote.origin.proxy remote.origin.uploadpack remote.origin.receivepack; do + if config_output=$(git -C "$repository" config --get-all "$config_key" 2>/dev/null); then + [ -z "$config_output" ] || return 1 + else + config_status=$? + [ "$config_status" -eq 1 ] || return 1 + fi + done + if config_output=$(git -C "$repository" config --get-regexp \ + '^(http\..*\.proxy|http\.proxy|remote\..*\.(proxy|uploadpack|receivepack))$' \ + 2>/dev/null); then + [ -z "$config_output" ] || return 1 + else + config_status=$? + [ "$config_status" -eq 1 ] || return 1 + fi + if config_output=$(git -C "$repository" config --get-regexp \ + '^url\..*\.[iI]nstead[Oo]f$' 2>/dev/null); then + [ -z "$config_output" ] || return 1 + else + config_status=$? + [ "$config_status" -eq 1 ] || return 1 + fi + resolved_host=$host + case "$transport" in + ssh|git+ssh) + [ -x /usr/bin/ssh ] || return 1 + ssh_args=(-G) + [ -z "$user" ] || ssh_args+=(-l "$user") + [ -z "$port" ] || ssh_args+=(-p "$port") + ssh_args+=("$host") + if fm_run_bounded_capture --combine-stderr ssh_output "$TEARDOWN_UPSTREAM_TIMEOUT" \ + /usr/bin/ssh "${ssh_args[@]}"; then + ssh_status=0 + else + ssh_status=$? + fi + fm_process_tree_cleanup_verified || return 1 + [ "$ssh_status" -eq 0 ] || return 1 + [ "$(printf '%s\n' "$ssh_output" | awk '$1 == "hostname" { count++ } END { print count + 0 }')" -eq 1 ] \ + || return 1 + resolved_host=$(printf '%s\n' "$ssh_output" | awk '$1 == "hostname" { print $2; exit }') + effective_user=$(printf '%s\n' "$ssh_output" | awk '$1 == "user" { print $2; exit }') + effective_port=$(printf '%s\n' "$ssh_output" | awk '$1 == "port" { print $2; exit }') + proxy_command=$(printf '%s\n' "$ssh_output" | awk '$1 == "proxycommand" { $1=""; sub(/^ /, ""); print; exit }') + proxy_jump=$(printf '%s\n' "$ssh_output" | awk '$1 == "proxyjump" { print $2; exit }') + local_command=$(printf '%s\n' "$ssh_output" | awk '$1 == "localcommand" { $1=""; sub(/^ /, ""); print; exit }') + remote_command=$(printf '%s\n' "$ssh_output" | awk '$1 == "remotecommand" { $1=""; sub(/^ /, ""); print; exit }') + permit_local=$(printf '%s\n' "$ssh_output" | awk '$1 == "permitlocalcommand" { print $2; exit }') + [ -z "$user" ] || [ "$effective_user" = "$user" ] || return 1 + [ -z "$port" ] || [ "$effective_port" = "$port" ] || return 1 + [ -z "$proxy_command" ] || [ "$proxy_command" = none ] || return 1 + [ -z "$proxy_jump" ] || [ "$proxy_jump" = none ] || return 1 + [ -z "$local_command" ] || [ "$local_command" = none ] || return 1 + [ -z "$remote_command" ] || [ "$remote_command" = none ] || return 1 + [ -z "$permit_local" ] || [ "$permit_local" = no ] || return 1 + ;; + http) effective_port=${port:-80} ;; + https) effective_port=${port:-443} ;; + git) effective_port=${port:-9418} ;; + *) return 1 ;; + esac + case "$resolved_host" in + ''|-*|*[!A-Za-z0-9._:-]*) return 1 ;; + esac + case "$effective_port" in + ''|*[!0-9]*) return 1 ;; + esac + if fm_run_bounded_capture --combine-stderr resolution "$TEARDOWN_UPSTREAM_TIMEOUT" \ + python3 - "$resolved_host" <<'PY' +import ipaddress +import os +import re +import socket +import subprocess +import sys + +host = sys.argv[1].lower().rstrip(".") + +def addresses(name): + values = set() + for result in socket.getaddrinfo(name, None, socket.AF_UNSPEC, socket.SOCK_STREAM): + value = result[4][0].split("%", 1)[0] + values.add(ipaddress.ip_address(value)) + return values + +def local_interface_addresses(): + values = set() + commands = ( + ("/sbin/ifconfig",), + ("/usr/sbin/ifconfig",), + ("/usr/sbin/ip", "-o", "addr", "show"), + ("/sbin/ip", "-o", "addr", "show"), + ) + output = None + for command in commands: + if not os.path.isfile(command[0]) or not os.access(command[0], os.X_OK): + continue + completed = subprocess.run( + command, + check=False, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + text=True, + timeout=5, + ) + if completed.returncode == 0: + output = completed.stdout + break + if output is None: + raise OSError("local interface inventory unavailable") + for match in re.findall(r"\binet6?\s+(?:addr:)?([0-9A-Fa-f:.]+)(?:%[^\s]+)?", output): + try: + values.add(ipaddress.ip_address(match)) + except ValueError: + pass + if not values: + raise OSError("local interface inventory is empty") + return values + +try: + local_names = { + "localhost", + socket.gethostname().lower().rstrip("."), + socket.getfqdn().lower().rstrip("."), + } + if host in local_names: + raise OSError("remote hostname names this machine") + injected = os.environ.get("FM_TEARDOWN_TEST_NETWORK_ADDRESSES", "") + if ( + os.environ.get("FM_ACCOUNT_ROUTING_TEST_LAB") + == "firstmate-account-routing-test-lab-v1" + and injected + ): + remote = { + ipaddress.ip_address(value.strip()) + for value in injected.split(",") + if value.strip() + } + else: + remote = addresses(host) + local = local_interface_addresses() + for name in tuple(local_names): + if not name: + continue + try: + local.update(addresses(name)) + except OSError: + pass + if not remote: + raise OSError("remote hostname has no addresses") + for address in remote: + if ( + address.is_loopback + or address.is_link_local + or address.is_unspecified + or address.is_multicast + or address in local + ): + raise OSError("remote hostname resolves to local or ephemeral storage") + print(",".join(sorted(str(address) for address in remote))) +except (OSError, ValueError, socket.gaierror, subprocess.SubprocessError): + raise SystemExit(1) +PY + then + resolution_status=0 + else + resolution_status=$? + fi + fm_process_tree_cleanup_verified || return 1 + [ "$resolution_status" -eq 0 ] && [ -n "$resolution" ] || return 1 + printf 'network\t%s\t%s\t%s\t%s\t%s\n' \ + "$transport" "$url" "$resolved_host" "$effective_port" "$resolution" +} + +validate_surviving_object_graph_bound() { + local repository=$1 objects=$2 retiring_home=$3 label=$4 + FM_SURVIVING_REPOSITORY_LABEL=$label python3 - "$repository" "$objects" "$retiring_home" <<'PY' +import os +import stat +import subprocess +import sys +import time + +repository, initial, retiring_home = map(os.path.realpath, sys.argv[1:]) +label = os.environ["FM_SURVIVING_REPOSITORY_LABEL"] +held = [] +visited = set() + +def confined(path): + try: + return os.path.commonpath((retiring_home, path)) == retiring_home + except ValueError: + return False + +def retain(path, expected_directory): + metadata = os.lstat(path) + if stat.S_ISLNK(metadata.st_mode): + raise OSError(f"redirected object storage entry: {path}") + if expected_directory and not stat.S_ISDIR(metadata.st_mode): + raise OSError(f"object storage directory is not a directory: {path}") + if not expected_directory and not stat.S_ISREG(metadata.st_mode): + raise OSError(f"object storage entry is not a regular file: {path}") + flags = os.O_RDONLY + if expected_directory: + flags |= os.O_DIRECTORY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + descriptor = os.open(path, flags) + opened = os.fstat(descriptor) + expected = ( + metadata.st_dev, + metadata.st_ino, + stat.S_IFMT(metadata.st_mode), + metadata.st_size, + ) + actual = ( + opened.st_dev, + opened.st_ino, + stat.S_IFMT(opened.st_mode), + opened.st_size, + ) + if expected != actual: + os.close(descriptor) + raise OSError(f"object storage identity changed at {path}") + if confined(os.path.realpath(path)): + os.close(descriptor) + raise OSError(f"object storage depends on retiring home: {path}") + held.append((path, descriptor, expected)) + return descriptor, opened + +def inspect(objects): + objects = os.path.normpath(objects) + directory, metadata = retain(objects, True) + identity = (metadata.st_dev, metadata.st_ino) + if identity in visited: + return + visited.add(identity) + object_device = metadata.st_dev + for name in sorted(os.listdir(directory)): + path = os.path.join(objects, name) + item = os.stat(name, dir_fd=directory, follow_symlinks=False) + if item.st_dev != object_device: + raise OSError(f"object storage crosses a filesystem boundary: {path}") + if stat.S_ISLNK(item.st_mode): + raise OSError(f"redirected object storage entry: {path}") + if stat.S_ISDIR(item.st_mode): + inspect(path) + elif stat.S_ISREG(item.st_mode): + retain(path, False) + else: + raise OSError(f"unsafe object storage entry: {path}") + alternates = os.path.join(objects, "info", "alternates") + http_alternates = os.path.join(objects, "info", "http-alternates") + if os.path.lexists(http_alternates): + raise OSError(f"HTTP alternates are not durable proof: {http_alternates}") + if not os.path.lexists(alternates): + return + alternate_fd, _ = retain(alternates, False) + with os.fdopen(os.dup(alternate_fd), "r", encoding="utf-8") as stream: + entries = [line.rstrip("\n") for line in stream] + if not entries or any(not entry or "\x00" in entry for entry in entries): + raise OSError(f"malformed alternates file: {alternates}") + for entry in entries: + if entry.startswith('"') or entry.endswith('"'): + raise OSError(f"quoted alternates are ambiguous: {alternates}") + candidate = entry if os.path.isabs(entry) else os.path.join(objects, entry) + inspect(os.path.realpath(candidate)) + +def verify_retained(): + for path, descriptor, expected in held: + metadata = os.lstat(path) + current = ( + metadata.st_dev, + metadata.st_ino, + stat.S_IFMT(metadata.st_mode), + metadata.st_size, + ) + opened = os.fstat(descriptor) + retained = ( + opened.st_dev, + opened.st_ino, + stat.S_IFMT(opened.st_mode), + opened.st_size, + ) + if current != expected or retained != expected or stat.S_ISLNK(metadata.st_mode): + raise OSError(f"object storage identity changed during graph proof: {path}") + +def run(arguments, input_data=None): + environment = os.environ.copy() + environment["GIT_NO_REPLACE_OBJECTS"] = "1" + completed = subprocess.run( + ["git", "-C", repository, *arguments], + check=False, + input=input_data, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + env=environment, + ) + verify_retained() + return completed + +try: + inspect(initial) + marker = os.environ.get("FM_TEARDOWN_TEST_OBJECT_SCAN_MARKER", "") + release = os.environ.get("FM_TEARDOWN_TEST_OBJECT_SCAN_RELEASE", "") + scan_root = os.environ.get("FM_TEARDOWN_TEST_OBJECT_SCAN_ROOT", "") + if ( + os.environ.get("FM_ACCOUNT_ROUTING_TEST_LAB") + == "firstmate-account-routing-test-lab-v1" + and marker + and release + and scan_root + and initial == os.path.realpath(scan_root) + ): + with open(marker, "w", encoding="utf-8"): + pass + deadline = time.monotonic() + 10 + while not os.path.exists(release): + if time.monotonic() >= deadline: + raise OSError("object storage test mutation did not release") + time.sleep(0.01) + verify_retained() + revision_arguments = ["rev-list", "--objects", "--missing=print", "--all", "--reflog"] + head = run(["rev-parse", "--verify", "HEAD^{object}"]) + if head.returncode == 0: + revision_arguments.append("HEAD") + objects = run(revision_arguments) + if objects.returncode != 0: + raise OSError("reachable objects cannot be enumerated") + object_ids = [] + for line in objects.stdout.splitlines(): + if not line: + continue + if line.startswith("?"): + raise OSError(f"promised object is missing: {line[1:].split()[0]}") + object_id = line.split(" ", 1)[0] + if not object_id: + raise OSError("malformed reachable-object inventory") + object_ids.append(object_id) + if not object_ids: + raise OSError("reachable-object inventory is empty") + checked = run( + ["cat-file", "--batch-check=%(objectname) %(objecttype) %(objectsize)"], + "\n".join(object_ids) + "\n", + ) + if checked.returncode != 0: + raise OSError("reachable objects cannot be inspected") + records = checked.stdout.splitlines() + if len(records) != len(object_ids): + raise OSError("reachable-object inspection is incomplete") + for requested, record in zip(object_ids, records): + fields = record.split() + if ( + len(fields) != 3 + or fields[0] != requested + or fields[1] in ("missing", "promisor") + or not fields[2].isdigit() + ): + raise OSError(f"reachable object is unavailable: {requested}") + checked = run(["fsck", "--full", "--strict", "--no-dangling"]) + if checked.returncode != 0: + raise OSError("required Git objects are incomplete") + verify_retained() +except (OSError, UnicodeError, subprocess.SubprocessError) as error: + print(f"REFUSED: {label} complete object graph is unavailable: {error}", file=sys.stderr) + raise SystemExit(1) +finally: + for _, descriptor, _ in reversed(held): + os.close(descriptor) +PY +} + +validate_surviving_repository_authority_locked() { + local repository=$1 retiring_home=$2 repository_container=${3:-$1} + local label=${4:-repository} common git_dir objects listed records path kind canonical + local allowed_retiring_worktree=${5:-} listed_common bare_repository count=0 refs ref + local shallow partial_status + [ "$(exact_git_repository_root "$repository" "$repository_container")" = "$repository" ] \ + || return 1 + git_history_rewrite_state_is_clean "$repository" "$label" || return 1 + [ -z "${GIT_OBJECT_DIRECTORY:-}" ] \ + && [ -z "${GIT_ALTERNATE_OBJECT_DIRECTORIES:-}" ] || { + echo "REFUSED: $label uses ambient Git object storage that cannot be proven durable" >&2 + return 1 + } + common=$(git -C "$repository" rev-parse --git-common-dir 2>/dev/null) || return 1 + case "$common" in /*) ;; *) common="$repository/$common" ;; esac + common=$(fm_checkout_trusted_dir "$common") || return 1 + git_dir=$(git -C "$repository" rev-parse --absolute-git-dir 2>/dev/null) || return 1 + git_dir=$(fm_checkout_trusted_dir "$git_dir") || return 1 + objects=$(git -C "$repository" rev-parse --git-path objects 2>/dev/null) || return 1 + case "$objects" in /*) ;; *) objects="$repository/$objects" ;; esac + objects=$(fm_checkout_trusted_dir "$objects") || return 1 + for path in "$common" "$git_dir" "$objects"; do + case "$path/" in + "$retiring_home/"*) + echo "REFUSED: $label Git storage depends on the retiring home at $path" >&2 + return 1 + ;; + esac + done + shallow=$(git -C "$repository" rev-parse --is-shallow-repository 2>/dev/null) || return 1 + [ "$shallow" = false ] || { + echo "REFUSED: $label is shallow and does not prove a complete surviving object graph" >&2 + return 1 + } + if git -C "$repository" config --get extensions.partialClone >/dev/null 2>&1; then + echo "REFUSED: $label uses promisor or partial-clone object semantics" >&2 + return 1 + else + partial_status=$? + [ "$partial_status" -eq 1 ] || return 1 + fi + if git -C "$repository" config --get-regexp \ + '^remote\..*\.(promisor|partialclonefilter)$' >/dev/null 2>&1; then + echo "REFUSED: $label uses promisor or partial-clone object semantics" >&2 + return 1 + else + partial_status=$? + [ "$partial_status" -eq 1 ] || return 1 + fi + FM_SURVIVING_OBJECTS_LABEL=$label python3 - "$objects" "$retiring_home" <<'PY' || return 1 +import os +import stat +import sys + +initial, retiring_home = map(os.path.realpath, sys.argv[1:]) +label = os.environ["FM_SURVIVING_OBJECTS_LABEL"] +visited = set() + +def confined_to_retiring_home(path): + try: + return os.path.commonpath((retiring_home, path)) == retiring_home + except ValueError: + return False + +def trusted_directory(path): + path = os.path.normpath(path) + if not os.path.isabs(path): + raise OSError("object directory is not absolute") + current = os.path.sep + for component in path.split(os.path.sep): + if not component: + continue + current = os.path.join(current, component) + metadata = os.lstat(current) + if stat.S_ISLNK(metadata.st_mode): + raise OSError(f"redirected object path {current}") + metadata = os.lstat(path) + if not stat.S_ISDIR(metadata.st_mode) or os.path.realpath(path) != path: + raise OSError(f"unsafe object directory {path}") + return path + +def inspect(objects): + objects = trusted_directory(objects) + metadata = os.lstat(objects) + identity = (metadata.st_dev, metadata.st_ino) + if identity in visited: + return + visited.add(identity) + if confined_to_retiring_home(objects): + raise OSError(f"object directory depends on retiring home: {objects}") + object_device = metadata.st_dev + pending = [objects] + while pending: + current = pending.pop() + with os.scandir(current) as entries: + for entry in entries: + entry_metadata = entry.stat(follow_symlinks=False) + if stat.S_ISLNK(entry_metadata.st_mode): + raise OSError(f"redirected object storage entry: {entry.path}") + if entry_metadata.st_dev != object_device: + raise OSError(f"object storage crosses a filesystem boundary: {entry.path}") + canonical = os.path.realpath(entry.path) + if confined_to_retiring_home(canonical): + raise OSError(f"object storage entry depends on retiring home: {entry.path}") + if stat.S_ISDIR(entry_metadata.st_mode): + if canonical != entry.path: + raise OSError(f"redirected object storage directory: {entry.path}") + pending.append(entry.path) + elif not stat.S_ISREG(entry_metadata.st_mode): + raise OSError(f"unsafe object storage entry: {entry.path}") + info = os.path.join(objects, "info") + alternates = os.path.join(info, "alternates") + http_alternates = os.path.join(info, "http-alternates") + if os.path.lexists(http_alternates): + raise OSError(f"HTTP alternates are not durable proof: {http_alternates}") + if not os.path.lexists(alternates): + return + metadata = os.lstat(alternates) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode): + raise OSError(f"unsafe alternates file: {alternates}") + with open(alternates, "r", encoding="utf-8") as stream: + entries = [line.rstrip("\n") for line in stream] + if not entries or any(not entry or "\x00" in entry for entry in entries): + raise OSError(f"malformed alternates file: {alternates}") + for entry in entries: + if entry.startswith('"') or entry.endswith('"'): + raise OSError(f"quoted alternates are ambiguous: {alternates}") + candidate = entry if os.path.isabs(entry) else os.path.join(objects, entry) + inspect(os.path.realpath(candidate)) + +try: + inspect(initial) +except (OSError, UnicodeError) as error: + print(f"REFUSED: {label} object storage is not independently durable: {error}", file=sys.stderr) + raise SystemExit(1) +PY + bare_repository=$(git -C "$repository" rev-parse --is-bare-repository 2>/dev/null) || return 1 + listed=$(git -C "$repository" -c core.quotePath=false worktree list --porcelain 2>/dev/null) || { + echo "REFUSED: $label linked-worktree graph is uninspectable at $repository" >&2 + return 1 + } + records=$(printf '%s\n' "$listed" | awk ' + function emit() { + if (path != "") printf "%s\t%s\n", path, is_bare ? "bare" : "worktree" + path = "" + is_bare = 0 + } + /^worktree / { emit(); path = substr($0, 10); next } + /^bare$/ { is_bare = 1; next } + /^$/ { emit() } + END { emit() } + ') || return 1 + while IFS=$'\t' read -r path kind; do + [ -n "$path" ] || continue + canonical=$(fm_checkout_trusted_dir "$path") || { + echo "REFUSED: $label linked worktree is missing or redirected at $path" >&2 + return 1 + } + case "$canonical/" in + "$retiring_home/"*) + if [ -z "$allowed_retiring_worktree" ] \ + || [ "$canonical" != "$allowed_retiring_worktree" ]; then + echo "REFUSED: $label linked-worktree graph depends on the retiring home at $canonical" >&2 + return 1 + fi + ;; + esac + case "$kind" in + bare) + [ "$canonical" = "$common" ] || return 1 + if [ "$bare_repository" = true ]; then + [ "$canonical" = "$repository" ] || return 1 + count=$((count + 1)) + fi + ;; + worktree) + if [ "$canonical" = "$repository" ] \ + || { [ -f "$repository/.git" ] && [ "$canonical" = "$common" ]; }; then + count=$((count + 1)) + else + listed_common=$(fm_checkout_git_common_dir "$canonical") || return 1 + [ "$listed_common" = "$common" ] || return 1 + fi + ;; + *) return 1 ;; + esac + done <<EOF +$records +EOF + [ "$count" -eq 1 ] || { + echo "REFUSED: $label worktree identity is ambiguous at $repository" >&2 + return 1 + } + refs=$(git -C "$repository" for-each-ref --format='%(refname)' 2>/dev/null) || { + echo "REFUSED: $label refs are uninspectable at $repository" >&2 + return 1 + } + while IFS= read -r ref; do + [ -n "$ref" ] || continue + git -C "$repository" cat-file -e "$ref^{object}" 2>/dev/null || { + echo "REFUSED: $label ref $ref depends on unavailable objects" >&2 + return 1 + } + done <<EOF +$refs +EOF + validate_surviving_object_graph_bound \ + "$repository" "$objects" "$retiring_home" "$label" +} + +validate_surviving_repository_authority() { + local repository=$1 bare + bare=$(git -C "$repository" rev-parse --is-bare-repository 2>/dev/null) || return 1 + if [ "$bare" = true ]; then + validate_surviving_repository_authority_locked "$@" + return + fi + fm_checkout_lock_run "$repository" "$CHECKOUT_LOCK_ROOT" \ + validate_surviving_repository_authority_locked "$@" +} + +secondmate_remote_identity() { + local repository=$1 retiring_home=$2 url raw_urls parsed kind transport host user port path + local canonical bare git_dir url_count + raw_urls=$(git -C "$repository" config --get-all remote.origin.url 2>/dev/null) || return 1 + url_count=$(printf '%s\n' "$raw_urls" | awk 'NF { count++ } END { print count + 0 }') || return 1 + [ "$url_count" -eq 1 ] || return 1 + url=$(printf '%s\n' "$raw_urls" | sed -n '1p') || return 1 + [ "$(git -C "$repository" remote get-url origin 2>/dev/null)" = "$url" ] || return 1 + [ -n "$url" ] || return 1 + parsed=$(python3 - "$url" "$repository" <<'PY' +import ipaddress +import os +import re +import sys +import urllib.parse + +url, repository = sys.argv[1:] + +def loopback(host): + normalized = host.lower().rstrip(".") + if normalized == "localhost": + return True + try: + return ipaddress.ip_address(normalized).is_loopback + except ValueError: + return False + +try: + if re.match(r"^[A-Za-z]:", url): + raise ValueError + if "://" in url: + parsed = urllib.parse.urlparse(url) + scheme = parsed.scheme.lower() + host = parsed.hostname or "" + if scheme == "file": + if host and not loopback(host): + raise ValueError + path = urllib.parse.unquote(parsed.path) + if not os.path.isabs(path): + raise ValueError + print("local\t" + path) + elif scheme in ("ssh", "git+ssh", "git", "http", "https"): + if loopback(host): + if scheme not in ("ssh", "git+ssh", "git"): + raise ValueError + path = urllib.parse.unquote(parsed.path) + if not os.path.isabs(path): + raise ValueError + print("local\t" + path) + elif host: + print( + "network\t" + + scheme + + "\t" + + host + + "\t" + + (parsed.username or "-") + + "\t" + + (str(parsed.port) if parsed.port is not None else "-") + + "\t" + + url + ) + else: + raise ValueError + else: + raise ValueError + else: + scp = re.match(r"^(?:([^@/:]+)@)?([^/:]+):(.+)$", url) + if scp: + user, host, path = scp.groups() + if loopback(host): + if not os.path.isabs(path): + raise ValueError + print("local\t" + path) + else: + print("network\tssh\t" + host + "\t" + (user or "-") + "\t-\t" + url) + else: + path = url if os.path.isabs(url) else os.path.join(repository, url) + print("local\t" + path) +except (OSError, ValueError): + raise SystemExit(1) +PY + ) || return 1 + IFS=$'\t' read -r kind transport host user port path <<EOF +$parsed +EOF + if [ "$kind" = network ]; then + [ "$user" != - ] || user= + [ "$port" != - ] || port= + secondmate_network_remote_identity \ + "$repository" "$path" "$transport" "$host" "$user" "$port" + return $? + fi + [ "$kind" = local ] || return 1 + path=$transport + canonical=$(fm_checkout_trusted_dir "$path") || return 1 + bare=$(git -C "$canonical" rev-parse --is-bare-repository 2>/dev/null) || return 1 + if [ "$bare" = true ]; then + git_dir=$(git -C "$canonical" rev-parse --absolute-git-dir 2>/dev/null) || return 1 + git_dir=$(fm_checkout_trusted_dir "$git_dir") || return 1 + [ "$git_dir" = "$canonical" ] || return 1 + else + [ "$(exact_git_worktree_root "$canonical")" = "$canonical" ] || return 1 + fi + case "$canonical/" in + "$retiring_home/"*) return 1 ;; + esac + validate_surviving_repository_authority \ + "$canonical" "$retiring_home" "$canonical" "secondmate project landing authority" \ + || return 1 + printf 'local\t%s\n' "$canonical" +} + +secondmate_pinned_git_url() { + python3 - "$1" "$2" <<'PY' +import ipaddress +import sys +import urllib.parse + +url, address = sys.argv[1:] +parsed = urllib.parse.urlsplit(url) +if parsed.scheme.lower() != "git" or not parsed.hostname: + raise SystemExit(1) +ip = ipaddress.ip_address(address) +host = f"[{ip}]" if ip.version == 6 else str(ip) +port = f":{parsed.port}" if parsed.port is not None else "" +user = f"{parsed.username}@" if parsed.username else "" +print(urllib.parse.urlunsplit((parsed.scheme, user + host + port, parsed.path, parsed.query, parsed.fragment))) +PY +} + +run_secondmate_remote_probe() { + local output_name=$1 repository=$2 retiring_home=$3 identity kind transport url bound_host bound_port addresses + local pinned pinned_url curl_address ssh_command index + local -a probe_args + shift 3 + identity=$(secondmate_remote_identity "$repository" "$retiring_home") || return 1 + IFS=$'\t' read -r kind transport url bound_host bound_port addresses <<EOF +$identity +EOF + probe_args=("$@") + if [ "$kind" = local ]; then + fm_run_bounded_capture --combine-stderr "$output_name" "$TEARDOWN_UPSTREAM_TIMEOUT" \ + git -C "$repository" ls-remote "${probe_args[@]}" + return + fi + [ "$kind" = network ] && [ -n "$addresses" ] || return 1 + pinned=${addresses%%,*} + case "$transport" in + ssh|git+ssh) + ssh_command="/usr/bin/ssh -oHostName=$pinned -oHostKeyAlias=$bound_host" + fm_run_bounded_capture --combine-stderr "$output_name" "$TEARDOWN_UPSTREAM_TIMEOUT" \ + /usr/bin/env -u GIT_SSH -u GIT_SSH_COMMAND -u GIT_SSH_VARIANT \ + "GIT_SSH_COMMAND=$ssh_command" GIT_SSH_VARIANT=ssh \ + git -C "$repository" ls-remote "${probe_args[@]}" + ;; + http|https) + curl_address=$pinned + case "$curl_address" in *:*) curl_address="[$curl_address]" ;; esac + fm_run_bounded_capture --combine-stderr "$output_name" "$TEARDOWN_UPSTREAM_TIMEOUT" \ + /usr/bin/env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY \ + -u ALL_PROXY -u all_proxy -u NO_PROXY -u no_proxy \ + git -C "$repository" \ + -c "http.curloptResolve=$bound_host:$bound_port:$curl_address" \ + ls-remote "${probe_args[@]}" + ;; + git) + pinned_url=$(secondmate_pinned_git_url "$url" "$pinned") || return 1 + for index in "${!probe_args[@]}"; do + [ "${probe_args[index]}" != origin ] || probe_args[index]=$pinned_url + done + fm_run_bounded_capture --combine-stderr "$output_name" "$TEARDOWN_UPSTREAM_TIMEOUT" \ + git -C "$repository" ls-remote "${probe_args[@]}" + ;; + *) return 1 ;; + esac +} + +prepare_secondmate_remote_authority() { + local repository=$1 retiring_home=$2 identity kind transport url bound_host bound_port addresses authority state_root + local object_format fetch_status pinned pinned_url curl_address ssh_command + identity=$(secondmate_remote_identity "$repository" "$retiring_home") || return 1 + IFS=$'\t' read -r kind transport url bound_host bound_port addresses <<EOF +$identity +EOF + if [ "$kind" = local ]; then + printf 'local\t%s\n' "$transport" + return 0 + fi + [ "$kind" = network ] && [ -n "$transport" ] && [ -n "$url" ] || return 1 + pinned=${addresses%%,*} + [ -n "$pinned" ] || return 1 + state_root=$(fm_checkout_trusted_dir "$STATE") || return 1 + authority=$(mktemp -d "$state_root/.remote-authority.XXXXXX") || return 1 + object_format=$(git -C "$repository" rev-parse --show-object-format 2>/dev/null) || { + removal_tree_operation "$authority" "remote authority proof" remove || true + return 1 + } + git init --quiet --bare --object-format="$object_format" "$authority" >/dev/null 2>&1 || { + removal_tree_operation "$authority" "remote authority proof" remove || true + return 1 + } + case "$transport" in + ssh|git+ssh) + ssh_command="/usr/bin/ssh -oHostName=$pinned -oHostKeyAlias=$bound_host" + if /usr/bin/env -u GIT_SSH -u GIT_SSH_COMMAND -u GIT_SSH_VARIANT \ + -u GIT_CONFIG_COUNT -u GIT_CONFIG_PARAMETERS \ + GIT_CONFIG_NOSYSTEM=1 GIT_CONFIG_GLOBAL=/dev/null \ + "GIT_SSH_COMMAND=$ssh_command" GIT_SSH_VARIANT=ssh \ + git -C "$authority" fetch --quiet --force --no-tags --no-recurse-submodules \ + "$url" '+refs/heads/*:refs/heads/*'; then + fetch_status=0 + else + fetch_status=$? + fi + ;; + http|https) + curl_address=$pinned + case "$curl_address" in *:*) curl_address="[$curl_address]" ;; esac + if /usr/bin/env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY \ + -u ALL_PROXY -u all_proxy -u NO_PROXY -u no_proxy \ + -u GIT_CONFIG_COUNT -u GIT_CONFIG_PARAMETERS \ + GIT_CONFIG_NOSYSTEM=1 GIT_CONFIG_GLOBAL=/dev/null \ + git -C "$authority" -c http.proxy= -c http.followRedirects=false \ + -c "http.curloptResolve=$bound_host:$bound_port:$curl_address" \ + fetch --quiet --force --no-tags --no-recurse-submodules \ + "$url" '+refs/heads/*:refs/heads/*'; then + fetch_status=0 + else + fetch_status=$? + fi + ;; + git) + pinned_url=$(secondmate_pinned_git_url "$url" "$pinned") || fetch_status=1 + if [ -z "${pinned_url:-}" ]; then + fetch_status=1 + elif /usr/bin/env -u GIT_CONFIG_COUNT -u GIT_CONFIG_PARAMETERS \ + GIT_CONFIG_NOSYSTEM=1 GIT_CONFIG_GLOBAL=/dev/null \ + git -C "$authority" fetch --quiet --force --no-tags --no-recurse-submodules \ + "$pinned_url" '+refs/heads/*:refs/heads/*'; then + fetch_status=0 + else + fetch_status=$? + fi + ;; + *) fetch_status=1 ;; + esac + if [ "$fetch_status" -ne 0 ] \ + || ! validate_surviving_repository_authority \ + "$authority" "$retiring_home" "$authority" "network landing authority"; then + removal_tree_operation "$authority" "remote authority proof" remove || true + return 1 + fi + printf 'network\t%s\n' "$authority" +} + +cleanup_secondmate_remote_authority() { + local kind=$1 authority=$2 + [ "$kind" != network ] || removal_tree_operation "$authority" "remote authority proof" remove +} + +enumerate_secondmate_project_repositories() { + python3 - "$1" <<'PY' +import os +import stat +import sys + +root = sys.argv[1] +root = os.path.realpath(root) +repositories = {"."} +visited = set() +symlink_targets = set() + +def confined(path): + try: + return os.path.commonpath((root, path)) == root + except ValueError: + return False + +def walk(current, ancestors): + metadata = os.lstat(current) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode): + raise OSError("unsafe project directory") + identity = (metadata.st_dev, metadata.st_ino) + if identity in ancestors: + raise OSError("project directory cycle") + if identity in visited: + return + visited.add(identity) + relative = os.path.relpath(current, root) + if relative != ".": + git_marker = os.path.join(current, ".git") + if os.path.lexists(git_marker): + marker = os.lstat(git_marker) + if stat.S_ISLNK(marker.st_mode) or not ( + stat.S_ISDIR(marker.st_mode) or stat.S_ISREG(marker.st_mode) + ): + raise OSError("unsafe git marker") + repositories.add(relative) + else: + head = os.path.join(current, "HEAD") + config = os.path.join(current, "config") + objects = os.path.join(current, "objects") + refs = os.path.join(current, "refs") + if all(os.path.exists(path) for path in (head, config, objects, refs)): + if not os.path.isfile(head) or not os.path.isfile(config): + raise OSError("unsafe bare repository") + if not os.path.isdir(objects) or not os.path.isdir(refs): + raise OSError("unsafe bare repository") + repositories.add(relative) + return + with os.scandir(current) as entries: + ordered = sorted(entries, key=lambda entry: entry.name) + for entry in ordered: + if entry.name == ".git": + continue + path = entry.path + entry_metadata = os.lstat(path) + if stat.S_ISLNK(entry_metadata.st_mode): + target_metadata = os.stat(path) + if not stat.S_ISDIR(target_metadata.st_mode): + continue + target = os.path.realpath(path) + if not confined(target): + raise OSError("escaping project directory symlink") + target_identity = (target_metadata.st_dev, target_metadata.st_ino) + if target_identity in ancestors or target_identity == identity: + raise OSError("project directory symlink cycle") + symlink_targets.add(target_identity) + continue + if stat.S_ISDIR(entry_metadata.st_mode): + walk(path, ancestors | {identity}) + +try: + walk(root, set()) + if not symlink_targets.issubset(visited): + raise OSError("unaccounted project directory symlink") + for repository in sorted(repositories): + print(repository) +except OSError: + raise SystemExit(1) +PY +} + +validate_secondmate_repository_worktree_graph() { + local repository=$1 retiring_home=$2 repository_container=${3:-$1} + local common listed records path kind canonical count=0 bare_repository container_git submodule_admin=0 + bare_repository=$(git -C "$repository" rev-parse --is-bare-repository 2>/dev/null) || return 1 + common=$(git -C "$repository" rev-parse --git-common-dir 2>/dev/null) || return 1 + case "$common" in /*) ;; *) common="$repository/$common" ;; esac + common=$(fm_checkout_trusted_dir "$common") || return 1 + case "$common/" in + "$retiring_home/"*) ;; + *) + echo "REFUSED: secondmate project repository is a linked worktree owned outside the retiring home at $repository" >&2 + return 1 + ;; + esac + if [ -f "$repository/.git" ] && [ ! -L "$repository/.git" ]; then + container_git=$(git -C "$repository_container" rev-parse --absolute-git-dir 2>/dev/null) || return 1 + container_git=$(fm_checkout_trusted_dir "$container_git") || return 1 + case "$common/" in "$container_git/modules/"*) submodule_admin=1 ;; esac + fi + listed=$(git -C "$repository" -c core.quotePath=false worktree list --porcelain 2>/dev/null) || { + echo "REFUSED: secondmate project linked-worktree graph is uninspectable at $repository" >&2 + return 1 + } + records=$(printf '%s\n' "$listed" | awk ' + function emit() { + if (path != "") { + printf "%s\t%s\n", path, is_bare ? "bare" : "worktree" + } + path = "" + is_bare = 0 + } + /^worktree / { + emit() + path = substr($0, 10) + next + } + /^bare$/ { + is_bare = 1 + next + } + /^$/ { + emit() + } + END { + emit() + } + ') || return 1 + while IFS=$'\t' read -r path kind; do + [ -n "$path" ] || continue + canonical=$(fm_checkout_trusted_dir "$path") || { + echo "REFUSED: secondmate project linked worktree is missing or redirected from $repository" >&2 + return 1 + } + case "$kind" in + bare) + [ "$canonical" = "$common" ] || { + echo "REFUSED: secondmate project bare worktree identity drifted at $canonical" >&2 + return 1 + } + if [ "$bare_repository" = true ]; then + [ "$canonical" = "$repository" ] || return 1 + count=$((count + 1)) + fi + ;; + worktree) + count=$((count + 1)) + if [ "$canonical" = "$repository" ]; then + : + elif [ "$submodule_admin" -eq 1 ] && [ "$canonical" = "$common" ]; then + : + else + echo "REFUSED: secondmate project common Git directory owns another linked worktree at $canonical" >&2 + return 1 + fi + ;; + *) return 1 ;; + esac + done <<EOF +$records +EOF + [ "$count" -eq 1 ] || { + echo "REFUSED: secondmate project linked-worktree ownership is ambiguous at $repository" >&2 + return 1 + } +} + +validate_secondmate_declared_submodules() { + local repository=$1 container=${2:-$1} modules entries status key path submodule + modules="$repository/.gitmodules" + [ -e "$modules" ] || [ -L "$modules" ] || return 0 + [ -f "$modules" ] && [ ! -L "$modules" ] && [ -r "$modules" ] || return 1 + if entries=$(git -C "$repository" config --file .gitmodules \ + --get-regexp '^submodule\..*\.path$' 2>/dev/null); then + status=0 + else + status=$? + fi + [ "$status" -eq 0 ] && [ -n "$entries" ] || return 1 + while read -r key path; do + [ -n "$key" ] && [ -n "$path" ] || return 1 + submodule=$(fm_checkout_lexical_path "$repository/$path") || return 1 + case "$submodule/" in + "$repository/"*) ;; + *) return 1 ;; + esac + [ "$(exact_git_repository_root "$submodule" "$container")" = "$submodule" ] || return 1 + done <<EOF +$entries +EOF +} + +validate_secondmate_project_repository_landed_state() { + local repository=$1 source_repository=$2 retiring_home=$3 repository_container=${4:-$1} + local source_container=${5:-$2} dirty refs ref tip reflog_tips + local remote_tips remote_tip landed repository_identity source_identity bare stash_status + local authority_record authority_kind authority cleanup_status=0 + [ "$(exact_git_repository_root "$repository" "$repository_container")" = "$repository" ] || { + echo "REFUSED: secondmate project repository is not an exact repository root at $repository" >&2 + return 1 + } + [ "$(exact_git_repository_root "$source_repository" "$source_container")" = "$source_repository" ] || { + echo "REFUSED: registered source project repository is not an exact repository root at $source_repository" >&2 + return 1 + } + validate_surviving_repository_authority \ + "$source_repository" "$retiring_home" "$source_container" \ + "registered source project repository" || return 1 + git_history_rewrite_state_is_clean "$repository" "secondmate project repository" || return 1 + validate_secondmate_repository_worktree_graph \ + "$repository" "$retiring_home" "$repository_container" || return 1 + validate_secondmate_declared_submodules "$repository" "$repository_container" || { + echo "REFUSED: secondmate project submodule state is uninspectable at $repository" >&2 + return 1 + } + bare=$(git -C "$repository" rev-parse --is-bare-repository 2>/dev/null) || return 1 + if [ "$bare" != true ]; then + dirty=$(GIT_OPTIONAL_LOCKS=0 git -C "$repository" status --porcelain=v1 --untracked-files=all 2>/dev/null) || { + echo "REFUSED: secondmate project clone cleanliness is uninspectable at $repository" >&2 + return 1 + } + [ -z "$dirty" ] || { + echo "REFUSED: secondmate project clone has unlanded changes at $repository" >&2 + return 1 + } + fi + if git -C "$repository" show-ref --verify --quiet refs/stash 2>/dev/null; then + echo "REFUSED: secondmate project repository has retained stash history at $repository" >&2 + return 1 + else + stash_status=$? + fi + [ "$stash_status" -eq 1 ] || { + echo "REFUSED: secondmate project repository stash state is uninspectable at $repository" >&2 + return 1 + } + repository_identity=$(secondmate_remote_identity "$repository" "$retiring_home") || { + echo "REFUSED: secondmate project remote identity is unsafe or does not survive home removal at $repository" >&2 + return 1 + } + source_identity=$(secondmate_remote_identity "$source_repository" "$retiring_home") || { + echo "REFUSED: registered source project remote identity is unsafe or unreadable at $source_repository" >&2 + return 1 + } + [ "$repository_identity" = "$source_identity" ] || { + echo "REFUSED: secondmate project origin drifted from its registered source at $repository" >&2 + return 1 + } + authority_record=$(prepare_secondmate_remote_authority "$repository" "$retiring_home") || { + echo "REFUSED: secondmate project remote authority graph is incomplete or uninspectable at $repository" >&2 + return 1 + } + IFS=$'\t' read -r authority_kind authority <<EOF +$authority_record +EOF + remote_tips=$(GIT_NO_REPLACE_OBJECTS=1 git -C "$authority" \ + for-each-ref --format='%(objectname)' refs/heads 2>/dev/null) || { + cleanup_secondmate_remote_authority "$authority_kind" "$authority" || true + return 1 + } + [ -n "$remote_tips" ] || { + cleanup_secondmate_remote_authority "$authority_kind" "$authority" || true + echo "REFUSED: secondmate project repository has no live remote branches at $repository" >&2 + return 1 + } + refs=$(GIT_NO_REPLACE_OBJECTS=1 git -C "$repository" for-each-ref --format='%(refname)' 2>/dev/null) || { + cleanup_secondmate_remote_authority "$authority_kind" "$authority" || true + echo "REFUSED: secondmate project repository refs are uninspectable at $repository" >&2 + return 1 + } + reflog_tips=$(GIT_NO_REPLACE_OBJECTS=1 git -C "$repository" reflog --all --format='%H' 2>/dev/null) || { + cleanup_secondmate_remote_authority "$authority_kind" "$authority" || true + echo "REFUSED: secondmate project repository reflogs are uninspectable at $repository" >&2 + return 1 + } + refs=$(printf 'HEAD\n%s\n%s\n' "$refs" "$reflog_tips") + while IFS= read -r ref; do + [ -n "$ref" ] || continue + [ "$ref" != refs/stash ] || continue + tip=$(GIT_NO_REPLACE_OBJECTS=1 git -C "$repository" rev-parse "$ref^{commit}" 2>/dev/null) || { + cleanup_secondmate_remote_authority "$authority_kind" "$authority" || true + echo "REFUSED: secondmate project repository ref $ref is uninspectable at $repository" >&2 + return 1 + } + landed=0 + while IFS= read -r remote_tip; do + [ -n "$remote_tip" ] || continue + if GIT_NO_REPLACE_OBJECTS=1 git -C "$authority" cat-file -e "$tip^{commit}" 2>/dev/null \ + && GIT_NO_REPLACE_OBJECTS=1 git -C "$authority" \ + merge-base --is-ancestor "$tip" "$remote_tip" 2>/dev/null; then + landed=1 + break + fi + done <<EOF +$remote_tips +EOF + [ "$landed" -eq 1 ] || { + cleanup_secondmate_remote_authority "$authority_kind" "$authority" || true + echo "REFUSED: secondmate project repository ref $ref is not proven on a live remote branch at $repository" >&2 + return 1 + } + done <<EOF +$refs +EOF + cleanup_secondmate_remote_authority "$authority_kind" "$authority" || cleanup_status=$? + [ "$cleanup_status" -eq 0 ] || { + echo "REFUSED: remote authority proof cleanup could not be completed safely" >&2 + return 1 + } +} + +validate_secondmate_project_clones() { + local home=$1 registry=$2 expected_id=$3 expected_source=$4 projects_root source_projects_root + local expected listed project clone source_clone repositories relative repository source_repository + if ! expected=$(fm_secondmate_registry_query "$registry" query "$expected_id" projects); then + if [ "$registry" = "$PREPARED_REGISTRY_PATH" ] \ + && [ "$expected_id" = "$PREPARED_REGISTRY_ID" ] \ + && [ "$home" = "$PREPARED_REGISTRY_HOME" ] \ + && [ -n "$PREPARED_REGISTRY_BACKUP" ] \ + && fm_account_lifecycle_lock_owned "$PREPARED_REGISTRY_LOCK"; then + expected=$(fm_secondmate_registry_query \ + "$PREPARED_REGISTRY_BACKUP" query "$expected_id" projects) || return 1 + else + echo "REFUSED: secondmate project registration is unprovable for $expected_id" >&2 + return 1 + fi + fi + projects_root=$(fm_checkout_trusted_dir "$home/projects") || { + echo "REFUSED: secondmate projects directory is missing, redirected, or unreadable at $home/projects" >&2 + return 1 + } + listed=$(python3 - "$projects_root" <<'PY' +import os +import stat +import sys + +root = sys.argv[1] +try: + with os.scandir(root) as entries: + for entry in sorted(entries, key=lambda item: item.name): + metadata = entry.stat(follow_symlinks=False) + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode): + raise OSError("unsafe project entry") + print(entry.name) +except OSError: + raise SystemExit(1) +PY + ) || { + echo "REFUSED: secondmate project clones cannot be safely enumerated at $projects_root" >&2 return 1 + } + if [ -n "$expected" ]; then + expected=$(printf '%s\n' "$expected" | tr ',' '\n' | sed 's/^ *//;s/ *$//' | sort) || return 1 fi - if ! worktree_registered_for_project "$project" "$target"; then - echo "REFUSED: unsafe child worktree removal target $target is not a git worktree for ${project:-the recorded project}" >&2 + listed=$(printf '%s\n' "$listed" | sed '/^$/d' | sort) || return 1 + [ "$listed" = "$expected" ] || { + echo "REFUSED: secondmate project clones do not exactly match the registration for $expected_id" >&2 return 1 + } + if [ "$expected_source" = "$FM_ROOT" ] && [ -n "${FM_PROJECTS_OVERRIDE:-}" ]; then + source_projects_root=$FM_PROJECTS_OVERRIDE + else + source_projects_root="$expected_source/projects" fi - printf '%s\n' "$abs_target" -} - -safe_rm_rf() { - local target=$1 label=$2 - validate_removal_target "$target" "$label" >/dev/null || return 1 - rm -rf -- "$target" -} - -safe_rm_rf_child_worktree() { - local target=$1 project=$2 - validate_child_worktree_for_removal "$target" "$project" >/dev/null || return 1 - rm -rf -- "$target" + source_projects_root=$(fm_checkout_trusted_dir "$source_projects_root") || { + echo "REFUSED: registered source projects are unavailable or redirected at $source_projects_root" >&2 + return 1 + } + while IFS= read -r project; do + [ -n "$project" ] || continue + clone=$(fm_checkout_trusted_dir "$projects_root/$project") || return 1 + source_clone=$(fm_checkout_trusted_dir "$source_projects_root/$project") || { + echo "REFUSED: registered source project is unavailable or redirected for $project" >&2 + return 1 + } + validate_removal_tree_boundaries "$clone" "secondmate project clone" || return 1 + repositories=$(enumerate_secondmate_project_repositories "$clone") || { + echo "REFUSED: nested project repositories cannot be safely enumerated at $clone" >&2 + return 1 + } + while IFS= read -r relative; do + [ -n "$relative" ] || continue + if [ "$relative" = . ]; then + repository=$clone + source_repository=$source_clone + else + repository=$(fm_checkout_trusted_dir "$clone/$relative") || return 1 + source_repository=$(fm_checkout_trusted_dir "$source_clone/$relative") || { + echo "REFUSED: nested project repository has no registered source counterpart at $clone/$relative" >&2 + return 1 + } + fi + validate_secondmate_project_repository_landed_state \ + "$repository" "$source_repository" "$home" "$clone" "$source_clone" || return 1 + done <<EOF +$repositories +EOF + done <<EOF +$listed +EOF } validate_firstmate_home_for_removal() { - local home=$1 label=$2 expected_id=${3:-} abs_home_path marker_id conflict child_id child_home - [ -n "$home" ] || return 0 - [ -e "$home" ] || return 0 + local home=$1 label=$2 expected_id=${3:-} expected_source=${4:-$FM_ROOT} expected_registry=${5:-} expected_project + local abs_home_path metadata_home_root marker_id source_authority=${7:-1} + expected_project=${6:-$home} + [ -n "$home" ] && [ -e "$home" ] || { + echo "REFUSED: missing $label removal target ${home:-<empty>}" >&2 + return 1 + } abs_home_path=$(validate_removal_target "$home" "$label") || return 1 if [ ! -f "$abs_home_path/$SUB_HOME_MARKER" ]; then echo "REFUSED: unsafe $label removal target $home is not a seeded secondmate home" >&2 @@ -994,48 +3385,131 @@ validate_firstmate_home_for_removal() { echo "REFUSED: unsafe $label removal target $home is marked for secondmate ${marker_id:-unknown}, expected $expected_id" >&2 return 1 fi + [ -n "$expected_registry" ] || expected_registry=$(secondmate_registry_for_source "$expected_source") || return 1 + require_registered_secondmate_home "$expected_registry" "$expected_id" "$abs_home_path" || return 1 fi - validate_firstmate_operational_dirs_for_removal "$abs_home_path" "$label" || return 1 - conflict=$(registered_descendant_home_for_removal "$SECONDMATE_REG" "$abs_home_path" || true) - if [ -z "$conflict" ]; then - conflict=$(registered_descendant_home_for_removal "$abs_home_path/data/secondmates.md" "$abs_home_path" || true) - fi - if [ -n "$conflict" ]; then - IFS=$'\t' read -r child_id child_home <<EOF -$conflict -EOF - echo "REFUSED: unsafe $label removal target $home contains registered secondmate home $child_home for $child_id" >&2 + metadata_home_root=$(exact_git_worktree_root "$expected_project") || { + echo "REFUSED: secondmate project metadata is not an exact repository root: ${expected_project:-<missing>}" >&2 + return 1 + } + [ "$metadata_home_root" = "$abs_home_path" ] || { + echo "REFUSED: secondmate project metadata resolves to $metadata_home_root, not registered home $abs_home_path" >&2 return 1 + } + validate_firstmate_home_repository_identity "$abs_home_path" "$expected_source" || return 1 + if [ "$source_authority" -eq 1 ]; then + validate_surviving_repository_authority \ + "$expected_source" "$abs_home_path" "$expected_source" \ + "secondmate top-level source repository" "$abs_home_path" || return 1 fi + if [ -n "$expected_id" ] && firstmate_home_has_treehouse_slot "$abs_home_path" "$expected_source"; then + require_treehouse_task_lease "$abs_home_path" "$expected_id" || return 1 + fi + validate_secondmate_home_landed_state "$abs_home_path" "$expected_source" || return 1 + if [ -n "$expected_id" ]; then + validate_secondmate_project_clones \ + "$abs_home_path" "$expected_registry" "$expected_id" "$expected_source" || return 1 + fi + validate_firstmate_operational_dirs_for_removal "$abs_home_path" "$label" || return 1 + secondmate_state_metadata "$abs_home_path" >/dev/null || return 1 + fm_secondmate_registry_query "$abs_home_path/data/secondmates.md" validate >/dev/null || { + echo "REFUSED: child secondmate registry is malformed or uninspectable at $abs_home_path/data/secondmates.md" >&2 + return 1 + } printf '%s\n' "$abs_home_path" } +remove_explicit_firstmate_home_locked() { + local home=$1 label=$2 expected_id=$3 expected_source=$4 expected_registry=${5:-} expected_project validated + expected_project=${6:-$home} + validated=$(validate_firstmate_home_for_removal "$home" "$label" "$expected_id" "$expected_source" "$expected_registry" "$expected_project") || return 1 + [ "$validated" = "$home" ] || return 1 + require_empty_secondmate_state "$home" || return 1 + firstmate_home_has_treehouse_slot "$home" "$expected_source" && { + echo "error: $label became a Treehouse worktree before explicit removal" >&2 + return 1 + } + safe_rm_rf "$home" "$label" +} + +validate_treehouse_firstmate_home_locked() { + validate_firstmate_home_for_removal "$1" "secondmate home" "$3" "$2" >/dev/null \ + && require_empty_secondmate_state "$1" +} + remove_firstmate_home() { - local home=$1 label=$2 expected_id=${3:-} abs_home_path + local home=$1 label=$2 expected_id=${3:-} expected_source=${4:-$FM_ROOT} expected_registry=${5:-} expected_project abs_home_path + expected_project=${6:-$home} [ -n "$home" ] || return 0 [ -e "$home" ] || return 0 - abs_home_path=$(validate_firstmate_home_for_removal "$home" "$label" "$expected_id") || return 1 + abs_home_path=$(validate_firstmate_home_for_removal "$home" "$label" "$expected_id" "$expected_source" "$expected_registry" "$expected_project") || return 1 [ -n "$abs_home_path" ] || return 0 - if firstmate_home_has_treehouse_slot "$abs_home_path"; then + if firstmate_home_has_treehouse_slot "$abs_home_path" "$expected_source"; then command -v treehouse >/dev/null 2>&1 || { echo "error: treehouse command not found; cannot return $label $abs_home_path" >&2 return 1 } - teardown_treehouse_return "$abs_home_path" "$FM_ROOT" "$label" || { + teardown_treehouse_return "$abs_home_path" "$expected_source" "$label" "$expected_id" \ + validate_treehouse_firstmate_home_locked || { echo "error: treehouse return failed for $label $abs_home_path; lease may still be held" >&2 return 1 } return 0 fi - safe_rm_rf "$abs_home_path" "$label" + fm_checkout_lock_run "$abs_home_path" "$CHECKOUT_LOCK_ROOT" \ + remove_explicit_firstmate_home_locked "$abs_home_path" "$label" "$expected_id" "$expected_source" "$expected_registry" "$expected_project" +} + +validate_registered_secondmate_children() { + local home=$1 entries child_id child_home _child_projects child_meta meta_kind meta_home + entries=$(fm_secondmate_registry_query "$home/data/secondmates.md" list) || { + echo "REFUSED: registered secondmate children are unprovable at $home/data/secondmates.md" >&2 + return 1 + } + while IFS=$'\t' read -r child_id child_home _child_projects; do + [ -n "$child_id" ] || continue + child_meta="$home/state/$child_id.meta" + [ -f "$child_meta" ] && [ ! -L "$child_meta" ] && [ -r "$child_meta" ] || { + echo "REFUSED: registered secondmate $child_id has no inspectable child metadata; preserving $child_home" >&2 + return 1 + } + meta_kind=$(meta_value "$child_meta" kind) + [ "$meta_kind" = secondmate ] || { + echo "REFUSED: registered secondmate $child_id has mismatched child metadata" >&2 + return 1 + } + meta_home=$(meta_value "$child_meta" home) + [ -n "$meta_home" ] || meta_home=$(meta_value "$child_meta" worktree) + meta_home=$(canonical_existing_dir "$meta_home") || return 1 + child_home=$(canonical_existing_dir "$child_home") || return 1 + [ "$meta_home" = "$child_home" ] || { + echo "REFUSED: registered secondmate $child_id home does not match child metadata" >&2 + return 1 + } + done <<EOF +$entries +EOF +} + +require_empty_secondmate_registry() { + local home=$1 entries + entries=$(fm_secondmate_registry_query "$home/data/secondmates.md" list) || { + echo "REFUSED: child secondmate registry is unprovable at $home/data/secondmates.md" >&2 + return 1 + } + [ -z "$entries" ] || { + echo "REFUSED: secondmate home still registers child homes; preserving $home" >&2 + return 1 + } } validate_firstmate_home_children_removal() { - local home=$1 sub_state child_meta child_id child_wt child_proj child_kind child_home child_backend child_orca_worktree_id + local home=$1 sub_state child_metas child_meta child_id child_wt child_proj child_kind child_home child_backend child_orca_worktree_id sub_state="$home/state" - [ -d "$sub_state" ] || return 0 - for child_meta in "$sub_state"/*.meta; do - [ -e "$child_meta" ] || continue + child_metas=$(secondmate_state_metadata "$home") || return 1 + validate_registered_secondmate_children "$home" || return 1 + while IFS= read -r child_meta; do + [ -n "$child_meta" ] || continue child_id=$(basename "$child_meta" .meta) child_wt=$(meta_value "$child_meta" worktree) child_kind=$(meta_value "$child_meta" kind) @@ -1047,155 +3521,445 @@ validate_firstmate_home_children_removal() { fi child_backend=$(fm_backend_of_meta "$child_meta") if [ "$child_kind" = secondmate ]; then + child_proj=$(meta_value "$child_meta" project) child_home=$(meta_value "$child_meta" home) [ -n "$child_home" ] || child_home=$child_wt - validate_firstmate_home_for_removal "$child_home" "child firstmate home" "$child_id" >/dev/null || return 1 + validate_firstmate_home_for_removal "$child_home" "child firstmate home" "$child_id" "$home" "$home/data/secondmates.md" "$child_proj" >/dev/null || return 1 validate_firstmate_home_children_removal "$child_home" || return 1 elif [ "$child_backend" = orca ]; then + require_orca_task_metadata_identity "$child_meta" "$child_id" || return 1 child_orca_worktree_id=$(require_orca_worktree_id "$child_meta") || return 1 if [ -n "$child_wt" ] && [ -e "$child_wt" ]; then child_proj=$(meta_value "$child_meta" project) validate_child_worktree_for_removal "$child_wt" "$child_proj" >/dev/null || return 1 require_orca_worktree_path_match "$child_orca_worktree_id" "$child_wt" || return 1 + validate_child_worktree_landed_state "$child_meta" "$child_id" "$child_wt" "$child_proj" || return 1 fi - elif [ -n "$child_wt" ] && [ -e "$child_wt" ]; then + elif [ -n "$child_wt" ] && [ -d "$child_wt" ]; then child_proj=$(meta_value "$child_meta" project) validate_child_worktree_for_removal "$child_wt" "$child_proj" >/dev/null || return 1 + require_treehouse_task_lease "$(canonical_existing_dir "$child_wt")" "firstmate-$child_id" || return 1 + validate_child_worktree_landed_state "$child_meta" "$child_id" "$child_wt" "$child_proj" || return 1 + else + echo "error: retained child metadata for $child_id because its Treehouse worktree is missing or uninspectable" >&2 + return 1 fi - done + done <<EOF +$child_metas +EOF +} + +remove_child_orca_worktree_locked() { + local child_worktree=$1 child_project=$2 child_worktree_id=$3 child_id=$4 child_meta=$5 branch=HEAD boundary_token + validate_child_worktree_for_removal "$child_worktree" "$child_project" >/dev/null || return 1 + require_orca_worktree_path_match "$child_worktree_id" "$child_worktree" || return 1 + fm_backend_quiesce_worktree_terminals orca "$child_worktree_id" "fm-$child_id" "$(meta_value "$child_meta" terminal)" || return 1 + validate_child_worktree_landed_state "$child_meta" "$child_id" "$child_worktree" "$child_project" || return 1 + validate_removal_tree_boundaries "$child_worktree" "child Orca worktree" || return 1 + require_orca_worktree_path_match "$child_worktree_id" "$child_worktree" || return 1 + branch=$(git -C "$child_worktree" rev-parse --abbrev-ref HEAD 2>/dev/null) || return 1 + boundary_token=$(removal_tree_boundary_token "$child_worktree" "child Orca worktree") || return 1 + fm_backend_remove_worktree_bound \ + orca "$child_worktree_id" "$child_worktree" "$boundary_token" || return 1 + if [ "$branch" != "HEAD" ]; then + git -C "$child_project" branch -D "$branch" >/dev/null 2>&1 || true + fi + remove_worktree_compatibility_artifacts "$child_worktree" "removed child Orca worktree" } cleanup_firstmate_home_children() { - local home=$1 sub_state child_meta child_id child_t child_wt child_proj child_kind child_home child_backend child_orca_worktree_id child_return_rc child_account_lock child_endpoint_home + local home=$1 sub_state child_metas child_meta child_id child_wt child_proj child_kind child_prelock_kind child_home child_home_after child_home_lock child_registry_lock child_backend child_orca_worktree_id child_return_rc child_account_lock child_endpoint_home remaining_child_metas child_registry_prepared child_registry_update child_registry_backup sub_state="$home/state" - [ -d "$sub_state" ] || return 0 - for child_meta in "$sub_state"/*.meta; do - [ -e "$child_meta" ] || continue + child_metas=$(secondmate_state_metadata "$home") || return 1 + while IFS= read -r child_meta; do + [ -n "$child_meta" ] || continue child_id=$(basename "$child_meta" .meta) + [ -f "$child_meta" ] && [ ! -L "$child_meta" ] && [ -r "$child_meta" ] \ + || { echo "error: child metadata is unsafe for $child_id" >&2; return 1; } + child_wt=$(meta_value "$child_meta" worktree) + child_kind=$(meta_value "$child_meta" kind) + [ -n "$child_kind" ] || child_kind=ship + child_prelock_kind=$child_kind + child_home= + if [ "$child_kind" = secondmate ]; then + child_home=$(meta_value "$child_meta" home) + [ -n "$child_home" ] || child_home=$child_wt + [ -d "$child_home" ] || { + echo "error: retained child secondmate metadata for $child_id because its home is missing or uninspectable" >&2 + return 1 + } + child_home_lock=$(fm_secondmate_home_lifecycle_lock_acquire "$CHECKOUT_LOCK_ROOT" "$child_home") || return 1 + TEARDOWN_ACCOUNT_LOCKS+=("$child_home_lock") + fi child_account_lock=$(fm_account_lifecycle_lock_acquire "$sub_state" "$child_id") || return 1 TEARDOWN_ACCOUNT_LOCKS+=("$child_account_lock") - [ -f "$child_meta" ] || { echo "error: child metadata disappeared while teardown waited for $child_id" >&2; return 1; } + [ -f "$child_meta" ] && [ ! -L "$child_meta" ] && [ -r "$child_meta" ] \ + || { echo "error: child metadata changed while teardown waited for $child_id" >&2; return 1; } if managed_account_meta "$child_meta"; then if [ ! -f "$child_meta" ] || ! managed_account_meta "$child_meta"; then echo "error: managed child metadata changed while teardown waited for $child_id" >&2 return 1 fi fi - child_wt=$(meta_value "$child_meta" worktree) child_proj=$(meta_value "$child_meta" project) child_kind=$(meta_value "$child_meta" kind) [ -n "$child_kind" ] || child_kind=ship - child_home= + [ "$child_kind" = "$child_prelock_kind" ] \ + || { echo "error: child kind changed while teardown waited for $child_id" >&2; return 1; } if [ "$child_kind" = secondmate ]; then - child_home=$(meta_value "$child_meta" home) - [ -n "$child_home" ] || child_home=$child_wt + child_home_after=$(meta_value "$child_meta" home) + [ -n "$child_home_after" ] || child_home_after=$(meta_value "$child_meta" worktree) + [ "$child_home_after" = "$child_home" ] \ + || { echo "error: child secondmate home changed while teardown waited for $child_id" >&2; return 1; } + [ -f "$child_meta" ] && [ ! -L "$child_meta" ] && [ -r "$child_meta" ] || { + echo "error: child metadata changed while teardown waited for secondmate home $child_id" >&2 + return 1 + } fi child_backend=$(fm_backend_of_meta "$child_meta") - if [ "$child_backend" = orca ]; then - child_t=$(meta_value "$child_meta" terminal) - else - child_t=$(fm_backend_target_of_meta "$child_meta") - fi if [ "$child_backend" = orca ] && [ "$child_kind" != secondmate ]; then + require_orca_task_metadata_identity "$child_meta" "$child_id" || return 1 child_orca_worktree_id=$(require_orca_worktree_id "$child_meta") || return 1 if [ -n "$child_wt" ] && [ -e "$child_wt" ]; then validate_child_worktree_for_removal "$child_wt" "$child_proj" >/dev/null || return 1 fi fi + if [ "$child_kind" != secondmate ] && [ "$child_backend" != orca ]; then + if [ -z "$child_wt" ] || [ ! -d "$child_wt" ]; then + echo "error: retained child metadata for $child_id because its Treehouse worktree is missing or uninspectable" >&2 + return 1 + fi + validate_child_worktree_for_removal "$child_wt" "$child_proj" >/dev/null || return 1 + require_treehouse_task_lease "$(canonical_existing_dir "$child_wt")" "firstmate-$child_id" || return 1 + fi if managed_account_meta "$child_meta"; then child_endpoint_home=$(fm_backend_endpoint_home "$child_backend" "$child_kind" "$home" "$child_home") release_managed_account "$child_meta" "$child_id" "$child_endpoint_home" "$child_account_lock" "$home/data" || return 1 child_account_lock=$MANAGED_ACCOUNT_LOCK else - if [ -n "$child_t" ]; then - ( unset FM_ROOT_OVERRIDE; FM_HOME="$home" FM_ROOT="$home" fm_backend_kill "$child_backend" "$child_t" "$(meta_value "$child_meta" zellij_tab_id)" "fm-$child_id" "$(meta_value "$child_meta" tmux_session_target)" ) 2>/dev/null || true - fi + quiesce_child_endpoint "$child_meta" "$child_id" "$home" "$child_home" || return 1 fi if [ "$child_kind" = secondmate ]; then if [ -n "$child_home" ] && [ -d "$child_home" ]; then - cleanup_firstmate_home_children "$child_home" - remove_firstmate_home "$child_home" "child firstmate home" "$child_id" + cleanup_firstmate_home_children "$child_home" || return 1 + child_registry_lock=$(fm_secondmate_registry_lock_acquire "$CHECKOUT_LOCK_ROOT" "$home/data/secondmates.md") || return 1 + TEARDOWN_ACCOUNT_LOCKS+=("$child_registry_lock") + validate_firstmate_home_for_removal "$child_home" "child firstmate home" "$child_id" "$home" "$home/data/secondmates.md" "$child_proj" >/dev/null || return 1 + remaining_child_metas=$(secondmate_state_metadata "$child_home") || return 1 + [ -z "$remaining_child_metas" ] || return 1 + child_registry_prepared=$(prepare_secondmate_registry_removal "$child_id" "$child_home" "$home/data/secondmates.md" "$child_registry_lock") || return 1 + IFS=$'\t' read -r child_registry_update child_registry_backup <<EOF +$child_registry_prepared +EOF + PREPARED_REGISTRY_PATH="$home/data/secondmates.md" + PREPARED_REGISTRY_BACKUP=$child_registry_backup + PREPARED_REGISTRY_ID=$child_id + PREPARED_REGISTRY_HOME=$child_home + PREPARED_REGISTRY_LOCK=$child_registry_lock + activate_secondmate_registry_removal "$home/data/secondmates.md" "$child_registry_lock" "$child_registry_update" || { + rm -f "$child_registry_update" "$child_registry_backup" + return 1 + } + remove_firstmate_home "$child_home" "child firstmate home" "$child_id" "$home" "$home/data/secondmates.md" "$child_proj" || { + rollback_secondmate_registry_removal "$home/data/secondmates.md" "$child_registry_lock" "$child_registry_backup" + PREPARED_REGISTRY_PATH= + PREPARED_REGISTRY_BACKUP= + PREPARED_REGISTRY_ID= + PREPARED_REGISTRY_HOME= + PREPARED_REGISTRY_LOCK= + return 1 + } + rm -f "$child_registry_backup" + PREPARED_REGISTRY_PATH= + PREPARED_REGISTRY_BACKUP= + PREPARED_REGISTRY_ID= + PREPARED_REGISTRY_HOME= + PREPARED_REGISTRY_LOCK= + fm_account_lifecycle_lock_release "$child_registry_lock" || return 1 + child_registry_lock= fi elif [ "$child_backend" = orca ]; then if [ -n "$child_wt" ] && [ -d "$child_wt" ]; then validate_child_worktree_for_removal "$child_wt" "$child_proj" >/dev/null || return 1 - rm -f "$child_wt/.claude/settings.local.json" "$child_wt/.opencode/plugins/fm-turn-end.js" "$child_wt/.fm-grok-turnend" + fm_checkout_lock_run "$child_wt" "$CHECKOUT_LOCK_ROOT" \ + remove_child_orca_worktree_locked "$child_wt" "$child_proj" "$child_orca_worktree_id" "$child_id" "$child_meta" || return 1 + else + echo "error: child Orca worktree identity for $child_id is unavailable; refusing provider removal" >&2 + return 1 fi - fm_backend_remove_worktree "$child_backend" "$child_orca_worktree_id" || return 1 elif [ -n "$child_wt" ] && [ -d "$child_wt" ]; then validate_child_worktree_for_removal "$child_wt" "$child_proj" >/dev/null || return 1 - rm -f "$child_wt/.claude/settings.local.json" "$child_wt/.opencode/plugins/fm-turn-end.js" "$child_wt/.fm-grok-turnend" - if [ -n "$child_proj" ] && [ -d "$child_proj" ] && command -v treehouse >/dev/null 2>&1; then - if teardown_treehouse_return "$child_wt" "$child_proj" "child worktree"; then + if [ -n "$child_proj" ] && [ -d "$child_proj" ]; then + if ! command -v treehouse >/dev/null 2>&1; then + echo "error: retained child worktree $child_wt because Treehouse is unavailable; install or restore treehouse, then retry teardown" >&2 + return "$FM_CHECKOUT_TREEHOUSE_RETURN_UNAVAILABLE_STATUS" + fi + CHILD_RETURN_META=$child_meta + CHILD_RETURN_ID=$child_id + if teardown_treehouse_return "$child_wt" "$child_proj" "child worktree" \ + "firstmate-$child_id" validate_child_worktree_return_safety cleanup_returned_worktree; then : else child_return_rc=$? - if [ "$child_return_rc" -eq "$TEARDOWN_TREEHOUSE_LOCK_REFUSED" ]; then - return "$child_return_rc" - fi - safe_rm_rf_child_worktree "$child_wt" "$child_proj" + case "$child_return_rc" in + "$FM_CHECKOUT_LOCK_CONTENTION_STATUS") + echo "error: retained child worktree $child_wt because its common checkout mutation lock is busy" >&2 + ;; + "$FM_CHECKOUT_TREEHOUSE_RETURN_TIMEOUT_STATUS") + echo "error: retained child worktree $child_wt because its Treehouse return timed out" >&2 + ;; + *) + echo "error: retained child worktree $child_wt because its locked Treehouse return failed (status $child_return_rc); resolve the Treehouse failure, then retry teardown" >&2 + ;; + esac + return "$child_return_rc" fi + CHILD_RETURN_META= + CHILD_RETURN_ID= else + validate_child_worktree_landed_state "$child_meta" "$child_id" "$child_wt" "$child_proj" || return 1 safe_rm_rf_child_worktree "$child_wt" "$child_proj" fi fi remove_grok_turnend_auth "$sub_state" "$child_id" rm -f "$sub_state/$child_id.status" "$sub_state/$child_id.turn-ended" "$sub_state/$child_id.check.sh" "$sub_state/$child_id.meta" "$sub_state/$child_id.pi-ext.ts" "$sub_state/$child_id.grok-turnend-token" [ -z "$child_account_lock" ] || fm_account_lifecycle_lock_release "$child_account_lock" >/dev/null 2>&1 || true - done + done <<EOF +$child_metas +EOF +} + +prepare_secondmate_registry_removal() { + local id=$1 home=$2 reg=$3 registry_lock=$4 tmp backup reg_dir + fm_account_lifecycle_lock_owned "$registry_lock" || return 1 + require_registered_secondmate_home "$reg" "$id" "$home" || return 1 + fm_account_safe_file_destination "$reg" || return 1 + reg_dir=$(dirname "$reg") + tmp=$(mktemp "$reg_dir/.secondmates.XXXXXX") || return 1 + backup=$(mktemp "$reg_dir/.secondmates-backup.XXXXXX") || { + rm -f "$tmp" + return 1 + } + if ! fm_account_system_perl - "$reg" "$tmp" "$backup" "$id" <<'PERL' + my ($source, $destination, $backup, $expected) = @ARGV; + open my $input, q{<}, $source or exit 1; + open my $output, q{>}, $destination or exit 1; + open my $saved, q{>}, $backup or exit 1; + my $removed = 0; + while (my $line = <$input>) { + print {$saved} $line or exit 1; + if ($line =~ /^- ([A-Za-z0-9][A-Za-z0-9._-]*) / && $1 eq $expected) { + ++$removed; + next; + } + print {$output} $line or exit 1; + } + close $output or exit 1; + close $saved or exit 1; + exit 1 if $removed != 1; +PERL + then + rm -f "$tmp" "$backup" + return 1 + fi + printf '%s\t%s\n' "$tmp" "$backup" +} + +activate_secondmate_registry_removal() { + local reg=$1 registry_lock=$2 tmp=$3 + fm_account_lifecycle_lock_owned "$registry_lock" || return 1 + [ -f "$tmp" ] && [ ! -L "$tmp" ] || return 1 + fm_account_safe_file_destination "$reg" || return 1 + mv "$tmp" "$reg" +} + +rollback_secondmate_registry_removal() { + local reg=$1 registry_lock=$2 backup=$3 + fm_account_lifecycle_lock_owned "$registry_lock" || return 1 + [ -f "$backup" ] && [ ! -L "$backup" ] || return 1 + fm_account_safe_file_destination "$reg" || return 1 + mv "$backup" "$reg" +} + +validate_pending_orca_worktree_identity() { + local project_root worktree_root project_common worktree_common recorded_root + require_safe_task_metadata || return 1 + require_orca_task_metadata_identity "$META" "$ID" || return 1 + project_root=$(exact_git_worktree_root "$PROJ") || { + echo "error: quarantined Orca project is not an exact repository root: ${PROJ:-<missing>}" >&2 + return 1 + } + worktree_root=$(exact_git_worktree_root "$WT") || { + echo "error: quarantined Orca worktree is not an exact repository root: ${WT:-<missing>}" >&2 + return 1 + } + project_common=$(fm_checkout_git_common_dir "$project_root") || return 1 + worktree_common=$(fm_checkout_git_common_dir "$worktree_root") || return 1 + [ "$project_common" = "$worktree_common" ] || { + echo "error: quarantined Orca worktree does not belong to the recorded project" >&2 + return 1 + } + if [ -n "$(meta_value "$META" worktree)" ] && [ -e "$(meta_value "$META" worktree)" ]; then + recorded_root=$(exact_git_worktree_root "$(meta_value "$META" worktree)") || return 1 + [ "$recorded_root" = "$worktree_root" ] || { + echo "error: quarantined Orca worktree path drifted from retained metadata" >&2 + return 1 + } + fi + require_orca_worktree_path_match "$ORCA_WORKTREE_ID" "$worktree_root" +} + +pending_orca_endpoint_absent() { + local state + if [ -n "$T" ]; then + state=$(fm_backend_target_state orca "$T" "fm-$ID" "$ORCA_WORKTREE_ID") + case "$state" in + absent) ;; + present) ;; + *) + echo "error: quarantined Orca terminal identity or state is unproven for $ID" >&2 + return 1 + ;; + esac + fi + fm_backend_quiesce_worktree_terminals orca "$ORCA_WORKTREE_ID" "fm-$ID" "$T" } -remove_secondmate_registry_entry() { - local id=$1 tmp - [ -f "$SECONDMATE_REG" ] || return 0 - fm_account_safe_file_destination "$SECONDMATE_REG" || return 1 - tmp=$(mktemp "$DATA/.secondmates.XXXXXX") || return 1 - grep -vE "^- $id( |$)" "$SECONDMATE_REG" > "$tmp" || true - fm_account_safe_file_destination "$SECONDMATE_REG" || { rm -f "$tmp"; return 1; } - mv "$tmp" "$SECONDMATE_REG" +remove_pending_orca_worktree_locked() { + local boundary_token + validate_pending_orca_worktree_identity || return 1 + pending_orca_endpoint_absent || return 1 + validate_worktree_teardown_safety || return 1 + validate_pending_orca_worktree_identity || return 1 + validate_removal_tree_boundaries "$WT" "quarantined Orca worktree" || return 1 + validate_pending_orca_worktree_identity || return 1 + boundary_token=$(removal_tree_boundary_token "$WT" "quarantined Orca worktree") || return 1 + fm_backend_remove_worktree_bound \ + orca "$ORCA_WORKTREE_ID" "$WT" "$boundary_token" } +if [ "$ORCA_CLEANUP_PENDING" = 1 ]; then + [ "$KIND" != secondmate ] || { + echo "error: Orca cleanup quarantine cannot describe a secondmate" >&2 + exit 1 + } + if [ -z "$ORCA_WORKTREE_ID" ]; then + echo "error: quarantined Orca worktree id remains unavailable for $ID; refusing to close an unscoped terminal and retaining metadata for provider-assisted recovery by recorded project and task label" >&2 + exit 1 + fi + WT=$(fm_backend_worktree_path orca "$ORCA_WORKTREE_ID") || { + echo "error: quarantined Orca worktree path remains unprovable for $ID" >&2 + exit 1 + } + validate_pending_orca_worktree_identity || exit 1 + pending_orca_endpoint_absent || exit 1 + fm_checkout_lock_run "$WT" "$CHECKOUT_LOCK_ROOT" remove_pending_orca_worktree_locked || exit 1 + remove_grok_turnend_auth "$STATE" "$ID" + fm_backend_clear_transition "$BACKEND" "$STATE" "$T" || true + safe_remove_task_tmp "$TASK_TMP" || exit 1 + rm -f "$STATE/$ID.status" "$STATE/$ID.turn-ended" "$STATE/$ID.check.sh" "$STATE/$ID.meta" "$STATE/$ID.pi-ext.ts" "$STATE/$ID.grok-turnend-token" + [ -z "$ACCOUNT_DELETE_LOCK" ] || fm_account_lifecycle_lock_release "$ACCOUNT_DELETE_LOCK" >/dev/null 2>&1 || true + ACCOUNT_DELETE_LOCK= + echo "teardown $ID complete (Orca cleanup quarantine cleared)" + exit 0 +fi + if [ "$KIND" = secondmate ]; then [ -n "$HOME_PATH" ] || HOME_PATH=$WT - validate_firstmate_home_for_removal "$HOME_PATH" "secondmate home" "$ID" >/dev/null || exit 1 + [ "$PRELOCK_KIND" = secondmate ] && [ "$PRELOCK_HOME" = "$HOME_PATH" ] || { + echo "error: secondmate home identity changed while teardown waited for lifecycle ownership" >&2 + exit 1 + } + validate_firstmate_home_for_removal \ + "$HOME_PATH" "secondmate home" "$ID" "$FM_ROOT" "$SECONDMATE_REG" "$PROJ" 0 \ + >/dev/null || exit 1 if [ "$FORCE" = "--force" ]; then validate_firstmate_home_children_removal "$HOME_PATH" || exit 1 - quiesce_secondmate_endpoint || exit 1 fi -fi - -if [ "$KIND" = secondmate ] && [ "$FORCE" != "--force" ]; then - SUB_STATE="$HOME_PATH/state" - if [ -d "$SUB_STATE" ]; then - for child_meta in "$SUB_STATE"/*.meta; do - [ -e "$child_meta" ] || continue + quiesce_secondmate_endpoint || exit 1 + if [ "$FORCE" = "--force" ]; then + validate_firstmate_home_children_removal "$HOME_PATH" || exit 1 + else + SUB_STATE="$HOME_PATH/state" + CHILD_METAS=$(secondmate_state_metadata "$HOME_PATH") || exit 1 + if [ -n "$CHILD_METAS" ]; then + child_meta=${CHILD_METAS%%$'\n'*} echo "REFUSED: secondmate $ID still has in-flight work in $SUB_STATE." >&2 - echo "Found $(basename "$child_meta"). Let that home finish or explicitly discard with --force." >&2 + echo "Found $(basename "$child_meta"). Let that home finish, or use --force to retire only after every child is proven landed and quiescent." >&2 exit 1 - done + fi fi fi -if [ "$KIND" = scout ] && [ "$FORCE" != "--force" ]; then +if [ "$KIND" = scout ]; then REPORT="$DATA/$ID/report.md" if [ ! -f "$REPORT" ]; then echo "REFUSED: scout task $ID has no report at $REPORT." >&2 - echo "The report is the work product. Have the crewmate write it, or use --force after explicit discard approval." >&2 + echo "The report is the work product. Have the crewmate write it, then retry teardown." >&2 exit 1 fi fi +[ "$KIND" = secondmate ] || validate_teardown_target_identity || exit 1 + PROBE_HOME= ENDPOINT_HOME=$(fm_backend_endpoint_home "$BACKEND" "$KIND" "$FM_HOME" "$HOME_PATH") [ "$ENDPOINT_HOME" = "$FM_HOME" ] || PROBE_HOME=$ENDPOINT_HOME -quiesce_completion_report_endpoint() { - local endpoint_status zellij_tab +quiesce_task_endpoint() { + local endpoint_status zellij_tab scoped_target if [ "$MANAGED_ACCOUNT" = 1 ]; then quiesce_managed_account_endpoint "$META" "$ID" "$PROBE_HOME" return $? fi zellij_tab=$(meta_value "$META" zellij_tab_id) + scoped_target=$(meta_value "$META" tmux_session_target) + [ "$BACKEND" != orca ] || scoped_target=$ORCA_WORKTREE_ID + if [ "$BACKEND" = orca ]; then + quiesce_authoritative_orca_endpoint "$T" "$ORCA_WORKTREE_ID" "fm-$ID" || { + echo "error: task Orca endpoint authority or quiescence is unproven for $ID; retaining metadata" >&2 + return 1 + } + return 0 + fi + if managed_endpoint_is_gone "$BACKEND" "$T" "fm-$ID" "$PROBE_HOME" "$scoped_target"; then + return 0 + else + endpoint_status=$? + fi + if [ "$endpoint_status" -eq 2 ]; then + echo "error: task endpoint identity or state for $ID is unknown; retaining metadata" >&2 + return 1 + fi + if [ -n "$T" ]; then + if [ -n "$PROBE_HOME" ]; then + ( unset FM_ROOT_OVERRIDE; FM_HOME="$PROBE_HOME" FM_ROOT="$PROBE_HOME" fm_backend_kill "$BACKEND" "$T" "$zellij_tab" "fm-$ID" "$(meta_value "$META" tmux_session_target)" ) 2>/dev/null || { + echo "error: failed to stop task endpoint for $ID; retaining metadata" >&2 + return 1 + } + else + fm_backend_kill "$BACKEND" "$T" "$zellij_tab" "fm-$ID" "$(meta_value "$META" tmux_session_target)" 2>/dev/null || { + echo "error: failed to stop task endpoint for $ID; retaining metadata" >&2 + return 1 + } + fi + fi + if managed_endpoint_is_gone "$BACKEND" "$T" "fm-$ID" "$PROBE_HOME" "$scoped_target"; then + return 0 + else + endpoint_status=$? + fi + if [ "$endpoint_status" -eq 2 ]; then + echo "error: task endpoint state for $ID is unknown; retaining metadata" >&2 + else + echo "error: task endpoint for $ID is still alive; retaining metadata" >&2 + fi + return 1 +} + +quiesce_retained_direct_spawn_endpoint() { + local endpoint_status zellij_tab + zellij_tab=$(meta_value "$META" zellij_tab_id) if [ -n "$T" ]; then if [ -n "$PROBE_HOME" ]; then ( unset FM_ROOT_OVERRIDE; FM_HOME="$PROBE_HOME" FM_ROOT="$PROBE_HOME" fm_backend_kill "$BACKEND" "$T" "$zellij_tab" "fm-$ID" "$(meta_value "$META" tmux_session_target)" ) 2>/dev/null || true @@ -1209,53 +3973,54 @@ quiesce_completion_report_endpoint() { endpoint_status=$? fi if [ "$endpoint_status" -eq 2 ]; then - echo "error: completion-report endpoint state for $ID is unknown; retaining metadata" >&2 + echo "error: retained direct-spawn endpoint state for $ID is unknown; retaining its worktree and metadata" >&2 else - echo "error: completion-report endpoint for $ID is still alive; retaining metadata" >&2 + echo "error: retained direct-spawn endpoint for $ID is still alive; retaining its worktree and metadata" >&2 fi return 1 } -report_gated_safety_refusal() { - [ "$REPORT_GATED" = 1 ] || return 0 - echo "The completion-report endpoint has already been shut down; the worktree and task metadata are preserved for a safe retry." >&2 +post_quiescence_safety_refusal() { + [ "$KIND" != secondmate ] || return 0 + echo "The task endpoint has already been shut down; the worktree and task metadata are preserved for a safe retry." >&2 } -if [ "$REPORT_GATED" = 1 ]; then - quiesce_completion_report_endpoint || exit 1 +if [ "$DIRECT_SPAWN_CLEANUP" = pending ]; then + quiesce_retained_direct_spawn_endpoint || exit 1 + validate_teardown_target_identity || { post_quiescence_safety_refusal; exit 1; } +elif [ "$KIND" != secondmate ]; then + quiesce_task_endpoint || exit 1 + validate_teardown_target_identity || { post_quiescence_safety_refusal; exit 1; } fi -if [ "$BACKEND" = orca ] && [ "$KIND" != scout ] && [ "$KIND" != secondmate ] && [ "$FORCE" != "--force" ]; then +if [ "$BACKEND" = orca ] && [ "$KIND" != secondmate ]; then if ! inspectable_git_worktree "$WT"; then - echo "REFUSED: Orca ship task $ID has no inspectable git worktree at ${WT:-<missing>}." >&2 - echo "Cannot verify dirty or unlanded work; restore the worktree path or get explicit OK to discard, then --force." >&2 - report_gated_safety_refusal + echo "REFUSED: Orca task $ID has no inspectable git worktree at ${WT:-<missing>}." >&2 + echo "Cannot verify dirty or unlanded work; restore the worktree path, then retry teardown." >&2 + post_quiescence_safety_refusal exit 1 fi - require_orca_worktree_path_match "$ORCA_WORKTREE_ID" "$WT" || { report_gated_safety_refusal; exit 1; } + require_orca_worktree_path_match "$ORCA_WORKTREE_ID" "$WT" || { post_quiescence_safety_refusal; exit 1; } ORCA_PATH_MATCH_VERIFIED=1 fi -if [ -d "$WT" ] && [ "$FORCE" != "--force" ]; then +if [ -d "$WT" ]; then if validate_worktree_teardown_safety; then : else safety_rc=$? if [ "$safety_rc" -eq "$TEARDOWN_WORKTREE_SAFETY_LOCK_BLOCKED" ]; then - cleanup_stale_lock_for_safety_check "$WT" || { report_gated_safety_refusal; exit 1; } - validate_worktree_teardown_safety || { report_gated_safety_refusal; exit 1; } + cleanup_stale_lock_for_safety_check "$WT" || { post_quiescence_safety_refusal; exit 1; } + validate_worktree_teardown_safety || { post_quiescence_safety_refusal; exit 1; } else - report_gated_safety_refusal + post_quiescence_safety_refusal exit 1 fi fi fi -# New tasks quiesce their endpoint, restore any pending rollback generation, -# and fail closed on their machine-global completion report before lease release -# or worktree removal. Tasks already in flight when this feature lands have no -# report_required marker and retain the legacy teardown contract. --force is an -# explicit discard, not a completion. +# Report-gated tasks restore any pending rollback generation and fail closed on +# their machine-global completion report before lease release or worktree removal. if [ "$REPORT_GATED" = 1 ]; then if [ "$MANAGED_ACCOUNT" = 1 ]; then reconcile_managed_account_rollback "$META" "$ID" "$DATA" || exit $? @@ -1269,64 +4034,126 @@ if [ "$MANAGED_ACCOUNT" = 1 ]; then fi if [ "$KIND" = secondmate ] && [ "$FORCE" = "--force" ]; then - cleanup_firstmate_home_children "$HOME_PATH" + cleanup_firstmate_home_children "$HOME_PATH" || exit 1 fi -# Best-effort: drop the local task branch so the shared repo does not accumulate refs. +[ "$KIND" = secondmate ] || validate_teardown_target_identity || exit 1 + +remove_orca_worktree_locked() { + local branch=HEAD boundary_token + validate_teardown_target_identity || return 1 + fm_backend_quiesce_worktree_terminals orca "$ORCA_WORKTREE_ID" "fm-$ID" "$T" || return 1 + validate_worktree_teardown_safety || return 1 + validate_teardown_target_identity || return 1 + validate_removal_tree_boundaries "$WT" "Orca worktree" || return 1 + validate_teardown_target_identity || return 1 + if [ -d "$WT" ]; then + branch=$(git -C "$WT" rev-parse --abbrev-ref HEAD 2>/dev/null) || return 1 + fi + boundary_token=$(removal_tree_boundary_token "$WT" "Orca worktree") || return 1 + fm_backend_remove_worktree_bound \ + "$BACKEND" "$ORCA_WORKTREE_ID" "$WT" "$boundary_token" || return 1 + if [ "$branch" != "HEAD" ]; then + git -C "$PROJ" branch -D "$branch" >/dev/null 2>&1 || true + fi + remove_worktree_compatibility_artifacts "$WT" "removed Orca worktree" +} + if [ "$BACKEND" = orca ] && [ "$KIND" != secondmate ]; then if [ "$ORCA_PATH_MATCH_VERIFIED" != 1 ]; then require_orca_worktree_path_match_if_present "$ORCA_WORKTREE_ID" "$WT" || exit 1 ORCA_PATH_MATCH_VERIFIED=1 fi - if [ -d "$WT" ]; then - branch=$(git -C "$WT" rev-parse --abbrev-ref HEAD 2>/dev/null || echo HEAD) - if [ "$branch" != "HEAD" ]; then - if git -C "$WT" checkout --detach -q 2>/dev/null; then - git -C "$WT" branch -D "$branch" >/dev/null 2>&1 || true - fi - fi - rm -f "$WT/.claude/settings.local.json" "$WT/.opencode/plugins/fm-turn-end.js" "$WT/.fm-grok-turnend" - fi - if [ "$MANAGED_ACCOUNT" = 0 ]; then - [ -z "$T_ORCA" ] || fm_backend_kill "$BACKEND" "$T" "$(meta_value "$META" zellij_tab_id)" "fm-$ID" "$(meta_value "$META" tmux_session_target)" 2>/dev/null || true - fi - fm_backend_remove_worktree "$BACKEND" "$ORCA_WORKTREE_ID" + fm_checkout_lock_run "$WT" "$CHECKOUT_LOCK_ROOT" remove_orca_worktree_locked || exit 1 elif [ -d "$WT" ] && [ "$KIND" != secondmate ]; then - branch=$(git -C "$WT" rev-parse --abbrev-ref HEAD 2>/dev/null || echo HEAD) - if [ "$branch" != "HEAD" ]; then - if git -C "$WT" checkout --detach -q 2>/dev/null; then - git -C "$WT" branch -D "$branch" >/dev/null 2>&1 || true - fi - fi - # Remove our hook file so a reused pool worktree cannot fire signals for a dead task. - rm -f "$WT/.claude/settings.local.json" "$WT/.opencode/plugins/fm-turn-end.js" "$WT/.fm-grok-turnend" # Kills remaining processes in the worktree (including the agent), resets, returns # to pool. treehouse resolves the pool from the working directory, so run it from # the project. teardown_treehouse_return tolerates transient and stale git locks # left by a killed crew process; see the script header for retry and stale-lock proof. post_lock_cleanup_check= - if [ "$FORCE" != "--force" ] && [ "$KIND" != scout ] && [ "$KIND" != secondmate ]; then + if [ "$KIND" != secondmate ]; then post_lock_cleanup_check=validate_worktree_teardown_safety fi - teardown_treehouse_return "$WT" "$PROJ" "worktree" "$post_lock_cleanup_check" || { + teardown_treehouse_return "$WT" "$PROJ" "worktree" "firstmate-$ID" "$post_lock_cleanup_check" cleanup_returned_worktree || { echo "error: treehouse return failed for worktree $WT; teardown aborted" >&2 exit 1 } fi -if [ "$MANAGED_ACCOUNT" = 0 ] && [ "$BACKEND" != orca ] && [ "$SECONDMATE_ENDPOINT_QUIESCED" = 0 ]; then - fm_backend_kill "$BACKEND" "$T" "$(meta_value "$META" zellij_tab_id)" "fm-$ID" "$(meta_value "$META" tmux_session_target)" 2>/dev/null || true +if [ "$DIRECT_SPAWN_CLEANUP" = pending ] && [ -n "$DIRECT_SPAWN_BACKUP" ]; then + case "$DIRECT_SPAWN_BACKUP" in + ".$ID.meta.rollback."*) ;; + *) echo "error: invalid direct spawn metadata backup for $ID; retaining cleanup state" >&2; exit 1 ;; + esac + direct_spawn_backup_path="$STATE/$DIRECT_SPAWN_BACKUP" + [ -f "$direct_spawn_backup_path" ] && [ ! -L "$direct_spawn_backup_path" ] || { + echo "error: direct spawn metadata backup is unavailable for $ID; retaining cleanup state" >&2 + exit 1 + } + [ -n "$DIRECT_SPAWN_ARTIFACTS" ] || { + echo "error: direct spawn artifact backup is unavailable for $ID; retaining cleanup state" >&2 + exit 1 + } + direct_spawn_restore_lock=$(fm_account_meta_lock_acquire "$STATE" "$ID") || exit 1 + if ! fm_account_restore_artifacts "$STATE" "$ID" "$DIRECT_SPAWN_ARTIFACTS" "$TASK_TMP" 1 \ + || ! fm_account_meta_merge_extensions "$META" "$direct_spawn_backup_path" \ + || ! fm_account_safe_file_destination "$META" \ + || ! mv "$direct_spawn_backup_path" "$META"; then + fm_account_meta_lock_release "$direct_spawn_restore_lock" >/dev/null 2>&1 || true + echo "error: failed to restore prior task state for $ID; retaining direct spawn cleanup metadata" >&2 + exit 1 + fi + if ! rm -rf "${STATE:?}/${DIRECT_SPAWN_ARTIFACTS:?}"; then + fm_account_meta_lock_release "$direct_spawn_restore_lock" >/dev/null 2>&1 || true + echo "error: failed to remove restored direct spawn artifact backup for $ID" >&2 + exit 1 + fi + fm_account_meta_lock_release "$direct_spawn_restore_lock" || exit 1 + fm_backend_clear_transition "$BACKEND" "$STATE" "$T" || true + [ -z "$ACCOUNT_DELETE_LOCK" ] || fm_account_lifecycle_lock_release "$ACCOUNT_DELETE_LOCK" >/dev/null 2>&1 || true + echo "cleaned failed direct spawn for $ID and restored the prior task generation" + exit 0 fi if [ "$KIND" = secondmate ]; then [ -n "$HOME_PATH" ] || HOME_PATH=$WT - remove_firstmate_home "$HOME_PATH" "secondmate home" "$ID" - remove_secondmate_registry_entry "$ID" + SECONDMATE_REGISTRY_LOCK=$(fm_secondmate_registry_lock_acquire "$CHECKOUT_LOCK_ROOT" "$SECONDMATE_REG") || exit 1 + TEARDOWN_ACCOUNT_LOCKS+=("$SECONDMATE_REGISTRY_LOCK") + validate_firstmate_home_for_removal "$HOME_PATH" "secondmate home" "$ID" "$FM_ROOT" "$SECONDMATE_REG" "$PROJ" >/dev/null || exit 1 + FINAL_CHILD_METAS=$(secondmate_state_metadata "$HOME_PATH") || exit 1 + [ -z "$FINAL_CHILD_METAS" ] || { + echo "error: secondmate $ID gained child state before its final removal boundary" >&2 + exit 1 + } + require_empty_secondmate_registry "$HOME_PATH" || exit 1 + SECONDMATE_REGISTRY_PREPARED=$(prepare_secondmate_registry_removal "$ID" "$HOME_PATH" "$SECONDMATE_REG" "$SECONDMATE_REGISTRY_LOCK") || exit 1 + IFS=$'\t' read -r SECONDMATE_REGISTRY_UPDATE SECONDMATE_REGISTRY_BACKUP <<EOF +$SECONDMATE_REGISTRY_PREPARED +EOF + PREPARED_REGISTRY_PATH=$SECONDMATE_REG + PREPARED_REGISTRY_BACKUP=$SECONDMATE_REGISTRY_BACKUP + PREPARED_REGISTRY_ID=$ID + PREPARED_REGISTRY_HOME=$HOME_PATH + PREPARED_REGISTRY_LOCK=$SECONDMATE_REGISTRY_LOCK + activate_secondmate_registry_removal "$SECONDMATE_REG" "$SECONDMATE_REGISTRY_LOCK" "$SECONDMATE_REGISTRY_UPDATE" || { + rm -f "$SECONDMATE_REGISTRY_UPDATE" "$SECONDMATE_REGISTRY_BACKUP" + exit 1 + } + remove_firstmate_home "$HOME_PATH" "secondmate home" "$ID" "$FM_ROOT" "$SECONDMATE_REG" "$PROJ" || { + rollback_secondmate_registry_removal "$SECONDMATE_REG" "$SECONDMATE_REGISTRY_LOCK" "$SECONDMATE_REGISTRY_BACKUP" + exit 1 + } + rm -f "$SECONDMATE_REGISTRY_BACKUP" + PREPARED_REGISTRY_PATH= + PREPARED_REGISTRY_BACKUP= + PREPARED_REGISTRY_ID= + PREPARED_REGISTRY_HOME= + PREPARED_REGISTRY_LOCK= fi remove_grok_turnend_auth "$STATE" "$ID" fm_backend_clear_transition "$BACKEND" "$STATE" "$T" || true # Remove the per-task temp root (/tmp/fm-<id>/, incl. its gotmp/) recorded by spawn. # Read before the state-file rm below; empty (pre-fix tasks without tasktmp=) is a no-op. -[ -n "$TASK_TMP" ] && rm -rf "$TASK_TMP" +[ -z "$TASK_TMP" ] || safe_remove_task_tmp "$TASK_TMP" || exit 1 rm -f "$STATE/$ID.status" "$STATE/$ID.turn-ended" "$STATE/$ID.check.sh" "$STATE/$ID.meta" "$STATE/$ID.pi-ext.ts" "$STATE/$ID.grok-turnend-token" [ -z "$ACCOUNT_DELETE_LOCK" ] || fm_account_lifecycle_lock_release "$ACCOUNT_DELETE_LOCK" >/dev/null 2>&1 || true if [ "$KIND" != scout ] && [ "$KIND" != secondmate ] && [ "$MODE" != local-only ]; then diff --git a/bin/fm-wake-lib.sh b/bin/fm-wake-lib.sh index cf6c5f9d40f..3489affb26b 100755 --- a/bin/fm-wake-lib.sh +++ b/bin/fm-wake-lib.sh @@ -1,7 +1,12 @@ #!/usr/bin/env bash # Shared durable wake queue and portable lock helpers. -FM_WAKE_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FM_WAKE_LIB_SOURCE=${BASH_SOURCE[0]} +case "$FM_WAKE_LIB_SOURCE" in + */*) FM_WAKE_LIB_DIR="$(cd "${FM_WAKE_LIB_SOURCE%/*}" && pwd)" ;; + *) FM_WAKE_LIB_DIR="$(pwd)" ;; +esac +unset FM_WAKE_LIB_SOURCE FM_WAKE_DEFAULT_ROOT="$(cd "$FM_WAKE_LIB_DIR/.." && pwd)" FM_ROOT="${FM_ROOT_OVERRIDE:-${FM_ROOT:-$FM_WAKE_DEFAULT_ROOT}}" FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" @@ -89,12 +94,14 @@ fm_watcher_healthy() { } fm_lock_clean_known_files() { - local lockdir=$1 + local lockdir=$1 target=$2 confined + confined=$(fm_lock_confined_target "$lockdir" "$target") || return 1 rm -f \ - "$lockdir/pid" \ - "$lockdir/fm-home" \ - "$lockdir/pid-identity" \ - "$lockdir/watcher-path" \ + "$confined/pid" \ + "$confined/fm-home" \ + "$confined/pid-identity" \ + "$confined/process-group" \ + "$confined/watcher-path" \ 2>/dev/null || true } @@ -112,6 +119,24 @@ fm_lock_owner_dir() { mktemp -d "${lock_abs}.owner.XXXXXX" 2>/dev/null } +fm_lock_confined_target() { + local lockdir=$1 target=$2 lock_abs target_abs parent lock_base target_base + lock_abs=$(fm_lock_abs_path "$lockdir") || return 1 + target_abs=$(fm_lock_abs_path "$target") || return 1 + [ -d "$target_abs" ] && [ ! -L "$target_abs" ] || return 1 + parent=$(dirname "$lock_abs") + [ "$(dirname "$target_abs")" = "$parent" ] || return 1 + if [ "$target_abs" != "$lock_abs" ]; then + lock_base=$(basename "$lock_abs") + target_base=$(basename "$target_abs") + case "$target_base" in + "$lock_base".owner.*) ;; + *) return 1 ;; + esac + fi + printf '%s\n' "$target_abs" +} + fm_lock_prepare_owner() { local ownerdir=$1 mypid back mypid=${BASHPID:-$$} @@ -121,30 +146,35 @@ fm_lock_prepare_owner() { } fm_lock_link_owner() { - local lockdir=$1 owner + local lockdir=$1 owner resolved + [ -L "$lockdir" ] || return 1 owner=$(readlink "$lockdir" 2>/dev/null) || return 1 [ -n "$owner" ] || return 1 case "$owner" in - /*) printf '%s\n' "$owner" ;; - *) printf '%s/%s\n' "$(dirname "$lockdir")" "$owner" ;; + /*) resolved=$owner ;; + *) resolved="$(dirname "$lockdir")/$owner" ;; esac + fm_lock_confined_target "$lockdir" "$resolved" } fm_lock_points_to_owner() { - local lockdir=$1 ownerdir=$2 actual - actual=$(readlink "$lockdir" 2>/dev/null) || return 1 - [ "$actual" = "$ownerdir" ] + local lockdir=$1 ownerdir=$2 actual expected + actual=$(fm_lock_link_owner "$lockdir") || return 1 + expected=$(fm_lock_confined_target "$lockdir" "$ownerdir") || return 1 + [ "$actual" = "$expected" ] } fm_lock_discard_owner() { - local ownerdir=$1 + local lockdir=$1 ownerdir=$2 confined [ -n "$ownerdir" ] || return 0 - fm_lock_clean_known_files "$ownerdir" - rmdir "$ownerdir" 2>/dev/null || true + confined=$(fm_lock_confined_target "$lockdir" "$ownerdir") || return 1 + fm_lock_clean_known_files "$lockdir" "$confined" || return 1 + rmdir "$confined" 2>/dev/null || true } fm_lock_remove_stray_owner_link() { local lockdir=$1 ownerdir=$2 stray + [ -d "$lockdir" ] && [ ! -L "$lockdir" ] || return 0 stray="$lockdir/$(basename "$ownerdir")" if [ -L "$stray" ] && [ "$(readlink "$stray" 2>/dev/null || true)" = "$ownerdir" ]; then rm -f "$stray" 2>/dev/null || true @@ -165,23 +195,23 @@ fm_lock_claim() { local lockdir=$1 ownerdir=$2 allowed_steal_owner=${3:-} mypid back mypid=${BASHPID:-$$} if ! { printf '%s\n' "$mypid" > "$ownerdir/pid"; } 2>/dev/null; then - fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 1 fi back=$(cat "$ownerdir/pid" 2>/dev/null || true) if [ "$back" != "$mypid" ]; then - fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 1 fi if ! fm_lock_points_to_owner "$lockdir" "$ownerdir"; then - fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 1 fi if fm_lock_claim_blocked_by_steal "$lockdir" "$allowed_steal_owner"; then if fm_lock_points_to_owner "$lockdir" "$ownerdir"; then rm -f "$lockdir" 2>/dev/null || true fi - fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 1 fi return 0 @@ -192,11 +222,11 @@ fm_lock_try_create() { FM_LOCK_OWNER_DIR= ownerdir=$(fm_lock_owner_dir "$lockdir") || return 1 if [ -e "$lockdir" ] || [ -L "$lockdir" ]; then - fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 1 fi if ! fm_lock_prepare_owner "$ownerdir"; then - fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 1 fi if ln -s "$ownerdir" "$lockdir" 2>/dev/null && fm_lock_points_to_owner "$lockdir" "$ownerdir"; then @@ -210,19 +240,19 @@ fm_lock_try_create() { else fm_lock_remove_stray_owner_link "$lockdir" "$ownerdir" fi - fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 1 } fm_lock_remove_path() { local lockdir=$1 ownerdir if [ -L "$lockdir" ]; then - ownerdir=$(fm_lock_link_owner "$lockdir" 2>/dev/null || true) + ownerdir=$(fm_lock_link_owner "$lockdir") || return 1 rm -f "$lockdir" 2>/dev/null || return 1 - [ -n "$ownerdir" ] && fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 0 fi - fm_lock_clean_known_files "$lockdir" + fm_lock_clean_known_files "$lockdir" "$lockdir" || return 1 rmdir "$lockdir" 2>/dev/null } @@ -239,6 +269,44 @@ fm_lock_mid_acquire_is_fresh() { return 1 } +fm_lock_process_group_guarded() { + local lockdir=$1 ownerdir guard group groups + if [ -L "$lockdir" ]; then + ownerdir=$(fm_lock_link_owner "$lockdir" 2>/dev/null) || return 1 + elif [ -d "$lockdir" ]; then + ownerdir=$lockdir + else + return 1 + fi + guard="$ownerdir/process-group" + [ -e "$guard" ] || [ -L "$guard" ] || return 1 + [ -f "$guard" ] && [ ! -L "$guard" ] || { + FM_LOCK_HELD_PID=unknown + return 0 + } + group=$(cat "$guard" 2>/dev/null || true) + case "$group" in + ''|*[!0-9]*) + FM_LOCK_HELD_PID=unknown + return 0 + ;; + esac + FM_LOCK_HELD_PID=$group + groups=$(ps -axo pgid= 2>/dev/null) || return 0 + if printf '%s\n' "$groups" | awk -v group="$group" '$1 == group { found = 1 } END { exit !found }'; then + return 0 + fi + if [ -L "$lockdir" ]; then + fm_lock_points_to_owner "$lockdir" "$ownerdir" || return 0 + elif [ "$ownerdir" != "$lockdir" ]; then + return 0 + fi + [ "$(cat "$guard" 2>/dev/null || true)" = "$group" ] || return 0 + rm -f "$guard" 2>/dev/null || return 0 + FM_LOCK_HELD_PID= + return 1 +} + fm_lock_recheck_stale_owner() { local lockdir=$1 expected_owner=$2 expected_pid=$3 actual_pid if [ -n "$expected_owner" ]; then @@ -266,6 +334,10 @@ fm_lock_try_acquire() { return 0 fi + if fm_lock_process_group_guarded "$lockdir"; then + return 1 + fi + pid=$(cat "$lockdir/pid" 2>/dev/null || true) if fm_pid_alive "$pid"; then FM_LOCK_HELD_PID=$pid @@ -343,16 +415,18 @@ fm_lock_release() { if [ -L "$lockdir" ]; then ownerdir=$(fm_lock_link_owner "$lockdir" 2>/dev/null || true) [ -n "$ownerdir" ] || return 0 + { [ ! -e "$ownerdir/process-group" ] && [ ! -L "$ownerdir/process-group" ]; } || return 0 pid=$(cat "$ownerdir/pid" 2>/dev/null || true) [ "$pid" = "$current" ] || return 0 fm_lock_points_to_owner "$lockdir" "$ownerdir" || return 0 rm -f "$lockdir" 2>/dev/null || return 0 - fm_lock_discard_owner "$ownerdir" + fm_lock_discard_owner "$lockdir" "$ownerdir" return 0 fi + { [ ! -e "$lockdir/process-group" ] && [ ! -L "$lockdir/process-group" ]; } || return 0 pid=$(cat "$lockdir/pid" 2>/dev/null || true) [ "$pid" = "$current" ] || return 0 - fm_lock_clean_known_files "$lockdir" + fm_lock_clean_known_files "$lockdir" "$lockdir" || return 0 rmdir "$lockdir" 2>/dev/null || true } diff --git a/docs/architecture.md b/docs/architecture.md index 26206fb0645..032abb92ff1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -80,15 +80,15 @@ Herdr is experimental and can be selected explicitly or by runtime auto-detectio Herdr's container shape is workspace-per-home plus tab-per-task: the primary home uses workspace label `firstmate`, secondmate homes use `2ndmate-<secondmate-id>`, and recovery/list-live scopes to the current `FM_HOME`'s workspace. Zellij is experimental and selected only explicitly: treehouse remains its worktree provider too, and its full verification - the resolved "gaps to verify" list from the original design report, the unconditional-exit-0 CLI quirk and its mitigation, the focus-steal-on-new-tab finding, the home-scoped tab-title collision fix, and known gaps - is recorded in `docs/zellij-backend.md`. Zellij's container shape is simpler than herdr's: one shared `firstmate` session, one tab per task, with no per-home workspace split; visible tab titles are scoped by the active home label plus a short hash of the resolved `FM_ROOT` path. -Orca is experimental and selected only explicitly for eligible pre-cutover tasks: Orca owns both worktree and terminal lifecycle, records `orca_worktree_id=` and `terminal=`, and removes worktrees through `orca worktree rm` only after the usual firstmate teardown checks pass. -New tasks refuse Orca; [docs/orca-backend.md](orca-backend.md#eligibility) owns the rationale and exact legacy eligibility rule. +Orca is experimental and legacy-recovery-only. Existing terminal supervision remains explicit, while respawn and destructive teardown fail closed until the provider exposes empirically verified task/worktree/terminal authority and complete terminal inventory. +New tasks refuse Orca; [docs/orca-backend.md](orca-backend.md#eligibility) owns the rationale, supervision boundary, and re-enablement criteria. cmux is experimental, GUI-first, macOS-only, and can be selected explicitly or by runtime auto-detection from its primary `CMUX_WORKSPACE_ID` marker plus documented fallback signals: treehouse remains its worktree provider (cmux is a session provider only, like herdr/zellij), and its full verification - the socket access setup requirement with Automation mode recommended, the read-screen-fails-on-a-fresh-surface finding, the close-surface-refuses-on-the-last-surface finding, the source-verified runtime marker and fallback behavior, and known gaps - is recorded in `docs/cmux-backend.md`. cmux's container shape is one workspace per task with one surface, no per-home container split; workspace titles are scoped by the active home label plus a short hash of the resolved `FM_ROOT` path, and `--secondmate` spawns are refused, mirroring Orca. Codex App support is recorded in `docs/codex-app-backend.md`; it is not selectable as a runtime backend. ## Worktrees, not branches in your checkout -Crewmates never intentionally touch your project clone; [treehouse](https://github.com/kunchenguid/treehouse) pools clean worktrees for new tmux, herdr, zellij, and cmux tasks, while an eligible legacy Orca respawn creates its own worktree. +Crewmates never intentionally touch your project clone; [treehouse](https://github.com/kunchenguid/treehouse) pools clean worktrees for new tmux, herdr, zellij, and cmux tasks. For ship and scout work, `fm-spawn.sh` refuses to launch unless the resolved task path is a real git worktree root that is distinct from the project primary checkout. The firstmate repo has one extra exposure because it can dispatch crewmates to work on itself. @@ -124,17 +124,17 @@ Publication is idempotent, fail-closed, and searchable; [report-stack.md](report ## Dispatch profiles Crewmate and scout dispatch can stay on the static crewmate harness resolved by `config/crew-harness`, or it can use local dispatch profiles in `config/crew-dispatch.json`. -The dispatch file is intentionally judgment-based: firstmate reads the natural-language rules at intake, chooses the best matching rule, resolves that rule directly or through a supported selector, and passes concrete harness/model/effort plus optional Agent Fleet account-pool/profile axes to `fm-spawn.sh`. +The dispatch file is intentionally judgment-based: firstmate reads the natural-language rules at intake, chooses the best matching rule, resolves that rule directly or through a supported selector, and passes concrete harness/model/effort plus optional compatibility account-routing axes to `fm-spawn.sh`. The shell scripts validate the JSON shape and verified harness/effort combinations, and `fm-dispatch-select.sh` owns deterministic selector behavior, but they do not parse task intent or match the natural-language rules. The session-start bootstrap step surfaces either the active rule block or a concise invalid-config line at startup. When the file exists, `fm-spawn.sh` refuses crewmate and scout launches without an explicit harness, so `config/crew-harness` is only automatic when no dispatch profile file is active. Secondmate launches are exempt because they resolve the secondmate harness and any optional secondmate model or effort tokens instead. Unsupported effort values are still recorded in task meta when passed to `fm-spawn.sh`, but the launch template omits any effort flag that the selected harness does not accept. That keeps spawn launch compatible across claude, codex, grok, pi, and opencode while preserving the requested profile for later audit. -Account routing remains backend-neutral only in off and observe modes; production enforcement uses the certified backend contract owned by [configuration.md](configuration.md#agent-fleet-account-routing). -Routing remains default-off and adds no managed account fields or provider wrapper until an explicit account flag or enforce policy enables it; observe performs only a dry-run selection. -For quota-balanced account-pool candidates, dispatch compares Agent Fleet pool summaries and then atomically leases within the winning pool, never mixing that decision with quota-axi's default-account view. -Enforced routing rejects poolless quota-balanced candidates, while off and observe retain the legacy quota-only selector path. +New ship/scout observe and enforce account routing is backend-neutral and uses the direct profile-directory contract owned by [configuration.md](configuration.md#agent-fleet-account-routing). +Routing remains default-off and adds no account field or provider environment override until an explicit account flag or observe/enforce policy enables it. +For quota-balanced account-pool candidates, new dispatch deterministically uses the ordered first profile and passes its compatibility alias to spawn only as the activation signal for direct per-account selection. +The legacy Agent Fleet pool-summary selector is inactive deferred code tracked by `remove-fleet-routing-deadcode`; it is not available to new dispatches. ## Optional secondmates @@ -153,7 +153,7 @@ Secondmates are idle by default: after startup recovery reconciles only work alr When called with `FM_HOME=<this-firstmate-home>` or when `FM_HOME` is already set to the active firstmate home, metadata-routed `fm-send.sh` requests to a live `kind=secondmate` are prefixed with the from-firstmate marker from `bin/fm-marker-lib.sh`, so the secondmate returns terse answers through status lines and detailed answers through docs plus status pointers instead of replying only in its own chat. Explicit backend-target sends and direct human typing stay unmarked, so captain intervention in a secondmate pane remains conversational. After seeding a secondmate, `fm-backlog-handoff.sh` validates the fleet-specific handoff, then atomically delegates already-judged in-scope queued item moves to `tasks-axi mv` so the domain queue starts in the right place. -Idle secondmate panes are healthy; teardown is explicit and refuses while the secondmate home has in-flight work unless the captain has approved discard with `--force`. +Idle secondmate panes are healthy; teardown is explicit, proves the home identity and landed state before quiescing its endpoint, repeats those checks at the locked removal boundary, and uses `--force` only to recursively retire children that independently pass every safety proof. Secondmate homes converge conservatively to the primary's version and declared inheritable configuration at launch and during locked session start. The [`secondmate-provisioning` skill](../.agents/skills/secondmate-provisioning/SKILL.md) owns the full guarded sync, propagation, nudge, and mid-session configuration-push contract. @@ -229,13 +229,24 @@ Generalizable firstmate knowledge goes to shared tracked docs through the normal ## Local clones stay fresh -The locked session-start bootstrap step, PR-based teardown, and merged-PR wake handling refresh remote-backed project clones when the clone is safe to move. -Wake-time refreshes can target a single clone by project name, so the primary home also catches up when a secondmate reports a merge from its own home. +The installed checkout-refresh owner polls tracked upstream default-branch tips independently of Firstmate tasks and reacts to changes from any contributor. +A 15-minute full-refresh backstop bounds drift when a signal is missed, while the locked session-start sweep, PR-based teardown, and merged-PR wake handling remain additional refresh paths. +The covered set includes exact Git roots for the active home's project clones, Treehouse backing checkouts, configured paths, and matching-origin top-level clones such as a parallel checkout under `$HOME`. +Uninspectable active-home projects, unreadable or unenumerable scan roots and repository origins, invalid checkout-refresh configuration, malformed or unreadable Treehouse state, changed or missing prior checkout identities, unsafe alert state, and failed skill inventories invalidate coverage health until discovery and hygiene inspection complete. +Treehouse skips dirty pool entries and fetches origin when Firstmate requests a durable task lease. +Firstmate verifies each task lease and every leased or explicit secondmate home is clean, belongs to the requested repository, and matches the upstream tip before creating an endpoint, seeding, or launching. +Remote-free local-only acquisitions use the requested repository's local default tip, while stale, dirty, changed, or unverifiable acquisitions are diagnosed and remain leased without destructive return. +Rollback returns a task lease only after re-proving its repository identity, cleanliness, and expected detached tip. Clean default-branch clones fast-forward to `origin/<default>`, and a clean detached HEAD that holds no unique commits is re-attached to the default branch before the same fast-forward path runs. +Registered local-only and remote-free clones are inspected without fetching and must be clean, on their local default branch, and at that branch's proven tip. +The shared mutation path holds one canonical repository lock for refresh, Treehouse acquisition, and every process-tree-bounded Treehouse return, and proves the fetched default ref against the live upstream `HEAD`, while teardown requires its locked pre-fetch and post-fetch probes to agree, so scheduler, preflight, teardown, and merged-PR wake callers neither race nor trust stale `origin/HEAD`. +Every signal probe also surfaces a new or growing inventory of non-ignored untracked files under repository skill directories in covered seed checkouts and Treehouse pool worktrees, while each full safe refresh quantifies all non-ignored untracked files in any dirty-checkout alarm. +That early hygiene signal prevents local skill drafts from silently accumulating until an upstream commit claims the same paths. Dirty clones, non-default branches, detached HEADs with unique commits, diverged defaults, and default branches checked out in another worktree are reported as `STUCK:` with their behind count and left untouched. +Gone local branches are retained unless remote reachability or default-branch content positively proves their work landed. Fetches blocked by an orphaned `.git/packed-refs.lock` use bounded retries and remove the lock only when the shared staleness proof can prove it abandoned; [configuration.md](configuration.md#toolchain) owns the recovery details and tuning knobs. Local-only projects, clones without an origin remote, and fetch failures remain benign skips. -The refresh also prunes local branches whose remote is gone and that no worktree still needs. +The session-start refresh may prune local branches whose remote is gone and that no worktree still needs; the independent cadence and spawn preflight disable pruning. ## Self-updates stay safe diff --git a/docs/configuration.md b/docs/configuration.md index 3b47fc20310..b2feb6dffe4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -25,7 +25,7 @@ The file format is unchanged in both modes; tasks-axi and manual edits produce t The runtime session-provider backend controls where task windows/endpoints are created, captured, sent to, watched, and killed. `tmux` is the verified reference backend (see [`docs/tmux-backend.md`](tmux-backend.md)); `herdr`, `zellij`, and `cmux` are experimental new-task spawn backends, while `orca` is retained only for eligible pre-cutover task recovery (see [`docs/herdr-backend.md`](herdr-backend.md), [`docs/zellij-backend.md`](zellij-backend.md), [`docs/orca-backend.md`](orca-backend.md), and [`docs/cmux-backend.md`](cmux-backend.md)). -Treehouse remains the worktree provider for tmux, herdr, zellij, and cmux, since herdr, zellij, and cmux are session providers only; eligible legacy Orca recoveries use Orca for both the task worktree and terminal endpoint. +Treehouse remains the worktree provider for tmux, herdr, zellij, and cmux, since herdr, zellij, and cmux are session providers only. Orca metadata may describe legacy Orca-owned worktrees and terminals, but lifecycle mutation is disabled until the authority capability documented in `docs/orca-backend.md` is empirically verified. New spawns choose the backend in this order: an explicit `--backend` flag firstmate passes when it spawns a task, then `FM_BACKEND`, then the first non-empty line of local gitignored `config/backend`, then runtime auto-detection from `$TMUX`, `HERDR_ENV=1`, or cmux runtime signals, then default `tmux`. If more than one runtime marker is present, detection resolves innermost-first: `$TMUX` is checked before `HERDR_ENV=1`, which is checked before cmux's primary `CMUX_WORKSPACE_ID` marker and its documented fallback signals - tmux or herdr started from inside a cmux terminal is the innermost, currently-executing layer, while cmux itself (a terminal application, not a nestable multiplexer) is always checked last. See [`docs/cmux-backend.md`](cmux-backend.md#runtime-auto-detection) for why cmux can be selected when `CMUX_WORKSPACE_ID` is absent. @@ -34,7 +34,7 @@ Zellij is never auto-detected; select it through local `config/backend`, `FM_BAC Orca is also never auto-detected and may be selected only under the [`docs/orca-backend.md`](orca-backend.md#eligibility) eligibility contract. Any value other than `tmux`, `herdr`, `zellij`, `orca`, or `cmux` is rejected until another adapter is implemented and verified. `fm-spawn.sh` accepts `tmux`, `herdr`, `zellij`, and `cmux` for new ship and scout tasks; `backend=cmux` still refuses `--secondmate` until its secondmate launch semantics are designed. -Every new task refuses `backend=orca` before any owned mutation, and Orca also refuses `--secondmate`; the Orca guide owns the rationale and exact legacy eligibility rule. +Every new task refuses `backend=orca` before any owned mutation, legacy respawn and destructive teardown currently fail closed on unavailable authority capability, and Orca also refuses `--secondmate`; the Orca guide owns the rationale and re-enablement contract. `codex-app` is not an accepted runtime backend yet; [`docs/codex-app-backend.md`](codex-app-backend.md) owns the Codex App boundary. The session-start secondmate liveness sweep uses a deeper `fm_backend_agent_alive` probe where verified. Today that probe can classify tmux and herdr secondmate endpoints as `alive`, `dead`, or `unknown`; zellij, Orca, and cmux report `unknown` until their own agent-process classifiers are verified. @@ -45,7 +45,7 @@ A backend spawn refusal from a missing dependency, version gate, or unauthentica Task meta records `backend=` only for a non-default backend; an absent `backend=` means `tmux`, preserving existing default-path meta files. A herdr task additionally records `herdr_session=`, `herdr_workspace_id=`, `herdr_tab_id=`, and `herdr_pane_id=`. A zellij task additionally records `zellij_session=`, `zellij_tab_id=`, and `zellij_pane_id=`. -An eligible legacy Orca task additionally records `orca_worktree_id=` and `terminal=`, with `window=fm-<id>` kept as the shared firstmate alias. +A legacy Orca task may additionally record `orca_worktree_id=` and `terminal=`, with `window=fm-<id>` kept as the shared firstmate alias. A cmux task additionally records `cmux_workspace_id=` and `cmux_surface_id=`. Task selectors for `fm-peek.sh`, `fm-send.sh`, and `fm-crew-state.sh` resolve centrally through `fm_backend_resolve_selector`. A selector containing `:` is passed through as an explicit backend endpoint escape hatch. @@ -167,25 +167,40 @@ For Pi secondmate launches, `fm-spawn.sh` starts Pi with `-e` pointed at the sec ## Agent Fleet account routing -Firstmate can route Claude and Codex launches through the machine-global `agent-fleet` CLI without reading profile homes, credentials, quota caches, or Agent Fleet state directly. -Agent Fleet's public, provider-neutral source and installation instructions live under [`tools/agent-fleet`](../tools/agent-fleet/README.md); the sealed cutover installs a regular native front door at the current passwd user's `~/.local/bin/agent-fleet` before enabling routing. -Account routing is default-off, so an unchanged installation makes no Agent Fleet calls, does not wrap the provider launch, and adds no managed account-routing fields to task metadata. -Routing never retrofits live task metadata or migrates existing sessions; the selected runtime backend, including Herdr, remains the observation and attachment layer rather than an account authority. +Firstmate routes new observe and enforce Claude and Codex ship/scout launches directly through account profile directories. +The exact discovery, fresh-usage, health, fallback, and Herdr-hook mechanics are owned by `bin/fm-account-directory.sh`'s header and help output. +The selected provider command receives `CLAUDE_CONFIG_DIR=<home>` or `CODEX_HOME=<home>`, and task metadata records the non-secret `account_home=<home>` for observability. +New ship/scout launches never ask Agent Fleet to enable a profile, establish identity, install a bundle, or acquire a lease. +They invoke Herdr's own integration installer against the selected profile directory and verify its per-profile hook file before launching. +Account credentials remain captain-owned and read-only to Firstmate; selection never authenticates, logs in, or invokes a model. + +Codex health and usage are genuinely readable per account. +Every selection performs a fresh per-account quota read instead of trusting a prior cache, and a Codex directory with no fresh general usage window is skipped. +This means an account re-authenticated immediately before spawn is eligible on that spawn. +Claude's config-directory-specific macOS Keychain credential is not currently distinguishable through non-interactive quota reads. +Claude therefore treats missing quota as unreadable rather than unhealthy, chooses the first real profile directory in stable sort order, and prints a loud `CLAUDE USAGE UNREADABLE` note explaining that keychain/quota-read gap. + +Account routing remains default-off. +An unchanged installation does not select an account directory, does not alter the provider launch, and adds no account field to task metadata. The production mode resolves in this order: explicit `--account-pool` or `--account-profile` enforces routing for that spawn, emergency `--no-account-routing` disables it for that spawn, the single value in local `config/account-routing-mode`, then `off`. The emergency bypass is printed loudly and persisted as `account_routing_emergency_bypass=1` in task metadata. Ambient `FM_ACCOUNT_ROUTING=off` cannot override the authoritative config. The valid modes are `off`, `observe`, and `enforce`. Bootstrap reports an `ACCOUNT_ROUTING` diagnostic when the configured policy is unreadable, contains multiple values, or names any other mode. -`observe` asks Agent Fleet for a read-only, non-leasing dry-run decision against the explicit task worktree, reports the non-secret pool/provider/profile choice, does not wrap the provider launch, and writes no managed account-routing fields even when Agent Fleet is unavailable. -`enforce` is a certified-Herdr-only production path. -It rejects tmux, zellij, cmux, and Orca before Fleet selection, lease acquisition, worktree creation, or endpoint creation because those backends cannot prove that the pane shell was sanitized before startup. -On Herdr it prepares the isolated runtime endpoint and worktree, passes that exact worktree to selection, recovery, execution, and resume, atomically reserves a profile immediately before provider binding, wraps the existing backend-neutral provider command with `agent-fleet exec`, and fails closed on selection, validation, or launch errors. -Without an explicit pool, enforced Claude and Codex tasks use the dynamic pools `claude-crew` and `codex-crew` respectively. -`--account-profile <profile>` pins one dynamic profile, using the supplied `--account-pool` when present and the reserved `explicit` pool otherwise. -Firstmate's direct spawn flags and `config/secondmate-account-pool` accept non-secret pool and profile aliases made only of letters, digits, dot, underscore, and dash, excluding values that begin with dot or dash; `config/crew-dispatch.json` deliberately narrows those fields to an alphanumeric first character. +`observe` and `enforce` both activate direct account-directory selection for a new Claude or Codex ship/scout launch on any supported runtime backend. +Neither mode invokes Agent Fleet selection or leases for a new ship/scout launch. +The existing `--account-pool`, `--account-profile`, and dispatch-profile fields remain compatibility activation inputs for those crewmates while the inactive-code removal is handled separately. +Their legacy aliases do not constrain the new usage-based account choice. +Secondmate integration is deferred: secondmate launches retain their pre-cutover Agent Fleet selection and lease behavior, including `config/secondmate-account-pool`. +Firstmate's spawn flags and `config/secondmate-account-pool` continue to accept aliases made only of letters, digits, dot, underscore, and dash, excluding values that begin with dot or dash; `config/crew-dispatch.json` deliberately narrows those fields to an alphanumeric first character. Account email addresses and filesystem paths are invalid in every input surface. -Managed task metadata records `account_pool=`, `account_profile=`, a home-namespaced and generation-unique `account_task=`, `account_attempt=`, and the real `provider_session_id=` learned from Agent Fleet's SessionStart mapping. -Spawn requires that generation's SessionStart mapping before reporting success, while the watcher can reconcile older managed metadata through `bin/fm-account-session-sync.sh --all`. +Direct ship/scout task metadata records only `account_home=` from this account mechanism. +It never creates `account_pool=`, `account_profile=`, `account_task=`, `account_attempt=`, or `provider_session_id=`. + +Existing ship/scout tasks that already carry `account_profile=` metadata remain legacy Agent Fleet managed generations. +That compatibility path is recovery-only for ordinary crewmates and is not used for any new ship/scout task. +Secondmate launches continue to create and recover legacy Agent Fleet managed generations until their dedicated direct-account integration is designed. +Bootstrap requires Agent Fleet for enforced secondmate routing and when legacy `account_profile=` or pending rollback metadata exists, while new direct ship/scout routing requires `jq`, `quota-axi`, and Herdr's integration installer instead. Same-profile recovery is sticky and fail-closed: `bin/fm-spawn.sh <id> --resume-account` validates existing task metadata and Agent Fleet's session mapping, uses `lease recover` rather than new-task quota selection, resumes the recorded provider session without replaying the brief as a new prompt, and requires a higher monotonic `session_event_seq` from a SessionStart accepted after its local launch gate before committing the recovered lease. Wall-clock `updated_at` remains diagnostic only and never decides launch freshness. Schema-1 mappings remain readable as virtual sequence zero; the next same-binding SessionStart atomically migrates them to schema 2 / sequence 1, while a changed binding is rejected without modifying the legacy record. @@ -195,20 +210,15 @@ Continuation verifies a bounded repository identity before replacement and fails Continuation inherits the predecessor pool only when the provider is unchanged; a provider change with no explicit pool or profile resolves the target provider's standard pool. If predecessor lease or session cleanup fails after that binding, the replacement stays committed with retry metadata, and rerunning the same `--continue-account` command completes cleanup without creating another endpoint or account attempt. If pre-bind rollback cleanup fails, metadata records `account_rollback_cleanup=pending` plus an exact predecessor backup when applicable, and recovery or teardown retries that failed attempt before restoring or recycling task state. -Bootstrap uses that managed recovery path for a confidently dead secondmate, but deliberately defers an unmanaged generation until an operator makes the explicit routing decision owned by the `secondmate-provisioning` skill's "Recovery" section. +Bootstrap uses the managed recovery path for a confidently dead secondmate carrying `account_profile=` and deliberately defers an unmanaged generation until an operator makes the explicit routing decision owned by the `secondmate-provisioning` skill's "Recovery" section. Teardown kills the recorded endpoint and releases the Agent Fleet lease and session mapping only after the backend confirms absence; a live or unknown endpoint state retains metadata and storage for retry. Off and observe mode support the tmux, Herdr, zellij, and cmux session backends. -Production enforce mode supports only a Herdr server carrying Firstmate's live process-bound schema-v2 closed-shell certificate, including the exact SHA-256 and opened-file identity of its content-addressed worker helper and managed terminal configuration. -A manually started, restored, pre-upgrade, or post-update stale Herdr server is not certified and routed spawn refuses it; stop that server only after proving the session idle, then let Firstmate launch it on the next spawn. -Test labs retain fake backends behind their explicit opt-in so backend-neutral routing behavior remains covered without weakening production. -`config/secondmate-account-pool` optionally selects the primary's dynamic pool for secondmate launches when routing is already enabled; it does not activate routing by itself and is deliberately not inherited into the secondmate home. -An explicit per-spawn account pool or profile overrides that secondmate pool, and an explicit profile without a pool uses only the reserved `explicit` pool. +`config/secondmate-account-pool` selects the Agent Fleet pool for secondmate launches when routing is already enabled; it does not activate routing by itself and is deliberately not inherited into the secondmate home. +An explicit per-spawn account pool or profile overrides that secondmate pool. `config/account-routing-mode` is inherited, so a secondmate can apply the same off/observe/enforce policy to its own crewmates while resolving its own pools from dispatch profiles or the standard provider defaults. -Production does not resolve Agent Fleet from ambient `PATH` and never executes `FM_AGENT_FLEET_BIN` or `FM_DISPATCH_AGENT_FLEET` overrides. -Spawn/control operations fail closed on such an override. -Quota-balanced dispatch remains advisory: it surfaces the refusal and falls back to the first ordered candidate, preserving that candidate's `account_pool`; the later spawn still enforces that pool through the fixed production front door, so this fallback cannot become a default-account launch. -It verifies the fixed passwd-home `~/.local/bin/agent-fleet` as a physical current-user/root-owned non-writable regular executable with safe ancestry and reuses that exact path for selection, execution, recovery, and cleanup. -Environment mode and executable overrides exist only behind the unmistakable `FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1` test/lab opt-in. +The legacy secondmate and recovery implementation resolves Agent Fleet through its sealed production front door and keeps its environment overrides behind `FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1`. +Its remaining crew-dispatch pool-summary branch and isolated new-crewmate fixture path are deferred to follow-up task `remove-fleet-routing-deadcode`; neither is a real new ship/scout launch path. +The direct account-directory module has a separate unmistakable `FM_ACCOUNT_DIRECTORY_TEST_LAB=firstmate-account-directory-test-lab-v1` opt-in for deterministic filesystem, quota, and installer fixtures. ## Crew dispatch profiles (config/crew-dispatch.json) @@ -225,27 +235,28 @@ This section is the single owner of the canonical schema and its per-field seman { "when": "<natural-language condition describing a kind of task>", "use": [ - { "harness": "<adapter>", "model": "<optional model>", "effort": "<low|medium|high|xhigh|max, optional>", "account_pool": "<optional Agent Fleet pool>", "account_profile": "<optional pinned profile>" } + { "harness": "<adapter>", "model": "<optional model>", "effort": "<low|medium|high|xhigh|max, optional>", "account_pool": "<optional compatibility activation alias>", "account_profile": "<optional compatibility activation alias>" } ], "select": "<optional strategy>", "why": "<optional rationale that helps firstmate choose>" } ], - "default": { "harness": "<adapter>", "model": "<optional model>", "effort": "<optional effort>", "account_pool": "<optional Agent Fleet pool>", "account_profile": "<optional pinned profile>" } + "default": { "harness": "<adapter>", "model": "<optional model>", "effort": "<optional effort>", "account_pool": "<optional compatibility activation alias>", "account_profile": "<optional compatibility activation alias>" } } ``` Per rule, `when` and `use` are required. `use` may be a single profile object or an ordered array of profile objects; the single-object form stays fully backward-compatible, and every profile needs `harness`. `use.model`, `use.effort`, `use.account_pool`, `use.account_profile`, and `why` are optional. -Account pool and profile fields are valid only for `claude` and `codex`, and selecting either field makes that spawn enforce account routing even when the global mode is off or observe. +Account pool and profile fields are valid only for `claude` and `codex`, and selecting either field activates direct account-directory routing even when the global mode is off or observe. `select` is optional and currently supports `quota-balanced`. Absent `select` means use the first array element, or the only object in the single-object form; the first array element is the deterministic tie-break and the ultimate fallback. `default` is optional. An omitted model or effort means the selected harness uses its own default for that axis. If a selected profile carries an effort value the chosen harness does not accept, `fm-spawn.sh` records the requested `effort=` in task meta for traceability but omits the launch flag, and bootstrap reports the invalid harness/effort pair as a `CREW_DISPATCH` diagnostic when it is visible in the file. `quota-balanced` selection is deterministic and implemented by `bin/fm-dispatch-select.sh`, whose header owns the general-window rules, the 20 point stale-clear freshness margin, vendor-availability handling, and the degrade-to-first-element fallbacks; quota trouble never blocks dispatch. -Any quota-balanced candidate carrying `account_profile` is invalid because a pinned profile is a direct per-spawn override; pool-aware candidates must all carry `account_pool`, and the selector compares only Agent Fleet `pool status` summaries before passing the winning pool to atomic selection. +Any quota-balanced candidate carrying `account_profile` remains invalid for compatibility, and pool-aware candidates must all carry `account_pool`. +The winning profile's account field activates the later per-account direct selection; its legacy alias does not pin an account directory. When account routing is `enforce`, every quota-balanced candidate must carry `account_pool`; `off` and `observe` retain the legacy poolless `quota-axi` compatibility path. That pool-aware path never falls through to `quota-axi` default-account data, preventing one quota source from choosing a provider while a different account source chooses the concrete profile. See [`docs/examples/crew-dispatch.json`](examples/crew-dispatch.json) for a starting point to copy into local `config/crew-dispatch.json`. @@ -256,21 +267,100 @@ If no dispatch rule fits, firstmate uses the dispatch profile `default` when pre Because the spawn backstop is gated by file presence, any fallback path after a missing match, validation error, or missing `jq` still passes a resolved harness explicitly until the file is fixed or removed. Secondmate homes inherit this file from the primary, so a secondmate's own crewmates apply the same dispatch profile behavior. +## Checkout refresh + +`bin/fm-checkout-refresh.sh` keeps worktree seed checkouts current independently of Firstmate's own PR lifecycle. +Its header owns the exact discovery, configuration-file, cadence, state, and command contracts. +The default covered set is every clone under the active home's `projects/`, every backing checkout referenced by a Treehouse pool under `~/.treehouse`, and every top-level clone under `$HOME` whose `origin` URL matches one of those tracked checkouts. +Matching by origin discovers parallel clones such as `~/relvino` without embedding a captain-specific path in shared code. +Optional `path` and shallow `scan` directives in the gitignored `config/checkout-refresh` file extend that set. +Every declared checkout and matching-origin scan result must resolve to its exact canonical Git worktree root, so nested directories can never redirect refresh to an enclosing repository. +Every scan root must also be readable, searchable, and successfully enumerable rather than treating an unreadable directory as empty. +A symlinked or unreadable configuration file, an unknown or malformed directive, or an invalid or unenumerable declared path or scan root makes coverage unhealthy. +An exact Git repository found during a scan must have inspectable remote metadata even when it does not match the currently tracked origins. +The owner persists the canonical path and actual origin identity of every covered checkout plus the identity of each discovered external clone, and retains each prior identity until it is inspectable and unchanged. + +Each `FM_HOME` owns a distinct background identity and state directory, so primary and secondmate homes can cover their own projects without displacing one another. +Checkout-level owner locks are held by the shared `fm-fleet-sync.sh` mutation path, Treehouse acquisition, and every authorized Treehouse return, so home-scoped services, spawn, secondmate seeding, teardown, and merged-PR wake handling serialize overlapping checkouts safely. +Each Treehouse return is process-tree bounded by `FM_TREEHOUSE_RETURN_TIMEOUT`, which defaults to 60 seconds, and the common checkout lock is released only after that process tree is gone. +Each background owner probes every covered checkout's remote default-branch tip every 60 seconds. +Any upstream-tip change triggers `fm-fleet-sync.sh` immediately, regardless of who pushed or merged it. +Every `fm-fleet-sync.sh` invocation repeats the live upstream-default probe after fetching and proves that the fetched ref matches the live tip instead of trusting a checkout's possibly stale `origin/HEAD`. +Each shared checkout mutation is process-tree bounded by `FM_CHECKOUT_REFRESH_SYNC_TIMEOUT`, including direct teardown and merged-PR wake calls. +Teardown holds that same checkout lock while its landed-content fallback probes, fetches, re-probes, and compares the live default, and it retains the worktree unless both live branch identity and tip remain unchanged. +If the upstream default branch changes, a checkout still on the old branch is reported as `STUCK:` and left unchanged under the fast-forward-only posture. +A full safe refresh runs at least every 15 minutes even when no change signal is observed, so transient network failures, a missed probe, or lost local state cannot create unbounded drift. +The owner advances a checkout's successful cadence and tip state only after a benign inspection result, and any failed or unsafe alert-state write keeps coverage unhealthy and forces the next run to reinspect before the backstop. +Every probe also inventories non-ignored untracked files in covered seed checkouts and Treehouse pool worktrees under `.agents/skills`, `.claude/skills`, `.codex/skills`, and `skills`. +Gitignored files are intentional local material and remain outside this collision guard. +A new or growing inventory produces a durable `HYGIENE:` alert immediately, while forced session-start and spawn-preflight checks repeat any unresolved alert for an operator. +An inventory read failure preserves the prior hygiene alert and marks the latest coverage result unhealthy until a complete scan succeeds. +Unreadable active-home project directories, unreadable or malformed Treehouse state, or a covered checkout that becomes uninspectable during either refresh pass surface an actionable diagnostic and mark the latest coverage result unhealthy. +The ordinary safe-refresh warning separately quantifies every non-ignored untracked file and the subset under those skill directories, so other untracked accumulation is bounded by the same 15-minute backstop. +These checks inspect paths only and never delete, move, stash, reset, or edit a draft. +The signal interval and backstop are configurable through `FM_CHECKOUT_REFRESH_INTERVAL` and `FM_CHECKOUT_REFRESH_BACKSTOP`. +Every cadence and spawn-preflight refresh disables gone-branch pruning and retains `fm-fleet-sync.sh`'s fail-safe behavior: a dirty, diverged, or non-default checkout is left untouched and recorded as an alert. +Registered `local-only` checkouts and repositories without `origin` are fully inspected against their proven local `main` or `master` tip without fetching, including when a local-only checkout still has a remote configured. +The forced session-start mode prunes a gone branch only when its tip is reachable from a surviving remote ref or its content is already present in the live default branch. +An unproven branch is retained and surfaced as `STUCK:`. + +Treehouse v2.0 already excludes dirty pool entries, fetches `origin`, and resets only an available clean detached worktree to the freshest default ref. +Firstmate surfaces matching dirty pool entries, acquires the selected path with `treehouse get --lease`, and verifies the durable lease before creating its endpoint. +That synchronous acquisition holds the common-Git-directory mutation lock and is process-tree bounded by `FM_TREEHOUSE_ACQUIRE_TIMEOUT`, which defaults to 60 seconds. +The accepted lease must be clean, belong to the requested repository, have the same origin identity, and match the live upstream default-branch tip. +Remote-free `local-only` acquisitions use the same repository and cleanliness proof, with the requested checkout's local `main` or `master` tip as their freshness authority. +Treehouse-acquired secondmate homes receive the same proof before seeding. +They use the same locked, bounded acquisition entrypoint as ordinary task worktrees. +Explicit secondmate homes are refreshed and must independently match the same live upstream or local default tip before seeding and again before launch. +A stale, dirty, or uninspectable acquisition remains durably leased without forced return and is surfaced for manual recovery. +If an unmanaged spawn fails after publishing metadata or task artifacts, it restores the prior task generation before returning only a worktree whose repository identity, cleanliness, and expected detached tip are re-proven. +That makes the acquisition proof explicit even if the background owner was offline. +Orca is an explicit legacy-recovery-only exception because this change creates no new Orca tasks or acquisitions. + +### Limitations / deferred + +- A checkout without `origin` still uses the local default-tip proof without first proving that its registered project mode is explicitly `local-only`. +- Home-scoped refresh owners still enumerate the shared user-level Treehouse root without filtering pool entries by owning `FM_HOME`. +- Secondmate home acquisition still relies on Treehouse's dirty-entry skip and does not run Firstmate's `pool-preflight` before requesting its durable lease. +- The rare SIGKILL lock-to-guard handoff race, exact wrapped-exit-code fidelity, and setup-failure diagnostic precision remain deferred under `clone-refresh-followup-edges`; current fail-safe behavior may retain or refuse work or require retry, but must not claim healthy coverage or discard unlanded work. + +The consolidated `clone-refresh-followup-edges` follow-up should tighten explicit local-only mode proof, add home-filtered pool ownership, apply pool preflight to secondmate acquisition, and close the three process-ownership edges above. +These are bounded completeness limitations: coverage health fails closed when Treehouse enumeration cannot be proved, and rollback retains any worktree whose repository identity and expected detached tip cannot be re-proven, so the deferred formalization does not create a false healthy-coverage signal or a destructive data-loss path. + +On macOS, bootstrap reports `MISSING: checkout-refresh` until that home's per-user LaunchAgent is installed. +After the captain approves the background owner, install it with: + +```sh +bin/fm-bootstrap.sh install checkout-refresh +``` + +The locked session-start sweep still runs the same broad discovery and a forced refresh, so the service has an operator-visible backstop. +The LaunchAgent is intentionally independent of the Firstmate watcher and continues polling when no fleet task or Firstmate session is active. +Only the LaunchAgent's scheduled invocation advances its heartbeat, while foreground session and manual runs update coverage health without refreshing scheduler liveness. +The heartbeat proves scheduler liveness only, while `ensure` independently requires the latest coverage result to be healthy. +Its definition persists the configured Treehouse root, refresh interval, and backstop, and health validation rejects any installed value that no longer matches the current configuration. +Health validation also binds the loaded job's effective arguments, interval, RunAtLoad state, explicit environment, and complete inherited/default environment surface to that definition before accepting its coverage record. +Inherited launchd variables use a narrow harmless allowlist; undeclared variables, including alternate Git configuration roots and URL-rewrite controls, make the loaded identity untrusted rather than allowing fresh-looking health records to stand in for proven coverage. +macOS launchd is the primary fleet scheduler in this release. +Scheduler installation and health checks dispatch through an adapter seam, while a future Linux cron or systemd adapter remains explicit follow-up work rather than silently claiming coverage today. + ## Toolchain On session start the first mate detects what its required toolchain is missing or too old and lists each problem with either an exact install command or manual instructions. It installs automatically supported tools only after you say go; manual-only tools remain for you to install from the printed instructions. Required tools come in two parts: a universal toolchain every home needs regardless of backend, and a per-backend delta that follows the runtime backend actually resolved for this home. -The universal toolchain is node, python3, git, gh with GitHub auth via `gh auth login`, no-mistakes v1.31.2 or newer, gh-axi, chrome-devtools-axi, lavish-axi, compatible tasks-axi per "Backlog backend" above, and quota-axi. +The universal toolchain is node, python3, git, gh with GitHub auth via `gh auth login`, Perl, no-mistakes v1.31.2 or newer, gh-axi, chrome-devtools-axi, lavish-axi, compatible tasks-axi per "Backlog backend" above, and quota-axi. This section is the single owner of that universal toolchain list; backend guides' prerequisites point here and add only their backend-specific tools. In that list, no-mistakes runs the validation pipeline, gh-axi, chrome-devtools-axi, and lavish-axi cover GitHub, browser, and rich-review operations, and tasks-axi plus quota-axi back backlog mutations and quota-balanced dispatch. The per-backend delta is required only for the backend resolved from `FM_BACKEND`, then `config/backend`, then runtime auto-detection, then default `tmux`, so a home is never told to install a tool an inactive backend or feature would need. -That delta is owned in code by `fm_backend_required_tools` in `bin/fm-backend.sh`: the resolved backend's own session-provider CLI (`tmux`, `herdr`, `zellij`, `orca`, or `cmux`), `jq` for the JSON-emitting experimental adapters (`herdr`, `zellij`, `cmux`) whose spawn and liveness paths parse the backend's JSON output, `nohup` and `perl` for Herdr's portable detached `setsid` server launcher, and the `treehouse` worktree provider for every session-provider-only backend (`tmux`, `herdr`, `zellij`, `cmux`). +That delta is owned in code by `fm_backend_required_tools` in `bin/fm-backend.sh`: the resolved backend's own session-provider CLI (`tmux`, `herdr`, `zellij`, `orca`, or `cmux`), `jq` for the JSON-emitting experimental adapters (`herdr`, `zellij`, `cmux`) whose spawn and liveness paths parse the backend's JSON output, `nohup` for Herdr's portable detached `setsid` server launcher, and the `treehouse` worktree provider for every session-provider-only backend (`tmux`, `herdr`, `zellij`, `cmux`). Backend tool availability uses the adapter's own executable resolver, so bootstrap and spawn agree on supported non-`PATH` locations such as cmux's bundled CLI. An unknown resolved backend emits `BACKEND_INVALID` and blocks dispatch instead of silently dropping its dependency delta or falling back to tmux. -For an eligible pre-cutover task, Orca provides both the task worktree and terminal endpoint (see "Runtime backend" above), so `backend=orca` requires only `orca` on top of the universal toolchain and skips both `treehouse` and every other backend's session CLI. +Legacy Orca metadata refers to an Orca-owned task worktree and terminal endpoint (see "Runtime backend" above). Existing read-only supervision requires the `orca` CLI, while lifecycle mutation remains unavailable until the authority capability gate is backed by real provider evidence. A herdr, zellij, or cmux home is therefore never told `tmux` is missing, and the `treehouse` durable-lease upgrade check runs only for the backends that actually use treehouse. -Bootstrap reports missing Agent Fleet and `jq` whenever local routing mode or dispatch configuration can enforce routing; observe mode degrades to the legacy launch when either is unavailable, and off mode never invokes them. +Bootstrap reports missing `jq`, fixed system Perl, or Herdr whenever local routing mode, dispatch configuration, or existing ship/scout `account_home=` metadata can activate direct account-directory launches. +It reports missing Agent Fleet when enforced secondmate routing is configured or legacy task metadata still carries `account_profile=` or pending rollback cleanup and may need managed recovery. +Observe and enforce modes both use direct account-directory routing for new ship/scout launches; off mode leaves them on the provider's default identity while still allowing recorded direct generations to recover through fresh selection. When `config/crew-dispatch.json` exists, bootstrap also requires `jq` for dispatch profile validation. When X mode is opted in, bootstrap also requires `curl` and `jq` before arming the relay poll shim. `tasks-axi` and `quota-axi` are required bootstrap tools in every profile, the same class as `lavish-axi`. @@ -278,10 +368,11 @@ An absent or incompatible `tasks-axi` reports `MISSING: tasks-axi (install: npm An absent `quota-axi` reports `MISSING: quota-axi (install: npm install -g quota-axi)`; `bin/fm-dispatch-select.sh` still degrades to the first profile at runtime when quota data is unavailable. Bootstrap also reports a `TANGLE:` line when `FM_ROOT` is on a named non-default branch; follow the printed checkout remediation rather than treating it as an installable tool problem. In a read-only session that did not get the fleet lock, the same line is advisory and omits the checkout command. -The locked session-start bootstrap step also runs a best-effort project clone refresh through `fm-fleet-sync.sh`. +The locked session-start bootstrap step also runs a best-effort covered-checkout refresh through `fm-checkout-refresh.sh`, which delegates individual safe updates to `fm-fleet-sync.sh`. It emits `FLEET_SYNC:` for skipped refreshes that may matter, recovered self-heals, and `STUCK:` alarms. +Checkout discovery and configuration diagnostics use the same relay so an omitted configured clone cannot remain silent. Normal completed runs keep local-only and no-origin skips silent. -If bootstrap kills a timed-out refresh, it replays any completed `fm-fleet-sync.sh` output before the aggregate timeout skip so no finished result is lost. +If bootstrap kills a timed-out refresh, it replays any completed checkout-refresh output before the aggregate timeout skip so no finished result is lost. A killed refresh (or a teardown process kill) can leave an orphaned `.git/packed-refs.lock` in a clone, which makes the next refresh's fetch fail with Git's `Unable to create '...packed-refs.lock': File exists`. On that signature only, `fm-fleet-sync.sh` retries the fetch with a bounded wait for the lock to self-clear, then removes the lock and retries once more only when it can prove the lock stale, exactly like the `fm-teardown.sh` `index.lock` recovery. It never removes a live lock, leaves any other failure shape untouched, and prints every wait, retry, and removal to stderr plus a one-line `recovered:` summary to stdout on success so that this session-start relay still surfaces the recovery. @@ -294,9 +385,10 @@ It uses the same live secondmate discovery and propagation helper as bootstrap, That live discovery starts from `state/*.meta` records with `kind=secondmate`; `data/secondmates.md` only backfills `home=` for older or incomplete meta records. Skipped items, such as a destination checkout that does not yet gitignore the item, are visible warnings but not hard failures. -### Herdr detached launcher prerequisites +### Portable process-control prerequisites -Bootstrap requires `nohup` and `perl` to be discoverable on `PATH` as availability gates, while the production Herdr adapter executes only the fixed system binaries documented in [herdr-backend.md](herdr-backend.md#control-plane-and-filesystem-hardening). +Bootstrap requires `perl` on every backend because checkout refresh and account control use it to bound and reap complete process trees. +Herdr additionally requires `nohup`, while its production adapter executes only the fixed system binaries documented in [herdr-backend.md](herdr-backend.md#control-plane-and-filesystem-hardening). Both commands ship with macOS and are commonly supplied by the platform's coreutils and Perl packages on other Unix-like systems. If bootstrap reports either command missing, restore or install the corresponding platform package, confirm it with `command -v nohup` or `command -v perl`, and rerun bootstrap. Bootstrap treats these as manual prerequisites because package names and command exposure differ across supported platforms. @@ -456,6 +548,13 @@ FM_WEDGE_DEMAND_INSPECT_COUNT=3 # consecutive unchanged wedge or permission-s FM_WATCH_TRIAGE_LOG_MAX_BYTES=262144 # size cap for the watcher's absorbed-wake debug log FM_FLEET_SYNC_BOOTSTRAP_TIMEOUT= # optional seconds allowed for bootstrap's best-effort clone refresh; unset/blank defaults to max(20, 5 + 3 * origin-backed-project-count) FM_FLEET_PRUNE=1 # set to 0 to skip pruning local branches whose upstream is gone +FM_CHECKOUT_REFRESH_INTERVAL=60 # seconds between background upstream-tip probes +FM_CHECKOUT_REFRESH_BACKSTOP=900 # maximum seconds between full safe refresh attempts +FM_CHECKOUT_REFRESH_PROBE_TIMEOUT=15 # seconds allowed for one upstream-tip probe +FM_CHECKOUT_REFRESH_SYNC_TIMEOUT=60 # seconds allowed for one checkout refresh +FM_TREEHOUSE_ACQUIRE_TIMEOUT=60 # seconds allowed for one durable task-worktree acquisition +FM_TREEHOUSE_RETURN_TIMEOUT=60 # seconds allowed for one Treehouse worktree return +# FM_TREEHOUSE_ROOT is unset by default; setting it empty is malformed, while a non-empty value overrides ~/.treehouse FM_STALE_WORKTREE_LOCK_AGE_SECS=30 # min mtime age before fm-teardown.sh treats a leftover worktree git index.lock as provably stale FM_TREEHOUSE_RETURN_LOCK_RETRIES=3 # retries after a treehouse return fails on the transient git index.lock signature FM_TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS=1 # seconds fm-teardown.sh waits before each retry after that signature diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index d4a782acefe..b0324a9ed21 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -20,7 +20,7 @@ Prerequisites: - `herdr` itself, protocol 14 or newer (0.7.1 and 0.7.3 verified) - see [herdr.dev](https://herdr.dev) for install instructions. - `jq`, required to parse herdr's JSON output: `brew install jq` (or your platform's package manager). -- `nohup` and `perl`, required by the detached server launcher; [`docs/configuration.md`](configuration.md#herdr-detached-launcher-prerequisites) owns the platform-specific recovery guidance. +- `nohup` and the universal `perl` runtime, required by the detached server launcher; [`docs/configuration.md`](configuration.md#portable-process-control-prerequisites) owns the platform-specific recovery guidance. - The universal firstmate prerequisites - a verified crew harness plus the required toolchain, owned by [`docs/configuration.md`](configuration.md) ("Harness support", "Toolchain"); treehouse still provides the worktree, herdr only provides the session. Select herdr by putting `herdr` in a local `config/backend` file - the durable way to pick it - or by exporting `FM_BACKEND=herdr` when you launch your harness for a one-off session; telling the first mate in chat to use herdr also works. diff --git a/docs/orca-backend.md b/docs/orca-backend.md index 4d0e0b94911..27c611d0141 100644 --- a/docs/orca-backend.md +++ b/docs/orca-backend.md @@ -6,12 +6,13 @@ Firstmate agents operating this backend should load the agent-only [`firstmate-o ## Eligibility -Every new task is report-required, and `backend=orca` refuses it before any owned mutation because Orca has no reliable endpoint-absence proof for report-gated teardown. -Only a pre-existing task whose metadata has no `report_required` marker is eligible to recover or continue operating on Orca; new work must use tmux, Herdr, zellij, or cmux. +Every new task is report-required, and `backend=orca` refuses it before any owned mutation under the legacy-recovery-only policy. +Pre-existing Orca metadata can still be inspected and supervised through non-destructive helpers, but respawn and destructive teardown currently fail closed because the verified CLI evidence does not establish task/worktree/terminal binding or a complete worktree terminal inventory. +New work and legacy respawn must use tmux, Herdr, zellij, or cmux until that capability is empirically verified and the adapter gate is deliberately enabled. ## Setup -For an eligible legacy task, Orca is macOS-only, explicit-only (never auto-detected), and has no secondmate support. +For an existing legacy task, Orca is macOS-only, explicit-only (never auto-detected), and has no secondmate support. Prerequisites: @@ -20,13 +21,13 @@ Prerequisites: - `node`, used by firstmate's adapter to parse Orca's JSON output and to gate spawns on runtime readiness. - The universal firstmate prerequisites - a verified crew harness plus the required toolchain, owned by [`docs/configuration.md`](configuration.md) ("Harness support", "Toolchain") - with `orca` as the only backend-specific tool, since Orca replaces both the session multiplexer CLI and the `treehouse` worktree provider that the other backends require. -Select Orca only for an eligible legacy recovery by passing `--backend orca` or exporting `FM_BACKEND=orca` for that recovery. +Existing legacy Orca metadata is inspection- and supervision-only until lifecycle authority is empirically verified. +Do not pass `--backend orca` or export `FM_BACKEND=orca` to respawn it; use tmux, Herdr, zellij, or cmux for any replacement generation. Do not make Orca the durable `config/backend` for a home that launches new work. It is never auto-detected. -Before an eligible respawn mutates any repo or worktree state, firstmate runs `orca status --json` and requires the app to report `reachable=true` and `state="ready"` - start the Orca app and wait for it to finish loading before respawning. -Spawn fails closed if the runtime is not ready. -The first eligible respawn against a given project also auto-registers that project's repo in Orca (`orca repo add --path`) if it is not already registered - no manual registration step is needed. +Before any future eligible respawn may mutate repo or worktree state, firstmate requires both a ready runtime and the lifecycle-authority capability. +The authority gate currently fails closed before repo registration, worktree creation, or terminal creation. Watching and attaching: Orca owns both the worktree and the terminal for its tasks, so there is nothing to attach to outside the Orca app itself - open the app and find the terminal for the task (recorded as `terminal=<handle>` in the task's meta, with `window=fm-<id>` as the shared firstmate alias). You do not need to open the app for routine supervision: from an active firstmate session, `bin/fm-peek.sh <id>` reads a task's terminal without opening Orca, and `FM_HOME=<this-firstmate-home> bin/fm-send.sh <id> "<text>"` steers it unless `FM_HOME` is already set to the active firstmate home (the stable `fm-<id>` alias also works; Enter and Ctrl-C are supported; Escape is not). @@ -34,12 +35,12 @@ You do not need to open the app for routine supervision: from an active firstmat Do not manufacture pre-cutover metadata or spawn a trivial Orca task for an end-to-end check. Use the focused fake-Orca suites below, or verify the recorded fields and terminal only while performing an actual eligible recovery. -Limitations: `--secondmate` spawns refuse `backend=orca` (secondmate-home semantics need a separate design), Escape is unsupported, Orca is macOS-only and explicit-only, and it exposes no stable CLI version marker, so spawn gates on runtime reachability instead of a version floor - see "Limitations" below for the complete list. +Limitations: lifecycle spawn and destructive teardown are disabled pending verified authority support, `--secondmate` spawns refuse `backend=orca`, Escape is unsupported, Orca is macOS-only and explicit-only, and it exposes no stable CLI version marker. ## Status PR #210 landed the primitive Orca terminal adapter: bounded capture, text send, Enter, Ctrl-C interrupt, and close for already-created Orca terminals. -This follow-up retains the full ship/scout task lifecycle for a task that meets the eligibility contract above: respawn, metadata, send/peek/watch/crew-state routing from metadata, and guarded teardown through Orca. +The verified evidence supports those terminal primitives plus create `id/path`; it does not support the stronger identity and inventory claims needed for safe lifecycle mutation. ## Task Shape @@ -50,7 +51,7 @@ The normal firstmate invariant still applies: a ship or scout task must run outs ## Metadata -An eligible Orca respawn records the normal task fields plus these Orca-specific fields: +The disabled Orca respawn design would record the normal task fields plus these Orca-specific fields: ```text backend=orca @@ -64,10 +65,12 @@ worktree=<absolute path to the Orca-created git worktree> `fm-teardown.sh <id>` uses the same recorded fields after loading `state/<id>.meta`. For Orca, `window=` keeps the stable firstmate alias while `terminal=` carries the stable Orca terminal handle that backend operations use. The recorded `backend=orca` field tells shared call sites to route capture, send, interrupt, and close through `bin/backends/orca.sh` instead of tmux assumptions. +The guarded implementation pre-arms discovery metadata before provider creation and retains every returned identity with `orca_cleanup_pending=1` if rollback cannot prove absence. +Production creation is disabled until provider-supported discovery and complete terminal inventory make that quarantine recoverable without guessing. ## Lifecycle -Eligible legacy respawn: +The disabled legacy respawn design is: 1. Ensure the project repo is registered in Orca, adding it with `orca repo add --path` when needed. 2. Create an independent Orca worktree with `orca worktree create --repo id:<repo> --name fm-<id> --no-parent --setup skip`. @@ -85,18 +88,25 @@ Operation routing: - `fm-watch.sh` treats Orca as a pull backend with no native busy-state primitive, so it falls back to the same terminal-tail busy regex used for tmux, zellij, and cmux. - `fm-crew-state.sh` reads the recorded Orca terminal when no no-mistakes run-step applies. -Teardown: +The guarded teardown design, exercised only in the synthetic authority lab, is: -- Eligible legacy scout teardown still requires `data/<id>/report.md` unless `--force` explicitly discards it; it does not automatically publish that pre-cutover report to the machine-global stack. +- Eligible legacy scout teardown still requires `data/<id>/report.md`; `--force` does not bypass the report. - [report-stack.md](report-stack.md) owns the explicit legacy archival path. - Ship teardown still refuses dirty or unlanded work before any terminal/worktree cleanup. - Ship teardown resolves `orca_worktree_id` back through Orca and verifies it matches the inspected `worktree=` path before removing anything; mismatches or uninspectable paths preserve metadata and fail closed. -- After the existing firstmate safety checks pass, teardown closes the recorded Orca terminal and releases the recorded worktree through `orca worktree rm --worktree id:<orca_worktree_id> --force`. +- Before close, teardown requires Orca's authoritative terminal read to bind the recorded terminal to the recorded worktree id; missing or mismatched identity preserves every resource. +- After the existing firstmate safety checks pass, teardown closes the recorded Orca terminal and requires a `terminal_handle_stale` read result before removal. +- The final project/worktree identity and landed-work checks, provider removal, and post-removal branch cleanup run under the common checkout lock. +- Provider removal also requires a fresh fail-closed filesystem-boundary proof for the exact recorded worktree root and a provider operation that atomically binds the expected canonical path and descendant boundary token through removal; mounted, redirected, missing, identity-drifted, or unsupported provider targets remain quarantined. +- Missing terminal identity, a live terminal, an ambiguous read result, or a close failure retains the worktree and metadata. +- A spawn-abort quarantine without a recorded terminal can proceed only when Orca explicitly reports that the retained worktree has no terminals. - Teardown does not raw-delete Orca worktrees. +- Until both lifecycle authority and identity-bound removal capability are empirically verified, teardown retains all Orca resources before endpoint close or worktree removal. ## Limitations - `--secondmate` spawns still refuse `backend=orca`; secondmate-home semantics need a separate design. +- Respawn and destructive teardown are disabled because the current verified CLI evidence cannot bind the task label, terminal, worktree, and repository or enumerate every attached terminal. - Escape is unsupported because the current Orca terminal send primitive exposes Enter and interrupt-style input but no verified Escape operation. - Orca is explicit-only and is not selected by runtime auto-detection. - Orca currently exposes no stable CLI version or protocol marker. Unlike the herdr/zellij/cmux docs, this backend intentionally gates spawn support on runtime reachability from `orca status --json` rather than a version floor. @@ -105,23 +115,27 @@ Teardown: Real-Orca smoke verification was run against `/usr/local/bin/orca` with `/Applications/Orca.app` reporting bundle version `1.4.116`; `orca status --json` reported `result.runtime.reachable=true` and `result.runtime.state="ready"`. The verified terminal creation handle field is `result.terminal.handle` from `orca terminal create --json`; worktree creation returned `result.worktree.id` and `result.worktree.path` in the same smoke run. +No recorded command/output establishes `worktree.name`, a complete `worktree.terminals` inventory, or an authoritative query binding a terminal to a task label, worktree, and repository. +The adapter therefore does not claim those fields in production and fails lifecycle capability validation before creation or destructive cleanup. Firstmate intentionally ignores speculative terminal-handle shapes such as bare `result.id` and nested `result.worktree.terminal` until a real Orca smoke run proves them. -Fake-Orca tests cover: +The fake-Orca authority lab is synthetic regression scaffolding, not provider evidence. It covers: - helper parsing for repo registration, worktree creation, verified implicit-terminal reuse, terminal creation, terminal sends, and worktree removal; - rejection of undocumented terminal-handle result shapes; +- retention and cleanup of pathless or malformed-create quarantines, including spawn retry refusal while cleanup remains pending; - runtime readiness gating through `orca status --json`; -- eligible legacy `fm-spawn.sh --backend orca` metadata creation and harness launch; +- synthetic legacy `fm-spawn.sh --backend orca` metadata creation and harness launch inside the authority lab; - `fm-peek.sh`, `fm-send.sh`, and `fm-crew-state.sh` routing through recorded Orca metadata; - slash-command popup placeholder handling that requires a second Enter before `fm-send.sh` reports submission; - scout teardown releasing an Orca worktree through `orca worktree rm`; +- terminal-state classification for live reads, stale-handle absence on nonzero exit, ambiguous failures, and terminal/worktree identity drift; - ship teardown failing closed when the recorded Orca worktree id is missing, cannot resolve to a path, or resolves to a different path than `worktree=`. Run the focused suite with: ```sh -tests/fm-backend-orca.test.sh +FM_TEST_FOCUSED=review-round-orca-authority tests/fm-backend-orca.test.sh tests/fm-backend.test.sh tests/fm-bootstrap.test.sh ``` diff --git a/docs/report-stack.md b/docs/report-stack.md index 0194b199c83..947f486422c 100644 --- a/docs/report-stack.md +++ b/docs/report-stack.md @@ -29,7 +29,7 @@ A safety refusal after quiescence preserves all work and metadata but leaves the If a required heading is absent or lacks substantive content, publication names every missing or empty section, identifies the exact report source to edit, and gives the publish and teardown retry commands. Publication failure leaves the prior durable entry unchanged and stops teardown before destructive cleanup, preserving the task for repair and retry. Tasks that were already in flight at cutover lack the marker and retain the earlier teardown contract. -An explicit `--force` teardown is a discard and does not create a completion report. +`--force` does not bypass completion-report publication or any work-retention proof. Retiring a persistent secondmate is also not a completion; ordinary tasks completed inside its home publish to the same machine-global stack. ## Stored entry diff --git a/docs/scripts.md b/docs/scripts.md index f972a54cb99..481f49b0137 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -10,7 +10,8 @@ The shared no-mistakes gate refusal used by every directly invocable mutating co | `fm-session-start.sh` | Compose lock, bootstrap, and wake drain into the single ordered session-start digest | | `fm-autocompact.sh` | Bridge Claude context compaction through a durable anchor and session-start recovery | | `fm-bootstrap.sh` | Detect toolchain and fleet problems, run the locked session-start sweeps, and install approved tools | -| `fm-fleet-sync.sh` | Refresh project clones with safe fast-forwards, self-heals, `STUCK:` reports, branch pruning, and bounded recovery from an orphaned `.git/packed-refs.lock` | +| `fm-checkout-refresh.sh` | Discover worktree seed checkouts, react to live upstream-default changes, surface untracked skill drafts, enforce the timed backstop, and manage the home-scoped scheduler | +| `fm-fleet-sync.sh` | Serialize live-default-proven project refreshes with safe fast-forwards, `STUCK:` reports, branch pruning, and orphaned-lock recovery | | `fm-fleet-snapshot.sh` | Print the read-only structured fleet snapshot JSON (schema `fm-fleet-snapshot.v1`) | | `fm-fleet-view.sh` | Render the fleet snapshot as a human Markdown view | | `fm-bearings-snapshot.sh` | Project the fleet snapshot to the compact TOON bearings view; local-only unless `--include-prs` | @@ -29,8 +30,9 @@ The shared no-mistakes gate refusal used by every directly invocable mutating co | `fm-supervision-instructions.sh` | Render the session-start primary-harness supervision block or the one-line repair instruction | | `fm-home-seed.sh` | Transactionally provision a secondmate home and maintain `data/secondmates.md` | | `fm-spawn.sh` | Spawn, native-resume, or provider-neutrally continue crews on the resolved account and runtime backend | -| `fm-dispatch-select.sh` | Resolve a matched crew-dispatch rule through legacy quota or Agent Fleet pool summaries | -| `fm-account-routing-lib.sh` | Own Firstmate's shell-side Agent Fleet mode, lease, exec, resume, and release contract | +| `fm-dispatch-select.sh` | Resolve a matched crew-dispatch rule through quota or the deferred legacy pool-summary branch | +| `fm-account-directory.sh` | Select a direct Claude/Codex account directory and install its per-profile Herdr hook | +| `fm-account-routing-lib.sh` | Own routing mode plus legacy Agent Fleet lease, exec, resume, and release recovery | | `fm-account-session-sync.sh` | Reconcile real Agent Fleet provider-session mappings into managed task metadata | | `fm-account-continuation.sh` | Build a verified task-owned packet for fresh cross-profile continuation | | `fm-backend.sh` | Runtime-backend selection, meta helpers, selector resolution, and operation dispatch | @@ -60,6 +62,8 @@ The shared no-mistakes gate refusal used by every directly invocable mutating co | `fm-supervision-lib.sh` | Shared in-flight-work-without-fresh-watcher-beacon predicate | | `fm-ff-lib.sh` | Shared guarded fast-forward helper for origin pulls and local secondmate syncs | | `fm-lock-lib.sh` | Shared "is this git lock provably abandoned?" proof used by teardown and fleet-sync | +| `fm-checkout-lock-lib.sh` | Shared common-Git-directory lock identity and ownership for checkout mutation | +| `fm-process-tree-lib.sh` | Shared bounded command runner that terminates and reaps complete process trees | | `fm-config-inherit-lib.sh` | Shared primary-to-secondmate inheritable-config propagation | | `fm-tasks-axi-lib.sh` | Shared backlog-backend selector and `tasks-axi` compatibility probe | | `fm-wake-drain.sh` | Atomically drain queued watcher wakes, then assert watcher liveness | diff --git a/tests/fm-account-directory.test.sh b/tests/fm-account-directory.test.sh new file mode 100755 index 00000000000..ea25c0b2190 --- /dev/null +++ b/tests/fm-account-directory.test.sh @@ -0,0 +1,927 @@ +#!/usr/bin/env bash +# Behavior tests for direct per-account usage selection and Herdr hook setup. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +SELECTOR="$ROOT/bin/fm-account-directory.sh" +TMP_ROOT=$(fm_test_tmproot fm-account-directory-tests) +ACCOUNT_ROOT="$TMP_ROOT/accounts" +FAKEBIN=$(fm_fakebin "$TMP_ROOT") +QUOTA_LOG="$TMP_ROOT/quota.log" +HERDR_LOG="$TMP_ROOT/herdr.log" +TREEHOUSE_LOG="$TMP_ROOT/treehouse.log" + +mkdir -p "$ACCOUNT_ROOT/codex" "$ACCOUNT_ROOT/claude" + +cat > "$FAKEBIN/quota-axi" <<'SH' +#!/usr/bin/env bash +set -u +[ "${1:-}" = --provider ] && [ "${2:-}" = codex ] && [ "${3:-}" = --json ] || exit 64 +account=${CODEX_HOME##*/} +cache_file=$XDG_CACHE_HOME/quota-axi/quotas.json +[ "$XDG_CACHE_HOME" = "$CODEX_HOME/.agent-fleet-quota-cache" ] || exit 65 +[ ! -e "$cache_file" ] || exit 66 +case "${QUOTA_AXI_HOSTILE+x}${AGENT_FLEET_HOSTILE+x}${XDG_CONFIG_HOME+x}" in + '') ;; + *) exit 67 ;; +esac +printf '%s\t%s\n' "$CODEX_HOME" "$XDG_CACHE_HOME" >> "$FM_FAKE_QUOTA_LOG" +remaining=$(cat "$CODEX_HOME/test-remaining") +mkdir -p "$(dirname "$cache_file")" +printf '{"cached":true}\n' > "$cache_file" +if [ "$remaining" = hang ]; then + sleep 30 + exit 0 +fi +if [ "$remaining" = none ]; then + printf '%s\n' '{"providers":[{"provider":"codex","state":{"status":"auth_required"},"windows":[]}]}' + exit 1 +fi +five=${remaining%%,*} +week=${remaining#*,} +cat <<JSON +{"providers":[{"provider":"codex","state":{"status":"fresh"},"windows":[ +{"id":"five_hour","kind":"session","percentRemaining":$five}, +{"id":"weekly","kind":"weekly","percentRemaining":$week}, +{"id":"model:test:5h","kind":"model","percentRemaining":100} +]}]} +JSON +SH +chmod +x "$FAKEBIN/quota-axi" + +cat > "$FAKEBIN/herdr" <<'SH' +#!/usr/bin/env bash +set -u +[ "${1:-}" = integration ] && [ "${2:-}" = install ] || exit 64 +case "${3:-}" in + codex) + [ -n "${CODEX_HOME:-}" ] || exit 65 + printf 'codex\t%s\n' "$CODEX_HOME" >> "$FM_FAKE_HERDR_LOG" + printf '#!/usr/bin/env bash\n' > "$CODEX_HOME/herdr-agent-state.sh" + ;; + claude) + [ -n "${CLAUDE_CONFIG_DIR:-}" ] || exit 66 + printf 'claude\t%s\n' "$CLAUDE_CONFIG_DIR" >> "$FM_FAKE_HERDR_LOG" + mkdir -p "$CLAUDE_CONFIG_DIR/hooks" + printf '#!/usr/bin/env bash\n' > "$CLAUDE_CONFIG_DIR/hooks/herdr-agent-state.sh" + ;; + *) exit 67 ;; +esac +if [ -n "${FM_FAKE_HERDR_DRIFT_WORKTREE:-}" ]; then + git -C "$FM_FAKE_HERDR_DRIFT_WORKTREE" switch --quiet -c drifted-account-prepare || exit 68 +fi +SH +chmod +x "$FAKEBIN/herdr" + +run_selector() { + FM_ACCOUNT_DIRECTORY_TEST_LAB=firstmate-account-directory-test-lab-v1 \ + FM_ACCOUNT_DIRECTORY_ROOT="$ACCOUNT_ROOT" \ + FM_ACCOUNT_DIRECTORY_QUOTA_AXI="$FAKEBIN/quota-axi" \ + FM_ACCOUNT_DIRECTORY_HERDR="$FAKEBIN/herdr" \ + FM_FAKE_QUOTA_LOG="$QUOTA_LOG" FM_FAKE_HERDR_LOG="$HERDR_LOG" \ + "$SELECTOR" "$@" +} + +set_remaining() { + local account=$1 remaining=$2 + mkdir -p "$ACCOUNT_ROOT/codex/$account/.agent-fleet-quota-cache/quota-axi" + printf '%s\n' "$remaining" > "$ACCOUNT_ROOT/codex/$account/test-remaining" + printf '{"stale":true}\n' > "$ACCOUNT_ROOT/codex/$account/.agent-fleet-quota-cache/quota-axi/quotas.json" +} + +reset_accounts() { + rm -rf "$ACCOUNT_ROOT/codex" "$ACCOUNT_ROOT/claude" + mkdir -p "$ACCOUNT_ROOT/codex" "$ACCOUNT_ROOT/claude" + : > "$QUOTA_LOG" + : > "$HERDR_LOG" +} + +test_codex_picks_highest_fresh_minimum_and_skips_no_window() { + local out err + reset_accounts + set_remaining 1 80,40 + set_remaining 2 none + set_remaining 3 90,75 + out=$(QUOTA_AXI_HOSTILE=1 AGENT_FLEET_HOSTILE=1 XDG_CONFIG_HOME=/hostile \ + run_selector select codex 2>"$TMP_ROOT/codex-select.err") + err=$(cat "$TMP_ROOT/codex-select.err") + [ "$out" = "$ACCOUNT_ROOT/codex/3" ] || fail "Codex did not choose the account with the highest minimum remaining usage: $out" + assert_contains "$err" "codex account $ACCOUNT_ROOT/codex/2 skipped: no freshly readable usage window" \ + "Codex no-window account was not visibly skipped" + assert_contains "$err" "selected codex account $ACCOUNT_ROOT/codex/3 with fresh remaining score=75" \ + "Codex selection did not report its fresh score" + [ "$(wc -l < "$QUOTA_LOG" | tr -d ' ')" = 3 ] || fail "Codex selection did not read every discovered account" + pass "Codex selects the highest fresh general-window minimum and skips only unreadable accounts" +} + +test_codex_rechecks_health_on_every_selection() { + local first second calls + reset_accounts + set_remaining 1 30,20 + set_remaining 2 none + first=$(run_selector select codex 2>"$TMP_ROOT/recheck-first.err") + [ "$first" = "$ACCOUNT_ROOT/codex/1" ] || fail "initial Codex selection ignored the only healthy account" + + set_remaining 2 100,100 + second=$(run_selector select codex 2>"$TMP_ROOT/recheck-second.err") + [ "$second" = "$ACCOUNT_ROOT/codex/2" ] || fail "freshly re-authenticated Codex account stayed cached as unhealthy" + calls=$(grep -c "^$ACCOUNT_ROOT/codex/2"$'\t' "$QUOTA_LOG" || true) + [ "$calls" = 2 ] || fail "Codex account health was not re-read on both selections" + pass "Codex health is read fresh at selection time so a newly authenticated account is immediately eligible" +} + +test_codex_fails_when_no_account_has_a_fresh_window() { + local out status + reset_accounts + set_remaining 1 none + set_remaining 2 none + out=$(run_selector select codex 2>&1) + status=$? + expect_code 1 "$status" "Codex selection with no healthy accounts should fail closed" + assert_contains "$out" "no healthy Codex account has a freshly readable usage window" \ + "Codex all-unhealthy failure was not actionable" + pass "Codex refuses selection when every discovered account lacks fresh readable usage" +} + +test_codex_timeout_skips_wedged_account() { + local out err + reset_accounts + set_remaining 1 hang + set_remaining 2 90,85 + out=$(FM_ACCOUNT_DIRECTORY_QUOTA_TIMEOUT_SECONDS=1 \ + run_selector select codex 2>"$TMP_ROOT/codex-timeout.err") + err=$(cat "$TMP_ROOT/codex-timeout.err") + [ "$out" = "$ACCOUNT_ROOT/codex/2" ] || fail "wedged Codex account prevented selection of a later healthy account: $out" + assert_contains "$err" "codex account $ACCOUNT_ROOT/codex/1 skipped: quota read timed out after 1s" \ + "Codex timeout was not classified as an unreadable account" + pass "Codex bounds each usage read and continues to later healthy accounts" +} + +test_claude_uses_stable_first_without_treating_usage_as_health() { + local out err + reset_accounts + mkdir -p "$ACCOUNT_ROOT/claude/2" "$ACCOUNT_ROOT/claude/1" + out=$(run_selector select claude 2>"$TMP_ROOT/claude-select.err") + err=$(cat "$TMP_ROOT/claude-select.err") + [ "$out" = "$ACCOUNT_ROOT/claude/1" ] || fail "Claude fallback did not use stable bytewise directory order: $out" + assert_contains "$err" "CLAUDE USAGE UNREADABLE" "Claude fallback did not carry the required obvious warning" + assert_contains "$err" "config-dir-specific macOS Keychain credential" \ + "Claude fallback did not explain the keychain/quota-read gap" + [ ! -s "$QUOTA_LOG" ] || fail "Claude fallback called quota-axi even though per-directory usage is known unreadable" + pass "Claude deterministically selects the first directory and explains why usage is not a health signal" +} + +test_default_root_uses_passwd_home_not_ambient_home() { + local passwd_home hostile_home expected out + passwd_home="$TMP_ROOT/passwd-home" + hostile_home="$TMP_ROOT/hostile-home" + expected="$passwd_home/.local/share/agent-fleet/accounts/claude/1" + mkdir -p "$expected" "$hostile_home/.local/share/agent-fleet/accounts/claude/0" + out=$(HOME="$hostile_home" \ + FM_ACCOUNT_DIRECTORY_TEST_LAB=firstmate-account-directory-test-lab-v1 \ + FM_ACCOUNT_DIRECTORY_PASSWD_HOME="$passwd_home" \ + "$SELECTOR" select claude 2>"$TMP_ROOT/passwd-home.err") + [ "$out" = "$expected" ] || fail "ambient HOME redirected account discovery away from the passwd home: $out" + pass "default account discovery ignores ambient HOME and stays under the passwd home" +} + +test_prepare_installs_and_verifies_per_account_herdr_hooks() { + local codex_home claude_home + reset_accounts + set_remaining 1 90,80 + mkdir -p "$ACCOUNT_ROOT/claude/1" + + codex_home=$(run_selector prepare codex 2>"$TMP_ROOT/prepare-codex.err") + claude_home=$(run_selector prepare claude 2>"$TMP_ROOT/prepare-claude.err") + [ -f "$codex_home/herdr-agent-state.sh" ] || fail "Codex Herdr hook was not installed in the selected profile home" + [ -f "$claude_home/hooks/herdr-agent-state.sh" ] || fail "Claude Herdr hook was not installed in the selected profile home" + assert_grep $'codex\t'"$ACCOUNT_ROOT/codex/1" "$HERDR_LOG" "Herdr installer did not receive CODEX_HOME" + assert_grep $'claude\t'"$ACCOUNT_ROOT/claude/1" "$HERDR_LOG" "Herdr installer did not receive CLAUDE_CONFIG_DIR" + pass "prepare uses Herdr's own installer and verifies each selected profile hook" +} + +make_spawn_fakebin() { + local fakebin=$1 + cat > "$fakebin/tmux" <<'SH' +#!/usr/bin/env bash +set -u +case "$*" in + *"#{session_name}"*"#{window_name}"*) + [ -f "${FM_FAKE_ENDPOINT_FILE:?}" ] || exit 1 + printf 'firstmate\t%s\n' "${FM_FAKE_ENDPOINT_LABEL:?}" + exit 0 + ;; + *"#{pane_current_path}"*) printf '%s\n' "${FM_FAKE_PANE_PATH:-}"; exit 0 ;; +esac +case "${1:-}" in + display-message) + case " $* " in + *" -t "*) [ -f "${FM_FAKE_ENDPOINT_FILE:?}" ] || exit 1 ;; + esac + printf 'firstmate\n' + exit 0 + ;; + list-windows) exit 0 ;; + has-session|new-session) exit 0 ;; + new-window) + touch "${FM_FAKE_ENDPOINT_FILE:?}" + printf '@1\n' + exit 0 + ;; + kill-window) + [ "${FM_FAKE_KILL_RETAIN:-0}" = 1 ] || rm -f "${FM_FAKE_ENDPOINT_FILE:?}" + exit 0 + ;; + send-keys) + prev= + for argument in "$@"; do + if [ "$prev" = -l ]; then + printf '%s\n' "$argument" >> "$FM_FAKE_LAUNCH_LOG" + fi + prev=$argument + done + exit 0 + ;; +esac +exit 0 +SH + chmod +x "$fakebin/tmux" + cat > "$fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +set -u +printf '%s\n' "$*" >> "${FM_FAKE_TREEHOUSE_LOG:?}" +if [ "${1:-}" = get ]; then + git -C "${FM_FAKE_PANE_PATH:?}" switch --quiet --detach + printf '%s\n' "$FM_FAKE_PANE_PATH" +fi +[ "${1:-}" = return ] || exit 0 +[ "${FM_FAKE_TREEHOUSE_RETURN_FAIL:-0}" != 1 ] || exit 71 +target=${@: -1} +if [ "$target" = . ]; then + target=${FM_FAKE_PANE_PATH:?} +fi +common=$(git -C "$target" rev-parse --git-common-dir) || exit 72 +case "$common" in /*) ;; *) common="$target/$common" ;; esac +git --git-dir="$common" worktree remove --force "$target" +SH + chmod +x "$fakebin/treehouse" + cat > "$fakebin/forbidden-agent-fleet" <<'SH' +#!/usr/bin/env bash +printf 'called\n' >> "$FM_FAKE_AGENT_FLEET_LOG" +exit 99 +SH + chmod +x "$fakebin/forbidden-agent-fleet" +} + +run_direct_spawn() { + local home=$1 worktree=$2 launch_log=$3 + shift 3 + : > "$launch_log" + FM_ROOT_OVERRIDE="${FM_TEST_ROOT_OVERRIDE:-}" FM_HOME="$home" \ + FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ + FM_PROJECTS_OVERRIDE="$home/projects" FM_CONFIG_OVERRIDE="$home/config" \ + FM_SPAWN_NO_GUARD=1 FM_FAKE_PANE_PATH="$worktree" TMUX="fake,1,0" \ + FM_FAKE_LAUNCH_LOG="$launch_log" FM_FAKE_ENDPOINT_FILE="$home/state/.fake-endpoint" \ + FM_FAKE_ENDPOINT_LABEL="fm-${1:-unknown}" FM_FAKE_KILL_RETAIN="${FM_FAKE_KILL_RETAIN:-0}" \ + FM_FAKE_HERDR_DRIFT_WORKTREE="${FM_FAKE_HERDR_DRIFT_WORKTREE:-}" \ + FM_FAKE_TREEHOUSE_LOG="$TREEHOUSE_LOG" \ + FM_FAKE_TREEHOUSE_RETURN_FAIL="${FM_FAKE_TREEHOUSE_RETURN_FAIL:-0}" \ + FM_TREEHOUSE_ROOT="$home/treehouse-pools" \ + PATH="$FAKEBIN:$PATH" \ + FM_ACCOUNT_DIRECTORY_TEST_LAB=firstmate-account-directory-test-lab-v1 \ + FM_ACCOUNT_DIRECTORY_ROOT="$ACCOUNT_ROOT" \ + FM_ACCOUNT_DIRECTORY_QUOTA_AXI="$FAKEBIN/quota-axi" \ + FM_ACCOUNT_DIRECTORY_HERDR="$FAKEBIN/herdr" \ + FM_FAKE_QUOTA_LOG="$QUOTA_LOG" FM_FAKE_HERDR_LOG="$HERDR_LOG" \ + FM_AGENT_FLEET_BIN="$FAKEBIN/forbidden-agent-fleet" \ + FM_FAKE_AGENT_FLEET_LOG="$TMP_ROOT/agent-fleet.log" \ + "$ROOT/bin/fm-spawn.sh" "$@" +} + +make_spawn_case() { + local name=$1 harness=$2 id=$3 case_dir home project worktree launch_log + case_dir="$TMP_ROOT/spawn-$name" + home="$case_dir/home" + project="$case_dir/project" + worktree="$case_dir/worktree" + launch_log="$case_dir/launch.log" + mkdir -p "$home/data/$id" "$home/projects" "$home/state" "$home/config" "$home/treehouse-pools" + printf '%s\n' "$harness" > "$home/config/crew-harness" + printf 'brief for %s\n' "$id" > "$home/data/$id/brief.md" + touch "$home/state/.last-watcher-beat" + fm_git_worktree "$project" "$worktree" "wt-$name" + printf '%s\n' "$home|$project|$worktree|$launch_log" +} + +read_spawn_case() { + IFS='|' read -r SPAWN_HOME SPAWN_PROJECT SPAWN_WORKTREE SPAWN_LAUNCH_LOG <<EOF +$1 +EOF +} + +test_spawn_uses_direct_codex_home_without_agent_fleet() { + local record id out launch meta + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 30,20 + set_remaining 2 100,95 + id=direct-codex-z1 + record=$(make_spawn_case direct-codex codex "$id") + read_spawn_case "$record" + printf '%s\n' enforce > "$SPAWN_HOME/config/account-routing-mode" + + if ! out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" 2>&1); then + fail "direct Codex spawn failed before launch"$'\n'"$out" + fi + launch=$(cat "$SPAWN_LAUNCH_LOG") + meta=$SPAWN_HOME/state/$id.meta + assert_contains "$out" "selected direct codex account home $ACCOUNT_ROOT/codex/2" \ + "spawn did not report its direct Codex account" + assert_contains "$launch" "CODEX_HOME='$ACCOUNT_ROOT/codex/2' codex" \ + "spawn did not scope Codex to the selected account home" + assert_grep "account_home=$ACCOUNT_ROOT/codex/2" "$meta" "spawn metadata omitted the selected account home" + if grep -q '^account_profile=' "$meta"; then fail "new direct spawn wrote legacy managed profile metadata"; fi + if grep -q '^account_pool=' "$meta"; then fail "new direct spawn wrote legacy managed pool metadata"; fi + [ ! -s "$TMP_ROOT/agent-fleet.log" ] || fail "new direct spawn invoked Agent Fleet" + pass "new enforced Codex spawn uses CODEX_HOME and never enters Agent Fleet" +} + +test_spawn_uses_direct_claude_fallback_and_hook() { + local record id out launch meta + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + mkdir -p "$ACCOUNT_ROOT/claude/2" "$ACCOUNT_ROOT/claude/1" + id=direct-claude-z2 + record=$(make_spawn_case direct-claude claude "$id") + read_spawn_case "$record" + + out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-claude-pool 2>&1) + launch=$(cat "$SPAWN_LAUNCH_LOG") + meta=$SPAWN_HOME/state/$id.meta + assert_contains "$out" "CLAUDE USAGE UNREADABLE" "spawn hid the required Claude quota-read warning" + assert_contains "$launch" "CLAUDE_CONFIG_DIR='$ACCOUNT_ROOT/claude/1' claude" \ + "spawn did not scope Claude to the deterministic first account home" + assert_grep "account_home=$ACCOUNT_ROOT/claude/1" "$meta" "Claude spawn metadata omitted account_home" + [ -f "$ACCOUNT_ROOT/claude/1/hooks/herdr-agent-state.sh" ] || fail "Claude spawn did not install its per-account Herdr hook" + [ ! -s "$TMP_ROOT/agent-fleet.log" ] || fail "new direct Claude spawn invoked Agent Fleet" + pass "new account-flagged Claude spawn uses deterministic CLAUDE_CONFIG_DIR with an explicit warning" +} + +test_observe_spawn_uses_direct_directory_without_agent_fleet() { + local record id out launch meta + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 75,70 + id=direct-observe-z3 + record=$(make_spawn_case direct-observe codex "$id") + read_spawn_case "$record" + printf '%s\n' observe > "$SPAWN_HOME/config/account-routing-mode" + + out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" 2>&1) + launch=$(cat "$SPAWN_LAUNCH_LOG") + meta=$SPAWN_HOME/state/$id.meta + assert_contains "$launch" "CODEX_HOME='$ACCOUNT_ROOT/codex/1' codex" \ + "observe launch did not use direct Codex selection" + assert_grep "account_home=$ACCOUNT_ROOT/codex/1" "$meta" "observe metadata omitted the direct account home" + [ ! -s "$TMP_ROOT/agent-fleet.log" ] || fail "observe launch invoked Agent Fleet" + assert_not_contains "$out" "fm-account-routing: observe" "observe launch entered the legacy dry-run selector" + pass "observe mode uses direct account-directory routing without Agent Fleet" +} + +test_direct_spawn_and_recovery_support_detached_worktree() { + local record id meta expected_head out launch + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 90,85 + id=direct-detached-z4 + record=$(make_spawn_case direct-detached codex "$id") + read_spawn_case "$record" + git -C "$SPAWN_WORKTREE" switch --quiet --detach + expected_head=$(git -C "$SPAWN_WORKTREE" rev-parse --verify HEAD) + + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool >/dev/null 2>&1 + meta="$SPAWN_HOME/state/$id.meta" + assert_grep "worktree_git_ref=refs/heads/fm/$id" "$meta" \ + "detached direct spawn did not record its authoritative task branch" + assert_grep "worktree_git_setup_head=$expected_head" "$meta" \ + "detached direct spawn did not retain its exact pre-setup HEAD" + git -C "$SPAWN_WORKTREE" switch --quiet -c "fm/$id" + + rm -f "$SPAWN_HOME/state/.fake-endpoint" + out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1) + launch=$(cat "$SPAWN_LAUNCH_LOG") + assert_contains "$out" "spawned $id harness=codex" \ + "direct recovery rejected an intentionally detached worktree" + assert_contains "$launch" "CODEX_HOME='$ACCOUNT_ROOT/codex/1' codex" \ + "detached direct recovery did not launch with direct account routing" + assert_grep "worktree_git_ref=refs/heads/fm/$id" "$meta" \ + "direct recovery did not preserve the authoritative task branch" + if grep -q '^worktree_git_setup_' "$meta"; then + fail "direct recovery did not adopt the completed task-branch transition" + fi + pass "direct routing safely adopts the required detached-to-task-branch transition" +} + +test_direct_recovery_preserves_recorded_task_context() { + local record id out meta launch project_name generation recorded_project recorded_worktree meta_tmp + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 95,90 + set_remaining 2 30,20 + id=direct-recovery-z4 + record=$(make_spawn_case direct-recovery codex "$id") + read_spawn_case "$record" + + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --harness codex --model gpt-recorded --effort high \ + --account-pool legacy-codex-pool --scout >/dev/null 2>&1 + meta=$SPAWN_HOME/state/$id.meta + generation=$(sed -n 's/^generation_id=//p' "$meta") + recorded_project=$(sed -n 's/^project=//p' "$meta") + recorded_worktree=$(sed -n 's/^worktree=//p' "$meta") + meta_tmp=$(mktemp "$SPAWN_HOME/state/.direct-recovery-meta.XXXXXX") + awk ' + /^mode=/ { print "mode=direct-PR"; next } + /^yolo=/ { print "yolo=on"; next } + { print } + END { print "dispatch_profile_required=1" } + ' "$meta" > "$meta_tmp" + mv "$meta_tmp" "$meta" + project_name=$(basename "$SPAWN_PROJECT") + printf '%s\n' "- $project_name [local-only] - changed policy (added 2026-07-23)" > "$SPAWN_HOME/data/projects.md" + printf '%s\n' '{"version":1,"rules":[],"default":{"harness":"claude"}}' > "$SPAWN_HOME/config/crew-dispatch.json" + set_remaining 1 20,15 + set_remaining 2 90,85 + rm -f "$SPAWN_HOME/state/.fake-endpoint" + + out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1) + launch=$(cat "$SPAWN_LAUNCH_LOG") + assert_contains "$out" "selected direct codex account home $ACCOUNT_ROOT/codex/2" \ + "direct recovery did not select a fresh account directory" + assert_contains "$launch" "CODEX_HOME='$ACCOUNT_ROOT/codex/2' codex" \ + "direct recovery did not launch with the freshly selected account" + assert_contains "$launch" "--model 'gpt-recorded'" \ + "direct recovery did not preserve the recorded model" + assert_contains "$launch" "model_reasoning_effort=\"high\"" \ + "direct recovery did not preserve the recorded effort" + assert_not_contains "$launch" "treehouse get" \ + "direct recovery reconstructed or replaced the recorded worktree" + assert_grep "kind=scout" "$meta" "direct recovery did not preserve scout kind" + assert_grep "project=$recorded_project" "$meta" "direct recovery changed project identity" + assert_grep "worktree=$recorded_worktree" "$meta" "direct recovery changed worktree identity" + assert_grep "worktree_git_dir=" "$meta" "direct recovery dropped the exact worktree Git-dir" + assert_grep "worktree_git_dir_identity=" "$meta" "direct recovery dropped the worktree Git-dir identity" + assert_grep "worktree_git_ref=refs/heads/" "$meta" "direct recovery dropped the worktree branch identity" + assert_grep "harness=codex" "$meta" "direct recovery changed the recorded harness" + assert_grep "model=gpt-recorded" "$meta" "direct recovery changed the recorded model" + assert_grep "effort=high" "$meta" "direct recovery changed the recorded effort" + assert_grep "mode=direct-PR" "$meta" "direct recovery re-resolved the recorded delivery mode" + assert_grep "yolo=on" "$meta" "direct recovery re-resolved the recorded yolo setting" + assert_grep "report_required=1" "$meta" "direct recovery dropped the report requirement" + assert_grep "generation_id=$generation" "$meta" "direct recovery replaced the task generation identity" + assert_grep "dispatch_profile_required=1" "$meta" "direct recovery dropped dispatch-profile metadata" + assert_grep "account_home=$ACCOUNT_ROOT/codex/2" "$meta" "direct recovery did not update account_home" + [ ! -s "$TMP_ROOT/agent-fleet.log" ] || fail "direct recovery invoked Agent Fleet" + pass "direct recovery preserves recorded task context while refreshing account selection" +} + +test_direct_recovery_rejects_secondmate_metadata() { + local record id meta out status + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + id=direct-secondmate-refused-z4 + record=$(make_spawn_case direct-secondmate-refused codex "$id") + read_spawn_case "$record" + meta="$SPAWN_HOME/state/$id.meta" + printf '%s\n' \ + 'kind=secondmate' \ + 'account_home=/accounts/codex/1' \ + > "$meta" + + if out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct recovery accepted secondmate metadata" + assert_contains "$out" "--recover-direct-account supports only recorded ship or scout tasks" \ + "direct recovery did not identify its crewmate-only scope" + [ ! -s "$QUOTA_LOG" ] || fail "rejected secondmate recovery read direct account quota" + [ ! -s "$HERDR_LOG" ] || fail "rejected secondmate recovery installed a direct account hook" + pass "direct account recovery refuses secondmate metadata before selection" +} + +test_direct_recovery_rejects_worktree_from_another_project() { + local record id meta unrelated meta_tmp out status recorded_git_dir + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 90,85 + id=direct-project-identity-z5 + record=$(make_spawn_case direct-project-identity codex "$id") + read_spawn_case "$record" + + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool >/dev/null 2>&1 + meta="$SPAWN_HOME/state/$id.meta" + unrelated="$TMP_ROOT/unrelated-direct-recovery" + fm_git_init_commit "$unrelated" + meta_tmp=$(mktemp "$SPAWN_HOME/state/.direct-project-meta.XXXXXX") + awk -v worktree="$unrelated" ' + /^worktree=/ { print "worktree=" worktree; next } + { print } + ' "$meta" > "$meta_tmp" + mv "$meta_tmp" "$meta" + rm -f "$SPAWN_HOME/state/.fake-endpoint" + : > "$QUOTA_LOG" + : > "$HERDR_LOG" + recorded_git_dir=$(sed -n 's/^worktree_git_dir=//p' "$meta") + + if out=$(GIT_DIR="$recorded_git_dir" GIT_COMMON_DIR="$SPAWN_PROJECT/.git" GIT_WORK_TREE="$unrelated" \ + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct recovery launched in a worktree from another project" + assert_contains "$out" "redirected or unprovable Git metadata" \ + "direct recovery project-identity refusal was not actionable" + [ ! -e "$SPAWN_HOME/state/.fake-endpoint" ] || fail "project-identity mismatch created a replacement endpoint" + [ ! -s "$QUOTA_LOG" ] || fail "project-identity mismatch read account quota before refusing recovery" + [ ! -s "$HERDR_LOG" ] || fail "project-identity mismatch installed a profile hook before refusing recovery" + pass "direct recovery proves the recorded worktree belongs to the recorded project" +} + +test_direct_recovery_requires_recorded_brief() { + local record id out status + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 90,85 + id=direct-brief-z6 + record=$(make_spawn_case direct-brief codex "$id") + read_spawn_case "$record" + + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool >/dev/null 2>&1 + rm -f "$SPAWN_HOME/state/.fake-endpoint" "$SPAWN_HOME/data/$id/brief.md" + : > "$QUOTA_LOG" + : > "$HERDR_LOG" + + if out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct recovery launched without its recorded brief" + assert_contains "$out" "no brief at $SPAWN_HOME/data/$id/brief.md" \ + "direct recovery missing-brief refusal was not actionable" + [ ! -e "$SPAWN_HOME/state/.fake-endpoint" ] || fail "missing direct recovery brief created a replacement endpoint" + [ ! -s "$QUOTA_LOG" ] || fail "missing direct recovery brief read account quota before refusing recovery" + [ ! -s "$HERDR_LOG" ] || fail "missing direct recovery brief installed a profile hook before refusing recovery" + pass "direct recovery refuses to launch without the recorded brief" +} + +test_direct_recovery_rejects_changed_worktree_identity() { + local record id meta original_worktree wrong_worktree redirected_worktree meta_tmp out status + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 90,85 + id=direct-worktree-identity-z7 + record=$(make_spawn_case direct-worktree-identity codex "$id") + read_spawn_case "$record" + + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool >/dev/null 2>&1 + meta="$SPAWN_HOME/state/$id.meta" + original_worktree=$(sed -n 's/^worktree=//p' "$meta") + wrong_worktree="$TMP_ROOT/wrong-linked-worktree" + git -C "$SPAWN_PROJECT" worktree add --quiet -b wrong-linked-worktree "$wrong_worktree" + wrong_worktree=$(cd "$wrong_worktree" && pwd -P) + meta_tmp=$(mktemp "$SPAWN_HOME/state/.direct-worktree-meta.XXXXXX") + awk -v worktree="$wrong_worktree" ' + /^worktree=/ { print "worktree=" worktree; next } + { print } + ' "$meta" > "$meta_tmp" + mv "$meta_tmp" "$meta" + rm -f "$SPAWN_HOME/state/.fake-endpoint" + : > "$QUOTA_LOG" + : > "$HERDR_LOG" + + if out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct recovery launched in another linked worktree" + assert_contains "$out" "no longer has its exact Git-dir identity" \ + "wrong linked-worktree refusal did not identify the exact Git-dir mismatch" + [ ! -s "$QUOTA_LOG" ] || fail "wrong linked worktree read account quota before refusing recovery" + [ ! -s "$HERDR_LOG" ] || fail "wrong linked worktree installed a profile hook before refusing recovery" + + redirected_worktree="$(cd "$TMP_ROOT" && pwd -P)/redirected-direct-worktree" + ln -s "$original_worktree" "$redirected_worktree" + meta_tmp=$(mktemp "$SPAWN_HOME/state/.direct-worktree-meta.XXXXXX") + awk -v worktree="$redirected_worktree" ' + /^worktree=/ { print "worktree=" worktree; next } + { print } + ' "$meta" > "$meta_tmp" + mv "$meta_tmp" "$meta" + : > "$QUOTA_LOG" + : > "$HERDR_LOG" + + if out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct recovery followed a symlinked worktree path" + assert_contains "$out" "is redirected or non-canonical" \ + "symlinked worktree refusal did not identify path redirection" + [ ! -s "$QUOTA_LOG" ] || fail "symlinked worktree read account quota before refusing recovery" + [ ! -s "$HERDR_LOG" ] || fail "symlinked worktree installed a profile hook before refusing recovery" + + meta_tmp=$(mktemp "$SPAWN_HOME/state/.direct-worktree-meta.XXXXXX") + awk -v worktree="$original_worktree" ' + /^worktree=/ { print "worktree=" worktree; next } + { print } + ' "$meta" > "$meta_tmp" + mv "$meta_tmp" "$meta" + git -C "$original_worktree" switch --quiet -c diverted-direct-recovery + : > "$QUOTA_LOG" + : > "$HERDR_LOG" + + if out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct recovery launched after the recorded worktree changed branches" + assert_contains "$out" "changed branch identity" \ + "changed-branch refusal did not identify the recorded branch mismatch" + [ ! -s "$QUOTA_LOG" ] || fail "changed worktree branch read account quota before refusing recovery" + [ ! -s "$HERDR_LOG" ] || fail "changed worktree branch installed a profile hook before refusing recovery" + pass "direct recovery rejects wrong, redirected, and branch-changed worktrees" +} + +test_direct_recovery_rechecks_identity_after_account_prepare() { + local record id out status + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 90,85 + id=direct-recheck-z7 + record=$(make_spawn_case direct-recheck codex "$id") + read_spawn_case "$record" + + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool >/dev/null 2>&1 + rm -f "$SPAWN_HOME/state/.fake-endpoint" + : > "$QUOTA_LOG" + : > "$HERDR_LOG" + + if out=$(FM_FAKE_HERDR_DRIFT_WORKTREE="$SPAWN_WORKTREE" \ + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct recovery ignored worktree identity drift during account preparation" + assert_contains "$out" "changed branch identity" \ + "post-prepare identity drift refusal was not actionable" + [ ! -e "$SPAWN_HOME/state/.fake-endpoint" ] || fail "post-prepare identity drift created a replacement endpoint" + [ -s "$QUOTA_LOG" ] || fail "post-prepare identity test did not reach fresh quota selection" + [ -s "$HERDR_LOG" ] || fail "post-prepare identity test did not reach Herdr installation" + pass "direct recovery rechecks exact identity immediately before endpoint creation" +} + +test_direct_recovery_tracks_retained_replacement_endpoint() { + local record id meta out status backup_name artifacts_name retry launch + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 90,85 + set_remaining 2 20,15 + id=direct-retained-endpoint-z7 + record=$(make_spawn_case direct-retained-endpoint codex "$id") + read_spawn_case "$record" + + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool >/dev/null 2>&1 + meta="$SPAWN_HOME/state/$id.meta" + set_remaining 1 20,15 + set_remaining 2 95,90 + rm -f "$SPAWN_HOME/state/.fake-endpoint" + rm -rf "/tmp/fm-$id" + : > "/tmp/fm-$id" + + if out=$(FM_FAKE_KILL_RETAIN=1 \ + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct recovery failure fixture unexpectedly succeeded" + [ -f "$SPAWN_HOME/state/.fake-endpoint" ] || fail "retained-endpoint fixture did not keep the replacement endpoint alive" + assert_grep "direct_recovery_cleanup=pending" "$meta" "failed recovery did not record pending direct cleanup" + assert_grep "tmux_window_id=@1" "$meta" "failed recovery did not record the replacement endpoint identity" + assert_grep "tmux_session_target=firstmate:fm-$id" "$meta" "failed recovery did not record the replacement endpoint scope" + assert_grep "account_home=$ACCOUNT_ROOT/codex/2" "$meta" "failed recovery metadata still presented the old account home as current" + backup_name=$(sed -n 's/^direct_recovery_backup=//p' "$meta") + artifacts_name=$(sed -n 's/^direct_recovery_artifacts=//p' "$meta") + [ -f "$SPAWN_HOME/state/$backup_name" ] || fail "failed recovery did not retain its prior metadata backup" + [ -d "$SPAWN_HOME/state/$artifacts_name" ] || fail "failed recovery did not retain its artifact backup" + + rm -f "/tmp/fm-$id" "$SPAWN_HOME/state/.fake-endpoint" + retry=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1) + launch=$(cat "$SPAWN_LAUNCH_LOG") + assert_contains "$retry" "cleaned retained direct recovery endpoint for $id" \ + "direct recovery did not reconcile the retained replacement endpoint" + assert_contains "$launch" "CODEX_HOME='$ACCOUNT_ROOT/codex/2' codex" \ + "direct recovery did not launch after reconciling the retained endpoint" + if grep -q '^direct_recovery_' "$meta"; then fail "successful retry left direct recovery markers in metadata"; fi + [ ! -e "$SPAWN_HOME/state/$backup_name" ] || fail "successful retry left the retained metadata backup" + [ ! -e "$SPAWN_HOME/state/$artifacts_name" ] || fail "successful retry left the retained artifact backup" + rm -rf "/tmp/fm-$id" + pass "failed direct recovery tracks and reconciles a retained replacement endpoint" +} + +test_new_direct_spawn_tracks_retained_endpoint_and_worktree() { + local record id meta out status recovery_out recovery_status recorded_worktree + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 90,85 + id=direct-new-retained-z8 + record=$(make_spawn_case direct-new-retained codex "$id") + read_spawn_case "$record" + : > "/tmp/fm-$id" + + if out=$(FM_FAKE_KILL_RETAIN=1 \ + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "new retained-endpoint failure fixture unexpectedly succeeded" + meta="$SPAWN_HOME/state/$id.meta" + recorded_worktree=$(cd "$SPAWN_WORKTREE" && pwd -P) + [ -f "$SPAWN_HOME/state/.fake-endpoint" ] || fail "new direct spawn fixture did not retain its endpoint" + assert_grep "direct_spawn_cleanup=pending" "$meta" \ + "new direct spawn did not record pending endpoint cleanup" + assert_grep "rollback_pending=1" "$meta" \ + "new direct spawn did not fail closed against duplicate spawn" + assert_grep "worktree=$recorded_worktree" "$meta" \ + "new direct spawn did not retain its worktree path" + assert_grep "worktree_git_dir=" "$meta" \ + "new direct spawn did not retain its exact worktree Git-dir" + assert_grep "worktree_git_dir_identity=" "$meta" \ + "new direct spawn did not retain its worktree Git-dir identity" + assert_grep "worktree_git_ref=refs/heads/" "$meta" \ + "new direct spawn did not retain its authoritative branch" + assert_grep "tmux_window_id=@1" "$meta" \ + "new direct spawn did not retain its endpoint identity" + assert_grep "account_home=$ACCOUNT_ROOT/codex/1" "$meta" \ + "new direct spawn did not retain its selected account home" + + if recovery_out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" --recover-direct-account 2>&1); then + recovery_status=0 + else + recovery_status=$? + fi + [ "$recovery_status" -ne 0 ] || fail "direct recovery bypassed pending new-spawn cleanup" + assert_contains "$recovery_out" "failed direct spawn cleanup is pending" \ + "pending new-spawn cleanup refusal was not actionable" + rm -f "/tmp/fm-$id" "$SPAWN_HOME/state/.fake-endpoint" + pass "new direct spawn tracks retained endpoint and worktree state" +} + +test_failed_new_direct_spawn_returns_worktree_after_endpoint_cleanup() { + local record id out status recorded_worktree + reset_accounts + set_remaining 1 90,85 + id=direct-new-rollback-z9 + record=$(make_spawn_case direct-new-rollback codex "$id") + read_spawn_case "$record" + recorded_worktree=$(cd "$SPAWN_WORKTREE" && pwd -P) + : > "$TREEHOUSE_LOG" + : > "/tmp/fm-$id" + + if out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "new direct rollback fixture unexpectedly succeeded" + if ! grep -Fq "return --force ." "$TREEHOUSE_LOG"; then + fail "failed new direct spawn did not return its worktree: output=$out treehouse=$(cat "$TREEHOUSE_LOG")" + fi + [ ! -e "$SPAWN_WORKTREE" ] || fail "failed new direct spawn left its worktree registered" + [ ! -e "$SPAWN_HOME/state/$id.meta" ] || fail "successful direct rollback left task metadata" + [ ! -e "$SPAWN_HOME/state/.fake-endpoint" ] || fail "successful direct rollback left its endpoint" + rm -f "/tmp/fm-$id" + pass "failed new direct spawn removes its endpoint and returns its worktree" +} + +test_failed_new_direct_spawn_retains_cleanup_when_worktree_return_fails() { + local record id out status meta + reset_accounts + set_remaining 1 90,85 + id=direct-new-return-fail-z9 + record=$(make_spawn_case direct-new-return-fail codex "$id") + read_spawn_case "$record" + : > "$TREEHOUSE_LOG" + : > "/tmp/fm-$id" + + if out=$(FM_FAKE_TREEHOUSE_RETURN_FAIL=1 \ + run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" --account-pool legacy-codex-pool 2>&1); then + status=0 + else + status=$? + fi + [ "$status" -ne 0 ] || fail "direct return-failure fixture unexpectedly succeeded" + meta="$SPAWN_HOME/state/$id.meta" + [ -d "$SPAWN_WORKTREE" ] || fail "direct return failure lost the retained worktree" + assert_grep "direct_spawn_cleanup=pending" "$meta" \ + "direct return failure did not record pending cleanup" + assert_grep "rollback_pending=1" "$meta" \ + "direct return failure did not fail closed" + [ ! -e "$SPAWN_HOME/state/.fake-endpoint" ] || fail "direct return failure retained an already-removed endpoint" + rm -f "/tmp/fm-$id" + pass "direct spawn persists cleanup state when worktree return cannot be confirmed" +} + +test_routing_off_keeps_default_provider_launch() { + local record id out launch meta + reset_accounts + : > "$TMP_ROOT/agent-fleet.log" + set_remaining 1 100,100 + id=direct-off-z3 + record=$(make_spawn_case direct-off codex "$id") + read_spawn_case "$record" + printf '%s\n' off > "$SPAWN_HOME/config/account-routing-mode" + + out=$(run_direct_spawn "$SPAWN_HOME" "$SPAWN_WORKTREE" "$SPAWN_LAUNCH_LOG" \ + "$id" "$SPAWN_PROJECT" 2>&1) + launch=$(cat "$SPAWN_LAUNCH_LOG") + meta=$SPAWN_HOME/state/$id.meta + assert_contains "$out" "spawned $id harness=codex" "routing-off spawn failed" + assert_not_contains "$launch" "CODEX_HOME=" "routing-off launch unexpectedly selected an account directory" + if grep -q '^account_home=' "$meta"; then fail "routing-off metadata unexpectedly recorded an account home"; fi + [ ! -s "$QUOTA_LOG" ] || fail "routing-off spawn read per-account quota" + [ ! -s "$HERDR_LOG" ] || fail "routing-off spawn ran the Herdr profile installer" + [ ! -s "$TMP_ROOT/agent-fleet.log" ] || fail "routing-off spawn invoked Agent Fleet" + pass "routing off preserves the provider's default identity and performs no account selection" +} + +make_spawn_fakebin "$FAKEBIN" + +if [ "${FM_TEST_FOCUSED:-}" = direct-recovery-lifecycle ]; then + test_direct_spawn_and_recovery_support_detached_worktree + test_direct_recovery_preserves_recorded_task_context + test_direct_recovery_rejects_secondmate_metadata + test_failed_new_direct_spawn_returns_worktree_after_endpoint_cleanup + test_failed_new_direct_spawn_retains_cleanup_when_worktree_return_fails + exit 0 +fi + +test_codex_picks_highest_fresh_minimum_and_skips_no_window +test_codex_rechecks_health_on_every_selection +test_codex_fails_when_no_account_has_a_fresh_window +test_codex_timeout_skips_wedged_account +test_claude_uses_stable_first_without_treating_usage_as_health +test_default_root_uses_passwd_home_not_ambient_home +test_prepare_installs_and_verifies_per_account_herdr_hooks +test_spawn_uses_direct_codex_home_without_agent_fleet +test_spawn_uses_direct_claude_fallback_and_hook +test_observe_spawn_uses_direct_directory_without_agent_fleet +test_direct_spawn_and_recovery_support_detached_worktree +test_direct_recovery_preserves_recorded_task_context +test_direct_recovery_rejects_secondmate_metadata +test_direct_recovery_rejects_worktree_from_another_project +test_direct_recovery_requires_recorded_brief +test_direct_recovery_rejects_changed_worktree_identity +test_direct_recovery_rechecks_identity_after_account_prepare +test_direct_recovery_tracks_retained_replacement_endpoint +test_new_direct_spawn_tracks_retained_endpoint_and_worktree +test_failed_new_direct_spawn_returns_worktree_after_endpoint_cleanup +test_failed_new_direct_spawn_retains_cleanup_when_worktree_return_fails +test_routing_off_keeps_default_provider_launch + +echo "# all fm-account-directory tests passed" diff --git a/tests/fm-account-routing.test.sh b/tests/fm-account-routing.test.sh index 2111e86b15e..92650e4c18c 100755 --- a/tests/fm-account-routing.test.sh +++ b/tests/fm-account-routing.test.sh @@ -4,6 +4,9 @@ # credential, real endpoint, global config, or live worker is touched. set -u export FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 +export FM_ACCOUNT_ROUTING_LEGACY_NEW_LAUNCH_TEST=firstmate-remove-fleet-routing-deadcode-fixture-v1 +export FM_ORCA_TEST_LAB=firstmate-orca-test-lab-v1 +export FM_ORCA_TEST_AUTHORITY_CAPABILITIES=verified-v1 # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" @@ -121,6 +124,20 @@ case "${1:-}" in exit 0 ;; send-keys) + if [ -n "${FM_FAKE_TMUX_GATE_SEND_MATCH:-}" ]; then + case "$*" in + *"$FM_FAKE_TMUX_GATE_SEND_MATCH"*) + [ -z "${FM_FAKE_TMUX_GATE_SEND_MARKER:-}" ] || touch "$FM_FAKE_TMUX_GATE_SEND_MARKER" + while [ -n "${FM_FAKE_TMUX_GATE_SEND_RELEASE:-}" ] \ + && [ ! -f "$FM_FAKE_TMUX_GATE_SEND_RELEASE" ]; do + sleep 0.05 + done + ;; + esac + fi + if [ -n "${FM_FAKE_TMUX_FAIL_SEND_MATCH:-}" ]; then + case "$*" in *"$FM_FAKE_TMUX_FAIL_SEND_MATCH"*) exit 72 ;; esac + fi if [ -n "${FM_FAKE_LAUNCH_LOG:-}" ]; then prev= for arg in "$@"; do @@ -176,6 +193,23 @@ SH [ -z "${FM_FAKE_TREEHOUSE_LOG:-}" ] || printf '%s\n' "$*" >> "$FM_FAKE_TREEHOUSE_LOG" [ -z "${FM_FAKE_LIFECYCLE_LOG:-}" ] || printf 'treehouse %s\n' "$*" >> "$FM_FAKE_LIFECYCLE_LOG" [ -z "${FM_FAKE_TREEHOUSE_SLEEP:-}" ] || sleep "$FM_FAKE_TREEHOUSE_SLEEP" +if [ "${1:-}" = return ] && [ -n "${FM_EXPECT_CHECKOUT_LOCK:-}" ]; then + [ -e "$FM_EXPECT_CHECKOUT_LOCK" ] || [ -L "$FM_EXPECT_CHECKOUT_LOCK" ] || exit 91 + lock_pid=$(cat "$FM_EXPECT_CHECKOUT_LOCK/pid" 2>/dev/null || true) + kill -0 "$lock_pid" 2>/dev/null || exit 92 + [ -z "${FM_EXPECT_CHECKOUT_LOCK_MARKER:-}" ] || touch "$FM_EXPECT_CHECKOUT_LOCK_MARKER" +fi +if [ "${1:-}" = return ] && [ -n "${FM_FAKE_TREEHOUSE_RETURN_CHILD_PID_FILE:-}" ]; then + [ -z "${FM_FAKE_TREEHOUSE_RETURN_MARKER:-}" ] || : > "$FM_FAKE_TREEHOUSE_RETURN_MARKER" + ( + trap '' HUP TERM + while :; do sleep 0.1; done + ) & + printf '%s\n' "$!" > "$FM_FAKE_TREEHOUSE_RETURN_CHILD_PID_FILE" +fi +if [ "${1:-}" = get ]; then + printf '%s\n' "${FM_FAKE_TREEHOUSE_PATH:?}" +fi exit 0 SH chmod +x "$fakebin/treehouse" @@ -327,20 +361,27 @@ SH } make_case() { - local name=$1 harness=$2 case_dir home proj wt fakebin + local name=$1 harness=$2 case_dir home proj wt fakebin first_id shift 2 + first_id=${1:-} case_dir="$TMP_ROOT/$name" home="$case_dir/home" proj="$case_dir/project" - wt="$case_dir/wt" + wt="$case_dir/treehouse-pools/pool/slot/wt" fakebin=$(make_fakebin "$case_dir/fake") - mkdir -p "$home/data" "$home/projects" "$home/state" "$home/config" + mkdir -p "$home/data" "$home/projects" "$home/state" "$home/config" "$(dirname "$wt")" printf '%s\n' "$harness" > "$home/config/crew-harness" fm_git_worktree "$proj" "$wt" "wt-$name" + git -C "$wt" checkout --quiet --detach + fm_git_add_origin "$proj" "$case_dir/origin.git" + [ -z "$first_id" ] || printf '{"worktrees":[{"path":"%s","leased":true,"lease_holder":"firstmate-%s","destroying":false}]}\n' \ + "$wt" "$first_id" > "$case_dir/treehouse-pools/pool/treehouse-state.json" touch "$home/state/.last-watcher-beat" for id in "$@"; do mkdir -p "$home/data/$id" printf 'brief for %s\n' "$id" > "$home/data/$id/brief.md" + printf '# Completion\n\n## Summary\n\nCompleted %s.\n\n## What changed\n\nExercised the behavior fixture.\n\n## Verification\n\nVerified by the account-routing suite.\n\n## Visual evidence\n\nNot applicable for this shell behavior fixture.\n\n## Artifacts\n\nThe test logs are the only artifacts.\n\n## Follow-ups\n\nNo follow-up work remains.\n' \ + "$id" > "$home/data/$id/completion.md" done printf '%s|%s|%s|%s|%s\n' "$case_dir" "$home" "$proj" "$wt" "$fakebin" } @@ -367,13 +408,27 @@ EOF run_spawn() { local id=$1 - FM_ROOT_OVERRIDE='' FM_HOME="$HOME_DIR" \ + FM_ROOT_OVERRIDE="${FM_TEST_ROOT_OVERRIDE:-}" FM_HOME="$HOME_DIR" \ FM_STATE_OVERRIDE="$HOME_DIR/state" FM_DATA_OVERRIDE="$HOME_DIR/data" \ FM_PROJECTS_OVERRIDE="$HOME_DIR/projects" FM_CONFIG_OVERRIDE="$HOME_DIR/config" \ FM_SPAWN_NO_GUARD=1 FM_FAKE_PANE_PATH="${FM_TEST_PANE_PATH:-$WT_DIR}" FM_FAKE_LAUNCH_LOG="$LAUNCH_LOG" \ FM_FAKE_TMUX_LOG="$TMUX_LOG" FM_FAKE_AF_LOG="$AF_LOG" \ FM_FAKE_TREEHOUSE_LOG="$TREEHOUSE_LOG" FM_FAKE_LIFECYCLE_LOG="$LIFECYCLE_LOG" \ + FM_EXPECT_CHECKOUT_LOCK="${FM_EXPECT_CHECKOUT_LOCK:-}" \ + FM_EXPECT_CHECKOUT_LOCK_MARKER="${FM_EXPECT_CHECKOUT_LOCK_MARKER:-}" \ + FM_FAKE_TREEHOUSE_PATH="$WT_DIR" FM_TREEHOUSE_ROOT="$CASE_DIR/treehouse-pools" \ + FM_FAKE_TREEHOUSE_SLEEP="${FM_FAKE_TREEHOUSE_SLEEP:-}" \ + FM_FAKE_TREEHOUSE_RETURN_CHILD_PID_FILE="${FM_FAKE_TREEHOUSE_RETURN_CHILD_PID_FILE:-}" \ + FM_FAKE_TREEHOUSE_RETURN_MARKER="${FM_FAKE_TREEHOUSE_RETURN_MARKER:-}" \ + FM_TEST_REAL_PS="${FM_TEST_REAL_PS:-}" \ + FM_TREEHOUSE_ACQUIRE_TIMEOUT="${FM_TREEHOUSE_ACQUIRE_TIMEOUT:-60}" \ FM_FAKE_ORCA_LOG="$ORCA_LOG" \ + FM_FAKE_TMUX_FAIL_SEND_MATCH="${FM_FAKE_TMUX_FAIL_SEND_MATCH:-}" \ + FM_FAKE_TMUX_GATE_SEND_MATCH="${FM_FAKE_TMUX_GATE_SEND_MATCH:-}" \ + FM_FAKE_TMUX_GATE_SEND_MARKER="${FM_FAKE_TMUX_GATE_SEND_MARKER:-}" \ + FM_FAKE_TMUX_GATE_SEND_RELEASE="${FM_FAKE_TMUX_GATE_SEND_RELEASE:-}" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$CASE_DIR/checkout-refresh-state" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$CASE_DIR/checkout-refresh-locks" \ FM_FAKE_ENDPOINT_FILE="$CASE_DIR/endpoint-live" FM_FAKE_TMUX_LABEL_FILE="$CASE_DIR/tmux-label" \ FM_FAKE_AF_RESUME_ARM="$CASE_DIR/resume-arm" FM_FAKE_AF_SESSION_REFRESHED="$CASE_DIR/session-refreshed" \ FM_FAKE_AF_RESUME_READY="$HOME_DIR/state/.$id.account-native-ready" FM_FAKE_AF_RESUME_GO="$HOME_DIR/state/.$id.account-native-go" \ @@ -389,6 +444,9 @@ run_teardown() { FM_FAKE_TMUX_LOG="$TMUX_LOG" FM_FAKE_AF_LOG="$AF_LOG" \ FM_FAKE_TREEHOUSE_LOG="$TREEHOUSE_LOG" FM_FAKE_ENDPOINT_FILE="$CASE_DIR/endpoint-live" \ FM_FAKE_TMUX_LABEL_FILE="$CASE_DIR/tmux-label" \ + FM_TREEHOUSE_ROOT="$CASE_DIR/treehouse-pools" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$CASE_DIR/checkout-refresh-state" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$CASE_DIR/checkout-refresh-locks" \ FM_AGENT_FLEET_BIN="$FAKEBIN_DIR/agent-fleet" \ TMUX="fake,1,0" PATH="$FAKEBIN_DIR:$PATH" "$TEARDOWN" "$@" } @@ -450,6 +508,297 @@ test_off_is_byte_compatible_and_never_calls_agent_fleet() { pass "routing off makes no Agent Fleet call and preserves launch/meta bytes" } +test_failed_freshness_proof_rolls_back_unmanaged_resources() { + local id rec out status default_branch + id='checkout-freshness-rollback-z1a' + rec=$(make_case checkout-freshness-rollback claude "$id") + read_case "$rec" + default_branch=$(git -C "$PROJ_DIR" branch --show-current) + printf '%s\n' upstream > "$PROJ_DIR/upstream.txt" + git -C "$PROJ_DIR" add upstream.txt + git -C "$PROJ_DIR" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm upstream + git -C "$PROJ_DIR" push -q -u origin "$default_branch" + + if out=$(run_spawn "$id" "$PROJ_DIR"); then + status=0 + else + status=$? + fi + + [ "$status" -ne 0 ] || fail "stale acquired worktree passed the spawn freshness proof" + assert_contains "$out" "acquired worktree is stale" \ + "failed freshness proof did not identify the stale acquired worktree" + assert_not_grep '^new-window ' "$TMUX_LOG" \ + "failed unmanaged freshness proof created an endpoint before verification" + assert_not_grep 'return --force' "$TREEHOUSE_LOG" \ + "failed unmanaged freshness proof force-returned an unverified acquired worktree" + assert_contains "$out" "retained unsafe acquired worktree" \ + "failed unmanaged freshness proof did not surface retain-only cleanup" + assert_absent "$CASE_DIR/endpoint-live" \ + "failed unmanaged freshness proof left its endpoint alive" + assert_absent "$HOME_DIR/state/$id.meta" \ + "failed unmanaged freshness proof published task metadata" + pass "failed freshness proofs unwind endpoints and retain unverified worktrees" +} + +test_local_only_spawn_uses_local_default_tip() { + local id rec out status + id='checkout-local-only-z1b' + rec=$(make_case checkout-local-only claude "$id") + read_case "$rec" + git -C "$PROJ_DIR" remote remove origin + printf '%s\n' '- project [local-only] - local project (added 2026-07-23)' \ + > "$HOME_DIR/data/projects.md" + + out=$(run_spawn "$id" "$PROJ_DIR") + status=$? + + [ "$status" -eq 0 ] || fail "remote-free local-only spawn failed its local freshness proof: $out" + assert_contains "$out" "spawned $id" "remote-free local-only spawn did not complete" + assert_not_grep 'return --force' "$TREEHOUSE_LOG" \ + "successful remote-free local-only spawn returned its acquired worktree" + pass "remote-free local-only spawns prove the local default tip" +} + +test_dirty_acquisition_is_retained_without_force_return() { + local id rec draft out status + id='checkout-dirty-retain-z1c' + rec=$(make_case checkout-dirty-retain claude "$id") + read_case "$rec" + draft="$WT_DIR/.agents/skills/unlanded/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# unlanded work' > "$draft" + + if out=$(run_spawn "$id" "$PROJ_DIR"); then + status=0 + else + status=$? + fi + + [ "$status" -ne 0 ] || fail "spawn accepted a dirty acquired pool worktree" + assert_contains "$out" "acquired worktree is dirty" \ + "dirty acquisition refusal did not identify the retained worktree" + assert_contains "$out" "retained unsafe acquired worktree" \ + "dirty acquisition cleanup did not surface its retain-only action" + assert_grep 'get --lease --lease-holder firstmate-checkout-dirty-retain-z1c' "$TREEHOUSE_LOG" \ + "dirty acquisition was not durably leased before verification" + assert_not_grep 'return --force' "$TREEHOUSE_LOG" \ + "dirty acquisition was returned through the destructive Treehouse path" + grep -Fq '# unlanded work' "$draft" || fail "dirty acquisition cleanup changed its draft" + assert_absent "$CASE_DIR/endpoint-live" \ + "dirty acquisition refusal left its prepared endpoint alive" + assert_absent "$HOME_DIR/state/$id.meta" \ + "dirty acquisition refusal published task metadata" + pass "dirty pool acquisitions are diagnosed and retained untouched" +} + +test_treehouse_acquisition_timeout_is_bounded_before_endpoint_creation() { + local id rec out status + id='checkout-acquire-timeout-z1f' + rec=$(make_case checkout-acquire-timeout claude "$id") + read_case "$rec" + + if out=$(FM_FAKE_TREEHOUSE_SLEEP=3 FM_TREEHOUSE_ACQUIRE_TIMEOUT=1 run_spawn "$id" "$PROJ_DIR"); then + status=0 + else + status=$? + fi + + [ "$status" -ne 0 ] || fail "hung Treehouse acquisition unexpectedly completed" + assert_contains "$out" "Treehouse worktree acquisition timed out after 1s" \ + "bounded Treehouse acquisition did not surface its distinct timeout" + assert_not_grep '^new-window ' "$TMUX_LOG" \ + "timed-out Treehouse acquisition created an endpoint" + assert_not_grep 'return --force' "$TREEHOUSE_LOG" \ + "timed-out Treehouse acquisition returned a path it never acquired" + pass "Treehouse acquisition is process-tree bounded before endpoint creation" +} + +test_changed_acquisition_is_retained_during_unmanaged_rollback() { + local id rec marker release out_file spawn_pid + id='checkout-changed-rollback-z1g' + rec=$(make_case checkout-changed-rollback pi "$id") + read_case "$rec" + marker="$CASE_DIR/gotmp-send-started" + release="$CASE_DIR/gotmp-send-release" + out_file="$CASE_DIR/spawn.out" + + FM_FAKE_TMUX_GATE_SEND_MATCH=GOTMPDIR \ + FM_FAKE_TMUX_GATE_SEND_MARKER="$marker" \ + FM_FAKE_TMUX_GATE_SEND_RELEASE="$release" \ + FM_FAKE_TMUX_FAIL_SEND_MATCH=GOTMPDIR \ + run_spawn "$id" "$PROJ_DIR" > "$out_file" 2>&1 & + spawn_pid=$! + for _ in $(seq 1 100); do [ -f "$marker" ] && break; sleep 0.05; done + [ -f "$marker" ] \ + || { kill "$spawn_pid" 2>/dev/null || true; fail "changed-acquisition test never reached the post-install failure"; } + printf '%s\n' retained-commit > "$WT_DIR/retained-commit.txt" + git -C "$WT_DIR" add retained-commit.txt + git -C "$WT_DIR" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm retained-commit + touch "$release" + if wait "$spawn_pid"; then + fail "changed-acquisition rollback unexpectedly succeeded" + fi + + assert_not_grep 'return --force' "$TREEHOUSE_LOG" \ + "rollback force-returned an acquired worktree whose detached tip changed" + assert_grep 'retained-commit' "$WT_DIR/retained-commit.txt" \ + "rollback changed work committed after acquisition" + assert_grep 'retained acquired worktree' "$out_file" \ + "rollback did not diagnose the changed acquired tip" + pass "unmanaged rollback retains acquisitions whose detached tip changed" +} + +test_unmanaged_postinstall_failure_restores_prior_state() { + local id rec expected out status artifact expected_lock lock_marker + id='checkout-unmanaged-restore-z1d' + rec=$(make_case checkout-unmanaged-restore pi "$id") + read_case "$rec" + fm_write_meta "$HOME_DIR/state/$id.meta" \ + "window=firstmate:fm-$id" \ + "worktree=$WT_DIR" \ + "project=$PROJ_DIR" \ + "harness=pi" \ + "kind=ship" \ + "mode=no-mistakes" \ + "custom_extension=retain-me" + expected="$CASE_DIR/original.meta" + expected_lock=$(bash -c \ + '. "$1"; fm_checkout_lock_path "$2" "$3"' \ + _ "$ROOT/bin/fm-checkout-lock-lib.sh" "$WT_DIR" "$CASE_DIR/checkout-refresh-locks") + lock_marker="$CASE_DIR/checkout-return-held-lock" + cp "$HOME_DIR/state/$id.meta" "$expected" + for artifact in status turn-ended check.sh pi-ext.ts grok-turnend-token; do + printf 'prior-%s\n' "$artifact" > "$HOME_DIR/state/$id.$artifact" + done + + if out=$(FM_EXPECT_CHECKOUT_LOCK="$expected_lock" \ + FM_EXPECT_CHECKOUT_LOCK_MARKER="$lock_marker" \ + FM_FAKE_TMUX_FAIL_SEND_MATCH=GOTMPDIR run_spawn "$id" "$PROJ_DIR"); then + status=0 + else + status=$? + fi + + [ "$status" -ne 0 ] || fail "post-metadata unmanaged launch failure unexpectedly succeeded" + cmp -s "$HOME_DIR/state/$id.meta" "$expected" \ + || fail "post-metadata unmanaged failure did not restore prior metadata" + for artifact in status turn-ended check.sh pi-ext.ts grok-turnend-token; do + [ "$(cat "$HOME_DIR/state/$id.$artifact" 2>/dev/null)" = "prior-$artifact" ] \ + || fail "post-metadata unmanaged failure did not restore prior $artifact state" + done + assert_absent "$CASE_DIR/endpoint-live" \ + "post-metadata unmanaged failure left its endpoint alive" + assert_grep 'return --force .' "$TREEHOUSE_LOG" \ + "post-metadata unmanaged failure did not return its clean worktree" + assert_present "$lock_marker" \ + "spawn rollback did not hold the common checkout lock during Treehouse return" + [ -n "$out" ] || true + pass "unmanaged post-install failures restore prior lifecycle state transactionally" +} + +test_spawn_rollback_relays_unverified_treehouse_cleanup() { + local id rec out status real_ps lock group owner child_pid + id='checkout-unverified-return-z1h' + rec=$(make_case checkout-unverified-return pi "$id") + read_case "$rec" + real_ps=$(command -v ps) + cat > "$FAKEBIN_DIR/ps" <<'SH' +#!/usr/bin/env bash +if [ -f "${FM_FAKE_TREEHOUSE_RETURN_MARKER:-/nonexistent}" ] \ + && [ "$*" = "-axo pid=,pgid=" ]; then + exit 1 +fi +exec "${FM_TEST_REAL_PS:?}" "$@" +SH + chmod +x "$FAKEBIN_DIR/ps" + + if out=$(FM_TEST_REAL_PS="$real_ps" \ + FM_FAKE_TREEHOUSE_RETURN_CHILD_PID_FILE="$CASE_DIR/treehouse-return-child.pid" \ + FM_FAKE_TREEHOUSE_RETURN_MARKER="$CASE_DIR/treehouse-return-started" \ + FM_FAKE_TMUX_FAIL_SEND_MATCH=GOTMPDIR run_spawn "$id" "$PROJ_DIR"); then + status=0 + else + status=$? + fi + + [ "$status" -ne 0 ] || fail "spawn with unverified rollback cleanup unexpectedly succeeded" + lock=$(bash -c \ + '. "$1"; fm_checkout_lock_path "$2" "$3"' \ + _ "$ROOT/bin/fm-checkout-lock-lib.sh" "$WT_DIR" "$CASE_DIR/checkout-refresh-locks") + assert_present "$CASE_DIR/treehouse-return-child.pid" \ + "spawn rollback did not exercise a surviving Treehouse descendant" + assert_present "$lock" "spawn rollback released an unverified checkout lock" + assert_present "$lock/process-group" "spawn rollback lost the guarded process-group identity" + assert_contains "$out" "bounded command process cleanup could not be verified for anchored group" \ + "spawn rollback suppressed the bounded supervisor diagnostic" + assert_contains "$out" "Treehouse return process cleanup could not be verified" \ + "spawn rollback suppressed the locked-return cleanup diagnostic" + assert_contains "$out" "retained rollback worktree $WT_DIR" \ + "spawn rollback did not provide actionable retain-only recovery guidance" + group=$(cat "$lock/process-group") + child_pid=$(cat "$CASE_DIR/treehouse-return-child.pid") + kill -0 "$child_pid" 2>/dev/null \ + || fail "unverified Treehouse descendant exited before retention could be inspected" + owner=$(readlink "$lock") + kill -KILL -- "-$group" 2>/dev/null || true + for _ in $(seq 1 50); do + kill -0 "$group" 2>/dev/null || break + sleep 0.02 + done + ! kill -0 "$group" 2>/dev/null || fail "test cleanup could not terminate retained anchored group $group" + rm -f "$lock" + rm -rf "$owner" + pass "spawn rollback relays unverified cleanup and retains guarded resources" +} + +test_unmanaged_rollback_waits_for_metadata_lock() { + local id rec marker release out_file spawn_pid held + id='checkout-unmanaged-rollback-lock-z1e' + rec=$(make_case checkout-unmanaged-rollback-lock pi "$id") + read_case "$rec" + fm_write_meta "$HOME_DIR/state/$id.meta" \ + "window=firstmate:fm-$id" \ + "worktree=$WT_DIR" \ + "project=$PROJ_DIR" \ + "harness=pi" \ + "kind=ship" \ + "mode=no-mistakes" \ + "custom_extension=retain-me" + marker="$CASE_DIR/gotmp-send-started" + release="$CASE_DIR/gotmp-send-release" + out_file="$CASE_DIR/spawn.out" + + FM_FAKE_TMUX_GATE_SEND_MATCH=GOTMPDIR \ + FM_FAKE_TMUX_GATE_SEND_MARKER="$marker" \ + FM_FAKE_TMUX_GATE_SEND_RELEASE="$release" \ + FM_FAKE_TMUX_FAIL_SEND_MATCH=GOTMPDIR \ + run_spawn "$id" "$PROJ_DIR" > "$out_file" & + spawn_pid=$! + for _ in $(seq 1 100); do [ -f "$marker" ] && break; sleep 0.05; done + [ -f "$marker" ] \ + || { kill "$spawn_pid" 2>/dev/null || true; fail "unmanaged rollback-lock test never reached the post-install failure"; } + # shellcheck source=bin/fm-account-routing-lib.sh + . "$ROOT/bin/fm-account-routing-lib.sh" + held=$(fm_account_meta_lock_acquire "$HOME_DIR/state" "$id") \ + || { kill "$spawn_pid" 2>/dev/null || true; fail "unmanaged rollback-lock test could not acquire the writer lock"; } + touch "$release" + sleep 0.1 + printf '%s\n' 'x_request=req-concurrent-rollback' >> "$HOME_DIR/state/$id.meta" + fm_account_meta_lock_release "$held" \ + || { kill "$spawn_pid" 2>/dev/null || true; fail "unmanaged rollback-lock test could not release the writer lock"; } + if wait "$spawn_pid"; then + fail "unmanaged rollback-lock failure unexpectedly succeeded" + fi + assert_grep 'custom_extension=retain-me' "$HOME_DIR/state/$id.meta" \ + "unmanaged rollback lost the prior metadata generation" + assert_grep 'x_request=req-concurrent-rollback' "$HOME_DIR/state/$id.meta" \ + "unmanaged rollback discarded an extension written under the metadata lock" + pass "unmanaged rollback serializes generation restoration with metadata writers" +} + test_completion_contract_upgrade_is_contained_nonfollowing_and_atomic() { local id rec brief before after outside out status heading_count real_node hook raced_size local ready proceed original_task swapped_task spawn_pid spawn_rc @@ -813,6 +1162,62 @@ test_resume_uses_sticky_recovery_and_preserves_mapping_on_failure() { pass "resume uses below-reserve sticky recovery and never deletes mapping on a failed attempt" } +test_recovered_secondmate_kind_blocks_unsupported_cmux_backend_early() { + local id rec meta replacement out status + id=acct-cmux-sm-z9b + rec=$(make_case recovered-cmux-secondmate claude "$id") + read_case "$rec" + mkdir -p "$CASE_DIR/treehouse-pools" + out=$(FM_FAKE_AF_POOL=claude-crew run_spawn "$id" "$PROJ_DIR" --account-pool claude-crew) + status=$? + [ "$status" -eq 0 ] || fail "recovered cmux secondmate precondition spawn failed: $out" + meta="$HOME_DIR/state/$id.meta" + replacement="$meta.replacement" + sed 's/^kind=.*/kind=secondmate/;/^backend=/d' "$meta" > "$replacement" + printf 'backend=cmux\n' >> "$replacement" + mv "$replacement" "$meta" + rm -f "$CASE_DIR/endpoint-live" + clear_case_logs + out=$(run_spawn "$id" --resume-account) + status=$? + [ "$status" -ne 0 ] || fail "metadata-recovered cmux secondmate launched" + assert_contains "$out" "backend=cmux does not support --secondmate spawns yet" \ + "metadata-recovered secondmate missed the cmux backend guard" + assert_not_grep 'lease recover\|lease choose\|lease acquire' "$AF_LOG" \ + "unsupported recovered secondmate reached account mutation" + assert_not_grep '^new-window ' "$TMUX_LOG" \ + "unsupported recovered secondmate created an endpoint" + pass "metadata kind recovery precedes unsupported secondmate backend checks" +} + +test_managed_recovery_rejects_duplicate_kind_metadata() { + local id rec meta replacement out status + id=acct-duplicate-kind-z9c + rec=$(make_case duplicate-kind-recovery claude "$id") + read_case "$rec" + mkdir -p "$CASE_DIR/treehouse-pools" + out=$(FM_FAKE_AF_POOL=claude-crew run_spawn "$id" "$PROJ_DIR" --account-pool claude-crew) + status=$? + [ "$status" -eq 0 ] || fail "duplicate-kind recovery precondition spawn failed: $out" + meta="$HOME_DIR/state/$id.meta" + replacement="$meta.replacement" + sed '/^kind=/d;/^backend=/d' "$meta" > "$replacement" + printf '%s\n' 'kind=secondmate' 'kind=ship' 'backend=cmux' >> "$replacement" + mv "$replacement" "$meta" + rm -f "$CASE_DIR/endpoint-live" + clear_case_logs + out=$(run_spawn "$id" --resume-account) + status=$? + [ "$status" -ne 0 ] || fail "duplicate recovery kind metadata launched" + assert_contains "$out" "duplicate kind records" \ + "duplicate recovery kind metadata was not surfaced" + assert_not_grep 'lease recover\|lease choose\|lease acquire' "$AF_LOG" \ + "duplicate recovery kind metadata reached account mutation" + assert_not_grep '^new-window ' "$TMUX_LOG" \ + "duplicate recovery kind metadata created an endpoint" + pass "managed recovery requires exactly one canonical kind record" +} + test_managed_recovery_accepts_inherited_lifecycle_lock() { local id rec out status held account_task id=account-inherited-recovery-z9d @@ -1610,13 +2015,23 @@ test_native_resume_uses_private_launch_directory_and_cleans_it() { } make_seeded_secondmate_home() { - local home=$1 id=$2 - mkdir -p "$home/bin" "$home/data" "$home/state" "$home/config" "$home/projects" - cp "$ROOT/bin/fm-account-routing-lib.sh" "$home/bin/fm-account-routing-lib.sh" - cp "$ROOT/bin/fm-spawn.sh" "$home/bin/fm-spawn.sh" - printf '# Firstmate\n' > "$home/AGENTS.md" + local home=$1 id=$2 primary="$CASE_DIR/primary-root" + mkdir -p "$primary" + cp -R "$ROOT/bin" "$primary/bin" + printf '# Firstmate\n' > "$primary/AGENTS.md" + printf 'data/\nstate/\nconfig/\nprojects/\n.fm-secondmate-home\n' > "$primary/.gitignore" + git -C "$primary" init -q + git -C "$primary" add . + git -C "$primary" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm initial + git clone --quiet "$primary" "$home" + git -C "$home" checkout --quiet --detach + mkdir -p "$home/data" "$home/state" "$home/config" "$home/projects" printf '%s\n' "$id" > "$home/.fm-secondmate-home" printf 'charter\n' > "$home/data/charter.md" + printf -- '- %s - test secondmate (home: %s; scope: test; projects: ; added 2026-07-24)\n' \ + "$id" "$home" > "$HOME_DIR/data/secondmates.md" + FM_TEST_ROOT_OVERRIDE=$primary } test_secondmate_pool_is_nonactivating_and_noninherited() { @@ -1648,11 +2063,14 @@ test_secondmate_pool_routes_when_mode_is_enforced_and_mode_inherits() { printf 'enforce\n' > "$HOME_DIR/config/account-routing-mode" printf 'claude-captains\n' > "$HOME_DIR/config/secondmate-account-pool" - out=$(FM_FAKE_AF_POOL=claude-captains FM_TEST_PANE_PATH="$sm" run_spawn "$id" "$sm" --secondmate) + out=$(FM_ACCOUNT_ROUTING_LEGACY_NEW_LAUNCH_TEST='' \ + FM_FAKE_AF_POOL=claude-captains FM_TEST_PANE_PATH="$sm" \ + run_spawn "$id" "$sm" --secondmate) status=$? [ "$status" -eq 0 ] || fail "enforced secondmate spawn should succeed (exit $status): $out" assert_regex "lease choose --pool claude-captains --task .*-$id-.* --provider claude" "$AF_LOG" "secondmate did not use its primary-owned account pool" assert_grep 'account_pool=claude-captains' "$HOME_DIR/state/$id.meta" "secondmate meta lost its account pool" + assert_not_grep '^account_home=' "$HOME_DIR/state/$id.meta" "secondmate entered direct account-directory routing" [ "$(cat "$sm/config/account-routing-mode" 2>/dev/null)" = enforce ] || fail "account routing mode did not inherit into the secondmate home" assert_absent "$sm/config/secondmate-account-pool" "primary-only secondmate pool leaked into the child home" pass "secondmate routing uses the primary pool while the mode, but not that pool, inherits" @@ -1702,7 +2120,11 @@ test_enforced_secondmate_requires_routing_inheritance_and_capable_home() { sm="$CASE_DIR/secondmate-home" make_seeded_secondmate_home "$sm" "$id" sm=$(cd "$sm" && pwd -P) - rm -f "$sm/bin/fm-account-routing-lib.sh" + git -C "$FM_TEST_ROOT_OVERRIDE" rm -q bin/fm-account-routing-lib.sh + git -C "$FM_TEST_ROOT_OVERRIDE" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm 'pre-account-routing fixture' + git -C "$sm" fetch -q origin + git -C "$sm" checkout --quiet --detach FETCH_HEAD printf 'enforce\n' > "$HOME_DIR/config/account-routing-mode" out=$(FM_TEST_PANE_PATH="$sm" run_spawn "$id" "$sm" --secondmate) status=$? @@ -1750,7 +2172,11 @@ test_secondmate_routing_inheritance_is_authoritative_for_every_mode() { sm="$CASE_DIR/secondmate-home" make_seeded_secondmate_home "$sm" "$id" sm=$(cd "$sm" && pwd -P) - rm -f "$sm/bin/fm-account-routing-lib.sh" + git -C "$FM_TEST_ROOT_OVERRIDE" rm -q bin/fm-account-routing-lib.sh + git -C "$FM_TEST_ROOT_OVERRIDE" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm 'pre-account-routing fixture' + git -C "$sm" fetch -q origin + git -C "$sm" checkout --quiet --detach FETCH_HEAD out=$(FM_TEST_PANE_PATH="$sm" run_spawn "$id" "$sm" --secondmate) status=$? [ "$status" -eq 0 ] || fail "off secondmate did not preserve warn-and-launch behavior: $out" @@ -1901,7 +2327,7 @@ test_reservation_occurs_after_worktree_preparation() { rec=$(make_case order claude "$id") read_case "$rec" run_spawn "$id" "$PROJ_DIR" --account-pool claude-crew >/dev/null || fail "reservation order spawn failed" - treehouse_line=$(grep -n '^tmux send-keys .* treehouse get Enter$' "$LIFECYCLE_LOG" | head -1 | cut -d: -f1) + treehouse_line=$(grep -n '^treehouse get --lease --lease-holder firstmate-account-order-z14$' "$LIFECYCLE_LOG" | head -1 | cut -d: -f1) lease_line=$(grep -n 'agent-fleet .* lease choose ' "$LIFECYCLE_LOG" | head -1 | cut -d: -f1) [ -n "$treehouse_line" ] && [ -n "$lease_line" ] && [ "$lease_line" -gt "$treehouse_line" ] \ || fail "Agent Fleet reservation did not follow worktree preparation: $(tr '\n' '|' < "$LIFECYCLE_LOG")" @@ -2642,7 +3068,7 @@ test_failed_continuation_cleanup_restores_predecessor_for_retry() { } test_concurrent_continuations_serialize_before_mutation() { - local id rec marker gate first_pid second_pid first_rc second_rc lease_count endpoint_count second_lock_waiter + local id rec marker gate first_pid second_pid first_rc second_rc lease_count endpoint_count id=account-continuation-race-z21d rec=$(make_case continuation-race claude "$id") read_case "$rec" @@ -2663,14 +3089,8 @@ test_concurrent_continuations_serialize_before_mutation() { FM_FAKE_AF_PROFILE=claude-3 FM_FAKE_AF_POOL=explicit \ run_spawn "$id" --continue-account --account-profile claude-3 > "$CASE_DIR/second.out" 2>&1 & second_pid=$! - second_lock_waiter= - for _ in $(seq 1 100); do - second_lock_waiter=$(find "$HOME_DIR/state" -maxdepth 1 -type f \ - -name ".account-lifecycle-$id.owner.*" -print -quit) - [ -n "$second_lock_waiter" ] && break - sleep 0.05 - done - if [ -z "$second_lock_waiter" ]; then + sleep 0.2 + if ! kill -0 "$second_pid" 2>/dev/null; then touch "$gate" kill "$first_pid" "$second_pid" 2>/dev/null || true fail "second continuation never waited behind the first lifecycle owner" @@ -2718,10 +3138,23 @@ test_continuation_fails_closed_without_original_brief() { id=account-continue-nobrief-z22 rec=$(make_case continue-nobrief claude "$id") read_case "$rec" - run_spawn "$id" "$PROJ_DIR" --account-pool claude-crew >/dev/null || fail "continuation precondition spawn failed" - rm -f "$CASE_DIR/endpoint-live" "$HOME_DIR/data/$id/brief.md" + fm_write_meta "$HOME_DIR/state/$id.meta" \ + "window=firstmate:fm-$id" \ + "worktree=$WT_DIR" \ + "project=$PROJ_DIR" \ + "harness=claude" \ + "kind=ship" \ + "account_pool=claude-crew" \ + "account_profile=claude-2" \ + "account_task=$id" \ + "account_attempt=legacy" \ + "provider_session_id=sess-$id" + rm -f "$HOME_DIR/data/$id/brief.md" clear_case_logs - out=$(run_spawn "$id" --continue-account --account-profile claude-3) + out=$(FM_ROOT_OVERRIDE="$ROOT" FM_HOME="$HOME_DIR" FM_STATE_OVERRIDE="$HOME_DIR/state" \ + FM_DATA_OVERRIDE="$HOME_DIR/data" FM_FAKE_ENDPOINT_FILE="$CASE_DIR/endpoint-live" \ + FM_FAKE_TMUX_LOG="$TMUX_LOG" PATH="$FAKEBIN_DIR:$PATH" \ + "$CONTINUATION" "$id" missing-brief 2>&1) status=$? [ "$status" -ne 0 ] || fail "continuation without original brief unexpectedly succeeded" assert_not_grep '^new-window ' "$TMUX_LOG" "unsafe continuation created an endpoint" @@ -2744,7 +3177,7 @@ test_session_sync_cannot_recreate_metadata_after_teardown() { FM_FAKE_AF_RELEASE_MARKER="$release_marker" FM_FAKE_TREEHOUSE_SLEEP=1 \ run_teardown "$id" --force > "$CASE_DIR/teardown-stdout" 2> "$CASE_DIR/teardown-stderr" & teardown_pid=$! - for _ in 1 2 3 4 5 6 7 8 9 10; do + for _ in $(seq 1 100); do [ -f "$release_marker" ] && break sleep 0.1 done @@ -4705,13 +5138,7 @@ test_agent_fleet_contract_is_validated_before_routing() { [ "$status" -ne 0 ] || fail "incompatible Agent Fleet contract unexpectedly enforced routing" assert_not_grep 'lease (choose|acquire)' "$AF_LOG" "incompatible Agent Fleet contract mutated a lease" assert_contains "$out" "unsupported Agent Fleet contract version 1" "contract mismatch was not actionable" - - clear_case_logs - if out=$(FM_ACCOUNT_ROUTING=observe FM_FAKE_AF_CONTRACT_VERSION=1 run_spawn "$id" "$PROJ_DIR"); then status=0; else status=$?; fi - [ "$status" -eq 0 ] || fail "observe mode should degrade on an incompatible Agent Fleet contract" - assert_not_grep ' choose ' "$AF_LOG" "incompatible observe contract still queried selection" - assert_contains "$out" "observe contract unavailable" "observe contract fallback was not surfaced" - pass "Agent Fleet contract v2 is required before observation or enforcement" + pass "Agent Fleet contract v2 is required before legacy fixture enforcement" } test_agent_fleet_entrypoint_is_physically_pinned_per_operation() { @@ -5015,81 +5442,27 @@ SH pass "production Fleet control, selection, recovery, and workers close authority environment before startup" } -test_production_enforce_refuses_hostile_uncertified_parent_shell() { - local id rec marker hook fake_ps trace out status startup_out - id=uncertified-hostile-shell - rec=$(make_case uncertified-hostile-shell claude "$id") - read_case "$rec" - marker="$CASE_DIR/hostile-parent-ps4-ran" - hook="$CASE_DIR/hostile-parent-bash-env" - fake_ps="$CASE_DIR/hostile-parent-ps" - trace="$CASE_DIR/hostile-parent-trace" - # shellcheck disable=SC2016 # The variable expands when Bash sources the hook. - printf '%s\n' ': > "$FM_HOSTILE_PARENT_MARKER"' > "$hook" - cat > "$fake_ps" <<'SH' -#!/bin/sh -printf '%s\n' 'Fri Jul 18 08:00:00 2026' -SH - chmod +x "$fake_ps" - - # shellcheck disable=SC2016 # Dollar expressions belong to the nested shell. - if startup_out=$(/usr/bin/env SHELLOPTS=xtrace \ - PS4=HOSTILE_PARENT_XTRACE: BASH_ENV="$hook" \ - FM_HOSTILE_PARENT_MARKER="$marker" \ - /bin/bash --noprofile --norc -c ' - exec 2>"$9" - exec /usr/bin/env -u FM_ACCOUNT_ROUTING_TEST_LAB \ - FM_ROOT_OVERRIDE= FM_HOME="$1" \ - FM_STATE_OVERRIDE="$1/state" FM_DATA_OVERRIDE="$1/data" \ - FM_PROJECTS_OVERRIDE="$1/projects" FM_CONFIG_OVERRIDE="$1/config" \ - FM_SPAWN_NO_GUARD=1 FM_FAKE_TMUX_LOG="$4" \ - FM_FAKE_ENDPOINT_FILE="$5" FM_FAKE_TMUX_LABEL_FILE="$6" \ - FM_ACCOUNT_TEST_HOOKS=firstmate-account-tests-v1 \ - FM_TEST_ACCOUNT_PS_BIN="${10}" \ - TMUX=fake,1,0 PATH="$3:$PATH" \ - "$7" "$8" "$2" --backend tmux --account-pool claude-crew - ' _ "$HOME_DIR" "$PROJ_DIR" "$FAKEBIN_DIR" "$TMUX_LOG" \ - "$CASE_DIR/endpoint-live" "$CASE_DIR/tmux-label" "$SPAWN" "$id" \ - "$trace" "$fake_ps" 2>&1); then - status=0 - else - status=$? - fi - out="$startup_out$(cat "$trace")" - - [ "$status" -ne 0 ] || fail "hostile uncertified tmux parent reached enforced routing" - [ -e "$marker" ] || fail "hostile parent-shell fixture never executed its inherited startup hook" - assert_contains "$out" "HOSTILE_PARENT_XTRACE:set -eu" \ - "hostile parent-shell fixture never imported SHELLOPTS=xtrace/PS4 into fm-spawn" - assert_contains "$out" "requires backend=herdr with a process-bound closed-shell certificate" \ - "uncertified hostile parent-shell refusal was not actionable" - [ ! -s "$AF_LOG" ] || fail "uncertified hostile parent shell reached Agent Fleet: $(cat "$AF_LOG")" - [ ! -s "$TMUX_LOG" ] || fail "uncertified hostile parent shell created a tmux endpoint" - assert_absent "$HOME_DIR/state/$id.meta" "uncertified hostile parent shell persisted managed metadata" - pass "production enforce refuses a genuinely hostile parent shell before Fleet, lease, or endpoint mutation" -} - test_agent_fleet_lifecycle_calls_are_bounded() { local id rec out status started elapsed id=account-control-timeout-z27 rec=$(make_case control-timeout claude "$id") read_case "$rec" started=$(date +%s) - if out=$(FM_FAKE_AF_SELECT_SLEEP=10 FM_ACCOUNT_CONTROL_TIMEOUT=1 run_spawn "$id" "$PROJ_DIR" --account-pool claude-crew); then status=0; else status=$?; fi + if out=$(FM_FAKE_AF_SELECT_SLEEP=20 FM_ACCOUNT_CONTROL_TIMEOUT=1 run_spawn "$id" "$PROJ_DIR" --account-pool claude-crew); then status=0; else status=$?; fi elapsed=$(( $(date +%s) - started )) [ "$status" -eq 0 ] || fail "timed-out lease choice was not reconciled through recovery: $out" - # Semantic: returns well under the 10s unbounded fake sleep; 8s absorbs full-sweep scheduling load (5s flaked in-sweep while passing standalone). - [ "$elapsed" -lt 8 ] || fail "lease choice timeout was not bounded (elapsed ${elapsed}s)" + # Semantic: returns well under the 20s unbounded fake sleep; 15s absorbs full-sweep scheduling load. + [ "$elapsed" -lt 15 ] || fail "lease choice timeout was not bounded (elapsed ${elapsed}s)" assert_grep 'lease recover ' "$AF_LOG" "timed-out lease choice did not reconcile ownership" rm -f "$CASE_DIR/endpoint-live" clear_case_logs started=$(date +%s) - if out=$(FM_FAKE_AF_RELEASE_SLEEP=10 FM_ACCOUNT_CONTROL_TIMEOUT=1 run_teardown "$id" --force 2>&1); then status=0; else status=$?; fi + if out=$(FM_FAKE_AF_RELEASE_SLEEP=20 FM_ACCOUNT_CONTROL_TIMEOUT=1 run_teardown "$id" --force 2>&1); then status=0; else status=$?; fi elapsed=$(( $(date +%s) - started )) [ "$status" -ne 0 ] || fail "ambiguous timed-out lease release unexpectedly completed teardown" - # Semantic: returns well under the 10s unbounded fake sleep; 8s absorbs full-sweep scheduling load (5s flaked in-sweep while passing standalone). - [ "$elapsed" -lt 8 ] || fail "lease release timeout was not bounded (elapsed ${elapsed}s)" + # Semantic: returns well under the 20s unbounded fake sleep; 15s absorbs full-sweep scheduling load. + [ "$elapsed" -lt 15 ] || fail "lease release timeout was not bounded (elapsed ${elapsed}s)" assert_present "$HOME_DIR/state/$id.meta" "ambiguous lease release discarded retry metadata" pass "Agent Fleet lease mutations are bounded and ambiguous outcomes retain ownership state" } @@ -5442,6 +5815,49 @@ if [ "${FM_TEST_FOCUSED:-}" = stale-reclaim-generation ]; then exit 0 fi +if [ "${FM_TEST_FOCUSED:-}" = checkout-freshness-cleanup ]; then + run_isolated_test test_failed_freshness_proof_rolls_back_unmanaged_resources + run_isolated_test test_local_only_spawn_uses_local_default_tip + run_isolated_test test_dirty_acquisition_is_retained_without_force_return + run_isolated_test test_unmanaged_postinstall_failure_restores_prior_state + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-40 ]; then + run_isolated_test test_failed_freshness_proof_rolls_back_unmanaged_resources + run_isolated_test test_local_only_spawn_uses_local_default_tip + run_isolated_test test_dirty_acquisition_is_retained_without_force_return + run_isolated_test test_unmanaged_postinstall_failure_restores_prior_state + run_isolated_test test_unmanaged_rollback_waits_for_metadata_lock + run_isolated_test test_reservation_occurs_after_worktree_preparation + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-durable-secondmate ]; then + run_isolated_test test_recovered_secondmate_kind_blocks_unsupported_cmux_backend_early + run_isolated_test test_managed_recovery_rejects_duplicate_kind_metadata + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-8 ]; then + run_isolated_test test_unmanaged_postinstall_failure_restores_prior_state + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-4 ]; then + run_isolated_test test_failed_freshness_proof_rolls_back_unmanaged_resources + run_isolated_test test_treehouse_acquisition_timeout_is_bounded_before_endpoint_creation + run_isolated_test test_changed_acquisition_is_retained_during_unmanaged_rollback + run_isolated_test test_unmanaged_postinstall_failure_restores_prior_state + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-12-ownership ]; then + run_isolated_test test_unmanaged_postinstall_failure_restores_prior_state + run_isolated_test test_spawn_rollback_relays_unverified_treehouse_cleanup + exit 0 +fi + if [ "${FM_TEST_FOCUSED:-}" = continuation-status-timeout ]; then run_isolated_test test_continuation_bounds_no_mistakes_status_snapshot exit 0 @@ -5452,11 +5868,67 @@ if [ "${FM_TEST_FOCUSED:-}" = packet-rollback-aba ]; then exit 0 fi +test_secondmate_home_lock_key_fails_closed_without_fixed_hasher() { + local home state out status + home="$TMP_ROOT/secondmate-lock-key-home" + state="$TMP_ROOT/secondmate-lock-key-state" + mkdir -p "$home" "$state" + set +e + out=$(FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ + FM_ACCOUNT_TEST_HOOKS=firstmate-account-tests-v1 \ + bash -c ' + . "$1/bin/fm-account-routing-lib.sh" + FM_CHECKOUT_SYSTEM_PERL_BIN= + fm_secondmate_home_lifecycle_lock_acquire "$2" "$3" + ' _ "$ROOT" "$state" "$home" 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "secondmate home lock accepted an unavailable fixed hasher" + assert_contains "$out" "lock identity is unavailable" \ + "secondmate home lock hash failure was not surfaced" + [ -z "$(find "$state" -mindepth 1 -print -quit)" ] \ + || fail "secondmate home lock hash failure created a fallback lock" + pass "secondmate home lifecycle locks fail closed without fixed hashing" +} + +test_secondmate_home_lock_key_uses_filesystem_identity() { + local home dotted case_alias missing_upper missing_lower key dotted_key case_key missing_upper_key missing_lower_key + home="$TMP_ROOT/secondmate-lock-identity-home" + mkdir -p "$home" + dotted="$home/." + key=$(bash -c '. "$1/bin/fm-account-routing-lib.sh"; fm_account_stable_path_key "$2" directory' _ "$ROOT" "$home") \ + || fail "secondmate home identity key was unavailable" + dotted_key=$(bash -c '. "$1/bin/fm-account-routing-lib.sh"; fm_account_stable_path_key "$2" directory' _ "$ROOT" "$dotted") \ + || fail "dotted secondmate home identity key was unavailable" + [ "$key" = "$dotted_key" ] || fail "one physical home produced different keys through a dotted alias" + case_alias=$(printf '%s\n' "$home" | tr '[:lower:]' '[:upper:]') + if [ "$case_alias" != "$home" ] && [ -d "$case_alias" ]; then + case_key=$(bash -c '. "$1/bin/fm-account-routing-lib.sh"; fm_account_stable_path_key "$2" directory' _ "$ROOT" "$case_alias") \ + || fail "case-aliased secondmate home identity key was unavailable" + [ "$key" = "$case_key" ] || fail "one physical home produced different keys through a case alias" + fi + missing_upper="$TMP_ROOT/Missing-Secondmate-Home" + missing_lower="$TMP_ROOT/missing-secondmate-home" + missing_upper_key=$(bash -c '. "$1/bin/fm-account-routing-lib.sh"; fm_account_stable_path_key "$2" directory' _ "$ROOT" "$missing_upper") \ + || fail "missing secondmate home identity key was unavailable" + missing_lower_key=$(bash -c '. "$1/bin/fm-account-routing-lib.sh"; fm_account_stable_path_key "$2" directory' _ "$ROOT" "$missing_lower") \ + || fail "case-folded missing secondmate home identity key was unavailable" + [ "$missing_upper_key" = "$missing_lower_key" ] \ + || fail "case aliases for one not-yet-created home produced different lifecycle keys" + pass "secondmate home lifecycle keys follow filesystem identity" +} + if [ "${FM_TEST_FOCUSED:-}" = symlink-artifacts ]; then run_isolated_test test_task_owned_account_artifacts_reject_symlink_paths exit 0 fi +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-races ]; then + run_isolated_test test_secondmate_home_lock_key_fails_closed_without_fixed_hasher + run_isolated_test test_secondmate_home_lock_key_uses_filesystem_identity + exit 0 +fi + if [ "${FM_TEST_FOCUSED:-}" = tail-safety ]; then run_isolated_test test_task_owned_account_artifacts_reject_symlink_paths run_isolated_test test_account_lineage_rejects_parent_swap_during_transaction @@ -5512,6 +5984,11 @@ if [ "${FM_TEST_FOCUSED:-}" = explicit-secondmate-route ]; then exit 0 fi +if [ "${FM_TEST_FOCUSED:-}" = secondmate-direct-scope ]; then + run_isolated_test test_secondmate_pool_routes_when_mode_is_enforced_and_mode_inherits + exit 0 +fi + if [ "${FM_TEST_FOCUSED:-}" = review-round-10 ]; then run_isolated_test test_managed_recovery_accepts_inherited_lifecycle_lock run_isolated_test test_native_resume_requires_fresh_sessionstart_evidence @@ -5742,7 +6219,6 @@ if [ "${FM_TEST_FOCUSED:-}" = signal-handoff ]; then fi if [ "${FM_TEST_FOCUSED:-}" = gate-e-workspace ]; then - run_isolated_test test_observe_is_dry_run_only run_isolated_test test_enforce_pool_wraps_backend_and_records_real_session run_isolated_test test_resume_uses_sticky_recovery_and_preserves_mapping_on_failure run_isolated_test test_lease_signal_handoff_publishes_cleanup_ownership @@ -5762,21 +6238,27 @@ fi if [ "${FM_TEST_FOCUSED:-}" = production-fleet-environment ]; then run_isolated_test test_production_fleet_environment_is_closed_before_control_and_worker_exec - run_isolated_test test_production_enforce_refuses_hostile_uncertified_parent_shell exit 0 fi run_isolated_test test_reserved_generation_is_durable_before_lease_mutation run_isolated_test test_off_is_byte_compatible_and_never_calls_agent_fleet +run_isolated_test test_failed_freshness_proof_rolls_back_unmanaged_resources +run_isolated_test test_local_only_spawn_uses_local_default_tip +run_isolated_test test_dirty_acquisition_is_retained_without_force_return +run_isolated_test test_treehouse_acquisition_timeout_is_bounded_before_endpoint_creation +run_isolated_test test_changed_acquisition_is_retained_during_unmanaged_rollback +run_isolated_test test_unmanaged_postinstall_failure_restores_prior_state run_isolated_test test_completion_contract_upgrade_is_contained_nonfollowing_and_atomic run_isolated_test test_completion_contract_ignores_raw_html_headings -run_isolated_test test_observe_is_dry_run_only run_isolated_test test_enforce_pool_wraps_backend_and_records_real_session run_isolated_test test_explicit_profile_uses_explicit_pool run_isolated_test test_enforce_failure_rolls_back_prepared_endpoint run_isolated_test test_pane_failure_happens_before_account_reservation run_isolated_test test_batch_partial_failure_releases_only_failed_item run_isolated_test test_resume_uses_sticky_recovery_and_preserves_mapping_on_failure +run_isolated_test test_recovered_secondmate_kind_blocks_unsupported_cmux_backend_early +run_isolated_test test_managed_recovery_rejects_duplicate_kind_metadata run_isolated_test test_managed_recovery_accepts_inherited_lifecycle_lock run_isolated_test test_inherited_lifecycle_handoff_releases_on_child_abort run_isolated_test test_inherited_lifecycle_lock_rejects_owner_aba @@ -5820,7 +6302,6 @@ run_isolated_test test_unknown_spawn_endpoint_retains_lease_for_retry run_isolated_test test_rollback_retry_rechecks_live_endpoint_before_release run_isolated_test test_failed_secondmate_rollback_preserves_home_for_relaunch run_isolated_test test_failed_secondmate_respawn_rollback_restores_prior_state -run_isolated_test test_observe_invalid_response_remains_advisory run_isolated_test test_explicit_secondmate_profile_ignores_configured_pool run_isolated_test test_enforced_orca_is_rejected_before_owned_resource_creation run_isolated_test test_cross_profile_continuation_for_harness claude claude-2 claude-3 claude @@ -5868,13 +6349,14 @@ run_isolated_test test_account_lock_owner_controls_reject_symlinks run_isolated_test test_linux_stat_selection_avoids_filesystem_stat_output run_isolated_test test_stale_reclaim_guard_is_owned_before_lock_removal run_isolated_test test_task_owned_account_artifacts_reject_symlink_paths +run_isolated_test test_secondmate_home_lock_key_fails_closed_without_fixed_hasher +run_isolated_test test_secondmate_home_lock_key_uses_filesystem_identity run_isolated_test test_account_lineage_rejects_parent_swap_during_transaction run_isolated_test test_agent_fleet_contract_is_validated_before_routing run_isolated_test test_agent_fleet_entrypoint_is_physically_pinned_per_operation run_isolated_test test_agent_fleet_validation_ignores_ambient_system_tool_shadows run_isolated_test test_production_routing_ignores_ambient_mode_and_forbids_binary_override run_isolated_test test_production_fleet_environment_is_closed_before_control_and_worker_exec -run_isolated_test test_production_enforce_refuses_hostile_uncertified_parent_shell run_isolated_test test_agent_fleet_lifecycle_calls_are_bounded run_isolated_test test_agent_fleet_selection_timeout_is_scoped_and_backward_compatible run_isolated_test test_unsuccessful_lease_mutations_always_reconcile diff --git a/tests/fm-backend-orca.test.sh b/tests/fm-backend-orca.test.sh index 7f5d1208b8e..63e04b4f2c9 100755 --- a/tests/fm-backend-orca.test.sh +++ b/tests/fm-backend-orca.test.sh @@ -2,6 +2,9 @@ # tests/fm-backend-orca.test.sh - fake-Orca-CLI unit tests for the Orca # terminal adapter primitives in bin/backends/orca.sh. set -u +export FM_ORCA_TEST_LAB=firstmate-orca-test-lab-v1 +export FM_ORCA_TEST_AUTHORITY_CAPABILITIES=verified-v1 +export FM_ORCA_TEST_BOUND_REMOVAL_CAPABILITIES=verified-v1 # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" @@ -23,16 +26,144 @@ next=$(( $(cat "$COUNT_FILE" 2>/dev/null || echo 0) + 1 )) for a in "$@"; do printf '\x1f%s' "$a"; done printf '\n' } >> "$LOG" +if [ "${1:-} ${2:-}" = "worktree create" ] \ + && [ "${FM_ORCA_QUARANTINE_STATE_READONLY:-0}" = 1 ] \ + && [ -n "${FM_STATE_OVERRIDE:-}" ]; then + chmod 500 "$FM_STATE_OVERRIDE" +fi if [ "${1:-}" = status ] && [ "${FM_ORCA_STATUS_RESPONSE:-ready}" != sequence ]; then printf '{"ok":true,"result":{"runtime":{"reachable":true,"state":"ready"}}}\n' exit 0 fi n=$next echo "$n" > "$COUNT_FILE" +if [ -f "$RESP/$n.out" ]; then + if [ "${1:-} ${2:-}" = "worktree show" ]; then + worktree_label= + worktree_terminal= + if [ -n "${FM_STATE_OVERRIDE:-}" ] && [ -d "$FM_STATE_OVERRIDE" ]; then + worktree_id=$(node -e ' +const fs = require("fs"); +const data = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); +const result = data.result || {}; +const worktree = result.worktree || result.item || result; +process.stdout.write(String(worktree.id || worktree.worktreeId || result.worktreeId || "")); +' "$RESP/$n.out") + for metadata in "$FM_STATE_OVERRIDE"/*.meta; do + [ -f "$metadata" ] || continue + if grep -Fxq "orca_worktree_id=$worktree_id" "$metadata"; then + worktree_label="fm-$(basename "$metadata" .meta)" + worktree_terminal=$(sed -n 's/^terminal=//p' "$metadata" | tail -1) + break + fi + done + fi + node -e ' +const fs = require("fs"); +const data = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); +const result = data.result || {}; +const worktree = result.worktree || result.item || result; +const label = process.argv[2] || ""; +const terminal = process.argv[3] || ""; +if (label && !worktree.name && !worktree.title && !result.worktreeName) { + worktree.name = label; +} +if (label && !Array.isArray(worktree.terminals) && !Array.isArray(result.terminals)) { + worktree.terminals = terminal ? [{handle: terminal, title: label}] : []; +} +process.stdout.write(JSON.stringify(data) + "\n"); +' "$RESP/$n.out" "$worktree_label" "$worktree_terminal" > "$RESP/.worktree-show" || exit 1 + cat "$RESP/.worktree-show" + elif [ "${1:-} ${2:-}" = "worktree create" ]; then + requested_name= + previous= + for argument in "$@"; do + [ "$previous" != --name ] || requested_name=$argument + previous=$argument + done + node -e ' +const fs = require("fs"); +const data = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); +const result = data.result || {}; +const worktree = result.worktree || result.item || null; +if (worktree && !worktree.name && !worktree.title && !result.worktreeName) { + worktree.name = process.argv[2] || ""; +} +process.stdout.write(JSON.stringify(data) + "\n"); +' "$RESP/$n.out" "$requested_name" || exit 1 + else + cat "$RESP/$n.out" + fi + if [ -f "$RESP/$n.exit" ]; then + exit "$(cat "$RESP/$n.exit")" + fi + exit 0 +fi if [ -f "$RESP/$n.exit" ]; then exit "$(cat "$RESP/$n.exit")" fi -[ -f "$RESP/$n.out" ] && cat "$RESP/$n.out" +case "${1:-} ${2:-}" in + "terminal read") + if [ -f "$RESP/.terminal-closed" ]; then + printf '{"ok":false,"error":{"code":"terminal_handle_stale","message":"terminal handle stale"}}\n' + exit 1 + fi + terminal= + previous= + for argument in "$@"; do + [ "$previous" != --terminal ] || terminal=$argument + previous=$argument + done + if [ -f "$RESP/.terminal-worktree-override" ]; then + worktree=$(cat "$RESP/.terminal-worktree-override") + else + worktree="wt-${terminal#term-}" + fi + if [ -f "$RESP/.terminal-title" ]; then + title=$(cat "$RESP/.terminal-title") + elif [ -n "${FM_STATE_OVERRIDE:-}" ] && [ -d "$FM_STATE_OVERRIDE" ]; then + title= + for metadata in "$FM_STATE_OVERRIDE"/*.meta; do + [ -f "$metadata" ] || continue + if grep -Fxq "terminal=$terminal" "$metadata"; then + title="fm-$(basename "$metadata" .meta)" + break + fi + done + [ -n "$title" ] || title="fm-${terminal#term-}" + else + title="fm-${terminal#term-}" + fi + printf '{"ok":true,"result":{"terminal":{"handle":"%s","title":"%s","worktreeId":"%s","tail":[]}}}\n' "$terminal" "$title" "$worktree" + ;; + "terminal close") + : > "$RESP/.terminal-closed" + printf '{"ok":true,"result":{"closed":true}}\n' + ;; + "worktree show") + if [ -f "$RESP/.worktree-show" ]; then + if [ -f "$RESP/.terminal-closed" ]; then + node -e ' +const fs = require("fs"); +const data = JSON.parse(fs.readFileSync(process.argv[1], "utf8")); +const result = data.result || {}; +const worktree = result.worktree || result.item || result; +worktree.terminals = []; +process.stdout.write(JSON.stringify(data) + "\n"); +' "$RESP/.worktree-show" + else + cat "$RESP/.worktree-show" + fi + fi + ;; + "worktree rm") + if [ -n "${FM_ORCA_REMOVE_ERROR:-}" ]; then + printf '{"ok":false,"error":{"code":"worktree_not_removed","message":"worktree not removed"}}\n' + else + printf '{"ok":true,"result":{"removed":true}}\n' + fi + ;; +esac exit 0 SH chmod +x "$fb/orca" @@ -56,9 +187,16 @@ neutral_fm_root() { # <dir> -> echoes a minimal root with a quiet guard exit 0 SH chmod +x "$root/bin/fm-guard.sh" + fm_git_init_commit "$root" printf '%s\n' "$root" } +initialize_secondmate_home_repo() { + local home=$1 source=$2 + fm_git_init_commit "$home" + git -C "$home" remote add origin "$source" +} + add_tmux_fake() { local fb=$1 cat > "$fb/tmux" <<'SH' @@ -361,15 +499,43 @@ test_send_key_refuses_escape_until_supported() { pass "fm_backend_orca_send_key: refuses Escape instead of mapping it to interrupt" } -test_kill_is_best_effort_close() { - orca_case kill +test_kill_propagates_close_failure() { + local status + orca_case kill-failure printf '1\n' > "$RESP/1.exit" + set +e PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ bash -c '. "$0/bin/backends/orca.sh"; fm_backend_orca_kill term-123' "$ROOT" - expect_code 0 $? "kill should stay best-effort when Orca close fails" + status=$? + set -e + [ "$status" -ne 0 ] || fail "kill should propagate an Orca close failure" assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-123'$'\x1f''--json' \ "kill did not call orca terminal close" - pass "fm_backend_orca_kill: calls terminal close and stays best-effort" + pass "fm_backend_orca_kill: propagates terminal close failure" +} + +test_terminal_state_classifies_closed_live_and_ambiguous_orca() { + local out + orca_case terminal-state-closed + printf '{"ok":false,"error":{"code":"terminal_handle_stale","message":"terminal handle stale"}}\n' > "$RESP/1.out" + printf '1\n' > "$RESP/1.exit" + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + bash -c '. "$0/bin/fm-backend.sh"; fm_backend_target_state orca term-closed fm-task' "$ROOT" ) + [ "$out" = absent ] || fail "stale Orca terminal should classify absent even when read exits nonzero, got '$out'" + + orca_case terminal-state-live + printf '{"ok":true,"result":{"terminal":{"handle":"term-live","title":"fm-task","tail":[]}}}\n' > "$RESP/1.out" + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + bash -c '. "$0/bin/fm-backend.sh"; fm_backend_target_state orca term-live fm-task' "$ROOT" ) + [ "$out" = present ] || fail "valid Orca terminal read should classify present, got '$out'" + + orca_case terminal-state-ambiguous + printf '{"ok":false,"error":{"code":"runtime_unavailable","message":"runtime unavailable"}}\n' > "$RESP/1.out" + printf '1\n' > "$RESP/1.exit" + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + bash -c '. "$0/bin/fm-backend.sh"; fm_backend_target_state orca term-unknown fm-task' "$ROOT" ) + [ "$out" = unknown ] || fail "ambiguous Orca read failure should classify unknown, got '$out'" + pass "fm_backend_target_state: classifies Orca terminal absence soundly" } test_remove_worktree_refuses_empty_id() { @@ -385,15 +551,37 @@ test_remove_worktree_refuses_empty_id() { } test_remove_worktree_rejects_orca_error_json() { - local out status + local out status token orca_case remove-error-json + token=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa printf '{"ok":false,"error":{"code":"worktree_not_found","message":"worktree not found"}}\n' > "$RESP/1.out" out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ - bash -c '. "$0/bin/backends/orca.sh"; fm_backend_orca_remove_worktree wt-gone' "$ROOT" 2>&1 ) + bash -c '. "$0/bin/backends/orca.sh"; fm_backend_orca_remove_worktree_bound wt-gone /tmp/orca-wt "$1"' "$ROOT" "$token" 2>&1 ) status=$? - [ "$status" -ne 0 ] || fail "remove_worktree should fail on Orca ok:false JSON" - assert_contains "$out" "worktree not found" "remove_worktree should surface the Orca removal error" - pass "fm_backend_orca_remove_worktree: fails closed on ok:false JSON" + [ "$status" -ne 0 ] || fail "bound remove_worktree should fail on Orca ok:false JSON" + assert_contains "$out" "worktree not found" "bound remove_worktree should surface the Orca removal error" + assert_contains "$(cat "$LOG")" $'--expected-path\x1f/tmp/orca-wt' \ + "bound remove_worktree omitted the provider path precondition" + assert_contains "$(cat "$LOG")" $'--expected-boundary-token\x1f'"$token" \ + "bound remove_worktree omitted the filesystem-boundary precondition" + pass "fm_backend_orca_remove_worktree_bound: fails closed on provider errors" +} + +test_remove_worktree_requires_bound_provider_capability() { + local out status token + orca_case remove-boundary-capability + token=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ORCA_TEST_BOUND_REMOVAL_CAPABILITIES=unavailable \ + bash -c '. "$0/bin/backends/orca.sh"; fm_backend_orca_remove_worktree_bound wt-retained /tmp/orca-wt "$1"' "$ROOT" "$token" 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "Orca removal proceeded without a bound provider capability" + assert_contains "$out" "identity-bound provider capability" \ + "unbound Orca removal did not surface its provider limitation" + [ ! -s "$LOG" ] || fail "unbound Orca removal reached the provider" + pass "Orca removal retains worktrees without a bound provider capability" } test_worktree_path_resolves_id() { @@ -409,7 +597,7 @@ test_worktree_path_resolves_id() { } test_json_get_ignores_undocumented_terminal_id_shapes() { - local out status wt_id wt_path term + local out status wt_id wt_path rest term orca_case parser-pruned-terminal-shapes set +e @@ -425,17 +613,18 @@ test_json_get_ignores_undocumented_terminal_id_shapes() { out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ bash -c '. "$0/bin/backends/orca.sh"; fm_backend_orca_worktree_create /repo/path fm-task' "$ROOT" ) wt_id=${out%%$'\t'*} - wt_path=${out#*$'\t'} - term=${wt_path#*$'\t'} - wt_path=${wt_path%%$'\t'*} + rest=${out#*$'\t'} + wt_path=${rest%%$'\t'*} + rest=${rest#*$'\t'} + term=${rest%%$'\t'*} [ "$wt_id" = wt-123 ] || fail "worktree helper should still print worktree id, got '$wt_id'" [ "$wt_path" = /tmp/orca-wt ] || fail "worktree helper should still print worktree path, got '$wt_path'" - [ "$term" = "$wt_path" ] || fail "worktree helper should ignore undocumented result.worktree.terminal and omit an implicit terminal, got '$out'" + [ -z "$term" ] || fail "worktree helper should ignore undocumented result.worktree.terminal and omit an implicit terminal, got '$out'" pass "fm_backend_orca_json_get: ignores undocumented terminal id shapes" } test_worktree_and_terminal_helpers_parse_json() { - local out wt_id wt_path term + local out wt_id wt_path rest term orca_case lifecycle-helpers printf '1\n' > "$RESP/1.exit" printf '{"ok":true,"result":{"repo":{"id":"repo-123"}}}\n' > "$RESP/2.out" @@ -444,7 +633,8 @@ test_worktree_and_terminal_helpers_parse_json() { out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ bash -c '. "$0/bin/backends/orca.sh"; fm_backend_orca_worktree_create /repo/path fm-task' "$ROOT" ) wt_id=${out%%$'\t'*} - wt_path=${out#*$'\t'} + rest=${out#*$'\t'} + wt_path=${rest%%$'\t'*} [ "$wt_id" = wt-123 ] || fail "worktree helper should print worktree id, got '$wt_id'" [ "$wt_path" = /tmp/orca-wt ] || fail "worktree helper should print worktree path, got '$wt_path'" term=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ @@ -461,23 +651,49 @@ test_worktree_and_terminal_helpers_parse_json() { pass "Orca lifecycle helpers: register repo, create worktree, create terminal, parse stable ids" } -test_worktree_create_removes_worktree_when_path_missing() { +test_worktree_create_retains_partial_authority_when_path_missing() { local out status orca_case lifecycle-missing-path printf '1\n' > "$RESP/1.exit" printf '{"ok":true,"result":{"repo":{"id":"repo-no-path"}}}\n' > "$RESP/2.out" printf '{"ok":true,"result":{"worktree":{"id":"wt-no-path"},"terminal":{"handle":"term-no-path"}}}\n' > "$RESP/3.out" + set +e out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ bash -c '. "$0/bin/backends/orca.sh"; fm_backend_orca_worktree_create /repo/path fm-task' "$ROOT" 2>&1 ) status=$? + set -e [ "$status" -ne 0 ] || fail "worktree helper should fail when Orca omits the worktree path" - assert_contains "$out" "orca worktree create did not return a path for fm-task" \ + assert_contains "$out" "orca worktree create returned incomplete or unsuccessful authority for fm-task" \ "worktree helper did not explain the missing path" - assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-no-path'$'\x1f''--json' \ - "worktree helper did not close the implicit terminal when path parsing failed" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-no-path'$'\x1f''--force'$'\x1f''--json' \ - "worktree helper did not remove the pathless Orca worktree" - pass "fm_backend_orca_worktree_create: removes created worktree when path is missing" + assert_contains "$out" $'wt-no-path\t\tterm-no-path\trecorded\trepo-no-path' \ + "worktree helper did not return every partial create identity" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "worktree helper closed a terminal before durable quarantine" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "worktree helper removed a pathless worktree before absence proof" + pass "fm_backend_orca_worktree_create: returns partial authority without cleanup" +} + +test_worktree_create_never_cleans_partial_response_inline() { + local out status + orca_case lifecycle-close-failure + printf '1\n' > "$RESP/1.exit" + printf '{"ok":true,"result":{"repo":{"id":"repo-close-failure"}}}\n' > "$RESP/2.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-close-failure"},"terminal":{"handle":"term-close-failure"}}}\n' > "$RESP/3.out" + printf '{"ok":false,"error":{"code":"terminal_close_failed","message":"terminal close failed"}}\n' > "$RESP/4.out" + set +e + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + bash -c '. "$0/bin/backends/orca.sh"; fm_backend_orca_worktree_create /repo/path fm-task' "$ROOT" 2>&1 ) + status=$? + set -e + expect_code 2 "$status" "pathless Orca worktree partial-response status" + assert_contains "$out" $'wt-close-failure\t\tterm-close-failure' \ + "partial response did not return durable Orca cleanup identity" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "worktree helper attempted cleanup before its caller durably quarantined identity" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "worktree helper removed a partial-response worktree" + pass "Orca worktree creation delegates partial-response cleanup to quarantine" } test_spawn_preserves_orca_metadata_when_pathless_worktree_cleanup_fails() { @@ -505,15 +721,18 @@ test_spawn_preserves_orca_metadata_when_pathless_worktree_cleanup_fails() { "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1 ) status=$? [ "$status" -ne 0 ] || fail "Orca spawn should fail when path parsing and cleanup fail" - assert_contains "$out" "orca worktree create did not return a path" \ + assert_contains "$out" "orca worktree create returned incomplete or unsuccessful authority" \ "pathless worktree failure should explain the missing path" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-pathless-cleanup'$'\x1f''--force'$'\x1f''--json' \ - "pathless cleanup should attempt helper-backed worktree removal" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "pathless cleanup removed a worktree without terminal-absence proof" assert_present "$state/$id.meta" "failed pathless cleanup should preserve metadata" assert_grep "window=fm-$id" "$state/$id.meta" "preserved pathless metadata missing stable window alias" assert_grep "backend=orca" "$state/$id.meta" "preserved pathless metadata missing backend=orca" assert_grep "orca_worktree_id=wt-pathless-cleanup" "$state/$id.meta" "preserved pathless metadata missing Orca worktree id" assert_no_grep "terminal=" "$state/$id.meta" "preserved pathless metadata should not invent a terminal handle" + assert_no_grep '^worktree=' "$state/$id.meta" "preserved pathless metadata should not record an empty worktree path" + assert_grep 'orca_cleanup_pending=1' "$state/$id.meta" "preserved pathless metadata missing cleanup quarantine" + assert_grep 'orca_cleanup_phase=spawn-abort' "$state/$id.meta" "preserved pathless metadata missing cleanup phase" assert_no_grep "report_required=" "$state/$id.meta" "preserved pathless metadata must keep the legacy no-report contract" pass "fm-spawn.sh --backend orca: preserves metadata when pathless cleanup fails" } @@ -548,6 +767,10 @@ test_legacy_respawn_writes_orca_metadata_and_launches_harness() { assert_grep "window=fm-$id" "$state/$id.meta" "meta missing stable Orca window alias" assert_grep "terminal=term-spawn" "$state/$id.meta" "meta missing terminal handle" assert_grep "orca_worktree_id=wt-spawn" "$state/$id.meta" "meta missing Orca worktree id" + assert_grep "orca_repo_id=repo-spawn" "$state/$id.meta" "meta missing Orca repo id" + assert_grep "orca_expected_task=fm-$id" "$state/$id.meta" "meta missing expected Orca task" + assert_grep "orca_discovery_label=fm-$id" "$state/$id.meta" "meta missing Orca discovery label" + assert_grep "orca_provider_scope=repo-path:$proj" "$state/$id.meta" "meta missing Orca provider scope" assert_grep "worktree=$wt" "$state/$id.meta" "meta missing Orca worktree path" assert_no_grep "report_required=" "$state/$id.meta" "legacy respawn must preserve the absent report_required marker" assert_not_contains "$(cat "$log")" $'orca\x1f''terminal'$'\x1f''create' \ @@ -626,6 +849,36 @@ test_spawn_refuses_orca_respawn_of_report_required_task() { pass "fm-spawn.sh --backend orca: refuses respawning a report-required task before any owned mutation" } +test_spawn_refuses_malformed_legacy_orca_report_metadata() { + local proj data state config id out status + id="orcamalformedreportz2" + proj="$TMP_ROOT/malformed-report-project" + data="$TMP_ROOT/malformed-report-data" + state="$TMP_ROOT/malformed-report-state" + config="$TMP_ROOT/malformed-report-config" + fm_git_init_commit "$proj" + mkdir -p "$data/$id" "$state" "$config" + printf 'brief\n' > "$data/$id/brief.md" + touch "$state/.last-watcher-beat" + seed_legacy_task_meta "$state" "$id" "$proj" + printf 'report_required=0\n' >> "$state/$id.meta" + orca_case malformed-report-refusal + set +e + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1 ) + status=$? + set -e + [ "$status" -ne 0 ] || fail "malformed legacy Orca report metadata was launched" + assert_contains "$out" "invalid report_required metadata for $id" \ + "malformed legacy Orca report metadata was not diagnosed" + assert_grep 'report_required=0' "$state/$id.meta" \ + "malformed legacy metadata was rewritten during refusal" + [ ! -s "$LOG" ] || fail "malformed legacy Orca metadata reached backend mutation" + pass "Orca recovery accepts only an absent report marker" +} + test_spawn_refuses_report_required_orca_batch_pair_before_mutation() { local proj data state config id out status id="orcabatchrefusez7" @@ -841,6 +1094,40 @@ test_spawn_refuses_orca_when_runtime_not_ready() { pass "fm-spawn.sh --backend orca: refuses before mutation when Orca runtime is not ready" } +test_spawn_refuses_orca_without_verified_authority_capabilities() { + local proj data state config id out status + id="orcacapabilityz7" + proj="$TMP_ROOT/capability-project" + data="$TMP_ROOT/capability-data" + state="$TMP_ROOT/capability-state" + config="$TMP_ROOT/capability-config" + fm_git_init_commit "$proj" + mkdir -p "$data/$id" "$state" "$config" + printf 'brief\n' > "$data/$id/brief.md" + touch "$state/.last-watcher-beat" + orca_case capability-absence + seed_legacy_task_meta "$state" "$id" "$proj" + add_dead_tmux_fake "$FB" + + set +e + out=$(env -u FM_ORCA_TEST_LAB -u FM_ORCA_TEST_AUTHORITY_CAPABILITIES \ + PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "spawn accepted Orca without verified lifecycle authority capabilities" + assert_contains "$out" "Orca lifecycle authority is disabled" \ + "capability refusal did not explain the unsupported authority boundary" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''repo' \ + "capability refusal reached Orca repository mutation" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''create' \ + "capability refusal reached Orca worktree creation" + pass "Orca spawn fails closed when lifecycle authority capabilities are unavailable" +} + test_spawn_refuses_orca_nonisolated_worktree() { local proj data state config id out status id="orcabadwtz4" @@ -858,27 +1145,117 @@ test_spawn_refuses_orca_nonisolated_worktree() { printf '1\n' > "$RESP/1.exit" printf '{"ok":true,"result":{"repo":{"id":"repo-bad"}}}\n' > "$RESP/2.out" printf '{"ok":true,"result":{"worktree":{"id":"wt-bad","path":"%s"},"terminal":{"handle":"term-bad"}}}\n' "$proj" > "$RESP/3.out" + set +e out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1 ) status=$? + set -e expect_code 1 "$status" "fm-spawn.sh --backend orca should refuse a primary checkout worktree" assert_contains "$out" "orca worktree create did not yield an isolated worktree" \ "Orca spawn should reuse the isolated-worktree guard" - assert_grep "window=legacy:fm-$id" "$state/$id.meta" "aborted Orca spawn must leave the legacy meta unchanged" - assert_no_grep "backend=orca" "$state/$id.meta" "aborted Orca spawn must not record Orca endpoint metadata" - assert_no_grep "orca_worktree_id=" "$state/$id.meta" "aborted Orca spawn must not record an Orca worktree id" + assert_grep "backend=orca" "$state/$id.meta" "aborted Orca spawn did not retain cleanup metadata" + assert_grep "orca_worktree_id=wt-bad" "$state/$id.meta" "aborted Orca spawn lost the unsafe provider worktree id" + assert_grep "terminal=term-bad" "$state/$id.meta" "aborted Orca spawn lost the unsafe terminal handle" + assert_grep "orca_cleanup_pending=1" "$state/$id.meta" "aborted Orca spawn did not remain quarantined" assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''create' \ "Orca spawn should validate the worktree before creating a terminal" - assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-bad'$'\x1f''--json' \ - "Orca spawn should close the implicit terminal after validation aborts" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-bad'$'\x1f''--force'$'\x1f''--json' \ - "Orca spawn should remove the worktree after validation aborts" - pass "fm-spawn.sh --backend orca: refuses non-isolated worktrees and closes implicit terminals" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "Orca spawn closed a terminal whose returned worktree was the primary checkout" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "Orca spawn removed a provider worktree whose isolation was unproven" + pass "fm-spawn.sh --backend orca: quarantines non-isolated worktree responses" +} + +test_spawn_quarantines_unrelated_orca_worktree() { + local proj unrelated data state config id out status + id="orcaunrelatedz5" + proj="$TMP_ROOT/unrelated-source-project" + unrelated="$TMP_ROOT/unrelated-provider-worktree" + data="$TMP_ROOT/unrelated-data" + state="$TMP_ROOT/unrelated-state" + config="$TMP_ROOT/unrelated-config" + fm_git_init_commit "$proj" + fm_git_init_commit "$unrelated" + mkdir -p "$data/$id" "$state" "$config" + printf 'brief\n' > "$data/$id/brief.md" + touch "$state/.last-watcher-beat" + orca_case unrelated-worktree + seed_legacy_task_meta "$state" "$id" "$proj" + add_dead_tmux_fake "$FB" + printf '1\n' > "$RESP/1.exit" + printf '{"ok":true,"result":{"repo":{"id":"repo-unrelated"}}}\n' > "$RESP/2.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-unrelated","path":"%s"},"terminal":{"handle":"term-unrelated"}}}\n' \ + "$unrelated" > "$RESP/3.out" + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "spawn accepted an Orca worktree from an unrelated repository" + assert_contains "$out" "returned a worktree from an unrelated repository" \ + "unrelated Orca repository identity was not surfaced" + assert_grep "orca_worktree_id=wt-unrelated" "$state/$id.meta" \ + "unrelated Orca worktree identity was not quarantined" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "spawn closed a terminal attached to an unrelated repository" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "spawn removed an unrelated provider worktree" + pass "Orca spawn quarantines unrelated provider worktrees" } -test_spawn_removes_orca_worktree_when_terminal_create_fails() { +test_spawn_quarantines_unbound_orca_terminal() { + local proj wt data state config id out status + id="orcaunboundtermz6" + proj="$TMP_ROOT/unbound-terminal-project" + wt="$TMP_ROOT/unbound-terminal-wt" + data="$TMP_ROOT/unbound-terminal-data" + state="$TMP_ROOT/unbound-terminal-state" + config="$TMP_ROOT/unbound-terminal-config" + fm_git_worktree "$proj" "$wt" "fm/$id" + mkdir -p "$data/$id" "$state" "$config" + printf 'brief\n' > "$data/$id/brief.md" + touch "$state/.last-watcher-beat" + orca_case unbound-terminal + seed_legacy_task_meta "$state" "$id" "$proj" + add_dead_tmux_fake "$FB" + printf '1\n' > "$RESP/1.exit" + printf '{"ok":true,"result":{"repo":{"id":"repo-unbound"}}}\n' > "$RESP/2.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-unbound","path":"%s"},"terminal":{"handle":"term-unbound"}}}\n' \ + "$wt" > "$RESP/3.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-unbound","name":"fm-%s","path":"%s","terminals":[{"handle":"term-unbound","title":"fm-%s"}]}}}\n' \ + "$id" "$wt" "$id" > "$RESP/4.out" + printf 'fm-another-task\n' > "$RESP/.terminal-title" + printf 'wt-unbound\n' > "$RESP/.terminal-worktree-override" + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "spawn accepted a terminal not bound to its expected task" + assert_contains "$out" "terminal is not authoritatively bound" \ + "unbound Orca terminal authority was not surfaced" + assert_grep "terminal=term-unbound" "$state/$id.meta" \ + "unbound Orca terminal was not retained in quarantine" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''send' \ + "spawn sent launch commands through an unbound terminal" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "spawn closed a terminal whose task binding was unproven" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "spawn removed a worktree while terminal authority was unproven" + pass "Orca spawn quarantines terminals without task authority" +} + +test_spawn_quarantines_orca_worktree_when_terminal_create_fails() { local proj wt data state config id out status id="orcatermfailz8" proj="$TMP_ROOT/terminal-fail-project" @@ -903,16 +1280,18 @@ test_spawn_removes_orca_worktree_when_terminal_create_fails() { "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1 ) status=$? [ "$status" -ne 0 ] || fail "Orca spawn should fail when terminal creation fails" - assert_grep "window=legacy:fm-$id" "$state/$id.meta" "terminal-create abort must leave the legacy meta unchanged after successful cleanup" - assert_no_grep "backend=orca" "$state/$id.meta" "terminal-create abort should not record Orca metadata after successful cleanup" - assert_no_grep "orca_worktree_id=" "$state/$id.meta" "terminal-create abort should not record an Orca worktree id after successful cleanup" + assert_grep "window=fm-$id" "$state/$id.meta" "terminal-create abort must retain the cleanup alias" + assert_grep "backend=orca" "$state/$id.meta" "terminal-create abort should retain Orca cleanup metadata" + assert_grep "orca_worktree_id=wt-terminal-fail" "$state/$id.meta" "terminal-create abort should retain the Orca worktree id" assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''create'$'\x1f''--worktree'$'\x1f''id:wt-terminal-fail'$'\x1f''--title'$'\x1f'"fm-$id"$'\x1f''--json' \ "Orca spawn should attempt terminal creation before abort cleanup" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-terminal-fail'$'\x1f''--force'$'\x1f''--json' \ - "Orca spawn should remove the worktree when terminal creation fails" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "Orca spawn removed a worktree after ambiguous terminal creation" assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ "Orca spawn should not close a terminal when no handle was recorded" - pass "fm-spawn.sh --backend orca: removes worktree when terminal creation fails" + assert_grep 'orca_cleanup_pending=1' "$state/$id.meta" \ + "ambiguous terminal creation did not quarantine the worktree" + pass "fm-spawn.sh --backend orca: quarantines ambiguous terminal creation" } test_spawn_preserves_orca_metadata_when_abort_cleanup_fails() { @@ -941,8 +1320,8 @@ test_spawn_preserves_orca_metadata_when_abort_cleanup_fails() { "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1 ) status=$? [ "$status" -ne 0 ] || fail "Orca spawn should fail when terminal creation and abort cleanup fail" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-cleanup-fail'$'\x1f''--force'$'\x1f''--json' \ - "Orca spawn should attempt helper cleanup before preserving metadata" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "Orca spawn removed a worktree while terminal absence was unknown" assert_present "$state/$id.meta" "failed Orca abort cleanup should preserve metadata" assert_grep "window=fm-$id" "$state/$id.meta" "preserved metadata missing stable window alias" assert_grep "backend=orca" "$state/$id.meta" "preserved metadata missing backend=orca" @@ -951,6 +1330,74 @@ test_spawn_preserves_orca_metadata_when_abort_cleanup_fails() { pass "fm-spawn.sh --backend orca: preserves metadata when abort cleanup fails" } +test_spawn_retains_orca_worktree_when_abort_close_fails() { + local proj data state config id out status + id="orcaabortclosez4" + proj="$TMP_ROOT/abort-close-project" + data="$TMP_ROOT/abort-close-data" + state="$TMP_ROOT/abort-close-state" + config="$TMP_ROOT/abort-close-config" + fm_git_init_commit "$proj" + mkdir -p "$data/$id" "$state" "$config" + printf 'brief\n' > "$data/$id/brief.md" + touch "$state/.last-watcher-beat" + orca_case abort-close-failure + seed_legacy_task_meta "$state" "$id" "$proj" + add_dead_tmux_fake "$FB" + printf '1\n' > "$RESP/1.exit" + printf '{"ok":true,"result":{"repo":{"id":"repo-abort-close"}}}\n' > "$RESP/2.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-abort-close","path":"%s"},"terminal":{"handle":"term-abort-close"}}}\n' "$proj" > "$RESP/3.out" + printf '{"ok":false,"error":{"code":"terminal_close_failed","message":"terminal close failed"}}\n' > "$RESP/4.out" + set +e + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1 ) + status=$? + set -e + [ "$status" -ne 0 ] || fail "Orca spawn should fail after non-isolated worktree creation" + assert_contains "$out" "retaining Orca cleanup metadata" \ + "abort close failure did not surface durable retention" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "abort cleanup removed an Orca worktree after terminal close failed" + assert_grep 'terminal=term-abort-close' "$state/$id.meta" \ + "abort close failure did not preserve terminal identity" + assert_grep 'orca_worktree_id=wt-abort-close' "$state/$id.meta" \ + "abort close failure did not preserve worktree identity" + assert_grep 'orca_cleanup_pending=1' "$state/$id.meta" \ + "abort close failure did not mark durable cleanup state" + assert_no_grep 'report_required=' "$state/$id.meta" \ + "abort close failure changed the legacy report contract" + pass "Orca abort cleanup retains worktrees until terminal absence is proven" +} + +test_teardown_rejects_symlinked_orca_task_metadata() { + local state data config neutral out status + state="$TMP_ROOT/orca-meta-alias-state" + data="$TMP_ROOT/orca-meta-alias-data" + config="$TMP_ROOT/orca-meta-alias-config" + mkdir -p "$state" "$data" "$config" + fm_write_meta "$state/bar.meta" \ + 'window=fm-bar' 'terminal=term-bar' 'worktree=/missing/bar' 'project=/missing/project' \ + 'harness=claude' 'kind=ship' 'mode=no-mistakes' 'backend=orca' 'orca_worktree_id=wt-bar' + ln -s bar.meta "$state/foo.meta" + orca_case teardown-meta-alias + neutral=$(neutral_fm_root "$CASE_DIR/meta-alias-neutral") + set +e + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + "$ROOT/bin/fm-teardown.sh" foo --force 2>&1 ) + status=$? + set -e + [ "$status" -ne 0 ] || fail "symlinked Orca task metadata was accepted" + assert_contains "$out" "task metadata must be a real readable file for foo" \ + "symlinked Orca task metadata was not diagnosed" + assert_present "$state/foo.meta" "symlinked task metadata was removed" + assert_present "$state/bar.meta" "aliased task metadata target was removed" + [ ! -s "$LOG" ] || fail "symlinked Orca task metadata reached backend mutation" + pass "Orca teardown binds real metadata to the requested task" +} + test_spawn_refuses_invalid_state_before_orca_resource_creation() { local proj data state_file config id out status id="orcametafailz9" @@ -1088,7 +1535,7 @@ test_scout_teardown_removes_orca_worktree_via_helper() { expect_code 0 "$rc" "Orca scout teardown should succeed once report exists"$'\n'"$out" assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-teardown'$'\x1f''--json' \ "teardown did not close the recorded Orca terminal" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-teardown'$'\x1f''--force'$'\x1f''--json' \ + assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-teardown' \ "teardown did not remove the Orca worktree through orca worktree rm" assert_absent "$state/$id.meta" "teardown should remove task metadata" pass "fm-teardown.sh backend=orca: scout report gate then helper-backed worktree removal" @@ -1132,7 +1579,7 @@ test_scout_teardown_refuses_orca_id_path_mismatch() { pass "fm-teardown.sh backend=orca: scout teardown refuses id/path mismatches" } -test_teardown_removes_orca_worktree_when_path_missing() { +test_teardown_refuses_orca_worktree_when_path_missing() { local proj wt data state config id out rc neutral id="orcamissingpathz7" proj="$TMP_ROOT/missing-path-project" @@ -1140,6 +1587,7 @@ test_teardown_removes_orca_worktree_when_path_missing() { data="$TMP_ROOT/missing-path-data" state="$TMP_ROOT/missing-path-state" config="$TMP_ROOT/missing-path-config" + fm_git_init_commit "$proj" mkdir -p "$data/$id" "$state" "$config" printf 'report\n' > "$data/$id/report.md" touch "$state/.last-watcher-beat" @@ -1155,13 +1603,12 @@ test_teardown_removes_orca_worktree_when_path_missing() { "$ROOT/bin/fm-teardown.sh" "$id" 2>&1 ) rc=$? set -e - expect_code 0 "$rc" "Orca teardown should release helpers even when the path is absent"$'\n'"$out" - assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-missing-path'$'\x1f''--json' \ - "teardown did not close the recorded Orca terminal when the path was absent" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-missing-path'$'\x1f''--force'$'\x1f''--json' \ - "teardown did not remove the recorded Orca worktree when the path was absent" - assert_absent "$state/$id.meta" "successful helper cleanup should remove task metadata" - pass "fm-teardown.sh backend=orca: releases terminal/worktree when path is absent" + [ "$rc" -ne 0 ] || fail "Orca teardown should refuse when its worktree path is absent" + assert_contains "$out" "teardown worktree metadata is not an exact repository root" \ + "pathless Orca teardown should surface the unprovable target identity" + [ ! -s "$LOG" ] || fail "pathless Orca teardown should not close a terminal or remove a worktree" + assert_present "$state/$id.meta" "pathless Orca teardown should preserve task metadata" + pass "fm-teardown.sh backend=orca: refuses pathless provider removal" } test_teardown_preserves_metadata_when_orca_remove_error_json() { @@ -1172,18 +1619,19 @@ test_teardown_preserves_metadata_when_orca_remove_error_json() { data="$TMP_ROOT/remove-error-data" state="$TMP_ROOT/remove-error-state" config="$TMP_ROOT/remove-error-config" + fm_git_worktree "$proj" "$wt" "fm/$id" mkdir -p "$data/$id" "$state" "$config" printf 'report\n' > "$data/$id/report.md" touch "$state/.last-watcher-beat" fm_write_meta "$state/$id.meta" \ - "window=fm-$id" "worktree=$wt" "project=$proj" \ + "window=fm-$id" "terminal=term-remove-error" "worktree=$wt" "project=$proj" \ "harness=claude" "kind=scout" "mode=no-mistakes" "yolo=off" \ "backend=orca" "orca_worktree_id=wt-remove-error" orca_case remove-error-teardown - printf '{"ok":false,"error":{"code":"worktree_not_removed","message":"worktree not removed"}}\n' > "$RESP/1.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-remove-error","path":"%s"}}}\n' "$wt" > "$RESP/1.out" neutral=$(neutral_fm_root "$CASE_DIR/neutral") set +e - out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" FM_ORCA_REMOVE_ERROR=1 \ FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ "$ROOT/bin/fm-teardown.sh" "$id" 2>&1 ) rc=$? @@ -1283,12 +1731,47 @@ test_ship_teardown_removes_orca_worktree_when_id_path_matches() { "teardown did not resolve the Orca worktree id before removal" assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-ship-match'$'\x1f''--json' \ "teardown did not close the matched Orca terminal" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-ship-match'$'\x1f''--force'$'\x1f''--json' \ + assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-ship-match' \ "teardown did not remove the matched Orca worktree" assert_absent "$state/$id.meta" "successful matched teardown should remove task metadata" pass "fm-teardown.sh backend=orca: ship teardown requires a matching Orca id path" } +test_ship_teardown_rejects_orca_mounted_removal_root() { + local proj wt data state config id out rc neutral + id="orcamountedrootz6" + proj="$TMP_ROOT/mounted-root-project" + wt="$TMP_ROOT/mounted-root-wt" + data="$TMP_ROOT/mounted-root-data" + state="$TMP_ROOT/mounted-root-state" + config="$TMP_ROOT/mounted-root-config" + fm_git_worktree "$proj" "$wt" "fm/$id" + mkdir -p "$data/$id" "$state" "$config" + touch "$state/.last-watcher-beat" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "terminal=term-mounted-root" "worktree=$wt" "project=$proj" \ + "harness=claude" "kind=ship" "mode=local-only" "yolo=off" \ + "backend=orca" "orca_worktree_id=wt-mounted-root" + orca_case mounted-root + printf '{"ok":true,"result":{"worktree":{"id":"wt-mounted-root","path":"%s"}}}\n' "$wt" > "$RESP/1.out" + neutral=$(neutral_fm_root "$CASE_DIR/neutral") + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" \ + FM_CONFIG_OVERRIDE="$config" FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ + FM_TEARDOWN_TEST_MOUNT_PATH="$wt" \ + "$ROOT/bin/fm-teardown.sh" "$id" 2>&1) + rc=$? + set -e + expect_code 1 "$rc" "mounted Orca worktree root must block provider removal" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "mounted Orca worktree reached provider removal" + assert_present "$state/$id.meta" "mounted Orca worktree removed retry metadata" + assert_contains "$out" "crosses an untrusted filesystem boundary" \ + "mounted Orca worktree was not surfaced" + pass "Orca removal rejects mounted worktree roots" +} + test_ship_teardown_refuses_orca_unresolvable_worktree_id() { local proj wt data state config id out rc neutral id="orcashipunresolvedz1" @@ -1395,7 +1878,7 @@ test_teardown_refuses_orca_missing_worktree_id() { pass "fm-teardown.sh backend=orca: refuses missing worktree ids before cleanup" } -test_teardown_removes_orca_worktree_without_terminal_handle() { +test_teardown_refuses_orca_worktree_without_terminal_handle() { local proj wt data state config id out rc neutral id="orcanotermz0" proj="$TMP_ROOT/no-terminal-project" @@ -1420,13 +1903,15 @@ test_teardown_removes_orca_worktree_without_terminal_handle() { "$ROOT/bin/fm-teardown.sh" "$id" 2>&1 ) rc=$? set -e - expect_code 0 "$rc" "Orca teardown should remove a worktree even when no terminal was ever recorded"$'\n'"$out" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-no-terminal'$'\x1f''--force'$'\x1f''--json' \ - "teardown did not remove the partial Orca worktree" + [ "$rc" -ne 0 ] || fail "Orca teardown should refuse when terminal identity is missing" + assert_contains "$out" "missing terminal" \ + "missing Orca terminal identity was not surfaced" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "teardown removed an Orca worktree without proving endpoint absence" assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ "teardown should not close a terminal when no terminal handle is recorded" - assert_absent "$state/$id.meta" "successful partial cleanup should remove task metadata" - pass "fm-teardown.sh backend=orca: removes partial worktree-only metadata" + assert_present "$state/$id.meta" "unknown Orca endpoint state should retain metadata" + pass "fm-teardown.sh backend=orca: refuses unquiesced partial metadata" } test_secondmate_force_teardown_removes_orca_child_via_orca() { @@ -1450,9 +1935,11 @@ test_secondmate_force_teardown_removes_orca_child_via_orca() { "harness=claude" "kind=ship" "mode=no-mistakes" "yolo=off" \ "backend=orca" "orca_worktree_id=wt-child-cleanup" orca_case secondmate-child-cleanup - printf '{"ok":true,"result":{"worktree":{"id":"wt-child-cleanup","path":"%s"}}}\n' "$childwt" > "$RESP/1.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-child-cleanup","name":"fm-%s","path":"%s","terminals":[{"handle":"term-child-cleanup","title":"fm-%s"}]}}}\n' \ + "$child_id" "$childwt" "$child_id" > "$RESP/1.out" add_tmux_fake "$FB" neutral=$(neutral_fm_root "$CASE_DIR/neutral") + initialize_secondmate_home_repo "$subhome" "$neutral" set +e out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ FM_ROOT_OVERRIDE="$neutral" FM_HOME="$home" "$ROOT/bin/fm-teardown.sh" domain --force 2>&1 ) @@ -1461,7 +1948,7 @@ test_secondmate_force_teardown_removes_orca_child_via_orca() { expect_code 0 "$rc" "forced secondmate teardown should remove Orca child work through Orca"$'\n'"$out" assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-child-cleanup'$'\x1f''--json' \ "child cleanup did not close the recorded Orca terminal" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-child-cleanup'$'\x1f''--force'$'\x1f''--json' \ + assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-child-cleanup' \ "child cleanup did not remove the Orca worktree through orca worktree rm" assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f'"fm-$child_id" \ "child cleanup closed the stable alias instead of the Orca terminal" @@ -1495,6 +1982,7 @@ test_secondmate_force_teardown_refuses_orca_child_id_path_mismatch() { printf '{"ok":true,"result":{"worktree":{"id":"wt-child-mismatch","path":"%s"}}}\n' "$other_wt" > "$RESP/1.out" add_tmux_fake "$FB" neutral=$(neutral_fm_root "$CASE_DIR/neutral") + initialize_secondmate_home_repo "$subhome" "$neutral" set +e out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ FM_ROOT_OVERRIDE="$neutral" FM_HOME="$home" "$ROOT/bin/fm-teardown.sh" domain --force 2>&1 ) @@ -1511,7 +1999,7 @@ test_secondmate_force_teardown_refuses_orca_child_id_path_mismatch() { pass "fm-teardown.sh --force: refuses Orca child id/path mismatches" } -test_secondmate_force_teardown_removes_partial_orca_child() { +test_secondmate_force_teardown_retains_partial_orca_child() { local home subhome childproj childwt child_id neutral out rc home="$TMP_ROOT/orca-partial-child-parent" subhome="$TMP_ROOT/orca-partial-child-secondmate" @@ -1535,18 +2023,22 @@ test_secondmate_force_teardown_removes_partial_orca_child() { printf '{"ok":true,"result":{"worktree":{"id":"wt-partial-child","path":"%s"}}}\n' "$childwt" > "$RESP/1.out" add_tmux_fake "$FB" neutral=$(neutral_fm_root "$CASE_DIR/neutral") + initialize_secondmate_home_repo "$subhome" "$neutral" set +e out=$( PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ FM_ROOT_OVERRIDE="$neutral" FM_HOME="$home" "$ROOT/bin/fm-teardown.sh" domain --force 2>&1 ) rc=$? set -e - expect_code 0 "$rc" "forced secondmate teardown should remove partial Orca child state"$'\n'"$out" - assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-partial-child'$'\x1f''--force'$'\x1f''--json' \ - "partial child cleanup did not remove the Orca worktree through orca worktree rm" + [ "$rc" -ne 0 ] || fail "forced secondmate teardown should refuse partial Orca child state" + assert_contains "$out" "child endpoint identity for $child_id is missing" \ + "partial Orca child refusal did not surface missing endpoint identity" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "partial child cleanup removed an Orca worktree without quiescence proof" assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ "partial child cleanup should not close a terminal when no terminal handle is recorded" - assert_absent "$home/state/domain.meta" "parent metadata should be removed after forced partial cleanup" - pass "fm-teardown.sh --force: removes partial Orca secondmate children" + assert_present "$home/state/domain.meta" "partial child refusal removed parent metadata" + assert_present "$subhome/state/$child_id.meta" "partial child refusal removed child metadata" + pass "fm-teardown.sh --force: retains partial Orca secondmate children" } test_dispatcher_sources_orca_and_routes_primitives() { @@ -1559,6 +2051,450 @@ test_dispatcher_sources_orca_and_routes_primitives() { pass "fm-backend dispatcher: accepts orca and routes capture through bin/backends/orca.sh" } +test_spawn_refuses_cleanup_pending_orca_task_before_mutation() { + local proj data state config id out status + id="orcacleanupblockz9" + proj="$TMP_ROOT/cleanup-block-project" + data="$TMP_ROOT/cleanup-block-data" + state="$TMP_ROOT/cleanup-block-state" + config="$TMP_ROOT/cleanup-block-config" + fm_git_init_commit "$proj" + mkdir -p "$data/$id" "$state" "$config" + printf 'brief\n' > "$data/$id/brief.md" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "project=$proj" "harness=claude" "kind=ship" "mode=local-only" \ + "backend=orca" "orca_worktree_id=wt-retained" "terminal=term-retained" \ + "orca_cleanup_pending=1" "orca_cleanup_phase=spawn-abort" "orca_terminal_proof=recorded" + orca_case cleanup-pending-block + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "spawn reused an Orca cleanup quarantine" + assert_contains "$out" "Orca cleanup is pending for $id" \ + "cleanup-pending spawn refusal did not explain the supported cleanup path" + assert_grep 'orca_worktree_id=wt-retained' "$state/$id.meta" \ + "cleanup-pending spawn overwrote the retained worktree identity" + [ ! -s "$LOG" ] || fail "cleanup-pending spawn reached Orca mutation" + pass "Orca cleanup quarantine blocks respawn without identity loss" +} + +test_pathless_orca_quarantine_has_supported_cleanup() { + local proj wt data state config id out status neutral + id="orcapathcleanupz7" + proj="$TMP_ROOT/path-cleanup-project" + wt="$TMP_ROOT/path-cleanup-wt" + data="$TMP_ROOT/path-cleanup-data" + state="$TMP_ROOT/path-cleanup-state" + config="$TMP_ROOT/path-cleanup-config" + fm_git_worktree "$proj" "$wt" "fm/$id" + mkdir -p "$data/$id" "$state" "$config" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "project=$proj" "harness=claude" "kind=ship" "mode=local-only" \ + "backend=orca" "orca_worktree_id=wt-path-cleanup" "terminal=term-path-cleanup" \ + "orca_cleanup_pending=1" "orca_cleanup_phase=spawn-abort" "orca_terminal_proof=recorded" \ + "orca_expected_task=fm-$id" + orca_case pathless-supported-cleanup + printf '{"ok":true,"result":{"worktree":{"id":"wt-path-cleanup","path":"%s"}}}\n' "$wt" > "$RESP/1.out" + neutral=$(neutral_fm_root "$CASE_DIR/neutral") + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$CASE_DIR/checkout-state" \ + "$ROOT/bin/fm-teardown.sh" "$id" 2>&1) + status=$? + set -e + + expect_code 0 "$status" "pathless Orca quarantine cleanup"$'\n'"$out" + assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-path-cleanup' \ + "pathless quarantine cleanup did not quiesce the retained terminal" + assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-path-cleanup' \ + "pathless quarantine cleanup did not remove the retained provider worktree" + assert_absent "$state/$id.meta" "successful pathless quarantine cleanup retained metadata" + pass "pathless Orca quarantine has a supported cleanup path" +} + +test_idless_orca_quarantine_refuses_unscoped_terminal_close() { + local proj data state config id out status neutral + id="orcaidlesscleanupz4" + proj="$TMP_ROOT/idless-cleanup-project" + data="$TMP_ROOT/idless-cleanup-data" + state="$TMP_ROOT/idless-cleanup-state" + config="$TMP_ROOT/idless-cleanup-config" + fm_git_init_commit "$proj" + mkdir -p "$data/$id" "$state" "$config" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "project=$proj" "harness=claude" "kind=ship" "mode=local-only" \ + "backend=orca" "terminal=term-idless" \ + "orca_cleanup_pending=1" "orca_cleanup_phase=spawn-abort" "orca_terminal_proof=recorded" \ + "orca_expected_task=fm-$id" "orca_discovery_label=fm-$id" "orca_provider_scope=repo-path:$proj" + orca_case idless-unscoped-cleanup + neutral=$(neutral_fm_root "$CASE_DIR/neutral") + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + "$ROOT/bin/fm-teardown.sh" "$id" 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "idless Orca quarantine closed an unscoped terminal" + assert_contains "$out" "refusing to close an unscoped terminal" \ + "idless Orca quarantine did not surface its missing worktree authority" + [ ! -s "$LOG" ] || fail "idless Orca quarantine reached provider mutation" + assert_present "$state/$id.meta" "idless Orca quarantine removed retained metadata" + pass "idless Orca quarantines retain unscoped terminals" +} + +test_teardown_refuses_orca_terminal_worktree_identity_drift() { + local proj wt data state config id out status neutral + id="orcatermdriftz5" + proj="$TMP_ROOT/terminal-drift-project" + wt="$TMP_ROOT/terminal-drift-wt" + data="$TMP_ROOT/terminal-drift-data" + state="$TMP_ROOT/terminal-drift-state" + config="$TMP_ROOT/terminal-drift-config" + fm_git_worktree "$proj" "$wt" "fm/$id" + mkdir -p "$data/$id" "$state" "$config" + printf 'report\n' > "$data/$id/report.md" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "terminal=term-terminal-drift" "worktree=$wt" "project=$proj" \ + "harness=claude" "kind=scout" "mode=no-mistakes" "backend=orca" \ + "orca_worktree_id=wt-terminal-drift" + orca_case terminal-worktree-drift + printf '{"ok":true,"result":{"worktree":{"id":"wt-terminal-drift","path":"%s"}}}\n' "$wt" > "$RESP/1.out" + printf 'wt-other\n' > "$RESP/.terminal-worktree-override" + neutral=$(neutral_fm_root "$CASE_DIR/neutral") + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + "$ROOT/bin/fm-teardown.sh" "$id" 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "teardown closed a terminal bound to another Orca worktree" + assert_contains "$out" "task Orca endpoint authority or quiescence is unproven" \ + "terminal/worktree identity drift was not surfaced" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "terminal/worktree identity drift closed the recorded terminal" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "terminal/worktree identity drift removed the provider worktree" + assert_present "$state/$id.meta" "terminal/worktree identity drift removed metadata" + pass "Orca teardown rejects terminal/worktree identity drift" +} + +test_spawn_quarantines_create_response_without_worktree_id() { + local proj data state config id out status + id="orcapartialidz8" + proj="$TMP_ROOT/partial-id-project" + data="$TMP_ROOT/partial-id-data" + state="$TMP_ROOT/partial-id-state" + config="$TMP_ROOT/partial-id-config" + fm_git_init_commit "$proj" + mkdir -p "$data/$id" "$state" "$config" + printf 'brief\n' > "$data/$id/brief.md" + touch "$state/.last-watcher-beat" + orca_case partial-create-id + seed_legacy_task_meta "$state" "$id" "$proj" + add_dead_tmux_fake "$FB" + printf '1\n' > "$RESP/1.exit" + printf '{"ok":true,"result":{"repo":{"id":"repo-partial-id"}}}\n' > "$RESP/2.out" + printf '{"ok":true,"result":{"worktree":{"path":"%s"},"terminal":{"handle":"term-partial-id"}}}\n' "$proj" > "$RESP/3.out" + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "spawn accepted an Orca create response without a worktree id" + assert_grep "worktree=$proj" "$state/$id.meta" \ + "partial create quarantine dropped the returned worktree path" + assert_grep 'terminal=term-partial-id' "$state/$id.meta" \ + "partial create quarantine dropped the returned terminal" + assert_grep 'orca_repo_id=repo-partial-id' "$state/$id.meta" \ + "partial create quarantine dropped the returned repo identity" + assert_grep "orca_expected_task=fm-$id" "$state/$id.meta" \ + "partial create quarantine dropped expected task identity" + assert_grep "orca_discovery_label=fm-$id" "$state/$id.meta" \ + "partial create quarantine dropped the discovery label" + assert_grep "orca_provider_scope=repo-path:$proj" "$state/$id.meta" \ + "partial create quarantine dropped the provider scope" + assert_no_grep '^orca_worktree_id=' "$state/$id.meta" \ + "partial create quarantine invented a missing worktree id" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "spawn closed a partial-response terminal without worktree authority" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "spawn removed a partial-response worktree without provider id" + pass "Orca partial create responses retain every available identity" +} + +test_orca_quarantine_write_failure_keeps_prearmed_blocker() { + local proj data state config id out status + id="orcaquarantinewritez2" + proj="$TMP_ROOT/quarantine-write-project" + data="$TMP_ROOT/quarantine-write-data" + state="$TMP_ROOT/quarantine-write-state" + config="$TMP_ROOT/quarantine-write-config" + fm_git_init_commit "$proj" + mkdir -p "$data/$id" "$state" "$config" + printf 'brief\n' > "$data/$id/brief.md" + touch "$state/.last-watcher-beat" + orca_case quarantine-write-failure + seed_legacy_task_meta "$state" "$id" "$proj" + add_dead_tmux_fake "$FB" + printf '1\n' > "$RESP/1.exit" + printf '{"ok":true,"result":{"repo":{"id":"repo-quarantine-write"}}}\n' > "$RESP/2.out" + printf '{"ok":true,"result":{"worktree":{"id":"wt-quarantine-write","path":"%s"}}}\n' "$proj" > "$RESP/3.out" + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ORCA_QUARANTINE_STATE_READONLY=1 \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1) + status=$? + chmod 700 "$state" + set -e + + [ "$status" -ne 0 ] || fail "spawn ignored an Orca quarantine update failure" + assert_grep 'orca_cleanup_pending=1' "$state/$id.meta" \ + "quarantine write failure lost the pre-armed retry blocker" + assert_grep 'orca_cleanup_phase=spawn-preparing' "$state/$id.meta" \ + "quarantine write failure replaced the durable pre-create phase" + assert_grep "orca_discovery_label=fm-$id" "$state/$id.meta" \ + "quarantine write failure lost the pre-armed discovery label" + assert_grep "orca_provider_scope=repo-path:$proj" "$state/$id.meta" \ + "quarantine write failure lost the pre-armed provider scope" + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$ROOT" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + FM_PROJECTS_OVERRIDE="$TMP_ROOT/unused-projects" FM_SPAWN_NO_GUARD=1 \ + "$ROOT/bin/fm-spawn.sh" "$id" "$proj" claude --backend orca 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "spawn reused a task after quarantine publication failed" + assert_contains "$out" "Orca cleanup is pending for $id" \ + "pre-armed quarantine did not block retry" + pass "Orca quarantine publication failures retain a durable retry blocker" +} + +test_teardown_rejects_cross_task_orca_terminal_label() { + local proj wt data state config id out status neutral + id="orcacrosstaskz4" + proj="$TMP_ROOT/cross-task-project" + wt="$TMP_ROOT/cross-task-wt" + data="$TMP_ROOT/cross-task-data" + state="$TMP_ROOT/cross-task-state" + config="$TMP_ROOT/cross-task-config" + fm_git_worktree "$proj" "$wt" "fm/$id" + mkdir -p "$data/$id" "$state" "$config" + printf 'report\n' > "$data/$id/report.md" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "terminal=term-cross-task" "worktree=$wt" "project=$proj" \ + "harness=claude" "kind=scout" "mode=no-mistakes" "backend=orca" \ + "orca_worktree_id=wt-cross-task" + orca_case cross-task-terminal + printf '{"ok":true,"result":{"worktree":{"id":"wt-cross-task","path":"%s"}}}\n' "$wt" > "$RESP/1.out" + printf 'fm-other-task\n' > "$RESP/.terminal-title" + printf 'wt-cross-task\n' > "$RESP/.terminal-worktree-override" + neutral=$(neutral_fm_root "$CASE_DIR/neutral") + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + "$ROOT/bin/fm-teardown.sh" "$id" 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "teardown accepted a terminal labeled for another task" + assert_contains "$out" "task Orca endpoint authority or quiescence is unproven" \ + "cross-task Orca terminal drift was not surfaced" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "cross-task Orca terminal drift closed another task" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "cross-task Orca terminal drift removed the provider worktree" + pass "Orca teardown binds terminals to the expected task label" +} + +test_teardown_rejects_cross_task_orca_worktree_label() { + local proj wt data state config id out status neutral + id="orcaworktreetaskz7" + proj="$TMP_ROOT/worktree-task-project" + wt="$TMP_ROOT/worktree-task-wt" + data="$TMP_ROOT/worktree-task-data" + state="$TMP_ROOT/worktree-task-state" + config="$TMP_ROOT/worktree-task-config" + fm_git_worktree "$proj" "$wt" "fm/$id" + mkdir -p "$data/$id" "$state" "$config" + printf 'report\n' > "$data/$id/report.md" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "terminal=term-worktree-task" "worktree=$wt" "project=$proj" \ + "harness=claude" "kind=scout" "mode=no-mistakes" "backend=orca" \ + "orca_worktree_id=wt-worktree-task" + orca_case cross-task-worktree + printf '{"ok":true,"result":{"worktree":{"id":"wt-worktree-task","name":"fm-other-task","path":"%s","terminals":[{"handle":"term-worktree-task","title":"fm-%s"}]}}}\n' \ + "$wt" "$id" > "$RESP/1.out" + printf "fm-%s\n" "$id" > "$RESP/.terminal-title" + printf 'wt-worktree-task\n' > "$RESP/.terminal-worktree-override" + neutral=$(neutral_fm_root "$CASE_DIR/neutral") + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + "$ROOT/bin/fm-teardown.sh" "$id" 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "teardown accepted an Orca worktree labeled for another task" + assert_contains "$out" "task Orca endpoint authority or quiescence is unproven" \ + "cross-task Orca worktree drift was not surfaced" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "cross-task Orca worktree drift closed a terminal" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "cross-task Orca worktree drift removed the provider worktree" + pass "Orca teardown binds worktrees to the expected task label" +} + +test_teardown_quiesces_unrecorded_orca_terminals() { + local proj wt data state config id out status neutral + id="orcaunrecordedz3" + proj="$TMP_ROOT/unrecorded-project" + wt="$TMP_ROOT/unrecorded-wt" + data="$TMP_ROOT/unrecorded-data" + state="$TMP_ROOT/unrecorded-state" + config="$TMP_ROOT/unrecorded-config" + fm_git_worktree "$proj" "$wt" "fm/$id" + mkdir -p "$data/$id" "$state" "$config" + printf 'report\n' > "$data/$id/report.md" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "terminal=term-recorded-stale" "worktree=$wt" "project=$proj" \ + "harness=claude" "kind=scout" "mode=no-mistakes" "backend=orca" \ + "orca_worktree_id=wt-unrecorded" + orca_case unrecorded-terminal + printf '{"ok":true,"result":{"worktree":{"id":"wt-unrecorded","path":"%s","terminals":[{"handle":"term-unrecorded","title":"fm-%s"}]}}}\n' \ + "$wt" "$id" > "$RESP/1.out" + printf '{"ok":false,"error":{"code":"terminal_handle_stale","message":"terminal handle stale"}}\n' > "$RESP/2.out" + printf '1\n' > "$RESP/2.exit" + printf '{"ok":false,"error":{"code":"terminal_handle_stale","message":"terminal handle stale"}}\n' > "$RESP/4.out" + printf '1\n' > "$RESP/4.exit" + printf "fm-%s\n" "$id" > "$RESP/.terminal-title" + printf 'wt-unrecorded\n' > "$RESP/.terminal-worktree-override" + neutral=$(neutral_fm_root "$CASE_DIR/neutral") + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + "$ROOT/bin/fm-teardown.sh" "$id" --force 2>&1) + status=$? + set -e + + expect_code 0 "$status" "unrecorded Orca terminal cleanup"$'\n'"$out" + assert_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close'$'\x1f''--terminal'$'\x1f''term-unrecorded' \ + "teardown did not quiesce the unrecorded terminal" + assert_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm'$'\x1f''--worktree'$'\x1f''id:wt-unrecorded' \ + "teardown did not remove the terminal-free worktree" + pass "Orca teardown quiesces every terminal attached to the worktree" +} + +test_teardown_rejects_live_recorded_terminal_missing_from_inventory() { + local proj wt data state config id out status neutral + id="orcaomittedterminalz8" + proj="$TMP_ROOT/omitted-terminal-project" + wt="$TMP_ROOT/omitted-terminal-wt" + data="$TMP_ROOT/omitted-terminal-data" + state="$TMP_ROOT/omitted-terminal-state" + config="$TMP_ROOT/omitted-terminal-config" + fm_git_worktree "$proj" "$wt" "fm/$id" + mkdir -p "$data/$id" "$state" "$config" + printf 'report\n' > "$data/$id/report.md" + fm_write_meta "$state/$id.meta" \ + "window=fm-$id" "terminal=term-omitted" "worktree=$wt" "project=$proj" \ + "harness=claude" "kind=scout" "mode=no-mistakes" "backend=orca" \ + "orca_worktree_id=wt-omitted" + orca_case omitted-live-terminal + printf '{"ok":true,"result":{"worktree":{"id":"wt-omitted","name":"fm-%s","path":"%s","terminals":[]}}}\n' \ + "$id" "$wt" > "$RESP/1.out" + printf "fm-%s\n" "$id" > "$RESP/.terminal-title" + printf 'wt-omitted\n' > "$RESP/.terminal-worktree-override" + neutral=$(neutral_fm_root "$CASE_DIR/neutral") + + set +e + out=$(PATH="$FB:$PATH" FM_ORCA_LOG="$LOG" FM_ORCA_RESPONSES="$RESP" \ + FM_ROOT_OVERRIDE="$neutral" FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ + "$ROOT/bin/fm-teardown.sh" "$id" 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "teardown accepted a live recorded terminal omitted from the worktree inventory" + assert_contains "$out" "task Orca endpoint authority or quiescence is unproven" \ + "recorded-terminal inventory disagreement was not surfaced" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''terminal'$'\x1f''close' \ + "inventory disagreement closed the recorded terminal" + assert_not_contains "$(cat "$LOG")" $'orca\x1f''worktree'$'\x1f''rm' \ + "inventory disagreement removed the provider worktree" + assert_present "$state/$id.meta" "inventory disagreement removed task metadata" + pass "Orca teardown rejects a live recorded terminal omitted from inventory" +} + +if [ "${FM_TEST_FOCUSED:-}" = review-round-orca-quiescence ]; then + test_kill_propagates_close_failure + test_terminal_state_classifies_closed_live_and_ambiguous_orca + test_scout_teardown_removes_orca_worktree_via_helper + test_teardown_refuses_orca_worktree_without_terminal_handle + test_secondmate_force_teardown_retains_partial_orca_child + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-orca-final ]; then + test_worktree_create_retains_partial_authority_when_path_missing + test_worktree_create_never_cleans_partial_response_inline + test_spawn_refuses_malformed_legacy_orca_report_metadata + test_spawn_retains_orca_worktree_when_abort_close_fails + test_teardown_rejects_symlinked_orca_task_metadata + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-orca-quarantine ]; then + test_spawn_preserves_orca_metadata_when_pathless_worktree_cleanup_fails + test_spawn_quarantines_orca_worktree_when_terminal_create_fails + test_spawn_refuses_cleanup_pending_orca_task_before_mutation + test_pathless_orca_quarantine_has_supported_cleanup + test_teardown_refuses_orca_terminal_worktree_identity_drift + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-orca-authority ]; then + test_spawn_refuses_orca_without_verified_authority_capabilities + test_worktree_create_retains_partial_authority_when_path_missing + test_spawn_refuses_orca_nonisolated_worktree + test_spawn_quarantines_unrelated_orca_worktree + test_spawn_quarantines_unbound_orca_terminal + test_spawn_quarantines_create_response_without_worktree_id + test_orca_quarantine_write_failure_keeps_prearmed_blocker + test_idless_orca_quarantine_refuses_unscoped_terminal_close + test_teardown_rejects_cross_task_orca_terminal_label + test_teardown_rejects_cross_task_orca_worktree_label + test_teardown_quiesces_unrecorded_orca_terminals + test_teardown_rejects_live_recorded_terminal_missing_from_inventory + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-13-safety ]; then + test_ship_teardown_rejects_orca_mounted_removal_root + test_remove_worktree_requires_bound_provider_capability + exit 0 +fi + test_capture_reads_terminal_tail_json test_capture_falls_back_to_text_fields test_capture_fails_on_orca_error_json @@ -1576,41 +2512,61 @@ test_send_helpers_reject_orca_error_json test_send_key_enter_and_interrupt test_send_key_refuses_unknown_key test_send_key_refuses_escape_until_supported -test_kill_is_best_effort_close +test_kill_propagates_close_failure +test_terminal_state_classifies_closed_live_and_ambiguous_orca test_remove_worktree_refuses_empty_id test_remove_worktree_rejects_orca_error_json +test_remove_worktree_requires_bound_provider_capability test_worktree_path_resolves_id test_dispatcher_sources_orca_and_routes_primitives +test_spawn_refuses_cleanup_pending_orca_task_before_mutation +test_pathless_orca_quarantine_has_supported_cleanup +test_teardown_refuses_orca_terminal_worktree_identity_drift test_json_get_ignores_undocumented_terminal_id_shapes test_worktree_and_terminal_helpers_parse_json -test_worktree_create_removes_worktree_when_path_missing +test_worktree_create_retains_partial_authority_when_path_missing +test_worktree_create_never_cleans_partial_response_inline test_spawn_preserves_orca_metadata_when_pathless_worktree_cleanup_fails test_legacy_respawn_writes_orca_metadata_and_launches_harness test_spawn_refuses_new_report_required_orca_task_before_mutation test_spawn_refuses_orca_respawn_of_report_required_task +test_spawn_refuses_malformed_legacy_orca_report_metadata test_spawn_refuses_report_required_orca_batch_pair_before_mutation test_report_required_orca_refusal_preserves_competing_lifecycle_state test_report_required_orca_recovery_preserves_inherited_lifecycle_state test_spawn_refuses_orca_secondmate_before_home_mutation test_spawn_refuses_orca_when_runtime_not_ready +test_spawn_refuses_orca_without_verified_authority_capabilities test_spawn_refuses_orca_nonisolated_worktree -test_spawn_removes_orca_worktree_when_terminal_create_fails +test_spawn_quarantines_unrelated_orca_worktree +test_spawn_quarantines_unbound_orca_terminal +test_spawn_quarantines_orca_worktree_when_terminal_create_fails test_spawn_preserves_orca_metadata_when_abort_cleanup_fails +test_spawn_retains_orca_worktree_when_abort_close_fails test_spawn_refuses_invalid_state_before_orca_resource_creation test_peek_send_and_crew_state_route_through_orca_meta test_peek_and_crew_state_fail_closed_on_orca_error_json test_target_exists_rejects_orca_error_json test_scout_teardown_removes_orca_worktree_via_helper test_scout_teardown_refuses_orca_id_path_mismatch -test_teardown_removes_orca_worktree_when_path_missing +test_teardown_refuses_orca_worktree_when_path_missing test_teardown_preserves_metadata_when_orca_remove_error_json test_scout_teardown_refuses_orca_missing_report_when_path_missing test_ship_teardown_refuses_orca_missing_worktree_path test_ship_teardown_removes_orca_worktree_when_id_path_matches +test_ship_teardown_rejects_orca_mounted_removal_root test_ship_teardown_refuses_orca_unresolvable_worktree_id test_ship_teardown_refuses_orca_id_path_mismatch test_teardown_refuses_orca_missing_worktree_id -test_teardown_removes_orca_worktree_without_terminal_handle +test_teardown_refuses_orca_worktree_without_terminal_handle +test_teardown_rejects_symlinked_orca_task_metadata test_secondmate_force_teardown_removes_orca_child_via_orca test_secondmate_force_teardown_refuses_orca_child_id_path_mismatch -test_secondmate_force_teardown_removes_partial_orca_child +test_secondmate_force_teardown_retains_partial_orca_child +test_spawn_quarantines_create_response_without_worktree_id +test_orca_quarantine_write_failure_keeps_prearmed_blocker +test_idless_orca_quarantine_refuses_unscoped_terminal_close +test_teardown_rejects_cross_task_orca_terminal_label +test_teardown_rejects_cross_task_orca_worktree_label +test_teardown_quiesces_unrecorded_orca_terminals +test_teardown_rejects_live_recorded_terminal_missing_from_inventory diff --git a/tests/fm-backend.test.sh b/tests/fm-backend.test.sh index ba129f24235..a09459b60c2 100755 --- a/tests/fm-backend.test.sh +++ b/tests/fm-backend.test.sh @@ -25,6 +25,8 @@ # The real tmux smoke test (create session, send text + Enter, capture, list, # kill) lives in tests/fm-backend-tmux-smoke.test.sh. set -u +export FM_ORCA_TEST_LAB=firstmate-orca-test-lab-v1 +export FM_ORCA_TEST_AUTHORITY_CAPABILITIES=verified-v1 # shellcheck source=tests/lib.sh . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" @@ -123,7 +125,7 @@ BASE_REF=$(resolve_base_ref) \ # tmux-only conformance run the tmux adapter's behavior is what is under test, # and that is unchanged by any later (e.g. non-tmux backend) addition to # fm-backend.sh's own dispatch surface. -OLD_BIN_UNCHANGED_SIBLINGS="fm-gate-refuse-lib.sh fm-guard.sh fm-lock-lib.sh fm-tangle-lib.sh fm-tmux-lib.sh fm-composer-lib.sh fm-marker-lib.sh fm-wake-lib.sh fm-classify-lib.sh fm-transition-lib.sh fm-ff-lib.sh fm-config-inherit-lib.sh fm-account-routing-lib.sh fm-report-contract-lib.sh fm-tasks-axi-lib.sh fm-project-mode.sh fm-harness.sh fm-crew-state.sh fm-backend.sh" +OLD_BIN_UNCHANGED_SIBLINGS="fm-gate-refuse-lib.sh fm-guard.sh fm-lock-lib.sh fm-tangle-lib.sh fm-tmux-lib.sh fm-composer-lib.sh fm-marker-lib.sh fm-wake-lib.sh fm-supervision-lib.sh fm-classify-lib.sh fm-transition-lib.sh fm-ff-lib.sh fm-config-inherit-lib.sh fm-process-tree-lib.sh fm-checkout-lock-lib.sh fm-account-routing-lib.sh fm-report-contract-lib.sh fm-tasks-axi-lib.sh fm-project-mode.sh fm-harness.sh fm-crew-state.sh fm-backend.sh" OLD_BIN_REFACTORED="fm-send.sh fm-peek.sh fm-watch.sh fm-spawn.sh fm-teardown.sh" build_old_bin() { # <name> -> echoes root dir (root/bin/<script> is the entry point) @@ -186,14 +188,16 @@ test_resolve_base_ref_refuses_unverified_merge_parent() { pass "resolve_base_ref: unverified topic merges do not supply a baseline" } -test_herdr_required_tools_include_detached_launcher_dependencies() { +test_herdr_required_tools_include_backend_specific_launcher_dependencies() { local required tool required=$(fm_backend_required_tools herdr) || fail "Herdr should have a required-tool registry entry" - for tool in herdr jq nohup perl treehouse; do + for tool in herdr jq nohup treehouse; do fm_backend_list_contains "$required" "$tool" \ || fail "Herdr required tools should include $tool" done - pass "fm_backend_required_tools: Herdr includes detached launcher dependencies" + fm_backend_list_contains "$required" perl \ + && fail "Herdr backend delta duplicated the universal Perl dependency" + pass "fm_backend_required_tools: Herdr owns only its backend-specific launcher delta" } test_backend_name_precedence() { @@ -1104,13 +1108,46 @@ make_teardown_fakebin() { # <dir> -> echoes fakebin dir; logs tmux+treehouse ca cat > "$fb/tmux" <<'SH' #!/usr/bin/env bash set -u -{ printf 'tmux'; for a in "$@"; do printf '\x1f%s' "$a"; done; printf '\n'; } >> "${FM_TMUX_LOG:?}" -exit 0 +live="${FM_TMUX_LOG:?}.live" +target= +previous= +for argument in "$@"; do + [ "$previous" != -t ] || target=$argument + previous=$argument +done +case "${1:-}" in + display-message) + [ -e "$live" ] || exit 1 + case "${*: -1}" in + *session_name*window_name*) printf 'firstmate\t%s\n' "${target#*:}" ;; + *pane_current_command*) printf 'bash\n' ;; + *pane_id*) printf '%%1\n' ;; + esac + ;; + list-windows) + [ -e "$live" ] || { + printf "can't find session: firstmate\n" >&2 + exit 1 + } + printf '%s\n' "${target#*:}" + ;; + has-session) [ -e "$live" ] ;; + kill-window) + { printf 'tmux'; for argument in "$@"; do printf '\x1f%s' "$argument"; done; printf '\n'; } \ + >> "$FM_TMUX_LOG" + rm -f "$live" + ;; +esac SH cat > "$fb/treehouse" <<'SH' #!/usr/bin/env bash set -u -{ printf 'treehouse'; for a in "$@"; do printf '\x1f%s' "$a"; done; printf '\n'; } >> "${FM_TMUX_LOG:?}" +target=${3:-} +case "$target" in + .|/dev/fd/*) target=$(cd "$target" && pwd -P) || exit 1 ;; +esac +printf 'treehouse\x1f%s\x1f%s\x1f%s\x1fCONTEXT=%s\n' \ + "${1:-}" "${2:-}" "$target" "${FM_TREEHOUSE_RETURN_PROJECT:-}" >> "${FM_TMUX_LOG:?}" exit 0 SH chmod +x "$fb/tmux" "$fb/treehouse" @@ -1126,6 +1163,7 @@ SH run_teardown_case() { local script=$1 fmroot=$2 fb=$3 log=$4 state=$5 data=$6 config=$7 id=$8 : > "$log" + : > "$log.live" env PATH="$fb:$PATH" FM_ROOT_OVERRIDE="$fmroot" \ FM_STATE_OVERRIDE="$state" FM_DATA_OVERRIDE="$data" FM_CONFIG_OVERRIDE="$config" \ FM_TMUX_LOG="$log" \ @@ -1136,9 +1174,12 @@ test_teardown_conformance_old_vs_new() { local old_bin fb proj wt id local state_old state_new config_old config_new data log_old log_new out_old out_new rc_old rc_new old_bin=$(build_old_bin teardown-old) - proj="$TMP_ROOT/teardown-project"; wt="$TMP_ROOT/teardown-wt" + proj="$TMP_ROOT/teardown-project"; wt="$TMP_ROOT/teardown-pool/1/teardown-wt" id="teardownconform1" fm_git_worktree "$proj" "$wt" "fm/$id" + cat > "$TMP_ROOT/teardown-pool/treehouse-state.json" <<EOF +{"worktrees":[{"name":"1","path":"$wt","leased":true,"lease_holder":"firstmate-$id"}]} +EOF fb=$(make_teardown_fakebin "$TMP_ROOT/teardown-fake") data="$TMP_ROOT/teardown-data" @@ -1163,14 +1204,19 @@ test_teardown_conformance_old_vs_new() { expect_code 0 "$rc_old" "old fm-teardown.sh (scout, report present) should succeed"$'\n'"$out_old" expect_code 0 "$rc_new" "new fm-teardown.sh (scout, report present) should succeed"$'\n'"$out_new" - diff -u "$log_old" "$log_new" > "$TMP_ROOT/teardown-diff.txt" 2>&1 \ - || fail "fm-teardown.sh: tmux+treehouse command log differs old vs new"$'\n'"$(cat "$TMP_ROOT/teardown-diff.txt")" + grep '^tmux' "$log_old" > "$TMP_ROOT/teardown-old-tmux.log" + grep '^tmux' "$log_new" > "$TMP_ROOT/teardown-new-tmux.log" + diff -u "$TMP_ROOT/teardown-old-tmux.log" "$TMP_ROOT/teardown-new-tmux.log" \ + > "$TMP_ROOT/teardown-diff.txt" 2>&1 \ + || fail "fm-teardown.sh: tmux command log differs old vs new"$'\n'"$(cat "$TMP_ROOT/teardown-diff.txt")" assert_contains "$(cat "$log_new")" "treehouse"$'\x1f''return'$'\x1f''--force'$'\x1f'"$wt" \ "teardown did not call treehouse return --force <worktree>" + assert_contains "$(cat "$log_new")" $'\x1f'"CONTEXT=$proj" \ + "teardown did not preserve Treehouse project context" assert_contains "$(cat "$log_new")" "tmux"$'\x1f''kill-window'$'\x1f''-t'$'\x1f'"firstmate:fm-$id" \ "teardown did not call tmux kill-window -t <window>" - pass "fm-teardown.sh: treehouse return + tmux kill-window command log is byte-identical old vs new for a scout task" + pass "fm-teardown.sh: Treehouse return preserves project context and tmux teardown compatibility" } # --- backend selection loudly refuses an unknown backend -------------------- @@ -1292,6 +1338,11 @@ if [ "${FM_TEST_FOCUSED:-}" = review-round-27 ]; then exit 0 fi +if [ "${FM_TEST_FOCUSED:-}" = review-round-13-safety ]; then + test_teardown_conformance_old_vs_new + exit 0 +fi + if [ "${FM_TEST_FOCUSED:-}" = review-findings ]; then test_meta_get_and_backend_of_meta exit 0 @@ -1319,7 +1370,7 @@ fi if [ "${FM_TEST_FOCUSED:-}" = review-round-34 ]; then test_resolve_base_ref_uses_single_parent_when_main_is_head - test_herdr_required_tools_include_detached_launcher_dependencies + test_herdr_required_tools_include_backend_specific_launcher_dependencies exit 0 fi @@ -1338,7 +1389,7 @@ fi test_resolve_base_ref_uses_single_parent_when_main_is_head test_resolve_base_ref_refuses_unverified_merge_parent -test_herdr_required_tools_include_detached_launcher_dependencies +test_herdr_required_tools_include_backend_specific_launcher_dependencies test_backend_name_precedence test_backend_detect_precedence test_backend_detect_cmux_fallback_bundle_id diff --git a/tests/fm-bootstrap.test.sh b/tests/fm-bootstrap.test.sh index 72c3912839d..773734b4810 100755 --- a/tests/fm-bootstrap.test.sh +++ b/tests/fm-bootstrap.test.sh @@ -15,6 +15,8 @@ # override, blank-env defaulting, partial-output relay, and pre-launch timeout # scan. set -u +export FM_ORCA_TEST_LAB=firstmate-orca-test-lab-v1 +export FM_ORCA_TEST_AUTHORITY_CAPABILITIES=verified-v1 # shellcheck source=tests/lib.sh disable=SC1091 . "$(dirname "${BASH_SOURCE[0]}")/lib.sh" @@ -24,12 +26,13 @@ TMP_ROOT=$(fm_test_tmproot fm-bootstrap-tests) export FM_BACKEND_CMUX_BUNDLE_BIN="$TMP_ROOT/no-bundled-cmux" # Hermetic runtime-backend detection. These cases pin the backend per-home via -# config/backend; the dev shell's ambient runtime markers ($TMUX inside tmux, -# HERDR_ENV inside herdr, CMUX_* inside a cmux terminal) must not leak into -# fm_backend_name and flip a default-backend case onto a non-tmux backend. Unset -# them once so the suite resolves the tmux reference backend unless a case says -# otherwise - the same hermeticity discipline as pinning PATH via BASE_PATH. -unset TMUX TMUX_PANE HERDR_ENV HERDR_PANE_ID HERDR_SESSION HERDR_SOCKET_PATH \ +# config/backend; the dev shell's explicit FM_BACKEND and ambient runtime +# markers ($TMUX inside tmux, HERDR_ENV inside herdr, CMUX_* inside a cmux +# terminal) must not leak into fm_backend_name and override a case. Unset them +# once so the suite resolves the configured backend, or the tmux reference +# backend when no case says otherwise - the same hermeticity discipline as +# pinning PATH via BASE_PATH. +unset FM_BACKEND TMUX TMUX_PANE HERDR_ENV HERDR_PANE_ID HERDR_SESSION HERDR_SOCKET_PATH \ CMUX_WORKSPACE_ID CMUX_SURFACE_ID CMUX_SOCKET_PATH CMUX_TAB_ID CMUX_PANEL_ID 2>/dev/null || true # A fake toolchain where every required tool is present and gh is authenticated. @@ -452,7 +455,7 @@ test_herdr_install_requires_manual_action() { test_herdr_detach_dependencies_have_manual_guidance() { local tool case_dir fakebin bash_env out install_out status instructions - instructions="https://github.com/ruby-dlee/firstmate/blob/main/docs/configuration.md#herdr-detached-launcher-prerequisites" + instructions="https://github.com/ruby-dlee/firstmate/blob/main/docs/configuration.md#portable-process-control-prerequisites" for tool in nohup perl; do case_dir="$TMP_ROOT/herdr-missing-$tool" mkdir -p "$case_dir/home/config" @@ -486,6 +489,29 @@ SH pass "bootstrap: Herdr detach dependencies have actionable manual guidance" } +test_perl_is_a_universal_process_control_dependency() { + local case_dir fakebin bash_env out + case_dir="$TMP_ROOT/universal-perl" + mkdir -p "$case_dir/home/config" + printf '%s\n' manual > "$case_dir/home/config/backlog-backend" + fakebin=$(make_fake_toolchain "$case_dir") + bash_env="$case_dir/no-perl.bash" + cat > "$bash_env" <<'SH' +command() { + if [ "${1:-}" = -v ] && [ "${2:-}" = perl ]; then + return 1 + fi + builtin command "$@" +} +SH + out=$(PATH="$fakebin:$BASE_PATH" BASH_ENV="$bash_env" \ + FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_contains "$out" 'MISSING_MANUAL: perl' \ + "a non-Herdr home did not validate the checkout refresher's Perl dependency" + pass "bootstrap validates Perl for every backend" +} + test_cmux_bundled_cli_satisfies_dependency() { local case_dir fakebin bundle out case_dir="$TMP_ROOT/cmux-bundled-cli" @@ -764,8 +790,9 @@ SH out=$(PATH="$fakebin:$BASE_PATH" BASH_ENV="$bash_env" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") - assert_contains "$out" 'MISSING_MANUAL: agent-fleet (instructions: https://github.com/ruby-dlee/firstmate/blob/main/docs/configuration.md#agent-fleet-account-routing)' "enforce mode did not report manual Agent Fleet installation" + assert_contains "$out" 'MISSING_MANUAL: agent-fleet' "enforce mode did not retain the secondmate Agent Fleet dependency" assert_contains "$out" 'MISSING: jq (install: brew install jq # or the platform' "enforce mode did not report missing jq" + assert_contains "$out" 'MISSING_MANUAL: herdr' "enforce mode did not report the direct hook installer" case_dir="$TMP_ROOT/account-routing-dispatch" mkdir -p "$case_dir/home/config" @@ -777,7 +804,7 @@ SH out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") - assert_contains "$out" 'MISSING_MANUAL: agent-fleet (instructions: https://github.com/ruby-dlee/firstmate/blob/main/docs/configuration.md#agent-fleet-account-routing)' "account-routed dispatch profile did not report manual Agent Fleet installation" + assert_not_contains "$out" 'MISSING_MANUAL: agent-fleet' "account-routed dispatch profile still required Agent Fleet" assert_contains "$out" 'CREW_DISPATCH: active config/crew-dispatch.json' "account dependency preflight suppressed dispatch validation" case_dir="$TMP_ROOT/account-routing-observe" @@ -785,12 +812,75 @@ SH printf '%s\n' manual > "$case_dir/home/config/backlog-backend" printf '%s\n' observe > "$case_dir/home/config/account-routing-mode" fakebin=$(make_fake_toolchain "$case_dir") + rm -f "$fakebin/herdr" rm -f "$fakebin/agent-fleet" + bash_env="$case_dir/no-jq.bash" + cat > "$bash_env" <<'SH' +command() { + if [ "${1:-}" = -v ] && [ "${2:-}" = jq ]; then + return 1 + fi + builtin command "$@" +} +jq() { + return 127 +} +SH out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + BASH_ENV="$bash_env" FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_not_contains "$out" 'MISSING_MANUAL: agent-fleet' "observe mode treated Agent Fleet as a new-launch dependency" + assert_contains "$out" 'MISSING: jq' "observe mode did not report the direct selector dependency" + assert_contains "$out" 'MISSING_MANUAL: herdr' "observe mode did not report the direct hook installer" + case_dir="$TMP_ROOT/account-routing-legacy-recovery" + mkdir -p "$case_dir/home/config" "$case_dir/home/state" + printf '%s\n' manual > "$case_dir/home/config/backlog-backend" + printf '%s\n' 'account_profile=codex-1' > "$case_dir/home/state/legacy.meta" + fakebin=$(make_fake_toolchain "$case_dir") + rm -f "$fakebin/agent-fleet" + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") - assert_not_contains "$out" 'MISSING: agent-fleet' "observe-only mode treated optional Agent Fleet as required" - assert_not_contains "$out" 'MISSING: jq' "observe-only mode treated advisory JSON parsing as required" - pass "bootstrap reports dependencies only when configuration can enforce routing" + assert_contains "$out" 'MISSING_MANUAL: agent-fleet' "legacy managed metadata no longer reported its recovery dependency" + + case_dir="$TMP_ROOT/account-routing-pending-rollback" + mkdir -p "$case_dir/home/config" "$case_dir/home/state" + printf '%s\n' manual > "$case_dir/home/config/backlog-backend" + printf '%s\n' 'account_rollback_cleanup=pending' > "$case_dir/home/state/legacy.meta" + fakebin=$(make_fake_toolchain "$case_dir") + rm -f "$fakebin/agent-fleet" + out=$(PATH="$fakebin:$BASE_PATH" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ + FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_contains "$out" 'MISSING_MANUAL: agent-fleet' "pending rollback metadata no longer reported its cleanup dependency" + + case_dir="$TMP_ROOT/account-routing-direct-metadata" + mkdir -p "$case_dir/home/config" "$case_dir/home/state" + printf '%s\n' manual > "$case_dir/home/config/backlog-backend" + printf '%s\n' off > "$case_dir/home/config/account-routing-mode" + printf '%s\n' 'account_home=/accounts/codex/1' > "$case_dir/home/state/direct.meta" + fakebin=$(make_fake_toolchain "$case_dir") + rm -f "$fakebin/herdr" + bash_env="$case_dir/no-jq.bash" + cat > "$bash_env" <<'SH' +command() { + if [ "${1:-}" = -v ] && [ "${2:-}" = jq ]; then + return 1 + fi + builtin command "$@" +} +jq() { + return 127 +} +SH + out=$(PATH="$fakebin:$BASE_PATH" BASH_ENV="$bash_env" FM_HOME="$case_dir/home" FM_ROOT_OVERRIDE="$case_dir/home" \ + FM_ACCOUNT_DIRECTORY_TEST_LAB=firstmate-account-directory-test-lab-v1 \ + FM_ACCOUNT_DIRECTORY_PERL_BIN="$case_dir/missing-perl" \ + FM_FAKE_TREEHOUSE_LEASE_HELP=1 "$ROOT/bin/fm-bootstrap.sh") + assert_contains "$out" 'MISSING: jq' "direct metadata did not retain the selector dependency when routing was off" + assert_contains "$out" 'MISSING_MANUAL: herdr' "direct metadata did not retain the hook dependency when routing was off" + assert_contains "$out" 'MISSING_MANUAL: perl' "direct metadata did not preflight the fixed passwd-home resolver" + assert_not_contains "$out" 'MISSING_MANUAL: agent-fleet' "direct metadata incorrectly restored the legacy recovery dependency" + pass "bootstrap requires direct launch tools for crews and Agent Fleet for enforced secondmates or legacy recovery" } test_agent_fleet_install_requires_manual_release() { @@ -874,6 +964,16 @@ if [ "${FM_TEST_FOCUSED:-}" = review-round-35 ]; then exit 0 fi +if [ "${FM_TEST_FOCUSED:-}" = account-directory-cutover ]; then + test_account_routing_dependency_preflight + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-40 ]; then + test_perl_is_a_universal_process_control_dependency + exit 0 +fi + test_bootstrap_reporting test_no_mistakes_min_version test_git_is_required_with_supported_install_instruction @@ -882,6 +982,7 @@ test_session_provider_backends_do_not_require_tmux test_session_provider_backends_gate_own_cli_not_tmux test_herdr_install_requires_manual_action test_herdr_detach_dependencies_have_manual_guidance +test_perl_is_a_universal_process_control_dependency test_cmux_bundled_cli_satisfies_dependency test_unknown_backend_reports_invalid_configuration test_json_backends_require_jq_not_tmux diff --git a/tests/fm-checkout-refresh.test.sh b/tests/fm-checkout-refresh.test.sh new file mode 100755 index 00000000000..693c1e0e3d3 --- /dev/null +++ b/tests/fm-checkout-refresh.test.sh @@ -0,0 +1,2407 @@ +#!/usr/bin/env bash +# Behavior tests for the checkout-refresh discovery, upstream signal, timed +# backstop, independent coverage and scheduler health, untracked skill-draft +# hygiene, safety posture, worktree freshness proof, and LaunchAgent definition. +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" +# shellcheck source=bin/fm-checkout-lock-lib.sh +. "$ROOT/bin/fm-checkout-lock-lib.sh" + +fm_git_identity fmtest fmtest@example.invalid + +TMP_ROOT=$(fm_test_tmproot fm-checkout-refresh-tests) +TEST_HOME="$TMP_ROOT/user" +FM_TEST_HOME="$TMP_ROOT/fm-home" +STATE_ROOT="$TMP_ROOT/refresh-state" +LOCK_ROOT="$TMP_ROOT/refresh-locks" +mkdir -p "$TEST_HOME/.treehouse" "$FM_TEST_HOME/projects" "$FM_TEST_HOME/config" "$STATE_ROOT" + +checkout_state_key() { + local path + path=$(fm_checkout_trusted_dir "$1") || return 1 + fm_checkout_hash_value "$path" "${2:-24}" +} + +checkout_lock_key() { + fm_checkout_stable_path_key "$1" directory 0 "${2:-24}" +} + +commit_file() { + local dir=$1 file=$2 content=$3 message=$4 + printf '%s\n' "$content" > "$dir/$file" + git -C "$dir" add "$file" + git -C "$dir" commit -qm "$message" +} + +build_origin() { + local work="$TMP_ROOT/work-$1" remote="$TMP_ROOT/remotes/$1.git" remote_abs + mkdir -p "$TMP_ROOT/remotes" + git init -q "$work" + git -C "$work" symbolic-ref HEAD refs/heads/main + commit_file "$work" file.txt v0 C0 + git clone --quiet --bare "$work" "$remote" + remote_abs=$(cd "$remote" && pwd -P) + git -C "$work" remote add origin "file://$remote_abs" + git -C "$work" push -q -u origin main + printf '%s\n' "$remote_abs" +} + +clone_from() { + local remote=$1 destination=$2 + git clone --quiet "file://$remote" "$destination" +} + +advance_origin() { + local message=$2 work="$TMP_ROOT/work-$1" + commit_file "$work" file.txt "$message" "$message" + git -C "$work" push -q origin main +} + +switch_origin_default() { + local work="$TMP_ROOT/work-$1" remote="$TMP_ROOT/remotes/$1.git" + git -C "$work" checkout -q -b trunk + commit_file "$work" trunk.txt trunk default-trunk + git -C "$work" push -q origin trunk + git -C "$remote" symbolic-ref HEAD refs/heads/trunk +} + +run_refresh() { + HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" FM_CHECKOUT_REFRESH_LOCK_ROOT="$LOCK_ROOT" \ + FM_TREEHOUSE_ROOT="$TEST_HOME/.treehouse" \ + FM_CHECKOUT_REFRESH_TEST=1 \ + "$ROOT/bin/fm-checkout-refresh.sh" "$@" +} + +run_isolated_refresh() { + local home=$1 state_root=$2 + shift 2 + mkdir -p "$home/user/.treehouse" + HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$state_root" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$state_root-locks" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_TEST=1 \ + "$ROOT/bin/fm-checkout-refresh.sh" "$@" +} + +run_manifest_failure_refresh() { + local failure=$1 + shift + FM_CHECKOUT_TEST_MANIFEST_FAILURE="$failure" run_isolated_refresh "$@" +} + +assert_refresh_state() { + local state_root=$1 expected=$2 coverage_epoch coverage + coverage_epoch=$(sed -n '1p' "$state_root/coverage-health" 2>/dev/null || true) + coverage=$(sed -n '2p' "$state_root/coverage-health" 2>/dev/null || true) + case "$coverage_epoch" in ''|*[!0-9]*) fail "coverage health timestamp is missing" ;; esac + [ "$coverage" = "$expected" ] \ + || fail "expected $expected coverage health, found ${coverage:-missing}" +} + +assert_heartbeat_value() { + local state_root=$1 expected=$2 actual + actual=$(sed -n '1p' "$state_root/heartbeat" 2>/dev/null || true) + [ "$actual" = "$expected" ] \ + || fail "expected heartbeat $expected, found ${actual:-missing}" +} + +assert_head_matches_origin() { + local checkout=$1 + [ "$(git -C "$checkout" rev-parse HEAD)" = "$(git -C "$checkout" rev-parse origin/main)" ] \ + || fail "$checkout did not reach origin/main" +} + +write_launch_agent_fixture() { + local path=$1 label=$2 home=$3 state=$4 generation=${5:-} + { + printf '%s\n' '<?xml version="1.0" encoding="UTF-8"?>' + printf '%s\n' '<plist version="1.0"><dict>' + printf '<key>Label</key><string>%s</string>\n' "$label" + printf '%s\n' '<key>ProgramArguments</key><array>' + printf '%s\n' '<string>/bin/bash</string>' + printf '<string>%s/bin/fm-checkout-refresh.sh</string>\n' "$ROOT" + printf '%s\n' '<string>run-once</string><string>--scheduled</string></array>' + printf '%s\n' '<key>StartInterval</key><integer>900</integer>' + printf '%s\n' '<key>RunAtLoad</key><true/>' + printf '%s\n' '<key>EnvironmentVariables</key><dict>' + printf '<key>FM_HOME</key><string>%s</string>\n' "$(cd "$home" && pwd -P)" + printf '<key>FM_CHECKOUT_REFRESH_STATE_ROOT</key><string>%s</string>\n' "$state" + [ -z "$generation" ] \ + || printf '<key>FM_CHECKOUT_REFRESH_GENERATION</key><string>%s</string>\n' "$generation" + printf '%s\n' '</dict></dict></plist>' + } > "$path" +} + +plist_generation() { + sed -n 's#.*<key>FM_CHECKOUT_REFRESH_GENERATION</key><string>\([^<]*\)</string>.*#\1#p' "$1" +} + +write_stateful_launchctl_fake() { + local path=$1 + cat > "$path" <<'SH' +#!/usr/bin/env bash +state=${FM_FAKE_LAUNCHCTL_STATE:?} +mkdir -p "$state" +[ -z "${FM_FAKE_LAUNCHCTL_LOG:-}" ] || printf '%s\n' "$*" >> "$FM_FAKE_LAUNCHCTL_LOG" +absent() { + echo "Could not find service \"${1:-unknown}\" in domain" >&2 + exit 3 +} +case "${1:-}" in + print) + target=${2:-} + label=${target##*/} + record="$state/$label" + [ -f "$record" ] || absent "$label" + plist=$(cat "$record") + [ -f "$plist" ] || exit 4 + python3 - "$target" "$plist" <<'PY' +import os +import plistlib +import sys + +target, path = sys.argv[1:] +with open(path, "rb") as stream: + definition = plistlib.load(stream) +arguments = definition["ProgramArguments"] +environment = definition["EnvironmentVariables"] +print(f"{target} = {{") +print(f" path = {path}") +print(f" program = {arguments[0]}") +print(" arguments = {") +for argument in arguments: + print(f" {argument}") +print(" }") +print(" inherited environment = {") +inherited = os.environ.get("FM_TEST_LAUNCHCTL_INHERITED_ENV", "") +if inherited: + key, separator, value = inherited.partition("=") + if not separator or not key: + raise SystemExit(5) + print(f" {key} => {value}") +print(" }") +print(" default environment = {") +default = os.environ.get("FM_TEST_LAUNCHCTL_DEFAULT_ENV", "") +if default: + key, separator, value = default.partition("=") + if not separator or not key: + raise SystemExit(5) + print(f" {key} => {value}") +print(" }") +print(" environment = {") +for key, value in environment.items(): + print(f" {key} => {value}") +extra = os.environ.get("FM_TEST_LAUNCHCTL_EXTRA_ENV", "") +if extra: + key, separator, value = extra.partition("=") + if not separator or not key: + raise SystemExit(5) + print(f" {key} => {value}") +print(" }") +interval = os.environ.get( + "FM_TEST_LAUNCHCTL_INTERVAL", + str(definition["StartInterval"]), +) +run_at_load = os.environ.get( + "FM_TEST_LAUNCHCTL_RUN_AT_LOAD", + "true" if definition["RunAtLoad"] else "false", +) +print(f" run interval = {interval} seconds") +print(f" run at load = {run_at_load}") +print("}") +PY + ;; + bootout) + target=${2:-} + label=${target##*/} + [ "${FM_TEST_LAUNCHCTL_BOOTOUT_FAIL_LABEL:-}" != "$label" ] || { + echo "injected bootout failure for $label" >&2 + exit 70 + } + [ -f "$state/$label" ] || absent "$label" + rm -f "$state/$label" + ;; + bootstrap) + plist=${3:-} + label=$(python3 - "$plist" <<'PY' +import plistlib +import sys +with open(sys.argv[1], "rb") as stream: + print(plistlib.load(stream)["Label"]) +PY +) + printf '%s\n' "$plist" > "$state/$label" + ;; + kickstart) + target=${2:-} + label=${target##*/} + [ -f "$state/$label" ] || absent "$label" + plist=$(cat "$state/$label") + if [ -n "${FM_TEST_LOGICAL_STATE:-}" ] \ + && [ -n "${FM_TEST_LOGICAL_PLIST:-}" ] \ + && [ "$plist" = "$FM_TEST_LOGICAL_PLIST" ]; then + generation=$(sed -n 's#.*<key>FM_CHECKOUT_REFRESH_GENERATION</key><string>\([^<]*\)</string>.*#\1#p' "$plist") + now=$(date +%s) + printf '%s\n' "$now" > "$FM_TEST_LOGICAL_STATE/heartbeat" + printf '%s\nhealthy\n' "$now" > "$FM_TEST_LOGICAL_STATE/coverage-health" + printf '%s\n' "$generation" > "$FM_TEST_LOGICAL_STATE/scheduler-generation" + fi + ;; + *) exit 2 ;; +esac +SH + chmod +x "$path" +} + +mark_launch_agent_loaded() { + local state=$1 label=$2 plist=$3 + mkdir -p "$state" + printf '%s\n' "$plist" > "$state/$label" +} + +test_discovery_covers_projects_treehouse_external_and_config() { + local remote project external pool_worktree explicit_remote explicit custom_root scanned out + remote=$(build_origin relvino) + project="$FM_TEST_HOME/projects/relvino" + external="$TEST_HOME/relvino" + clone_from "$remote" "$project" + clone_from "$remote" "$external" + project=$(cd "$project" && pwd -P) + external=$(cd "$external" && pwd -P) + + pool_worktree="$TEST_HOME/.treehouse/relvino-test/1/relvino" + mkdir -p "$(dirname "$pool_worktree")" + git -C "$project" worktree add --quiet --detach "$pool_worktree" main + printf '{"worktrees":[{"name":"1","path":"%s"}]}\n' "$pool_worktree" \ + > "$TEST_HOME/.treehouse/relvino-test/treehouse-state.json" + + explicit_remote=$(build_origin explicit) + explicit="$TMP_ROOT/explicit-checkout" + clone_from "$explicit_remote" "$explicit" + custom_root="$TMP_ROOT/custom-scan" + scanned="$custom_root/relvino-copy" + mkdir -p "$custom_root" + clone_from "$remote" "$scanned" + explicit=$(cd "$explicit" && pwd -P) + scanned=$(cd "$scanned" && pwd -P) + { + printf 'path %s\n' "$explicit" + printf 'scan %s\n' "$custom_root" + } > "$FM_TEST_HOME/config/checkout-refresh" + + out=$(run_refresh discover) + + assert_contains "$out" "$project" "projects/ checkout was not discovered" + assert_contains "$out" "$external" "matching-origin top-level clone was not discovered" + assert_contains "$out" "$explicit" "configured checkout path was not discovered" + assert_contains "$out" "$scanned" "configured shallow scan root was not discovered" + assert_not_contains "$out" "$pool_worktree" "Treehouse pool worktree was treated as a mutable backing checkout" + [ "$(printf '%s\n' "$out" | grep -Fxc "$project")" -eq 1 ] \ + || fail "Treehouse backing checkout was not deduplicated with projects/ checkout" + pass "discovery covers projects, Treehouse backing checkouts, matching-origin clones, and config" +} + +test_uninspectable_active_project_invalidates_coverage_health() { + local project out status + project="$FM_TEST_HOME/projects/relvino" + chmod 000 "$project" + printf '%s\n' preserved-project-heartbeat > "$STATE_ROOT/heartbeat" + + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + chmod 700 "$project" + + [ "$status" -ne 0 ] || fail "uninspectable active-home project reported healthy coverage" + assert_contains "$out" "incomplete active-home project coverage at $project" \ + "uninspectable active-home project was not surfaced" + assert_refresh_state "$STATE_ROOT" unhealthy + pass "uninspectable active-home projects invalidate coverage health" +} + +test_nested_active_project_invalidates_coverage_health() { + local container projects nested nested_state out status + container="$TMP_ROOT/active-project-container" + fm_git_init_commit "$container" + projects="$container/projects" + nested="$projects/nested-directory" + nested_state="$TMP_ROOT/nested-active-state" + mkdir -p "$nested" "$nested_state" + printf '%s\n' preserved-nested-heartbeat > "$nested_state/heartbeat" + + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_PROJECTS_OVERRIDE="$projects" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$nested_state" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$TMP_ROOT/nested-active-locks" \ + FM_TREEHOUSE_ROOT="$TMP_ROOT/nested-active-treehouse" \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --force 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "nested non-repository active project reported healthy coverage" + assert_contains "$out" "active-home project is not an exact inspectable Git repository root: $nested" \ + "nested non-repository active project was not surfaced" + assert_refresh_state "$nested_state" unhealthy + pass "active projects must be exact canonical Git repository roots" +} + +test_discovery_rejects_nested_configured_and_scanned_paths() { + local remote seed outer configured_child scan_root scanned_child scanned_canonical out err + remote=$(build_origin exact-discovery) + seed="$FM_TEST_HOME/projects/exact-discovery" + outer="$TMP_ROOT/exact-discovery-outer" + configured_child="$outer/configured-child" + scan_root="$outer/scan-root" + scanned_child="$scan_root/scanned-child" + out="$TMP_ROOT/exact-discovery.out" + err="$TMP_ROOT/exact-discovery.err" + clone_from "$remote" "$seed" + clone_from "$remote" "$outer" + mkdir -p "$configured_child" "$scanned_child" + scanned_canonical=$(cd "$scanned_child" && pwd -P) + { + printf 'path %s\n' "$configured_child" + printf 'scan %s\n' "$scan_root" + } > "$FM_TEST_HOME/config/checkout-refresh" + + run_refresh discover > "$out" 2> "$err" \ + || fail "exact-root discovery fixture failed" + + assert_no_grep "^$configured_child$" "$out" \ + "configured nested directory was emitted as a checkout" + assert_no_grep "^$scanned_canonical$" "$out" \ + "scanned nested directory was emitted as a clone" + assert_grep "configured checkout is not an exact inspectable Git repository root: $configured_child" \ + "$err" "configured nested directory was not surfaced" + assert_grep "discovered clone is not an exact inspectable Git repository root: $scanned_canonical" \ + "$err" "scanned nested directory was not surfaced" + rm -f "$FM_TEST_HOME/config/checkout-refresh" + rm -rf "$seed" "$outer" + pass "configured and scanned checkouts require exact Git roots" +} + +test_discovery_provenance_failures_invalidate_coverage() { + local fixture home state remote project pool out status scan fakebin real_git + fixture="$TMP_ROOT/discovery-provenance" + home="$fixture/home" + state="$fixture/state" + scan="$fixture/scan" + fakebin="$fixture/fakebin" + mkdir -p "$home/user" "$home/projects" "$home/config" "$state" "$scan" "$fakebin" + remote=$(build_origin discovery-provenance) + project="$home/projects/relvino" + clone_from "$remote" "$project" + pool="$home/user/.treehouse/relvino" + mkdir -p "$pool" + printf '{"worktrees":[{"name":"bad","path":"%s"}]}\n' "$project" \ + > "$pool/treehouse-state.json" + + set +e + out=$(run_isolated_refresh "$home" "$state" discover 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "unrelated Treehouse backing checkout was accepted" + assert_contains "$out" "Treehouse worktree identity or registration is not inspectable: $project" \ + "unrelated Treehouse state path was not surfaced" + + rm -rf "$home/user/.treehouse" + ln -s "$fixture/missing-project" "$home/projects/broken-project" + printf 'scan %s\n' "$scan" > "$home/config/checkout-refresh" + ln -s "$fixture/missing-scan-target" "$scan/broken-candidate" + set +e + out=$(run_isolated_refresh "$home" "$state" run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "broken discovery symlinks reported healthy coverage" + assert_contains "$out" "broken active-home project symlink" \ + "broken active-project symlink was not surfaced" + assert_refresh_state "$state" unhealthy + + rm -f "$home/projects/broken-project" + set +e + out=$(run_isolated_refresh "$home" "$state" run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "broken scanned symlink reported healthy coverage" + assert_contains "$out" "broken scan candidate symlink" \ + "broken scanned symlink was not surfaced" + assert_refresh_state "$state" unhealthy + + rm -f "$scan/broken-candidate" + clone_from "$remote" "$scan/scanned-repo" + : > "$scan/scanned-repo/.fm-fail-git-probe" + real_git=$(command -v git) + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = -C ] && [ -f "${2:-}/.fm-fail-git-probe" ] \ + && [ "${3:-}" = rev-parse ] && [ "${4:-}" = --is-inside-work-tree ]; then + exit 128 +fi +exec "$FM_REAL_GIT" "$@" +SH + chmod +x "$fakebin/git" + set +e + out=$(FM_REAL_GIT="$real_git" PATH="$fakebin:$PATH" \ + run_isolated_refresh "$home" "$state" run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "uninspectable discovered Git identity reported healthy coverage" + assert_contains "$out" "discovered Git identity cannot be inspected or disproved:" \ + "discovered rev-parse failure was classified as a non-Git directory" + assert_refresh_state "$state" unhealthy + pass "discovery provenance failures invalidate coverage health" +} + +test_upstream_tip_signal_refreshes_between_firstmate_events() { + local project external remote out + project="$FM_TEST_HOME/projects/relvino" + external="$TEST_HOME/relvino" + remote=$(git -C "$project" remote get-url origin) + : "$remote" + + run_refresh run-once --force >/dev/null + advance_origin relvino C1 + out=$(run_refresh run-once) + + assert_contains "$out" "synced" "upstream-tip change did not trigger a refresh" + assert_head_matches_origin "$project" + assert_head_matches_origin "$external" + pass "any observed upstream default-tip change refreshes all covered clones" +} + +test_periodic_backstop_repairs_drift_without_a_new_tip() { + local external="$TEST_HOME/relvino" prior + prior=$(git -C "$external" rev-parse HEAD^) + git -C "$external" reset --hard -q "$prior" + find "$STATE_ROOT" -type f -name '*.last' -exec sh -c 'printf "0\n" > "$1"' _ {} \; + + run_refresh run-once >/dev/null + + assert_head_matches_origin "$external" + pass "periodic backstop repairs local drift even when the observed upstream tip is unchanged" +} + +test_live_default_change_is_surfaced_without_switching_branches() { + local project before out + project=$(cd "$FM_TEST_HOME/projects/relvino" && pwd -P) + before=$(git -C "$project" rev-parse HEAD) + switch_origin_default relvino + + out=$(run_refresh run-once --force) + + assert_contains "$out" "relvino: STUCK: on branch main" \ + "a live upstream default-branch change was not surfaced as an unsafe checkout" + [ "$(git -C "$project" rev-parse HEAD)" = "$before" ] \ + || fail "default-branch change moved the checkout" + [ "$(git -C "$project" branch --show-current)" = main ] \ + || fail "default-branch change switched the checkout" + pass "live default-branch changes are excluded and surfaced without mutation" +} + +test_skill_drafts_surface_on_every_probe_without_log_spam() { + local project draft_one draft_two out key alert status + project=$(cd "$FM_TEST_HOME/projects/relvino" && pwd -P) + draft_one="$project/.agents/skills/local-one/SKILL.md" + draft_two="$project/skills/local-two/SKILL.md" + mkdir -p "$(dirname "$draft_one")" "$(dirname "$draft_two")" + printf '%s\n' '# local one' > "$draft_one" + + out=$(run_refresh run-once) + assert_contains "$out" "HYGIENE: 1 untracked skill-draft files" \ + "a new untracked skill draft was not surfaced between refresh events" + assert_contains "$out" ".agents/skills/local-one/SKILL.md" \ + "the hygiene alert did not identify the draft" + grep -Fq '# local one' "$draft_one" || fail "hygiene probe changed an untracked draft" + + out=$(run_refresh run-once) + assert_not_contains "$out" "HYGIENE:" \ + "an unchanged hygiene inventory was repeatedly logged by the background probe" + + printf '%s\n' '# local two' > "$draft_two" + out=$(run_refresh run-once) + assert_contains "$out" "HYGIENE: 2 untracked skill-draft files" \ + "growth in the untracked skill-draft inventory was not surfaced" + + set +e + out=$(run_refresh preflight "$project") + status=$? + set -e + [ "$status" -ne 0 ] || fail "spawn preflight accepted a checkout containing untracked drafts" + assert_contains "$out" "HYGIENE: 2 untracked skill-draft files" \ + "spawn preflight did not repeat the unresolved hygiene alert" + + out=$(run_refresh run-once --force --verbose) + assert_contains "$out" "HYGIENE: 2 untracked skill-draft files" \ + "an operator-visible forced refresh did not repeat the unresolved hygiene alert" + assert_contains "$out" "STUCK:" \ + "the safe refresh did not refuse the checkout containing untracked drafts" + assert_contains "$out" "2 untracked, 2 under repository skill directories" \ + "the safe refresh did not quantify untracked skill drafts" + grep -Fq '# local one' "$draft_one" || fail "safe refresh discarded the first draft" + grep -Fq '# local two' "$draft_two" || fail "safe refresh discarded the second draft" + + key=$(checkout_state_key "$project") + alert="$STATE_ROOT/$key.hygiene-alert" + [ -f "$alert" ] || fail "the unresolved hygiene alert was not persisted" + rm -rf "$project/.agents" "$project/skills" + run_refresh run-once >/dev/null + [ ! -e "$alert" ] || fail "the hygiene alert did not clear after drafts were reconciled" + pass "skill-draft accumulation surfaces promptly, persists, and never changes draft contents" +} + +test_preflight_rejects_hygiene_without_an_origin() { + local checkout="$TMP_ROOT/no-origin-checkout" draft out status + fm_git_init_commit "$checkout" + draft="$checkout/.agents/skills/local-only/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# local only' > "$draft" + + set +e + out=$(run_refresh preflight "$checkout") + status=$? + set -e + + [ "$status" -ne 0 ] || fail "preflight accepted untracked skill drafts in a no-origin checkout" + assert_contains "$out" "HYGIENE: 1 untracked skill-draft files" \ + "no-origin preflight swallowed its hygiene finding" + grep -Fq '# local only' "$draft" || fail "no-origin preflight changed the draft" + pass "preflight treats hygiene as actionable independently of sync eligibility" +} + +test_treehouse_pool_skill_drafts_are_inventoried() { + local pool_worktree draft out key alert + pool_worktree="$TEST_HOME/.treehouse/relvino-test/1/relvino" + pool_worktree=$(cd "$pool_worktree" && pwd -P) + draft="$pool_worktree/.agents/skills/pool-draft/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# pool draft' > "$draft" + + out=$(run_refresh run-once) + assert_contains "$out" "$pool_worktree: HYGIENE: 1 untracked skill-draft files" \ + "an untracked draft in a Treehouse pool worktree was not surfaced" + grep -Fq '# pool draft' "$draft" || fail "pool hygiene inventory changed the draft" + + key=$(checkout_state_key "$pool_worktree") + alert="$STATE_ROOT/$key.hygiene-alert" + [ -f "$alert" ] || fail "the pool-worktree hygiene alert was not persisted" + rm -rf "$pool_worktree/.agents" + run_refresh run-once >/dev/null + [ ! -e "$alert" ] || fail "the pool-worktree hygiene alert did not clear" + pass "Treehouse pool worktrees participate in skill-draft hygiene detection" +} + +test_ignored_skill_files_are_outside_the_collision_guard() { + local source="$TMP_ROOT/ignored-source" worktree="$TMP_ROOT/ignored-worktree" draft out + fm_git_worktree "$source" "$worktree" ignored-skill + git -C "$worktree" checkout --quiet --detach + printf '%s\n' '.agents/skills/' >> "$source/.git/info/exclude" + draft="$worktree/.agents/skills/intentional/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# intentional ignored material' > "$draft" + + run_refresh verify-worktree "$worktree" "$source" \ + || fail "an ignored skill file made a clean local acquisition fail" + out=$(run_refresh preflight "$worktree") \ + || fail "preflight rejected an acquisition containing only ignored skill material" + assert_not_contains "$out" "HYGIENE:" \ + "ignored skill material entered the untracked-draft collision inventory" + grep -Fq '# intentional ignored material' "$draft" \ + || fail "ignored skill-file inspection changed its contents" + pass "gitignored skill files remain outside the non-ignored collision guard" +} + +test_pool_preflight_surfaces_dirty_worktrees_without_blocking_clean_selection() { + local project pool_worktree before out + project=$(cd "$FM_TEST_HOME/projects/relvino" && pwd -P) + pool_worktree="$TEST_HOME/.treehouse/relvino-test/1/relvino" + pool_worktree=$(cd "$pool_worktree" && pwd -P) + before=$(cat "$pool_worktree/file.txt") + printf '%s\n' dirty-pool-change >> "$pool_worktree/file.txt" + + out=$(run_refresh pool-preflight "$project" 2>&1) \ + || fail "inspectable dirty pool entries should remain skippable while another clean entry may be selected" + assert_contains "$out" "$pool_worktree: skipped: dirty Treehouse pool worktree remains unavailable for acquisition" \ + "pre-acquisition pool inspection did not surface the dirty entry" + grep -Fq dirty-pool-change "$pool_worktree/file.txt" \ + || fail "pool preflight changed the dirty worktree" + printf '%s\n' "$before" > "$pool_worktree/file.txt" + pass "pool preflight surfaces dirty entries and leaves them unavailable untouched" +} + +test_bootstrap_relays_hygiene_alerts() { + local project draft out config_backup config_real + project=$(cd "$FM_TEST_HOME/projects/relvino" && pwd -P) + draft="$project/.agents/skills/bootstrap-draft/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# bootstrap draft' > "$draft" + config_backup=$(mktemp "$TMP_ROOT/checkout-refresh-config.XXXXXX") + cp "$FM_TEST_HOME/config/checkout-refresh" "$config_backup" + printf '%s\n' 'unexpected directive' >> "$FM_TEST_HOME/config/checkout-refresh" + + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" FM_TREEHOUSE_ROOT="$TEST_HOME/.treehouse" \ + FM_CHECKOUT_REFRESH_BOOTSTRAP_TEST=1 \ + "$ROOT/bin/fm-bootstrap.sh" 2>/dev/null) + mv "$config_backup" "$FM_TEST_HOME/config/checkout-refresh" + + assert_contains "$out" "FLEET_SYNC: $project: HYGIENE: 1 untracked skill-draft files" \ + "session-start bootstrap did not relay the unresolved hygiene alert" + assert_contains "$out" "FLEET_SYNC: checkout-refresh: skipped: unknown config directive 'unexpected'" \ + "session-start bootstrap swallowed checkout discovery diagnostics" + + config_real="$TMP_ROOT/checkout-refresh-real" + mv "$FM_TEST_HOME/config/checkout-refresh" "$config_real" + ln -s "$config_real" "$FM_TEST_HOME/config/checkout-refresh" + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" FM_TREEHOUSE_ROOT="$TEST_HOME/.treehouse" \ + FM_CHECKOUT_REFRESH_BOOTSTRAP_TEST=1 \ + "$ROOT/bin/fm-bootstrap.sh" 2>/dev/null) + rm "$FM_TEST_HOME/config/checkout-refresh" + mv "$config_real" "$FM_TEST_HOME/config/checkout-refresh" + assert_contains "$out" "FLEET_SYNC: checkout-refresh: skipped: unsafe config path" \ + "session-start bootstrap suppressed the unsafe configuration warning" + grep -Fq '# bootstrap draft' "$draft" || fail "bootstrap refresh changed the draft" + rm -rf "$project/.agents" + run_refresh run-once >/dev/null + pass "session-start bootstrap relays hygiene and discovery diagnostics" +} + +test_treehouse_discovery_failure_invalidates_coverage_health() { + local treehouse_root pool_dir bad_state missing_path="$TMP_ROOT/missing-treehouse-worktree" out status + treehouse_root=$(cd "$TEST_HOME/.treehouse" && pwd -P) + pool_dir="$treehouse_root/relvino-test" + bad_state="$treehouse_root/broken/treehouse-state.json" + mkdir -p "$(dirname "$bad_state")" + printf '%s\n' '{"worktrees":[' > "$bad_state" + printf '%s\n' preserved-heartbeat > "$STATE_ROOT/heartbeat" + + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "malformed Treehouse state reported healthy checkout coverage" + assert_contains "$out" "incomplete Treehouse coverage at $bad_state" \ + "malformed Treehouse state was not surfaced" + assert_refresh_state "$STATE_ROOT" unhealthy + + printf '%s\n' '{}' > "$bad_state" + printf '%s\n' preserved-schema-heartbeat > "$STATE_ROOT/heartbeat" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "Treehouse state without a worktrees field reported healthy coverage" + assert_contains "$out" "worktrees is required" \ + "missing Treehouse worktrees schema was not surfaced" + assert_refresh_state "$STATE_ROOT" unhealthy + + printf '{"worktrees":[{"path":"%s"}]}\n' "$missing_path" > "$bad_state" + printf '%s\n' preserved-path-heartbeat > "$STATE_ROOT/heartbeat" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "uninspectable declared Treehouse worktree reported healthy coverage" + assert_contains "$out" "Treehouse worktree identity or registration is not inspectable: $missing_path" \ + "uninspectable declared Treehouse worktree was not surfaced" + assert_refresh_state "$STATE_ROOT" unhealthy + rm -rf "$(dirname "$bad_state")" + + chmod 000 "$treehouse_root" + printf '%s\n' preserved-root-heartbeat > "$STATE_ROOT/heartbeat" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + chmod 700 "$treehouse_root" + [ "$status" -ne 0 ] || fail "unreadable Treehouse root reported healthy coverage" + assert_contains "$out" "configured root is unsafe or unreadable" \ + "unreadable Treehouse root was not surfaced" + assert_refresh_state "$STATE_ROOT" unhealthy + + chmod 000 "$pool_dir" + printf '%s\n' preserved-pool-heartbeat > "$STATE_ROOT/heartbeat" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + chmod 700 "$pool_dir" + [ "$status" -ne 0 ] || fail "unreadable Treehouse pool reported healthy coverage" + assert_contains "$out" "Treehouse pool is unreadable" \ + "unreadable Treehouse pool was not surfaced" + assert_refresh_state "$STATE_ROOT" unhealthy + pass "unreadable roots, malformed schemas, and uninspectable paths invalidate coverage health" +} + +test_raw_treehouse_root_symlink_invalidates_coverage_health() { + local real_root linked_root linked_parent linked_child missing_root out status + real_root="$TMP_ROOT/treehouse-root-real" + linked_root="$TMP_ROOT/treehouse-root-link" + mkdir -p "$real_root" + ln -s "$real_root" "$linked_root" + + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" FM_CHECKOUT_REFRESH_LOCK_ROOT="$LOCK_ROOT" \ + FM_TREEHOUSE_ROOT="$linked_root" FM_CHECKOUT_REFRESH_TEST=1 \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --force 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "symlinked raw Treehouse root reported healthy coverage" + assert_contains "$out" "configured root is unsafe or unreadable: $linked_root" \ + "symlinked raw Treehouse root was resolved before rejection" + assert_refresh_state "$STATE_ROOT" unhealthy + + linked_parent="$TMP_ROOT/treehouse-parent-link" + linked_child="$linked_parent/pools" + ln -s "$real_root" "$linked_parent" + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" FM_CHECKOUT_REFRESH_LOCK_ROOT="$LOCK_ROOT" \ + FM_TREEHOUSE_ROOT="$linked_child/" FM_CHECKOUT_REFRESH_TEST=1 \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "Treehouse root with a symlinked ancestor and trailing slash reported healthy coverage" + assert_contains "$out" "configured root is unsafe or unreadable: $linked_child/" \ + "Treehouse ancestor symlink was hidden by normalization" + + missing_root="$TMP_ROOT/configured-treehouse-missing" + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" FM_CHECKOUT_REFRESH_LOCK_ROOT="$LOCK_ROOT" \ + FM_TREEHOUSE_ROOT="$missing_root" FM_CHECKOUT_REFRESH_TEST=1 \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "explicitly configured missing Treehouse root reported healthy coverage" + assert_contains "$out" "configured root is unsafe or unreadable: $missing_root" \ + "missing configured Treehouse root was treated as benignly absent" + pass "raw, ancestor-symlinked, and missing configured Treehouse roots fail closed" +} + +test_empty_treehouse_and_identity_tool_failures_fail_closed() { + local out status + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" FM_CHECKOUT_REFRESH_LOCK_ROOT="$LOCK_ROOT" \ + FM_TREEHOUSE_ROOT='' FM_CHECKOUT_REFRESH_TEST=1 \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "explicitly empty Treehouse root fell back to the default" + assert_contains "$out" "configured root is unsafe or unreadable:" \ + "explicitly empty Treehouse root was not surfaced" + + set +e + out=$(FM_CHECKOUT_TEST_DISABLE_SYSTEM_PERL=1 run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "missing fixed identity tool reported healthy coverage" + assert_contains "$out" "checkout-refresh home identity is unavailable" \ + "fixed identity-tool failure did not fail before state aliasing" + pass "empty Treehouse configuration and identity-tool failures fail closed" +} + +test_config_git_metadata_and_non_git_races_fail_closed() { + local real_config linked_config remote source redirected scan candidate out status + real_config="$TMP_ROOT/config-real" + linked_config="$TMP_ROOT/config-linked" + mkdir -p "$real_config" + printf '# valid\n' > "$real_config/checkout-refresh" + ln -s "$real_config" "$linked_config" + set +e + out=$(FM_CONFIG_OVERRIDE="$linked_config" run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "config through a symlinked ancestor reported healthy coverage" + assert_contains "$out" "unsafe config path" \ + "config ancestor redirect was not surfaced" + rm -f "$linked_config" + + remote=$(build_origin redirected-git) + source="$TMP_ROOT/redirected-git-source" + redirected="$TMP_ROOT/redirected-git-candidate" + clone_from "$remote" "$source" + mkdir -p "$redirected" + ln -s "$source/.git" "$redirected/.git" + printf 'path %s\n' "$redirected" > "$FM_TEST_HOME/config/checkout-refresh" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "redirected Git metadata reported healthy coverage" + assert_contains "$out" "configured checkout is not an exact inspectable Git repository root" \ + "redirected Git metadata was not surfaced" + rm -f "$FM_TEST_HOME/config/checkout-refresh" + + scan="$TMP_ROOT/non-git-race-scan" + candidate="$scan/candidate" + mkdir -p "$candidate" + printf 'scan %s\n' "$scan" > "$FM_TEST_HOME/config/checkout-refresh" + set +e + out=$(FM_CHECKOUT_TEST_CREATE_GIT_AT="$candidate" run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "concurrent Git metadata creation was classified as non-Git" + assert_contains "$out" "discovered Git identity cannot be inspected or disproved" \ + "concurrent Git metadata creation was not surfaced" + rm -f "$FM_TEST_HOME/config/checkout-refresh" + pass "config, Git metadata, and non-Git classification races fail closed" +} + +test_skill_inventory_failure_preserves_alert_and_invalidates_coverage() { + local project draft key alert prior fakebin real_git out status + project=$(cd "$FM_TEST_HOME/projects/relvino" && pwd -P) + draft="$project/.agents/skills/inventory-failure/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# retained draft' > "$draft" + run_refresh run-once >/dev/null + key=$(checkout_state_key "$project") + alert="$STATE_ROOT/$key.hygiene-alert" + [ -f "$alert" ] || fail "inventory-failure setup did not persist a hygiene alert" + prior=$(cat "$alert") + printf '%s\n' preserved-inventory-heartbeat > "$STATE_ROOT/heartbeat" + fakebin="$TMP_ROOT/inventory-fakebin" + real_git=$(command -v git) + mkdir -p "$fakebin" + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +if [ "${3:-}" = ls-files ]; then + exit 74 +fi +exec "${FM_TEST_REAL_GIT:?}" "$@" +SH + chmod +x "$fakebin/git" + + set +e + out=$(FM_TEST_REAL_GIT="$real_git" PATH="$fakebin:$PATH" run_refresh run-once --force 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "skill inventory failure reported healthy coverage" + assert_contains "$out" "HYGIENE: inventory failed - preserving the prior alert" \ + "skill inventory failure was not surfaced" + [ "$(cat "$alert")" = "$prior" ] || fail "skill inventory failure changed the prior alert" + assert_refresh_state "$STATE_ROOT" unhealthy + rm -rf "$fakebin" "$project/.agents" + run_refresh run-once >/dev/null + pass "skill inventory failures preserve alerts and invalidate coverage health" +} + +test_lock_root_failure_invalidates_coverage_before_preparation() { + local state_root="$TMP_ROOT/lock-root-health-state" bad_lock="$TMP_ROOT/lock-root-health-file" + local out status now + mkdir -p "$state_root" + printf '%s\n' occupied > "$bad_lock" + printf '%s\n' manual-heartbeat > "$state_root/heartbeat" + now=$(date +%s) + printf '%s\n%s\n' "$now" healthy > "$state_root/coverage-health" + + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$state_root" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$bad_lock" \ + FM_TREEHOUSE_ROOT="$TEST_HOME/.treehouse" \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --force 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "unsafe lock-root preparation preserved healthy coverage" + assert_contains "$out" "unsafe checkout-refresh lock directory: $bad_lock" \ + "unsafe lock-root preparation was not surfaced" + assert_refresh_state "$state_root" unhealthy + assert_heartbeat_value "$state_root" manual-heartbeat + pass "lock-root preparation failures invalidate coverage without refreshing liveness" +} + +test_reinspection_failure_invalidates_coverage_health() { + local remote home state_root lock_root treehouse project fakebin real_git out key alert + local initial_head reinspection_count + remote=$(build_origin reinspection) + home="$TMP_ROOT/reinspection-home" + state_root="$TMP_ROOT/reinspection-state" + lock_root="$TMP_ROOT/reinspection-locks" + treehouse="$TMP_ROOT/reinspection-treehouse" + project="$home/projects/reinspection" + fakebin="$TMP_ROOT/reinspection-fakebin" + real_git=$(command -v git) + fm_git_init_commit "$home" + mkdir -p "$home/projects" "$home/config" "$state_root" "$treehouse" "$fakebin" + clone_from "$remote" "$project" + project=$(cd "$project" && pwd -P) + home=$(cd "$home" && pwd -P) + initial_head=$(git -C "$project" rev-parse HEAD) + advance_origin reinspection changed-after-discovery + printf '%s\n' manual-reinspection-heartbeat > "$state_root/heartbeat" + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = -C ] \ + && [ "${2:-}" = "${FM_TEST_REINSPECTION_TARGET:?}" ] \ + && [ "${3:-}" = rev-parse ] \ + && [ "${4:-}" = --show-toplevel ]; then + count=$(cat "${FM_TEST_REINSPECTION_COUNT:?}" 2>/dev/null || printf 0) + count=$((count + 1)) + printf '%s\n' "$count" > "$FM_TEST_REINSPECTION_COUNT" + if [ "$count" -gt 1 ]; then + printf '%s\n' "${FM_TEST_REINSPECTION_ENCLOSING:?}" + exit 0 + fi +fi +exec "${FM_TEST_REAL_GIT:?}" "$@" +SH + chmod +x "$fakebin/git" + + out=$(HOME="$TEST_HOME" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$state_root" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$lock_root" \ + FM_TREEHOUSE_ROOT="$treehouse" \ + FM_TEST_REAL_GIT="$real_git" FM_TEST_REINSPECTION_TARGET="$project" \ + FM_TEST_REINSPECTION_ENCLOSING="$home" \ + FM_TEST_REINSPECTION_COUNT="$TMP_ROOT/reinspection-count" \ + PATH="$fakebin:$PATH" \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --force 2>&1) + + assert_contains "$out" "$project: skipped: covered checkout became uninspectable during refresh" \ + "covered-checkout reinspection failure was not surfaced" + assert_refresh_state "$state_root" unhealthy + assert_heartbeat_value "$state_root" manual-reinspection-heartbeat + reinspection_count=$(cat "$TMP_ROOT/reinspection-count") + [ "$reinspection_count" -eq 3 ] \ + || fail "both post-discovery passes did not repeat the exact-root proof" + [ "$(git -C "$project" rev-parse HEAD)" = "$initial_head" ] \ + || fail "identity-drifted covered path was refreshed through its enclosing repository" + key=$(checkout_state_key "$project") + alert="$state_root/$key.alert" + assert_grep "covered checkout became uninspectable during refresh" "$alert" \ + "covered-checkout reinspection failure did not persist an alert" + pass "covered-checkout reinspection failures invalidate coverage health" +} + +test_scheduler_liveness_is_scheduler_owned() { + local home="$TMP_ROOT/scheduler-owned-home" state_root="$TMP_ROOT/scheduler-owned-state" + local lock_root="$TMP_ROOT/scheduler-owned-locks" treehouse="$TMP_ROOT/scheduler-owned-treehouse" + local project heartbeat + project="$home/projects/local" + mkdir -p "$home/projects" "$home/config" "$state_root" "$treehouse" + fm_git_init_commit "$project" + printf '%s\n' manual-sentinel > "$state_root/heartbeat" + + HOME="$TEST_HOME" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$state_root" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$lock_root" \ + FM_TREEHOUSE_ROOT="$treehouse" \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --force >/dev/null + assert_refresh_state "$state_root" healthy + assert_heartbeat_value "$state_root" manual-sentinel + + HOME="$TEST_HOME" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$state_root" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$lock_root" \ + FM_TREEHOUSE_ROOT="$treehouse" \ + "$ROOT/bin/fm-checkout-refresh.sh" run-once --scheduled --force >/dev/null + heartbeat=$(sed -n '1p' "$state_root/heartbeat" 2>/dev/null || true) + case "$heartbeat" in ''|*[!0-9]*) fail "scheduled run did not advance scheduler liveness" ;; esac + assert_refresh_state "$state_root" healthy + pass "only scheduler-owned runs advance the liveness heartbeat" +} + +test_unreadable_scan_root_invalidates_coverage_health() { + local home="$TMP_ROOT/unreadable-scan-home" state_root="$TMP_ROOT/unreadable-scan-state" + local scan_root="$TMP_ROOT/unreadable-scan-root" canonical_scan project out status + project="$home/projects/local" + mkdir -p "$home/user" "$home/projects" "$home/config" "$state_root" "$scan_root" + fm_git_init_commit "$project" + printf 'scan %s\n' "$scan_root" > "$home/config/checkout-refresh" + canonical_scan=$(cd "$scan_root" && pwd -P) + chmod 111 "$scan_root" + + out=$(run_isolated_refresh "$home" "$state_root" run-once --force 2>&1) + status=$? + chmod 700 "$scan_root" + + [ "$status" -ne 0 ] || fail "unreadable scan root reported successful coverage" + assert_contains "$out" "scan root is unreadable or cannot be enumerated: $canonical_scan" \ + "unreadable scan root was silently enumerated as empty" + assert_refresh_state "$state_root" unhealthy + pass "scan roots must be readable and successfully enumerable" +} + +test_unreadable_scanned_origin_invalidates_coverage_health() { + local remote home="$TMP_ROOT/unreadable-origin-home" state_root="$TMP_ROOT/unreadable-origin-state" + local project scan_root candidate canonical_candidate fakebin real_git out status + remote=$(build_origin unreadable-scanned-origin) + project="$home/projects/seed" + scan_root="$home/shared" + candidate="$scan_root/candidate" + fakebin="$home/fakebin" + real_git=$(command -v git) + mkdir -p "$home/user" "$home/projects" "$home/config" "$state_root" "$scan_root" "$fakebin" + clone_from "$remote" "$project" + clone_from "$remote" "$candidate" + canonical_candidate=$(cd "$candidate" && pwd -P) + printf 'scan %s\n' "$scan_root" > "$home/config/checkout-refresh" + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = -C ] \ + && [ "${2:-}" = "${FM_TEST_UNREADABLE_ORIGIN_TARGET:?}" ] \ + && [ "${3:-}" = remote ] \ + && [ "$#" -eq 3 ]; then + exit 74 +fi +exec "${FM_TEST_REAL_GIT:?}" "$@" +SH + chmod +x "$fakebin/git" + + out=$(FM_TEST_REAL_GIT="$real_git" FM_TEST_UNREADABLE_ORIGIN_TARGET="$canonical_candidate" \ + PATH="$fakebin:$PATH" \ + run_isolated_refresh "$home" "$state_root" run-once --force 2>&1) + status=$? + + [ "$status" -ne 0 ] || fail "unreadable scanned origin reported successful coverage" + assert_contains "$out" "discovered checkout origin identity cannot be inspected: $canonical_candidate" \ + "unreadable scanned origin was treated as an irrelevant directory" + assert_refresh_state "$state_root" unhealthy + pass "scanned repository origin failures invalidate coverage" +} + +test_failed_alert_persistence_forces_reinspection() { + local remote home="$TMP_ROOT/alert-persistence-home" state_root="$TMP_ROOT/alert-persistence-state" + local project key alert last_file out status + remote=$(build_origin alert-persistence) + project="$home/projects/alert-persistence" + mkdir -p "$home/user" "$home/projects" "$home/config" "$state_root" + clone_from "$remote" "$project" + run_isolated_refresh "$home" "$state_root" run-once --force >/dev/null + key=$(checkout_state_key "$project") + alert="$state_root/$key.alert" + last_file="$state_root/$key.last" + printf '%s\n' 1 > "$last_file" + mkdir "$alert" + printf '%s\n' dirty > "$project/untracked.txt" + + out=$(run_isolated_refresh "$home" "$state_root" run-once --force 2>&1) + status=$? + + [ "$status" -ne 0 ] || fail "failed checkout-alert persistence reported success" + assert_contains "$out" "STUCK:" "failed alert write did not surface the unsafe checkout" + assert_contains "$out" "checkout alert cannot be persisted" \ + "failed alert write was not surfaced" + [ "$(cat "$last_file")" = 1 ] || fail "failed refresh advanced checkout cadence state" + assert_refresh_state "$state_root" unhealthy + + out=$(run_isolated_refresh "$home" "$state_root" run-once 2>&1) + status=$? + + [ "$status" -ne 0 ] || fail "run after failed alert persistence reported success" + assert_contains "$out" "STUCK:" \ + "prior unhealthy coverage did not force pre-backstop reinspection" + assert_contains "$out" "checkout alert cannot be persisted" \ + "second alert persistence failure was hidden" + [ "$(cat "$last_file")" = 1 ] || fail "failed retry advanced checkout cadence state" + assert_refresh_state "$state_root" unhealthy + pass "failed alert persistence cannot become healthy before reinspection" +} + +test_local_authority_is_fully_inspected_and_tracks_origin_identity() { + local remote home="$TMP_ROOT/local-authority-home" state_root="$TMP_ROOT/local-authority-state" + local project initial out status fakebin real_git old_tip + remote=$(build_origin local-authority) + project="$home/projects/local-authority" + mkdir -p "$home/user" "$home/projects" "$home/config" "$home/data" "$state_root" + clone_from "$remote" "$project" + project=$(cd "$project" && pwd -P) + printf -- '- local-authority [local-only] - test project (added 2026-07-23)\n' > "$home/data/projects.md" + initial=$(git -C "$project" rev-parse HEAD) + advance_origin local-authority upstream-only-change + + out=$(run_isolated_refresh "$home" "$state_root" run-once --force --verbose) + + assert_contains "$out" "$project: already current at local main" \ + "local-only checkout with origin did not use its local default tip" + assert_refresh_state "$state_root" healthy + [ "$(git -C "$project" rev-parse HEAD)" = "$initial" ] \ + || fail "local-only checkout advanced to its remote" + + printf '%s\n' draft > "$project/untracked.txt" + out=$(run_isolated_refresh "$home" "$state_root" run-once --force 2>&1) + assert_contains "$out" "with uncommitted changes (1 untracked)" \ + "local-only untracked work was not surfaced" + assert_refresh_state "$state_root" unhealthy + status=0 + run_isolated_refresh "$home" "$state_root" preflight "$project" >/dev/null 2>&1 || status=$? + [ "$status" -ne 0 ] || fail "local-only preflight accepted untracked work" + rm -f "$project/untracked.txt" + + git -C "$project" checkout -q -b feature + out=$(run_isolated_refresh "$home" "$state_root" run-once --force 2>&1) + assert_contains "$out" "on non-default branch feature" \ + "local-only non-default branch was not surfaced" + assert_refresh_state "$state_root" unhealthy + status=0 + run_isolated_refresh "$home" "$state_root" preflight "$project" >/dev/null 2>&1 || status=$? + [ "$status" -ne 0 ] || fail "local-only preflight accepted a non-default branch" + git -C "$project" checkout -q main + + commit_file "$project" local.txt local local-default-advance + old_tip=$(git -C "$project" rev-parse HEAD^) + git -C "$project" checkout -q --detach "$old_tip" + out=$(run_isolated_refresh "$home" "$state_root" run-once --force 2>&1) + assert_contains "$out" "detached HEAD at stale local tip" \ + "stale local-only checkout was not surfaced" + assert_refresh_state "$state_root" unhealthy + git -C "$project" checkout -q main + + fakebin="$home/fakebin" + real_git=$(command -v git) + mkdir -p "$fakebin" + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = -C ] \ + && [ "${2:-}" = "${FM_TEST_LOCAL_STATUS_TARGET:?}" ] \ + && [ "${3:-}" = status ]; then + exit 74 +fi +exec "${FM_TEST_REAL_GIT:?}" "$@" +SH + chmod +x "$fakebin/git" + out=$(FM_TEST_REAL_GIT="$real_git" FM_TEST_LOCAL_STATUS_TARGET="$project" \ + PATH="$fakebin:$PATH" \ + run_isolated_refresh "$home" "$state_root" run-once --force 2>&1) + assert_contains "$out" "local checkout cleanliness cannot be inspected" \ + "unreadable local-only status was treated as clean" + assert_refresh_state "$state_root" unhealthy + rm -rf "$fakebin" + + run_isolated_refresh "$home" "$state_root" run-once --force >/dev/null + git -C "$project" remote remove origin + out=$(run_isolated_refresh "$home" "$state_root" run-once --force 2>&1) + assert_contains "$out" "covered checkout origin identity drifted" \ + "persisted local-only origin removal was not surfaced" + assert_refresh_state "$state_root" unhealthy + pass "local authority inspects safety and preserves origin identity" +} + +test_dirty_nondefault_and_diverged_checkouts_are_untouched() { + local remote dirty feature diverged dirty_head feature_head diverged_head out + remote=$(build_origin safety) + dirty="$FM_TEST_HOME/projects/safety-dirty" + feature="$FM_TEST_HOME/projects/safety-feature" + diverged="$FM_TEST_HOME/projects/safety-diverged" + clone_from "$remote" "$dirty" + clone_from "$remote" "$feature" + clone_from "$remote" "$diverged" + printf 'uncommitted\n' >> "$dirty/file.txt" + git -C "$feature" checkout -q -b feature + commit_file "$diverged" local.txt local local-divergence + dirty_head=$(git -C "$dirty" rev-parse HEAD) + feature_head=$(git -C "$feature" rev-parse HEAD) + diverged_head=$(git -C "$diverged" rev-parse HEAD) + advance_origin safety C1 + + out=$(run_refresh run-once --force) + + assert_contains "$out" "safety-dirty: STUCK:" "dirty checkout did not surface STUCK" + assert_contains "$out" "safety-feature: STUCK:" "non-default checkout did not surface STUCK" + assert_contains "$out" "safety-diverged: STUCK:" "diverged checkout did not surface STUCK" + [ "$(git -C "$dirty" rev-parse HEAD)" = "$dirty_head" ] || fail "dirty checkout HEAD moved" + [ "$(git -C "$feature" rev-parse HEAD)" = "$feature_head" ] || fail "feature checkout HEAD moved" + [ "$(git -C "$diverged" rev-parse HEAD)" = "$diverged_head" ] || fail "diverged checkout HEAD moved" + grep -Fq uncommitted "$dirty/file.txt" || fail "dirty checkout contents were discarded" + [ "$(git -C "$feature" branch --show-current)" = feature ] || fail "feature checkout branch changed" + pass "background refresh preserves and surfaces dirty, non-default, and diverged work" +} + +test_refresh_locks_recover_stale_owners_and_surface_contention() { + local run_lock="$STATE_ROOT/.run-lock" checkout common key checkout_lock alias out + mkdir -p "$run_lock" + touch -t 200001010000 "$run_lock" + + out=$(run_refresh run-once --force) + assert_not_contains "$out" "refresh already running" \ + "an abandoned ownerless run lock was not recovered" + [ ! -e "$run_lock" ] || fail "recovered run lock was not released" + + mkdir -p "$run_lock" + printf '%s\n' "$$" > "$run_lock/pid" + out=$(run_refresh run-once --force) + assert_contains "$out" "checkout-refresh: skipped: refresh already running (pid $$)" \ + "live run-lock contention was silent" + rm -rf "$run_lock" + + checkout=$(cd "$FM_TEST_HOME/projects/relvino" && pwd -P) + common=$(git -C "$checkout" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$checkout/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(checkout_lock_key "$common") + checkout_lock="$LOCK_ROOT/$key.lock" + mkdir -p "$checkout_lock" + printf '%s\n' "$$" > "$checkout_lock/pid" + out=$(run_refresh run-once --force) + assert_contains "$out" "$checkout: skipped: refresh already running (pid $$)" \ + "shared-checkout lock contention was not surfaced" + alias="$TMP_ROOT/relvino-checkout-alias" + ln -s "$checkout" "$alias" + out=$(run_refresh preflight "$alias" 2>&1) || true + assert_contains "$out" "checkout-refresh preflight target must be an exact inspectable Git repository root: $alias" \ + "a symlink alias was not rejected before lock resolution" + rm -rf "$checkout_lock" + pass "refresh locks recover abandoned owners and serialize every checkout alias" +} + +test_session_mode_preserves_gone_branch_pruning() { + local remote work project out + remote=$(build_origin prune) + work="$TMP_ROOT/work-prune" + git -C "$work" checkout -q -b merged + commit_file "$work" merged.txt merged merged + git -C "$work" push -q -u origin merged + git -C "$work" checkout -q main + project="$FM_TEST_HOME/projects/prune" + clone_from "$remote" "$project" + git -C "$project" checkout -q -b merged --track origin/merged + git -C "$project" checkout -q main + git -C "$work" push -q origin --delete merged + + run_refresh run-once --force >/dev/null + git -C "$project" show-ref --verify --quiet refs/heads/merged \ + || fail "cadence refresh pruned a gone branch" + + out=$(run_refresh run-once --force --session) + git -C "$project" show-ref --verify --quiet refs/heads/merged \ + || fail "session refresh deleted a gone branch without landed-work proof" + assert_contains "$out" "retained gone branch merged because landed work cannot be proved" \ + "session refresh did not surface an unproven gone branch" + + git -C "$work" merge -q --no-ff merged -m land-merged + git -C "$work" push -q origin main + run_refresh run-once --force --session >/dev/null + if git -C "$project" show-ref --verify --quiet refs/heads/merged; then + fail "session refresh retained a gone branch after its content landed" + fi + pass "session pruning requires positive landed-work proof" +} + +test_config_and_external_identity_fail_closed() { + local remote project external original_origin out status config_real + remote=$(build_origin identity-history) + project="$FM_TEST_HOME/projects/identity-history" + external="$TEST_HOME/identity-history" + clone_from "$remote" "$project" + clone_from "$remote" "$external" + run_refresh run-once --force >/dev/null + + original_origin=$(git -C "$external" remote get-url origin) + git -C "$external" remote set-url origin file://"$TMP_ROOT/remotes/unrelated.git" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "external origin drift reported successful coverage" + assert_contains "$out" "prior external checkout origin changed at" \ + "external origin drift was silently dropped from discovery" + assert_refresh_state "$STATE_ROOT" unhealthy + git -C "$external" remote set-url origin "$original_origin" + run_refresh run-once --force >/dev/null + + printf '%s\n' 'unexpected directive' > "$FM_TEST_HOME/config/checkout-refresh" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "unknown checkout-refresh config reported successful coverage" + assert_contains "$out" "unknown config directive" "unknown config was not surfaced" + assert_refresh_state "$STATE_ROOT" unhealthy + rm -f "$FM_TEST_HOME/config/checkout-refresh" + + { + printf 'path %s\n' "$TMP_ROOT/missing-configured-checkout" + printf 'scan %s\n' "$TMP_ROOT/missing-configured-scan" + } > "$FM_TEST_HOME/config/checkout-refresh" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "invalid configured coverage paths reported success" + assert_contains "$out" "configured checkout is not an exact inspectable Git repository root" \ + "invalid configured checkout was not surfaced" + assert_contains "$out" "configured scan root is not a directory" \ + "invalid configured scan root was not surfaced" + rm -f "$FM_TEST_HOME/config/checkout-refresh" + + config_real="$TMP_ROOT/checkout-refresh-symlink-target" + printf '# valid target\n' > "$config_real" + ln -s "$config_real" "$FM_TEST_HOME/config/checkout-refresh" + set +e + out=$(run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "symlinked checkout-refresh config reported success" + assert_contains "$out" "unsafe config path" "symlinked config was not surfaced" + assert_refresh_state "$STATE_ROOT" unhealthy + rm -f "$FM_TEST_HOME/config/checkout-refresh" "$config_real" + pass "configuration and prior external identity failures invalidate coverage" +} + +test_public_entrypoints_reject_nested_repository_paths() { + local source worktree nested alias_parent alias_source out status + source="$TMP_ROOT/exact-entry-source" + worktree="$TMP_ROOT/exact-entry-worktree" + fm_git_worktree "$source" "$worktree" exact-entry + nested="$source/nested" + mkdir -p "$nested" + set +e + out=$(run_refresh preflight "$nested" 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "preflight accepted a nested repository path" + assert_contains "$out" "must be an exact inspectable Git repository root" \ + "nested preflight refusal was unclear" + set +e + run_refresh acquire-worktree "$nested" nested-test >/dev/null 2>&1 + status=$? + set -e + [ "$status" -ne 0 ] || fail "worktree acquisition accepted a nested repository path" + set +e + run_refresh verify-worktree "$worktree" "$nested" >/dev/null 2>&1 + status=$? + set -e + [ "$status" -ne 0 ] || fail "worktree verification accepted a nested source path" + alias_parent="$TMP_ROOT/exact-entry-parent-link" + ln -s "$(dirname "$source")" "$alias_parent" + alias_source="$alias_parent/$(basename "$source")/" + set +e + out=$(run_refresh preflight "$alias_source" 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "preflight accepted a repository through a symlinked ancestor and trailing slash" + assert_contains "$out" "must be an exact inspectable Git repository root" \ + "ancestor-symlinked preflight refusal was unclear" + mkdir -p "$source/relative-caller" + ( + cd "$source/relative-caller" || exit 1 + run_refresh preflight .. >/dev/null + ) || fail "parent-relative exact repository root was rejected" + pass "public refresh entrypoints require lexical and canonical repository roots" +} + +test_explicit_secondmate_home_requires_live_default_tip() { + local remote source home status + remote=$(build_origin explicit-home-freshness) + source="$TMP_ROOT/explicit-home-source" + home="$TMP_ROOT/explicit-secondmate-home" + clone_from "$remote" "$source" + git clone --quiet "$source" "$home" + run_refresh verify-home "$home" "$source" \ + || fail "fresh explicit secondmate home failed live-tip verification" + advance_origin explicit-home-freshness C1 + set +e + run_refresh verify-home "$home" "$source" >/dev/null 2>&1 + status=$? + set -e + [ "$status" -ne 0 ] || fail "stale explicit secondmate home passed live-tip verification" + run_refresh preflight "$source" >/dev/null + run_refresh preflight "$home" >/dev/null + run_refresh verify-home "$home" "$source" \ + || fail "refreshed explicit secondmate home failed live-tip verification" + pass "explicit secondmate homes require proven live-tip freshness" +} + +test_lock_owner_symlink_cannot_escape_state_directory() { + local outside lock status + outside="$TEST_HOME/external-lock-owner" + lock="$STATE_ROOT/escaped-lock" + mkdir -p "$outside" + printf '%s\n' 999999 > "$outside/pid" + printf '%s\n' 999999 > "$outside/process-group" + ln -s "$outside" "$lock" + set +e + FM_HOME="$FM_TEST_HOME" FM_ROOT="$ROOT" \ + bash -c '. "$1/bin/fm-wake-lib.sh"; FM_LOCK_STALE_AFTER=0; fm_lock_try_acquire "$2"' \ + bash "$ROOT" "$lock" >/dev/null 2>&1 + status=$? + set -e + [ "$status" -ne 0 ] || fail "lock acquisition accepted an external owner target" + assert_present "$outside/pid" "lock cleanup deleted an external owner pid" + assert_present "$outside/process-group" "lock cleanup deleted an external process-group guard" + [ -L "$lock" ] || fail "lock cleanup rewrote a malformed external owner link" + pass "lock cleanup remains confined to state-owned owner directories" +} + +test_worktree_freshness_verification_fails_closed() { + local remote primary worktree unrelated local_source local_worktree before status dirty tip + remote=$(build_origin verify) + primary="$TMP_ROOT/verify-primary" + worktree="$TMP_ROOT/verify-worktree" + clone_from "$remote" "$primary" + before=$(git -C "$primary" rev-parse HEAD) + git -C "$primary" worktree add --quiet --detach "$worktree" "$before" + advance_origin verify C1 + + set +e + HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" \ + "$ROOT/bin/fm-checkout-refresh.sh" verify-worktree "$worktree" "$primary" >/dev/null 2>&1 + status=$? + set -e + [ "$status" -ne 0 ] || fail "stale acquired worktree passed freshness verification" + + git -C "$primary" fetch -q origin + git -C "$worktree" checkout --quiet --detach origin/main + HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$STATE_ROOT" \ + "$ROOT/bin/fm-checkout-refresh.sh" verify-worktree "$worktree" "$primary" \ + || fail "fresh acquired worktree failed verification" + tip=$(git -C "$worktree" rev-parse HEAD) + run_refresh verify-returnable "$worktree" "$primary" "$tip" \ + || fail "unchanged detached acquisition failed return-safety verification" + git -C "$worktree" switch --quiet -c return-unsafe + set +e + run_refresh verify-returnable "$worktree" "$primary" "$tip" >/dev/null 2>&1 + status=$? + set -e + [ "$status" -ne 0 ] || fail "an attached acquired worktree passed return-safety verification" + git -C "$worktree" checkout --quiet --detach "$tip" + + unrelated="$TMP_ROOT/verify-unrelated" + fm_git_init_commit "$unrelated" + set +e + run_refresh verify-worktree "$unrelated" "$primary" >/dev/null 2>&1 + status=$? + set -e + [ "$status" -ne 0 ] || fail "an unrelated repository passed worktree identity verification" + + local_source="$TMP_ROOT/verify-local-source" + local_worktree="$TMP_ROOT/verify-local-worktree" + fm_git_worktree "$local_source" "$local_worktree" local-acquisition + git -C "$local_worktree" checkout --quiet --detach + run_refresh verify-worktree "$local_worktree" "$local_source" \ + || fail "clean remote-free worktree failed its local default-tip proof" + commit_file "$local_source" local.txt advanced advance-local-default + set +e + run_refresh verify-worktree "$local_worktree" "$local_source" >/dev/null 2>&1 + status=$? + set -e + [ "$status" -ne 0 ] || fail "stale remote-free worktree passed its local default-tip proof" + + git -C "$local_worktree" reset --hard -q "$(git -C "$local_source" rev-parse HEAD)" + dirty="$local_worktree/.agents/skills/retained/SKILL.md" + mkdir -p "$(dirname "$dirty")" + printf '%s\n' '# retain me' > "$dirty" + set +e + run_refresh verify-worktree "$local_worktree" "$local_source" >/dev/null 2>&1 + status=$? + set -e + [ "$status" -eq 3 ] || fail "dirty acquired worktree did not return the retain-only status" + grep -Fq '# retain me' "$dirty" || fail "dirty worktree verification changed its draft" + pass "acquisition proof validates repository identity, local freshness, and cleanliness" +} + +test_bounded_refresh_terminates_descendants() { + local remote checkout fakebin real_git out status parent_pid child_pid + remote=$(build_origin bounded) + checkout="$FM_TEST_HOME/projects/bounded" + clone_from "$remote" "$checkout" + fakebin="$TMP_ROOT/bounded-fakebin" + real_git=$(command -v git) + mkdir -p "$fakebin" + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +if [ "${3:-}" = fetch ]; then + trap '' TERM + printf '%s\n' "$BASHPID" > "${FM_TEST_FETCH_PARENT:?}" + ( + trap '' TERM + printf '%s\n' "$BASHPID" > "${FM_TEST_FETCH_CHILD:?}" + while :; do sleep 1; done + ) & + wait +fi +exec "${FM_TEST_REAL_GIT:?}" "$@" +SH + chmod +x "$fakebin/git" + + set +e + out=$(FM_TEST_REAL_GIT="$real_git" FM_TEST_FETCH_PARENT="$TMP_ROOT/fetch-parent.pid" \ + FM_TEST_FETCH_CHILD="$TMP_ROOT/fetch-child.pid" FM_CHECKOUT_REFRESH_SYNC_TIMEOUT=1 \ + PATH="$fakebin:$PATH" run_refresh run-once --force 2>&1) + status=$? + set -e + [ "$status" -eq 0 ] || fail "bounded refresh command failed unexpectedly: $out" + assert_contains "$out" "refresh timed out after 1s" \ + "bounded refresh did not report its timeout" + assert_refresh_state "$STATE_ROOT" unhealthy + parent_pid=$(cat "$TMP_ROOT/fetch-parent.pid") + child_pid=$(cat "$TMP_ROOT/fetch-child.pid") + if kill -0 "$parent_pid" 2>/dev/null || kill -0 "$child_pid" 2>/dev/null; then + fail "bounded refresh returned while a fetch descendant was still alive" + fi + rm -rf "$fakebin" "$checkout" + pass "bounded refresh terminates and reaps its complete descendant tree" +} + +test_acquisition_honors_shared_checkout_lock() { + local source fakebin common key lock out status marker + source="$TMP_ROOT/acquisition-lock-source" + fakebin="$TMP_ROOT/acquisition-lock-fakebin" + marker="$TMP_ROOT/acquisition-lock-called" + fm_git_init_commit "$source" + mkdir -p "$fakebin" + cat > "$fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +touch "${FM_TEST_TREEHOUSE_CALLED:?}" +printf '%s\n' "$PWD/acquired" +SH + chmod +x "$fakebin/treehouse" + common=$(git -C "$source" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$source/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(checkout_lock_key "$common") + lock="$LOCK_ROOT/$key.lock" + mkdir -p "$lock" + printf '%s\n' "$$" > "$lock/pid" + + set +e + out=$(FM_TEST_TREEHOUSE_CALLED="$marker" PATH="$fakebin:$PATH" \ + run_refresh acquire-worktree "$source" firstmate-lock-test 2>&1) + status=$? + set -e + + [ "$status" -ne 0 ] || fail "Treehouse acquisition bypassed the shared checkout lock" + assert_contains "$out" "Treehouse acquisition already running for $source (pid $$)" \ + "contended Treehouse acquisition did not identify the shared lock owner" + [ ! -e "$marker" ] || fail "Treehouse ran while the shared checkout lock was held" + rm -rf "$lock" + pass "Treehouse acquisition serializes through the common Git lock" +} + +test_launch_agent_definition_is_home_scoped_with_scheduler_seam() { + local fakebin fake_state agents log plist second_home second_plist key second_key install_state_base install_state_root + local loaded_drift + local custom_treehouse="$TMP_ROOT/custom-treehouse" other_treehouse="$TMP_ROOT/other-treehouse" out status now generation + fakebin="$TMP_ROOT/fakebin" + fake_state="$TMP_ROOT/fake-launchctl-state" + agents="$TMP_ROOT/LaunchAgents" + log="$TMP_ROOT/launchctl.log" + install_state_base="$TMP_ROOT/install-state" + mkdir -p "$fakebin" "$fake_state" "$agents" "$custom_treehouse" "$other_treehouse" + custom_treehouse=$(cd "$custom_treehouse" && pwd -P) + other_treehouse=$(cd "$other_treehouse" && pwd -P) + write_stateful_launchctl_fake "$fakebin/launchctl" + + HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_TREEHOUSE_ROOT="$custom_treehouse" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$install_state_base" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + "$ROOT/bin/fm-checkout-refresh.sh" install + + key=$(checkout_state_key "$FM_TEST_HOME" 16) + plist="$agents/com.firstmate.checkout-refresh.$key.plist" + install_state_root="$install_state_base/homes/$key" + assert_grep '<key>StartInterval</key><integer>60</integer>' "$plist" \ + "LaunchAgent does not carry the upstream signal cadence" + assert_grep '<key>FM_CHECKOUT_REFRESH_BACKSTOP</key><string>900</string>' "$plist" \ + "LaunchAgent does not persist the timed backstop" + assert_grep 'fm-checkout-refresh.sh</string>' "$plist" \ + "LaunchAgent does not invoke the checkout refresher" + assert_grep '<string>--scheduled</string>' "$plist" \ + "LaunchAgent does not identify its scheduler-owned invocation" + assert_grep "<key>FM_HOME</key><string>$(cd "$FM_TEST_HOME" && pwd -P)</string>" "$plist" \ + "LaunchAgent does not bind the active Firstmate home" + assert_grep "<key>FM_TREEHOUSE_ROOT</key><string>$custom_treehouse</string>" "$plist" \ + "LaunchAgent does not persist the configured Treehouse root" + assert_grep "<key>FM_CHECKOUT_REFRESH_STATE_ROOT</key><string>$install_state_base/homes/$key</string>" "$plist" \ + "LaunchAgent does not use home-scoped state" + assert_grep "<key>FM_CHECKOUT_REFRESH_LOCK_ROOT</key><string>$install_state_base/locks</string>" "$plist" \ + "LaunchAgent does not use the shared checkout lock root" + assert_grep 'bootstrap' "$log" "LaunchAgent was not bootstrapped" + assert_grep 'kickstart' "$log" "LaunchAgent was not started" + now=$(date +%s) + printf '%s\n' "$now" > "$install_state_root/heartbeat" + printf '%s\n%s\n' "$((now - 1))" healthy > "$install_state_root/coverage-health" + generation=$(plist_generation "$plist") + [ "${#generation}" -eq 32 ] || fail "LaunchAgent scheduler generation is missing" + printf '%s\n' "$generation" > "$install_state_root/scheduler-generation" + HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_TREEHOUSE_ROOT="$custom_treehouse" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$install_state_base" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure \ + || fail "matching LaunchAgent scheduler configuration was reported unhealthy" + + printf '%s\n%s\n' "$now" unhealthy > "$install_state_root/coverage-health" + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_TREEHOUSE_ROOT="$custom_treehouse" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$install_state_base" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "LaunchAgent health accepted an unhealthy latest coverage run" + assert_contains "$out" "latest coverage run is missing or unhealthy" \ + "LaunchAgent health did not diagnose the failed coverage run" + printf '%s\n%s\n' "$now" healthy > "$install_state_root/coverage-health" + loaded_drift="$TMP_ROOT/loaded-launch-agent-drift.plist" + cp "$plist" "$loaded_drift" + sed -i.bak 's#<key>FM_HOME</key><string>[^<]*</string>#<key>FM_HOME</key><string>/stale/home</string>#' "$loaded_drift" + rm -f "$loaded_drift.bak" + mark_launch_agent_loaded "$fake_state" "com.firstmate.checkout-refresh.$key" "$loaded_drift" + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_TREEHOUSE_ROOT="$custom_treehouse" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$install_state_base" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "LaunchAgent health trusted a stale same-label loaded job" + assert_contains "$out" "loaded identity is missing or untrusted" \ + "loaded LaunchAgent identity drift was not surfaced before health" + mark_launch_agent_loaded "$fake_state" "com.firstmate.checkout-refresh.$key" "$plist" + + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_TREEHOUSE_ROOT="$other_treehouse" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$install_state_base" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "LaunchAgent health accepted a different Treehouse root" + assert_contains "$out" "different Treehouse root" \ + "LaunchAgent Treehouse-root drift was not diagnosed" + + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_TREEHOUSE_ROOT="$custom_treehouse" FM_CHECKOUT_REFRESH_INTERVAL=61 \ + FM_CHECKOUT_REFRESH_STATE_BASE="$install_state_base" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "LaunchAgent health accepted a different refresh interval" + assert_contains "$out" "different refresh interval" \ + "LaunchAgent refresh-interval drift was not diagnosed" + + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_TREEHOUSE_ROOT="$custom_treehouse" FM_CHECKOUT_REFRESH_BACKSTOP=901 \ + FM_CHECKOUT_REFRESH_STATE_BASE="$install_state_base" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "LaunchAgent health accepted a different refresh backstop" + assert_contains "$out" "different refresh backstop" \ + "LaunchAgent refresh-backstop drift was not diagnosed" + + second_home="$TMP_ROOT/fm-home-two" + mkdir -p "$second_home/projects" "$second_home/config" + HOME="$TEST_HOME" FM_HOME="$second_home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$install_state_base" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + "$ROOT/bin/fm-checkout-refresh.sh" install + second_key=$(checkout_state_key "$second_home" 16) + second_plist="$agents/com.firstmate.checkout-refresh.$second_key.plist" + [ -f "$plist" ] && [ -f "$second_plist" ] \ + || fail "installing a second home displaced the first home's LaunchAgent" + assert_grep "<key>FM_HOME</key><string>$(cd "$second_home" && pwd -P)</string>" "$second_plist" \ + "second LaunchAgent does not bind its own Firstmate home" + assert_grep "<key>FM_CHECKOUT_REFRESH_STATE_ROOT</key><string>$install_state_base/homes/$second_key</string>" "$second_plist" \ + "second LaunchAgent does not use its own state directory" + + set +e + out=$(HOME="$TEST_HOME" FM_HOME="$FM_TEST_HOME" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_PLATFORM=Linux \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "Linux scheduler seam silently reported background coverage" + assert_contains "$out" "no Linux scheduler adapter yet" \ + "Linux scheduler seam did not report its explicit platform limitation" + pass "scheduler ownership is home-scoped and Linux remains an explicit adapter seam" +} + +test_logical_home_state_migrates_and_ambiguity_fails_closed() { + local home state_base logical_key current_physical_key physical_key logical physical agents fakebin fake_state physical_plist logical_plist now out status + local rollback_home rollback_logical_key rollback_physical_key rollback_logical rollback_physical rollback_physical_plist rollback_logical_plist + local staging_home staging_logical_key staging_physical_key staging_logical staging_physical staging_physical_plist + home="$TMP_ROOT/state-migration-home" + state_base="$TMP_ROOT/state-migration-base" + mkdir -p "$home/projects" "$home/config" "$home/user/.treehouse" + logical_key=$(checkout_state_key "$home" 16) + current_physical_key=$(fm_checkout_physical_path_key "$home" directory 16) + physical_key=1111111111111111 + [ "$physical_key" != "$logical_key" ] || physical_key=2222222222222222 + [ "$physical_key" != "$current_physical_key" ] || physical_key=3333333333333333 + logical="$state_base/homes/$logical_key" + physical="$state_base/homes/$physical_key" + agents="$TMP_ROOT/state-migration-agents" + fakebin="$TMP_ROOT/state-migration-fakebin" + fake_state="$TMP_ROOT/state-migration-launchctl-state" + physical_plist="$agents/com.firstmate.checkout-refresh.$physical_key.plist" + logical_plist="$agents/com.firstmate.checkout-refresh.$logical_key.plist" + mkdir -p "$physical" "$agents" "$fakebin" "$fake_state" + printf 'preserved\n' > "$physical/external-identities" + now=$(date +%s) + printf '%s\n%s\n' "$now" "$ROOT/bin/fm-checkout-refresh.sh" > "$physical/heartbeat" + printf '%s\nhealthy\n' "$now" > "$physical/coverage-health" + write_launch_agent_fixture "$physical_plist" \ + "com.firstmate.checkout-refresh.$physical_key" "$home" "$physical" + write_stateful_launchctl_fake "$fakebin/launchctl" + mark_launch_agent_loaded "$fake_state" \ + "com.firstmate.checkout-refresh.$physical_key" "$physical_plist" + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LOGICAL_PLIST="$logical_plist" \ + FM_TEST_LOGICAL_STATE="$logical" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + [ -f "$logical/external-identities" ] || fail "physical home state was not migrated to the logical namespace" + [ ! -e "$physical" ] || fail "physical home state namespace survived migration" + [ -f "$logical_plist" ] || fail "physical LaunchAgent was not migrated to the logical label" + [ ! -e "$physical_plist" ] || fail "physical LaunchAgent survived logical-label migration" + + rollback_home="$TMP_ROOT/state-migration-rollback-home" + mkdir -p "$rollback_home/projects" "$rollback_home/config" "$rollback_home/user/.treehouse" + rollback_logical_key=$(checkout_state_key "$rollback_home" 16) + rollback_physical_key=4444444444444444 + [ "$rollback_physical_key" != "$rollback_logical_key" ] || rollback_physical_key=5555555555555555 + rollback_logical="$state_base/homes/$rollback_logical_key" + rollback_physical="$state_base/homes/$rollback_physical_key" + rollback_physical_plist="$agents/com.firstmate.checkout-refresh.$rollback_physical_key.plist" + rollback_logical_plist="$agents/com.firstmate.checkout-refresh.$rollback_logical_key.plist" + mkdir -p "$rollback_physical" + printf 'preserved-on-failure\n' > "$rollback_physical/external-identities" + printf '%s\n%s\n' "$now" "$ROOT/bin/fm-checkout-refresh.sh" > "$rollback_physical/heartbeat" + printf '%s\nhealthy\n' "$now" > "$rollback_physical/coverage-health" + write_launch_agent_fixture "$rollback_physical_plist" \ + "com.firstmate.checkout-refresh.$rollback_physical_key" \ + "$rollback_home" "$rollback_physical" + mark_launch_agent_loaded "$fake_state" \ + "com.firstmate.checkout-refresh.$rollback_physical_key" "$rollback_physical_plist" + set +e + out=$(HOME="$rollback_home/user" FM_HOME="$rollback_home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$rollback_home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_CHECKOUT_REFRESH_ACTIVATION_TIMEOUT=1 \ + FM_TEST_PHYSICAL_PLIST="$rollback_physical_plist" \ + FM_FAKE_LAUNCHCTL_LOG="$TMP_ROOT/state-migration-rollback-launchctl.log" \ + "$ROOT/bin/fm-checkout-refresh.sh" install 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "failed logical LaunchAgent activation reported success" + assert_contains "$out" "active scheduler generation" \ + "copied legacy health records were accepted as fresh logical scheduler proof" + [ -f "$rollback_physical/external-identities" ] \ + || fail "failed LaunchAgent migration did not restore physical state" + [ -f "$rollback_physical_plist" ] || fail "failed LaunchAgent migration removed the prior definition" + [ ! -e "$rollback_logical" ] || fail "failed LaunchAgent migration left a logical state namespace" + [ ! -e "$rollback_logical_plist" ] || fail "failed LaunchAgent migration left a logical definition" + assert_grep "bootstrap gui/$(id -u) $rollback_physical_plist" "$TMP_ROOT/state-migration-rollback-launchctl.log" \ + "failed logical health verification did not restart the prior LaunchAgent" + + staging_home="$TMP_ROOT/state-migration-stage-home" + mkdir -p "$staging_home/projects" "$staging_home/config" "$staging_home/user/.treehouse" + staging_logical_key=$(checkout_state_key "$staging_home" 16) + staging_physical_key=6666666666666666 + [ "$staging_physical_key" != "$staging_logical_key" ] || staging_physical_key=7777777777777777 + staging_logical="$state_base/homes/$staging_logical_key" + staging_physical="$state_base/homes/$staging_physical_key" + staging_physical_plist="$agents/com.firstmate.checkout-refresh.$staging_physical_key.plist" + mkdir -p "$staging_physical" + printf 'preserved-stage\n' > "$staging_physical/external-identities" + write_launch_agent_fixture "$staging_physical_plist" \ + "com.firstmate.checkout-refresh.$staging_physical_key" \ + "$staging_home" "$staging_physical" + mark_launch_agent_loaded "$fake_state" \ + "com.firstmate.checkout-refresh.$staging_physical_key" "$staging_physical_plist" + set +e + out=$(HOME="$staging_home/user" FM_HOME="$staging_home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$staging_home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_CHECKOUT_REFRESH_TEST=1 \ + FM_CHECKOUT_TEST_HOME_MIGRATION_FAILURE=stage \ + FM_FAKE_LAUNCHCTL_LOG="$TMP_ROOT/state-migration-stage-launchctl.log" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "partial home-state staging failure reported success" + [ -f "$staging_physical/external-identities" ] || fail "partial staging failure damaged prior state" + [ -f "$staging_physical_plist" ] || fail "partial staging failure removed prior LaunchAgent" + [ ! -e "$staging_logical" ] || fail "partial staging failure published a logical namespace" + + mkdir -p "$state_base/homes/$current_physical_key" + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Linux \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "ambiguous state namespaces were accepted" + assert_contains "$out" "ambiguous checkout-refresh home state namespaces" \ + "ambiguous state namespaces did not fail closed" + pass "logical home state migration is discoverable, staged, and rollback-safe" +} + +test_launch_agent_label_and_custom_legacy_state_are_authoritative() { + local home state_base custom_state agents fakebin fake_state logical_key logical_label logical_plist legacy_plist out status + home="$TMP_ROOT/launch-agent-identity-home" + state_base="$TMP_ROOT/launch-agent-identity-state-base" + custom_state="$TMP_ROOT/launch-agent-custom-state" + agents="$TMP_ROOT/launch-agent-identity-agents" + fakebin="$TMP_ROOT/launch-agent-identity-fakebin" + fake_state="$TMP_ROOT/launch-agent-identity-launchctl-state" + mkdir -p "$home/projects" "$home/config" "$home/user/.treehouse" \ + "$custom_state" "$agents" "$fakebin" "$fake_state" + logical_key=$(checkout_state_key "$home" 16) + logical_label="com.firstmate.checkout-refresh.$logical_key" + logical_plist="$agents/$logical_label.plist" + legacy_plist="$agents/com.firstmate.checkout-refresh.plist" + write_launch_agent_fixture "$logical_plist" \ + "com.firstmate.checkout-refresh.aaaaaaaaaaaaaaaa" "$home" "$custom_state" + write_stateful_launchctl_fake "$fakebin/launchctl" + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "renamed LaunchAgent definition was accepted" + assert_contains "$out" "LaunchAgent namespaces cannot be safely enumerated" \ + "LaunchAgent Label drift was not surfaced" + rm -f "$logical_plist" + + printf 'legacy-state\n' > "$custom_state/external-identities" + write_launch_agent_fixture "$legacy_plist" \ + "com.firstmate.checkout-refresh" "$home" "$custom_state" + mark_launch_agent_loaded "$fake_state" \ + "com.firstmate.checkout-refresh" "$legacy_plist" + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LAUNCHCTL_BOOTOUT_FAIL_LABEL=com.firstmate.checkout-refresh \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "legacy LaunchAgent bootout failure allowed replacement activation" + assert_present "$legacy_plist" "failed legacy bootout removed its tracking definition" + assert_absent "$logical_plist" "failed legacy bootout activated a duplicate logical scheduler" + assert_contains "$out" "cannot quiesce checkout-refresh LaunchAgent" \ + "legacy bootout failure was not surfaced" + HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LOGICAL_PLIST="$logical_plist" \ + FM_TEST_LOGICAL_STATE="$custom_state" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure >/dev/null \ + || fail "unsuffixed custom-state LaunchAgent was not migrated" + assert_present "$logical_plist" "legacy LaunchAgent was not migrated to the logical label" + assert_absent "$legacy_plist" "unsuffixed legacy LaunchAgent survived migration" + assert_present "$custom_state/external-identities" "custom state was not preserved" + assert_absent "$state_base/homes/$logical_key" "custom state was silently moved to the default namespace" + HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure >/dev/null \ + || fail "logical custom-state LaunchAgent could not rediscover its namespace" + pass "LaunchAgent Label and custom legacy state remain authoritative" +} + +test_loaded_launch_agent_controls_and_untracked_legacy_job_fail_closed() { + local home state_base state_root agents fakebin fake_state key label plist generation now out status + local legacy_home legacy_key legacy_plist legacy_shadow legacy_state_root log + home="$TMP_ROOT/loaded-control-home" + state_base="$TMP_ROOT/loaded-control-state" + agents="$TMP_ROOT/loaded-control-agents" + fakebin="$TMP_ROOT/loaded-control-fakebin" + fake_state="$TMP_ROOT/loaded-control-launchctl" + log="$TMP_ROOT/loaded-control-launchctl.log" + mkdir -p "$home/projects" "$home/config" "$home/user/.treehouse" "$agents" "$fakebin" "$fake_state" + write_stateful_launchctl_fake "$fakebin/launchctl" + HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + "$ROOT/bin/fm-checkout-refresh.sh" install >/dev/null + key=$(checkout_state_key "$home" 16) + label="com.firstmate.checkout-refresh.$key" + plist="$agents/$label.plist" + state_root="$state_base/homes/$key" + generation=$(plist_generation "$plist") + now=$(date +%s) + printf '%s\n' "$now" > "$state_root/heartbeat" + printf '%s\nhealthy\n' "$now" > "$state_root/coverage-health" + printf '%s\n' "$generation" > "$state_root/scheduler-generation" + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LAUNCHCTL_EXTRA_ENV="FM_PROJECTS_OVERRIDE=$TMP_ROOT/stale-projects" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "loaded LaunchAgent accepted an undeclared checkout control" + assert_contains "$out" "loaded identity is missing or untrusted" \ + "undeclared loaded LaunchAgent control was not surfaced before health" + + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LAUNCHCTL_INHERITED_ENV="FM_CONFIG_OVERRIDE=$TMP_ROOT/stale-config" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "loaded LaunchAgent accepted an inherited checkout control" + assert_contains "$out" "loaded identity is missing or untrusted" \ + "inherited loaded LaunchAgent control was not surfaced before health" + + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LAUNCHCTL_DEFAULT_ENV="GIT_CONFIG_PARAMETERS='remote.origin.url=/stale'" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "loaded LaunchAgent accepted a default Git control" + assert_contains "$out" "loaded identity is missing or untrusted" \ + "default loaded LaunchAgent control was not surfaced before health" + + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LAUNCHCTL_INHERITED_ENV="XDG_CONFIG_HOME=$TMP_ROOT/redirecting-git-config" \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "loaded LaunchAgent accepted an inherited Git config root" + assert_contains "$out" "loaded identity is missing or untrusted" \ + "inherited Git config root was not surfaced before health" + + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LAUNCHCTL_INTERVAL=1 \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "loaded LaunchAgent accepted a stale run interval" + assert_contains "$out" "loaded identity is missing or untrusted" \ + "loaded LaunchAgent interval drift was not surfaced before health" + + set +e + out=$(HOME="$home/user" FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_TEST_LAUNCHCTL_RUN_AT_LOAD=false \ + "$ROOT/bin/fm-checkout-refresh.sh" ensure 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "loaded LaunchAgent accepted disabled RunAtLoad" + assert_contains "$out" "loaded identity is missing or untrusted" \ + "loaded LaunchAgent RunAtLoad drift was not surfaced before health" + + legacy_home="$TMP_ROOT/untracked-legacy-home" + mkdir -p "$legacy_home/projects" "$legacy_home/config" "$legacy_home/user/.treehouse" + legacy_key=$(checkout_state_key "$legacy_home" 16) + legacy_plist="$agents/com.firstmate.checkout-refresh.$legacy_key.plist" + legacy_state_root="$state_base/homes/$legacy_key" + legacy_shadow="$TMP_ROOT/untracked-legacy-definition.plist" + write_launch_agent_fixture "$legacy_shadow" \ + "com.firstmate.checkout-refresh" "$legacy_home" "$legacy_state_root" + mark_launch_agent_loaded "$fake_state" "com.firstmate.checkout-refresh" "$legacy_shadow" + set +e + out=$(HOME="$legacy_home/user" FM_HOME="$legacy_home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_TREEHOUSE_ROOT="$legacy_home/user/.treehouse" \ + FM_CHECKOUT_REFRESH_PLATFORM=Darwin \ + FM_CHECKOUT_REFRESH_LAUNCH_AGENTS_DIR="$agents" \ + FM_CHECKOUT_REFRESH_LAUNCHCTL="$fakebin/launchctl" \ + FM_FAKE_LAUNCHCTL_STATE="$fake_state" \ + FM_FAKE_LAUNCHCTL_LOG="$log" \ + FM_TEST_LAUNCHCTL_BOOTOUT_FAIL_LABEL=com.firstmate.checkout-refresh \ + "$ROOT/bin/fm-checkout-refresh.sh" install 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "untracked live legacy LaunchAgent allowed replacement activation" + assert_absent "$legacy_plist" "replacement activated before untracked legacy absence was proven" + assert_present "$legacy_state_root/legacy-launch-agent.quarantine" \ + "untracked legacy LaunchAgent identity was not durably retained" + assert_contains "$out" "cannot quiesce checkout-refresh LaunchAgent" \ + "untracked legacy LaunchAgent bootout failure was not surfaced" + pass "loaded scheduler controls and untracked legacy jobs fail closed" +} + +test_same_path_replacement_and_manifest_failures_are_unhealthy() { + local home state checkout original_checkout first_remote second_remote out status + home="$TMP_ROOT/replacement-home" + state="$TMP_ROOT/replacement-state" + checkout="$TMP_ROOT/replacement-checkout" + original_checkout="$TMP_ROOT/replacement-checkout-original" + mkdir -p "$home/projects" "$home/config" "$home/user/.treehouse" "$state" + first_remote=$(build_origin replacement-first) + second_remote=$(build_origin replacement-second) + clone_from "$first_remote" "$checkout" + printf 'path %s\n' "$checkout" > "$home/config/checkout-refresh" + run_isolated_refresh "$home" "$state" run-once --force >/dev/null + mv "$checkout" "$original_checkout" + clone_from "$second_remote" "$checkout" + set +e + out=$(run_isolated_refresh "$home" "$state" run-once --force 2>&1) + set -e + assert_contains "$out" "covered checkout" \ + "same-path checkout replacement was not tied to prior coverage" + assert_contains "$out" "identity drifted" \ + "same-path checkout replacement was baselined instead of surfaced" + assert_refresh_state "$state" unhealthy + + rm -rf "$checkout" + clone_from "$first_remote" "$checkout" + set +e + out=$(run_isolated_refresh "$home" "$state" run-once --force 2>&1) + set -e + assert_contains "$out" "physical identity drifted" \ + "same-origin physical replacement was baselined instead of surfaced" + assert_refresh_state "$state" unhealthy + + set +e + out=$(run_manifest_failure_refresh append "$home" "$state" run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "manifest append failure remained healthy" + assert_refresh_state "$state" unhealthy + set +e + out=$(run_manifest_failure_refresh sort "$home" "$state" run-once --force 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "manifest sort failure remained healthy" + assert_refresh_state "$state" unhealthy + pass "same-path replacement and manifest failures invalidate coverage" +} + +test_legacy_identity_requires_physical_binding_and_migrates_transactionally() { + local home state checkout remote origin stable_key physical_key legacy_prefix stable_prefix out + home="$TMP_ROOT/legacy-identity-home" + state="$TMP_ROOT/legacy-identity-state" + checkout="$TMP_ROOT/legacy-identity-checkout" + mkdir -p "$home/projects" "$home/config" "$home/user/.treehouse" "$state" + remote=$(build_origin legacy-identity) + clone_from "$remote" "$checkout" + origin=$(git -C "$checkout" remote get-url origin) + printf 'path %s\n' "$checkout" > "$home/config/checkout-refresh" + stable_key=$(checkout_state_key "$checkout") + physical_key=$(fm_checkout_physical_path_key "$checkout" directory 24) + legacy_prefix="$state/aaaaaaaaaaaaaaaaaaaaaaaa" + [ "${legacy_prefix##*/}" != "$physical_key" ] || legacy_prefix="$state/bbbbbbbbbbbbbbbbbbbbbbbb" + printf '%s\norigin %s\n' "$checkout" "$origin" > "$legacy_prefix.identity" + printf 'legacy-tip\n' > "$legacy_prefix.tip" + set +e + out=$(run_isolated_refresh "$home" "$state" run-once --force 2>&1) + set -e + assert_contains "$out" "filename does not match the current physical checkout" \ + "unbindable legacy checkout identity did not surface its missing physical proof" + assert_present "$legacy_prefix.identity" "unbindable legacy identity history was rewritten" + assert_absent "$state/$stable_key.identity" "unbindable legacy identity was rebaselined" + assert_refresh_state "$state" unhealthy + + mv "$legacy_prefix.identity" "$state/$physical_key.identity" + mv "$legacy_prefix.tip" "$state/$physical_key.tip" + legacy_prefix="$state/$physical_key" + stable_prefix="$state/$stable_key" + set +e + out=$(FM_CHECKOUT_TEST_IDENTITY_MIGRATION_FAILURE=publish \ + run_isolated_refresh "$home" "$state" run-once --force 2>&1) + set -e + assert_present "$legacy_prefix.identity" "partial identity publish failure lost legacy identity" + assert_present "$legacy_prefix.tip" "partial identity publish failure lost legacy tip" + assert_absent "$stable_prefix.identity" "partial identity publish failure exposed destination identity" + assert_absent "$stable_prefix.tip" "partial identity publish failure exposed destination tip" + assert_refresh_state "$state" unhealthy + + set +e + out=$(FM_CHECKOUT_TEST_IDENTITY_MIGRATION_CRASH=after-first-publish \ + run_isolated_refresh "$home" "$state" run-once --force 2>&1) + set -e + assert_present "$legacy_prefix.identity" "interrupted identity migration hid authoritative legacy history" + assert_present "$stable_prefix.identity" "interrupted identity migration did not publish its first durable replacement" + assert_present "$stable_prefix.identity-migration" "interrupted identity migration did not retain its recovery journal" + assert_refresh_state "$state" unhealthy + + run_isolated_refresh "$home" "$state" run-once --force >/dev/null \ + || fail "interrupted physically bound legacy identity did not recover" + [ "$(awk 'END { print NR + 0 }' "$stable_prefix.identity")" -eq 3 ] \ + || fail "migrated legacy identity did not retain its physical binding" + assert_absent "$legacy_prefix.identity" "successful identity migration retained the legacy identity name" + assert_absent "$legacy_prefix.tip" "successful identity migration retained the legacy tip name" + assert_absent "$stable_prefix.identity-migration" "recovered identity migration retained its journal" + assert_refresh_state "$state" healthy + pass "legacy checkout identities require binding and publish transactionally" +} + +test_lock_key_failure_cannot_construct_a_shared_lock_path() { + local out status + set +e + out=$(bash -c ' + set -u + . "$1/bin/fm-checkout-lock-lib.sh" + fm_checkout_lock_key() { return 1; } + fm_checkout_lock_path "$1" "$2" + ' bash "$ROOT" "$TMP_ROOT/failed-lock-root" 2>&1) + status=$? + set -e + [ "$status" -ne 0 ] || fail "failed lock-key derivation returned a lock path" + [ -z "$out" ] || fail "failed lock-key derivation emitted a fallback lock path: $out" + pass "lock-key derivation failure cannot collapse onto a shared lock path" +} + +test_logical_lock_keys_survive_creation_and_atomic_replacement() { + local home registry replacement missing_home_key created_home_key missing_registry_key replaced_registry_key + home="$TMP_ROOT/aba-secondmate-home" + registry="$TMP_ROOT/aba-data/secondmates.md" + mkdir -p "$(dirname "$registry")" + missing_home_key=$(fm_checkout_stable_path_key "$home" directory 1 24) + missing_registry_key=$(fm_checkout_stable_path_key "$registry" file 1 24) + mkdir -p "$home" + printf 'initial\n' > "$registry" + created_home_key=$(fm_checkout_stable_path_key "$home" directory 1 24) + replacement="$registry.replacement" + printf 'replacement\n' > "$replacement" + mv "$replacement" "$registry" + replaced_registry_key=$(fm_checkout_stable_path_key "$registry" file 1 24) + [ "$missing_home_key" = "$created_home_key" ] || fail "home lifecycle lock key changed after home creation" + [ "$missing_registry_key" = "$replaced_registry_key" ] || fail "registry lock key changed after atomic replacement" + pass "logical lifecycle keys survive path creation and replacement" +} + +if [ "${FM_TEST_FOCUSED:-}" = review-round-14 ]; then + test_bounded_refresh_terminates_descendants + test_launch_agent_definition_is_home_scoped_with_scheduler_seam + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-durable-identity ]; then + test_launch_agent_definition_is_home_scoped_with_scheduler_seam + test_logical_home_state_migrates_and_ambiguity_fails_closed + test_launch_agent_label_and_custom_legacy_state_are_authoritative + test_loaded_launch_agent_controls_and_untracked_legacy_job_fail_closed + test_same_path_replacement_and_manifest_failures_are_unhealthy + test_legacy_identity_requires_physical_binding_and_migrates_transactionally + test_lock_key_failure_cannot_construct_a_shared_lock_path + test_logical_lock_keys_survive_creation_and_atomic_replacement + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-15 ]; then + test_lock_root_failure_invalidates_coverage_before_preparation + test_reinspection_failure_invalidates_coverage_health + test_scheduler_liveness_is_scheduler_owned + test_launch_agent_definition_is_home_scoped_with_scheduler_seam + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-16 ]; then + test_reinspection_failure_invalidates_coverage_health + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-13-safety ]; then + test_loaded_launch_agent_controls_and_untracked_legacy_job_fail_closed + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-6 ]; then + test_nested_active_project_invalidates_coverage_health + test_bounded_refresh_terminates_descendants + test_acquisition_honors_shared_checkout_lock + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-7 ]; then + test_discovery_rejects_nested_configured_and_scanned_paths + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-safety ]; then + test_config_and_external_identity_fail_closed + test_public_entrypoints_reject_nested_repository_paths + test_explicit_secondmate_home_requires_live_default_tip + test_session_mode_preserves_gone_branch_pruning + test_lock_owner_symlink_cannot_escape_state_directory + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-followups ]; then + test_unreadable_scan_root_invalidates_coverage_health + test_unreadable_scanned_origin_invalidates_coverage_health + test_failed_alert_persistence_forces_reinspection + test_local_authority_is_fully_inspected_and_tracks_origin_identity + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-provenance ]; then + test_discovery_provenance_failures_invalidate_coverage + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-symlinks ]; then + test_discovery_covers_projects_treehouse_external_and_config + test_treehouse_discovery_failure_invalidates_coverage_health + test_raw_treehouse_root_symlink_invalidates_coverage_health + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-authority ]; then + test_empty_treehouse_and_identity_tool_failures_fail_closed + test_config_git_metadata_and_non_git_races_fail_closed + test_public_entrypoints_reject_nested_repository_paths + exit 0 +fi + +test_discovery_covers_projects_treehouse_external_and_config +test_uninspectable_active_project_invalidates_coverage_health +test_nested_active_project_invalidates_coverage_health +test_discovery_rejects_nested_configured_and_scanned_paths +test_upstream_tip_signal_refreshes_between_firstmate_events +test_periodic_backstop_repairs_drift_without_a_new_tip +test_live_default_change_is_surfaced_without_switching_branches +test_skill_drafts_surface_on_every_probe_without_log_spam +test_preflight_rejects_hygiene_without_an_origin +test_treehouse_pool_skill_drafts_are_inventoried +test_ignored_skill_files_are_outside_the_collision_guard +test_pool_preflight_surfaces_dirty_worktrees_without_blocking_clean_selection +test_bootstrap_relays_hygiene_alerts +test_treehouse_discovery_failure_invalidates_coverage_health +test_raw_treehouse_root_symlink_invalidates_coverage_health +test_empty_treehouse_and_identity_tool_failures_fail_closed +test_config_git_metadata_and_non_git_races_fail_closed +test_skill_inventory_failure_preserves_alert_and_invalidates_coverage +test_lock_root_failure_invalidates_coverage_before_preparation +test_reinspection_failure_invalidates_coverage_health +test_scheduler_liveness_is_scheduler_owned +test_unreadable_scan_root_invalidates_coverage_health +test_unreadable_scanned_origin_invalidates_coverage_health +test_discovery_provenance_failures_invalidate_coverage +test_failed_alert_persistence_forces_reinspection +test_local_authority_is_fully_inspected_and_tracks_origin_identity +test_dirty_nondefault_and_diverged_checkouts_are_untouched +test_refresh_locks_recover_stale_owners_and_surface_contention +test_session_mode_preserves_gone_branch_pruning +test_config_and_external_identity_fail_closed +test_public_entrypoints_reject_nested_repository_paths +test_explicit_secondmate_home_requires_live_default_tip +test_lock_owner_symlink_cannot_escape_state_directory +test_worktree_freshness_verification_fails_closed +test_bounded_refresh_terminates_descendants +test_acquisition_honors_shared_checkout_lock +test_launch_agent_definition_is_home_scoped_with_scheduler_seam +test_logical_home_state_migrates_and_ambiguity_fails_closed +test_launch_agent_label_and_custom_legacy_state_are_authoritative +test_loaded_launch_agent_controls_and_untracked_legacy_job_fail_closed +test_same_path_replacement_and_manifest_failures_are_unhealthy +test_legacy_identity_requires_physical_binding_and_migrates_transactionally +test_lock_key_failure_cannot_construct_a_shared_lock_path +test_logical_lock_keys_survive_creation_and_atomic_replacement diff --git a/tests/fm-daemon.test.sh b/tests/fm-daemon.test.sh index 5b854f678bb..32f59c535d2 100755 --- a/tests/fm-daemon.test.sh +++ b/tests/fm-daemon.test.sh @@ -6,6 +6,8 @@ # fm-send swallow reporting, composer-pending ANSI parsing). The operator-visible # inject flow lives in fm-afk-inject-e2e and fm-wake-daemon-lifecycle-e2e. set -u +export FM_ORCA_TEST_LAB=firstmate-orca-test-lab-v1 +export FM_ORCA_TEST_AUTHORITY_CAPABILITIES=verified-v1 # shellcheck source=tests/wake-helpers.sh . "$(dirname "${BASH_SOURCE[0]}")/wake-helpers.sh" diff --git a/tests/fm-dispatch-select.test.sh b/tests/fm-dispatch-select.test.sh index 777b5492015..dedb3e73db4 100755 --- a/tests/fm-dispatch-select.test.sh +++ b/tests/fm-dispatch-select.test.sh @@ -309,8 +309,8 @@ SH pass "dispatch pins its parser and input tools without ambient PATH resolution" } -test_account_pool_summary_owns_provider_quota_choice() { - local fakebin af_log quota_marker out pooled +test_account_pool_defers_selection_to_direct_spawn() { + local fakebin af_log quota_marker out pooled err fakebin=$(fm_fakebin "$TMP_ROOT/agent-fleet-pools") af_log="$TMP_ROOT/agent-fleet-pools/calls.log" quota_marker="$TMP_ROOT/agent-fleet-pools/quota-called" @@ -323,13 +323,15 @@ SH chmod +x "$fakebin/quota-axi" pooled='[{"harness":"claude","model":"sonnet","account_pool":"claude-crew"},{"harness":"codex","model":"gpt-5","account_pool":"codex-crew"}]' out=$(FM_FAKE_AF_LOG="$af_log" FM_DISPATCH_AGENT_FLEET="$fakebin/agent-fleet" \ - FM_DISPATCH_QUOTA_AXI="$fakebin/quota-axi" "$ROOT/bin/fm-dispatch-select.sh" --select quota-balanced "$pooled") - [ "$out" = '{"harness":"codex","model":"gpt-5","account_pool":"codex-crew"}' ] \ - || fail "Agent Fleet pool headroom should choose codex, got: $out" - assert_grep 'pool status --pool claude-crew --provider claude' "$af_log" "claude pool summary was not queried" - assert_grep 'pool status --pool codex-crew --provider codex' "$af_log" "codex pool summary was not queried" + FM_DISPATCH_QUOTA_AXI="$fakebin/quota-axi" "$ROOT/bin/fm-dispatch-select.sh" --select quota-balanced "$pooled" \ + 2>"$TMP_ROOT/agent-fleet-pools/error.log") + err=$(cat "$TMP_ROOT/agent-fleet-pools/error.log") + [ "$out" = '{"harness":"claude","model":"sonnet","account_pool":"claude-crew"}' ] \ + || fail "pooled dispatch did not preserve ordered direct-routing activation: $out" + [ ! -e "$af_log" ] || fail "new pooled dispatch queried Agent Fleet: $(cat "$af_log")" [ ! -e "$quota_marker" ] || fail "account_pool selection consulted default-account quota-axi" - pass "account_pool quota-balanced selection consumes only Agent Fleet pool summaries" + assert_contains "$err" 'deferring account selection to spawn' "pooled dispatch did not explain direct selection ownership" + pass "account_pool dispatch defers account choice to direct spawn selection" } test_degraded_pool_summary_is_diagnostic_only() { @@ -412,7 +414,7 @@ test_fully_pooled_dispatch_ignores_overridden_ambient_mode() { out=$(FM_ACCOUNT_ROUTING=malformed FM_DISPATCH_AGENT_FLEET="$fakebin/agent-fleet" \ "$ROOT/bin/fm-dispatch-select.sh" --select quota-balanced "$pooled") \ || fail "fully pooled dispatch parsed overridden ambient routing policy" - [ "$out" = '{"harness":"codex","account_pool":"codex-crew"}' ] \ + [ "$out" = '{"harness":"claude","account_pool":"claude-crew"}' ] \ || fail "fully pooled dispatch returned the wrong selection: $out" mixed='[{"harness":"claude","account_pool":"claude-crew"},{"harness":"codex"}]' @@ -526,6 +528,12 @@ SH pass "pool-summary failures degrade safely while every pinned quota-balanced candidate is rejected" } +if [ "${FM_TEST_FOCUSED:-}" = account-directory-cutover ]; then + test_account_pool_defers_selection_to_direct_spawn + test_fully_pooled_dispatch_ignores_overridden_ambient_mode + exit 0 +fi + if [ "${FM_TEST_FOCUSED:-}" = review-round-13 ]; then test_enforced_quota_balancing_rejects_poolless_candidates test_fully_pooled_dispatch_ignores_overridden_ambient_mode @@ -540,17 +548,10 @@ test_bad_quota_json_falls_back_to_first test_stale_with_cache_needs_clear_margin_to_beat_fresh test_vendor_absent_or_unusable_falls_back_conservatively test_backward_compatible_first_selection -test_account_pool_summary_owns_provider_quota_choice -test_degraded_pool_summary_is_diagnostic_only -test_account_pool_query_timeout_falls_back -test_slow_valid_pool_status_uses_selection_class_timeout -test_invalid_pool_status_timeout_is_rejected +test_account_pool_defers_selection_to_direct_spawn test_dispatch_ignores_hostile_path_jq_and_dirname test_enforced_quota_balancing_rejects_poolless_candidates test_fully_pooled_dispatch_ignores_overridden_ambient_mode -test_agent_fleet_binary_precedence_matches_routing -test_production_dispatch_override_is_never_executed test_account_fields_survive_direct_selection -test_pooled_failures_degrade_without_default_account_quota echo "# all fm-dispatch-select tests passed" diff --git a/tests/fm-fleet-sync.test.sh b/tests/fm-fleet-sync.test.sh index b1fcd0a38e2..d31c7e052d1 100755 --- a/tests/fm-fleet-sync.test.sh +++ b/tests/fm-fleet-sync.test.sh @@ -11,6 +11,8 @@ # instead of a quiet skip. # The pre-existing fast-forward / already-current / local-only / no-origin paths # must be unchanged, and bootstrap must relay the new outcomes as FLEET_SYNC lines. +# Every origin-backed path must use the live upstream default and the shared +# canonical repository lock, including callers outside checkout-refresh. # # It also pins the orphaned .git/packed-refs.lock recovery in the fetch step # (fetch_with_packed_refs_lock_guard, backed by bin/fm-lock-lib.sh's shared @@ -81,13 +83,25 @@ advance_origin() { git -C "$work" push -q origin main } +switch_origin_default() { + local home=$1 name=$2 work remote + work="$home/work-$name" + remote="$home/remotes/$name.git" + git -C "$work" checkout -q -b trunk + commit_file "$work" trunk.txt trunk default-trunk + git -C "$work" push -q -u origin trunk + git -C "$remote" symbolic-ref HEAD refs/heads/trunk +} + head_sha() { git -C "$1" rev-parse HEAD; } # run_sync <home> [args...]: run fleet-sync against an isolated home, stdout only. run_sync() { local home=$1 shift - FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" "$ROOT/bin/fm-fleet-sync.sh" "$@" 2>/dev/null + FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$home/checkout-refresh-state" \ + "$ROOT/bin/fm-fleet-sync.sh" "$@" 2>/dev/null } # --- packed-refs.lock fixtures ---------------------------------------------- @@ -191,11 +205,137 @@ run_sync_guarded() { realgit=$(command -v git) PATH="$fakebin:$PATH" REAL_GIT_FOR_TEST="$realgit" \ FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$home/checkout-refresh-state" \ "$ROOT/bin/fm-fleet-sync.sh" "$@" >"$outf" 2>"$errf" } # --- tests ------------------------------------------------------------------ +test_status_failure_is_never_treated_as_clean() { + local home clone fakebin out err before + home=$(new_home) + clone=$(build_pair "$home" status-unreadable) + advance_origin "$home" status-unreadable C1 + before=$(head_sha "$clone") + fakebin="$home/fakebin" + out="$home/status-unreadable.out" + err="$home/status-unreadable.err" + mkdir -p "$fakebin" + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +for argument in "$@"; do + if [ "$argument" = status ]; then + exit 74 + fi +done +exec "${REAL_GIT_FOR_TEST:?}" "$@" +SH + chmod +x "$fakebin/git" + run_sync_guarded "$home" "$fakebin" "$out" "$err" "$clone" + assert_contains "$(cat "$out")" "working tree cleanliness cannot be inspected" \ + "status failure was not surfaced" + [ "$(head_sha "$clone")" = "$before" ] \ + || fail "status failure was treated as clean and advanced the checkout" + pass "unreadable git status cannot produce a clean refresh" +} + +test_direct_and_batch_sync_reject_nested_repository_paths() { + local home clone nested out before status batch_home batch_nested batch_out batch_before batch_status + home=$(new_home) + clone=$(build_pair "$home" nested-direct) + advance_origin "$home" nested-direct C1 + nested="$clone/child" + mkdir -p "$nested" + before=$(head_sha "$clone") + + out=$(run_sync "$home" "$nested") + status=$? + + assert_contains "$out" "target must be an exact canonical Git repository root" \ + "direct fleet sync accepted a nested repository path" + [ "$status" -ne 0 ] || fail "direct fleet sync reported success for a nested repository path" + [ "$(head_sha "$clone")" = "$before" ] || fail "direct nested target mutated its enclosing repository" + + batch_home=$(new_home) + fm_git_init_commit "$batch_home/projects" + batch_nested="$batch_home/projects/child" + mkdir -p "$batch_nested" + batch_before=$(head_sha "$batch_home/projects") + + batch_out=$(run_sync "$batch_home") + batch_status=$? + + assert_contains "$batch_out" "target must be an exact canonical Git repository root" \ + "batch fleet sync accepted a nested repository path" + [ "$batch_status" -ne 0 ] || fail "batch fleet sync reported success for a nested repository path" + [ "$(head_sha "$batch_home/projects")" = "$batch_before" ] \ + || fail "batch nested target mutated its enclosing repository" + pass "direct and batch sync reject nested repository paths" +} + +test_direct_and_batch_sync_reject_symlink_repository_paths() { + local home clone alias ancestor alias_parent out before status batch_home batch_clone batch_alias batch_out batch_before batch_status projects_target projects_link + home=$(new_home) + clone=$(build_pair "$home" symlink-direct) + advance_origin "$home" symlink-direct C1 + alias="$home/symlink-direct" + ln -s "$clone" "$alias" + before=$(head_sha "$clone") + + out=$(run_sync "$home" "$alias") + status=$? + + assert_contains "$out" "target must be an exact canonical Git repository root" \ + "direct fleet sync accepted a symlink repository path" + [ "$status" -ne 0 ] || fail "direct fleet sync reported success for a symlink repository path" + [ "$(head_sha "$clone")" = "$before" ] || fail "direct symlink target mutated its repository" + + ancestor="$home/ancestor-link" + alias_parent=$(dirname "$clone") + ln -s "$alias_parent" "$ancestor" + set +e + out=$(run_sync "$home" "$ancestor/$(basename "$clone")/") + status=$? + set -e + [ "$status" -ne 0 ] || fail "direct fleet sync accepted a symlinked ancestor with a trailing slash" + [ "$(head_sha "$clone")" = "$before" ] || fail "ancestor-symlinked direct target mutated its repository" + + batch_home=$(new_home) + batch_clone=$(build_pair "$batch_home" symlink-batch-target) + advance_origin "$batch_home" symlink-batch-target C1 + mv "$batch_clone" "$batch_home/symlink-batch-target" + batch_clone="$batch_home/symlink-batch-target" + batch_alias="$batch_home/projects/symlink-batch" + ln -s "$batch_clone" "$batch_alias" + batch_before=$(head_sha "$batch_clone") + + set +e + batch_out=$(run_sync "$batch_home") + batch_status=$? + set -e + + assert_contains "$batch_out" "target must be an exact canonical Git repository root" \ + "batch fleet sync accepted a symlinked projects entry" + [ "$batch_status" -ne 0 ] || fail "batch fleet sync reported success for a symlinked projects entry" + [ "$(head_sha "$batch_clone")" = "$batch_before" ] \ + || fail "batch symlink target mutated its external repository" + + projects_target="$batch_home/projects-target" + projects_link="$batch_home/projects-root-link" + mkdir -p "$projects_target" + ln -s "$projects_target" "$projects_link" + set +e + batch_out=$(FM_HOME="$batch_home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_PROJECTS_OVERRIDE="$projects_link/" \ + "$ROOT/bin/fm-fleet-sync.sh" 2>&1) + batch_status=$? + set -e + [ "$batch_status" -ne 0 ] || fail "batch fleet sync accepted a symlinked projects root" + assert_contains "$batch_out" "projects root contains an unsafe" \ + "symlinked projects root refusal was unclear" + pass "direct and batch sync reject leaf, ancestor, and projects-root symlinks" +} + test_detached_clean_ancestor_recovers() { local home clone out before after home=$(new_home) @@ -259,20 +399,26 @@ test_detached_clean_ancestor_with_diverged_local_default_is_stuck_untouched() { } test_dirty_is_stuck_untouched() { - local home clone out before + local home clone out before draft home=$(new_home) clone=$(build_pair "$home" gamma) advance_origin "$home" gamma C1 before=$(head_sha "$clone") printf 'uncommitted edit\n' >> "$clone/file.txt" + draft="$clone/.agents/skills/local-draft/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# local draft' > "$draft" out=$(run_sync "$home" "$clone") assert_contains "$out" "gamma: STUCK:" "dirty clone reports STUCK" assert_contains "$out" "uncommitted changes" "STUCK names the dirty state" + assert_contains "$out" "1 untracked, 1 under repository skill directories" \ + "STUCK quantifies untracked skill drafts" assert_contains "$out" "1 commits behind origin/main" "STUCK quantifies how far behind" [ "$(head_sha "$clone")" = "$before" ] || fail "dirty clone HEAD was moved" grep -q "uncommitted edit" "$clone/file.txt" || fail "dirty working-tree change was discarded" + grep -q "# local draft" "$draft" || fail "untracked skill draft was discarded" pass "dirty working tree is reported STUCK and left untouched" } @@ -325,6 +471,129 @@ test_on_default_clean_behind_fast_forwards() { pass "on-default clean behind clone still fast-forwards" } +test_live_default_probe_overrides_stale_origin_head() { + local home clone out before stale + home=$(new_home) + clone=$(build_pair "$home" live-default) + advance_origin "$home" live-default C1 + switch_origin_default "$home" live-default + before=$(head_sha "$clone") + stale=$(git -C "$clone" symbolic-ref --short refs/remotes/origin/HEAD) + [ "$stale" = "origin/main" ] || fail "live-default fixture did not retain stale origin/HEAD" + + out=$(run_sync "$home" "$clone") + + assert_contains "$out" "live-default: STUCK: on branch main" \ + "direct fleet sync did not use the live upstream default" + assert_contains "$out" "commits behind origin/trunk" \ + "live-default warning did not identify the authoritative branch" + [ "$(head_sha "$clone")" = "$before" ] || fail "former default branch was fast-forwarded" + [ "$(git -C "$clone" branch --show-current)" = "main" ] || fail "former default checkout was switched" + [ "$(git -C "$clone" symbolic-ref --short refs/remotes/origin/HEAD)" = "origin/main" ] \ + || fail "fixture's stale origin/HEAD unexpectedly changed" + pass "direct sync proves the live upstream default and leaves the former default untouched" +} + +test_direct_sync_honors_shared_checkout_lock() { + local home clone before common key lock_root lock out + home=$(new_home) + clone=$(build_pair "$home" shared-lock) + advance_origin "$home" shared-lock C1 + before=$(head_sha "$clone") + common=$(git -C "$clone" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$clone/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(printf '%s' "$common" | shasum -a 256 | awk '{print substr($1,1,24)}') + lock_root="$home/checkout-locks" + lock="$lock_root/$key.lock" + mkdir -p "$lock" + printf '%s\n' "$$" > "$lock/pid" + + out=$(FM_CHECKOUT_REFRESH_LOCK_ROOT="$lock_root" \ + FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + "$ROOT/bin/fm-fleet-sync.sh" "$clone" 2>/dev/null) + + assert_contains "$out" "$clone: skipped: refresh already running (pid $$)" \ + "direct fleet sync bypassed the shared checkout lock" + [ "$(head_sha "$clone")" = "$before" ] || fail "direct sync mutated a contended checkout" + rm -rf "$lock" + pass "direct sync serializes through the shared canonical checkout lock" +} + +test_direct_sync_timeout_terminates_descendants() { + local home clone fakebin real_git out status parent_pid child_pid common key lock_root lock + home=$(new_home) + clone=$(build_pair "$home" direct-timeout) + advance_origin "$home" direct-timeout C1 + fakebin="$home/direct-timeout-fakebin" + real_git=$(command -v git) + mkdir -p "$fakebin" + common=$(git -C "$clone" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$clone/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(printf '%s' "$common" | shasum -a 256 | awk '{print substr($1,1,24)}') + lock_root="$home/checkout-refresh-state/locks" + lock="$lock_root/$key.lock" + cat > "$fakebin/git" <<'SH' +#!/usr/bin/env bash +is_fetch=0 +for arg in "$@"; do + [ "$arg" = fetch ] && is_fetch=1 +done +if [ "$is_fetch" -eq 1 ]; then + if { [ -e "$FM_TEST_EXPECT_LOCK" ] || [ -L "$FM_TEST_EXPECT_LOCK" ]; } \ + && lock_pid=$(cat "$FM_TEST_EXPECT_LOCK/pid" 2>/dev/null) \ + && kill -0 "$lock_pid" 2>/dev/null; then + : > "$FM_TEST_LOCK_BEFORE_MUTATION" + fi + trap ' + if [ -e "$FM_TEST_EXPECT_LOCK" ] || [ -L "$FM_TEST_EXPECT_LOCK" ]; then + : > "$FM_TEST_LOCK_DURING_CLEANUP" + fi + ' TERM + printf '%s\n' "$BASHPID" > "${FM_TEST_FETCH_PARENT:?}" + ( + trap '' TERM + printf '%s\n' "$BASHPID" > "${FM_TEST_FETCH_CHILD:?}" + while :; do sleep 0.1; done + ) & + while :; do wait || true; done +fi +exec "${FM_TEST_REAL_GIT:?}" "$@" +SH + chmod +x "$fakebin/git" + + set +e + out=$(FM_TEST_REAL_GIT="$real_git" \ + FM_TEST_FETCH_PARENT="$home/direct-fetch-parent.pid" \ + FM_TEST_FETCH_CHILD="$home/direct-fetch-child.pid" \ + FM_TEST_EXPECT_LOCK="$lock" \ + FM_TEST_LOCK_BEFORE_MUTATION="$home/lock-before-mutation" \ + FM_TEST_LOCK_DURING_CLEANUP="$home/lock-during-cleanup" \ + FM_CHECKOUT_REFRESH_SYNC_TIMEOUT=1 \ + FM_HOME="$home" FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_STATE_BASE="$home/checkout-refresh-state" \ + PATH="$fakebin:$PATH" \ + "$ROOT/bin/fm-fleet-sync.sh" "$clone" 2>&1) + status=$? + set -e + + [ "$status" -eq 0 ] || fail "direct bounded fleet sync failed unexpectedly: $out" + assert_contains "$out" "direct-timeout: skipped: refresh timed out after 1s" \ + "direct fleet sync did not surface its process-tree timeout" + parent_pid=$(cat "$home/direct-fetch-parent.pid") + child_pid=$(cat "$home/direct-fetch-child.pid") + if kill -0 "$parent_pid" 2>/dev/null || kill -0 "$child_pid" 2>/dev/null; then + fail "direct fleet sync returned while a fetch descendant was still alive" + fi + assert_present "$home/lock-before-mutation" \ + "direct fleet sync started mutation before the supervising process acquired the checkout lock" + assert_present "$home/lock-during-cleanup" \ + "direct fleet sync released the checkout lock before descendant cleanup" + assert_absent "$lock" "direct fleet sync retained its checkout lock after verified cleanup" + pass "direct fleet sync bounds and reaps its mutation process tree" +} + test_already_current_unchanged() { local home clone out before home=$(new_home) @@ -370,6 +639,26 @@ test_local_only_skipped() { pass "local-only clone is skipped (benign), not flagged STUCK" } +test_external_path_honors_basename_delivery_mode() { + local home clone external out before remote + home=$(new_home) + clone=$(build_pair "$home" external-mode) + remote=$(git -C "$clone" remote get-url origin) + external="$home/outside/external-mode" + mkdir -p "$(dirname "$external")" + git clone --quiet "$remote" "$external" + advance_origin "$home" external-mode C1 + mkdir -p "$home/data" + printf -- '- external-mode [local-only] - test project (added 2026-07-23)\n' > "$home/data/projects.md" + before=$(head_sha "$external") + + out=$(run_sync "$home" "$external") + + assert_contains "$out" "skipped: local-only project" "external clone ignored the basename registry mode" + [ "$(head_sha "$external")" = "$before" ] || fail "external local-only clone was advanced" + pass "external clone paths honor the basename project delivery mode" +} + test_single_project_by_bare_name_resolves() { local home out home=$(new_home) @@ -434,6 +723,34 @@ test_single_project_unresolvable_name_still_skips() { pass "single-project form leaves a genuinely bad name unresolved" } +test_parent_relative_exact_root_and_lock_identity_failure() { + local home clone caller out before status + home=$(new_home) + clone=$(build_pair "$home" parent-relative) + advance_origin "$home" parent-relative C1 + caller="$home/caller" + mkdir -p "$caller" + out=$( + cd "$caller" || exit 1 + run_sync "$home" ../projects/parent-relative + ) + assert_contains "$out" "parent-relative: synced" \ + "parent-relative exact repository root was rejected" + [ "$(head_sha "$clone")" = "$(git -C "$clone" rev-parse origin/main)" ] \ + || fail "parent-relative fleet sync did not reach origin/main" + + before=$(head_sha "$clone") + set +e + FM_CHECKOUT_TEST_DISABLE_SYSTEM_PERL=1 bash -c \ + '. "$1"; fm_checkout_lock_path "$2" "$3" >/dev/null' \ + _ "$ROOT/bin/fm-checkout-lock-lib.sh" "$clone" "$home/locks" + status=$? + set -e + [ "$status" -ne 0 ] || fail "checkout lock identity succeeded without its fixed identity tool" + [ "$(head_sha "$clone")" = "$before" ] || fail "lock identity failure mutated the checkout" + pass "parent-relative roots work and lock identity failures fail closed" +} + test_whole_fleet_form() { local home behind current out home=$(new_home) @@ -603,6 +920,65 @@ test_non_signature_fetch_failure_is_not_retried() { pass "a non-packed-refs.lock fetch failure keeps today's behavior (no retry)" } +test_expected_origin_is_rechecked_inside_mutation_lock() { + local home clone original replacement before out + home=$(new_home) + clone=$(build_pair "$home" origin-lock-proof) + replacement=$(build_pair "$home" origin-lock-replacement) + original=$(git -C "$clone" remote get-url origin) + before=$(head_sha "$clone") + out=$(FM_FLEET_SYNC_EXPECTED_ORIGIN_KIND=origin \ + FM_FLEET_SYNC_EXPECTED_ORIGIN_VALUE="$original" \ + FM_FLEET_SYNC_TEST=1 \ + FM_FLEET_SYNC_TEST_DRIFT_ORIGIN_TO="$(git -C "$replacement" remote get-url origin)" \ + run_sync "$home" "$clone") + assert_contains "$out" "checkout repository or origin identity drifted before mutation" \ + "origin drift under the checkout lock was not surfaced" + [ "$(head_sha "$clone")" = "$before" ] || fail "origin drift under lock mutated the checkout" + pass "expected origin is rechecked under the checkout mutation lock" +} + +if [ "${FM_TEST_FOCUSED:-}" = review-round-6 ]; then + test_live_default_probe_overrides_stale_origin_head + test_direct_sync_honors_shared_checkout_lock + test_direct_sync_timeout_terminates_descendants + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-12-ownership ]; then + test_direct_sync_honors_shared_checkout_lock + test_direct_sync_timeout_terminates_descendants + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-safety ]; then + test_status_failure_is_never_treated_as_clean + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-followups ]; then + test_direct_and_batch_sync_reject_nested_repository_paths + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-symlinks ]; then + test_direct_and_batch_sync_reject_symlink_repository_paths + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-authority ]; then + test_parent_relative_exact_root_and_lock_identity_failure + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-durable-identity ]; then + test_expected_origin_is_rechecked_inside_mutation_lock + exit 0 +fi + +test_status_failure_is_never_treated_as_clean +test_direct_and_batch_sync_reject_nested_repository_paths +test_direct_and_batch_sync_reject_symlink_repository_paths test_detached_clean_ancestor_recovers test_detached_unique_commit_is_stuck_untouched test_detached_clean_ancestor_with_diverged_local_default_is_stuck_untouched @@ -610,14 +986,19 @@ test_dirty_is_stuck_untouched test_non_default_branch_is_stuck_untouched test_diverged_is_stuck_untouched test_on_default_clean_behind_fast_forwards +test_live_default_probe_overrides_stale_origin_head +test_direct_sync_honors_shared_checkout_lock +test_direct_sync_timeout_terminates_descendants test_already_current_unchanged test_no_origin_skipped test_local_only_skipped +test_external_path_honors_basename_delivery_mode test_single_project_by_bare_name_resolves test_single_project_by_bare_name_ignores_cwd_shadow test_single_project_by_projects_relative_name_resolves test_single_project_by_projects_relative_name_ignores_cwd_shadow test_single_project_unresolvable_name_still_skips +test_parent_relative_exact_root_and_lock_identity_failure test_whole_fleet_form test_bootstrap_relays_recovered_and_stuck test_orphaned_stale_packed_refs_lock_recovers @@ -625,3 +1006,4 @@ test_live_packed_refs_lock_is_never_removed test_live_git_cwd_in_clone_dir_blocks_removal test_transient_packed_refs_lock_self_clears test_non_signature_fetch_failure_is_not_retried +test_expected_origin_is_rechecked_inside_mutation_lock diff --git a/tests/fm-secondmate-safety.test.sh b/tests/fm-secondmate-safety.test.sh index 2c8a96d785f..f1c956e44dd 100755 --- a/tests/fm-secondmate-safety.test.sh +++ b/tests/fm-secondmate-safety.test.sh @@ -13,6 +13,30 @@ set -u TMP_ROOT=$(fm_test_tmproot fm-secondmate-safety) export FM_BACKEND=tmux +make_live_default_firstmate_worktree() { + local destination=$1 name=$2 remote remote_abs head source + remote="$TMP_ROOT/remotes/$name.git" + source="$TMP_ROOT/sources/$name" + mkdir -p "$TMP_ROOT/remotes" + git clone --quiet --bare "$ROOT" "$remote" + head=$(git -C "$ROOT" rev-parse HEAD) + git -C "$remote" update-ref refs/heads/main "$head" + git -C "$remote" symbolic-ref HEAD refs/heads/main + remote_abs=$(cd "$remote" && pwd -P) + mkdir -p "$TMP_ROOT/sources" + git clone --quiet "file://$remote_abs" "$source" + git -C "$source" worktree add --quiet --detach "$destination" main + printf '%s\n' "$source" +} + +write_secondmate_registration() { + local home=$1 id=$2 target=$3 target_abs + mkdir -p "$home/data" + target_abs=$(cd "$target" && pwd -P) || return 1 + printf -- '- %s - test domain (home: %s; scope: test domain; projects: ; added 2026-07-23)\n' \ + "$id" "$target_abs" > "$home/data/secondmates.md" +} + test_fm_home_parameterization() { local brief home_one home_two out @@ -117,7 +141,7 @@ EOF if FM_HOME="$home" "$ROOT/bin/fm-home-seed.sh" validate >/dev/null 2>"$err"; then fail "registry validation accepted two secondmates with the same home" fi - grep -F 'duplicate secondmate home assignment' "$err" >/dev/null \ + grep -F 'secondmate registry is malformed, duplicated, redirected, or uninspectable' "$err" >/dev/null \ || fail "registry validation did not explain duplicate home assignment" pass "home seed validation rejects duplicate home routes" } @@ -139,7 +163,7 @@ EOF if FM_HOME="$home" "$ROOT/bin/fm-home-seed.sh" validate >/dev/null 2>"$err"; then fail "registry validation accepted two homes for the same secondmate id" fi - grep -F 'duplicate secondmate id assignment' "$err" >/dev/null \ + grep -F 'secondmate registry is malformed, duplicated, redirected, or uninspectable' "$err" >/dev/null \ || fail "registry validation did not explain duplicate id assignment" pass "home seed validation rejects duplicate id routes" } @@ -161,26 +185,44 @@ EOF if FM_HOME="$home" "$ROOT/bin/fm-home-seed.sh" validate >/dev/null 2>"$err"; then fail "registry validation accepted nested secondmate homes" fi - grep -F 'overlapping secondmate home assignment' "$err" >/dev/null \ + grep -F 'secondmate registry is malformed, duplicated, redirected, or uninspectable' "$err" >/dev/null \ || fail "registry validation did not explain nested home assignment" pass "home seed validation rejects nested home routes" } +test_home_seed_validate_rejects_partial_registry_rows() { + local home registered err + home="$TMP_ROOT/partial-registry-home" + registered="$TMP_ROOT/partial-registry-target" + err="$TMP_ROOT/partial-registry.err" + mkdir -p "$home/data" "$registered" + printf '%s\n' "- partial - incomplete (home: $registered; scope: partial; added 2026-07-23)" \ + > "$home/data/secondmates.md" + + if FM_HOME="$home" "$ROOT/bin/fm-home-seed.sh" validate >/dev/null 2>"$err"; then + fail "registry validation accepted an entry with a missing projects field" + fi + assert_grep 'secondmate registry is malformed, duplicated, redirected, or uninspectable' "$err" \ + "partial registry row was not surfaced" + pass "home seed validation rejects partial registry rows" +} + test_home_seed_uses_treehouse_acquired_home() { - local home acquired acquired_abs fakebin log lease out + local home acquired acquired_abs fakebin log lease out source home="$TMP_ROOT/dash-home" acquired="$TMP_ROOT/dash-acquired-home" mkdir -p "$home/projects" "$home/data" "$home/state" fm_git_init_commit "$home/projects/alpha" fm_git_add_origin "$home/projects/alpha" "$TMP_ROOT/remotes/dash-alpha.git" printf '%s\n' '- alpha [direct-PR] - alpha project (added 2026-06-22)' > "$home/data/projects.md" - git clone --quiet "$ROOT" "$acquired" + source=$(make_live_default_firstmate_worktree "$acquired" dash-firstmate) fakebin=$(make_fake_tmux "$TMP_ROOT/dash-fake") log="$TMP_ROOT/dash-fake/tmux.log" lease="$TMP_ROOT/dash-fake/lease" out=$(PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TREEHOUSE_HOME="$acquired" FM_FAKE_TMUX_LOG="$log" \ FM_FAKE_TREEHOUSE_LEASE_FILE="$lease" \ + FM_ROOT_OVERRIDE="$source" \ FM_SECONDMATE_CHARTER='dash acquired scope' FM_SECONDMATE_SCOPE='dash acquired scope' \ "$ROOT/bin/fm-home-seed.sh" dash - alpha) \ || fail "seed failed for a treehouse-acquired home" @@ -196,8 +238,164 @@ test_home_seed_uses_treehouse_acquired_home() { pass "home seeding durably leases treehouse-acquired dash homes under the secondmate id" } +test_home_seed_acquisition_honors_shared_checkout_lock() { + local home acquired fakebin log err source common key lock_root lock state_root + home="$TMP_ROOT/dash-lock-home" + acquired="$TMP_ROOT/dash-lock-acquired-home" + err="$TMP_ROOT/dash-lock.err" + mkdir -p "$home/projects" "$home/data" "$home/state" + fm_git_init_commit "$home/projects/alpha" + fm_git_add_origin "$home/projects/alpha" "$TMP_ROOT/remotes/dash-lock-alpha.git" + printf '%s\n' '- alpha [direct-PR] - alpha project (added 2026-06-22)' > "$home/data/projects.md" + source=$(make_live_default_firstmate_worktree "$acquired" dash-lock-firstmate) + fakebin=$(make_fake_tmux "$TMP_ROOT/dash-lock-fake") + log="$TMP_ROOT/dash-lock-fake/tmux.log" + state_root="$TMP_ROOT/dash-lock-state" + lock_root="$TMP_ROOT/dash-lock-locks" + common=$(git -C "$source" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$source/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(printf '%s' "$common" | shasum -a 256 | awk '{print substr($1,1,24)}') + lock="$lock_root/$key.lock" + mkdir -p "$lock" "$state_root" + printf '%s\n' "$$" > "$lock/pid" + + if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TREEHOUSE_HOME="$acquired" \ + FM_FAKE_TMUX_LOG="$log" FM_ROOT_OVERRIDE="$source" \ + FM_CHECKOUT_REFRESH_STATE_ROOT="$state_root" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$lock_root" \ + FM_SECONDMATE_CHARTER='dash lock scope' FM_SECONDMATE_SCOPE='dash lock scope' \ + "$ROOT/bin/fm-home-seed.sh" dash-lock - alpha >/dev/null 2>"$err"; then + fail "secondmate acquisition bypassed the shared checkout lock" + fi + + grep -F "Treehouse acquisition already running for $source (pid $$)" "$err" >/dev/null \ + || fail "secondmate acquisition did not surface shared-lock contention" + if [ -f "$log" ] && grep -F 'treehouse get' "$log" >/dev/null; then + fail "secondmate acquisition invoked Treehouse while the shared lock was held" + fi + pass "secondmate acquisition uses the common locked entrypoint" +} + +test_home_seed_rejects_stale_treehouse_acquired_home() { + local home acquired acquired_abs fakebin log err source before + home="$TMP_ROOT/dash-stale-home" + acquired="$TMP_ROOT/dash-stale-acquired-home" + err="$TMP_ROOT/dash-stale.err" + mkdir -p "$home/projects" "$home/data" "$home/state" + fm_git_init_commit "$home/projects/alpha" + fm_git_add_origin "$home/projects/alpha" "$TMP_ROOT/remotes/dash-stale-alpha.git" + printf '%s\n' '- alpha [direct-PR] - alpha project (added 2026-06-22)' > "$home/data/projects.md" + source=$(make_live_default_firstmate_worktree "$acquired" dash-stale-firstmate) + before=$(git -C "$acquired" rev-parse HEAD) + printf '%s\n' upstream > "$source/upstream.txt" + git -C "$source" add upstream.txt + git -C "$source" -c user.name='Firstmate Tests' -c user.email='tests@example.invalid' \ + commit -qm upstream + git -C "$source" push -q origin main + acquired_abs=$(cd "$acquired" && pwd -P) + fakebin=$(make_fake_tmux "$TMP_ROOT/dash-stale-fake") + log="$TMP_ROOT/dash-stale-fake/tmux.log" + + if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TREEHOUSE_HOME="$acquired" FM_FAKE_TMUX_LOG="$log" \ + FM_ROOT_OVERRIDE="$source" \ + FM_SECONDMATE_CHARTER='dash stale scope' FM_SECONDMATE_SCOPE='dash stale scope' \ + "$ROOT/bin/fm-home-seed.sh" dash-stale - alpha >/dev/null 2>"$err"; then + fail "seed accepted a stale Treehouse-acquired secondmate home" + fi + + grep -F 'acquired worktree is stale' "$err" >/dev/null \ + || fail "stale secondmate-home refusal did not identify the upstream mismatch" + grep -F 'retaining unsafe treehouse-acquired home' "$err" >/dev/null \ + || fail "stale secondmate-home refusal did not surface retain-only cleanup" + grep -F "treehouse return --force $acquired_abs" "$log" >/dev/null \ + && fail "stale secondmate-home refusal force-returned an unverifiable lease" + [ -d "$acquired" ] || fail "stale secondmate-home refusal removed the acquired worktree" + [ "$(git -C "$acquired" rev-parse HEAD)" = "$before" ] \ + || fail "stale secondmate-home refusal changed the acquired tip" + if [ -f "$home/data/secondmates.md" ] && grep -F -- '- dash-stale ' "$home/data/secondmates.md" >/dev/null; then + fail "stale secondmate-home refusal wrote a registry route" + fi + pass "stale secondmate acquisitions remain leased without destructive rollback" +} + +test_home_seed_retains_dirty_treehouse_acquired_home() { + local home acquired acquired_abs fakebin log err source draft + home="$TMP_ROOT/dash-dirty-home" + acquired="$TMP_ROOT/dash-dirty-acquired-home" + err="$TMP_ROOT/dash-dirty.err" + mkdir -p "$home/projects" "$home/data" "$home/state" + fm_git_init_commit "$home/projects/alpha" + fm_git_add_origin "$home/projects/alpha" "$TMP_ROOT/remotes/dash-dirty-alpha.git" + printf '%s\n' '- alpha [direct-PR] - alpha project (added 2026-06-22)' > "$home/data/projects.md" + source=$(make_live_default_firstmate_worktree "$acquired" dash-dirty-firstmate) + acquired_abs=$(cd "$acquired" && pwd -P) + draft="$acquired/.agents/skills/unlanded/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# unlanded secondmate work' > "$draft" + rm "$acquired/AGENTS.md" + fakebin=$(make_fake_tmux "$TMP_ROOT/dash-dirty-fake") + log="$TMP_ROOT/dash-dirty-fake/tmux.log" + + if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TREEHOUSE_HOME="$acquired" FM_FAKE_TMUX_LOG="$log" \ + FM_ROOT_OVERRIDE="$source" \ + FM_SECONDMATE_CHARTER='dash dirty scope' FM_SECONDMATE_SCOPE='dash dirty scope' \ + "$ROOT/bin/fm-home-seed.sh" dash-dirty - alpha >/dev/null 2>"$err"; then + fail "seed accepted a dirty Treehouse-acquired secondmate home" + fi + + grep -F 'acquired worktree is dirty' "$err" >/dev/null \ + || fail "dirty secondmate-home refusal did not identify the unlanded work" + grep -F 'retaining unsafe treehouse-acquired home' "$err" >/dev/null \ + || fail "dirty secondmate-home refusal did not surface retain-only cleanup" + grep -F "treehouse return --force $acquired_abs" "$log" >/dev/null \ + && fail "dirty secondmate-home refusal used destructive Treehouse return" + [ -d "$acquired" ] || fail "dirty secondmate-home refusal removed the acquired worktree" + grep -Fq '# unlanded secondmate work' "$draft" \ + || fail "dirty secondmate-home refusal changed its draft" + [ ! -e "$acquired/AGENTS.md" ] \ + || fail "dirty secondmate-home refusal restored or rewrote its tracked deletion" + if [ -f "$home/data/secondmates.md" ] && grep -F -- '- dash-dirty ' "$home/data/secondmates.md" >/dev/null; then + fail "dirty secondmate-home refusal wrote a registry route" + fi + pass "dirty secondmate acquisitions are retained untouched for recovery" +} + +test_home_seed_retains_repository_mismatch_acquisition() { + local home expected_acquired acquired expected_source fakebin log err acquired_abs + home="$TMP_ROOT/dash-mismatch-home" + expected_acquired="$TMP_ROOT/dash-mismatch-expected-worktree" + acquired="$TMP_ROOT/dash-mismatch-acquired-home" + err="$TMP_ROOT/dash-mismatch.err" + mkdir -p "$home/projects" "$home/data" "$home/state" + fm_git_init_commit "$home/projects/alpha" + fm_git_add_origin "$home/projects/alpha" "$TMP_ROOT/remotes/dash-mismatch-alpha.git" + printf '%s\n' '- alpha [direct-PR] - alpha project (added 2026-06-22)' > "$home/data/projects.md" + expected_source=$(make_live_default_firstmate_worktree "$expected_acquired" dash-mismatch-expected) + make_live_default_firstmate_worktree "$acquired" dash-mismatch-unrelated >/dev/null + acquired_abs=$(cd "$acquired" && pwd -P) + fakebin=$(make_fake_tmux "$TMP_ROOT/dash-mismatch-fake") + log="$TMP_ROOT/dash-mismatch-fake/tmux.log" + + if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TREEHOUSE_HOME="$acquired" FM_FAKE_TMUX_LOG="$log" \ + FM_ROOT_OVERRIDE="$expected_source" \ + FM_SECONDMATE_CHARTER='dash mismatch scope' FM_SECONDMATE_SCOPE='dash mismatch scope' \ + "$ROOT/bin/fm-home-seed.sh" dash-mismatch - alpha >/dev/null 2>"$err"; then + fail "seed accepted an acquired home from an unrelated repository" + fi + + grep -F 'acquired worktree repository mismatch' "$err" >/dev/null \ + || fail "repository-mismatch acquisition was not diagnosed" + grep -F 'retaining unsafe treehouse-acquired home' "$err" >/dev/null \ + || fail "repository-mismatch acquisition did not surface retain-only cleanup" + grep -F "treehouse return --force $acquired_abs" "$log" >/dev/null \ + && fail "repository-mismatch acquisition was force-returned" + [ -d "$acquired" ] || fail "repository-mismatch acquisition was removed" + pass "repository-mismatch secondmate acquisitions remain durably retained" +} + test_home_seed_returns_treehouse_acquired_home_on_assignment_failure() { - local home acquired acquired_abs fakebin log err + local home acquired acquired_abs fakebin log err source common key lock_root expected_lock lock_marker home="$TMP_ROOT/dash-fail-home" acquired="$TMP_ROOT/dash-fail-acquired-home" err="$TMP_ROOT/dash-fail.err" @@ -205,13 +403,23 @@ test_home_seed_returns_treehouse_acquired_home_on_assignment_failure() { fm_git_init_commit "$home/projects/alpha" fm_git_add_origin "$home/projects/alpha" "$TMP_ROOT/remotes/dash-fail-alpha.git" printf '%s\n' '- alpha [direct-PR] - alpha project (added 2026-06-22)' > "$home/data/projects.md" - git clone --quiet "$ROOT" "$acquired" + source=$(make_live_default_firstmate_worktree "$acquired" dash-fail-firstmate) acquired_abs=$(cd "$acquired" && pwd -P) + common=$(git -C "$acquired" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$acquired/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(printf '%s' "$common" | shasum -a 256 | awk '{print substr($1,1,24)}') + lock_root="$TMP_ROOT/dash-fail-locks" + expected_lock="$lock_root/$key.lock" + lock_marker="$TMP_ROOT/dash-fail-return-held-lock" printf 'other\n' > "$acquired/.fm-secondmate-home" fakebin=$(make_fake_tmux "$TMP_ROOT/dash-fail-fake") log="$TMP_ROOT/dash-fail-fake/tmux.log" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TREEHOUSE_HOME="$acquired" FM_FAKE_TMUX_LOG="$log" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$lock_root" \ + FM_EXPECT_CHECKOUT_LOCK="$expected_lock" FM_EXPECT_CHECKOUT_LOCK_MARKER="$lock_marker" \ + FM_ROOT_OVERRIDE="$source" \ FM_SECONDMATE_CHARTER='dash acquired scope' FM_SECONDMATE_SCOPE='dash acquired scope' \ "$ROOT/bin/fm-home-seed.sh" dash - alpha >/dev/null 2>"$err"; then fail "seed reused an acquired home marked for another secondmate" @@ -219,6 +427,8 @@ test_home_seed_returns_treehouse_acquired_home_on_assignment_failure() { grep -F 'already marked for other' "$err" >/dev/null || fail "seed did not explain acquired marked-home rejection" grep -F "treehouse return --force $acquired_abs" "$log" >/dev/null \ || fail "failed acquired seed did not return the home through treehouse" + [ -f "$lock_marker" ] \ + || fail "secondmate rollback did not hold the common checkout lock during Treehouse return" if [ -f "$home/data/secondmates.md" ] && grep -F -- '- dash ' "$home/data/secondmates.md" >/dev/null; then fail "failed acquired seed left a registry route" fi @@ -226,7 +436,7 @@ test_home_seed_returns_treehouse_acquired_home_on_assignment_failure() { } test_home_seed_warns_when_acquired_home_return_fails() { - local home acquired acquired_abs fakebin log err lease + local home acquired acquired_abs fakebin log err lease source home="$TMP_ROOT/dash-return-fail-home" acquired="$TMP_ROOT/dash-return-fail-acquired-home" err="$TMP_ROOT/dash-return-fail.err" @@ -234,7 +444,7 @@ test_home_seed_warns_when_acquired_home_return_fails() { fm_git_init_commit "$home/projects/alpha" fm_git_add_origin "$home/projects/alpha" "$TMP_ROOT/remotes/dash-return-fail-alpha.git" printf '%s\n' '- alpha [direct-PR] - alpha project (added 2026-06-22)' > "$home/data/projects.md" - git clone --quiet "$ROOT" "$acquired" + source=$(make_live_default_firstmate_worktree "$acquired" dash-return-fail-firstmate) acquired_abs=$(cd "$acquired" && pwd -P) printf 'other\n' > "$acquired/.fm-secondmate-home" fakebin=$(make_fake_tmux "$TMP_ROOT/dash-return-fail-fake") @@ -243,6 +453,7 @@ test_home_seed_warns_when_acquired_home_return_fails() { if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TREEHOUSE_HOME="$acquired" FM_FAKE_TMUX_LOG="$log" \ FM_FAKE_TREEHOUSE_LEASE_FILE="$lease" FM_FAKE_TREEHOUSE_RETURN_FAIL=1 \ + FM_ROOT_OVERRIDE="$source" \ FM_SECONDMATE_CHARTER='dash acquired scope' FM_SECONDMATE_SCOPE='dash acquired scope' \ "$ROOT/bin/fm-home-seed.sh" dash - alpha >/dev/null 2>"$err"; then fail "seed reused an acquired home after return failure setup" @@ -437,6 +648,131 @@ test_home_seed_no_projects_end_to_end() { pass "home seeding scaffolds, registers, and spawns a project-less home end to end" } +test_home_seed_serializes_with_home_retirement() { + local home sub registry_sub state holder_pid waited status + home="$TMP_ROOT/seed-retirement-home" + sub="$TMP_ROOT/seed-retirement-subhome" + state="$TMP_ROOT/seed-retirement-state" + mkdir -p "$home/projects" "$home/data" "$home/state" + bash -c ' + . "$1/bin/fm-account-routing-lib.sh" + lock=$(fm_secondmate_home_lifecycle_lock_acquire "$2/locks" "$3") || exit 1 + : > "$4" + while [ ! -f "$5" ]; do sleep 0.05; done + fm_account_lifecycle_lock_release "$lock" + ' _ "$ROOT" "$state" "$sub" "$TMP_ROOT/seed-retirement-ready" "$TMP_ROOT/seed-retirement-release" & + holder_pid=$! + waited=0 + while [ ! -f "$TMP_ROOT/seed-retirement-ready" ] && [ "$waited" -lt 200 ]; do + sleep 0.05 + waited=$((waited + 1)) + done + [ -f "$TMP_ROOT/seed-retirement-ready" ] || { + : > "$TMP_ROOT/seed-retirement-release" + wait "$holder_pid" || true + fail "home lifecycle lock holder did not start" + } + set +e + FM_HOME="$home" FM_SECONDMATE_CHARTER='serialized firstmate domain' \ + FM_SECONDMATE_SCOPE='serialized firstmate work' \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state" \ + FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS=0 \ + "$ROOT/bin/fm-home-seed.sh" serialized "$sub" --no-projects \ + > "$TMP_ROOT/seed-retirement.out" 2> "$TMP_ROOT/seed-retirement.err" + status=$? + set -e + : > "$TMP_ROOT/seed-retirement-release" + wait "$holder_pid" || fail "home lifecycle lock holder failed to release" + [ "$status" -ne 0 ] || fail "home seed ignored concurrent retirement ownership" + assert_contains "$(cat "$TMP_ROOT/seed-retirement.err")" "secondmate home lifecycle lock" \ + "home seed lifecycle contention was not surfaced" + assert_absent "$sub" "blocked home seed created a retiring home" + if [ -f "$home/data/secondmates.md" ]; then + assert_no_grep '^- serialized ' "$home/data/secondmates.md" \ + "blocked home seed registered a retiring home" + fi + registry_sub="$TMP_ROOT/seed-registry-subhome" + bash -c ' + . "$1/bin/fm-account-routing-lib.sh" + lock=$(fm_secondmate_registry_lock_acquire "$2/locks" "$3") || exit 1 + : > "$4" + while [ ! -f "$5" ]; do sleep 0.05; done + fm_account_lifecycle_lock_release "$lock" + ' _ "$ROOT" "$state" "$home/data/secondmates.md" \ + "$TMP_ROOT/seed-registry-ready" "$TMP_ROOT/seed-registry-release" & + holder_pid=$! + waited=0 + while [ ! -f "$TMP_ROOT/seed-registry-ready" ] && [ "$waited" -lt 200 ]; do + sleep 0.05 + waited=$((waited + 1)) + done + [ -f "$TMP_ROOT/seed-registry-ready" ] || { + : > "$TMP_ROOT/seed-registry-release" + wait "$holder_pid" || true + fail "registry lifecycle lock holder did not start" + } + set +e + FM_HOME="$home" FM_SECONDMATE_CHARTER='serialized registry domain' \ + FM_SECONDMATE_SCOPE='serialized registry work' \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state" \ + FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS=0 \ + "$ROOT/bin/fm-home-seed.sh" registrylocked "$registry_sub" --no-projects \ + > "$TMP_ROOT/seed-registry.out" 2> "$TMP_ROOT/seed-registry.err" + status=$? + set -e + : > "$TMP_ROOT/seed-registry-release" + wait "$holder_pid" || fail "registry lifecycle lock holder failed to release" + [ "$status" -ne 0 ] || fail "home seed ignored concurrent registry ownership" + assert_absent "$registry_sub" "registry-locked seed retained a partially provisioned home" + if [ -f "$home/data/secondmates.md" ]; then + assert_no_grep '^- registrylocked ' "$home/data/secondmates.md" \ + "registry-locked seed overwrote the shared registry" + fi + pass "secondmate home seeding serializes with retirement and registry updates" +} + +test_home_seed_requires_parent_lifecycle_authority() { + local home sub state holder_pid waited status + home="$TMP_ROOT/parent-seed-lock-home" + sub="$TMP_ROOT/parent-seed-lock-subhome" + state="$TMP_ROOT/parent-seed-lock-state" + mkdir -p "$home/projects" "$home/data" "$home/state" + bash -c ' + . "$1/bin/fm-account-routing-lib.sh" + lock=$(fm_secondmate_home_lifecycle_lock_acquire "$2/locks" "$3") || exit 1 + : > "$4" + while [ ! -f "$5" ]; do sleep 0.05; done + fm_account_lifecycle_lock_release "$lock" + ' _ "$ROOT" "$state" "$home" "$TMP_ROOT/parent-seed-lock-ready" "$TMP_ROOT/parent-seed-lock-release" & + holder_pid=$! + waited=0 + while [ ! -f "$TMP_ROOT/parent-seed-lock-ready" ] && [ "$waited" -lt 200 ]; do + sleep 0.05 + waited=$((waited + 1)) + done + [ -f "$TMP_ROOT/parent-seed-lock-ready" ] || { + : > "$TMP_ROOT/parent-seed-lock-release" + wait "$holder_pid" || true + fail "parent home lifecycle lock holder did not start" + } + set +e + FM_HOME="$home" FM_SECONDMATE_CHARTER='nested seed domain' \ + FM_SECONDMATE_SCOPE='nested seed work' \ + FM_CHECKOUT_REFRESH_STATE_BASE="$state" \ + FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS=0 \ + "$ROOT/bin/fm-home-seed.sh" nested-seed "$sub" --no-projects \ + > "$TMP_ROOT/parent-seed-lock.out" 2> "$TMP_ROOT/parent-seed-lock.err" + status=$? + set -e + : > "$TMP_ROOT/parent-seed-lock-release" + wait "$holder_pid" || fail "parent home lifecycle lock holder failed to release" + [ "$status" -ne 0 ] || fail "nested seed bypassed parent home lifecycle authority" + assert_absent "$sub" "parent-locked seed created a child home" + assert_grep 'secondmate home lifecycle lock' "$TMP_ROOT/parent-seed-lock.err" \ + "parent home lock contention was not surfaced" + pass "nested home seeding requires parent lifecycle authority" +} + test_home_seed_refuses_projectful_reused_charter_for_projectless_home() { local home reusable_sub stale_sub stale_brief stale_brief_before err home="$TMP_ROOT/no-projects-reused-charter-home" @@ -1135,6 +1471,7 @@ SH log="$TMP_ROOT/spawn-validate-fake/tmux.log" err="$TMP_ROOT/spawn-validate.err" + write_secondmate_registration "$home" domain "$subhome" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$subhome" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted an unseeded home" @@ -1146,6 +1483,7 @@ SH grep -F 'new-window' "$log" >/dev/null && fail "spawn created a window before validation" printf 'other\n' > "$wronghome/.fm-secondmate-home" + write_secondmate_registration "$home" domain "$wronghome" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$wronghome" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted a home marked for another secondmate" @@ -1154,6 +1492,7 @@ SH printf 'domain\n' > "$marker_only/.fm-secondmate-home" printf 'charter\n' > "$marker_only/data/charter.md" + write_secondmate_registration "$home" domain "$marker_only" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$marker_only" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted a marked home missing AGENTS.md" @@ -1168,12 +1507,14 @@ SH grep -F 'not a firstmate home (missing bin/)' "$err" >/dev/null || fail "spawn did not explain missing bin" printf 'domain\n' > "$home/.fm-secondmate-home" + write_secondmate_registration "$home" domain "$home" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$home" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted the active home" fi grep -F 'secondmate home cannot be the active firstmate home' "$err" >/dev/null || fail "spawn did not reject active home" + write_secondmate_registration "$home" domain "$ROOT" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$ROOT" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted the firstmate repo root" @@ -1182,6 +1523,7 @@ SH printf 'domain\n' > "$active_descendant/.fm-secondmate-home" printf 'charter\n' > "$active_descendant/data/charter.md" + write_secondmate_registration "$home" domain "$active_descendant" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$active_descendant" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted a home inside the active firstmate home" @@ -1190,6 +1532,7 @@ SH printf 'domain\n' > "$active_ancestor/.fm-secondmate-home" printf 'charter\n' > "$active_ancestor/data/charter.md" + write_secondmate_registration "$ancestor_active_home" domain "$active_ancestor" if PATH="$fakebin:$PATH" FM_HOME="$ancestor_active_home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$active_ancestor" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted a home containing the active firstmate home" @@ -1198,6 +1541,7 @@ SH printf 'domain\n' > "$root_descendant/.fm-secondmate-home" printf 'charter\n' > "$root_descendant/data/charter.md" + write_secondmate_registration "$home" domain "$root_descendant" if PATH="$fakebin:$PATH" FM_ROOT_OVERRIDE="$fakeroot" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$root_descendant" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted a home inside the firstmate repo" @@ -1206,6 +1550,7 @@ SH printf 'domain\n' > "$root_ancestor/.fm-secondmate-home" printf 'charter\n' > "$root_ancestor/data/charter.md" + write_secondmate_registration "$home" domain "$root_ancestor" if PATH="$fakebin:$PATH" FM_ROOT_OVERRIDE="$root_inside" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-validate-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$root_ancestor" codex --secondmate >/dev/null 2>"$err"; then fail "secondmate spawn accepted a home containing the firstmate repo" @@ -1235,6 +1580,7 @@ test_secondmate_spawn_refuses_operational_dirs_outside_subhome() { if [ "$opdir" = data ]; then printf 'charter\n' > "$sink/charter.md" fi + write_secondmate_registration "$home" domain "$subhome" : > "$log" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-opdir-fake/pane.txt" \ "$ROOT/bin/fm-spawn.sh" domain "$subhome" codex --secondmate >/dev/null 2>"$err"; then @@ -1247,6 +1593,70 @@ test_secondmate_spawn_refuses_operational_dirs_outside_subhome() { pass "secondmate spawn refuses operational directories outside the subhome" } +test_secondmate_spawn_requires_exact_registration_and_target_home_lock() { + local home target other state_base fakebin log err ready release holder_pid status + home="$TMP_ROOT/spawn-registration-home" + target="$TMP_ROOT/spawn-registration-target" + other="$TMP_ROOT/spawn-registration-other" + state_base="$TMP_ROOT/spawn-registration-locks" + fakebin=$(make_fake_tmux "$TMP_ROOT/spawn-registration-fake") + log="$TMP_ROOT/spawn-registration-fake/tmux.log" + err="$TMP_ROOT/spawn-registration.err" + ready="$TMP_ROOT/spawn-registration.ready" + release="$TMP_ROOT/spawn-registration.release" + mkdir -p "$home/data" "$home/state" "$target" "$other" + + set +e + PATH="$fakebin:$PATH" FM_HOME="$home" FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-registration-fake/pane.txt" \ + "$ROOT/bin/fm-spawn.sh" domain "$target" codex --secondmate >/dev/null 2>"$err" + status=$? + set -e + [ "$status" -ne 0 ] || fail "secondmate spawn accepted an unregistered explicit home" + grep -F 'registry is malformed, missing, or does not uniquely register domain' "$err" >/dev/null \ + || fail "unregistered explicit home refusal did not identify registry authority" + + write_secondmate_registration "$home" domain "$other" + set +e + PATH="$fakebin:$PATH" FM_HOME="$home" FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-registration-fake/pane.txt" \ + "$ROOT/bin/fm-spawn.sh" domain "$target" codex --secondmate >/dev/null 2>"$err" + status=$? + set -e + [ "$status" -ne 0 ] || fail "secondmate spawn accepted a home different from its registration" + grep -F 'does not match the exact registration' "$err" >/dev/null \ + || fail "explicit registration drift was not surfaced" + + write_secondmate_registration "$home" domain "$target" + FM_HOLDER_ROOT="$ROOT" FM_HOLDER_STATE="$state_base" FM_HOLDER_HOME="$target" \ + FM_HOLDER_READY="$ready" FM_HOLDER_RELEASE="$release" \ + bash -c ' + set -eu + . "$FM_HOLDER_ROOT/bin/fm-checkout-lock-lib.sh" + . "$FM_HOLDER_ROOT/bin/fm-account-routing-lib.sh" + lock=$(fm_secondmate_home_lifecycle_lock_acquire "$FM_HOLDER_STATE/locks" "$FM_HOLDER_HOME") + : > "$FM_HOLDER_READY" + while [ ! -f "$FM_HOLDER_RELEASE" ]; do sleep 0.05; done + fm_account_lifecycle_lock_release "$lock" + ' & + holder_pid=$! + while [ ! -f "$ready" ]; do sleep 0.05; done + set +e + PATH="$fakebin:$PATH" FM_HOME="$home" FM_CHECKOUT_REFRESH_STATE_BASE="$state_base" \ + FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS=0 \ + FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/spawn-registration-fake/pane.txt" \ + "$ROOT/bin/fm-spawn.sh" domain "$target" codex --secondmate >/dev/null 2>"$err" + status=$? + set -e + : > "$release" + wait "$holder_pid" + [ "$status" -ne 0 ] || fail "secondmate spawn bypassed the declared target home lock" + grep -F 'secondmate home lifecycle lock' "$err" >/dev/null \ + || fail "target home lifecycle lock contention was not surfaced" + grep -F 'new-window' "$log" >/dev/null && fail "target home lock contention reached endpoint launch" + pass "secondmate spawn proves registration and owns the target home lock" +} + test_fm_send_refuses_bare_window_without_home_meta() { # The happy path (a bare fm-<id> resolves the window recorded in THIS home's # meta and never a foreign same-named window) is asserted in the lifecycle e2e. @@ -1385,8 +1795,8 @@ EOF pass "secondmate teardown raw-removes plain-clone homes" } -test_secondmate_force_teardown_discards_child_work() { - local home subhome childproj childwt fakebin log +test_secondmate_force_teardown_retains_unlanded_child_work() { + local home subhome childproj childwt fakebin log err rc home="$TMP_ROOT/force-teardown-home" subhome="$TMP_ROOT/force-teardown-subhome" childproj="$subhome/projects/alpha" @@ -1417,20 +1827,24 @@ yolo=off EOF fakebin=$(make_fake_tmux "$TMP_ROOT/force-teardown-fake") log="$TMP_ROOT/force-teardown-fake/tmux.log" + err="$TMP_ROOT/force-teardown-fake/teardown.err" + printf '%s\n' retained > "$childwt/untracked-child.txt" if PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/force-teardown-fake/pane.txt" \ "$ROOT/bin/fm-teardown.sh" domain >/dev/null 2>&1; then fail "teardown allowed a secondmate with in-flight child work" fi + set +e PATH="$fakebin:$PATH" FM_HOME="$home" FM_FAKE_TMUX_LOG="$log" FM_FAKE_TMUX_CAPTURE="$TMP_ROOT/force-teardown-fake/pane.txt" \ - "$ROOT/bin/fm-teardown.sh" domain --force >/dev/null 2>/dev/null \ - || fail "force teardown failed to discard child work" - [ ! -d "$subhome" ] || fail "force teardown did not remove the retired secondmate home" - [ ! -d "$childwt" ] || fail "force teardown did not remove child worktree" - [ ! -e "$home/state/domain.meta" ] || fail "teardown did not clear parent meta" - grep -F -- '- domain ' "$home/data/secondmates.md" >/dev/null && fail "force teardown did not remove secondmate registry route" - grep -F 'kill-window -t firstmate:fm-child' "$log" >/dev/null || fail "force teardown did not kill child window" - grep -F 'kill-window -t firstmate:fm-domain' "$log" >/dev/null || fail "force teardown did not kill parent window" - pass "secondmate force teardown discards child work" + "$ROOT/bin/fm-teardown.sh" domain --force >/dev/null 2>"$err" + rc=$? + set -e + [ "$rc" -ne 0 ] || fail "force teardown discarded unlanded child work" + assert_present "$subhome" "force teardown removed the parent home" + assert_present "$childwt/untracked-child.txt" "force teardown discarded untracked child work" + assert_present "$home/state/domain.meta" "force teardown removed parent retry metadata" + assert_present "$subhome/state/child.meta" "force teardown removed child retry metadata" + assert_grep '- domain ' "$home/data/secondmates.md" "force teardown removed the parent registry route" + pass "secondmate force teardown retains unlanded child work" } test_secondmate_force_teardown_preserves_child_on_unproven_lock() { @@ -2098,13 +2512,52 @@ EOF pass "fm-backlog-handoff refuses Done items under whitespace section headings and unsafe homes" } +if [ "${FM_TEST_FOCUSED:-}" = checkout-freshness ]; then + test_home_seed_rejects_stale_treehouse_acquired_home + test_home_seed_retains_dirty_treehouse_acquired_home + test_home_seed_retains_repository_mismatch_acquisition + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-6 ]; then + test_home_seed_acquisition_honors_shared_checkout_lock + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-8 ]; then + test_home_seed_returns_treehouse_acquired_home_on_assignment_failure + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-races ]; then + test_home_seed_serializes_with_home_retirement + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-secondmate-authority ]; then + test_home_seed_validate_rejects_partial_registry_rows + test_home_seed_requires_parent_lifecycle_authority + test_secondmate_force_teardown_retains_unlanded_child_work + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-durable-secondmate ]; then + test_secondmate_spawn_requires_exact_registration_and_target_home_lock + exit 0 +fi + test_fm_home_parameterization test_lock_status_is_per_home test_seed_allows_overlapping_clones_and_drops_owner test_home_seed_validate_rejects_duplicate_homes test_home_seed_validate_rejects_duplicate_ids test_home_seed_validate_rejects_nested_homes +test_home_seed_validate_rejects_partial_registry_rows test_home_seed_uses_treehouse_acquired_home +test_home_seed_acquisition_honors_shared_checkout_lock +test_home_seed_rejects_stale_treehouse_acquired_home +test_home_seed_retains_dirty_treehouse_acquired_home +test_home_seed_retains_repository_mismatch_acquisition test_home_seed_returns_treehouse_acquired_home_on_assignment_failure test_home_seed_warns_when_acquired_home_return_fails test_home_seed_does_not_return_unsafe_acquired_home @@ -2113,6 +2566,8 @@ test_home_seed_refuses_missing_filled_charter test_home_seed_refuses_placeholder_charter test_home_seed_refuses_empty_charter_fields test_home_seed_no_projects_end_to_end +test_home_seed_serializes_with_home_retirement +test_home_seed_requires_parent_lifecycle_authority test_home_seed_refuses_projectful_reused_charter_for_projectless_home test_home_seed_refuses_projectless_conversion_of_populated_home test_home_seed_refuses_projectless_home_with_uninspectable_projects @@ -2137,11 +2592,12 @@ test_home_seed_refuses_operational_dirs_outside_subhome test_home_seed_refuses_symlinked_leaf_files test_secondmate_spawn_requires_seeded_matching_home test_secondmate_spawn_refuses_operational_dirs_outside_subhome +test_secondmate_spawn_requires_exact_registration_and_target_home_lock test_fm_send_refuses_bare_window_without_home_meta test_secondmate_teardown_retires_empty_home test_secondmate_teardown_refuses_failed_leased_home_return test_secondmate_teardown_removes_plain_clone_home_without_treehouse_return -test_secondmate_force_teardown_discards_child_work +test_secondmate_force_teardown_retains_unlanded_child_work test_secondmate_force_teardown_preserves_child_on_unproven_lock test_secondmate_force_teardown_allows_operational_dir_symlinks_inside_home test_secondmate_force_teardown_refuses_operational_dir_symlink_outside_home diff --git a/tests/fm-spawn-batch.test.sh b/tests/fm-spawn-batch.test.sh index 4f6f10c5ef4..85578793859 100755 --- a/tests/fm-spawn-batch.test.sh +++ b/tests/fm-spawn-batch.test.sh @@ -35,6 +35,8 @@ test_batch_dispatches_every_pair() { out=$(run_spawn nope-batch-a-z1=projects/none-a nope-batch-b-z2=projects/none-b) status=$? [ "$status" -ne 0 ] || fail "batch with missing briefs should exit non-zero" + printf '%s\n' "$out" | grep -F 'unbound variable' >/dev/null \ + && fail "batch cleanup expanded an uninitialized single-task identity" printf '%s\n' "$out" | grep -F 'batch: FAILED to spawn nope-batch-a-z1 (projects/none-a)' >/dev/null \ || fail "first pair was not dispatched/reported" printf '%s\n' "$out" | grep -F 'batch: FAILED to spawn nope-batch-b-z2 (projects/none-b)' >/dev/null \ @@ -53,6 +55,8 @@ test_batch_mode_boundaries() { out=$(run_spawn $args) status=$? [ "$status" -ne 0 ] || fail "$label: expected non-zero exit" + printf '%s\n' "$out" | grep -F 'unbound variable' >/dev/null \ + && fail "$label: batch cleanup expanded an uninitialized task identity" if [ -n "$expect" ]; then printf '%s\n' "$out" | grep -F "$expect" >/dev/null || fail "$label: missing '$expect'" fi diff --git a/tests/fm-tangle-guard.test.sh b/tests/fm-tangle-guard.test.sh index a6f24eaa593..04ef03cd29d 100755 --- a/tests/fm-tangle-guard.test.sh +++ b/tests/fm-tangle-guard.test.sh @@ -149,8 +149,8 @@ test_brief_assertion_precedes_branch() { # --- GUARD 1b: fm-spawn isolation abort ------------------------------------- -# A fake tmux that reports FM_FAKE_PANE_PATH as the post-`treehouse get` pane cwd -# (so the spawn's worktree-resolution loop resolves to a path we control), names +# A fake tmux that reports FM_FAKE_PANE_PATH as the leased-worktree pane cwd +# (so the spawn's endpoint verification resolves to a path we control), names # the session on '#S', and swallows window ops. Echoes the fakebin dir. make_spawn_fakebin() { local dir=$1 fakebin @@ -169,7 +169,11 @@ esac exit 0 SH chmod +x "$fakebin/tmux" - fm_fake_exit0 "$fakebin" treehouse + cat > "$fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +printf '%s\n' "${FM_FAKE_TREEHOUSE_PATH:?}" +SH + chmod +x "$fakebin/treehouse" printf '%s\n' "$fakebin" } @@ -181,6 +185,7 @@ run_spawn() { FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ FM_PROJECTS_OVERRIDE="$home/projects" FM_CONFIG_OVERRIDE="$home/config" \ FM_SPAWN_NO_GUARD=1 FM_FAKE_PANE_PATH="$pane" TMUX="fake,1,0" \ + FM_FAKE_TREEHOUSE_PATH="$pane" FM_TREEHOUSE_ROOT="$home/treehouse-pools" \ PATH="$fakebin:$PATH" \ "$ROOT/bin/fm-spawn.sh" "$id" "$proj" codex 2>&1 } @@ -248,7 +253,12 @@ esac exit 0 SH chmod +x "$fakebin/tmux" - fm_fake_exit0 "$fakebin" treehouse + cat > "$fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +[ -n "${FM_TMUX_REC:-}" ] && printf 'treehouse %s\n' "$*" >> "$FM_TMUX_REC" +printf '%s\n' "${FM_FAKE_TREEHOUSE_PATH:?}" +SH + chmod +x "$fakebin/treehouse" printf '%s\n' "$fakebin" } @@ -260,7 +270,8 @@ run_spawn_record() { FM_STATE_OVERRIDE="$home/state" FM_DATA_OVERRIDE="$home/data" \ FM_PROJECTS_OVERRIDE="$home/projects" FM_CONFIG_OVERRIDE="$home/config" \ FM_SPAWN_NO_GUARD=1 FM_FAKE_PANE_PATH="$pane" TMUX="fake,1,0" \ - FM_TMUX_REC="$rec" \ + FM_TMUX_REC="$rec" FM_FAKE_TREEHOUSE_PATH="$pane" \ + FM_TREEHOUSE_ROOT="$home/treehouse-pools" \ PATH="$fakebin:$PATH" \ "$ROOT/bin/fm-spawn.sh" "$id" "$proj" codex 2>&1 } @@ -292,11 +303,12 @@ test_spawn_tmux_window_construction() { assert_grep "set-window-option -t @spawnwid allow-rename off" "$rec" \ "must disable allow-rename on the spawned window" - # Bug 2 fix (b): treehouse-get and the worktree wait loop target the stable id. - assert_grep "send-keys -t @spawnwid treehouse get Enter" "$rec" \ - "treehouse get must be sent to the stable window id" + assert_grep "treehouse get --lease --lease-holder firstmate-rec-win-gg7" "$rec" \ + "Treehouse worktree acquisition was not a durable task lease" + assert_grep "new-window -dP -F #{window_id} -t firstmate: -n fm-rec-win-gg7 -c $wt" "$rec" \ + "the endpoint did not start directly in its verified leased worktree" assert_grep "display-message -p -t @spawnwid #{pane_current_path}" "$rec" \ - "the worktree wait loop must query the stable window id, not the name" + "leased-worktree endpoint verification must query the stable window id" pass "fm-spawn: appends windows by session-colon, pins the name, and targets the window id" } diff --git a/tests/fm-teardown.test.sh b/tests/fm-teardown.test.sh index bef270f3fb0..4a6fdb86a29 100755 --- a/tests/fm-teardown.test.sh +++ b/tests/fm-teardown.test.sh @@ -26,7 +26,7 @@ # (c) local-only + merged into local main, no remote -> ALLOW (no regression) # (d) no-mistakes + HEAD on origin remote-tracking branch -> ALLOW (no regression) # (e) no-mistakes + unpushed, no PR, content not in default -> REFUSE (safety) -# (f) local-only + truly unpushed + --force -> ALLOW (escape hatch) +# (f) local-only + truly unpushed + --force -> REFUSE (force retains work) # (g) no-mistakes + squash-merged PR, exact PR head -> ALLOW (squash fix) # (h) no-mistakes + no PR but content already in default -> ALLOW (content fallback) # (i) no-mistakes + dirty worktree, even when work landed -> REFUSE (dirty wins) @@ -63,6 +63,72 @@ export REAL_GIT_FOR_TEST REAL_STAT_FOR_TEST=$(command -v stat) export REAL_STAT_FOR_TEST +write_treehouse_lease() { + local worktree=$1 holder=$2 slot pool state + slot=$(cd "$(dirname "$worktree")" && pwd -P) + pool=$(cd "$(dirname "$slot")" && pwd -P) + state="$pool/treehouse-state.json" + python3 - "$state" "$(cd "$worktree" && pwd -P)" "$holder" <<'PY' +import json +import sys + +state, path, holder = sys.argv[1:] +with open(state, "w", encoding="utf-8") as stream: + json.dump( + { + "worktrees": [ + { + "name": "1", + "path": path, + "leased": True, + "lease_holder": holder, + } + ] + }, + stream, + ) +PY +} + +prepare_secondmate_home_fixture() { + local case_dir=$1 id=${2:-task-x1} root_default default root_tip exclude home_abs + mkdir -p "$case_dir/data" "$case_dir/wt/data" "$case_dir/wt/state" "$case_dir/wt/config" \ + "$case_dir/wt/projects" "$case_dir/source-projects" + printf '%s\n' "$id" > "$case_dir/wt/.fm-secondmate-home" + home_abs=$(cd "$case_dir/wt" && pwd -P) + printf '%s\n' "- $id - test secondmate (home: $home_abs; scope: test; projects: test; added 2026-07-23)" \ + > "$case_dir/data/secondmates.md" + root_default=$(git -C "$ROOT" symbolic-ref --quiet --short refs/remotes/origin/HEAD 2>/dev/null || printf 'origin/main') + default=${root_default#origin/} + root_tip=$(git -C "$ROOT" rev-parse "$root_default") + git -C "$case_dir/project" fetch --quiet "$ROOT" "$root_tip" + git -C "$case_dir/project" checkout --quiet --detach + git -C "$case_dir/wt" checkout --quiet -B "$default" "$root_tip" + git -C "$case_dir/project" branch -D fm/task-x1 >/dev/null 2>&1 || true + git -C "$case_dir/project" remote set-url origin "$ROOT" + git -C "$case_dir/project" update-ref "refs/remotes/origin/$default" "$root_tip" + git -C "$case_dir/project" symbolic-ref refs/remotes/origin/HEAD "refs/remotes/origin/$default" + git -C "$case_dir/wt" reflog expire --expire=now --all + git clone --quiet "$case_dir/origin.git" "$case_dir/source-projects/test" + git clone --quiet "$case_dir/origin.git" "$case_dir/wt/projects/test" + exclude=$(git -C "$case_dir/wt" rev-parse --git-path info/exclude) + printf '%s\n' '.fm-secondmate-home' '/data/' '/state/' '/config/' '/projects/' >> "$exclude" + cat > "$case_dir/fakebin/git" <<'SH' +#!/usr/bin/env bash +case " $* " in + *" ls-remote --symref origin HEAD "*|*" ls-remote --symref origin HEAD") + remote_head=$("$REAL_GIT_FOR_TEST" -C "$FM_FAKE_FIRSTMATE_SOURCE" \ + symbolic-ref --quiet --short refs/remotes/origin/HEAD 2>/dev/null || printf 'origin/main') + remote_tip=$("$REAL_GIT_FOR_TEST" -C "$FM_FAKE_FIRSTMATE_SOURCE" rev-parse "$remote_head") + printf 'ref: refs/heads/%s\tHEAD\n%s\tHEAD\n' "${remote_head#origin/}" "$remote_tip" + exit 0 + ;; +esac +exec "$REAL_GIT_FOR_TEST" "$@" +SH + chmod +x "$case_dir/fakebin/git" +} + # Build a fresh sandbox for one test case. Sets up: # $CASE/state/ - firstmate state dir (with a fresh watcher beacon) # $CASE/fakebin/ - mocks for treehouse, tmux (PATH-prepended by caller) @@ -81,6 +147,12 @@ make_case() { cat > "$fakebin/treehouse" <<'SH' #!/usr/bin/env bash # `treehouse return --force <wt>`: succeed silently. +[ -z "${FM_EXPECT_CHECKOUT_LOCK:-}" ] || { + [ -e "$FM_EXPECT_CHECKOUT_LOCK" ] || [ -L "$FM_EXPECT_CHECKOUT_LOCK" ] || exit 91 + lock_pid=$(cat "$FM_EXPECT_CHECKOUT_LOCK/pid" 2>/dev/null || true) + kill -0 "$lock_pid" 2>/dev/null || exit 92 + [ -z "${FM_EXPECT_CHECKOUT_LOCK_MARKER:-}" ] || touch "$FM_EXPECT_CHECKOUT_LOCK_MARKER" +} [ -z "${FM_EXPECT_CHILD_LINEAGE_PATH:-}" ] || [ -f "$FM_EXPECT_CHILD_LINEAGE_PATH" ] || { echo "child lineage missing before home removal: $FM_EXPECT_CHILD_LINEAGE_PATH" >&2 exit 96 @@ -107,7 +179,12 @@ exit 0 SH cat > "$fakebin/tmux" <<'SH' #!/usr/bin/env bash -# tmux kill-window etc.: succeed silently. +state="$(dirname "$0")/.tmux-live" +case "${1:-}" in + display-message) [ -f "$state" ]; exit $? ;; + list-windows) [ ! -f "$state" ] || printf '%s\n' fm-task-x1; exit 0 ;; + kill-window) rm -f "$state"; exit 0 ;; +esac exit 0 SH # Default gh-axi mock: no PR is associated with the branch, and viewing any PR @@ -130,6 +207,7 @@ esac exit 0 SH chmod +x "$fakebin/treehouse" "$fakebin/tmux" "$fakebin/gh-axi" "$fakebin/gh" + : > "$fakebin/.tmux-live" # Bare origin so the clone has an `origin` remote and origin/HEAD. git init -q --bare "$case_dir/origin.git" @@ -145,6 +223,7 @@ SH git -C "$case_dir/project" remote set-head origin main 2>/dev/null || true # Add a worktree on a fresh task branch; that branch is where the crewmate commits. git -C "$case_dir/project" worktree add -q -b fm/task-x1 "$case_dir/wt" main + write_treehouse_lease "$case_dir/wt" firstmate-task-x1 # Fresh watcher beacon so fm-guard stays quiet. touch "$case_dir/state/.last-watcher-beat" @@ -180,6 +259,7 @@ write_meta() { local case_dir=$1 mode=$2 kind=$3 fm_write_meta "$case_dir/state/task-x1.meta" \ "window=fm-task-x1" \ + "tmux_session_target=firstmate:fm-task-x1" \ "worktree=$case_dir/wt" \ "project=$case_dir/project" \ "kind=$kind" \ @@ -422,6 +502,60 @@ SH chmod +x "$case_dir/fakebin/treehouse" } +add_stale_lock_on_first_return_treehouse() { + local case_dir=$1 + cat > "$case_dir/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = return ]; then + shift + wt="" + for a in "$@"; do + case "$a" in + --force) ;; + *) wt=$a ;; + esac + done + lock=$(git -C "$wt" rev-parse --git-path index.lock 2>/dev/null || true) + case "$lock" in + /*|'') ;; + *) lock="$wt/$lock" ;; + esac + if [ ! -f "${TREEHOUSE_FIRST_RETURN_MARKER:?}" ]; then + : > "$TREEHOUSE_FIRST_RETURN_MARKER" + mkdir -p "$(dirname "$lock")" + : > "$lock" + touch -t 200001010000 "$lock" + fi + if [ -n "$lock" ] && [ -e "$lock" ]; then + echo "fatal: Unable to create '$lock': File exists." >&2 + exit 128 + fi +fi +exit 0 +SH + chmod +x "$case_dir/fakebin/treehouse" +} + +add_hanging_treehouse() { + local case_dir=$1 + cat > "$case_dir/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = return ]; then + ( + trap '' TERM + while :; do + sleep 1 + done + ) & + child=$! + printf '%s\n' "$child" > "${TREEHOUSE_RETURN_CHILD_PID_FILE:?}" + wait "$child" +fi +exit 0 +SH + chmod +x "$case_dir/fakebin/treehouse" +} + git_index_lock_path() { local dir=$1 lock abs_dir lock=$(git -C "$dir" rev-parse --git-path index.lock) @@ -434,6 +568,15 @@ git_index_lock_path() { esac } +checkout_lock_path() { + local dir=$1 lock_root=$2 common key + common=$(git -C "$dir" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$dir/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(printf '%s' "$common" | shasum -a 256 | awk '{print substr($1,1,24)}') + printf '%s/%s.lock\n' "$lock_root" "$key" +} + # fakebin/lsof stub: no process ever holds anything open (lsof's not-found exit # code), so a lock's staleness is decided by age alone. add_lsof_no_holder() { @@ -523,12 +666,35 @@ run_teardown() { local case_dir=$1; shift FM_ROOT_OVERRIDE="$ROOT" \ FM_STATE_OVERRIDE="$case_dir/state" \ + FM_DATA_OVERRIDE="${FM_DATA_OVERRIDE:-$case_dir/data}" \ FM_CONFIG_OVERRIDE="$case_dir/config" \ + FM_PROJECTS_OVERRIDE="${FM_PROJECTS_OVERRIDE:-$case_dir/source-projects}" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="${FM_CHECKOUT_REFRESH_LOCK_ROOT:-$case_dir/checkout-locks}" \ + FM_EXPECT_CHECKOUT_LOCK="${FM_EXPECT_CHECKOUT_LOCK:-}" \ + FM_EXPECT_CHECKOUT_LOCK_MARKER="${FM_EXPECT_CHECKOUT_LOCK_MARKER:-}" \ + FM_FAKE_FIRSTMATE_SOURCE="${FM_FAKE_FIRSTMATE_SOURCE:-$ROOT}" \ + FM_TEARDOWN_TEST_MOUNT_PATH="${FM_TEARDOWN_TEST_MOUNT_PATH:-}" \ + HOME="${FM_TEST_TEARDOWN_HOME:-$HOME}" \ FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ PATH="$case_dir/fakebin:$PATH" \ "$TEARDOWN" task-x1 "$@" } +run_teardown_named() { + local case_dir=$1 task=$2 + shift 2 + FM_ROOT_OVERRIDE="$ROOT" \ + FM_STATE_OVERRIDE="$case_dir/state" \ + FM_DATA_OVERRIDE="$case_dir/data" \ + FM_CONFIG_OVERRIDE="$case_dir/config" \ + FM_PROJECTS_OVERRIDE="${FM_PROJECTS_OVERRIDE:-$case_dir/source-projects}" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$case_dir/checkout-locks" \ + FM_FAKE_FIRSTMATE_SOURCE="$ROOT" \ + FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ + PATH="$case_dir/fakebin:$PATH" \ + "$TEARDOWN" "$task" "$@" +} + test_local_only_fork_remote_allows() { local case_dir rc case_dir=$(make_case fork-allow) @@ -537,7 +703,7 @@ test_local_only_fork_remote_allows() { add_fork_with_pushed_branch "$case_dir" set +e - run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" rc=$? set -e @@ -1000,7 +1166,7 @@ SH } test_content_in_default_fallback_allows() { - local case_dir rc + local case_dir rc common key expected_lock lock_marker case_dir=$(make_case content-landed) write_meta "$case_dir" no-mistakes ship # No pr= recorded and the default gh-axi mock reports no PR, so the merged-PR path @@ -1008,14 +1174,24 @@ test_content_in_default_fallback_allows() { # the same net change has independently landed on origin/main via a squash commit. wt_commit_file "$case_dir" feature.txt hello "add feature" land_on_origin_main "$case_dir" feature.txt hello + common=$(git -C "$case_dir/wt" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$case_dir/wt/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(printf '%s' "$common" | shasum -a 256 | awk '{print substr($1,1,24)}') + expected_lock="$case_dir/checkout-locks/$key.lock" + lock_marker="$case_dir/checkout-return-held-lock" set +e - run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + FM_EXPECT_CHECKOUT_LOCK="$expected_lock" \ + FM_EXPECT_CHECKOUT_LOCK_MARKER="$lock_marker" \ + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" rc=$? set -e expect_code 0 "$rc" "content-landed: teardown should succeed when content is already in the default branch" ! grep -q REFUSED "$case_dir/stderr" || fail "content-landed: teardown printed a REFUSED line" + assert_present "$lock_marker" \ + "normal teardown did not hold the common checkout lock during Treehouse return" pass "worktree whose content already landed in the default branch is torn down (content fallback)" } @@ -1038,6 +1214,160 @@ test_content_fallback_refreshes_stale_origin_ref() { pass "content fallback refreshes origin default before comparing trees" } +test_content_fallback_uses_live_default() { + local case_dir rc baseline + case_dir=$(make_case content-live-default) + write_meta "$case_dir" no-mistakes ship + wt_commit_file "$case_dir" feature.txt hello "add feature" + land_on_origin_main "$case_dir" feature.txt hello + baseline=$(git --git-dir="$case_dir/origin.git" rev-parse main^) + git --git-dir="$case_dir/origin.git" update-ref refs/heads/trunk "$baseline" + git --git-dir="$case_dir/origin.git" symbolic-ref HEAD refs/heads/trunk + [ "$(git -C "$case_dir/project" symbolic-ref --short refs/remotes/origin/HEAD)" = origin/main ] \ + || fail "live-default teardown fixture did not preserve stale origin/HEAD" + + set +e + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "live-default teardown must refuse content absent from live trunk" + assert_present "$case_dir/wt" "live-default teardown discarded the task worktree" + assert_present "$case_dir/state/task-x1.meta" "live-default teardown removed task metadata" + assert_grep "task content is not present in authoritative refs/remotes/origin/trunk" \ + "$case_dir/stderr" "live-default teardown did not identify the authoritative branch" + pass "teardown landing proof uses the live upstream default" +} + +test_content_fallback_reprobes_live_default_after_fetch() { + local case_dir rc baseline + case_dir=$(make_case content-default-race) + write_meta "$case_dir" no-mistakes ship + wt_commit_file "$case_dir" feature.txt hello "add feature" + land_on_origin_main "$case_dir" feature.txt hello + baseline=$(git --git-dir="$case_dir/origin.git" rev-parse main^) + git --git-dir="$case_dir/origin.git" update-ref refs/heads/trunk "$baseline" + cat > "$case_dir/fakebin/git" <<'SH' +#!/usr/bin/env bash +real=${REAL_GIT_FOR_TEST:?} +is_fetch=0 +for arg in "$@"; do + [ "$arg" = fetch ] && is_fetch=1 +done +if [ "$is_fetch" -eq 1 ]; then + "$real" "$@" + status=$? + if [ "$status" -eq 0 ]; then + "$real" --git-dir="${FM_TEST_ORIGIN:?}" symbolic-ref HEAD refs/heads/trunk + fi + exit "$status" +fi +exec "$real" "$@" +SH + chmod +x "$case_dir/fakebin/git" + + set +e + FM_TEST_ORIGIN="$case_dir/origin.git" \ + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "default rename during landing proof must refuse teardown" + assert_present "$case_dir/wt" "default-rename race discarded the task worktree" + assert_present "$case_dir/state/task-x1.meta" "default-rename race removed task metadata" + assert_grep "live origin default changed during landing proof" "$case_dir/stderr" \ + "default-rename race did not surface the changed live authority" + pass "teardown re-probes live default after fetching" +} + +test_content_fallback_honors_shared_checkout_lock() { + local case_dir rc common key lock_root lock + case_dir=$(make_case content-shared-lock) + write_meta "$case_dir" no-mistakes ship + wt_commit_file "$case_dir" feature.txt hello "add feature" + land_on_origin_main "$case_dir" feature.txt hello + common=$(git -C "$case_dir/wt" rev-parse --git-common-dir) + case "$common" in /*) ;; *) common="$case_dir/wt/$common" ;; esac + common=$(cd "$common" && pwd -P) + key=$(printf '%s' "$common" | shasum -a 256 | awk '{print substr($1,1,24)}') + lock_root="$case_dir/checkout-locks" + lock="$lock_root/$key.lock" + mkdir -p "$lock" + printf '%s\n' "$$" > "$lock/pid" + + set +e + FM_CHECKOUT_REFRESH_LOCK_ROOT="$lock_root" \ + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "contended checkout lock must refuse teardown landing proof" + assert_present "$case_dir/wt" "lock contention discarded the task worktree" + assert_present "$case_dir/state/task-x1.meta" "lock contention removed task metadata" + assert_grep "checkout mutation already running for $case_dir/wt (pid $$)" \ + "$case_dir/stderr" "teardown did not surface shared checkout lock contention" + rm -rf "$lock" + pass "teardown landing proof holds the shared checkout lock" +} + +test_locked_return_reuses_checkout_lock_for_landing_recheck() { + local case_dir rc lock marker + case_dir=$(make_case locked-return-landing-recheck) + write_meta "$case_dir" no-mistakes ship + wt_commit_file "$case_dir" feature.txt hello "add feature" + land_on_origin_main "$case_dir" feature.txt hello + add_stale_lock_on_first_return_treehouse "$case_dir" + add_lsof_no_holder "$case_dir" + lock=$(git_index_lock_path "$case_dir/wt") + marker="$case_dir/treehouse-first-return" + + set +e + TREEHOUSE_FIRST_RETURN_MARKER="$marker" \ + FM_TREEHOUSE_RETURN_LOCK_RETRIES=0 \ + FM_TREEHOUSE_RETURN_LOCK_RETRY_WAIT_SECS=0 \ + FM_STALE_WORKTREE_LOCK_AGE_SECS=1 \ + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 0 "$rc" "locked landing recheck should reuse its already-held checkout lock" + assert_present "$marker" "locked landing recheck did not exercise Treehouse return recovery" + assert_absent "$lock" "locked landing recheck left the stale Git lock behind" + assert_not_contains "$(cat "$case_dir/stderr")" "checkout mutation already running" \ + "locked landing recheck tried to reacquire its non-reentrant checkout lock" + pass "locked Treehouse recovery reuses its checkout lock for landing proof" +} + +test_treehouse_return_timeout_reaps_children_before_unlock() { + local case_dir rc child_pid_file child_pid lock + case_dir=$(make_case treehouse-return-timeout) + write_meta "$case_dir" no-mistakes ship + wt_commit "$case_dir" "shippable work" + git -C "$case_dir/wt" push -q origin fm/task-x1 + git -C "$case_dir/project" fetch -q origin + add_hanging_treehouse "$case_dir" + child_pid_file="$case_dir/treehouse-return-child.pid" + lock=$(checkout_lock_path "$case_dir/wt" "$case_dir/checkout-locks") + + set +e + TREEHOUSE_RETURN_CHILD_PID_FILE="$child_pid_file" FM_TREEHOUSE_RETURN_TIMEOUT=1 \ + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "timed-out Treehouse return should retain the task" + assert_present "$child_pid_file" "timed-out Treehouse return did not start its descendant" + child_pid=$(cat "$child_pid_file") + ! kill -0 "$child_pid" 2>/dev/null \ + || fail "timed-out Treehouse return left descendant $child_pid alive" + assert_absent "$lock" "checkout lock remained held after the Treehouse return process tree was reaped" + assert_present "$case_dir/wt" "timed-out Treehouse return removed the worktree" + assert_present "$case_dir/state/task-x1.meta" "timed-out Treehouse return removed task metadata" + assert_grep "Treehouse return timed out after 1s" "$case_dir/stderr" \ + "timed-out Treehouse return did not surface its timeout" + pass "Treehouse return timeout reaps descendants before releasing checkout lock" +} + test_dirty_worktree_refuses() { local case_dir rc pr_head case_dir=$(make_case dirty-wt) @@ -1423,7 +1753,7 @@ test_fractional_legacy_retry_wait_refuses_without_arithmetic_error() { pass "fractional legacy retry wait remains supported without arithmetic" } -test_local_only_force_overrides_unpushed() { +test_local_only_force_retains_unpushed() { local case_dir rc case_dir=$(make_case force-override) write_meta "$case_dir" local-only ship @@ -1434,9 +1764,12 @@ test_local_only_force_overrides_unpushed() { rc=$? set -e - expect_code 0 "$rc" "force-override: --force should bypass the unpushed-work check" - ! grep -q REFUSED "$case_dir/stderr" || fail "force-override: REFUSED printed despite --force" - pass "local-only worktree with unpushed work is torn down under --force (escape hatch)" + expect_code 1 "$rc" "force-retention: --force must not bypass the unpushed-work check" + assert_present "$case_dir/wt" "force-retention removed a worktree with unpushed work" + assert_present "$case_dir/state/task-x1.meta" "force-retention removed task metadata" + assert_grep 'work not yet merged' "$case_dir/stderr" \ + "force-retention did not surface the unpushed work" + pass "force teardown retains local-only unpushed work" } add_fake_agent_fleet() { @@ -1470,7 +1803,7 @@ SH chmod +x "$case_dir/fakebin/agent-fleet" "$case_dir/fakebin/tmux" } -test_managed_force_teardown_releases_lease_and_session() { +test_managed_force_teardown_retains_unlanded_lease_and_session() { local case_dir af_log rc case_dir=$(make_case managed-force-release) af_log="$case_dir/agent-fleet.log" @@ -1491,11 +1824,12 @@ test_managed_force_teardown_releases_lease_and_session() { rc=$? set -e - expect_code 0 "$rc" "managed-force-release: teardown should succeed: $(cat "$case_dir/stderr")" - assert_grep 'lease release --task fm-home-task-x1-attempt-a1 --force' "$af_log" "managed teardown did not release the lease" - assert_grep 'session remove --task fm-home-task-x1-attempt-a1' "$af_log" "managed teardown did not remove the session mapping" - assert_absent "$case_dir/state/task-x1.meta" "managed teardown left task metadata" - pass "managed teardown releases its lease and session mapping only after endpoint removal" + expect_code 1 "$rc" "managed-force-retention: teardown must retain unlanded work" + assert_no_grep 'lease release' "$af_log" "managed force teardown released the retained lease" + assert_no_grep 'session remove' "$af_log" "managed force teardown removed the retained session mapping" + assert_present "$case_dir/state/task-x1.meta" "managed force teardown removed retry metadata" + assert_present "$case_dir/wt" "managed force teardown removed unlanded work" + pass "managed force teardown retains unlanded lease and session state" } test_managed_teardown_retains_lease_when_endpoint_state_is_unknown() { @@ -1658,8 +1992,7 @@ test_managed_child_teardown_locks_generation_before_snapshot() { kill_started="$case_dir/kill-started" allow_kill="$case_dir/allow-kill" : > "$af_log" - mkdir -p "$case_dir/wt/data" "$case_dir/wt/state" "$case_dir/wt/config" "$case_dir/wt/projects" - printf '%s\n' task-x1 > "$case_dir/wt/.fm-secondmate-home" + prepare_secondmate_home_fixture "$case_dir" fm_write_meta "$case_dir/state/task-x1.meta" \ 'window=fm-task-x1' \ 'tmux_session_target=firstmate:fm-task-x1' \ @@ -1669,6 +2002,7 @@ test_managed_child_teardown_locks_generation_before_snapshot() { 'mode=secondmate' \ "home=$case_dir/wt" fm_git_worktree "$child_project" "$child_worktree" child-branch + write_treehouse_lease "$child_worktree" "firstmate-$child_id" fm_write_meta "$case_dir/wt/state/$child_id.meta" \ "window=fm-$child_id" \ "tmux_session_target=firstmate:fm-$child_id" \ @@ -1765,8 +2099,7 @@ test_forced_secondmate_child_uses_child_home_for_endpoint_verification() { child_id=child-zellij-x2 : > "$af_log" : > "$zellij_log" - mkdir -p "$case_dir/wt/data" "$case_dir/wt/state" "$case_dir/wt/config" "$case_dir/wt/projects" - printf '%s\n' task-x1 > "$case_dir/wt/.fm-secondmate-home" + prepare_secondmate_home_fixture "$case_dir" fm_write_meta "$case_dir/state/task-x1.meta" \ 'window=fm-task-x1' \ 'tmux_session_target=firstmate:fm-task-x1' \ @@ -1776,6 +2109,7 @@ test_forced_secondmate_child_uses_child_home_for_endpoint_verification() { 'mode=secondmate' \ "home=$case_dir/wt" fm_git_worktree "$child_project" "$child_worktree" child-branch + write_treehouse_lease "$child_worktree" "firstmate-$child_id" fm_write_meta "$case_dir/wt/state/$child_id.meta" \ 'window=firstmate:9' \ "worktree=$child_worktree" \ @@ -1836,8 +2170,7 @@ test_forced_secondmate_quiesces_parent_before_child_cleanup() { child_id=child-after-quiesce-x4 parent_live="$case_dir/parent-live" parent_quiesced="$case_dir/parent-quiesced" - mkdir -p "$case_dir/wt/data" "$case_dir/wt/state" "$case_dir/wt/config" "$case_dir/wt/projects" - printf '%s\n' task-x1 > "$case_dir/wt/.fm-secondmate-home" + prepare_secondmate_home_fixture "$case_dir" fm_write_meta "$case_dir/state/task-x1.meta" \ 'window=fm-task-x1' \ 'tmux_session_target=firstmate:fm-task-x1' \ @@ -1847,6 +2180,7 @@ test_forced_secondmate_quiesces_parent_before_child_cleanup() { 'mode=secondmate' \ "home=$case_dir/wt" fm_git_worktree "$child_project" "$child_worktree" child-branch + write_treehouse_lease "$child_worktree" "firstmate-$child_id" fm_write_meta "$case_dir/wt/state/$child_id.meta" \ "window=fm-$child_id" \ "worktree=$child_worktree" \ @@ -1882,6 +2216,259 @@ SH pass "forced secondmate teardown quiesces and verifies the parent before child cleanup" } +setup_forced_secondmate_child_case() { + local name=$1 child_id=$2 + FORCED_CHILD_CASE_DIR=$(make_case "$name") + FORCED_CHILD_PROJECT="$FORCED_CHILD_CASE_DIR/child-project" + FORCED_CHILD_WORKTREE="$FORCED_CHILD_CASE_DIR/child-worktree" + FORCED_CHILD_PARENT_LIVE="$FORCED_CHILD_CASE_DIR/parent-live" + prepare_secondmate_home_fixture "$FORCED_CHILD_CASE_DIR" + fm_write_meta "$FORCED_CHILD_CASE_DIR/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$FORCED_CHILD_CASE_DIR/wt" \ + "project=$FORCED_CHILD_CASE_DIR/project" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$FORCED_CHILD_CASE_DIR/wt" + fm_git_worktree "$FORCED_CHILD_PROJECT" "$FORCED_CHILD_WORKTREE" child-branch + write_treehouse_lease "$FORCED_CHILD_WORKTREE" "firstmate-$child_id" + fm_write_meta "$FORCED_CHILD_CASE_DIR/wt/state/$child_id.meta" \ + "window=fm-$child_id" \ + "worktree=$FORCED_CHILD_WORKTREE" \ + "project=$FORCED_CHILD_PROJECT" \ + 'kind=ship' \ + 'mode=local-only' + : > "$FORCED_CHILD_PARENT_LIVE" + cat > "$FORCED_CHILD_CASE_DIR/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +case "${1:-}" in + display-message) [ -f "$FM_FAKE_PARENT_LIVE" ]; exit $? ;; + list-panes) exit 0 ;; + kill-window) rm -f "$FM_FAKE_PARENT_LIVE"; exit 0 ;; +esac +exit 0 +SH + chmod +x "$FORCED_CHILD_CASE_DIR/fakebin/tmux" +} + +test_forced_secondmate_retains_child_on_treehouse_failure() { + local case_dir child_worktree child_id lock child_pid_file child_ready_file child_pid term_marker rc + child_id=child-return-failure-x6 + setup_forced_secondmate_child_case secondmate-child-return-failure "$child_id" + case_dir=$FORCED_CHILD_CASE_DIR + child_worktree=$FORCED_CHILD_WORKTREE + lock=$(checkout_lock_path "$child_worktree" "$case_dir/checkout-locks") + child_pid_file="$case_dir/treehouse-child.pid" + child_ready_file="$case_dir/treehouse-child.ready" + term_marker="$case_dir/treehouse-child-terminated-under-lock" + cat > "$case_dir/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = return ]; then + ( + trap ' + if [ -e "$FM_EXPECT_CHECKOUT_LOCK" ] || [ -L "$FM_EXPECT_CHECKOUT_LOCK" ]; then + : > "$TREEHOUSE_RETURN_CHILD_TERM_MARKER" + fi + exit 0 + ' TERM + : > "$TREEHOUSE_RETURN_CHILD_READY_FILE" + while :; do + sleep 1 + done + ) & + child=$! + printf '%s\n' "$child" > "$TREEHOUSE_RETURN_CHILD_PID_FILE" + while [ ! -f "$TREEHOUSE_RETURN_CHILD_READY_FILE" ]; do + : + done + exit 17 +fi +exit 0 +SH + chmod +x "$case_dir/fakebin/treehouse" + + set +e + FM_FAKE_PARENT_LIVE="$FORCED_CHILD_PARENT_LIVE" \ + FM_EXPECT_CHECKOUT_LOCK="$lock" \ + TREEHOUSE_RETURN_CHILD_PID_FILE="$child_pid_file" \ + TREEHOUSE_RETURN_CHILD_READY_FILE="$child_ready_file" \ + TREEHOUSE_RETURN_CHILD_TERM_MARKER="$term_marker" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 17 "$rc" "forced secondmate cleanup should preserve the failed Treehouse return status" + assert_present "$child_pid_file" "failed Treehouse return did not start its descendant" + child_pid=$(cat "$child_pid_file") + ! kill -0 "$child_pid" 2>/dev/null \ + || fail "failed Treehouse return left descendant $child_pid alive" + assert_present "$term_marker" "failed Treehouse return released the checkout lock before terminating descendants" + assert_absent "$lock" "failed Treehouse return left the checkout lock held" + assert_present "$child_worktree" "failed Treehouse return deleted the child worktree" + assert_present "$case_dir/wt/state/$child_id.meta" "failed Treehouse return removed child retry metadata" + assert_present "$case_dir/state/task-x1.meta" "failed Treehouse return removed parent retry metadata" + assert_grep "retained child worktree $child_worktree because its locked Treehouse return failed (status 17)" \ + "$case_dir/stderr" "forced secondmate cleanup did not report failed-return retention" + pass "forced secondmate cleanup retains failed returns after reaping descendants" +} + +test_forced_secondmate_retains_unverified_process_group() { + local case_dir child_worktree child_id lock child_pid_file rc group anchor_state owner + child_id=child-return-unverified-x8 + setup_forced_secondmate_child_case secondmate-child-return-unverified "$child_id" + case_dir=$FORCED_CHILD_CASE_DIR + child_worktree=$FORCED_CHILD_WORKTREE + lock=$(checkout_lock_path "$child_worktree" "$case_dir/checkout-locks") + child_pid_file="$case_dir/treehouse-unverified-child.pid" + cat > "$case_dir/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +if [ "${1:-}" = return ]; then + ( + trap '' TERM + while :; do + sleep 1 + done + ) & + printf '%s\n' "$!" > "$TREEHOUSE_RETURN_CHILD_PID_FILE" +fi +exit 0 +SH + cat > "$case_dir/fakebin/ps" <<'SH' +#!/usr/bin/env bash +exit 1 +SH + chmod +x "$case_dir/fakebin/treehouse" "$case_dir/fakebin/ps" + + set +e + FM_FAKE_PARENT_LIVE="$FORCED_CHILD_PARENT_LIVE" \ + TREEHOUSE_RETURN_CHILD_PID_FILE="$child_pid_file" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 76 "$rc" "unverified Treehouse process cleanup should fail distinctly" + assert_present "$child_pid_file" "unverified Treehouse return did not start its descendant" + assert_present "$lock" "unverified process cleanup released the checkout lock" + assert_present "$lock/process-group" "unverified process cleanup lost its guarded group identity" + group=$(cat "$lock/process-group") + anchor_state=$(ps -p "$group" -o pid= -o pgid= 2>/dev/null | awk '{$1=$1; print}') + [ "$anchor_state" = "$group $group" ] \ + || fail "unverified process cleanup did not retain its identity-pinned group anchor" + assert_present "$child_worktree" "unverified process cleanup deleted the child worktree" + assert_present "$case_dir/wt/state/$child_id.meta" "unverified process cleanup removed child retry metadata" + assert_present "$case_dir/state/task-x1.meta" "unverified process cleanup removed parent retry metadata" + assert_grep "bounded command process cleanup could not be verified" "$case_dir/stderr" \ + "unverified process cleanup did not surface the supervisor failure" + assert_grep "Treehouse return process cleanup could not be verified" "$case_dir/stderr" \ + "unverified process cleanup did not surface retain-only Treehouse handling" + owner=$(readlink "$lock") + kill -KILL -- "-$group" 2>/dev/null || true + for _ in $(seq 1 50); do + kill -0 "$group" 2>/dev/null || break + sleep 0.02 + done + ! kill -0 "$group" 2>/dev/null || fail "test cleanup could not terminate retained anchored group $group" + rm -f "$lock" + rm -rf "$owner" + pass "unverified Treehouse process cleanup retains worktree and checkout lock" +} + +test_bounded_runner_preserves_command_status_125() { + local rc + # shellcheck source=bin/fm-process-tree-lib.sh + . "$ROOT/bin/fm-process-tree-lib.sh" + if fm_run_bounded 2 sh -c 'exit 125'; then + rc=0 + else + rc=$? + fi + expect_code 125 "$rc" "bounded runner changed a legitimate command exit 125" + [ "$FM_PROCESS_TREE_CLEANUP_STATUS" = verified ] \ + || fail "bounded runner confused command exit 125 with cleanup failure" + pass "bounded runner preserves command exit 125 separately from cleanup state" +} + +test_forced_secondmate_retains_child_when_treehouse_unavailable() { + local case_dir child_worktree child_id rc + child_id=child-return-unavailable-x7 + setup_forced_secondmate_child_case secondmate-child-return-unavailable "$child_id" + case_dir=$FORCED_CHILD_CASE_DIR + child_worktree=$FORCED_CHILD_WORKTREE + rm -f "$case_dir/fakebin/treehouse" + + set +e + PATH=/usr/bin:/bin FM_FAKE_PARENT_LIVE="$FORCED_CHILD_PARENT_LIVE" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 127 "$rc" "forced secondmate cleanup should surface unavailable Treehouse distinctly" + assert_present "$child_worktree" "unavailable Treehouse deleted the child worktree" + assert_present "$case_dir/wt/state/$child_id.meta" "unavailable Treehouse removed child retry metadata" + assert_present "$case_dir/state/task-x1.meta" "unavailable Treehouse removed parent retry metadata" + assert_grep "retained child worktree $child_worktree because Treehouse is unavailable" \ + "$case_dir/stderr" "forced secondmate cleanup did not report unavailable-return retention" + pass "forced secondmate cleanup retains child worktree when Treehouse is unavailable" +} + +test_forced_secondmate_retains_child_on_checkout_lock_contention() { + local case_dir child_project child_worktree child_id lock_root lock parent_live rc + case_dir=$(make_case secondmate-child-checkout-contention) + child_project="$case_dir/child-project" + child_worktree="$case_dir/child-worktree" + child_id=child-contention-x5 + lock_root="$case_dir/checkout-locks" + prepare_secondmate_home_fixture "$case_dir" + fm_write_meta "$case_dir/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$case_dir/wt" \ + "project=$case_dir/project" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$case_dir/wt" + fm_git_worktree "$child_project" "$child_worktree" child-branch + write_treehouse_lease "$child_worktree" "firstmate-$child_id" + fm_write_meta "$case_dir/wt/state/$child_id.meta" \ + "window=fm-$child_id" \ + "worktree=$child_worktree" \ + "project=$child_project" \ + 'kind=ship' \ + 'mode=local-only' + lock=$(checkout_lock_path "$child_worktree" "$lock_root") + mkdir -p "$lock" + printf '%s\n' "$$" > "$lock/pid" + parent_live="$case_dir/parent-live" + : > "$parent_live" + cat > "$case_dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +case "${1:-}" in + display-message) [ -f "$FM_FAKE_PARENT_LIVE" ]; exit $? ;; + list-panes) exit 0 ;; + kill-window) rm -f "$FM_FAKE_PARENT_LIVE"; exit 0 ;; +esac +exit 0 +SH + chmod +x "$case_dir/fakebin/tmux" + + set +e + FM_FAKE_PARENT_LIVE="$parent_live" FM_CHECKOUT_REFRESH_LOCK_ROOT="$lock_root" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 75 "$rc" "forced secondmate cleanup should surface checkout lock contention distinctly" + assert_present "$child_worktree" "checkout lock contention deleted the child worktree" + assert_present "$case_dir/wt/state/$child_id.meta" "checkout lock contention removed child retry metadata" + assert_present "$case_dir/state/task-x1.meta" "checkout lock contention removed parent retry metadata" + assert_grep "checkout mutation already running for $child_worktree (pid $$)" "$case_dir/stderr" \ + "forced secondmate cleanup did not surface checkout lock contention" + assert_grep "retained child worktree $child_worktree because its common checkout mutation lock is busy" \ + "$case_dir/stderr" "forced secondmate cleanup did not report retention on contention" + pass "forced secondmate cleanup retains child worktree on checkout lock contention" +} + test_herdr_teardown_clears_escalation_marker() { local case_dir marker case_dir=$(make_case herdr-marker-cleanup) @@ -2079,28 +2666,440 @@ SH pass "required report teardown quiesces before its final safety validation" } -test_teardown_refuses_unsafe_tasktmp_metadata() { - local case_dir sentinel rc - case_dir=$(make_case unsafe-tasktmp) - sentinel="$case_dir/must-survive" - write_meta "$case_dir" local-only ship - mkdir -p "$sentinel" - printf 'preserve\n' > "$sentinel/marker" - printf 'tasktmp=%s\n' "$sentinel" >> "$case_dir/state/task-x1.meta" - - set +e - run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" - rc=$? - set -e - expect_code 1 "$rc" "unsafe tasktmp teardown exit" - assert_present "$sentinel/marker" "teardown deleted a metadata-selected arbitrary directory" - assert_present "$case_dir/state/task-x1.meta" "unsafe tasktmp refusal removed task metadata" - assert_grep 'unsafe task temp path' "$case_dir/stderr" "unsafe teardown tasktmp refusal was unclear" - pass "teardown only removes its exact task temp root" -} - -test_teardown_rejects_malformed_report_requirement() { - local case_dir rc +test_legacy_teardown_revalidates_after_quiescence() { + local case_dir live rc + case_dir=$(make_case legacy-post-quiesce-safety) + live="$case_dir/legacy-endpoint-live" + write_meta "$case_dir" no-mistakes ship + cat > "$case_dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +case "${1:-}" in + display-message) [ -f "$FM_FAKE_REPORT_LIVE" ]; exit $? ;; + list-windows) exit 0 ;; + kill-window) + printf 'late work\n' > "$FM_FAKE_WORKTREE/late-work.txt" + rm -f "$FM_FAKE_REPORT_LIVE" + exit 0 + ;; +esac +exit 0 +SH + chmod +x "$case_dir/fakebin/tmux" + : > "$live" + + set +e + FM_FAKE_REPORT_LIVE="$live" FM_FAKE_WORKTREE="$case_dir/wt" \ + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "legacy teardown: post-quiescence dirty work must refuse return" + assert_absent "$live" "legacy teardown validated before endpoint quiescence" + assert_present "$case_dir/wt/late-work.txt" "legacy teardown discarded post-quiescence work" + assert_present "$case_dir/state/task-x1.meta" "legacy teardown removed metadata after safety refusal" + assert_grep 'endpoint has already been shut down; the worktree and task metadata are preserved' "$case_dir/stderr" \ + "legacy teardown did not explain its post-quiescence fail-safe state" + pass "legacy teardown quiesces before landed-work validation" +} + +test_teardown_rejects_nested_metadata_roots_before_quiescence() { + local case_dir marker nested tmp rc + case_dir=$(make_case nested-teardown-root) + write_meta "$case_dir" no-mistakes ship + nested="$case_dir/project/nested" + marker="$case_dir/endpoint-killed" + mkdir -p "$nested" + tmp="$case_dir/meta.tmp" + sed "s#^project=.*#project=$nested#" "$case_dir/state/task-x1.meta" > "$tmp" + mv "$tmp" "$case_dir/state/task-x1.meta" + cat > "$case_dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +[ "${1:-}" != kill-window ] || : > "$FM_FAKE_KILL_MARKER" +exit 0 +SH + chmod +x "$case_dir/fakebin/tmux" + + set +e + FM_FAKE_KILL_MARKER="$marker" run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "nested teardown project metadata must be rejected" + assert_absent "$marker" "teardown quiesced an endpoint before validating metadata roots" + assert_present "$case_dir/state/task-x1.meta" "nested metadata refusal removed task metadata" + assert_grep 'not an exact inspectable repository root' "$case_dir/stderr" \ + "nested metadata root refusal was unclear" + pass "teardown validates exact repository identity before mutation" +} + +test_teardown_rejects_drifted_treehouse_task_lease() { + local case_dir marker rc + case_dir=$(make_case drifted-treehouse-task-lease) + write_meta "$case_dir" no-mistakes ship + write_treehouse_lease "$case_dir/wt" firstmate-other-task + marker="$case_dir/endpoint-killed" + cat > "$case_dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +[ "${1:-}" != kill-window ] || : > "$FM_FAKE_KILL_MARKER" +exit 0 +SH + chmod +x "$case_dir/fakebin/tmux" + + set +e + FM_FAKE_KILL_MARKER="$marker" run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "drifted Treehouse task lease must be rejected" + assert_absent "$marker" "teardown stopped an endpoint before proving Treehouse task ownership" + assert_present "$case_dir/state/task-x1.meta" "Treehouse ownership refusal removed task metadata" + assert_grep "expected 'firstmate-task-x1'" "$case_dir/stderr" \ + "Treehouse ownership refusal did not identify the expected task holder" + pass "teardown requires the recorded Treehouse lease holder" +} + +test_teardown_rechecks_treehouse_lease_after_locked_safety() { + local case_dir count_file return_marker state rc branch + case_dir=$(make_case lease-drift-during-locked-safety) + write_meta "$case_dir" no-mistakes ship + count_file="$case_dir/status-count" + return_marker="$case_dir/treehouse-returned" + state="$TMP_ROOT/treehouse-state.json" + cat > "$case_dir/fakebin/git" <<'SH' +#!/usr/bin/env bash +set -u +case " $* " in + *" status --porcelain"*) + count=0 + [ ! -f "$FM_FAKE_STATUS_COUNT" ] || count=$(cat "$FM_FAKE_STATUS_COUNT") + count=$((count + 1)) + printf '%s\n' "$count" > "$FM_FAKE_STATUS_COUNT" + if [ "$count" -eq 2 ]; then + python3 - "$FM_FAKE_TREEHOUSE_STATE" <<'PY' +import json +import sys + +path = sys.argv[1] +with open(path, encoding="utf-8") as stream: + state = json.load(stream) +state["worktrees"][0]["lease_holder"] = "firstmate-other-task" +with open(path, "w", encoding="utf-8") as stream: + json.dump(state, stream) +PY + fi + ;; +esac +exec "$REAL_GIT_FOR_TEST" "$@" +SH + cat > "$case_dir/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +: > "$FM_FAKE_TREEHOUSE_RETURNED" +exit 0 +SH + chmod +x "$case_dir/fakebin/git" "$case_dir/fakebin/treehouse" + + set +e + FM_FAKE_STATUS_COUNT="$count_file" FM_FAKE_TREEHOUSE_STATE="$state" \ + FM_FAKE_TREEHOUSE_RETURNED="$return_marker" \ + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "lease reassignment during locked safety checks must abort return" + assert_absent "$return_marker" "Treehouse return ran after the task lease changed" + assert_present "$case_dir/wt" "lease reassignment removed the task worktree" + assert_present "$case_dir/state/task-x1.meta" "lease reassignment removed task metadata" + branch=$("$REAL_GIT_FOR_TEST" -C "$case_dir/wt" symbolic-ref --quiet --short HEAD) + [ "$branch" = fm/task-x1 ] || fail "lease reassignment detached or deleted the task branch" + assert_grep "ownership changed during final safety checks" "$case_dir/stderr" \ + "lease reassignment was not surfaced at the locked mutation boundary" + pass "locked teardown rechecks ownership before return mutation" +} + +test_secondmate_rejects_drifted_home_repository_identity() { + local case_dir marker rc + case_dir=$(make_case secondmate-home-identity-drift) + mkdir -p "$case_dir/wt/data" "$case_dir/wt/state" "$case_dir/wt/config" "$case_dir/wt/projects" + printf '%s\n' task-x1 > "$case_dir/wt/.fm-secondmate-home" + fm_write_meta "$case_dir/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$case_dir/wt" \ + "project=$case_dir/project" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$case_dir/wt" + marker="$case_dir/endpoint-killed" + cat > "$case_dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +[ "${1:-}" != kill-window ] || : > "$FM_FAKE_KILL_MARKER" +exit 0 +SH + chmod +x "$case_dir/fakebin/tmux" + + set +e + FM_FAKE_KILL_MARKER="$marker" run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "secondmate home repository identity drift must block teardown" + assert_absent "$marker" "secondmate endpoint was stopped before home identity was proved" + assert_present "$case_dir/wt" "identity-drifted secondmate home was removed" + assert_present "$case_dir/state/task-x1.meta" "identity-drifted secondmate metadata was removed" + assert_grep "secondmate home repository identity does not match" "$case_dir/stderr" \ + "secondmate home identity drift was not surfaced" + pass "secondmate teardown proves its home repository identity" +} + +test_normal_secondmate_retires_proven_detached_head() { + local case_dir rc + case_dir=$(make_case normal-secondmate-quiescence) + prepare_secondmate_home_fixture "$case_dir" + git -C "$case_dir/wt" checkout --quiet --detach + fm_write_meta "$case_dir/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$case_dir/wt" \ + "project=$case_dir/project" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$case_dir/wt" + + set +e + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 0 "$rc" "normal secondmate retirement should quiesce and complete" + assert_absent "$case_dir/fakebin/.tmux-live" "normal secondmate retirement left its endpoint alive" + assert_absent "$case_dir/wt" "normal secondmate retirement retained its home" + pass "normal secondmate retirement proves endpoint absence" +} + +test_forced_secondmate_retains_untracked_skill_draft() { + local case_dir draft rc + case_dir=$(make_case secondmate-untracked-skill) + prepare_secondmate_home_fixture "$case_dir" + fm_write_meta "$case_dir/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$case_dir/wt" \ + "project=$case_dir/wt" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$case_dir/wt" + draft="$case_dir/wt/.claude/skills/new-skill/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' draft > "$draft" + + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "forced secondmate retirement must retain an untracked skill draft" + assert_present "$case_dir/fakebin/.tmux-live" "dirty-home refusal stopped the secondmate endpoint" + assert_present "$draft" "forced secondmate retirement discarded an untracked skill draft" + assert_present "$case_dir/state/task-x1.meta" "dirty-home refusal removed secondmate metadata" + assert_grep "new-skill/SKILL.md" "$case_dir/stderr" \ + "dirty-home refusal did not surface the untracked skill draft" + pass "forced secondmate retirement retains untracked skill drafts" +} + +test_forced_secondmate_retains_unique_detached_head() { + local case_dir rc + case_dir=$(make_case secondmate-clean-unique-commit) + prepare_secondmate_home_fixture "$case_dir" + git -C "$case_dir/wt" checkout --quiet --detach + fm_write_meta "$case_dir/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$case_dir/wt" \ + "project=$case_dir/wt" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$case_dir/wt" + printf '%s\n' unique > "$case_dir/wt/unique.txt" + git -C "$case_dir/wt" add unique.txt + git -C "$case_dir/wt" -c user.name=tests -c user.email=tests@example.invalid \ + commit --quiet -m unique + + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "forced secondmate retirement must retain a clean unique commit" + assert_present "$case_dir/fakebin/.tmux-live" "unique-commit refusal stopped the secondmate endpoint" + assert_present "$case_dir/wt/unique.txt" "forced secondmate retirement discarded a clean unique commit" + assert_present "$case_dir/state/task-x1.meta" "unique-commit refusal removed secondmate metadata" + assert_grep "not proven in authoritative" "$case_dir/stderr" \ + "unique-commit refusal did not surface the unlanded ref" + pass "forced secondmate retirement retains clean unique commits" +} + +test_forced_secondmate_retains_stash() { + local case_dir rc + case_dir=$(make_case secondmate-retained-stash) + prepare_secondmate_home_fixture "$case_dir" + fm_write_meta "$case_dir/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$case_dir/wt" \ + "project=$case_dir/wt" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$case_dir/wt" + printf '%s\n' retained > "$case_dir/wt/retained-stash.txt" + git -C "$case_dir/wt" add retained-stash.txt + git -C "$case_dir/wt" stash push --quiet + + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "forced secondmate retirement must retain stash history" + assert_present "$case_dir/wt" "forced secondmate retirement removed a home with a stash" + assert_present "$case_dir/state/task-x1.meta" "stash refusal removed secondmate metadata" + [ -n "$(git -C "$case_dir/wt" stash list)" ] || fail "forced secondmate retirement discarded the stash" + assert_grep "retained stash history" "$case_dir/stderr" \ + "stash refusal was not surfaced" + pass "forced secondmate retirement retains stash history" +} + +test_forced_secondmate_retains_unlanded_child_work() { + local case_dir child_id child_worktree rc + child_id=child-unlanded-x7 + setup_forced_secondmate_child_case secondmate-child-unlanded "$child_id" + case_dir=$FORCED_CHILD_CASE_DIR + child_worktree=$FORCED_CHILD_WORKTREE + printf '%s\n' retained > "$child_worktree/untracked-child.txt" + + set +e + FM_FAKE_PARENT_LIVE="$FORCED_CHILD_PARENT_LIVE" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "forced secondmate cleanup must retain unlanded child work" + assert_present "$child_worktree/untracked-child.txt" "forced cleanup discarded untracked child work" + assert_present "$case_dir/wt/state/$child_id.meta" "forced cleanup removed child retry metadata" + assert_present "$case_dir/state/task-x1.meta" "forced cleanup removed parent retry metadata" + assert_grep "uncommitted changes" "$case_dir/stderr" \ + "forced child retention did not surface uncommitted work" + pass "forced secondmate cleanup retains unlanded child work" +} + +test_forced_secondmate_retains_unquiesced_unmanaged_child() { + local case_dir child_id child_project child_worktree parent_live child_live rc + case_dir=$(make_case secondmate-unmanaged-child-live) + child_id=child-live-x9 + child_project="$case_dir/child-project" + child_worktree="$case_dir/child-worktree" + parent_live="$case_dir/parent-live" + child_live="$case_dir/child-live" + prepare_secondmate_home_fixture "$case_dir" + fm_write_meta "$case_dir/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$case_dir/wt" \ + "project=$case_dir/project" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$case_dir/wt" + fm_git_worktree "$child_project" "$child_worktree" child-branch + write_treehouse_lease "$child_worktree" "firstmate-$child_id" + fm_write_meta "$case_dir/wt/state/$child_id.meta" \ + "window=fm-$child_id" \ + "tmux_session_target=firstmate:fm-$child_id" \ + "worktree=$child_worktree" \ + "project=$child_project" \ + 'kind=ship' \ + 'mode=local-only' + : > "$parent_live" + : > "$child_live" + cat > "$case_dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +set -u +target= +prev= +for arg in "$@"; do + [ "$prev" = -t ] && target=$arg + prev=$arg +done +case "${1:-}" in + display-message) + case "$target" in + *task-x1) [ -f "$FM_FAKE_PARENT_LIVE" ] ;; + *child-live-x9) [ -f "$FM_FAKE_CHILD_LIVE" ] ;; + *) exit 1 ;; + esac + exit $? + ;; + list-windows) + [ -f "$FM_FAKE_PARENT_LIVE" ] && printf '%s\n' fm-task-x1 + [ -f "$FM_FAKE_CHILD_LIVE" ] && printf '%s\n' fm-child-live-x9 + ;; + kill-window) + case "$target" in + *child-live-x9) exit 74 ;; + *) rm -f "$FM_FAKE_PARENT_LIVE" ;; + esac + ;; +esac +exit 0 +SH + chmod +x "$case_dir/fakebin/tmux" + + set +e + FM_FAKE_PARENT_LIVE="$parent_live" FM_FAKE_CHILD_LIVE="$child_live" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "a live unmanaged child must block forced secondmate cleanup" + assert_present "$child_live" "the child endpoint fixture unexpectedly disappeared" + assert_present "$case_dir/wt" "forced cleanup removed the parent home after child quiescence failed" + assert_present "$child_worktree" "forced cleanup removed a worktree with a live child endpoint" + assert_present "$case_dir/wt/state/$child_id.meta" "forced cleanup removed live-child retry metadata" + assert_present "$case_dir/state/task-x1.meta" "forced cleanup removed parent metadata after child quiescence failed" + assert_grep "child endpoint for $child_id is still alive" "$case_dir/stderr" \ + "forced cleanup did not surface the surviving child endpoint" + pass "forced secondmate cleanup retains unquiesced unmanaged children" +} + +test_teardown_retains_untracked_claude_skill_draft() { + local case_dir draft rc + case_dir=$(make_case retained-claude-skill) + write_meta "$case_dir" no-mistakes ship + draft="$case_dir/wt/.claude/skills/draft/SKILL.md" + mkdir -p "$(dirname "$draft")" + printf '%s\n' '# draft' > "$draft" + + set +e + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "untracked .claude skill draft must refuse teardown" + assert_present "$draft" "teardown discarded an untracked .claude skill draft" + assert_present "$case_dir/state/task-x1.meta" "skill-draft refusal removed task metadata" + assert_grep 'has uncommitted changes' "$case_dir/stderr" \ + "skill-draft refusal did not surface uncommitted work" + pass "teardown retains untracked .claude skill drafts" +} + +test_teardown_refuses_unsafe_tasktmp_metadata() { + local case_dir sentinel rc + case_dir=$(make_case unsafe-tasktmp) + sentinel="$case_dir/must-survive" + write_meta "$case_dir" local-only ship + mkdir -p "$sentinel" + printf 'preserve\n' > "$sentinel/marker" + printf 'tasktmp=%s\n' "$sentinel" >> "$case_dir/state/task-x1.meta" + + set +e + run_teardown "$case_dir" > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "unsafe tasktmp teardown exit" + assert_present "$sentinel/marker" "teardown deleted a metadata-selected arbitrary directory" + assert_present "$case_dir/state/task-x1.meta" "unsafe tasktmp refusal removed task metadata" + assert_grep 'unsafe task temp path' "$case_dir/stderr" "unsafe teardown tasktmp refusal was unclear" + pass "teardown only removes its exact task temp root" +} + +test_teardown_rejects_malformed_report_requirement() { + local case_dir rc case_dir=$(make_case malformed-report-required) write_meta "$case_dir" local-only ship printf 'report_required=0\n' >> "$case_dir/state/task-x1.meta" @@ -2108,21 +3107,1184 @@ test_teardown_rejects_malformed_report_requirement() { run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" rc=$? set -e - expect_code 1 "$rc" "malformed report_required teardown exit" - assert_present "$case_dir/state/task-x1.meta" "malformed report_required metadata was destructively bypassed" - assert_grep 'invalid report_required metadata' "$case_dir/stderr" \ - "malformed report_required refusal was unclear" + expect_code 1 "$rc" "malformed report_required teardown exit" + assert_present "$case_dir/state/task-x1.meta" "malformed report_required metadata was destructively bypassed" + assert_grep 'invalid report_required metadata' "$case_dir/stderr" \ + "malformed report_required refusal was unclear" + + case_dir=$(make_case duplicate-report-required) + write_meta "$case_dir" local-only ship + printf 'report_required=1\nreport_required=1\n' >> "$case_dir/state/task-x1.meta" + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "duplicate report_required teardown exit" + assert_present "$case_dir/state/task-x1.meta" "duplicate report_required metadata was destructively bypassed" + pass "teardown treats only one exact report_required marker as valid" +} + +write_secondmate_meta() { + local case_dir=$1 home + home=${2:-$case_dir/wt} + fm_write_meta "$case_dir/state/task-x1.meta" \ + 'window=fm-task-x1' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + "worktree=$home" \ + "project=$home" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$home" +} + +test_secondmate_state_enumeration_fails_closed() { + local case_dir rc + case_dir=$(make_case missing-secondmate-state) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + rmdir "$case_dir/wt/state" + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "missing secondmate state teardown exit" + assert_present "$case_dir/wt" "missing state allowed secondmate home removal" + assert_present "$case_dir/state/task-x1.meta" "missing state allowed secondmate metadata removal" + assert_grep 'secondmate child state is unprovable' "$case_dir/stderr" \ + "missing secondmate state was not surfaced" + + case_dir=$(make_case unreadable-secondmate-state) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + chmod 100 "$case_dir/wt/state" + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + chmod 700 "$case_dir/wt/state" + expect_code 1 "$rc" "unreadable secondmate state teardown exit" + assert_present "$case_dir/wt" "unreadable state allowed secondmate home removal" + assert_present "$case_dir/state/task-x1.meta" "unreadable state allowed secondmate metadata removal" + pass "secondmate child-state enumeration fails closed" +} + +test_secondmate_missing_treehouse_child_is_retained() { + local case_dir child_id missing_worktree rc + case_dir=$(make_case missing-treehouse-child) + child_id=missing-child + missing_worktree="$case_dir/missing-child-worktree" + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + fm_write_meta "$case_dir/wt/state/$child_id.meta" \ + "window=fm-$child_id" \ + "tmux_session_target=firstmate:fm-$child_id" \ + "worktree=$missing_worktree" \ + "project=$case_dir/project" \ + 'harness=claude' \ + 'kind=ship' \ + 'mode=local-only' + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "missing Treehouse child teardown exit" + assert_present "$case_dir/wt/state/$child_id.meta" \ + "missing Treehouse child metadata was forgotten" + assert_present "$case_dir/wt" "missing Treehouse child allowed parent home removal" + assert_grep 'Treehouse worktree is missing or uninspectable' "$case_dir/stderr" \ + "missing Treehouse child lease blocker was not surfaced" + pass "missing Treehouse children retain their metadata and parent home" +} + +test_secondmate_registry_home_drift_blocks_removal() { + local case_dir copied rc + case_dir=$(make_case secondmate-registry-drift) + prepare_secondmate_home_fixture "$case_dir" + copied="$case_dir/copied-home" + cp -R "$case_dir/wt" "$copied" + write_secondmate_meta "$case_dir" "$copied" + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "secondmate registry drift teardown exit" + assert_present "$copied" "registry drift allowed copied home removal" + assert_present "$case_dir/wt" "registry drift damaged the registered home" + assert_present "$case_dir/data/secondmates.md" "registry drift removed the real registration" + assert_grep 'secondmate registry home for task-x1' "$case_dir/stderr" \ + "secondmate registry drift was not surfaced" + pass "secondmate retirement requires exact registry-home ownership" +} + +test_retained_direct_spawn_requires_confirmed_endpoint_quiescence() { + local case_dir rc + case_dir=$(make_case retained-direct-spawn-quiescence) + write_meta "$case_dir" local-only ship + printf '%s\n' \ + 'tmux_session_target=firstmate:fm-task-x1' \ + 'account_home=/tmp/direct-account-home' \ + 'direct_spawn_cleanup=pending' \ + 'rollback_pending=1' >> "$case_dir/state/task-x1.meta" + cat > "$case_dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +case "${1:-}" in + kill-window) exit 0 ;; + list-windows) echo "control plane unavailable" >&2; exit 1 ;; + display-message) exit 1 ;; +esac +exit 0 +SH + chmod +x "$case_dir/fakebin/tmux" + + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + + expect_code 1 "$rc" "retained direct-spawn teardown should fail on unknown endpoint state" + assert_grep 'retained direct-spawn endpoint state for task-x1 is unknown' "$case_dir/stderr" \ + "retained direct-spawn teardown did not explain the endpoint blocker" + assert_present "$case_dir/wt/.git" "retained direct-spawn teardown recycled the worktree without endpoint proof" + assert_present "$case_dir/state/task-x1.meta" "retained direct-spawn teardown erased cleanup metadata without endpoint proof" + pass "retained direct-spawn teardown requires confirmed endpoint quiescence" +} + +test_secondmate_registry_duplicate_home_blocks_removal() { + local case_dir home rc + case_dir=$(make_case secondmate-registry-duplicate-home) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + home=$(cd "$case_dir/wt" && pwd -P) + printf '%s\n' "- other-secondmate - duplicate home (home: $home; scope: test; projects: test; added 2026-07-23)" \ + >> "$case_dir/data/secondmates.md" + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "duplicate secondmate registry home teardown exit" + assert_present "$case_dir/wt" "duplicate registry home allowed secondmate removal" + assert_present "$case_dir/state/task-x1.meta" "duplicate registry home allowed metadata removal" + assert_grep 'secondmate registry is malformed, duplicated, redirected' "$case_dir/stderr" \ + "duplicate secondmate registry home was not surfaced" + pass "secondmate retirement rejects registry home aliases" +} + +test_secondmate_retirement_retains_idle_registered_child() { + local case_dir child_home parent_home rc + case_dir=$(make_case secondmate-idle-registered-child) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + child_home="$case_dir/idle-child-home" + mkdir -p "$child_home" + parent_home=$(cd "$case_dir/wt" && pwd -P) + child_home=$(cd "$child_home" && pwd -P) + printf '%s\n' "- idle-child - idle child (home: $child_home; scope: idle; projects: ; added 2026-07-23)" \ + > "$parent_home/data/secondmates.md" + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "idle registered child must block parent retirement" + assert_present "$case_dir/wt" "idle registered child allowed parent home removal" + assert_present "$child_home" "idle registered child home was removed" + assert_present "$case_dir/state/task-x1.meta" "idle registered child allowed parent metadata removal" + assert_grep 'has no inspectable child metadata' "$case_dir/stderr" \ + "idle registered child was not surfaced" + pass "parent retirement retains idle externally registered children" +} + +test_secondmate_retirement_retains_unlanded_project_clone() { + local case_dir rc + case_dir=$(make_case secondmate-unlanded-project-clone) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + printf 'draft\n' > "$case_dir/wt/projects/test/unlanded.txt" + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "unlanded nested project clone must block retirement" + assert_present "$case_dir/wt/projects/test/unlanded.txt" "force retirement discarded nested project work" + assert_present "$case_dir/wt" "unlanded nested project clone allowed parent home removal" + assert_present "$case_dir/state/task-x1.meta" "unlanded nested project clone allowed metadata removal" + assert_grep 'project clone has unlanded changes' "$case_dir/stderr" \ + "nested project clone work was not surfaced" + pass "force retirement retains unlanded nested project clones" +} + +test_secondmate_project_tags_do_not_prove_landing() { + local case_dir clone rc + case_dir=$(make_case secondmate-project-tag-only) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + clone="$case_dir/wt/projects/test" + printf 'tag only\n' > "$clone/tag-only.txt" + git -C "$clone" add tag-only.txt + git -C "$clone" commit -qm "tag-only project work" + git -C "$clone" tag tag-only-proof + git -C "$clone" push -q origin refs/tags/tag-only-proof + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "tag-only project reachability must block retirement" + assert_present "$clone/tag-only.txt" "tag-only project work was discarded" + assert_present "$case_dir/state/task-x1.meta" "tag-only project work allowed metadata removal" + assert_grep 'not proven on a live remote branch' "$case_dir/stderr" \ + "tag-only reachability was not distinguished from a durable branch" + pass "remote tags alone never prove secondmate project work landed" +} + +test_secondmate_project_origin_authority_survives_home_removal() { + local drift_case drift_clone drift_origin in_home_case in_home_clone in_home_origin rc + drift_case=$(make_case secondmate-project-origin-drift) + prepare_secondmate_home_fixture "$drift_case" + write_secondmate_meta "$drift_case" + drift_clone="$drift_case/wt/projects/test" + drift_origin="$drift_case/drift-origin.git" + git clone --quiet --bare "$drift_case/origin.git" "$drift_origin" + git -C "$drift_clone" remote set-url origin "$drift_origin" + set +e + run_teardown "$drift_case" --force > "$drift_case/stdout" 2> "$drift_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "drifted project origin must block retirement" + assert_present "$drift_clone" "drifted project origin allowed clone removal" + assert_grep 'origin drifted from its registered source' "$drift_case/stderr" \ + "drifted project origin was not surfaced" + + in_home_case=$(make_case secondmate-project-in-home-origin) + prepare_secondmate_home_fixture "$in_home_case" + write_secondmate_meta "$in_home_case" + in_home_clone="$in_home_case/wt/projects/test" + in_home_origin="$in_home_case/wt/data/in-home-origin.git" + git clone --quiet --bare "$in_home_case/origin.git" "$in_home_origin" + git -C "$in_home_clone" remote set-url origin "$in_home_origin" + set +e + run_teardown "$in_home_case" --force > "$in_home_case/stdout" 2> "$in_home_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "in-home project origin must block retirement" + assert_present "$in_home_origin" "in-home landing authority was deleted" + assert_grep 'does not survive home removal' "$in_home_case/stderr" \ + "in-home project landing authority was not surfaced" + pass "project landing authority is bound and survives secondmate removal" +} + +test_secondmate_retirement_recurses_into_ignored_nested_repositories() { + local case_dir source_clone clone nested_origin nested rc + case_dir=$(make_case secondmate-nested-project-repository) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + source_clone="$case_dir/source-projects/test" + clone="$case_dir/wt/projects/test" + nested_origin="$case_dir/nested-origin.git" + git clone --quiet --bare "$case_dir/origin.git" "$nested_origin" + git -C "$source_clone" -c protocol.file.allow=always submodule add -q \ + "$nested_origin" vendor/nested + git -C "$source_clone" commit -qm "register nested project fixture" + git -C "$source_clone" push -q origin main + git -C "$clone" pull -q --ff-only + git -C "$clone" -c protocol.file.allow=always submodule update -q --init + nested="$clone/vendor/nested" + printf 'unlanded nested work\n' > "$nested/unlanded.txt" + git -C "$nested" add unlanded.txt + git -C "$nested" commit -qm "unlanded nested repository work" + git -C "$clone" add vendor/nested + git -C "$clone" commit -qm "reference unlanded nested repository work" + git -C "$clone" push -q origin main + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "unpushed submodule repository must block retirement" + assert_present "$nested/unlanded.txt" "unpushed submodule repository work was discarded" + assert_present "$case_dir/state/task-x1.meta" "nested repository work allowed metadata removal" + assert_grep 'not proven on a live remote branch' "$case_dir/stderr" \ + "unlanded nested repository ref was not surfaced" + pass "secondmate retirement recursively proves submodule repositories" +} + +test_secondmate_retirement_rejects_linked_worktree_graphs() { + local nested_case nested_source nested_clone nested_worktree nested_exclude external_case external_clone external_worktree rc + nested_case=$(make_case secondmate-nested-linked-worktree) + prepare_secondmate_home_fixture "$nested_case" + write_secondmate_meta "$nested_case" + nested_source="$nested_case/source-projects/test" + nested_clone="$nested_case/wt/projects/test" + nested_worktree="$nested_clone/linked-owned-elsewhere" + nested_exclude="$(git -C "$nested_clone" rev-parse --absolute-git-dir)/info/exclude" + printf '%s\n' '/linked-owned-elsewhere/' >> "$nested_exclude" + git clone --quiet "$nested_case/origin.git" "$nested_source/linked-owned-elsewhere" + git -C "$nested_source" worktree add -q -b linked-retirement "$nested_worktree" main + set +e + run_teardown "$nested_case" --force > "$nested_case/stdout" 2> "$nested_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "nested linked worktree must block retirement" + assert_present "$nested_worktree" "nested linked worktree was removed without unregistering its owner" + assert_present "$nested_case/state/task-x1.meta" "nested linked worktree allowed metadata removal" + assert_grep 'linked-worktree graph depends on the retiring home' "$nested_case/stderr" \ + "nested linked-worktree ownership was not surfaced" + + external_case=$(make_case secondmate-external-linked-worktree) + prepare_secondmate_home_fixture "$external_case" + write_secondmate_meta "$external_case" + external_clone="$external_case/wt/projects/test" + external_worktree="$external_case/external-linked-worktree" + git -C "$external_clone" worktree add -q -b external-retirement "$external_worktree" main + set +e + run_teardown "$external_case" --force > "$external_case/stdout" 2> "$external_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "external linked worktree must block common-directory removal" + assert_present "$external_worktree" "external linked worktree lost its common Git directory" + assert_present "$external_case/state/task-x1.meta" "external linked worktree allowed metadata removal" + assert_grep 'owns another linked worktree' "$external_case/stderr" \ + "external linked worktree was not surfaced" + pass "secondmate retirement retains every linked-worktree graph" +} + +test_secondmate_retirement_accounts_for_directory_symlinks() { + local escape_case escape_clone escape_exclude external cycle_case cycle_clone cycle_exclude rc + escape_case=$(make_case secondmate-project-symlink-escape) + prepare_secondmate_home_fixture "$escape_case" + write_secondmate_meta "$escape_case" + escape_clone="$escape_case/wt/projects/test" + external="$escape_case/external-repository" + git clone --quiet "$escape_case/origin.git" "$external" + ln -s "$external" "$escape_clone/escaping-repository" + escape_exclude="$(git -C "$escape_clone" rev-parse --absolute-git-dir)/info/exclude" + printf '%s\n' '/escaping-repository' >> "$escape_exclude" + set +e + run_teardown "$escape_case" --force > "$escape_case/stdout" 2> "$escape_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "escaping repository symlink must block retirement" + assert_present "$escape_case/wt" "escaping repository symlink allowed home removal" + assert_grep 'nested project repositories cannot be safely enumerated' "$escape_case/stderr" \ + "escaping repository symlink was not surfaced" + + cycle_case=$(make_case secondmate-project-symlink-cycle) + prepare_secondmate_home_fixture "$cycle_case" + write_secondmate_meta "$cycle_case" + cycle_clone="$cycle_case/wt/projects/test" + mkdir -p "$cycle_clone/cycle" + ln -s .. "$cycle_clone/cycle/back" + cycle_exclude="$(git -C "$cycle_clone" rev-parse --absolute-git-dir)/info/exclude" + printf '%s\n' '/cycle/' >> "$cycle_exclude" + set +e + run_teardown "$cycle_case" --force > "$cycle_case/stdout" 2> "$cycle_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "repository symlink cycle must block retirement" + assert_present "$cycle_case/wt" "repository symlink cycle allowed home removal" + assert_grep 'nested project repositories cannot be safely enumerated' "$cycle_case/stderr" \ + "repository symlink cycle was not surfaced" + pass "secondmate retirement accounts for directory symlinks" +} + +test_secondmate_retirement_rejects_loopback_and_stale_tracking_authority() { + local loopback_case loopback_clone loopback_source loopback_origin stale_case stale_clone unique_tip rc + loopback_case=$(make_case secondmate-loopback-origin) + prepare_secondmate_home_fixture "$loopback_case" + write_secondmate_meta "$loopback_case" + loopback_clone="$loopback_case/wt/projects/test" + loopback_source="$loopback_case/source-projects/test" + loopback_origin="$loopback_case/wt/data/loopback-origin.git" + git clone --quiet --bare "$loopback_case/origin.git" "$loopback_origin" + git -C "$loopback_clone" remote set-url origin "ssh://localhost$loopback_origin" + git -C "$loopback_source" remote set-url origin "ssh://localhost$loopback_origin" + set +e + run_teardown "$loopback_case" --force > "$loopback_case/stdout" 2> "$loopback_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "in-home loopback origin must block retirement" + assert_present "$loopback_origin" "loopback landing authority inside the home was deleted" + assert_grep 'does not survive home removal' "$loopback_case/stderr" \ + "loopback landing authority was assumed durable" + + stale_case=$(make_case secondmate-stale-remote-tracking-ref) + prepare_secondmate_home_fixture "$stale_case" + write_secondmate_meta "$stale_case" + stale_clone="$stale_case/wt/projects/test" + git -C "$stale_clone" checkout -q -b stale-only + printf 'stale tracking work\n' > "$stale_clone/stale-tracking.txt" + git -C "$stale_clone" add stale-tracking.txt + git -C "$stale_clone" commit -qm "stale tracking work" + unique_tip=$(git -C "$stale_clone" rev-parse HEAD) + git -C "$stale_clone" update-ref refs/remotes/origin/deleted "$unique_tip" + git -C "$stale_clone" checkout -q main + git -C "$stale_clone" branch -D stale-only >/dev/null + set +e + run_teardown "$stale_case" --force > "$stale_case/stdout" 2> "$stale_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "stale remote-tracking ref must block retirement" + assert_present "$stale_case/wt" "stale remote-tracking work was discarded" + assert_present "$stale_case/state/task-x1.meta" "stale remote-tracking ref allowed metadata removal" + assert_grep 'ref refs/remotes/origin/deleted is not proven on a live remote branch' "$stale_case/stderr" \ + "stale remote-tracking work was not surfaced" + pass "loopback and stale remote-tracking authority never prove landing" +} + +test_secondmate_retirement_rejects_mount_boundaries() { + local case_dir mounted rc root_case + case_dir=$(make_case secondmate-project-mount-boundary) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + mounted="$case_dir/wt/projects/test/mounted-storage" + mkdir -p "$mounted" + set +e + FM_TEARDOWN_TEST_MOUNT_PATH="$mounted" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "mounted project subtree must block retirement" + assert_present "$case_dir/wt" "mounted project subtree allowed home removal" + assert_present "$case_dir/state/task-x1.meta" "mounted project subtree allowed metadata removal" + assert_grep 'crosses an untrusted filesystem boundary' "$case_dir/stderr" \ + "mounted project subtree was not surfaced" + + root_case=$(make_case secondmate-mounted-removal-root) + prepare_secondmate_home_fixture "$root_case" + write_secondmate_meta "$root_case" + set +e + FM_TEARDOWN_TEST_MOUNT_PATH="$root_case/wt" \ + run_teardown "$root_case" --force > "$root_case/stdout" 2> "$root_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "mounted secondmate home root must block retirement" + assert_present "$root_case/wt" "mounted secondmate home root was traversed" + assert_present "$root_case/state/task-x1.meta" "mounted secondmate home removed metadata" + assert_grep 'crosses an untrusted filesystem boundary' "$root_case/stderr" \ + "mounted secondmate home root was not surfaced" + + root_case=$(make_case treehouse-mounted-return-root) + write_meta "$root_case" local-only ship + set +e + FM_TEARDOWN_TEST_MOUNT_PATH="$root_case/wt" \ + run_teardown "$root_case" --force > "$root_case/stdout" 2> "$root_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "mounted Treehouse return root must block teardown" + assert_present "$root_case/wt" "mounted Treehouse worktree was returned" + assert_present "$root_case/state/task-x1.meta" "mounted Treehouse return removed metadata" + pass "secondmate retirement refuses mounted deletion boundaries" +} + +test_secondmate_retirement_rejects_effective_ssh_redirects() { + local case_dir clone source redirect local_origin rc + case_dir=$(make_case secondmate-effective-ssh-redirect) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + clone="$case_dir/wt/projects/test" + source="$case_dir/source-projects/test" + local_origin="$case_dir/wt/data/redirected-origin.git" + redirect="$case_dir/fakebin/redirect-ssh" + git clone --quiet --bare "$case_dir/origin.git" "$local_origin" + cat > "$redirect" <<'SH' +#!/usr/bin/env bash +exec git-upload-pack "${FM_REDIRECT_ORIGIN:?}" +SH + chmod +x "$redirect" + git -C "$clone" remote set-url origin ssh://8.8.8.8/repository.git + git -C "$source" remote set-url origin ssh://8.8.8.8/repository.git + git -C "$clone" config core.sshCommand "$redirect" + git -C "$source" config core.sshCommand "$redirect" + set +e + FM_REDIRECT_ORIGIN="$local_origin" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "effective SSH redirect must block retirement" + assert_present "$local_origin" "effective SSH redirect landing authority was deleted" + assert_present "$case_dir/state/task-x1.meta" "effective SSH redirect allowed metadata removal" + assert_grep 'remote identity is unsafe' "$case_dir/stderr" \ + "effective SSH transport override was not surfaced" + pass "effective Git SSH transport must match landing validation" +} + +test_secondmate_retirement_rejects_incomplete_surviving_authority() { + local case_dir source rc promisor_case promisor_source + case_dir=$(make_case secondmate-shallow-source-authority) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + source="$case_dir/source-projects/test" + rm -rf "$source" + git clone --quiet --depth 1 "file://$case_dir/origin.git" "$source" + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "shallow source authority must block retirement" + assert_present "$case_dir/wt" "shallow source authority allowed home removal" + assert_grep 'is shallow and does not prove a complete surviving object graph' \ + "$case_dir/stderr" "shallow source authority was accepted" + + promisor_case=$(make_case secondmate-promisor-source-authority) + prepare_secondmate_home_fixture "$promisor_case" + write_secondmate_meta "$promisor_case" + promisor_source="$promisor_case/source-projects/test" + git -C "$promisor_source" config core.repositoryFormatVersion 1 + git -C "$promisor_source" config extensions.partialClone origin + git -C "$promisor_source" config remote.origin.promisor true + set +e + run_teardown "$promisor_case" --force > "$promisor_case/stdout" 2> "$promisor_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "promisor source authority must block retirement" + assert_present "$promisor_case/wt" "promisor source authority allowed home removal" + assert_grep 'uses promisor or partial-clone object semantics' \ + "$promisor_case/stderr" "promisor source authority was accepted" + pass "surviving authorities require complete local object graphs" +} - case_dir=$(make_case duplicate-report-required) - write_meta "$case_dir" local-only ship - printf 'report_required=1\nreport_required=1\n' >> "$case_dir/state/task-x1.meta" +test_secondmate_retirement_validates_top_level_source_storage() { + local case_dir owner source root_ref root_tip rc + case_dir=$(make_case secondmate-top-source-storage) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + owner="$case_dir/wt/data/top-source-owner" + source="$case_dir/top-source" + root_ref=$(git -C "$ROOT" symbolic-ref --quiet --short refs/remotes/origin/HEAD 2>/dev/null || printf 'origin/main') + root_tip=$(git -C "$ROOT" rev-parse "$root_ref") + git clone --quiet "$ROOT" "$owner" + git -C "$owner" checkout --quiet --detach "$root_tip" + git -C "$owner" branch --force main "$root_tip" + git -C "$owner" update-ref refs/remotes/origin/main "$root_tip" + git -C "$owner" symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main + git -C "$owner" worktree add --quiet "$source" main + git -C "$case_dir/project" remote set-url origin "$source" + set +e + FM_ROOT_OVERRIDE="$source" \ + FM_HOME="$source" \ + FM_STATE_OVERRIDE="$case_dir/state" \ + FM_DATA_OVERRIDE="$case_dir/data" \ + FM_CONFIG_OVERRIDE="$case_dir/config" \ + FM_PROJECTS_OVERRIDE="$case_dir/source-projects" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$case_dir/checkout-locks" \ + FM_FAKE_FIRSTMATE_SOURCE="$source" \ + FM_ACCOUNT_ROUTING_TEST_LAB=firstmate-account-routing-test-lab-v1 \ + PATH="$case_dir/fakebin:$PATH" \ + "$TEARDOWN" task-x1 --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "top-level source storage inside retiring home must block retirement" + assert_present "$owner/.git" "top-level source storage inside retiring home was deleted" + assert_present "$case_dir/state/task-x1.meta" "top-level source storage allowed metadata removal" + assert_grep 'secondmate top-level source repository Git storage depends on the retiring home' \ + "$case_dir/stderr" "top-level source storage graph was not validated" + pass "top-level source storage must survive secondmate retirement" +} + +test_secondmate_retirement_rejects_local_network_aliases() { + local case_dir clone source ssh_home local_origin rc + case_dir=$(make_case secondmate-local-network-alias) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + clone="$case_dir/wt/projects/test" + source="$case_dir/source-projects/test" + local_origin="$case_dir/wt/data/local-alias-origin.git" + ssh_home="$case_dir/ssh-home" + mkdir -p "$ssh_home/.ssh" + chmod 700 "$ssh_home/.ssh" + git clone --quiet --bare "$case_dir/origin.git" "$local_origin" + printf '%s\n' \ + 'Host local-store-alias' \ + ' HostName localhost' \ + > "$ssh_home/.ssh/config" + chmod 600 "$ssh_home/.ssh/config" + git -C "$clone" remote set-url origin "ssh://local-store-alias$local_origin" + git -C "$source" remote set-url origin "ssh://local-store-alias$local_origin" + set +e + FM_TEST_TEARDOWN_HOME="$ssh_home" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "local SSH alias must block retirement" + assert_present "$local_origin" "local SSH alias landing authority was deleted" + assert_present "$case_dir/state/task-x1.meta" "local SSH alias allowed metadata removal" + assert_grep 'remote identity is unsafe' "$case_dir/stderr" \ + "local SSH alias was assumed to be durable network storage" + pass "network-shaped local aliases never prove durable landing" +} + +test_secondmate_retirement_rejects_in_home_remote_object_storage() { + local case_dir clone source in_home_origin external_authority rc + case_dir=$(make_case secondmate-remote-object-alternate) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + clone="$case_dir/wt/projects/test" + source="$case_dir/source-projects/test" + in_home_origin="$case_dir/wt/data/in-home-object-authority.git" + external_authority="$case_dir/external-shared-authority.git" + git clone --quiet --bare "$case_dir/origin.git" "$in_home_origin" + git clone --quiet --bare --shared "$in_home_origin" "$external_authority" + git -C "$clone" remote set-url origin "$external_authority" + git -C "$source" remote set-url origin "$external_authority" set +e run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" rc=$? set -e - expect_code 1 "$rc" "duplicate report_required teardown exit" - assert_present "$case_dir/state/task-x1.meta" "duplicate report_required metadata was destructively bypassed" - pass "teardown treats only one exact report_required marker as valid" + expect_code 1 "$rc" "remote object alternate inside the home must block retirement" + assert_present "$in_home_origin" "remote object authority inside the home was deleted" + assert_present "$case_dir/state/task-x1.meta" "in-home remote object storage allowed metadata removal" + assert_grep 'object storage is not independently durable' "$case_dir/stderr" \ + "remote object alternates were not included in survival proof" + pass "landing authority object storage must survive home removal" +} + +test_secondmate_retirement_rejects_source_common_dir_in_home() { + local case_dir source owner rc + case_dir=$(make_case secondmate-source-common-in-home) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + source="$case_dir/source-projects/test" + owner="$case_dir/wt/data/source-owner" + rm -rf "$source" + git clone --quiet "$case_dir/origin.git" "$owner" + git -C "$owner" worktree add --quiet --detach "$source" main + set +e + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "source project common directory inside the home must block retirement" + assert_present "$owner/.git" "source project common directory inside the home was deleted" + assert_present "$case_dir/state/task-x1.meta" "source common-directory drift allowed metadata removal" + assert_grep 'registered source project repository Git storage depends on the retiring home' \ + "$case_dir/stderr" "source project storage graph was not validated" + pass "registered source storage must survive secondmate removal" +} + +test_teardown_removal_roots_fail_closed() { + local missing_case retained caller rc symlink_case target + missing_case=$(make_case missing-secondmate-removal-root) + prepare_secondmate_home_fixture "$missing_case" + write_secondmate_meta "$missing_case" + retained="$missing_case/retained-home" + caller="$missing_case/caller" + mv "$missing_case/wt" "$retained" + mkdir -p "$caller" + printf 'caller sentinel\n' > "$caller/sentinel" + set +e + ( + cd "$caller" || exit 1 + run_teardown "$missing_case" --force + ) > "$missing_case/stdout" 2> "$missing_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "missing removal root must fail closed" + assert_present "$caller/sentinel" "missing removal root deleted the caller working directory" + assert_present "$retained/.fm-secondmate-home" "missing removal root deleted retained home data" + assert_present "$missing_case/state/task-x1.meta" "missing removal root removed retry metadata" + assert_grep 'missing secondmate home removal target' "$missing_case/stderr" \ + "missing removal root was not surfaced" + + symlink_case=$(make_case symlinked-secondmate-removal-root) + prepare_secondmate_home_fixture "$symlink_case" + write_secondmate_meta "$symlink_case" + target="$symlink_case/retained-home" + mv "$symlink_case/wt" "$target" + ln -s "$target" "$symlink_case/wt" + set +e + run_teardown "$symlink_case" --force > "$symlink_case/stdout" 2> "$symlink_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "symlinked removal root must fail closed" + assert_present "$target/.fm-secondmate-home" "symlinked removal root traversed its target" + assert_present "$symlink_case/state/task-x1.meta" "symlinked removal root removed retry metadata" + assert_grep 'error:' "$symlink_case/stderr" "symlinked removal root was not surfaced" + pass "missing and redirected removal roots never select another directory" +} + +test_treehouse_return_stays_bound_to_validated_root() { + local case_dir moved redirect marker rc + case_dir=$(make_case treehouse-return-root-swap) + write_meta "$case_dir" local-only ship + moved="$case_dir/moved-worktree" + redirect="$case_dir/redirect-target" + marker="$case_dir/bound-root-observed" + mkdir -p "$case_dir/wt/retained-descendant" + printf 'validated worktree\n' > "$case_dir/wt/.bound-root-identity" + printf 'retained descendant\n' > "$case_dir/wt/retained-descendant/identity" + git -C "$case_dir/wt" add .bound-root-identity retained-descendant/identity + git -C "$case_dir/wt" -c user.email=t@t -c user.name=t \ + commit -qm "record bound worktree identity" + add_fork_with_pushed_branch "$case_dir" + rm -f "$case_dir/fakebin/.tmux-live" + cat > "$case_dir/fakebin/treehouse" <<'SH' +#!/usr/bin/env bash +[ "$1" = return ] && [ "$2" = --force ] || exit 91 +[ "$3" = "." ] || exit 92 +[ "$FM_TREEHOUSE_RETURN_PROJECT" = "$FM_TREEHOUSE_EXPECT_PROJECT" ] || exit 93 +bound_target=$3 +old_ifs=$IFS +IFS=, +set -- $FM_TREEHOUSE_RETURN_BOUNDARY_FDS +IFS=$old_ifs +[ "$#" -ge 2 ] || exit 95 +for descriptor in "$@"; do + [ -d "/dev/fd/$descriptor" ] || exit 96 +done +mv "$FM_TREEHOUSE_SWAP_TARGET" "$FM_TREEHOUSE_MOVED_TARGET" || exit 92 +mkdir -p "$FM_TREEHOUSE_REDIRECT_TARGET" || exit 93 +printf 'redirect target\n' > "$FM_TREEHOUSE_REDIRECT_TARGET/.bound-root-identity" +ln -s "$FM_TREEHOUSE_REDIRECT_TARGET" "$FM_TREEHOUSE_SWAP_TARGET" || exit 94 +[ "$(cat "$bound_target/.bound-root-identity")" = "validated worktree" ] || exit 97 +[ "$(cat "$bound_target/retained-descendant/identity")" = "retained descendant" ] || exit 98 +: > "$FM_TREEHOUSE_BOUND_MARKER" +exit 0 +SH + chmod +x "$case_dir/fakebin/treehouse" + set +e + FM_TREEHOUSE_SWAP_TARGET="$case_dir/wt" \ + FM_TREEHOUSE_MOVED_TARGET="$moved" \ + FM_TREEHOUSE_REDIRECT_TARGET="$redirect" \ + FM_TREEHOUSE_EXPECT_PROJECT="$case_dir/project" \ + FM_TREEHOUSE_BOUND_MARKER="$marker" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + [ "$rc" -eq 0 ] || [ "$rc" -eq 1 ] || fail "identity-bound Treehouse return exited unexpectedly" + [ -e "$marker" ] || fail \ + "Treehouse return did not execute from the validated worktree descriptor: $(cat "$case_dir/stderr")" + assert_grep 'redirect target' "$redirect/.bound-root-identity" \ + "Treehouse return traversed the replacement symlink target" + pass "Treehouse return remains bound across pathname replacement" +} + +test_teardown_distinguishes_dead_and_live_harness_processes() { + local dead_case live_case rc + dead_case=$(make_case dead-harness-endpoint) + write_meta "$dead_case" local-only ship + cat > "$dead_case/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +state="$(dirname "$0")/.tmux-live" +case "${1:-}" in + display-message) + [ -f "$state" ] || exit 1 + case " $* " in *pane_current_command*) printf 'zsh\n' ;; esac + ;; + list-windows) [ ! -f "$state" ] || printf '%s\n' fm-task-x1 ;; + kill-window) rm -f "$state" ;; +esac +exit 0 +SH + chmod +x "$dead_case/fakebin/tmux" + run_teardown "$dead_case" --force > "$dead_case/stdout" 2> "$dead_case/stderr" \ + || fail "dead harness endpoint false-refused teardown: $(cat "$dead_case/stderr")" + assert_absent "$dead_case/fakebin/.tmux-live" \ + "dead harness process left its managed endpoint behind" + assert_absent "$dead_case/state/task-x1.meta" "dead harness endpoint retained task metadata" + + live_case=$(make_case live-harness-endpoint) + write_meta "$live_case" local-only ship + cat > "$live_case/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +case "${1:-}" in + display-message) + case " $* " in *pane_current_command*) printf 'codex\n' ;; esac + exit 0 + ;; + list-windows) printf '%s\n' fm-task-x1 ;; + kill-window) exit 0 ;; +esac +exit 0 +SH + chmod +x "$live_case/fakebin/tmux" + set +e + run_teardown "$live_case" --force > "$live_case/stdout" 2> "$live_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "live harness process must block teardown" + assert_present "$live_case/state/task-x1.meta" "live harness process removed retry metadata" + assert_grep 'endpoint for task-x1 is still alive' "$live_case/stderr" \ + "live harness process was not surfaced" + pass "teardown distinguishes dead harnesses from live processes" +} + +test_secondmate_retirement_retains_reflog_and_rewritten_history() { + local reflog_case clone unique_tip rc rewrite_case grafts + reflog_case=$(make_case secondmate-reflog-only-history) + prepare_secondmate_home_fixture "$reflog_case" + write_secondmate_meta "$reflog_case" + clone="$reflog_case/wt/projects/test" + git -C "$clone" checkout -q -b reflog-only + printf 'recoverable history\n' > "$clone/reflog-only.txt" + git -C "$clone" add reflog-only.txt + git -C "$clone" commit -qm "reflog-only project work" + unique_tip=$(git -C "$clone" rev-parse HEAD) + git -C "$clone" reset -q --hard HEAD^ + git -C "$clone" checkout -q main + git -C "$clone" branch -D reflog-only >/dev/null + set +e + run_teardown "$reflog_case" --force > "$reflog_case/stdout" 2> "$reflog_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "reflog-only project history must block retirement" + assert_present "$reflog_case/wt" "reflog-only project history allowed home removal" + assert_present "$reflog_case/state/task-x1.meta" "reflog-only history removed retry metadata" + assert_grep "$unique_tip" "$reflog_case/stderr" "reflog-only commit was not inventoried" + + rewrite_case=$(make_case secondmate-grafted-history) + prepare_secondmate_home_fixture "$rewrite_case" + write_secondmate_meta "$rewrite_case" + clone="$rewrite_case/wt/projects/test" + grafts=$(git -C "$clone" rev-parse --git-path info/grafts) + case "$grafts" in /*) ;; *) grafts="$clone/$grafts" ;; esac + mkdir -p "$(dirname "$grafts")" + printf '%s\n' "$(git -C "$clone" rev-parse HEAD)" > "$grafts" + set +e + run_teardown "$rewrite_case" --force > "$rewrite_case/stdout" 2> "$rewrite_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "grafted project history must block retirement" + assert_present "$rewrite_case/wt" "grafted project history allowed home removal" + assert_grep 'uses local grafted history' "$rewrite_case/stderr" \ + "grafted history was not surfaced" + pass "retirement retains reflog-only and rewritten Git history" +} + +test_secondmate_retirement_rejects_http_proxy_and_object_redirects() { + local proxy_case clone source rc object_case objects pack redirected + proxy_case=$(make_case secondmate-scoped-http-proxy) + prepare_secondmate_home_fixture "$proxy_case" + write_secondmate_meta "$proxy_case" + clone="$proxy_case/wt/projects/test" + source="$proxy_case/source-projects/test" + git -C "$clone" remote set-url origin https://example.com/repository.git + git -C "$source" remote set-url origin https://example.com/repository.git + git -C "$clone" config 'http.https://example.com.proxy' http://127.0.0.1:9 + git -C "$source" config 'http.https://example.com.proxy' http://127.0.0.1:9 + set +e + run_teardown "$proxy_case" --force > "$proxy_case/stdout" 2> "$proxy_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "URL-scoped HTTP proxy must block retirement" + assert_present "$proxy_case/wt" "URL-scoped HTTP proxy allowed home removal" + assert_grep 'remote identity is unsafe' "$proxy_case/stderr" \ + "URL-scoped HTTP proxy was not surfaced" + + object_case=$(make_case secondmate-object-file-redirect) + prepare_secondmate_home_fixture "$object_case" + write_secondmate_meta "$object_case" + source="$object_case/source-projects/test" + git -C "$source" gc --quiet --prune=now + objects=$(git -C "$source" rev-parse --git-path objects) + case "$objects" in /*) ;; *) objects="$source/$objects" ;; esac + pack=$(find "$objects/pack" -type f -name '*.pack' -print -quit) + [ -n "$pack" ] || fail "object redirect fixture did not create a pack" + redirected="$object_case/wt/data/$(basename "$pack")" + mv "$pack" "$redirected" + ln -s "$redirected" "$pack" + set +e + run_teardown "$object_case" --force > "$object_case/stdout" 2> "$object_case/stderr" + rc=$? + set -e + expect_code 1 "$rc" "redirected pack storage must block retirement" + assert_present "$redirected" "redirected pack storage inside home was deleted" + assert_present "$object_case/state/task-x1.meta" "redirected object storage removed retry metadata" + assert_grep 'redirected object storage entry' "$object_case/stderr" \ + "redirected pack storage was not surfaced" + pass "HTTP routing and object-file redirects never prove durable landing" +} + +test_secondmate_network_fetches_pin_validated_addresses() { + local case_dir clone source tip rc count + case_dir=$(make_case secondmate-pinned-network-authority) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + clone="$case_dir/wt/projects/test" + source="$case_dir/source-projects/test" + tip=$(git -C "$source" rev-parse refs/remotes/origin/main) + git -C "$clone" remote set-url origin https://example.com/repository.git + git -C "$source" remote set-url origin https://example.com/repository.git + cat > "$case_dir/fakebin/git" <<'SH' +#!/usr/bin/env bash +set -u +network_operation= +pin= +any_pin= +repository= +previous= +network_target= +for argument in "$@"; do + if [ "$previous" = -C ]; then + repository=$argument + fi + case "$argument" in + ls-remote|fetch) network_operation=$argument ;; + http.curloptResolve=example.com:443:*) pin=$argument; any_pin=$argument ;; + http.curloptResolve=*) any_pin=$argument ;; + https://example.com/repository.git) network_target=1 ;; + esac + previous=$argument +done +case " $* " in + *" ls-remote --symref origin HEAD "*|*" ls-remote --symref origin HEAD") + if [ "$repository" = "$FM_FAKE_FIRSTMATE_SOURCE" ]; then + [ -n "$any_pin" ] || exit 88 + remote_head=$("$FM_REAL_GIT" -C "$FM_FAKE_FIRSTMATE_SOURCE" \ + symbolic-ref --quiet --short refs/remotes/origin/HEAD 2>/dev/null || printf 'origin/main') + remote_tip=$("$FM_REAL_GIT" -C "$FM_FAKE_FIRSTMATE_SOURCE" rev-parse "$remote_head") + printf 'ref: refs/heads/%s\tHEAD\n%s\tHEAD\n' "${remote_head#origin/}" "$remote_tip" + exit 0 + fi + ;; +esac +if [ -n "$network_operation" ]; then + if [ -z "$network_target" ] && [ -n "$repository" ]; then + remote=$("$FM_REAL_GIT" -C "$repository" remote get-url origin 2>/dev/null || true) + [ "$remote" != https://example.com/repository.git ] || network_target=1 + fi + [ -n "$network_target" ] || exec "$FM_REAL_GIT" "$@" + printf '%s\t%s\n' "$network_operation" "$pin" >> "$FM_GIT_PIN_LOG" + [ -n "$pin" ] || exit 88 + if [ "$network_operation" = ls-remote ]; then + printf 'ref: refs/heads/main\tHEAD\n%s\tHEAD\n' "$FM_PINNED_TIP" + exit 0 + fi + exit 89 +fi +exec "$FM_REAL_GIT" "$@" +SH + chmod +x "$case_dir/fakebin/git" + : > "$case_dir/pinned-network.log" + set +e + FM_REAL_GIT="$(command -v git)" \ + FM_GIT_PIN_LOG="$case_dir/pinned-network.log" \ + FM_PINNED_TIP="$tip" \ + FM_TEARDOWN_TEST_NETWORK_ADDRESSES=203.0.113.10 \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + expect_code 1 "$rc" "synthetic pinned authority fetch stops before retirement" + count=$(wc -l < "$case_dir/pinned-network.log" | tr -d ' ') + [ "$count" -ge 1 ] || fail \ + "network authority did not exercise its graph probe: $(cat "$case_dir/pinned-network.log"); teardown: $(cat "$case_dir/stderr")" + if grep -v $'\thttp.curloptResolve=example.com:443:' "$case_dir/pinned-network.log" >/dev/null; then + fail "network authority re-resolved an unpinned hostname: $(cat "$case_dir/pinned-network.log")" + fi + assert_present "$case_dir/wt" "failed pinned authority proof removed the secondmate home" + pass "network authority fetches retain validated address bindings" +} + +test_surviving_object_storage_is_bound_through_graph_proof() { + local case_dir source objects pack redirected marker release teardown_pid rc waited + case_dir=$(make_case secondmate-object-storage-toctou) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + source="$case_dir/source-projects/test" + git -C "$source" gc --quiet --prune=now + objects=$(git -C "$source" rev-parse --git-path objects) + case "$objects" in /*) ;; *) objects="$source/$objects" ;; esac + pack=$(find "$objects/pack" -type f -name '*.pack' -print -quit) + [ -n "$pack" ] || fail "object-storage identity fixture did not create a pack" + redirected="$case_dir/wt/data/$(basename "$pack")" + marker="$case_dir/object-scan-ready" + release="$case_dir/object-scan-release" + FM_TEARDOWN_TEST_OBJECT_SCAN_ROOT="$objects" \ + FM_TEARDOWN_TEST_OBJECT_SCAN_MARKER="$marker" \ + FM_TEARDOWN_TEST_OBJECT_SCAN_RELEASE="$release" \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" & + teardown_pid=$! + waited=0 + while [ ! -f "$marker" ] && [ "$waited" -lt 200 ]; do + sleep 0.05 + waited=$((waited + 1)) + done + if [ ! -f "$marker" ]; then + : > "$release" + wait "$teardown_pid" || true + fail "object-storage graph proof did not expose its retained-identity boundary" + fi + mv "$pack" "$redirected" + ln -s "$redirected" "$pack" + : > "$release" + set +e + wait "$teardown_pid" + rc=$? + set -e + expect_code 1 "$rc" "object storage replacement during graph proof must fail closed" + assert_present "$case_dir/wt" "object storage replacement allowed secondmate home removal" + assert_grep 'object storage identity changed during graph proof' "$case_dir/stderr" \ + "object storage replacement was not detected under retained identities" + pass "surviving object storage remains identity-bound through graph proof" +} + +test_secondmate_retirement_serializes_child_spawn() { + local case_dir child_project rc teardown_pid spawn_rc waited + case_dir=$(make_case secondmate-retirement-child-race) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + child_project="$case_dir/wt/projects/child-project" + fm_git_init_commit "$child_project" + mkdir -p "$case_dir/wt/data/child" + printf '%s\n' 'Do bounded child work.' > "$case_dir/wt/data/child/brief.md" + cat > "$case_dir/fakebin/tmux" <<'SH' +#!/usr/bin/env bash +state="$(dirname "$0")/.tmux-live" +started="$(dirname "$0")/.retirement-started" +release="$(dirname "$0")/.retirement-release" +case "${1:-}" in + display-message) [ -f "$state" ]; exit $? ;; + list-windows) [ ! -f "$state" ] || printf '%s\n' fm-task-x1; exit 0 ;; + kill-window) + : > "$started" + while [ ! -f "$release" ]; do sleep 0.05; done + rm -f "$state" + exit 0 + ;; +esac +exit 0 +SH + chmod +x "$case_dir/fakebin/tmux" + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" & + teardown_pid=$! + waited=0 + while [ ! -f "$case_dir/fakebin/.retirement-started" ] && [ "$waited" -lt 200 ]; do + sleep 0.05 + waited=$((waited + 1)) + done + [ -f "$case_dir/fakebin/.retirement-started" ] || { + : > "$case_dir/fakebin/.retirement-release" + wait "$teardown_pid" || true + fail "secondmate retirement did not reach the serialized quiescence boundary" + } + set +e + FM_HOME="$case_dir/wt" \ + FM_ROOT_OVERRIDE="$ROOT" \ + FM_CHECKOUT_REFRESH_LOCK_ROOT="$case_dir/checkout-locks" \ + FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS=0 \ + FM_SPAWN_NO_GUARD=1 \ + PATH="$case_dir/fakebin:$PATH" \ + "$ROOT/bin/fm-spawn.sh" child child-project claude \ + > "$case_dir/spawn-stdout" 2> "$case_dir/spawn-stderr" + spawn_rc=$? + set -e + : > "$case_dir/fakebin/.retirement-release" + set +e + wait "$teardown_pid" + rc=$? + set -e + expect_code 1 "$spawn_rc" "child spawn during secondmate retirement exit" + assert_grep 'secondmate home lifecycle lock' "$case_dir/spawn-stderr" \ + "child spawn did not contend on the retiring secondmate home" + expect_code 0 "$rc" "serialized secondmate retirement exit" + assert_absent "$case_dir/wt" "serialized secondmate retirement retained the home" + assert_absent "$case_dir/state/task-x1.meta" "serialized secondmate retirement retained metadata" + pass "secondmate retirement serializes child spawn through removal" +} + +test_nested_secondmate_cleanup_requires_child_home_lock() { + local case_dir nested holder_pid waited rc + case_dir=$(make_case nested-secondmate-home-lock) + prepare_secondmate_home_fixture "$case_dir" + write_secondmate_meta "$case_dir" + nested="$case_dir/nested-home" + git clone --quiet "$case_dir/wt" "$nested" + mkdir -p "$nested/data" "$nested/state" "$nested/config" "$nested/projects" + printf '%s\n' nested > "$nested/.fm-secondmate-home" + printf '%s\n' "- nested - nested secondmate (home: $nested; scope: nested; projects: ; added 2026-07-23)" \ + > "$case_dir/wt/data/secondmates.md" + fm_write_meta "$case_dir/wt/state/nested.meta" \ + 'window=fm-nested' \ + 'tmux_session_target=firstmate:fm-nested' \ + "worktree=$nested" \ + "project=$nested" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$nested" + bash -c ' + . "$1/bin/fm-account-routing-lib.sh" + lock=$(fm_secondmate_home_lifecycle_lock_acquire "$2" "$3") || exit 1 + : > "$4" + while [ ! -f "$5" ]; do sleep 0.05; done + fm_account_lifecycle_lock_release "$lock" + ' _ "$ROOT" "$case_dir/checkout-locks" "$nested" \ + "$case_dir/nested-lock-ready" "$case_dir/nested-lock-release" & + holder_pid=$! + waited=0 + while [ ! -f "$case_dir/nested-lock-ready" ] && [ "$waited" -lt 200 ]; do + sleep 0.05 + waited=$((waited + 1)) + done + [ -f "$case_dir/nested-lock-ready" ] || { + : > "$case_dir/nested-lock-release" + wait "$holder_pid" || true + fail "nested secondmate lock holder did not start" + } + set +e + FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS=0 \ + run_teardown "$case_dir" --force > "$case_dir/stdout" 2> "$case_dir/stderr" + rc=$? + set -e + : > "$case_dir/nested-lock-release" + wait "$holder_pid" || fail "nested secondmate lock holder failed to release" + expect_code 1 "$rc" "nested secondmate home lock teardown exit" + assert_present "$nested" "nested lock contention allowed child home removal" + assert_present "$case_dir/wt/state/nested.meta" "nested lock contention removed child metadata" + assert_present "$case_dir/wt" "nested lock contention allowed parent home removal" + assert_grep 'secondmate home lifecycle lock' "$case_dir/stderr" \ + "nested secondmate home lock contention was not surfaced" + pass "recursive secondmate cleanup acquires each child home lock" +} + +test_secondmate_registry_updates_are_locked_and_literal() { + local case_dir id other_home holder_pid waited rc + case_dir=$(make_case secondmate-registry-locked-literal) + id='foo.bar' + prepare_secondmate_home_fixture "$case_dir" "$id" + fm_write_meta "$case_dir/state/$id.meta" \ + "window=fm-$id" \ + "tmux_session_target=firstmate:fm-$id" \ + "worktree=$case_dir/wt" \ + "project=$case_dir/wt" \ + 'kind=secondmate' \ + 'mode=secondmate' \ + "home=$case_dir/wt" + other_home="$case_dir/other-home" + mkdir -p "$other_home" + printf '%s\n' "- fooxbar - retained neighbor (home: $other_home; scope: neighbor; projects: test; added 2026-07-23)" \ + >> "$case_dir/data/secondmates.md" + bash -c ' + . "$1/bin/fm-account-routing-lib.sh" + lock=$(fm_secondmate_registry_lock_acquire "$2" "$3") || exit 1 + : > "$4" + while [ ! -f "$5" ]; do sleep 0.05; done + fm_account_lifecycle_lock_release "$lock" + ' _ "$ROOT" "$case_dir/checkout-locks" "$case_dir/data/secondmates.md" \ + "$case_dir/registry-lock-ready" "$case_dir/registry-lock-release" & + holder_pid=$! + waited=0 + while [ ! -f "$case_dir/registry-lock-ready" ] && [ "$waited" -lt 200 ]; do + sleep 0.05 + waited=$((waited + 1)) + done + [ -f "$case_dir/registry-lock-ready" ] || { + : > "$case_dir/registry-lock-release" + wait "$holder_pid" || true + fail "registry lock holder did not start" + } + set +e + FM_ACCOUNT_LIFECYCLE_LOCK_WAIT_SECONDS=0 \ + run_teardown_named "$case_dir" "$id" --force > "$case_dir/locked-stdout" 2> "$case_dir/locked-stderr" + rc=$? + set -e + expect_code 1 "$rc" "registry-locked teardown exit" + assert_present "$case_dir/wt" "registry lock contention allowed home removal" + assert_grep 'fooxbar' "$case_dir/data/secondmates.md" \ + "registry lock contention overwrote a neighboring registration" + : > "$case_dir/registry-lock-release" + wait "$holder_pid" || fail "registry lock holder failed to release" + run_teardown_named "$case_dir" "$id" --force > "$case_dir/stdout" 2> "$case_dir/stderr" \ + || fail "dotted-id teardown failed after registry lock release: $(cat "$case_dir/stderr")" + assert_grep '- fooxbar ' "$case_dir/data/secondmates.md" \ + "retiring foo.bar removed the literal neighbor fooxbar" + assert_no_grep '- foo.bar ' "$case_dir/data/secondmates.md" \ + "retiring foo.bar left its exact registry entry" + pass "secondmate registry updates are serialized and compare ids literally" } if [ "${FM_TEST_FOCUSED:-}" = tasktmp-safety ]; then @@ -2131,12 +4293,12 @@ if [ "${FM_TEST_FOCUSED:-}" = tasktmp-safety ]; then fi if [ "${FM_TEST_FOCUSED:-}" = managed-force-release ]; then - test_managed_force_teardown_releases_lease_and_session + test_managed_force_teardown_retains_unlanded_lease_and_session exit 0 fi if [ "${FM_TEST_FOCUSED:-}" = managed-endpoint-identity ]; then - test_managed_force_teardown_releases_lease_and_session + test_managed_force_teardown_retains_unlanded_lease_and_session test_managed_teardown_retains_lease_when_endpoint_state_is_unknown exit 0 fi @@ -2152,12 +4314,136 @@ if [ "${FM_TEST_FOCUSED:-}" = review-round-35-pr ]; then exit 0 fi +if [ "${FM_TEST_FOCUSED:-}" = direct-spawn-cleanup ]; then + test_retained_direct_spawn_requires_confirmed_endpoint_quiescence + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-teardown-state ]; then + test_secondmate_state_enumeration_fails_closed + test_secondmate_missing_treehouse_child_is_retained + test_secondmate_registry_home_drift_blocks_removal + exit 0 +fi + if [ "${FM_TEST_FOCUSED:-}" = legacy-pr-generation ]; then test_pr_check_backfills_legacy_generation_and_records_state test_pr_check_backfills_legacy_generation_before_race_check exit 0 fi +if [ "${FM_TEST_FOCUSED:-}" = review-round-6 ]; then + test_content_in_default_fallback_allows + test_content_fallback_refreshes_stale_origin_ref + test_content_fallback_uses_live_default + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-7 ]; then + test_content_fallback_reprobes_live_default_after_fetch + test_content_fallback_honors_shared_checkout_lock + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-8 ]; then + test_content_in_default_fallback_allows + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-9 ]; then + test_forced_secondmate_retains_child_on_checkout_lock_contention + test_locked_return_reuses_checkout_lock_for_landing_recheck + test_treehouse_return_timeout_reaps_children_before_unlock + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-10-treehouse-return ]; then + test_forced_secondmate_retains_child_on_treehouse_failure + test_forced_secondmate_retains_child_when_treehouse_unavailable + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-11-process-group ]; then + test_forced_secondmate_retains_child_on_treehouse_failure + test_treehouse_return_timeout_reaps_children_before_unlock + test_forced_secondmate_retains_unverified_process_group + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-12-ownership ]; then + test_forced_secondmate_retains_child_on_treehouse_failure + test_treehouse_return_timeout_reaps_children_before_unlock + test_forced_secondmate_retains_unverified_process_group + test_bounded_runner_preserves_command_status_125 + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-refresh-safety ]; then + test_legacy_teardown_revalidates_after_quiescence + test_teardown_rejects_nested_metadata_roots_before_quiescence + test_teardown_retains_untracked_claude_skill_draft + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-teardown-ownership ]; then + test_teardown_rejects_drifted_treehouse_task_lease + test_teardown_rechecks_treehouse_lease_after_locked_safety + test_secondmate_rejects_drifted_home_repository_identity + test_normal_secondmate_retires_proven_detached_head + test_forced_secondmate_retains_untracked_skill_draft + test_forced_secondmate_retains_unique_detached_head + test_forced_secondmate_retains_stash + test_forced_secondmate_retains_unlanded_child_work + test_forced_secondmate_retains_unquiesced_unmanaged_child + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-teardown-lifecycle ]; then + test_secondmate_registry_duplicate_home_blocks_removal + test_secondmate_retirement_serializes_child_spawn + test_nested_secondmate_cleanup_requires_child_home_lock + test_secondmate_registry_updates_are_locked_and_literal + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-durable-secondmate ]; then + test_normal_secondmate_retires_proven_detached_head + test_secondmate_retirement_retains_idle_registered_child + test_secondmate_retirement_retains_unlanded_project_clone + test_secondmate_project_tags_do_not_prove_landing + test_secondmate_project_origin_authority_survives_home_removal + test_secondmate_retirement_recurses_into_ignored_nested_repositories + test_secondmate_retirement_rejects_linked_worktree_graphs + test_secondmate_retirement_accounts_for_directory_symlinks + test_secondmate_retirement_rejects_loopback_and_stale_tracking_authority + test_secondmate_retirement_rejects_mount_boundaries + test_secondmate_retirement_rejects_effective_ssh_redirects + test_secondmate_retirement_rejects_incomplete_surviving_authority + test_secondmate_retirement_validates_top_level_source_storage + test_secondmate_retirement_rejects_local_network_aliases + test_secondmate_retirement_rejects_in_home_remote_object_storage + test_secondmate_retirement_rejects_source_common_dir_in_home + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-13-safety ]; then + test_teardown_removal_roots_fail_closed + test_treehouse_return_stays_bound_to_validated_root + test_teardown_distinguishes_dead_and_live_harness_processes + test_secondmate_retirement_retains_reflog_and_rewritten_history + test_secondmate_retirement_rejects_http_proxy_and_object_redirects + test_secondmate_retirement_rejects_incomplete_surviving_authority + test_secondmate_network_fetches_pin_validated_addresses + test_surviving_object_storage_is_bound_through_graph_proof + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = review-round-13-network ]; then + test_secondmate_network_fetches_pin_validated_addresses + test_surviving_object_storage_is_bound_through_graph_proof + exit 0 +fi + test_local_only_fork_remote_allows test_teardown_prompts_tasks_axi_done_when_compatible test_teardown_manual_backend_prompts_hand_edit_even_when_tasks_axi_present @@ -2165,20 +4451,66 @@ test_local_only_truly_unpushed_refuses test_local_only_merged_to_local_main_allows test_no_mistakes_origin_remote_allows test_no_mistakes_truly_unpushed_refuses -test_local_only_force_overrides_unpushed -test_managed_force_teardown_releases_lease_and_session +test_local_only_force_retains_unpushed +test_managed_force_teardown_retains_unlanded_lease_and_session test_managed_teardown_retains_lease_when_endpoint_state_is_unknown test_managed_release_failure_preserves_unrecycled_worktree_for_retry test_managed_teardown_locks_generation_before_endpoint_cleanup test_managed_child_teardown_locks_generation_before_snapshot test_forced_secondmate_child_uses_child_home_for_endpoint_verification test_forced_secondmate_quiesces_parent_before_child_cleanup +test_forced_secondmate_retains_child_on_treehouse_failure +test_forced_secondmate_retains_unverified_process_group +test_forced_secondmate_retains_child_when_treehouse_unavailable +test_forced_secondmate_retains_child_on_checkout_lock_contention test_herdr_teardown_clears_escalation_marker test_required_report_blocks_then_publishes_before_cleanup test_required_report_restores_rollback_generation_before_publish test_required_report_revalidates_after_quiescence +test_legacy_teardown_revalidates_after_quiescence +test_teardown_rejects_nested_metadata_roots_before_quiescence +test_teardown_rejects_drifted_treehouse_task_lease +test_teardown_rechecks_treehouse_lease_after_locked_safety +test_secondmate_rejects_drifted_home_repository_identity +test_normal_secondmate_retires_proven_detached_head +test_forced_secondmate_retains_untracked_skill_draft +test_forced_secondmate_retains_unique_detached_head +test_forced_secondmate_retains_stash +test_forced_secondmate_retains_unlanded_child_work +test_forced_secondmate_retains_unquiesced_unmanaged_child +test_secondmate_registry_duplicate_home_blocks_removal +test_secondmate_retirement_retains_idle_registered_child +test_secondmate_retirement_retains_unlanded_project_clone +test_secondmate_project_tags_do_not_prove_landing +test_secondmate_project_origin_authority_survives_home_removal +test_secondmate_retirement_recurses_into_ignored_nested_repositories +test_secondmate_retirement_rejects_linked_worktree_graphs +test_secondmate_retirement_accounts_for_directory_symlinks +test_secondmate_retirement_rejects_loopback_and_stale_tracking_authority +test_secondmate_retirement_rejects_mount_boundaries +test_secondmate_retirement_rejects_effective_ssh_redirects +test_secondmate_retirement_rejects_incomplete_surviving_authority +test_secondmate_retirement_validates_top_level_source_storage +test_secondmate_retirement_rejects_local_network_aliases +test_secondmate_retirement_rejects_in_home_remote_object_storage +test_secondmate_retirement_rejects_source_common_dir_in_home +test_teardown_removal_roots_fail_closed +test_treehouse_return_stays_bound_to_validated_root +test_teardown_distinguishes_dead_and_live_harness_processes +test_secondmate_retirement_retains_reflog_and_rewritten_history +test_secondmate_retirement_rejects_http_proxy_and_object_redirects +test_secondmate_network_fetches_pin_validated_addresses +test_surviving_object_storage_is_bound_through_graph_proof +test_secondmate_retirement_serializes_child_spawn +test_nested_secondmate_cleanup_requires_child_home_lock +test_secondmate_registry_updates_are_locked_and_literal +test_teardown_retains_untracked_claude_skill_draft test_teardown_refuses_unsafe_tasktmp_metadata test_teardown_rejects_malformed_report_requirement +test_secondmate_state_enumeration_fails_closed +test_secondmate_missing_treehouse_child_is_retained +test_secondmate_registry_home_drift_blocks_removal +test_retained_direct_spawn_requires_confirmed_endpoint_quiescence test_squash_merged_branch_deleted_allows test_squash_merged_pr_allows_when_head_ancestor_of_pr_head test_no_pr_recorded_discovers_merged_pr_by_branch_allows @@ -2190,6 +4522,11 @@ test_pr_check_serializes_with_account_session_updates test_pr_check_rejects_reused_task_generation test_content_in_default_fallback_allows test_content_fallback_refreshes_stale_origin_ref +test_content_fallback_uses_live_default +test_content_fallback_reprobes_live_default_after_fetch +test_content_fallback_honors_shared_checkout_lock +test_locked_return_reuses_checkout_lock_for_landing_recheck +test_treehouse_return_timeout_reaps_children_before_unlock test_dirty_worktree_refuses test_gh_error_and_content_absent_refuses test_stale_index_lock_cleared_and_teardown_succeeds diff --git a/tests/lib.sh b/tests/lib.sh index a7c0504e85b..8597685ee7f 100644 --- a/tests/lib.sh +++ b/tests/lib.sh @@ -69,6 +69,7 @@ fm_test_cleanup() { fm_test_tmproot() { local prefix=${1:-fm-test} root root=$(mktemp -d "${TMPDIR:-/tmp}/${prefix}.XXXXXX") + root=$(cd "$root" && pwd -P) if [ "${#FM_TEST_CLEANUP_DIRS[@]}" -eq 0 ]; then trap fm_test_cleanup EXIT fi diff --git a/tests/secondmate-helpers.sh b/tests/secondmate-helpers.sh index 6dd4d30288c..3dd0981cb91 100644 --- a/tests/secondmate-helpers.sh +++ b/tests/secondmate-helpers.sh @@ -70,9 +70,9 @@ SH cat > "$fakebin/treehouse" <<'SH' #!/usr/bin/env bash set -u -printf 'treehouse %s\n' "$*" >> "${FM_FAKE_TMUX_LOG:-/dev/null}" case "${1:-}" in get) + printf 'treehouse %s\n' "$*" >> "${FM_FAKE_TMUX_LOG:-/dev/null}" # Durable lease: print only the worktree path to stdout (banners to stderr), # and record the lease holder so tests can assert it is set and later cleared. shift @@ -94,6 +94,12 @@ case "${1:-}" in exit 0 ;; return) + if [ -n "${FM_EXPECT_CHECKOUT_LOCK:-}" ]; then + [ -e "$FM_EXPECT_CHECKOUT_LOCK" ] || [ -L "$FM_EXPECT_CHECKOUT_LOCK" ] || exit 91 + lock_pid=$(cat "$FM_EXPECT_CHECKOUT_LOCK/pid" 2>/dev/null || true) + kill -0 "$lock_pid" 2>/dev/null || exit 92 + [ -z "${FM_EXPECT_CHECKOUT_LOCK_MARKER:-}" ] || touch "$FM_EXPECT_CHECKOUT_LOCK_MARKER" + fi shift target= while [ $# -gt 0 ]; do @@ -103,6 +109,10 @@ case "${1:-}" in esac shift done + case "$target" in + .|/dev/fd/*) target=$(cd "$target" && pwd -P) || exit 18 ;; + esac + printf 'treehouse return --force %s\n' "$target" >> "${FM_FAKE_TMUX_LOG:-/dev/null}" [ -z "${FM_FAKE_TREEHOUSE_RETURN_FAIL:-}" ] || exit 17 [ -n "${FM_FAKE_TREEHOUSE_LEASE_FILE:-}" ] && rm -f "$FM_FAKE_TREEHOUSE_LEASE_FILE" [ -n "$target" ] && rm -rf -- "$target"