fix(worker): resolve a staged cloud credential in the home that staged it

[ruby-dlee]

The defect, reproduced first

Since the compartment-child task-home split (#278/#279), a secondmate compartment's child stages its plaintext provider credential at <task home>/state/<id>.cloud-account/auth.json, and its task home is the COMPARTMENT's home. withdraw and surrender re-derived the PRIMARY's state directory, because the controller document has exactly one home and every lifecycle command for the child runs with FM_HOME on the primary. They removed a path that never existed, reported success, and the credential stayed in the compartment home with nothing left that would ever take it.

Reproduced before touching any production code, end to end through the real bin/fm-spawn.sh and the real bin/fm-worker-lifecycle.sh:

queued smc-withdraw generation gen-parent for one isolated author worker
not ok - withdraw left the child's staged provider credential in the compartment home

Correction to the brief this branch was written from: bin/fm-teardown.sh:130 is the STATE= definition, not a remover, and teardown resolves its own task's home correctly. Teardown's real gap was different: cleanup_firstmate_home_children, the loop that reaps a secondmate home's children when the parent goes first, had no cloud-state removal at all.

What changed

Removers do not infer a home. Each is handed the state directory of the home whose task it is:

• the spawn's rollback and its re-spawn sweep pass $STATE, the directory they just staged into;
• teardown passes its own task's home, and each secondmate home's own state directory when it reaps that home's children;
• withdraw and surrender are told by the CONTROLLER. The authorized task home is already durable on the queue item and the worker record, so both commands echo it back on an FM-TASK-HOME line beside their receipt. The value the remover uses is the value the spawn passed to --task-home and the controller authorized under its own lock.

The credential is NOT relocated to the primary's state. The task-home split stays exactly as #278 landed it.

The per-task cloud file set is now enumerated once, in bin/fm-cloud-state-lib.sh. The library, the re-spawn sweep and the spawn rollback used to spell it out separately, which is what allowed a file to be created by one lane and removed by none.

Why not a per-task record file

The first version of this PR recorded the staging home in <controller state>/<id>.cloud-task-home. Review showed that is destructive. Task ids are HOME-SCOPED, so the same id can be live in a compartment home and in the primary at once, and a stale record redirected an unrelated primary task's teardown into the compartment. Executed against that implementation, same fixture:

--- after the REJECTED library ---
PRESENT  primary/state/X.cloud-account/auth.json     <- the leak this PR exists to fix
DELETED  compartment/state/X.cloud-account/auth.json <- a LIVE child's credential
DELETED  compartment/state/X.cloud-payload/repo.bundle
DELETED  compartment/state/X.worker-result.json

and against what is on this branch now:

--- after the FIXED library, same fixture ---
DELETED  primary/state/X.cloud-account/auth.json
PRESENT  compartment/state/X.cloud-account/auth.json
PRESENT  compartment/state/X.cloud-payload/repo.bundle
PRESENT  compartment/state/X.worker-result.json

run_a_removal_never_reaches_a_same_id_task_in_another_home pins this. It deliberately seeds the rejected design's record file, which nothing writes and nothing reads any more, so the case stays red against any remover that resolves a home from the task id.

Lanes covered

Lane	Proof
withdraw	real spawn + real withdraw, tests/fm-spawn-cloud.test.sh
surrender	real surrender through the wrapper, tests/fm-worker-lifecycle.test.sh
teardown, task's own home	driven in tests/fm-cloud-state.test.sh
teardown, secondmate-home child reaping	structural only, not driven (see below)
post-staging refusal rollback	real controller refusal after staging, tests/fm-spawn-cloud.test.sh

The reaping-loop call is checked by parsing cleanup_firstmate_home_children and requiring the removal to appear before the metadata removal that ends the child. It is not execution-driven, and deliberately so: the loop is followed by the removal of the whole home, which would make any after-the-fact file assertion pass whether or not the call exists. Correctness there follows by composition with the library tests, not by execution.

Structural, not a comment

• test_compartment_child_staging_and_removal_resolve_the_same_home runs a real compartment-child spawn, asks the filesystem which home actually holds the credential, and fails if the controller's recorded task home resolves a different one.
• run_no_second_enumeration_of_the_cloud_file_set fails if any bin/*.sh other than the library removes a per-task cloud file by name.

Admit-red mutations

1. fm_worker_receipt_state_dir forced to answer the controller's own state directory (the pre-change behavior). With the audit fixed this is now refused loudly rather than reaching the old assertion text: ELASTIC WORKER REFUSED: surrendered task-1 but its staged provider credential remains, and the withdraw lane reds on not ok - withdraw left the child's staged provider credential in the compartment home.
2. The previous head's id-keyed library restored verbatim: not ok - the primary's own credential was left behind.
3. The receipt sourced from the worker record again: AssertionError: surrender left a compartment child's staged provider credential in its task home.

Every red is the credential surviving or a live home being destroyed, never a message differing. Each was driven after watching the unmutated head go green first.

Ordinary crewmate lane is unchanged

Base commit versus this branch, identical fixture scaffolding, run tonight:

IDENTICAL: artifact listing (47 entries)
IDENTICAL: task metadata keys (24 entries)
IDENTICAL: spawn stdout (normalized)

Scope, since the earlier count was quoted without it: 47 is every entry in the whole task-home snapshot (checkout-refresh-state, config, data, projects, state, treehouse-pools). Under state/ alone it is 30 entries, 31 counting state/ itself.

Deliberate behavior change in the rollback

The rollback now also removes .cloud-execute-dispatched and the three worker-* artifacts, because it shares one entry point with the re-spawn sweep. At the rollback point none of them can exist: the sweep at the start of the same spawn removed them and no execute has run. The outcome directory is untouched, and its neighbouring preservation of an unlanded bundle is unchanged and asserted.

Rebased onto #280 (d8154a61)

#280 moved the credential write: the account directory is now written once, by the narrowing step, AFTER the lease exists. Consequences re-checked on the rebased head, not assumed:

• The withdraw lane still stages a real credential before the withdraw, because a queued-but-unassigned request has already succeeded and been narrowed. Reproduction and lane proof are unchanged and still pass.
• The rollback lane now proves the transport removal rather than the credential removal, and its comment says so.
• feat(worker): lease one Pi account per placement so concurrent crewmates never collide #280 added a NEW credential-bearing exit that depends on this fix: when spawn_cloud_bind_leased_account fails after a successful request, the spawn calls withdraw itself, through spawn_cloud_lifecycle, which runs with FM_HOME on the primary. For a compartment child that withdraw is exactly the lane this PR repairs.
• The rebase conflict was itself the drift this PR is about. feat(worker): lease one Pi account per placement so concurrent crewmates never collide #280 added <id>.worker-request.out to the re-spawn sweep only, so nothing removed it at the END of a task's life. It is in the single enumeration now, so teardown, withdraw and surrender cover it, and a test asserts that.

Shard 7 red on the first rebased head, and why

d9b2220a failed Behavior tests (shard 7/8). It reproduced locally on the first run of tests/fm-worker-lifecycle.test.sh (the suite that shard carries), so it was deterministic, not a flake, and not a #280 interaction.

Root cause was mine. Closing F2 meant carrying parent_task on the FM-TASK-HOME line so the reader could apply the stager's own marker-content check. parent_task and task_home both land on the QUEUE ITEM at admission, but only task_home is copied onto the worker record, and the surrender path sourced the receipt from the worker. The parent was therefore always absent, the line was never printed, and surrender fell back to the controller's own state directory: the leak this branch exists to close, reintroduced on exactly one lane by the hardening that closed the reader.

The receipt now reads the queue item, and stays silent when a worker record's task_home disagrees with its own item's. The surrender fixture had used a convenient shape rather than the real admission shape; it now writes both fields where admission writes them, which is what made the defect reachable.

Re-driven admit-reds after the fix, all red on the credential surviving:

1. receipt reader forced to the controller's own state directory: withdraw and surrender both red;
2. the rejected id-keyed library restored: red;
3. the receipt sourced from the worker record again (the defect above): surrender red.

Second review: five findings

The safety net could not catch the failure it exists to catch (blocking). It re-used the directory the removal had just resolved, so when resolution fell back wrongly, which is the entire defect this branch closes, it inspected the controller's own home, found nothing because nothing was ever staged there, and the command exited 0 announcing success. That is why the leak was silent in production rather than loud. Subjects are now fixed independently of the resolution: the home the controller NAMED for this task, read raw and only stat'ed, plus the controller's own state directory. Driven, same F1 mutation both times:

resolver forced to fall back, NEW audit: ELASTIC WORKER REFUSED: surrendered task-1 but its staged provider credential remains
resolver forced to fall back, OLD audit: no refusal at all; the command exits 0

The task home no longer travels on a parsed line in a mixed stdout stream. It has its own channel, --task-home-out, written by the controller and read as two fixed lines. On a shared stream its safety rested on two unrelated invariants holding forever, that stderr is never folded in and that no id can contain a space or newline; either relaxing would let another line decide where a removal is aimed.

The lease pointer was a live instance of this PR's own class. .cloud-worktree was excluded from the task-end set on the stated grounds that teardown removed it with the task's metadata. grep -n 'cloud-worktree' bin/fm-teardown.sh returns nothing, so it was created by the spawn lane and removed by no task-end lane. It is in the one set now.

The enumeration invariant caught one spelling. It now reads bin/*.py and bin/*.mjs as well as bin/*.sh, and catches flagless rm, globs over name stems, and find -delete; all three were driven and now fail the guard. It cannot catch a name assembled from a variable or from string concatenation, and the test says so rather than implying otherwise.

The teardown reaping guard failed open, and my previous claim about it was false. It was a source check, and three escapes walked through it: the call wrapped in a never-true condition (shell does not force re-indentation, so the indentation comparison held), the owner shadowed in the bash keyword form, and sub_state re-pointed at a directory that does not exist, which leaves the call textually perfect and aimed at nothing. The stated reason for keeping a source check was also wrong: five return 1 paths run between the reap and remove_firstmate_home, so a home whose children are reaped can survive.

It is a run-time seam now. The loop's own text is extracted from the shipped script and EXECUTED against stubs, with the owner replaced by a recorder writing OUTSIDE the home, so what happens to the home afterwards is irrelevant. It asserts on a call that happened and on the arguments it happened with. All four escapes driven against it, all four now red. A redefinition at the script's top level is outside the extracted text, so a separate check refuses that in both spellings.

The audit hung forever on a FIFO. The reader gated the channel and the audit did not: two functions applying different trust to one input, which is the seam that produced the original defect. They share the gate now. Driven: pre-fix killed after hanging (exit 137), post-fix returns immediately.

The enumeration guard fired on innocent code. rsync --delete and the operator-advice strings this wrapper prints beside its own refusals were all flagged. Narrowed, with the three true positives re-driven to confirm they still fail it.

Three texts asserted things the code does not do. docs/azure-workers.md still described the deleted FM-TASK-HOME line and called a pure delegate a group selector; the audit's comment said the named home is "never followed" when [ -e ] follows symlinks. All three now say what the code does, including the one symlink shape that stays silent (a named home that is itself a symlink to an empty marked decoy). That shape was silent before this PR too, so it is not a regression, and the other two symlink shapes now fail loudly.

Accepted consequences, recorded rather than fixed

• withdraw and surrender now remove <id>.cloud-worktree, which bin/fm-spawn-cloud-monitor.sh:245 reads. Both commands end the task, and the monitor treats an unreadable pointer as "the leased worktree is gone" and keeps the bundle for manual landing, so the loss is a reported landing, never lost commits.
• The file channel introduces one exotic fail path: an unwritable TMPDIR makes the write raise after the receipt is printed and after save_state, leaving the task surrendered with no removal. The wrapper's own mktemp would have failed first.

Notes

• Test-run safety is a property of these units, not of the harness. There is no az shim in tests/ and tests/run.sh scrubs no ambient FM_*; what makes these lanes safe is FM_WORKER_PROVIDER_COMMAND being pinned in run_child_spawn, stand_up_compartment and run_child_lifecycle, so no lane can select the real provider. run_child_lifecycle now also prepends the fakebin to PATH like its siblings.
• No credential contents appear in this diff, its tests, or this body. The tests reuse the suite's own pre-existing synthetic fixture literal.