feat(worker): lease one Pi account per placement so concurrent crewmates never collide

[ruby-dlee]

R5's "Work" line: multi-profile account selection for crewmate and worker placement, reusing the projection tool and the account-lease identity already in the worker request path.

Two defects this closes, both security-relevant

Neither is a refactor. Both were live on the cloud placement lane.

1. Every crewmate carried the entire account pool. bin/fm-spawn.sh staged the WHOLE POOLED auth.json from ${PI_CODING_AGENT_DIR:-~/.pi/agent} onto each worker's account disk. That is a credential-blast-radius problem quite apart from collisions: one compromised, misbehaving, or merely buggy crewmate held live OAuth credentials for all eight upstream accounts, not the one it was placed on. It also guaranteed the collision, because every worker then resolved the pool's first slot no matter what the controller thought it had assigned.

Review round 2 found the first version of this fix incomplete: narrowing AFTER the request left a window in which the pool was staged, the lease was already durable, and the tracking monitor was already polling and willing to dispatch that directory. The pool is now not staged at all (below).

2. The duplicate-account guard was decorative and could never fire. account_binding was digest({task, account_home}) - unique per task BY CONSTRUCTION - so ensure_unique_bindings's account clause was structurally incapable of raising on this lane, for any input. docs/azure-workers.md has stated "No two active tasks share a VM, account lease, browser profile, or writable task disk" the entire time. The VM, browser-profile and task-disk halves were enforced. The account lease half was not enforced at all, and no test could have caught it, because the guard's own precondition was unreachable.

The four design questions

What is the unit of exclusion, and why? The upstream account, identified by bin/fm-pi-account-home.py's account_digest (a truncated SHA-256 of the account id, never token material). Not the profile name and not the account-home path: both are handles, and a re-login can point two slots at one account while a projection can put one account behind two paths. What a concurrent crewmate actually contends for - the rate limit, the ban, the session - belongs to the account. Two profiles resolving to one account are therefore ONE lease, proved by a unit that rewrites both fixture profiles to the same accountId and shows the second placement refuse.

How is the lease acquired and released, and what happens on a crash between them? There is no separate lease record. Selection happens inside command_request, in the same controller_lock hold and the same save_state that writes the queue entry, so the queue entry is the lease. The held set is derived from the non-complete queue, so there is no window in which an account is held by something the queue does not show, and no second document a crash could leave holding a profile nothing owns. Release is the existing machinery, unchanged: release proof to complete, withdraw, or surrender. A crash before the commit leaves nothing; after it, an ordinary queued entry that status lists and withdraw clears. The projection written just before the commit can survive with no entry - that directory is inert, not a lease, and the next placement may take it.

One new handback path (second commit): a spawn that gets past the request but cannot bind the account it was handed now withdraws its own queued entry rather than leaving the lease held by work that will never run, and names the manual withdraw if even that fails. Without it the pool lost one account every time that happened.

What happens when all profiles are leased? It refuses, by name, listing every leased profile and the task holding it, and creates no queue entry. There is no fallback to a shared or arbitrary profile. Same for an unreadable pool, an empty pool, a pool with no projectable profile, and a single-profile home naming no upstream account.

Deterministic or arbitrary, and does replay reuse? Deterministic: the first free profile in the pool's sorted (lexicographic) name order. Replaying the same task generation reuses the same profile, because the replay path short-circuits on the existing entry before selecting anything. A different generation is a different placement and takes the next free account.

Round-2 review: two blockers, both fixed

Both were reproduced against the previous head with no mutation, so neither was theoretical.

Blocker 1: the projection key and the lease key were two different functions of the pool. The lease keyed on the upstream account (correct), but the projected home keyed on the profile's local SLOT NAME. Those disagree exactly where my own design rationale said they could: an operator re-logging slot openai-codex from one account to another produces two placements with two correct, distinct bindings that both project into accounts/openai-codex, so the second write_home replaces the credential the first placement's still-live lease points at. controller.json then reports two accounts while the disk holds one. I reasoned about re-login for the exclusion unit and not for the projection key.

Fixed at the single point that caused it: the home is now accounts/<account-binding>, so the directory name and the exclusion key are literally the same string and the divergence is unrepresentable. A second, defensive refusal declines to project over an account home a live queue entry still names; it is dead while the keys agree and load-bearing if anyone re-keys later. A consequence the review did not name: spawn_cloud_bind_leased_account derived the profile name from the path's last component, which is now the binding, so the controller reports account-profile explicitly and the spawn reads that.

On the review's aggravating factor (replay re-stages from account_home on every call, so a resumed spawn stages whatever is at that path NOW): I believe the same keying change resolves it rather than it needing a separate fix, and I am stating that as REASONING TO BE VERIFIED rather than as a proven claim, because no test in this PR exercises a replay across a pool mutation. The argument is that write_home only ever writes accounts/<binding> with the credential whose upstream account hashes to that binding, so by construction what sits at that path is what the lease names; a pool change moves the profile to a NEW directory and leaves the old one holding the account the old lease still points at. If a reviewer disagrees, the case to attack is a replay after the leased profile's account changed.

Blocker 2: the pooled auth.json still reached the request-to-narrow window. Staging ran before the lease existed, while the monitor pane was already polling, and bin/fm-spawn-cloud-monitor.sh guarded --account-dir only with [ -d ... ]. Killed in that window, all four fixture accounts were staged with the lease already durable.

Fixed by removing the window rather than guarding it, which is the stronger of the two directions offered: the payload step no longer copies the pooled auth.json at all, and the account directory is written exactly once, by the narrowing step, after the lease exists. settings.json still stages there because it is configuration, not credential material. The monitor guard was added as well, not instead, because the monitor outlives the spawn and dispatches independently, so the shape belongs at the point of use. One correction to the suggested placement: the guard must sit BEFORE the O_EXCL dispatch claim. Behind it, a refusal leaves the marker taken and both owners stand down forever. Ahead of it, a not-yet-narrowed directory simply does not claim and the next poll retries.

Both smaller items are done: the exhaustion refusals now name the remedy, and the monitor fixture's account ids are scoped per task so a future third concurrent task cannot hit phantom exhaustion.

Reuse, not reinvention

• bin/fm-pi-account-home.py is imported as a module and supplies all three things placement needs: read_pool, account_digest, and prepare_root/write_home. Nothing here re-derives an account identity or re-implements how an account home is written.
• The lease identity stays account_binding, the field already in the request path, and the exclusion stays ensure_unique_bindings, the screen already there. Only the identity's meaning changed, from "this task's lease of this home" to "this upstream account" - which is what makes the existing screen able to fire at all.
• authoritative_request_bindings now returns account_pool_home instead of a per-task account_binding: the task's metadata still proves which pool it may draw from, and the controller decides which profile of that pool, because that decision must exclude every other concurrent placement and no task-local document can see them.

Compartment children

Covered by construction, not by a special case. A compartment child goes through the same command_request, and FM_HOME still names the primary's controller for both it and an ordinary crewmate (#278/#279 moved the task home, never the money document), so both draw from the one document under the one lock. A unit places a compartment, its child, and an ordinary crewmate - the last two concurrently - and asserts three distinct accounts.

Where the projected homes live

Under $FM_HOME/state/azure-workers/accounts/<profile> (overridable with FM_PI_ACCOUNT_HOME_ROOT), not the shared ~/.local/share/agent-fleet/accounts/pi roster. Those homes belong to the crosscheck reviewer lane; a placement rewriting one mid-review would swap a running reviewer's credential underneath it. It also makes the root follow FM_HOME, so a fixture home cannot write into a real one.

Proof

Every leg ran with an explicit fixture provider, a fixture FM_HOME, and the operator's ambient FM_*/AZURE*/ARM_* variables scrubbed. Each placement unit additionally shadows az with a recording stub that fails, and asserts it was never invoked. az vm list across the subscription is empty: nothing was created.

python3 tests/run-one.py tests/fm-worker-placement.test.sh - 8 units, all green:

• The acceptance sentence, as far as it can be taken locally. Eight placements started together and left to race the controller lock take eight distinct profiles and eight distinct upstream accounts, read back from controller.json, with each leased home verified to hold exactly one slot whose accountId matches its binding. The assignment order is scrambled relative to task order on every run, which is the race actually happening.
• Exhaustion refuses, verbatim: ELASTIC WORKER REFUSED: provider-account placement is exhausted: all 3 distinct upstream accounts in <pool>/auth.json are leased (openai-codex -> task-1, openai-codex-2 -> task-2, openai-codex-3 -> task-3); refusing to place task-4 on a shared upstream account, with the queue unmutated.
• Crash safety, with a stated floor. The unit times an uninterrupted placement, then spreads 24 SIGKILLs across that calibrated window. The shipped assertion is 0 < survivors < attempts: the floor is one - at least one kill must have landed inside the window and at least one placement must have survived it, so the unit cannot silently pass with zero kills landing. That is deliberately the weakest non-vacuous floor rather than a rate, so the pass condition does not depend on machine load; in practice roughly half land (10 of 24 on a recent local run). No account is held twice, every projected home is named by a live entry, and withdrawing the survivors returns the whole pool, proved by a fresh placement taking the first account again.
• Released-account return, replay determinism, two-profiles-one-account, and status reporting placements in both renderings.

python3 tests/run-one.py tests/fm-spawn-cloud.test.sh - green. The assigned-lane unit asserts the end-to-end claim through the real bin/fm-spawn.sh: the staged state/<id>.cloud-account/auth.json holds exactly one slot, byte-equal to the leased profile's projected home; the meta records worker_account_profile/worker_account_home; those match the queue entry; and the pool it came from held more than one account, so a selection genuinely happened. Units also cover the lease handback and the request-to-narrow window, the latter killing the real spawn inside the window and asserting the lease is durable while nothing pooled is staged:

# window state: lease durable, staged slots = no auth.json at all

tests/fm-worker-lifecycle.test.sh (36 ok, the documented hermetic baseline), fm-secondmate-cloud-monitor, fm-azure-pilot, fm-worker-authority-secondmate, fm-worker-outcome-transport, fm-worker-supervisor: all green. bin/fm-lint.sh clean over all 247 scripts.

ADMIT-red mutation

Mutated placement_account_binding to fold the task back into the lease identity - the exact pre-change shape, one line, at the single point where the lease identity is defined. It disables selection and the screen together, because both read that one identity. The concurrent unit went RED on an actual collision read back from durable state, not on a message:

#   openai-codex task-1
#   openai-codex task-2
...
#   openai-codex task-8
AssertionError: ['openai-codex', 'openai-codex', 'openai-codex', 'openai-codex',
                 'openai-codex', 'openai-codex', 'openai-codex', 'openai-codex']
not ok - concurrent placements did not take distinct upstream accounts

Reverted; the suite is green again.

Two test-fixture edits, and why they are not papering over failures

A reviewer seeing fixture edits inside a placement PR should look hard at them. Both are in tests/fm-secondmate-cloud-monitor.test.sh:

1. make_lifecycle_task created an EMPTY account directory. Placement refuses a directory it cannot identify an upstream account in, so that is no longer a usable account authority. The fixture now seeds a real pool.
2. The end-to-end lane's pi-agent-home was a single thin profile, and that lane places a compartment and a child of that compartment. With one account in the pool the child now correctly refuses as exhausted. That is the new invariant working, not a regression - and it means the old one-profile fixture could only ever have passed while the invariant was unenforced. It was pinning the bug. The pool now holds three accounts.

Both fixtures were already unrealistic against the real lane, whose account home is the multi-profile Pi coding-agent home.

Hermeticity fix included

test_cloud_spawn_fails_closed_when_the_lifecycle_refuses_the_request relied on the ambient environment NOT carrying Azure identity, and its fixture id is cloud-noenv-c7. An operator shell exports the real subscription, tenant, resource group and image; inherited there, that unit's request is admitted and its reconcile reaches the real Azure adapter - a test that believes it is exercising a fake, creating a billable VM tagged with its own hardcoded fixture id. It now unsets the identity variables explicitly and pins the provider to its case fixture.

Operational consequence: the pool is now a concurrency ceiling

Worth a reviewer's attention because it changes fleet behavior, not just correctness. Concurrent placements are bounded by min(FM_AZURE_WORKER_MAX, distinct upstream accounts in the pool). With the fleet's eight Pi accounts, the ninth concurrent placement refuses even though MAX_WORKERS is 16 and quota, budget and capacity would all admit it. Compartments compete in the same pool: one compartment plus its four children takes five of the eight before an ordinary crewmate is placed.

This is the requirement rather than a regression - sixteen crewmates never could run on eight accounts without sharing one, they just used to do it silently - but it halves effective author parallelism, and raising the ceiling means adding signed-in profiles on distinct upstream accounts, not turning a knob. Recorded in docs/azure-workers.md and in R5's status so an operator does not have to discover it from a refusal.

What is NOT proven: the acceptance is NOT met

R5's acceptance sentence is "concurrent crewmates run on distinct pi profiles with no account collision". This PR proves placement up to, and including, the credential the worker is handed. It does not prove the run leg: no part of this touched real Azure compute, so what the guest agent does with that credential is unobserved here. Treat the acceptance as open. That leg needs a live worker under the owner's supervision, and R5's status text in docs/azure-requirements.md says so rather than claiming DONE.

Deploy note: in-flight cloud placements need a manual withdraw across this merge

Found in re-review, verified end to end, and NOT otherwise recorded here.

A queue entry minted by base main carries the old per-task account_binding and no
account_home. Replaying that generation on this head is REFUSED:

ELASTIC WORKER REFUSED: task generation already exists with different queue identity

because authoritative_request_bindings now returns account_pool_home rather than a per-task
account_binding. The refusal is loud and fail-closed, the legacy binding blocks no real account
(a fresh placement alongside it succeeds), and withdraw clears it. But any cloud placement that is
in flight at deploy time needs a manual bin/fm-worker-lifecycle.sh withdraw before it can be
replayed.

Three non-blocking follow-ups from re-review, recorded so they are not lost

1. bin/fm-secondmate-cloud-monitor.sh did not receive the point-of-use guard. For KIND=secondmate
   the tracking pane is that script, launched before spawn_cloud_dispatch, so it shares the same
   request-to-narrow window. It can no longer leak a pooled credential (nothing pooled is staged),
   so the residual is availability only: leg 1 can dispatch against a directory holding just
   settings.json, burning the exactly-once leg marker on a credential-less dispatch. The same
   rationale used for the crewmate monitor applies here.
2. The defensive refusal in select_placement_account aborts rather than skipping to the next free
   binding, so one corrupt entry blocks the lane even with other accounts free. Dead code while the
   keys agree, and its message names no remedy.
3. After a re-login, the controller reports the account_profile slot name recorded at lease time,
   which may now name a different upstream account. The CREDENTIAL is correct; only the
   worker_account_profile label is stale. Observability nit.

Correction to a claim made earlier in this PR's discussion: placing the monitor guard behind the
O_EXCL claim is NOT a permanent wedge. reclaim_stale_dispatch releases a held marker with an empty
result after wall + 300 (default 3900s), so it is a roughly 65 minute stall that self-heals. The
shipped ordering (guard before the claim) remains the correct one.