Skip to content

feat(worker): run provider mutations outside the fleet lock under per-slot leases (C2 3 of 3) - #261

Merged
ruby-dlee merged 3 commits into
mainfrom
feat/c2-lock-discipline
Aug 19, 2026
Merged

feat(worker): run provider mutations outside the fleet lock under per-slot leases (C2 3 of 3)#261
ruby-dlee merged 3 commits into
mainfrom
feat/c2-lock-discipline

Conversation

@ruby-dlee

Copy link
Copy Markdown
Owner

What

C2's final change, per C2-DESIGN.md section 3, shipped as ONE PR as the design requires: a half-serialized controller in which some commands hold the fleet lock across provider calls and others do not is worse than either end state.

  • slot_lease: non-blocking per-slot flock; the ONLY waited-on lock in the system stays the fleet lock, making deadlock impossible. The kernel drops a dead owner's lease, which is what makes its claim drainable at once.
  • provider_mutate is the only mutate path (provider_call refuses 'mutate'); it requires a live lease on the exact slot.
  • claim_pending (durable claim before the provider; refuses a different key on a claimed slot, closing the blind-overwrite/double-guest-run defect), apply_pending (fresh load, claim re-check, transactional apply, pop, save), drain_pending (post-convergence, live-owner skip, stale-snapshot re-read, strict/lenient modes).
  • reconcile: inventory unlocked, plan+claim in one short hold, mutate unlocked, apply in a fresh hold, refusals recorded on a CLEAN load; drain runs AFTER convergence so a wedged replay cannot stop the fleet. Planner and classification refresh skip claimed slots. Worst lock holds drop from ~8h25m (execute) to milliseconds.
  • abandon-claim: the sanctioned, evidence-preserving exit from a deterministically refused claim (the version-skew execute is the recorded case; the old exit was the blind overwrite). It replays the mutation itself under the lease, requires the result to bind the exact key - proving the mutation complete and its result final under key-idempotency - records the refusal verbatim with the result digest, then clears the claim.
  • Capacity commands deliberately keep their single holds (merged_specialized_reservations ignores non-reserved locals; splitting them double-admits real money). execute_action and replay_pending are deleted; resume drains only its own slot, strictly.

Verification

18 units green. New proofs:

  • concurrent_mutations_do_not_serialize: two reconciles park two creates INSIDE the provider on a real FIFO rendezvous (arrival = O_EXCL file per idempotency key); asserts two map claims with distinct assignment generations, both leases held in status --json, readers and unrelated withdraw proceeding while parked, exactly one provider call per key after release, and a ONE-child positive control that must fail the two-arrival wait. Structurally unsatisfiable under the old fleet lock.
  • wedged_slot_does_not_stop_the_fleet: a guaranteed-refusing claim is reported replay-refused and retained while a queued task still gets its worker; plus the planner never proposes a mutation for a claimed slot even when its released worker is otherwise due destruction.
  • The e2e models the real recovery for a version-skewed execute: the retry refuses ('still has an unapplied execute action'), a wrong key refuses abandon, abandon-claim clears it with the refusal recorded, then the retry lands.
  • The call-site unit drives the real claim/mutate/apply/drain chain at the raw provider boundary: mutate ban, durable-claim-before-provider from the FILE, different-key refusal, wrong-lease refusal, failed apply leaving the point-3 image intact on disk, lenient drain retaining refused claims, honest drain clearing them, and stale apply refused after the claim is gone.

Mutations against the committed tree, each red: claim different-key refusal dropped; apply_pending claim re-check dropped (survived the first sweep, pinned by the stale-apply probe); drain dropping refused claims; planner claimed-slot skip dropped (survived the first sweep, pinned by the released-worker plan probe); mutate ban dropped; lease slot binding dropped; abandon-claim key match dropped.

bin/fm-lint.sh clean; fm-behavior-shards.sh --check 8 clean (file duration re-budgeted 8000->12000ms).

…-slot leases (C2 3 of 3)

The fleet lock now covers only short read-validate-claim and apply sections.
Every provider mutation goes through provider_mutate under a non-blocking
per-slot flock lease (provider_call refuses 'mutate' outright), claimed
durably by claim_pending before the provider is touched and applied by
apply_pending against a FRESH load, never the caller's pre-call object.
Mutations for different slots run concurrently; an execute no longer blocks
status, dry reconcile, requests, or withdrawals for its whole guest run.

Reconcile fetches inventory unlocked, plans and claims in one short hold,
mutates unlocked, applies in a fresh hold, and records refusals on a CLEAN
load. The drain of stranded claims runs AFTER convergence with strict=False,
skipping any slot whose lease a live process holds, so a wedged or
hours-long replay cannot stop the fleet converging; the planner and the
classification refresh skip claimed slots, whose durable records are
deliberately not yet the truth. resume drains only its own slot, strictly.

claim_pending refuses a different key on a claimed slot, closing the blind
overwrite that silently discarded the first claim's replay obligation and
ran the guest twice. The sanctioned exit that overwrite used to provide is
now explicit: abandon-claim takes the lease, replays the mutation itself,
requires the provider result to bind the exact idempotency key (the
mutation is provably complete and its result final under key-idempotency),
records the refusal verbatim with the result digest in cleanup_refusals,
and only then clears the claim. The capacity commands keep their single
holds deliberately: merged_specialized_reservations ignores non-reserved
locals, so a split would let two concurrent reserves each admit against a
budget that fits one.

The two-writer proof parks two creates INSIDE the provider on a real FIFO
rendezvous and asserts a two-entry pending_actions map with two distinct
assignment generations - a shape structurally unsatisfiable under the old
fleet-lock discipline, with a one-child positive control that must fail the
two-arrival wait. The fixture provider gains an flock around its
read-modify-write so a lost update cannot masquerade as a controller bug.
…e refusal, close abandon's race

The review proved the LOCK_NB discipline had no behavioral pin (a blocking
lease survived the suite; the static marker was satisfied by a docstring):
the concurrent unit now probes slot 1's lease from a bounded subprocess
while the parked child owns it, requiring an immediate SlotBusy. abandon-
claim gains the drain's under-lease re-read before re-sending, so a claim a
concurrent drain applied is never mutated again without a durable claim
naming it. The strict drain mode is pinned against the wedged claim (resume
depends on it). The abandon docstring and help now say the proof is
obtained BY submitting and what a refused create leaves behind; dead loop
and stale execute_action comments removed.
…w path the ban exposes

fm-azure-pilot's bound probe called provider_call with 'mutate' directly,
which the lock discipline now refuses; the timeout bound under test lives on
_provider_call_raw, which every mutation reaches through provider_mutate.
The probe now asserts the ban fires AND the raw path carries the bound.
@ruby-dlee
ruby-dlee merged commit 2d5ba8a into main Aug 19, 2026
13 checks passed
@ruby-dlee
ruby-dlee deleted the feat/c2-lock-discipline branch August 19, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant