feat(worker): per-slot pending_actions map, load fence, and revision CAS (C2 2 of 3) - #260
Merged
Conversation
…CAS (C2 change 2 of 3)
The scalar pending_action could hold exactly one in-flight provider mutation,
which is what forces the whole fleet to serialize behind one lock. This lands
the durable groundwork while everything stays fully serialized:
- pending_actions: {slot: action} replaces the scalar. The stored claim is a
deep copy of the exact minted action (make_action aliases live queue and
worker objects, and apply mutates some of them in place; a claim that can
change after it was hashed turns verification into a random wedge), and
verify_state re-derives every stored claim's idempotency key at load, so a
hand-edited or truncated claim refuses before it can replay. Bounded by
MAX_WORKERS, never the operator-settable env limit, so lowering
FM_AZURE_WORKER_MAX cannot make an existing file unloadable; claims for
slots without worker records stay loadable, because turning a recoverable
wedge into a file that refuses even status helps nobody.
- The legacy scalar is migrated onto the map at load (the action always
carried its own slot) and permanently holds a sentinel string an OLD
binary's verify_state refuses loudly, so a rollback cannot read None, plan
fresh work, and blind-overwrite a live claim. Rollback is a one-way door:
cp controller.json controller.json.pre-c2 before deploying.
- FencedState + load fence: save_state refuses any document that was not
produced by load_state inside the lock hold that is committing it. Moving a
provider call between a load and its save fails on the first run.
- Revision CAS: save_state re-reads the on-disk revision and refuses when it
moved, naming both numbers. Last-writer-wins here would not corrupt the
file; it would silently forget another writer's cloud resource identities
and re-admit a VM that exists and is billing.
- controller_lock refuses re-entrant acquisition (it deadlocked before) and
stamps a per-process epoch the fence checks.
- status --json gains pending_mutations [{slot, type}]; the text mode prints
it when non-empty, so an operator can finally see a slot mid-mutation.
- replay_pending drains the map in slot order; execute_action claims its own
slot; the withdraw guard fans out over every entry naming the slot in its
refusal; surrender and resume gate on the map.
The lock discipline itself (provider calls outside the fleet lock, slot
leases, drain-after-convergence) is deliberately NOT here; it ships as the
next change in one piece.
…n the claim-pop ordering The review found the migration paving over a corrupt legacy scalar (a non-dict, non-sentinel value the old binary refused loudly), silently destroying whatever replay obligation the corrupted bytes used to be; it now refuses before the sentinel overwrite, with a red-path test. The one load-bearing surviving mutation - popping the slot's claim before the apply instead of after, which would let a later refusal-handler save erase the replay obligation and wedge the slot silently - is pinned by asserting the claim on the CALLER'S object after a failed apply. Two more tripwires get red-path tests: a legacy scalar disagreeing with its slot's map entry, and a self-hashing claim naming a slot outside 1..MAX_WORKERS. The C2 requirement text is updated to record which half landed and which blocker remains.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
C2's second change, per the judged design (C2-DESIGN.md): the durable state shape and tripwires that the lock-discipline change (3 of 3) requires, landed while the controller stays fully serialized so any defect surfaces under today's semantics.
Verification
16 units green including three new/rewritten proofs: legacy_scalar_migration (real minted scalar -> map + sentinel + revision, through the real wrapper), state_fence_and_revision_cas (re-entrancy refusal, stale-hold refusal, unfenced-dict refusal, tampered-claim load refusal, CAS refusal naming both revisions), and restart replays per-slot keys exactly once with TWO parked claims (a shape the scalar could never hold). The e2e withdraw guard now plants two REAL minted claims and asserts the refusal names the second slot ('slot 9'), because verify_state rejects the old fake-key fixture at load, exactly as the design predicted. execute_action's durable claim is pinned at the provider boundary by reading the FILE before and after a modeled provider crash.
Mutations, each run against the committed tree and restored from it, each turning the suite red: migration keeps the dict scalar; revision CAS dropped; load fence dropped; self-hash dropped from verify_state; pending_mutations projection dropped; execute_action claims nothing durable (this one survived the first sweep and got the file-reading probe above); re-entrancy refusal dropped (deadlocks the suite - detected by timeout, it can never pass).
bin/fm-lint.sh and fm-behavior-shards.sh --check 8 clean. Deploy note: after merge, cp $FM_HOME/state/azure-workers/controller.json{,.pre-c2}, run status --live, verify revision advanced, pending_actions {}, sentinel present; the live file's scalar is currently null so the migration is a no-op on this machine.