Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions bin/fm-worker-lifecycle.py
Original file line number Diff line number Diff line change
Expand Up @@ -4274,8 +4274,6 @@ def command_surrender(env, args):
raise LifecycleError("--confirm-subscription must exactly match FM_AZURE_SUBSCRIPTION_ID")
with controller_lock(env):
state = load_state(env)
if state.get("pending_actions"):
raise LifecycleError("a pending provider action exists; reconcile first")
key = request_key(args.task, args.task_generation)
item = state["queue"].get(key)
if item is not None and item.get("status") == "complete":
Expand All @@ -4289,6 +4287,8 @@ def command_surrender(env, args):
worker = state["workers"].get(str(item.get("slot")))
if worker is None or worker.get("queue_key") != key:
raise LifecycleError("surrender task has no exact durable worker owner")
if (state.get("pending_actions") or {}).get(str(worker["slot"])) is not None:
raise LifecycleError("the worker slot has a pending provider action; reconcile it first")
existing = worker.get("release_proof")
if existing is not None:
if isinstance(existing.get("surrender"), dict):
Expand Down
2 changes: 1 addition & 1 deletion docs/azure-workers.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ No age or cost override can convert retained-for-investigation into safe deletio
### Operator surrender for unrecoverable ordinary authority

`surrender` releases one exact ASSIGNED worker whose ordinary release authority can no longer be minted, for example when local teardown consumed the task metadata before any receipt existed.
Surrender is refusal-first, not a shortcut: the command runs the ordinary authority itself and refuses when that succeeds (and fails closed when the authority tool breaks rather than refuses), refuses live compute (the VM must be deallocated or stopped), refuses to replace an ordinary release proof, refuses while a pending provider action exists, refuses when the controller's own execution records show repository work whose landing is unproven unless the operator passes `--confirm-discard-unlanded`, and demands an operator `--reason` plus the same explicit confirmation pair as withdraw.
Surrender is refusal-first, not a shortcut: the command runs the ordinary authority itself and refuses when that succeeds (and fails closed when the authority tool breaks rather than refuses), refuses live compute (the VM must be deallocated or stopped), refuses to replace an ordinary release proof, refuses while a pending provider action exists on that worker's slot, refuses when the controller's own execution records show repository work whose landing is unproven unless the operator passes `--confirm-discard-unlanded`, and demands an operator `--reason` plus the same explicit confirmation pair as withdraw.
The minted bundle keeps the `fm.worker-release/v2` shape the deallocate/delete-compute/reset machinery fences on, but every authority verdict is `surrendered` - `release` rejects that verdict, so a surrender bundle can never replay through the ordinary release command - and a top-level `surrender` block records the operator reason and the ordinary authority's refusal verbatim.
Surrendering a parent whose queue still holds non-complete compartment children refuses unless the operator passes `--confirm-orphan-children`; with the flag, every live child's queue entry gains a durable `reparented_to: primary` note under the same lock hold that records the surrender, the `surrender` block records the orphan count, and the captain drives those children from the local home from then on.
After the proof is recorded, reconcile owns deallocation, compute deletion, and reset exactly as for an ordinary release, and the wrapper removes the task's locally staged provider credential keyed off the command's own `FM-SURRENDERED` receipt.
Expand Down
12 changes: 11 additions & 1 deletion tests/fm-worker-lifecycle.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4019,9 +4019,19 @@ def expect_refusal(state, call_args, fragment, *, attempt=None, provider=None):
# Malformed identities refuse before anything else runs.
expect_refusal(base_state(), args(task="../escape"), "bounded identifier characters")

# A pending provider action blocks the whole lane.
# A pending provider action blocks only its own worker slot. An unrelated
# stranded claim must not prevent a dark worker from taking its sanctioned
# surrender path.
state = base_state()
state["pending_actions"] = {"2": {"type": "execute", "request": {"task": "other", "task_generation": "gen-9"}}}
expect_refusal(
state, args(), "non-assigned or ambiguous",
attempt=lambda *_a: "WORKER AUTHORITY REFUSED: fixture refusal",
provider=lambda *_a, **_k: {"inventory": {"workers": []}},
)

state = base_state()
state["pending_actions"] = {"1": {"type": "execute", "request": {"task": "task-1", "task_generation": "gen-1"}}}
expect_refusal(state, args(), "pending provider action")

# A converged entry names its credential recovery instead of a generic refusal.
Expand Down
Loading