feat(worker): daily spend bound, idle deallocate, and cooldown stamp (C3)

[ruby-dlee]

Requirement

docs/azure-requirements.md C3 ("Cost guard"): a day's spend cannot quietly reach 100 dollars. Three parts: a daily spend bound with an explicit operator override, an idle-release path for an assigned worker whose task ended (the live example: wkr-04 idled ~4h with no release proof until its TTL fired), and the R10 metering/standing-cost booking note.

What this builds

1. Daily spend bound (create and resume only)

• FM_AZURE_WORKER_DAILY_BOUND_USD, default 100 per the requirement sentence. Unset means 100; an explicit zero, negative, or non-numeric value refuses loudly at environment() instead of meaning "no bound".
• Cost Management gives month-to-date actual only, so day spend is computed honestly: a durable daily_cost_baseline record {utc_day, actual_usd_at_day_start} is snapshotted under the existing fleet lock on the first observation of each new UTC day (planner entry, live status, resume), and day spend = current actual - baseline, clamped at zero. The UTC month boundary is also a UTC day boundary, so the month-to-date reset always coincides with a baseline roll.
• Once day spend >= bound, every NEW compute-creating or compute-resuming provider action refuses: the planner's create branch returns an admission-refused action with the exact string naming the day, spend, and bound, and command_resume raises the same string before touching any worker field. An unreadable actual fails the bound closed.
• Wind-down is never blocked: deallocate/delete-compute/reset planning runs before the create branch in next_reconcile_action, and release, surrender, withdraw, and the claim-exempt message lane never route through the check. execute on an already-assigned worker also stays allowed: that capacity is already held and billing, so refusing its work burns the same money for nothing. Both choices are documented in docs/azure-workers.md.
• Override: FM_AZURE_WORKER_DAILY_BOUND_OVERRIDE=<utc-day> admits past the bound only when it names the exact current UTC day. Its use is printed loudly (reconcile text output, resume stdout), carried on the admitted action itself (daily_bound_override field, visible in public_action/JSON output), and recorded durably in daily_bound_override_used, which status prints as DAILY BOUND OVERRIDE USED.
• ACM-lag honesty, stated in code and docs: actual lags hours, so this bound is a backstop on RECORDED spend, not a real-time meter. The same-day protectors ahead of it are the existing per-mutation cumulative admission and the idle deallocate below; the bound guarantees a day cannot keep admitting new compute after the recorded number crosses it. No fake intraday precision.

2. Idle deallocate (not release)

• FM_AZURE_WORKER_IDLE_RELEASE_SECONDS, default 14400 (the wkr-04 four hours), floor 600, cap 604800.
• Reconcile plans an automatic DEALLOCATE for a worker that is: classification assigned with the VM still running, queue item still assigned, no pending_actions claim on the slot (the planner already skips claimed slots, and an in-flight execute always holds a claim), and last_execution_at (stamped only when a durably applied execution result exists) older than the threshold.
• Provable signals only: a worker with no recorded execution is never idle-deallocated, because nothing in controller state proves its task ended; its per-VM TTL stays the backstop for that case, and this is stated in the docstring and docs.
• Deallocate, deliberately not release/surrender: releasing requires authority receipts a machine cannot mint; deallocation is reversible and stops the compute spend, which is what C3 wants. The boundary sentence in the docs: idle-deallocate stops compute cost unattended; the ordinary release stays human-driven.
• The apply stamps idle_deallocated_at durably, and status output LOUDLY lists every idle-deallocated worker (IDLE-DEALLOCATED WORKER: ... RELEASE IT PROPERLY) plus a idle_deallocated_workers array in JSON, until the ordinary release/reset completes.

3. Cooldown-stamp fix (adjacent known gap)

A release-proved worker whose VM was already dark but whose cooldown_started_at was null waited forever: only the controller's own deallocate apply stamps the field, and surrender's dark-compute gate requires an operator-side deallocate that never does. The planner previously computed started = now on every pass, so the cooldown clock restarted forever and delete-compute never became due (live: slot 1, d2-probe-20260819). The planner now stamps the field durably under the lock on first observation of that exact shape, so the clock starts and delete-compute becomes due after cooldown_seconds.

4. Docs and C3 status

docs/azure-workers.md gains the daily bound, override, ACM-lag, and idle-deallocate contract sentences in the admission, cooldown, and operator-override sections; the wrapper header lists the three new env vars. docs/azure-requirements.md C3 moves to BUILT (live acceptance pending) with the honest ACM-lag caveat and the R10 booking note: the Slack listener runs on the operator mac in v1 (standing Azure cost ~0) and its per-submitter daily ledger lives under $FM_HOME/state/crosscheck-slack.

C2 machinery untouched

No change to pending_actions claims, FencedState, the load fence, revision CAS, slot_lease, provider_mutate, claim_pending/apply_pending/drain_pending, or assert_scoped. The new planner writes (baseline roll, cooldown stamp) happen under the fleet lock in the plan phase, before any claim, exactly like the existing classification refreshes; the one new apply-phase write (idle_deallocated_at) is inside the worker's own compartment.

Tests

Three new units in tests/fm-worker-lifecycle.test.sh (suite: 22 -> 25 executed units, all green, exit 0):

• daily_bound_and_idle_matrix (direct module): env parsing (unset default 100; "0"/"-5"/"abc"/"nan"/"inf"/empty refuse; idle floor/cap), baseline roll on injected relative datetimes (no hardcoded wall-clock dates), exact refusal string equality, exact-day override recorded / wrong-day override refused / unreadable actual fails closed, idle boundary at exactly 14400 vs 14399 seconds, never-executed and dark-VM and non-assigned-item exclusions, claimed-slot skip through next_reconcile_action, and the cooldown stamp landing once with delete-compute due after cooldown_seconds (injected time).
• daily_bound_cli (fixture provider, end to end): baseline snapshot on first reconcile; exact refusal string through the real CLI; wind-down (deallocate/delete-compute/reset) still applying while the bound is tripped; wrong-day override refused with the mismatch named; exact-day override admitting with loud print, action field, durable record, and status output; the resume call site refusing then admitting under override; a stale yesterday baseline rolling to today; and loud env-parse refusals. A UTC-midnight crossing mid-unit exits 99 and the unit reruns once on a fresh fixture, so no assertion depends on a date that can go stale.
• idle_deallocate_cli (fixture provider, end to end): below-threshold no-op, past-threshold unattended deallocate (fixture VM goes dark, idle_release on the action, durable marker, queue item stays assigned, no release minted), loud status listing, no re-deallocation of dark compute, never-executed worker untouched, and the ordinary release/reset path completing afterwards.

static_contract also pins the new machinery names in the controller and the new contract sentences in the doc. Adjacent suites re-run green: fm-azure-pilot, fm-spawn-cloud, fm-cloud-state, fm-gate-refuse.

Mutation proofs (each applied to the production call site, proven red, reverted)

1. Planner bound check dropped (if bound_refusal is not None: -> if False:): daily_bound_cli red - "AssertionError: [{... 'type': 'create'}]" (a create was admitted past the tripped bound). 24/25 prior-and-other units unaffected until the failing unit.
2. Override made day-agnostic (if override == day: -> if override:): daily_bound_and_idle_matrix red at the wrong-day refusal assert.
3. Cooldown stamp dropped by restoring the exact pre-fix line (started = parse_time(...) if ... else now): daily_bound_and_idle_matrix red at the stamp/delete-compute-due asserts ("AssertionError: None"); all 22 pre-existing units still green, proving the regression is caught only by the new coverage.
4. Idle threshold weakened two ways: return True goes red in the existing end_to_end_lifecycle (premature deallocate perturbs the action sequence - defense in depth), and the subtler >= threshold // 2 goes red precisely at the new 14399-second boundary assert.
5. Resume call-site refusal dropped (raise LifecycleError(bound_refusal) -> clears the refusal): daily_bound_cli red at the resume-blocked assert (returncode 0 instead of 2).

Final clean run after reverting: 25/25 ok, exit 0.

Honest limits

• The bound reacts to RECORDED spend; an intraday burst inside ACM's reporting lag is caught by cumulative admission and idle deallocate, not by this bound. Documented rather than papered over.
• Specialized capacity-reserve/capacity-reserve-shape stay governed by the existing cumulative admission only; the daily bound scope is exactly the worker create/resume lanes the requirement names.
• A worker that never executed is outside the idle path by design (no provable end-of-task signal); its TTL remains the backstop.
• tests/behavior-test-durations.tsv still carries the suite's old 12s baseline; the suite is slower now and the TSV refreshes from CI recordings per its own header.
• Live billable acceptance of C3 has not run; the requirement doc says so.

Review round 2 (adversarial review findings, commit dc408f3)

Finding 1 (MAJOR) - specialized reserves now sit under the daily bound. capacity-reserve and capacity-reserve-shape gate NEW reservation admissions through the same daily_bound_refusal as create/resume, with the same FM_AZURE_WORKER_DAILY_BOUND_OVERRIDE lane; the disposable runner performs these automatically, so the ungated lane could quietly burn past 100 with no human anywhere. The three capacity commands stay fully locked and unsplit: the gate sits inside each reserve entrance's existing single lock hold, immediately before capacity_admission, identically at both. Exemptions, both documented: capacity-release is wind-down and stays ungated; a lineage re-admission of an already-reserved shape constituent is already-held accounting (the execute rationale) and skips the gate. The refusal string now names the reserve lane ("new compute and reservations (create/resume/capacity-reserve) are refused"), and reserve JSON output carries daily_bound_override when an override admitted. docs/azure-requirements.md C3 acceptance is unscoped again. The pre-existing shared_shape_cli unit pins the daily bound to 100000 with a comment, because its 1499-pressure step exists to reach the CUMULATIVE refusal and the daily bound has its own dedicated units.

Finding 2 (MINOR) - steer counts as recency, and the terminal boundary is documented. idle_deallocate_due now takes the NEWEST durable activity stamp (last_execution_at, last_steer_at) against the threshold, so a worker steered minutes ago is never deallocated; the never-executed exclusion is unchanged. Docs now state plainly that idle-deallocate is operationally terminal for the assignment (execute cannot run on deallocated compute, resume needs the VM absent, no power-on lane exists; release or surrender is the exit) and that FM_AZURE_WORKER_IDLE_RELEASE_SECONDS is the knob compartment operators must raise - specifically the pending secondmate monitor renews legs at 14,400 seconds, exactly the default threshold, so it must run with the knob above its renewal cadence.

Finding 3 (MINOR) - FM_AZURE_WORKER_IDLE_RELEASE_SECONDS=abc now refuses through the loud LifecycleError lane ("must be an integer between 600 and 604800") instead of a raw ValueError traceback, mirroring the daily-bound parse.

Finding 4 (NOTE) - override use is recorded as an effect, not an intent. daily_bound_refusal no longer writes daily_bound_override_used; the new record_daily_override_use is called only after the admission decision actually admitted (planner create after admission_result, resume after its guard - resume has no further gate - and both reserve entrances after reserved lands). A unit proves bound-tripped + override-named + cumulative-refusal leaves no durable "used" claim.

New/updated tests (still 25 executed units, all green, exit 0): matrix adds non-numeric idle parse, steer-recency boundary (recent steer shields, old steer does not), check-never-records + record_daily_override_use semantics, and the planner effect-not-intent scenario (override + cumulative refusal -> no record; override + room -> create carries the day and records). daily_bound_cli adds: reserve refused past the tripped bound with the exact string, shape refused with the exact string, reserve admitted under the exact-day override (output field + durable record), and capacity-release applying while tripped. idle_deallocate_cli adds the steer-recency reconcile check and the loud non-numeric env refusal (asserts no Traceback).

New mutation proofs (applied, red, reverted):
6. capacity-reserve daily gate dropped (if readmission_id is None: -> if False:): daily_bound_cli red - the reservation came back "status": "reserved" past the tripped bound.
7. capacity-reserve-shape daily gate dropped (gate result forced to None, None): daily_bound_cli red - the shape came back reserved past the tripped bound.

CI lint: the SC2034 attempt warning is fixed by using the variable in the midnight-retry notice line; full-repo bin/fm-lint.sh run locally, rc=0.