Skip to content

fix(crosscheck): queue shared Azure capacity #328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ruby-dlee merged 2 commits into from
Aug 25, 2026
Merged

fix(crosscheck): queue shared Azure capacity #328

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a0eadab
fix(crosscheck): wait for shared Azure capacity
ruby-dlee Aug 25, 2026
0ef961a
no-mistakes(review): Route transient SKU capacity through durable all…
ruby-dlee Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 95 additions & 52 deletions bin/fm-crosscheck-azure.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,14 @@
MODEL_CAPTURE_BYTES = 16 * 1024 * 1024
MAX_ACTIVE_REVIEWS = 4
MAX_REVIEW_PACKET_BYTES = 1500 * 1024
CAPACITY_RETRY_SECONDS = 5
TRANSIENT_CAPACITY_REFUSALS = frozenset(
{
"exact selected-family observed-plus-reserved capacity is exhausted",
"specialized observed-plus-reserved demand exceeds its shared 40-vCPU shape",
"combined observed-plus-reserved demand would consume the shared East US ceiling",
}
)
STAGING_CONTAINER = "validation-shards"
AZURE_EVIDENCE_PATH_PATTERN = (
r"^\.crosscheck/(reproductions|mutations)/"
Expand Down Expand Up @@ -214,8 +222,9 @@ def preflight_reviewer_credential(core: Any, config: dict[str, str]) -> dict[str
same treatment any other environment fault gets.

The margin covers the review, not the wait in front of it, so this is
called twice: once to fail fast, and once after the lane is held, which
is the call that actually stands between a dead token and a paid VM.
called three times: once to fail fast, once after the lane is held, and
once after shared capacity is admitted. The final call stands between a
token that expired in either queue and Azure-staged data or a paid VM.

It also covers the gap between the check and the reviewer's first token:
scope verification, VM create, boot, and bundle upload all happen after
Expand Down Expand Up @@ -418,13 +427,6 @@ def verify_scope_and_foundation(config: dict[str, Any]) -> Any:
runner_env = runner.environment()
runner.scope_gate(runner_env)
runner.foundation_gate(runner_env)
runner.sku_quota_gate(
runner_env,
{
"sku": config["reviewer_sku"],
"sku_family": runner.SKU_FAMILY[config["reviewer_sku"]],
},
)
runner.budget_gate(
runner_env, {"sku": config["reviewer_sku"], "network_bytes": 0}
)
Expand Down Expand Up @@ -1213,13 +1215,33 @@ def reserve_model_capacity(config: dict[str, Any], identity: dict[str, Any], run
compartment holds one exact reservation with a cushioned worst-case amount
until its compute absence is proved.
"""
fence = hashlib.sha256(os.urandom(32)).hexdigest()
reservation_id = "ccm-" + identity["review_generation"][:12]
sku = config["reviewer_sku"]
family = runner.SKU_FAMILY[sku]
rate = runner.retail_rate(runner.environment(), sku)
amount = round(float(rate) * 24.0 * 1.5 + 5.0, 6)
result = shared_capacity_command([
# The allocator persists a queued row before returning its refusal. Bind
# the fence to the complete stable review identity so a command restarted
# after an ambiguous transport interruption reattaches to that exact row
# instead of stranding it behind a newly random fence.
fence = hashlib.sha256(canonical_bytes({
"schema": "fm.azure-crosscheck-capacity-fence/v1",
"reservation_id": reservation_id,
"review_generation": identity["review_generation"],
"subscription_binding": hashlib.sha256(
config["subscription"].encode("utf-8")
).hexdigest(),
"sku": sku,
"sku_family": family,
})).hexdigest()
capacity = {
"reservation_id": reservation_id,
"fence": fence,
"sku": sku,
"sku_family": family,
"amount_usd": amount,
}
arguments = [
"capacity-reserve",
"--reservation-id", reservation_id,
"--fence-binding", fence,
Expand All @@ -1229,30 +1251,37 @@ def reserve_model_capacity(config: dict[str, Any], identity: dict[str, Any], run
"--vcpus", str(runner.SKU_VCPUS[sku]),
"--amount-usd", str(amount),
"--confirm-subscription", config["subscription"],
])
if result.returncode != 0:
raise AzureCrosscheckError(
"shared allocator refused the model reservation: "
+ (result.stderr or result.stdout or "").strip()[-400:]
)
try:
reservation = json.loads(result.stdout)
except json.JSONDecodeError as exc:
raise AzureCrosscheckError("shared allocator returned a malformed model reservation") from exc
if (
reservation.get("reservation_id") != reservation_id
or reservation.get("status") not in ("reserved", "queued")
):
raise AzureCrosscheckError("shared allocator returned a model reservation with the wrong identity")
capacity = {
"reservation_id": reservation_id,
"fence": fence,
"sku": sku,
"sku_family": family,
"amount_usd": amount,
}
if reservation["status"] != "reserved":
]
wait_seconds = config["queue_wait_seconds"]
deadline = time.monotonic() + wait_seconds
while True:
result = shared_capacity_command(arguments)
if result.returncode != 0:
raise AzureCrosscheckError(
"shared allocator refused the model reservation: "
+ (result.stderr or result.stdout or "").strip()[-400:]
)
try:
reservation = json.loads(result.stdout)
except json.JSONDecodeError as exc:
raise AzureCrosscheckError(
"shared allocator returned a malformed model reservation"
) from exc
if (
not isinstance(reservation, dict)
or reservation.get("reservation_id") != reservation_id
or reservation.get("status") not in ("reserved", "queued")
):
raise AzureCrosscheckError(
"shared allocator returned a model reservation with the wrong identity"
)
if reservation["status"] == "reserved":
return capacity
reason = str(reservation.get("reason") or "capacity unavailable")[:300]
timed_out = time.monotonic() >= deadline
if reason in TRANSIENT_CAPACITY_REFUSALS and not timed_out:
time.sleep(min(CAPACITY_RETRY_SECONDS, max(0.0, deadline - time.monotonic())))
continue
try:
# capacity-reserve persists even refused candidates as queued. No
# model compute can exist yet, but capacity-release still asks the
Expand All @@ -1264,10 +1293,15 @@ def reserve_model_capacity(config: dict[str, Any], identity: dict[str, Any], run
"shared allocator queued the model compartment and its exact "
"zero-compute release failed: " + reason + "; " + str(exc)
) from exc
if timed_out and reason in TRANSIENT_CAPACITY_REFUSALS:
raise AzureCrosscheckError(
"shared allocator capacity queue wait exceeded {} seconds: {}".format(
wait_seconds, reason
)
)
raise AzureCrosscheckError(
"shared allocator queued the model compartment: " + reason
)
return capacity


def release_model_capacity(config: dict[str, Any], reservation: dict[str, Any]) -> None:
Expand Down Expand Up @@ -2071,10 +2105,8 @@ def run_azure_review(
# blocks in FIFO order for up to queue_wait_seconds - 7200 by default and
# 86400 at the maximum - which is far longer than the review margin, so a
# credential admitted as usable can be long dead by the time a lane frees.
# Under load, which is exactly when spend is highest, the first check is
# the one that proves nothing. This second check is the one that gates
# spend: every billable action happens after it, and it costs one local
# file read.
# This second check refuses that drift before foundation inspection. A
# third check after shared-capacity admission gates staging and compute.
preflight_reviewer_credential(core, config)
return _run_azure_review_in_lane(
core=core, root=root, home=home, task_id=task_id, pr_url=pr_url,
Expand Down Expand Up @@ -2236,20 +2268,26 @@ def _run_azure_review_in_lane(
}
uploaded: set[str] = set()
resources: dict[str, Any] | None = None
model_capacity: dict[str, Any] | None = None
cleanup_error: Exception | None = None
ledger_identity: dict[str, Any] | None = None
try:
with measured_phase(phase_timer, "create"):
model_capacity = reserve_model_capacity(azure, identity, runner)
except core.CrosscheckToolError:
raise
except Exception as exc:
# Reservation refusal happens before the model-resource cleanup
# window below exists. Normalize both allocator refusals and local
# subprocess failures here so the core records this exact reviewer
# as a tool failure and can advance to the next screened account.
raise core.CrosscheckToolError(str(exc)) from exc
try:
try:
with measured_phase(phase_timer, "create"):
model_capacity = reserve_model_capacity(azure, identity, runner)
except core.CrosscheckToolError:
raise
except Exception as exc:
# Normalize allocator refusals and local subprocess failures so
# the core records this exact reviewer as a tool failure. The
# outer cleanup window releases any admitted reservation.
raise core.CrosscheckToolError(str(exc)) from exc
preflight_reviewer_credential(core, config)
require_stable_reviewer_credential(
core,
config,
(credential, source, identifier, reviewer_account_identity),
)
with measured_phase(phase_timer, "stage"):
upload_blob(azure, input_path, staged["input_blob"])
uploaded.add(staged["input_blob"])
Expand Down Expand Up @@ -2401,13 +2439,18 @@ def _run_azure_review_in_lane(
cleanup_model_vm(azure, resources, identity)
except Exception as exc:
cleanup_error = exc
if cleanup_error is None:
if cleanup_error is None and model_capacity is not None:
try:
release_model_capacity(azure, model_capacity)
except Exception as exc:
cleanup_error = exc
blob_cleanup_errors: list[str] = []
for blob in sorted(uploaded | {staged["output_blob"]}):
expected_blobs = (
uploaded | {staged["output_blob"]}
if model_capacity is not None
else uploaded
)
for blob in sorted(expected_blobs):
try:
delete_exact_blob(azure, blob)
except Exception as exc:
Expand Down
11 changes: 7 additions & 4 deletions docs/azure-crosscheck.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ A wrong account, model, generation, VM, boot, request, transport, or cleanup ide
The model compartment's egress allowlist is Azure-provided DNS plus the exact provider API endpoint, and a provider auth host is not on it.
A reviewer CLI inside the compartment therefore cannot refresh an expired session, so a dead credential buys a real VM and returns a tool failure instead of a verdict.

Every review runs `bin/fm-credential-expiry.py` against the selected reviewer's account home twice: once before the FIFO lane wait, before any Azure call and before any staged object, and again once the lane is held.
The second check is the one that gates spend. The margin covers the review, not the queue in front of it, and `FM_AZURE_CROSSCHECK_QUEUE_WAIT_SECONDS` bounds that queue at 7200 seconds by default and 86400 at its maximum, so a credential admitted before the wait can be long dead by the time a lane frees - under load, which is when spend is highest.
Every review runs `bin/fm-credential-expiry.py` against the selected reviewer's account home three times: before the FIFO lane wait, once the lane is held, and after shared capacity is admitted.
The third check gates Azure staging and compute because `FM_AZURE_CROSSCHECK_QUEUE_WAIT_SECONDS` bounds both lane and shared-capacity waiting at 7200 seconds by default and 86400 at its maximum.
The credential must be `usable` and must still be usable after the review deadline (`FM_CROSSCHECK_REVIEWER_TIMEOUT_SECONDS`); `refreshable` is refused because it is not recoverable inside the compartment.
A refusal is an ordinary tool failure, so the roster records the account and rotates to the next policy-screened reviewer rather than ending the review.
The preflight reads expiry instants and account paths only, and never emits token material.
Expand Down Expand Up @@ -172,7 +172,8 @@ It creates no role assignment, provider registration, public path, support ticke
There is no warm review compute and no review queue daemon.
Zero waiting reviews means zero model, tool, or verifier VMs.
Review capacity is owned by the released whole-fleet allocator in [Elastic task workers](azure-workers.md): every model compartment reserves one exact SKU/family/cost constituent through `capacity-reserve` before compute and releases it only after proven compute absence, tool and verifier invocations reserve through the released runner's own shared-allocator bridge, and review demand shares the 40-vCPU specialized envelope with no-mistakes validation under the single 128-vCPU East US ceiling.
A queued shared reservation refuses the review rather than creating capacity, and the local software cap (default four active model compartments, configurable one through eight) remains only a concurrency safety bound, never a capacity authority.
A queued shared reservation caused by exact-family or shared-capacity pressure is retried with the same durable reservation identity until `FM_AZURE_CROSSCHECK_QUEUE_WAIT_SECONDS` expires, while budget, daily-bound, credential, identity, and other allocator failures remain immediate.
Timeout releases the exact queued reservation before failing, and the local software cap (default four active model compartments, configurable one through eight) remains only a concurrency safety bound, never a capacity authority.
Regional and exact-family quota can impose a lower effective ceiling.

Two admitted reviews have distinct review generations, staged object prefixes, model VMs, tool invocations, verifier invocations, process trees, scratch, credentials, and cleanup authorization.
Expand Down Expand Up @@ -277,11 +278,13 @@ The home-local configuration is optional and gitignored:
"enabled": true,
"provider_host": "exact-provider-host.example",
"provider_port": 443,
"reviewer_sku": "Standard_D4as_v6",
"model_image_id": "/subscriptions/.../resourceGroups/.../providers/Microsoft.Compute/galleries/.../images/.../versions/1.0.0"
}
```

Omitting `reviewer_sku` spreads the default four lanes across the reviewed SKU families.
An explicit `reviewer_sku` remains an opt-in diagnostic override that pins every lane.

The required environment is the accepted foundation's existing `FM_HOME`, `FM_AZURE_TENANT_ID`, `FM_AZURE_SUBSCRIPTION_ID`, `FM_AZURE_NAMING_PREFIX`, `FM_AZURE_STORAGE_NAME`, `FM_AZURE_OWNER_TAG`, `FM_AZURE_DEPLOYMENT_GENERATION`, and independently accepted `FM_AZURE_BLOB_PE_NIC_RESOURCE_GUID`, plus the exact image through the config or `FM_CROSSCHECK_AZURE_MODEL_IMAGE_ID`.
The standard Crosscheck reviewer roster remains `config/crosscheck-reviewer.json` and keeps its existing account/model policy.

Expand Down
3 changes: 3 additions & 0 deletions docs/azure-requirements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1128,6 +1128,9 @@ commits only on success, and is refused if its effects reach outside the one com
owns.
The later per-slot pending map and lock-discipline changes completed that work.

Crosscheck model admission now retains the shipped four-lane FIFO model while transient exact-family or shared-capacity pressure polls one durable allocator reservation identity within the configured queue wait.
Timeout releases that exact queued identity, non-capacity refusals remain immediate, and reviewer credentials are rechecked after admission before staging or billable compute.

One thing not to do, found while designing this: the three capacity commands stay fully locked.
`merged_specialized_reservations` ignores local reservations whose status is not `reserved`, so a
candidate parked by one concurrent reserve is invisible to another's admission arithmetic and two
Expand Down
Loading
Loading