Repository navigation
Scale batch GPUs from admissible queued work - #23824
Merged
Merged
Conversation
Replace age-driven capacity demand with an indexed count of admitted first attempts and allowed retries. Existing held/unevaluated transitions stay excluded across mixed worker versions, while custody and paid authorization remain unchanged. Split OD/Spot HPAs keep a total floor2/cap24 without hard topology spread; verify both deployment roles and fresh deduplicated counts. The target remains provisional pending throughput qualification before apply. Lead owns deployment after the current5c verdict; this commit changes source. Failure-Class: none
Move the existing batch index families beside the new capacity count index. Preserve generated requirements and shrink the oversized central registry.
Git-on-my-level
had a problem deploying
to
development
October 27, 2026 18:43 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Keep production at four workers with the scaler off. Accept the pinned Firestore SDK's real empty count and keep optional capacity sampling failures out of recovery requests. Sample capacity only in dedicated workers. Update characterization goldens only for the intended unattempted metadata; retain synthetic private failure traces for hermetic investigation. Failure-Class: none
Copy the extracted batch index declarations into the temporary readiness action checkout so it exercises the same imports as the real repository. Validation: backend workflow-contract shard, 94 passed.
Member
Author
|
Final delta review: source SHIP at The reviewer identified a minor simulator gap: successful simulated jobs retain Production defaults remain fixed4 with autoscaling and balanced pools off. Target16 stays provisional until the approved worker benchmark, producer+sampler deployment, READY indexes, actual eligible-unit count and release approval are complete. |
Git-on-my-level
marked this pull request as ready for review
October 27, 2026 20:29
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This was referenced Oct 28, 2026
Open
Git-on-my-level
added a commit
that referenced
this pull request
Oct 28, 2026
…05065) (#23860) Reworks the dormant queue scaler (#23824, merged disabled) into the qualified cutover shape accepted for the 100% capacity plan: **demand-driven, min 3 / max 16, OD-first** (lead f05065; David b38eea — peak OK today, skip 50%). - `replicaCount` 4 → **3**: the ratified batch floor (lead 114a22/724762; op1 readback 3 desired / 3 Ready on 10-28). - Latent autoscaling budget 2/24 → **3/16**. Stays disabled in values; enabling happens only in op1's reviewed qualification window during the 25% soak, where a real scale-up doubles as the scheduling proof and op1 raises `batch-pipeline-pool`'s node maximum (3 → 16) in the same window. - **OD-first placement**: the worker pins to `batch-pipeline-pool` so the guaranteed floor and scale-out never depend on Spot (persistent L4 Spot stockout; Spot L4 bills above on-demand — N6 10-26). - `balancedPools` stays machinery-only; the chart rejects it in production when the stage, values namespace, or Helm release namespace marks production, and the docs now name the single OD HPA as the cutover. Going above 16 is a second, separately qualified step; never pre-provision. - `targetQueuedUnitsPerWorker=16` remains provisional until op1 benchmarks useful completions/queue latency on the qualified worker. Tests: environment/policy pins move to replicas 3; the balanced design test uses a non-production explicit even budget (2/24); production values reject balanced mode at min=3 or min=4 even with overrides, and a production Helm release namespace rejects balanced mode when chart values say dev. Production cutover remains a single OD HPA at min3/max16, pinned to OD with no `-spot` deployment; the adapter contract renders the single-pool cutover. Local targeted backend shard: 77 passed across `test_batch_pipeline_chart.py` and `test_monitoring_adapter_hpa_contract.py`. Failure-Class: none UI-Evidence: none -- reason: chart/docs/tests only; no web/admin/app paths <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/BasedHardware/omi/pull/23860?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="View guided diff" src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a> <a href="https://www.cubic.dev/action/auto-fix/pr/BasedHardware/omi/23860?returnTo=https%3A%2F%2Fgithub.com%2FBasedHardware%2Fomi%2Fpull%2F23860&source=description" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img alt="Turn on auto-fix" src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
YinkaAdepoju
pushed a commit
to YinkaAdepoju/omi
that referenced
this pull request
Oct 28, 2026
Rework the merged-but-dormant queue scaler (BasedHardware#23824) into the qualified cutover shape the lead accepted for the 100% capacity plan (f05065, David b38eea: peak OK today, skip 50%): demand-driven, min 3 / max 16, OD-first. - prod values: replicaCount 4 -> 3 (the ratified floor, lead 114a22/724762; op1 readback 3 desired / 3 Ready) and the latent autoscaling budget 2/24 -> 3/16. The worker pins to batch-pipeline-pool (on-demand) so the guaranteed floor and scale-out never depend on Spot: the L4 Spot stockout is persistent and Spot L4 bills above on-demand (N6 10-26). - balancedPools stays machinery-only: the template already refuses odd or undersized totals, so the production 3/16 budget refuses it by design; the docs now name the single OD HPA as the cutover. - Above 16 replicas is a second, separately qualified step; op1 raises the OD node-pool maximum (3 -> 16) in the same reviewed qualification window that enables the scaler, using a real scale-up during the 25% soak as the scheduling proof. - targetQueuedUnitsPerWorker=16 stays provisional until op1 benchmarks useful completions and queue latency on the qualified worker. Tests: the environment/policy pins move to replicas 3; the balanced design test gets an explicit even budget (2/24) so it keeps proving the machinery; new cases pin the production cutover (single OD HPA min3/max16, OD nodeSelector, no -spot deployment) and the odd-floor balanced refusal; the HPA-adapter contract test renders the actual single-pool cutover. Failure-Class: none
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img](https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a](https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a){kind=link}
![https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source](https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source){kind=link}
![https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img](https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img){kind=link}
![https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a](https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a){kind=link}
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parked allowance holds periodically re-enter
readyfor policy evaluation while keeping their original enqueue time. The existing age-only HPA could therefore request the GPU ceiling for one old held row. Use a fresh indexed count of admitted first attempts and allowed retries, and keep raw age as an SLO diagnostic.New admission records
compute_decision.outcome=unattempted. Existing worker versions already overwrite that with held/allowed or remove it on an unevaluated refusal, so mixed versions cannot leave a separate capacity eligibility bit true. The count also requires ready, merge-pending and currently available work. Unmarked legacy debt remains durable and drainable by warm workers; paid compute still requires the existing per-lease funding/privacy authorization.Production values retain four fixed workers with autoscaling and balanced pools off. The prepared cutover renders two pool-pinned deployments/HPAs, each min1/max12 (total2/24), with separate ownership/scrapes and one shared metric scope. The adapter deduplicates workers before summing the three priorities, requires all three fresh classes and never fabricates zero. AverageValue divides the shared demand across both pools. Deployment reporting rejects a missing, unready, zero-sized or wrong-image Spot companion. The query driver and generated prod/dev composite indexes are included.
Only the dedicated worker's supervised sampler runs capacity aggregates. Failed capacity queries retain the prior values and freshness without failing recovery/projection requests. The pinned Firestore SDK decodes a genuine integer-zero aggregate as
0.0; that exact result is accepted, while missing, fractional and malformed results remain unavailable. The real loopback gauntlet caught this boundary; focused tests now exercise the installed SDK decoder. Synthetic gauntlet processes retain failure tracebacks in private process logs, separately from sanitized evidence.Validation: targeted backend files run through
backend/test.shwith explicit file lists underomi-heavy/nice19. The18-file pass found one stale adapter-test assumption about the production default; the final five-file rerun covers that correction, capacity, chart, metrics and characterization. The other business, runtime, readiness and seven Firestore guard files passed; query-shape coverage includes511 cases. The full batch family of the hermetic Sync Cloud Tasks gauntlet passed, including recovery, process drain/kill, privacy, burst and two workers. A prior burst admission503 did not recur and remains an unresolved test finding; it is not claimed fixed. Canonical formatting, diff hygiene and all38make preflightchecks passed.Characterization goldens intentionally add only
final_job.compute_decision.outcome=unattempted:75 additions across11 files. Structural comparison confirms no other golden changes; paid-call counts, outcomes and transitions are unchanged. First review required preserving fixed4 until qualification; that fix is included. Final delta review and new-head CI remain pending. The initial CI development-index prebuild passed add-only validation then failed WIF authentication; the release owner identifies that as the existing PR-event identity condition issue, with no IAM change proposed here.Activation hold: queued-unit target16 is provisional slot arithmetic, not qualified throughput, and is unused by the production default. Before enabling the scaler, benchmark useful completions and queue latency on the exact worker and pin the measured target. The benchmark owner is bramp3, review deadline2026-10-28 09Z; the release owner approves the measured values and op1 applies them. No production configuration, admission cohort, GPU allocation, account data or paid provider was changed by this PR work. Verify the new indexes READY, both worker roles/images and the live external count before the approved cutover. Capture the current release first; rollback restores its pinned image/configuration and four Ready workers with normal SIGTERM drain. The fixed4 fallback remains in force until qualification.
Regional/global GPU quota was independently checked read-only: global12000 limit,22 allocated GPU quota units across43 regions; us-central1 L4OD160/16used and Spot32/1used. This is quota headroom, not stock availability or a billing/savings claim. No new workflows or IAM changes.
Cutover requires both the CPU admission producer that records
unattemptedand the GPU sampler image. A worker-only image roll can export a fresh zero count while older producers still create unmarked work; it remains a fixed4 instrumentation roll. Verify the actual serving producer revision, new-unit marker/count and new worker provenance before enabling either HPA. The observer's artifact pins and source qualification must follow the approved CPU/worker rollout; source merge is not installed/runtime acceptance.Failure-Class: none