Repository navigation
perf(batch): prepared flights, VAD segmentation reuse and CUDA-graph TDT on the GPU worker (+61% L4 bench) - #24384
Merged
Merged
Conversation
…GPU worker Ports three accepted GPU-efficiency hill-climb ticks onto main, each behind a ship-on kill switch read through ProviderConfig: - BATCH_PIPELINE_PREPARE_FLIGHT_ENABLED (worker.prepareFlightEnabled): a lane waiting for the GPU prepares every leading unit of its flight that fits the shared 1800-audio-second decode budget, and the chain returns finished preparations when its GPU section ends instead of after publication. - BATCH_PIPELINE_VAD_SEGMENTATION_REUSE_ENABLED (worker.vadSegmentationReuseEnabled): a unit diarized at full length takes the raw segmentation scores its own VAD pass computed instead of a second forward. Dev L4 bench (g2-standard-8, simulated storage IO, prod chart env): main 70.35 / 70.24 -> this 113.65 / 112.85 audio-h per GPU-h; GPU section 99.9 -> 61.7 s; diarize 19.9 -> 15.9 s. Gate vs the L4 main run: word edit rate 0.0, speaker agreement 1.0, no lost speech or class changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ports lane/gpe-cgraph (explicit, asserted decoding mode; a mismatch in either direction quarantines the process) and turns it on by default on L4 evidence. BATCH_PIPELINE_ASR_CUDA_GRAPHS_ENABLED / worker.asrCudaGraphsEnabled: false restores asserted eager decoding. Dev L4 paired runs (same tree, eager vs graphs, prod chart env, simulated IO): ASR stage 13.6 -> 10.1 s (-26%), audio-h per GPU-h 57.1 -> 58.7, word edit rate 0, speaker agreement 1.0, CUDA peak +0.15 GiB. On the full stack the ASR stage drops 13.5 -> 10.0 s; the bench objective stays ~113 because the L4 bench is then bound by its simulated artifact IO. The T4 FP32 fallback from the hill climb is not ported: prod batch runs only on L4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
9 issues found across 20 files
Confidence score: 3/5
batch_pipeline_models.pycan fail worker startup when CUDA graphs are disabled because NeMo usescuda_graphs_mode='no_graphs', notNone. Treatno_graphsas disabled.feature-flag-registry.mdcan incorrectly report Helm-declared batch-worker flags as undeclared, hiding their kill switches. Include the batch-worker chart when generating the registry.- The benchmark can produce misleading comparisons: graph capture may be timed, and the recorded config and documentation can disagree with the mode actually run. Warm each graph configuration before timing and keep the receipt and instructions aligned with the selected mode.
batch_pipeline_runtime.mdoverstates the preparation guarantee: dispatch can wait on a decode future while holding the GPU lock. Describe preparation as reducing fetch waits rather than preventing them.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="backend/tests/unit/test_batch_pipeline_runtime.py">
<violation number="1" location="backend/tests/unit/test_batch_pipeline_runtime.py:318">
P2: This startup test does not verify the kill switch it claims to cover: a regression that ignored `provider_config().asr_cuda_graphs` during worker construction would still pass. Exercise startup with the environment set to `false` and assert both the model setting and `worker.words.cuda_graphs` are `False`; also set the enabled value explicitly so ambient deployment variables cannot make the test non-hermetic.</violation>
</file>
<file name="backend/tests/unit/test_batch_pipeline_worker_bundle.py">
<violation number="1" location="backend/tests/unit/test_batch_pipeline_worker_bundle.py:867">
P2: This assertion is timing-dependent because the three prepared decodes run concurrently and `decoded` records completion order. Compare an order-independent collection so the test verifies exactly-once decoding without flaking under scheduler variation.</violation>
</file>
<file name="backend/docs/listen_pusher_pipeline/changes/2026-11-02-batch-gpu-flight-prep-vad-reuse.md">
<violation number="1" location="backend/docs/listen_pusher_pipeline/changes/2026-11-02-batch-gpu-flight-prep-vad-reuse.md:21">
P2: The readback instruction misstates the metric granularity and omits its production gate: `worker_flight` is one observation per GPU flight, not per `fetch_decode` unit, and the histogram is normally disabled in production. Distinguish the per-flight and per-unit series and state that stage profiling must be enabled for this readback.</violation>
</file>
<file name="backend/diarizer/batch_pipeline_runtime.md">
<violation number="1" location="backend/diarizer/batch_pipeline_runtime.md:51">
P2: Preparation overlaps fetches but cannot guarantee that the GPU section never waits: dispatch calls `future.result()` for unfinished prepared decodes while the GPU lock is held. Change this to “reduces fetch waits” or qualify the claim for preparations that finish before dispatch.</violation>
</file>
<file name="backend/docs/feature-flag-registry.md">
<violation number="1" location="backend/docs/feature-flag-registry.md:901">
P2: These flags are declared by the batch-worker Helm deployment, so listing them as undeclared falsely says they run code defaults and hides their Helm kill switches. Update the registry generator to inspect this chart’s template or values wiring, then regenerate the document.</violation>
</file>
<file name="backend/diarizer/batch_pipeline_benchmark.py">
<violation number="1" location="backend/diarizer/batch_pipeline_benchmark.py:64">
P3: The benchmark receipt contradicts itself when the graph kill switch is false: the actual decoder runs eager mode, but each row’s serialized `config.asr_cuda_graphs` remains `True`. Populate the row config with the selected `cuda_graphs` value or omit that field from the per-cell configuration.</violation>
<violation number="2" location="backend/diarizer/batch_pipeline_benchmark.py:64">
P2: Graph-enabled rows include one-time CUDA graph capture in their timings: the only warm-up is a single 30-second batch before the size/context loops, so batch-8/32 and context-2 rows use a new graph during measurement. Warm each row with its actual first batch before starting the timer, or report capture and steady-state timings separately.</violation>
<violation number="3" location="backend/diarizer/batch_pipeline_benchmark.py:76">
P3: The benchmark documentation still describes this script as pinning eager decoding, but the new default runs CUDA-graph decoding. Update the qualification instructions and historical note to state the selected mode, or explicitly set `BATCH_PIPELINE_ASR_CUDA_GRAPHS_ENABLED=false` for the documented eager baseline.</violation>
</file>
<file name="backend/diarizer/batch_pipeline_models.py">
<violation number="1" location="backend/diarizer/batch_pipeline_models.py:92">
P1: The eager kill switch can fail worker startup even though CUDA graphs are disabled: NeMo represents the disabled decoder with `cuda_graphs_mode='no_graphs'`, but this check accepts only `None`. Accept the disabled mode as well while continuing to require `full_graph is None`.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| if cuda_graphs: | ||
| actual = computer.allow_cuda_graphs is True and mode in _GRAPH_MODES | ||
| else: | ||
| actual = computer.allow_cuda_graphs is False and mode is None and computer.full_graph is None |
There was a problem hiding this comment.
P1: The eager kill switch can fail worker startup even though CUDA graphs are disabled: NeMo represents the disabled decoder with cuda_graphs_mode='no_graphs', but this check accepts only None. Accept the disabled mode as well while continuing to require full_graph is None.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/diarizer/batch_pipeline_models.py, line 92:
<comment>The eager kill switch can fail worker startup even though CUDA graphs are disabled: NeMo represents the disabled decoder with `cuda_graphs_mode='no_graphs'`, but this check accepts only `None`. Accept the disabled mode as well while continuing to require `full_graph is None`.</comment>
<file context>
@@ -58,27 +71,38 @@ def pin_parakeet_decoding(model):
+ if cuda_graphs:
+ actual = computer.allow_cuda_graphs is True and mode in _GRAPH_MODES
+ else:
+ actual = computer.allow_cuda_graphs is False and mode is None and computer.full_graph is None
+ if not actual:
raise RuntimeError('worker CUDA graph policy violated; restart required')
</file context>
Suggested change
| actual = computer.allow_cuda_graphs is False and mode is None and computer.full_graph is None | |
| actual = computer.allow_cuda_graphs is False and mode in (None, 'no_graphs') and computer.full_graph is None |
| assert asr.cfg.decoding.greedy.use_cuda_graph_decoder is False | ||
| # Graph decoding ships on; the kill switch restores asserted eager decoding. | ||
| assert asr.cfg.decoding.greedy.use_cuda_graph_decoder is True | ||
| assert worker.words.cuda_graphs is True |
There was a problem hiding this comment.
P2: This startup test does not verify the kill switch it claims to cover: a regression that ignored provider_config().asr_cuda_graphs during worker construction would still pass. Exercise startup with the environment set to false and assert both the model setting and worker.words.cuda_graphs are False; also set the enabled value explicitly so ambient deployment variables cannot make the test non-hermetic.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/tests/unit/test_batch_pipeline_runtime.py, line 318:
<comment>This startup test does not verify the kill switch it claims to cover: a regression that ignored `provider_config().asr_cuda_graphs` during worker construction would still pass. Exercise startup with the environment set to `false` and assert both the model setting and `worker.words.cuda_graphs` are `False`; also set the enabled value explicitly so ambient deployment variables cannot make the test non-hermetic.</comment>
<file context>
@@ -239,7 +313,9 @@ def test_startup_pins_precision_reuses_segmentation_and_owns_one_gpu(monkeypatch
- assert asr.cfg.decoding.greedy.use_cuda_graph_decoder is False
+ # Graph decoding ships on; the kill switch restores asserted eager decoding.
+ assert asr.cfg.decoding.greedy.use_cuda_graph_decoder is True
+ assert worker.words.cuda_graphs is True
fake_torch.cuda.device_count = lambda: 2
startup_calls = list(calls)
</file context>
| finally: | ||
| prepared.close() | ||
| assert [r.segment_id for r in results] == ['a', 'b', 'c', 'd'] | ||
| assert decoded == ['a', 'b', 'c', 'd'] |
There was a problem hiding this comment.
P2: This assertion is timing-dependent because the three prepared decodes run concurrently and decoded records completion order. Compare an order-independent collection so the test verifies exactly-once decoding without flaking under scheduler variation.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/tests/unit/test_batch_pipeline_worker_bundle.py, line 867:
<comment>This assertion is timing-dependent because the three prepared decodes run concurrently and `decoded` records completion order. Compare an order-independent collection so the test verifies exactly-once decoding without flaking under scheduler variation.</comment>
<file context>
@@ -850,3 +850,129 @@ def test_strict_vad_shadow_sampling_skips_inspect_and_counters_without_changing_
+ finally:
+ prepared.close()
+ assert [r.segment_id for r in results] == ['a', 'b', 'c', 'd']
+ assert decoded == ['a', 'b', 'c', 'd']
+ assert worker._decode_budget.seconds == 0
+
</file context>
Suggested change
| assert decoded == ['a', 'b', 'c', 'd'] | |
| assert sorted(decoded) == ['a', 'b', 'c', 'd'] |
Comment on lines
+21
to
+22
| roll: `omi_batch_pipeline_stage_seconds` for `worker_flight` (GPU section) per | ||
| `fetch_decode` unit, and the `diarize` stage mean, against the pre-roll window. |
There was a problem hiding this comment.
P2: The readback instruction misstates the metric granularity and omits its production gate: worker_flight is one observation per GPU flight, not per fetch_decode unit, and the histogram is normally disabled in production. Distinguish the per-flight and per-unit series and state that stage profiling must be enabled for this readback.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/docs/listen_pusher_pipeline/changes/2026-11-02-batch-gpu-flight-prep-vad-reuse.md, line 21:
<comment>The readback instruction misstates the metric granularity and omits its production gate: `worker_flight` is one observation per GPU flight, not per `fetch_decode` unit, and the histogram is normally disabled in production. Distinguish the per-flight and per-unit series and state that stage profiling must be enabled for this readback.</comment>
<file context>
@@ -0,0 +1,24 @@
+ themselves. `false` restores the second forward.
+
+Both are worker-only kill switches, read once at startup. Readback after the
+roll: `omi_batch_pipeline_stage_seconds` for `worker_flight` (GPU section) per
+`fetch_decode` unit, and the `diarize` stage mean, against the pre-roll window.
+See [runtime](../../../diarizer/batch_pipeline_runtime.md) and
</file context>
Suggested change
| roll: `omi_batch_pipeline_stage_seconds` for `worker_flight` (GPU section) per | |
| `fetch_decode` unit, and the `diarize` stage mean, against the pre-roll window. | |
| roll (with stage profiling enabled): `omi_batch_pipeline_stage_seconds` for `worker_flight` (one GPU-section observation per flight), `fetch_decode` (one per unit), and the `diarize` stage mean, against the pre-roll window. |
| before publication, with renewal still active. A pull admits at most 32 units, | ||
| exclusive behind one asyncio lock. Each waiting lane prepares the leading | ||
| units of its flight that fit one shared 1800-audio-second decode budget, using | ||
| the same artifact IO pool (default five), so the GPU section does not wait on |
There was a problem hiding this comment.
P2: Preparation overlaps fetches but cannot guarantee that the GPU section never waits: dispatch calls future.result() for unfinished prepared decodes while the GPU lock is held. Change this to “reduces fetch waits” or qualify the claim for preparations that finish before dispatch.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/diarizer/batch_pipeline_runtime.md, line 51:
<comment>Preparation overlaps fetches but cannot guarantee that the GPU section never waits: dispatch calls `future.result()` for unfinished prepared decodes while the GPU lock is held. Change this to “reduces fetch waits” or qualify the claim for preparations that finish before dispatch.</comment>
<file context>
@@ -39,10 +46,18 @@ legacy provider clients into the GPU image is unnecessary.
-before publication, with renewal still active. A pull admits at most 32 units,
+exclusive behind one asyncio lock. Each waiting lane prepares the leading
+units of its flight that fit one shared 1800-audio-second decode budget, using
+the same artifact IO pool (default five), so the GPU section does not wait on
+fetches (`BATCH_PIPELINE_PREPARE_FLIGHT_ENABLED`, Helm
+`worker.prepareFlightEnabled`, default on; `false` prepares only the first
</file context>
Suggested change
| the same artifact IO pool (default five), so the GPU section does not wait on | |
| the same artifact IO pool (default five), so the GPU section usually avoids waiting on |
| - `BATCH_CONVERSATION_SPEAKERS_ENABLED` — Default on within persisted authoritative batch records; link unit speaker groups acoustically and name conversation groups (fail: open) | ||
| - `BATCH_JOBSTORE_MODE` — Default firestore; shadow mirrors committed batch job-store writes to the staging Postgres and counts read parity (Firestore stays the only authority) (fail: closed) | ||
| - `BATCH_LEGACY_CAPACITY_FALLBACK_ENABLED` — Default on; while the GPU ring reads saturated (see BATCH_VENDOR_OVERFLOW_MIN_READY_UNITS), a new fresh upload from an owner whose legacy sync is still on stays on legacy sync instead of queueing behind the ring. false queues it on batch as before (fail: open) | ||
| - `BATCH_PIPELINE_ASR_CUDA_GRAPHS_ENABLED` — Default on unless false; batch worker Parakeet TDT greedy_batch label-looping decoding runs NeMo CUDA graphs instead of eager kernels (same algorithm, identical words); the chosen mode is asserted before and after every transcription and a mismatch quarantines the process (fail: open) |
There was a problem hiding this comment.
P2: These flags are declared by the batch-worker Helm deployment, so listing them as undeclared falsely says they run code defaults and hides their Helm kill switches. Update the registry generator to inspect this chart’s template or values wiring, then regenerate the document.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/docs/feature-flag-registry.md, line 901:
<comment>These flags are declared by the batch-worker Helm deployment, so listing them as undeclared falsely says they run code defaults and hides their Helm kill switches. Update the registry generator to inspect this chart’s template or values wiring, then regenerate the document.</comment>
<file context>
@@ -895,6 +898,7 @@ their code default (`fail` tells you which way a missing value resolves).
- `BATCH_CONVERSATION_SPEAKERS_ENABLED` — Default on within persisted authoritative batch records; link unit speaker groups acoustically and name conversation groups (fail: open)
- `BATCH_JOBSTORE_MODE` — Default firestore; shadow mirrors committed batch job-store writes to the staging Postgres and counts read parity (Firestore stays the only authority) (fail: closed)
- `BATCH_LEGACY_CAPACITY_FALLBACK_ENABLED` — Default on; while the GPU ring reads saturated (see BATCH_VENDOR_OVERFLOW_MIN_READY_UNITS), a new fresh upload from an owner whose legacy sync is still on stays on legacy sync instead of queueing behind the ring. false queues it on batch as before (fail: open)
+- `BATCH_PIPELINE_ASR_CUDA_GRAPHS_ENABLED` — Default on unless false; batch worker Parakeet TDT greedy_batch label-looping decoding runs NeMo CUDA graphs instead of eager kernels (same algorithm, identical words); the chosen mode is asserted before and after every transcription and a mismatch quarantines the process (fail: open)
- `BATCH_PIPELINE_ASR_LEFT_CONTEXT_SECONDS` — Default 0 (off); seconds of the same capture stream's contiguous predecessor speech run prefixed to a unit whose speech starts at sample 0, so Parakeet TDT does not start mid-utterance (fail: closed)
- `BATCH_PIPELINE_ASR_LEFT_CONTEXT_UNITS` — Process-local predecessor tails kept for ASR left context (default 64, range [0, 1024]; each <=8 s int16, 10 min TTL, keyed by owner, privacy generation and capture stream: source, device and lock from the staged intake context, or the record conversation) (fail: closed)
</file context>
| model = nemo_asr.models.ASRModel.from_pretrained(args.model).cuda().eval() | ||
| pin_parakeet_decoding(model) | ||
| provider = ParakeetWords(model, enforce_eager=True) | ||
| cuda_graphs = provider_config().asr_cuda_graphs |
There was a problem hiding this comment.
P2: Graph-enabled rows include one-time CUDA graph capture in their timings: the only warm-up is a single 30-second batch before the size/context loops, so batch-8/32 and context-2 rows use a new graph during measurement. Warm each row with its actual first batch before starting the timer, or report capture and steady-state timings separately.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/diarizer/batch_pipeline_benchmark.py, line 64:
<comment>Graph-enabled rows include one-time CUDA graph capture in their timings: the only warm-up is a single 30-second batch before the size/context loops, so batch-8/32 and context-2 rows use a new graph during measurement. Warm each row with its actual first batch before starting the timer, or report capture and steady-state timings separately.</comment>
<file context>
@@ -61,8 +61,9 @@ def main():
model = nemo_asr.models.ASRModel.from_pretrained(args.model).cuda().eval()
- pin_parakeet_decoding(model)
- provider = ParakeetWords(model, enforce_eager=True)
+ cuda_graphs = provider_config().asr_cuda_graphs
+ pin_parakeet_decoding(model, cuda_graphs=cuda_graphs)
+ provider = ParakeetWords(model, cuda_graphs=cuda_graphs)
</file context>
| 'torch': torch.__version__, | ||
| 'model': args.model, | ||
| 'cuda_graph_policy': 'persistently_disabled_and_asserted', | ||
| 'cuda_graph_policy': 'enabled_and_asserted' if cuda_graphs else 'persistently_disabled_and_asserted', |
There was a problem hiding this comment.
P3: The benchmark documentation still describes this script as pinning eager decoding, but the new default runs CUDA-graph decoding. Update the qualification instructions and historical note to state the selected mode, or explicitly set BATCH_PIPELINE_ASR_CUDA_GRAPHS_ENABLED=false for the documented eager baseline.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/diarizer/batch_pipeline_benchmark.py, line 76:
<comment>The benchmark documentation still describes this script as pinning eager decoding, but the new default runs CUDA-graph decoding. Update the qualification instructions and historical note to state the selected mode, or explicitly set `BATCH_PIPELINE_ASR_CUDA_GRAPHS_ENABLED=false` for the documented eager baseline.</comment>
<file context>
@@ -72,7 +73,7 @@ def main():
'torch': torch.__version__,
'model': args.model,
- 'cuda_graph_policy': 'persistently_disabled_and_asserted',
+ 'cuda_graph_policy': 'enabled_and_asserted' if cuda_graphs else 'persistently_disabled_and_asserted',
'rows': rows,
'scope': 'ASR only; whole conversation input, internal windows; no persistence',
</file context>
| model = nemo_asr.models.ASRModel.from_pretrained(args.model).cuda().eval() | ||
| pin_parakeet_decoding(model) | ||
| provider = ParakeetWords(model, enforce_eager=True) | ||
| cuda_graphs = provider_config().asr_cuda_graphs |
There was a problem hiding this comment.
P3: The benchmark receipt contradicts itself when the graph kill switch is false: the actual decoder runs eager mode, but each row’s serialized config.asr_cuda_graphs remains True. Populate the row config with the selected cuda_graphs value or omit that field from the per-cell configuration.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At backend/diarizer/batch_pipeline_benchmark.py, line 64:
<comment>The benchmark receipt contradicts itself when the graph kill switch is false: the actual decoder runs eager mode, but each row’s serialized `config.asr_cuda_graphs` remains `True`. Populate the row config with the selected `cuda_graphs` value or omit that field from the per-cell configuration.</comment>
<file context>
@@ -61,8 +61,9 @@ def main():
model = nemo_asr.models.ASRModel.from_pretrained(args.model).cuda().eval()
- pin_parakeet_decoding(model)
- provider = ParakeetWords(model, enforce_eager=True)
+ cuda_graphs = provider_config().asr_cuda_graphs
+ pin_parakeet_decoding(model, cuda_graphs=cuda_graphs)
+ provider = ParakeetWords(model, cuda_graphs=cuda_graphs)
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
Productionizes the GPU-efficiency gains from the batch-worker hill climb (avo-lite
avo/gpueff), as three ship-on kill switches on the dedicated batch GPU worker. Batch at 100% and moving Japanese off Soniox both need the extra GPU headroom.false= off)BATCH_PIPELINE_PREPARE_FLIGHT_ENABLEDworker.prepareFlightEnabledBATCH_PIPELINE_VAD_SEGMENTATION_REUSE_ENABLEDworker.vadSegmentationReuseEnabledBATCH_PIPELINE_ASR_CUDA_GRAPHS_ENABLEDworker.asrCudaGraphsEnabledlane/gpe-cgraph, turned on by the L4 run belowThe switches are read once at startup through
ProviderConfig, like the shared embedding trunk (#24352). They are registered inconfig/feature-flags.yamlasops_killand added tobackend/config/flag_budget.json(count 222) andconfig/deployment-setting-classification.json.Not ported:
admissionstage averages 1.49 s per call; the bench simulates 0.05 s. More calls would cost more GPU-held time than they save.Product invariants affected
none
How it was verified
Dev bench (
gpueff-avo/eval/run.sh): the real worker (load_worker) runs on 248 public-audio units (7055 audio-s) with the prod chart env, 2 lanes and flights of 8. Storage IO is simulated with fixed sleeps. The gate (gate.py) checks no lost speech, no result-class changes, word edit rate ≤1%, speaker agreement ≥0.97 and no words on silent units.L4 (g2-standard-8, engine-pool, real L4, simulated IO):
Every run's gate against the L4 main run passed with word edit rate 0.0, speaker agreement 1.0, 0 lost speech and 0 class changes. That is +61% audio-h/GPU-h on L4 against main. CUDA graphs cut ASR by 26% with identical words. On the full stack the bench objective stays flat, because the L4 bench is then bound by its simulated artifact IO, so that saving shows up as less GPU-busy time.
Caveats (simulated IO): the prepared-flight gain comes from overlapping fetches with the GPU section. Prod
fetch_decodeaverages 0.27 s per unit (2 h aggregate) against ~0.43 s simulated, and prod is not saturated. Expect the prod gain from that lever to be smaller than the bench shows, and largest under batch load. VAD reuse and CUDA graphs remove GPU work outright, so they don't depend on IO.Safety of the early release (t6): the pump cancels its lost-lease watcher on GPU entry and decides refunds (
skipped) before dispatch, so releasing finished preparations after the GPU section can't change lease, refund or publication outcomes. Only done futures are closed, and close is idempotent. The decoded-memory bound is unchanged: 1800 s of preparation plus a 7200 s group, and buffers are freed sooner. If any prepared lease is lost while waiting, the whole prepared prefix is refunded with no attempt charge. That costs some throughput, never correctness.Tests
backend/test.shthroughomi-heavy,BACKEND_UNIT_TEST_FILE_LIST= all 69tests/unit/test_batch_pipeline*.pyfiles plustest_batch_worker_vad_recovery.py,test_batch_unit_policy.pyandtest_flag_budget.py(72 files): all passed on the t5/t6/t8 commit. On the final tree, 71 passed.test_batch_pipeline_admission.py::test_slow_dependency_cannot_grant_after_deadline[seal]failed one wall-clock< 1 sassertion on a loaded host. The file is untouched here, and it passed 50/50 on rerun.test_batch_pipeline_worker_bundle.py).test_batch_pipeline_flight.py).test_batch_pipeline_runtime.py).test_batch_pipeline_chart.py).pytest --noconftest -k pyannote), 9 passed, includingtest_vad_segmentation_reuse_matches_serial_pyannote_exactly.make preflight: see the CI run.UI evidence
UI-Evidence: none -- backend GPU worker only
Failure class (fixes)
Failure-Class: none
🤖 Generated with Claude Code