Skip to content

[CUDA] Runtime physical columns for batched-sequence off-graph KV (ABI 2) - #25324

Open
Gasoonjia wants to merge 10 commits into
pytorch:mainfrom
Gasoonjia:runtime-columns-abi2-pr1
Open

Gasoonjia wants to merge 10 commits into
pytorch:mainfrom
Gasoonjia:runtime-columns-abi2-pr1

Conversation

@Gasoonjia

@Gasoonjia Gasoonjia commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A CUDA batched-sequence off-graph KV cache currently allocates the exported maximum owner width on its first forward. Sequence-info ABI 2 carries the runtime physical pitch independently of sequence IDs, so columns start with one owner plus scratch and grow geometrically on admission. After a 500 ms idle grace period, the Runner can compact underused columns while retaining one spare owner column per live owner to avoid repeated resize and graph recapture across bursts. This changes physical width; per-owner row sizing and Ring headroom remain separate follow-ups.

Review the pool/cache lifecycle first, then executor/Runner integration, lowering/export metadata and tests. Replacement copies retained KV transactionally, invalidates bindings and captured graphs, and releases replaced allocations. Same-width compaction preserves valid mappings without relocation. Replacement temporarily holds old and new allocations; failed admission removes the unpublished owner. Post-commit per-device allocator trimming does not allocate a host snapshot that could interrupt pitch reconciliation.

The backend aborts a failed prepared forward independently of logging, fencing queued work before storage can move again. Abort does not roll back claimed logical rows. Prepare/compaction reject active forwards, and raw execution rejects re-entrant execution or maintenance. Execution, session verbs and maintenance follow the Executor engine-thread contract and use a valid current-thread stream.

ABI 1 remains the default when the optional ABI constant is absent. The cache registry distinguishes fixed and runtime-column kinds. Manifest schema version remains 1; the validated runtime_columns flag selects sequence-info ABI 2. Export is opt-in through --runtime-kv-columns, requires CUDA batched-sequence layout, and excludes DFlash. The existing attention algorithm and Ring addressing are preserved apart from reading the runtime pitch.

Validation

The local Runner/scheduler/prefix-cache/opaque-input C++ suites passed 228 tests. The follow-up passed 6 CI configuration tests, 2 workload tests (8 C++ cases), 10 CUDA workflow tests, formatting and diff checks. An exact-source host reproduction confirms that failed-forward abort remains active with logging disabled. CUDA/Triton execution is unavailable on this Mac; current-head GPU CI and Claude review are pending.

CUDA CI builds the release runtime and ABI-2 native probe with logging disabled. Backend tests cover prepare/rebind/program/commit failure recovery and capture/replay counters across recapture. The allocator test forbids host allocation during post-commit per-device trimming. Native qualification exports identical tiny ABI-1/ABI-2 models and checks logits/argmax through growth, wrapped Ring cloning, sparse compaction, reuse and a 5-to-3-to-5 owner burst. It verifies capture/replay through counters, rejects eager fallback, compares live KV allocations and checks driver-reported allocator memory contraction. Same-width tests assert that warmup state is not reset; the CI wrapper forwards idle maintenance.

Built on upstream main f63b372b702c50535b063c29a843b4adf63beffe. Authored with OpenAI Codex assistance.

Decouple sequence IDs from physical storage columns so the
batched-sequence cache can start with 2 physical columns and grow or
shrink at runtime instead of always allocating max_seqs + 1 columns.

- CudaKVPool gains resize_columns() with declared vs physical column
  counts, transactional replacement copies, and binding invalidation on
  column moves (mirroring grow()).
- CudaBatchedSequenceCache tracks IDs and physical columns separately,
  uploads a 5-wide seq_info whose last field is the runtime pitch, and
  adds ensure_columns()/compact_columns() behind an inflight guard.
- The lowering pass gains manifest version 2 with runtime_columns, the
  SDPA kernel reads the runtime pitch via INFO_WIDTH, and CudaExecutor
  reads get_offgraph_kv_sequence_info_abi (1 or 2) from the program.
- ABI-1 behaviour is unchanged: the new paths are gated on
  CacheConfig::runtime_sequence_columns and manifest version 2.

Tests: extended test_cuda_kv_cache and
test_offgraph_kv_batched_sequence, plus an opt-in Muse-Glimmer
runtime KV probe. The SDPA ring-scan logic from the experiment patch
was intentionally not ported: main's ring handling is kept and only
the runtime-pitch change is included.

This is PR 1 of the runtime-columns series; per-owner capacities
(ABI 3) build on the ID/column split introduced here.
@pytorch-bot

pytorch-bot Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25324

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 2 Pending, 7 Unrelated Failures

As of commit 952486c with merge base f63b372 (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 20, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@Gasoonjia Gasoonjia left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the runtime-physical-columns (ABI 2) change. The core pool mechanics look right, but I have one functional concern about failure recovery and a few smaller items.

What looks good

  • CudaKVPool::resize_columns runs its steps in a sound order: it allocates every replacement, queues the 2D copies, synchronizes, and only then calls invalidate_bindings() and discards the old storage. Every failure path before the commit point rolls back to the old allocations, bindings, allocated_bytes_ and columns_. FailedColumnGrowthKeepsOldStorageAndBindingsUsable covers this.
  • committed is reported separately from errors after the commit point (the trim sync), and ensure_columns poisons the cache in that case. That is the right split.
  • invalidate_bindings() is shared with grow(). It reaches every handle (Replay -> recapture, Warmup -> at least one eager step), and ContractionInvalidatesEveryGraphAndBinding checks it.
  • An ABI mismatch between the program and the runtime fails at load: check_compiled compares the side-buffer shape ([S,4] vs [S,5]), and FixedInfoABIIsRejected covers it.
  • The pitch handling is consistent across C++, the Python writes and Triton. Every seq_info row, padded segments included, carries columns_, so the decode kernel can safely read only row 0 and the prefill kernel can read it per segment. The dummy column moves to columns_ - 1 with the pitch.

Main concern

  1. A failed execute leaves inflight_ set, and ABI-2 caches stay unusable until clear() (inline). In ABI-1 a failed forward does not block later steps.

Other items (inline)
2. Columns grow one at a time. Each admission past the current width runs a device-wide sync, a full KV reallocation and copy, and an allocator trim, all inside prepare_step. Peak memory during a resize is old + new storage, so near max_seqs this can OOM where ABI-1's upfront allocation would have fit.
3. Changing compile_wrapper_opt_level from O0 to O2 in export_solo.py is unrelated and affects ABI-1 exports too.
4. ABI 2 combined with DFlash is rejected only by the export script. The runtime (CudaExecutor::create / the cache factory) still accepts it.
5. The seq_rm guard against step_seq_ids_ applies only in runtime mode. An abandoned declare_step then blocks removal, and ABI-1 still has the latent seqs_.at() throw.
6. The ABI-1 seq_clone failure path changed: a device-wide sync and possible poisoning. So ABI-1 is not strictly untouched (minor).

Nits / docs

  • make_cuda_batched_sequence_kv_cache's comment still says the step buffers are [segments, 4] (cuda_kv_cache.cpp, ~L1576).
  • The sdpa_varlen docstring for seq_info (~L410) still says Int32[S, 4].
  • In LowerOffGraphKVPass, self._manifest.get("runtime_columns", False) is redundant now that parse_offgraph_kv_manifest calls setdefault. self._manifest["runtime_columns"] is enough.
  • When columns grow with an identity mapping, one cudaMemcpy2DAsync per layer of width slot * old_columns could replace the per-column copies (layers x seqs x 2 calls).

Test gaps

  • No test covers a step that is prepared and never committed (execute failure) in ABI-2.
  • No executor-level test of get_offgraph_kv_sequence_info_abi validation (ABI 3; ABI 2 on the cell layout; ABI 2 with a draft method).
  • The Python test runs batched_sequence_step eagerly. No AOTI-compiled ABI-2 program runs in CI; the Muse-Glimmer probe covers that only manually. The probe is ~290 lines of opt-in diagnostics that CI never builds. Consider moving it to a follow-up or a dev-tools location.

This review was generated with AI assistance (Claude Code).

Comment thread backends/cuda/runtime/cuda_kv_cache.cpp Resolved
Comment thread backends/cuda/runtime/cuda_kv_cache.cpp Resolved
Comment thread backends/cuda/runtime/cuda_kv_pool.cpp Resolved
Comment thread backends/cuda/runtime/cuda_kv_cache.cpp Resolved
Comment thread backends/cuda/runtime/cuda_kv_cache.cpp Resolved
- Revert the unrelated O0 -> O2 change in export_solo.py.
- Recover a prepared-but-uncommitted ABI-2 step in the next
  prepare_step (fence all streams, then clear inflight_) instead of
  wedging the cache until clear() drops every sequence; add a
  regression test.
- Grow physical columns geometrically and trim the allocator only on
  the compact path, avoiding one full KV copy per admission.
- Reject runtime columns + DFlash in CudaExecutor and in the
  batched-sequence cache factory.
- Gate the seq_clone fence/poison on runtime columns so ABI-1 clone
  failures are unchanged; refuse seq_rm for a declared sequence in
  both ABIs.
- Fix the lintrunner CMake formatting failure and stale seq_info
  width comments.
- Pool resize comment and lowering .get() redundancy left as-is:
  resize_columns is only called from ensure_columns, and the .get()
  keeps LowerOffGraphKVPass usable with manifests not built by the
  parser.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Thanks for the review — addressed in b649b06 (see replies to each inline comment). The one CI failure on the first push was lintrunner CMake formatting only, also fixed in this update. Remaining known gaps, not addressed here: per-column copy merging in resize_columns, and a compiled ABI-2 run in CI (the opt-in probe is not built by CI); both are follow-up work, not blockers for the ABI-2 plumbing itself.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 53m 50s —— View job


PR Review: #25324

This PR adds a runtime-pitch seq_info ABI ([S, 5], ABI 2) so the CUDA batched-sequence KV cache can allocate fewer physical columns than the compiled maximum, but ensure_columns widens the storage on every step until it reaches that maximum (by trace, two of the PR's own C++ tests fail), and the new seq_rm refusal leaks sequences in both ABIs, so I'm requesting changes.

Detailed review

Summary

The PR lets the batched-sequence cache start at 2 physical columns, grow as sequences are admitted, and shrink with compact_columns. The kernels get the pitch from seq_info[:, 4].

Two problems block it:

  • The growth policy added in b649b06 resizes on every prepare_step and seq_clone until the width reaches max_seqs + 1, which cancels the memory saving.
  • seq_rm now refuses in states that ordinary forward failures leave behind, and every caller ignores the refusal.

"Test CUDA Builds" (run 44647068649) was still queued when I wrote this, so the test failures below come from tracing the code, not from CI.

Backward Compatibility

  • seq_rm refusals leak sequences (major). backends/cuda/runtime/cuda_kv_cache.cpp:998-1009 now returns false while inflight_ is set or while the id is in step_seq_ids_, in both ABIs.
    • Callers ignore the result. CudaExecutor::close_session calls SessionTable::close (extension/llm/batching/util/session_table.cpp:110-111), which erases the session anyway. SequenceGuard (extension/llm/batching/executor_utils.h:71-75) never retries.
    • So a refusal keeps the sequence, its id and its column until clear(). Once max_seqs ids have leaked, seq_new fails permanently.
    • ABI 1 (existing programs): when prepare_step fails after declare_step, step_seq_ids_ is never cleared (1171). Example failures: an OOM during row growth in pool_.prepare (1121), a failed upload (1154-1157), or the claim check (1164). Closing those sessions then leaks them. On the base branch this seq_rm succeeded.
    • ABI 2: after any failed forward, inflight_ stays set until the next prepare_step. The usual response, closing the failed sessions, leaks every one of them.
    • Both cases contradict the BatchControl contract (extension/llm/cache/cache.h:128-130: "False = an unknown sequence").
    • compact_columns (1226-1230) has the same guards, so memory can't be reclaimed after a failure until another step runs.
    • Suggested fix:
      • Drop the inflight_ check. It isn't needed: prepare_step fences every stream before reusing storage (1059-1071), and seq_clone and compact_columns already refuse while a step is in flight.
      • For a declared id, clear step_seq_ids_ before erasing it. A following prepare_step then returns InvalidArgument (1081-1087) instead of throwing from seqs_.at.
      • Have compact_columns fence and recover the same way prepare_step does. Its step_seq_ids_.empty() guard isn't needed for safety, because prepare_step resolves columns at 1141.
    • Fix this →

API Design

  • seq_new admits a sequence it can't place. When no physical column is free, seq_new (cuda_kv_cache.cpp:936-947) admits the sequence with column -1. Once the Code Quality fix below lands, this happens every time the current columns are full. The column is only allocated by the next prepare_step's ensure_columns call (1119; the loop at 1327-1341), even when that step doesn't include the new sequence.
    • If that resize fails, every prepare_step fails until the new sequence is removed, including steps whose sequences already have columns.
    • This breaks the admission-gate contract (cache.h:104-108: "a step that passes has room for its tokens"). declare_step (899-930) never looks at columns, so it passes and the failure shows up as a forward error.
    • Fix: allocate the column in seq_new as seq_clone does (977-981), return nullopt on failure, and add "insufficient resources" to the seq_new doc (cache.h:114-117), as the seq_clone doc already does.
    • Fix this →
  • The ABI is declared in two places that must agree.
    • The runtime selects ABI 2 only from the constant method "get_offgraph_kv_sequence_info_abi". It is a string literal at backends/cuda/batching/cuda_executor.cpp:375, read with value_or(1).
    • Only examples/models/muse-glimmer/export/export_solo.py:696 emits that method, also as a literal.
    • The pass picks the seq_info width from the manifest's version-2 runtime_columns (backends/cuda/passes/lower_offgraph_kv.py:98-107).
    • So if any other exporter lowers with runtime_columns, the program is configured as ABI 1 and rejected in note_handle. The pool expects [S, 4] (cuda_kv_cache.cpp:847, checked at cuda_kv_pool.cpp:722).
    • Fix: define the name once, next to kMaxSeqsMethod (cuda_executor.h:53), and have one helper in the pass produce both the manifest fields and the constant-method entry.
    • Fix this →
  • A CUDA-only switch sits in the backend-neutral config. CacheConfig::runtime_sequence_columns (extension/llm/cache/cache.h:181-182) only affects the CUDA batched-sequence cache.
    • Only make_cuda_batched_sequence_kv_cache reads it. make_cuda_sequence_kv_cache (cuda_kv_cache.cpp:1526), make_cuda_cell_kv_cache (1556) and every other backend's cache silently ignore true.
    • Fix: reject it in those factories, or make it a parameter of make_cuda_batched_sequence_kv_cache.
    • Fix this →

Infrastructure

  • The probe is never built or run. examples/models/muse-glimmer/CMakeLists.txt:147-170 builds it only under MUSE_GLIMMER_BUILD_RUNTIME_KV_PROBE, which defaults to OFF, and nothing under .ci/ or .github/ turns it on.
    • Yet the RuntimeColumnPitchTest docstring (backends/cuda/tests/test_offgraph_kv_batched_sequence.py:268-270) names "the C++ cache and model probe" as the tests for the smaller allocations.
    • Either build and run the probe in the existing muse-glimmer CUDA CI job, or drop it and that part of the docstring.

Testing

  • Nothing checks that the width stays put or that growth is geometric. GrowthAndSparseIdCompactionPreserveAllKV checks widths only at 1, 2, 4, 8 and 16 live sessions (test_cuda_kv_cache.cpp:2829). At those counts, growing to exactly live + 1 and doubling the usable width give the same widths, so the geometric growth that b649b06 adds is never exercised.
    • Add a test that runs repeated steps over the same sequences and checks that physical_columns and column_resize_count don't change, plus one admission count that isn't a power of two. The Code Quality fix link includes this.
  • Other missing coverage:
    • CudaKVPoolTest.FailedColumnGrowthKeepsOldStorageAndBindingsUsable (test_cuda_kv_cache.cpp:3062) only runs with trim_unused=true. The growth path uses false (cuda_kv_cache.cpp:1353).
    • No test calls seq_rm or seq_clone between a failed step and the next prepare_step, or calls seq_rm on a declared sequence under ABI 1.
    • The executor's ABI checks (cuda_executor.cpp:372-387) have no test: ABI 2 without the sequence layout, ABI 2 with DFlash, and unknown values.
    • RuntimeColumnPitchTest never runs the multi-split path with a 5-wide seq_info. rows is at most 64 (test_offgraph_kv_batched_sequence.py:276), so _num_splits (sdpa_varlen.py:385-388) always returns 1. The "split" path named in the comment at test_offgraph_kv_batched_sequence.py:329 is the decode kernel with a single split.
    • test_lowered_program_matches_the_neutral_op (test_offgraph_kv_batched_sequence.py:477) only lowers the version-1 (4-wide) ABI, so no test checks the numerics of a lowered ABI-2 program.
    • Fix this →

Performance

  • Column resizes copy more than they need to. In backends/cuda/runtime/cuda_kv_pool.cpp:529-564, resize_columns allocates and copies every allocated row of each retained column (logical_rows, lines 530 and 544). grow() copies only live_rows (402-403).
    • With a large initial_capacity, or after the rows have grown, a column resize copies mostly empty rows.
    • When one prepare_step needs both a column resize (cuda_kv_cache.cpp:1119) and row growth (1121-1124), the KV cache is copied twice, each time with old and new storage allocated at once.
    • Fix: pass the live row count, as grow() does. Consider a single combined resize for steps that need both.
    • Fix this → (this link covers the live-rows part only)

Code Quality

  • Blocker: ensure_columns grows on every call. In backends/cuda/runtime/cuda_kv_cache.cpp:1306-1309, the non-compact target never keeps the current width.
    • grown = max(live + 1, 2 * (columns_ - 1) + 1) is always greater than columns_ once columns_ >= 2. So target != columns_ (1342) holds until the width reaches max_seqs_ + 1.
    • Every prepare_step (1118-1120, unconditional under runtime columns) and seq_clone (978) therefore resizes, however many sequences are live.
    • Each resize does a device-wide sync, copies the full KV cache, and calls invalidate_bindings() (cuda_kv_pool.cpp:569), which sends every CUDA graph back to warmup.
    • With one live sequence and max_seqs = 21, the first five steps widen the storage 2 → 3 → 5 → 9 → 17 → 22, and the first step after compact_columns starts the climb again. The memory saving is gone after about log2(max_seqs) steps, and each of those steps pays for a copy and a graph recapture.
    • By trace, two of the PR's own tests fail:
      • GrowthAndSparseIdCompactionPreserveAllKV:
        • The first one-session step widens to 3 columns, but 2845 expects 2 (sessions + 1).
        • before has 22 columns, but 2889 assumes 17.
        • The step after compaction widens from 2 to 3 columns, so the scratch column checked at 2905 is 2 instead of 1, and the pitch checked at 2917 is 3 instead of 2.
      • CloneAndContractionCopyWrappedRingHistory: two single-sequence steps reach 5 columns and the clone reaches 9, but 2948 expects 3.
    • Fix: keep columns_ when live + 1 <= columns_. Otherwise grow to min(max_seqs_ + 1, max(live + 1, 2 * (columns_ - 1) + 1)). With that change, both tests pass by trace.
    • The comment at 1301 ("doubling the live columns") should then say that the usable width doubles.
    • Fix this →

Specific Comments

  • cuda_kv_cache.cpp:802-819: the class comment still describes fixed columns and a 4-field seq_info row.
  • cuda_kv_cache.cpp:1064-1065: the comment says "treat the step as aborted", but the step is kept. Its rows stay claimed (1161-1170), and UncommittedStepRecoversOnNextPrepare checks for pos == 2.
  • cuda_kv_cache.cpp:1518-1519: the poisoned_ comment says the flag is set when a clear() fence fails, but it is now also set at 990, 1067 and 1362.
  • cuda_kv_cache.cpp:1503, 1507: the mutable columns_ and inflight_ sit in the middle of the const members.
  • cuda_kv_cache.cpp:1278: free_sequence_id's loop variable is named column, which is now misleading because ids are no longer columns under ABI 2.
  • cuda_kv_cache.cpp:1312-1318: relocate is dead code. It is only set when compact is true, and compact alone already satisfies the condition at 1342.
  • cuda_kv_cache.h:33-35: the growth_count doc says it identifies the addresses a program was bound to, but column resizes now move the storage without incrementing it. The new fields at 37-40 are undocumented.
  • cuda_kv_cache.h:92: the compact_columns(int) parameter is unnamed.
  • cuda_executor.h:230: compact_kv_columns(int spare_columns = 1) has no doc saying what spare_columns counts or when it may be called.
  • cuda_kv_pool.h:88-90: the doc says the resize "commits only after every replacement copy passes", but with trim_unused it can return an error after committing (cuda_kv_pool.cpp:582-588). Document *committed.
  • lower_offgraph_kv.py:314-315: the slot formula r * columns + c is wrong under runtime_columns, where the pitch is seq_info[0, 4] (329). Line 325 says "ABI-5" where it means the 5-wide ABI 2.
  • sdpa_varlen.py:408-409: with a 5-wide seq_info, key.stride(0) is ignored and rows are addressed at seq_info[:, 4] * stride(1) (94-97, 254-257), but the doc still says only the head dimension must be contiguous.

Fix the Specific Comments →

Recommendation

Request Changes.

  • The growth policy (Code Quality) cancels the PR's memory saving, and by trace it fails two of the PR's own tests.
  • The seq_rm refusal (Backward Compatibility) also leaks sequences in existing ABI-1 programs.
  • The API Design items should be settled before the ABI-2 contract ships.

Review generated by Claude Code (AI) using the /pr-review skill.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude review effort=max

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

PR Review: #25324

  • Gather context (PR metadata, diff against main, all 18 files)
  • Run the independent reviews
  • Consolidate and deduplicate the findings
  • Fact-check each finding against the code
  • Re-check line numbers for the fix links (all confirmed against HEAD)
  • Check how callers handle seq_new and seq_clone refusals after a failed forward
  • Write up and post the final review

What the caller check found:

  • ServingRuntime rejects a refused open with CapacityExceeded (extension/llm/serving/serving_runtime.cpp:1275-1287).
  • A refused clone makes clone_prefix fall back to a full prefill (serving_runtime.cpp:860-904).
  • The refusal only lifts at the next prepare_step. While idle keyed sessions or retained snapshots fill the current width, new sessions can keep being rejected.

The draft still leans toward Request Changes.

View job run

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review effort=max

Please review the current PR head (47a2cd7) and post the final review. The previous run (44797892208) failed before publishing its final result.

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 59m 4s —— View job


PR Review: #25324

Adds an opt-in ABI 2 that lets the CUDA batched-sequence KV cache grow and compact its physical column width at runtime; Request Changes, because compaction and step recovery treat a flag that every normal forward sets as proof of an abandoned step, and nothing in serving ever compacts, so the memory saving is gone after the first burst to peak concurrency.

Detailed review

Summary

Adds manifest version 2 / runtime_columns to the off-graph KV lowering (a 5-wide seq_info whose field 4 is the physical pitch), CudaKVPool::resize_columns, geometric column growth and compact_columns() in CudaBatchedSequenceCache, ABI selection in CudaExecutor, and a native probe plus CI job that checks ABI 2 against ABI 1. Request Changes: the in-flight step state cannot tell a live forward from an abandoned one, and ABI 2 has no shrink path outside the probe. The Backward Compatibility and API Design items should be settled before exported programs depend on ABI 2. examples/models/muse-glimmer/tests/runtime_kv_columns_probe.cpp was only lightly reviewed.

Thread Safety

  • inflight_ is true during every forward, yet compact_columns and prepare_step treat it as an abandoned step (backends/cuda/runtime/cuda_kv_cache.cpp:1194).
    • Every successful prepare_step sets it and only commit_step (1208) clears it. While it is set, seq_new (943-944) refuses to grow and seq_clone (971-972) refuses outright. But compact_columns (1256-1262) and prepare_step (1080-1093) fence all streams, clear it and carry on, as the comment at 1027-1028 describes. The fence only waits for work that is already enqueued.
    • The mutex contract at 1535-1536 lets the runner's verbs run on any thread. Suppose a compaction lands after prepare_step (cuda_backend.cpp:829-830) but before the step's kernels are enqueued (839-925). It reallocates and frees the storage bound at 831, while the step's uploaded seq_cols/seq_info still carry the old mapping and pitch. The forward then writes freed blocks, and invalidate_bindings (cuda_kv_pool.cpp:441-461) races the unlocked read of cuda_graph_state.phase at cuda_backend.cpp:839. commit_step still returns Ok because it never checks inflight_.
    • compact_kv_columns is documented as "serialized with execute" (cuda_executor.h:240). The only enforcement is raw_executing_ (cuda_executor.h:316), a plain bool written on the engine thread, so the check at cuda_executor.cpp:1116-1117 is itself a data race when it runs on another thread.
    • After a real failure the flag stays set until the next prepare, compact or clear(). In that window every seq_clone, and any seq_new that needs a new column, returns nullopt, which SessionTable reports as a full table.
    • Make abandonment explicit. Add an abort_step() next to prepare_step/commit_step, and call it from a scope guard in cuda_backend.cpp on every exit between prepare_step and commit_offgraph_kv_step (896, 926). Then have compact_columns and prepare_step return InvalidState while inflight_ is set. Update UncommittedStepRecoversOnNextPrepare and FailedStepAllowsCloseAndFencesBeforeReuse to go through the abort. Fix this →

Backward Compatibility

  • seq_rm now refuses a pending DFlash owner, which only changes the existing ABI-1 path (backends/cuda/runtime/cuda_kv_cache.cpp:1019-1022). The check is not gated on runtime_columns_, and runtime columns are rejected with DFlash (1637), so it only affects ABI-1 DFlash draft caches. A removal that used to succeed now returns false and keeps the sequence and its id until clear(). extension/llm/cache/cache.h:129-131 still says "False = an unknown sequence", and MuseGlimmerCudaDFlashExecutor::close_session (cuda_dflash_executor.cpp:828-831) ignores the result. Drop the check from this PR, or document the new failure in the BatchControl contract and handle it in the callers. Fix this →

API Design

  • ABI 2 is built outside the cache factory (backends/cuda/batching/cuda_executor.cpp:551-560). The executor calls make_cuda_batched_sequence_kv_cache(geometry, cfg, true) directly, while the registered kBatchedSequence builder (cuda_kv_cache.cpp:1706-1709) always builds the fixed-column cache. A runner that asks the factory for (backend_id, kind), as the registry comment at 1695-1698 describes, therefore gets an ABI-1 cache for an ABI-2 program. The layout is chosen by a defaulted trailing bool (cuda_kv_cache.h:229). Register the runtime-column cache under its own kind (or carry the choice in CacheConfig) and build through CacheFactory::global().build in both branches. Fix this →
  • The manifest schema version doubles as the runtime-columns flag (backends/cuda/passes/lower_offgraph_kv.py:88-108). Version 2 is accepted exactly when runtime_columns is true on a batched-sequence layout, so the two fields encode one bit. offgraph_kv_constant_methods (160) then exports manifest["version"] as the runtime's seq_info ABI. Any later manifest schema change would have to become seq-info ABI 3, and no other layout can ever use version 2. Export the ABI from runtime_columns and keep version for the schema (or drop the redundant field). Update export_solo.py:344-364 and the tests to match. Fix this →

Testing

  • The lowered runtime-pitch test cannot detect a dropped pitch (backends/cuda/tests/test_offgraph_kv_batched_sequence.py:565-572). It fills seq_info[:, 4] with MAX_SEQS + 1, which is the declared column count. test_lowered_runtime_pitch_matches_the_neutral_op therefore still passes if lower_offgraph_kv.py:788 stops passing runtime_columns and the step falls back to the declared width (342). Only the slower end-to-end CUDA probe would catch that. Use a pitch below the declared width (live sequences + 1, with the scratch column at pitch - 1). Then fill the storage past rows * pitch with NaN and assert it stays NaN, as RuntimeColumnPitchTest (268-365) does for the eager op. Fix this →

Performance

  • Nothing in serving shrinks the columns back (backends/cuda/runtime/cuda_kv_cache.cpp:1329-1333). compact_columns() is the only shrink path, and its only non-test caller is the probe (runtime_kv_columns_probe.cpp:272). The batching runner and worker never call compact_kv_columns. Growth passes trim_unused=false (1384-1392), and the CudaAllocator pool keeps freed blocks reserved (release threshold UINT64_MAX). After one burst to max_seqs, a serving process keeps the widest storage live (9 columns at max_seqs = 8). It also keeps whichever of the replaced 2-, 3- and 5-column blocks the pool has not reused. For any workload that ever reaches peak concurrency, ABI 2 then saves nothing over ABI 1 and can reserve more. Compact from the serving loop at idle boundaries with hysteresis, and trim after a committed growth. Otherwise, state in the export help text (export_solo.py:1238-1241) and the compact_kv_columns doc that ABI 2 does not shrink in serving yet. Fix this →
  • Compaction is not stable, so a call with nothing to gain still relocates everything (backends/cuda/runtime/cuda_kv_cache.cpp:1344-1348). Columns are reassigned in id order. Suppose id 0 is removed and a compaction to width 2 leaves id 1 in column 0. seq_new then returns id 0 and grows to width 3 with it in column 1. A later compact_columns(0) keeps width 3 but swaps the two owners, so relocate is set (cuda_kv_pool.cpp:491). The call pays a full reallocation and copy and invalidates every graph and binding, with no memory saved. Keep owners already below target - 1 in place and move only the rest, so such a call takes the early return at cuda_kv_pool.cpp:516. Add a test that the second compaction leaves column_resize_count unchanged. Fix this →

Code Quality

  • Single-use helper in a public header (backends/cuda/batching/cuda_executor.h:57-62). detail::valid_sequence_info_abi is a one-liner used once, at cuda_executor.cpp:375-381. It exists only so SequenceInfoABIRejectsUnsupportedCombinations (test/test_cuda_executor.cpp:42-51) can call it, which goes against CLAUDE.md's rule on trivial single-use helpers. Inline the condition and drop the truth-table test. Fix this →

Specific Comments

  • backends/cuda/runtime/cuda_kv_cache.cpp:1004-1006 - the comment narrates history ("previously just returned nullopt"); state the invariant instead.
  • backends/cuda/runtime/cuda_kv_cache.h:208-209 - stale: still says the program declares seq_info as [segments, 4]; ABI 2 declares 5 columns.
  • backends/cuda/batching/cuda_executor.h:54-55 - kSequenceInfoABIMethod is read with value_or(1), but unlike kMinSelectedRowsMethod (64-66) it has no "Optional: ... Defaults to 1." comment.
  • examples/models/muse-glimmer/export/export_solo.py:258 - MAX_KV_SEQS_METHOD is now unused; its only use was replaced by offgraph_kv_constant_methods at 695.

Fix this →

Recommendation

Request Changes

Before compaction is safe to expose, inflight_ has to tell a live forward from an abandoned one. ABI 2 also needs either a serving-side shrink path or documentation that it is probe-only. Settle the Backward Compatibility and API Design items before exported programs depend on manifest version 2 / ABI 2. The Testing, compaction-stability and Code Quality items are minor.

Review generated by Claude Code (AI) using the /pr-review skill.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review effort=max

Please review current head 89714b77fe94670571d25d356c9063ccf91dced2 and post the final recommendation. This revision includes upstream main c1ab845be19a341330ad3f0ae2883ba6655db168 and updated runtime-column lifecycle, serving integration, and tests. Local validation: 228 actual Runner/scheduler/prefix/opaque C++ tests and 8 CI/workload checks passed. GPU CI for this head is pending.

Review requested by OpenAI Codex on behalf of the PR author.

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 42m 25s —— View job


PR Review: #25324

Recommendation: Approve. There are no blockers at 89714b7. I found one undefined threading contract (execution_mutex_), a compact_columns that can grow storage, a manifest version field that no longer means anything, loose GPU-probe assertions, and resize and idle-policy costs. GPU CI for this head is still pending, and this approval assumes it passes.

Detailed review

Summary

The PR adds ABI 2 for the CUDA batched-sequence off-graph KV cache. A 5-wide seq_info carries the runtime physical column pitch. Owner columns start at one, grow geometrically when sessions are admitted, and compact while the Runner is idle through a new Executor::on_idle() hook.

None of the findings blocks the PR. I only skimmed examples/models/muse-glimmer/tests/runtime_kv_columns_probe.cpp.

Thread Safety

  • execution_mutex_ fails callers instead of serializing them (backends/cuda/batching/cuda_executor.cpp:697-705, 1102-1110, 1116-1122).
    • execute_raw, compact_kv_columns and on_idle all use try-lock, so whichever call arrives second fails instead of waiting.
    • The header calls compact_kv_columns "an idle boundary serialized with execute" (cuda_executor.h:236). A caller who trusts that and compacts from a maintenance thread while a Runner serves makes the colliding execute() return false. The Runner then condemns the batch and poisons its sessions (extension/llm/batching/runner.cpp:1157-1169). Today this is safe only because every caller (the engine thread, the probe) is single-threaded.
    • last_cache_activity_ is written without the mutex in open_session/close_session/clone (654-665), and on_idle reads it.
    • A re-entrant execute_raw from inside consume now try-locks a std::mutex the thread already owns, which is undefined behavior. The removed raw_executing_ flag rejected this deterministically.
    • Fix: pick one contract and document it. The smallest option matches Executor::on_idle's documented engine-thread model: drop the mutex, restore the re-entrancy flag, and make compact_kv_columns/on_idle refuse while it is set. The alternative is to block in execute_raw/compact_kv_columns, try-lock only in on_idle, and make the clock atomic. Fix this →

API Design

  • compact_columns(spare) can grow the storage (backends/cuda/runtime/cuda_kv_cache.cpp:1338-1339).
    • The target is min(max_seqs, max(1, live + spare)) + 1, and nothing clamps it to the current columns_.
    • For example, compact_kv_columns(4) with 2 live sequences at width 3 resizes to 7 columns. That costs a full KV copy and invalidates every binding and graph.
    • The docs (cuda_kv_cache.h:99-100, cuda_executor.h:236-237) promise compaction. on_idle is not affected, because its 2 * live + 1 < physical guard keeps the target below the current width.
    • Fix: clamp the target to columns_ when compacting, or rename the method and document it as a resize. Fix this →
  • The manifest version no longer means anything (backends/cuda/passes/lower_offgraph_kv.py:88-89, 99-103, 155).
    • Version 2 is accepted, but only runtime_columns selects the ABI. {"version": 2, "runtime_columns": false} lowers as ABI 1, and {"version": 1, "runtime_columns": true} lowers as ABI 2. backends/cuda/tests/test_offgraph_kv_batched_sequence.py:494-503 asserts both cases.
    • Nothing emits version 2. batched_sequence_manifest (examples/models/muse-glimmer/export/export_solo.py:341-362) adds runtime_columns but keeps the base manifest's "version": 1 (examples/models/muse-glimmer/source_transformations/cuda.py:457, 733).
    • Fix: either keep version 1, since the key is optional and additive, or require version == 2 exactly when runtime_columns is set.
    • The same validation also accepts runtime_columns with context_window (a DFlash draft), which the runtime cannot serve. CudaExecutor::create rejects ABI 2 with a DFlash block, and the draft cache is always kBatchedSequence (examples/models/muse-glimmer/runtime/cuda_dflash_executor.cpp:692-695). Only export_solo.py:477 stops this today, so the pass should reject it too. Fix this →

Testing

  • The graph-qualification check cannot catch recapture churn (examples/models/muse-glimmer/tests/check_runtime_kv_columns.py:143-151).
    • It counts forward_2 captures with >=. A regression that recaptures on every step still passes, for example a spurious invalidate_bindings() on the new resize path.
    • Replay is never observed directly, even though the assertion message claims it is.
    • The workload is deterministic: 10 forward_2 steps with exactly 2 captures for ABI 2 (the first, and one after compaction) and 1 for ABI 1. Asserting == would prove that the remaining steps replayed. Fix this →
  • ABI 2 is never compared with ABI 1 on memory (check_runtime_kv_columns.py:180, 197).
    • The ABI-1 responses are discarded (fixed, _ = run(...)), so the memory assertion only compares ABI 2 with itself across compaction.
    • The module docstring promises to "Compare actual ABI-2 CUDA allocations and logits with the fixed-column ABI". Using less KV memory than ABI 1 is the point of the PR, but nothing asserts it. Fix this →
  • The retention tests cannot detect mapping errors (backends/cuda/runtime/test/test_cuda_kv_cache.cpp).
    • GrowthAndSparseIdCompactionPreserveAllKV seeds distinct values per column, but after each growth it checks only column 0 (2853-2858, 2876-2884). A wrong destination offset or swap for columns 1..N-1 would pass.
    • CloneAndContractionCopyWrappedRingHistory fills both columns with a uniform 1234 + layer (2960). The test is named for the wrapped ring copy, but a wrong row order or wrap offset in that copy would not show.
    • Fix: use seed()-style values that depend on row and column, and check every retained column. Fix this →

Performance

  • Column growth always trims the shared allocator and never falls back to a smaller width.
    • ensure_columns always passes trim_unused=true (backends/cuda/runtime/cuda_kv_cache.cpp:1386), so the parameter is effectively constant in production.
    • As a result, every growth on admission pays a device-wide sync plus two CudaAllocator::release_cached_memory calls (backends/cuda/runtime/cuda_kv_pool.cpp:517-519, 588-594). Those calls also release unused blocks cached by other allocator users.
    • Growth only tries the doubled width, which needs about 3× the old storage at its transient peak (old plus new). If that allocation fails, seq_new returns nullopt, which reads as a capacity refusal (extension/llm/cache/cache.h:115), even when live + 1 columns would fit.
    • Fix: trim and retry only after an allocation failure, then fall back to live + 1. Keep the unconditional trim for compaction. Fix this →
  • The idle compaction policy is hard-coded and thrashes under bursty load (backends/cuda/batching/cuda_executor.cpp:1112-1136).
    • on_idle fixes three values: a 500 ms grace period, compaction when 2 * live + 1 < physical, and one spare column.
    • Once storage is more than about twice the live count, any idle gap over 500 ms shrinks it to live + 2 columns (2 when no session remains). The next burst of N sessions then pays about log2(N) resizes, each with a device sync, a KV copy, allocator trims, and recapture of every graph in use.
    • The clock is reset before the call (1128), so a compaction that keeps failing retries every 500 ms, each time with a sync, a trim and an Error log.
    • The fixed grace period also forces the probe to sleep 550 ms (runtime_kv_columns_probe.cpp:277). The threshold and grace logic have no unit test beyond that single call.
    • Fix: move the grace period, threshold and spare count into CudaExecutorConfig, so they can be tuned and unit-tested. Fix this →

Specific Comments

  • backends/cuda/runtime/cuda_kv_cache.cpp:809-810: the line wrap is broken. The line ends after "The last column is scratch," and the sentence continues on the next line. Line 818 repeats the scratch-column note.
  • backends/cuda/runtime/cuda_kv_cache.cpp:1328: "doubling the live columns" reads as 2 * live owners. But live (1327) already counts the sequence being admitted (seq_new emplaces it first, at 949), and the code doubles the previous owner width 2 * (columns_ - 1), which is 2 * (live - 1) at growth time.
  • backends/cuda/runtime/cuda_kv_cache.cpp:1024: "verbs run between forwards" contradicts seq_new (943-944), which admits a sequence during an in-flight ABI-2 forward when a free column exists.
  • backends/cuda/runtime/cuda_kv_cache.cpp:947-948, 988-989: free_physical_column() is called twice, and the -1 sentinel only works because ensure_columns assigns a column later. Store the optional once.
  • backends/cuda/runtime/cuda_kv_cache.cpp:1124-1126: this ensure_columns call cannot resize, because admission already sized the columns. Writing it as an invariant check would state what it guards.
  • backends/cuda/runtime/cuda_backend.cpp:794-813: kv_prepared is never reset after commit_step, so the guard calls abort_step() after every successful forward. That call is a no-op only because abort_step checks inflight_.
  • backends/cuda/runtime/cuda_kv_pool.h:89: "Dummy data" should be "the scratch column", the term used everywhere else.
  • extension/llm/cache/cache.h:115-117: the rewrap leaves "Ids may also be chosen by the caller;" on a line by itself.
  • backends/cuda/passes/lower_offgraph_kv.py:333: the docstring says seq_info[:, 4], but line 337 reads seq_info[0, 4]. The prefill kernel reads the pitch per segment (sdpa_varlen.py:94-97), while the decode kernel (254-257) and this write read row 0. Reading row 0 everywhere would enforce the "shared by every segment" contract by construction.
  • backends/cuda/passes/lower_offgraph_kv.py:154: "get_offgraph_kv_max_seqs" is a string literal next to the named OFFGRAPH_KV_SEQUENCE_INFO_ABI_METHOD. test_offgraph_kv_batched_sequence.py:553 repeats the literal.
  • examples/models/muse-glimmer/export/export_solo.py:256-257: this comment described the removed MAX_KV_SEQS_METHOD. It is now orphaned and reads as if it described DRAFT_MAX_KV_SEQS_METHOD.
  • backends/cuda/batching/cuda_executor.cpp:1130: the NotSupported filter is dead code. ABI 2 always builds a kBatchedSequenceRuntime cache (551-552), and that cache only returns NotSupported from compact_columns when runtime columns are off (cuda_kv_cache.cpp:1246-1249).
  • examples/models/muse-glimmer/tests/check_runtime_kv_columns.py:95, 119: mkdir() is called without exist_ok, so a rerun with the same --output-dir fails.
  • examples/models/muse-glimmer/tests/runtime_kv_columns_probe.cpp:95: live_KV_bytes is the only mixed-case key among snake_case keys.
  • examples/models/muse-glimmer/CMakeLists.txt:151-152: the probe guard checks only EXECUTORCH_BUILD_CUDA, not TARGET cuda_batching as the batching block does (285-294). An install without the LLM extensions therefore fails at link time instead of with the clear error message.
  • .github/workflows/cuda.yml:699-710: the job now copies the muse-glimmer-batching-cuda Makefile recipe by hand (presets, -DMUSE_GLIMMER_BUILD_BATCHING=ON, runner/worker targets) and adds logging. Later changes to the Makefile target will not reach this job.

Fix this →

Recommendation

Approve

Nothing blocks approval. The two items most worth fixing before merge are the threading contract under Thread Safety and the >= capture check under Testing. The new runtime-column ctests and the probe run only in GPU CI, which was still pending for 89714b7. This approval assumes those jobs pass.


Review generated by Claude Code (AI) using the /pr-review skill.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review effort=max

Please review final head bd49b8559ca0197cb683335423c35cb9b8f4158b and post a recommendation. The preceding revision passed Lint and all four CUDA runtime/native-batching/system/kernel jobs. This follow-up clarifies the engine-thread maintenance contract, restores deterministic re-entrancy rejection, makes compaction shrink-only, removes redundant per-forward bookkeeping, and strengthens capture, memory and retention tests. GPU CI is rerunning for this head.

Requested by OpenAI Codex on behalf of the PR author.

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 42m 28s —— View job


Recommendation: Approve, contingent on the GPU CI rerun passing at bd49b85: there are no blockers, but idle compaction repeats a full KV resize and CUDA-graph recapture for each burst of concurrent sessions separated by more than 500 ms, the lowering accepts a manifest version: 2 that changes nothing, and a few contract docs and tests have gaps.

Detailed review

PR Review: #25324 — [CUDA] Runtime physical columns for batched-sequence off-graph KV (ABI 2)

Summary

This PR adds ABI 2 for the CUDA batched-sequence off-graph KV cache. A fifth seq_info field carries the runtime physical column pitch, so owner columns start at one, double on admission, and shrink during idle maintenance through a new Executor::on_idle() hook. Nothing here blocks merging. The most consequential issue is the idle compaction policy (Performance). The remaining items are interface validation, contract docs, two test gaps and small simplifications.

API Design

  • Manifest version: 2 is accepted but means nothing, and runtime_columns is accepted with context_window even though no runtime serves that combination.

    • lower_offgraph_kv.py:88 now accepts version 1 or 2, but only runtime_columns (:99-103, :155) chooses the ABI.
    • No writer emits version 2: batched_sequence_manifest keeps version 1 and just adds the flag.
    • test_offgraph_kv_batched_sequence.py:494-503 asserts that both versions behave identically. Either make version 2 mean and require runtime columns, or keep accepting only version 1.
    • Only export_cuda_batching rejects runtime_columns plus context_window (the DFlash draft form), and only for its own callers (export_solo.py:477). MuseGlimmerCudaDFlashExecutor picks kBatchedSequence without reading the ABI method (cuda_dflash_executor.cpp:694). A draft built by any other exporter would load as ABI 1 against a program that declares a 5-wide seq_info.
    • Reject the combination in parse_offgraph_kv_manifest so it fails at export. Fix this →
  • sdpa_varlen documents a 5-wide layout precondition but never checks it. The docstring requires dense column and head lanes for 5-wide seq_info (sdpa_varlen.py:409-411), but the host only checks stride(-1) == 1 (:456). The kernels take the row pitch as field4 * stride(1) (:95-97, :255-257). A key/value view whose column stride is not the per-column footprint, such as a permuted [rows, H, pitch, D] buffer, is then addressed wrongly with no error. ABI 1 handles that view correctly through stride(0). Validate the layout when seq_info.shape[1] == 5. Fix this →

  • Only compaction is rejected from inside execute_raw's consumer.

    • compact_kv_columns refuses while raw_executing_ is set (cuda_executor.cpp:1103-1107, documented at cuda_executor.h:237-239).
    • open_session() and clone() pass straight through to seq_new()/seq_clone() (cuda_executor.cpp:653-666, extension/llm/batching/util/session_table.cpp:83,127). Under ABI 2 they do the same storage replacement from the consumer: a device-wide sync, a reallocation and an invalidate_bindings() recapture.
    • Either reject resizing session calls from the consumer too, or drop the compaction special case and document both as safe between slices. Either way, "execute_raw rejects re-entrant execution" (cuda_executor.h:20-21) would then have one meaning. Fix this →
  • Executor::on_idle() doesn't say how often it runs or that it blocks the engine. extension/llm/batching/executor.h:97-100 says only "when no batch is ready". RunnerImpl::run_() calls it on every idle loop iteration, as often as every 20 ms, and processes no commands while it runs (runner.cpp:798-811). A slow implementation, such as a compaction that copies live KV, therefore delays admissions and cancellations. Document both, so implementers know to rate-limit internally and keep the no-op path cheap. Fix this →

Testing

  • No test covers the backend's abort guard. The cache tests call abort_step() directly (test_cuda_kv_cache.cpp:3057, :3134). No test drives CudaBackend::execute through a failure after prepare_step (cuda_backend.cpp:794-815: a failed rebind_for_execute, AOTI run or commit) to show that the ScopeGuard clears inflight_. If the guard regresses, an ABI-2 cache wedges. Every later prepare_step would fail with "a forward is still prepared" (cuda_kv_cache.cpp:1076), and seq_new/seq_clone would return nullopt (:943, :970-971). Fix this →

  • IdleHookRunsBetweenBatchesOnTheEngineThread cannot fail for the property it is named after. on_idle and execute share the single engine thread. So EXPECT_FALSE(in_execute()) and the unchanged count during the held execute (runner_test.cpp:386, :404-406) hold by construction. If on_idle() moved ahead of execute_one_batch_() (runner.cpp:798-804) and so also ran on busy iterations, the test would still pass. Queue several ready batches and assert that no on_idle call lands between them. Fix this →

Performance

  • Idle compaction repeats a full resize and recapture for each burst of concurrent sessions separated by more than 500 ms.

    • Growth doubles owner columns on admission (cuda_kv_cache.cpp:1335-1339). CudaExecutor::on_idle() compacts back to live + 1 owners plus scratch once 500 ms pass with no cache activity and 2 * live + 1 < physical (cuda_executor.cpp:1113-1124, cuda_kv_cache.cpp:1340-1343).
    • Example: with one long-lived session, a burst of two more sessions grows the columns 2 → 3 → 5, and a 500 ms pause compacts 5 → 3. Every later burst then grows 3 → 5 and compacts back.
    • Each transition costs a device-wide sync, a copy of every live owner's KV, allocator trims, and a recapture plus one eager warmup step for every graph bound to the pool (cuda_kv_pool.cpp:442-462, :515-578).
    • Growth runs in open_session() on the engine thread, so both the request being admitted and every decoding session wait for it.
    • A failed compaction retries every 500 ms (cuda_executor.cpp:1123-1130). A shrink that cannot allocate repeats the sync and trims each time, and a poisoned cache logs an Error each time.
    • The thresholds are hard-coded. Only the probe's single idle op exercises the policy, and the no-op branches have no unit test.
    • Suggested fix: add hysteresis against recent peak occupancy, or make the grace and spare count configurable. Back off after a failure, and unit-test the thresholds. Fix this →
  • Column growth holds old and new storage at the same time.

    • resize_columns allocates every replacement layer before it discards the old ones (cuda_kv_pool.cpp:531-578).
    • With rows pre-allocated (initial_capacity == capacity), ABI 1 never reallocates. ABI 2's last growth, by contrast, peaks at old + new columns. For max_seqs = 8 that is 5 + 9 = 14 columns, or 1.56× ABI 1's fixed footprint, and close to 2× for other max_seqs (19 vs 10 columns at max_seqs = 9).
    • A GPU sized for ABI 1 can therefore refuse the last admissions under ABI 2. seq_new returns nullopt and the cache stays usable.
    • Neither the --runtime-kv-columns help (export_solo.py:1238-1239) nor the factory doc (cuda_kv_cache.h:229-231) mentions this. Fix this →

Code Quality

  • ensure_columns takes inputs that never vary.

    • ensure_columns(bool compact, int spare_columns, cudaStream_t stream) (cuda_kv_cache.cpp:1328) receives cudaStreamPerThread from all three callers (:956, :997, :1264).
    • spare_columns only matters when compact is set, but grown is computed in both modes.
    • The || compact at :1371 has no effect: a same-width compaction keeps every column, so resize_columns returns early (cuda_kv_pool.cpp:509-511). If the callers passed a target width instead, the mode flag could go.
    • The pool's trim_unused is always true in production (cuda_kv_cache.cpp:1390). Only FailedColumnGrowthKeepsOldStorageAndBindingsUsable passes false. Fix this →
  • free_physical_column() duplicates assignment that ensure_columns already does. seq_new and seq_clone pre-pick the lowest free column (cuda_kv_cache.cpp:947, :988, :1315-1326). The second loop in ensure_columns (:1356-1370) would assign that same column to an owner seeded with -1, and seq_clone reads the final column back afterwards (:1003). Fix this →

Specific Comments

  • backends/cuda/runtime/cuda_kv_cache.cpp:809-810: the wrap is broken, and "The last column is scratch," ends a short line.
  • backends/cuda/triton/kernels/sdpa_varlen.py:95: prefill reads the shared pitch from each segment's row, but decode (:255) and the lowering (lower_offgraph_kv.py:337) read row 0. Read row 0 everywhere, to match "shared by every segment".
  • backends/cuda/passes/lower_offgraph_kv.py:154: "get_offgraph_kv_max_seqs" is a bare literal next to the new named constant. This PR deleted its only Python constant (MAX_KV_SEQS_METHOD), and the test repeats the literal (test_offgraph_kv_batched_sequence.py:553).
  • examples/models/muse-glimmer/export/export_solo.py:256-257: stale comment. It describes the deleted MAX_KV_SEQS_METHOD and now sits above the draft constant's own comment.
  • examples/models/muse-glimmer/export/export_solo.py:1239: "sustained serving idleness" overstates the 500 ms grace in CudaExecutor::on_idle().
  • examples/models/muse-glimmer/tests/check_runtime_kv_columns.py:96, :120: these mkdir() calls lack exist_ok, so a rerun with the same --output-dir raises FileExistsError. main() creates the root with exist_ok=True (:180).
  • extension/llm/cache/cache.h:115-116: the new nullopt wording leaves out two refusals, a poisoned cache and, under ABI 2, a prepared forward (cuda_kv_cache.cpp:943). The rewrap also leaves "Ids may also be chosen by the caller;" on a short line.
  • extension/llm/cache/README.md:139-142: the kind table is missing kind::kBatchedSequenceRuntime / batched-sequence-runtime (cache_registry.h:79-81).

Fix this →

Recommendation

Approve, contingent on the GPU CI rerun passing at bd49b85. None of the issues above blocks merging an opt-in, experimental ABI. The idle policy should be tuned or made configurable before --runtime-kv-columns is recommended for bursty serving traffic. The manifest validation and the two test gaps are cheap to close, either in this PR or in a follow-up.

Review steps
  • Gather context: diffed against upstream base c1ab845, which reproduces the PR's 22 files (+2179/−123)
  • Run three independent reviews
  • Consolidate and deduplicate findings
  • Fact-check each finding against the code
  • Post the final review and recommendation

Review generated by Claude Code (AI) using the /pr-review skill.

Merge upstream f63b372 without changing the runtime-columns feature delta. Integration verified with assistance from OpenAI Codex.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review effort=max

Please review the current head 670bafe9295faa28ddf001ef30ff049c3db03570 and post a final recommendation. It merges the latest upstream main f63b372b702c50535b063c29a843b4adf63beffe; the CUDA runtime, lowering, executor and Runner feature code is unchanged from the previously qualified head bd49b8559. The merged tree passes 228 local native C++ tests and 8 CI/workload tests. Current-head CI is rerunning. Please verify the integration and any outstanding issues against this exact head.

Requested by OpenAI Codex on behalf of the PR author.

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 41m 24s —— View job


Request changes: the merge with f63b372 is clean and the feature code is unchanged from bd49b85, but in the default release (logging-off) build the ABI-2 abort guard in cuda_backend.cpp compiles away, so one failed forward permanently wedges the off-graph KV cache, and the only CI job that runs ABI 2 end to end builds with logging on and cannot catch it.

Detailed review of 670bafe

PR Review: #25324

Summary

Adds ABI 2 for the CUDA batched-sequence off-graph KV cache: seq_info gains a fifth field carrying the runtime physical column pitch, so owner columns start at 2, grow geometrically on admission, and shrink through compact_columns() / CudaExecutor::on_idle() instead of being fixed at max_seqs + 1.

Integration: 670bafe has parents bd49b85 and f63b372 (= origin/main). Of the 22 PR files, upstream touched only examples/models/muse-glimmer/CMakeLists.txt (the muse_glimmer_sampling test and the MLX DFlash block); the probe block merged intact, and the runtime, lowering, executor and Runner code is unchanged from bd49b85, so every finding below predates the merge.

CI at review time: tests were not rerun in this job (no GPU). unittest-cuda-muse-glimmer-batching (the only job that drives ABI 2 through the backend) and the other GPU jobs in run 44919195267 were still running. The only failure so far, MLX test-mlx-voxtral-realtime (run 44919244989), is in code this PR does not touch.

Found: one blocker plus the CI gap that hides it, one design issue in the idle-compaction policy, and several minor items.

Code Quality

  • Blocker: backends/cuda/runtime/cuda_backend.cpp:795-801 compiles the ABI-2 abort out of release builds. abort_step() is the condition of ET_CHECK_OR_LOG_ERROR, and that macro is ((void)0) under ET_LOG_ENABLED=0 (runtime/platform/log.h:214), so the call disappears along with the log. This is the shipped configuration:

    • EXECUTORCH_ENABLE_LOGGING defaults to on only for Debug (tools/cmake/preset/default.cmake:9-22).
    • llm-release-cuda, which make muse-glimmer-batching-cuda uses, is a Release build and does not override it.
    • The root CMakeLists.txt:263-268 adds -DET_LOG_ENABLED=0 before backends/cuda is added (line 937).

    prepare_step sets inflight_ at line 812. Any later error leaves it set for good: rebind_for_execute, mutable_state_rebind_for_execute, a replay input-shape mismatch, cudaGraphLaunch/cudaStreamSynchronize, run_once, or a failed commit. After that:

    • every later prepare_step fails with "a forward is still prepared" (cuda_kv_cache.cpp:1075-1076);
    • seq_new/seq_clone return nullopt (cuda_kv_cache.cpp:943);
    • compact_columns is refused (cuda_kv_cache.cpp:1260-1263);
    • nothing on the CudaExecutor path calls clear().

    So one transient failure disables the executor until the process exits. The unit tests call abort_step() directly, so they cannot see this. Call it unconditionally and log the result:

    const Error aborted = handle->kv_cache->abort_step();
    if (aborted != Error::Ok) {
      ET_LOG(Error, "offgraph_kv: failed to abort prepared forward");
    }

    Fix this →

  • Minor: backends/cuda/runtime/cuda_kv_pool.cpp:564-567, 571-574. When the column copy or stream sync fails in resize_columns, it rolls back and returns a bare Error::Internal. Every other CUDA failure in this file logs cudaGetErrorString (for example lines 178-202, 289, 358, 417). A failed admission or compaction resize therefore leaves no diagnostic.

Infrastructure

  • Major: the ABI-2 CI job no longer tests the configuration users build (.github/workflows/cuda.yml:698-700, examples/models/muse-glimmer/tests/check_runtime_kv_columns.py:144-159, .ci/scripts/tests/test_muse_glimmer_batching_ci.py:34).

    • The job replaces make muse-glimmer-batching-cuda with cmake --preset llm-release-cuda -DEXECUTORCH_ENABLE_LOGGING=ON.
    • It needs logging because the checker qualifies graph capture by counting the Info line "CUDA graph: captured and instantiated for 'forward_2". The CI unit test pins the flag.
    • As a result, CI never runs the logging-off build where the blocker above is live, and capture qualification depends on the exact log wording.

    Report capture/recapture counts as a metric (next to method_calls() or in the KV metrics), assert on that, and go back to the default preset. Fix this →

  • Minor: .ci/scripts/muse_glimmer_batching_workload.h:103-151. EvidenceExecutor overrides every Executor virtual except the new on_idle(), and the base version is a no-op (extension/llm/batching/executor.h:100). So when the smoke (muse_glimmer_batching_smoke.cpp:75) runs against a --runtime-kv-columns export, it never compacts and gives no sign of it. Fix this →

  • Minor: examples/models/muse-glimmer/CMakeLists.txt:147-153. MUSE_GLIMMER_BUILD_RUNTIME_KV_PROBE checks only EXECUTORCH_BUILD_CUDA, but tests/runtime_kv_columns_probe.cpp:21 includes <unistd.h> and uses dup/dup2/fdopen/getpid. Enabling it in the Windows CUDA build therefore fails at compile time, not at configure time. Add a WIN32 FATAL_ERROR next to the CUDA check, or use the _dup/_fdopen equivalents.

Testing

  • Minor: examples/models/muse-glimmer/tests/check_runtime_kv_columns.py:193-205 asserts memory contraction only through live_KV_bytes, which is the pool's own allocated_bytes counter. The probe already reports allocator_reserved_bytes and device_used_bytes (runtime_kv_columns_probe.cpp:98-121), but nothing asserts them. A compaction that updates the counter without returning memory to the device (for example a regressed release_cached_memory call in cuda_kv_pool.cpp) still passes, even though the docstring promises "actual ABI-2 CUDA allocations".
  • Minor: backends/cuda/runtime/test/test_cuda_kv_cache.cpp:3175, 3181 sets phase = Warmup and expects Warmup after the no-op compactions. That cannot detect an unwanted invalidate_bindings(), which leaves a Warmup handle in Warmup and only raises warmup_remaining to at least 1 (cuda_kv_pool.cpp:459). Set warmup_remaining = 0 and assert it stays 0, as lines 1019-1031 do. Today only column_resize_count catches this case.

API Design

  • Minor: backends/cuda/passes/lower_offgraph_kv.py:88-103 accepts manifest version 2, but the version has no meaning:

    • runtime_columns works with version 1, and version 2 works without it (test_offgraph_kv_batched_sequence.py:494-503 accepts all four combinations).
    • No producer emits version 2: batched_sequence_manifest (export_solo.py:341-362) keeps the sequence manifest's version 1 and only adds runtime_columns.

    Either make version 2 mean runtime columns (require it and emit it), or stop accepting 2. Fix this →

Performance

  • Major: idle compaction has no hysteresis against geometric growth (backends/cuda/batching/cuda_executor.cpp:1109-1131 with backends/cuda/runtime/cuda_kv_cache.cpp:1335-1344). After 500 ms idle, on_idle() compacts whenever 2*live+1 < physical, down to live + 1 spare, while admission doubles the width.

    • A load that alternates between 3 and 5 live sessions with gaps of 500 ms or more cycles 9 → 5 → 9 columns forever. Compaction goes to min(max_seqs, 3+1)+1 = 5, then the fifth admission regrows to 2*(5-1)+1 = 9.
    • Per the comment at cuda_kv_cache.cpp:1330-1332, each resize is a device-wide sync plus a full KV copy. It also costs a graph recapture and an allocator trim.
    • Regrowth runs inside open_session on the engine thread, so it stalls decode for every live session.
    • The 500 ms, 2*live+1 and spare = 1 constants are hard-coded, and the 2*live+1 threshold is untested.

    Compacting to 2*live+1 (spare = live) makes the policy stable. Making the delay and spare configurable in CudaExecutorConfig would also let deployments opt out. Fix this →

Specific Comments

  • backends/cuda/runtime/cuda_kv_cache.cpp:1071-1074 (and 1196): an ABI-2 cache reports "uncommitted or failed draft co-batch; clear required" once a failed resize or abort fence has poisoned it. There is no draft co-batch in that case, and CudaExecutor never calls clear().
  • backends/cuda/runtime/cuda_kv_cache.cpp:809-810 and extension/llm/cache/cache.h:115-117: leftovers from reflowing. A short line breaks each paragraph mid-sentence ("The last column is scratch," / "Ids may also be chosen by the caller;").

Recommendation

Request Changes

The release-build abort guard (cuda_backend.cpp:795-801) takes only a few lines to fix. As the code stands, one failed ABI-2 forward permanently disables the executor, and CI cannot detect it because the only ABI-2 job builds with logging on. Both of these block approval. The idle-compaction thrash should be fixed or explicitly justified. The minor items can follow.


Review generated by Claude Code (AI) using the /pr-review skill.

Task checklist
  • Verify merge 670bafe against origin/main (f63b372) and the previous head (bd49b85)
  • Run three independent reviews over the 22 changed files
  • Consolidate and fact-check findings against the exact head
  • Check current-head CI
  • Post final recommendation

Abort prepared forwards independently of logging, qualify capture and replay through counters, and retain idle compaction headroom across bursts. Add release-path recovery, allocation-free trimming, and native burst regression coverage. Authored with OpenAI Codex assistance.
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review effort=max

Please review current head 952486c401a769bf03b2fd1e7b81a98e8bee156c and post a final recommendation. This follow-up executes failed-forward abort independently of logging and adds backend recovery tests; native graph qualification now reads counters and runs logging OFF. Idle compaction retains proportional headroom and the native probe exercises burst reuse without regrowth. Per-device allocator trim avoids post-commit host allocation, with a fault-injection test. The follow-up also addresses the manifest/version, idle-wrapper, Windows probe and no-op warmup checks.

Local CI/workload/workflow checks passed. Current-head CUDA and MLX qualification is pending; this Mac cannot run CUDA. Please review the whole PR against upstream main f63b372b702c50535b063c29a843b4adf63beffe, including release failure paths and native test adequacy.

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 58m 39s —— View job


PR Review: #25324

Approve once Test CUDA Builds (still running) passes on 952486c: there are no correctness blockers, but in release builds a poisoned ABI-2 cache or a repeatedly failing idle compaction leaves no trace and only shows up as "At capacity" refusals.

Detailed review

Summary

This PR adds ABI 2 to the CUDA batched-sequence off-graph KV cache. A fifth seq_info field carries the physical column pitch, so storage grows geometrically on admission and compacts after an idle grace period. The PR also adds a transactional resize_columns, abort_step for failed forwards, per-method CUDA graph counters, a per-device allocator trim and a native ABI-1-vs-ABI-2 probe in CUDA CI.

There are no correctness blockers. The main gap is that failures in release builds are invisible (first API Design item). Smaller issues:

  • resize_columns can still fail after it commits.
  • Every growth trims the allocator twice.
  • Native tests miss poison/recovery, the idle grace period and replay after recapture.

Code Quality

  • CudaKVPool::resize_columns can fail after it commits. It commits the new allocations and columns_ at backends/cuda/runtime/cuda_kv_pool.cpp:597-609. It then runs a fallible cudaStreamSynchronize at :611-614, before the trim. release_cached_memory returns void, so that sync is the only step that can fail after the commit. That one step accounts for:

    • the committed out-param;
    • the split-state branch in ensure_columns (backends/cuda/runtime/cuda_kv_cache.cpp:1382-1392: adopt target, remap owners, then poison), which is untested;
    • the caveat at cuda_kv_pool.h:92-93;
    • the constraint at extension/cuda/cuda_allocator.cpp:848-849.

    Return right after the commit, and have ensure_columns run the sync and trim once it has updated columns_ and the owner columns, poisoning on failure. That removes both the out-param and the split state.
    Fix this →

Testing

  • The poison and recovery paths have no native tests. No test reaches these poison sites:

    • the clone copy (backends/cuda/runtime/cuda_kv_cache.cpp:1007);
    • the abort_step fence (:1218);
    • the post-commit resize (:1389-1391).

    No runtime-column test calls clear(). The only calls in backends/cuda/runtime/test/test_cuda_kv_cache.cpp are at :377, :2003, :2115 and :2418. So nothing checks that prepare_step, seq_new, seq_clone and compact_columns refuse while poisoned, or that clear() makes the cache usable again.

    The six new backend recovery tests in test_cuda_backend_execute.cpp use a fake PreparedCache. They show that the delegate calls abort_step, not what the real cache does next. extension/cuda/test/cuda_allocator_exit.cpp:68-114 already interposes CUDA runtime symbols in a dedicated binary. Doing the same for cudaStreamSynchronize and cudaMemcpy2DAsync would reach all three sites.
    Fix this →

  • CudaExecutor's ABI-2 policy has no unit tests, and the probe goes around it. backends/cuda/batching/test/test_cuda_executor.cpp has no runtime-columns, on_idle or compact_kv_columns coverage. The probe (examples/models/muse-glimmer/tests/runtime_kv_columns_probe.cpp) has three gaps:

    • It always sleeps 550 ms before on_idle (:291), so the "too soon" early return at backends/cuda/batching/cuda_executor.cpp:1117-1121 never runs.
    • It sends new, remove and clone through raw cache_control() (:271-284), so the executor's activity updates in open, close and clone (cuda_executor.cpp:653-666) never run.
    • It calls cudaDeviceSynchronize after every op and in snapshot (:298, :80, and :187 before reading logits). That hides cross-stream ordering bugs in the ABI-2 clone and resize paths, which run on cudaStreamPerThread (cuda_kv_cache.cpp:995-999).

    Fix this →

  • The graph-replay check uses cumulative totals. examples/models/muse-glimmer/tests/check_runtime_kv_columns.py:169-176 reads only the final snapshot.

    • captures == 2 proves a recapture happened.
    • replays > captures is already true from the replays before compaction plus the new graph's first launch, which counts as a replay (backends/cuda/runtime/cuda_backend.cpp:1336).

    So the check doesn't show that the recaptured forward_2 graph is ever replayed. Compare forward_2 in steps[7] (before_compaction, already used at :226) against steps[12] (decode-1-4): captures should grow by exactly one and replays by more than one.
    Fix this →

API Design

  • In release builds, a poisoned cache and failing compactions are invisible and last until restart. Once poisoned_ is set, these refuse:

    • prepare_step (backends/cuda/runtime/cuda_kv_cache.cpp:1069-1072);
    • seq_new (:943);
    • seq_clone (:969);
    • compact_columns (:1249-1267).

    Only clear() (:884-901) resets it. Nothing in CudaExecutor, SessionTable or the batching runner calls clear() on the target cache. The only caller in the tree is for the dflash draft cache (examples/models/muse-glimmer/runtime/cuda_dflash_executor.cpp:1390). An operator has no signal of what happened:

    • SessionTable (extension/llm/batching/util/session_table.cpp:83) gets nullopt from seq_new. The runner reports this as "At capacity" and counts it in sessions_refused (extension/llm/batching/runner.cpp:891-895).
    • The seq_new doc (extension/llm/cache/cache.h:114-117) doesn't mention poisoning.
    • OffGraphKVMetrics (cuda_kv_cache.h:29-43) has no poisoned or failed-resize field.
    • on_idle reports failure only through ET_LOG (backends/cuda/batching/cuda_executor.cpp:1130-1135). That compiles out when EXECUTORCH_ENABLE_LOGGING is OFF, which is the default outside Debug and how llm-release-cuda builds.
    • on_idle also resets last_cache_activity_ before compacting (:1127). A compaction that keeps failing without poisoning (e.g. an allocation failure that rolls back) is retried every 500 ms of idleness. Each retry does a device-wide sync and an allocator trim.

    The new poisoned_ check in seq_new also applies to ABI 1.

    Suggested changes:

    • Expose poison and failed resizes in OffGraphKVMetrics.
    • Document in cache.h both meanings of nullopt and that clear() recovers.
    • Either give the runner a recovery path (drain, then clear()) or fail loudly.

    This should land before ABI 2 serves production traffic.
    Fix this →

  • sdpa_varlen doesn't validate its 5-wide preconditions. The docstring requires contiguous column and head lanes for 5-wide seq_info (backends/cuda/triton/kernels/sdpa_varlen.py:409-411), but the wrapper only checks stride(-1) (:456-457). Prefill reads the pitch from each segment's row (:94-97), while decode reads it from row 0 only (:254-257). This is safe today because the cache writes the same pitch into every row. Another caller that passes strided K/V, or rows with different field-4 values, gets wrong attention instead of an error.
    Fix this →

  • graph_metrics double-counts a failed first launch and only covers methods bound to the KV pool.

    • ++captures (backends/cuda/runtime/cuda_backend.cpp:1323) runs before the first cudaGraphLaunch (:1330-1335). If that launch fails, graph_launched stays false and :900-901 also increments capture_failures. One failure is then counted both as a capture and as a capture failure, which contradicts "Successful captures/launches" in cuda_kv_cache.h:99.
    • CudaExecutor::graph_metrics forwards to the KV cache (cuda_executor.cpp:1103-1104), and the pool sums only the handles bound to it (cuda_kv_pool.cpp:442-445). A graph-captured method with no off-graph KV therefore always reports zeros.

    Fix this →

Performance

  • Every growth trims the allocator twice. ensure_columns passes trim_unused=true (backends/cuda/runtime/cuda_kv_cache.cpp:1380), so resize_columns calls CudaAllocator::release_cached_memory twice:

    • before allocating (cuda_kv_pool.cpp:530-532);
    • after the commit (:610-616).

    The post-commit trim is needed because the pool's release threshold is UINT64_MAX (extension/cuda/cuda_allocator.cpp:65). The pre-allocation trim also returns every other method's cached workspace to the driver on each admission-driven growth, and the next forwards have to reserve it again. Allocate first, and trim and retry only if an allocation fails.
    Fix this →

Specific Comments

  • backends/cuda/runtime/cuda_kv_cache.cpp:809-810: the comment wraps early after "The last column is scratch,".
  • extension/llm/cache/cache.h:116: the comment wraps early, leaving "Ids may also be chosen by the caller;" alone on its line.
  • backends/cuda/runtime/cuda_kv_cache.cpp:973-975, :1401-1404: "this thread's stream" is only true for ABI 2. ABI 1 uses pool_.stream() (:998-999).
  • examples/models/muse-glimmer/CMakeLists.txt:154-167: the probe checks EXECUTORCH_BUILD_CUDA but not TARGET cuda_batching. An install without the LLM batching libraries fails at link time instead of with the clear FATAL_ERROR used at :288-297.

Recommendation

Approve, conditional on Test CUDA Builds passing on 952486c. That run covers the new probe and the CUDA unit tests and is still in progress.

Fix the release-build observability problem (first API Design item) before ABI 2 serves production traffic, either here or in a quick follow-up. The other items can be follow-ups.

The MLX failure (test-mlx-voxtral-realtime in run 44924434986, all-<unk> output) is unrelated to this PR.


Review generated by Claude Code (AI) using the /pr-review skill.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant