Repository navigation
Conversation
Decouple sequence IDs from physical storage columns so the batched-sequence cache can start with 2 physical columns and grow or shrink at runtime instead of always allocating max_seqs + 1 columns. - CudaKVPool gains resize_columns() with declared vs physical column counts, transactional replacement copies, and binding invalidation on column moves (mirroring grow()). - CudaBatchedSequenceCache tracks IDs and physical columns separately, uploads a 5-wide seq_info whose last field is the runtime pitch, and adds ensure_columns()/compact_columns() behind an inflight guard. - The lowering pass gains manifest version 2 with runtime_columns, the SDPA kernel reads the runtime pitch via INFO_WIDTH, and CudaExecutor reads get_offgraph_kv_sequence_info_abi (1 or 2) from the program. - ABI-1 behaviour is unchanged: the new paths are gated on CacheConfig::runtime_sequence_columns and manifest version 2. Tests: extended test_cuda_kv_cache and test_offgraph_kv_batched_sequence, plus an opt-in Muse-Glimmer runtime KV probe. The SDPA ring-scan logic from the experiment patch was intentionally not ported: main's ring handling is kept and only the runtime-pitch change is included. This is PR 1 of the runtime-columns series; per-owner capacities (ABI 3) build on the ID/column split introduced here.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25324
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 2 Pending, 7 Unrelated FailuresAs of commit 952486c with merge base f63b372 ( NEW FAILURES - The following jobs have failed:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
BROKEN TRUNK - The following jobs failed but was present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
Gasoonjia
left a comment
There was a problem hiding this comment.
Review of the runtime-physical-columns (ABI 2) change. The core pool mechanics look right, but I have one functional concern about failure recovery and a few smaller items.
What looks good
CudaKVPool::resize_columnsruns its steps in a sound order: it allocates every replacement, queues the 2D copies, synchronizes, and only then callsinvalidate_bindings()and discards the old storage. Every failure path before the commit point rolls back to the old allocations, bindings,allocated_bytes_andcolumns_.FailedColumnGrowthKeepsOldStorageAndBindingsUsablecovers this.committedis reported separately from errors after the commit point (the trim sync), andensure_columnspoisons the cache in that case. That is the right split.invalidate_bindings()is shared withgrow(). It reaches every handle (Replay -> recapture, Warmup -> at least one eager step), andContractionInvalidatesEveryGraphAndBindingchecks it.- An ABI mismatch between the program and the runtime fails at load:
check_compiledcompares the side-buffer shape ([S,4]vs[S,5]), andFixedInfoABIIsRejectedcovers it. - The pitch handling is consistent across C++, the Python writes and Triton. Every
seq_inforow, padded segments included, carriescolumns_, so the decode kernel can safely read only row 0 and the prefill kernel can read it per segment. The dummy column moves tocolumns_ - 1with the pitch.
Main concern
- A failed execute leaves
inflight_set, and ABI-2 caches stay unusable untilclear()(inline). In ABI-1 a failed forward does not block later steps.
Other items (inline)
2. Columns grow one at a time. Each admission past the current width runs a device-wide sync, a full KV reallocation and copy, and an allocator trim, all inside prepare_step. Peak memory during a resize is old + new storage, so near max_seqs this can OOM where ABI-1's upfront allocation would have fit.
3. Changing compile_wrapper_opt_level from O0 to O2 in export_solo.py is unrelated and affects ABI-1 exports too.
4. ABI 2 combined with DFlash is rejected only by the export script. The runtime (CudaExecutor::create / the cache factory) still accepts it.
5. The seq_rm guard against step_seq_ids_ applies only in runtime mode. An abandoned declare_step then blocks removal, and ABI-1 still has the latent seqs_.at() throw.
6. The ABI-1 seq_clone failure path changed: a device-wide sync and possible poisoning. So ABI-1 is not strictly untouched (minor).
Nits / docs
make_cuda_batched_sequence_kv_cache's comment still says the step buffers are[segments, 4](cuda_kv_cache.cpp, ~L1576).- The
sdpa_varlendocstring forseq_info(~L410) still saysInt32[S, 4]. - In
LowerOffGraphKVPass,self._manifest.get("runtime_columns", False)is redundant now thatparse_offgraph_kv_manifestcallssetdefault.self._manifest["runtime_columns"]is enough. - When columns grow with an identity mapping, one
cudaMemcpy2DAsyncper layer of widthslot * old_columnscould replace the per-column copies (layers x seqs x 2calls).
Test gaps
- No test covers a step that is prepared and never committed (execute failure) in ABI-2.
- No executor-level test of
get_offgraph_kv_sequence_info_abivalidation (ABI 3; ABI 2 on the cell layout; ABI 2 with a draft method). - The Python test runs
batched_sequence_stepeagerly. No AOTI-compiled ABI-2 program runs in CI; the Muse-Glimmer probe covers that only manually. The probe is ~290 lines of opt-in diagnostics that CI never builds. Consider moving it to a follow-up or a dev-tools location.
This review was generated with AI assistance (Claude Code).
- Revert the unrelated O0 -> O2 change in export_solo.py. - Recover a prepared-but-uncommitted ABI-2 step in the next prepare_step (fence all streams, then clear inflight_) instead of wedging the cache until clear() drops every sequence; add a regression test. - Grow physical columns geometrically and trim the allocator only on the compact path, avoiding one full KV copy per admission. - Reject runtime columns + DFlash in CudaExecutor and in the batched-sequence cache factory. - Gate the seq_clone fence/poison on runtime columns so ABI-1 clone failures are unchanged; refuse seq_rm for a declared sequence in both ABIs. - Fix the lintrunner CMake formatting failure and stale seq_info width comments. - Pool resize comment and lowering .get() redundancy left as-is: resize_columns is only called from ensure_columns, and the .get() keeps LowerOffGraphKVPass usable with manifests not built by the parser.
|
Thanks for the review — addressed in b649b06 (see replies to each inline comment). The one CI failure on the first push was lintrunner CMake formatting only, also fixed in this update. Remaining known gaps, not addressed here: per-column copy merging in resize_columns, and a compiled ABI-2 run in CI (the opt-in probe is not built by CI); both are follow-up work, not blockers for the ABI-2 plumbing itself. |
|
@claude review effort=max |
|
Claude finished @Gasoonjia's task in 53m 50s —— View job PR Review: #25324This PR adds a runtime-pitch Detailed reviewSummaryThe PR lets the batched-sequence cache start at 2 physical columns, grow as sequences are admitted, and shrink with Two problems block it:
"Test CUDA Builds" (run 44647068649) was still queued when I wrote this, so the test failures below come from tracing the code, not from CI. Backward Compatibility
API Design
Infrastructure
Testing
Performance
Code Quality
Specific Comments
RecommendationRequest Changes.
Review generated by Claude Code (AI) using the |
|
@claude review effort=max |
PR Review: #25324
|
|
Claude finished @Gasoonjia's task in 59m 4s —— View job PR Review: #25324Adds an opt-in ABI 2 that lets the CUDA batched-sequence KV cache grow and compact its physical column width at runtime; Request Changes, because compaction and step recovery treat a flag that every normal forward sets as proof of an abandoned step, and nothing in serving ever compacts, so the memory saving is gone after the first burst to peak concurrency. Detailed reviewSummaryAdds manifest version 2 / Thread Safety
Backward Compatibility
API Design
Testing
Performance
Code Quality
Specific Comments
RecommendationRequest Changes Before compaction is safe to expose, Review generated by Claude Code (AI) using the |
|
@claude /pr-review effort=max Please review current head Review requested by OpenAI Codex on behalf of the PR author. |
|
Claude finished @Gasoonjia's task in 42m 25s —— View job PR Review: #25324Recommendation: Approve. There are no blockers at Detailed reviewSummaryThe PR adds ABI 2 for the CUDA batched-sequence off-graph KV cache. A 5-wide None of the findings blocks the PR. I only skimmed Thread Safety
API Design
Testing
Performance
Specific Comments
RecommendationApprove Nothing blocks approval. The two items most worth fixing before merge are the threading contract under Thread Safety and the Review generated by Claude Code (AI) using the |
|
@claude /pr-review effort=max Please review final head Requested by OpenAI Codex on behalf of the PR author. |
|
Claude finished @Gasoonjia's task in 42m 28s —— View job Recommendation: Approve, contingent on the GPU CI rerun passing at Detailed reviewPR Review: #25324 — [CUDA] Runtime physical columns for batched-sequence off-graph KV (ABI 2)SummaryThis PR adds ABI 2 for the CUDA batched-sequence off-graph KV cache. A fifth API Design
Testing
Performance
Code Quality
Specific Comments
RecommendationApprove, contingent on the GPU CI rerun passing at Review steps
Review generated by Claude Code (AI) using the |
Merge upstream f63b372 without changing the runtime-columns feature delta. Integration verified with assistance from OpenAI Codex.
|
@claude /pr-review effort=max Please review the current head Requested by OpenAI Codex on behalf of the PR author. |
|
Claude finished @Gasoonjia's task in 41m 24s —— View job Request changes: the merge with Detailed review of
|
Abort prepared forwards independently of logging, qualify capture and replay through counters, and retain idle compaction headroom across bursts. Add release-path recovery, allocation-free trimming, and native burst regression coverage. Authored with OpenAI Codex assistance.
|
@claude /pr-review effort=max Please review current head Local CI/workload/workflow checks passed. Current-head CUDA and MLX qualification is pending; this Mac cannot run CUDA. Please review the whole PR against upstream main |
|
Claude finished @Gasoonjia's task in 58m 39s —— View job PR Review: #25324Approve once Detailed reviewSummaryThis PR adds ABI 2 to the CUDA batched-sequence off-graph KV cache. A fifth There are no correctness blockers. The main gap is that failures in release builds are invisible (first API Design item). Smaller issues:
Code Quality
Testing
API Design
Performance
Specific Comments
RecommendationApprove, conditional on Fix the release-build observability problem (first API Design item) before ABI 2 serves production traffic, either here or in a quick follow-up. The other items can be follow-ups. The MLX failure ( Review generated by Claude Code (AI) using the |

Summary
A CUDA batched-sequence off-graph KV cache currently allocates the exported maximum owner width on its first forward. Sequence-info ABI 2 carries the runtime physical pitch independently of sequence IDs, so columns start with one owner plus scratch and grow geometrically on admission. After a 500 ms idle grace period, the Runner can compact underused columns while retaining one spare owner column per live owner to avoid repeated resize and graph recapture across bursts. This changes physical width; per-owner row sizing and Ring headroom remain separate follow-ups.
Review the pool/cache lifecycle first, then executor/Runner integration, lowering/export metadata and tests. Replacement copies retained KV transactionally, invalidates bindings and captured graphs, and releases replaced allocations. Same-width compaction preserves valid mappings without relocation. Replacement temporarily holds old and new allocations; failed admission removes the unpublished owner. Post-commit per-device allocator trimming does not allocate a host snapshot that could interrupt pitch reconciliation.
The backend aborts a failed prepared forward independently of logging, fencing queued work before storage can move again. Abort does not roll back claimed logical rows. Prepare/compaction reject active forwards, and raw execution rejects re-entrant execution or maintenance. Execution, session verbs and maintenance follow the Executor engine-thread contract and use a valid current-thread stream.
ABI 1 remains the default when the optional ABI constant is absent. The cache registry distinguishes fixed and runtime-column kinds. Manifest schema version remains 1; the validated
runtime_columnsflag selects sequence-info ABI 2. Export is opt-in through--runtime-kv-columns, requires CUDA batched-sequence layout, and excludes DFlash. The existing attention algorithm and Ring addressing are preserved apart from reading the runtime pitch.Validation
The local Runner/scheduler/prefix-cache/opaque-input C++ suites passed 228 tests. The follow-up passed 6 CI configuration tests, 2 workload tests (8 C++ cases), 10 CUDA workflow tests, formatting and diff checks. An exact-source host reproduction confirms that failed-forward abort remains active with logging disabled. CUDA/Triton execution is unavailable on this Mac; current-head GPU CI and Claude review are pending.
CUDA CI builds the release runtime and ABI-2 native probe with logging disabled. Backend tests cover prepare/rebind/program/commit failure recovery and capture/replay counters across recapture. The allocator test forbids host allocation during post-commit per-device trimming. Native qualification exports identical tiny ABI-1/ABI-2 models and checks logits/argmax through growth, wrapped Ring cloning, sparse compaction, reuse and a 5-to-3-to-5 owner burst. It verifies capture/replay through counters, rejects eager fallback, compares live KV allocations and checks driver-reported allocator memory contraction. Same-width tests assert that warmup state is not reset; the CI wrapper forwards idle maintenance.
Built on upstream main
f63b372b702c50535b063c29a843b4adf63beffe. Authored with OpenAI Codex assistance.