Repository navigation
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24572
Note: Links to docs will display an error until the docs builds have been completed. ❌ 3 New Failures, 1 Unrelated Failure, 5 Unclassified FailuresAs of commit c87270e with merge base 765e973 ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
Run each partitioner’s edge transforms after caller-supplied passes so backends can use the standard lowering entry point. QNN contributes its existing transforms through the new hook.
Reserve names for constant tensors already present in the exported program before lifting new constants, so methods with folded tables retain distinct constants.
Choose the registered IOManager and its logits, method names and step sizes, rejecting unknown contracts and invalid metadata. Load and reset the selected methods; route prompt tails to decode with a side-effect-free capacity check. Preserve module and I/O-buffer lifetimes and expose a backend-neutral allocator hook.
Route backend options through QnnPartitioner.from_options and register an allocator that supplies QNN shared-memory buffers through the generic IOManager hook. The LLM runner remains backend-neutral.
Encode graph output positions and restore their order at both runtime tensor-allocation entry points. Preserve context-binary order and legacy mutable outputs, and ignore malformed or overflowing position labels without throwing. This runtime change needs Qualcomm review.
0ef7a34 to
21d2bdc
Compare
On QNN SDK 2.48 an HTP execute that binds some tensors as registered rpcmem and others as plain client buffers fails (prefill DMA error 1100, decode transport error 1003) or produces wrong results; 2.50 and 2.51 runtimes run the same binding correctly. LLM runners place their caches, masks, positions and logits in rpcmem through the I/O buffer allocator, but delegate inputs and outputs that ExecuTorch plans internally (CPU-computed masks and RoPE tables, new K/V rows) stay in planned memory. When any tensor of an execute is shared, the backend now copies the remaining ones through persistent rpcmem staging buffers, one block per graph owned by the QnnManager and registered once, so each execute is all shared or all client memory. Executes without shared tensors are unchanged. Authored with AI assistance (Metamate / agentcloud).
Move backend-neutral static-cache rewriting into the LLM extension and keep QNN partitioning and tuning in the backend. Detect sliding layers from model configuration, write every cache and counter in place on the caller buffer (export fails otherwise, since a missed write becomes full-cache CPU copies), and register a versioned runner that handles fixed chunks, padding, counters and reset. Quantized caches, sha attention and full-length sliding mode are excluded. Authored with AI assistance (Metamate / agentcloud).
Use authoritative positions, float-cache version and dtype checks, exact buffer geometry and selected methods. Bind optional HF state outputs to existing buffers, preserve input aliases and register each concrete IOManager from its own implementation. Float-only fixtures cover reset, padding, malformed metadata and near-context tails.
21d2bdc to
c87270e
Compare
LLM runner: validate and drive hf_static float-cache programs
Resetting only logical position while leaving cache counters stale corrupts subsequent generation. The hf_static IOManager supplies authoritative
start_poscounters, validates version-1 metadata and buffer geometry, and drives fixed methods through shared float-cache buffers.The program keeps HF's static-cache tensor interface with a fixed prefill/decode padding protocol. Window masks exclude padding, and a tail that does not fit as a padded prefill runs through decode before execution. Logits remain at
3 * n_layers; the mutation prefix is opaque. Optional ordered state suffix outputs bind to existing buffers rather than allocating another cache.Register the hf_static manager from this PR, replacing PR 3's inert builtin definition. Review validation/state handling, registration, build wiring and float32/float16/repeated-state fixtures. C++ has no transformers dependency; Python loading needs PR 6's guarded local-branch hook. Quantized caches, sha and full-length sliding mode are excluded.
Buffer ownership, fixed-shape execution and reset are implemented. Inspection, adopt/fork/evict hooks and arbitrary context extension remain follow-ups.
Files
Test Plan
Commit
cb09f3b3e2b9e529875812fa6377908c9f351e95on892e50425f5a3e90ff8d3c8802fdfb31f8599518. Validated locally with Python 3.12, pinned Torch/TorchAO, the local transformers exporter branch and QAIRT SDK 2.48.223 runner tests passed, including all 77 hf_static cases; 2 existing skips; 0 failures.
cmake --build /data/users/sidart/qnnflow/gh_prs/build-split-native-pr07-final -j4 --target test_runner ctest --test-dir /data/users/sidart/qnnflow/gh_prs/build-split-native-pr07-final -R "^test_runner$" --output-on-failure -VThe two existing runner skips are
TextDecoderRunnerTest.LoadUsesMethodNameandStepWithAllModels.No device/physical rpcmem, two-target FCB, real-model export or full QNN device-suite result is claimed. Buck could not start because the local launcher requires a missing
.buck2. The base already contains #23563 and #23183; neither is a separate prerequisite commit here.Related PRs (review in order)
Each PR includes the commits of the ones above it; review only its own top commit.