Skip to content

Document the hf_static v2 NPU cache layout - #25351

Draft
psiddh wants to merge 15 commits into
pytorch:mainfrom
psiddh:npu/pr13-docs
Draft

psiddh wants to merge 15 commits into
pytorch:mainfrom
psiddh:npu/pr13-docs

Conversation

@psiddh

@psiddh psiddh commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Documents the v2 NPU cache layout: what is in the contract, why each property is there, and the measured effect of each.

Related PRs (review in order)

  1. Partitioner: run backend-contributed edge transform passes #24566 Partitioner: run backend-contributed edge transform passes
  2. Export: avoid reused lifted-constant names #24567 Export: avoid reused lifted-constant names
  3. LLM runner: select I/O contracts and accept custom buffer allocators #24568 LLM runner: select I/O contracts and accept custom buffer allocators
  4. QNN: construct partitioners from options and allocate LLM I/O in rpcmem #24569 QNN: construct partitioners from options and allocate LLM I/O in rpcmem
  5. QNN: bind delegate outputs in exported program order #24570 QNN: bind delegate outputs in exported program order
  6. QNN: never mix shared and client buffers in one graph execute #25345 QNN: never mix shared and client buffers in one graph execute
  7. Export hf_static float-cache programs and register their Python runner #24571 Export hf_static float-cache programs and register their Python runner
  8. LLM runner: validate and drive hf_static float-cache programs #24572 LLM runner: validate and drive hf_static float-cache programs
  9. Document the version-1 hf_static I/O contract #24573 Document the version-1 hf_static I/O contract
  10. portable: copy consecutive index_put_ rows in one block #25346 portable: copy consecutive index_put_ rows in one block
  11. LLM runner: never sample more logits than the model returns #25347 LLM runner: never sample more logits than the model returns
  12. hf_static v2: transposed key caches, full-length sliding caches, per-head attention #25348 hf_static v2: transposed key caches, full-length sliding caches, per-head attention
  13. LLM runner: drive hf_static v2 programs #25349 LLM runner: drive hf_static v2 programs
  14. QNN: export quantized hf_static programs in the NPU cache layout #25350 QNN: export quantized hf_static programs in the NPU cache layout
  15. Document the hf_static v2 NPU cache layout #25351 Document the hf_static v2 NPU cache layout

Each PR includes the commits of the ones above it; review only its own top commit.

Run each partitioner’s edge transforms after caller-supplied passes so backends can use the standard lowering entry point. QNN contributes its existing transforms through the new hook.
Reserve names for constant tensors already present in the exported program before lifting new constants, so methods with folded tables retain distinct constants.
Choose the registered IOManager and its logits, method names and step sizes, rejecting unknown contracts and invalid metadata. Load and reset the selected methods; route prompt tails to decode with a side-effect-free capacity check. Preserve module and I/O-buffer lifetimes and expose a backend-neutral allocator hook.
Route backend options through QnnPartitioner.from_options and register an allocator that supplies QNN shared-memory buffers through the generic IOManager hook. The LLM runner remains backend-neutral.
Encode graph output positions and restore their order at both runtime tensor-allocation entry points. Preserve context-binary order and legacy mutable outputs, and ignore malformed or overflowing position labels without throwing. This runtime change needs Qualcomm review.
On QNN SDK 2.48 an HTP execute that binds some tensors as registered rpcmem and others as plain client buffers fails (prefill DMA error 1100, decode transport error 1003) or produces wrong results; 2.50 and 2.51 runtimes run the same binding correctly. LLM runners place their caches, masks, positions and logits in rpcmem through the I/O buffer allocator, but delegate inputs and outputs that ExecuTorch plans internally (CPU-computed masks and RoPE tables, new K/V rows) stay in planned memory. When any tensor of an execute is shared, the backend now copies the remaining ones through persistent rpcmem staging buffers, one block per graph owned by the QnnManager and registered once, so each execute is all shared or all client memory. Executes without shared tensors are unchanged.

Authored with AI assistance (Metamate / agentcloud).
Move backend-neutral static-cache rewriting into the LLM extension and keep QNN partitioning and tuning in the backend. Detect sliding layers from model configuration, write every cache and counter in place on the caller buffer (export fails otherwise, since a missed write becomes full-cache CPU copies), and register a versioned runner that handles fixed chunks, padding, counters and reset. Quantized caches, sha attention and full-length sliding mode are excluded.

Authored with AI assistance (Metamate / agentcloud).
Use authoritative positions, float-cache version and dtype checks, exact buffer geometry and selected methods. Bind optional HF state outputs to existing buffers, preserve input aliases and register each concrete IOManager from its own implementation. Float-only fixtures cover reset, padding, malformed metadata and near-context tails.
Define fixed methods, cache metadata, padding, absolute counters and state-output bindings, and link the contract from the Qualcomm guide. State float-cache support and CPU-tested sliding correctness without claiming unmeasured device performance or a zero-copy Python host API.
A static-cache step writes its new rows at consecutive positions. With keys stored [1, H, D, L] each written element lands in its own position along the last dimension, so the in-place kernel issued one memcpy per element (917k per Qwen3 prefill chunk, 11.9 ms on a Galaxy S25). Rows bound for consecutive positions are now copied together, which keeps the result identical for any index order, duplicates included, and cuts that write to well under a millisecond.

Authored with AI assistance (Metamate / agentcloud).
get_llm_metadata replaced the program's get_vocab_size with the tokenizer's size, and the sampler reads that many logits per row. Gemma 3's tokenizer has 262145 entries for 262144 logits, so greedy decoding read one float past the logits buffer and crashed (SIGSEGV in Sampler::sample_argmax once the buffer ended on a page boundary). The runner now samples over the smaller of the two when the program reports its vocabulary.

Authored with AI assistance (Metamate / agentcloud).
…head attention

Contract version 2 lets a program store each key cache as [1, H, D, L] (get_key_layout = "transposed") and hold integer caches whose contents a runner treats as opaque. NPUs read keys as the scores' right-hand side in that layout; on a Galaxy S25 a quantized Qwen3-0.6B decodes 59.9 instead of 45.7 tok/s with it. The step writes its key rows along the last dimension, attention reads the cache without a per-step transpose, and the rest of HF's per-step signature is unchanged. Sliding-window layers may also take context-length caches with the window masked (specialize(sliding="full")), which needs no per-step shift and works with either key layout; window-sized caches keep HF's layout. A per-head attention form ("sha") writes out the single-head graph QNN's mha2sha makes of the concat form, for programs whose quantized K/V cannot be expanded for grouped-query heads, and the embedding can gather by int32 ids converted on CPU (the input stays int64). The Python runner accepts versions 1 and 2. Exporters keep writing version 1 until a backend opts in.

Authored with AI assistance (Metamate / agentcloud).
HfStaticIOManager accepts contract version 2: keys stored [1, H, D, L] when get_key_layout is "transposed", uint8/uint16 caches whose contents it treats as opaque (the program writes its rows in place, masked slots are never read, so a reset still zero-fills), and uint16 logits, which it dequantizes with get_logits_scale / get_logits_zero_point before sampling. Version 1 programs load and run as before; a version 2 program without a known key layout, whose cache shapes disagree with it, or with uint16 logits but no encoding, is rejected at load. The tiny decoder fixture gains a transposed-key variant that runs every generation test, and load-only probes cover integer caches, quantized logits and the new rejections.

Authored with AI assistance (Metamate / agentcloud).
With a quantizer, export_hf_static now produces contract version 2: every cache is stored as unsigned 16-bit with one encoding per cache across methods (the symmetric grid attention reads, offset by 32768), keys are stored [1, H, D, L], sliding layers take context-length caches with the window masked, and the logits leave HTP as uint16 with their encoding recorded for the runner. HTP then reads the caches in its native type and layout with no per-step conversion; the step's rows leave the delegate already in the cache's encoding, and the CPU write copies bytes (checked at export). Quantized prompt chunks attend per head, and the embedding gathers by int32 ids (an int64 id cast inside the graph makes HTP gather ~300x slower: SmolLM2 decodes 344 instead of 272 tok/s). On a Galaxy S25, quantized Qwen3-0.6B moves from 49.2 decode / 744 prefill tok/s with float caches to above Qualcomm's own runner's 80.7 decode (see the contract document for the measured steps). Float programs and `cache_io="float"` keep version 1. Sliding layers take context-length caches by default for float programs too (no per-step window shift: Gemma3-1B fp16 decodes 27.4 instead of 22.5 tok/s, 99% of Qualcomm's runner). BuildQuantIo also retypes the delegate's reference module so a CPU op may consume a quantized delegate output, and calibration can cap the tokens generated per prompt.

The quantized-cache encoding code is ported from the pre-split stack 2 branch (24ce101bb7).

Authored with AI assistance (Metamate / agentcloud).
The contract document adds version 2: which properties it carries (transposed keys, opaque integer caches with recorded encodings, context-length sliding caches, uint16 logits), the measurement behind each, what stays private to the program, and what was measured and left out. The Qualcomm backend guide notes that quantized exports now default to version 2.

Authored with AI assistance (Metamate / agentcloud).
@pytorch-bot

pytorch-bot Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25351

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 1 Unrelated Failure, 5 Unclassified Failures

As of commit 6fea0aa with merge base 765e973 (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 20, 2026
@winskuo-quic

Copy link
Copy Markdown
Contributor

Hi @psiddh,
Thanks for all the PR stacks.
Could you provide the instructions on how I could lower a QNN PTE from HF Exporter and run it on this branch?
Thanks.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants