Skip to content

Commit 3660bab

Browse files
committed
Document the hf_static v1 contract and runner boundaries
Define fixed methods, cache metadata, padding, absolute counters and state-output bindings, and link the contract from the Qualcomm guide. State float-cache support and CPU-tested sliding correctness without claiming unmeasured device performance or a zero-copy Python host API.
1 parent c87270e commit 3660bab

3 files changed

Lines changed: 109 additions & 0 deletions

File tree

‎docs/source/backends-qualcomm.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -545,6 +545,30 @@ cd $env:EXECUTORCH_ROOT
545545
.\qnn_executor_runner.exe --model_path .\deeplab_v3\dlv3_qnn.pte
546546
```
547547

548+
### Export a Hugging Face LLM
549+
550+
`executorch.backends.qualcomm.utils.hf_static` traces a Hugging Face decoder with HF's static-cache exporter and lowers it to QNN. The model can use a tuned quantization recipe while its cache I/O stays floating-point. The following is an export configuration example, not a device-validation or performance result:
551+
552+
```bash
553+
python -m executorch.backends.qualcomm.utils.hf_static \
554+
--model Qwen/Qwen3-0.6B --soc_model SM8750 --max_seq_len 1024 \
555+
--quantize --hybrid --output qwen3.pte
556+
```
557+
558+
Each step keeps HF's static-cache interface: token IDs, attention masks, position IDs and full K/V caches in; logits and the same caches updated in place out. The one adaptation is fixed shapes: fixed-length methods and a padding / counter protocol replace HF's single variable-length `forward`, because Hexagon Tensor Processor (HTP) execution needs static shapes. This adaptation is labeled `hf_static` v1.
559+
560+
`export_hf_static()` produces `decode` (one token) and optionally `prefill` (`prefill_len` tokens), sharing weights and cache buffers. The command above adds `prefill` with `--hybrid`. Cache rows are written by CPU kernels into runner-owned buffers. A runner must follow the [version-1 contract](llm-hf-static-io-contract.md), including fixed query lengths, padding rows in sliding masks, absolute counters and reset; matching HF's tensor order alone is not sufficient.
561+
562+
ExecuTorch's LLM runner uses `HfStaticIOManager` without a transformers dependency. HF's own Python runner can load these programs only with `transformers.exporters.register_io_contract`, which exists only in a local transformers branch for this work, not stock transformers. ExecuTorch guards that hook import with `try` / `except ImportError`: importing `executorch.extension.llm.io_contracts` registers the Python adapters only when the hook is available. Without it, registration is skipped and stock HF loading is not enabled. Python native bindings may clone returned outputs; a zero-copy Python host API is not claimed.
563+
564+
KV-cache policies, including eviction, long-context handling, reset and fork, belong in the runner's IOManager, not the QNN backend or exporter. This series implements fixed-shape execution and reset. Cache inspection and adopt / fork / evict hooks are follow-up work, not APIs provided by the current IOManager.
565+
566+
`hf_static` v1 uses float32 / float16 caches, even with quantized model computation. Auto attention uses split for decode and quantized computation, and concat for floating-point prefill.
567+
568+
For the transformers export used here, the Python branch that shifts HF's sliding cache once full is traced away. `hf_static` replaces those writes and determines sliding layers from model configuration. A prompt method must not exceed the smallest sliding cache length. Padding rows in a window mask are all-false, so the shift counts real tokens only. See the [counter and padding protocol](llm-hf-static-io-contract.md#positions-counters-and-padding), including the counter difference from HF eager state. These model window writes do not supply a general runner eviction policy.
569+
570+
**Validation boundary:** CPU tests provide sliding-window correctness evidence only. They do not establish sliding-window device support, longer-context device support or device latency. Device behavior requires separate validation. Quantized caches, `sha` prefill and full-length sliding-cache mode are held back from this series.
571+
548572
### Run [Android LlamaDemo](https://github.com/meta-pytorch/executorch-examples/tree/main/llm/android/LlamaDemo) with QNN backend
549573

550574
`$DEMO_APP` refers to the root of the executorch android demo, i.e., the directory containing `build.gradle.kts`.

‎docs/source/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,5 +179,6 @@ llm/working-with-llms
179179
advanced-topics-section
180180
tools-section
181181
api-section
182+
llm-hf-static-io-contract
182183
support-section
183184
```
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Hugging Face static-cache I/O contract
2+
3+
**Version:** 1 (`get_io_contract = "hf_static"`)
4+
5+
## TL;DR
6+
7+
Each step keeps Hugging Face's static-cache interface: token IDs, attention masks, position IDs and full K/V caches in; logits and the same caches updated in place out. The runner owns the buffers; the program writes into them.
8+
9+
The one adaptation is fixed shapes. Instead of HF's single variable-length `forward`, ExecuTorch exports fixed-length `prefill` / `decode` methods and requires the padding / counter protocol below, because Hexagon Tensor Processor (HTP) execution needs static shapes. This adaptation is labeled `hf_static` v1. `decode` is required and `prefill` is optional; the program is not a drop-in replacement for an arbitrary HF runner.
10+
11+
## Goals / Non-Goals
12+
13+
This contract defines the inputs, outputs, metadata and runner responsibilities for batch-one float-cache generation. It does not define runtime-owned dynamic caches, cache-state import / fork, quantized caches, or arbitrary query lengths. CPU tests provide sliding-window correctness evidence only; they do not establish sliding-window device support, longer-context device support or device latency.
14+
15+
## Methods and metadata
16+
17+
`decode` takes one token. An optional `prefill` method takes a fixed `N` tokens per call. Both methods share the same cache shapes, dtypes and buffers. If `prefill` is absent, prompts run through `decode`.
18+
19+
Required metadata methods:
20+
21+
| Method | Value / meaning |
22+
|---|---|
23+
| `get_io_contract` | `"hf_static"` |
24+
| `get_io_contract_version` | `1`; a version-1 runner rejects newer versions |
25+
| `get_max_seq_len` | Maximum total context length; not necessarily the longest physical cache when all layers slide |
26+
| `get_n_layers` | Number of key / value / counter triples |
27+
| `get_cache_dtype` | `"float32"` or `"float16"`, matching every cache input in every method |
28+
29+
The exporter also records model vocabulary / BOS / EOS metadata when available and `use_kv_cache = True`. Runners validate contract metadata against method tensor metadata before allocating or writing buffers. Version 1 has no per-cache quantization scales or zero points.
30+
31+
## Inputs and outputs
32+
33+
Inputs are flattened in HF's exported order:
34+
35+
1. `input_ids`: int64 `[1, N]` (`N = 1` for decode).
36+
2. One or more bool attention masks: `[1, 1, N, W]`, in the exported order.
37+
3. `position_ids`: int64 `[1, N]`.
38+
4. Per layer, in layer order: key cache `[1, H_kv, L_i, D]`, value cache of the same shape and dtype, and a scalar int64 counter.
39+
40+
Full-attention cache length `L_i` is the context length. A sliding-attention layer keeps `min(sliding_window, context_length)` rows. Sliding layers are determined from model configuration / HF cache layer types, not cache-size comparisons or mask names. Every layer's attention must be rewritten; an incomplete match is an export error.
41+
42+
Cache and counter state must be unplanned CPU inputs in the contiguous layout the runner allocates. The Python loader rejects memory-planned or device-resident state: runtime-owned copies would otherwise lose updates to the caller's buffers. Its public `TensorInfo` metadata does not expose dimension order, so externally produced programs must have their contiguous cache layout checked at export.
43+
44+
The exported Python module returns logits followed by the cache / counter state. The low-level ExecuTorch method prepends `3 * n_layers` user-input mutation outputs, then exposes logits at index `3 * n_layers`. The mutation prefix is opaque: its internal order need not match input order (for example, counters can precede K/V). Mutation-prefix outputs write into their input buffers.
45+
46+
Real HF graphs can also return the original `3 * n_layers` state tensors again after logits. Runtimes accept either valid layout and do not treat that repeated state as a second owned cache. The optional ordered state suffix may contain unplanned functional outputs (notably counters); the C++ runner binds each unplanned suffix output to the corresponding existing state buffer.
47+
48+
The C++ IOManager and registered Python host driver consume the fixed logits index and retain their existing state buffers. Python native bindings may clone returned outputs, so callers must not infer program aliasing from returned Torch `data_ptr()` values. Those binding-owned copies are separate from the driver's persistent cache buffers and from the exported graph. The Python host API is not claimed to be zero-copy.
49+
50+
In a padded chunk, Python may mark all padding rows false; the window-mask marking is mandatory, while unused full-attention padding rows can remain causal as in the C++ driver. Logits are floating-point `[1, N, vocabulary]`. For a padded prompt call, sample from row `real_tokens - 1`, not the last padded row.
51+
52+
## Positions, counters and padding
53+
54+
The runner's `start_pos` is the source of truth. Before every call, set every counter to `start_pos` and supply positions `start_pos .. start_pos + N - 1`. Real tokens must fit the total context. A fixed-length padded call must also fit physically; near the context boundary, run the remaining real tokens through `decode` instead of rejecting a prompt that fits.
55+
56+
A prompt chunk with `r < N` real tokens repeats the last real token in its padding. Full-attention masks are causal by absolute position. In a sliding-window mask, each real query row must have at least one true entry and every padding row must be all-false. The program counts nonempty rows to determine `r`; this is part of the fixed-shape adaptation, not standard HF runner behavior.
57+
58+
The program may advance counters by the static query length `N`, including padding. The runner must restore the logical count to `start_pos + r` after a padded call, or overwrite all counters with the next logical `start_pos` before execution. A reset returns the logical position and counters to zero. Unwritten cache slots need not contain zeros because masks exclude them.
59+
60+
Sliding caches retain the most recent rows in chronological order. When a call crosses the window, the program shifts the retained rows and writes the real token rows on CPU. `prefill_len` must not exceed the smallest sliding cache length. Padding writes must use the last real row's data whenever they share its destination, so correctness does not depend on duplicate-index write order.
61+
62+
HF's eager `StaticSlidingWindowLayer` may stop advancing its tensor `cumulative_length` once full, while its Python bookkeeping tracks the absolute position. This program's counters continue advancing as absolute positions. Cache state cannot be handed between HF eager and this contract by copying counters alone.
63+
64+
## Runners and boundaries
65+
66+
ExecuTorch's C++ `HfStaticIOManager` and the registered Python `hf_static` runner implement this protocol. The C++ IOManager does not require transformers. The Python runner declares `drives_generation = True` for HF's `ExportedGenerator` contract hook.
67+
68+
HF's own Python runner can load these programs only when transformers provides `transformers.exporters.register_io_contract`. For this work, that hook exists only in a local transformers branch, not stock transformers. ExecuTorch guards the hook import with `try` / `except ImportError`. Importing `executorch.extension.llm.io_contracts` registers the adapters only when the hook is available; otherwise registration is skipped, and stock HF loading is not enabled.
69+
70+
Generation takes one unpadded sequence with standard absolute positions; caller-defined mask / position semantics are not supported. The runner's own fixed-chunk padding is supported.
71+
72+
Cache allocation is backend-neutral. An optional I/O allocator can place buffers in QNN shared memory. Graph rewriting and CPU writes live in the LLM extension; QNN partitioning and HTP tuning remain in the Qualcomm backend. This allocation interface does not establish device support or zero-copy behavior for every backend / runner combination.
73+
74+
KV-cache policies, including eviction, long-context handling, reset and fork, belong in the runner's IOManager, not in QNN partitioning or the model exporter. Version 1 implements buffer ownership, the fixed-shape protocol and reset; it does not provide arbitrary context extension or cache-state transfer from HF eager execution. The program's sliding-cache shift / write implements the exported model's window semantics, not a general runner eviction policy.
75+
76+
Cache inspection and adopt / fork / evict hooks are follow-up work and are not exposed by the current IOManager. They need explicit state-ownership semantics and an appropriate contract version rather than silently changing version 1.
77+
78+
## Version history
79+
80+
| Version | Definition |
81+
|---|---|
82+
| 1 | Batch-one fixed-length decode / prefill, float caches, in-place state mutation, sliding-mask padding protocol |
83+
84+
Experimental quantized caches, `sha` attention and full-length sliding-cache mode are excluded from this contract and the upstream series.

0 commit comments

Comments
 (0)