The prefix cache accelerates multi-turn conversations by snapshotting the KV cache state at turn boundaries. On subsequent requests that share the same system prompt / early conversation, the server restores from a snapshot instead of recomputing the full prefill — saving both latency and compute.
Request 1: [system + user1 + assistant1 + user2]
↑ boundary — snapshot here
Request 2: [system + user1 + assistant1 + user2 + assistant2 + user3]
└── restore snapshot ──────────┘ └── diff-prefill ──┘
┌─────────────────────────────────────────────────────────┐
│ HTTP Server │
│ • Tokenizes prompt │
│ • Calls PrefixCache for lookup / prepare │
└────────────────────────┬────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ PrefixCache (LRU logic) │
│ • detect chat boundaries via ChatMarkers │
│ • SHA-1 hash prefix at each boundary │
│ • LRU eviction when cap is reached │
│ • Two RAM pools: inline prefix + exact prefill │
└────────────────────────┬────────────────────────────────┘
│ snapshot_save / restore_and_generate
▼
┌─────────────────────────────────────────────────────────┐
│ ModelBackend (per-arch) │
│ • snapshot_save(slot): copy KV cache → snap_backend_ │
│ • restore_and_generate(slot, req): snap → KV + decode │
│ • snapshot_free(slot): release snapshot memory │
└─────────────────────────────────────────────────────────┘
Caches KV state at turn boundaries within a conversation. The boundary
detector uses ChatMarkers to find end-of-message + start-of-next-role
token sequences. DeepSeek's template closes only assistant turns with an end
marker (the system text and user turns end where the next role starts), so
for that family every role marker is a boundary: the system text, each
completed turn, and the generation prompt. A first turn therefore snapshots
its system prompt, and a new session on the same system prompt restores it.
For tool-using chat templates, tool definitions are rendered in the system
prefix. The first safe boundary therefore includes the complete tool schema:
the first request pays that prefill once, and later requests with the same
tools restore it through the normal native cache. See
TOOL_PREFIX_CACHE.md for the correctness contract and
benchmark.
- lookup(): Finds the longest cached prefix matching the current prompt.
- prepare_inline_snap(): Selects a slot and cut-point for snapshotting after the current prefill completes.
- confirm_inline_snap(): Commits the entry after successful save.
- LRU eviction: oldest entry's slot is reused when capacity is reached.
Caches the entire prompt's KV state in RAM, keyed on the tokens served: the raw prompt, or a compressed prompt's served tokens. A repeated PFlash request is served the same tokens by its view without a drafter pass, so a hit skips the prefill entirely -- the fastest path for repeated prompts.
- Separate slot pool starting at
cap(inline slots are[0, cap)). - Keyed on SHA-1 of the full served prompt (not just a prefix).
The two flags are intentionally separate:
--prefix-cache-slotscontrols turn-boundary prefix snapshots.--prefill-cache-slotscontrols exact full-prompt snapshots.
The backend can save one snapshot during a generation. For requests carrying tools, the reusable inline system/tool boundary takes priority over an exact full-prompt snapshot; for ordinary requests, the exact cache keeps priority. If the preferred cache has no useful boundary or available slot, the server falls back to the other tier. A restore only reserves a new inline snapshot when the selected boundary advances beyond the restored prefix.
The disk prefix cache is a separate persistence/overflow layer for token-keyed
snapshots. Today it is integrated with the inline/effective-prompt path; exact
prefill snapshots are kept in RAM unless a dedicated raw-prompt disk path is
added. With the default full policy a lookup probes the whole prompt, every
chat boundary and the deepest stored entry that prefixes the prompt, deepest
first, so a restart recovers the sessions the previous process persisted
(probes are index lookups; only a hit reads a file). See
Disk Session Cache.
The disk session cache keeps agent and chat sessions on an SSD, so a session whose RAM snapshots are gone (another session evicted them, the server restarted, the user switched model) restores its KV and prefills only the new suffix. It is on by default.
| Flag | Default | Meaning |
|---|---|---|
--kv-cache-dir <path>|off |
$XDG_CACHE_HOME/lucebox/kv-cache, else ~/.cache/lucebox/kv-cache |
Directory of the cache; off (or --no-kv-cache) disables it. |
--kv-cache-budget <MB> |
131072 (128 GB) |
Hard cap on the bytes of all snapshot files, clamped to the free space. |
--kv-cache-ttl <hours> |
72 |
Delete files unused for this long; 0 = never. |
--kv-cache-shutdown-flush <s> |
120 |
Longest a graceful shutdown spends writing RAM snapshots; -1 = no limit. |
--kv-cache-cold-max <N> |
10240 |
Longest head kept when a deeper state of its session is written. |
When something is written. Only when a snapshot leaves RAM, never after a turn:
- a RAM eviction: a new snapshot takes the slot of an older one (its reservation's victim), the resident budget evicts several to make room for a large one (see Several evictions for one capture), or it evicts one after a commit;
- a graceful shutdown (SIGTERM/SIGINT): one last eviction of everything still
in RAM, most recently used first, for at most
--kv-cache-shutdown-flushseconds. The log line[disk-cache] shutdown flush:gives the sessions, files, bytes and seconds, and what the limit dropped.
A crash, kill -9 or power loss loses what was only in RAM; the states written
before it still restore. An active session therefore writes nothing while it
stays resident: a 90-120K-token Qwen3.8-27B session with q4_0 KV is 1.5-2.2 GB
per snapshot.
What is written. Since #835 every request snapshots the end of its last message, a new turn's snapshot keeps the previous turn's one while the budget holds it (an edit of the last message resumes there), and the turns before it are pruned. On eviction (and at shutdown) the disk gets:
- each conversation's newest snapshot: the inline snapshot of the last
request, or the agent-turn state with the generated reply (tool path,
--agent-turn-cache), on the single-sequence and the concurrent (paged) paths alike; - the system/tools heads (protected pins and the shallowest entries).
It does not get the kept previous turn (an edit backup) while a deeper
snapshot of its conversation is resident (PrefixCache::persist_on_evict):
it only serves an edit of the last message, which the deeper snapshot's file
cannot, and writing it would double the bytes per conversation. Once the
deeper snapshot is gone, the kept turn is that conversation's newest and is
written like any other. Pruned (superseded) turns are not written either.
When a session's state is written, its shallower files on disk are deleted,
except a head at or below --kv-cache-cold-max tokens or, without one, the
shallowest (a longer head).
Compressed prompts are not raw history and get no disk key,
except a prompt served from a PFlash unified view: its
key is the tokens it served, and the view itself is kept on disk too, so the
conversation's next request serves the same tokens again and restores them. A
request whose disk scope is not full gives its snapshots no key either.
Shortened conversations. With the disk cache on, the unified view keeps its
views (what each shortened conversation served, and which lines it left out) in
<kv-cache-dir>/.pflash-views: a view is written when it leaves the RAM view
store and at shutdown, and read when a request finds no view in RAM, so neither
a restart nor an eviction turns a shortened conversation cold. One file per
view, a one-line header indexed at startup; at most 256 views, the least
recently written dropped first, and the cache's TTL applies, counted from a
view's last write and checked at startup and at every lookup (a view in use
returns to RAM and is written again when it leaves). Views written for another
model, template or context (the cache's identity salt), or under older paging
rules (a version in the same identity), are never read. A view
file that does not load, or whose offsets and per-line and per-unit data do not
agree, is deleted and the request proceeds without it. A conversation served
raw after a PFlash failure loses its views in RAM
and on disk, so no later turn continues them.
The drafter's own scoring state stays in RAM only: after a restart, the first
question that needs recall scores the earlier messages again.
How it is written. Eviction hands the snapshot's buffer to the disk writer
without a copy (Qwen keeps snapshots in host memory; other backends copy to
host first) and the request continues at once. The snapshot is indexed
immediately, counts against the budget, and a lookup serves it from memory
until its file exists. A writer thread writes <file>.tmp, fdatasyncs it,
renames it into place and syncs the directory. Queued writes are bounded
(4 GiB of payload); past that the evicting request waits, logged as
[disk-cache] waited.
Restore. With the default full policy a lookup probes the whole prompt,
every chat boundary and the deepest stored key that prefixes the prompt
(index lookups; only a hit reads a file). A disk state deeper than the best
RAM hit wins over it: a session evicted to disk usually still matches the
shared head in RAM. If that read fails, the RAM hit is used. The single-sequence path reads the
file on the request that restores it (the only request running). The
concurrent scheduler reads it on a reader thread while the request waits at
the head of the queue and the live streams keep decoding, then restores it
into the new sequence's pages through a staging checkpoint. A hit's state is
bit-identical to the RAM snapshot it was written from (raw bytes in the
cache's own type; q4_0 stays q4_0).
A session's prompt is rendered with the exact text the model generated for its earlier tool calls (ToolMemory, which a new process does not have). Each file stores that replay text and startup loads it back into ToolMemory.
Budget. Every layout directory under the cache directory is indexed, also
files of another model, KV type or server, so the cap covers the whole
directory. The budget never exceeds the cache's bytes plus the free space
minus a reserve (the larger of 1 GiB and 5% of the disk, at most 16 GiB).
When a write is queued, this server's least recently used files are deleted
until it fits; before it is renamed into place, the writer merges the
directory (see Several servers) and deletes the least recently used files
of any model until the directory fits. A hit rewrites the file's last_used
field in place, so LRU order survives restarts and other servers see it.
TTL. A file unused for longer than --kv-cache-ttl hours is deleted at
startup, before each write, and when a lookup meets it.
Safety.
- Format v5 ends with an XXH64 checksum of the whole file (the header with
last_usedzeroed). A hit verifies it while reading: a corrupt or partial file is a logged miss, it is deleted and the prompt is prefilled. Startup deletes leftover.tmpfiles nobody writes, files of older formats and truncated files. - A file is restored only when its identity matches: model path, size, mtime
and a content fingerprint (SHA-1 of the first and last MiB), KV cache types,
K rotation,
--max-ctx, chat template and draft model, plus the snapshot's tensor names, types and shapes. Qwen checks every adopted tensor against the live cache (the paged restore against the page table too). - Each file stores the exact key token ids; a hit compares them with the prompt.
- The models of one process share the directory's store (
DiskStore: one index, one writer, one budget and TTL), each restoring only the files its own identity wrote. Other servers can share the directory too (below).
Several servers. Servers on one machine share one cache directory, one budget and one TTL (POSIX; on Windows the first server keeps the directory and the others run without a disk cache):
- Each holds a shared lock on
<dir>/.lockfor its lifetime. An older server, which takes it exclusively, keeps newer ones off, and is kept off by them (both log it). - Each model identity is served by one server at a time
(
<dir>/.identity-<salt>.lock): its files, pins file and views are its alone, so no two servers write the same file. A second server of the same model and settings runs without a disk cache and logs it. - Before a write is renamed into place, the writer thread takes
<dir>/.index.lockand merges the directory into its index: the files other servers wrote, the ones they deleted, and theirlast_usedtimes. The TTL and the budget then apply to the whole directory, and the file is renamed. Only the writer thread deletes other servers' files, right after that merge; a request never waits on another server. - A file another server deletes while it is being read stays readable to
the end; one deleted earlier is a miss (
is gone) and leaves the index. - A temporary file is locked while it is written, so a server starting up removes only temporaries nobody writes.
- Each server keeps the whole directory within its own
--kv-cache-budgetand--kv-cache-ttl: with different values, the smaller one wins once that server writes. Queued writes count for the server that queued them only, so the directory exceeds the budget by at most the files being written at that moment.
Without RAM slots. With --prefix-cache-slots 0 nothing leaves RAM, so
nothing new is written; the entries already on disk are still served. Only
the fixed:N scope (--disk-prefix-cache N) keeps its scoped-prefix write.
The auto scope and the cold-head and continued checkpoints this mode used
to write are retired; auto and auto:N read as full, and
--kv-cache-interval is accepted and ignored.
Engines. Concurrent engines keep their checkpoints in the backend's
snapshot slots (Qwen paged, qwen4exp), where the disk cache writes and
reads them. A capture keeps the checkpoint it replaces aside until the
scheduler claims it: written when worth keeping, freed otherwise. A backend
whose snapshots are not plain ggml tensors packs them itself
(ModelBackend::snapshot_exports): qwen4exp stores its snapshot buffer as
one tensor and its host fields (positions, tokens, draft state, logits, each
strip's place) as another, hands a snapshot in system memory to the writer
without a copy, and keeps a disk read in place.
Not covered. Single-sequence --paged-attention keeps no snapshots, and
the concurrent DeepSeek4 engine keeps its checkpoints outside the backend's
snapshot slots: both run without a disk cache (logged at startup).
Cache-aware admission. PFlash's admission counts the deeper of the RAM
prefix snapshot and the deepest disk entry for the prompt
(DiskPrefixCache::longest_prefix, an index lookup that reads no file; ties
go to RAM). A conversation restored after a restart, or written out
on eviction, is then cached history: the view serves it raw, from disk,
and pages only new long content. The disk counts only with --kv-cache-dir
set and the request's disk scope full. With the disk cache on, the view's
stats carry a cached object with the tokens and source.
A shared prefix is what a client repeats at the start of every request: its
tool schemas and system prompt, up to the first user turn, or up to where its
projects differ. --shared-prefix auto (the default) keeps them across
conversations, and with the disk cache on across restarts and model swaps
(server/kv/shared_prefixes, server/kv/prefix_pins); with
--kv-cache-dir off the pins and the ring live in memory only.
The cut. A request with tools, or whose prefix before the first user turn
is at least 512 tokens, captures that head first, until it is restored. When
it agrees with the prefix region of one of the last 8 such requests (the
ring) on at least 512 tokens, it captures where they agree instead: at the
deepest chat boundary inside the agreement, else mid-message, backed off to
the end of the last line. Both serving paths cut the same way. A learned cut
yields to the deep-first rule like the head does. (This was PPP's pin-end
annotation; LUCE_PPP=0, LUCE_PPP_LCP_WINDOW and LUCE_PPP_MIN_PIN_TOKENS
still work for one release, with a warning.)
Pins. A shared-prefix entry remembers the prompt that captured it, up to its last message. A restore by a prompt that does not extend that one comes from another conversation (a later turn of the capturing conversation extends it) and pins the entry: at once when its cut is a chat boundary, inside a message only once the cut recurs in 3 of the ring's regions.
- Its KV is written to disk after the response (single-sequence) or when the scheduler goes idle. The file is marked pinned in its header, so the TTL, the LRU and ancestor pruning skip it, also after a restart. The budget never deletes a pinned file: a save that would need its room is skipped. Two cuts saved at one chunk start share a file, which stays pinned while any pin lists it; the mark is set once the catalog lists the pin.
- The pin is listed in
<kv-cache-dir>/pins/<tokenizer+template id>.json, with the ring, so pins and cuts survive a restart. The file is written to a temporary, fsynced and renamed (and the directory fsynced). The models of one process with the same token identity share it, each with its own disk file per pin (up to 3 layouts per pin, the least recently hit dropped). - An engine without a disk tier (DeepSeek's concurrent engine) pins the token ids only, and warms them after a restart (below).
- Under
auto, a disk hit on a pinned file counts as the pin's hit, so a head served only from disk does not expire (freezecounts no hits). - Caps: 16 pins and 16 GiB of files per catalog, least recently hit first; a pin with no hit for 14 days is dropped, and its files fall under the TTL again.
GET /v1/cache/shared-prefixeslists the pins (id, tokens, source, client, hits, last hit, files);DELETE /v1/cache/shared-prefixes/{id}drops one./propsreportsshared_prefixes{mode, pins}.
Startup and warming. At startup a pin with a disk file for the current
settings is served by disk hits, like any file. A pinned file of this layout
that no pin lists (a crash between the mark and the catalog's save) is
unpinned. A pin that does not fit in --max-ctx is kept for settings that
can serve it.
On the concurrent path, the pins with no file for these settings (an engine
without a disk tier, or after a --max-ctx, KV type, draft or model change)
are warmed once the scheduler has been idle for 5 seconds (a client
reconnecting right after a restart comes first): the pin's prompt, plus one
token, is planned, admitted and captured on the engine alone, as a request's
prefill is, and retires at the capture.
- Into RAM only while the warmed pins fit min(one checkpoint at
--max-ctx, the prefix budget less two of them), and never in place of another pin. - A request arriving during a warm waits for the engine step in progress. That can be a whole head: DeepSeek's staged prefill runs up to 2048 rows in one step (about 36 s for 1.75K tokens with experts streamed from SSD). The warm is then preempted and dropped: each pin is tried once per process.
- A pin deleted during its warm stays deleted.
The single-sequence path warms nothing: after a settings change its first request prefills the head, and the head is pinned again for the new layout.
What a hit guarantees. A warmed checkpoint is the one a request's own
capture at the same cut saves (the same xxh64 on Qwen; on DeepSeek a warmed
hit's output is the captured hit's), and a hit from it gives what a hit from
a captured checkpoint gives. A hit is not
always byte for byte a cold run: where a capture or a restore splits the
prefill (DeepSeek on both paths, Qwen's concurrent engine), the numerics of
the split differ by the model's rounding noise, on main too. Under auto
the learned cut moves where prefill is split, so on DeepSeek a request's
output can depend on the requests before it; freeze holds the cuts still.
freeze serves the known pins (and warms them) but learns nothing: the ring,
the pins and their hits stay put (none is added, none expires), so cuts don't
move during a benchmark. off is plain prefix caching.
Naive snapshots stored full max_ctx KV tensors regardless of actual cache occupancy. For short prefixes this was extremely wasteful:
| Model | max_ctx | Full snapshot | Right-sized (29 tokens) |
|---|---|---|---|
| Qwen3.5-27B (Q8_0 KV) | 64000 | ~1.5 GB | ~0.17 MB |
| Gemma4 | 32768 | ~0.5 GB | ~0.05 MB |
| Laguna | 16384 | ~0.3 GB | ~0.03 MB |
With 3 inline snapshots at cur_pos=29,138,265 the old code allocated ~4.5 GB of GPU memory (on a 22 GB card already holding ~17 GB for model+cache), causing spill to system RAM and 5× decode slowdowns.
Two complementary fixes:
-
Right-sized allocation: KV tensors are allocated as
[head_dim, cur_pos, n_head_kv]instead of[head_dim, max_ctx, n_head_kv]. This reduces per-snapshot memory from ~1.5 GB to a few MB for typical prefix lengths. -
Platform-aware backend: Snapshots are stored on system RAM (CPU backend) for discrete GPUs, keeping VRAM free for model weights and active cache.
// Right-sized KV allocation in snapshot_target_cache():
ggml_tensor * K = ggml_new_tensor_3d(snap.ctx, sk->type,
sk->ne[0], snap_pos, sk->ne[2]);
// snap_pos = cache.cur_pos (e.g., 29 instead of 64000)Buffer reuse: When the same slot is saved at the same cur_pos, the
existing buffer is reused (no free+alloc). Only when cur_pos changes is
the buffer freed and a new (still tiny) one allocated.
Strip copy: Since right-sized KV tensors have different ne[1] than the
full-size cache, save/restore uses per-head strip copies via
ggml_backend_tensor_get/set — the same pattern as thin snapshots.
Snapshots are stored on a snapshot backend selected at init time based on the compute backend's memory characteristics:
// common/snapshot_backend.h
ggml_backend_t create_snapshot_backend(ggml_backend_t compute_backend);
void free_snapshot_backend(ggml_backend_t snap, ggml_backend_t compute);Decision logic:
compute_backend's default buffer type is host-accessible?
├── YES (unified memory: Metal, AMD iGPU/HALO)
│ └── return compute_backend (no copy needed, same physical RAM)
└── NO (discrete VRAM: CUDA, HIP)
└── return ggml_backend_cpu_init() (system RAM, off-GPU)
Platform behavior:
| Platform | GPU type | Snapshot storage | Copy cost |
|---|---|---|---|
| CUDA (discrete) | RTX 2080 Ti, etc. | System RAM (CPU backend) | GPU↔CPU via PCIe at save/restore |
| Metal (Apple Silicon) | Unified | Same as compute (no-op) | Zero (same memory) |
| AMD HALO / iGPU | Unified | Same as compute (no-op) | Zero (same memory) |
| HIP (discrete) | RX 7900, etc. | System RAM (CPU backend) | GPU↔CPU via PCIe at save/restore |
Right-sized snapshots use ggml_backend_tensor_get/set with explicit
offsets for KV tensors (since source and destination have different shapes).
SSM/conv state (fixed-size) uses ggml_backend_tensor_copy() directly.
Each backend adds a single member and three code points:
// Header:
ggml_backend_t snap_backend_ = nullptr;
// init():
snap_backend_ = create_snapshot_backend(compute_backend_);
// snapshot_save(): right-sized alloc + partial copy
// KV: [head_dim, cur_pos, n_head_kv] on snap_backend_
// SSM/conv: full-size on snap_backend_
// target_feat: [fc_in, min(cur_pos, cap)] on snap_backend_
// shutdown(): free in correct order
for (auto & s : snapshots_) free_snapshot(s); // free tensors first
free_snapshot_backend(snap_backend_, compute_backend_); // then backend| Server flag | Default | Description |
|---|---|---|
--prefix-cache-slots N |
32 | Max turn-boundary prefix cache slots |
--prefix-cache-max-mib auto|N |
auto | Resident RAM limit for single-sequence prefix snapshots; auto keeps room for three snapshots at --max-ctx, capped at a quarter of the memory available at startup (a readable cgroup v2 container limit is included; otherwise total physical RAM is used); 0 is unlimited. Enforced for Qwen and DeepSeek4 (not DeepSeek4 mixed-backend splits): other backends cannot size snapshots, so auto stays unlimited there and an explicit limit is rejected at startup |
--concurrent-prefix-cache-max-mib N |
auto | Resident RAM limit for copied concurrent paged checkpoints; auto = 2 x --max-concurrency + 1 checkpoints at --max-ctx, at least 4096 MiB; above that, at most 1/4 of available memory; 0 is unlimited |
--prefill-cache-slots N |
0 | Max exact full-prompt prefill cache slots |
--skip-park |
false | Skip parking draft model during compress |
With right-sized, CPU-resident snapshots the limiting resource is system RAM, not VRAM. A snapshot of a hybrid model also copies the full recurrent state, so the fixed part dominates short prefixes. For Qwen3.5/3.6-27B with q8_0 KV and a DFlash draft, one snapshot is about 34 KiB per token plus about 350 MiB fixed: 0.7 GiB at 10K tokens, 1.0 GiB at 20K and 3.5 GiB at 94K.
Every chat boundary (and every extra cut a cache may take) starts a prefill
chunk, in a cold prefill and after a restore alike (restore_points). A
snapshot therefore lands exactly on the boundary it was requested at, and a
restored prefix plus the suffix prefill reproduces a cold prefill. Apart from
the last chunk of the prompt, Qwen never ends a chunk within 16 tokens of its
start, or 64 after restoring a generated-turn checkpoint, so the cache does not
request a snapshot that close to the restored prefix. The
generation prompt's own boundary starts a chunk only when the request
snapshots there, since no saved state lies past it otherwise.
After a tool-call turn the server also keeps the state the generation left behind, keyed by the prompt plus the generated tokens the next request renders identically (see Agent Turn Cache). Qwen defers that snapshot: while the next request continues the conversation it is never copied to system RAM, and it is copied out only before other work would overwrite the live state, or when it is evicted to disk. Restoring the snapshot the live state already holds copies nothing back either.
Coding agents grow one conversation turn by turn, and each turn's snapshot is a strict prefix of the next. After committing a snapshot, the single-sequence path frees the ancestors it supersedes, keeping only the shallowest one (the shared system/tools head) and protected tool pins. A conversation therefore holds about two snapshots instead of one per turn.
State backups. A superseded turn repeats the per-position rows of the
snapshot that supersedes it; only the state of its own position is its own.
Where the backend can drop a snapshot's rows, the superseded turns are
therefore kept instead of freed, as state-only backups. A prompt that
departs from the conversation before its end (an edited or retried last
message, a second request from the same point, a client that drops its last
turns) restores the deepest backup it still extends, with the rows of the
newer snapshot ([pc] state backup slot=N ... restores with the rows of slot=M), and prefills only what follows.
The policy is the same for every backend and both serving paths
(KvCache::trim_superseded). Backups thin out with distance: the previous
turn is kept, then one at least twice as far behind the newest snapshot as the
last one kept, and so on, so a conversation holds a number that grows with the
logarithm of its length and a prompt that departs d tokens back resumes within
about 2d of its end. A backup is never written to disk and leaves RAM with the
last snapshot whose rows can restore it. Under the resident limit, the backups
of the conversation being served are evicted after the leaves of the other
conversations, not before.
A backend takes part by saying what one backup holds and by dropping and
borrowing rows (ModelBackend::snapshot_state_bytes_estimate,
snapshot_drop_kv, GenerateRequest::restore_kv_slot; for concurrent
serving SeqEngine::estimate_prefix_state_bytes, drop_prefix_kv,
PrefixStorePlan::restore_kv):
| Backend | A backup keeps | Size | Rows it borrows |
|---|---|---|---|
| Qwen3.5/3.6/3.8, MoE variants | recurrent and conv state, drafter features | about 370 MB (27B) | attention K/V |
| Qwen3.8-Flash-Next | recurrent, conv and PLE state, draft hidden state, logits | about 118 MB | attention K/V, indexer keys and pooled blocks, draft-layer K/V |
| DeepSeek V4 / V4.1 | raw window, compressor windows, HC state, logits, features | tens of MB | compressed and index rows |
Single-sequence and concurrent serving both take part for these three. They
rely on a prefix's rows not depending on how it was reached, which holds
because these backends start a prefill chunk at every restore point
(GenerateRequest::restore_points). Not covered, and unchanged: a KVFlash pool
(Qwen keeps state-only rungs of its own beside the live cache there), layer
split and cluster serving, and the other backends, which keep the previous
turn whole while the limit holds it. Chains that are abandoned,
for example after a client compacts its context, are bounded by
--prefix-cache-max-mib: when a new snapshot would not fit, the cache replaces
the oldest leaf first. When no single entry makes enough room, it evicts several,
least recently used first (see below), and skips the capture only if nothing
can make room or the snapshot is larger than the whole limit. Snapshots
the new one supersedes (including the one it restored from) count as freed,
because they are pruned as soon as it is committed; a failed capture keeps them.
After each commit the cache also evicts, oldest leaf first, until the committed
snapshots fit the limit again (never evicting the new snapshot or a protected
tools pin), so a capture that lands shorter than it reserved does not leave the
cache over budget. The limit covers committed snapshots: the
new snapshot is allocated before the ones it replaces are freed, so while a
request runs, process memory can exceed the limit by up to one snapshot.
Concurrent paged serving measures the exact backend allocation required for
each checkpoint before copying it. The cache keeps committed checkpoints under
--concurrent-prefix-cache-max-mib: when necessary it replaces one eligible least-recently-used
entry, and when no single eligible entry can make enough room it evicts several
(below). The configured limit covers resident committed checkpoint buffers.
During an atomic replacement, the new buffer and the entries it replaces can
coexist briefly, so transient process memory can exceed the limit by up to one
checkpoint.
Several evictions for one capture. Both paths use the same rule. A capture
that fits the limit at all, but that no single entry can make room for, plans
several evictions, least recently used first: edit backups (not written to
disk while their deeper snapshot is resident), then leaves. The plan never
takes a protected pin, a head that other resident entries extend, the
capture's own ancestors or restore source, or an entry another capture in
flight writes, evicts or counts as freed. It is all or nothing: it exists only
if the capture fits once those entries are gone, otherwise the capture is
skipped and nothing is touched. The entries leave RAM only when the capture
commits (the single-sequence path writes and frees them just before the
commit), so a capture that fails or is cancelled evicts none of them; each one
goes to the disk cache like any other eviction. Without this a long session
that arrives after many short ones (subagent threads) was never captured, so
it never reached the disk either. A checkpoint larger than the whole limit is
still skipped (budget_skips).
By default the limit is sized when the scheduler starts, from the batch
engine's own estimate of one checkpoint at --max-ctx: every slot keeps its
conversation's restore point and the capture in flight, and all slots share the
system/tools head, so the budget holds 2 x --max-concurrency + 1
checkpoints. It is never below 4096 MiB (the former fixed default); above that
it is capped at a quarter of the available memory (the container limit when one
applies, else total physical memory when availability cannot be read), so a host
with less than 16 GiB keeps the former 4096 MiB. With a fixed 4096 MiB, four Qwen3.8-27B coding agents with
20K-token conversations could not store their deeper checkpoints: every turn fell
back to the system/tools head and re-read most of the conversation.
| Scenario | Typical prefix length | Recommended cap |
|---|---|---|
| Single-user chat | 200–2000 tokens | 16–32 |
| Multi-session agent | 500–5000 tokens | 32–63 |
| Batch / benchmark | N/A (cold starts) | 4 |
The in-memory cache limit is 63 slots. Backend slot 63 is reserved for disk cache staging, preventing disk loads from overwriting a live in-memory entry.
KVFlash (the --kvflash option) keeps only a fixed pool of KV rows on the GPU and
pages the rest. That breaks the ordinary prefix snapshot, which copies rows
assuming they sit contiguously, so the Qwen3.5 backend has its own restore
paths. The switches are listed in ENVIRONMENT.md
(LUCE_KVFLASH_SESSION, LUCE_KVFLASH_RESTORE_CONSUME,
LUCE_KVFLASH_STATE_LADDER_MIB).
Pooled snapshots. When the live cache has outgrown the pool (or the pager
is no longer identity-mapped), a snapshot is saved as a pager blob stored inside
the prefix snapshot instead of as dense KV rows. It is taken at the current
position and only used for a restore when that position is a multiple of the
64-token pager chunk. The blob is saved only when
LUCE_KVFLASH_RESTORE_CONSUME=1; otherwise the snapshot is skipped with a
one-time log line.
Never on disk. The disk cache (see Disk Session Cache above) writes a
snapshot only when it leaves RAM. take_prefix_snapshot frees a pooled
snapshot instead of handing it over, because an OwnedSnapshot cannot carry
the pager blob. A file on disk therefore never lacks the KV it claims to hold.
Suffix-only restore. With LUCE_KVFLASH_RESTORE_CONSUME=1, a pooled
snapshot rebuilds the pager and KV pool, and the request prefills only the
tokens after the snapshot. Without it a pooled snapshot is dropped and the
request re-prefills the whole prompt from position 0. A pooled snapshot is
never read as a dense snapshot and never half-applied.
DeltaNet state ladder and live restore. Qwen3.5 layers include DeltaNet
(recurrent) layers whose state cannot be rebuilt from KV rows. While a pooled
prefill runs, the backend captures state-only checkpoints (recurrent and
convolution state, no KV) at ladder positions, evicting the least recently used
ones past LUCE_KVFLASH_STATE_LADDER_MIB (default 3072). The KV rows of the
running session are still in the pool, so a new request whose prompt shares a
prefix with the live session can resume from the largest checkpoint at or
below the shared prefix, without any saved snapshot. The ladder is cleared at
every cold prefill and is not used with paged attention.
Choosing the resume point. For each request the backend compares the server snapshot cut (0 if the snapshot is dropped or refused) with the live cut and takes the larger. A tie keeps the server snapshot, which needs no state restore.
Turn checkpoints. At generation end the backend records a checkpoint at the
final cache position, and the prompt is cut at chat turn boundaries, so the
next turn can resume exactly where the last one ended. These belong to the same
switch: LUCE_KVFLASH_SESSION=0 disables the ladder, live restore, and turn
checkpoints together. It is on by default and only matters when KVFlash is
active.
512-token prefill grid. Pooled prefill (a prompt too long for the pool, or a suffix prefill after a pooled
restore; independent of LUCE_KVFLASH_SESSION) runs on an absolute grid of 512 tokens, a multiple of the
64-token chunk. A batch never starts off the grid
except the first one after an off-grid resume, so where a cut falls does not
depend on where an earlier request started. Turn cuts (also independent of the session
switch, so the session-off reference runs the same batches) are the only other batch boundaries, and they depend only on the prompt tokens, so a restore at a cut
replays exactly the batches a cold prefill runs from there on. This is what
lets restored and cold answers match byte for byte.
restorable_prefix hook. ModelBackend::restorable_prefix(prompt, slot, slot_tokens) returns how many prompt tokens a restore would actually skip. The
default returns slot_tokens (trust the snapshot). The Qwen3.5 backend
returns the larger of the server snapshot cut and the live cut, computed from
the same arguments the restore itself uses, so callers such as prefix-cache
accounting see what the backend will take rather than what the snapshot holds.
With KVFlash inactive it returns slot_tokens.
PFlash admission uses it too: HttpServer::ram_restorable_prefix_tokens finds the
snapshot the prefix cache would restore (or none) and returns the backend's
answer for it, so a KVFlash live session counts as restorable even when the
cache holds no matching snapshot.
| File | Role |
|---|---|
server/kv/prefix_cache.{h,cpp} |
LRU logic: entries and their state (disk key, aux blob, agent-turn mark), reservations, eviction, resident budget |
server/kv/chat_boundaries.{h,cpp} |
Chat markers and boundary detection |
server/kv/cuts.{h,cpp} |
Capture cut and restore points |
server/kv/prefix_hash.{h,cpp} |
Prefix keys (truncated SHA-1 of the token ids) |
server/kv/kv_cache.{h,cpp} |
KvCache, one per model: the request protocol of both serving paths (single-sequence match/admit/release; concurrent plan/admitted/captured, with the disk reader thread), what happens when a checkpoint leaves RAM (written to disk when it should be kept, then freed), and the shutdown flush |
server/kv/shared_prefixes.{h,cpp} |
The shared-prefix cut and its ring (--shared-prefix) |
server/kv/prefix_pins.{h,cpp} |
Pinned shared prefixes and their catalog, <kv-cache-dir>/pins/<identity>.json |
server/kv/checkpoint_store.{h,cpp} |
Checkpoint handles and the stores every request-time snapshot call goes through (BackendCheckpoints, EngineCheckpoints), including the disk-staging checkpoint |
server/kv/disk_store.{h,cpp} |
Disk snapshots: the file format and DiskStore, one per directory per process (lock, index, writer, budget, TTL) |
server/kv/disk_prefix_cache.{h,cpp} |
DiskPrefixCache, one model's view of its directory's DiskStore (identity salt, layout, lookups, saves), and the disk scope policies |
server/kv/parallel_prefix_txn.h |
The concurrent path's capture transaction |
common/snapshot_backend.h |
Platform-aware snapshot backend selection |
common/model_backend.h |
snapshot_save/free/used/cur_pos interface |
qwen35/qwen35_target_graph.cpp |
snapshot_target_cache(), restore_target_cache() |
laguna/laguna_target_graph.cpp |
laguna_snapshot_alloc/save/restore() |
gemma4/gemma4_backend.cpp |
Inline snapshot allocation + copy |
- Unified memory: Near-instant (just memcpy, no PCIe)
- Discrete GPU: PCIe transfer (right-sized: typically < 1ms for short prefixes)
- Amortized over the full prefill time (typically seconds for long prompts)
- Unified memory: Near-instant
- Discrete GPU: PCIe transfer (e.g., 4096 tokens → ~6ms)
- Always faster than re-computing prefill (which takes seconds)
With right-sized snapshots, typical save/restore transfers are small:
- cur_pos=265 → ~4.6 MB → < 1ms over PCIe
- cur_pos=4096 → ~70 MB → ~6ms over PCIe
The old full-size approach (1.5 GB per snapshot) is eliminated. System RAM footprint is proportional to actual token count, enabling many more cached prefixes with no VRAM cost.
scripts/bench_agentx_cache.py replays the public coding-agent traces of the AgentX
benchmark against a running server and reports the share of prompt tokens read from the
cache, next to the share a cache that never misses would read on the same requests. The
client is the official AgentX harness running its agentx scenario unchanged, so the
harness itself checks the benchmark's rules and marks the run submission_valid; the
report prints that verdict.
python3 server/scripts/bench_agentx_cache.py run --url http://127.0.0.1:8080 \
--model qwen3.8-27b --tokenizer Qwen/Qwen3.8-27B --sessions 1 --out results/agentx-1The default corpus is the official 256K one (every request at most 256,000 tokens): serve
it with --max-ctx 262144. The traces hold the shape of real Claude Code sessions and no
text, so the client sends filler text of the recorded lengths, replays the recorded
replies, and asks for each reply at its recorded length with ignore_eos. They carry no tool calls: the Agent Turn Cache,
which keeps the state a tool-call turn leaves behind, does not take part, and every turn
resumes from the end of the previous prompt.