enable Anthropic/OpenRouter prompt caching (cache_control on system prefix)#2
Open
Shooksie wants to merge 3 commits into
Open
enable Anthropic/OpenRouter prompt caching (cache_control on system prefix)#2Shooksie wants to merge 3 commits into
Shooksie wants to merge 3 commits into
Conversation
…refix)
Set an ephemeral cache_control breakpoint on the first system message for
Anthropic models routed through OpenRouter, so the stable system prefix is
cached across agentic steps instead of re-paying the full prompt each turn.
- ChatRequest gains a cache_system_prompt flag and a custom Serialize that
rewrites only the first system message into the multipart content-array
form { type: text, text, cache_control: { type: ephemeral } }. All other
messages keep their plain-string content, so the cacheable prefix stays
byte-stable and nothing else churns.
- Gated via wants_anthropic_cache_breakpoint(model, api_base): requires both
an Anthropic model (anthropic/ or claude) AND an OpenRouter route (base url
or openrouter/ prefix). OpenAI-native and custom non-OpenRouter endpoints
stay plain strings (they auto-cache / can reject the field).
- The read side already reports cache_read tokens, so caching is verifiable
live with no further change.
Bump to 0.1.6 (Cargo.toml + plugin.toml).
Per OpenRouter's prompt-caching docs, exactly two families require explicit cache_control breakpoints: Anthropic Claude AND Alibaba Qwen. OpenAI, DeepSeek, Grok/xAI, Moonshot/Kimi, Groq, and Gemini 2.5 all cache automatically and must stay plain strings. - Rename wants_anthropic_cache_breakpoint -> wants_explicit_cache_breakpoint; keep a #[deprecated] alias forwarding to the new name for API compatibility. - Match the model's vendor SEGMENT (anthropic/qwen/alibaba), not a bare substring, so auto-cache models that merely mention an explicit-cache family in their id (e.g. deepseek/deepseek-r1-distill-qwen-32b) correctly stay plain. - Still require the OpenRouter route (base url contains openrouter, or model has the openrouter/ namespace prefix). - Extend the test truth table: Qwen-via-OpenRouter -> multipart system with cache_control ephemeral; DeepSeek/Gemini-2.5/Grok/Kimi/Groq/OpenAI all stay plain (incl. the deepseek-distill-qwen regression case). Version stays 0.1.6.
…onfig seam
- Multi-breakpoint prompt caching: place cache_control:{ephemeral} across the
system message + tools (if sent) + a rolling breakpoint on the growing
conversation tail (up to Anthropic's 4-breakpoint cap), for the explicit-cache
families (Anthropic/Qwen) via OpenRouter — so multi-step agent loops reuse all
prior turns from cache, not just the system prompt. Auto-cache families stay plain.
- OpenRouter response caching: opt-in X-OpenRouter-Cache (+ TTL) header; hits bill
as 0 tokens. Logs X-OpenRouter-Cache-Status at debug.
- Config seam: provider reads an optional cache config from the runtime contract
(+ env fallbacks) with sensible defaults; ao-cli YAML->contract plumbing is a
separate follow-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The provider already reads cache usage (
cache_read_input_tokensinUsageInfo, surfaced ascache_readtoken events) but it never set a cache breakpoint. Anthropic-via-OpenRouter caches only when a content part carriescache_control— caching is not automatic. So every agentic step re-paid the full system prompt; the read side reportedcache_read: 0forever.Change
ChatRequestgainscache_system_prompt: boolplus a customSerializeimpl. When the flag is set, only the firstsystemmessage is rewritten into the OpenRouter/Anthropic multipart form:{"role":"system","content":[{"type":"text","text":"<sys>","cache_control":{"type":"ephemeral"}}]}content(derivedChatMessageserialization is untouched), so the cacheable prefix stays byte-stable and no other message shape changes.ChatMessageliterals (38 of them) and session storage are completely unchanged.wants_anthropic_cache_breakpoint(model, api_base)requires both an Anthropic model (anthropic/orclaude) and an OpenRouter route (base url containsopenrouter, or model has theopenrouter/namespace prefix). OpenAI-native endpoints (which auto-cache and can reject an unexpectedcache_control) and custom non-OpenRouter OpenAI-compatible bases stay plain strings.Before / After
cache_readstays 0, full system prompt billed every step.cache_read, so caching can be verified live (watchcache_readgo non-zero after the first step).Tests
cache_control.type == "ephemeral"; OpenAI-native → system message stays a plain string with nocache_control.anthropic/on a custom non-OpenRouter base all false).cargo fmt/cargo clippy --all-targetsclean;cargo testgreen (159 passed).Version
Bumped 0.1.5 -> 0.1.6 (
Cargo.toml+plugin.toml) so the media team and other fleets can repin to pick up caching.