Repository navigation
[FEATURE] Caching support for all models #1140
Description
Goal
Every model provider supports prompt caching. strategy="auto" picks the best mechanism for each provider×model using one documented resolution table, in both SDKs. CacheConfig stays backward compatible.
Mechanisms
The different mechanisms that we vend as strategies:
- A: provider caches the prefix on its side (implicit)
- B: client places cache points
- C: provider places the cache point
- D: pre-created cache resource. Modifiers:
cache_key,ttl.
Already landed
#3681 #3949 #3985 #4186 #3986 #4055 #4127 #4232 #4083 #4080 #4193 #4219 #4617
1. Core
-
strategybuckets + resolution table ([FEATURE] CacheConfig: strategy buckets + per-provider auto resolution table #5126) - Config surface unification ([FEATURE] Move cache_config onto BaseModelConfig #5130)
2. Provider gaps
- Anthropic native automatic as opt-in →
provider([FEATURE] Anthropic: native automatic caching as opt-in strategy="provider" #3573) - Gemini/Google caching →
resource([FEATURE] Add explicit context caching support for Gemini models #1060, feat: add managed prompt caching to the Gemini and Google providers #4137) - Bedrock Nova →
sdk([FEATURE] Create Nova model cache strategy #1571, [FEATURE] Amazon Nova tool caching with cachePoint in messages #449) - LiteLLM Anthropic-family →
sdk(feat(litellm): route cache_config per underlying provider #3682) - OpenAI gpt-5.6+ cache points as
sdkopt-in
3. Correctness
- Cache-aware context management ([FEATURE] Context Manager Cache aware mode #4176)
- Keep injected context from breaking implicit prefix caches ([BUG] harness-py: per-call ContextInjector reminders defeat implicit prompt caching (GPT-6 Luna on Bedrock) #4879)
4. Caching on by default
- Turn on caching by default
Metadata
Metadata
Assignees
Labels
Type
Fields
Language
Python
Projects
- Status
Activity
[-][FEATURE] Prompt caching support for all models[/-][+][FEATURE] Caching support for all models[/+]feat(anthropic): add prompt caching support to direct Anthropic API
chosw1029 commentedon Apr 19, 2026
Hi — we're hitting this on the direct Anthropic API path and have a working patch that adds parity with
BedrockModel:AnthropicModel.stream()/format_request()now acceptsystem_prompt_contentand emit thesystemfield in Anthropic list-form withcache_control: {"type": "ephemeral"}on the text block immediately preceding acachePoint.format_chunk()metadata events exposecacheReadInputTokens/cacheWriteInputTokens(matchingBedrockModel's field names so existing observability code works unchanged).Verified against a live Claude Sonnet 4.5 run: subsequent requests with an identical system prefix now report
cache_read_input_tokens > 0where the pre-patch code reported0.Opening a PR shortly. Happy to adjust scope if you'd like anything trimmed or staged differently.
joshwand commentedon May 19, 2026
Is this ticket limited to system prompt caching, or does it include arbitrary cache points for multi-turn conversations?
feat(gemini): plumb through cache tokens in metadata events
6 remaining items
mkmeral commentedon Nov 2, 2026
Related #5700
joshwand commentedon Nov 4, 2026
@maxmilian It's good to see at least one provider supporting it, but my question was more one of intent/scope—would arbitrary and/or automatic cache points be supported across all models, with a uniform interface? A few data points I notice:
cache_configset,AnthropicModelstrips all cache points except the first in the last user message, so manual placement and automatic caching are mutually exclusive thereLiteLLMModelinheritsOpenAIModel's message formatting, which dropscachePointblocks outside the system prompt, regardless of the target modelSeems like there's several different levels at which it could be supported:
(cc @joshwand-aws)