[EPIC] Model support updates #5636

Human Overview

The latest updates from model providers (including OpenAI's caching on latest models as well as Anthropic's preserved thinking) have highlighted potential improvements we need to make in our SDK. Especially considering there are multiple models and model families per model providers, this has added to complexity of our implementation.

The goal of this epic is to standardize how we are handling these features across model providers, and how we can scale discovery/implementation/testing of these models.

Summary

New models ship every week, and each one differs in context window, output limit, effort levels, how it caches, how it replays reasoning, and what it rejects. Today that knowledge is spread across hand-maintained tables in both SDKs, the harness and the CLI. New models fail until someone patches them, and caching only works well on a few providers.

This epic makes model support declarative, tested and cost-efficient:

1. Model knowledge: one data file describes every model and is kept current automatically. Every model we add is tested end to end, and Strands keeps working when providers change their rules.
2. Caching: one caching interface for every provider, and SDK features that keep the cached prefix intact.
3. Usage, cost and budgets: one usage shape across providers, complete accounting, and caps that cover everything an agent spends.

Workstream 1: Model knowledge

Goal: Every model the SDK can call is described in one data file, which is kept current without hand edits. Every model in that file passes the same end-to-end scenarios, and provider rule changes are caught by tests before users hit them.

Tasks

• One source of truth: the harness and CLI read model info instead of their own max-token, effort, thinking and context-window tables ([BUG] harness Claude max_tokens tables disagree with shared/model-info.json (opus-4-1 128k vs 32k) #5680, Harness: Bedrock effort levels do not apply to openai.gpt-6.1-terra (exact-match on openai.gpt-6-astra) #5561, [FEATURE] Support reasoning effort on caller- constructed model instances #4804)
• Capabilities cover what features need: reasoning and effort levels, input modalities, tool support, server-side compaction ([FEATURE] Gate server-side compaction through model metadata capabilities #5466), and cache facts
• Automated onboarding and updates: new and changed models land in the data file through a scheduled job and PR (seeded from models.dev and provider APIs, with our overrides on top) ([FEATURE] Automated model onboarding and updates #5803)
• End-to-end tests for every model in model info, generated from the data file and run through the Strands harness ([FEATURE] Model canary: smoke-test every model in the context-window tables nightly #5634): real multi-turn runs covering tool use, caching (asserting cache reads), structured output, context management and summarization, injected context and memory, reasoning replay, and session restore. Runs nightly and files an issue on failure (ci: add a nightly integ canary that reports failures to issues #5577)
• Preserved-thinking and reasoning-replay compatibility across providers and model switches (Epic: Anthropic distillation protection (preserved thinking and reasoning_extraction refusals) #5160, [FEATURE] Cross-model reasoning content: capture, replay, and safe interoperability #4779, [FEATURE] Model canary: run preserved-thinking paths with thinking-block binding enforced #5635)
• Provider outcomes are typed and classified: stop reasons ([FEATURE] Centralize SDK stop-reason classification and outcome metadata (Python + TypeScript) #5168), context overflow, output cut off by max_tokens

Bugs to close: #780, #5753, #5754, #5562, #5556, #5202, #5203, #5498, #5766, #5499, #5786, #5692, #5739, #5742, #3769, #4158, #5740, #5428

Workstream 2: Caching

Goal: One cache_config works on every provider and is resolved from model info. SDK features keep the cached prefix intact, so long-running agents read most of their input from the cache on every provider that supports caching.

Tracking: #1140

Tasks

• OpenAI explicit cache points on GPT-5.6+, for the OpenAI API and Bedrock Mantle ([FEATURE] OpenAIResponsesModel: support explicit prompt cache breakpoints for GPT-5.6+ (py + ts) #5802, PR fix(openai): preserve prompt cache across temporary context injections #5418), and the harness passes its caching setting through (feat(harness): pass the caching setting to OpenAI as an SDK cache config #5472)
• One config surface and resolution table for every provider ([FEATURE] CacheConfig: strategy buckets + per-provider auto resolution table #5126, [FEATURE] Move cache_config onto BaseModelConfig #5130)
• Provider gaps: Gemini ([FEATURE] Add explicit context caching support for Gemini models #1060), Nova ([FEATURE] Create Nova model cache strategy #1571), LiteLLM (feat(litellm): route cache_config per underlying provider #3682), Anthropic native automatic caching ([FEATURE] Anthropic: native automatic caching as opt-in strategy="provider" #3573), OpenAI Chat Completions
• Refactor context manager summarization (Model.summarize()) to account for caching: the summary request reuses the agent's cached prefix instead of a fresh request (PR feat(context-manager): batch summarization replays the agent's own request plus one instruction #5299; evaluated in [FEATURE] Evaluate base summarization strategies across models before fixing defaults #5734, alongside [FEATURE] Separate native compaction from base summarization, replacing the placeholder Model.summarize() #5733)
• Features keep the prefix stable: injected context ([BUG] harness-py: per-call ContextInjector reminders defeat implicit prompt caching (GPT-6 Luna on Bedrock) #4879, via Epic: durable injected context as content blocks #5587), structured output (Agent.structured_output_async() doesn't forward system_prompt_content, silently drops non-text blocks (e.g. cachePoint) #3496), tool order ([FEATURE] Sort tool specs sent to the model by default (opt-out) so MCP tool order doesn't bust the prompt cache #5700), tool selection (Epic: dynamic tool loading (design 0021) #5668), router stickiness, sliding window trimming
• Caching on by default, with an opt-out

Bugs to close: #4879, #3496, #3348, #3758, #5700

Workstream 3: Usage, cost and budgets

Goal: Every model call, including auxiliary ones, is reported in one usage shape across providers, and developers can cap what an agent spends once, at construction.

Tasks

• One usage convention across providers: cache reads and writes reported the same way, reasoning tokens as their own field, cache writes split by TTL ([FEATURE] Normalize token usage to one semconv-aligned convention #4618, [FEATURE] Report cache write tokens split by TTL in Usage #5257)
• Complete accounting: auxiliary calls, compaction and retries show up in usage ([FEATURE] Auxiliary LLM calls (summarization, HITL classifier, routing classifier, memory extraction) bypass usage metrics — token spend is invisible or orphaned #3863, [FEATURE] Include compaction calls in token usage #5404, feat: route DecisionModel usage through the auxiliary LLM-call instrumentation #4956, [BUG] AfterModelCallEvent.retry drops discarded call metrics from accumulated_metrics #4333)
• Limits can be set once at construction instead of on every call, and cover model calls, tool calls and wall-clock time as well as turns and tokens; overflow recovery is bounded (builds on feat(agent): enforce invoke limits and usage accounting across the whole agent tree #4667; ContextWindowOverflowException recovery recurses without convergence check — non-terminating retry loop ending in RecursionError #3338)

Optional scope (to discuss with the team): dollar cost estimates from reference pricing (e.g. models.dev) or user-supplied prices, and budgets in dollars (#1428). Unknown cost would be reported as None, never 0.

Bugs to close: #3338, #4333, #3602