feat(ai): default claude-* models to automatic Anthropic prompt caching - #3923
Conversation
Claude-family models routed through non-canonical Anthropic-compatible
gateways (Cloudflare AI Gateway, GitHub Copilot, GitLab Duo, Vercel AI
Gateway, zenmux, ...) previously received no generated cache markers, so
long agentic sessions paid full input cost on every turn. The provider now
emits top-level `cache_control: { type: "ephemeral" }` (automatic caching
per platform.claude.com prompt-caching docs) for any claude-* model id,
matching the canonical-API default. Non-Claude models on unknown compatible
endpoints keep the no-cache default; `promptCacheMode: none|explicit` and
per-request `cacheRetention: none` still opt out. Non-canonical endpoints
keep the default ~5m cache lifetime unless they opt into
`supportsLongCacheRetention`.
Confidence: high
Scope-risk: narrow
Reversibility: trivial
Tested: anthropic-cache, anthropic-stream-envelope, anthropic-alignment, issue-814 repro, biome, ai typecheck
Regenerate artifacts/issue-3670-anthropic-cache-eval.json so the pinned providerSourceBlobOid/providerSourceSha256 match the committed anthropic.ts after the automatic-caching default change. Confidence: high Scope-risk: narrow Reversibility: trivial
The automatic-caching classifier previously read `wireModelId ?? id` while the Anthropic transport serializes `params.model = model.id`, so a custom model with a differing wire id could get a cache decision for a different model than the one dispatched. Classify `model.id` only and add a table-driven matrix covering prefixed/uppercase claude ids, non-canonical long-retention opt-in, promptCacheMode none, cacheRetention none, `-claude-` false positives, and wire-id override inertness. Confidence: high Scope-risk: narrow Reversibility: trivial Tested: anthropic-cache suite (10 tests), packages/ai check (biome + tsc)
Regenerate artifacts/issue-3670-anthropic-cache-eval.json so the pinned provider source identity matches the classifier change in 483ba5f. Confidence: high Scope-risk: narrow Reversibility: trivial
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00d32034bc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| : isCanonicalApi || isClaudeFamilyModel(model) | ||
| ? "automatic" |
There was a problem hiding this comment.
When a user-defined anthropic-messages endpoint exposes a claude-* model but rejects top-level cache_control and requires block-level markers, this identifier-only default now makes every request fail upstream. The updated AnthropicCompat documentation explicitly recognizes such endpoints and recommends promptCacheMode: "explicit", but the coding-agent models configuration still validates compat with OpenAICompatSchema, which does not admit promptCacheMode, so CLI users cannot select that advertised mode (short of disabling caching entirely with cacheRetention: "none"). Restrict this default to known-capable endpoints or expose the Anthropic compatibility setting through the configuration schema.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 9dc8a8d. The shared models compatibility schema now admits compat.promptCacheMode (none, explicit, automatic) and compat.supportsLongCacheRetention at provider, model, and model-override levels; the generated schemas/models.schema.json is updated. Focused schema validation and ModelRegistry propagation/precedence tests cover all three configuration levels. The branch is also merged with current dev, and Anthropic/config/package checks are clean.
Automatic caching for claude-* models must remain configurable on custom Anthropic-compatible endpoints that reject top-level cache_control or lack one-hour retention. The models config schema previously admitted only the OpenAI compatibility fields, making the documented promptCacheMode escape hatch unavailable to CLI users. Expose promptCacheMode and supportsLongCacheRetention through the shared model compatibility schema at provider, model, and model-override levels, regenerate models.schema.json, and lock propagation/validation with focused tests. Confidence: high Scope-risk: narrow Reversibility: trivial Tested: models config Anthropic cache compat, focused model registry propagation, coding-agent biome + tsc
…-default-for-anthropic-models
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b3f7ba2a36
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }); | ||
|
|
||
| test("generated JSON schema exposes Anthropic prompt-cache compatibility", async () => { | ||
| const schema = (await import("../../../schemas/models.schema.json")) as Record<string, unknown>; |
There was a problem hiding this comment.
This new test uses await import() even though the repository contract explicitly prohibits inline imports and requires top-level imports; import the JSON schema statically instead.
AGENTS.md reference: AGENTS.md:L101-L105
Useful? React with 👍 / 👎.
| : isCanonicalApi || isClaudeFamilyModel(model) | ||
| ? "automatic" | ||
| : "none"; |
There was a problem hiding this comment.
When a non-Claude model on a noncanonical Anthropic-compatible endpoint sets the newly accepted compat.promptCacheMode: "automatic", this ternary ignores that value and falls through to the model-name classifier, selecting "none" and emitting no cache marker. This makes the only meaningful opt-in use of automatic ineffective for custom identifiers; handle promptCacheMode === "automatic" before applying the endpoint/model defaults.
Useful? React with 👍 / 👎.
The compatibility schema now exposes promptCacheMode to custom endpoints, so the automatic enum value must actively opt a non-Claude, non-canonical Anthropic-compatible model into top-level caching instead of falling through to the conservative none default. Honor the explicit automatic mode, lock it with a payload test, and update the models configuration guidance and generated docs index. Confidence: high Scope-risk: narrow Reversibility: trivial Tested: focused Anthropic/config/registry tests, packages/ai biome + tsc
Regenerate the Anthropic cache-eval source identity after the explicit automatic prompt-cache mode change. Confidence: high Scope-risk: narrow Reversibility: trivial
|
@codex review |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Keep the Anthropic cache-compat schema regression test within the repo's top-level-import contract while preserving the generated-schema assertions. Confidence: high Scope-risk: narrow Reversibility: trivial Tested: focused schema test, Biome, coding-agent typecheck
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
GitLab Duo dispatches before the shared option mapper, so a configured model.cacheRetention value was not forwarded to Anthropic. With automatic Claude-family caching, that could silently turn cacheRetention:none into an ephemeral cache marker. Resolve request-over-model retention at the Anthropic cache-control boundary and cover GitLab Duo's special dispatch for both configured none and an explicit request override. Confidence: high Scope-risk: narrow Reversibility: trivial Tested: 86 Anthropic/GitLab tests, packages/ai biome + tsc
Regenerate the Anthropic cache-eval source identity after moving model cache retention fallback to the provider boundary. Confidence: high Scope-risk: narrow Reversibility: trivial
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…-default-for-anthropic-models # Conflicts: # packages/coding-agent/src/internal-urls/docs-index.generated.ts
1 participant
What
Claude-family models routed through non-canonical Anthropic-compatible gateways (Cloudflare AI Gateway, GitHub Copilot, GitLab Duo, Vercel AI Gateway, zenmux, ...) previously received no generated prompt-cache markers, so long agentic sessions paid full input cost on every turn. This PR makes the
anthropic-messagesprovider emit Anthropic's automatic caching (cache_control: { type: "ephemeral" }at the request top level, per platform.claude.com prompt-caching docs) for everyclaude-*model id — the same default canonicalapi.anthropic.comalready had.Behavior
claude-*onapi.anthropic.comttl: "1h"claude-*on non-canonical gateway{type:"ephemeral"}(default ~5m)claude-*gateway +compat.supportsLongCacheRetention: true{type:"ephemeral", ttl:"1h"}claude-*on unknown endpointEscape hatches preserved:
compat.promptCacheMode: "none" | "explicit"and per-requestcacheRetention: "none"still win. Classification keys off the dispatchedmodel.id(last path segment, case-insensitiveclaude-prefix) —wireModelIdis inert for this transport, so the cache decision always describes the model actually posted.Changes
packages/ai/src/providers/anthropic.ts—isClaudeFamilyModelhelper;getCacheControlmode is nowautomaticforisCanonicalApi || isClaudeFamilyModel(model)packages/ai/src/types.ts—promptCacheModeJSDoc updatedpackages/ai/test/anthropic-cache.test.ts— defaults test extended + new table-driven mode/identifier matrix (prefixed/uppercase ids, long-retention opt-in, both opt-outs,-claude-false positive, wire-id inertness)packages/ai/test/anthropic-stream-envelope.test.ts— cache TTL cases extendedpackages/ai/CHANGELOG.md— Unreleased entryartifacts/issue-3670-anthropic-cache-eval.json— regenerated source-identity pinsVerification
bun testtargeted suites: cache 10, stream-envelope 26, alignment 47, issue-814 2, cache-eval write+validate 2+2 — 89 pass / 0 fail; 11/11 adversarial payload probesbun --cwd=packages/ai run check(biome + tsc) cleansha256:f92bc40c…): cleaner PASS, architect CLEAR/APPROVE, executor QA passed, terminal critic OKAYcheck:tshas one pre-existing failure inverify-gjc-sdk-canonicalization.ts --self-test(stale fixture expects exit 1 where verification now passes); reproduces on the base commit and in the parent checkout, unrelated to this diff (verified: no non-packages/ai/artifactsfiles changed)Review follow-up
Addressed Codex P1 in
9dc8a8d9c: customanthropic-messagesendpoints can configurecompat.promptCacheMode(none,explicit,automatic) andcompat.supportsLongCacheRetentionat provider, model, and model-override levels. Generated schema + focused validation and registry propagation tests included.Synced the branch with current
devin merge commitb3f7ba2a3; GitHub now reports the PR mergeable.Final independent review found and fixed GitLab Duo retention precedence in
4c71ba5db: Anthropic cache resolution now falls back tomodel.cacheRetentionat the provider boundary, so special dispatch wrappers preserve configurednoneand explicit request options still win. Added a GitLab Duo payload regression; eval identity refreshed in9c8cc8328.