feat(caching): Anthropic prompt caching with cache-aware cost reporting - #903
Open
addyCooks wants to merge 2 commits into
Open
feat(caching): Anthropic prompt caching with cache-aware cost reporting#903addyCooks wants to merge 2 commits into
addyCooks wants to merge 2 commits into
Conversation
…orting Mark the stable prompt prefix with Anthropic cache breakpoints so multi-turn sessions read it back out of cache instead of resending it at full price every turn. The system prompt now travels as a message rather than the `system:` string, since a bare string cannot carry `providerOptions`. Two breakpoints are placed: one on the system entry, which also covers the tool schemas because they precede it in Anthropic's prompt order, and one on the final message so the next turn's prefix ends there. Prefixes below Anthropic's minimum cacheable length get no breakpoints, and the whole path is off unless the provider uses the anthropic SDK, with `promptCaching: false` to opt out. Cost reporting is made cache-aware in the same change, otherwise every cache hit would be billed at the full input rate and overstate spend. Cache read and write counts flow from the SDK through ApiUsage, ApiCallRecord, and ResponseUsage; `priceTokens` becomes the single pricing function shared by the per-response indicator and /usage, billing reads and writes at their own models.dev rates and falling back to the input rate where cache pricing is unknown. Closes Nano-Collective#888.
Moving the system prompt into the messages array trips a bare `console.warn` in the SDK's prompt standardiser on every request. It is not gated by AI_SDK_LOG_WARNINGS, so it would print to stderr on each turn and corrupt the Ink frame. Declare `allowSystemInMessages: true` on the caching path, where the system message is one we placed ourselves, and leave the SDK's warning in place everywhere else. Adds handleChat coverage for the cache-marked system message, the absence of the warning, and the unchanged non-anthropic path.
addyCooks
requested review from
Avtrkrb,
akramcodez and
will-lamerton
as code owners
August 19, 2026 10:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Marks the stable prompt prefix with Anthropic cache breakpoints so multi-turn sessions
read it back out of cache instead of resending it at full price every turn.
withCacheBreakpoints()folds the system prompt into the message array and places twobreakpoints: the system entry, and the final message of the turn. The system breakpoint
also covers tool schemas, since tools precede system in Anthropic's prefix order. The
system:param is dropped when caching is active a bare string can't carryproviderOptions, and sending both would duplicate the prompt. Below Anthropic'sminimum cacheable length the prompt is still folded in but no breakpoints are emitted.
Anthropic-only: OpenAI and OpenRouter prefix-cache automatically, and local models have
no cache to address.
"promptCaching": falseon the provider config opts out.Cost reporting is made cache-aware in the same PR without it, every cache hit would be
billed at the full input rate and overstate spend.
cacheReadTokens/cacheWriteTokensflow from
usage.inputTokenDetailsthroughApiUsage→ApiCallRecord→ResponseUsage,and
priceTokens()becomes the single pricing function shared by the per-responseindicator and
/usage, billing reads and writes at their own models.dev rates.Closes #888.
Type of Change
Changeset
pnpm changeset) describing this change for the changelogTesting
Automated Tests
.spec.ts/tsxfilespnpm test:allcompletes successfully)Manual Testing
Checklist