Repository navigation
Conversation
github-actions
Bot
added
documentation
jimbrub
changed the title
|
|
||
| Auto mode's tool-result truncation runs eagerly, when the result is added and before the model sees it, so it doesn't affect preserved thinking. A truncate strategy with `preserveRecent > 0` instead truncates results the model has already seen, which invalidates later thinking blocks. | ||
|
|
||
| `Model` gets a non-abstract `compact()` that throws by default, so `SummarizeStrategy` can call it without importing `AnthropicModel`, the only provider that overrides it. The parameters mirror `OffloadConditions`. `utilization` is passed through only for logging, since `apply()` has already decided to compact. `preservedMessages` takes the same count-or-ratio as `preserveRecent` and sets the cut point. The cut respects pinned messages and never separates a `toolUse` from its `toolResult`. In Python the signature is `compact(messages, *, utilization=None, preserved_messages=0)`. |
There was a problem hiding this comment.
What about Anthropic Model on Bedrock?
There was a problem hiding this comment.
I have been calling compaction through AsyncAnthropicBedrock. BedrockModel (Converse) can't compact, because Converse has no compaction parameter or block. Today, setting the flag on it fails with only a logged strategy error, so the section proposes rejecting the flag at config time. One gap: Python's AnthropicModel can't take a Bedrock client in its constructor yet, so the doc lists that as a follow-up.
I am not too sure if this addresses your comment
| * ran against — so this never falls back to `context.agent.model` behind the customer's | ||
| * back if they configured `agent.auxModel`. | ||
| */ | ||
| private async _applyServerSide(context: ContextState, model: Model): Promise<boolean> { |
There was a problem hiding this comment.
are there any other thresholds that we can set on anthropic server side compaction? ie. custom prompt? summary length? etc
There was a problem hiding this comment.
i have added information about the parameters compaction accepts
9d43938 to
c2ed1a4
Compare
…up open questions, and add compact() reference code
…and note caching was off
…update open questions
|
Can you look at this design document? Assess whether or not to implement this change in the Strands SDK Scope out what needs to be done to implement |
|
|
||
| Auto mode's tool-result truncation runs eagerly, when the result is added and before the model sees it, so it doesn't affect preserved thinking. A truncate strategy with `preserveRecent > 0` instead truncates results the model has already seen, which invalidates later thinking blocks. | ||
|
|
||
| `Model` gets a non-abstract `compact()` that throws by default, so `SummarizeStrategy` can call it without importing `AnthropicModel`, the only provider that overrides it. The parameters mirror `OffloadConditions`. `utilization` is passed through only for logging, since `apply()` has already decided to compact. `preservedMessages` takes the same count-or-ratio as `preserveRecent` and sets the cut point. The cut respects pinned messages and never separates a `toolUse` from its `toolResult`. `systemPrompt` and `toolSpecs` are the live conversation's, passed through unchanged, because the compaction request must match the prefix the thinking blocks are bound to. `instructions` carries the strategy's custom summarization prompt (`SummarizeConfig.systemPrompt`) into the provider's compaction instructions. In Python the signature is `compact(messages, *, utilization=None, preserved_messages=0, tool_specs=None, system_prompt=None, instructions=None)`. |
There was a problem hiding this comment.
Can we take the approach here that there's a default client-side, which just gets overridden by the model provider? For example look at how token counting is implemented
There was a problem hiding this comment.
I like how token counting is implemented. Linking the code here:
harness-sdk/strands-ts/src/models/model.ts
Line 375 in 3aca339
|
|
||
| Auto mode also truncates tools results automatically which will not affect preserved thinking models because the tool result is changed before a model sees the tool result and therefore not thinking block can be formed on the tool result. | ||
|
|
||
| This effects customers that use the `"auto"` mode or any that use ContextManagement techniques that ever modify context. This also only effects customers that are using `preserved_thinking` Anthropic models. With this paradigm shift into LLM providers having restrictions on using their native APIs to modify context, there needs to be a universal way to switch to server side context management techniques for the specific LLM provider you are dealing with at that point in time. |
There was a problem hiding this comment.
What is "this" that is effecting customers? Server side preserved thinking?
There was a problem hiding this comment.
The "this" is just the change to the preserved thinking models anthropic vends. Which is adding a check on prefixes tied to thinking blocks and that the content of these prefixes is unchanged, otherwise anthropic will return a 400 error.
|
|
||
| Agent developers are finding new ways to increase agent performance when modifying the context window to save performance. It is a common strategy to limit token usage. How do agent developers continue to modify every bit of data in an agent's context history while respecting LLM provider's restrictions? | ||
|
|
||
| Strands modifies content in the context window through the current ContextManagement [methods](https://github.com/strands-agents/harness-sdk/tree/main/strands-ts/src/context-manager/methods). These functions will be affected by Anthropic's preserved thinking changes. |
There was a problem hiding this comment.
How are these impacted by anthropics preserved thinking changes?
There was a problem hiding this comment.
The summarize and truncate methods under context manager may modify a prefix that is tied to a thinking block. Once this modification happens then Antrhopic will return a 400 error
| * | ||
| * @defaultValue false | ||
| */ | ||
| requiresServerSideCompaction?: boolean |
There was a problem hiding this comment.
Instead of a bool, it might be better as a dict so we can expand upon it later
There was a problem hiding this comment.
i kinda love the bool | Config tho approach tbh we've taken in the SDK.
initial release of just bool and then expand with drop down config later
There was a problem hiding this comment.
This field will be added to the model.info/metadata model capabilities
| get requiresServerSideCompaction(): boolean { | ||
| return this.getConfig().requiresServerSideCompaction ?? false |
There was a problem hiding this comment.
We already have a getConfig, we probably dont need this
|
|
||
| ## Approach 1: Server-side context management versus client-side context management | ||
|
|
||
| Anthropic's preserved thinking models will return a 400 if the prefix of a thinking block has been modified. This check means that in our SDK, whenever summarize is called and preserved thinking is enabled, Strands needs to summarize the conversation through Claude's API. |
There was a problem hiding this comment.
I believe this page here: https://platform.claude.com/docs/en/build-with-claude/context-editing has everything that is relevant to this document.
Also see compaction here: https://platform.claude.com/docs/en/build-with-claude/compaction
|
|
||
| ### Summarize Server Side through Compaction | ||
|
|
||
| Anthropic's answer to vended context modification is compaction. The public surface, `Offload.summarize(target).when({ utilization, preserveRecent })`, does not change. Behind it, `apply()` resolves the model as it does today (`this._config.model ?? agent.auxModel`). If `_isRetroactive` is set and that model's `requiresServerSideCompaction` is `true`, `SummarizeStrategy` calls `model.compact(...)` instead of generating a client-side summary. A customer who set `agent.auxModel` is still routed there, and `agent.model` is never read directly. `compact()` takes the customer's existing conditions as-is, so there is no second API to learn. |
There was a problem hiding this comment.
A customer who set
agent.auxModelis still routed there,
Clarifying question just to make sure I understand:
- if auxModel is anthropic and supports server-side compaction, we'd use that?
- if aux model does not support server side compaction, but main model does, we'd use the aux model client side still?
mainly wondering/trying to tease out if there's any tricky behavior here
There was a problem hiding this comment.
The auxModel behavior is something I am still weighing. My initial thought is to always respect if a Strands user set an auxModel, even if it breaks preserved thinking.
Anthropic preserved thinking models can be used to summarize one another: https://platform.claude.com/docs/en/build-with-claude/preserved-thinking#switching-models
| @@ -0,0 +1,844 @@ | |||
| # Server Side Compaction Support | |||
There was a problem hiding this comment.
- can we merge requiresServerSideCompaction in to some of the model metadata stuff that Murat proposed? Gives us one central place and one single mechanism for defining those things per model. Some users require it and some do not, based on account creation date. Could we do something smart with 4xx responses as the default, and then additionally allow customers to control it with force enable/force disable?
- support multiple providers using anthropic models, e.g. bedrock converse, bedrock mantle, anthropic, litellm, vercel. They should all work
- should we discuss the open questions?
There was a problem hiding this comment.
- I plan on using the model metadata stuff to track if server-side-compaction is supported by models (to start, this is only true for fable 5.1, opus 5.5, haiku 5.5, and sonnet 5.5)
- I have ran server side compaction tests through bedrock's invoke model api, I am unsure how this performs through the other model provider api's and will need to test more
- The open questions were just kind of there to see if anyone on the team had strong opinions, just some design questions for me to get a better understanding on when customers may be using context management techniques. I really just had questions about implementing this through the model info/metadata stuff, but I believe the meeting cleared this up
|
|
||
| The SDK needs a path in the `ContextManagement` techniques which tells the context manager that it needs to modify context through the provider. This path needs to allow for various model providers. There needs to be a flag set during an agent session that indicates when the contextManager needs to refer to the model provider code for how to modify context. | ||
|
|
||
| To trigger this path, the following change is proposed to the `Model` class: |
There was a problem hiding this comment.
actually now i'm thinking if we should merge capabilities and config 🤔 probably right?
| messages: Message[], | ||
| conditions: { | ||
| utilization?: number | ||
| preservedMessages?: number |
There was a problem hiding this comment.
why do we need to pass these down, can't context manager handle this and only pass the messages we want compacted? i wouldn't defer this responsibility to model providers, it'll just be duplicated logic everywhere
There was a problem hiding this comment.
we should support fields the model provider supports in my mind, but everything else can be handled in context manager
There was a problem hiding this comment.
We do not need to track utilization in compact, I will remove it. My current compact function needs the preserved messages to establish where to split the conversation history to call compact on everything before the preserved recent messages.
I have a question:
A thinking model decides that it needs to summarize. There are two ways the agent can compact the conversation natively.
- It can call compact on the entire conversation history, then append the recent messages to the end of this.
- It can split the history between the history before the preserve content and then call compaction only on that and add back in the recent messages.
I have currently tested out option 2. The concern I have is that with approach 2, there may be some useful thinking blocks in the preserve recent history that would be dropped because we aren't compacting them, but approach one could duplicate content from the preserve recent in the summary.
| */ | ||
| async compact( | ||
| messages: Message[], | ||
| conditions: { |
There was a problem hiding this comment.
Why are these parameters in a nested object instead of being passed directly to compact?
| // strands-ts/src/models/model.ts | ||
| export interface BaseModelConfig { | ||
| contextWindowLimit?: number | ||
| requiresServerSideCompaction?: boolean // Default should be not supported, also should have ModelConfig attached |
There was a problem hiding this comment.
nit for api bar raising:
requires server side compaction versus supports server side compaction?
i think we should enable server-side compaction by default if it is supported and eventually give the user an opt out. that opt out can live likely on context manager?
| import { AnthropicModel } from '@strands-agents/sdk/models/anthropic' | ||
|
|
||
| const agent = new Agent({ | ||
| model: new AnthropicModel({ |
There was a problem hiding this comment.
Does serverside compaction only work if AnthropicModel is used? If I use BedrockModel, does this feature not work?
Reason being, I can pass in a modelId into model, and it will automatically use the bedrock model provider. Should we change this behavior for the new wave of preserved thinking models?
|
|
||
| ## Open Questions | ||
|
|
||
| - [ ] **One capability switch for Part 2.** `requiresServerSideCompaction` only describes compaction. Should it become a broader capability, for example one per technique, once Part 2 adds tool-use clearing, thinking clearing, and system prompt and tool list changes? Strategies would then ask the model what it supports instead of checking an Anthropic-specific flag. |
There was a problem hiding this comment.
yes we need this and should also add this as a part of this scope imo.
same kinda thing where if the flag is true, we hotswap our own implementation for the server-side equivalent.
| if (this._isMessageLevel && !context.overflow && context.utilization < this._utilizationThreshold!) { | ||
| return false | ||
| } | ||
| if (this._isRetroactive && model?.requiresServerSideCompaction) { |
There was a problem hiding this comment.
is there any way for summarization to be not retroactive?
There was a problem hiding this comment.
if something is summarized before the model sees it. for truncating or summarizing tool results, this would be a non retroactive summary
|
|
||
| | Setting | How Strands sets it | | ||
| |---|---| | ||
| | `compaction.instructions` (up to 16,384 characters) | From `SummarizeConfig.systemPrompt`. It replaces Anthropic's default summarization prompt entirely, so it should say what the summary must keep and tell the model not to call tools. | |
There was a problem hiding this comment.
do we override for everyone? can we just default to anthropic's instructions if customer didn't provide any? idk if it matters tbh
There was a problem hiding this comment.
If no systemPrompt for summarizer model is specified on the Strands side, then we would allow compact to just use Anthropic's default instructions. I am not sure if this answers your question
|
|
||
| ### Anthropic models on Bedrock | ||
|
|
||
| The two ways to run Claude on Bedrock behave differently: |
There was a problem hiding this comment.
so what is our approach? I don't get it from this section
|
|
||
| Anthropic allows LLM users to call Claude Native API to "compact" a conversation. Compacting does not allow for the amount of control that the Strands SDK provides in terms of summarizing a re-shaping context. In order to avoid dropping thinking blocks, Strands needs to use Anthropic's `compact` function to modify the context otherwise 400 errors will be returned by Anthropic. | ||
|
|
||
| The SDK needs to allow for a CompactionBlock datatype, matching the shape Anthropic's SDK already returns (`BetaCompactionBlock`/`BetaCompactionBlockParam` — `content`, `signature`, optionally `encrypted_content`): |
There was a problem hiding this comment.
why? can't we use one of the existing ones?
Also isn't this backwards incompatible? As a customer, I'd get this new block type, no?
There was a problem hiding this comment.
also does this only apply to Anthropic? so if i switch to openai from anthropic as model provider, what happens to this block type?
There was a problem hiding this comment.
Good call. reworked this to reuse TextBlock instead of adding a block type. The compaction is now stored as a normal summary text block with an optional opaque providerData field (same pattern as ToolUseBlock.reasoningSignature), which only AnthropicModel reads to send the signed block back. It's additive, so there's no new type for customers to handle: they see a text block with the summary
| model: new AnthropicModel({ | ||
| modelId: 'claude-fable-5-1', | ||
| params: { thinking: { type: 'adaptive' } }, | ||
| requiresServerSideCompaction: true, // the only new line |
There was a problem hiding this comment.
If the user leaves requiresServerSideCompaction as false on preserved thinking models, should we warn at construction time? Or better yet, set it for them by model type. My concern is that users who don't know about this setting will silently use a broken compaction strategy by default.
There was a problem hiding this comment.
I agree. I think moving this field into the models metadata should help to clear this up for users
…el.compact() and model metadata
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| #### Codebase changes | ||
|
|
||
| Where is the "path" checked in the codebase for summarize to function normally to users? |
There was a problem hiding this comment.
Issue: A few authoring artifacts / notes-to-self are still in the doc body and read as unfinished:
- Line 13: a leftover HTML comment
<!-- Is there any mention of thinking blocks in any other design document? --> - Line 327: the heading sub-line is itself a question — "Where is the "path" checked in the codebase for summarize to function normally to users?"
- Line 778 (appendix script): inline
# this potentially needs to be Offload.summarize perhaps
Suggestion: Resolve or remove these. Open design questions are better consolidated under the existing "Open Questions" section so reviewers know they're intentional rather than stray edits.
|
|
||
| ## Alternative: Categorizing Prefixes as Protected? | ||
|
|
||
| ## Alternative: Drop Thinking Blocks |
There was a problem hiding this comment.
Issue: Two of the "Alternative" sections are unfinished — ## Alternative: Categorizing Prefixes as Protected? (line 469) has no body at all, and ## Alternative: Drop Thinking Blocks ends on the dangling fragment "Anthropic does not recommend" (line 473).
Suggestion: Either complete these (even one or two sentences each on the pros/cons and why rejected, matching the other well-written alternatives above) or remove the headers. As-is they read as TODOs and leave a reader unsure whether an alternative was actually considered — which undercuts an otherwise strong "Alternatives" section.
|
|
||
| Server-side summarization through compaction with additional model providers. | ||
|
|
||
| OpenAI compaction uses the same `providerData` field with `provider: 'openai'`, so no new block type is needed. Open item: if OpenAI's compaction output has no readable summary text, the block would be an empty `text` with opaque `data`, and other providers would need to drop it rather than send an empty text block. OpenAI's actual output shape hasn't been checked yet. |
There was a problem hiding this comment.
Issue: The decision to store compaction as a TextBlock + providerData (rather than a dedicated CompactionBlock) rests on the invariant that text always holds a human-readable summary, so "every other provider ignores providerData and sends plain text" (line 305) stays safe. This Part 3 note flags a case where that invariant may not hold: "if OpenAI's compaction output has no readable summary text, the block would be an empty text... and other providers would need to drop it rather than send an empty text block."
That "other providers must special-case dropping the block" is exactly the cross-provider handling burden the rejected CompactionBlock alternative (line 455) was faulted for — so the chosen approach doesn't fully escape it; it just defers it.
Suggestion: State the invariant explicitly in "Storing the compaction as a summary" (e.g. a providerData-bearing TextBlock MUST carry a meaningful text summary), and note that providers which can't produce readable summary text are out of scope for this field. If that can't be guaranteed, this is worth revisiting before Part 3 rather than after, since it affects the Part 1 type contract.
| }) | ||
| ``` | ||
|
|
||
| Without native compaction, a client-side summary on these models fails the next request with a 400, because the kept thinking blocks no longer match the conversation that produced them. That's why `AnthropicModel.compact()` leaves the history unchanged instead of falling back to one on a history-bound model. |
There was a problem hiding this comment.
Issue: On a history-bound model, AnthropicModel.compact() deliberately returns undefined and leaves history unchanged when native compaction is unavailable or fails (lines 141-142, 419), to avoid invalidating preserved thinking. But the strategy fires precisely because of overflow/utilization — so "leave history unchanged" means the next request overflows too, and the agent can make no progress. The doc doesn't define what happens in that state.
The answer seems to already be in the doc: the "Client-side summarization that stays valid on preserved-thinking models" appendix (lines 482/496) describes a keep-tail summary with thinking stripped from the preserved tail that passes the prefix check on these models — yet it's left as a "candidate fallback" and the Proposal never adopts it.
Suggestion: Promote this to a defined behavior in the Proposal: when native compaction is unavailable or fails on a history-bound model, fall back to the valid strip-thinking summary rather than no-op'ing. That closes the "stuck agent on overflow" gap and also gives Bedrock Converse customers (line 245) a working path. (Related to lizradway's fallback thread.)
|
Assessment: Comment Docs-only design doc; the direction (route summarization through the provider's native compaction via an overridable Review themes
Nice work on the empirical grounding — the eval tables comparing native vs. client-side paths make the tradeoffs easy to evaluate. |
Human Overview
This design document is the first step in additions to the ContextManager class and the Model class to allow for server-side (native) context management. LLM providers may start forcing the context window to be managed through the server-side APIs, and the Strands SDK needs to be able to still support the core functionalities of ContextManager, even when restrictions are placed.
Description
Replaces the
0021-preserved-thinking*drafts with a single, re-scoped design doc (0022) for keeping Strands' context management working on preserved-thinking Claude models (Opus 5.5, Fable 5.1, Haiku 5.5, Sonnet 5.5). These models bind each thinking block's signature to the exact system prompt, tool list, and history that produced it; any client-side edit to that prefix (summarization, sliding-window trimming, truncation) invalidates the block and the next request fails.The doc narrows scope to Part 1 only: a
requiresServerSideCompactionflag onModelthat, when true, routes the summarizer through the provider's native compaction API instead of rewriting history locally. Tool-list and system-prompt mutation (Part 2) are explicitly called out as unsolved and deferred. A new Current State section grounds the proposal in today's actual behavior — context-reduction strategies have no concept of preserved thinking and will drop or invalidate thinking blocks already, which the sliding-window and summarizer repro scripts in the appendix demonstrate against live Bedrock models.This PR is docs-only; no SDK code changes are included.
Related Issues
Related to #4933
Documentation PR
N/A, design doc only.
Type of Change
Documentation update
Testing
Design doc only, no code changes. The appendix includes manual repro scripts run against Bedrock (
sliding_window_thinking.py-style tests) that reproduce the 400 the design is meant to fix; these aren't part of the SDK test suite.hatch run prepareAdditional Details
Extended context (optional)
Open questions tracked in the doc (not yet resolved): how to align with Anthropic's compact API shape, whether other providers need their own compaction block type, and what happens for Bedrock customers without a direct Anthropic API key. Part 2 (tool list / system prompt mutation) is intentionally left as TODO pending a follow-up design.
Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.