[FEATURE] Evaluate base summarization strategies across models before fixing defaults #5734

SDK Language

Both / Not language-specific

Problem Statement

The base summarization defaults are currently chosen by argument, not by measurement:

• whether message 0 is kept;
• the request shape;
• where the summary goes;
• which model writes it.

The one live eval so far (gist from #5299) compared two request shapes on 6 Bedrock models. It showed that these choices move the results a lot:

• answered-instead-of-summarized: 8% → 1%;
• fact recall: 90% → 83%;
• an 88% silent text-only fallback on main for tool histories.

That eval measured one summary in isolation. It did not measure what the agent does next, repeated passes, or direct Anthropic/OpenAI providers.

Proposed Solution

A reusable eval harness (kept outside the published package, for example under strands-py/tests_integ/ or a scripts directory) that runs a full agent through repeated summarization and scores the outcome.

Knobs (each a summarizer or strategy setting):

• Keep message 0 verbatim vs. summarize it.
• Request shape:
  • flattened transcript in one user message (main);
  • replay of the agent's own request with a trailing instruction (feat(context-manager): batch summarization replays the agent's own request plus one instruction #5299);
  • native compaction where available.
• Summarizer model: aux_model (small) vs. the agent's main model.
• System prompt / instruction variant (see [FEATURE] Refine the base summarizer system prompt #5735).
• Placement: a standalone summary message vs. joining a neighbor.
• Summary cue: the rendered [Summarized: …] header vs. plain text (once summaries carry metadata.offloaded).
• preserve_recent and trigger threshold.

Models: the gist's 6 Bedrock models, plus at least one direct Anthropic and one OpenAI provider, and a preserved-thinking model, which can only run the native path.

Metrics:

• Per summary (from the gist):
  • answered-instead rate (LLM judge);
  • planted-fact recall;
  • model calls per summary;
  • fallback/rejection rate;
  • summary length;
  • prompt-cache hit rate.
• Downstream (new):
  • task success after N summarization passes;
  • goal drift: does the agent still pursue message 0's task, judged against the original;
  • tokens freed per pass;
  • total cost.

Dataset: the gist's 8 tool-using slices, plus long multi-turn tasks that force 3+ passes. Some should open with a task statement that later turns depend on (the message-0 question), and some with tool results carrying identifiers.

Acceptance

• The harness runs from one command against any provider/model list and writes a results table like the gist's.

• A results write-up that recommends defaults for:

  • the message-0 policy;
  • the request shape;
  • the summarizer model;

  posted to this issue and linked from [FEATURE] Separate native compaction from base summarization, replacing the placeholder Model.summarize() #5733.

• Defaults changed only in a separate PR that cites the results.

Related: #5410 (also covers Part 1's "comparison of native and client-side summaries"), #5470, #5299, #5733, #5735