[benchmark][BEAM] Address retrieval and context construction issues exposed by the 128k run

[hrw991009]

Summary

The first complete BEAM 128k diagnostic run finished all 400 questions without execution errors, but the evidence chain exposed several retrieval and context-construction issues that prevent the result from being treated as a reliable measurement of OpenContext semantic memory.

This issue tracks five problems and possible minimal solutions. The goal is to make the next BEAM 128k run attributable, efficient, and reproducible without changing the existing nugget scoring behavior.

See: benchmark/beam/docs/128k-v1-test-report.md

Current baseline

Metric	Result
Questions completed	400 / 400
Execution errors	0
Nugget Mean	0.5305
Pass Rate	56.75%
Hit@8	92.11%
Mean Source Recall@8	0.8027
All Required Sources Retrieved	65.35%
Precision@8	0.2011
MRR	0.2958
Total token usage	31,456,385

This is a diagnostic baseline, not an official leaderboard claim or a pure semantic-retrieval score.

Problem 1: Final Top-K results are almost entirely lexical-only

Of the 3,200 final Top-K hits:

• 3,199 were lexical-only;
• 1 contained both lexical and semantic;
• 0 were semantic-only.

This does not prove that semantic search was never executed. The current trace only records the merged Top-K, so semantic candidates may have been absent, ineffective, or removed during merging.

Possible solutions

• Record semantic and lexical candidate lists before merging.
• Record candidate counts, ranks, scores, and final channel membership.
• Explicitly pass and record mergeStrategy: "rrf" instead of relying on daemon defaults.
• Add a semantic-only fixture whose query and relevant memory have no direct keyword overlap.
• Verify raw-message count, embedded-row count, vector dimensions, and ANN index count.
• Record the daemon backend, embedding provider/model, token limit, and merge strategy in the run manifest.

The first implementation should prefer instrumentation and a small fixture over adding a new retrieval framework.

Problem 2: Retrieval chunks exceed the embedding input limit

The runner currently combines 20 turns into one memory message.

Observed chunk sizes:

Metric	Characters
Average	41,712
P50	40,834
P95	58,259
Maximum	376,965

The repository's local embedding provider defaults to 512 tokens and enables truncation. If that provider is active, a large part of each chunk cannot contribute to its semantic representation.

Possible solutions

• Use one upstream source turn as the default retrieval unit.
• Split an individual turn further when it exceeds the configured embedding limit.
• Preserve source ID, speaker, timestamp, and deterministic sub-chunk ID after splitting.
• Use a small token overlap for split turns where necessary.
• Validate chunk token length before ingest and report truncation explicitly.
• Consider a longer-context embedding model only after correcting the retrieval-unit size.

The preferred first approach is token-aware chunking aligned with upstream source turns.

Problem 3: Multi-evidence recall and ranking are incomplete

Hit@8 is high, but only 65.35% of applicable questions retrieved every required source. Precision@8 and MRR are also low.

This particularly affects event ordering, summarization, temporal reasoning, and multi-session reasoning.

Possible solutions

• Overfetch semantic and lexical candidates separately before fusion.
• Fuse candidates using RRF so different score scales are not compared directly.
• Add a lightweight reranker over the small fused candidate set.
• Preserve timestamps and organize temporal evidence chronologically.
• For questions requiring several facts, retrieve separate evidence facets and deduplicate the combined results.
• Evaluate existing query rewrite or iterative retrieval only after the base semantic path is verified.
• Do not increase final Top-K as the first response, because that would further increase prompt size and cost.

The first comparison should keep the final Top-K unchanged and measure before/after retrieval metrics on a fixed sample.

Problem 4: Answerer context and token usage are excessive

Each question currently receives eight complete chunks.

Observed Answerer input:

Metric	Result
Average context characters	343,439
Average prompt tokens	74,191
P95 prompt tokens	100,504
Maximum prompt tokens	105,620

The Answerer consumed 29.7M input tokens across 400 questions. Large, noisy prompts increase cost and make it harder to identify updates, timestamps, and event order.

Possible solutions

• Introduce an explicit Answerer prompt-token budget.
• Send smaller retrieval units instead of complete 20-turn chunks.
• Deduplicate overlapping or repeated evidence.
• Keep only relevant spans and necessary neighboring turns.
• Prefer results that add previously uncovered source evidence.
• Record prompt size before and after trimming, including removed hit IDs.
• Keep timestamps and source IDs in the final evidence.
• Treat prompt caching as a cost optimization only, not as a quality fix.

A reasonable initial calibration target is a P95 Answerer prompt below 16K tokens.

Problem 5: Many answers fail even when annotated sources are present

There were 94 context_present_answer_failed cases. These include:

• knowledge update: 18;
• contradiction resolution: 13;
• temporal reasoning: 13;
• instruction following: 12;
• multi-session reasoning: 10.

This status only means that chunks containing every annotated source turn entered the prompt. It does not prove that the relevant evidence was concise, correctly ordered, or easy for the Answerer to use.

Possible solutions

• Select a fixed stratified sample of 30–50 failed questions.
• Run each question with:
  1. the actual retrieved context;
  2. only the upstream gold evidence;
  3. optionally, the complete conversation as an oracle comparison.
• Keep the Answerer, Judge, and scoring prompt unchanged during this comparison.
• Sort temporal, event-ordering, contradiction, and knowledge-update evidence chronologically.
• Use structured Answerer output containing the answer and cited evidence IDs.
• Compare the current Flash Answerer with a stronger model only on the fixed diagnostic sample.
• Change the Answerer or prompt only if failures remain when clean gold evidence is supplied.

This experiment should separate retrieval/context failures from Answerer capability before another full paid run.

Suggested implementation order

1. Add pre-merge semantic/lexical diagnostics and a semantic calibration fixture.
2. Replace fixed 20-turn chunks with token-aware source-aligned chunks.
3. Add explicit fusion behavior and a bounded Answerer context.
4. Run fixed-sample retrieval and gold-evidence comparisons.
5. Review the evidence and authorize a fresh full 128k run separately.

Scope

• Improve the existing BEAM retrieval and context path.
• Reuse current OpenContext search, RRF, reasoning, and reranking capabilities where appropriate.
• Preserve upstream source provenance.
• Preserve the existing BEAM nugget scoring behavior.
• Add only the tests and diagnostics required to verify these five problems.

Out of scope

• Adding another benchmark or dataset.
• Building a new general benchmark framework.
• Changing nugget atoms or the Judge rubric.
• Increasing Top-K or changing models only to improve the headline score.
• Running 500k, 1m, or 10m.
• Running a new full paid 128k evaluation without separate approval.
• Treating dataset annotation errors as OpenContext failures.

Acceptance criteria

• A controlled semantic fixture proves that the configured embedding and ANN path works.
• Trace output shows semantic candidates, lexical candidates, and the final merged results separately.
• The run manifest records the effective retrieval and daemon configuration.
• Retrieval units fit within the configured embedding token limit.
• Upstream source IDs remain traceable after token-aware splitting.
• A fixed calibration sample reports before/after Recall@K, Hit@K, Precision@K, MRR, complete-source retrieval, and channel distribution.
• The Answerer context has an explicit token budget and reports average and P95 usage.
• Gold-evidence comparison separates retrieval/context failures from Answerer failures.
• Existing nugget scoring and pass-threshold behavior remain unchanged.
• Offline fixture, typecheck, lint, resume, and diagnostic tests pass.
• No large dataset download or paid full benchmark run is required for implementation acceptance.