Normalize sampler results between synthesis and no-synthesis paths

[dan-s-w]

Problem

format_results is documented as returning either a list of search results or a single already-written answer. But the two downstream paths each require one specific shape, and neither was enforced.

parallel_pro and parallel_ultra hit this. ParallelTaskSampler.format_results returns a str, while both configs leave needs_synthesis at its True default. So trim_results_to_model_limit iterated the answer one character at a time and rejoined it with a separator between every character:

'R\n---\ne\n---\nv\n---\ne\n---\nn\n---\nu\n---\ne\n---\n \n---\nw\n---\na\n---\ns ...'

That is what the synthesis model received instead of the answer — at roughly 7x the token count. It doesn't crash, which is why it went unnoticed; the model can still partially read it. Worth considering whether this depresses parallel_pro's 34.45% on the FinSearchComp T2 table in the README.

The mirror case also exists. exa_research_pro and tavily_research_pro return list[str] with needs_synthesis=False, so generated_answer reaches the grader as a Python list repr (['the answer']) rather than the answer text.

Approach

Fixed in the base class rather than in ParallelTaskSampler, because the mismatch is a contract problem — any new sampler can reintroduce it, and two already had. Samplers stay free to return whichever shape is natural for their provider; normalize_formatted_results coerces to what the configured path needs.

The abstract format_results signature is corrected to -> str | list[str], which is what implementations were already doing.

Heads up on scores

This changes what actually reaches the synthesis model and the grader for four samplers: parallel_pro, parallel_ultra, exa_research_pro, tavily_research_pro. Their numbers are expected to move. I'd treat existing results for those four as not comparable to post-merge runs.

Happy to split the no-synthesis half (the Exa/Tavily join) into its own PR if you'd rather land the two behaviour changes separately.

Tests

tests/test_result_normalization.py — pure function, no network. Includes a test asserting the synthesis-side join round-trips the text intact, which is the specific corruption above.

🤖 Generated with Claude Code