Capture reasoning traces from OpenRouter's 'reasoning' field - #2
Open
oy2017 wants to merge 1 commit into
Open
Conversation
OpenRouter returns reasoning in message.reasoning; the harness only read message.reasoning_content (DeepSeek/GLM native name), so traces were silently dropped. Fall back to 'reasoning' so traces are stored as <think> blocks in raw_output. Parsers are unaffected (all take the last bracketed match). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the existing
reasoning_content→<think>reconstruction (present since the initial release,b9691bf) to also cover OpenRouter's field name for the same data: OpenRouter's OpenAI-compatible API returns reasoning traces inmessage.reasoning, notmessage.reasoning_content(the DeepSeek/GLM native convention the code currently reads). The storage format is unchanged — traces are prepended toraw_outputas<think>\n…\n</think>\nexactly as the existing mechanism does; this PR only makes that mechanism reachable for one more API dialect.Without the fallback, runs through OpenRouter silently drop the traces while still paying for them: reasoning tokens show up in
usage.completion_tokens_details.reasoning_tokens, but the trace text never reachesraw_output. We hit this on full-benchmark runs ofz-ai/glm-5.2(~10.8M reasoning tokens billed, zero traces stored).Changes
api_model.py(OpenAIModel._call_api): readmessage.reasoning_content, falling back tomessage.reasoning.azure_batch_model.py: same fallback in the batch-results path (dict access). Theor-chain also tolerates an explicit"reasoning_content": null.Verification
OpenAIModelagainst OpenRouterz-ai/glm-5.2: returned output now begins with the full<think>trace.game_behavior_dictatorontencent/hy3via OpenRouter through the full CLI pipeline: traces stored inraw_output, 10/10 parsed, 0 failed parses, metrics computed normally.parse_numeric_output,parse_push_pull_output, boxed/bracket list parsing) all take the LAST match, so bracketed decoy values inside the trace (e.g. a trace quoting the[$x]format) cannot shadow the final answer — verified on trace-bearing outputs containing decoys.No behavior change for backends that don't emit a
reasoningfield (OpenAI/Azure, DeepSeek-style native endpoints, self-hosted servers).🤖 Generated with Claude Code