refactor(python-sdk): llm.ThinkingConfig #1847
Open
+11,251
−10,592
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
v2-docs | 165afa7 | Jan 09 2026, 02:07 AM |
willbakst
reviewed
Jan 8, 2026
| "minimal": 0, | ||
| "low": 1024, | ||
| "medium": 2048, | ||
| "high": 8192, |
Collaborator
There was a problem hiding this comment.
hmm, this seems weird:
- should be
LEVEL_TO_BUDGET? - I feel like this should map to a percentage of max tokens? Seems weird that we hard limit the user to 8192 thinking budget tokens
| from ..model_id import AnthropicModelId, model_name | ||
|
|
||
| DEFAULT_MAX_TOKENS = 16000 | ||
| # TODO: Change DEFAULT_FORMAT_MODE to strict when strict is no longer a beta feature. | ||
| DEFAULT_FORMAT_MODE = "tool" | ||
|
|
||
| # Effort level to token budget mapping for thinking conversion | ||
| EFFORT_TO_BUDGET = { | ||
| "minimal": 0, |
Collaborator
There was a problem hiding this comment.
seems weird that "minimal" here is 0, meaning actually disabled? that seems semantically incorrect. I thought we spoke about having something like disabled: True in the config for this? I feel like minimal should still set some form of budget?
| from typing import TypedDict | ||
| from typing import Literal, TypedDict | ||
|
|
||
| ThinkingLevel = Literal["minimal", "low", "medium", "high"] |
Collaborator
There was a problem hiding this comment.
nit: missing docstring. I like having the longer one in ThinkingConfig, so a single line here suffices (can even be 1:1 copy-paste with the short description below)
| from ..model_id import GoogleModelId, model_name | ||
| from ..model_info import MODELS_WITHOUT_STRUCTURED_OUTPUT_AND_TOOLS_SUPPORT | ||
|
|
||
| UNKNOWN_TOOL_ID = "google_unknown_tool_id" | ||
|
|
||
| EFFORT_TO_BUDGET = { |
llm.ThinkingConfig has level (minimal / low / medium / high) and can be unset for auto. It also has encode_thoughts_as_text I dropped include_summary because really it's only Google that let's you enable/disable summaries (openai allows auto / verbose / concise, but concise is only for computer use models apparently). Since we don't have at least 2 providers with consistent semantics, I am just turning them on for Google by default for consistent behavior. If someone wants the ability to disable them for Google we can add that in 2.x once we get a request for it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
llm.ThinkingConfig has level (minimal / low / medium / high) and can be
unset for auto. It also has encode_thoughts_as_text
I dropped include_summary because really it's only Google that let's you
enable/disable summaries (openai allows auto / verbose / concise, but
concise is only for computer use models apparently). Since we don't have
at least 2 providers with consistent semantics, I am just turning them
on for Google by default for consistent behavior. If someone wants the
ability to disable them for Google we can add that in 2.x once we get a
request for it.