feat: add result metadata and structural list truncation to execute_code - #22
Closed
juliusguay wants to merge 1 commit into
Closed
feat: add result metadata and structural list truncation to execute_code#22juliusguay wants to merge 1 commit into
juliusguay wants to merge 1 commit into
Conversation
Adds result_type and result_count fields to ExecutionResult so the LLM has an authoritative anchor for result shape without recounting from the payload. result_count is computed before sanitize_output runs, so it reflects the true pre-truncation total for list results. Large list/tuple results are now capped at MAX_RESULT_ITEMS (200) at element boundaries rather than mid-JSON, preventing the model from attempting to complete malformed structures with invented data. The truncation notice is appended as the final list element so the cutoff is explicit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
juliusguay
requested review from
josha-bentley,
rgonzalo-orellana and
silvestre-perret-bentley
as code owners
June 1, 2026 21:19
Member
|
Solved by #18. Agreed to close this PR in an internal discussion with @JJGIV2010. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
result_type,result_count, and JSON-aware list truncation to theexecute_codetool response. This establishes a result-metadata contract thatanchors the LLM's narrative and caps unbounded payloads at clean element
boundaries.
Problem
execute_codereturned raw results with no payload cap. A 300-member forceextraction returned all 300 items unbounded. If a downstream character truncator
cut the JSON mid-list, the LLM would attempt to "complete" the malformed structure
with invented data — a fabrication pattern with real consequences in structural
analysis workflows.
Without a count field, there was no way for the LLM to distinguish a truncated
list from a complete one. A model reporting "38 members failed" from a truncated
result where only 38 were visible is indistinguishable from one that saw all members.
Changes
MAX_RESULT_ITEMS = 200added tosandbox/const.pysanitize_outputinsandbox/stdio_helpers.pynow truncates lists at elementboundaries and appends a truncation notice as the final item (no mid-JSON cuts)
_classify_result()runs beforesanitize_outputinexecutor.pysoresult_countreflects the true pre-truncation totalresult_typeandresult_countadded toExecutionResultand emitted into_dict()execute_codedocstring and server instructions updated to declareresult_countas the authoritative countToken impact
For large models where the list exceeds 200 items,
result_countbecomes theonly way to know the true total — the LLM can accurately report "5,000 members
analyzed, showing first 200" without fabricating counts.
For small models (< 200 items), no truncation triggers and
result_countadds~2 tokens of overhead only.
Tests
11 new tests in
TestResultAnchoringcovering:"list"classification (COM APIs likeGetPrimaryLoadCaseNumbersreturn tuples)result_count== true total when list is cappedresult_type: "null"All 132 executor tests pass.
Questions or feedback: julius.guay@bentley.com