Skip to content

feat: add result metadata and structural list truncation to execute_code - #22

Closed
juliusguay wants to merge 1 commit into
BentleySystems:mainfrom
juliusguay:feat/result-metadata
Closed

feat: add result metadata and structural list truncation to execute_code#22
juliusguay wants to merge 1 commit into
BentleySystems:mainfrom
juliusguay:feat/result-metadata

Conversation

@juliusguay

Copy link
Copy Markdown

Summary

Adds result_type, result_count, and JSON-aware list truncation to the
execute_code tool response. This establishes a result-metadata contract that
anchors the LLM's narrative and caps unbounded payloads at clean element
boundaries.

Problem

execute_code returned raw results with no payload cap. A 300-member force
extraction returned all 300 items unbounded. If a downstream character truncator
cut the JSON mid-list, the LLM would attempt to "complete" the malformed structure
with invented data — a fabrication pattern with real consequences in structural
analysis workflows.

Without a count field, there was no way for the LLM to distinguish a truncated
list from a complete one. A model reporting "38 members failed" from a truncated
result where only 38 were visible is indistinguishable from one that saw all members.

Changes

  • MAX_RESULT_ITEMS = 200 added to sandbox/const.py
  • sanitize_output in sandbox/stdio_helpers.py now truncates lists at element
    boundaries and appends a truncation notice as the final item (no mid-JSON cuts)
  • _classify_result() runs before sanitize_output in executor.py so
    result_count reflects the true pre-truncation total
  • result_type and result_count added to ExecutionResult and emitted in
    to_dict()
  • execute_code docstring and server instructions updated to declare
    result_count as the authoritative count

Token impact

Scenario Before After Savings
300-member force extraction, single turn ~4,500 tokens ~3,012 tokens 33%
3-turn session (history re-sent each turn) ~13,500 tokens ~9,036 tokens ~4,500 tokens

For large models where the list exceeds 200 items, result_count becomes the
only way to know the true total — the LLM can accurately report "5,000 members
analyzed, showing first 200" without fabricating counts.

For small models (< 200 items), no truncation triggers and result_count adds
~2 tokens of overhead only.

Tests

11 new tests in TestResultAnchoring covering:

  • Type classification for all result types (scalar, bool, string, list, dict, null)
  • COM tuple → "list" classification (COM APIs like GetPrimaryLoadCaseNumbers return tuples)
  • Pre-truncation count invariant: result_count == true total when list is capped
  • Truncation notice appended as final element (not mid-array)
  • Error paths return result_type: "null"

All 132 executor tests pass.


Questions or feedback: julius.guay@bentley.com

Adds result_type and result_count fields to ExecutionResult so the LLM
has an authoritative anchor for result shape without recounting from the
payload. result_count is computed before sanitize_output runs, so it
reflects the true pre-truncation total for list results.

Large list/tuple results are now capped at MAX_RESULT_ITEMS (200) at
element boundaries rather than mid-JSON, preventing the model from
attempting to complete malformed structures with invented data. The
truncation notice is appended as the final list element so the cutoff
is explicit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@silvestre-perret-bentley

Copy link
Copy Markdown
Member

Solved by #18.

Agreed to close this PR in an internal discussion with @JJGIV2010.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants