Skip to content

fix: UI sample_size ignored due to Planner LLM default override#29

Open
JoonYong-Park wants to merge 1 commit intomainfrom
fix/sample-size-override
Open

fix: UI sample_size ignored due to Planner LLM default override#29
JoonYong-Park wants to merge 1 commit intomainfrom
fix/sample-size-override

Conversation

@JoonYong-Park
Copy link
Member

Summary

  • Fixed a bug where the custom sample size set in the UI was always overridden by the Planner LLM's default value (100)
  • The orchestrator now explicitly overrides sample_size in both plan_metadata.config and plan_yaml with the user's actual selection after receiving the planner response

Root Cause

In generate_leaderboard(), only query.query (the text string) was passed to planner_agent.create_evaluation_plan(). The user's query.sample_size (derived from the UI's sample_scale setting) was never forwarded. Since the Planner LLM prompt defaults to sample_size: 100, the UI setting was silently ignored.

Before / After

Before — Custom(10) set but 100 samples evaluated

image

After — Custom(7) set and 7 samples evaluated correctly

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant