Repository navigation
Optional improvement-over-parent / cost-aware bandit reward (#2) - #11
Merged
Merged
Conversation
Closes #2. Adds an opt-in reward mode for the Thompson model-selection bandit. The default rewards a model by the child's absolute combined_score, which over-credits a model that merely mutated an already-strong parent. The new "improvement" mode rewards the gain over the parent (regressions clamped to 0), crediting a model for what it actually contributed; an optional cost discount makes cheaper models win on ties. - llm/ensemble.shape_reward(child, parent, mode, cost, cost_coefficient): "absolute" (default) | "improvement", cost-discounted, clamped to [0,1]. - Both reward sites (parallel process_parallel + serial iteration) compute the shaped reward; parent + config are in scope, ensemble exposes last-call cost. No ensemble-construction plumbing touched. - config: reward_mode / reward_cost_aware / reward_cost_coefficient on LLMConfig, in to_dict. Defaults preserve current behavior exactly. Default-off: with reward_mode="absolute" the reward equals the old value, so the default loop is unchanged. Quick A/B (auto, best setup, 2 seeds, only reward_mode differs) was inconclusive: improvement 0.3960 (seeds 0.4406 / 0.3514) vs absolute 0.3891 (0.3841 / 0.3942) -- Δ +0.007, within noise, with notably higher variance. The improvement signal (child - parent) is itself noisy, so this mode is shipped off-by-default for experimentation rather than promoted; a noise-aware re-evaluation pass would be the natural way to make it pay off. 10 new tests; full suite green (295). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2. Opt-in mode, default off — no change to default behavior.
What
Adds a config-selectable reward mode for the Thompson model-selection bandit.
absolute(default): reward a model by the child's owncombined_score— the current behavior, which over-credits a model that merely mutated an already-strong parent.improvement: reward the gain over the parent (regressions clamped to 0), crediting a model for what it actually contributed. Optional cost discount (reward_cost_aware) so cheaper models win on otherwise-equal contribution.How
llm/ensemble.shape_reward(child, parent, mode, cost, cost_coefficient)— pure helper, clamped to[0,1]for the Beta-Thompson update.process_parallel, serialiteration) compute the shaped reward —parent+ config are in scope and the ensemble exposes last-call cost, so no ensemble-construction plumbing was touched.config:reward_mode/reward_cost_aware/reward_cost_coefficientonLLMConfig, into_dict. Defaults reproduce the old reward exactly.Benchmark (honest: inconclusive)
auto task, best setup, 2 seeds, only
reward_modediffers:Δ +0.007 — within noise, with higher variance: improvement produced both the best run and a below-baseline run. The
child − parentsignal is itself noisy, so the bandit chases lucky deltas. Shipped off by default for experimentation rather than promoted; a noise-aware re-evaluation (treating fitness as a belief) is the natural way to make this mode pay off, and is tracked separately.Tests
10 new tests in
tests/test_reward_mode.py(absolute = child score, improvement = gain, regression → 0, no-parent fallback, cost discount, clamping, non-numeric safety, config round-trip). Full suite: 295 OK.🤖 Generated with Claude Code