Skip to content

Optional improvement-over-parent / cost-aware bandit reward (#2) - #11

Merged
aktasbatuhan merged 1 commit into
mainfrom
feat/improvement-reward-mode
Jun 2, 2026
Merged

aktasbatuhan merged 1 commit into
mainfrom
feat/improvement-reward-mode

Conversation

@aktasbatuhan

Copy link
Copy Markdown
Member

Closes #2. Opt-in mode, default off — no change to default behavior.

What

Adds a config-selectable reward mode for the Thompson model-selection bandit.

  • absolute (default): reward a model by the child's own combined_score — the current behavior, which over-credits a model that merely mutated an already-strong parent.
  • improvement: reward the gain over the parent (regressions clamped to 0), crediting a model for what it actually contributed. Optional cost discount (reward_cost_aware) so cheaper models win on otherwise-equal contribution.

How

  • llm/ensemble.shape_reward(child, parent, mode, cost, cost_coefficient) — pure helper, clamped to [0,1] for the Beta-Thompson update.
  • Both reward sites (parallel process_parallel, serial iteration) compute the shaped reward — parent + config are in scope and the ensemble exposes last-call cost, so no ensemble-construction plumbing was touched.
  • config: reward_mode / reward_cost_aware / reward_cost_coefficient on LLMConfig, in to_dict. Defaults reproduce the old reward exactly.

Benchmark (honest: inconclusive)

auto task, best setup, 2 seeds, only reward_mode differs:

arm mean seeds
absolute (default) 0.3891 0.3841, 0.3942 (tight)
improvement 0.3960 0.4406, 0.3514 (wide)

Δ +0.007 — within noise, with higher variance: improvement produced both the best run and a below-baseline run. The child − parent signal is itself noisy, so the bandit chases lucky deltas. Shipped off by default for experimentation rather than promoted; a noise-aware re-evaluation (treating fitness as a belief) is the natural way to make this mode pay off, and is tracked separately.

Tests

10 new tests in tests/test_reward_mode.py (absolute = child score, improvement = gain, regression → 0, no-parent fallback, cost discount, clamping, non-numeric safety, config round-trip). Full suite: 295 OK.

🤖 Generated with Claude Code

Closes #2.

Adds an opt-in reward mode for the Thompson model-selection bandit. The default
rewards a model by the child's absolute combined_score, which over-credits a
model that merely mutated an already-strong parent. The new "improvement" mode
rewards the gain over the parent (regressions clamped to 0), crediting a model
for what it actually contributed; an optional cost discount makes cheaper models
win on ties.

- llm/ensemble.shape_reward(child, parent, mode, cost, cost_coefficient):
  "absolute" (default) | "improvement", cost-discounted, clamped to [0,1].
- Both reward sites (parallel process_parallel + serial iteration) compute the
  shaped reward; parent + config are in scope, ensemble exposes last-call cost.
  No ensemble-construction plumbing touched.
- config: reward_mode / reward_cost_aware / reward_cost_coefficient on LLMConfig,
  in to_dict. Defaults preserve current behavior exactly.

Default-off: with reward_mode="absolute" the reward equals the old value, so the
default loop is unchanged.

Quick A/B (auto, best setup, 2 seeds, only reward_mode differs) was inconclusive:
improvement 0.3960 (seeds 0.4406 / 0.3514) vs absolute 0.3891 (0.3841 / 0.3942) --
Δ +0.007, within noise, with notably higher variance. The improvement signal
(child - parent) is itself noisy, so this mode is shipped off-by-default for
experimentation rather than promoted; a noise-aware re-evaluation pass would be
the natural way to make it pay off. 10 new tests; full suite green (295).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@aktasbatuhan
aktasbatuhan merged commit cb0b2f2 into main Jun 2, 2026
1 check passed
@aktasbatuhan
aktasbatuhan deleted the feat/improvement-reward-mode branch June 11, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[optional mode] Improvement-over-parent, cost-aware reward for model selection

1 participant