Skip to content

[optional mode] Improvement-over-parent, cost-aware reward for model selection #2

Closed

Description

@aktasbatuhan

What

Add an optional, config-gated reward mode for the LLM-ensemble bandit (Thompson sampling) that rewards a model by the improvement it actually contributed, not the absolute score of what it produced.

  • New mode: reward = improvement of the child over its parent's score, clamped at zero so regressions don't earn reward (optionally passed through a shaping function that emphasizes rare large jumps).
  • Optionally blend in a cost term so that, on otherwise-equal contribution, cheaper models are preferred.
  • This is selectable via config and off by default. The existing absolute-combined_score reward remains the default behavior; the new mode does not replace it.

Why

The current bandit rewards models by the absolute combined_score of the program they generate. That over-credits a model that merely mutated an already-strong parent without adding much, and under-credits a model that makes a rare, bold jump from a weak parent. Rewarding the marginal improvement a model contributed should bias model selection toward models that actually drive progress, improving sample efficiency. The optional cost term lets us trade quality against spend. Shipping it as an opt-in mode lets us A/B it against the current system without committing the whole pipeline to it.

Where

  • kaievolve/llm/ensemble.py — update_reward path; thread the parent score through so the delta can be computed.
  • Config flag to select reward mode (and cost weighting).

Validation

Benchmark run with the mode on vs off (both tasks); compare best score, sample efficiency, and cost.

Activity

  1. added a commit that references this issue on Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions