You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[optional mode] Improvement-over-parent, cost-aware reward for model selection #2
Add an optional, config-gated reward mode for the LLM-ensemble bandit (Thompson sampling) that rewards a model by the improvement it actually contributed, not the absolute score of what it produced.
New mode: reward = improvement of the child over its parent's score, clamped at zero so regressions don't earn reward (optionally passed through a shaping function that emphasizes rare large jumps).
Optionally blend in a cost term so that, on otherwise-equal contribution, cheaper models are preferred.
This is selectable via config and off by default. The existing absolute-combined_score reward remains the default behavior; the new mode does not replace it.
Why
The current bandit rewards models by the absolute combined_score of the program they generate. That over-credits a model that merely mutated an already-strong parent without adding much, and under-credits a model that makes a rare, bold jump from a weak parent. Rewarding the marginal improvement a model contributed should bias model selection toward models that actually drive progress, improving sample efficiency. The optional cost term lets us trade quality against spend. Shipping it as an opt-in mode lets us A/B it against the current system without committing the whole pipeline to it.
Where
kaievolve/llm/ensemble.py — update_reward path; thread the parent score through so the delta can be computed.
Config flag to select reward mode (and cost weighting).
Validation
Benchmark run with the mode on vs off (both tasks); compare best score, sample efficiency, and cost.
Activity
Merge pull request #11 from firstbatchxyz/feat/improvement-reward-mode