Repository navigation
Noise-aware fitness + a stochastic benchmark task (#12) - #14
Merged
Merged
Conversation
Closes #12. For STOCHASTIC evaluators a single evaluation is a noisy sample of a program's true quality, so selection (MAP-Elites elites, archive, parent sampling, best tracking) can be fooled into keeping a mediocre program that got a lucky draw. Conceptually this treats a noisy score as *partial observability* of the true fitness, and a re-evaluation as an "observation action" that reduces the uncertainty before committing — the framing is borrowed from contingent planning under partial observability: Jörg Hoffmann and Ronen I. Brafman, "Contingent Planning via Heuristic Forward Search with Implicit Belief States", ICAPS 2005. https://cdn.aaai.org/ICAPS/2005/ICAPS05-008.pdf - config: evaluator.re_evaluations (default 1 = unchanged). When >1, each program is evaluated that many times and the numeric metrics (incl. combined_score) are averaged (variance ~1/n), recording combined_score_std and n_evals. Leave at 1 for deterministic tasks. - evaluator: after a successful eval, average re_evaluations independent runs via _average_metrics (numeric-in-every-sample keys averaged; non-numeric and bool keys take the last). Only re-evaluates programs that ran successfully. - examples/noisy_optimization/: a purpose-built stochastic task — evolve an 8-vector toward a hidden target; evaluate() reports true quality + fresh entropy-seeded Gaussian noise each call; true_evaluate() is noiseless, used only by the benchmark to judge the selected program. construct() runs in an isolated subprocess so the candidate can't read the target out of the evaluator's memory (a reward-hack the agent found and we closed). Validated A/B (3 seeds, judged by the TRUE score of the program each run selected): re_evaluations=3 vs 1 lifts the selected best's true score 0.710 -> 0.855 (+0.145) and halves the noise-inflation of the reported best (+0.18 -> +0.08). 9 new tests; full suite green (311). Default-off, so deterministic tasks are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12.
Problem
For stochastic evaluators, a single evaluation is a noisy sample of a program's true quality. KaiEvolve makes every keep/discard decision on
combined_score(MAP-Elites elites, archive, parent sampling, best-program tracking, the bandit reward), so a mediocre program that gets a lucky draw can displace a genuinely better one — and the reported "best" may not reproduce.Idea (and the paper that seeded it)
Treat a noisy score as partial observability of the true fitness, and a re-evaluation as an "observation action" that reduces the uncertainty before committing. The framing is borrowed from contingent planning under partial observability:
Change
config:evaluator.re_evaluations(default 1 = unchanged). When >1, each program is evaluated that many times and the numeric metrics (incl.combined_score) are averaged (variance ~1/n), recordingcombined_score_std+n_evals.evaluator: after a successful eval, averagere_evaluationsindependent runs via_average_metrics(a key is averaged only when numeric in every sample; non-numeric / bool take the last). Re-evaluates only programs that ran successfully.examples/noisy_optimization/: a purpose-built stochastic task — evolve an 8-vector toward a hidden target;evaluate()reports true quality + fresh entropy-seeded Gaussian noise each call;true_evaluate()is noiseless, used only by the benchmark to judge the selected program.construct()runs in an isolated subprocess so the candidate can't read the target out of the evaluator's memory — a reward-hack the agent found (it usedinspect.stack()to stealTARGET) and we closed.Validation (A/B, judged by the TRUE noiseless score of the selected best)
3 seeds, 30 iters; only
re_evaluationsdiffers:Δ true-score +0.145, and the noise-inflation of the reported best is halved (+0.18 → +0.08) — both as designed. (3 seeds, so treatment variance is high, but baseline is tightly stuck at ~0.71 and never escapes, while treatment reaches scores it can't.)
Tests
9 new tests (
_average_metricslogic + the task's noisiness +true_evaluatedeterminism). Full suite green (311). Default-off, so deterministic tasks are unaffected.🤖 Generated with Claude Code