You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[optional mode] Second-stage novelty check that judges semantic difference #3
Add an optional, config-gated second stage to novelty detection: a lightweight LLM judge that decides whether a borderline candidate is meaningfully different.
Today novelty is decided purely by embedding cosine similarity against the population, with a fixed threshold.
New optional stage: when a candidate is above the similarity threshold (i.e., looks like a near-duplicate), escalate only that borderline candidate to a cheap LLM judge that answers "is this a substantive change?" before we spend a full evaluation on it.
Off by default and config-gated. The embedding-only path stays the default; the judge only runs on borderline cases when enabled.
Why
Embedding similarity catches textually-similar programs but can't reliably distinguish a trivial rename or reformat from a substantive algorithmic change (or the reverse). A cheap LLM judge applied only to borderline candidates can:
avoid wasting evaluations on true duplicates, and
avoid discarding changes that are semantically novel despite looking similar.
We already compute the embeddings, so the gate runs on a small subset and adds little cost. Making it optional lets us measure whether the extra LLM call pays for itself before adopting it broadly.
Where
kaievolve/novelty.py — add the second-stage judge between the embedding check and kaievolve/evaluator.py.
Config flag to enable the judge (and pick the model/threshold band it applies to).
Validation
Benchmark run with the judge on vs off; watch evaluation count, duplicate rate, and final score.
Tested, not merged — the second-stage LLM novelty judge does not earn its keep on our tasks
Implemented the rescue-only judge (branch feat/novelty-judge-mode, default-off): when the embedding check flags a near-duplicate (sim ≥ 0.95), a cheap LLM judge decides whether the change is meaningfully different in approach; if so it's rescued as novel. It only ever loosens the embedding gate (rescue, never reject), so it can't starve exploration. Benchmarked in two parts.
Part 1 — controlled precision on real near-duplicate pairs
Embedded the 979 unique-code programs from the completed ablation runs locally, found the 15,398 pairs in the near-duplicate band (sim ≥ 0.95, 3.2% of all pairs), and ran the production judge on a 60-pair sample.
Spot-check: a mix — some genuinely different (single-start → multi-start, added hexagonal layout) and some plain refactors (function renames, import reordering).
A naive read of this looks like a strong win. It is misleading, because those pairs are distinct cross-lineage programs from across all runs — that distribution does not occur as in-loop duplicates.
Part 2 — in-loop A/B (the decision-maker)
Novelty enabled in both arms (so the judge can fire), auto task, 30 iters, 2 seeds; the only difference is llm_judge_enabled. Judge activity confirmed in logs (Novelty LLM judge enabled on all 4 workers; baseline has none).
arm
best-score
judge rescues
regen retries
cost
novelty on, judge off
0.4055 ±0.015
—
22
$0.226
novelty on, judge on
0.4057 ±0.022
0
23
$0.289 (+27%)
The judge fired on every in-loop duplicate, rescued zero, left the score unchanged (within noise), and added +27% cost.
Why
In the actual loop, an embedding-flagged duplicate is the LLM regenerating near-identical code from the same parent — a genuine repeat — which the judge correctly declines to rescue. Part 1's 75% came from comparing programs that evolved on separate lineages and only look similar; that case doesn't arise as an in-loop "duplicate." So the judge is correct to rescue ~nothing here, and is pure cost overhead.
Decision
Not merging. On our optimization tasks the judge adds cost for no benefit.
It could help in a domain where textually-similar code genuinely hides different logic (e.g. multi-file / real-software evolution), which our current tasks don't produce as in-loop duplicates. Re-open / revisit there if/when we benchmark such a task.
Notes:
The branch feat/novelty-judge-mode is shelved (not deleted).
A real robustness bug was surfaced and fixed on that branch: _judge_meaningfully_different was swallowing all exceptions and silently returning "not different" — it now logs loudly and self-disables after repeated failures. (It lives only inside the judge code, so there's nothing to fix on main.)
Activity
aktasbatuhan commentedon Jun 2, 2026
Tested, not merged — the second-stage LLM novelty judge does not earn its keep on our tasks
Implemented the rescue-only judge (branch
feat/novelty-judge-mode, default-off): when the embedding check flags a near-duplicate (sim ≥ 0.95), a cheap LLM judge decides whether the change is meaningfully different in approach; if so it's rescued as novel. It only ever loosens the embedding gate (rescue, never reject), so it can't starve exploration. Benchmarked in two parts.Part 1 — controlled precision on real near-duplicate pairs
Embedded the 979 unique-code programs from the completed ablation runs locally, found the 15,398 pairs in the near-duplicate band (sim ≥ 0.95, 3.2% of all pairs), and ran the production judge on a 60-pair sample.
A naive read of this looks like a strong win. It is misleading, because those pairs are distinct cross-lineage programs from across all runs — that distribution does not occur as in-loop duplicates.
Part 2 — in-loop A/B (the decision-maker)
Novelty enabled in both arms (so the judge can fire), auto task, 30 iters, 2 seeds; the only difference is
llm_judge_enabled. Judge activity confirmed in logs (Novelty LLM judge enabledon all 4 workers; baseline has none).The judge fired on every in-loop duplicate, rescued zero, left the score unchanged (within noise), and added +27% cost.
Why
In the actual loop, an embedding-flagged duplicate is the LLM regenerating near-identical code from the same parent — a genuine repeat — which the judge correctly declines to rescue. Part 1's 75% came from comparing programs that evolved on separate lineages and only look similar; that case doesn't arise as an in-loop "duplicate." So the judge is correct to rescue ~nothing here, and is pure cost overhead.
Decision
Not merging. On our optimization tasks the judge adds cost for no benefit.
It could help in a domain where textually-similar code genuinely hides different logic (e.g. multi-file / real-software evolution), which our current tasks don't produce as in-loop duplicates. Re-open / revisit there if/when we benchmark such a task.
Notes:
feat/novelty-judge-modeis shelved (not deleted)._judge_meaningfully_differentwas swallowing all exceptions and silently returning "not different" — it now logs loudly and self-disables after repeated failures. (It lives only inside the judge code, so there's nothing to fix onmain.)