Skip to content

[optional mode] Second-stage novelty check that judges semantic difference #3

Closed as not planned

Description

@aktasbatuhan

What

Add an optional, config-gated second stage to novelty detection: a lightweight LLM judge that decides whether a borderline candidate is meaningfully different.

  • Today novelty is decided purely by embedding cosine similarity against the population, with a fixed threshold.
  • New optional stage: when a candidate is above the similarity threshold (i.e., looks like a near-duplicate), escalate only that borderline candidate to a cheap LLM judge that answers "is this a substantive change?" before we spend a full evaluation on it.
  • Off by default and config-gated. The embedding-only path stays the default; the judge only runs on borderline cases when enabled.

Why

Embedding similarity catches textually-similar programs but can't reliably distinguish a trivial rename or reformat from a substantive algorithmic change (or the reverse). A cheap LLM judge applied only to borderline candidates can:

  • avoid wasting evaluations on true duplicates, and
  • avoid discarding changes that are semantically novel despite looking similar.

We already compute the embeddings, so the gate runs on a small subset and adds little cost. Making it optional lets us measure whether the extra LLM call pays for itself before adopting it broadly.

Where

  • kaievolve/novelty.py — add the second-stage judge between the embedding check and kaievolve/evaluator.py.
  • Config flag to enable the judge (and pick the model/threshold band it applies to).

Validation

Benchmark run with the judge on vs off; watch evaluation count, duplicate rate, and final score.

Activity

  1. aktasbatuhan commented on Jun 2, 2026

    @aktasbatuhan
    MemberAuthor

    Tested, not merged — the second-stage LLM novelty judge does not earn its keep on our tasks

    Implemented the rescue-only judge (branch feat/novelty-judge-mode, default-off): when the embedding check flags a near-duplicate (sim ≥ 0.95), a cheap LLM judge decides whether the change is meaningfully different in approach; if so it's rescued as novel. It only ever loosens the embedding gate (rescue, never reject), so it can't starve exploration. Benchmarked in two parts.

    Part 1 — controlled precision on real near-duplicate pairs

    Embedded the 979 unique-code programs from the completed ablation runs locally, found the 15,398 pairs in the near-duplicate band (sim ≥ 0.95, 3.2% of all pairs), and ran the production judge on a 60-pair sample.

    • Rescue rate: 45/60 (75%) judged "meaningfully different."
    • Spot-check: a mix — some genuinely different (single-start → multi-start, added hexagonal layout) and some plain refactors (function renames, import reordering).

    A naive read of this looks like a strong win. It is misleading, because those pairs are distinct cross-lineage programs from across all runs — that distribution does not occur as in-loop duplicates.

    Part 2 — in-loop A/B (the decision-maker)

    Novelty enabled in both arms (so the judge can fire), auto task, 30 iters, 2 seeds; the only difference is llm_judge_enabled. Judge activity confirmed in logs (Novelty LLM judge enabled on all 4 workers; baseline has none).

    arm best-score judge rescues regen retries cost
    novelty on, judge off 0.4055 ±0.015 — 22 $0.226
    novelty on, judge on 0.4057 ±0.022 0 23 $0.289 (+27%)

    The judge fired on every in-loop duplicate, rescued zero, left the score unchanged (within noise), and added +27% cost.

    Why

    In the actual loop, an embedding-flagged duplicate is the LLM regenerating near-identical code from the same parent — a genuine repeat — which the judge correctly declines to rescue. Part 1's 75% came from comparing programs that evolved on separate lineages and only look similar; that case doesn't arise as an in-loop "duplicate." So the judge is correct to rescue ~nothing here, and is pure cost overhead.

    Decision

    Not merging. On our optimization tasks the judge adds cost for no benefit.

    It could help in a domain where textually-similar code genuinely hides different logic (e.g. multi-file / real-software evolution), which our current tasks don't produce as in-loop duplicates. Re-open / revisit there if/when we benchmark such a task.

    Notes:

    • The branch feat/novelty-judge-mode is shelved (not deleted).
    • A real robustness bug was surfaced and fixed on that branch: _judge_meaningfully_different was swallowing all exceptions and silently returning "not different" — it now logs loudly and self-disables after repeated failures. (It lives only inside the judge code, so there's nothing to fix on main.)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions