Skip to content

fix(config): load evaluator.re_evaluations from YAML (#12 was unenable-able) - #15

Merged
aktasbatuhan merged 1 commit into
mainfrom
fix/re-evaluations-config
Jun 3, 2026
Merged

aktasbatuhan merged 1 commit into
mainfrom
fix/re-evaluations-config

Conversation

@aktasbatuhan

Copy link
Copy Markdown
Member

Bug

load_config filtered the evaluator: YAML section through a hand-maintained allowlist that didn't include re_evaluations (added for #12). So evaluator.re_evaluations was silently dropped on load — setting it had no effect, the noise-aware re-evaluation never ran, and #12's A/B was invalid (both arms ran re_eval=1). The #12 unit tests passed only because they built EvaluatorConfig(re_evaluations=3) directly, bypassing the YAML path.

Fix

Derive the allowed keys from EvaluatorConfig's dataclass fields instead of a hardcoded set, so any field loads from YAML automatically and can't silently drop again. The allowlist's real purpose (rejecting keys that aren't config fields) is preserved.

evaluator_config_fields = {f.name for f in fields(EvaluatorConfig)}

Adds tests/test_config_loading.py covering the YAML-load path that was untested (the gap that hid this).

Note on #12

With this fix, re_evaluations=3 actually runs (verified: combined_score_std now persists). I re-ran the #12 A/B properly — it's inconclusive/negative, not the +0.145 originally reported (that run had re_eval stuck at 1). #12 stays merged but default-off and explicitly unproven; the record is corrected on issue #12. This PR is the bug fix that makes the feature enable-able at all.

🤖 Generated with Claude Code

The evaluator section was filtered through a hand-maintained allowlist that
omitted re_evaluations (added for #12), so it was silently dropped on load:
setting evaluator.re_evaluations in YAML had no effect, the noise-aware
re-evaluation never ran, and #12's A/B was invalid (both arms ran re_eval=1).

Derive the allowlist from EvaluatorConfig's dataclass fields instead, so any
field loads from YAML automatically and can't silently drop again. Add
test_config_loading.py to cover the YAML-load path that was untested.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@aktasbatuhan
aktasbatuhan merged commit d8dd55e into main Jun 3, 2026
1 check passed
@aktasbatuhan
aktasbatuhan deleted the fix/re-evaluations-config branch June 4, 2026 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant