[11-3] Run the verifier-normalized frontier-LLM reference comparison

[duckyquang]

Goal 11, Task 3 — [11-3] · GEML Goals 6–12 (roadmap, README §5) · Size: M

Independence

Hard prerequisites (genuinely required): [8-2], [9-0]. These cannot be stubbed: it runs frontier LLMs over the frozen proof and SR benchmarks through the same verifier so verified-correct vs claimed-correct rates are comparable.

What can start early, before the prerequisites merge: claim the issue, write the config and the tiny-fixture smoke test — only the work that touches real merged code or production artifacts must wait.

• Exclusive write set — no other issue edits your owned paths, so your PR cannot conflict with parallel work.
• Branch: issue/11-3-<short-slug>.

Objective

Run the frontier-LLM reference on the frozen benchmarks, verifier-normalized: report verified-correct vs claimed-correct rates so LLM numbers are comparable to the pipeline's, as a reference — not a controlled baseline.

Exclusive file ownership

Owned paths — this branch may edit only these paths:

• src/geml/experiments/goal11/llm_reference.py
• configs/goal11_llm.yaml
• docs/goals/goal11/LLM_REFERENCE.md
• tests/experiments/test_goal11_llm_smoke.py

Read-only dependencies:

• Benchmarks [8-2]/[9-0]
• verifier [2-6]

Tasks

• Prompt frontier LLMs on the frozen benchmark problems; capture raw claimed answers.
• Verify every claim through the standard verifier tiers; report verified-correct and claimed-correct separately.
• Record model versions, dates, prompts, and sampling settings for reproducibility.
• Frame results explicitly as a reference ceiling with uncontrolled training data.
• Add a fixture smoke test with a mocked LLM client.

Acceptance criteria

• Every LLM number is paired: claimed vs verified.
• Model/version/date metadata is complete.
• No controlled-comparison claim is made.

Validation

• python -m pytest tests/experiments/test_goal11_llm_smoke.py
• Run the production command in configs/goal11_llm.yaml.

Out of scope

• Do not fold into gate verdicts; [12-0] consumes this as context.

PR checklist

• I edited only the owned paths listed above.
• I did not inspect or reuse the previous GEML implementation.
• Tests use temporary/tiny fixtures and do not require production outputs.
• Failures and unsupported cases are retained and reported, not silently dropped.
• I included exact reproduction/validation commands in the PR description.
• I documented any scientific metric or assumption change.

Clean-room rule: Implement only from the current repository specifications and authoritative public sources. Use the v0 prototype only to understand what was measured and why — never as a code template. If an interface is ambiguous, stop and ask rather than modifying another issue's contract.