Skip to content

[11-2] Run the full three-track evaluation at scale and publish the Gate G11 verdict #81

Description

@duckyquang

Goal 11, Task 2 — [11-2] · GEML Goals 6–12 (roadmap, README §5) · Size: XL

Independence

Hard prerequisites (genuinely required): [11-0], [6-5], [7-3], [8-3], [9-1]. These cannot be stubbed: it re-runs the merged equivalence, rewrite/proof, and SR tracks at v3 scale and tests the parameter-efficiency hypothesis explicitly.

What can start early, before the prerequisites merge: claim the issue, write the config and the tiny-fixture smoke test — only the work that touches real merged code or production artifacts must wait.

  • Exclusive write set — no other issue edits your owned paths, so your PR cannot conflict with parallel work.
  • Branch: issue/11-2-<short-slug>.

Objective

Run the final at-scale evaluation across all three tracks (equivalence, rewrite/proof, SR), test the parameter-efficiency hypothesis explicitly (small GNN vs larger transformer at matched accuracy), and publish the Gate G11 verdict.

Exclusive file ownership

Owned paths — this branch may edit only these paths:

  • src/geml/experiments/goal11/run_final_eval.py
  • configs/goal11_final.yaml
  • src/geml/analysis/goal11/final_eval.py
  • src/geml/plots/goal11_final.py
  • docs/goals/goal11/GOAL11_SUMMARY.md
  • docs/goals/goal11/GATE_G11.md
  • tests/experiments/test_goal11_smoke.py

Read-only dependencies:

  • v3 data [11-0]
  • track machinery [6-5]/[7-3]/[8-3]/[9-1]

Tasks

  • Re-run the three tracks at v3 scale with the existing machinery and 3 seeds per cell.
  • Test parameter efficiency: find the smallest GNN matching the transformer's accuracy (and vice versa); report parameter/FLOP ratios at matched accuracy.
  • Report both denominators, multi-seed variance, and OOD axes everywhere.
  • Apply Gate G11 exactly: every headline number has both denominators, multi-seed variance, and scaling caveats resolved or stated; publish the verdict.
  • Persist everything under outputs/final/goal11/ with manifests.

Acceptance criteria

  • GATE_G11.md passes only if every headline number meets the evidence bar.
  • The parameter-efficiency verdict is stated at matched accuracy, not cherry-picked points.
  • All failure rows retained.

Validation

  • python -m pytest tests/experiments/test_goal11_smoke.py
  • Run the production command in configs/goal11_final.yaml.

Out of scope

  • Do not run the LLM comparison; [11-3] owns it.

PR checklist

  • I edited only the owned paths listed above.
  • I did not inspect or reuse the previous GEML implementation.
  • Tests use temporary/tiny fixtures and do not require production outputs.
  • Failures and unsupported cases are retained and reported, not silently dropped.
  • I included exact reproduction/validation commands in the PR description.
  • I documented any scientific metric or assumption change.

Clean-room rule: Implement only from the current repository specifications and authoritative public sources. Use the v0 prototype only to understand what was measured and why — never as a code template. If an interface is ambiguous, stop and ask rather than modifying another issue's contract.

Metadata

Metadata

Assignees

No one assigned

    Labels

    clean-roomClean-room rule applies: no previous GEML implementationgoal:11Goal 11 — scale-up & final comparisongroup-bArchitecture & learning workstreampriority:p0Critical pathtype:experimentProduction runs over real artifacts

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions