[11-1] Analyze compute-normalized efficiency at fixed corpus scale

[duckyquang]

Task [11-1] | GEML Goals 6-12 workshop plan | Size: M | Revised: 2026-07-26

Authoritative scope revision: This body supersedes the earlier production-scale wording. Goals 1-5 and their 250k-v1 corpus, EML trees, DAGs, motifs, and reports are immutable inputs: do not regenerate or modify them. The project is targeting a resource-bounded MathNLP workshop submission. Preserve failures, unsupported cases, invalid outputs, and timeouts as explicit rows.

Dependencies

Hard prerequisites: result manifests from Goals 6-9 and the frozen [11-0] workshop manifest.

Objective

Replace data-scaling curves with fixed-scale, compute-normalized evidence: compare representation quality against parameters, FLOPs, wall time, memory, and alpha without claiming 10-100x behavior.

Exclusive file ownership

Owned paths - edit only these paths:

• src/geml/experiments/goal11/run_scaling.py - analysis orchestration only; no data scaling
• src/geml/analysis/goal11/scaling.py
• src/geml/plots/goal11_scaling.py
• configs/goal11_scaling.yaml - fixed-scale comparison config
• docs/goals/goal11/SCALING_CURVES.md - document fixed-scale efficiency and deferred scaling
• tests/experiments/test_goal11_scaling_smoke.py

Read-only dependencies:

• [11-0] manifest
• Goals 5-9 summaries and per-seed result tables

Tasks

• Validate comparable configs/budgets before joining quality with parameters, FLOPs, runtime, memory, and alpha.
• Produce fixed-scale Pareto tables/plots for equivalence and policy results; add proof/SR resource outcomes where definitions are compatible.
• Report raw per-seed points and uncertainty; do not fit or extrapolate data-scaling exponents.
• Separate GPU-parallel wall-clock speedup from total GPU-hours and include failed/incomplete cells.
• State explicitly that 10-100x scaling was not run and cannot be inferred.

Acceptance criteria

• Every point traces to a frozen result row and compatible budget/config.
• Efficiency claims use measured compute/resource fields, not parameter count alone.
• No data-scaling curve, exponent, or extrapolation is presented.
• The document answers only fixed-scale parameter/compute efficiency.

Validation

• python -m pytest tests/experiments/test_goal11_scaling_smoke.py
• Rebuild the fixed-scale efficiency tables/plots.
• Run the repository-wide standard validation commands.

No individual test, lint, or smoke command may exceed approximately 30 minutes. Production experiment cells may run longer only when sharded, resumable, checkpointed, and independently auditable.

Out of scope

• New training, corpus generation, subsample curves, extrapolation, or claims about 10-100x behavior.

PR checklist

• I edited only the owned paths above and did not change another issue's contracts.
• I used only current repository specifications, the complete issue, authoritative public mathematical sources, and official library documentation.
• Tests use tiny hand-written or temporary fixtures and do not require outputs/ or production artifacts.
• I retained and reported every failure, timeout, unsupported input, and validation error.
• I recorded configuration hashes, exact seeds, commit/package/hardware metadata, and any scientific assumption or metric change.
• I ran python -m pytest, python -m ruff check ., and python -m ruff format . --check before handoff.

Clean-room rule: Do not inspect or reuse the prototype repository, its code, tests, schemas, helpers, architecture, or commit history. Stop and ask when a shared or scientific interface is ambiguous.