diff --git a/AAAI_READINESS.md b/AAAI_READINESS.md deleted file mode 100644 index 8cc1030..0000000 --- a/AAAI_READINESS.md +++ /dev/null @@ -1,119 +0,0 @@ -# AAAI readiness — honest verdict, reframe, next steps - -> **Superseded for writing purposes (2026-07-21):** the project adopted the **orchestrator reframe** -> (learned structure proposals + classical value solvers + checker). Start from [OUTLINE.md](OUTLINE.md) -> (the team writing guide) and [README.md](README.md) (the framing); this file remains the -> point-in-time readiness verdict that motivated the reframe. - -**Date:** 2026-07-20 · **Deadline:** 2026-07-27 (7 days) · **Canonical results:** `paper/RESULTS.md` -+ `paper/PROVENANCE.md`. Everything else in `paper/*.md` is working notes (consolidate — see §5). - ---- - -## 1. Is this AAAI-ready? — **Not for the main track as-is. Workshop-viable with honest framing.** - -Be blunt: tonight's rigor pass (adding the random-multistart control) was the right thing to do, -and it **shrank the claim**. The honest state: - -- **What's solid:** the system works end-to-end; the learned solver converges; entrapment (noise - escapes deterministic traps) is real with CIs; the amortized-inference **crossover** (learned - beats random search at n≥3) is genuine and controlled. -- **What's weak for a main-track bar:** - 1. **The learned model has no advantage over *random multi-start* on the hard suite** — the - "win over classical refinement" is really the *hybrid recipe*, not the denoiser. (A8.1.) - 2. **The one place the denoiser wins (high-dim crossover) is on *synthetic, independent* - bundled traps.** A reviewer will say: "random search failing in high dimension is the curse - of dimensionality; a model that learns each variable's marginal root is expected to win — - demonstrated only on toy problems it was built for." That criticism is fair right now. - 3. **Entrapment = textbook annealed Langevin** (novelty is confirmatory, not new). - 4. **All problems are synthetic**; MATH coverage is 0/48; no real domain. - 5. **No `.tex` exists yet.** - -**Verdict:** a defensible **workshop** paper (honest system + entrapment RQ + the crossover as -preliminary evidence). **Main-track needs the §3 experiment to land** — otherwise the novel -contribution is too narrow. - ---- - -## 2. What must be reframed (do NOT ship the old framing) - -| Old (wrong) framing | Corrected framing | -|---|---| -| "Learned diffusion solver beats classical refinement." | "A proposal+polish **hybrid** beats cold-start Langevin; the *learned* proposal beats **random** search only in high dimension." | -| Headline = A8.1 hard-suite numbers. | Headline = the **dimension-scaling crossover** (learned > random for n≥3). A8.1 becomes "the hybrid recipe" evidence, with the random control shown. | -| "Solves math problems." | "Numeric constraint solver; 0/48 MATH coverage; targets the constraint-shaped slice." (Reality check, not a result.) | -| Report `refine` numbers as system results. | `refine`/`random` are baselines; always labelled; every number has N + CI/z-test. | -| "Structure invention" as a claim. | "Menu-based structure selection (with predicted defining value)" — built and trained; clean-protocol numbers pending (see §3b, `paper/notes/REVIEW_ATTACKS.md` #2/#3). | - ---- - -## 3. The decisive experiment — RAN IT. Result: the learned advantage does NOT survive coupling. - -We built the coupled chained-bilinear family (`x_i+x_{i+1}=s_i, x_i·x_{i+1}=p_i`) and ran the -scaling with the existing controls (R7 in `RESULTS.md`). **Outcome: the learned model ties or -loses to random restart at every dimension (0/5).** The R5 high-dim advantage was an -**independence artifact** — it required per-variable-separable solutions (memorizable marginals) -*and* random restart collapsing (must hit all n basins by chance). Under coupling, neither holds, -and the diffusion model provides **no advantage over random search + refinement**. - -**Implication:** the "amortized learned proposal beats classical search" route to a main-track -claim is **closed**. What remains genuinely solid is narrower: the learned solver *converges* -(engineering), entrapment/noise escapes deterministic traps (real but textbook Langevin), and the -hybrid recipe beats cold-start Langevin (but so does random restart). That is a **workshop-level** -contribution. - -## 3b. The remaining main-track shots -- **Menu-based structure selection (with predicted defining value).** BUILT: the D3PM - structure policy is trained on the aux-required families (fixed graph certified - unsolvable without the auxiliary), and the end-to-end eval harness runs it with - positive/negative controls. Honest naming: the model selects one of K candidate - auxiliary structures and predicts its defining value — it does not synthesize - structure from an open vocabulary, so claim-language is "selection", not - "invention". **Clean-protocol numbers pending**: the preliminary run is withdrawn - (test seeds == validation seeds, plus a train/eval data-source mismatch in the - harness — both now fixed; see `paper/RESULTS.md` R8 and `paper/notes/REVIEW_ATTACKS.md` - #2/#3). Regenerate under the seed-space v1 protocol before citing anything. -- **A real (narrow) domain.** Get the model training+solving on geometry or a real - polynomial-system slice — moves it from "synthetic toy" to "real problems." ~2–3 days, medium - risk. Biggest credibility jump. -- Honest note: a rigorous *"when do learned proposals help constraint solving — and when they - don't"* study (including the coupled negative) is a legitimate paper, but reads workshop, not - main-track, at AAAI. - ---- - -## 4. Next steps to the deadline (priority order) - -1. **[P0, ~1 day] Coupled high-dim scaling experiment (§3)** — decides the paper's ceiling. -2. **[P0, today] Start the `.tex`** — intro/method/related-work are experiment-independent - (`paper/notes/related_work.md` is ready). Write to the *corrected* framing (§2). -3. **[P0] Pick the target** — AAAI main vs an AAAI/NeurIPS workshop vs a different venue. Default - to **workshop** unless §3 lands; decide at the team meeting. -4. **[P1] CoT baseline properly** (needs a fresh, funded key): N≥100, k≥4, stronger model, - Wilson CIs — only if we want an H1/LLM comparison. Otherwise cut it. -5. **[P1] One honest limitations paragraph** — CircleLine failure, cross-family 2/4, learned - loses to random at low n, synthetic-only, no proofs. Reviewers reward this. -6. **[P2] Rotate the OpenAI + Gemini keys** (exposed in the working chat). - ---- - -## 5. Doc consolidation (the sprawl you flagged) - -There are ~10 `paper/*.md` + root docs. Keep **two canonical**, treat the rest as source notes: - -- **Canonical:** `paper/RESULTS.md` (all results, corrected) · `paper/PROVENANCE.md` (every number's - command/seed/commit). -- **Source notes (fold into the .tex, then archive):** `learned_solver_fix.md`, - `dimension_scaling_result.md`, `math_coverage.md`, `related_work.md`, `h2_reframe.md`, - `ablation_reframe.md`, `hard_suite_table.md`. -- **Process docs (not paper):** `FIXING_PLAN.md`, `MEETING_NOTES.md`, `SUMMARY.md`, this file. - -Suggest: move source notes under `paper/notes/` so `paper/` top level is just the `.tex` + the two -canonical `.md`s. - ---- - -## Bottom line -Honest, rigorous, working system with one genuine (but narrow) result. **Workshop-ready now; -main-track hinges on the §3 coupled-family experiment.** Don't oversell — the corrected framing is -what makes it survivable. Decide the target at the meeting; start the `.tex` today regardless. diff --git a/HANDOFF.md b/HANDOFF.md deleted file mode 100644 index 79366bb..0000000 --- a/HANDOFF.md +++ /dev/null @@ -1,87 +0,0 @@ -# MARC — session handoff (fresh-context, 2026-07-22) - -**AAAI abstract deadline:** 2026-07-27. Repo `saidlaboratory/MARC`. -Run scripts with `PYTHONPATH=.` (some also need `:scripts`); MacBook, torch 2.12, MPS. -Canonical docs: `paper/RESULTS.md`, `paper/PROVENANCE.md` (every number → command/seed/branch), -`paper/ABSTRACT.md`, `paper/tex/marc.tex` (the paper), `paper/notes/*.md`. - -## Conventions (IMPORTANT) -- Commits authored by **@ImSpxrsh, NO Claude co-author** (git is already configured for this). -- **Do NOT put "Generated with Claude Code" in PR bodies** (team preference). -- Every rate carries N + Wilson CI or z-test. Negatives are reported as primary findings. -- `main` push is blocked by the harness; work on a branch, push, open a PR; the team merges. -- Abstract is em-dash-free (AI-tell hygiene); the paper body uses `---` as house style (fine). -- Before staging: `git checkout -- marc/graph/__pycache__` (tracked pyc noise reappears). - -## The paper in one line -A **controlled study / characterization** of *when* learned proposals help continuous algebraic -constraint solving. Two acts: (1) value-diffusion is mostly a boundary/negative unified by a -**factorization law**; (2) relocating learning to the **discrete structural-repair** decision is a -decisive **positive** (the repair ranker). MARC (factor graphs + GNN diffusion denoiser + CAS + -exact checker) is the *instrument*, not the claimed contribution — keep it framed that way (the -diffusion model's novelty is weak; that is fine because we do not claim it as the contribution). - -## DONE + merged to main this session -- **Abstract v5** (repair co-headlined + geometry + real-systems validation), em-dash-free, ~265w. - `paper/tex/marc.tex` + `paper/ABSTRACT.md`. (PRs #113, #115.) -- **Geometry learned arm (R25)** — the law's live prediction, TESTED and refuted, which sharpened - it: a trained denoiser ties random on the coupled geometry point-chain and collapses with it - (0.625/0.175/0.025/0.000, 0/4 wins). Corrected the law to **two conditions**: learning helps iff - (1) reachability collapses AND (2) the solution is per-variable separable. (PR #114.) -- Earlier merged: factorization law R9 (MAE 0.012), repair ranker R20–R24 (0.997 vs 0.236, - p<1e-70, beats cheap-probe on accuracy+cost, 0.982±0.006 multiseed), fixes #103/#104, R8 regen. - -## OPEN PR / branches -- **PR #116 `sparsh/real-systems`** — **External validity (R26)**: eight NAMED real systems - (robotics IK, trilateration, Rosenbrock/Himmelblau, cyclic-4, circle/conic). Classical **LM - solves 8/8**; gradient-polish random restart 4/8. No learning-favorable regime (real = low-dim + - coupled → classical suffices, consistent with the law). Answers **synthetic-only**. Merged main - in; mergeable. **Action: get it merged.** -- Branches `sparsh/crossover-families` work below is not yet pushed at handoff — see next section. - -## WINS LANDED this session (PR #117 open) — both experiments finished -1. **Crossover replication + learned-beats-LM** — the strongest new win. - `PYTHONPATH=.:scripts python3 scripts/run_crossover_families.py --K 8 --test 40 --epochs 200 --ntrain 200` - → `results/p_scaling/crossover_families.{json,log}`. Establishes TWO things: - (a) the R5 amortization crossover **replicates** across 3 structurally different separable - families (baseline / double_well / wide_roots) — not one designed family; - (b) the learned proposal beats **LM (the strong classical solver), not just random** — LM ALSO - collapses ~p^n (measured on baseline: LM 0.825/0.575/0.200/0.100/0.000 at n=1/2/3/4/6). This - closes the "did you compare to a real solver?" attack on R5. - Files already created (uncommitted): `scripts/run_crossover_families.py`, - `tests/test_crossover_families.py` (passing). **DONE — landed as R27 in PR #117.** (RESULTS.md + - PROVENANCE, add a paper table/paragraph to the R5/scaling section of `marc.tex`, commit to a - branch `sparsh/crossover-families`, push, PR (NO Claude footer). -2. **3-seed geometry hardening** — `scripts/run_pointchain_learned.py --seeds 3` - → `results/p_geometry/pointchain_learned_3seed.log`. Hardens R25 (ties reproduce). When done, - update R25's note to "3 seeds, tie robust". - -## Honest WIN inventory (what the paper can claim) -- **STRONG:** repair ranker (R20–R24) beats controls + cheap-probe on accuracy AND cost, multiseed-robust. -- **STRONG (new):** learned beats BOTH random and LM at high-dim separable; crossover replicates. -- **SOLID:** entrapment R2 (0.525±0.086); factorization law R9 (parameter-free MAE 0.012, 3-family - validation incl. the geometry refutation); structure-selection R16 (0.410 vs 0.200 in-pattern, sig). -- **EXTERNAL VALIDITY (new):** R26 real systems (LM 8/8) — answers synthetic-only. -- **HONEST NEGATIVES:** coupling kills value-learning (R7); geometry (R25); CircleLine; transfer 2/4 - (R4); K=16 repair advantage gone (cost-only, R24). - -## Highest-value NEXT wins (priority order) -1. **A real-domain POSITIVE = the single biggest lever for main-track.** The repair positive is on - a synthetic task construction. The real analog is **geometry auxiliary construction** - (AlphaGeometry's domain): geometry problems unsolvable without an auxiliary point/line, a menu of - candidate auxiliaries (exactly one solvable, certified), train the ranker. Hard but decisive — - moves the paper from "coin-flip" to "strong." -2. **More separable families** for the crossover (extend `run_crossover_families`) — cheap breadth. -3. **Repair ranker: more generalization axes** (held-out patterns, harder negatives). -4. **Tighter CIs** on R5/R25 (more seeds). - -## Honest main-track verdict -Borderline / credible-but-not-a-lock (~coin-flip). Strengths: the law (falsifiable, parameter-free, -3-family), the repair positive, the controlled protocol, external validity, learned-beats-LM. -Weaknesses: strongest positive is a synthetic task construction; diffusion model weak/negative -(fine if framed as instrument); fundamentally an analysis paper. Biggest lever: a real-domain -positive (item 1). - -## Health -- Full suite `PYTHONPATH=. python3 -m pytest -q` was **404 passing**; +new tests → ~406–408. -- Result JSONs whitelisted in `.gitignore` (results/ is ignored; add `!results/p_.../x.json` per file). diff --git a/MEETING_NOTES.md b/MEETING_NOTES.md deleted file mode 100644 index 379ec27..0000000 --- a/MEETING_NOTES.md +++ /dev/null @@ -1,123 +0,0 @@ -# MARC — Meeting Notes - -**Date:** 2026-07-19 · **AAAI deadline:** 2026-07-27 (8 days) · **Branch of latest work:** `sparsh/a1-hard-suite` - -Snapshot for the team meeting: what changed, where we stand, decisions needed, and next moves against the deadline. Written to be honest — the caveats are in here on purpose. - ---- - -## 0. Overnight update (rigor pass) — see `paper/RESULTS.md` for the canonical, cited results - -**⚠️ Two honest corrections from adding the random-multistart control (read before the meeting):** -- **A8.1 reframed:** the learned hybrid beats *cold-start Langevin*, but a **random-init + - polish** control ties/beats it on all 4 families. So the contribution on these families is the - **hybrid recipe (proposal + polish)**, *not* the learned denoiser — the learned proposal has - no advantage over random multi-start when solutions are small/dense. (Nearly shipped an - over-claim; the control caught it.) -- **Dimension scaling reframed (stronger + honest):** the learned proposal beats random restart - **only in high dimension** — random wins at n≤2 (0.875 vs 0.675), learned wins at n≥4 where - random collapses to ~0 (n=4: learned 0.650 vs random 0.000). *That crossover* is the real - amortized-inference result. New headline: "amortized learned proposals beat random search in - high-dim non-convex constraint solving," not "learned beats classical everywhere." -- **Cross-family generalization (H1 transfer):** leave-one-out. Learned model **solves 2/4 - held-out families it never trained on** (0.683, p<1e-4); fails on 2/4. Partial, honest — - a dissimilar training family (CircleLine) can disrupt transfer. -- **MATH reality-check:** honest 0/48 coverage + scope breakdown (no inflation). -- **Rigor + positioning docs added:** `paper/RESULTS.md`, `PROVENANCE.md`, `related_work.md`, - `math_coverage.md`; `metrics.wilson_interval` + `two_proportion_z`. 209 tests green. -- All on [PR #53](https://github.com/saidlaboratory/MARC/pull/53), authored by @ImSpxrsh. - ---- - -## 1. TL;DR - -- **The learned solver now works.** It went from *diverging / 0% solve* to actually solving. Five real bugs fixed. (`paper/notes/learned_solver_fix.md`) -- **Two genuine results in hand:** (a) noise escapes entrapment where deterministic descent is 100% trapped; (b) on high-dim non-convex problems the learned model does per-instance inference that beats classical refinement **and** a trivial prior. -- **NEW (today): the eval suite is de-saturated** — the #1 paper blocker (A1) is fixed, and the first hard-suite ablation shows the **learned hybrid beats refine** (0.625 vs 0.350 vs 0.000). This answers the "what does the denoiser add?" review attack. -- **Reality check:** these are toy problems (linear systems, bilinear systems, constructed traps). This is **not** competition math (IMO/USAMO) and not SOTA. It's a plausible AAAI submission **if scoped honestly**. -- **Still no paper draft (.tex).** That is the critical path (C1). - ---- - -## 2. What we accomplished this session - -### Engineering / correctness -- **Fixed the learned diffusion solver (0% → solves).** Root causes: (1) denoiser never received the noised input `x_t`; (2) equation constants absent from graph tensors; (3) timestep didn't condition variables; (4) inference guidance exploded; (5) a 1-variable `squeeze` bug. Method = diffusion proposal + energy-descent polish. -- **Architectural contribution — per-instance inference.** The model was collapsing to "predict the mean solution." Fix: condition each variable on its **incident constraint constants** + a **direct constant→output skip** (message-passing LayerNorm was washing out the magnitude). Recovers roots at mean-error 0.9 (was 5.4). -- **Test coverage:** 159 → 208 tests, all green. New coverage for structure toys, embeddings, rollout, CoT baseline, dimension-scaling, hard templates. -- **CoT baseline runs on Gemini** (free-tier, OpenAI-compatible endpoint, backoff + resume cache). -- **Merged to `main`:** PR #50 (solver converges + ablations + entrapment), PR #51 (inference + scaling + SUMMARY). Quang's `FIXING_PLAN.md` merged (#52). - -### Science results -| Result | Numbers | File | -|---|---|---| -| Learned solver converges (convex) | 0% → 100% in-dist & held-out | `paper/notes/learned_solver_fix.md` | -| Entrapment (RQ2) | deterministic 100% trapped → Langevin 0.475; reduction **0.525 ± 0.086**, N=200 | `results/p1_entrapment/` | -| Dimension scaling | learned beats determ (0) / Langevin (→0) / mean-prior (0); decays 0.68→0.10 over n=1..6 | `paper/notes/dimension_scaling_result.md` | -| **A1 hard suite (new)** | bilinear traps de-saturate: determ 0.000, Langevin 0.175–0.35 | `scripts/run_hard_eval.py` | -| **A8.1 hybrid ablation (new)** | learned_hybrid beats refine on both hard families: BilinearSystem **0.625** vs 0.350/0.000; BilinearProduct **0.725** vs 0.125/0.000 | `results/p_hard/hard_eval.json` | - ---- - -## 3. Fixing-plan status (P0 items) - -| ID | Item | Status | -|---|---|---| -| C2 | Merge PR #50 + doc sweep | ✅ merged (#50, #51) | -| **A1** | Eval suite saturated → hard tier | ✅ **done today** (bilinear templates de-saturate) | -| **A8.1** | Hybrid vs refine-only ablation | ✅ **done, rigorous** — 4 non-convex families + Wilson CIs + z-test. Learned hybrid significantly beats Langevin on **3/4** (p<0.01); honest **failure on CircleLine** (0.000). `paper/figures/hard_suite_table.md` | -| A2 | Headline numbers from learned model, not refine | ✅ hard-suite headline table + figure (`paper/figures/hard_suite_table.md`, `fig_hard_suite.pdf`) | -| A4/A5 | Guidance / purist ablations degenerate | ✅ reframed (`paper/notes/ablation_reframe.md`): replace with A8.1; guidance sweep on hard checkpoint optional (P1) | -| A6 | CoT baseline too thin (N=25, k=1) | ⏳ needs Gemini key + N≥100, k≥4, stronger tier | -| A7 | H2 null result | ✅ reframed as "preliminary" (`paper/notes/h2_reframe.md`) | -| A3 | Geometry training template | ⏳ P1 — not started (Davin) | -| A8.3 | Entrapment on a non-convex family | ⏳ P0 framing — bilinear suite now exists for it | -| C1 | **No paper (.tex)** | ❌ **not started — critical path (Quang)** | -| C3 | Provenance table | ✅ `paper/PROVENANCE.md` started (R1–R12) | - ---- - -## 4. AAAI timeline (from FIXING_PLAN.md) - -Two parallel tracks; **the paper track must never wait on experiments.** - -- **Now → Jul 21:** hard-suite generator (done) → run learned/refine/CoT grid on it (A1/A2/A6). Start LaTeX scaffold + intro/method/related-work (≈60% of the paper has zero experiment dependency). -- **Jul 22:** finish hybrid ablation (A8.1) + guidance/purist re-runs (A4/A5). -- **Jul 23 — DECISION GATE:** does the learned solver separate from CoT on the hard suite? - - *Yes (CIs disjoint):* H1 claims section, consider main-track framing. - - *No / CoT wins:* workshop framing — system + entrapment (RQ2) + hybrid ablation as the contributions; H1/H2 preliminary. **The draft is written to survive this either way.** -- **Jul 24–25:** final numbers into tables; provenance table; internal read. -- **Jul 26–27:** revisions, license, abstract, **submit**. - ---- - -## 5. Decisions to make at the meeting - -1. **Main-track vs workshop framing.** Default to workshop-survivable; the Jul 23 gate can only upgrade. Agree we write to the honest floor. -2. **Owners for the paper sections** (per plan): Intro/method (Quang), System (Davin), Entrapment (Sparsh), H1 (Akash), H2 preliminary (Quang). **Who starts the .tex today?** -3. **CoT baseline budget.** A6 needs a stronger model tier + N≥100 → real (small) API spend. Approve? Which model as primary baseline? -4. **Scope honesty in claims.** Agree the house style: `refine` always labeled a classical baseline; every number carries solver + N + CI + provenance; no "formally verified", no unqualified geometry/NL claims. -5. **License** (C4): MIT vs Apache-2.0 — needs all four sign-offs (15 min). -6. **GPU?** If one materializes, launch the D=512/L=8 scale plan immediately (a mid-week checkpoint upgrades every table). If not, scope scale explicitly. - ---- - -## 6. Open risks / honest caveats (say these out loud) - -- **Toy problems only.** Biggest solved: 3×3 linear or 2–3 var bilinear. Not natural math, no proofs, **not IMO/USAMO** — a different research program. -- **Checkpoints are toy-scale** (D=128, CPU-minutes). Not a scaling claim. -- **Dimension-scaling degrades at n=6** (learned 0.10) — the honest weak spot; it beats baselines but is not dimension-immune. -- **H2 is a null result** on an untrained proxy; must be reframed as preliminary, not sold. -- **Mechanism novelty is moderate** — amortized learned proposal beating blind search is a known principle; our contribution is the concrete constraint-solving instance + the conditioning architecture. -- **API keys (OpenAI + Gemini) were pasted in a working chat — rotate them.** - ---- - -## 7. Immediate next moves (this week, concrete) - -1. **Finish A8.1** (running) → if hybrid > refine on both families, it's a headline ablation. Commit `run_hard_eval.py` + results. -2. **A2:** regenerate `summary_table.md` + figures from the learned checkpoint on the hard suite; label `refine` as a baseline row. -3. **A4/A5:** re-run guidance/purist sweeps on the hard suite (degenerate "w=0, 0.000" numbers cannot ship). -4. **A6:** with a Gemini/OpenAI key, scale CoT to N≥100, k≥4, add Wilson CIs, sweep perturbation Δ. -5. **C1:** scaffold the AAAI LaTeX **today** and port intro/method/related-work from CONCEPT.md / README. -6. **C3:** start `paper/PROVENANCE.md` (figure → script → commit → seed) now, not at the end. diff --git a/NIGHT_SESSION.md b/NIGHT_SESSION.md deleted file mode 100644 index c419c9e..0000000 --- a/NIGHT_SESSION.md +++ /dev/null @@ -1,54 +0,0 @@ -# Overnight session — the factorization law (2026-07-21) - -**Branch:** `sparsh/crossover-theory` (committed locally, **not pushed** — open a PR when ready). -**Starting point:** HANDOFF.md's honest verdict (workshop-level; R5 positive + R7 negative solid). -**What this session added:** the unifying *scientific* result the prior state lacked — a -falsifiable, parameter-free **law** that predicts both R5 and R7 (and now a real domain) from -one measured quantity. This is the main-track-shaped contribution: an analysis/understanding -result with a general principle, not a new SOTA claim. Nothing falsified was re-inflated. - -## The result (R9 — see `paper/notes/crossover_law.md`, `paper/RESULTS.md`) -All methods share one polish + one checker. Define single-start reachability `q(n)`. Then -best-of-K random restart is exactly `1-(1-q(n))^K`. The regime is set by how `q(n)` decays: -- **Separable (independent traps):** `q(n)=v^n` provably. Measured slope **−1.03 (R²=0.98)**, - v=0.27; a single constant reproduces the whole random curve **parameter-free, MAE 0.012**; - expected restarts explode **3.7→600**. Random collapses ⇒ learning wins (R5). -- **Coupled bilinear:** slope **−0.13 (R²=0.96)**, expected restarts flat **2→4**. Random - survives ⇒ learning ties, classical LM dominates (R7). -- **Geometry (real domain, new):** syntactically coupled but reachability **collapses** - (slope −0.77, R²=0.999; q=0.653/0.147/0.027/0.007) — per-point reflection ambiguity + - spurious basins compound. ⇒ the diagnostic - is the *measured slope*, not the syntactic label; geometry is flagged **learning-favorable**. - -**One-line law:** a learned proposal beats classical search iff single-start reachability -decays with dimension (equivalently, iff the acceptance basins effectively factorize). Steep -slope ⇒ learning helps; flat slope ⇒ it cannot. - -## Artifacts produced -- `scripts/run_crossover_theory.py` — measures q(n) with the *identical* generators/refine/checker - as R5/R7 (600 trials, Wilson CIs), tests the law, predicts the random curve. `--no-geometry` - to skip the slower real-domain family. Reproduce: `PYTHONPATH=. python3 scripts/run_crossover_theory.py --trials 600 --K 8 --seed 20260721`. -- `marc/data/geometry.py` — `build_point_chain_graph` / `make_point_chain`: scalable coupled - geometry family (2k vars), integer solutions the checker accepts. -- `paper/main.tex` — full AAAI-2026 draft built on the law (needs the official `aaai2026.sty`). -- `paper/ABSTRACT.md`, `paper/refs.bib`, `paper/notes/crossover_law.md` (derivation + limits). -- `paper/RESULTS.md` R9; `paper/PROVENANCE.md` R17–R19; `paper/figures/fig_crossover_theory.pdf`. -- `results/p_crossover/crossover_theory.json` (tracked via .gitignore whitelist). -- Tests: `tests/test_crossover_theory.py` (law algebra, dichotomy, geometry). **Suite: 370 → passing.** - -## Honest status / what's NOT claimed -- No claim a learned solver beats classical search in general — the coupled result is the - opposite, reported as a primary finding. The contribution is the law that says *when*. -- Geometry: the law *predicts* learning can help there (steep collapse). Training the geometry - denoiser and running it vs the random-restart control is the flagged next experiment (not done). -- Crossover point is not claimed sharply: the higher-N (600) re-measurement puts random weaker - at low n than the N=40 R5 table, so the robust claim is the geometric collapse, not a specific n*. - -## Next steps (priority) -1. **Train the geometry denoiser** on `make_point_chain` and eval vs random-restart + LM — the - law's live prediction; a geometry *positive* would materially strengthen the paper. -2. Drop the official `aaai2026.sty`/`.bst` in `paper/`, compile `main.tex`, fill the two ⟨geo⟩ - slots from `crossover_theory.json`. -3. Optionally still wire `MARC_CKPT` into the eval scripts (HANDOFF §5) — but the law already - explains why scale won't rescue the coupled negative (factorization is a problem property). -4. Push `sparsh/crossover-theory`, open a PR (author @ImSpxrsh, no Claude co-author per repo convention). diff --git a/OUTLINE.md b/OUTLINE.md deleted file mode 100644 index 0c369f9..0000000 --- a/OUTLINE.md +++ /dev/null @@ -1,112 +0,0 @@ -# OUTLINE — MARC v0.2: the orchestrator reframe - -**Purpose of this document:** everything a team member needs to understand the new framing and start writing — the thesis, the story arc, the evidence and where it lives, section-by-section paper guidance, framing rules, and what is blocked on the overnight run. Read this before touching the `.tex`. - -**Deadline context:** AAAI abstract deadline July 27, 2026. The `.tex` does not exist yet. Writing starts now against this outline; final numbers drop in from the clean overnight run. - ---- - -## 1. The thesis in three sentences - -Solving a constraint problem involves two kinds of decisions: **continuous** (what values satisfy the equations — owned by classical solvers, which we measured to be near-unbeatable) and **discrete** (what representation makes the problem tractable at all — which auxiliary variable, substitution, or defining relation to introduce — where classical solvers have *nothing* and enumeration grows combinatorially). MARC learns the discrete decision: a structural prior over constraint-graph augmentations, run as a discrete-diffusion policy, whose proposals are completed by classical value solvers and accepted only by an exact symbolic checker. One line: **a neural mathematician's instinct for "introduce `d = x − y`," bolted onto solvers that finish the job and a checker that keeps everyone honest.** - -## 2. Why this framing (and why it is credible) - -1. **Our own controls chose it.** We bet on value diffusion first and falsified it publicly: learned value proposals tie/lose to random restart on coupled systems (R7); Levenberg–Marquardt with restarts saturates the hard families at 1.000; the GNN could not even overfit `Ax=b` from raw coefficients. Meanwhile the structure-selection policy is the one learned component that beat its controls. -2. **The economics invert.** For values, enumeration/search was cheap → the learned model had no room. For structure, the candidate space is combinatorial → one policy forward vs. exponentially many candidate-solve attempts, an advantage that *grows* with difficulty. -3. **Precedent.** AlphaGeometry (Nature 2024) is the same division of labor — neural auxiliary-construction proposals + symbolic engine — in geometry-specific machinery. MARC generalizes the pattern to arbitrary constraint graphs with an exact checker in the loop. Cite it early and prominently; it converts "weird pivot" into "recognized paradigm, generalized." -4. **The negative results are the motivation, not the embarrassment.** The paper's credibility rests on: we measured where learning does not help, and located where it does. - -## 3. The story arc (this is the paper's spine) - -1. **Problem:** LLM math reasoning is memorization-prone and unverifiable; we want derive-not-recall on a checkable substrate. -2. **Substrate:** constraint graphs + exact CAS residuals + a two-stage checker (numeric, then symbolic-exact). Training reward comes only from the checker. -3. **First bet, measured and closed:** value diffusion. Present R7 (coupled negative), the LM column, and the random-restart control *as findings* — with CIs. This section buys the trust the main claim spends. -4. **The surviving observation:** entrapment (noise escapes deterministic traps, 0.525 ± 0.086) — stochasticity matters in *search*; it never required a learned denoiser. -5. **The reframe:** the discrete/continuous decision split (§1 above). What classical solvers cannot do at any budget: change the representation. -6. **The system:** aux-required problem families (fixed graph certifiably unsolvable; only the right augmentation fixes it), candidate menus with certified exactly-one-solvable structure and hard negatives, the structure policy (absorbing-D3PM over padded slots, graph-conditioned encoder, value head predicting the defining constant), classical solvers downstream, checker gate. -7. **Results:** invention/solve rates vs. the control battery (random-slot, no-context, always-none, gold-oracle, enumeration), cross-pattern holdout, amortization costs, hard-negative confusion — all under the clean seed protocol, multi-seed, Holm-corrected. -8. **Honest scope:** menu-based selection with predicted defining value = rungs 1–2 of a ladder whose endpoint (free-form generation) is future work. State the ladder explicitly. - -## 4. Evidence inventory (what exists, where it lives, what may be cited) - -| Evidence | Status | Source of truth | -|---|---|---| -| Entrapment: noise reduces trapping 1.000 → 0.475, reduction 0.525 ± 0.086, N=200 | **Solid, citable** | `paper/RESULTS.md` R2; `results/p1_entrapment/` | -| Coupled negative (R7): learned ties/loses random at every n | **Solid, citable** — it is the motivation | `paper/RESULTS.md` R7; `results/p_coupled/coupled.json` | -| LM/exact classical columns on hard + coupled suites | **Solid** (regenerate rows in the overnight run) | `scripts/run_hard_eval.py`, `run_coupled_eval.py` outputs | -| Hybrid beats cold-start Langevin (but so does random) | Solid, cite with both halves | `paper/RESULTS.md` R3 | -| Cross-family value-transfer (partial, 2/4) | Citable with caveats | `paper/RESULTS.md` R4 | -| Dimension scaling | Only the **unified-v2** methodology run | `results/p_scaling/scaling.json` (`methodology: "unified-v2"`) | -| Structure selection 0.45/0.53 vs random 0.125 | **WITHDRAWN — never cite** (eval seeds == validation seeds; see R8) | `paper/RESULTS.md` R8 | -| Structure selection, clean protocol | **Pending the overnight run** — the paper's headline slot | will land in `results/p5_invention/invention.json` + `invention_heldout.json` | -| MATH coverage 0/48 | Citable as scope reality-check | `results/p_math/coverage.json` | - -**Citation law (non-negotiable):** -- Structure-selection numbers only from runs whose JSON has `seed_hygiene.overlap_instances: 0`. -- Every rate: N + Wilson CI. Every comparison: z-test, Holm-corrected within the declared family. -- `refine`/`lm`/`exact`/`random` are always labeled classical baselines, never system results. -- The word **"invents"** is reserved for the ladder's endpoint. Current claims say **"menu-based structure selection (with predicted defining value)."** -- Positive controls are reported with their by-construction caveats (`positive_control.by_construction` in the eval JSON). - -## 5. Section-by-section writing guide - -Suggested owners in (parentheses) — reassign freely at the meeting. - -### §1 Introduction (Quang) -Lead with the discrete/continuous decision split, not with diffusion. The AlphaGeometry sentence appears by paragraph 2. The contribution list: (i) an honest, controlled study closing the learned-value-proposal route on constraint graphs; (ii) a general substrate + certified problem families for studying learned structure augmentation; (iii) a trained structure policy beating controls under a contamination-proof protocol; (iv) the amortization analysis. Do not promise invention. - -### §2 Substrate & verification (Davin) -Constraint graphs (variable/factor nodes, expression strings), CAS residuals/energy, two-stage checker, why checker-only reward. Source: `marc/graph/`, `marc/cas/`, TECHNICAL_GUIDE §§3–7. Half a page; this is plumbing, written confidently. - -### §3 The value-diffusion study (honest negative) (Sparsh) -The controlled experiments and what killed each claim: random-restart control shrank R5; R7 coupled families closed the route; LM saturates; the `Ax=b` overfit probe. Entrapment as the surviving (pre-registered) positive. Tone: measurement, not apology. Sources: RESULTS.md R1–R7, `paper/notes/learned_solver_fix.md`, `results/p_coupled/`. - -### §4 Method: the structure policy (Quang) -Aux-required families + certificates (exact rank for linear; **empirical probes for nonlinear — state plainly they are probabilistic claims at a stated budget**), menu construction with hard negatives and randomized gold support, padded-slot representation, absorbing-D3PM forward/reverse, the graph-conditioned encoder, predicted defining value. The ladder (§8 below) closes this section. Sources: `marc/structure/`, `marc/data/aux_required.py`, TECHNICAL_GUIDE §10/§14. - -### §5 Experimental protocol (Sparsh) -The part reviewers will probe hardest — write it proudly: disjoint seed spaces asserted at checkpoint load (contamination is *impossible by protocol*, and we say why the protocol exists — we caught ourselves once); multi-seed pooled Wilson CIs; Holm correction over the declared comparison family; the full control battery (random-slot, no-context, always-none, gold-oracle, enumeration); cross-pattern holdout; hard-negative confusion; amortization measurement. Source: `scripts/run_invention_eval.py`, `paper/notes/REVIEW_ATTACKS.md`. - -### §6 Results (Akash, once the overnight lands) -Placeholder tables NOW with the exact JSON keys they will be filled from (`samplers.*.invention_rate`, `arms.enumeration.*`, `comparisons_holm.*`, `invention_heldout.json`). Structure: main table (policy vs. controls, both samplers), cross-pattern table, amortization figure (policy cost vs. enumeration cost as K grows), hard-negative confusion, value-solver context rows. - -### §7 Related work (Akash) -AlphaGeometry (closest relative — generalize, don't compete), D3PM/DiGress (formalism), neural algorithmic reasoning (independent evidence for delegating numerics), neural CO/SAT solvers (we learn representation change, not search), RLVR/verifier training, PoT/PAL (computation delegation taken to its end). The README's prior-art table is the skeleton. - -### §8 Limitations & the ladder (Quang) -Rungs: menu selection → predicted value → compositional/multi-aux → free-form generation. Current work = rungs 1–2. Also: synthetic families only, MATH coverage 0/48, empirical certificates are budget-relative, single training seed caveat where applicable. Reviewers reward this section — write it first, not last. - -## 6. Anticipated reviews and our answers (from `paper/notes/REVIEW_ATTACKS.md`) - -| Attack | Our answer | -|---|---| -| "Selection from K candidates is classification, not invention." | Correct — and we say so (the ladder). The claim is amortized structure choice under verification, with a continuous value head (rung 2) already beyond pure classification. | -| "Enumeration solves your task exactly." | Reported as an arm, expected 1.00 — the claim is **cost**: measured policy-vs-enumeration economics, widening with K. | -| "The policy could be matching family signatures, not reading the graph." | Randomized gold support kills the signature; the no-context ablation and hard-negative confusion measure graph-reading directly. | -| "Your earlier numbers were contaminated." | Yes — we caught it, withdrew them (R8), and rebuilt the protocol so the eval refuses contaminated seeds. This is a strength; cite the protocol. | -| "Value diffusion failed; why keep the diffusion formalism at all?" | For structure, discrete diffusion is the natural formalism for ABSENT→active instantiation (D3PM), and the single-shot ablation is always reported alongside — if single-shot wins at scale, the paper says so. | -| "Toy scale." | Conceded in Limitations; the certified-family methodology and the protocol are contributions independent of scale. | - -## 7. What is blocked vs. writable today - -**Writable now (≈70% of the paper):** §§1, 2, 3, 4, 5, 7, 8 — everything except final result numbers. The negative-result section is fully numbered already. - -**Blocked on the overnight run (Sparsh, MacBook M5 — see `RUNBOOK_OVERNIGHT.md`):** §6 tables; the abstract's headline sentence; the amortization figure. The harness produces every number the placeholders need in one command. - -**Decision gate after the run:** policy beats controls on nonlinear families with cross-pattern holdout intact → submit AAAI July 27. Signal is weaker → the same paper, honestly hedged, goes to the next cycle (ICLR ~6 weeks out) with rungs 3+ matured; do not force it. - -## 8. Glossary (use these terms, exactly) - -- **Structure policy** — the learned model proposing augmentations (never "the solver"). -- **Menu-based structure selection** — the current capability (rungs 1–2). Not "invention." -- **Aux-required family** — problems whose fixed graph is certifiably unsolvable without the correct augmentation. -- **Certificate** — proof of unsolvability: `exact` (linear rank theorem) or `empirical` (solver-probe at a stated budget; probabilistic). -- **Enumeration arm** — try every candidate, first checker-accept wins; the exact ceiling and the cost baseline. -- **Amortization** — policy inference cost vs. enumeration cost; the economic claim. -- **Seed hygiene** — disjoint train/val/test seed ranges, asserted mechanically; `overlap_instances: 0` or the number does not exist. -- **Classical baselines** — `refine` (Langevin), `lm` (Levenberg–Marquardt), `exact` (linear), `random` (multi-start). Always labeled, never headlines. - ---- - -*Questions on framing → this file + `paper/notes/REVIEW_ATTACKS.md` first, then Quang. Questions on any number → `paper/PROVENANCE.md` has its command, seed, and commit.* diff --git a/OVERNIGHT_RESULTS.md b/OVERNIGHT_RESULTS.md deleted file mode 100644 index fbf440b..0000000 --- a/OVERNIGHT_RESULTS.md +++ /dev/null @@ -1,89 +0,0 @@ -# Overnight run — results & status - -**Date:** 2026-07-20 · **Machine:** Apple Silicon (MPS), torch 2.12 · **Driver:** -`scripts/run_overnight.py` (crash-safe manifest). Full artifacts land in -`results/overnight/` (MANIFEST.json, SUMMARY.md, per-phase logs) and the per-experiment -`results/**/*.json`. - -## Pre-flight (done) -- **Pulled main**; sibling training/structure scripts present (`train_scale.py`, - `train_structure_policy.py`, `run_invention_eval.py`). No open PRs to merge. -- **Env:** MPS available; `PYTORCH_ENABLE_MPS_FALLBACK=1` set. -- **Fixed the one stale test** (`test_invention_eval::test_evaluate_full_schema_and_pooling`): - the eval emits a guarded `reverse:policy_value_vs_random` Holm comparison when `pv_active` - (`_GoldStub` supplies a callable `predicted_pin`); updated the expected set (4→5) and `m` - (4→5). **Full suite green: 365 passed.** -- **Smoke run (`--smoke --force`) validated the whole pipeline:** 19 phases ok, 2 justified - skips (`eval_invention_heldout` — training fell back to `toys` data; `eval_cot` — key absent - in smoke), tests-gate excepted. Every training + eval phase executes end-to-end. - -## Real run (in progress) -Launched: `caffeinate -is nohup python3 scripts/run_overnight.py > overnight.out 2>&1 &` -with `GEMINI_API_KEY` exported so **`eval_cot` runs** (was the only skip in smoke). - -Phases (in order): `env_check, tests, train_stage_a, train_stage_b, train_structure_policy, -eval_p1_learned, eval_p1_refine, eval_main, eval_main_learned, eval_hard, eval_crossfamily, -eval_coupled, eval_dimension_scaling, eval_geometry, eval_h2, eval_structure_toys, -eval_invention, eval_invention_heldout, eval_math_coverage, eval_cot, figures, summarize`. - -- **Stage-A D512/L8 training dominates** (14h timeout; Stage-B 8h; structure policy 4h). Pace - checked early per the runbook; if projected Stage-A > ~12h, drop to D256/L6 in - `marc/configs/train/scale.yaml`. **[pace + any config change recorded below]** -- Eval battery + figures + CoT run after training; results overwrite the toy-scale numbers with - trained-checkpoint numbers. - -## Pace / decisions -- **Pace on MPS: ~280 examples/sec** at D512/L8 → Stage-A (n_train 10k × 50 epochs = 500k - examples) projects to **~30 min**, far under the 12h threshold. **No config downsizing needed** - — `scale.yaml` left at D512/L8, device auto→mps. -- **Loss behaviour: unstable but not diverging.** Epoch 1 loss starts ~20, spikes to ~300–390, - then trends down to ~20–60 by end of epoch 1; grad norms are large and volatile (300–3300, no - gradient clipping in the config, data unnormalised at D512). `best.pt` captures the - lowest-loss state. Expect a usable-but-noisy checkpoint; the eval phases report whatever solve - rates it achieves. -- Consequence: the whole run (Stage-A ~30 min + Stage-B GRPO + structure policy + full eval - battery + CoT) should finish in a few hours, not a full night. - -## Run outcome -**22 phases: 17 ok, 4 skipped, 1 failed.** Skips: `tests`/`train_stage_a`/`train_stage_b` -(intentional `--skip` on relaunch), `eval_invention_heldout` (policy trained on `toys` -fallback). Failed: `eval_main_learned` — killed after being stuck 3h+ (the D512 learned -solver's diffusion+guidance loop is too slow over the full perturbation/length suite on MPS; -the problems are convex/saturated anyway). - -### Two operational findings (matter for future runs) -1. **Stage-B GRPO diverged at D512** (loss 12.9 → 134,892; reward ~−9e8) and was ~35 min/epoch - (→ 8h timeout). Cut it; used the clean **Stage-A** checkpoint (loss **0.60**, good DSM - convergence — the first real-scale trained model). -2. **The eval scripts retrain their own small models and do NOT load the D512 checkpoint.** So - the scaled training does not reach the differentiating experiments (hard/coupled/dimension). - Only the convex evals used it (saturated at 1.0). **Fix needed:** wire `MARC_CKPT` into - `run_hard_eval`/`run_coupled_eval`/`run_dimension_scaling` so scaled runs actually test the - trained model. - -## Results (this run) — the honest pattern holds and sharpens -Consistent theme across every experiment: **the learned model beats *naive* baselines -(fixed / cold-start / deterministic) but ties or loses to *random* selection/restart** — except -in the high-dimensional *independent* regime where random suffers the curse of dimensionality. - -| Experiment | Key numbers | Read | -|---|---|---| -| Convex (p1, main, main-learned, CoT) | learned = refine = CoT = **1.000**, gap 0 | saturated; no signal | -| **Hard (bilinear, non-convex)** | learned = random on 3/4 (0.55/0.68/0.68), fails CircleLine | **learned ties random** | -| **Coupled** (chained bilinear) | learned 0.23/0.53/0.52/0.33/0.48 vs random 0.48/0.60/0.52/0.37/0.47 | **learned ≤ random at every n** | -| **Dimension scaling** (independent) | learned 0.95/0.95/**0.975**/**0.925**/0.25 vs random 1.0/0.725/0.075/0.0/0.0 | **learned ≫ random for n≥3** (random collapses); the one real win, but it is amortizing over the curse of dimensionality on *independent* traps | -| **Structure invention** (policy, valid: seed-overlap 0) | invention rate ~0.19; policy **> fixed** (p<1e-4) and **> no-context** (p<1e-4) but **= random** (p=0.28) | learned beats naive, **ties random** structure selection | -| Geometry (refine) | in-dist solve **0.56** | a non-saturated, harder real-ish domain (candidate for future signal) | - -**Bottom line:** the full-scale harness run **confirms the prior honest findings** — no new -main-track positive. The learned solver's only clean advantage over random is on independent -high-dimensional traps (a curse-of-dimensionality amortization), and it disappears under -coupling and on the structure-invention policy. `paper/RESULTS.md` / `PROVENANCE.md` remain the -canonical write-ups; these numbers reproduce them under the full pipeline. - -## Honest note carried forward -The scaled run tests whether training at D512/L8 changes the prior conclusions (the learned -solver did **not** beat random restart on coupled systems, structure/auxiliary invention hurt -the numeric solver, LLM-verify < LLM-direct). If the scaled model still ties the classical -baselines, that strengthens the honest characterization; it does not, by itself, create a -main-track positive. Report the numbers straight either way. diff --git a/SUMMARY.md b/SUMMARY.md deleted file mode 100644 index 0bd9e95..0000000 --- a/SUMMARY.md +++ /dev/null @@ -1,69 +0,0 @@ -# MARC — project summary - -**MARC** is a graph-diffusion approach to constraint solving: math problems are encoded as -**factor graphs** (variables + constraint factors), a **GNN denoiser** proposes candidate -assignments by reverse diffusion, and a **CAS** (SymPy) checks/guides them. This file is an -honest snapshot of what currently exists and works. - -## What works today - -### Pipeline (P0–P4, all implemented, 205 tests passing) -- **Graph core** — `VariableNode`/`FactorNode`/`Edge`, `FactorGraph`, JSON I/O, PyG - `HeteroData` builder (now encodes each factor's constant term). -- **CAS** — residuals, energy, energy-gradient, numeric+symbolic checker. -- **Diffusion** — cosine schedule, forward corruption, DDIM sampler, CAS-guided `solve()`. -- **Model** — `GraphDenoiser`: bipartite message passing with timestep + constant conditioning. -- **Training** — Stage A (denoising) + Stage B (GRPO RL against the checker reward). -- **Refinement** — energy-gradient / annealed-Langevin solver (`marc/refine`). -- **Eval** — solve rate, generalization gap, entrapment, perturbation, length extrapolation; - ablations (noise / guidance / purist reward); CoT LLM baseline. -- **Data** — generators for linear systems (2×2, 3×3) and geometry; structure toys; NL parser - for a small set of sentence templates. - -### Results -1. **The learned solver converges.** It previously diverged (inference ≈ 1e4, 0% solve). After - fixing five bugs — the denoiser never saw the noised input; equation constants were absent - from the graph tensors; the timestep didn't condition variables; inference guidance exploded; - a 1-variable `squeeze` bug — it now **solves 100%** on in-distribution and held-out linear - systems (generalization gap 0), and Stage-A loss drops from a flat ~1.0 to ~0.37. - Method = diffusion proposal + energy-descent polish. See `paper/notes/learned_solver_fix.md`. -2. **Noise escapes entrapment (H1/RQ2).** On 200 non-convex problems, deterministic descent is - **100% trapped**; annealed-noise (Langevin) descent cuts entrapment to 0.48 — reduction - **0.52 ± 0.09** (95% CI excludes 0). `results/p1_entrapment/`. -3. **Learned inference beats classical + prior, scaling in dimension.** On bundled non-convex - traps with per-instance-varying solutions, the learned model beats deterministic (0), - Langevin (→0 by n=3), and a mean-prior (0) at every dimension, though it also degrades at - n=6 (0.68 → 0.10). Required a specific architectural fix (condition variables on incident - constraint constants + a direct skip to the output). See `paper/notes/dimension_scaling_result.md` - and `scripts/run_dimension_scaling.py`. -4. **CoT baseline** runs on Gemini (`gemini-flash-lite-latest`) via the OpenAI-compatible - endpoint, with backoff + resume cache; full N=25 completed (in-dist 1.0, held-out 1.0). - -## What this is NOT (honest scope) -- **Toy problems only.** Largest solved: a 3×3 linear system, or a bundle of 1-variable cubic - traps. Problems arrive *already encoded* as factor graphs with explicit residuals. -- **No proofs, no natural math.** It finds numeric solutions to pre-encoded equations. It does - **not** do olympiad/competition math (IMO/USAMO): no NL understanding of real problems, no - multi-step proof, no lemma invention. That is a different research program entirely. -- **Checkpoints are toy-scale** CPU runs (D=128, L=4, minutes) — enough to exercise the - pipeline, not a scaling claim. Full-scale training (`results/p4_scale/roadmap.md`) is future - work. -- The scaling result's mechanism (amortized learned proposals beating blind search in high-D) - is a **known principle**; the contribution is a concrete graph-constraint-solving instance of - it plus the conditioning architecture it needs. - -## Key entry points -| Area | Path | -|---|---| -| Solve one problem | `scripts/solve_one.py` | -| Train checkpoints | `scripts/train_p2_checkpoints.py` | -| Main paper eval | `scripts/run_main_eval.py` | -| Entrapment ablation | `python -m marc.eval.ablations.noise_ablation --graphs 200` | -| Dimension scaling | `scripts/run_dimension_scaling.py` | -| CoT baseline | `GEMINI_API_KEY=… python -m marc.eval.baselines.cot_baseline` | -| Investigation notes | `paper/notes/learned_solver_fix.md`, `paper/notes/dimension_scaling_result.md` | - -## Status for AAAI -A **working** method with one clean scaling result and honest, well-characterized limitations — -a plausible submission if scoped to small constraint solving. It is not a state-of-the-art or -competition-math result, and the write-ups say so. diff --git a/marc/refine/noise.py b/marc/refine/noise.py deleted file mode 100644 index 2eceedd..0000000 --- a/marc/refine/noise.py +++ /dev/null @@ -1,4 +0,0 @@ -import torch - -def inject_noise(x: torch.Tensor, sigma: float) -> torch.Tensor: - return x + sigma * torch.randn_like(x) \ No newline at end of file diff --git a/results/p1_entrapment/energy_hist.png b/results/p1_entrapment/energy_hist.png deleted file mode 100644 index e4bf7da..0000000 Binary files a/results/p1_entrapment/energy_hist.png and /dev/null differ diff --git a/results/p1_entrapment/entrapment_bar.png b/results/p1_entrapment/entrapment_bar.png deleted file mode 100644 index aa22fdb..0000000 Binary files a/results/p1_entrapment/entrapment_bar.png and /dev/null differ diff --git a/results/p1_entrapment/trajectories.png b/results/p1_entrapment/trajectories.png deleted file mode 100644 index 4442832..0000000 Binary files a/results/p1_entrapment/trajectories.png and /dev/null differ diff --git a/run_logs/RUN_1.md b/run_logs/RUN_1.md deleted file mode 100644 index 9866acd..0000000 --- a/run_logs/RUN_1.md +++ /dev/null @@ -1,152 +0,0 @@ -# RUN 1 — full-scale overnight harness (2026-07-20/21) - -Driver: `scripts/run_overnight.py` on MacBook M5 (MPS, torch 2.12). Outcome: 22 phases, -17 ok / 4 skipped / 1 failed. Canonical numbers: `paper/RESULTS.md` + `paper/PROVENANCE.md`; -raw outcome: `OVERNIGHT_RESULTS.md`. - -## Summary of results - -The run confirms the honest pattern from the small-scale experiments and sharpens it: the -learned model beats naive baselines (fixed / cold-start / deterministic) everywhere, but ties -or loses to *random* selection/restart everywhere except one regime — independent -high-dimensional traps, where random restart collapses under the curse of dimensionality and -the learned proposal holds. - -| Experiment | Key numbers | Read | -|---|---|---| -| Convex (p1, main, CoT) | all 1.000, gap 0 | saturated, no signal | -| Hard bilinear | learned = random on 3/4 (0.55/0.68/0.68); CircleLine 0.00 | ties random | -| Coupled chained bilinear | learned ≤ random at every n (0.23–0.48 vs 0.37–0.60) | ties/loses, 0/5 wins | -| Dimension scaling (independent) | learned 0.95/0.95/0.975/0.925/0.25 vs random 1.0/0.725/0.075/0/0 | learned ≫ random for n≥3 | -| Structure-selection policy | > fixed and > no-context (p<1e-4); = random (p=0.28) | ties random selection | -| Geometry (refine) | 0.56 in-dist | non-saturated; best future lead | -| LLM (Gemini) + MARC | direct ≈0.80 vs formalize-then-solve ≈0.00 on MATH | formalization is the bottleneck | - -Training: Stage-A D512/L8 converged cleanly (DSM loss 0.60, ~30 min at ~280 ex/s on MPS). -Stage-B GRPO diverged (loss 12.9 → 134,892; reward ~−9e8) and was cut. `eval_main_learned` -was killed after 3h+ stuck (D512 diffusion+guidance too slow on MPS over the full -perturbation suite; that eval is convex/saturated anyway). - -One caveat that limits what this run can claim: the differentiating eval scripts -(`run_hard_eval`, `run_coupled_eval`, `run_dimension_scaling`) retrain their own small models -and never load the D512 checkpoint. So the scaled model was only ever tested on the saturated -convex evals. The negatives above are established for the small models; "scale doesn't change -them" is expected but not yet measured. - -## What we have that's good - -1. A working end-to-end neuro-symbolic system: factor graphs + GNN diffusion denoiser + exact - SymPy residuals/energy/gradients + exact accept checker + classical Langevin fallback. -2. The learned solver converges (was diverging at 0%; five real bugs found and fixed — - `paper/learned_solver_fix.md`). First real-scale trained checkpoint (D512/L8, loss 0.60). -3. A genuine, controlled positive: the dimension-scaling crossover. Random restart wins n≤2, - learned wins n≥3 (0.925 vs 0.000 at n=4). Clean amortized-inference story. -4. The entrapment result (RQ2): deterministic descent 100% trapped → annealed Langevin 0.475; - reduction 0.525 ± 0.086, 95% CI excludes 0, N=200, pre-registered. -5. Partial cross-family transfer: the hybrid solves 2/4 held-out families it never trained on - at 0.683 (p<1e-4), plus the finding that a pathological training family disrupts transfer. -6. An architectural finding: variables must be conditioned directly on incident constraint - constants (LayerNorm washes magnitude out); the constant→output skip cut mean|err| 5.4 → 0.9. -7. Statistical hygiene throughout: every solve rate has N + Wilson CI or z-test; the - random-multistart control most papers skip; provenance for every number (R1–R14). -8. Rigorous negatives, properly controlled: the coupled family (R7), structure selection vs - random (R8 protocol), LLM-verify vs LLM-direct. These are informative, not just failures. -9. Crash-safe overnight harness with manifest/skip/relaunch, 365-test green suite, MATH - coverage reality check (0/48, ~20% constraint-shaped). -10. Geometry at 0.56 in-dist: a non-saturated, real-ish domain where signal is still possible. - -## Every flaw - -1. **No main-track positive.** All three angles are negative: learned proposal ties/loses to - random restart on hard and coupled families; structure selection ties random; LLM+MARC - formalization scores ~0 vs ~0.80 direct. -2. **The one positive is narrow and synthetic.** The n≥3 crossover needs per-variable-separable - solutions *and* random restart collapsing; it vanishes under coupling. A reviewer reads it - as "learned marginals on toy problems built for the model." -3. **The R7 diagnosis is structural:** the model effectively learns per-variable marginals, so - any coupled (joint) solution space kills the advantage. This is an architecture limit, not a - tuning problem. -4. **Harness gap: the differentiating evals never load the trained D512 checkpoint** — they - retrain small models. The scaled run was largely wasted on saturated convex evals. -5. **Stage-B GRPO diverges at D512** (unbounded shaping reward — a raw energy delta that can - reach ~1e16 within the ±1e4 rollout clamp — plus unnormalized inputs; reward ~−9e8), - ~35 min/epoch. No usable RL stage. Gradient clipping is already in place (`grad_clip: 1.0` - in `scale.yaml`, applied in both stages) and doesn't save it. -6. **CircleLine fails completely** (0.00 in-dist and cross-family), and cross-family transfer - fails on 2/4; adding CircleLine to the training mix collapses BilinearSystem transfer - (0.70 → 0.00). -7. **Learned degrades at n=6 even on independent traps** (0.25/0.10) and loses to random at n≤2. -8. **R8 structure-selection numbers are withdrawn** (test seeds == validation seeds, plus a - train/eval data-source mismatch); clean seed-space-v1 numbers not yet regenerated. Even the - valid rerun ties random (p=0.28). -9. **All problems are synthetic.** MATH coverage 0/48; no real domain result yet; geometry only - has the classical-refine number (0.56), no learned-vs-random comparison there. -10. **Entrapment is confirmatory, not novel** — textbook annealed Langevin. -11. **MPS is too slow for D512 diffusion+guidance at eval scale** (the 3h+ hang); Stage-A loss - is volatile (spikes to ~390 in epoch 1) despite the `grad_clip: 1.0` already in the config. -12. **No `.tex` exists** with 6 days to deadline; `paper/*.md` sprawl (~10 files) not yet - consolidated. -13. Hygiene debt: OpenAI + Gemini keys exposed in a working chat (need rotation); a pending - results commit may still need pull+push; Gemini free tier quota caps the CoT baseline - (small N, weak model). - -## Solutions - -1. **(Flaw 4, cheapest, do first)** Wire `MARC_CKPT` into `run_hard_eval`, `run_coupled_eval`, - `run_dimension_scaling` so they load the D512 checkpoint. Rerun those three with - `--skip tests,train_stage_a,train_stage_b`. Either scale changes a conclusion (new result) - or every claim upgrades from "small model ties random" to "scale doesn't rescue it." -2. **(Flaws 1–3, the only real path to a positive)** Attack the joint-distribution limit - directly: condition the denoiser across the coupling structure (message passing that - preserves constraint-constant magnitude along chains, or an autoregressive/joint proposal - head) and re-test on the coupled family. Until the proposal is joint, R7 will reproduce. -3. **(Flaw 9)** Run learned-vs-random on geometry. It is the one non-saturated, non-separable, - real-ish domain; a win there is worth more than all the synthetic tables combined. ~2–3 days. -4. **(Flaw 5)** Before any Stage-B retry: input normalization, reward scaling/clipping (the - shaping reward is a raw, unbounded energy delta; a −9e8 reward guarantees divergence and - gradient clipping — already applied — can't rescue it), lower LR. Or drop Stage-B for this - paper — Stage-A is the checkpoint that matters. -5. **(Flaw 6)** Treat CircleLine as a diagnostic: characterize why (solution manifold geometry - vs training distribution) and report it as the failure-mode analysis section. Curriculum or - family-weighting for the transfer-collapse effect. -6. **(Flaw 8)** Regenerate R8 under seed-space v1 (disjoint ranges, `overlap_instances: 0` in - the results JSON) via the fixed harness; cite nothing until then. -7. **(Flaw 7)** Report the n=6 dropoff and n≤2 loss openly and scope the claim to the crossover - window. Optionally probe whether more restarts (best-of-K scaling curve) shifts the crossover. -8. **(Flaws 10, 12)** Write the paper to the corrected framing now: thesis = "learned proposals - amortize *independent* search, not *joint* search — a controlled characterization of when - learning helps constraint solving." Entrapment becomes supporting evidence, not a claim of - novelty. Start the `.tex` today (intro/method/related-work are experiment-free; - `related_work.md` is ready); consolidate `paper/*.md` into `paper/notes/`. -9. **(Flaw 11)** Cap eval-suite sizes for D512 on MPS and add a per-phase wall-clock budget to - the harness (`scale.yaml` already carries `grad_clip: 1.0`). -10. **(Flaw 13)** Rotate both API keys; `git pull --no-rebase origin main` then push the pending - results commit; fund a real key if the CoT baseline stays in the paper (N≥100, k≥4, - stronger model) — otherwise cut it. - -## AAAI main-track confidence - -*Reviewed as by a harsh AAAI PC member before scoring:* - -> The central claim reduces to: a diffusion model beats random multi-start only when solutions -> are per-variable separable and dimension makes random search infeasible — and the authors' -> own controls show the advantage disappears the moment variables couple. That is a negative -> result about the method they built. The one positive (R5) is on synthetic trap families -> constructed to exhibit exactly the property the model exploits; no real domain, 0/48 on MATH, -> and the entrapment result is standard annealed Langevin. The structure-"selection" component -> ties random choice, and the first version of that experiment was seed-contaminated. Transfer -> fails on 2/4 families including a total collapse. The statistical rigor and the honesty of the -> controls are genuinely above average — I would say this is the best-executed negative result -> I've reviewed this cycle — but AAAI main track wants a contribution, not a well-documented -> boundary. As a "when does learning help" characterization study it lacks the breadth (one -> problem class, one architecture) that would make the negative itself the contribution. Weak -> reject; would be a strong workshop paper. - -**Main-track acceptance confidence: 8/100.** - -The rigor floor is real (controls, CIs, provenance) and prevents a desk-reject-quality score, -but with no positive on coupled/realistic problems, no real domain, and 6 days to deadline with -no `.tex`, acceptance would require a lenient committee, not a good draft. Contingencies: if -the D512-checkpoint rerun flips a coupled result, ~20/100; if geometry produces a controlled -learned-vs-random win, ~30–35/100. Workshop acceptance with the honest framing: ~85/100 — -that is the venue this project is currently built for. diff --git a/scripts/make_supplementary.sh b/scripts/make_supplementary.sh index ac9d8bd..1ead7ff 100755 --- a/scripts/make_supplementary.sh +++ b/scripts/make_supplementary.sh @@ -28,6 +28,8 @@ rm -f paper/tex/*.fls paper/tex/*.aux paper/tex/*.log paper/tex/*.out \ rm -rf run_logs 2>/dev/null || true # §3 scrub names / possessives / milestone tags / org URL / author line (BSD sed -i '') +# staged camera-ready author block in the paper source (names live only at camera-ready) +[ -f paper/tex/marc_aaai.tex ] && sed -i '' '/% Submission is double-blind/,/affiliations{SAID Laboratory}/d' paper/tex/marc_aaai.tex # SPECIFIC substitutions FIRST (full author line, org, paths) — before the bare-name # catch-all, which would otherwise mangle "Quang Bui" into "the author Bui". [ -f README.md ] && sed -i '' \ diff --git a/scripts/solve_one.py b/scripts/solve_one.py deleted file mode 100644 index ec37567..0000000 --- a/scripts/solve_one.py +++ /dev/null @@ -1,92 +0,0 @@ -#!/usr/bin/env python3 -"""Solve a single constraint graph and report the checker verdict. - -Loads a FactorGraph JSON, runs a solver (default: the learned diffusion solver -with a trained checkpoint if one exists, else the gradient-refinement baseline), -and prints the best candidate plus the checker's accept/reject. - - python scripts/solve_one.py # two_equations, learned - python scripts/solve_one.py --solver refine # classical baseline - python scripts/solve_one.py --sample path/to.json --k 16 -""" -import argparse -from pathlib import Path - -from marc.graph.serialize import load_graph -from marc.cas.checker import Checker -from marc.eval.runner import Problem -from marc.eval.solver import load_solver - -# preference order for an auto-selected learned checkpoint -DEFAULT_CKPTS = [ - "checkpoints/denoiser_stage_b_standard.pt", - "checkpoints/denoiser_stage_b_purist.pt", - "checkpoints/denoiser_stage_a.pt", -] - - -def _pick_checkpoint(explicit: str | None) -> str | None: - if explicit: - return explicit - for c in DEFAULT_CKPTS: - if Path(c).exists(): - return c - return None - - -def main() -> None: - ap = argparse.ArgumentParser(description="MARC: solve one constraint graph and check it.") - ap.add_argument("--sample", default="marc/data/examples/two_equations.json", - help="input FactorGraph JSON") - ap.add_argument("--solver", default="refine", choices=["learned", "refine", "dummy"]) - ap.add_argument("--weights", default=None, help="denoiser checkpoint (learned solver)") - ap.add_argument("--k", type=int, default=8, help="candidates / best-of-N rollouts") - args = ap.parse_args() - - G = load_graph(args.sample) - checker = Checker() - # Checker verifies candidates against the graph's constraints; the placeholder - # solution just fixes the expected dimensionality for the solver contract. - problem = Problem(id=Path(args.sample).stem, graph=G, solution=[0.0] * len(G.variables)) - - solver_name = args.solver - kwargs: dict = {} - if solver_name == "learned": - ckpt = _pick_checkpoint(args.weights) - if ckpt is None: - print("No trained checkpoint found — falling back to --solver refine.") - solver_name = "refine" - else: - print(f"Using learned solver with checkpoint: {ckpt}") - kwargs["checkpoint"] = ckpt - - solver = load_solver(solver_name, **kwargs) - print(f"Solving {args.sample} with '{solver_name}' (best-of-{args.k})…\n") - candidates = solver.sample(problem, args.k) - - # first checker-accepted candidate wins; otherwise keep the lowest-residual one - best, best_res, accepted = None, float("inf"), False - for x in candidates: - if x is None: - continue - res = checker.verify(G, x) - if res.accepted: - best, accepted = x, True - break - if res.max_residual < best_res: - best, best_res = x, res.max_residual - - print("-" * 50) - if best is None: - print("Solver returned no finite candidate.") - return - names = [v.id for v in G.variables] - print("Solution:", {n: round(float(v), 4) for n, v in zip(names, best)}) - if accepted: - print("Checker: ACCEPTED ✅") - else: - print(f"Checker: REJECTED ❌ (max|residual| = {best_res:.3g})") - - -if __name__ == "__main__": - main()