Skip to content

Cortex phase 4: migration — split the epics, backfill the rulings, move the science batch records #383

Description

@Jammy2211

Overview

Phase 4 of the cortex-birth epic (ledger draft/feature/pyautocortex/cortex_birth_epic.md; phases 0–3 shipped in #377/#379/#380/#382). The Cortex holds no phases and no rulings yet while the Mind still carries every science phase and the two 2026-08-31 science batches, and the rulings the human made that day live in three places with one cited file that does not exist. This phase moves the science half of four epics into the Cortex as phases, encodes the 2026-08-31/09-01 rulings as Cortex rulings of record, transcribes the science batch records in the Cortex grammar, and fixes the stale pointers. Nothing changes science state — it relocates the record of it.

The inventory corrected the prompt in twelve places, all absorbed into the plan: euclid#48 is a merged PR (clears) and phase 3b has no issue (opened at filing as a gate ref, decision 55); cluster phase 11 lives under draft/research/autolens/; the am rulings exist only in the review file; Migrated-from: needs a schema key (decision 53); intra-Cortex sequencing is planned + Ready when: (decision 54); the REWIND is one ruling on a synthetic phase, all-legacy_wrong phases are dropped from ready, subhalo gets one phase per lens, transcribed batch records are rewritten in the Cortex grammar and the Mind originals stay intact (decision 56); the subhalo runs are further along than every ledger says (all four witnesses landed).

Plan

  • Mind (slice B, first): open the three gate/follow-up issues (euclid 3b, legacy_point sweep, ell_comps disk constraint) without leaving draft/; remove the moved science prompts; close the subhalo active entry with a stub record; renumber euclid 3a/3b/6c/7 → 3/4/8/9; cortex-half: on four epics; queue feat: pyauto-status shell function for cross-repo drift dashboard #3 retired, Dashboard: flag non-main branches and print follow-up commands #9 noted; jax-compile-stall sentence struck; write the missing pm review and the pm record's three fields; one science vocabulary in the review docs; the decision-55 rule in REFERENCE + create_issue.
  • Cortex (slice A): Migrated-from: keys + decisions 53–56; 4 euclid + 11 inference_programme + 4 subhalo + 2 graphical-ep phases; rulings R-20260831-01..07 and R-20260901-01..04; am/pm batch records + reviews + the am packet copy; four epics.md entries; re-render.
  • Science ledgers (slice C, laptop): DECISIONS.md correction entry mapping each 2026-08-31+ decision to its R- id; CORTEX.md id list; subhalo wiki rows filled with the landed witnesses and citing the rulings; the .gitignore inline-comment fix.
  • PRs: PyAutoCortex → PyAutoMind → autolens_profiling; subhalo commits on main.
Detailed implementation plan

Decisions (recorded as docs/schema_decisions.md 53–56)

  • 53 — provenance header. Migrated-from: joins PHASE_KEYS and RULING_KEYS (one
    tuple edit each + REFERENCE + tests); value = the Mind path or ledger anchor the file was
    transcribed from. The prompt's requirement stands; the current schema would have flagged
    it as drift. Batch records may carry - migrated-from: (no key check there).
  • 54 — intra-Cortex sequencing. Gates: stays GitHub-refs-only. A phase that waits on
    another Cortex phase's ruling is State: planned with a body line Ready when: <phase> accepted (ruling R-…) under ## Question; the human moves it to ready after the
    ruling. A phase whose dev gate has no issue yet is planned with Ready when: <Mind prompt> is issued — add its ref to Gates: and move to gated.
  • 55 — Cortex-spawned dev follow-ups get an issue at filing without leaving draft/.
    The Mind prompt carries a body line Issue: <url> (opened <date> as a Cortex gate ref; reuse in start_dev — never open a second); create_issue reuses it. Mind
    REFERENCE.md gains the rule. Same mechanism gives the 3b gate its ref.
  • 56 — legacy-born encodings. Pre-Cortex runs are legacy (reusable) or legacy_wrong
    (quarantined) run lines with where:; a phase whose runs are all legacy_wrong is ruled
    drop from ready (it cannot be pulled). A programme-wide directive (the REWIND) is
    one ruling on a synthetic phase inference_programme/rewind_2026_08_31 whose runs are
    the two ids the directive names, with the directive verbatim as the ruling body. One
    phase per lens for the subhalo project. Transcribed batch records are rewritten in the
    Cortex grammar (member slug = phase slug), the Mind originals stay intact with a
    - moved-to: line — a verbatim history must live somewhere live (deviation from the
    prompt's "stub"); dev-only members of the am slot are noted, not transcribed.

The migration map (what lands in the Cortex)

phases/euclid/ (project euclid) — from draft/research/euclid/…, draft/feature/euclid/resimulate…

Cortex Mind file (removed from Mind) State Notes
4 dr1_prelim_10_lens_science_run draft/research/euclid/dr1_prelim_10_lens_science_run.md gated Gates: euclid#48, euclid#<3b issue> #48 is the merged 3a PR (clears); the 3b issue is opened by decision 55 on cpu_vis_lp_jax_vis_pix_numba_submission.md
5 resimulate_fitted_lens_simulator draft/feature/euclid/resimulate_fitted_lens_simulator.md planned — Ready when: euclid 4 accepted keeps the note that it also carries a real simulator.py PR (that PR is filed as a Mind dev prompt when the phase opens)
6 sersic_index_recovery (was 6a) draft/research/euclid/sersic_index_recovery.md planned — Ready when: euclid 5 accepted
7 magnification_robustness (was 6b) draft/research/euclid/magnification_robustness.md planned — same 6c stays in Mind as its Phase: 6
Each: Witness: = the prompt's acceptance sentence condensed to one checkable line;
Budget: from the prompt (4: 48:00; 5/6/7: 24:00); Review-minutes: carried;
Epic: euclid-dr1-prep; Lane: local-dev; body Question/Witness/Where to look from the
prompt text; Migrated-from:; the dead Autonomy: safe/Unattended: ready headers die.
Mind side: 3a → Phase: 3, 3b → Phase: 4, 6c → Phase: 8, 7 → Phase: 9 (order kept,
collisions gone); the epic ledger gets a renumbering table + "science half → Cortex".

phases/inference_programme/ (project inference_programme)

Cortex Source Runs State / ruling
1 rewind_2026_08_31 DECISIONS.md:1443 341908_5 legacy_wrong (where: output/legacy_wrong/searches/nautilus/…slam_source_pix_nn…), 341874_24 legacy_wrong (_void_341874_24) dropped — R-20260831-01 body = the REWIND entry verbatim
2 smc_probe_342018, 3 knn_rerun_342016_7, 4 refs_5_6_342016_56, 5 slogdet_ab_342017 am batch members (review file) legacy (MGE) / legacy_wrong (mesh) per the REWIND table; where: from the mirror dropped — R-20260831-02..05, body = the review's verbatim note ("…discard this and start again…"), Batch: 2026-08-31-am
6 failed_submissions_342008_10 am member (UNREVIEWED, carried) 342008,342009,342010 legacy_wrong (died in 1 s; CWD trap) dropped — R-20260901-nn body "carried, then superseded by the rewind"
7 mge_pos_ref_reuse pm ruling 340210_9 legacy → pulled → awaiting-ruling accepted — R-20260901-01 (Batch: 2026-08-31-pm)
8 mge_fp64_retro_baseline pm ruling the run behind results/baselines/InferenceRefs_v1/INDEX.json's mge_fp64 row (job id from the mirror's .out/search.log; if unknowable, the run-dir hash with where: and the id 0 is NOT allowed — find it) legacy accepted — R-20260901-02
9 delaunay_fp64_retro_baseline pm ruling the delaunay run in output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/ legacy_wrong dropped — R-20260901-03, Follow-ups: autolens_profiling#203, body carries the PositionsLH directive verbatim
10 inference_refs_v1_redo pm carried member jax-inference-phase1-refs 342091_[0-8,10] — pre-Cortex runs, so legacy lines with pulled_to: the mirror dirs awaiting-ruling — no ruling yet (mesh rows spent per R-20260901-03; the human rules next slot)
epics.md (Cortex): ## jax-inference-profiling title/ledger (autolens_profiling/results/notes/inference/PROGRAMME.md)/mind-half: PyAutoMind/epics.md#jax-inference-profiling/status "REWOUND to Phase 1; refs redo awaiting ruling". Mind epics.md entry gains - cortex-half: and a one-line status "science half → Cortex".

phases/subhalo_validation/ (project subhalo_validation) — one phase per lens

Cortex Runs (all legacy, where: = output/subhalo/detect/<pipeline>/<lens>/) State / ruling
1 delaunay_adapt_split_pl_sersic_0 342027_0 accepted — R-20260831-06 (results_summary text verbatim + the review note; Batch: 2026-08-31-am; follow-ups in prose: subhalo[2] off, RectangularBilinear next)
2 delaunay_adapt_split_pl_eff_0 342027_1 awaiting-ruling; R-20260831-07 leave-to-finish (--also with 3) then pulled today (witness −0.2425 landed)
3 delaunay_adapt_split_pl_eff_1_outer 342027_2, 342093_0 (adapt_split_fix rerun, --stop_after, "awaiting the human's judgement") awaiting-ruling; R-20260831-07 leave-to-finish
4 rectangular_adapt_pl_sersic_0 342094_0 (after: none), 342095_0 (after: 342094_0) awaiting-ruling (witness +0.2538 landed 2026-09-01; JSON untracked by the user's ruling — Where to look names it)
The Mind active.md entry closes (removed) + active/follow_up_wave_adapt_split_and_rectangular.md removed + stub complete/2026/09/subhalo-followup-moved-to-cortex.md (one paragraph, ## Original prompt folded in for the record's sake). state.md/results_summary.md/journal: fill the landed witness rows as facts and cite R-20260831-06/07; do not invent rulings for 342093/342095.

phases/slope_hierarchy/ and phases/ic50_workspace/ (graphical-ep)

  • slope_hierarchy/1 n25_scale_up from draft/research/graphical_ep/slope_hierarchy_n25_scale_up.md (all science) — planned, Epic: graphical-ep, Ready when: graphical-ep phase 2 (Mind) is issued — add its ref.
  • ic50_workspace/1 ep_scale_up from draft/research/graphical_ep/ic50_ep_scale_up.md — planned, same; the dev half is the existing Mind prompt feature/autofit/ep_lbfgs_jax.md (named in Where to look). ep_campaign.md phase map rows 3/4 point at the Cortex; its malformed header (Status: wrap) fixed.
  • Cortex epics.md ## graphical-ep + Mind cortex-half:. Project rows stay dormant (a planned phase on a dormant project is legal).

phases/inference_programme/11 cluster_extended_source_inference (cluster-strong-lensing phase 11)

From draft/research/autolens/cluster_extended_source_inference.md — planned, Epic: cluster-strong-lensing, Ready when: cluster phase 10 (Mind draft) is issued — add its ref. Cortex epics.md ## cluster-strong-lensing + Mind cortex-half:; the arc ledger's
line 11 points at the Cortex. (Its "Phase: 10" collision with cluster_regime_narrative.md
is noted in the ledger, not fixed here.)

Batch records and reviews (Cortex)

  • batches/2026-08-31-am.md — transcribed: dispatched: (board opened) 2026-08-29T22:00Z,
    review-at: 2026-08-31T13:00Z, members = the 7 science members as <phase slug>: <phase path> — <runs> — <minutes> — <state>, - migrated-from:, notes: names the dev
    retrospective euclid-phase2-pr46 as Mind-only. reviews/2026-08-31-am.md in the Cortex
    grammar (HEALTH per the original; decision: drop for the four rejected, accept,
    leave-to-finish, (none) + ruled: no for the carried one; notes verbatim).
    packets/2026-08-31-am.html copied (archive of record from now on; the Mind's stays at
    its Pages URL for localStorage — both noted).
  • batches/2026-08-31-pm.md — new: members inference_refs_v1_redo + the four subhalo
    phases + the three ruling phases; refreshed: lines from the Mind record's three
    refreshes; reviewed-at: 2026-09-01T15:13 (transcribed); review: batches/reviews/2026-08-31-pm.md
    (Cortex grammar; the three rulings ruled: yes, the carried members (none)).

Mind-side fixes

  • queue.md feat: pyauto-status shell function for cross-repo drift dashboard #3 → retired with - moved-to: PyAutoCortex phases/subhalo_validation/…; Dashboard: flag non-main branches and print follow-up commands #9
    gains note: superseded by cortex-birth (phase 5 retires it).
  • epics.md jax-compile-stall: strike the "follow-up filed under draft/research/ci/"
    sentence → "follow-up never filed (the resume door was never written)".
  • Two new drafts with issues at filing (decision 55): draft/maintenance/autolens_profiling/legacy_point_output_sweep.md
    (laptop action; body from DECISIONS.md:1553-1559) and
    draft/feature/autogalaxy/ell_comps_joint_disk_constraint.md (from PROGRAMME W10 + the
    8B crash channel PyAutoFit#1535); issues on autolens_profiling and PyAutoGalaxy.
    R-20260901-03's Follow-ups: cites prompt: shelve euclid-eceb-editorial-revision — untrackable, retired from the board #203 and the sweep issue.
  • batches/2026-08-31-pm.md gains reviewed-at: 2026-09-01T15:13 (transcribed),
    review-minutes-actual: (not given), review: batches/reviews/2026-08-31-pm.md; that
    review file written: science members transcribed from DECISIONS.md + the record; dev
    members UNREVIEWED with note "merged via /prm 2026-09-01; no packet ruling recorded".
  • batches/reviews/AGENTS.md:24merge | tweak | reject | defer # science members (Cortex vocabulary): accept | rerun | drop | leave-to-finish; packets/TEMPLATE.md:64
    dev: merge|tweak|reject|defer|structure-ok|UNREVIEWED · science: accept|rerun|drop|leave-to-finish.
  • REFERENCE.md: the decision-55 rule (Issue: line on Cortex-spawned drafts; create_issue
    reuses it) — and PyAutoMind/skills/create_issue/SKILL.md one sentence.
  • Moved prompts: git rm in Mind (history stays), the Cortex files carry Migrated-from:.

autolens_profiling (PR) and the science ledgers (main commits)

  • DECISIONS.md: one appended entry 2026-09-01 — Rulings of record (Cortex backfill)
    mapping each 2026-08-31+ decision to its R- id (never editing the originals); CORTEX.md
    gains the id list.
  • subhalo wiki/project/results_summary.md + state.md + journal: fill the landed witness
    rows (facts), cite R-20260831-06/07, note 342093/342095 await the human. Also fix the
    broken inline comment on the .gitignore sources/ line (the user's "next sweep" —
    cheap while here). Commit on main, push.

Execution (Opus, sequential where refs are needed)

  • Issue via /create_issue; prompt → active/; worktree cortex-migration for
    PyAutoCortex + PyAutoMind + autolens_profiling; subhalo edited in place.
  • Slice B first (Mind): the three issues (3b gate, sweep, ell_comps) + all Mind-side
    edits + the Mind pm review + git rm of the moved prompts + stub record + epics/queue/
    vocab/REFERENCE; lifecycle.py check OK; dashboard regen; local commit.
  • Slice A (Cortex): schema keys + decisions 53–56, all phase files (via cortex.py new
    • move + rule --today with rule_phase(now=) from Python for the 15:13 stamps),
      both batch records + reviews + packet copy, epics.md entries; cortex.py check OK;
      pytest green; dashboard re-render (Brain main); local commit.
  • Slice C (laptop): DECISIONS correction entry + CORTEX.md (worktree autolens_profiling,
    PR) and the subhalo wiki citations + gitignore fix (main, push).
  • Ship: Cortex PR → Mind PR (ledger + docs; feature-branch PR since REFERENCE/skills are
    code) → autolens_profiling PR; /prm under the standing authorisation.

Verification

  • Prompt witness: lifecycle.py check OK; cortex.py check OK; Mind dashboard Epics shows
    euclid-dr1-prep with phases 0–4, 8, 9 only (3a/3b renumbered) and the entry's
    cortex-half: (rendered? no — the Mind renderer ignores it; the epic ledger's link is the
    visible half — say so); Cortex dashboard shows euclid phases 4 gated on euclid#48 +
    euclid#<3b> and 5–7 planned; every DECISIONS.md entry from 2026-08-31 cites an R- id
    that resolves (the correction entry).
  • No science prompt remains under PyAutoMind/draft/research/ or active/ for the moved
    set; git log --follow on each Cortex phase names its Migrated-from:.
  • Cortex: pytest green; the pm/am reviews parse; gates --grade (dry) on euclid 4 reports
    PyAutoScientist 3b-3: firewall allowlist rider (clone conductor) #48 cleared and the 3b issue open → stays gated.

Branch Survey

Repository Current Branch Dirty?
./PyAutoCortex main clean
./PyAutoMind main clean
./autolens_profiling main clean (two other live worktrees on other branches)
/mnt/c/Users/Jammy/Science/subhalo_validation main (private remote) ?? results/rectangular_adapt/ (stays untracked by ruling)

Suggested branch: feature/cortex-migration (worktree root ~/Code/PyAutoLabs-wt/cortex-migration/)

Original Prompt

Click to expand starting prompt

Cortex phase 4 — migration: split the epics, backfill the rulings, move the science batch records, fix the stale pointers

Type: feature
Target: pyautocortex
Repos:

  • PyAutoCortex
  • PyAutoMind
  • autolens_profiling
    Themes:
  • mind-workflow
  • dashboard
    Difficulty: large
    Autonomy: supervised
    Priority: high
    Status: draft
    Consequence: judge
    Witness: lifecycle.py check and cortex.py check both exit 0; the Mind dashboard's Epics section shows euclid-dr1-prep with phases 0–3b, 6c, 7 only and a "cortex half" link; the Cortex dashboard shows euclid-dr1-prep phases 4, 5, 6a, 6b gated on euclid#48 + the 3b issue; every ruling in DECISIONS.md since 2026-08-31 cites an R- id that resolves
    Review-minutes: 30
    Unattended: needs-slicing
    Epic: cortex-birth
    Phase: 4
    Parent: draft/feature/pyautocortex/cortex_birth_epic.md
    Filed: 2026-09-01

Phase 4 of 7 in the PyAutoCortex birth epic. Gates: phases 2 and 3. Gates
phase 6. Slices if needed: (a) epics + prompts, (b) rulings backfill, (c) batch
records + stale pointers.

Nothing here changes science state — it relocates the record of it. Every move
is a git mv-shaped edit with the source path recorded in the destination's
header (Migrated-from:), so the Mind's history and the Cortex's join up.

Task

(a) Split the epics — the migration map from the epic ledger

Moves to the Cortex From Gates: it carries
euclid-dr1-prep 4, 5, 6a, 6b draft/research/euclid/{dr1_prelim_10_lens_science_run,sersic_index_recovery,magnification_robustness}.md, draft/feature/euclid/resimulate_fitted_lens_simulator.md 4: euclid#48 (3a) + 3b's issue; 5: phase 4's ruling; 6a/6b: phase 5's ruling
jax-inference-profiling (whole) epics.md entry; PROGRAMME.md stays in autolens_profiling as commentary autolens_profiling#200, #201
graphical-ep phases 3, 4 science halves draft/research/graphical_ep/ (split each into a Mind dev prompt + a Cortex phase) the dev halves' issues, when opened
cluster-strong-lensing phase 11 draft/feature/autolens/ arc member phase 10's issue
subhalo-followup-adapt-split-rectangular active.md + active/follow_up_wave_adapt_split_and_rectangular.md none — runs 342093/342094/342095 already in flight; state running or pulled per the current pull

Rules: Cortex phase numbers are distinct integers within a project (euclid 5
stays 5; 6a/6b become 6 and 7 — record the renumbering in both ledgers). The
Mind keeps its Phase: integers but fixes its own collisions (3a/3b → 3/4,
6c/7 → 8/9 — or whatever preserves order; document it). The euclid science
prompts' Autonomy: safe / Unattended: ready headers are wrong (ledger:
"supervised, never an autonomous ship gate") — they die with the move; the
Cortex phase files have no autonomy header at all. Mind epics.md entries gain
- cortex-half: PyAutoCortex/epics.md#<slug>; Cortex epics.md entries carry
- mind-half:. The Mind's active.md subhalo entry closes with a pointer to
the Cortex phase (no complete/ record — it never had an issue; write a
one-line complete/2026/09/subhalo-followup-moved-to-cortex.md so
lifecycle.py check stays clean).

Stays in Mind (ruled): euclid 0–3b, 6c, 7; every dev phase of every epic; the
numba-vs-jax-sparse verdict; the pyautoreduce local-data research prompts
(default: stay — decide and record).

(b) Backfill the rulings

  • Every ruling in autolens_profiling/results/notes/inference/DECISIONS.md
    from 2026-08-31 onward (the REWIND entry, the batch-am rulings, the
    2026-09-01 pm rulings) becomes a Cortex ruling R-… with the human's words
    verbatim; the DECISIONS entry gains a trailing Ruling of record: R-… line
    (append a correction entry — never edit the original text, per its own rule).
    Earlier entries are commentary and stay untouched.
  • The subhalo rulings (pl_sersic_0 ACCEPTED 2026-08-31, pl_eff
    leave-to-finish, the 2026-08-31 tweak rulings) become R-… entries; the
    journal and results_summary.md cite them.
  • The three pm legacy-reuse rulings (mge-pos-ref-reuse,
    mge-fp64-retro-baseline, delaunay-fp64-retro-baseline REJECTED) become
    rulings with the legacy/legacy_wrong run states recorded on the phase.

(c) Move the batch records and fix the stale pointers

  • PyAutoMind/batches/2026-08-31-am.md + packets/2026-08-31-am.html +
    reviews/2026-08-31-am.md move to the Cortex (an all-science slot; the one
    dev retrospective member euclid-phase2-pr46 stays listed with a note). The
    Mind keeps a one-line stub record pointing across so ledger_merge and the
    packet archive's stable-path rule are not broken: the HTML stays at the Mind
    Pages URL (localStorage notes are keyed by path) and the Cortex copy is the
    archive of record from now on.
  • batches/2026-08-31-pm.md: the two carried science members and three ruling
    members are recorded in a new Cortex batch record 2026-08-31-pm.md; the
    Mind record keeps its 9 dev members and gains the missing reviewed-at:,
    review-minutes-actual: and review: fields; write
    PyAutoMind/batches/reviews/2026-08-31-pm.md
    from the DECISIONS.md
    rulings (dated as the human's review, 2026-09-01 15:13, marked
    "transcribed") so the citation resolves.
  • Fix: queue.md feat: pyauto-status shell function for cross-repo drift dashboard #3 path (draft/research/subhalo_validation/… → the Cortex
    phase; then retire the entry — science leaves the Mind queue); queue.md Dashboard: flag non-main branches and print follow-up commands #9
    (superseded by this epic — phase 5 retires it; leave a note now);
    epics.md jax-compile-stall's nonexistent draft/research/ci/ follow-up
    (file it or strike the sentence); DECISIONS.md's nonexistent
    draft/maintenance/autolens_profiling/legacy_point_output_sweep.md and
    PROGRAMME W10's nonexistent
    draft/feature/autogalaxy/ell_comps_joint_disk_constraint.md (file both as
    Mind drafts with their issues created — they are Cortex-spawned dev
    follow-ups, and the gate rule says the ref exists at filing);
    batches/reviews/AGENTS.md:24 and packets/TEMPLATE.md:64 collapse to the
    one Cortex vocabulary for science members (dev vocabulary untouched).
  • Regenerate both dashboards (pyauto-brain intake dashboard --apply,
    pyauto-brain cortex dashboard --apply); Brain is already merged (phase 2).

Acceptance

  • The witness above; git log --follow on every moved file reaches its Mind
    history via the Migrated-from: header; no science prompt remains under
    PyAutoMind/draft/research/ or active/.
  • PRs: PyAutoCortex, PyAutoMind, autolens_profiling (DECISIONS correction
    entries + CORTEX.md), subhalo_validation (journal citations; commits on main).

Out of scope

  • Rulings before 2026-08-31 (commentary; not backfilled).
  • The batch conductor (phase 5). This phase moves records; it does not change
    how the next batch is planned.
  • Any RAL action. legacy_point/ relocation directed 2026-09-01 is a laptop
    action already queued in the science ledger, not part of this phase.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions