Skip to content

refactor: rename the multi example package to multi_dataset #408

Description

@Jammy2211

Overview

The multi example package is renamed to multi_dataset across every workspace, doc and downstream consumer.

multi is ambiguous: it collides conceptually with the sibling multi_galaxy package, with the multi_gaussian_expansion feature folders, and with the general adjective "multi-" used throughout the prose ("multi-wavelength", "multi-plane", "multi-band"). multi_dataset names what the package actually is — the examples for fitting multiple datasets simultaneously via a factor graph.

Scope is the full rename, confirmed with the user: package directories and dataset paths and output path_prefix values. That last part is a breaking change for existing users' on-disk output/multi/ directories, and is accepted — it needs a release-note line.

Plan

  • Phase 1 — user workspaces (autolens_workspace, autogalaxy_workspace): git mv the scripts/, notebooks/, markdown/ and dataset/ packages, rewrite every reference and the folder-naming prose, sweep the path-keyed sidecar configs, regenerate notebooks/markdown/indexes.
  • Phase 2 — test / profiling / dev (autolens_workspace_test, autogalaxy_workspace_test, autolens_profiling, autolens_workspace_developer, HowToLens). Currently blocked — see the branch survey.
  • Phase 3 — downstream reference consumers (PyAutoLens/docs, PyAutoGalaxy/docs, autolens_assistant, autolens_jax_joss). Gated on phase 1 merging, because these are GitHub blob/main URLs and paired_example pointers that 404 or dangle until the workspace move lands.
  • Verify each repo by a post-move grep for the old path shape returning zero live hits, and by sidecar entry counts being preserved rather than merely changed.
Old New
scripts/multi/ scripts/multi_dataset/
notebooks/multi/ notebooks/multi_dataset/
markdown/multi/ markdown/multi_dataset/
dataset/multi/ dataset/multi_dataset/
dataset_type = "multi" dataset_type = "multi_dataset"
output/multi/... (path_prefix) output/multi_dataset/...
Detailed implementation plan

Affected Repositories

  • autolens_workspace (primary) — 89 tracked files under multi/, 40 files with references
  • autogalaxy_workspace — 61 tracked files, 35 files with references
  • autolens_workspace_test — 15 tracked files, 16 files with references
  • autogalaxy_workspace_test — 11 tracked files, 12 files with references
  • autolens_profiling — 3 tracked files, 7 files with references
  • autolens_jax_joss — 8 files with references
  • autolens_assistant — 5 files with references
  • PyAutoGalaxy — 2 files (docs URLs)
  • PyAutoLens — 1 file (docs URLs)
  • autolens_workspace_developer — 2 files
  • HowToLens — 1 comment

Branch Survey

Repository Current Branch Dirty?
./autolens_workspace main clean
./autogalaxy_workspace main clean
./autolens_workspace_test main clean
./autogalaxy_workspace_test main clean
./autolens_profiling main DIRTY (untracked dataset/imaging/jwst_lw/, unrelated)
./autolens_jax_joss main clean
./autolens_assistant main DIRTY (M README.md, unrelated)
./PyAutoLens main clean
./PyAutoGalaxy main clean
./autolens_workspace_developer main clean
./HowToLens main clean

Worktree conflict (hand-checked, not via worktree_check_conflict): autolens_workspace_test, autolens_profiling and autolens_workspace_developer are real worktrees (not symlinks) checked out on feature/point-source-chi-squared-variants under ~/Code/PyAutoLabs-wt/point-source-chi-squared-variants/, held by that task's in-flight phase 3. Phase 2 is blocked until it releases them. autogalaxy_workspace_test is free.

Suggested branch: feature/multi-package-rename-multi-dataset

Implementation Steps

Phase 1 — autolens_workspace + autogalaxy_workspace

  1. git mv scripts/multi scripts/multi_dataset, and the same for notebooks/, markdown/, and (autolens only) dataset/multi.
  2. Rewrite in-script path values: dataset_type = "multi""multi_dataset"; path_prefix=Path("multi", "modeling"); path_prefix=Path("multi") / "features"; the nested path_prefix=Path("slam", "multi", "simultaneous") (a leading-multi/ sweep misses this one); dataset_main_path = Path("dataset", "multi", "imaging", ...); dataset_path = Path("dataset") / "multi" / "rxj1131".
  3. Rewrite cross-package references from sibling packages: interferometer/features/datacube, weak/features/strong_lensing, guides/modeling, guides/results/aggregator, imaging, point_source/start_here.py, misc/database/scrape, misc/searches, misc/simulators, gallery.
  4. Rewrite folder-naming prose (no path-shaped regex catches these): README.md, scripts/README.md, AGENTS.md, root start_here.ipynb, the moved packages' own README.md and features/README.md.
  5. Sweep the path-keyed sidecars by hand (see "Traps" below).
  6. Regenerate notebooks + markdown + workspace_index.json + llms-full.txt.

Phase 2 — same mechanical pattern in the test/profiling/dev repos, plus their config/build/no_run.yaml SLOW and BOOTSTRAP-TARGET entries and smoke_tests.txt entries.

Phase 3 — GitHub URL rewrites in the two model_cookbook.md files and PyAutoGalaxy/docs/overview/overview_2_new_user_guide.md (which includes a version-pinned Colab URL, blob/2026.7.29.2/notebooks/multi/...), the */multi package prose in both overview_3_features.md, the autolens_assistant wiki/skills pointers, and the autolens_jax_joss paired_example fields in benchmarks/*.py, results/*.json, results/quick/*.json, results/RESULTS.md and README.md.

Traps

1. Path-keyed sidecars fail OPEN on a git mv — no error, just silent coverage loss. Sweep by hand; prove by entry count, not content:

  • smoke_tests.txt — all four workspaces, including a commented-out # multi/start_here.py line in autolens_workspace
  • config/build/no_run.yaml — both _test repos (SLOW + BOOTSTRAP-TARGET entries; one comment also names dataset/multi/lens_sersic)
  • config/build/profile_release.yamlautolens_workspace has - pattern: "multi/start_here", a pattern match, so it silently stops matching rather than erroring
  • .script_sizes.json — both workspaces; edit by hand, do not run check_sizes.sh --update (it sweeps in unrelated repo-wide drift)
  • .navigator_check_ignore — currently no multi entries, but re-check after the move
  • autolens_workspace_developer/.gitignorejax_profiling/dataset/multi/
  • HowToLens/config/build/profile_smoke.yaml — a comment naming multi/start_here

2. multi/ is not always a path — do NOT blind-sed. In scripts/point_source/features/multiple_sources/ and its notebooks, "the multi/factor-graph API" is English meaning multi-dataset, not the folder. Those must stay (or be reworded to "multi-dataset/factor-graph"). Conversely, prose naming the folder — "Unlike other multi simulators", "The multi package extends…", "Checkout the autolens_workspace/*/multi package" — does need rewriting and no path-shaped regex catches it. The sweep is grep-assisted with eyes on each hit, not automated.

3. Out of scope, deliberately: PyAutoMind/complete/** and draft/** historical records (a log of what happened, not live references — do not rewrite history); generated .artifacts-* directories; and every multi token that is not this package — multi_galaxy, multi_gaussian_expansion, multipoles, multiplane, MultiStartProdigy, multiprocessing, and all prose uses of "multi-wavelength" / "multi-band".

Verification

  • Post-move grep for the old path shape returns zero live hits per repo.
  • Sidecar entry counts preserved (not merely changed).
  • Notebooks/markdown regenerated via generate.py from the repo root with the short project key (al = 308 scripts, ag = 132) — from the workspace root it prints "0 scripts" and silently regenerates nothing.
  • features/ subpackage names are unchanged throughout.

Key Files

  • */scripts/multi/ — the package being moved (both workspaces + both _test repos + profiling)
  • */smoke_tests.txt, */config/build/no_run.yaml, */config/build/profile_release.yaml, */.script_sizes.json — path-keyed sidecars that fail open
  • */workspace_index.json, autolens_workspace/llms-full.txt — generated indexes, regenerate
  • PyAutoLens/docs/general/model_cookbook.md, PyAutoGalaxy/docs/general/model_cookbook.md, PyAutoGalaxy/docs/overview/overview_2_new_user_guide.md — GitHub blob + pinned Colab URLs
  • autolens_jax_joss/benchmarks/*.py, autolens_jax_joss/results/**paired_example pointers
  • autolens_assistant/skills/al_multi_dataset.md, autolens_assistant/wiki/core/** — assistant pointers

Phasing note

pyauto-brain feature scored this too-large (28) and proposed the generic design → core_api → workspace_examples → docs split. That is the repo-count difficulty heuristic and is wrong here — there is no design step and no API to change. Overridden in favour of the per-repo, merge-dependency-ordered split above; the override is recorded in the prompt.

Original Prompt

Click to expand starting prompt

Rename the multi example package to multi_dataset

Type: refactor
Target: workspaces
Repos:

  • autolens_workspace
  • autogalaxy_workspace
  • autolens_workspace_test
  • autogalaxy_workspace_test
  • autolens_profiling
  • autolens_jax_joss
  • autolens_assistant
  • PyAutoLens
  • PyAutoGalaxy
    Difficulty: large
    Autonomy: supervised
    Priority: normal
    Status: formalised

Original request

Rename the multi package as multi_dataset, which is a huge endeavor with docs,
URLs and everything so be thorough

Why

multi is ambiguous — it collides conceptually with the sibling multi_galaxy
package, with multi_gaussian_expansion feature folders, and with the general
adjective "multi-" used throughout the prose ("multi-wavelength", "multi-plane",
"multi-band"). multi_dataset names what the package actually is: the examples
for fitting multiple datasets simultaneously via a factor graph.

Scope (confirmed with the user)

Full rename, including dataset and output paths:

Old New
scripts/multi/ scripts/multi_dataset/
notebooks/multi/ notebooks/multi_dataset/
markdown/multi/ markdown/multi_dataset/
dataset/multi/ dataset/multi_dataset/
dataset_type = "multi" dataset_type = "multi_dataset"
output/multi/... (path_prefix) output/multi_dataset/...

All nine repos above are in scope. This is a breaking change for existing users'
on-disk output/multi/ directories — that is accepted.

Surveyed surface (~330 path-shaped references)

Directories to git mv (scripts / notebooks / markdown / dataset):

  • autolens_workspace: scripts, notebooks, markdown, dataset (incl.
    dataset/multi/rxj1131 real data)
  • autogalaxy_workspace: scripts, notebooks, markdown
  • autolens_workspace_test: scripts/multi (jax_likelihood, visualization,
    images), dataset/multi/lens_sersic
  • autogalaxy_workspace_test: scripts/multi (jax_grad, jax_likelihood)
  • autolens_profiling: scripts/multi/likelihood_runtime,
    dataset/multi/imaging, results/runtime/multi

Path-keyed sidecars that fail OPEN on a git mv (must be swept explicitly —
see [[feedback_extension_filtered_grep_misses_dotfiles]] and
[[feedback_no_run_suffix_entries_break_on_moves]]):

  • smoke_tests.txt — all four workspaces have multi/... entries (autolens has
    a commented-out # multi/start_here.py line too)
  • config/build/no_run.yaml — autolens_workspace_test (SLOW + BOOTSTRAP-TARGET
    entries, incl. a comment naming dataset/multi/lens_sersic),
    autogalaxy_workspace_test (SLOW + BOOTSTRAP-TARGET)
  • .script_sizes.json — autolens_workspace + autogalaxy_workspace (path keys;
    do not run check_sizes.sh --update, which sweeps in unrelated repo-wide
    drift — see [[project_dspl_terminology_rename]])
  • .navigator_check_ignore — currently no multi entries, but re-check
  • autolens_workspace_developer/.gitignorejax_profiling/dataset/multi/
  • HowToLens/config/build/profile_smoke.yaml — comment naming
    multi/start_here
  • config/build/profile_release.yaml — autolens_workspace has
    - pattern: "multi/start_here" (a pattern match, so it silently stops
    matching after the move rather than erroring)

Generated indexes keyed by path (regenerate, do not hand-edit):

  • workspace_index.json — both workspaces (dozens of multi/... entries)
  • llms-full.txt — autolens_workspace

Package-listing prose that names the folder:

  • autolens_workspace/README.md, scripts/README.md, AGENTS.md
  • autolens_workspace/start_here.ipynb (root) — */multi package pointer
  • the moved packages' own README.md / features/README.md bodies
  • guides/results/aggregator/ (both workspaces) — "The multi package of the
    workspace illustrates…"

Output path_prefix values (in-scope per the confirmed scope):

  • path_prefix=Path("multi", "modeling"), Path("multi") / "features"
  • path_prefix=Path("slam", "multi", "simultaneous") — note the nested
    form under output/slam/, which a naive leading-multi/ sweep misses
  • dataset_main_path = Path("dataset", "multi", "imaging", dataset_name) and
    dataset_path = Path("dataset") / "multi" / "rxj1131"

Ambiguity trap — do NOT blind-sed

multi appears as an English word adjacent to a slash in prose that is not
a path. In scripts/point_source/features/multiple_sources/ and its notebooks,
"the multi/factor-graph API" means multi-dataset / factor-graph, not
multi/. Those must stay (or be reworded to "multi-dataset/factor-graph"),
not rewritten to multi_dataset/factor-graph. Every replacement needs eyes on
it; the sweep is grep-assisted, not sed-automated.

Conversely, prose that names the folder — "Unlike other multi simulators",
"The multi package extends…", "Checkout the autolens_workspace/*/multi
package" — does need rewriting, and a path-shaped regex will not catch it.

Docs & URLs:

  • PyAutoLens/docs/general/model_cookbook.md — 2 GitHub blob URLs
  • PyAutoGalaxy/docs/general/model_cookbook.md — 2 GitHub blob URLs
  • PyAutoGalaxy/docs/overview/overview_2_new_user_guide.md — a GitHub blob URL
    and a version-pinned Colab URL (.../blob/2026.7.29.2/notebooks/multi/...)
  • PyAutoLens/docs/overview/overview_3_features.md,
    PyAutoGalaxy/docs/overview/overview_3_features.md*/multi package prose
  • In-notebook Binder / Colab / "download" links inside the moved notebooks
    themselves

Cross-repo consumers:

  • autolens_jax_jossREADME.md, benchmarks/*.py (paired_example=),
    results/*.json, results/quick/*.json, results/RESULTS.md
  • autolens_assistantskills/al_multi_dataset.md (already named
    multi_dataset!), wiki/core/api/datasets.md,
    wiki/core/api/analysis_objects.md, wiki/core/concepts/multi_wavelength.md
  • Sibling workspace scripts referencing multi/ cross-links:
    interferometer/features/datacube, weak/features/strong_lensing,
    guides/modeling, imaging, misc/database/scrape,
    misc/searches, misc/simulators, gallery

Explicitly OUT of scope:

  • PyAutoMind/complete/** and PyAutoMind/draft/** historical records — these
    are a log of what happened, not live references. Do not rewrite history.
  • Generated .artifacts-* directories.
  • Any word containing multi that is not this package: multi_galaxy,
    multi_gaussian_expansion, multipoles, multiplane, MultiStartProdigy,
    multiprocessing, and all prose uses of "multi-wavelength"/"multi-band".

Phasing (Brain override — recorded)

pyauto-brain feature scored this too-large (28) and proposed the generic
design → core_api → workspace_examples → docs split. That is the repo-count
heuristic ([[feedback_brain_repo_count_difficulty_proxy]]) and is wrong here —
there is no design step and no API to change. Overridden in favour of a
per-repo split ordered by merge dependency:

  • Phase 1 — user workspaces (autolens_workspace, autogalaxy_workspace).
    The bulk of the work; everything downstream points at these.
  • Phase 2 — test / profiling / dev (autolens_workspace_test,
    autogalaxy_workspace_test, autolens_profiling,
    autolens_workspace_developer, HowToLens comment).
  • Phase 3 — downstream reference consumers (PyAutoLens/docs,
    PyAutoGalaxy/docs, autolens_assistant, autolens_jax_joss).
    Gated on phase 1 merging — these are GitHub blob/main URLs and
    paired_example pointers that 404 / dangle until the workspace move lands.

Phase 2 is BLOCKED at time of writing: autolens_workspace_test,
autolens_profiling and autolens_workspace_developer are real worktrees held
by the in-flight point-source-chi-squared-variants phase 3
(feature/point-source-chi-squared-variants). autogalaxy_workspace_test is
free. Phase 2 starts once that task releases its claim.

Execution notes

  • Library-first merge gate does not apply cleanly here: the library-side
    changes are docs-only (URLs). But the docs URLs point at
    autolens_workspace@main, so the workspace PRs must merge first or the
    doc links 404 in the interim.
  • README.md bodies inside the moved packages describe "the multi folder" —
    rewrite the prose, not just the paths.
  • The features/ subpackage names stay unchanged.
  • Prove each repo with a post-move grep for the old path shape returning zero
    live hits, and by the .script_sizes.json / smoke_tests.txt entry counts
    being preserved (not merely changed) — see
    [[feedback_smoke_entry_path_and_count]].
  • Regenerate notebooks/markdown via generate.py from the repo root with the
    short project key (al = 308 scripts, ag = 132) — see
    [[feedback_generate_py_cwd_and_project_key]].

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions