diff --git a/complete/2026/08/defer-scipy-sparse-import.md b/complete/2026/08/defer-scipy-sparse-import.md new file mode 100644 index 00000000..1beeeed6 --- /dev/null +++ b/complete/2026/08/defer-scipy-sparse-import.md @@ -0,0 +1,163 @@ +# Defer the eager scipy.sparse (and scipy.spatial) imports + +- shipped: 2026-08-22 — @PyAutoArray#477 (`91d8b97`, merged `6bbde1a` from + `claude/defer-scipy-sparse-import`). No issue was opened; the work went + straight to a PR off the back of the pynufft removal that surfaced it. +- classification: maintenance (libraries) — import-time performance, single + repo, no API change. Difficulty `small`, autonomy `safe`. +- result: `import autoarray` **464.4 ms → 183.7 ms** (medians of 15 runs, + Python 3.13, dev extras) — a **281 ms** saving, ~2.8× the ~0.10 s the prompt + estimated. Suites green: autoarray 1179 passed, autogalaxy 1103 passed / + 1 skipped, autolens 532 passed / 1 skipped. +- changes: `inversion/mesh/mesh_geometry/delaunay.py` (drop the module-scope + `import scipy.spatial`; two of its three use sites already had local imports, + so this finished a deferral left half-done), plus + `operators/derivative_util.py` and `operators/coarse_interp_util.py` (move + `from scipy.sparse import csr_matrix` into the four functions that use it). + +## The correction — the prompt's own premise was wrong + +Filed on the theory that `derivative_util.py:30`'s module-scope +`from scipy.sparse import csr_matrix` was costing ~0.11 s of every import. +Deferring it changed nothing, because `scipy.sparse` was never being imported +from there. Traced with a `sys.meta_path` hook: + +``` +autoarray/__init__.py:80 + -> inversion/mesh/mesh_geometry/delaunay.py:2 import scipy.spatial + -> scipy/spatial/__init__.py:111 from ._kdtree import * + -> scipy/spatial/_kdtree.py:4 from ._ckdtree import cKDTree +``` + +`scipy.spatial` (134 ms) pulls `scipy.sparse` (154 ms) in transitively, so the +`csr_matrix` import was riding on a subtree already paid for. Deferring +`scipy.spatial` **as well** is what removes both — and was required to satisfy +the prompt's own acceptance criterion. The `csr_matrix` deferrals were kept +anyway: no longer load-bearing on their own, but they keep `scipy.sparse` off +the import path independently of what `scipy.spatial` happens to pull in. + +**The lesson, now hit twice** (the pynufft removal made the same mistake first): +a module's `importtime` **cumulative** figure is not its **exclusive** cost. +Attributing a saving to a dependency requires checking who else pulls its +subtree in. + +## Traps (for the next reader) + +- The prompt warned that the deferral "must survive unpickling in + multiprocessing workers, so hang it off the call sites rather than `__init__` + alone", citing `transformer.py:_load_nufftax()`. That precedent did **not** + apply here: every use site is inside a function, so a plain function-local + import suffices — it runs on every call and hits `sys.modules` after the + first. `_load_nufftax()` needed a module-level cache only because unpickled + `TransformerNUFFT` instances in Pool workers never re-run `__init__`. +- `delaunay.py` already had local `import scipy.spatial` at two of its three use + sites — a half-finished deferral that bought nothing, because the surviving + module-scope import kept the subtree on the path. A partial deferral of a + heavy module is worth exactly zero. + +## Lifecycle note + +The implementation recorded its own correction back into the prompt on +2026-08-22 (`79d057ce`) but left the file in `draft/` with `Status: in-flight`. +The 2026-08-24 completed-prompt reconciliation sweep (`06d76dbb`) touched this +file to repoint a cross-reference but passed over it — the `in-flight` header +read as live work rather than shipped work — so it kept rendering as pickable +backlog. Picked off the dashboard by `/start_dev` on 2026-08-25, found already +merged, and recorded here. + +Verified against `PyAutoArray@main` at record time: a module-scope grep for +`scipy.sparse` / `scipy.spatial` across `autoarray/` returns nothing — every +surviving reference is function-local. + +## Original prompt + +# Defer the eager scipy.sparse import in derivative_util (~0.10 s of import) + +Type: maintenance +Target: libraries +Repos: +- @PyAutoArray +Difficulty: small +Autonomy: safe +Priority: normal +Status: in-flight +Filed: 2026-08-22 (backfilled from git) + +## Where this came from + +Found 2026-08-22 while measuring the pynufft removal +(`complete/2026/08/remove-pynufft-legacy-transformer.md`). That task +assumed removing pynufft would take ~0.23 s off `import autoarray`. It takes +~10 ms. The reason is the real target: + +`autoarray/operators/derivative_util.py:30` does + +```python +from scipy.sparse import csr_matrix +``` + +at module scope. `scipy.sparse` costs **0.106 s cumulative** and is imported on +every `import autoarray`, whether or not anything touches a derivative +operator. pynufft's apparent 0.19 s was ~95 % this same shared subtree — +removing pynufft did not remove it, because `derivative_util` pulls it in +anyway. + +## Evidence (Python 3.13, dev extras, median of 7 runs) + +| | `import autoarray` | +|---|---| +| main, pynufft installed | 369.8 ms | +| pynufft removed | 359.9 ms | + +`python -X importtime` on the pynufft-removed branch still shows +`scipy.sparse` at 0.106 s cumulative. + +## Task + +Defer the `csr_matrix` import into the functions that build the sparse +operators (the same pattern `transformer.py` already uses for `nufftax` via +`_load_nufftax()`). Check for other eager `scipy.sparse` importers before +assuming this is the only one — the win only lands if *no* module-scope import +of it survives on the `import autoarray` path. + +Note the precedent in `transformer.py`: the deferral must survive unpickling in +multiprocessing workers, so hang it off the call sites rather than `__init__` +alone. + +## Acceptance + +- `python -X importtime -c "import autoarray" | grep scipy.sparse` is empty. +- Median `import autoarray` drops by ~0.10 s against the same measurement + method above (record the before/after numbers in the PR). +- Full suite green; sparse-operator behaviour unchanged. + +## Correction + result (implemented 2026-08-22, PyAutoArray#477) + +**This prompt's own premise was wrong, in exactly the way the pynufft one +was.** Deferring `derivative_util.py:30` changed nothing: `scipy.sparse` was +never imported from there. Traced with a `sys.meta_path` hook: + +``` +autoarray/__init__.py:80 + -> inversion/mesh/mesh_geometry/delaunay.py:2 import scipy.spatial + -> scipy/spatial/__init__.py:111 from ._kdtree import * + -> scipy/spatial/_kdtree.py:4 from ._ckdtree import cKDTree +``` + +`scipy.spatial` (134 ms) pulls `scipy.sparse` (154 ms) transitively, so the +`csr_matrix` import was riding on a subtree already paid for. Deferring +`scipy.spatial` as well is what removes both — and is required to satisfy this +prompt's own acceptance criterion. + +The general lesson, now hit twice: **a module's `importtime` cumulative figure +is not its exclusive cost.** Attributing a saving to a dependency requires +checking who else pulls its subtree in. + +Result: `import autoarray` **464.4 ms -> 183.7 ms** (medians of 15 runs, +Python 3.13, dev extras) — a 281 ms saving, ~2.8x this prompt's ~0.10 s +estimate, because `scipy.spatial`'s own cost comes off too. Both greps are +empty. Suites green: autoarray 1179, autogalaxy 1103/1 skipped, +autolens 532/1 skipped. + +Note `delaunay.py` already had local `import scipy.spatial` at two of its three +use sites — this deferral had been started and left half-done. diff --git a/complete/2026/08/remove-pynufft-legacy-transformer.md b/complete/2026/08/remove-pynufft-legacy-transformer.md index 5ebcfad8..2b232411 100644 --- a/complete/2026/08/remove-pynufft-legacy-transformer.md +++ b/complete/2026/08/remove-pynufft-legacy-transformer.md @@ -67,8 +67,11 @@ The 2026-08-19 filing had two factual errors, both found during implementation: The removal is still worth doing — an unmaintained dependency, one dead class, and a `dev` extra that is broken against SciPy >= 1.17 — but **not** on -import-time grounds. The real 0.10 s win is filed separately as -`draft/maintenance/libraries/defer_scipy_sparse_import.md`. +import-time grounds. The real win shipped separately as +`complete/2026/08/defer-scipy-sparse-import.md` — and it was 281 ms, not the +0.10 s estimated here: `scipy.sparse` was never imported from +`derivative_util.py` at all; `scipy.spatial` pulls it in transitively, and +deferring both is what removed the subtree. ## The Intel-macOS decision (settled 2026-08-22) diff --git a/complete/index.md b/complete/index.md index 51997c66..729b5d14 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1136 records across 7 buckets. +1137 records across 7 buckets. ## Highlights @@ -72,6 +72,7 @@ _(curate hard-won records here — survives regeneration.)_ - [database-guide-info-inline](2026/08/database-guide-info-inline.md) — the dataset_1d database guide chain failed on any fresh checkout — - [database-guide-sample-weight-threshold](2026/08/database-guide-sample-weight-threshold.md) — `guides/results/database/start_here.py` ran its own Nautilus fits capped at n_like_max=300 then indexed sample… - [dataset-allowlist-small-datasets-guard](2026/08/dataset-allowlist-small-datasets-guard.md) +- [defer-scipy-sparse-import](2026/08/defer-scipy-sparse-import.md) - [delaunay-nan-probe-fix](2026/08/delaunay-nan-probe-fix.md) - [delaunay-nn-laptop-gpu-profile](2026/08/delaunay-nn-laptop-gpu-profile.md) — Added the CPU, RTX 2060, and A100 DelaunayNN profiling sweep and result artifacts at matched PyAuto source rev… - [dep-floors-source-chain-ci](2026/08/dep-floors-source-chain-ci.md) diff --git a/dashboard.html b/dashboard.html index 01417309..863801f0 100644 --- a/dashboard.html +++ b/dashboard.html @@ -185,7 +185,7 @@

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

- +

Last updated 2026-08-25. This page is generated from active/, draft/ and the registry files, so it is only as current as they are. dashboard_refresh.yml re-renders it on every push to main — that heals a stale page, but not a stale prompt: a task that shipped without its prompt advancing to complete/ keeps rendering here as pickable backlog. Reconciling those is the refresh below.

Fix release JAX runtime compatibility and likelihood parity🐛 bughealth_fixestoo-largesupervisedhigh

Fix JIT quick-update visualization output regressions🐛 bughealth_fixestoo-largesupervisedhigh

Quick wins (small enough, and safe enough to run unattended)

-

Defer the eager scipy.sparse import in derivative_util (~0.10 s of import)🧹 maintenancelibrariessmallsafenormal

The weekly smoke run's timings land in results-* under no discoverable name🧹 maintenancepyautoheartsmallsafelow

In flight markdown version

Issued — each has an open GitHub issue and usually a branch.

@@ -251,7 +250,7 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

140 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

+

139 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

feature — 27

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

@@ -307,12 +306,11 @@

Backlog

interferometer/start_here.py OOM in nightly release-validation integrate leg🐛 bugautolens

-maintenance — 25 +maintenance — 24

autocti_workspace has no Navigator Check, so its CI can never roll…🧹 maintenancecimediumsupervisedhigh

Untrack the generated FITS test artifacts in autoarray🧹 maintenancelibrariessmallsupervisedmedium

smoke_install.sh's stale jax<0.7 pin — CI is on the right jax…🧹 maintenancecilowsupervisedmedium

autolens_workspace_developer rectangular experiments — Gut stash + rename🧹 maintenanceautolens_workspace_developersmallsupervisednormal

-

Defer the eager scipy.sparse import in derivative_util (~0.10 s of import)🧹 maintenancelibrariessmallsafenormal

Mirror drifted library config keys into the workspace configs🧹 maintenanceworkspacessmallsupervisednormal

euclid: CRLF has reached the HPC submit scripts AGENTS.md warns about🧹 maintenanceworkspacessmallsupervisednormal

Un-park imaging/features/scaling_relation/slam once PyAutoArray#431 merges🧹 maintenanceworkspacessmallsupervisednormal

@@ -500,12 +498,6 @@

Backlog -2026-08-22 -filed -Defer the eager scipy.sparse import in derivative_util (~0.10 s of… - - - 2026-08-21 filed Rectangular mesh split: Bilinear (fast CPU default) vs RTU… @@ -697,6 +689,12 @@

Backlog HowToLens ch4 tutorial 3: mask overlay is never actually drawn + +2026-08-03 +filed +Un-park imaging/features/scaling_relation/slam once PyAutoArray#431… + +

Epics markdown version

diff --git a/dashboard.md b/dashboard.md index ba547e35..15e2bdd5 100644 --- a/dashboard.md +++ b/dashboard.md @@ -45,7 +45,7 @@ anything you could not verify. | [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 5 | -| [Backlog](#backlog) (`draft/`) | 140 | +| [Backlog](#backlog) (`draft/`) | 139 | ## Start here @@ -149,14 +149,6 @@ anything you could not verify. **Quick wins** (small enough, and safe enough to run unattended) -
📋 Defer the eager scipy.sparse import in derivative_util (~0.10 s of import) — libraries · small · safe · normal - -``` -/start_dev draft/maintenance/libraries/defer_scipy_sparse_import.md -``` - -
-
📋 The weekly smoke run's timings land in results-* under no discoverable name — pyautoheart · small · safe · low ``` @@ -261,7 +253,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**140** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below. +**139** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below.
feature — 27 @@ -658,7 +650,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-maintenance — 25 +maintenance — 24
📋 autocti_workspace has no Navigator Check, so its CI can never roll… — ci · medium · supervised · high @@ -692,14 +684,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-
📋 Defer the eager scipy.sparse import in derivative_util (~0.10 s of import) — libraries · small · safe · normal - -``` -/start_dev draft/maintenance/libraries/defer_scipy_sparse_import.md -``` - -
-
📋 Mirror drifted library config keys into the workspace configs — workspaces · small · supervised · normal ``` @@ -1256,15 +1240,14 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | The reconstruction noise map describes a different estimator than the… | | 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | | 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the numpy-only… | -| 2026-08-22 | filed | Defer the eager scipy.sparse import in derivative_util (~0.10 s of… | | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-20 | filed | Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… | +| 2026-08-20 | filed | Numba CPU likelihood phase 1: batched MGE convolution +… |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| -| 2026-08-20 | filed | Numba CPU likelihood phase 1: batched MGE convolution +… | | 2026-08-19 | filed | status.sh --repos sources a file that no longer exists | | 2026-08-19 | filed | jax 0.11 breaks beta/gamma message log_partition under jit… | | 2026-08-19 | filed | autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on… | @@ -1274,12 +1257,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | RTD organism docs currency: Nerves page, organ-count drift, hands.md… | | 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | +| 2026-08-18 | parked | single-source-density-design |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| -| 2026-08-18 | parked | single-source-density-design | | 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | @@ -1289,12 +1272,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-09 | found | isothermal-ell-sph-oversampling-at-the-cusp | | 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | +| 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| -| 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | | 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead | | 2026-08-06 | filed | Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two… | @@ -1304,6 +1287,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | | 2026-08-04 | filed | Nightly release has been blocked 8 nights running — triage the streak | | 2026-08-04 | filed | HowToLens ch4 tutorial 3: mask overlay is never actually drawn | +| 2026-08-03 | filed | Un-park imaging/features/scaling_relation/slam once PyAutoArray#431… |
diff --git a/draft/maintenance/libraries/defer_scipy_sparse_import.md b/draft/maintenance/libraries/defer_scipy_sparse_import.md deleted file mode 100644 index 73305c40..00000000 --- a/draft/maintenance/libraries/defer_scipy_sparse_import.md +++ /dev/null @@ -1,90 +0,0 @@ -# Defer the eager scipy.sparse import in derivative_util (~0.10 s of import) - -Type: maintenance -Target: libraries -Repos: -- @PyAutoArray -Difficulty: small -Autonomy: safe -Priority: normal -Status: in-flight -Filed: 2026-08-22 (backfilled from git) - -## Where this came from - -Found 2026-08-22 while measuring the pynufft removal -(`complete/2026/08/remove-pynufft-legacy-transformer.md`). That task -assumed removing pynufft would take ~0.23 s off `import autoarray`. It takes -~10 ms. The reason is the real target: - -`autoarray/operators/derivative_util.py:30` does - -```python -from scipy.sparse import csr_matrix -``` - -at module scope. `scipy.sparse` costs **0.106 s cumulative** and is imported on -every `import autoarray`, whether or not anything touches a derivative -operator. pynufft's apparent 0.19 s was ~95 % this same shared subtree — -removing pynufft did not remove it, because `derivative_util` pulls it in -anyway. - -## Evidence (Python 3.13, dev extras, median of 7 runs) - -| | `import autoarray` | -|---|---| -| main, pynufft installed | 369.8 ms | -| pynufft removed | 359.9 ms | - -`python -X importtime` on the pynufft-removed branch still shows -`scipy.sparse` at 0.106 s cumulative. - -## Task - -Defer the `csr_matrix` import into the functions that build the sparse -operators (the same pattern `transformer.py` already uses for `nufftax` via -`_load_nufftax()`). Check for other eager `scipy.sparse` importers before -assuming this is the only one — the win only lands if *no* module-scope import -of it survives on the `import autoarray` path. - -Note the precedent in `transformer.py`: the deferral must survive unpickling in -multiprocessing workers, so hang it off the call sites rather than `__init__` -alone. - -## Acceptance - -- `python -X importtime -c "import autoarray" | grep scipy.sparse` is empty. -- Median `import autoarray` drops by ~0.10 s against the same measurement - method above (record the before/after numbers in the PR). -- Full suite green; sparse-operator behaviour unchanged. - -## Correction + result (implemented 2026-08-22, PyAutoArray#477) - -**This prompt's own premise was wrong, in exactly the way the pynufft one -was.** Deferring `derivative_util.py:30` changed nothing: `scipy.sparse` was -never imported from there. Traced with a `sys.meta_path` hook: - -``` -autoarray/__init__.py:80 - -> inversion/mesh/mesh_geometry/delaunay.py:2 import scipy.spatial - -> scipy/spatial/__init__.py:111 from ._kdtree import * - -> scipy/spatial/_kdtree.py:4 from ._ckdtree import cKDTree -``` - -`scipy.spatial` (134 ms) pulls `scipy.sparse` (154 ms) transitively, so the -`csr_matrix` import was riding on a subtree already paid for. Deferring -`scipy.spatial` as well is what removes both — and is required to satisfy this -prompt's own acceptance criterion. - -The general lesson, now hit twice: **a module's `importtime` cumulative figure -is not its exclusive cost.** Attributing a saving to a dependency requires -checking who else pulls its subtree in. - -Result: `import autoarray` **464.4 ms -> 183.7 ms** (medians of 15 runs, -Python 3.13, dev extras) — a 281 ms saving, ~2.8x this prompt's ~0.10 s -estimate, because `scipy.spatial`'s own cost comes off too. Both greps are -empty. Suites green: autoarray 1179, autogalaxy 1103/1 skipped, -autolens 532/1 skipped. - -Note `delaunay.py` already had local `import scipy.spatial` at two of its three -use sites — this deferral had been started and left half-done. diff --git a/draft/maintenance/workspaces/pynufft_removal_downstream_residue.md b/draft/maintenance/workspaces/pynufft_removal_downstream_residue.md index 69852f2e..3f71b710 100644 --- a/draft/maintenance/workspaces/pynufft_removal_downstream_residue.md +++ b/draft/maintenance/workspaces/pynufft_removal_downstream_residue.md @@ -126,8 +126,9 @@ environment still has `pynufft 2025.1.1` installed at all. coverage**, which is why the break went unnoticed. If a minimal smoke tier is added under that prompt, `dataset_setup/interferometer.py` is a strong candidate for it. -- Unrelated to `draft/maintenance/libraries/defer_scipy_sparse_import.md` (the - real ~0.10s import-time win) and to the `nufftax` dependency itself. +- Unrelated to `complete/2026/08/defer-scipy-sparse-import.md` (the real + import-time win, shipped 2026-08-22 at 281 ms) and to the `nufftax` + dependency itself. ## Phase split (decided 2026-08-23)