Skip to content

Commit 5537884

Browse files
claudeclaude
committed
Merge branch 'claude/small-datasets-regime-stamp-s3i9o7'
Mind state for the small-datasets regime stamp and its three follow-ups, all of which shipped and merged as code PRs: PyAutoNerves#154 stamp the regime at the FITS writer funnel PyAutoArray#474 prefer the stamp over the shape heuristic PyAutoArray#476 reuse an already-correctly-capped dataset PyAutoGalaxy#584 remove the duplicate plot_utils module PyAutoNerves#155 stop writing the literal [''] header comment Brings across four completion records, the deferred point-source JSON follow-up, the post-release autoarray floor bump, and the parked PyAutoNerves release with the evidence that it cannot be gated from a cloud session. # Conflicts: # complete/index.md # dashboard.html # dashboard.md
2 parents a21b4e1 + a7e5091 commit 5537884

11 files changed

Lines changed: 763 additions & 131 deletions

‎complete/2026/08/fits-header-comment-literal-list.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
- issue: none — shipped directly as a PR (small, single-repo, cosmetic)
2+
- completed: 2026-08-22
3+
- library-pr: https://github.com/PyAutoLabs/PyAutoNerves/pull/155 (merged 0ecefa0)
4+
- workspace-pr: none — no workspace change needed
5+
6+
`hdu_list_for_output_from` passed a LIST, `[""]`, as the FITS card comment.
7+
astropy does not reject it — it `str()`s it — so every card written from a
8+
`header_dict` landed on disk reading `PIXSCAY = 0.1 / ['']`. Four such cards on
9+
every `Imaging` dataset the stack writes, rendering for anyone opening a PyAuto
10+
FITS in DS9, astropy or any external tool. Found while implementing the regime
11+
stamp (`complete/2026/08/small-datasets-regime-stamp.md`) and deliberately kept
12+
out of that change so a cosmetic fix did not ride a behavioural one.
13+
14+
Fixed by passing `""`. Verified on disk: the card becomes a plain
15+
`PIXSCAY = 0.1` — no comment, no trailing slash, byte-size unchanged. Both `""`
16+
and dropping the third tuple element produce byte-identical output; `""` was
17+
chosen to keep the intent explicit and the diff minimal.
18+
19+
**DECISION taken explicitly, not by omission.** The prompt raised adding REAL
20+
per-key comments, since `PIXSCAY`/`ORIGINY` are not self-describing. Declined:
21+
autonerves receives an opaque `header_dict` and does not know the key vocabulary
22+
— those are autoarray's `Mask2DKeys`. Hardcoding their meanings in the base
23+
serialization layer couples it to a downstream key set, the wrong direction for
24+
the dependency. If descriptive comments are wanted the CALLER should supply them,
25+
which is a separate API change to `header_dict`'s shape.
26+
27+
**TRAPS**
28+
- astropy accepts a non-string comment silently and `str()`s it. It will not warn
29+
you that you passed the wrong type; the evidence is only visible on disk. Check
30+
rendered cards, not just `header[key]`.
31+
- The `except ValueError: float(value)` fallback beside it is unrelated and was
32+
left alone — bool never raises there, so it is unreachable for the stamp.
33+
- `test_autonerves/files/array_out.fits` is a tracked test WRITE TARGET, so its
34+
bytes change with the comment and it must be refreshed in the same commit.
35+
Verified byte-stable across repeated runs, and identical with
36+
PYAUTO_SMALL_DATASETS exported AND unset — the autouse conftest fixture from
37+
#154 is what makes that deterministic.
38+
39+
Tests: 166 passed, green both env states. CI green on 3.12 / 3.13 / nojax.
40+
41+
**Gate note.** Heart was not consulted — no PyAutoHeart checkout in this
42+
web-github session; the per-repo suite fallback was used and CI agreed.
43+
44+
**Completes the three follow-ups** filed off the regime-stamp task, with
45+
`complete/2026/08/should-simulate-capped-branch-reuse.md` and
46+
`complete/2026/08/plot-utils-duplicate-modules.md`.
47+
48+
**STILL OPEN, and the stamp is NOT LIVE until it is done:** PyAutoNerves needs a
49+
release. `autoarray/pyproject.toml` floors `autonerves>=2026.8.22.1`, which was
50+
the newest release on PyPI and predates the stamp, so an installed-from-PyPI
51+
autoarray sees no card and falls back to the shape heuristic. Also still unrun:
52+
the workspace smoke suite, which has never exercised any of this.
53+
54+
## Original prompt
55+
56+
# Every header_dict FITS card carries the literal comment text ['']
57+
58+
Type: bug
59+
Target: pyautonerves
60+
Repos:
61+
- @PyAutoNerves
62+
Difficulty: small
63+
Autonomy: supervised
64+
Priority: low
65+
Status: formalised
66+
67+
Found 2026-08-22 while implementing the regime stamp (PyAutoNerves#153,
68+
`complete/2026/08/small-datasets-regime-stamp.md`). Pre-existing, cosmetic, and
69+
deliberately left out of that change so a header-card fix did not ride along with
70+
a behavioural one.
71+
72+
`autonerves/fitsable.py` (lines 138 and 140 on current main) passes a **list** as
73+
the FITS comment:
74+
75+
```python
76+
header.append((key_str, value, [""]))
77+
except ValueError:
78+
header.append((key_str, float(value), [""]))
79+
```
80+
81+
astropy does not reject it — it `str()`s the list. So every card written from a
82+
`header_dict` lands on disk carrying the literal three-character comment `['']`:
83+
84+
```
85+
PIXSCAY = 0.1 / ['']
86+
PIXSCAX = 0.1 / ['']
87+
ORIGINY = 0.0 / ['']
88+
ORIGINX = 0.0 / ['']
89+
```
90+
91+
Verified empirically with astropy 8.0.1. Every `Imaging` dataset the stack writes
92+
has this on four cards.
93+
94+
## Impact
95+
96+
Cosmetic only. Nothing reads comments — the FITS header consumers across
97+
PyAutoArray, PyAutoGalaxy, PyAutoLens and the workspaces all index values by key
98+
name, verified during PyAutoNerves#153. It is visible to anyone opening a PyAuto
99+
FITS in DS9, `astropy`, or any external tool, which is the argument for fixing it:
100+
it reads as a serialization bug in output we hand to other people.
101+
102+
The intent was presumably an empty comment. `""` gives that; the list gives the
103+
rendered repr of a list.
104+
105+
## Suggested scope
106+
107+
1. Replace `[""]` with `""` at both sites, or drop the third tuple element.
108+
2. Consider whether `header_dict` should carry real comments — the four keys it
109+
writes (`PIXSCAY`/`PIXSCAX`/`ORIGINY`/`ORIGINX`) are not self-describing to an
110+
outside reader, and the slot is already there.
111+
3. Note this changes the bytes of every FITS the stack writes, exactly as the
112+
regime stamp did. The same finding applies: no hash, golden-file or checksum
113+
pin exists over any `.fits` in the stack, and the change is byte-size neutral
114+
at current header sizes. But the tracked test fixtures that the suites rewrite
115+
will need refreshing, and the autouse conftest fixtures added in
116+
PyAutoNerves#154 / PyAutoArray#474 keep that deterministic.

‎complete/2026/08/plot-utils-duplicate-modules.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
- issue: none — shipped directly as a PR (small, single-repo, behaviour-preserving)
2+
- completed: 2026-08-22
3+
- library-pr: https://github.com/PyAutoLabs/PyAutoGalaxy/pull/584 (merged 506f153)
4+
- workspace-pr: none — no workspace change needed
5+
6+
`autogalaxy/util/plot_utils.py` and `autogalaxy/plot/plot_utils.py` were
7+
byte-identical (md5 `a6eed88a228524d55403418fdf01d32a`), both defining the plot
8+
helpers and `fits_array`. Found while auditing every FITS write path in the stack
9+
for PyAutoNerves#153 (`complete/2026/08/small-datasets-regime-stamp.md`).
10+
11+
`util/` is the real home: `plot/__init__.py:9` imports the public
12+
`plot_array`/`plot_grid`/`fits_array` from it, eight internal modules import it,
13+
PyAutoLens references it thirteen times. The duplicate was deleted.
14+
15+
**Why it was worth doing rather than tolerating.** Both copies sat on a FITS write
16+
path. A change applied to one and not the other splits behaviour between callers
17+
with nothing to catch it — separate modules, so no test compares them and no
18+
import fails. That is a silent-divergence trap, not just untidiness.
19+
20+
**THE METHOD LESSON, and the reason this nearly shipped broken.** The importer
21+
search used a LINE-ANCHORED grep (`^from autogalaxy.plot.plot_utils`) and returned
22+
ZERO across every repo. The deletion looked provably safe. It was not:
23+
`test_autogalaxy/plot/mat_wrap/test_visuals.py:41` has an INDENTED, FUNCTION-LOCAL
24+
import, which an anchored pattern silently skips. The test suite caught it
25+
immediately (passed on clean main, `ModuleNotFoundError` with the change).
26+
27+
**When removing a module, search UNANCHORED.** Python's function-local imports do
28+
not sit at column zero, and a `^`-anchored grep reports a false negative that reads
29+
exactly like proof of safety. The unanchored search then found exactly one
30+
reference, which is what shipped.
31+
32+
**TRAPS**
33+
- `^from X` / `^import X` greps miss function-local, class-body and conditional
34+
imports. Never treat an anchored search as proof a module is dead.
35+
- Verify behaviour-preservation with an IDENTICAL test COUNT against clean main,
36+
not just "the suite passes" — a deleted or uncollected test also passes.
37+
- A dead-module deletion leaves dangling Sphinx cross-references. `galaxies_plots.py`
38+
pointed `:func:`~autogalaxy.plot.plot_utils._critical_curves_from`` at the deleted
39+
module; re-pointed. PyAutoGalaxy's CI has a `docs / docs-build` job that the other
40+
library repos lack, and it is the check that matters for this class of change.
41+
42+
**Debris found and removed.** A stray `/btw ok` line was sitting inside the
43+
`_critical_curves_from` docstring, having arrived in `3ca31bf` (#582) and been
44+
inherited by both copies. It renders into the API docs. Treated as committed junk,
45+
not as an instruction. Worth a wider sweep: a stray keystroke reached a merged PR
46+
once and may have siblings elsewhere.
47+
48+
Tests: 1099 passed / 5 skipped, an IDENTICAL count to clean main. CI green on
49+
3.12 / 3.13 / nojax / docs-build.
50+
51+
**Gate note.** Heart was not consulted — no PyAutoHeart checkout in this
52+
web-github session; the documented per-repo suite fallback was used, and CI agreed.
53+
54+
## Original prompt
55+
56+
# autogalaxy has two byte-identical plot_utils modules
57+
58+
Type: refactor
59+
Target: autogalaxy
60+
Repos:
61+
- @PyAutoGalaxy
62+
Difficulty: small
63+
Autonomy: safe
64+
Priority: low
65+
Status: formalised
66+
67+
Found 2026-08-22 while auditing every FITS write path in the stack for
68+
PyAutoNerves#153 (`complete/2026/08/small-datasets-regime-stamp.md`); unrelated to
69+
that change and deliberately not absorbed into it.
70+
71+
`autogalaxy/util/plot_utils.py` and `autogalaxy/plot/plot_utils.py` are
72+
**byte-identical** — same md5 (`a6eed88a228524d55403418fdf01d32a`). Both define
73+
`fits_array`, which routes to `output_to_fits`.
74+
75+
Import counts inside `autogalaxy/`:
76+
77+
- `util.plot_utils` — 8 importers
78+
- `plot.plot_utils` — 1 importer
79+
80+
So the `util` one is the de-facto home and `plot` one is near-dead, but neither
81+
is unused, which is why this needs a real check rather than a delete.
82+
83+
## Why it matters slightly more than ordinary duplication
84+
85+
Both modules are on a FITS write path. A future change to how figures are written
86+
— the regime stamp landed on exactly this surface — applied to one copy and not
87+
the other produces a silent behavioural split between callers, with nothing to
88+
catch it: they are separate modules, so no test compares them and no import fails.
89+
90+
## Suggested scope
91+
92+
1. Confirm the two files are still identical (they may have diverged since
93+
2026-08-22 — if they have, that divergence is itself the finding and this
94+
prompt should be rewritten around it).
95+
2. Check for external importers outside `autogalaxy/` before deleting either —
96+
PyAutoLens, the workspaces and any notebook may reach in.
97+
3. Keep `util/plot_utils.py`, re-point the single `plot.plot_utils` importer, and
98+
delete the duplicate. Or, if `plot/` is the intended long-term home, do the
99+
reverse — but pick one deliberately rather than by import count.
100+
4. Behaviour-preserving: no output should change.

‎complete/2026/08/should-simulate-capped-branch-reuse.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
- issue: none — shipped directly as a PR (small, single-repo follow-up of the closed PyAutoNerves#153)
2+
- completed: 2026-08-22
3+
- library-pr: https://github.com/PyAutoLabs/PyAutoArray/pull/476 (merged dc0c273)
4+
- workspace-pr: none — no workspace change needed
5+
6+
`should_simulate`'s `PYAUTO_SMALL_DATASETS=1` branch deleted and re-simulated
7+
**every** dataset unconditionally, paying a full simulation pass per dataset per
8+
smoke run across ~253 call sites for datasets that were already correct. That was
9+
correct when written and the docstring said why: it "cannot know the capped
10+
dataset on disk was produced by the SAME cap". The `SMALLDAT` stamp
11+
(`complete/2026/08/small-datasets-regime-stamp.md`) removed that limitation.
12+
13+
**THE FIX IS NOT THE ONE-LINER THE PARENT RECORD PREDICTED.** That record — and
14+
the filed prompt's parent framing — said `if stamp is not True:` was a cheap fix.
15+
It is not, and shipping it would have been worse than doing nothing.
16+
17+
`SMALLDAT = T` means "capped at whatever `SMALL_DATASETS_SHAPE_NATIVE` was when
18+
this file was written", NOT "capped at today's cap". If that constant ever
19+
changes, every dataset on disk goes on claiming `T` at the old size, and reusing
20+
on the stamp alone silently feeds stale wrong-sized data to a run that asked for
21+
the new cap — the exact silent-stale-dataset bug the stamp exists to prevent,
22+
reintroduced through the opposite branch.
23+
24+
Reuse therefore requires BOTH: stamped `T` **and** shape `== SMALL_DATASETS_SHAPE_NATIVE`
25+
(`_is_capped_at_the_current_cap`).
26+
27+
**This is the mirror image of `_stamp_contradicted_by_shape`** on the
28+
full-resolution branch, and exists for the same reason, which is the durable
29+
lesson from this pair of tasks: **the stamp records the writer's ENVIRONMENT, not
30+
a measured property of the data.** Neither branch may treat it as unfalsifiable.
31+
Any future consumer of `SMALLDAT` must corroborate it before acting destructively
32+
or before skipping work on its authority.
33+
34+
**TRAPS**
35+
- Reuse on the stamp alone is the bug. Both halves are load-bearing.
36+
- Interferometer datasets deliberately NEVER qualify: `data.fits` is
37+
`(n_visibilities, 2)`, its shape fixed by the committed uv file and unchanged by
38+
the cap, so shape cannot corroborate the stamp. Trusting the stamp alone for
39+
precisely the family whose corruption is invisible is the wrong trade. Written
40+
into the docstring, not left to fall out of the code.
41+
- Anything with no readable top-level `data.fits` (JSON-only, datacubes nesting
42+
theirs in `channel_XXX/`, multi_dataset's prefixed names) fails the check and
43+
regenerates, preserving prior behaviour for the families this cannot speak about.
44+
- The pre-existing test asserting unconditional deletion ENCODED the limitation
45+
being removed. It had to be rewritten, not deleted — a test that fails because
46+
the limitation it documents is gone is a signal, not an obstacle.
47+
48+
**Behaviour: exactly one row changes.** A dataset capped at the current cap is
49+
reused; different-cap-stamped, unstamped legacy, full-resolution, interferometer
50+
and no-`data.fits` all regenerate as before. Reuse requires positive evidence and
51+
everything else fails to provide it.
52+
53+
Tests: 1106 passed / 0 failed, green with `PYAUTO_SMALL_DATASETS=1` exported AND
54+
unset, tree clean both ways. CI green on 3.12 / 3.13 / nojax.
55+
56+
**Gate note.** Heart was not consulted — no PyAutoHeart checkout in this
57+
web-github session; the documented per-repo suite fallback was used, and CI
58+
subsequently agreed.
59+
60+
## Original prompt
61+
62+
# should_simulate's capped branch re-simulates every dataset, ignoring the stamp it now has
63+
64+
Type: maintenance
65+
Target: libraries
66+
Repos:
67+
- @PyAutoArray
68+
Difficulty: small
69+
Autonomy: supervised
70+
Priority: medium
71+
Status: formalised
72+
73+
Split out of PyAutoNerves#153 on 2026-08-22 (`complete/2026/08/small-datasets-regime-stamp.md`),
74+
which added the `SMALLDAT` regime stamp and deliberately did not touch this branch.
75+
76+
`autoarray/util/dataset_util.py should_simulate` still does this when
77+
`PYAUTO_SMALL_DATASETS=1`:
78+
79+
```python
80+
if os.environ.get("PYAUTO_SMALL_DATASETS") == "1":
81+
if Path(dataset_path).exists():
82+
shutil.rmtree(dataset_path)
83+
return not Path(dataset_path).exists()
84+
```
85+
86+
Unconditional. Every smoke run deletes and re-simulates **every** dataset, even
87+
one already written by a capped run at the same cap.
88+
89+
That was correct when written — the docstring says so explicitly: *"The small
90+
path is unconditional by design: it cannot know the capped dataset on disk was
91+
produced by the SAME cap, so it always regenerates."* The stamp removes exactly
92+
that limitation. `SMALLDAT = T` now says the writer capped it.
93+
94+
`if stamp is not True:` is the shape of the fix. The saving is one full
95+
simulation pass per dataset per smoke run, across ~253 `should_simulate` call
96+
sites in autolens_workspace.
97+
98+
## The trap that makes this not a one-liner
99+
100+
**A stamp of `T` does not mean "capped at the cap size in force now."** It means
101+
"capped at whatever `SMALL_DATASETS_SHAPE_NATIVE` was when it was written." If
102+
that constant ever changes, every dataset on disk still claims `T` while being
103+
the wrong size, and skipping regeneration would silently reuse it — the same
104+
class of silent-stale-dataset bug the stamp exists to prevent, reintroduced
105+
through the other branch.
106+
107+
So the reuse condition is not `stamp is True` alone. It needs the on-disk shape
108+
to also match the *current* cap, which `_on_disk_shape_native` already provides
109+
and `_is_small_datasets_on_disk` already compares with `== SMALL_DATASETS_SHAPE_NATIVE`.
110+
Reuse only when the stamp says capped **and** the shape matches today's cap;
111+
anything else regenerates.
112+
113+
Note this is the mirror image of `_stamp_contradicted_by_shape` on the full-regime
114+
branch, and for the same reason: the stamp records the writer's environment, not
115+
a property of the data. Neither branch should treat it as unfalsifiable.
116+
117+
Interferometer datasets are shape-invariant under the cap, so a shape check
118+
cannot corroborate them. Decide explicitly whether they reuse on the stamp alone
119+
or always regenerate — do not leave it to fall out of the code.
120+
121+
## Suggested scope
122+
123+
1. Reuse a capped dataset only when the stamp says `T` **and** the shape matches
124+
the current cap. Everything else regenerates, as today.
125+
2. Take the interferometer decision explicitly and write it in the docstring.
126+
3. Test both directions: same-cap dataset is reused; a dataset stamped `T` at a
127+
different shape is regenerated.
128+
4. Correct the docstring paragraph that says the small path cannot know.

0 commit comments

Comments
 (0)