fix: stamp the small-datasets regime into FITS headers at the writer funnel

[Jammy2211]

Summary

Closes the write half of #153.

PYAUTO_SMALL_DATASETS=1 caps simulated datasets to a reduced resolution, but nothing on disk recorded which regime a dataset was written under. A capped dataset could survive into a later full-resolution run and be loaded silently — the root cause of autolens_workspace_test#260.

PyAutoArray#471 closed the imaging case by inferring the regime from data.fits's shape. That inference is structurally blind to interferometer datasets: their visibility count is fixed by the committed uv file while the real-space grid behind it is capped, so a capped run writes a data.fits with identical NAXIS and different values. It fails silently — no shape mismatch, no assertion. A silent wrong answer in a likelihood is worse than a loud one.

This records the regime at write time instead. A card written by the same call that writes the data is truthful by construction, needs no call-site changes, and does not depend on the data looking different — the only discriminant that can reach the interferometer case.

The issue's premise was wrong about where the funnel is. output_to_fits is the only definition of that name, but not the only write path. The real inventory is 18 library FITS writes: 2 through write_hdu_list, 14 in PyAutoGalaxy/PyAutoLens calling hdu_list.writeto directly (neither repo even imports write_hdu_list), and 4 handing the list to PyAutoFit's paths.save_fits (verified at autofit/non_linear/paths/directory.py:131 to writeto it as-is). All 18 build through hdu_list_for_output_from.

So the stamp goes in both functions, which is load-bearing rather than belt-and-braces: hdu_list_for_output_from catches the 14 + 4, write_hdu_list catches HDULists assembled elsewhere. The public API steers callers toward the bypass — hdu_list_for_output_from is re-exported as aa./ag./al.hdu_list_for_output_from, write_hdu_list by none of them. Stamping only output_to_fits, as #153 proposed, would have missed 16 of 18.

The card is written in both regimes (T capped, F full), not only when capped. Absence then means unknown — written before the stamp existed — and readers fall back to their legacy heuristic. Absence must never read as "full resolution", or every pre-stamp dataset masquerades as full and the original bug returns.

API Changes

Additive only; nothing removed, renamed, or resignatured.

Two new public helpers in autonerves.fitsable (stamp_small_datasets_regime, plus the SMALLDAT key/comment constants) and one in autonerves.test_mode (small_datasets()).

One behaviour change worth downstream attention: hdu_list_for_output_from and write_hdu_list now add a SMALLDAT card to the headers they produce, so every FITS the libraries author gains one header card. write_hdu_list mutates the primary header of the HDUList it is passed, in place.

See full details below.

Test Plan

• python -m pytest test_autonerves/ — 165 passed
• Stamp verified across all three write paths (output_to_fits, multi-HDU builder, externally-built HDUList) in both regimes
• Boolean round-trips as a genuine FITS T/F, not a string or float — pinned by test, because bool("F") is True and readers must not coerce
• Exactly one card despite both funnels stamping (assignment, not append)
• Existing header_dict cards (PIXSCAY/PIXSCAX/ORIGINY/ORIGINX) undisturbed
• Suite green and working tree clean both with PYAUTO_SMALL_DATASETS=1 exported and unset

Reviewed adversarially before opening: 6 diverse lenses + independent refuters, 18 findings, 13 refuted, and every survivor actioned. It caught a real one — the committed FITS fixtures had become regime-dependent, so running the suite under the harness default left a dirty tree. Fixed with an autouse conftest fixture rather than rewriting the 14 tracked test write-targets, which would have buried a header-card change under a test refactor.

Full API Changes (for automation & release notes)

Added

• autonerves.test_mode.small_datasets() — returns True when PYAUTO_SMALL_DATASETS == "1". Sole reader of that env var in this package.
• autonerves.fitsable.stamp_small_datasets_regime(header) — records the regime into a FITS header in place, idempotently; returns the header.
• autonerves.fitsable.SMALL_DATASETS_HEADER_KEY — "SMALLDAT". Exactly 8 characters, and that ceiling is load-bearing: a 9-character keyword is silently promoted to HIERARCH by astropy rather than raising, and header.get("SMALLDAT") then returns None, which readers treat as unknown. An over-long key would not fail loudly — it would quietly un-fix the interferometer case. Pinned by test.
• autonerves.fitsable.SMALL_DATASETS_HEADER_COMMENT — the card's comment string.

Changed Behaviour

• hdu_list_for_output_from(...) — stamps SMALLDAT into the header it builds, so every HDU it produces carries it.
• write_hdu_list(hdu_list, ...) — stamps SMALLDAT into hdu_list[0].header in place before writing. Callers holding a reference to the passed HDUList will observe the added card. Guarded against an empty list.
• Net effect: every FITS written through this module gains one header card. Verified byte-size neutral in practice — a FITS header block holds 36 cards and a real dataset header carries ~10, so file sizes are unchanged (5760 → 5760 B) and the byte-size diagnostics used in autolens_workspace_test#260 still hold. At a 36-card boundary a header would spill into a second block; no PyAuto header is near that.

Not changed

• output_to_fits is untouched — it inherits the stamp via both functions it already calls.
• The JSON (dictable.output_to_json) and CSV (csvable.output_to_csv) funnels are deliberately not stamped. Rationale and the evidence behind it are in the follow-up prompt filed in PyAutoMind; in short, of 146 non-test output_to_json call sites only 18 serialise a dataset, and the two FITS-less dataset directories are either regime-invariant or excluded from harness execution today.

Scope, stated honestly

The stamp reaches every FITS the PyAuto libraries author. It is not universal: PyAutoFit's aggregator assembles some HDULists by hand (autofit/aggregator/summary/aggregate_fits.py), and workspace scripts using raw astropy (the lenstool converter in autolens_workspace) write unstamped files. Those read as absent → unknown → the safe direction, which is exactly why absence must never mean "full".

Migration

None required. Existing FITS without the card read as "unknown regime" and consumers fall back to their prior behaviour.

Generated by the PyAutoLabs agent workflow.

---

Generated by Claude Code