fix: validate adapt-image cache against the dataset mask - #517
Merged
Conversation
The per-galaxy adapt-image cache (`files/galaxy_images_snr.fits`) lives in a directory keyed by the search identifier, which encodes the model and the search but not the dataset. A rerun whose mask changed while the model stayed identical therefore lands on the previous run's cache and silently loads adapt images sized for the old mask, surfacing much later as a bare `IndexError` deep in the inversion when they are indexed by the new mask's slim indexes. Validate the cached mask against `result.mask` on load and treat a mismatch as a cache miss, so the images are recomputed on the current mask. `result.mask` resolves to `analysis.dataset.mask` and does not rebuild the maximum log likelihood fit, so the check costs nothing the cache was saving. Also corrects the docstring, which claimed staleness was "structurally guarded" — true for model/search changes, false for dataset changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #516.
Summary
The per-galaxy adapt-image cache (
files/galaxy_images_snr.fits) lives in a directory keyed by thesearch identifier, which encodes the model and the search but not the dataset. A rerun whose mask
changed while the model stayed identical therefore lands on the previous run's output directory and loads
its cache — adapt images sized for the old mask — silently. The mismatch surfaces much later as a bare
IndexErrordeep in the inversion.This validates the cached mask against
result.maskon load and treats a mismatch as a cache miss, so theimages are recomputed on the current mask.
Why the reported symptom looked like something else
Filed from the 2026-07-21 census as an off-by-one in PyAutoArray
mapper_util.adaptive_pixel_signals_from(IndexError: index 177 is out of bounds for axis 0 with size 177;index == size). That diagnosis was wrong —
adaptive_pixel_signals_fromis correct, and the bug is notmulti-plane / double-Einstein-ring specific.
PyAutoArray
656be94b(#396, merged 2026-07-19) changed thePYAUTO_SMALL_DATASETScap from 15x15 to16x16. The census ran against an
output/holding pre-#396 caches: cached adapt image 15x15 = 177pixels, current dataset mask 16x16 = 208 pixels. Slim indexes run to 207 against 177-length data, so the
first out-of-range index is 177 — which merely looks like an off-by-one.
Reproduction matrix (
double_einstein_ring/slam.py,PYAUTO_TEST_MODE=2):output/statemasks equal: True, 2828/2828)PYAUTO_SMALL_DATASETS=1(16x16)Verification
End-to-end, by planting a genuine stale cache (loose
files/and the archived copy inside each search's.zip, whichrestore()re-extracts):main: reproduces the census failure exactly —IndexError: index 177 is out of bounds for axis 0 with size 177, same frame, same numbers.source_pix[2]with the likelihood actuallyevaluated.
pytest test_autogalaxy— 1006 passed.Known limitation
A stale cache is not repaired on disk. The recomputed images are written to the loose
files/folder, butPaths.preserve_in_ziponly adds a member absent from the zip and never replaces one, so the nextrestore()re-extracts the stale copy. Such a search misses the cache on every run rather than once —correct, but without the caching win until its output is cleared. Filed separately as PyAutoFit#1414; the
docstring records it.
API Changes
None.
_galaxy_image_dict_from_cacheis private and gains an optionalmaskargument that defaults toNone(no validation), so existing callers are unaffected. No public signature, name or behaviour changesfor correctly-cached runs — the only behavioural difference is that a previously-silent stale load now
recomputes.
🤖 Generated with Claude Code