Skip to content

fix: validate adapt-image cache against the dataset mask - #517

Merged
Jammy2211 merged 1 commit into
mainfrom
feature/adapt-image-cache-mask-validation
Jul 22, 2026
Merged

fix: validate adapt-image cache against the dataset mask#517
Jammy2211 merged 1 commit into
mainfrom
feature/adapt-image-cache-mask-validation

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Closes #516.

Summary

The per-galaxy adapt-image cache (files/galaxy_images_snr.fits) lives in a directory keyed by the
search identifier, which encodes the model and the search but not the dataset. A rerun whose mask
changed while the model stayed identical therefore lands on the previous run's output directory and loads
its cache — adapt images sized for the old mask — silently. The mismatch surfaces much later as a bare
IndexError deep in the inversion.

This validates the cached mask against result.mask on load and treats a mismatch as a cache miss, so the
images are recomputed on the current mask.

Why the reported symptom looked like something else

Filed from the 2026-07-21 census as an off-by-one in PyAutoArray
mapper_util.adaptive_pixel_signals_from (IndexError: index 177 is out of bounds for axis 0 with size 177;
index == size). That diagnosis was wrong — adaptive_pixel_signals_from is correct, and the bug is not
multi-plane / double-Einstein-ring specific.

PyAutoArray 656be94b (#396, merged 2026-07-19) changed the PYAUTO_SMALL_DATASETS cap from 15x15 to
16x16. The census ran against an output/ holding pre-#396 caches: cached adapt image 15x15 = 177
pixels, current dataset mask 16x16 = 208 pixels. Slim indexes run to 207 against 177-length data, so the
first out-of-range index is 177 — which merely looks like an off-by-one.

Reproduction matrix (double_einstein_ring/slam.py, PYAUTO_TEST_MODE=2):

Data size output/ state Result
full (100x100) cleared PASS (masks equal: True, 2828/2828)
PYAUTO_SMALL_DATASETS=1 (16x16) cleared PASS (208/208)
either pre-#396 caches present IndexError: index 177 out of bounds for size 177

Verification

End-to-end, by planting a genuine stale cache (loose files/ and the archived copy inside each search's
.zip, which restore() re-extracts):

  • against stock main: reproduces the census failure exactly — IndexError: index 177 is out of bounds for axis 0 with size 177, same frame, same numbers.
  • against this branch: exit 0, the full pipeline runs through source_pix[2] with the likelihood actually
    evaluated.

pytest test_autogalaxy1006 passed.

Known limitation

A stale cache is not repaired on disk. The recomputed images are written to the loose files/ folder, but
Paths.preserve_in_zip only adds a member absent from the zip and never replaces one, so the next
restore() re-extracts the stale copy. Such a search misses the cache on every run rather than once —
correct, but without the caching win until its output is cleared. Filed separately as PyAutoFit#1414; the
docstring records it.

API Changes

None. _galaxy_image_dict_from_cache is private and gains an optional mask argument that defaults to
None (no validation), so existing callers are unaffected. No public signature, name or behaviour changes
for correctly-cached runs — the only behavioural difference is that a previously-silent stale load now
recomputes.

🤖 Generated with Claude Code

The per-galaxy adapt-image cache (`files/galaxy_images_snr.fits`) lives in a
directory keyed by the search identifier, which encodes the model and the search
but not the dataset. A rerun whose mask changed while the model stayed identical
therefore lands on the previous run's cache and silently loads adapt images sized
for the old mask, surfacing much later as a bare `IndexError` deep in the
inversion when they are indexed by the new mask's slim indexes.

Validate the cached mask against `result.mask` on load and treat a mismatch as a
cache miss, so the images are recomputed on the current mask. `result.mask`
resolves to `analysis.dataset.mask` and does not rebuild the maximum log
likelihood fit, so the check costs nothing the cache was saving.

Also corrects the docstring, which claimed staleness was "structurally guarded" —
true for model/search changes, false for dataset changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release PR queued for the next release build label Jul 22, 2026
@Jammy2211
Jammy2211 merged commit 64790e7 into main Jul 22, 2026
5 checks passed
@Jammy2211
Jammy2211 deleted the feature/adapt-image-cache-mask-validation branch July 22, 2026 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: validate adapt-image cache against dataset mask (stale cache IndexError)

1 participant