From 2e69416b11db101b4aa27abac43a774e35df1fd0 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 16:02:11 +0000 Subject: [PATCH 1/3] prompt: retire interferometer OOM prompt, issue JAX traceback harness fix (#186) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The OOM in draft/bug/autolens/interferometer_release_leg_oom.md was already root-caused and shipped before that prompt's own 2026-08-04 amendment was written: MultiStartProdigy ran 48 starts as one unbatched vmap (~86 GB in a single allocation), autolens_workspace#450 set batch_size=4, and Release Integrate run 30901054267 discharged the leg on 2026-08-04 (complete/2026/08/interferometer-start-here-integrate-oom.md). Re-verified before retiring: autolens_workspace and autogalaxy_workspace both carry batch_size=4 on the 48-start MultiStartProdigy in scripts/interferometer/start_here.py, and the interferometer shard is green in the two most recent nightly Release Integrate runs (32804373015 on 08-25, 33073386315 on 08-27). The prompt's NUFFT/transformer hypothesis was wrong. Retired to complete/archive/shelved/ with a superseded-by note rather than issued. The one item of the amendment that was never done — JAX_TRACEBACK_FILTERING=off in the release harness, still absent from PyAutoHeart at d576003 — is split out as its own task and issued as PyAutoLabs/PyAutoHeart#186. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SFoQCsxRTGjKk2YGe4R9Rs --- active.md | 12 +++ ...jax_traceback_filtering_release_harness.md | 89 +++++++++++++++++++ .../interferometer_release_leg_oom.md | 25 ++++++ dashboard.html | 22 ++--- dashboard.md | 37 ++++---- 5 files changed, 155 insertions(+), 30 deletions(-) create mode 100644 active/jax_traceback_filtering_release_harness.md rename {draft/bug/autolens => complete/archive/shelved}/interferometer_release_leg_oom.md (71%) diff --git a/active.md b/active.md index fb25a165..7b5df9f4 100644 --- a/active.md +++ b/active.md @@ -13,3 +13,15 @@ - registered: 2026-08-19 by the wake_up session — the issuing session (claude/autofit-priors-messages-audit-ylvenv) filed the prompt + issue but not this entry, tripping Lifecycle Drift on main. - repos-none-claimed: this entry claims NO repos — one line deliberately, not 2-space bullets. + +## jax-traceback-filtering-release-harness +- issue: https://github.com/PyAutoLabs/PyAutoHeart/issues/186 (issued 2026-08-27) +- issued: 2026-08-27 +- prompt: active/jax_traceback_filtering_release_harness.md +- status: library-dev (organ repo — PyAutoHeart CI workflow, not a PyAuto library or workspace) +- worktree: ~/Code/PyAutoLabs-wt/jax-traceback-filtering-release-harness +- session: web session https://claude.ai/code/session_01SFoQCsxRTGjKk2YGe4R9Rs (not `claude --resume`-able) +- origin: split out of the retired prompt `complete/archive/shelved/interferometer_release_leg_oom.md` + by /start_dev on 2026-08-27, when that prompt's headline OOM was found already shipped + (`complete/2026/08/interferometer-start-here-integrate-oom.md`). +- repos: diff --git a/active/jax_traceback_filtering_release_harness.md b/active/jax_traceback_filtering_release_harness.md new file mode 100644 index 00000000..f4d5f34a --- /dev/null +++ b/active/jax_traceback_filtering_release_harness.md @@ -0,0 +1,89 @@ +# Release/smoke script harness filters JAX tracebacks away, hiding the real exception + +Type: bug +Target: ci +Repos: +- @PyAutoHeart +Difficulty: small +Autonomy: safe +Priority: medium +Status: formalised +Filed: 2026-08-27 +Issued: 2026-08-27 + +## Symptom + +When a workspace script fails under JAX inside PyAutoHeart's script harness +(`.github/workflows/workspace-validation.yml`, job `run_scripts`, both the +`smoke` and `release` legs), JAX's default traceback filter removes its own +internal frames *and*, in practice, the exception that matters: + +``` +scripts/interferometer/start_here.py ... FAIL (19.5s) +For simplicity, JAX has removed its internal frames from the traceback of the +following exception. Set JAX_TRACEBACK_FILTERING=off to include these. +``` + +That is the whole failure record. The reader is told which script failed and +nothing about why. + +## Why it matters — the cost is measured, not hypothetical + +PyAutoHeart run 30516167217 (2026-07-30) failed both interferometer shards +(`autogalaxy` and `autolens`) with exactly the message above. Because the real +exception was filtered away, that night read as an unidentified failure. The +same defect surfaced its true cause only the following night, run +30607596240 (2026-07-31), where the message survived intact: + +``` +jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes. +``` + +One night was spent identifying a failure the harness already had the +information to name. The underlying OOM was root-caused and fixed the same day +(`autolens_workspace#450`; record: +`complete/2026/08/interferometer-start-here-integrate-oom.md`) — this task is +about the *harness*, which still filters, and will do the same to the next +JAX failure. + +A gate whose error messages are filtered away is the same category of problem +as the exit-code contract fixed in PyAutoBrain#196: the signal exists, the +harness discards it before a human sees it. + +## Current state (verified 2026-08-27) + +`JAX_TRACEBACK_FILTERING` appears nowhere in PyAutoHeart: + +``` +$ grep -rn "JAX_TRACEBACK_FILTERING" . # PyAutoHeart @ d576003 — no matches +``` + +Nor in the workspaces' `config/build/profile_release.yaml` / +`profile_smoke.yaml` defaults, both of which already pin every JAX-adjacent var +they care about (`JAX_ENABLE_X64`, `PYAUTO_DISABLE_JAX`, …) explicitly. + +## Proposed fix + +Set `JAX_TRACEBACK_FILTERING: "off"` once, in the workflow-level `env:` block of +`PyAutoHeart/.github/workflows/workspace-validation.yml` — the block that +already carries `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK`. One place covers both +the `smoke` and `release` legs and every workspace in the matrix, and +`run_python.py`'s per-script env profiles inherit it (a profile only *sets* the +keys it is given; it never clears inherited vars). + +Consider whether the same belongs in the other script-running workflows +(`smoke-tests.yml`, `workspace-smoke.yml`) for the same reason. + +## Exit criteria + +- A JAX exception raised inside a workspace script under `run_scripts` reports + its own exception type and message in the job log, not the + "JAX has removed its internal frames" placeholder. +- Verified against a real (or deliberately induced) JAX failure in a CI run, + not only by reading the diff. + +## Origin + +Split out of `draft/bug/autolens/interferometer_release_leg_oom.md` (its +2026-08-04 amendment, consequence 2) when that prompt was retired as superseded +— the OOM it described had already shipped, this harness gap had not. diff --git a/draft/bug/autolens/interferometer_release_leg_oom.md b/complete/archive/shelved/interferometer_release_leg_oom.md similarity index 71% rename from draft/bug/autolens/interferometer_release_leg_oom.md rename to complete/archive/shelved/interferometer_release_leg_oom.md index 17e5d642..da630395 100644 --- a/draft/bug/autolens/interferometer_release_leg_oom.md +++ b/complete/archive/shelved/interferometer_release_leg_oom.md @@ -1,3 +1,28 @@ +> **RETIRED 2026-08-27 — superseded, not actioned as written.** +> +> The OOM this prompt describes was root-caused and fixed before the prompt's own +> amendment was written. See `complete/2026/08/interferometer-start-here-integrate-oom.md`: +> `MultiStartProdigy` ran 48 starts as a single unbatched `vmap` (~86 GB in one +> allocation); `autolens_workspace#450` set `batch_size=4` and the leg was +> discharged by Release Integrate run 30901054267 on 2026-08-04 +> (`scripts/interferometer/start_here.py ... PASS (173.5s)`). +> +> Re-verified at retirement (2026-08-27): +> - `autolens_workspace` and `autogalaxy_workspace` **both** carry `batch_size=4` +> on the 48-start `MultiStartProdigy` in `scripts/interferometer/start_here.py`, +> so the amendment's consequence 1 ("it is not autolens-specific") is closed too. +> - The interferometer shard is green in the two most recent nightly Release +> Integrate runs — 32804373015 (2026-08-25, `autogalaxy`) and 33073386315 +> (2026-08-27, `autolens`). No recurrence. +> - The NUFFT/transformer hypothesis in "Notes for triage" was wrong: the +> allocation was the multi-start batch, not a dense transformer matrix. +> +> The amendment's consequence 2 — `JAX_TRACEBACK_FILTERING=off` in the release +> harness — was **not** done and is still open. It was split out as its own task: +> `draft/bug/ci/jax_traceback_filtering_release_harness.md`. + +--- + # interferometer/start_here.py OOM in nightly release-validation integrate leg Filed: 2026-07-31 (backfilled from git) diff --git a/dashboard.html b/dashboard.html index 85f95f1a..131ce3fc 100644 --- a/dashboard.html +++ b/dashboard.html @@ -204,7 +204,7 @@

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

- +

Last updated 2026-08-27. This page is generated from active/, draft/ and the registry files, so it is only as current as they are. dashboard_refresh.yml re-renders it on every push to main — that heals a stale page, but not a stale prompt: a task that shipped without its prompt advancing to complete/ keeps rendering here as pickable backlog. Reconciling those is the refresh below.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

+

Release/smoke script harness filters JAX tracebacks away, hiding the real exception — issue #186 — issued 2026-08-27 — library-dev (organ repo — PyAutoHeart CI workflow, not a PyAuto library or workspace)

Parked markdown version

3 task(s) @@ -269,7 +270,7 @@

Planned

latent-nan-guard-honest-run — planned 2026-07-22

Backlog markdown version

-

136 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

+

135 filed prompts, not started — sorted most-pickable first (priority, then size). 25 of them belong to an epic and are listed only under Epics below.

feature — 28

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix caching✨ featureautoarraymediumsupervisedhigh

@@ -344,7 +345,7 @@

Backlog

Regenerate setup_notebook-drifted notebooks in autogalaxy/autofit/HowToFit workspaces🧹 maintenanceworkspacessmallsupervisedlow

-bug — 17 +bug — 16

LensCalc NumPy Hessian step is too coarse for multi-plane tracers🐛 bugautogalaxylargesupervisedhigh

Fix release JAX runtime compatibility and likelihood parity🐛 bughealth_fixestoo-largesupervisedhigh

Fix JIT quick-update visualization output regressions🐛 bughealth_fixestoo-largesupervisedhigh

@@ -361,7 +362,6 @@

Backlog

Priors & Messages cleanup — tracker🐛 bugpriorstoo-largesupervisednormal

The reconstruction noise map describes a different estimator than the default solver🐛 bugautoarraymediumhuman-requiredlow

Point-source JSON datasets record no resolution regime🐛 bugpyautolensmediumsupervisedlow

-

interferometer/start_here.py OOM in nightly release-validation integrate leg🐛 bugautolens

docs — 12 @@ -413,6 +413,12 @@

Backlog 2026-08-27 +issued +Release/smoke script harness filters JAX tracebacks away, hiding the… + + + +2026-08-27 filed One construction path for plane-bound lensing quantities @@ -465,7 +471,7 @@

Backlog autocti_workspace has no Navigator Check, so its CI can never roll… - + 2026-08-24 filed Un-park multi_galaxy/features/scaling_relation/slam once a capped run… @@ -705,12 +711,6 @@

Backlog HowToLens ch4 tutorial 3: mask overlay is never actually drawn - -2026-08-03 -filed -Un-park imaging/features/scaling_relation/slam — the PyAutoArray#431… - -

Epics markdown version

diff --git a/dashboard.md b/dashboard.md index cc89c9a0..f0adc670 100644 --- a/dashboard.md +++ b/dashboard.md @@ -42,10 +42,10 @@ anything you could not verify. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 1 | +| [In flight](#in-flight) (`active/`) | 2 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 5 | -| [Backlog](#backlog) (`draft/`) | 136 | +| [Backlog](#backlog) (`draft/`) | 135 | ## Start here @@ -163,6 +163,14 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
+
📋 Release/smoke script harness filters JAX tracebacks away, hiding the real exception — issue #186 — issued 2026-08-27 — library-dev (organ repo — PyAutoHeart CI workflow, not a PyAuto library or workspace) + +``` +/start_dev active/jax_traceback_filtering_release_harness.md +``` + +
+ ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -247,7 +255,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**136** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below. +**135** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **25** of them belong to an epic and are listed only under [Epics](#epics) below.
feature — 28 @@ -777,7 +785,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-bug — 17 +bug — 16
📋 LensCalc NumPy Hessian step is too coarse for multi-plane tracers — autogalaxy · large · supervised · high @@ -907,14 +915,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-
📋 interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens - -``` -/start_dev draft/bug/autolens/interferometer_release_leg_oom.md -``` - -
-
@@ -1188,6 +1188,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| +| 2026-08-27 | issued | Release/smoke script harness filters JAX tracebacks away, hiding the… | | 2026-08-27 | filed | One construction path for plane-bound lensing quantities | | 2026-08-27 | filed | Multi-plane time delays | | 2026-08-27 | filed | LensCalc NumPy Hessian step is too coarse for multi-plane tracers | @@ -1197,12 +1198,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-24 | filed | wiki-currency's --check-version gate rots on every library main merge | | 2026-08-24 | filed | interferometer/jax_grad/gradient.py: eager and jitted likelihoods… | | 2026-08-24 | filed | autocti_workspace has no Navigator Check, so its CI can never roll… | -| 2026-08-24 | filed | Un-park multi_galaxy/features/scaling_relation/slam once a capped run… |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| +| 2026-08-24 | filed | Un-park multi_galaxy/features/scaling_relation/slam once a capped run… | | 2026-08-24 | filed | Induct PyAutoReduce into the PyAutoHands release machinery (date… | | 2026-08-24 | filed | Heart's local smoke runner cannot run any CTI workspace — no autocti… | | 2026-08-23 | filed | pynufft removal: unswept downstream residue (1 hard break + stale… | @@ -1212,12 +1213,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | Untrack the generated FITS test artifacts in autoarray | | 2026-08-22 | filed | The reconstruction noise map describes a different estimator than the… | | 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | -| 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the numpy-only… |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| +| 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the numpy-only… | | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-20 | filed | Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… | | 2026-08-20 | filed | Numba CPU likelihood phase 1: batched MGE convolution +… | @@ -1227,12 +1228,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | RTD organism docs currency: Nerves page, organ-count drift, hands.md… | | 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | -| 2026-08-18 | parked | single-source-density-design |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| +| 2026-08-18 | parked | single-source-density-design | | 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | @@ -1242,12 +1243,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-09 | found | isothermal-ell-sph-oversampling-at-the-cusp | | 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | -| 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| +| 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | | 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead | | 2026-08-06 | filed | Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two… | @@ -1257,7 +1258,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | | 2026-08-04 | filed | Nightly release has been blocked 8 nights running — triage the streak | | 2026-08-04 | filed | HowToLens ch4 tutorial 3: mask overlay is never actually drawn | -| 2026-08-03 | filed | Un-park imaging/features/scaling_relation/slam — the PyAutoArray#431… |
@@ -1522,12 +1522,11 @@ Continue the 'Expectation propagation (EP) campaign' epic. Its canonical state l ## Hygiene -4 prompt(s) without a metadata header — they show no facets above. Re-home or re-run intake on them when touched. +3 prompt(s) without a metadata header — they show no facets above. Re-home or re-run intake on them when touched.
Headerless prompts -- `draft/bug/autolens/interferometer_release_leg_oom.md` - `draft/docs/autolens_workspace/sampler_cli_output_workspace_sweep.md` - `draft/docs/workspaces/plot_coverage_followups.md` - `draft/research/autolens_profiling/cluster_gradient_search_benchmark.md` From 9395d83c565fbfa8142dd5111eccbda91f0902e2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 16:33:14 +0000 Subject: [PATCH 2/3] prompt: record PyAutoHeart#187 on jax-traceback-filtering-release-harness Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SFoQCsxRTGjKk2YGe4R9Rs --- active.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/active.md b/active.md index 7b5df9f4..6a09503e 100644 --- a/active.md +++ b/active.md @@ -24,4 +24,8 @@ - origin: split out of the retired prompt `complete/archive/shelved/interferometer_release_leg_oom.md` by /start_dev on 2026-08-27, when that prompt's headline OOM was found already shipped (`complete/2026/08/interferometer-start-here-integrate-oom.md`). +- library-pr: https://github.com/PyAutoLabs/PyAutoHeart/pull/187 (opened 2026-08-27) +- mind-pr: https://github.com/PyAutoLabs/PyAutoMind/pull/347 (registry + retirement; this session + is branch-scoped, so Mind state reaches main via PR rather than a direct push) - repos: + - PyAutoHeart (feature/jax-traceback-filtering-release-harness) From ea933a9ff41795374f4ec2a452a0a6651acb2d0d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 16:35:52 +0000 Subject: [PATCH 3/3] prompt: complete jax-traceback-filtering-release-harness (PyAutoHeart#187) PyAutoHeart#187 merged as c8b093e1; issue #186 closed. Record written, prompt folded out of active/, complete/index.md refreshed, dashboard regenerated. Also swept draft/triage/nightly_release_blocked_eight_nights.md, whose 07-30 and 07-31 rows and "What is actually left" list still pointed at the now-retired draft/bug/autolens/interferometer_release_leg_oom.md as live work. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SFoQCsxRTGjKk2YGe4R9Rs --- active.md | 16 -- ...jax_traceback_filtering_release_harness.md | 89 --------- ...jax-traceback-filtering-release-harness.md | 178 ++++++++++++++++++ complete/index.md | 3 +- dashboard.html | 17 +- dashboard.md | 20 +- .../nightly_release_blocked_eight_nights.md | 13 +- 7 files changed, 203 insertions(+), 133 deletions(-) delete mode 100644 active/jax_traceback_filtering_release_harness.md create mode 100644 complete/2026/08/jax-traceback-filtering-release-harness.md diff --git a/active.md b/active.md index 6a09503e..fb25a165 100644 --- a/active.md +++ b/active.md @@ -13,19 +13,3 @@ - registered: 2026-08-19 by the wake_up session — the issuing session (claude/autofit-priors-messages-audit-ylvenv) filed the prompt + issue but not this entry, tripping Lifecycle Drift on main. - repos-none-claimed: this entry claims NO repos — one line deliberately, not 2-space bullets. - -## jax-traceback-filtering-release-harness -- issue: https://github.com/PyAutoLabs/PyAutoHeart/issues/186 (issued 2026-08-27) -- issued: 2026-08-27 -- prompt: active/jax_traceback_filtering_release_harness.md -- status: library-dev (organ repo — PyAutoHeart CI workflow, not a PyAuto library or workspace) -- worktree: ~/Code/PyAutoLabs-wt/jax-traceback-filtering-release-harness -- session: web session https://claude.ai/code/session_01SFoQCsxRTGjKk2YGe4R9Rs (not `claude --resume`-able) -- origin: split out of the retired prompt `complete/archive/shelved/interferometer_release_leg_oom.md` - by /start_dev on 2026-08-27, when that prompt's headline OOM was found already shipped - (`complete/2026/08/interferometer-start-here-integrate-oom.md`). -- library-pr: https://github.com/PyAutoLabs/PyAutoHeart/pull/187 (opened 2026-08-27) -- mind-pr: https://github.com/PyAutoLabs/PyAutoMind/pull/347 (registry + retirement; this session - is branch-scoped, so Mind state reaches main via PR rather than a direct push) -- repos: - - PyAutoHeart (feature/jax-traceback-filtering-release-harness) diff --git a/active/jax_traceback_filtering_release_harness.md b/active/jax_traceback_filtering_release_harness.md deleted file mode 100644 index f4d5f34a..00000000 --- a/active/jax_traceback_filtering_release_harness.md +++ /dev/null @@ -1,89 +0,0 @@ -# Release/smoke script harness filters JAX tracebacks away, hiding the real exception - -Type: bug -Target: ci -Repos: -- @PyAutoHeart -Difficulty: small -Autonomy: safe -Priority: medium -Status: formalised -Filed: 2026-08-27 -Issued: 2026-08-27 - -## Symptom - -When a workspace script fails under JAX inside PyAutoHeart's script harness -(`.github/workflows/workspace-validation.yml`, job `run_scripts`, both the -`smoke` and `release` legs), JAX's default traceback filter removes its own -internal frames *and*, in practice, the exception that matters: - -``` -scripts/interferometer/start_here.py ... FAIL (19.5s) -For simplicity, JAX has removed its internal frames from the traceback of the -following exception. Set JAX_TRACEBACK_FILTERING=off to include these. -``` - -That is the whole failure record. The reader is told which script failed and -nothing about why. - -## Why it matters — the cost is measured, not hypothetical - -PyAutoHeart run 30516167217 (2026-07-30) failed both interferometer shards -(`autogalaxy` and `autolens`) with exactly the message above. Because the real -exception was filtered away, that night read as an unidentified failure. The -same defect surfaced its true cause only the following night, run -30607596240 (2026-07-31), where the message survived intact: - -``` -jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes. -``` - -One night was spent identifying a failure the harness already had the -information to name. The underlying OOM was root-caused and fixed the same day -(`autolens_workspace#450`; record: -`complete/2026/08/interferometer-start-here-integrate-oom.md`) — this task is -about the *harness*, which still filters, and will do the same to the next -JAX failure. - -A gate whose error messages are filtered away is the same category of problem -as the exit-code contract fixed in PyAutoBrain#196: the signal exists, the -harness discards it before a human sees it. - -## Current state (verified 2026-08-27) - -`JAX_TRACEBACK_FILTERING` appears nowhere in PyAutoHeart: - -``` -$ grep -rn "JAX_TRACEBACK_FILTERING" . # PyAutoHeart @ d576003 — no matches -``` - -Nor in the workspaces' `config/build/profile_release.yaml` / -`profile_smoke.yaml` defaults, both of which already pin every JAX-adjacent var -they care about (`JAX_ENABLE_X64`, `PYAUTO_DISABLE_JAX`, …) explicitly. - -## Proposed fix - -Set `JAX_TRACEBACK_FILTERING: "off"` once, in the workflow-level `env:` block of -`PyAutoHeart/.github/workflows/workspace-validation.yml` — the block that -already carries `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK`. One place covers both -the `smoke` and `release` legs and every workspace in the matrix, and -`run_python.py`'s per-script env profiles inherit it (a profile only *sets* the -keys it is given; it never clears inherited vars). - -Consider whether the same belongs in the other script-running workflows -(`smoke-tests.yml`, `workspace-smoke.yml`) for the same reason. - -## Exit criteria - -- A JAX exception raised inside a workspace script under `run_scripts` reports - its own exception type and message in the job log, not the - "JAX has removed its internal frames" placeholder. -- Verified against a real (or deliberately induced) JAX failure in a CI run, - not only by reading the diff. - -## Origin - -Split out of `draft/bug/autolens/interferometer_release_leg_oom.md` (its -2026-08-04 amendment, consequence 2) when that prompt was retired as superseded -— the OOM it described had already shipped, this harness gap had not. diff --git a/complete/2026/08/jax-traceback-filtering-release-harness.md b/complete/2026/08/jax-traceback-filtering-release-harness.md new file mode 100644 index 00000000..f76d5899 --- /dev/null +++ b/complete/2026/08/jax-traceback-filtering-release-harness.md @@ -0,0 +1,178 @@ +# Release/smoke script harness stopped filtering JAX tracebacks + +Issue: PyAutoLabs/PyAutoHeart#186 (CLOSED 2026-08-27) + +## What was wrong + +JAX removes its own frames from a traceback by default and, in practice, takes +the exception that names the failure with them. A workspace script failing under +JAX inside Heart's script harness reported only: + +``` +scripts/interferometer/start_here.py ... FAIL (19.5s) +For simplicity, JAX has removed its internal frames from the traceback of the +following exception. Set JAX_TRACEBACK_FILTERING=off to include these. +``` + +Which script failed, and nothing about why. + +The cost was measured, not hypothetical. Heart run 30516167217 (2026-07-30) +failed **both** interferometer shards (`autogalaxy` and `autolens`) with exactly +that placeholder, so that night read as an unidentified failure. The real cause — +`jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating +85898814480 bytes` — became legible only in run 30607596240 the following night, +where the message happened to survive. One night was spent identifying a failure +the harness already had the information to name. + +## What shipped + +| Repo | PR | Merged | Change | +|---|---|---|---| +| PyAutoHeart | [#187](https://github.com/PyAutoLabs/PyAutoHeart/pull/187) | 2026-08-27 (`c8b093e1`) | `JAX_TRACEBACK_FILTERING: "off"` — workflow-level `env:` in `workspace-validation.yml`, and the workspace-runner step's `env:` in `smoke-tests.yml`. 2 files, +18/−0 | + +Workflow-level `env:` reaches every job and step of `workspace-validation.yml`, +so both `mode=smoke` and `mode=release`, and every project in the matrix, inherit +it. `run_python.py`'s per-script env profiles inherit it too — a profile only +*sets* the keys it is given and never clears inherited vars, which is the +behaviour `profile_release.yaml`'s own header already relies on. +`workspace-smoke.yml` needed no edit: it is a thin caller of +`workspace-validation.yml`. + +CI before merge: Heart Tests run 33093776599, `pytest (3.12)` + `pytest (3.13)` +both success, `mergeable_state: clean`. Locally `python3 -m pytest -q -n auto` → +641 passed. + +## The exit criterion is deferred, not met + +The issue asked that a real JAX exception print its own type and message in a job +log, "verified against a real (or deliberately induced) JAX failure in a CI run, +not only by reading the diff". Nothing has failed under JAX since the merge, so +that has not been observed. The next JAX failure in `run_scripts` or the smoke +runner is the proof. This is recorded as shipped-but-unproven on purpose: the +whole defect is that the diff looks obviously right and the log is what has to +change. + +## Traps worth keeping + +- **`off` must be quoted in YAML.** Bare `off` is a YAML 1.1 boolean, so + `JAX_TRACEBACK_FILTERING: off` hands GitHub Actions `false`, not the string JAX + expects. Checked with `yaml.safe_load` rather than assumed. +- **A stale prompt can outlive its own fix by a month.** This task exists only + because `/start_dev` on + `draft/bug/autolens/interferometer_release_leg_oom.md` found the prompt's + headline OOM already shipped — + `complete/2026/08/interferometer-start-here-integrate-oom.md`, 2026-07-31, + `MultiStartProdigy` running 48 starts as one unbatched `vmap`. That prompt sat + in `draft/` as pickable backlog for 27 days, and its triage notes pointed at + the wrong subsystem (NUFFT/transformer). It was retired to + `complete/archive/shelved/` in the same run. **Check `complete/` for the slug + before planning a bug prompt**, not after. +- **A prompt's own amendment can be the only live part of it.** The retired + prompt's 2026-08-04 amendment carried two consequences. One ("it is not + autolens-specific") was already closed — both workspaces carry `batch_size=4`. + The other was this task. Reading only the headline would have lost it. + +## Original prompt + +`active/jax_traceback_filtering_release_harness.md`, filed and issued +2026-08-27 by the same `/start_dev` run that retired its parent. + +## Not done + +12 workspace scripts still call `MultiStartProdigy` with no `batch_size` +(autolens: `group/`, `point_source/`, `multi_dataset/`, `cluster/`, `weak/`, …; +autogalaxy: `multi_galaxy/modeling.py`, `multi_dataset/start_here.py`, +`cluster/modeling.py`). Cheaper datasets than a 108k-visibility interferometer +fit, so likely fine — surfaced during this run, deliberately not filed. + +## Original prompt + +# Release/smoke script harness filters JAX tracebacks away, hiding the real exception + +Type: bug +Target: ci +Repos: +- @PyAutoHeart +Difficulty: small +Autonomy: safe +Priority: medium +Status: formalised +Filed: 2026-08-27 +Issued: 2026-08-27 + +## Symptom + +When a workspace script fails under JAX inside PyAutoHeart's script harness +(`.github/workflows/workspace-validation.yml`, job `run_scripts`, both the +`smoke` and `release` legs), JAX's default traceback filter removes its own +internal frames *and*, in practice, the exception that matters: + +``` +scripts/interferometer/start_here.py ... FAIL (19.5s) +For simplicity, JAX has removed its internal frames from the traceback of the +following exception. Set JAX_TRACEBACK_FILTERING=off to include these. +``` + +That is the whole failure record. The reader is told which script failed and +nothing about why. + +## Why it matters — the cost is measured, not hypothetical + +PyAutoHeart run 30516167217 (2026-07-30) failed both interferometer shards +(`autogalaxy` and `autolens`) with exactly the message above. Because the real +exception was filtered away, that night read as an unidentified failure. The +same defect surfaced its true cause only the following night, run +30607596240 (2026-07-31), where the message survived intact: + +``` +jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes. +``` + +One night was spent identifying a failure the harness already had the +information to name. The underlying OOM was root-caused and fixed the same day +(`autolens_workspace#450`; record: +`complete/2026/08/interferometer-start-here-integrate-oom.md`) — this task is +about the *harness*, which still filters, and will do the same to the next +JAX failure. + +A gate whose error messages are filtered away is the same category of problem +as the exit-code contract fixed in PyAutoBrain#196: the signal exists, the +harness discards it before a human sees it. + +## Current state (verified 2026-08-27) + +`JAX_TRACEBACK_FILTERING` appears nowhere in PyAutoHeart: + +``` +$ grep -rn "JAX_TRACEBACK_FILTERING" . # PyAutoHeart @ d576003 — no matches +``` + +Nor in the workspaces' `config/build/profile_release.yaml` / +`profile_smoke.yaml` defaults, both of which already pin every JAX-adjacent var +they care about (`JAX_ENABLE_X64`, `PYAUTO_DISABLE_JAX`, …) explicitly. + +## Proposed fix + +Set `JAX_TRACEBACK_FILTERING: "off"` once, in the workflow-level `env:` block of +`PyAutoHeart/.github/workflows/workspace-validation.yml` — the block that +already carries `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK`. One place covers both +the `smoke` and `release` legs and every workspace in the matrix, and +`run_python.py`'s per-script env profiles inherit it (a profile only *sets* the +keys it is given; it never clears inherited vars). + +Consider whether the same belongs in the other script-running workflows +(`smoke-tests.yml`, `workspace-smoke.yml`) for the same reason. + +## Exit criteria + +- A JAX exception raised inside a workspace script under `run_scripts` reports + its own exception type and message in the job log, not the + "JAX has removed its internal frames" placeholder. +- Verified against a real (or deliberately induced) JAX failure in a CI run, + not only by reading the diff. + +## Origin + +Split out of `draft/bug/autolens/interferometer_release_leg_oom.md` (its +2026-08-04 amendment, consequence 2) when that prompt was retired as superseded +— the OOM it described had already shipped, this harness gap had not. diff --git a/complete/index.md b/complete/index.md index 9b79da84..237a3b56 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1153 records across 7 buckets. +1154 records across 7 buckets. ## Highlights @@ -125,6 +125,7 @@ _(curate hard-won records here — survives regeneration.)_ - [jax-likelihood-mass-sensitivity](2026/08/jax-likelihood-mass-sensitivity.md) — Audit + fix for median-literal mass blindness across all jax_likelihood/datacube scripts (phase 2 of the findi… - [jax-likelihood-smoke-pins-stale](2026/08/jax-likelihood-smoke-pins-stale.md) - [jax-stall-shared-preloads-retime-refutation](2026/08/jax-stall-shared-preloads-retime-refutation.md) +- [jax-traceback-filtering-release-harness](2026/08/jax-traceback-filtering-release-harness.md) - [knowledge-board](2026/08/knowledge-board.md) — auto-closed on merge - [lacosmic-cr-option-and-star-pass-decoupling](2026/08/lacosmic-cr-option-and-star-pass-decoupling.md) — both deliberately OPEN — closure is the default-flip decision, not this ship - [latex-docstrings-invalid-escape-warnings](2026/08/latex-docstrings-invalid-escape-warnings.md) diff --git a/dashboard.html b/dashboard.html index 131ce3fc..ed2588ff 100644 --- a/dashboard.html +++ b/dashboard.html @@ -204,7 +204,7 @@

PyAutoMindDashboard

Intent. Priority. Flow.

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date — what has been happening rather than what to do next.

-
  • 2In flight
  • 3Parked
  • 5Planned
  • 135Backlog
+
  • 1In flight
  • 3Parked
  • 5Planned
  • 135Backlog

Last updated 2026-08-27. This page is generated from active/, draft/ and the registry files, so it is only as current as they are. dashboard_refresh.yml re-renders it on every push to main — that heals a stale page, but not a stale prompt: a task that shipped without its prompt advancing to complete/ keeps rendering here as pickable backlog. Reconciling those is the refresh below.

@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — issued 2026-08-19 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract

-

Release/smoke script harness filters JAX tracebacks away, hiding the real exception — issue #186 — issued 2026-08-27 — library-dev (organ repo — PyAutoHeart CI workflow, not a PyAuto library or workspace)

Parked markdown version

3 task(s) @@ -413,12 +412,6 @@

Backlog 2026-08-27 -issued -Release/smoke script harness filters JAX tracebacks away, hiding the… - - - -2026-08-27 filed One construction path for plane-bound lensing quantities @@ -471,7 +464,7 @@

Backlog autocti_workspace has no Navigator Check, so its CI can never roll… - + 2026-08-24 filed Un-park multi_galaxy/features/scaling_relation/slam once a capped run… @@ -711,6 +704,12 @@

Backlog HowToLens ch4 tutorial 3: mask overlay is never actually drawn + +2026-08-03 +filed +Un-park imaging/features/scaling_relation/slam — the PyAutoArray#431… + +

Epics markdown version

diff --git a/dashboard.md b/dashboard.md index f0adc670..26688199 100644 --- a/dashboard.md +++ b/dashboard.md @@ -42,7 +42,7 @@ anything you could not verify. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 2 | +| [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 5 | | [Backlog](#backlog) (`draft/`) | 135 | @@ -163,14 +163,6 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
-
📋 Release/smoke script harness filters JAX tracebacks away, hiding the real exception — issue #186 — issued 2026-08-27 — library-dev (organ repo — PyAutoHeart CI workflow, not a PyAuto library or workspace) - -``` -/start_dev active/jax_traceback_filtering_release_harness.md -``` - -
- ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). @@ -1188,7 +1180,6 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | Date | Event | Task | |------|-------|------| -| 2026-08-27 | issued | Release/smoke script harness filters JAX tracebacks away, hiding the… | | 2026-08-27 | filed | One construction path for plane-bound lensing quantities | | 2026-08-27 | filed | Multi-plane time delays | | 2026-08-27 | filed | LensCalc NumPy Hessian step is too coarse for multi-plane tracers | @@ -1198,12 +1189,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-24 | filed | wiki-currency's --check-version gate rots on every library main merge | | 2026-08-24 | filed | interferometer/jax_grad/gradient.py: eager and jitted likelihoods… | | 2026-08-24 | filed | autocti_workspace has no Navigator Check, so its CI can never roll… | +| 2026-08-24 | filed | Un-park multi_galaxy/features/scaling_relation/slam once a capped run… |
… 10 more (40 left) | Date | Event | Task | |------|-------|------| -| 2026-08-24 | filed | Un-park multi_galaxy/features/scaling_relation/slam once a capped run… | | 2026-08-24 | filed | Induct PyAutoReduce into the PyAutoHands release machinery (date… | | 2026-08-24 | filed | Heart's local smoke runner cannot run any CTI workspace — no autocti… | | 2026-08-23 | filed | pynufft removal: unswept downstream residue (1 hard break + stale… | @@ -1213,12 +1204,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-22 | filed | Untrack the generated FITS test artifacts in autoarray | | 2026-08-22 | filed | The reconstruction noise map describes a different estimator than the… | | 2026-08-22 | filed | Point-source JSON datasets record no resolution regime | +| 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the numpy-only… |
… 10 more (30 left) | Date | Event | Task | |------|-------|------| -| 2026-08-22 | filed | Is Intel macOS a supported platform, and what is the numpy-only… | | 2026-08-21 | filed | Rectangular mesh split: Bilinear (fast CPU default) vs RTU… | | 2026-08-20 | filed | Numba CPU likelihood phase 2: kernel-CDF numba fast path (the 49-88%… | | 2026-08-20 | filed | Numba CPU likelihood phase 1: batched MGE convolution +… | @@ -1228,12 +1219,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-19 | filed | RTD organism docs currency: Nerves page, organ-count drift, hands.md… | | 2026-08-19 | filed | Deduplicate repos_sync.py's check/write pairs | | 2026-08-19 | filed | Bug in autocti_workspace: the dataset_1d results/database example… | +| 2026-08-18 | parked | single-source-density-design |
… 10 more (20 left) | Date | Event | Task | |------|-------|------| -| 2026-08-18 | parked | single-source-density-design | | 2026-08-18 | parked | prior-message-collapse-design | | 2026-08-18 | filed | @PyAutoFit TransformedMessage.logpdf/pdf omit the transform… | | 2026-08-17 | filed | Which other searches need prior-support handling — coverage audit… | @@ -1243,12 +1234,12 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-09 | found | isothermal-ell-sph-oversampling-at-the-cusp | | 2026-08-08 | parked | pyautoreduce-slacs1430-acs-comparison | | 2026-08-08 | filed | Regenerate autolens_workspace markdown/ so the MGE pages show… | +| 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… |
… 10 more (10 left) | Date | Event | Task | |------|-------|------| -| 2026-08-07 | filed | Regenerate setup_notebook-drifted notebooks in… | | 2026-08-06 | filed | Triage: Convolver "No blurring_image provided" warning in canonical… | | 2026-08-06 | filed | Rewrite PyAutoCTI docs/api — 55 of 89 autosummary entries are dead | | 2026-08-06 | filed | Dependency-cap refresh 2026-08: safe bumps, astropy 8 decision, two… | @@ -1258,6 +1249,7 @@ The 50 newest things to happen to the work in hand, newest first — issued, par | 2026-08-04 | filed | autolens_workspace_developer: broad stale-API rot (56 symbols, no CI) | | 2026-08-04 | filed | Nightly release has been blocked 8 nights running — triage the streak | | 2026-08-04 | filed | HowToLens ch4 tutorial 3: mask overlay is never actually drawn | +| 2026-08-03 | filed | Un-park imaging/features/scaling_relation/slam — the PyAutoArray#431… |
diff --git a/draft/triage/nightly_release_blocked_eight_nights.md b/draft/triage/nightly_release_blocked_eight_nights.md index 2d726090..075885d7 100644 --- a/draft/triage/nightly_release_blocked_eight_nights.md +++ b/draft/triage/nightly_release_blocked_eight_nights.md @@ -102,8 +102,8 @@ most of the set was already fixed or tracked by the time the streak was noticed. | Night | Failing legs | Cause | Status | |---|---|---|---| | 07-28 | 3 × `FileNotFoundError: dataset/…/data.fits` (autogalaxy/multi, autolens/imaging) | auto-simulate guards pointed at the wrong simulator | **FIXED** — see below | -| 07-30 | `interferometer/start_here.py` in **both** autolens and autogalaxy shards; `group/start_here.py` TIMEOUT | same OOM as 07-31, message eaten by the JAX traceback filter | `draft/bug/autolens/interferometer_release_leg_oom.md` | -| 07-31 | `interferometer/start_here.py` → `RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes`; `delaunay.py` TIMEOUT | ~86 GB single allocation; XLA hang | as above; autolens_workspace_test#245 | +| 07-30 | `interferometer/start_here.py` in **both** autolens and autogalaxy shards; `group/start_here.py` TIMEOUT | same OOM as 07-31, message eaten by the JAX traceback filter | **FIXED** — OOM `complete/2026/08/interferometer-start-here-integrate-oom.md`; filter `complete/2026/08/jax-traceback-filtering-release-harness.md` | +| 07-31 | `interferometer/start_here.py` → `RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes`; `delaunay.py` TIMEOUT | 48-start `MultiStartProdigy` vmapped unbatched; XLA hang | **FIXED** — autolens_workspace#450 (`batch_size=4`), discharged by run 30901054267; autolens_workspace_test#245 | | 08-01 | `guides/modeling/advanced/hierarchical.py` TIMEOUT; `delaunay.py` TIMEOUT | Nautilus deadlock; XLA hang | hierarchical **FIXED** (PyAutoFit#1443); #245 | | 08-03 | `graphical/ep.py` — "`log_likelihood_function` is always returning `nan`" | EP initializer | `draft/bug/autofit/graphical_ep_nan_likelihood_release_leg.md` | | 08-04 | `guides/results/database/start_here.py` `IndexError` | `samples_weight_threshold` pruning | **FIXED + MERGED** 08-04 09:40 — autolens_workspace#465, autogalaxy_workspace#202 | @@ -171,8 +171,13 @@ this needs its own prompt. - `draft/bug/autofit/covariance_interpolator_test_unseeded_rng.md` — Class B root cause. Smallest and highest value: it alone unblocks a live release. -- `draft/bug/autolens/interferometer_release_leg_oom.md` — filed 07-31, never - issued; now updated with the 07-30 both-workspaces recurrence. +- ~~`draft/bug/autolens/interferometer_release_leg_oom.md`~~ — **closed out + 2026-08-27.** Its OOM had already shipped on 07-31 (autolens_workspace#450, + `batch_size=4` on the 48-start `MultiStartProdigy`; + `complete/2026/08/interferometer-start-here-integrate-oom.md`), so the prompt + was retired to `complete/archive/shelved/`. The one live part of its 08-04 + amendment — the JAX traceback filter that ate the 07-30 message — shipped as + PyAutoHeart#187 (`complete/2026/08/jax-traceback-filtering-release-harness.md`). - `draft/bug/autofit/graphical_ep_nan_likelihood_release_leg.md` — new. - autolens_workspace_test#245 (delaunay hang) — already tracked, no action here. - Confirm on 08-05 that Check D self-healed and that PyAutoBrain#196 renders the