|
| 1 | +# Release/smoke script harness stopped filtering JAX tracebacks |
| 2 | + |
| 3 | +Issue: PyAutoLabs/PyAutoHeart#186 (CLOSED 2026-08-27) |
| 4 | + |
| 5 | +## What was wrong |
| 6 | + |
| 7 | +JAX removes its own frames from a traceback by default and, in practice, takes |
| 8 | +the exception that names the failure with them. A workspace script failing under |
| 9 | +JAX inside Heart's script harness reported only: |
| 10 | + |
| 11 | +``` |
| 12 | +scripts/interferometer/start_here.py ... FAIL (19.5s) |
| 13 | +For simplicity, JAX has removed its internal frames from the traceback of the |
| 14 | +following exception. Set JAX_TRACEBACK_FILTERING=off to include these. |
| 15 | +``` |
| 16 | + |
| 17 | +Which script failed, and nothing about why. |
| 18 | + |
| 19 | +The cost was measured, not hypothetical. Heart run 30516167217 (2026-07-30) |
| 20 | +failed **both** interferometer shards (`autogalaxy` and `autolens`) with exactly |
| 21 | +that placeholder, so that night read as an unidentified failure. The real cause — |
| 22 | +`jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating |
| 23 | +85898814480 bytes` — became legible only in run 30607596240 the following night, |
| 24 | +where the message happened to survive. One night was spent identifying a failure |
| 25 | +the harness already had the information to name. |
| 26 | + |
| 27 | +## What shipped |
| 28 | + |
| 29 | +| Repo | PR | Merged | Change | |
| 30 | +|---|---|---|---| |
| 31 | +| PyAutoHeart | [#187](https://github.com/PyAutoLabs/PyAutoHeart/pull/187) | 2026-08-27 (`c8b093e1`) | `JAX_TRACEBACK_FILTERING: "off"` — workflow-level `env:` in `workspace-validation.yml`, and the workspace-runner step's `env:` in `smoke-tests.yml`. 2 files, +18/−0 | |
| 32 | + |
| 33 | +Workflow-level `env:` reaches every job and step of `workspace-validation.yml`, |
| 34 | +so both `mode=smoke` and `mode=release`, and every project in the matrix, inherit |
| 35 | +it. `run_python.py`'s per-script env profiles inherit it too — a profile only |
| 36 | +*sets* the keys it is given and never clears inherited vars, which is the |
| 37 | +behaviour `profile_release.yaml`'s own header already relies on. |
| 38 | +`workspace-smoke.yml` needed no edit: it is a thin caller of |
| 39 | +`workspace-validation.yml`. |
| 40 | + |
| 41 | +CI before merge: Heart Tests run 33093776599, `pytest (3.12)` + `pytest (3.13)` |
| 42 | +both success, `mergeable_state: clean`. Locally `python3 -m pytest -q -n auto` → |
| 43 | +641 passed. |
| 44 | + |
| 45 | +## The exit criterion is deferred, not met |
| 46 | + |
| 47 | +The issue asked that a real JAX exception print its own type and message in a job |
| 48 | +log, "verified against a real (or deliberately induced) JAX failure in a CI run, |
| 49 | +not only by reading the diff". Nothing has failed under JAX since the merge, so |
| 50 | +that has not been observed. The next JAX failure in `run_scripts` or the smoke |
| 51 | +runner is the proof. This is recorded as shipped-but-unproven on purpose: the |
| 52 | +whole defect is that the diff looks obviously right and the log is what has to |
| 53 | +change. |
| 54 | + |
| 55 | +## Traps worth keeping |
| 56 | + |
| 57 | +- **`off` must be quoted in YAML.** Bare `off` is a YAML 1.1 boolean, so |
| 58 | + `JAX_TRACEBACK_FILTERING: off` hands GitHub Actions `false`, not the string JAX |
| 59 | + expects. Checked with `yaml.safe_load` rather than assumed. |
| 60 | +- **A stale prompt can outlive its own fix by a month.** This task exists only |
| 61 | + because `/start_dev` on |
| 62 | + `draft/bug/autolens/interferometer_release_leg_oom.md` found the prompt's |
| 63 | + headline OOM already shipped — |
| 64 | + `complete/2026/08/interferometer-start-here-integrate-oom.md`, 2026-07-31, |
| 65 | + `MultiStartProdigy` running 48 starts as one unbatched `vmap`. That prompt sat |
| 66 | + in `draft/` as pickable backlog for 27 days, and its triage notes pointed at |
| 67 | + the wrong subsystem (NUFFT/transformer). It was retired to |
| 68 | + `complete/archive/shelved/` in the same run. **Check `complete/` for the slug |
| 69 | + before planning a bug prompt**, not after. |
| 70 | +- **A prompt's own amendment can be the only live part of it.** The retired |
| 71 | + prompt's 2026-08-04 amendment carried two consequences. One ("it is not |
| 72 | + autolens-specific") was already closed — both workspaces carry `batch_size=4`. |
| 73 | + The other was this task. Reading only the headline would have lost it. |
| 74 | + |
| 75 | +## Original prompt |
| 76 | + |
| 77 | +`active/jax_traceback_filtering_release_harness.md`, filed and issued |
| 78 | +2026-08-27 by the same `/start_dev` run that retired its parent. |
| 79 | + |
| 80 | +## Not done |
| 81 | + |
| 82 | +12 workspace scripts still call `MultiStartProdigy` with no `batch_size` |
| 83 | +(autolens: `group/`, `point_source/`, `multi_dataset/`, `cluster/`, `weak/`, …; |
| 84 | +autogalaxy: `multi_galaxy/modeling.py`, `multi_dataset/start_here.py`, |
| 85 | +`cluster/modeling.py`). Cheaper datasets than a 108k-visibility interferometer |
| 86 | +fit, so likely fine — surfaced during this run, deliberately not filed. |
| 87 | + |
| 88 | +## Original prompt |
| 89 | + |
| 90 | +# Release/smoke script harness filters JAX tracebacks away, hiding the real exception |
| 91 | + |
| 92 | +Type: bug |
| 93 | +Target: ci |
| 94 | +Repos: |
| 95 | +- @PyAutoHeart |
| 96 | +Difficulty: small |
| 97 | +Autonomy: safe |
| 98 | +Priority: medium |
| 99 | +Status: formalised |
| 100 | +Filed: 2026-08-27 |
| 101 | +Issued: 2026-08-27 |
| 102 | + |
| 103 | +## Symptom |
| 104 | + |
| 105 | +When a workspace script fails under JAX inside PyAutoHeart's script harness |
| 106 | +(`.github/workflows/workspace-validation.yml`, job `run_scripts`, both the |
| 107 | +`smoke` and `release` legs), JAX's default traceback filter removes its own |
| 108 | +internal frames *and*, in practice, the exception that matters: |
| 109 | + |
| 110 | +``` |
| 111 | +scripts/interferometer/start_here.py ... FAIL (19.5s) |
| 112 | +For simplicity, JAX has removed its internal frames from the traceback of the |
| 113 | +following exception. Set JAX_TRACEBACK_FILTERING=off to include these. |
| 114 | +``` |
| 115 | + |
| 116 | +That is the whole failure record. The reader is told which script failed and |
| 117 | +nothing about why. |
| 118 | + |
| 119 | +## Why it matters — the cost is measured, not hypothetical |
| 120 | + |
| 121 | +PyAutoHeart run 30516167217 (2026-07-30) failed both interferometer shards |
| 122 | +(`autogalaxy` and `autolens`) with exactly the message above. Because the real |
| 123 | +exception was filtered away, that night read as an unidentified failure. The |
| 124 | +same defect surfaced its true cause only the following night, run |
| 125 | +30607596240 (2026-07-31), where the message survived intact: |
| 126 | + |
| 127 | +``` |
| 128 | +jax.errors.JaxRuntimeError: RESOURCE_EXHAUSTED: Out of memory allocating 85898814480 bytes. |
| 129 | +``` |
| 130 | + |
| 131 | +One night was spent identifying a failure the harness already had the |
| 132 | +information to name. The underlying OOM was root-caused and fixed the same day |
| 133 | +(`autolens_workspace#450`; record: |
| 134 | +`complete/2026/08/interferometer-start-here-integrate-oom.md`) — this task is |
| 135 | +about the *harness*, which still filters, and will do the same to the next |
| 136 | +JAX failure. |
| 137 | + |
| 138 | +A gate whose error messages are filtered away is the same category of problem |
| 139 | +as the exit-code contract fixed in PyAutoBrain#196: the signal exists, the |
| 140 | +harness discards it before a human sees it. |
| 141 | + |
| 142 | +## Current state (verified 2026-08-27) |
| 143 | + |
| 144 | +`JAX_TRACEBACK_FILTERING` appears nowhere in PyAutoHeart: |
| 145 | + |
| 146 | +``` |
| 147 | +$ grep -rn "JAX_TRACEBACK_FILTERING" . # PyAutoHeart @ d576003 — no matches |
| 148 | +``` |
| 149 | + |
| 150 | +Nor in the workspaces' `config/build/profile_release.yaml` / |
| 151 | +`profile_smoke.yaml` defaults, both of which already pin every JAX-adjacent var |
| 152 | +they care about (`JAX_ENABLE_X64`, `PYAUTO_DISABLE_JAX`, …) explicitly. |
| 153 | + |
| 154 | +## Proposed fix |
| 155 | + |
| 156 | +Set `JAX_TRACEBACK_FILTERING: "off"` once, in the workflow-level `env:` block of |
| 157 | +`PyAutoHeart/.github/workflows/workspace-validation.yml` — the block that |
| 158 | +already carries `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK`. One place covers both |
| 159 | +the `smoke` and `release` legs and every workspace in the matrix, and |
| 160 | +`run_python.py`'s per-script env profiles inherit it (a profile only *sets* the |
| 161 | +keys it is given; it never clears inherited vars). |
| 162 | + |
| 163 | +Consider whether the same belongs in the other script-running workflows |
| 164 | +(`smoke-tests.yml`, `workspace-smoke.yml`) for the same reason. |
| 165 | + |
| 166 | +## Exit criteria |
| 167 | + |
| 168 | +- A JAX exception raised inside a workspace script under `run_scripts` reports |
| 169 | + its own exception type and message in the job log, not the |
| 170 | + "JAX has removed its internal frames" placeholder. |
| 171 | +- Verified against a real (or deliberately induced) JAX failure in a CI run, |
| 172 | + not only by reading the diff. |
| 173 | + |
| 174 | +## Origin |
| 175 | + |
| 176 | +Split out of `draft/bug/autolens/interferometer_release_leg_oom.md` (its |
| 177 | +2026-08-04 amendment, consequence 2) when that prompt was retired as superseded |
| 178 | +— the OOM it described had already shipped, this harness gap had not. |
0 commit comments