Skip to content

Commit cbc2cb1

Browse files
James Nightingaleclaude
andcommitted
prompt: the captured stack says this is not a compile stall
The two 1800s runs dumped faulthandler tracebacks at 1440s. Both repos, both scripts, the same stack: parked in jax.block_until_ready / try_to_block — the EXECUTION half of the first call. func(*args, **kwargs) returned; what never returns is the wait for the result to materialize. So the epic's name, and every marker calling this an intermittent XLA compile stall, inherit a guess made before there was evidence. Corrected in the record, the epic entry, the ledger and the follow-up task so the smoke speed-up work does not restart from the wrong question. Also recorded: the heartbeat says 'still compiling' while the process sits in block_until_ready — log_on_first_compile cannot tell which half it is in. Fix the wording before anyone reads another of these logs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qk7hoavMnFyPtW4toYn8K
1 parent 358c13c commit cbc2cb1

6 files changed

Lines changed: 349 additions & 338 deletions

File tree

complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,3 +175,68 @@ imply the API one is resolved.
175175
chasing speedups on scripts that are hanging.
176176
- No script is un-quarantined by this phase — restoring coverage is phase 3,
177177
and depends on the fix.
178+
179+
## The stack, captured after close-out — it is NOT a compile stall
180+
181+
The two 1800s runs named above finished at 22:38 and both dumped a
182+
`faulthandler` traceback at 1440s, exactly as PyAutoFit#1518 intended. **The
183+
stack contradicts the premise this whole epic was filed under.**
184+
185+
`autogalaxy_workspace_test` run 32668061785, `imaging/jax_likelihood/mge_group.py`:
186+
187+
```
188+
Thread 0x00007fbf1f0e4b80 (most recent call first):
189+
File ".../jax/_src/api.py", line 2764 in try_to_block
190+
File ".../jax/_src/api.py", line 2781 in block_until_ready
191+
File ".../autofit/non_linear/jax_compile.py", line 264 in wrapper
192+
File ".../scripts/imaging/jax_likelihood/mge_group.py", line 175 in <module>
193+
```
194+
195+
`autolens_workspace_test` run 32668067325, `multi_dataset/jax_likelihood/mge.py`
196+
— different repo, different script, **same stack**:
197+
198+
```
199+
File ".../jax/_src/api.py", line 2781 in block_until_ready
200+
File ".../autofit/non_linear/jax_compile.py", line 264 in wrapper
201+
File ".../scripts/multi_dataset/jax_likelihood/mge.py", line 150 in <module>
202+
```
203+
204+
`jax_compile.py:264` is the `jax.block_until_ready(result)` call — the **second**
205+
half of the wrapped first call. The process is not tracing, lowering or
206+
compiling. `func(*args, **kwargs)` **returned**; what never returns is the wait
207+
for the result to materialize.
208+
209+
### What this overturns
210+
211+
- Every marker, issue and prompt in this epic calls it an *"intermittent XLA
212+
compile stall"*. On this evidence that name is **wrong**, and it has been
213+
wrong since the first quarantine on 2026-08-01. Anyone resuming should treat
214+
"compile stall" as a label inherited from a guess, not a finding.
215+
- The heartbeat line reads `JAX jit still compiling ... 1770s elapsed` while the
216+
process sits in `block_until_ready`. That wording is a defect in
217+
`log_on_first_compile` — the wrapper cannot tell which half it is in and says
218+
"compiling" regardless. **Fix the message to name the phase before anyone
219+
reads another one of these logs.**
220+
- Both entries are `AMBIGUOUS` even at the 1800s cap: 2/2 runs capped on both
221+
legs, no completion. Combined with the 300s round, that is **20 consecutive
222+
cap hits** for `mge_group.py` and `multi_dataset/mge.py` with zero
223+
completions — while `mge.py`'s own marker records it finishing in 32s
224+
standalone.
225+
226+
### Where this points instead
227+
228+
An execution/materialization hang, not a compiler one. The hypotheses worth
229+
testing are now different from the ones this epic queued up:
230+
231+
1. **Device transfer or the async dispatch queue deadlocking** on the runner —
232+
`try_to_block` is where a never-arriving buffer would park.
233+
2. **The persistent compilation cache** is still live but for a different
234+
reason than assumed: a cache *read* satisfying the compile instantly and
235+
leaving execution to hang is consistent with what is seen here.
236+
3. **The `vmap(jit)` ordering** result (80% -> 30%, p=0.070) needs reinterpreting
237+
in this light: the ordering changes the *shape of the executed computation*,
238+
not just the compile graph, which is a more plausible route to an execution
239+
hang than to a compile one.
240+
241+
Do not restart from "why is XLA slow to compile". Start from "why does
242+
`block_until_ready` never return".

0 commit comments

Comments
 (0)