multi_dataset/jax_likelihood/delaunay.py hangs to the 1800s cap in 3 of 4 release-integrate runs

[Jammy2211]

scripts/multi_dataset/jax_likelihood/delaunay.py has hung to the 1800s release-profile timeout in 3 of the last 4 release-integrate runs, and is now the sole blocker for release validation (the hierarchical.py Nautilus deadlock is fixed, PyAutoFit#1443).

History (release-integrate, autolens_test multi_dataset shard)

Run	Dispatched (UTC)	delaunay.py
30516167217	07-30 05:16	PASS
30607596240	07-31 05:43	TIMEOUT 1800s
30672739606	07-31 23:23	PASS (18.2s)
30686136529	08-01 05:35	TIMEOUT 1800s
30714531417	08-01 19:20	TIMEOUT 1800s

18s when it passes, 1800s cap when it doesn't → a hang, not slowness. Sibling scripts in the same shard (mge, mge_group, rectangular, rectangular_mge, lp, dataset_model) pass every time.

What is known

• First hang is the 07-31 05:43 run; the 07-30→07-31 library window includes PyAutoArray dfc5a988 (nufftax 0.6.x lift + vmap rank-guard) and 4332ec24 (interferometer sim jit) — but delaunay.py is imaging, and the 23:23 run passed at 18.2s with the same SHAs that hung at 05:35, so the trigger is environmental/intermittent, not a deterministic code regression.
• Key pip package versions (jax/jaxlib/nautilus/numpy/scipy/nufftax) are identical between the passing 23:23 job and the failing 08-01 19:20 job; same runner image (ubuntu24/20260720.247).
• The script is a JIT/vmap parity check on a 2-band Delaunay-pixelization FactorGraphModel — the hang profile is consistent with an XLA compile/autotune stall (cf. the compile-time investigation which found cold-compile dominated by autotuning; Delaunay also needs custom_jvp per the gradient audit).
• Local WSL repro under the release env does not hang but fails differently (vmap path returns resample FOM −1e99 for all 3 parameter sets vs −8853.07 direct; likely a local GPU-JAX environment artifact — needs checking on a clean CPU env before trusting).

Release-window handling (2026-08-01)

One Stage 3 re-dispatch is in flight. If delaunay hangs again it will be quarantined via config/build/no_run.yaml (NEEDS_FIX, pointing here) to unblock the release, and un-parked when this is fixed — recorded in this issue either way.

Next steps

• Reproduce on a clean CPU-only env (no GPU): does the vmap −1e99 mismatch appear there too, or is it WSL-local?
• If compile-stall: capture a py-spy/pyststack dump from a hung CI run (e.g. a debug workflow with a watchdog that dumps stacks at 600s) to see whether it sits in backend_compile_and_load/autotuning.
• Consider an XLA_FLAGS autotune cap for the release harness if autotuning is confirmed.

[Jammy2211]

Update 2026-08-01 ~21:30 UTC: the 20:23 re-dispatch (30716867386) cleared delaunay.py (18s) but hung autogalaxy_workspace_test scripts/multi_dataset/jax_likelihood/rectangular.py to the 1800s cap instead — first non-delaunay hang. This is a family-level intermittent XLA compile stall over the multi-dataset factor-graph JIT parity scripts, hopping between members (4 of the last 6 runs hung one member; per-script rates: delaunay 3/5, everything else 1/~90 script-runs).

Quarantined the two observed hangers to unblock the release (NEEDS_FIX entries pointing here): autolens_workspace_test b9b0703 (delaunay.py), autogalaxy_workspace_test 8024012 (rectangular.py). Un-park both when this is fixed. If another family member hangs on the next dispatch it gets the same treatment + logged here.