Skip to content

Commit 8cdfad5

Browse files
committed
Document the DFT/NUFFT trade-off on the sparse-operator setup path
Both transformers work with `apply_sparse_operator` and agree to ~3e-13 relative, but which is *faster* was undocumented, and the answer is not the one the codebase's existing 10,000-visibility guard implies. Measured on CPU (`use_jax=False`), the crossover is governed by the product `N_vis * N_pix`, not by visibility count alone — which follows from the DFT setup being `O(N_vis * N_pix)` against the NUFFT's `O((N_vis + N_pix) log N)` plus a fixed ~2s overhead. Seven measurements agree on a crossover near 1e7: N_vis x N_pix DFT/NUFFT time ratio 1.3e6 0.21x (DFT faster) 2.6e6 0.27x 5.2e6 0.70x 1.0e7 1.16x (crossover) 2.1e7 1.50x 2.3e7 1.41x 8.3e7 1.92x (NUFFT faster) So at a typical 64x64 mask the crossover sits near 5,000 visibilities, but on a 32x32 mask the DFT still wins at 4,000. Quoting a visibility count alone would be wrong on half the grids. Memory is the more decisive axis and is what makes this more than a micro-optimisation. The DFT's allocation grows with `N_vis * N_pix` (measured 60 -> 239 -> 293 -> 446 MB as the grid grows at fixed N_vis=4000) while the NUFFT path allocates nothing measurable beyond its working buffers. At 10.7 bytes per element that extrapolates to ~109 GB at 1M visibilities, which independently corroborates the ~123 GB figure recorded in bd18a76 and confirms the NUFFT is the only feasible path at ALMA scale. Adds the guidance as an INFO on the existing setup log, emitted only when a NUFFT-backed transformer is used below the crossover — i.e. only on an explicit, expensive, opt-in call, and never when the advice would be moot. Deliberately NOT a warning on `Interferometer` construction: that class defaults to `TransformerNUFFT`, so a small-dataset warning would fire on the default path for every tutorial, test and workspace example. The dangerous direction is already a hard `DatasetException` above 10,000 visibilities with an opt-out, so nothing is left unguarded by keeping this advisory quiet. Verified the advisory fires for small+NUFFT and stays silent for both small+DFT and large+NUFFT. Suite green: 1179 passed.
1 parent 46fc1c5 commit 8cdfad5

1 file changed

Lines changed: 29 additions & 0 deletions

File tree

autoarray/dataset/interferometer/dataset.py

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,21 @@ def apply_sparse_operator(
232232
the number of visibilities and the real-space mask resolution — potentially hours for large
233233
datasets). The result can be cached to disk and reloaded to avoid recomputation.
234234
235+
Both `TransformerDFT` and `TransformerNUFFT` are supported here and agree to ~3e-13 relative.
236+
Which is faster is governed by the product `N_vis * N_pix`, because the DFT setup cost scales
237+
as `O(N_vis * N_pix)` whereas the NUFFT scales as `O((N_vis + N_pix) log N)` on top of a fixed
238+
~2s overhead:
239+
240+
- below `N_vis * N_pix ~ 1e7`, `TransformerDFT` is faster (measured 0.2-0.7x the NUFFT time)
241+
- above it, `TransformerNUFFT` is faster (1.2-1.9x at 1e7-1e8)
242+
- above `~1e8`, the DFT's `O(N_vis * N_pix)` allocation makes it infeasible rather than merely
243+
slow — extrapolating a measured 446 MB at N_vis=4e3 / N_pix=1e4 gives ~109 GB at 1M
244+
visibilities, whereas the NUFFT path allocates nothing beyond its working buffers.
245+
246+
As a rule of thumb at a typical 64x64 mask the crossover sits near 5,000 visibilities, but on a
247+
coarser mask the DFT stays ahead well beyond that — it is the product that matters, not the
248+
visibility count alone.
249+
235250
Parameters
236251
----------
237252
nufft_precision_operator
@@ -264,6 +279,20 @@ def apply_sparse_operator(
264279
"INTERFEROMETER - Computing NUFFT Precision Operator; runtime scales with visibility count and mask resolution, CPU run times may exceed hours."
265280
)
266281

282+
n_vis = self.uv_wavelengths.shape[0]
283+
n_pix = self.real_space_mask.pixels_in_mask
284+
285+
if n_vis * n_pix < 10**7 and not isinstance(
286+
self.transformer, TransformerDFT
287+
):
288+
logger.info(
289+
f"INTERFEROMETER - This dataset is small for the NUFFT setup path "
290+
f"(N_vis x N_pix = {n_vis * n_pix:.1e}, below the ~1e7 crossover). "
291+
f"`TransformerDFT` computes this operator faster below that point; "
292+
f"`TransformerNUFFT` wins above it, and above ~1e8 it is the only "
293+
f"option that fits in memory. Both are supported here."
294+
)
295+
267296
nufft_precision_operator = self.psf_precision_operator_from(
268297
chunk_k=chunk_k,
269298
show_progress=show_progress,

0 commit comments

Comments
 (0)