ENH: Overlap-add processing for maxwell filter

[larsoner]

Algorithms that process data in sequential temporal windows benefit from overlap-add processing to reduce edge artifacts. OTP in our codebase already uses this. This PR adds that to maxwell_filter for tSSS and for movement compensation (which is trickier because the windows / estimation points themselves have variable duration / timing).

The MF code changes will be hard to review 😞 Fortunately, the maxwell_filter tests we have are pretty extensive, so we can sleep well at night knowing they still pass with very few modifications. And we see some improvements in tests with what will be the new defaults in 1.11. In some ways I think the code actually gets a bit clearer by the use of classes, for example for movement compensation / head position a class makes sense because the head position is a state you want to keep track of and get information / data / methods for. That way not all aspects are controlled by a monolithic, multi-indented function like now -- there is still a multi-indented monster, but its content is much more limited and thus easier to follow.

@drammock one point for debate is whether we should actually make the new defaults "smooth" (via deprecation) as currently implemented here. In theory and in my testing they are indeed better, but they will make us more different from standard maxwell filtering. But I think the gain in data SNR is probably worth it.

[drammock]

I had a look through the code, but not with a fine-toothed comb. Nothing jumps out as questionable, but I will try to do a more thorough look (and testing) next week. I'm a bit undecided about changing the defaults, but I haven't actually looked at the results / SNR of this yet, which could easily sway me.

[drammock]

[nitpick] this doesn't need to be a separate import statement I think? locally I can do

from numpy import (
    ndarray,
    sum as npsum,  # <---
    rollaxis
)

and it works.

[larsoner]

Indeed, but ruff wants it this way, it undoes the proposed change:

larsoner:~/python/mne-python$ git diff
diff --git a/mne/preprocessing/tests/test_maxwell.py b/mne/preprocessing/tests/test_maxwell.py
index 2f279440d1..ce28a04b0c 100644
--- a/mne/preprocessing/tests/test_maxwell.py
+++ b/mne/preprocessing/tests/test_maxwell.py
@@ -34,10 +34,8 @@ from mne.preprocessing import (
     annotate_movement,
     compute_maxwell_basis,
     find_bad_channels_maxwell,
-    maxwell_filter_prepare_emptyroom,
-)
-from mne.preprocessing import (
     maxwell_filter as _maxwell_filter_ola,
+    maxwell_filter_prepare_emptyroom,
 )
 from mne.preprocessing.maxwell import (
     _bases_complex_to_real,
larsoner:~/python/mne-python$ pre-commit run --all
ruff lint mne........................................................................Failed
- hook id: ruff
- files were modified by this hook

Found 1 error (1 fixed, 0 remaining).
...
larsoner:~/python/mne-python$ git diff
larsoner:~/python/mne-python$

[drammock]

weird, ruff

[drammock]

so IIUC this would change after a deprecation cycle right? And also the smoothing of headpos also gets enabled by default with 1.11 release?

[larsoner]

Yeah both get enabled by default in 1.11. I would be inclined to leave this after 1.11 actually to keep diff and blame clean, but we could do a bunch of renames to like maxwell_filter_old or something

[drammock]

So you're leaning toward not changing the defaults in 1.11? For me it depends on how big is the gain + how universal is the gain for different datasets. If it's (almost) always a (fairly big) improvement, there's a clear case to change the defaults, and our job becomes making sure it's widely communicated. If it's only sometimes an improvement, then I'd vote to leave the defaults as-is. I'm not in favor of maxwell_filter_old.

[larsoner]

I do want to change the defaults to the new smooth ones. Gains should be near universal.

I'm just saying I don't plan on changing this test file in particular because it'll lead to a bunch of code churn and blame loss. Like maxwell_filter(...) in this file would continue to mean "non-smooth" and maxwell_filter_ola would mean "smooth".

[drammock]

ahhh, OK, sorry I didn't realize this was in a test file. everything makes sense now 🤦🏻