Skip to content

Dual-timescale EWMA: recover from mid-stream level shifts (scaler, detrend) - #173

Draft
cboulay wants to merge 1 commit into
devfrom
feat-168-dual-timescale-ewma
Draft

Dual-timescale EWMA: recover from mid-stream level shifts (scaler, detrend)#173
cboulay wants to merge 1 commit into
devfrom
feat-168-dual-timescale-ewma

Conversation

@cboulay

@cboulay cboulay commented Jul 8, 2026

Copy link
Copy Markdown
Member

Closes #168.

Problem

Bias correction (#166) fixed cold start, but a sudden persistent level shift mid-stream (reference change, electrode impedance step, amplifier re-lock) still biases the EWMA — and downstream z-scores — for ~3·time_constant. In the scaler this reproduces the #166 collapse signature mid-session: the lagging mean biases every z-score one direction while the lagging second moment inflates the std, so features sit at a systematic z ≈ ±1 until the EWMA re-converges.

Design (as proposed in #168, decimated-detector variant)

Opt-in via EWMASettings.fast_time_constant (default None = off; detector-off output is bit-identical to before):

  • A fast estimate is advanced once per chunk from the chunk mean, using the chunk's effective forgetting factor (1-α_fast)^n — no second per-sample scan, so the full-rate cost is unchanged.
  • Divergence test at each chunk boundary: |fast − slow| > shift_threshold · (within-chunk residual std), with shift_hysteresis consecutive divergent chunks required (default 4·std, 2 chunks).
  • Per-channel snap via xp.where: the main estimate's scan state is set so its bias-corrected value equals the fast estimate; it then continues with its own time constant. A single stepping electrode doesn't disturb the other channels.
  • All detector work is elementwise on state at chunk boundaries (mean/sqrt/abs/where) — Array API portable, verified numpy/MLX parity to float32 tolerance.

Normalizing by the within-chunk residual std makes the detector robust to variance-only artifacts: a noise burst inflates the denominator, so no snap (asserted bit-identical in tests).

Scope

  • EWMATransformer: detector + snap.
  • AdaptiveStandardScaler: propagates the settings to both child EWMAs (mean and E[x²] snap on their respective inputs) — z-scores re-center within a few chunks of a shift.
  • Detrend: inherits through EWMATransformer, nothing to change.

Tests

  • Level shift: snapped estimate reaches the new level within 3 chunks; detector-off lags to end of stream.
  • Per-channel: un-shifted channel bit-identical to detector-off run; shifted channel snaps.
  • Stationary and variance-burst streams: bit-identical to detector-off (no false snaps).
  • Scaler: signed mean z re-centers (<0.3) vs. systematic one-sided bias (>0.7) without tracking.

Defaults / tuning

shift_threshold=4.0 and shift_hysteresis=2 are conservative first choices; both only matter when fast_time_constant is set. Blend-instead-of-hard-snap and a shared (rather than per-child) detector for the scaler were considered and deferred — noted as follow-ups in #168 if needed.

Closes #168. Bias correction (#166) fixed cold start, but a sudden persistent
level shift (reference change, impedance step) still biased the EWMA -- and
z-scores downstream -- for ~3*time_constant.

Opt-in via EWMASettings.fast_time_constant: a secondary estimate is advanced
once per chunk by the chunk's effective forgetting factor (decimated -- no
second per-sample scan). When it diverges from the main estimate by more than
shift_threshold * within-chunk residual std for shift_hysteresis consecutive
chunks, the main estimate's scan state snaps to the fast estimate, per
channel. All detector work is elementwise on state at chunk boundaries
(mean/sqrt/abs/where), so it is Array API portable; verified numpy/MLX parity.

Normalizing divergence by the within-chunk residual std makes the detector
insensitive to variance-only artifacts (bursts inflate the denominator), and
per-channel xp.where snapping lets a single electrode step without disturbing
the other channels' estimates.

AdaptiveStandardScaler propagates the settings to both child EWMAs (mean and
second moment snap on their respective inputs); Detrend inherits the feature
through EWMATransformer. Defaults are inert (fast_time_constant=None):
detector-off output is bit-identical.
@cboulay
cboulay force-pushed the feat-168-dual-timescale-ewma branch from 396941c to 4b82ed3 Compare July 8, 2026 14:50
@cboulay

cboulay commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

I don't plan to merge this anytime soon. It's a solution for a problem that I haven't really encountered yet and thus likely adds unnecessary complication.

@cboulay
cboulay marked this pull request as draft July 21, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adaptive EWMA statistics (scaler) are slow to re-converge after a mid-stream level shift; proposal: dual-timescale EWMA

1 participant