Skip to content

EWMA/scaler: add opt-in reset_on_resume for passthrough gaps - #200

Open
cboulay wants to merge 1 commit into
devfrom
feat/ewma-reset-on-resume
Open

EWMA/scaler: add opt-in reset_on_resume for passthrough gaps#200
cboulay wants to merge 1 commit into
devfrom
feat/ewma-reset-on-resume

Conversation

@cboulay

@cboulay cboulay commented Aug 7, 2026

Copy link
Copy Markdown
Member

Addresses #195, as an opt-in setting rather than a behaviour change.

The problem

EWMATransformer and AdaptiveStandardScalerTransformer short-circuit passthrough in __call__/__acall__, returning before _hash_message, so self._hash keeps whatever value it had when passthrough was switched on. Switching passthrough back off therefore resumes from the state as it was before the gap.

zi is an exponentially-weighted average over the recent past. During passthrough the filter sees none of the samples that go by, so on resume it describes a window that ended when passthrough began — and the gap is invisible in the state, so a 10 ms blip and a 10 minute outage resume identically. For the scaler that means z-scoring post-gap data against pre-gap mean and variance.

The change

reset_on_resume: bool = False on EWMASettings, inherited by AdaptiveStandardScalerSettings.

Default False preserves current behaviour — the three tests pinning seamless resume are untouched, and the existing passthrough docstring promise still holds. A short blip arguably should resume seamlessly, and resetting discards an estimate that may have taken many time_constants to converge.

True rebuilds from the first post-gap message instead, matching the position BinnedAggregateTransformer already takes deliberately for the same flag. With the existing bias correction the first post-gap output is exactly the first sample, and the estimate re-converges over time_constant. For the scaler, _reset_state rebuilds both child EWMAs, so a parent reset propagates without extra work.

Two details

The reset is driven from __call__/__acall__, not update_settings. So it also holds when settings are pushed by assignment (proc.settings = replace(...)) rather than through update_settings. Both passthrough and reset_on_resume stay in NONRESET_SETTINGS_FIELDS: a toggle with no messages in between leaves no gap and so should not reset.

The passthrough and empty-message short-circuits are now separate. They were one or:

if self.settings.passthrough or np.prod(message.data.shape) == 0:

An empty chunk carries no samples past the filter and so leaves no hole in zi's history; only passthrough does. Sharing one condition would have made every empty chunk look like a gap. Both are now routed through a small _skip() helper that documents the distinction, with a test pinning that an empty chunk — including one arriving during passthrough — leaves _hash untouched.

Tests

New, all exercising reset_on_resume=True (the default path is covered by the existing tests):

  • test_ewma_reset_on_resume_discards_state — output after the gap matches a never-used transformer, not one carrying the pre-gap estimate. Also asserts the state is left alone during the gap; only the hash is invalidated, so the rebuild happens on the next real message rather than eagerly.
  • test_ewma_reset_on_resume_via_update_settings — toggling queues no reset by itself; the first passthrough message does.
  • test_ewma_reset_on_resume_ignores_empty_messages — the case the split exists to protect.
  • TestAdaptiveStandardScalerPassthrough::test_reset_on_resume_discards_state — both child EWMAs are replaced on resume.

Full suite: 3693 passed, 5 skipped. Ruff clean.

EWMATransformer and AdaptiveStandardScalerTransformer short-circuit
passthrough before _hash_message, so switching passthrough back off
resumes from a `zi` describing an exponentially-weighted window that
ended when passthrough was switched on. The gap is invisible in the
state -- a 10 ms blip and a 10 minute outage resume identically -- and
for the scaler that means z-scoring post-gap data against pre-gap
statistics.

Add `reset_on_resume` to EWMASettings (inherited by
AdaptiveStandardScalerSettings), defaulting to False so current
behaviour is unchanged. True rebuilds from the first post-gap message
instead, matching the position BinnedAggregateTransformer already takes.

Two details worth noting:

- The reset is driven from __call__/__acall__ rather than from
  update_settings, so it also holds when settings are pushed by
  assignment. Both flags stay in NONRESET_SETTINGS_FIELDS: a toggle with
  no messages in between leaves no gap and so should not reset.

- The passthrough and empty-message short-circuits are now separate. An
  empty chunk passes no samples to the filter and so leaves no hole in
  zi's history; only passthrough does. Sharing one `or` would have made
  every empty chunk look like a gap.

Refs #195
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant