Skip to content

Replace busy-wait publishers in ResampleConcat and ResampleUnit with event-driven publishing - #178

Merged
cboulay merged 2 commits into
devfrom
feat-resampleconcat-event-driven
Jul 21, 2026
Merged

Replace busy-wait publishers in ResampleConcat and ResampleUnit with event-driven publishing#178
cboulay merged 2 commits into
devfrom
feat-resampleconcat-event-driven

Conversation

@cboulay

@cboulay cboulay commented Jul 21, 2026

Copy link
Copy Markdown
Member

Summary

ResampleConcat.output() and ResampleUnit.gen_resampled() both polled next(self.processor) in a while True loop with await asyncio.sleep(0) when no output was ready. sleep(0) yields to the event loop but never sleeps, so each unit spun a full CPU core whenever idle. This PR makes both units publish only when output can actually exist.

ResampleConcat: drain from the subscriber handlers

The polling publisher is removed entirely. on_reference / on_signal each push to the processor and then drain it, publishing every chunk it can currently produce (same idiom as Align). This is lossless because the composed resampler is always reference-driven (resample_rate=None), and in that mode output readiness only ever changes on new input — the wall-clock max_chunk_delay extrapolation applies to prescribed-rate mode only. A single next() consumes all currently-eligible reference values, so the drain loop runs at most twice per message.

Side benefit: backpressure is preserved — a subscriber does not complete until its outputs are published, so a slow consumer backs up the input queues instead of silently growing the resampler's internal buffers.

ResampleUnit: event wake with a timeout in prescribed-rate mode

This unit cannot publish from its handlers alone: in prescribed-rate mode with a finite max_chunk_delay, output is intended to become ready by wall clock with no input. The publisher therefore waits on an asyncio.Event set by both input handlers, using max_chunk_delay as a wait_for timeout when resample_rate is set and the delay is finite. The event is cleared before draining so a push landing mid-drain re-arms the next wait rather than being lost. on_signal is overridden via @ez.subscriber(BaseConsumerUnit.INPUT_SIGNAL) (the same pattern Downsample uses) to add the wake.

Pre-existing issue surfaced (not fixed here)

While testing the timed wake I found that the max_chunk_delay wall-clock extrapolation is currently unreachable at the processor level, and was under the old polling design too: after a drain the source buffer retains only ~2 samples, and ResampleProcessor.__next__ returns early on src.available() < 3 before evaluating b_project. The timed wake preserves the intended trigger for when that guard is reworked (linear interp1d only needs 2 points, so relaxing the guard for the extrapolation path looks feasible); the new integration test documents this and can be extended to assert extrapolated output once fixed.

Tests

  • tests/integration/ezmsg/test_resampleconcat_system.py (new): drives ResampleConcat through a live ez.run graph with interleaved reference/signal chunks at 100 vs 99.7 Hz; asserts output flows, has A+B channels, and that the published time axis stays strictly monotonic with both handlers publishing to the same stream. Terminates on quiet, which also proves nothing depends on a background publisher task.
  • tests/integration/ezmsg/test_resample_system.py (new): drives ResampleUnit in reference-driven and prescribed-rate modes; asserts output flows and is monotonic, with a documented pointer for the extrapolation assertion above.
  • Full suite: 3651 passed, 7 skipped.

cboulay added 2 commits July 21, 2026 17:47
The output() publisher polled next(processor) in a loop with
asyncio.sleep(0), spinning a full core whenever no output was ready.
Publish instead from the two subscriber handlers, draining the processor
after each push. This is lossless because the composed resampler is
always reference-driven (resample_rate=None), so output readiness only
changes on new input -- the wall-clock max_chunk_delay extrapolation
applies to prescribed-rate mode only. Also preserves backpressure: a
subscriber does not complete until its outputs are published.

Adds an integration test driving the unit through a live graph with
interleaved reference/signal chunks, asserting output flows and the
published time axis stays monotonic with both handlers publishing.
gen_resampled polled next(processor) with asyncio.sleep(0), spinning a
core whenever no output was ready. Wait instead on an asyncio.Event set
by both input handlers (cleared before draining so a push landing
mid-drain re-arms the wait). Unlike ResampleConcat, this unit cannot
publish from its handlers alone: in prescribed-rate mode with a finite
max_chunk_delay, output is meant to become ready by wall clock with no
input, so the event wait uses that delay as a timeout in that mode.

Note: the wall-clock extrapolation itself is currently unreachable at
the processor level (after a drain the source buffer retains ~2 samples
and __next__ returns early on src.available() < 3 before evaluating
b_project) -- true of the previous polling design as well. The timed
wake preserves the intended trigger for when that guard is reworked;
the new integration test documents this.

Adds integration tests driving ResampleUnit through a live graph in
both reference-driven and prescribed-rate modes.
@cboulay
cboulay merged commit f9f68f0 into dev Jul 21, 2026
14 checks passed
@cboulay
cboulay deleted the feat-resampleconcat-event-driven branch July 21, 2026 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant