Skip to content

De-hang resample-merge system tests: arm idle terminator at startup + absolute watchdog - #172

Merged
cboulay merged 1 commit into
devfrom
fix-ci-hang-resample-merge
Jul 8, 2026
Merged

De-hang resample-merge system tests: arm idle terminator at startup + absolute watchdog#172
cboulay merged 1 commit into
devfrom
fix-ci-hang-resample-merge

Conversation

@cboulay

@cboulay cboulay commented Jul 8, 2026

Copy link
Copy Markdown
Member

Fixes the semi-frequent CI hangs on test_resample_merge_system.py.

Root cause

TerminateOnTimeout.poll_terminate only checks the idle gap once last_msg_timestamp is set — i.e. the idle timer arms only on the first received message. The hubs in these tests deliberately never complete (while True: await asyncio.sleep(0.1), from the earlier de-flake, so a seized graph is distinguishable from exhausted input), which leaves the idle terminator as the graph's only exit path. But it is fed from the merged output: a run that produces zero output messages never arms the timer, and ez.run blocks until the CI job timeout.

Zero-output runs are timing-dependent and exactly what these tests court: the seize test's glitch lands ~30 ms in, and on a loaded runner the resampler warmup + merge alignment can lose the race, so no merged message precedes the seize. That matches the semi-frequent, CI-only signature.

Fix (belt and suspenders)

  1. ArmedTerminateOnTimeout — sets last_msg_timestamp in initialize(), so an all-idle graph tears down after the 4 s idle window and the test fails on its assertions (n_msgs > 0 / last_t = -inf) instead of hanging for hours.
  2. An unconnected armed terminator as a 30 s absolute deadline per graph run, in case future wiring defeats the idle terminator. A full run terminates via the idle path in well under 10 s, so it never fires in healthy runs.

If the seize test ever fails on a slow runner post-fix (zero output before the glitch), the remedy is a later glitch_at or larger dispatch_dt — visible failure beats a silent hang.

Worth considering upstream: TerminateOnTimeout arming at startup in ezmsg.util.terminate itself — an idle terminator that cannot fire from the all-idle state is a footgun for any graph shaped like this one.

All 4 tests pass locally (~18.5 s total).

… watchdog

TerminateOnTimeout only arms its idle timer on the first received message. The
hubs in these tests deliberately never complete (so a seized graph is
distinguishable from exhausted input), leaving the idle terminator as the only
exit path -- but it is fed from the merged output, so a run that produces zero
output messages never arms it and ez.run blocks until the CI job timeout. This
is timing-dependent (resampler warmup + merge alignment vs. the early glitch),
matching the semi-frequent CI hangs on this file.

- ArmedTerminateOnTimeout initializes last_msg_timestamp at startup, so a
  zero-output run tears down after the 4 s idle window and fails on the test's
  assertions instead of hanging.
- A second, unconnected armed terminator acts as a 30 s absolute deadline in
  case future wiring defeats the idle terminator.
@cboulay
cboulay merged commit 88ad077 into dev Jul 8, 2026
14 checks passed
@cboulay
cboulay deleted the fix-ci-hang-resample-merge branch July 8, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant