De-hang resample-merge system tests: arm idle terminator at startup + absolute watchdog - #172
Merged
Merged
Conversation
… watchdog TerminateOnTimeout only arms its idle timer on the first received message. The hubs in these tests deliberately never complete (so a seized graph is distinguishable from exhausted input), leaving the idle terminator as the only exit path -- but it is fed from the merged output, so a run that produces zero output messages never arms it and ez.run blocks until the CI job timeout. This is timing-dependent (resampler warmup + merge alignment vs. the early glitch), matching the semi-frequent CI hangs on this file. - ArmedTerminateOnTimeout initializes last_msg_timestamp at startup, so a zero-output run tears down after the 4 s idle window and fails on the test's assertions instead of hanging. - A second, unconnected armed terminator acts as a 30 s absolute deadline in case future wiring defeats the idle terminator.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the semi-frequent CI hangs on
test_resample_merge_system.py.Root cause
TerminateOnTimeout.poll_terminateonly checks the idle gap oncelast_msg_timestampis set — i.e. the idle timer arms only on the first received message. The hubs in these tests deliberately never complete (while True: await asyncio.sleep(0.1), from the earlier de-flake, so a seized graph is distinguishable from exhausted input), which leaves the idle terminator as the graph's only exit path. But it is fed from the merged output: a run that produces zero output messages never arms the timer, andez.runblocks until the CI job timeout.Zero-output runs are timing-dependent and exactly what these tests court: the seize test's glitch lands ~30 ms in, and on a loaded runner the resampler warmup + merge alignment can lose the race, so no merged message precedes the seize. That matches the semi-frequent, CI-only signature.
Fix (belt and suspenders)
ArmedTerminateOnTimeout— setslast_msg_timestampininitialize(), so an all-idle graph tears down after the 4 s idle window and the test fails on its assertions (n_msgs > 0/last_t = -inf) instead of hanging for hours.If the seize test ever fails on a slow runner post-fix (zero output before the glitch), the remedy is a later
glitch_ator largerdispatch_dt— visible failure beats a silent hang.Worth considering upstream:
TerminateOnTimeoutarming at startup inezmsg.util.terminateitself — an idle terminator that cannot fire from the all-idle state is a footgun for any graph shaped like this one.All 4 tests pass locally (~18.5 s total).