run_smoke: cap each entry and kill its process group on expiry - #146
Merged
Conversation
Both legs of this runner ran uncapped: run_script and execute_notebook each
called subprocess.run(capture_output=True) with no timeout at all. When an
entry leaves a grandchild holding the inherited stdout pipe, the parent
blocks on that pipe forever — even though the entry's own work finished and
the direct child exited. The gate then sits until the 6-hour GitHub Actions
ceiling, reporting nothing since the last completed entry. That is
autolens_workspace_test#196; this runner had the same shape in two places.
Reproduced against this exact file, then re-run after the change:
script leg — a script spawning a grandchild and exiting: old runner
printed "::group::script: hangs.py" and nothing more, still
hung when killed at 30s with BUILD_SCRIPT_TIMEOUT=4 set.
New: "[TIMEOUT (4s)] hangs.py — 4.0s", exit 1, no survivors.
notebook leg — a jupyter that hangs and spawns a grandchild: new runner
reports TIMEOUT (4s) and reaps the group. The
jupyter-not-found path still reports exit 127 with its own
message, unchanged.
Adopts what autolens_workspace_test already runs: the cap resolved per entry
via timeout_for(env), so a profile's BUILD_SCRIPT_TIMEOUT override is read
parent-side where the kill timer lives; the child in its own session so the
whole group dies; exit 124 rather than the signal, because a timeout means
"raise the cap or SLOW-skip it" and -9 would mislabel it an ordinary failure.
timeout_for and kill_group come from PyAutoHands build_util — guarded imports
with local fallbacks, so an older Hands on PYTHONPATH cannot break the gate.
A notebook timeout deliberately skips the regenerate-and-retry path: a
timeout is not a stale notebook, and retrying would burn a second full cap to
reach the same answer.
Also drops the vestigial _BUILD_DIR intermediate in autogalaxy_workspace,
the last real drift between the three copies of this runner; all three are
now byte-identical again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTPM1RmMvSuMvJkEntAMv8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leg A of the per-script timeout backport. Depends on PyAutoHands#257, which adds the
kill_grouphelper this imports — merge that first (the imports are guarded, so an older Hands falls back rather than breaking the gate).The defect
Both legs of this runner ran uncapped:
run_scriptandexecute_notebookeach calledsubprocess.run(capture_output=True)with notimeout=at all. When an entry leaves a grandchild holding the inherited stdout pipe, the parent blocks on that pipe forever — even though the entry's own work finished and the direct child exited. The gate then sits until the 6-hour GitHub Actions ceiling, reporting nothing since the last completed entry. That is autolens_workspace_test#196; this runner had the same shape in two places.This variant was not previously identified as exposed — the earlier survey framed the missing cap as a
workspace_test-tier divergence. It is six repos wide, and these three are the gates on the public workspaces.Reproduced, against this exact file
With
BUILD_SCRIPT_TIMEOUT=4:::group::script: hangs.pyand nothing more; still hung when killed at 30s[TIMEOUT (4s)] hangs.py — 4.0s, exit 1, zero survivorsjupyter nbconvert --execute[TIMEOUT (4s)], group reaped including the grandchild the fake jupyter forkedThe jupyter-not-found path still reports exit 127 with its own message, unchanged.
Notes on the notebook leg
A notebook timeout deliberately skips the regenerate-and-retry path. A timeout is not a stale notebook, and retrying would burn a second full cap to reach the same answer — doubling the slowest entry's cost.
Also
Drops the vestigial
_BUILD_DIRintermediate inautogalaxy_workspace(the sibling PR), the last real drift between the three copies of this runner. All three are byte-identical again after this series.Generated by Claude Code