fix(signals): Ctrl-C and SIGTERM do not stop the loop - #345
Open
Sofoklis-byte wants to merge 4 commits into
Open
fix(signals): Ctrl-C and SIGTERM do not stop the loop#345Sofoklis-byte wants to merge 4 commits into
Sofoklis-byte wants to merge 4 commits into
Conversation
The SIGINT/SIGTERM trap called cleanup(), which returns instead of exiting, so bash resumed the main loop: the operator saw "Ralph loop interrupted. Cleaning up..." and the loop kept running. Five faults in one control: 1. trap cleanup SIGINT SIGTERM — the handler never terminated. 2. The comment claimed "EXIT trap handles natural termination". There is no EXIT trap anywhere in the file; that trap line was the only one. 3. The _CLEANUP_DONE reentrancy guard, documented as covering an "EXIT + signal combination" that does not exist, made every signal after the first a silent no-op. 4. cleanup() read $? as trap_exit_code. In a trap handler that is the last completed command's status, not 128+n, so the "interrupted" status was recorded only when the preceding command happened to fail. 5. The claude child is backgrounded with a `local claude_pid`, invisible to the handler, so a SIGTERM to the script left the child running. Fix: a dedicated on_signal() that restores default dispositions first (so a second Ctrl-C always kills even if cleanup hangs), stops the child via a new CLAUDE_CHILD_PID global, runs cleanup, then re-raises the signal so the process dies with the correct 128+n status. _INTERRUPTED makes the interrupted-status record deterministic. Proof: 7/7 in --dry-run (no API call), each patched case paired with the unpatched original as a known-false control. SIGINT to pgroup patched DEAD 1s (130) · original ALIVE at 15s SIGTERM to pid patched DEAD 2s (143) · original ALIVE at 15s SIGINT x4 patched DEAD on first · original ALIVE, Loop frankbria#3 no signal patched still looping — normal path unaffected The original's control log reproduces the reported symptom exactly: "interrupted. Cleaning up..." followed by "Completed Loop frankbria#1". Not covered: dry-run returns before spawning a child, so the CLAUDE_CHILD_PID kill leg is reasoned, not measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five tests, each verified to FAIL against the unpatched loop and pass
against the fix:
- SIGTERM terminates the loop (exit 143)
- SIGINT terminates the loop (exit 130)
- repeated signals do not disarm the handler
- the bare `trap cleanup SIGINT SIGTERM` form is gone
- the child pid is reachable from the handler
All run under --dry-run, so no API calls are made.
Two harness notes worth keeping, both of which produced false results
before they were understood:
- A background job started from a non-interactive shell inherits SIGINT
as SIG_IGN, and bash cannot re-trap an inherited-ignored signal. A
naive harness shows the loop surviving Ctrl-C whether or not the bug
is present, which proves nothing. The perl wrapper resets SIGINT to
default before exec, as an interactive terminal would.
- `wait` is used rather than a kill -0 poll: a dead-but-unreaped child
still answers kill -0. A watchdog bounds the wait so a hung loop
cannot wedge the suite.
A sixth test was written and dropped: it asserted that no further
iteration is logged after the "interrupted" line, but that line is only
printed when the preceding command happened to fail, so under --dry-run
it never appears and the assertion matched nothing. It passed against
the buggy code and was therefore worthless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ctrl-C returned the prompt but the agent kept working and kept billing.
Measured mid-work under a real pty:
loop pid 94095 pgid 94095 <- foreground group; gets the Ctrl-C
bash pid 94100 pgid 94095 <- $! captured THIS, not the agent
gtimeout pid 94107 pgid 94107 <- gtimeout setpgid()s ITSELF
claude pid 94112 pgid 94107 <- the agent, in that other group
portable_timeout is a shell function, so backgrounding it forks a subshell;
bash sometimes execs gtimeout in its place and sometimes does not, varying
with execution context, so $! does not reliably name the agent. Ctrl-C
reaches only the foreground group, so the agent never sees it; on_signal
then TERMs the subshell and gtimeout plus the agent are orphaned and run on.
A process-group kill is not the answer either: the agent is not in our
group, so -$PGID signals the loop itself and spares the agent.
Add list_descendants() and kill_tree(), and call kill_tree from on_signal.
Membership is proven by parentage and enumerated BEFORE anything is
signalled -- once the root dies its descendants reparent to launchd and no
later walk can find them. Deepest-first TERM, 5s grace, then SIGKILL to
survivors; never signals itself, an ancestor, or a process group.
Written for bash 3.2, which #!/bin/bash on macOS means it must be: no
mapfile, no readarray, no array expansions that abort under set -u.
Evidence, control vs fixed:
real agent, real Ctrl-C 3 files after the stop, ran to completion,
$0.242 -> 0 files, killed at 5 turns, $0.105
zero-cost stub, repeated fails 3/3 -> passes 3/3
library selftest (3.2) control fails as it must -> 6/6
Not addressed here: CLAUDE_CHILD_PID is assigned only in the background
branch, so the child-kill leg is a no-op under --live, where the agent runs
as a foreground pipeline and is never registered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven tests, each verified to FAIL against unpatched main at 0c1d7bf and pass against the fix: list_descendants finds a child and a grandchild through a wrapper a single-pid TERM leaves the tree alive (known-false control) kill_tree clears the whole tree from a live root kill_tree reaches a descendant in a DIFFERENT process group kill_tree does not kill its own caller kill_tree escalates to SIGKILL past a TERM-ignorer the signal handler stops the tree, not a single pid They source the shipped ralph_loop.sh in a subshell, so the functions under test are the ones that ship and the loop's own traps never land in the bats shell. Trees are built from sleep; no API calls, no cost. The control is inside the suite rather than in a commit message: test 2 asserts the OLD single-pid TERM still fails to stop a tree. If it ever goes green, the kill_tree tests are proving nothing and it fails loudly. Test 4 is the production shape. gtimeout setpgid()s itself, so the agent is not in the loop's group; the test asserts the fixture actually straddles a group boundary before asserting the tree died, or it would pass on a fixture that never tested the thing. Two harness notes, both of which produced wrong results first: - Test 5 originally passed against unpatched main. kill_tree does not exist there, so nothing died and the caller trivially survived. It now also asserts the target died. A test that goes green against the buggy code is worthless -- the same fault this suite dropped a sixth test for. - Cleanup must not be a test's last command: killing an already-dead pid exits non-zero and fails the test for the wrong reason. Suite: 1286 tests, 13 failures, the same 13 pre-existing platform failures present on unpatched main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ctrl-C does not stop Ralph. It prints
Ralph loop interrupted. Cleaning up..., then the loop continues to the next iteration. Repeated Ctrl-C does nothing at all, andSIGTERMdoes not stop it either. In the run that surfaced this, onlykill -9ended the process — about seven minutes after the first stop attempt.The reassuring log line is the dangerous part: an operator watching an autonomous loop believes it has stopped when it has not.
Root cause
trap cleanup SIGINT SIGTERMinstalls a handler that returns instead of exiting.cleanup()ends with:# No exit here — EXIT trap handles natural terminationThere is no EXIT trap. That
trapline is the only one in the file. The comment describes a handler that does not exist, so nothing ever terminates the process, and bash resumes the loop as soon as the handler returns.Four further faults sit behind that one:
cleanup()returns; the signal handler therefore never terminates.EXIT trapthe comment relies on does not exist anywhere in the file._CLEANUP_DONEreentrancy guard — documented as covering an "EXIT + signal combination" that cannot occur — makes the second and every later signal a silent no-op. This is why repeated Ctrl-C cannot recover.cleanup()reads$?astrap_exit_code. In a trap handler that is the last completed command's status, not128+n, so theinterruptedstatus is recorded only when the preceding command happened to fail. Under--dry-runit never prints.claudeis backgrounded and its pid is alocal claude_pid, invisible to the handler, so aSIGTERMsent to the script leaves the child running.Fix
A dedicated
on_signal()replaces the bare trap:CLAUDE_CHILD_PID— this leg proved insufficient and is corrected by28cf3ca; see Follow-up below;cleanup()for teardown, unchanged in meaning;128+nstatus rather than falling out of a loop — anything supervising Ralph now sees a real interrupt;_INTERRUPTED, which makes the interrupted-status record deterministic rather than dependent on the previous command's exit code.cleanup()keeps its single-run guard and its existing behaviour. Normal (unsignalled) operation is unchanged.Tests
tests/integration/test_signal_handling.bats— five tests, each verified to fail against the unpatched loop and pass against the fix. All run under--dry-run, so they make no API calls.Two harness notes are recorded in the test file, because both produced false results before they were understood:
SIG_IGN, and bash cannot re-trap an inherited-ignored signal. A naive harness shows the loop surviving Ctrl-C whether or not the bug is present — it proves nothing. The tests reset SIGINT to default beforeexec, as an interactive terminal does.waitis used rather than akill -0poll, because a dead-but-unreaped child still answerskill -0. A watchdog bounds the wait so a hung loop cannot wedge the suite.A sixth test was written and dropped: it asserted that no further iteration is logged after the
interruptedline, but per fault 4 that line is not printed under--dry-run, so the assertion matched nothing and passed against the buggy code. It is mentioned here so it is not re-added later in good faith.Follow-up — the child-kill leg was insufficient (
28cf3ca)The original fix reached the child pid, and the tests above assert exactly that: the child pid is reachable from the handler. Reaching it turned out not to be enough. In production the operator pressed Ctrl-C, got the prompt back, and the agent wrote a file 2s later, committed 17s later, and finished 47s after the stop, billing the call in full.
Measured mid-work under a real pty:
portable_timeoutis a shell function, so backgrounding it forks a subshell; bash sometimes replaces that subshell with an exec ofgtimeoutand sometimes does not, varying with execution context.$!therefore does not reliably name the agent. Two consequences:TERMing$!kills the subshell;gtimeoutand the agent are orphaned and keep working.A process-group kill is not the answer either: the agent is not in our group, so
-$PGIDsignals the loop itself and spares the agent. (Measured — a candidate doing this killed the supervisor and left the target running.)28cf3caaddslist_descendants()andkill_tree()and calls the latter fromon_signal(). Membership is proven by parentage and enumerated before anything is signalled — once the root dies its descendants reparent tolaunchdand no later walk can find them. Deepest-firstTERM, 5s grace, thenSIGKILLto survivors; it never signals itself, an ancestor, or a process group. Written for bash 3.2, which#!/bin/bashon macOS requires: nomapfile, noreadarray, no array expansions that abort underset -u.Control vs fixed:
28cf3ca)Regression tests (
99f2323).tests/integration/test_process_tree_kill.bats— seven tests, each verified to fail against unpatchedmainat0c1d7bfand pass against the fix. They source the shippedralph_loop.shin a subshell, so the functions under test are the ones that ship; trees are built fromsleep, so there are no API calls.The control lives inside the suite: test 2 asserts the OLD single-pid TERM still fails to stop a tree, so if it ever goes green the
kill_treetests are proving nothing and it says so. Test 4 asserts the fixture actually straddles a process-group boundary before asserting the tree died.main—kill_treedoes not exist there, so nothing died and the caller trivially survived. It now also asserts the target died. This is the same fault the sixth signal test was dropped for, and it is recorded rather than quietly fixed.What this evidence is still not. The behavioural proof of the original defect came from an external pty harness and a faithful miniature of the loop's spawn-and-signal architecture. The control has never been exercised against a live
ralph_loop.shend to end.Known gap, not addressed here.
CLAUDE_CHILD_PIDis assigned only in the background branch. Under--livethe agent runs as a foreground pipeline and is never registered, so the child-kill leg is a no-op in that mode.Test suite status
Re-measured on macOS at
28cf3ca, not carried from an earlier run:npm testat99f2323npm testat unpatchedmain0c1d7bfnpm run test:e2eat28cf3caThe 13 failures are identical in name and number across both runs, so this PR introduces none of them. They are platform failures (
head: illegal line count -- -1, a Linux-onlynotify-sendcase, a BSDstatfallback case), none in code this PR touches. The count differs by 12 because this PR adds 5 signal tests and 7 process-tree tests.The e2e suite runs
ralph_loop.shas a real subprocess and includes termination signal during execution records interrupted status and preserves call count, which exercises the signal path this PR changes.Reproducing the bug