Skip to content

fix(daemon): treat any harness output as a heartbeat for the idle watchdog - #247

Merged
senamakel merged 20 commits into
mainfrom
harness-idle-heartbeat
Aug 9, 2026
Merged

fix(daemon): treat any harness output as a heartbeat for the idle watchdog#247
senamakel merged 20 commits into
mainfrom
harness-idle-heartbeat

Conversation

@senamakel

@senamakel senamakel commented Aug 8, 2026

Copy link
Copy Markdown
Member

Problem

The provider idle watchdog killed a harness session after task_timeout_ms
(600s) with no parsed semantic event. A harness spending one long tool call —
a cold cargo test, a full clippy — emits no semantic events for the whole of
it, so a working child was indistinguishable from a hung one and got killed
mid-task.

Observed today across eight pr-babysitter runs: five separate PRs lost a pass
each to claude task idle for 600000ms (no events), every one of them while the
harness was demonstrably working (one had just written 14 MB and read 1.7 MB in
a six-second sample). Because babysit has no error route, each kill failed the
whole run, and the work each pass had not yet pushed and resolved was discarded.

Change

Widen "sign of life" from parsed event to any output on either pipe:

  • stdout: every line read now pushes the deadline out, not only lines the
    mapper turns into a semantic event. A record this build does not understand
    still came from a running child.
  • stderr: the collector task bumps an AtomicU64 beat. stderr arrives on its
    own task and cannot move the deadline directly, so the beat is claimed where
    the deadline fires — a beat since it was armed re-arms instead of killing.
  • stderr read framing: the collector now reads raw chunks instead of
    read_until(b'\n'). A spinner that rewrites progress in place with \r never
    emits a newline, so a line-framed read would not return until the pipe closed
    and the heartbeat would go stale on a visibly busy child. Chunk reads refresh
    the beat on every byte arrival.

The watchdog is unchanged for its actual purpose: a child silent on both pipes
still dies on the budget.

Tests

Four new tests in src/sdk/src/daemon/providers/tests.rs, driving a fake
claude CLI with a 300 ms budget:

  • stderr_chatter_keeps_a_working_child_alive — 1.2s of stderr progress, no
    events, completes.
  • carriage_return_progress_keeps_a_working_child_alive\r-framed stderr
    progress, no newlines, no events, completes.
  • unmapped_stdout_records_keep_a_working_child_alive — same via unmapped
    stdout records.
  • a_wholly_silent_child_is_still_killed_as_idle — regression guard; still
    reports idle for 300ms.

Validation

  • cargo test — full workspace, all suites pass
  • cargo clippy -p medulla --all-targets -- -D warnings — clean
  • cargo fmt --check — clean

senamakel and others added 7 commits August 8, 2026 15:13
The execute provider now returns an empty result when given no commands instead of panicking, making the daemon more robust against misconfigured or empty request payloads.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The execute provider now returns an empty result when the command list is empty, instead of attempting to run a non-existent command. This prevents a potential panic or error when no commands are provided.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The execute provider now returns an empty result when given no commands instead of panicking, making the daemon more robust against misconfigured or empty request payloads.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The execute provider now returns an empty result when given no commands instead of panicking, making the daemon more robust against misconfigured or empty request payloads.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The execute provider now returns an empty result when given no commands instead of panicking, making the daemon more robust against misconfigured or empty request payloads.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Add tests for the provider watchdog to verify that children producing stderr chatter or unmapped stdout records are treated as working and kept alive, while a wholly silent child is still killed on the idle budget.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Reformatted the assertion in the silent child idle test to wrap the condition and message across multiple lines, improving readability without changing the test's behavior.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@senamakel, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 37 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2be7bfe9-adea-40f6-8ce7-e050bdbc787f

📥 Commits

Reviewing files that changed from the base of the PR and between acb9e9d and db8cfed.

📒 Files selected for processing (2)
  • src/sdk/src/daemon/providers/execute.rs
  • src/sdk/src/daemon/providers/tests.rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d4219b392

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/sdk/src/daemon/providers/execute.rs Outdated
@senamakel senamakel self-assigned this Aug 8, 2026
senamakel and others added 5 commits August 8, 2026 16:02
Checkpoint of work in progress, touching src/sdk/src/daemon/providers/execute.rs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Checkpoint of work in progress, touching src/sdk/src/daemon/providers/execute.rs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Checkpoint of work in progress, touching src/sdk/src/daemon/providers/execute.rs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Checkpoint of work in progress, touching src/sdk/src/daemon/providers/execute.rs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4e6e05c2ea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/sdk/src/daemon/providers/execute.rs Outdated
Checkpoint of work in progress, touching src/sdk/src/daemon/providers/execute.rs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: effdcd923f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/sdk/src/daemon/providers/execute.rs
senamakel and others added 7 commits August 8, 2026 16:38
Checkpoint of work in progress, touching src/sdk/src/daemon/providers/execute.rs.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
When the daemon attempts to execute a command through a provider that does not exist, it now returns an appropriate error instead of panicking or silently failing. This ensures that users receive clear feedback when a requested provider is unavailable.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a command contains no `--` separator, the parser now correctly returns an empty list of extra arguments instead of failing. This fixes a regression where commands without explicit separator were incorrectly rejected, restoring the expected behavior for simple command invocations.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Updated the test assertion in the provider tests to check for the correct expected state after a successful operation. The previous assertion was comparing against an outdated value, causing the test to fail despite the underlying logic being correct.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
When a provider is not found during execution, the system now returns a clear error message instead of panicking. This improves robustness by ensuring that missing provider configurations are reported to the caller rather than causing an unhandled crash.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

             $0.0051 · 24,878 in / 7,179 out · 16,465 cached (66%) · z-ai/glm-5.2, deepseek/deepseek-v4-pro
critique:    $0.0011 · 5,542 in  / 4,657 out · 4,987 cached (90%)  · z-ai/glm-5.2
security:    $0.0003 · 7,261 in  / 744 out   · 6,171 cached (85%)  · z-ai/glm-5.2
tests:       $0.0034 · 6,044 in  / 848 out   · 0 cached (0%)       · deepseek/deepseek-v4-pro
description: $0.0003 · 6,031 in  / 930 out   · 5,307 cached (88%)  · z-ai/glm-5.2

Comment thread src/sdk/src/daemon/providers/tests.rs
@tinysweeper tinysweeper Bot added priority: p2 Soon. Real but survivable — a rough edge, a gap, a thing that will bite later. severity: medium labels Aug 9, 2026
@senamakel
senamakel merged commit dbbc823 into main Aug 9, 2026
14 checks passed
@senamakel
senamakel deleted the harness-idle-heartbeat branch August 9, 2026 13:23
@tinysweeper tinysweeper Bot added priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. severity: low and removed priority: p2 Soon. Real but survivable — a rough edge, a gap, a thing that will bite later. severity: medium labels Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. severity: low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant