Add opt-in unattended resilience to the Pane Chat orchestrator skill - #609
Conversation
Both generated variants (runpane-orchestrator.md guide and the pane-orchestrator SKILL.md mirrored to Codex, Claude and Cursor) now ask one startup question after doctor and the watcher self-test, defaulting to yes, and carry a shared "Unattended resilience (when enabled)" section: caffeinate keep-awake with the battery/clamshell caveat, auto-resume of orchestrator-dispatched panes whose turn died from a sleep or network API error, guardrails (never BLOCKED panes, max 3 resumes per pane per hour, log every resume, hard stops unchanged), and a watcher self-test re-run after a detected wake. Files are rewritten unconditionally on every ensurePaneChatGuide, so existing installs pick the text up on next launch; the bootstrap prompt is unchanged so PANE_CHAT_BOOTSTRAP_VERSION stays at 9. Claude-Session: https://claude.ai/code/session_0171TCXkRCPujMN7aff1LHuc
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c3b64cbe2e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - Resume only when the composer is empty (the screen payload reports | ||
| \`composer.hasUndeliveredText: false\`) and the tail shows a | ||
| sleep/network death signature, one of: |
There was a problem hiding this comment.
For Claude and Cursor panels, detectPanelComposer in main/src/ipc/runpane.ts unconditionally returns hasUndeliveredText: false, so this guard passes even when a user has typed an unsent draft. On the next READY/IDLE event with a matching error still visible, panels submit appends the resume text and presses Enter, potentially submitting or corrupting the user's draft; exclude agents without reliable composer detection or add a real held-input check before auto-resuming.
Useful? React with 👍 / 👎.
| - On a READY or IDLE line for a pane you dispatched, read | ||
| \`runpane panels screen --panel <panel-id> --limit 80 --json\`. | ||
| - Resume only when the composer is empty (the screen payload reports | ||
| \`composer.hasUndeliveredText: false\`) and the tail shows a | ||
| sleep/network death signature, one of: |
There was a problem hiding this comment.
This searches an arbitrary 80-line current-screen snapshot after every READY or recurring IDLE event without tying the signature to the turn that just ended. After a pane successfully recovers, the earlier API error can remain visible above a short success response, causing the orchestrator to resume an already-completed pane repeatedly—up to three times per hour and again in later hours. Correlate the signature with the newest terminal outcome or record that the specific failure has already been handled.
Useful? React with 👍 / 👎.
| - Start \`caffeinate -dims\` in the background for the session and | ||
| record its PID. Kill that PID at session end. | ||
| - caffeinate cannot stop clamshell sleep on battery. Run | ||
| \`pmset -g batt\` once; if it reports battery power, tell the user | ||
| once to plug in and keep the lid open. |
There was a problem hiding this comment.
When resilience defaults to enabled on supported Linux or Windows installations, the generated instructions still require caffeinate and pmset, which are macOS-only commands; initialization therefore produces command-not-found failures and provides no keep-awake behavior. Add an explicit Darwin check and either skip this subsection or use a supported platform-specific mechanism elsewhere.
AGENTS.md reference: AGENTS.md:L13-L13
Useful? React with 👍 / 👎.
| - Then send a carriage return, because submit alone often only pastes: | ||
| \`printf '\\r' | runpane panels input --panel <panel-id> --input-file - --yes --json\`. |
There was a problem hiding this comment.
panels submit
The checked-in RunPane contract at contracts/runpane/contract.json:6247-6250 states that panels submit already stages text, sends the appropriate submit sequence, and verifies startup; sending another carriage return after it succeeds injects an extra keystroke into the now-running agent. The same contract's notes at lines 6297-6300 explicitly prohibit following this atomic command with another composer submission, so inspect the returned ok/blocked state instead of unconditionally pressing Enter.
AGENTS.md reference: AGENTS.md:L86-L86
Useful? React with 👍 / 👎.
Keep-awake now names three tiers from pmset -g log evidence: caffeinate covers lid-open idle sleep only; lid closed on AC needs `sudo pmset -c disablesleep 1`, which the orchestrator prints once for the user and continues without; battery in a bag relies on Power Nap plus TCP keepalive dark wakes, so auto-resume must be idempotent and fast, and startup checks `pmset -g custom` for powernap and tcpkeepalive without changing them. Cold-read fixes: the default-yes rule no longer waits for an answer and a later "no" kills caffeinate; BUSY joins the Liveness Contract key lines and IDLE is described per pane; the shared section points at the skill's Liveness Contract instead of assuming one in the guide; a HEARTBEAT gap that ends with queued lines is a wake, not a dead watch, and does not spend the re-arm; STUCK stays with the resubmit rule; signatures inside file or tool output do not count; a missing composer field means report, not resume; the retry ladder ends after one submit-composer. Claude-Session: https://claude.ai/code/session_0171TCXkRCPujMN7aff1LHuc
On AC the Mac must never deep-sleep with the lid closed, so remote control and the panes keep running. Tier 1 now names the AC-profile setting `sudo pmset -c disablesleep 1`, verified with `pmset -g | grep SleepDisabled`, and gives the orchestrator a four-step startup routine: check the setting; if 0, tell the user to run `! sudo pmset -c disablesleep 1` in the chat and note the revert; optionally offer the one-time NOPASSWD rule for /usr/bin/pmset so later sessions can apply it with `sudo -n`; re-check battery and the setting after any wake and remind once. Tier 2 (battery dark wakes) unchanged. Claude-Session: https://claude.ai/code/session_0171TCXkRCPujMN7aff1LHuc
1 participant
Description
A long unattended
/dorun died twice last night: the MacBook slept on battery with the lid closed, the Anthropic API became unreachable (ENOTFOUND), the in-flight review subagent stalled, and the turn ended. Pane Chat recovered it without any human input: when the watcher's queued events reached it during a later wake, it read the pane, saw the sleep and ENOTFOUND errors, and sent the pane a resume message on its own judgment. The skill text did not yet describe that move. This PR makes it a standard, opt-in behaviour of the generated orchestrator skill.Not exercised end to end. No live Pane Chat session has been put through sleep and recovery with this text. The tests prove the text is emitted, not that an agent following it recovers. The startup question will appear on every Pane Chat launch for every install after upgrade.
The change is text only, in
main/src/services/skillCacheManager.ts. Two shared constants (the startup prompt and the resilience section) are embedded in both emitted variants so those two blocks cannot drift. The variants are otherwise different documents:<PANE_DIR>/skills/pane-chat/runpane-orchestrator.md), which has no Liveness Contract of its own and points at the skill's<PANE_DIR>/skills/pane-chat/pane-orchestrator/SKILL.md), which owns the Liveness Contract and is mirrored byte-for-byte to<PANE_DIR>/.codex/skillsand<PANE_DIR>/.claude/skillsand converted to the.cursor/rulesfileStartup question. In Initialize, after doctor and
runpane watch --self-test, the orchestrator asks once per session (the text notes it lives in both files and one ask covers both): "Enable unattended resilience for this session? (keeps the Mac awake with caffeinate, auto-resumes panes whose turn died from a sleep/network API error, re-arms the watcher). Default: yes." The default is yes and the orchestrator does not wait for an answer. Any other prompt counts as yes, said in one line. An explicit "no" at any point disables it for the rest of the session and kills caffeinate if running.New section "Unattended resilience (when enabled)".
pmset -g logevidence. Lid open:caffeinate -dimsin the background, PID recorded and killed at session end; this covers idle sleep only. Lid closed on AC: the Mac must never deep-sleep with the lid closed on AC so remote control and the panes keep running. caffeinate cannot prevent clamshell sleep without an external display, so the mechanism is the AC-profile settingsudo pmset -c disablesleep 1(-cscopes it to the charger profile, battery behaviour unchanged; verify withpmset -g | grep SleepDisabled). With it on, closing the lid keeps the machine fully awake. The orchestrator cannot sudo, so at startup it checks the setting, tells the user in one line to run! sudo pmset -c disablesleep 1in the chat if it is 0 (noting the revert), optionally offers a one-time NOPASSWD sudoers rule for/usr/bin/pmsetso later sessions can apply it withsudo -n, and re-checkspmset -g battand the setting after any wake to remind once. Battery in a bag: nothing keeps the Mac awake; Power Nap plus TCP keepalive give short dark wakes every 5 to 15 minutes, pane agents retry inside those windows, and the run resumes when Wi-Fi is in range. The orchestrator relies on that by keeping auto-resume idempotent and fast, checkspmset -g customat startup and warns once ifpowernaportcpkeepaliveis 0 without changing them, and tells the user once if on battery that plugged in with the lid open is the only fully awake setup.runpane panels screen. Resume only whencomposer.hasUndeliveredTextis false (missing field means report, not resume) and the last thing the agent printed is one of the death signatures (sleep mid-response, can't reach the API server, ENOTFOUND, agent stalled, terminated early due to an API error, retry attempts exhausted). Submit a resume message viarunpane panels submitnaming the failure and pointing at the earliest incomplete gate of the runpane-orchestrator lifecycle, then a carriage return viarunpane panels input, then confirm via screen that the resume message is no longer held and the agent is working. If the resume message is still held,panels submit-composeronce, then report rather than retry. The whole sequence runs in one pass so it fits inside a wake window.runpane watch --self-testbefore trusting new lines and do not spend the re-arm. Each wake resets the re-arm allowance. A gap with nothing arriving while the Mac is awake is a dead watch.Two small edits outside the shared blocks: the skill's Liveness Contract key-lines list now includes BUSY (the watcher already emits it) and describes IDLE as the pane being quiet rather than nothing dispatched, and the guide's Initialize list is introduced as "Do these" since it mixes reads and actions.
Hard stops are untouched. The watcher in
packages/runpaneis untouched.Upgrade path.
writePaneChatGuiderewrites every target withfs.writeFileon everyensurePaneChatGuide, which runs at app boot and at every Pane Chat launch. There is no hash or version gate, so existing installs receive the new text on next launch. The bootstrap prompt text is unchanged, soPANE_CHAT_BOOTSTRAP_VERSIONstays at 9. A Pane Chat panel that is already open keeps its in-context copy until its terminal restarts.Type of Change
Testing
main/src/services/skillCacheManager.test.tsgains two tests:ensurePaneChatGuide, and asserts each was overwritten with the new sectionRan locally on the final diff:
pnpm lintpassespnpm typecheckpassespnpm --filter main exec vitest run: 960 passed, 2 skippedAlso rendered the generator into an isolated
PANE_DIRand read the resulting files by eye: backticks, theprintf '\r'line and theecho $!PID capture render correctly.Review notes: a Codex code-reviewer pass flagged that the guide's Initialize had no self-test step before the question; fixed. A Claude implementation-reviewer pass found no must-fix items; its polish items are applied. A zero-context cold read of the PR body and the rendered files then drove the wording fixes above: the unanswered-question case, BUSY missing from the key lines, the guide referencing a Liveness Contract it does not contain, the 120s gap meaning both wake and dead watch, STUCK versus auto-resume, signature matches inside file output, the missing composer-field fallback, and the open-ended retry ladder.
Checklist
pnpm typecheckandpnpm lintlocallypnpm electron-devCritical Areas Modified
None.
Additional Notes
The 3-per-hour resume cap and the set of dispatched panes live in the orchestrator's own memory and reset if Pane Chat restarts. That is as specified; a durable counter would need state outside the skill text.
https://claude.ai/code/session_0171TCXkRCPujMN7aff1LHuc