research: graceful restart improvements — reduce downtime and user impact

[nathanschram]

Context

Research into whether Untether's restart experience can be improved beyond the current graceful drain approach. Investigated zero-downtime patterns, Telegram-specific constraints, and subprocess preservation techniques.

TL;DR: True zero-downtime restart is impossible for our architecture (Telegram long-polling + long-lived subprocesses). But several practical improvements can reduce the gap from ~15-30s to ~5s and improve UX significantly.

Current approach (already good)

Untether already implements:

• Graceful drain: on SIGTERM, waits up to 120s for active runs to finish
• Per-chat "Restarting — waiting for your run to finish…" notifications
• Timeout notification: "Restart timed out — N run(s) interrupted"
• Progress persistence: orphan messages edited to "⚠️ interrupted by restart" on next startup
• KillMode=mixed + TimeoutStopSec=150 for clean subprocess cleanup

Why true zero-downtime is impossible

Telegram constraint: Only one process can call getUpdates per bot token. A second caller gets 409 Conflict. No framework (python-telegram-bot, aiogram, telethon) has solved this — they all recommend either "accept the gap" or "use webhook mode."

Subprocess constraint: Active Claude Code runs communicate via PTY FDs and stdout pipes. When the parent process exits, these FDs are destroyed. Reconnecting to a running subprocess's JSONL output stream after parent death requires either:

• SCM_RIGHTS fd transfer over Unix socket (complex, fragile)
• os.execv() with inheritable FDs (destroys asyncio event loop, requires full state serialisation)
• pidfd_getfd (Linux 5.6+, requires ptrace permissions)

None of these are production-ready patterns for our use case.

Practical improvements (recommended)

Tier 1: Low effort, high value

1. Persist update_id offset
Write the last confirmed Telegram update_id to a state file. On restart, resume from offset=last_id+1. Telegram holds undelivered updates for 24 hours. This means zero lost messages — just a brief polling gap (~5-15s).

Currently, the new process starts with offset=0 and might re-process recent updates (deduplication prevents issues, but it's wasteful).

2. Type=notify systemd service
Use sd_notify(READY=1) after the event loop starts and first getUpdates succeeds. This tells systemd the new process is actually healthy, not just "PID exists." Also enables STOPPING=1 during drain for better status reporting. Library: async-sdnotify on PyPI (or stdlib socket with $NOTIFY_SOCKET).

3. Reduce RestartSec
Currently uses systemd default (100ms). Could explicitly set RestartSec=2 for faster restart after drain completes.

Tier 2: Medium effort, targeted value

4. Socket activation for webhook/trigger server
Define untether.socket listening on the trigger port. During restarts, kernel queues incoming webhook HTTP requests. Zero dropped webhooks. Requires:

• untether.socket unit file
• Detect LISTEN_FDS env var in trigger server startup
• Use web.SockSite(runner, sock) instead of web.TCPSite

5. Better drain UX

• Broadcast restart notice to all active chats (not just those with running tasks) — "🔄 Untether is restarting, back in ~10s"
• /restart --force option to skip drain wait
• Per-run drain priority — let short-running tasks finish, cancel long ones first
• Drain progress in Telegram — edit a message with countdown: "Waiting for 2 runs to finish (45s remaining)"

6. Pre-restart state snapshot enhancement
Before exiting, write richer state: active session IDs, resume tokens, engine types, run durations. The new process reads this on startup and can:

• Show "Your previous run was interrupted. Resume with /continue" with session details
• Provide better resume guidance (which topic, which engine)

Tier 3: Future consideration (high effort)

7. Webhook mode for planned restarts
On /restart, temporarily call setWebhook pointing to the trigger HTTP server, restart, then deleteWebhook to resume polling. Eliminates the polling gap entirely for planned restarts. The trigger aiohttp server is already running. Complex but technically feasible.

8. Subprocess output redirection
Before exiting, redirect each child's stdout to a temp file (via /proc/<pid>/fd/), persist the path, and have the new process tail -f to resume JSONL translation. Fragile but would preserve runs across restarts.

What NOT to pursue

• Blue-green dual-process: Impossible — Telegram enforces single poller per token
• os.execv() in-place restart: Destroys asyncio event loop, requires full state serialisation
• reptyr terminal reattachment: Works for terminals, not for stdout pipes
• Redis/queue intermediary: Overkill for single-user service, doesn't solve subprocess problem

The honest conclusion

The hot-reload work in PR #285 (issue #269) is actually the highest-value improvement. It eliminates restarts for the most common config change scenario (trigger updates). The remaining restart cases (bot token, server port, engine binaries) are rare.

For those rare restarts, the current drain approach is solid. Tier 1 improvements (offset persistence + Type=notify) would reduce the gap to ~5s with minimal effort. Tier 2 improvements polish the UX but don't fundamentally change the architecture.

Related

• feat: hot-reload support for triggers, watchdog, and progress settings #269 — hot-reload trigger configuration (PR feat: hot-reload trigger config without restart #285)
• feat: unfreeze TelegramBridgeConfig for hot-reload of voice, files, chat_ids, timing settings #286 — unfreeze TelegramBridgeConfig for more hot-reloadable settings

[nathanschram]

Tier 1 implemented in v0.35.1rc4:

• Persist update_id to last_update_id.json with DebouncedOffsetWriter (5s/100-update debounce)
• Type=notify via stdlib sd_notify (READY=1 after first getUpdates, STOPPING=1 at drain)
• RestartSec=2 in contrib/untether.service

Tiers 2-3 deferred to v0.35.2+. See PR #292.

[nathanschram]

v0.35.1rc4 Implementation Status — Tier 1 Complete

All three Tier 1 practical improvements are implemented in PR #292.

What shipped

1. Persist update_id offset

• New src/untether/telegram/offset_persistence.py with DebouncedOffsetWriter:
  • Debounces writes to last_update_id.json (5s interval OR 100-update cap, whichever first)
  • _last_flush initialised to construction time so the first note is debounced properly
  • flush() called in poll_updates finally block (shutdown path)
• poll_updates loads saved offset on startup: offset = saved + 1
• poll_incoming gains on_offset_advanced: Callable[[int], None] | None callback; called inside the polling loop on each offset = upd.update_id + 1 advance
• Startup log: startup.offset.resumed last_update_id=N path=...
• State file lives alongside active_progress.json (sibling to config)

2. Type=notify systemd service

• New src/untether/sdnotify.py — stdlib-only (socket.AF_UNIX, SOCK_DGRAM), no dependency:
  • notify("READY=1") → True if sent, False if NOTIFY_SOCKET absent or send error
  • Abstract namespace socket support (@ prefix → \0 prefix)
  • All errors swallowed with debug log — never breaks startup
• poll_updates sends READY=1 after _send_startup succeeds
• _drain_and_exit sends STOPPING=1 at drain start
• Debug log: sdnotify.ready / sdnotify.stopping

3. RestartSec=2

• contrib/untether.service: RestartSec=10 → RestartSec=2
• Also: Type=simple → Type=notify, NotifyAccess=main added
• Key settings header comment updated with Type=notify, NotifyAccess=main, RestartSec=2 documentation

Test coverage

• test_sdnotify.py — 7 tests:
  • NOTIFY_SOCKET absent → False, empty → False
  • Filesystem socket → correct sendto addr
  • Abstract namespace (@...) → null-byte prefix
  • Send errors swallowed, socket creation errors swallowed
  • UTF-8 encoding of messages
• test_offset_persistence.py — 15 tests:
  • resolve_offset_path uses config sibling
  • Load: missing file → None, valid payload round-trip, corrupt JSON → None, wrong type → None, negative → None, string value → None
  • Save: round-trip, no leftover .tmp, creates parent dirs
  • DebouncedOffsetWriter: below interval no flush, interval triggers flush, max_pending forces flush, explicit flush() writes latest, flush with no pending is no-op

Remaining (Tiers 2-3, deferred to v0.35.2+)

• Tier 2: Socket activation for webhook server, better drain UX, pre-restart state snapshot
• Tier 3: Webhook mode for planned restarts, subprocess output redirection

Docs updated

• docs/reference/dev-instance.md — new "Readiness" and "Restart timing" sections
• docs/reference/integration-testing.md — R8 (update_id persistence), R9 (sd_notify READY), R10 (sd_notify STOPPING)
• .claude/rules/telegram-transport.md — offset persistence and sd_notify sections

[nathanschram]

Integration Test Results — v0.35.1rc4

Tested: 2026-04-14 against @untether_dev_bot (local editable source)
Branch: feature/v0.35.1rc4

Graceful restart Tier 1 (GR1–GR4) ✅

Test	Result	Details
GR1: update_id persistence	✅ PASS	Sent /ping (msg 51855), restarted service, sent another /ping (msg 51863, "up 16s") → no duplicate pong from pre-restart message; startup log shows startup.offset.resumed last_update_id=486776305 path=~/.untether-dev/last_update_id.json
GR2: sd_notify READY=1	⏭️ SKIP	Dev unit uses Type=simple (not Type=notify); sd_notify code paths covered by unit tests (7 tests in test_sdnotify.py)
GR3: sd_notify STOPPING=1 + drain	✅ PASS	Scheduled /at 5m test then restarted → logs show shutdown.requested → shutdown.draining active_runs=0 pending_at=1 → at.cancelled chat_id=8351408485 token=96fffc11f3b0 — pending /at delays properly cancelled during drain
GR4: RestartSec	✅ PASS	systemctl show untether-dev.service -p RestartUSec → 10s (dev unit); contrib template recommends 2s

Verified capabilities

• offset_persistence.py: DebouncedOffsetWriter persists update_id to last_update_id.json; on restart, poll_updates loads saved+1 as offset — no dropped or duplicate updates within Telegram's 24h retention window
• Drain integration: /at scheduler's active_count() is checked during drain; pending delays are cancelled with at.cancelled log
• Progress persistence: On restart, startup.orphan_cleanup count=2 appeared — orphan progress messages from prior instance correctly edited to "⚠️ interrupted by restart" with keyboard removed
• No update loss: Pre-restart /ping at msg 51855, post-restart /ping at msg 51863 — no gap, no duplication

Note: Full sd_notify end-to-end testing requires Type=notify + NotifyAccess=main in the systemd unit. The staging unit (contrib/untether.service) has these settings; test during staging phase.

Verdict: update_id persistence and drain logic work correctly. Ready for staging (sd_notify testable there with proper Type=notify unit).