Skip to content

fix(notifications): stop a failed reconciliation pass from exiting the Telegram daemon - #3907

Merged
Yeachan-Heo merged 1 commit into
Yeachan-Heo:devfrom
yazzang-homelab:fix/daemon-scan-failure-not-fatal
Aug 6, 2026
Merged

fix(notifications): stop a failed reconciliation pass from exiting the Telegram daemon#3907
Yeachan-Heo merged 1 commit into
Yeachan-Heo:devfrom
yazzang-homelab:fix/daemon-scan-failure-not-fatal

Conversation

@yazzang-homelab

Copy link
Copy Markdown
Contributor

What

A Telegram notification daemon whose reconciliation pass fails no longer exits the owner process. runScan() now reports the failure and lets the next scan interval retry it, and the queue-flush timer's fire-and-forget promise is handled the same way.

Why

The scan pass archives topics whose session is gone, and that archive persists through the shared topic authority. A momentarily unavailable authority (lock contention, or a compare-and-set that cannot complete) rejected out of both unguarded callers:

  • startScanTimer: void this.runScan() — fire-and-forget, no handler.
  • run: await this.runScan() — awaited inside the loop with no try.

Either one reaches process.on("unhandledRejection" | "uncaughtException") in packages/utils/src/postmortem.ts, which calls process.exit(1). The ownership-heartbeat and adoption-sweep timers already .catch(); these two did not.

Observed on a live installation, not inferred. After a session host was killed (a closed tmux window), notifications/daemon.log recorded two consecutive daemon generations dying seconds apart:

2026-08-05T23:14:23.672Z pid=732571 [Unhandled Rejection] Error: shared topic authority unavailable
    at .../sdk/bus/telegram-daemon.ts:8379:16
2026-08-05T23:16:17.656Z pid=735076 [Uncaught Exception] Error: shared topic authority unavailable
    at .../sdk/bus/telegram-daemon.ts:8379:16

telegram-daemon.heartbeat.json stops at the second timestamp, that pid is gone, and telegram-topics.json still held two topics stuck in disconnect_grace 20+ minutes later — ORPHAN_TOPIC_GRACE_MS is 60s, and orphanedAt for one of them is the crash instant. With no owner, those topics are exactly the reported symptom: an empty-shell thread that is never archived and answers nothing. Sessions that were still live lost their notifications at the same moment, because the whole owner went down.

This is deliberately scoped to the fatal boundary. The underlying authority contention (a 5s withFileLock budget: 50 retries x 100ms) is a separate question; a daemon that survives it retries every scan interval and converges, whereas a daemon that exits cannot.

run is a protected declaration in scripts/telegram-daemon-generation-manifest.json, so the guard is inside runScan and run is byte-identical to dev. No DAEMON_GENERATION bump is owed (v43 no protected changes), which also keeps this off the 52/53 generation queue held by #3844 and #3899.

Testing

  • New regression test, notifications-telegram-daemon.test.ts: a loaded topic whose session has no endpoint, plus a shared authority whose compareAndSet rejects. Firing the scan-timer tick reaches the authority, logs session scan failed, records no unhandledRejection, leaves running === true, and the next tick runs the pass again. Verified failing on dev (the rejection escapes and bun reports shared topic authority unavailable) and passing with the fix.
  • bun test packages/coding-agent/test/notifications-telegram-daemon.test.ts packages/coding-agent/test/notifications-topic-registry.test.ts → 607 pass, 0 fail.
  • bun test packages/coding-agent/test/notifications-telegram-daemon-2960-redteam.test.ts packages/coding-agent/test/sdk-host-wiring.test.ts → 84 pass, 0 fail.
  • bun scripts/telegram-daemon-generation-guard.ts (base 11e48d5bc, head edd15f917) → v43 no protected changes.
  • tsc --noEmit -p packages/coding-agent/tsconfig.json clean; biome check clean on both touched files.

GJC verdict

gajae.pr-review-verdict.v1 needs-human sha256:edd15f9172e4229ed437ff78eefbbf6c62ddc2b9 reviewer:human evidence:local bun test packages/coding-agent/test/notifications-telegram-daemon.test.ts (607 pass) + notifications/daemon.log crash evidence quoted above

  • Target branch is dev
  • bun check passes (full bun check not run locally; focused suites, typecheck, biome, and the daemon generation guard are green — CI covers the rest)
  • Tested locally
  • CHANGELOG updated (if user-facing)
  • Verdict above matches the exact PR head, not an earlier commit

…e Telegram daemon

The scan pass archives topics whose session is gone, and it persists that
through the shared topic authority. A momentarily unavailable authority
(lock contention, or a compare-and-set that cannot be completed) rejected
out of both `void this.runScan()` in the scan timer and `await
this.runScan()` in the run loop. Nothing caught either, so the rejection
reached the process-level fatal handler, which exits the process.

Observed on a live installation: two consecutive daemon generations died
with `shared topic authority unavailable` seconds after a session host was
killed, and every session topic was left in `disconnect_grace` for 20+
minutes with no owner to archive it or answer in it — the empty-shell
topics that closing a tmux window leaves behind. Sessions that were still
live lost their notifications at the same moment.

Lore-id: c4f7a2e9
Constraint: a reconciliation pass runs every scan interval -- a failed one must be retried, never fatal
Rejected: retry inside persistTopics | hides authority contention from the operator and still cannot bound a wedged lock
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: scan timer with a rejecting shared authority keeps the owner running, reports the failure, and retries on the next tick
Not-tested: live multi-account lock contention against a real Telegram supergroup
@Yeachan-Heo
Yeachan-Heo merged commit f14c6a0 into Yeachan-Heo:dev Aug 6, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants