fix(notifications): stop a failed reconciliation pass from exiting the Telegram daemon - #3907
Merged
Yeachan-Heo merged 1 commit intoAug 6, 2026
Conversation
…e Telegram daemon The scan pass archives topics whose session is gone, and it persists that through the shared topic authority. A momentarily unavailable authority (lock contention, or a compare-and-set that cannot be completed) rejected out of both `void this.runScan()` in the scan timer and `await this.runScan()` in the run loop. Nothing caught either, so the rejection reached the process-level fatal handler, which exits the process. Observed on a live installation: two consecutive daemon generations died with `shared topic authority unavailable` seconds after a session host was killed, and every session topic was left in `disconnect_grace` for 20+ minutes with no owner to archive it or answer in it — the empty-shell topics that closing a tmux window leaves behind. Sessions that were still live lost their notifications at the same moment. Lore-id: c4f7a2e9 Constraint: a reconciliation pass runs every scan interval -- a failed one must be retried, never fatal Rejected: retry inside persistTopics | hides authority contention from the operator and still cannot bound a wedged lock Confidence: high Scope-risk: narrow Reversibility: easy Tested: scan timer with a rejecting shared authority keeps the owner running, reports the failure, and retries on the next tick Not-tested: live multi-account lock contention against a real Telegram supergroup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A Telegram notification daemon whose reconciliation pass fails no longer exits the owner process.
runScan()now reports the failure and lets the next scan interval retry it, and the queue-flush timer's fire-and-forget promise is handled the same way.Why
The scan pass archives topics whose session is gone, and that archive persists through the shared topic authority. A momentarily unavailable authority (lock contention, or a compare-and-set that cannot complete) rejected out of both unguarded callers:
startScanTimer:void this.runScan()— fire-and-forget, no handler.run:await this.runScan()— awaited inside the loop with notry.Either one reaches
process.on("unhandledRejection" | "uncaughtException")inpackages/utils/src/postmortem.ts, which callsprocess.exit(1). The ownership-heartbeat and adoption-sweep timers already.catch(); these two did not.Observed on a live installation, not inferred. After a session host was killed (a closed tmux window),
notifications/daemon.logrecorded two consecutive daemon generations dying seconds apart:telegram-daemon.heartbeat.jsonstops at the second timestamp, that pid is gone, andtelegram-topics.jsonstill held two topics stuck indisconnect_grace20+ minutes later —ORPHAN_TOPIC_GRACE_MSis 60s, andorphanedAtfor one of them is the crash instant. With no owner, those topics are exactly the reported symptom: an empty-shell thread that is never archived and answers nothing. Sessions that were still live lost their notifications at the same moment, because the whole owner went down.This is deliberately scoped to the fatal boundary. The underlying authority contention (a 5s
withFileLockbudget: 50 retries x 100ms) is a separate question; a daemon that survives it retries every scan interval and converges, whereas a daemon that exits cannot.runis a protected declaration inscripts/telegram-daemon-generation-manifest.json, so the guard is insiderunScanandrunis byte-identical todev. NoDAEMON_GENERATIONbump is owed (v43 no protected changes), which also keeps this off the 52/53 generation queue held by #3844 and #3899.Testing
notifications-telegram-daemon.test.ts: a loaded topic whose session has no endpoint, plus a shared authority whosecompareAndSetrejects. Firing the scan-timer tick reaches the authority, logssession scan failed, records nounhandledRejection, leavesrunning === true, and the next tick runs the pass again. Verified failing ondev(the rejection escapes and bun reportsshared topic authority unavailable) and passing with the fix.bun test packages/coding-agent/test/notifications-telegram-daemon.test.ts packages/coding-agent/test/notifications-topic-registry.test.ts→ 607 pass, 0 fail.bun test packages/coding-agent/test/notifications-telegram-daemon-2960-redteam.test.ts packages/coding-agent/test/sdk-host-wiring.test.ts→ 84 pass, 0 fail.bun scripts/telegram-daemon-generation-guard.ts(base11e48d5bc, headedd15f917) →v43 no protected changes.tsc --noEmit -p packages/coding-agent/tsconfig.jsonclean;biome checkclean on both touched files.GJC verdict
devbun checkpasses (fullbun checknot run locally; focused suites, typecheck, biome, and the daemon generation guard are green — CI covers the rest)