[Improve] Harden the Discord gateway delivery and leadership paths#410
Merged
Conversation
Follow-ups from the PR #268 reliability review: - surface the dead-letter stream: quarantine depth is tracked in gateway status and shown as a failing diagnostic row in Discord settings, so undeliverable events are visible instead of accumulating silently - bound the durable streams: the inbound stream and dead-letter stream get approximate MAXLEN caps (DISCORD_GATEWAY_INBOUND_MAX_ENTRIES, DISCORD_GATEWAY_DEAD_LETTER_MAX_ENTRIES), and capacity pressure is reported in status before shedding starts - tolerate one transient Redis error on lease renewal: the 30s lease leaves room for two 10s renew intervals, so only a definitive ownership loss or a second consecutive failure tears down the Gateway connection (renewDetailed distinguishes lost from error) - restart the delivery backoff after a long healthy run instead of ratcheting toward the lifetime maximum forever - report supervisor death: the embedded supervisor takes an onFatal hook, wired to Sentry in the BullMQ host, and stamps the shared status so a silently-stopped Discord ingestion pages and shows in diagnostics The lease-renewal tolerance is covered indirectly (renewDetailed contract + simple streak logic); there is no service-level harness to drive the renew timer directly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
No new code issues found. See task
Reviewed fd390b3 |
…delivery refactor #409 replaced the deliveryAbort controller with a delivery object and an abortDelivery() helper so the forwarder can restart on secret rotation; the renew-error streak logic now abandons leadership through that helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sure warning - pruneOrphanedAttempts deletes retry counters whose stream entries were shed by the MAXLEN cap (acknowledge/quarantine clean up their own); runs on the status refresh timer so the attempts hash stays bounded with the stream during a long outage - the capacity pressure warning moves from lastError — which every successful delivery clears — to a dedicated capacityWarning status field whose only writer is the refresh timer, and Discord settings shows it as an Event backlog diagnostic row Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pruneOrphanedAttempts read the stream first: an event enqueued and attempted between XRANGE (empty) and HKEYS put a live counter in the deletion set, resetting its retry accounting past the configured maximum. Snapshotting the tracked ids first means a counter recorded after the snapshot can never be pruned; ordering is covered by tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reliability follow-ups to #268, from the gateway review that ran before it merged.
Changes
deadLetterDepthand Discord settings shows a failing Event delivery diagnostic row when anything is quarantined, instead of events accumulating in a stream nothing reads.MAXLENcaps (DISCORD_GATEWAY_INBOUND_MAX_ENTRIES, default 50k;DISCORD_GATEWAY_DEAD_LETTER_MAX_ENTRIES, default 1k). A multi-hour delivery outage can no longer grow Redis without limit; capacity pressure is reported in status before shedding starts.renewDetailedon the Redis lock distinguishes definitive ownership loss (immediate teardown) from a transient error (tolerated once).onFatalhook (wired to Sentry in the BullMQ host) and stamps the shared status, so silently-stopped Discord ingestion is visible in error tracking and diagnostics rather than only a console line.Validation
service.tshas no direct unit harness (the renew timer isn't injectable); it's covered by therenewDetailedcontract and kept deliberately simple.🤖 Generated with Claude Code