fix(whatsapp): catch wedged info queries in /health; make send approval asynchronous - #117
Merged
Merged
Conversation
The bridge can hold a live, linked websocket while its outbound info queries (IQ) are wedged — every usync query (device-list lookup, i.e. any send to a recipient whose devices are not cached) times out with "failed to send usync query: info query timed out" while /health still reported connected+linked, so the gateway-monitor never alerted (issue #115). A probe thread now completes a real IQ round trip once a minute (get_user_info/get_user_devices on our own JID, resolved via get_me(), with defensive fallbacks across neonize versions). Results fold into the connection state via _note_iq_result(): after two consecutive failures /health flips to connected: false with an error naming what actually breaks, so the existing monitor alerting fires unchanged. Only a successful probe clears the wedge — a reconnect that doesn't fix the IQ path can never flap health. On a sustained wedge the gateway also tears its connection down (at most once per WHATSAPP_IQ_RECONNECT_BACKOFF, default 600s) so main()'s outer loop reconnects. Tunables: WHATSAPP_IQ_PROBE_SECONDS (default 60, 0 disables), WHATSAPP_IQ_PROBE_FAILURES (default 2), WHATSAPP_IQ_RECONNECT_BACKOFF. The probe takes the client lock with a 10s acquire timeout and skips the round when the client is busy, so it never queues behind a long transfer. An older neonize without the needed methods logs once and falls back to link-state-only health. Closes #115. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbqfmpARas38AfiwtCjZbn
Approving a channel pending send on /sends ran the actual send synchronously inside the HTTP request: the web-gateway proxied POST /pending-sends/<id>/approve with a 30s timeout while the gateway's _complete_pending_send() called _push() inline. Any slow send (document upload, first-contact device-list lookup, Signal voice synthesis) outran the timeout and the page reported "gateway unreachable" even though the gateway was fine and still sending (issue #116). All three gateways now flip the entry to status "sending" under the store lock (so concurrent approvals cannot start two sends), return immediately, and execute the send in a background thread that records the terminal status — "approved", or "error" carrying the real error string. The caller gets a snapshot taken before the worker starts, so it always sees the "sending" transition. Rejection stays synchronous. The web-gateway now redirects an approval to the send's own page instead of /sends/next; that page renders live status ("Sending…" with auto-refresh, then "Sent ✓" or the gateway's error string, e.g. a usync timeout) so the user gets outcome feedback at approval time. Lifecycle tests updated for the async flow, plus a new error-path test asserting the real error string is persisted and surfaced. Closes #116. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbqfmpARas38AfiwtCjZbn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two halves of the same incident: a WhatsApp bridge whose outbound info queries (IQ/usync) were wedged looked healthy to the monitor (#115), and approving the affected send tripped the web-gateway's 30s proxy timeout into a misleading "gateway unreachable" (#116).
Closes #115. Closes #116.
#115 —
/healthcatches a wedged bridgeThe bridge can hold a live, linked websocket while outbound IQ queries are wedged: every send to a recipient without a cached device list fails with
failed to send usync query: info query timed out, yet/healthderivedconnectedfrom socket/link events only, sogateway-monitor.pynever alerted.get_user_info/get_user_deviceson our own JID (resolved viaget_me()), with defensive fallbacks across neonize versions; an older neonize without the methods logs once and falls back to link-state-only health./healthflips toconnected: falsewith an error naming what actually breaks, plus a dedicatediq_okfield — so the existing monitor alerting fires unchanged;connectednow means "can actually send", not "socket is open".main()'s outer loop reconnects, at most once perWHATSAPP_IQ_RECONNECT_BACKOFF(default 600s).WHATSAPP_IQ_PROBE_SECONDS(default 60,0disables),WHATSAPP_IQ_PROBE_FAILURES(default 2),WHATSAPP_IQ_RECONNECT_BACKOFF.#116 — asynchronous send approval
Approving a channel pending send on
/sendsran the actual send synchronously inside the HTTP request (web-gateway proxies the approve with a 30s timeout; the gateway's_complete_pending_send()called_push()inline). Any slow send — document upload, first-contact device-list lookup, Signal's voice synthesis — outran the timeout even on a healthy bridge.status: sendingunder the store lock (concurrent approvals cannot start two sends), answer immediately, and execute the send in a background thread that records the terminal status —approved, orerrorcarrying the real error string. The caller gets a snapshot taken before the worker starts, so it always sees thesendingtransition. Rejection stays synchronous./sends/next; that page live-refreshes throughSending…toSent ✓— or shows the gateway's actual error (e.g. the usync timeout) instead of a generic "unreachable" — so the user gets outcome feedback at approval time. Older gateways that still answer with a terminal status render exactly as before.Tests
tests/test_whatsapp_health.py: wedge debounce, error text naming the usync failure, recovery only via a successful probe, logged-out precedence, reconnect backoff.test_*_send_policy.pyupdated for the async approve flow (sending→ terminal, idempotent re-completion), plus a new error-path test asserting the real error string is persisted and surfaced.test_push_notify.py, which also fails onmainin this environment (broken systemcryptography, unrelated).Notes
/gatewayspage and monitor need no changes for WhatsApp gateway /health reports connected while outbound IQ (usync) queries are wedged #115 — they already key offconnected.retinue/gateway containers picking up the new scripts.🤖 Generated with Claude Code
https://claude.ai/code/session_01YbqfmpARas38AfiwtCjZbn
Generated by Claude Code