Skip to content

fix(whatsapp): catch wedged info queries in /health; make send approval asynchronous - #117

Merged
retog merged 2 commits into
mainfrom
claude/gateway-connection-monitoring-fc52co
Aug 16, 2026
Merged

fix(whatsapp): catch wedged info queries in /health; make send approval asynchronous#117
retog merged 2 commits into
mainfrom
claude/gateway-connection-monitoring-fc52co

Conversation

@retog

@retog retog commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Fixes the two halves of the same incident: a WhatsApp bridge whose outbound info queries (IQ/usync) were wedged looked healthy to the monitor (#115), and approving the affected send tripped the web-gateway's 30s proxy timeout into a misleading "gateway unreachable" (#116).

Closes #115. Closes #116.

#115/health catches a wedged bridge

The bridge can hold a live, linked websocket while outbound IQ queries are wedged: every send to a recipient without a cached device list fails with failed to send usync query: info query timed out, yet /health derived connected from socket/link events only, so gateway-monitor.py never alerted.

  • A probe thread now completes a real IQ round trip once a minuteget_user_info/get_user_devices on our own JID (resolved via get_me()), with defensive fallbacks across neonize versions; an older neonize without the methods logs once and falls back to link-state-only health.
  • After two consecutive failures (debounce) /health flips to connected: false with an error naming what actually breaks, plus a dedicated iq_ok field — so the existing monitor alerting fires unchanged; connected now means "can actually send", not "socket is open".
  • Only a successful probe clears the wedge — a reconnect that doesn't fix the IQ path can never flap health (relevant because in the observed incident even a container restart didn't clear it).
  • On a sustained wedge the gateway tears its connection down so main()'s outer loop reconnects, at most once per WHATSAPP_IQ_RECONNECT_BACKOFF (default 600s).
  • The probe takes the client lock with a 10s acquire timeout and skips the round when the client is busy, so it never queues behind a long transfer.
  • Tunables: WHATSAPP_IQ_PROBE_SECONDS (default 60, 0 disables), WHATSAPP_IQ_PROBE_FAILURES (default 2), WHATSAPP_IQ_RECONNECT_BACKOFF.

#116 — asynchronous send approval

Approving a channel pending send on /sends ran the actual send synchronously inside the HTTP request (web-gateway proxies the approve with a 30s timeout; the gateway's _complete_pending_send() called _push() inline). Any slow send — document upload, first-contact device-list lookup, Signal's voice synthesis — outran the timeout even on a healthy bridge.

  • All three gateways (Signal, WhatsApp, Telegram — the block is identical across them) now flip the entry to status: sending under the store lock (concurrent approvals cannot start two sends), answer immediately, and execute the send in a background thread that records the terminal status — approved, or error carrying the real error string. The caller gets a snapshot taken before the worker starts, so it always sees the sending transition. Rejection stays synchronous.
  • The web-gateway redirects an approval to the send's own page instead of /sends/next; that page live-refreshes through Sending… to Sent ✓ — or shows the gateway's actual error (e.g. the usync timeout) instead of a generic "unreachable" — so the user gets outcome feedback at approval time. Older gateways that still answer with a terminal status render exactly as before.

Tests

  • New tests/test_whatsapp_health.py: wedge debounce, error text naming the usync failure, recovery only via a successful probe, logged-out precedence, reconnect backoff.
  • Lifecycle tests in all three test_*_send_policy.py updated for the async approve flow (sending → terminal, idempotent re-completion), plus a new error-path test asserting the real error string is persisted and surfaced.
  • Full suite passes locally except test_push_notify.py, which also fails on main in this environment (broken system cryptography, unrelated).

Notes

🤖 Generated with Claude Code

https://claude.ai/code/session_01YbqfmpARas38AfiwtCjZbn


Generated by Claude Code

claude added 2 commits August 16, 2026 18:41
The bridge can hold a live, linked websocket while its outbound info
queries (IQ) are wedged — every usync query (device-list lookup, i.e.
any send to a recipient whose devices are not cached) times out with
"failed to send usync query: info query timed out" while /health still
reported connected+linked, so the gateway-monitor never alerted
(issue #115).

A probe thread now completes a real IQ round trip once a minute
(get_user_info/get_user_devices on our own JID, resolved via get_me(),
with defensive fallbacks across neonize versions). Results fold into the
connection state via _note_iq_result(): after two consecutive failures
/health flips to connected: false with an error naming what actually
breaks, so the existing monitor alerting fires unchanged. Only a
successful probe clears the wedge — a reconnect that doesn't fix the IQ
path can never flap health. On a sustained wedge the gateway also tears
its connection down (at most once per WHATSAPP_IQ_RECONNECT_BACKOFF,
default 600s) so main()'s outer loop reconnects.

Tunables: WHATSAPP_IQ_PROBE_SECONDS (default 60, 0 disables),
WHATSAPP_IQ_PROBE_FAILURES (default 2), WHATSAPP_IQ_RECONNECT_BACKOFF.
The probe takes the client lock with a 10s acquire timeout and skips the
round when the client is busy, so it never queues behind a long
transfer. An older neonize without the needed methods logs once and
falls back to link-state-only health.

Closes #115.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YbqfmpARas38AfiwtCjZbn
Approving a channel pending send on /sends ran the actual send
synchronously inside the HTTP request: the web-gateway proxied
POST /pending-sends/<id>/approve with a 30s timeout while the gateway's
_complete_pending_send() called _push() inline. Any slow send (document
upload, first-contact device-list lookup, Signal voice synthesis) outran
the timeout and the page reported "gateway unreachable" even though the
gateway was fine and still sending (issue #116).

All three gateways now flip the entry to status "sending" under the
store lock (so concurrent approvals cannot start two sends), return
immediately, and execute the send in a background thread that records
the terminal status — "approved", or "error" carrying the real error
string. The caller gets a snapshot taken before the worker starts, so it
always sees the "sending" transition. Rejection stays synchronous.

The web-gateway now redirects an approval to the send's own page instead
of /sends/next; that page renders live status ("Sending…" with
auto-refresh, then "Sent ✓" or the gateway's error string, e.g. a usync
timeout) so the user gets outcome feedback at approval time. Lifecycle
tests updated for the async flow, plus a new error-path test asserting
the real error string is persisted and surfaced.

Closes #116.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YbqfmpARas38AfiwtCjZbn
@retog
retog merged commit dd31daa into main Aug 16, 2026
1 check passed
@retog
retog deleted the claude/gateway-connection-monitoring-fc52co branch August 16, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants