In-app chat: agent shows typing indicator indefinitely, never replies

[obaid]

Summary

On a freshly-created agent in the hosted app, sending a message in the in-app Web Chat shows the typing indicator (· · ·) persistently but the agent never responds. Agent activity counters (sessions, messages, tokens) stay at 0 even after waiting several minutes.

Steps to reproduce

1. Create a new agent via the wizard (any defaults; Efficient / claude-haiku-4-5 model selected).
2. After deployment, click Open Chat on the Web Chat card under Channels (or navigate directly to /agents/<id>/chat).
3. Type any message and hit Enter.

Expected

Agent replies within seconds. The Overview tab Activity counters increment.

Actual

• User's message renders in the chat.
• Typing indicator appears immediately.
• Typing indicator persists for >90 seconds; no response arrives.
• Overview tab Activity counters remain 0/0/0 with Last active: Never.
• Last synced updates normally (e.g. 2m ago), so the agent record itself is being polled.

Environment

• Hosted production deploy.
• Agent deployed onto an already-running shared server (not a fresh one).
• Credit balance is well above zero, so auto-top-up exhaustion is not the cause.

Suspected area

Likely either:

1. The shared server's gateway/runtime isn't picking up the newly-created agent registration, or
2. Messages from the Web Chat ingress aren't being routed to the agent runtime.

The presence of a Restart Gateway button on the agent Overview suggests this is a recognized failure mode. Worth confirming whether new agents on existing shared servers require a manual gateway restart and, if so, automating that step in the deploy flow.

[obaid]

Root cause confirmed via prod debug

openclaw-gateway (user systemd service on the agent server) is dead with a hard config error and not auto-restarting. The Laravel app reaches the server fine over SSH, but curl http://127.0.0.1:18789/v1/responses (the gateway endpoint) connection-refuses — so chat hangs on the typing indicator forever.

Service state on the shared agent server

● openclaw-gateway.service - OpenClaw Gateway (v2026.5.3-1)
   Active: failed (Result: exit-code) since … 17min ago
   Process: ExecStart=/usr/bin/node /usr/lib/node_modules/openclaw/dist/index.js gateway --port 18789 (code=exited, status=78)

What it logs on startup

Config was last written by a newer OpenClaw (2026.5.19); current version is 2026.5.3-1.
Refusing to start the gateway service because this OpenClaw binary (2026.5.3-1) is older than the config last written by OpenClaw 2026.5.19.
Run the newer openclaw binary on PATH, or reinstall the intended gateway service from the newer install.

exit 78 is EX_CONFIG — OpenClaw's own version-downgrade safety check is refusing to operate the newer on-disk config with the older binary.

Timeline

• Gateway had been running fine for the previous several hours (normal [ws] health pings every minute).
• At the moment the new agent's AgentUpdateScriptService ran on the server, it hit Step 7: Restart Gateway (systemctl --user restart openclaw-gateway).
• The restart killed the running gateway cleanly. On re-start, OpenClaw 2026.5.3 binary detected newer config and refused to come up.
• All in-app chat, channel polling, etc. for every agent on this server have been broken since.
• Laravel log still reported Agent … update completed via script callback — so the script's health check on its own did not catch the failure, and the UI continues to show the agent as active.

Two related issues worth fixing together

1. The version-mismatch state is reachable from a normal deploy. Either the openclaw binary should be upgraded as part of AgentUpdateScriptService before the restart, or restart should be skipped when the binary is older than the on-disk config.
2. The script-callback's health check doesn't detect a failed gateway service. Worth adding an explicit systemctl --user is-active openclaw-gateway (or openclaw gateway call health) check before reporting update completed, and surfacing a failure on the agent's Overview tab instead of leaving it labeled active.

[obaid]

Confirmed via fresh-team E2E

Provisioned a brand-new team, which triggered a fresh Linode + fresh OpenClaw install, then ran the same E2E flow with the same Telegram bot that previously had no replies on the broken server.

Fresh server state:

• openclaw --version → OpenClaw 2026.5.19 (matches what the broken server's config had been bumped to)
• systemctl --user is-active openclaw-gateway → active
• curl http://127.0.0.1:18789/health → 200

Result on the fresh server, same flow:

• Telegram bot replies in seconds:

  "👋 Hey Obaid! Yes, I'm up and running…"
  "Hey Obaid! 👋 Great to hear from you. I'm all set up and working smoothly here on Telegram…"
  "Perfect! 🎉 I'm confirming receipt and I'm fully operational on the fresh server…"

• In-app chat replies in seconds:

  "Got it! ✓ In-app chat is working smoothly. Ready to go whenever you need me."

So the install path produces a healthy gateway. The bug is specifically the in-place upgrade path: agent-update scripts on an existing server can leave the binary at an older version while bumping config to a newer one, after which systemctl --user restart openclaw-gateway permanently fails with EX_CONFIG (78) and the script's health check doesn't notice.

Fix scope confirmation: the two action items in my earlier comment still apply — (1) ensure binary upgrade happens before config-incompatible restarts, and (2) treat a non-active gateway as an update completed failure so the agent's Overview tab surfaces it instead of staying labeled active.

[obaid]

Manual recovery on the affected server — successful

Recovery steps run on the broken shared server:

npm install -g openclaw@2026.5.19
systemctl --user reset-failed openclaw-gateway
systemctl --user restart openclaw-gateway

Post-state:

• openclaw --version → OpenClaw 2026.5.19
• systemctl --user is-active openclaw-gateway → active
• Node listening on 127.0.0.1:18789
• curl http://127.0.0.1:18789/healthz → 200
• openclaw gateway call health → {"ok": true}

End-to-end verification on the recovered server:

• The 2 messages that had queued in Telegram during the outage were automatically processed once the gateway came back, and the agent's activity feed shows: 2 new messages processed — Just now.
• A fresh in-app chat ping to a long-running agent returned a reply.

So the fix recipe is short. The remaining work is the code change so the next deploy doesn't re-introduce the same state.