fix: tolerate agent stalls and self-heal instead of dropping all ExApp routing

[oleksandr-nc]

Fixes #110: the outages are not HAProxy refusing to reconnect, they happen because the whole Python agent (SPOA auth, the control API and the frps Login hook share one event loop) stops responding under load, the SPOP check tolerates only about 6 seconds of that before all /exapps routing goes down, and nothing detects or restarts a hung agent.

🤖 AI (if applicable)

• The content of this PR was partly or fully generated using AI

[coderabbitai]

📝 Walkthrough

Walkthrough

The HaRP agent now provides a lightweight /heartbeat endpoint, pooled Nextcloud HTTP sessions, bounded asynchronous DNS resolution, SPOA error acknowledgements, and event-loop lag monitoring. Startup runs under Bash with process supervision, readiness checks, configurable heartbeat watchdog behavior, and wait -n exit handling. Health checks use the heartbeat endpoint. HAProxy adds 503 fallback responses and explicit agent health-check timing. Documentation describes the watchdog variables.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly captures the main change: self-healing the stalled agent to avoid dropping ExApp routing.
Description check	✅ Passed	The description matches the PR and explains the stalled-agent outage that this change addresses.
Linked Issues check	✅ Passed	Changes for [#110] add heartbeat supervision and HAProxy health checks that recover from FRP/agent stalls and reconnects.
Out of Scope Changes check	✅ Passed	All modified files support the self-healing/reconnect flow or its docs, with no clear unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a62ecaed-ffc5-48bb-8cc5-9a77b02dd30e

📥 Commits

Reviewing files that changed from the base of the PR and between 6ab3089 and 9ab37f3.

📒 Files selected for processing (5)

• README.md
• haproxy.cfg.template
• haproxy_agent.py
• healthcheck.sh
• start.sh

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

start.sh (1)

487-523: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Identify which process died in the exit message for faster incident triage.

Line 519 logs a generic "a HaRP process exited unexpectedly" without naming which of haproxy/agent/frps/frpc/watchdog actually died. Given this whole PR is about diagnosing/recovering from agent stalls, naming the culprit directly in the log would meaningfully speed up on-call triage, and can be done with a simple kill -0 scan after wait -n (no need for the Bash 5.1-only wait -n -p, keeping the existing Bash 4.3+ requirement).

🔍 Proposed diagnostic improvement

 wait -n
 rc=$?
-echo "ERROR: a HaRP process exited unexpectedly (status ${rc}), stopping the container."
+dead="unknown"
+for entry in "haproxy:$HAPROXY_PID" "agent:$AGENT_PID" "frps:$FRPS_PID" ${FRPC_PID:+"frpc:$FRPC_PID"} ${WATCHDOG_PID:+"watchdog:$WATCHDOG_PID"}; do
+    pid="${entry#*:}"
+    kill -0 "$pid" 2>/dev/null || { dead="${entry%%:*}"; break; }
+done
+echo "ERROR: HaRP process '${dead}' exited unexpectedly (status ${rc}), stopping the container."
 [ -n "${WATCHDOG_PID:-}" ] && kill "$WATCHDOG_PID" 2>/dev/null

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bf120867-d495-4498-b23f-ea7c6729da0f

📥 Commits

Reviewing files that changed from the base of the PR and between 9ab37f3 and 6ea04f9.

📒 Files selected for processing (6)

• Dockerfile
• README.md
• haproxy.cfg.template
• haproxy_agent.py
• healthcheck.sh
• start.sh

🚧 Files skipped from review as they are similar to previous changes (3)

• haproxy.cfg.template
• healthcheck.sh
• haproxy_agent.py

[coderabbitai]

🧹 Nitpick comments (1)

start.sh (1)

501-523: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Dedupe the hardcoded agent heartbeat URL.

http://127.0.0.1:8200/heartbeat is now hardcoded a third time (already literal at lines 468-469 for the readiness probe). If the agent's listen port/path ever changes and only one copy is updated, the watchdog would silently start treating a healthy agent as permanently unresponsive, repeatedly killing/restarting it.

♻️ Proposed fix: single source of truth for the URL

+AGENT_HEARTBEAT_URL="http://127.0.0.1:8200/heartbeat"
+
 # Wait deterministically for the agent to be ready (HTTP) and for SPOA (TCP).
 # Probe /heartbeat, not /info: /info can block on a K8s API reachability check.
-log "INFO: Waiting for HaRP Agent HTTP (GET http://127.0.0.1:8200/heartbeat) to be ready..."
-wait_for_http "http://127.0.0.1:8200/heartbeat" "$HP_WAIT_AGENT_HTTP" "$HP_WAIT_INTERVAL"
+log "INFO: Waiting for HaRP Agent HTTP (GET ${AGENT_HEARTBEAT_URL}) to be ready..."
+wait_for_http "$AGENT_HEARTBEAT_URL" "$HP_WAIT_AGENT_HTTP" "$HP_WAIT_INTERVAL"

-            if curl -fsS --noproxy '*' --max-time 5 http://127.0.0.1:8200/heartbeat >/dev/null 2>&1; then
+            if curl -fsS --noproxy '*' --max-time 5 "$AGENT_HEARTBEAT_URL" >/dev/null 2>&1; then

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 971b968c-d09f-4f62-aee6-83b481597c02

📥 Commits

Reviewing files that changed from the base of the PR and between 6ea04f9 and 8c40a82.

📒 Files selected for processing (6)

• Dockerfile
• README.md
• haproxy.cfg.template
• haproxy_agent.py
• healthcheck.sh
• start.sh

🚧 Files skipped from review as they are similar to previous changes (5)

• README.md
• healthcheck.sh
• haproxy.cfg.template
• Dockerfile
• haproxy_agent.py