[BUG] Cog supervision: a wedged-but-alive cog (work loop stalled, HTTP still 200 / health green) is never detected or restarted

[shaal]

Firmware version

cognitum-agent v0.22.7 (mode: seed)

Hardware

Raspberry Pi Zero 2 W (Rev 1.0)

Host OS

Raspbian GNU/Linux 13 (trixie) — kernel 6.12.47+rpt-rpi-v7 (armv7l)

Description

A cog whose work loop wedges — but whose process stays alive and keeps answering HTTP — is never detected or restarted. The agent treats "process is up / HTTP responds" as healthy, so a wedged cog stays "green" indefinitely.

This is the sibling failure mode to #52. That issue covers a cog that crashes/exits (segfault → zombie → not restarted). This one is the opposite: the cog never exits — it keeps returning HTTP 200 on / and /health — but its internal work loop has stalled and stopped making progress. Any supervision keyed on process-exit (the reap + auto-restart-on-crash that #52 asks for) cannot see this case, because there is nothing to reap.

Concrete instance (the doom cog, but the failure mode is cog-agnostic):

• The cog renders frames on a dedicated worker thread and publishes them behind a counter. After ~38h uptime (~6.4M frames) the worker thread wedged in its paced-sleep cycle and stopped producing frames.
• The cog's HTTP server thread stayed alive the whole time: GET / served the full page, and GET /health kept returning {"status":"ok"}.
• From a client this looks "stuck" — the page loads but never updates — rather than down, so nothing (and no one) flags it.
• It only recovered after a manual systemctl restart of the cog. CPU was ~2% the whole time (a wedge, not a runaway loop), so even a CPU-based heuristic wouldn't catch it.

Why the current health surface doesn't help:

• /health is app-controlled and trivial. A cog returning a static {"status":"ok"} proves only that its HTTP thread is scheduled — not that its actual work is progressing. process-alive ≠ work-progressing, and the agent currently can't tell the difference.
• I also still observe the zombie from Cog supervision: crashed cog not auto-restarted, left as a zombie #52 on the same device (a <defunct> cog child of the agent that was never reaped), so Cog supervision: crashed cog not auto-restarted, left as a zombie #52 is live too — but it's a separate symptom from this one.

Suggested direction (platform-level)

Give the agent a liveness signal independent of "process up / HTTP 200", and recover on staleness. Options, roughly in order of effort:

1. Liveness convention for cogs: define an optional cog-reported heartbeat / monotonic work counter (e.g. a required field in /health like {"status":"ok","ticks":<n>,"last_progress_ms":<n>}). The agent polls it and restarts the cog if the counter hasn't advanced within a cog-declared interval.
2. Agent-side staleness watchdog: for cogs that expose a frame/progress endpoint, the agent samples it and restarts on no-progress, with backoff and a clear "stalled/restarting" status (mirroring the status fix requested in Cog supervision: crashed cog not auto-restarted, left as a zombie #52).
3. Document the expectation for cog authors: a meaningful /health must reflect work progress, not just thread liveness — so this failure mode is designed out rather than discovered in the field.

(The cog-side half of this — making the doom cog's own /health report frame liveness and self-exit on a stalled render loop so the supervisor can restart it cleanly — is being handled in cognitum-one/cogs#28. This issue tracks the platform-level gap that applies to every cog.)

Steps to reproduce

1. Run any cog with a long-lived worker loop that can stall without the process exiting (the doom cog reproduces it after long uptime; a worker thread blocking or wedging while the HTTP thread keeps serving is enough).
2. Let the worker loop wedge (or simulate it: have the worker stop advancing its progress counter while the HTTP server keeps answering).
3. Observe: GET /health still returns {"status":"ok"}, the process stays alive, GET /api/v1/apps shows the cog healthy — but the cog produces no new work.
4. Nothing auto-detects or restarts it; recovery requires a manual stop/start (or systemctl restart).

Logs / console output

# client view — frame counter frozen, server still serving the last frame
$ curl -s -D - -o /dev/null 'http://<seed>:1993/frame?since=0'
HTTP/1.1 200 OK
X-Frame-Id: 6442296            # identical across repeated requests, minutes apart
$ curl -s -D - -o /dev/null 'http://<seed>:1993/frame?since=6442296'
HTTP/1.1 204 No Content        # no frame newer than the frozen one, ever
$ curl -s 'http://<seed>:1993/health'
{"status":"ok"}                # still "healthy"

# on-device — worker thread alive but parked, process near-idle (wedge, not spin)
$ for t in /proc/<pid>/task/*; do printf '%s %s %s\n' \
    "$(cat $t/comm)" "$(awk '{print $3}' $t/stat)" "$(cat $t/wchan)"; done
doom-engine     S   hrtimer_nanosleep     # worker parked in its paced sleep, not drawing
cog-doom-arm    S   inet_csk_accept       # HTTP accept thread — still up
cog-doom-arm    S   futex_wait_queue      # HTTP workers — parked
...
# CPU delta over 2s: ~4 ticks (~2%) => wedged, not a runaway loop

(Cog-side stdout was unavailable — journald had rotated by the time it was noticed, which is itself part of the problem: there's no retained signal that the loop stalled.)

Confirmation

• I searched existing issues first (related: Cog supervision: crashed cog not auto-restarted, left as a zombie #52 — crashed/zombie cog not restarted; this is the never-exits sibling)
• I'm running the latest firmware (cognitum-agent v0.22.7)

[proffesor-for-testing]

Confirmed and reproduced from source — this is a genuine platform gap, distinct from #52. The agent's entire notion of cog health is /proc/{pid} existence (src/cognitum-agent/src/api.rs:1486); there's no heartbeat, progress counter, staleness watchdog, or even any polling of a cog's /health anywhere in the agent (verified by grep across the whole crate). So a cog whose worker loop wedges while its HTTP thread keeps answering 200 stays "green" indefinitely, exactly as you describe — and the exit-keyed supervision #52 asks for can't see it (nothing to reap).

We'll track the platform-level liveness signal — an optional cog-declared heartbeat/progress field in /health (e.g. {"status":"ok","ticks":n,"last_progress_ms":n}) that the agent polls, restarting on staleness with backoff and a clear stalled/restarting status — as cognitum-one/seed#182. The doom cog's self-reporting half stays in cognitum-one/cogs#28. Severity: major. Thanks for the thorough /proc-level evidence and for separating this cleanly from #52.