fix(worker): recover stale tasks from dead workers after restart

[76131356]

Problem

When a Hindsight container is restarted (OOM, docker kill, host crash), operations stuck in processing status are never recovered. The existing recover_own_tasks() only matches tasks where worker_id equals the current worker's ID. After a restart, the new worker gets a different ID (hostname/container-id changes), so tasks from the previous worker remain stuck indefinitely.

This blocks ALL new consolidation work — the worker sees claimable=1 but my_active= none, with others: <dead_worker_id>:1 in the logs.

Reproduced consistently on self-hosted Docker deployment (v0.8.1 through v0.8.4). Related: #2165

Root Cause

# poller.py:recover_own_tasks()
WHERE status = 'processing' AND worker_id =   # ← only matches CURRENT worker

The worker_id defaults to socket.gethostname() (container hostname), which changes on every restart.

Fix

Added recover_stale_tasks() which finds ALL processing tasks whose claimed_at is older than 10 minutes and resets them to pending:

UPDATE {table}
SET status = 'pending', worker_id = NULL, claimed_at = NULL,
    error_message = 'auto-recovered: stale processing task'
WHERE status = 'processing'
  AND claimed_at < now() - interval '600 seconds'

Runs at two points:

1. At startup — after recover_own_tasks() in run()
2. Periodically — every 5 minutes during the idle polling loop

The 10-minute timeout avoids interfering with legitimately slow tasks (large consolidation batches) while recovering promptly after a crash.

Testing

• Verified on our self-hosted NAS (Z4Pro-O2RV) with Hindsight v0.8.4-slim
• Confirmed stale tasks are automatically recovered after container restart
• No interference with active consolidation tasks (tested with 24-memory batch)
• pending_consolidation count drops from 21→0 after recovery

Screenshots

Before fix (worker stuck indefinitely):

[WORKER_STATS] worker=e2ef8f1b7a3c slots=0/10 | others: ef231f6edf6f:1 | my_active: none
[PENDING_BREAKDOWN] consolidation: total=1 claimable=1 payload_null=0 retry_blocked=0 assigned=0

After fix (auto-recovered):

Worker e2ef8f1b7a3c recovered 1 stale tasks from dead workers (timeout=600s)
[WORKER_STATS] worker=e2ef8f1b7a3c slots=1/10 | my_active: consolidation:HERMES(1)

[koriyoshi2041]

I ran a focused worker slice locally on this branch:\n\nuv run pytest tests/test_worker.py -k 'decommission or worker_status or priority or recover'\n\nResult: 24 passed, 2 warnings.\n\nOne edge case I think is worth tightening before merge: recover_stale_tasks() resets any processing row older than 10 minutes, independent of whether the owning worker is actually dead. Since _recover_stale_if_due() runs periodically from other live workers, a legitimate long-running retain/consolidation task can cross the 600s threshold while its original worker is still executing it, get reset to pending, then be claimed again by another worker. That would turn a slow task into duplicate execution rather than just crash recovery.\n\nA safer shape might be to recover only explicitly decommissioned/known-dead worker IDs, or add a heartbeat/lease that is refreshed by active tasks and recover only when that lease is stale. If keeping the timeout-only approach, I’d at least add a regression test with worker A actively processing a task older than the threshold while worker B runs the stale sweep, so the intended behavior is explicit.

[benfrank241]

thanks for this, and the writeup is excellent — the restart→new-worker-id→orphaned-processing-tasks failure (#2165) is real, and recover_own_tasks() genuinely can't catch it since it keys on worker_id = self. cross-worker stale recovery is the right thing to add.

one correctness concern blocks it though: recovery keys on claimed_at age with no worker-liveness signal, so it can reset tasks a live worker is still legitimately processing. i checked the poller — there's no heartbeat: claimed_at is set once at claim, and updated_at is only touched on terminal transitions (completed/failed/retry), never bumped while a task runs. so a task that a live worker has been working for >10 min looks identical to one from a dead worker. recover_stale_tasks() resets it to pending, a second worker claims it, and now the same operation runs on two workers at once.

for consolidation that's worse than the bug it fixes — concurrent double-processing can produce duplicate observations / corrupt mental-model state, whereas the current failure is "stuck" (recoverable). and large consolidation/retain batches really do exceed 10 min (see #2644 — retains splitting into 100+ sub-batches).

the fix needs a liveness signal rather than a fixed claim-age timeout. two shapes that would work:

• add a periodic heartbeat that bumps updated_at (or a last_heartbeat column) on in-flight tasks in the polling loop, and key recovery on heartbeat staleness — i.e. "no activity in N minutes", not "claimed N minutes ago". that way a live-but-slow task keeps itself alive and only a truly-dead worker's tasks get reset.
• or a worker-lease/registry table where workers renew a lease and recovery targets only tasks whose worker's lease has expired.

also — this needs a test. it's a concurrency/recovery change in the worker path, which has had real churn recently (#2601/#2608/#2614), so a regression test that (a) a stale task past the timeout is recovered and (b) a recently-active/heartbeated task is not would give us confidence it won't reset live work.

happy to help think through the heartbeat approach if useful. direction's right — just want the recovery to distinguish "slow" from "dead" before it can go in.