Remote job worker supervisors accumulate one per remote operation and never exit, driving host load to 295

[stanzhang]

Summary

On a Linux remote secondmate host, fm-remote-job-worker.sh supervisors accumulate one per remote operation and never exit. Combined with a hole in the restart limiter, this drove a 24-core WSL2 host to a load average of 295, with 800-1000 concurrent worker processes and ~290 of them reparented to init.

Observed

Host: Ubuntu under WSL2, registered as a remote secondmate, reached over the normal remote transport.

$ cat /proc/loadavg
295.84 285.15 239.60 285/1811 1296283

$ ps -eo comm --no-headers | sort | uniq -c | sort -rn | head -3
    819 fm-remote-job-w
    101 ps
     16 bash

Top-level supervisors (ppid == 1), ages spanning minutes to hours, all still alive:

  16673  1  10666  /bin/bash .../bin/fm-remote-job-worker.sh
  19895  1    973  /bin/bash .../bin/fm-remote-job-worker.sh
  27863  1   1664  /bin/bash .../bin/fm-remote-job-worker.sh
  32945  1    496  /bin/bash .../bin/fm-remote-job-worker.sh
  35221  1   1207  /bin/bash .../bin/fm-remote-job-worker.sh

No systemd unit and no cron entry exist for these; they are started on demand by remote operations.

No single process was CPU-dominant. The load was entirely the aggregate of hundreds of short-lived workers churning.

Mechanism

1. The supervisor has no singleton guard.

worker_acquire_lock is called exactly once, at line 687, inside main(). main() runs for the --serve child and for the non-Linux path. The Linux supervisor entry point does not call it:

  '')
    if [ "$(fm_remote_job_platform)" = linux ]; then worker_supervise_linux; else main; fi
    ;;

worker_supervise_linux() acquires no lock of its own, so WORKER_LOCK serialises workers, not supervisors. Every remote operation that starts a bare worker therefore adds another permanent supervisor, and none of them can observe each other.

2. The restart limiter resets on a barely-healthy child.

    if [ $((SECONDS - started)) -ge "$FM_REMOTE_JOB_SUPERVISOR_HEALTHY_SECONDS" ]; then
      failures=0
      sleep 0.1
      continue
    fi

FM_REMOTE_JOB_SUPERVISOR_HEALTHY_SECONDS defaults to 10. A child that survives just past ten seconds and then exits resets failures to 0 and is restarted after 0.1s. FM_REMOTE_JOB_SUPERVISOR_MAX_RESTARTS (default 20) counts only consecutive sub-threshold failures, so it never fires on this path. One supervisor in that state restarts roughly ten times per second indefinitely; N accumulated supervisors multiply it.

The header comment shows runaway restarts were anticipated. The guard just does not cover a child that dies immediately after crossing the healthy threshold.

Impact

The affected host is a GPU machine intended for ML work. At load 295 it was effectively unusable for its actual purpose, and the secondmate's own status became unreadable from the parent home.

Clearing the processes dropped runnable tasks from ~290/1811 to 1/652 immediately, and the one-minute load average from 295 to 11 within a few minutes. The count then climbed again from 26 to 61 as further remote operations were performed, consistent with the per-operation accumulation above.

Suggested fix

1. Give the supervisor its own lock, distinct from the worker lock, so a second supervisor on the same account home refuses to start rather than coexisting.
2. Do not reset failures to zero on a bare threshold crossing, or require a longer healthy interval before resetting, so a child that dies just past the threshold cannot sustain a restart loop the maximum-restart guard never observes.

Possibly related, though a different failure mode: #2772 (remote job worker can swallow TERM after losing ownership).

Workaround

pkill -f "fm-remote-job-worke[r].sh"

Safe in the sense that it does not affect the secondmate agent itself; the next remote operation starts a fresh supervisor.

[kunchenguid]

Speaking as Kun's firstmate:

Inspected current main at 7b88520 (#2858). The leak is still present. No comments, no linked implementation PR.

Evidence on bin/fm-remote-job-worker.sh:

1. Supervisors are not singleton. worker_acquire_lock is called once, at line 687, inside main(). main() is the --serve child (and the non-Linux path). The Linux entrypoint does not take that lock:

'')
  if [ "$(fm_remote_job_platform)" = linux ]; then worker_supervise_linux; else main; fi

worker_supervise_linux() (745-787) never calls worker_acquire_lock. WORKER_LOCK therefore serializes workers, not supervisors. Every remote operation that starts a bare worker adds another permanent supervisor, and none of them can see each other.

2. The restart limiter resets on a barely-healthy child. Lines 772-775: if the child lived at least FM_REMOTE_JOB_SUPERVISOR_HEALTHY_SECONDS (default 10), failures is reset to 0 and the loop sleep 0.1 then continues. FM_REMOTE_JOB_SUPERVISOR_MAX_RESTARTS (default 20) only counts consecutive sub-threshold failures, so it never fires on this path. One supervisor in that state restarts about ten times per second; N accumulated supervisors multiply it.

VISION: this is a mechanics bug, not a new capability. It aligns with scripts owning the mechanics, a restart being a non-event (supervisors must not accumulate across operations), and honesty under load (a secondmate host at load 295 is not a fleet that can be looked away from). Authority, spine, vendor-independence, and scope are not in play beyond fixing the script that already owns this.

Related but different: #2772 (TERM swallowed after losing ownership). Not treating that as this issue.

Labeling ready-for-pr. Suggested direction from the report is sound: a supervisor lock distinct from the worker lock, and a limiter that cannot be reset by a child that dies immediately after the healthy threshold.