Correction: the original text of this issue blamed ChatMux for exhausting inotify. That was wrong. Measurement below shows ChatMux is a minor consumer; the real defect is how it reacts to exhaustion.
Symptom
The GJC native session watcher enters a permanent restart loop. On this host it ran 1752 consecutive failures over 14.5 hours (00:00:27 → 14:37:24), one cycle every 30s, and only recovered when an unrelated tmux kill-server happened to kill the stuck child.
While it loops, session discovery is unavailable, so resuming a session from the UI silently does not work. Nothing in the UI says anything is wrong.
Trigger
inotify watch exhaustion, ENOSPC:
Session watcher error for provider "codex" {
error: "ENOSPC: System limit for number of file watchers reached, watch '/home/…/rollout-….jsonl'" }
GJC session watcher failed. (child-exit code=1 signal=none)
GJC native session watcher failed. (child-exit code=1 signal=none; consecutive 25)
Who actually consumes the limit
Measured from /proc/*/fdinfo on a host sitting exactly at the cap (262144/262144):
| consumer |
watch descriptors |
share |
| VS Code Server |
109,309 |
42% |
omo agent processes |
135,276 |
52% |
| ChatMux server (chokidar) |
16,041 |
6% |
ChatMux native watcher (chatmux-core watch) |
972 |
0.4% |
ChatMux is not the cause. Deleting its watching entirely would recover 6% and cost the product its live session updates (the fallback reconcile is WATCHER_FALLBACK_RECONCILE_MS = 60_000, so transcript updates would lag up to a minute).
The actual bug
Reacting to exhaustion, not causing it:
- Infinite silent retry. The supervisor restarts the child every 30s forever. 1752 attempts produced no escalation and no user-visible signal.
- No degraded state surfaced. Discovery is down, the UI looks normal, and "resume does nothing" is the only symptom the user sees.
Suggested: after N consecutive failures, stop retrying at the cap and surface a degraded-discovery state to the client, ideally naming ENOSPC so the operator knows to raise fs.inotify.max_user_watches.
Before #37 this was undiagnosable — the failure logged one fixed string with no reason, exit code, or run length. The reason codes are what made the (child-exit code=1) line line up against the ENOSPC errors.
Optional efficiency follow-up
Not required to fix this issue, but the numbers are stark: chokidar uses 16,041 watches for claude/cursor/codex while the native Rust watcher uses 972 for the same job. Migrating the remaining providers from chokidar to chatmux-core watch would cut the ChatMux footprint ~16x with no feature loss. The codex root alone holds 11,214 transcript files, each individually watched.
Workaround
sudo sysctl -w fs.inotify.max_user_watches=524288 (persist in /etc/sysctl.d/). Note the binding limit here is max_user_watches, not max_user_instances (128, only 60 in use).
Symptom
The GJC native session watcher enters a permanent restart loop. On this host it ran 1752 consecutive failures over 14.5 hours (00:00:27 → 14:37:24), one cycle every 30s, and only recovered when an unrelated
tmux kill-serverhappened to kill the stuck child.While it loops, session discovery is unavailable, so resuming a session from the UI silently does not work. Nothing in the UI says anything is wrong.
Trigger
inotify watch exhaustion,
ENOSPC:Who actually consumes the limit
Measured from
/proc/*/fdinfoon a host sitting exactly at the cap (262144/262144):omoagent processeschatmux-core watch)ChatMux is not the cause. Deleting its watching entirely would recover 6% and cost the product its live session updates (the fallback reconcile is
WATCHER_FALLBACK_RECONCILE_MS = 60_000, so transcript updates would lag up to a minute).The actual bug
Reacting to exhaustion, not causing it:
Suggested: after N consecutive failures, stop retrying at the cap and surface a degraded-discovery state to the client, ideally naming
ENOSPCso the operator knows to raisefs.inotify.max_user_watches.Before #37 this was undiagnosable — the failure logged one fixed string with no reason, exit code, or run length. The reason codes are what made the
(child-exit code=1)line line up against the ENOSPC errors.Optional efficiency follow-up
Not required to fix this issue, but the numbers are stark: chokidar uses 16,041 watches for claude/cursor/codex while the native Rust watcher uses 972 for the same job. Migrating the remaining providers from chokidar to
chatmux-core watchwould cut the ChatMux footprint ~16x with no feature loss. The codex root alone holds 11,214 transcript files, each individually watched.Workaround
sudo sysctl -w fs.inotify.max_user_watches=524288(persist in/etc/sysctl.d/). Note the binding limit here ismax_user_watches, notmax_user_instances(128, only 60 in use).