Stop launchd killing the VM the autostart job exists to start - #72
Merged
Conversation
Root cause of the 2026-08-08 post-reboot outage, found after the fact. autostart/uk.mathewcsims.podman-autostart.plist was missing AbandonProcessGroup. From launchd.plist(5): "When a job dies, launchd kills any remaining processes with the same process group ID as the job." That only matters if podman's VM processes stay in the launching group. They do, confirmed live: gvproxy and krunkit run with PPID 1 (podman exited and they were reparented) but keep the PGID of the `podman machine start` that spawned them. No setsid, no detach. Inside a launchd job that group is the job's group, so when podman-autostart.sh finished its container listing and exited at 10:56:12, launchd tore down the group and took the VM with it. podman-watchdog found it dead at 10:57:41 and every Mac-hosted app was down. This explains two things nothing else did: there is no crash report for either process, because they were signalled rather than crashing; and re-running the SAME script by hand worked and stayed up, because the process group belonged to a shell session with no launchd job to be reaped against. Same script, same command, opposite outcome. Not deterministic — the stack has survived plenty of reboots, so there is a race between launchd's teardown and podman's children being reparented. This removes the race rather than winning it. The watchdog's auto-restart stays as the backstop for VM deaths we cannot name; this prevents the one we can, and the two should not be confused. The plist was not in the repo at all — only the script was, unlike podman-watchdog/ which tracks both. Now tracked, with the reinstall procedure and the `launchctl print | grep properties` check that proves the key actually took. Also records the wrong theory so it is not re-derived: Podman Desktop was suspected purely because it launches at login in the same second. Its settings.json has no machine-autostart preference and nothing shows it starting or stopping machines. Co-timing was the entire case. Installed and reloaded live; the loaded job now reports "properties = runatload | abandon process group | inferred program", and all 31 containers and every site came through the reload unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Caught reviewing this branch. The watchdog has the identical exposure and I had only fixed the autostart job. #71 gave the watchdog the ability to run `podman machine start` itself. Its plist lacks AbandonProcessGroup, so launchd would kill the VM the moment the watchdog run exited — and the failure mode is worse than a plain omission, because the script would already have alerted success: detect down -> machine start -> alert "restarted automatically" -> exit -> launchd kills the VM -> 2 min later detect down -> repeat Three cycles of that exhausts MAX_RESTART_ATTEMPTS, having sent three "restarted automatically" warnings, and ends with the stack still down and the watchdog refusing to try again. The feature would have been actively worse than not having it. It had not yet run against a real outage. Also swept every other launchd job in the repo rather than assuming these were the only two: contact-sync, kopia-mac-backup, kopia-verify, kopia-mirror, trivy-scan, paperless-task-alert and pf-lan-lockdown all run synchronously and leave no background process behind, so none of them need the key. The rule is narrow — a job needs it only if what it starts is meant to outlive it. Installed and reloaded live; the watchdog job now reports "properties = runatload | abandon process group | inferred program". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause of this morning's post-reboot outage, found after the fact. Follow-up to #71, which built the backstop.
The mechanism
autostart/uk.mathewcsims.podman-autostart.plistwas missingAbandonProcessGroup. Fromlaunchd.plist(5):That only matters if podman's VM processes stay in the launching group. They do — confirmed live:
PPID 1because podman exited and they were reparented;PGID 3581is thepodman machine startinvocation's group, which they never leave. Nosetsid.So: agent runs the script →
machine startspawns gvproxy/krunkit into the job's group → script exits at 10:56:12 → launchd tears down the group → watchdog finds the VM dead at 10:57:41.Why this is the answer and not another guess
Two things it explains that nothing else did:
Same script, same command, opposite outcome. That difference is the diagnosis.
Not deterministic — the stack has survived plenty of reboots, so there's a race between launchd's teardown and podman's children being reparented. This removes the race rather than winning it.
Also in here
podman-watchdog/which tracks both.settings.jsonhas no machine-autostart preference and nothing shows it starting or stopping machines. Co-timing was the entire case, which isn't one.Verification
Installed and reloaded live. The loaded job now reports:
All 31 containers and every site (
cp,blog,fj,docs,owl) came through the reload unaffected.plutil -lintclean.🤖 Generated with Claude Code