Skip to content

The launcher supervises its children, so a 3am death is restarted and written down - #18

Open
LeavesJ wants to merge 1 commit into
mainfrom
feat/launcher-supervision
Open

LeavesJ wants to merge 1 commit into
mainfrom
feat/launcher-supervision

Conversation

@LeavesJ

@LeavesJ LeavesJ commented Sep 4, 2026

Copy link
Copy Markdown
Owner

What

scripts/serve_invitee.py gains supervise(), its run loop. A child that exits on its own (the uvicorn server, or cloudflared) is restarted. The wait doubles from 1s to a 60s cap while the child keeps dying inside 60s of birth, resets after a longer life, and never gives up: a keyless child costs nothing while it cannot start, and it should come back the moment the cause is fixed. Server and tunnel have independent counters. Every restart is written into the tenant's server.log:

2026-09-04T07:41:02Z [ada] supervisor: server exited code=-15 after 4s; restart #1 in 1s
2026-09-04T07:41:03Z [ada] supervisor: server restarted (pid 81611)

grep supervisor: data/tenants/<slug>/server.log is the restart history.

Why

The beta died at 01:49 on 2026-09-03 with a clean SIGTERM sequence in the child and no cause ever established. Nothing noticed for hours: the launcher was waiting on a child that was gone, and the tunnel kept answering 302 to the world. This is the "supervision" item the handoff recommended next.

Stop semantics kept

SIGINT/SIGTERM set a flag; the loop terminates every child itself within one 0.5s poll, tunnel first, and returns the server's exit code. A stop is never followed by a restart, and the backoff wait is sliced so a stop during a 60s wait lands within a poll. The three pre-existing launcher subprocess tests still pass, which is the check that a launcher SIGTERM still takes everything down promptly.

What it does not cover, said in the docstring

The launcher itself being killed. That needs launchd on this machine, which changes the launch command J runs, so it is an operator decision and sits in the handoff rather than in code. No alert channel either: a desktop banner was considered and dropped (macOS-only, fires on the founder's screen every suite run, useless at 3am).

Verified

Seven tests, each watched to fail first. Six over supervise() with fake children and a fake clock: restart logged; stop terminates both and never restarts; backoff 1/2/4 then reset after a long life; 60s cap; tunnel restarted without touching the server; sliced wait. One real path: the launcher as a subprocess, its child found with pgrep -P and SIGTERMed, a new child pid observed, health back on the same port, restart #1 in the tenant log, launcher still alive.

felix gate pass: 1446 passed, 35 skipped.

Tier T1. Failure classes checked against the diff: stop-turns-into-restart-fight, crash loop spinning, zombie children (poll reaps, wait on stop), stale env on restart (same env dict as the first spawn, same as before), port rebind after a fast restart (proven by the real-path test), signal during sleep (PEP 475, noticed within one poll).

… restarted and written down

The beta died at 01:49 on 2026-09-03 with a clean SIGTERM sequence in the
child and no cause ever established, and nothing noticed for hours: the
launcher waited on a child that was gone while the tunnel kept answering
302 to the world. `serve_invitee.py` now runs `supervise()`: a child that
exits on its own is restarted, the wait doubles from 1s to a 60s cap while
it keeps dying young and resets after a long life, it never gives up, and
every restart lands in the tenant's server.log with the exit code, the
lifetime and the attempt number. Server and tunnel have independent
counters. A stop still terminates everything within one poll and is never
followed by a restart.

Not covered, and said so: the launcher itself being killed needs launchd,
which is an operator decision.

Seven tests: six over the loop with fake children and a fake clock, one
real path that SIGTERMs the launcher's child and watches a new one answer
on the same port with `restart #1` in the log. Each watched to fail first.
Gate: 1446 passed, 35 skipped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant