slots.md

Machine-wide agent slots

Several agent harnesses can share one machine. To keep the total load bounded, they cooperate through a directory of lock files. Any harness can join this protocol with a few dozen lines of code.

Where

• Slot directory: $AGENT_SLOTS_DIR if set. Otherwise /var/lib/worklane/agent-slots when it exists. Otherwise /var/tmp/agent-slots (macOS, Linux, WSL) or %ProgramData%\agent-slots (Windows).
• Config: $AGENT_SLOTS_CONFIG if set. Otherwise /etc/worklane/slots.json when the /var/lib directory is in use and that file exists. Otherwise config.json in the slot directory.
• In the engine's own test run (node's test runner, a test file run directly, the test preload's mark, or a process a test run started with its environment or the engine's childEnv()): only a slot directory inside the test machine (test/isolate.ts). Any other, or none, throws before a file is opened, so a test can never take a slot, write STOP or a budget, or read the machine's own (src/os/index.ts slotsDir, src/os/system-paths.ts testSlotsRefusal). A process started with an environment built from scratch is a test run only by how node started it. Outside a test run nothing changes.

On a machine that runs agents for weeks, use the /var/lib and /etc locations. Tmp cleaners such as systemd-tmpfiles delete files in /var/tmp that haven't changed for about 30 days, which would reset the cap and drop its history. /var/tmp suits single-user and development machines.

Files

File	Meaning
config (slots.json or config.json)	{"max_agents": N}. The machine's configured cap on concurrent agents across all harnesses, set once per machine. Default 2.
slots.lock	Held for a moment while a harness counts running agents and takes a slot.
.<resource filename>.flock	A permanent, empty native guard inode for that resource. It is retained after release and crashes; ordinary cleanup must never unlink, rename, link or replace it.
agent-<i>.lock	One running agent. The number of live files is capped, not the index: files may be numbered past the cap, so lowering it never strands an agent that is still running.
full-run.lock	A full test run is in progress. At most one per machine.
test-<i>.lock	One coordinator test run (docs/test-runs.md). Counted apart from agents and capped by the config's max_test_runs (default 2, 1..8), which only the machine helper changes (set-test-slots).
STOP	Emergency stop, {"by", "at", "reason"}. While it exists, every harness halts its running agents and takes no new slot.
github-budget-<key>.json	The GitHub API budget of one credential, shared by every process on the machine that uses it (src/github/budget.ts). <key> is the first 16 hex characters of the token's SHA-256, never the token. Per rate-limit resource (core, graphql): GitHub's last X-RateLimit-Limit / -Remaining / -Reset, less every call granted since, and the calls granted in the window by consumer; and pausedUntil, set by a secondary limit (403 or 429) for every caller on that token. A call takes one unit before it goes; at none left, or paused, it waits up to 10 s, then fails with GitHub's own kind of error (rate_limited or secondary_rate_limited, status 429) without reaching GitHub. Always replaced (a 0660 copy renamed over it), never written in place, so a coordinator of another user can update it. Bounded whoever wrote it: a pause counts only up to 15 min ahead (a longer Retry-After is cut to 15 min), a window only up to 1 h 5 min ahead; anything else (a reset too far or missing, a remaining that isn't a number of 0 or more) is "not known", and the call goes. Unreadable: starts afresh. A state it can't write, or a lock busy for 0.5 s, lets the call go. Keyed by token, so an App installation token's new token (about hourly) starts a new file: a pause in force is forgotten then, and the next limit GitHub meets sets it again. A file unchanged for 2 h is removed by the next process that takes from any budget (at most every 10 min per process). So far only the backlog's client (src/backlog/github.ts) takes from it. In a test run, a budget made without a directory is the test machine's (see Where above). The dashboard's Health page shows it to owners and writers (src/github/budget-health.ts): read without the lock (the file is only ever replaced whole), as a regular file only, at most 64 KiB, the 8 newest readable ones (at most 32 tried); only safe whole counts, and consumers named like the engine's own (others as "(other)"); credentials numbered there, never named by the key.
github-budget-<key>.lock	Held for a moment while one process reads, changes and replaces that state. Observing a response or pausing tries it once and never waits. Uses the shared native guard and process-identity recovery below; age or a future timestamp never permits bypassing a busy guard. Known-dead owners recover automatically. Unknown or legacy records stay busy; accounting then lets API calls go after at most 0.5 s, until an operator removes the abandoned record during quiescence.
update-drain.json	The engine updater draining the machine for an install, {"since", "until", "to"}. While it stands (until its until), no new agent slot, test slot or full-run lock is taken; what holds one already keeps it until it ends. With the /var/lib slot directory it lives in root's /var/lib/worklane/ (root-owned 0644), not in the slot directory, so only root writes or removes it. A flag past its until, unreadable, or more than 4 hours ahead is ignored.

Each JSON lock holds pid, owner, acquiredAt, place: {host, boot, pidns} and identity (Linux boot ID plus process start time). Test-run locks may also hold limit_ms. Older records without the place/identity proof remain readable but are not automatically taken over.

Native guard and recovery

src/locks.ts#tryLock takes a per-resource kernel guard before inspecting any JSON lock or takeover claim and holds it for the returned Lock's entire lifetime. src/slot-file.ts#trySlotFileGuard prepares an empty file and its exact 0444 mode through a private descriptor, then publishes it by exclusive hard link as .<filename>.flock. A crash before publication leaves no guard; after publication it leaves a complete usable inode, possibly with a leftover temporary hard link. This works with a restrictive creator umask and lets other coordinator users open the guard read-only.

The guard is opened without following a symlink and without blocking on a FIFO. It must be an empty regular file. Initial mode is 0444; the existing state privacy tidy may change it to 0660 without changing the inode or lock domain. Read-only mode is not the exclusion primitive. src/os/index.ts#tryNativeFlock passes only that descriptor as fd 3 to the native helper (/usr/bin/flock on Linux, /usr/bin/perl on macOS), with a minimal environment and a five-second helper timeout. Its nonblocking exclusive flock belongs to the shared open file description: helper exit closes its duplicate, without issuing LOCK_UN; the parent keeps the lock until it closes its descriptor. The retained descriptor is always above fd 2. If opening it would fill a closed standard-descriptor hole, acquisition throws before any native helper or JSON mutation and closes that descriptor. Keep standard streams open (redirect unused streams to /dev/null); Node/libuv helper spawning also requires valid stdio. Node-created descriptors are close-on-exec, so even an unrelated child using inherited standard streams does not retain ownership. Deliberately passing an extra duplicate can extend ownership until that duplicate closes. Release is idempotent, removes only the caller's complete JSON record, and closes the descriptor in finally, without reacquiring another lock.

This serializes claim inspection/removal/publication as well as the stale-lock read/removal. Two dead-claim reapers cannot validate one old claim and then remove each other's replacements. A crashed holder or takeover claimant releases its kernel ownership when its descriptors close; the next contender can automatically clear its known-dead JSON record/claim. If the short helper is still holding its explicit duplicate, recovery waits for that helper to exit too. A process externally stopped forever while holding a descriptor still holds the guard; ordinary scheduling and helper termination are assumptions. Live, hidden/unreadable, foreign, namespace-mismatched, malformed and legacy records remain busy, even after the kernel guard becomes available. Age is not death.

Supported backends: Linux uses /usr/bin/flock (util-linux), /proc/self/fd, and local ext2/3/4, XFS, Btrfs or tmpfs. macOS x86_64/arm64 uses the system /usr/bin/perl with native flock (fcntl emulation is refused) on local APFS/HFS. Its filesystem helper calls XNU fstatfs64 on inherited fd 3, validates the fixed 64-bit little-endian ABI and a buffer canary, and checks MNT_LOCAL and the exact filesystem name; it never parses human mount paths or guesses VFS registration numbers. Unsupported tooling, ABIs, unknown/network/FUSE/overlay filesystems and helper failures refuse acquisition. WSL2 uses the Linux backend; keep state/slots in the Linux filesystem. Native Windows agent execution still needs WSL2. Docker's image declares state/slot volumes; use the same supported local named volumes for every cooperating container, as in quickstart.md. An arbitrary host bind or remote volume is not assumed safe.

Darwin process scope is the host, kern.bootsessionuuid, and its single host PID space. Identity uses the immutable start timestamp reported by /bin/ps with C locale and UTC, read together with process state. Only leading zombie state proves exit; malformed or hidden output while signal zero succeeds is unknown. Its second-resolution identity may conservatively remain busy if a PID is reused within the same second. Linux retains boot/start-tick identity and PID namespace checks. Platform tests run the deterministic regression and crash boundaries on actual Ubuntu/macOS runners; their measured results, rather than this description, determine readiness. Other allowed filesystem families rely on their native locking semantics unless separately exercised.

Cooperative protocol and upgrade: all processes/harnesses sharing resources must adopt the guard. Stop old binaries before upgrading; old claim reapers ignore it. The JSON format and tryLock/acquireLock/Lock.release interfaces stay compatible, but mixed-version concurrency is not safe. Canonical guard names and their publication-temp aliases (with NFKC, default-ignorable removal and lowercase reservation) are refused by the ordinary slot-file mutation APIs. Export/import omits these machine-local inodes and their publication-temp aliases. Use canonical resource paths; filesystem-specific alias behavior beyond this normalization is not independently validated. Read-only mode does not prevent raw directory-entry mutation by a directory writer: direct external unlink/rename, mount changes, or removing/replacing parent directories while holders run are outside the protocol. Lifecycle cleanup must first quiesce all users.

Guard inodes accumulate for distinct resource names (for example, rotating GitHub budget keys); they are not garbage-collected online. Reclaim them only during a quiescent maintenance window. Adding the guard suffix and its publication temporary name must fit the filesystem's filename limit; failures refuse acquisition.

Rules

1. Create a lock atomically: write its JSON to a temp file in the same directory, then hard-link it to the lock's name (link() fails if the lock exists). A lock is never visible half-written. Exclusive create followed by a separate write is not enough: a competitor that reads the empty file in between takes it for stale.
2. Stale: while holding the resource's native guard, take over only a record from the same known host, boot and PID namespace whose recorded process is proved gone or has a different known start identity. Take an exclusive claim named by the stale JSON's hash, recheck the JSON, remove that stale record, and compete for the free JSON lock. Clear a dead takeover claim under the same native guard. Unreadable identity (including a hidden /proc entry for a process still answering signal zero) is never death. Liveness and identity decide staleness, not age.
3. Agents: take slots.lock; count live agent-*.lock files; if fewer than max_agents, take the first free agent-<i>.lock; release slots.lock. If at the cap, wait. A lower cap never stops running agents; new ones wait until the count is below it.
4. Full runs: take full-run.lock before starting a full suite. Projects can also declare an idle_probe command for runs started outside any harness. For example, a script that exits 0 only when no full run is live. With an agent user, the probe is a project command and runs in its sandbox, in a PID namespace of its own (issue #713): it can't see processes outside it, so it should look for a lock file or a port, not for a process (pgrep).
5. Release: under the guard already held, delete the JSON file only if its entire contents are still yours, then close the native guard descriptor. Never remove the permanent guard inode.
6. Never write a file in place: write a new temp file with a random name, created exclusively (O_EXCL), then rename it over the file (or hard-link it, for a lock). Set its mode through the descriptor you opened (fchmod), never by its name: anyone in the group can swap the name for a symlink, and a chmod by name follows it. A file here can be read-only to you (removing an instance leaves its files 0444), and replacing it needs only write access to the directory. A temp name built from your pid alone can be one a crashed process left, read-only, under the same pid. The engine makes every write here through one helper, writeSlotFile (src/slot-file.ts); test/slot-dir-writes.test.ts fails on any other. In the engine a path in this directory is a SlotPath, never a string (#772): an opaque value that only src/slot-file.ts reads, lists, removes or writes, so the type checker rejects it anywhere else a path goes, and at run time fs, path.join and string conversion refuse it. Only src/os/index.ts makes one; test/slot-path.test.ts checks that.

Emergency stop: worklane stop-all [reason] writes STOP; every Worklane coordinator on the machine aborts its agents within about 5 seconds (their tasks requeue) and starts none until worklane resume-all removes it. Writing it needs write access to the slot directory: run it as a coordinator user, or as root.

worklane slots shows what's held, and any stop in force. worklane queue full-run queues a full run behind both checks, in a detached runner that survives the session that queued it.