fix(daemon): enforce single-owner machine lock and loud version-skew

[fsafey]

Problem

On a host running both the CLI-spawned daemon (default profile) and the
Desktop-spawned daemon (a --profile desktop-<host> daemon), two daemon
processes register the same identity and runtime IDs and both write the same
checkout at once. The local-directory guard that is supposed to prevent
concurrent writes is a process-local sync.Mutex, so each daemon holds an
independent mutex and neither sees the other. The two existing single-instance
guards — the per-profile health-port bind in daemon.Run and the per-profile
PID file — key on the profile, so a daemon under a different profile is invisible
to them.

This was observed in production: CLI daemon (0.3.22) and Desktop daemon (0.4.2)
serving the same daemon id, both advertising the same Claude/Codex/Pi runtime
IDs, and both acquiring the same checkout for different tasks within the same
minute. The Desktop daemon also 404-looped on /prepare-lease — a route the
older server lacked — with no loud signal.

The upstream direction (fix/daemon-id-machine-scoped) deliberately makes CLI +
Desktop share one machine-scoped identity. This change is what makes that
shared-identity model safe: it enforces a single live owner of the machine.

Design decision (please weigh in)

A single authoritative ownership mechanism: a machine-global advisory file
lock at ~/.multica/daemon.lock, taken at the very top of daemon.Run before
the health bind or any registration.

• Machine-global, not identity-keyed. The invariant being protected is one
  daemon process per machine's shared checkouts — which the process-local mutex
  cannot enforce. Keying on the base config dir (shared by every profile) is also
  what catches the bug on today's main, where CLI and Desktop still mint
  different per-profile identities. It composes with
  fix/daemon-id-machine-scoped whether or not that lands.
• flock over a server lease. flock closes the pre-registration local-write
  window, needs no server round-trip, and the OS releases it on process death —
  so crash recovery and stale-owner recovery are automatic, with no TTL to tune
  and no stale window, and no manual database/file surgery.

The trade-off to confirm: this makes two concurrent per-profile daemons on
one machine mutually exclusive. The per-profile health-port scheme hints the
project may have intended to allow that (e.g. one daemon per backend). If you
want per-backend concurrency, the lock should be keyed differently and I'll
adjust. The break-glass MULTICA_DAEMON_ALLOW_MULTIPLE=1 is the escape hatch for
a deliberate multi-backend setup and the documented rollback.

What changed

Ownership (the core fix)

• server/internal/daemon/ownership.go + ownership_unix.go (flock) +
  ownership_windows.go (LockFileEx on a high sentinel byte so content reads
  stay unblocked). No new dependency — uses golang.org/x/sys, already vendored.
• daemon.Run acquires the lock first and releases it last (after
  deregisterRuntimes). A second daemon fails with an actionable error naming
  the incumbent (pid, version, profile, health port) and a profile-scoped stop
  hint (multica --profile <p> daemon stop) so the remedy actually reaches a
  cross-profile owner.
• Rolling-upgrade sweep: the launcher refuses to start while a daemon that
  does NOT hold the lock is alive on any known profile's health port. A
  lock-aware daemon acquires before binding health, so lock-free + health-alive
  can only be a pre-lock release (or a break-glass daemon) — without this, a new
  daemon would silently start alongside a legacy one and recreate the
  two-writer hazard.
• daemon start --takeover / daemon restart performs a graceful cross-profile
  handoff via the incumbent's recorded health port (cross-platform; no OS
  signals). The launcher distinguishes a live competitor (health up → fast
  conflict) from a shutting-down incumbent (health down → bounded wait sized to
  the full graceful-shutdown bound: 30s in-flight task drain + 5s deregister +
  margin = 45s) so restart never spuriously conflicts mid-shutdown.
• MULTICA_DAEMON_ALLOW_MULTIPLE break-glass / rollback.

Version compatibility (loud, not a 404 loop)

• The server reports server_version in the /api/daemon/register response.
• The daemon gates on it after the initial (synchronous) registration in
  preflight, before claiming any task; a version-mismatch exit deregisters its
  runtimes on the way out so the server does not keep routing tasks to a dead
  process until the sweeper catches up. Unreported/dev/unparsable versions are
  treated as unknown (warn + proceed, no false rejection).
• Backstop: a /prepare-lease 404 that means "route missing" logs one
  actionable error and stops the refresh loop, instead of Warn-ing every 15s
  forever. MinServerVersion = 0.3.28 is the evidence-backed floor (first
  release carrying /prepare-lease) — a "required route exists" floor, not a
  full compatibility matrix.

Docs: docs/daemon-single-owner.md — operator verification, takeover,
upgrade ordering, and rollback.

Cross-model review

The change was independently reviewed by GPT-5.6 and Claude Opus 4.8 (both high
effort); all findings from both are addressed in this revision:

• Rolling-upgrade blind spot → the launcher sweep (refuse while a pre-lock
  daemon is alive on any profile port), which also skips foreign-environment
  daemons (a WSL2/other-OS daemon visible via localhost forwarding, per the
  [Bug]: Desktop "Auto-start on launch" / "Auto-stop on quit" don't control the daemon running in WSL2 #3916 topology) rather than refusing with a stop hint the user can't act on.
• Prepare-lease 404 classification → positive matching on chi's plain-text
  router catch-all ("404 page not found") instead of excluding known bodies, so
  a transient handler-level 404 (JSON {"error":...} — the server maps DB
  hiccups in access checks to not-found) stays retryable and can never
  permanently stop the lease extender with a false version-skew verdict.
• Takeover never force-kills — a failed shutdown delivery (expected when the
  incumbent is already draining: it closes its health listener at the start of
  shutdown) now falls through to the bounded wait; a wedged incumbent fails
  loudly with a manual remedy instead of a SIGKILL that would skip the drain and
  orphan agent subprocesses mid-write.
• Deregister-on-version-exit, shutdown-sized waits (45s = 30s task drain
  • 5s deregister + margin), profile-scoped stop hints, and the sweep on
    the direct --foreground path too.

Known follow-up (deliberately out of scope): when the Desktop app's spawn of
daemon start fails with the ownership conflict, the app shows "stopped"
without the reason — startDaemon in
apps/desktop/src/main/daemon-manager.ts resolves {success:false, error: err.message} (the actionable text is in there via stderr) but
sendStatus({state:"stopped"}) carries no reason, and DaemonStatus has no
error field. Behavior is safe (the second daemon correctly does not start);
surfacing the reason needs a small DaemonStatus.reason? + UI change, best done
as a Desktop-side follow-up.

Test evidence

Rebased onto current main (ad7b23896, v0.4.3); merges clean. Ran locally
with Go 1.26.1 on darwin/arm64:

• go build ./... — pass (unix and GOOS=windows cross-build).
• go vet + gofmt -l on all changed files — clean.
• Full package suites (go test -count=1): internal/daemon, cmd/multica,
  pkg/agent — pass.
• go test -race on the ownership / version-gate / takeover / sweep tests — pass.

Scenario	Test
simultaneous start / duplicate identity	TestAcquireOwnership_SecondFailsWhileHeld
clean shutdown	TestAcquireOwnership_ReleaseAllowsReacquire
crash recovery (real 2nd process, SIGKILL)	TestAcquireOwnership_CrashRecovery
stale leftover lock file	TestAcquireOwnership_StaleLockFileReclaimed
version mismatch	TestCheckMinServerVersion, TestServerCompatibilityGate, TestPrepareLeaseExtender_UnsupportedRouteStopsLoudly
supported takeover	TestTakeoverDaemonOwner_ShutsDownAndWaits
rolling-upgrade sweep	TestDetectUnlockedDaemon
profile-scoped remedy	TestOwnershipConflict_ProfileAwareStopHint
break-glass	TestOwnershipBypassed

Scope note: this is code-level root-cause analysis plus a cross-process test
proving the lock mutually excludes two real OS processes on an isolated temp
base dir. Not run locally: go test -race ./... and the DB-gated handler
register tests (no local test Postgres) — both covered by this repo's backend
CI job, green on this PR.

Rollout

• Stop old daemons first. The upgrade sweep enforces this: a new daemon
  refuses to start while a pre-lock daemon is alive on any profile.
• Server-first (or matched) upgrade for the version handshake: a new daemon
  against an old server that predates server_version warns and relies on the
  /prepare-lease backstop rather than refusing startup.
• Rollback: export MULTICA_DAEMON_ALLOW_MULTIPLE=1 and restart — restores the
  prior no-guard behaviour without redeploying.

🤖 Generated with Claude Code

[vercel]

@fsafey is attempting to deploy a commit to the IndexLabs Team on Vercel.

A member of the Team first needs to authorize it.

[fsafey]

Updated: rebased onto current main (ad7b23896, v0.4.3) and folded in cross-model review findings — (1) launcher now refuses to start while a pre-lock daemon is alive on any profile's health port (closes the rolling-upgrade window where a new daemon could run alongside a legacy one), (2) a version-mismatch exit now deregisters runtimes on the way out, (3) handoff/takeover waits sized to the full graceful-shutdown bound (30s task drain + 5s deregister), (4) conflict error's stop hint is now profile-scoped. Single squashed commit d529be2e2; all suites + race pass locally; PR body updated to match.

[fsafey]

Second review pass (Claude Opus 4.8, high effort) on the previous revision surfaced four issues, all fixed in a2004f811: (1) prepare-lease 404 classification now positively matches chi's router catch-all instead of excluding known bodies — a transient handler-level 404 can no longer masquerade as version skew and permanently stop the lease extender; (2) --takeover no longer force-kills when shutdown delivery fails (that state is a normally-draining incumbent whose health listener already closed) — it waits, then fails loudly with a manual remedy; (3) the unlocked-daemon sweep skips foreign-environment daemons (WSL2/#3916 topology) and also runs on the direct --foreground path; (4) docs aligned with the 45s wait and no-force behavior. New tests cover each. Full suites + race green locally.