fix(daemon): enforce single-owner machine lock and loud version-skew

docs/daemon-single-owner.md

@@ -0,0 +1,122 @@
+# Single-daemon ownership and version compatibility
+
+A machine may run at most **one** Multica agent daemon at a time. Two daemons on
+one host — most commonly the CLI-spawned daemon and the Desktop-spawned daemon
+running under different profiles — share the same local checkouts and the same
+in-process local-directory mutex, so both can write the same repository at once
+and corrupt it (VWO-364 / VWO-365).
+
+This is enforced by a **machine-global advisory lock** and a **client/server
+version gate**. This document is the operator reference for both.
+
+## How ownership is enforced
+
+- On startup, before it binds its health port or registers any runtime, a
+  daemon takes an exclusive advisory lock on `~/.multica/daemon.lock`.
+- That path lives in the **base** config directory, which every profile shares,
+  so the lock is contended across profiles — unlike the per-profile health-port
+  and PID-file guards, which never see a daemon under a different profile.
+- The lock is held for the life of the process via an open file handle. The OS
+  releases it automatically when the process exits — cleanly **or on a crash** —
+  so there is no lease TTL to tune and no stale-lock window. A crashed owner's
+  lock is gone the instant the process dies.
+- A second daemon that tries to start while an owner is live fails immediately
+  with an actionable error naming the incumbent (pid, version, profile, health
+  port) instead of silently double-registering.
+- The launcher also refuses to start while a daemon that does NOT hold the lock
+  is alive on any known profile's health port — that can only be a daemon from a
+  release predating single-daemon ownership (or one started under the
+  break-glass env). This closes the rolling-upgrade window: when upgrading, stop
+  every old daemon (`multica [--profile <name>] daemon stop`) before starting
+  the new one; the sweep enforces it rather than silently running alongside.
+
+## How version compatibility is enforced
+
+- The server reports its build version in the daemon registration response
+  (`server_version`). The daemon compares it against the minimum server version
+  it requires (currently the first release carrying the task `prepare-lease`
+  route) and, if the server is **older**, fails to start with a clear error
+  rather than 404-looping on unsupported routes forever.
+- A server that reports **no** version (older than this field, or built without
+  a version stamp) is treated as "unknown": the daemon warns and proceeds. If it
+  then hits a missing route (a bare `prepare-lease` 404), it logs one loud error
+  and stops that retry loop — it never floods the log with repeated 404s.
+
+## Operator verification
+
+Confirm exactly one daemon owns the machine:
+
+```bash
+# 1. Exactly one daemon process.
+pgrep -fl 'multica daemon start --foreground'
+
+# 2. The lock records the live owner (pid, health port, version, profile).
+cat ~/.multica/daemon.lock
+
+# 3. The recorded pid is the running daemon.
+ps -p "$(python3 -c 'import json,os;print(json.load(open(os.path.expanduser("~/.multica/daemon.lock")))["pid"])')"
+
+# 4. Health endpoint agrees.
+multica daemon status
+```
+
+Confirm the second daemon is refused (safe to run — it will not start):
+
+```bash
+# With a daemon already running, this must exit non-zero with an ownership error:
+multica daemon start --profile some-other-profile
+# => "another Multica daemon already owns this machine (pid …) … `multica daemon stop` … `--takeover`"
+```
+
+Confirm version compatibility on a self-hosted server:
+
+```bash
+# The server build the daemon negotiated against.
+multica daemon logs | grep -Ei 'server version|incompatible|prepare-lease'
+```
+
+## Handing ownership over (supported takeover)
+
+To replace the running daemon (e.g. switch the owner from the CLI daemon to the
+Desktop daemon) without editing any file or database row:
+
+```bash
+# Ask the current owner to stop, wait for it to release, then start:
+multica daemon start --takeover
+# (or `multica daemon restart` for a same-profile stop+start)
+```
+
+`--takeover` reads the incumbent's health port from the lock file, requests a
+graceful shutdown over HTTP (cross-platform; no OS signals), waits up to 45s for
+the lock to free (sized to the incumbent's full graceful shutdown: a 30s
+in-flight task drain plus deregistration), then acquires it. If the incumbent
+does not release in time it fails loudly with a manual remedy — it never
+force-kills, since that would skip the task drain and could orphan agent
+subprocesses mid-write.
+
+## Rollback / break-glass
+
+The ownership guard can be disabled without redeploying:
+
+```bash
+# Restore the pre-change behavior (NO single-owner guard). Only for a deliberate
+# multi-backend setup that accepts the shared-checkout risk, or emergency rollback.
+export MULTICA_DAEMON_ALLOW_MULTIPLE=1
+multica daemon start
+```
+
+With the variable set (any value other than empty / `0` / `false`), the daemon
+skips lock acquisition and logs a loud warning on every start. Unset it and
+restart to restore the guard. The version gate is independent and is not
+affected by this variable.
+
+## Recovery scenarios
+
+| Situation | What happens | Operator action |
+|---|---|---|
+| Owner exits cleanly | Lock released on shutdown | Next daemon starts normally |
+| Owner crashes / is killed | OS drops the lock on process death | Next daemon starts immediately; no cleanup |
+| Leftover `daemon.lock` from a dead owner | File content is advisory only; no live lock | Next daemon acquires and overwrites it; no manual deletion |
+| Second daemon attempted | Refused with an actionable error | Use `daemon stop`, or `daemon start --takeover` |
+| Old (pre-lock) daemon still running during an upgrade | New daemon refuses to start alongside it | Stop the old daemon (`multica [--profile <name>] daemon stop`), then start |
+| Daemon newer than server | Startup fails with a version error (or one loud `prepare-lease` error) | Upgrade the server, or run a matching daemon |

server/cmd/multica/cmd_daemon.go

@@ -11,6 +11,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"runtime"
 	"sort"
 	"strconv"
 	"strings"
@@ -81,6 +82,7 @@ var daemonDiskUsageCmd = &cobra.Command{
 func init() {
 	f := daemonStartCmd.Flags()
 	f.Bool("foreground", false, "Run in the foreground instead of background")
+	f.Bool("takeover", false, "If another daemon already owns this machine, ask it to stop and take over instead of failing")
 	f.String("daemon-id", "", "Unique daemon identifier (env: MULTICA_DAEMON_ID)")
 	f.String("device-name", "", "Human-readable device name (env: MULTICA_DAEMON_DEVICE_NAME)")
 	f.String("runtime-name", "", "Runtime display name (env: MULTICA_AGENT_RUNTIME_NAME)")
@@ -101,6 +103,7 @@ func init() {
 	// restart shares all the same flags as start
 	rf := daemonRestartCmd.Flags()
 	rf.Bool("foreground", false, "Run in the foreground instead of background")
+	rf.Bool("takeover", false, "If another daemon already owns this machine, ask it to stop and take over instead of failing")
 	rf.String("daemon-id", "", "Unique daemon identifier (env: MULTICA_DAEMON_ID)")
 	rf.String("device-name", "", "Human-readable device name (env: MULTICA_DAEMON_DEVICE_NAME)")
 	rf.String("runtime-name", "", "Runtime display name (env: MULTICA_AGENT_RUNTIME_NAME)")
@@ -234,6 +237,179 @@ func healthPortForProfile(profile string) int {
 	return daemon.DefaultHealthPort + 1 + (h % 1000)
 }
 
+// prepareDaemonOwnership makes sure it is safe to (re)start a daemon on this
+// machine before one is spawned. At most one daemon may own the machine-global
+// lock at cli.ProfileDir("")/daemon.lock, which spans every profile — so it
+// catches a Desktop-spawned daemon running under a different profile that the
+// per-profile health-port guard cannot see (VWO-365).
+//
+// It runs in the LAUNCHER purely to give a fast, actionable error (or perform a
+// requested takeover); the spawned daemon's Run re-acquires the lock for real.
+//
+//   - break-glass (MULTICA_DAEMON_ALLOW_MULTIPLE set) → skip.
+//   - lock free → nil.
+//   - lock held, --takeover → ask the incumbent to stop, wait for release.
+//   - lock held, no --takeover → the actionable conflict error.
+//
+// ownershipHandoffWait bounds how long the launcher waits for a shutting-down
+// incumbent to release the machine lock. It covers the gap between a daemon's
+// health port closing (at the start of shutdown) and its ownership lock
+// releasing at the very end, so a `daemon restart`, a stop-then-start, or a
+// --takeover does not spuriously report a conflict/timeout mid-shutdown.
+// Sized to the full graceful-shutdown bound: pollLoop waits up to 30s for
+// in-flight tasks, then deregisterRuntimes has a 5s timeout — 45s adds margin.
+const ownershipHandoffWait = 45 * time.Second
+
+func prepareDaemonOwnership(cmd *cobra.Command) error {
+	if daemon.OwnershipBypassed() {
+		return nil
+	}
+	baseDir, err := cli.ProfileDir("")
+	if err != nil {
+		return fmt.Errorf("resolve base config dir for daemon ownership lock: %w", err)
+	}
+	free, incumbent, ok, err := daemon.ProbeOwnership(baseDir)
+	if err != nil {
+		return err
+	}
+	if !free {
+		if takeover, _ := cmd.Flags().GetBool("takeover"); takeover {
+			if err := takeoverDaemonOwner(baseDir, incumbent); err != nil {
+				return err
+			}
+		} else if ok && incumbent.HealthPort > 0 && daemonAliveOnPort(incumbent.HealthPort) {
+			// The lock is held and the incumbent is genuinely live: reject
+			// immediately with an actionable error.
+			return ownershipConflictErr(baseDir, incumbent, ok)
+		} else if !waitForOwnershipFree(baseDir, ownershipHandoffWait) {
+			// Not answering health means it is shutting down — the daemon a
+			// `restart`/`stop` just asked to exit, still draining tasks and
+			// deregistering before it releases the lock — so wait a bounded
+			// time for the clean handoff rather than reject a legitimate
+			// restart. A wedged daemon that never releases lands here.
+			return ownershipConflictErr(baseDir, incumbent, ok)
+		}
+	}
+	// The lock is (now) free. A daemon can still be running WITHOUT it: one
+	// from a release predating single-daemon ownership, or one started under
+	// the break-glass env. Starting alongside it would recreate the exact
+	// two-writer hazard the lock exists to prevent, so sweep every known
+	// profile's health port and refuse while such a daemon is alive.
+	return detectUnlockedDaemon(baseDir)
+}
+
+// detectUnlockedDaemon scans every known profile's health port for a daemon
+// running WITHOUT holding the machine lock. Only reachable while the lock is
+// free — and a lock-aware daemon acquires the lock BEFORE it binds its health
+// port — so an alive health endpoint here can only be a daemon from a pre-lock
+// release or one bypassing the guard. This closes the rolling-upgrade blind
+// spot: without it, upgrading the CLI while a legacy daemon runs under another
+// profile (e.g. the Desktop's desktop-<host> profile) would let the new daemon
+// start alongside it. Ports are derived from the profile directories that
+// actually exist plus the default profile, so the sweep is a handful of
+// fail-fast local probes, not a port scan.
+func detectUnlockedDaemon(baseDir string) error {
+	if daemon.OwnershipBypassed() {
+		return nil
+	}
+	ports := map[int]string{daemon.DefaultHealthPort: ""} // port -> profile ("" = default)
+	if entries, err := os.ReadDir(filepath.Join(baseDir, "profiles")); err == nil {
+		for _, e := range entries {
+			if e.IsDir() {
+				ports[healthPortForProfile(e.Name())] = e.Name()
+			}
+		}
+	}
+	for port, profile := range ports {
+		health := probeDaemonHealth(port)
+		if !daemonAlive(health) {
+			continue
+		}
+		// A daemon visible on localhost but running in an environment this
+		// launcher can't manage — e.g. a Linux daemon inside WSL2 reachable
+		// through Windows localhost forwarding (#3916) — is not sharing this
+		// home's checkouts the way a native daemon does, and its lifecycle is
+		// not ours to gate on. Skip it rather than refuse with a stop hint the
+		// user can't act on. A daemon that reports no OS (older release) is
+		// treated as native: fail safe toward refusing.
+		if hostOS, _ := health["os"].(string); hostOS != "" && hostOS != runtime.GOOS {
+			continue
+		}
+		label := "the default profile"
+		stopHint := "`multica daemon stop`"
+		if profile != "" {
+			label = fmt.Sprintf("profile %q", profile)
+			stopHint = fmt.Sprintf("`multica --profile %s daemon stop`", profile)
+		}
+		return fmt.Errorf(
+			"a daemon that does not hold the machine ownership lock is running for %s (health port %d) — likely an older Multica release from before single-daemon ownership; stop it with %s and retry, or set MULTICA_DAEMON_ALLOW_MULTIPLE=1 to bypass the guard at your own risk",
+			label, port, stopHint,
+		)
+	}
+	return nil
+}
+
+func ownershipConflictErr(baseDir string, incumbent daemon.OwnerInfo, hasInfo bool) error {
+	return &daemon.OwnershipConflict{Path: daemon.OwnershipLockPath(baseDir), Incumbent: incumbent, HasInfo: hasInfo}
+}
+
+// daemonAliveOnPort reports whether a daemon health endpoint answers on port.
+func daemonAliveOnPort(port int) bool {
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	return daemonAlive(checkDaemonHealthOnPort(ctx, port))
+}
+
+// probeDaemonHealth is an indirection over checkDaemonHealthOnPort so tests can
+// stub port liveness instead of binding real well-known ports (which would
+// collide with an actual daemon on a developer machine). Returns the raw
+// health map; a dead port yields a map daemonAlive() rejects.
+var probeDaemonHealth = func(port int) map[string]any {
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	return checkDaemonHealthOnPort(ctx, port)
+}
+
+// waitForOwnershipFree polls until the machine lock is free or timeout elapses.
+func waitForOwnershipFree(baseDir string, timeout time.Duration) bool {
+	deadline := time.Now().Add(timeout)
+	for time.Now().Before(deadline) {
+		if free, _, _, err := daemon.ProbeOwnership(baseDir); err == nil && free {
+			return true
+		}
+		time.Sleep(200 * time.Millisecond)
+	}
+	return false
+}
+
+// takeoverDaemonOwner asks the incumbent daemon to shut down (via its recorded
+// health port — cross-platform, no OS signals) and waits until it releases the
+// machine lock. The OS drops the incumbent's advisory lock the instant its
+// process exits, so a successful shutdown frees the lock with no stale window.
+func takeoverDaemonOwner(baseDir string, incumbent daemon.OwnerInfo) error {
+	if incumbent.HealthPort <= 0 {
+		return fmt.Errorf("cannot take over: owner health port unknown (owner pid %d); stop it manually with `multica daemon stop`", incumbent.PID)
+	}
+	fmt.Fprintf(os.Stderr, "Taking over from daemon pid %d (profile %q) on health port %d...\n",
+		incumbent.PID, incumbent.Profile, incumbent.HealthPort)
+	if err := requestDaemonShutdown(incumbent.HealthPort); err != nil {
+		// A refused connection here is EXPECTED when the incumbent is already
+		// shutting down: it closes its health listener at the start of shutdown
+		// while holding the lock through the 30s task drain + deregister. Never
+		// force-kill on this path — SIGKILL would skip the drain and the
+		// deregister, and orphaned agent subprocesses could still be writing a
+		// checkout when the new owner starts dispatching. Fall through to the
+		// bounded wait; a genuinely wedged incumbent fails loudly below with a
+		// manual remedy instead of being forced.
+		fmt.Fprintf(os.Stderr, "Shutdown request not delivered (%v); waiting for the daemon to release the machine on its own...\n", err)
+	}
+	if waitForOwnershipFree(baseDir, ownershipHandoffWait) {
+		fmt.Fprintln(os.Stderr, "Previous daemon released the machine; continuing.")
+		return nil
+	}
+	return fmt.Errorf("takeover timed out: daemon pid %d did not release the machine lock within %s (it may still be draining an in-flight task); retry, or if it is truly stuck stop it manually (kill %d) and start again", incumbent.PID, ownershipHandoffWait, incumbent.PID)
+}
+
 // --- daemon start ---
 
 func runDaemonStart(cmd *cobra.Command, _ []string) error {
@@ -261,6 +437,14 @@ func runDaemonBackground(cmd *cobra.Command) error {
 		return fmt.Errorf("%s is already running (pid %v). Use 'daemon restart' to restart it", label, int(pid))
 	}
 
+	// Reject (or take over) a daemon owning this machine under ANY profile — the
+	// cross-profile case the same-profile health check above misses (VWO-365).
+	// Done before spawning so the user sees an actionable error immediately, not
+	// the generic "may not have started" timeout after the child fails to lock.
+	if err := prepareDaemonOwnership(cmd); err != nil {
+		return err
+	}
+
 	// Resolve current executable so the foreground child reuses this binary.
 	exePath, err := selfexec.Resolve()
 	if err != nil {
@@ -424,6 +608,30 @@ func runDaemonForeground(cmd *cobra.Command) error {
 
 	profile := resolveProfile(cmd)
 
+	// Honor --takeover on a direct `daemon start --foreground --takeover`: free
+	// the incumbent before Run tries to acquire the machine lock. Without
+	// --takeover we skip the lock pre-check — Run's acquire is the sole
+	// authority and returns the conflict error itself — so the common
+	// background→foreground child path (which inherits no --takeover) does not
+	// re-probe the lock.
+	if takeover, _ := cmd.Flags().GetBool("takeover"); takeover {
+		if err := prepareDaemonOwnership(cmd); err != nil {
+			return err
+		}
+	} else if baseDir, err := cli.ProfileDir(""); err == nil {
+		// The unlocked-daemon sweep, however, must run on EVERY launch path:
+		// Run's lock acquire cannot see a pre-lock daemon (it holds no lock), so
+		// without this the documented --foreground debugging path could start
+		// alongside a legacy daemon during the rolling-upgrade window. Cheap —
+		// a handful of fail-fast local probes — and a no-op re-check when this
+		// process is the background launcher's child (the launcher swept
+		// moments earlier). The takeover branch above already ends with the
+		// same sweep inside prepareDaemonOwnership.
+		if err := detectUnlockedDaemon(baseDir); err != nil {
+			return err
+		}
+	}
+
 	// Pick the log sink. A user who runs `daemon start --foreground` in a shell
 	// keeps live, colored logging on their terminal (a documented debugging
 	// path — see docs troubleshooting). A detached/background child, whose