feat(serve): only offer machines that aren't busy (#2838)

Author: steipete

CHANGELOG.md

@@ -6,6 +6,10 @@
 
 - Coordinator: remove per-owner lease lifetime overrides from the production configuration; every owner uses the global lease policy, and the deploy guard now protects the complete live override document generically. [PR 2830](https://github.com/openclaw/crabbox/pull/2830).
 
+### Added
+
+- Serve: offer machines only while host CPU, memory, user activity, and power policies allow, with availability reasons, portal measurements, and a read-only local probe; running leases continue.
+
 ### Fixed
 
 - Release verification: accept provenance schema 4 (egress runtime packs) in the installed-Homebrew verifier, checking it against the same exact filesystem-layout archive inventory as schema 3. [PR 2840](https://github.com/openclaw/crabbox/pull/2840).

docs/commands/serve.md

@@ -9,6 +9,7 @@ crabbox serve --cpus 24 --memory 48g
 crabbox serve --runtime podman --cpus 8 --memory 16g --advertise-address 100.64.1.2
 crabbox serve machines --json
 crabbox serve status --json
+crabbox serve status --probe --json
 crabbox serve cache prune
 crabbox serve install --cpus 24 --memory 48g
 crabbox serve uninstall
@@ -28,6 +29,13 @@ crabbox serve uninstall
 | `--state-dir` | Private machine identity/status directory |
 | `--no-cache-volumes` | Disable managed cache volumes |
 | `--drain-on-exit` | Remove containers on orderly agent shutdown |
+| `--availability` | `auto` (default) or `always` to disable availability gating |
+| `--max-host-cpu` | Maximum non-serve host CPU fraction, default `0.5` |
+| `--require-user-idle` | Minimum console idle duration, default `0` on desktops or `10m` on laptops |
+| `--require-ac-power` | Require AC on laptops, default `true`; ignored on desktops |
+| `--max-memory-pressure` | Busy at `warn` or `critical`, default `critical` |
+| `--probe` | Read-only local availability sampling (`status` only); no coordinator needed |
+| `--probe-duration` | Probe observation period, `1m`–`1h`, default `7m` |
 
 `CRABBOX_SERVE_MEMORY` supplies the memory default. `serve.advertiseAddresses`
 or comma-separated `CRABBOX_SERVE_ADVERTISE_ADDRESSES` configures explicit
@@ -51,8 +59,9 @@ for details, fallback behavior, and image evidence.
 Each poll starts at least `--poll-interval` after the prior poll start and heartbeat response.
 
 `status` checks coordinator access and reads the local machine UUID, last successful coordinator contact,
-active assignments, caps, and draining state. `machines` queries owner-visible
-coordinator inventory. Both support `--json`.
+active assignments, caps, draining state, and availability with reasons. `machines` queries owner-visible
+coordinator inventory. Both support `--json`. See [availability](../providers/serve.md#availability) for
+signals, configuration, laptop defaults, and hysteresis.
 
 `install` writes and loads a per-user LaunchAgent on macOS or a systemd user
 unit on Linux, preserving the supplied flags. It uses the current executable

docs/features/portal.md

@@ -187,9 +187,11 @@ dedicated host opens the matching visibility-only detail page. Unattached host
 details and host-pinned launch commands require admin authentication.
 
 Owners enabled for [serve](../providers/serve.md) also see a **Machines** section
-below the fleet and a **machines online** readout when they have donated machines.
-It shows connection and draining status, last heartbeat age, used/offered and free
-CPU and memory, allocated lease links and limits, runtime and agent versions,
+below the fleet and a **machines available** readout when they have donated machines.
+It counts online, non-draining machines whose availability permits new leases.
+The section shows availability and reasons, measured host CPU, console idle time,
+power source, connection and draining status, last heartbeat age, used/offered and
+free CPU and memory, allocated lease links and limits, runtime and agent versions,
 labels, and the advertised SSH front door. Search and status filters apply only
 to this section. Machine visibility is owner- and organization-scoped, including
 for admins; disabled owners and owners without machines see no serve section.

docs/providers/serve.md

@@ -70,6 +70,9 @@ public lease keys, deadlines, and status, outside the repository.
 Every container gets a hard CPU limit, memory limit, and low CPU shares. The
 agent and coordinator enforce the offered CPU, memory, and `--max-leases` caps.
 CPU shares reduce contention but do not reserve idle CPU for the host.
+Availability sampling starts after the first reconciliation and image-preparation
+kickoff, and reuses reconciliation's container inventory. Busy or unknown
+availability does not delay image preparation or existing lease deadlines.
 The agent starts only its configured local-container image; a different image
 in an assignment fails with `serve_image_mismatch`.
 Containers have no host Docker socket, privileged mode, host network, or host
@@ -108,6 +111,69 @@ is retained for run reporting and cleanup. Fixed-ID acquisition does not switch
 providers. `CRABBOX_SERVE_MACHINE` and `CRABBOX_SERVE_FALLBACK` provide environment
 overrides.
 
+## Availability
+
+By default, a machine accepts new leases only while the owner is not using it.
+The agent samples host signals approximately every ten seconds, independently
+of heartbeats and deadline enforcement. Busy or unknown machines advertise zero
+free capacity; existing leases continue at their existing low CPU priority.
+Availability never kills a running lease.
+
+| Policy | Desktop/server default | Laptop default (battery detected) |
+| --- | --- | --- |
+| `--max-host-cpu` | `0.5` | `0.5` |
+| `--require-user-idle` | `0` (disabled) | `10m` |
+| `--require-ac-power` | Ignored | `true` |
+| `--max-memory-pressure` | `critical` | `critical` |
+
+CPU is a time-weighted 60-second average of total host utilization minus this
+agent's owned, labeled containers' runtime CPU usage, normalized by host logical
+cores and clamped at zero. On macOS the agent uses
+Mach host CPU counter deltas (or interval `top` samples in builds without cgo), the kernel memory-pressure
+level, IOHIDSystem console idle time, and `pmset` power information. On Linux it
+uses `/proc/stat`, `MemAvailable` (warning below 10%, critical below 5%), and
+sysfs power information. Linux console idle time is unknown: enabling an idle
+requirement there conservatively prevents availability. Unsupported or failed
+required measurements are unknown, never treated as idle. Unknown form factor
+uses laptop defaults.
+
+CPU above the threshold, memory pressure at or above the selected level, or
+battery power must persist for at least 60 seconds before becoming busy.
+Recent console activity makes the machine busy immediately. All required
+conditions must be clear for five minutes before availability returns, including
+at startup after the first complete CPU window. Unknown measurements immediately
+prevent new placement. `--availability always` explicitly disables gating.
+
+Configuration uses `serve.availability.mode`, `maxHostCPU`, `requireUserIdle`,
+`requireACPower`, and `maxMemoryPressure`. Flags override environment variables,
+which override configuration and detected defaults. The corresponding environment
+variables are `CRABBOX_SERVE_AVAILABILITY`, `CRABBOX_SERVE_MAX_HOST_CPU`,
+`CRABBOX_SERVE_REQUIRE_USER_IDLE`, `CRABBOX_SERVE_REQUIRE_AC_POWER`, and
+`CRABBOX_SERVE_MAX_MEMORY_PRESSURE`.
+
+```yaml
+serve:
+  availability:
+    mode: auto
+    maxHostCPU: 0.5
+    requireUserIdle: 10m
+    requireACPower: true
+    maxMemoryPressure: critical
+```
+
+`crabbox serve status` and `crabbox serve machines` show availability and reasons.
+The portal shows the same state, CPU, console idle time, and power source; its
+machine tile counts online, non-draining available machines. Legacy agents
+without availability data retain their previous placement behavior. Capacity
+errors include owner-visible machine states and reasons.
+
+For a read-only local measurement, run `crabbox serve status --probe --json`.
+It samples for seven minutes by default to include the startup and recovery windows,
+without registering, changing services, starting containers, or writing state.
+`--probe-duration 1m` through `1h` changes the observation period; a short probe
+may correctly report unknown while warming up. Use the agent's `--state-dir`
+and `--runtime` to subtract its own containers.
+
 ## Prepared container images
 
 After the first heartbeat and reconciliation, Serve probes the configured image
@@ -189,8 +255,8 @@ in 1–65535. The agent binds each advertised IPv4 address separately on that
 same port, never a wildcard address. If some binds fail, it logs the failures
 and advertises only the addresses that bound; no bound addresses is fatal.
 The agent re-detects candidates on each heartbeat and rebinds when addresses
-change, including DHCP and Wi-Fi changes. Capacity is controlled solely by
-`--max-leases` and the CPU/memory offer.
+change, including DHCP and Wi-Fi changes. Capacity is controlled by availability,
+`--max-leases`, and the CPU/memory offer.
 
 Automatic LAN candidates come from up, non-loopback, broadcast-capable
 interfaces in `10/8`, `172.16/12`, or `192.168/16`. Tunnel and runtime bridge

internal/cli/config.go

@@ -1743,11 +1743,20 @@ func baseConfig() Config {
 }
 
 type ServeConfig struct {
-	AdvertiseAddresses []string `yaml:"advertiseAddresses,omitempty"`
-	NoLANCandidates    bool     `yaml:"noLANCandidates,omitempty"`
-	SSHPort            int      `yaml:"sshPort,omitempty"`
-	Machine            string   `yaml:"machine,omitempty"`
-	Fallback           string   `yaml:"fallback,omitempty"`
+	Availability       ServeAvailabilityConfig `yaml:"availability,omitempty"`
+	AdvertiseAddresses []string                `yaml:"advertiseAddresses,omitempty"`
+	NoLANCandidates    bool                    `yaml:"noLANCandidates,omitempty"`
+	SSHPort            int                     `yaml:"sshPort,omitempty"`
+	Machine            string                  `yaml:"machine,omitempty"`
+	Fallback           string                  `yaml:"fallback,omitempty"`
+}
+
+type ServeAvailabilityConfig struct {
+	Mode              string   `yaml:"mode,omitempty"`
+	MaxHostCPU        *float64 `yaml:"maxHostCPU,omitempty"`
+	RequireUserIdle   string   `yaml:"requireUserIdle,omitempty"`
+	RequireACPower    *bool    `yaml:"requireACPower,omitempty"`
+	MaxMemoryPressure string   `yaml:"maxMemoryPressure,omitempty"`
 }
 
 type fileConfig struct {

internal/providers/localcontainer/served_stats.go

@@ -0,0 +1,44 @@
+package localcontainer
+
+import (
+	"context"
+	"fmt"
+	"math"
+	"strconv"
+	"strings"
+)
+
+// CPUCores reports runtime CPU usage in host-core units: 100% equals one core.
+// The caller supplies only its owned running containers, never a runtime-wide total.
+func (r *ServedRuntime) CPUCores(ctx context.Context, containers []ServedContainer) (float64, error) {
+	if len(containers) == 0 {
+		return 0, nil
+	}
+	args := []string{"stats", "--no-stream", "--no-trunc", "--format", "{{.ID}} {{.CPUPerc}}"}
+	ids := map[string]bool{}
+	for _, c := range containers {
+		ids[c.ID] = true
+		args = append(args, c.ID)
+	}
+	result, err := r.b.docker(ctx, args, nil, nil)
+	if err != nil {
+		return 0, err
+	}
+	total := 0.0
+	for _, line := range strings.Split(strings.TrimSpace(result.Stdout), "\n") {
+		fields := strings.Fields(line)
+		if len(fields) != 2 || !ids[fields[0]] || !strings.HasSuffix(fields[1], "%") {
+			return 0, fmt.Errorf("invalid owned-container CPU stats")
+		}
+		value, err := strconv.ParseFloat(strings.TrimSuffix(fields[1], "%"), 64)
+		if err != nil || value < 0 || math.IsNaN(value) || math.IsInf(value, 0) {
+			return 0, fmt.Errorf("invalid owned-container CPU usage")
+		}
+		total += value / 100
+		delete(ids, fields[0])
+	}
+	if len(ids) > 0 {
+		return 0, fmt.Errorf("missing owned-container CPU stats")
+	}
+	return total, nil
+}

internal/providers/localcontainer/served_stats_test.go

@@ -0,0 +1,44 @@
+package localcontainer
+
+import (
+	"context"
+	"fmt"
+	"reflect"
+	"testing"
+
+	core "github.com/openclaw/crabbox/internal/cli"
+)
+
+type statsRunner struct {
+	output string
+	args   []string
+}
+
+func (r *statsRunner) Run(_ context.Context, req core.LocalCommandRequest) (core.LocalCommandResult, error) {
+	r.args = req.Args
+	return core.LocalCommandResult{Stdout: r.output}, nil
+}
+func TestServedCPUCores(t *testing.T) {
+	for _, tc := range []struct {
+		output string
+		want   float64
+		bad    bool
+	}{
+		{"owned-a 150.00%\nowned-b 50.00%", 2, false},
+		{"owned-a 0.00%\nowned-b 0.00%", 0, false},
+		{"unowned 900.00%", 0, true}, {"owned-a 10.00%", 0, true},
+		{"owned-a NaN%\nowned-b 0%", 0, true}, {"owned-a -1%\nowned-b 0%", 0, true},
+	} {
+		t.Run(fmt.Sprint(tc.output), func(t *testing.T) {
+			runner := &statsRunner{output: tc.output}
+			rt := NewServedRuntime(core.Config{LocalContainer: core.LocalContainerConfig{Runtime: "docker"}}, core.Runtime{Exec: runner})
+			got, err := rt.CPUCores(t.Context(), []ServedContainer{{ID: "owned-a"}, {ID: "owned-b"}})
+			if (err != nil) != tc.bad || got != tc.want {
+				t.Fatalf("got %v %v", got, err)
+			}
+			if !reflect.DeepEqual(runner.args, []string{"stats", "--no-stream", "--no-trunc", "--format", "{{.ID}} {{.CPUPerc}}", "owned-a", "owned-b"}) {
+				t.Fatalf("unscoped stats: %v", runner.args)
+			}
+		})
+	}
+}