Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,15 @@
- CI: run CLI coverage in three isolated shards, retaining atomic whole-suite coverage and the 90% core gate; report the slowest top-level tests without summing nested timings.
- Coordinator monthly reserved-USD budgets are raised to $100,000 fleet-wide, $50,000 per owner and $100,000 per org; reservations from finished leases had exhausted the per-owner cap at roughly a quarter of the estimated spend.
- CLI: `crabbox list` streams and shows live leases by default; `--retained` includes kept terminal leases. Add `--jsonl` streaming and uncapped `--all` lease lookups while preserving the buffered `--json` array.
- Direct cleanup now expires `keep=true` leases and reduces running/provisioning expiry grace from 12 hours to 1 hour; unowned machines remain untouched.
- Direct leases: preserve provider user-data limits with compressed Tencent Cloud bootstrap, retain renewed leases during cleanup, and normalize Sealos ownership labels. [PR 2806](https://github.com/openclaw/crabbox/pull/2806).

### Added

- Blacksmith Testbox: configure the native total readiness wait (queue and hydration) through `blacksmith.waitTimeout`, `CRABBOX_BLACKSMITH_WAIT_TIMEOUT`, or `--blacksmith-wait-timeout`, preserving the native 15-minute default when unset.
- Serve leases can fall back to an owner-authorized, end-to-end encrypted coordinator relay when LAN and tailnet routes are unavailable.
- Coordinator: reconcile retained terminal AWS leases from the current index with scoped cleanup proof, bounded resumable passes, opt-in owned deletion, and `crabbox admin retained-reconcile` status.
- Direct Linux guest expiry guards across shared-bootstrap providers, plus `directLeases: deny` and trusted provider exceptions to require coordinator allocation.
- Serve agents advertise LAN and tailnet front-door candidates, with pinned-key LAN-first client selection and per-lease route caching.
- Portal: show owners their donated serve machines, online status, capacity, allocated leases, and runtime details.

Expand Down
21 changes: 15 additions & 6 deletions docs/commands/cleanup.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,20 +81,29 @@ What cleanup does depends on the selected provider:

### Deletion decisions for direct machines

Selection is label-driven. Cleanup reads the `keep`, `state`, `expires_at`, and
Selection is label-driven. Cleanup requires `crabbox=true` and reads `state`, `expires_at`, and
`ttl` labels written when the machine was created. The decision is conservative:

- skip machines labeled `keep=true`, even after recorded expiry;
- `keep=true` does not exempt a machine from expiry; it only suppresses automatic release after `run`;
- for `running` or `provisioning` machines, skip until well past expiry — delete
only once the expiry time plus a 12-hour stale window has elapsed;
only once the expiry time plus a 1-hour stale window has elapsed;
- for `leased`, `ready`, or `active` machines, delete once expired;
- always delete machines in `failed`, `released`, or `expired` states;
- for any other machine, delete only if `expires_at`/`ttl` parses and has passed;
skip if the expiry label is missing or still in the future.

Resources without these labels are skipped (`reason=missing labels`), so
Resources without `crabbox=true` are skipped (`reason=missing labels`), so
non-Crabbox machines are never touched.

Supported direct Linux guests also run an expiry guard every two minutes.
GCP self-deletes; AWS powers off and terminates; other covered providers power
off only. Stopped machines can remain billable (including Hetzner and Azure),
so run cleanup to delete them. See the
[provider coverage table](../features/lifecycle-cleanup.md#guest-side-expiry-guard).

`directLeases: deny` blocks new remote allocations, not cleanup, stop, list, or
status of existing machines.

### Namespace local cleanup

For `provider=namespace-devbox`, cleanup removes only Crabbox-owned Namespace SSH
Expand All @@ -115,12 +124,12 @@ For direct machine providers, each candidate prints one decision line. `--dry-ru
prints the same lines but makes no provider calls:

```text
skip server id=12345 name=crabbox-blue-lobster reason=keep=true
skip server id=12345 name=crabbox-blue-lobster reason=not expired
delete server id=67890 name=crabbox-amber-crab
stop server id=11223 name=crabbox-green-heron
```

`skip` lines include a `reason=` (for example `keep=true`, `state=running`,
`skip` lines include a `reason=` (for example `state=running`,
`missing expires_at`, `not expired`). Without `--dry-run`, each `delete` line is
followed by the actual provider delete call; a failed delete returns the provider
error and stops the sweep. Stop-only providers such as Hostinger print `stop`
Expand Down
40 changes: 38 additions & 2 deletions docs/features/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,51 @@ repository remain valid. An inherited user-config or environment credential is
not sent to a repository-defined destination unless the operator explicitly
overrides that destination through its environment variable or CLI flag.

## Direct lease policy

`directLeases: allow|deny` defaults to `allow`. Set it in trusted user/system
configuration to require coordinator-managed remote allocation on this machine:

```yaml
directLeases: deny
directLeaseAllow: [hetzner]
```

`CRABBOX_DIRECT_LEASES` overrides the policy; `CRABBOX_DIRECT_LEASE_ALLOW`
overrides the allowlist with comma-separated provider names. Names and aliases
resolve to canonical provider names. Repository configuration may tighten the
policy to `deny`; it cannot override an inherited deny with `allow` or add
allowlist entries. Repository `directLeaseAllow` is ignored. Environment is a
trusted operator override and can explicitly restore `allow`.

When denied, creation fails before configuring or calling the direct provider:

```text
direct-provider leases are disabled (directLeases: deny); configure a coordinator or add <provider> to directLeaseAllow
```

This applies to warmup/prewarm, new runs, fixed lease IDs, checkpoint forks,
and shards. Stop, cleanup, status, list, and runs against existing leases remain
available. A stale reused lease cannot be replaced with a new direct allocation.
A coordinator URL exempts only providers actually allocating through the
coordinator; registered-direct mode and providers without coordinator support
still obey the policy.

Providers that provision nothing remote are always exempt: `ssh`,
`local-container`, `docker-sandbox`, `apple-container`, `apple-machine`,
`apple-vm`, `anthropic-sandbox-runtime`, `mxc`, `multipass`, `firecracker`,
`lume`, `tart`, `hyperv`, and `windows-sandbox`. Providers capable of selecting
remote hosts, such as Parallels and Incus, are not automatically exempt.

## Precedence

```text
flags > env > repo-local crabbox.yaml/.crabbox.yaml > user config > defaults
```

Lowest precedence is applied first: defaults, then user config, then repo
config, then env vars, then flags. Each layer only overrides fields that are
explicitly set; unset fields fall through to the layer below.
config, then env vars, then flags. Except for the tightening-only repository direct-lease policy above, each layer
only overrides fields that are explicitly set; unset fields fall through to the layer below.

Lease-creating commands validate OS selector compatibility after resolving the
provider, target, and Windows mode from all layers. For example,
Expand Down
86 changes: 54 additions & 32 deletions docs/features/lifecycle-cleanup.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ Read this when:

A lease holds a remote box until it is released or expires. Two independent
paths reclaim the underlying resources: the **brokered** path, owned by the
coordinator, and the **direct** path, owned by the local CLI (and, for GCP, a
guest-side guard). Which one applies depends on whether the provider runs
coordinator, and the **direct** path, owned by the local CLI and supported Linux
guest-side guards. Which one applies depends on whether the provider runs
through a coordinator.

## Brokered lifecycle
Expand Down Expand Up @@ -587,36 +587,58 @@ Use `--dry-run` to print what would be deleted without touching anything. The
sweep is conservative; for each candidate machine `shouldCleanupServer`
(`internal/cli/pool.go`) decides from the machine's Crabbox labels:

- skip machines with no labels, or labeled `keep=true`, even after recorded expiry;
- `running` / `provisioning`: delete only when stale — past `expires_at` plus a
12-hour safety window;
- `leased` / `ready` / `active`: delete once past `expires_at`;
- `failed` / `released` / `expired`: delete;
- otherwise: delete once past `expires_at`, skip if `expires_at` is missing or
still in the future.

For this to work, every direct-provider machine must carry Crabbox labels/tags
(at least `crabbox`, `state`, and `expires_at`) so the sweep can identify owned
resources without touching unrelated infrastructure.

### GCP guest-side expiry guard

A direct GCP lease can outlive the local CLI that created it — if `cleanup`
never runs, the VM would leak. To guard against this, direct GCP leases install
a self-deleting guard (`cloudInitGCPExpiryGuardFiles` in
`internal/cli/bootstrap.go`): a systemd timer runs every 2 minutes, reads the
instance's own labels via the GCP metadata server, and deletes the instance when
it is clearly expired. Unlike the manual CLI sweep, this automatic expiry path
also reaps kept leases:

- exits unless `crabbox=true` and the provider label is absent or `gcp`;
- retains `keep=true` machines until their recorded expiry, leaving those without valid expiry metadata alone;
- `failed` / `released` / `expired`: delete;
- `running` / `provisioning`: delete only past `expires_at` plus 12 hours;
- `leased` / `ready` / `active` (and unlabeled state): delete once past
`expires_at`.

So an expired GCP box can reclaim itself even if the operator's machine is gone.
- skip machines unless `crabbox=true`; unrelated or unlabeled machines are never reclaimed;
- `running` / `provisioning`: reclaim only after `expires_at` plus **1 hour**;
- `leased` / `ready` / `active`: reclaim after `expires_at`;
- `failed` / `released` / `expired`: reclaim immediately;
- other states: reclaim only after a valid expiry; missing or invalid expiry is left alone.

`keep=true` only suppresses automatic release after `run`. It does **not**
exempt a direct lease from expiry, matching brokered leases. This changes the
old direct sweep behavior, which skipped kept leases and allowed a 12-hour
running/provisioning grace. Cleanup retains its legacy `ttl` expiry-label
fallback. New bootstraps use `expires_at` (Unix seconds).

### Guest-side expiry guard

Direct Linux VMs on the shared cloud-init path install one root-owned script,
`/usr/local/sbin/crabbox-expiry-guard`, and a systemd service/timer. The timer
runs every two minutes, beginning two minutes after boot, before optional
bootstrap work completes. It uses the same decision cases as the CLI sweep;
tests execute these cases through both Go and the shell guard. No provider
credentials are stored in guard files. Missing, ambiguous, or unavailable
ownership metadata fails closed. Expiry-based decisions require a valid deadline;
terminal `failed` / `released` / `expired` states remain immediately reclaimable.

| Direct Linux provider | Deadline source | Reclaim action |
| --- | --- | --- |
| GCP | Current labels fetched through the metadata server's service-account token and Compute API (existing mechanism) | Delete this VM through the Compute API |
| AWS | Instance tags via IMDSv2; new direct Linux launches enable instance metadata tags | Power off; `InstanceInitiatedShutdownBehavior=terminate` terminates the instance |
| Azure | IMDS `compute/tagsList` | `systemctl poweroff` |
| Hetzner | Root-owned `/etc/crabbox/lease-expiry` | `systemctl poweroff` |
| DigitalOcean, Vultr, Linode | Root-owned `/etc/crabbox/lease-expiry` | `systemctl poweroff` |
| OVHcloud, Scaleway, Nebius, Lambda, Tencent Cloud | Root-owned `/etc/crabbox/lease-expiry` | `systemctl poweroff` |
| Incus, Multipass, Firecracker | Root-owned `/etc/crabbox/lease-expiry` | `systemctl poweroff` |

Powering off stops the workload; it does **not** guarantee that billing stops.
Providers that bill stopped machines, including Hetzner, still require
`crabbox cleanup` to delete them. Azure guest shutdown does not deallocate its
VM. Provider disks and other retained resources can also remain billable.
GCP retains its existing self-delete permissions requirement.

File-backed bootstraps write exactly four nonsecret fields: `crabbox`, `state`,
`keep`, and `expires_at`. `RefreshGuestLeaseExpiry(ctx, target, fields)` in
`internal/cli/guest_expiry.go`
is the hook for a file-backed provider's lease-lifetime update: it validates the
fields and writes mode 0600 through the existing SSH transport, publishing by
atomic rename in `/etc/crabbox`. Ordinary CLI state touches and heartbeats refresh this file as well as provider
labels. Other deadline updates must use this hook, not only change provider labels. No new `extend` command is introduced here.

Brokered leases never install this guard; the coordinator owns their lifecycle.
Windows and macOS targets do not install it. Providers using prepared templates,
custom bootstrap, hosted sandboxes, delegated execution, or static hosts rather
than the shared cloud-init path also do not install it. The explicit uncovered
provider list is in [Providers](providers.md#direct-lease-expiry-coverage).

## Claims and `--reclaim`

Expand Down
37 changes: 36 additions & 1 deletion docs/features/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,40 @@ selection metadata, so registration and documentation drift fail the docs gate.
`crabbox providers --json` remains the low-level compiled spec. `crabbox doctor`
checks whether the selected provider is usable from the current environment.

## Direct lease expiry coverage

The shared Linux guest guard covers **AWS, Azure, GCP, Hetzner, DigitalOcean,
Vultr, Linode, OVHcloud, Scaleway, Nebius, Lambda, Tencent Cloud, Incus,
Multipass, and Firecracker**. See the [deadline source and reclaim action
table](lifecycle-cleanup.md#guest-side-expiry-guard). Coordinator-managed leases
and Windows/macOS targets are excluded.

The following providers do not use that shared Linux bootstrap and receive no
guard (some have provider-native expiry instead):

- `agent-sandbox`, `anthropic-sandbox-runtime`, `apple-container`, `apple-machine`, `apple-vm`, `ascii-box`, `aws-lambda-microvm`, `azure-dynamic-sessions`.
- `azure-sandbox`, `blacksmith-testbox`, `blaxel`, `boxd`, `cloud-run-sandbox`, `cloudflare`, `cloudflare-dynamic-workers`, `cloudflare-sandbox`.
- `coder`, `codesandbox`, `crownest`, `cua`, `cubesandbox`, `daytona`, `docker-sandbox`, `e2b`.
- `exe-dev`, `external`, `fastapi-cloud`, `freestyle`, `github-codespaces`, `hostinger`, `hyperv`, `islo`.
- `kubevirt`, `local-container`, `lume`, `machine0`, `modal`, `morph`, `mxc`, `namespace-devbox`.
- `namespace-instance`, `nomad`, `nvidia-brev`, `opencomputer`, `opensandbox`, `orgo`, `parallels`, `phala`.
- `proxmox`, `railway`, `runpod`, `sealos-devbox`, `semaphore`, `serve`, `smolvm`, `sprites`.
- `ssh`, `superserve`, `tart`, `tenki`, `tensorlake`, `unikraft-cloud`, `upstash-box`, `vast`.
- `vercel-sandbox`, `wandb`, `windows-sandbox`, `xcp-ng`.

In particular, prepared/custom-bootstrapped VMs such as Proxmox, XCP-ng,
Parallels, and Apple VM remain uncovered; their existing provider cleanup still
applies. This guard does not add a bootstrap path to hosted sandboxes or static
hosts. A guest shutdown is not deletion: stopped machines can remain billable
and still need `crabbox cleanup`.

To forbid new direct remote allocations, set
[`directLeases: deny`](configuration.md#direct-lease-policy), with optional
trusted `directLeaseAllow` exceptions. The always-exempt local/static set is
`ssh`, `local-container`, `docker-sandbox`, `apple-container`, `apple-machine`,
`apple-vm`, `anthropic-sandbox-runtime`, `mxc`, `multipass`, `firecracker`,
`lume`, `tart`, `hyperv`, and `windows-sandbox`.

## Machine classes

An explicitly selected canonical `--class tiny|small|standard|fast|large|beast` maps to an
Expand Down Expand Up @@ -166,7 +200,8 @@ describes retry order.

`crabbox list` marks brokered machines as `orphan=no-active-lease` when their
provider label references a lease no longer active in the coordinator. This is an
operator hint only; `keep=true` machines are never deleted automatically.
operator hint only; listing does not delete machines. Recorded expiry still
applies regardless of `keep=true`.

The structured quota preflight and `provisioningAttempts` metadata belong to the
brokered Worker path; direct AWS fallback can still retry provider types but
Expand Down
15 changes: 8 additions & 7 deletions docs/providers/gcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -466,13 +466,14 @@ Three independent safety nets enforce expiry:

- Direct GCP VMs set Compute Engine `maxRunDuration` with termination action
`DELETE`, so the TTL hard cap is enforced by the platform.
- Each VM installs a guest-side `crabbox-gcp-expiry-guard` systemd timer that
reads live instance labels through the metadata service and Compute API, then
self-deletes expired leases, including kept warmups, when the attached service
account can delete the VM. New VMs receive this behavior; existing VMs retain
their installed guard and must be recreated to receive the updated idle reaper.
Manual `crabbox cleanup` continues to skip kept leases; use explicit `stop`
to remove an existing kept VM.
- New direct Linux VMs install the shared `crabbox-expiry-guard` systemd timer.
It reads live instance labels through the metadata service and Compute API,
then self-deletes expired leases, including kept warmups, when the attached
service account can delete the VM. The timer runs every two minutes; running
and provisioning states have a one-hour expiry grace. Existing VMs retain
their installed guard and must be recreated to receive this version.
Manual `crabbox cleanup` also expires kept leases. Brokered leases do not
install this guard; their coordinator owns expiry.
- Cleanup removes stale local GCP claim files whose lease IDs no longer appear in
provider inventory.

Expand Down
4 changes: 4 additions & 0 deletions docs/providers/tencentcloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ through CVM user data, writes Crabbox ownership metadata as Tencent Cloud tags,
waits for public IPv4 and SSH readiness, then uses the normal Crabbox
SSH/sync/run/stop/cleanup path.

Linux cloud-init is gzip-compressed before base64 encoding. Crabbox keeps the
encoded user-data field within CVM's 16 KiB limit and rejects oversized payloads
before requesting an instance.

The provider is **direct-only** in this release. It does not run through the
coordinator, so the local CLI must have Tencent Cloud API credentials and direct
cleanup remains the operator's responsibility.
Expand Down
2 changes: 1 addition & 1 deletion internal/cli/actions.go
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ func (a App) actionsHydrate(ctx context.Context, args []string) (err error) {
}
defer stopHeartbeat()
} else if sshBackend, ok := backend.(SSHLeaseBackend); ok {
_, err := sshBackend.Touch(ctx, TouchRequest{Lease: LeaseTarget{Server: server, SSH: target, LeaseID: leaseID}, State: blank(server.Labels["state"], "ready"), IdleTimeout: cfg.IdleTimeout})
_, err := touchLeaseWithGuestExpiry(ctx, sshBackend, cfg, TouchRequest{Lease: LeaseTarget{Server: server, SSH: target, LeaseID: leaseID}, State: blank(server.Labels["state"], "ready"), IdleTimeout: cfg.IdleTimeout})
if err != nil {
fmt.Fprintf(a.Stderr, "warning: touch failed for %s: %v\n", leaseID, err)
}
Expand Down
Loading
Loading