Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 14 additions & 10 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ Creation-time network policies support unrestricted networking, blocking new
flows except the YuanRong control and published sandbox-port routes, or denying
exact and leading-wildcard DNS names.
Experimental whole-device NVIDIA GPU requests require runsc. Configurable
writable-storage requests are supported by runsc and Firecracker.
writable-storage requests and hard limits are supported by runsc and
Firecracker.

Use AKernel when a task needs an isolated remote environment with command
execution, file operations, interactive PTYs, port forwarding, or reverse
Expand Down Expand Up @@ -196,7 +197,8 @@ a configured runtime as an advertised runtime.

Firecracker supports commands, files, PTYs, network policies, published ports,
reverse tunnels, read-only EROFS image roots and mounts, explicit `storage_mb`
quotas, and recovery across sandboxd restarts. Its root and filesystem image
requests and `storage_limit_mb` hard limits, and recovery across sandboxd
restarts. Its root and filesystem image
mounts must be local or image-provider-backed regular EROFS files. It rejects
OCI/Nydus directory roots, directory mounts, writable live host binds, NVIDIA
GPUs, and nested KVM rather than weakening their semantics.
Expand All @@ -213,9 +215,9 @@ direct configuration, build an image with `AKERNEL_ENABLE_RUNC=true`, then use
`AKERNEL_ENABLE_RUNC=true` for standalone,
`node.config.sandboxd.enableRunc=true` for Helm, or `enable_runc=true` for
Terraform. Runc uses the host kernel and therefore has a different isolation
boundary from runsc. It does not support experimental GPU or explicit
`storage_mb` requests. Its optional `enableKVM` extra configuration requires a
usable `/dev/kvm` device.
boundary from runsc. It does not support experimental GPU, explicit
`storage_mb` requests, or `storage_limit_mb` limits. Its optional `enableKVM`
extra configuration requires a usable `/dev/kvm` device.

The bundled sandboxd configuration enables per-sandbox network ACLs. Pooled TAP
networking requires the host `tun` module and a usable `/dev/net/tun`. The
Expand Down Expand Up @@ -390,9 +392,10 @@ explicit override for multi-homed environments.

The standalone sandboxd filestore is a loop-mounted ext4 image under the
bind-mounted `deploy/standalone/data/` directory. Explicit `storage_mb`
quotas for runsc and Firecracker use this local-disk filestore. Without an
explicit quota, runsc retains its configured memory-backed overlay while
Firecracker creates its configured sparse ext4 default.
requests and `storage_limit_mb` hard limits for runsc and Firecracker use this
local-disk filestore. Without explicit storage values, runsc retains its
configured memory-backed overlay while Firecracker creates its configured
sparse ext4 default.

Terraform-managed Alibaba Cloud node pools instead attach a dedicated 300 GiB
ESSD by default, have ACK format it as XFS, and mount it at `/home/akernel`.
Expand Down Expand Up @@ -509,9 +512,10 @@ python sdk/python/tests/integration/test_sandbox.py -v

python sdk/python/benchmarks/sandbox_pressure.py --runtime firecracker
python sdk/python/benchmarks/sandbox_pressure.py \
--runtime firecracker --storage-mb 256
--runtime firecracker --storage-mb 256 --storage-limit-mb 512
python sdk/python/benchmarks/sandbox_pressure.py \
--xpu gpu:a10:1 --storage-mb 256 --processes 1 --threads 1
--xpu gpu:a10:1 --storage-mb 256 --storage-limit-mb 512 \
--processes 1 --threads 1
```

## Maintenance Rules
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,9 @@ with Sandbox(xpu="gpu:l20:1") as sandbox:
```

GPU sandboxes require a compatible NVIDIA node and the gVisor `runsc`
runtime. `storage_mb` is measured in MiB and is supported by `runsc` and
Firecracker.
runtime. Writable storage controls are measured in MiB: `storage_mb` requests
scheduler capacity, while `storage_limit_mb` sets the rootfs writable-layer
hard limit. Both are supported by `runsc` and Firecracker.

See the complete [basic usage example](./sdk/python/examples/basic_usage.py), the [sandbox runtime example](./sdk/python/examples/sandbox_runtime.py), and the other [SDK examples](./sdk/python/examples/) for more operations.

Expand Down
31 changes: 22 additions & 9 deletions sdk/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ Sandbox(
*,
xpu: str | None = None,
storage_mb: int | None = None,
storage_limit_mb: int | None = None,
network_policy: NetworkPolicy | None = None,
dockerfile: DockerfileLaunch | None = None,
extra_config: Mapping[str, object] | None = None,
Expand All @@ -125,17 +126,29 @@ supported. The bundled backend currently requires the gVisor `runsc` runtime
and a node configured for gVisor nvproxy. Runtime compatibility is validated
by the backend rather than the SDK.

Set the writable root filesystem quota in MiB:
Set the writable root filesystem scheduling request and hard limit in MiB:

```python
with Sandbox(storage_mb=20 * 1024) as sandbox:
with Sandbox(storage_mb=10 * 1024, storage_limit_mb=20 * 1024) as sandbox:
print(sandbox.commands.run("df -h /").stdout)
```

The bundled backend currently requires `runsc` for an explicit `storage_mb`
quota and uses sandboxd's disk-backed XFS filestore. Runtime compatibility is
validated by the backend. When `storage_mb` is omitted, sandboxd retains its
configured default 10 GiB memory-backed writable overlay. See
`storage_mb` is the amount reserved by the scheduler. `storage_limit_mb` is
the writable root filesystem's hard limit. Both default to `None`:

| `storage_mb` | `storage_limit_mb` | Behavior |
|---|---|---|
| `None` | `None` | No explicit storage reservation; use the runtime's configured writable-layer limit. |
| request | `None` | Reserve the request; use the same value as the hard limit. |
| `None` | limit | Reserve the limit and use it as the hard limit. |
| request | limit | Reserve the request and enforce the limit; the limit must be at least the request. |

Explicit storage values are supported by `runsc` and Firecracker. Runtime
compatibility is validated by the backend. With neither value set, the bundled
deployment keeps its configured 10 GiB writable-layer limit: runsc uses its
memory-backed overlay and Firecracker uses a sparse ext4 overlay image.
`SandboxInfo` reports the requested values; it does not resolve an omitted
value to the runtime default. See
[`examples/gpu_sandbox.py`](./examples/gpu_sandbox.py) and
[`examples/storage_sandbox.py`](./examples/storage_sandbox.py).

Expand Down Expand Up @@ -238,8 +251,8 @@ with Sandbox(
`enableKVM` is owned by the runc backend and requires a usable `/dev/kvm` on
the selected node. Runc supports OCI/EROFS root filesystems, read-only mounts,
networking, command execution, and the default writable overlay. Experimental
GPU requests remain runsc-only; explicit `storage_mb` quotas are supported by
runsc and Firecracker. See the
GPU requests remain runsc-only; explicit `storage_mb` requests and
`storage_limit_mb` limits are supported by runsc and Firecracker. See the
[sandbox runtime comparison](../../src/sandboxd/doc/runtime.md) for the
runtime capability boundaries.

Expand Down Expand Up @@ -590,7 +603,7 @@ not part of the default test suite.
| `CommandResult` | `stdout`, `stderr`, `exit_code` |
| `CommandInfo` | `pid`, `command`, `running` |
| `EntryInfo` | `name`, `path`, `type`, `size`, `permissions`, `modified_time` |
| `SandboxInfo` | `id`, `state`, `cpu`, `memory`, `image`, `xpu`, `storage_mb` |
| `SandboxInfo` | `id`, `state`, `cpu`, `memory`, `image`, `xpu`, `storage_mb`, `storage_limit_mb` |
| `NodeInfo` | `id`, `status`, `capacity`, `allocatable`, `labels` |
| `S3Config` | `endpoint`, `bucket`, `object`, optional credentials |
| `Mount` | `target`, one source, and `type` |
Expand Down
1 change: 1 addition & 0 deletions sdk/python/akernel_sdk/_backends/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ class SandboxSpec:
node_id: str | None
xpu: str | None
storage_mb: int | None
storage_limit_mb: int | None
network_policy: NetworkPolicy | None
extra_config: Mapping[str, object]

Expand Down
4 changes: 4 additions & 0 deletions sdk/python/akernel_sdk/_backends/openyuanrong_sandbox.py
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,7 @@ def get_info(self) -> SandboxInfo:
image=value.image,
xpu=self._spec.xpu,
storage_mb=self._spec.storage_mb,
storage_limit_mb=self._spec.storage_limit_mb,
)

def reload(self) -> bool:
Expand Down Expand Up @@ -415,6 +416,9 @@ def create(self, spec: SandboxSpec) -> BackendSession:
node_id=spec.node_id,
xpu=spec.xpu,
storage_mb=spec.storage_mb,
storage_limit_mb=(
spec.storage_limit_mb if spec.storage_limit_mb is not None else 0
),
network=network,
extra_config=dict(spec.extra_config),
create_timeout=create_timeout,
Expand Down
2 changes: 2 additions & 0 deletions sdk/python/akernel_sdk/_backends/openyuanrong_sdk.py
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,7 @@ def get_info(self) -> SandboxInfo:
image=self._spec.image,
xpu=self._spec.xpu,
storage_mb=self._spec.storage_mb,
storage_limit_mb=self._spec.storage_limit_mb,
)

def reload(self) -> bool:
Expand Down Expand Up @@ -306,6 +307,7 @@ def create(self, spec: SandboxSpec) -> BackendSession:
node_id=spec.node_id,
xpu=spec.xpu,
storage_mb=spec.storage_mb,
storage_limit_mb=spec.storage_limit_mb,
network_policy=spec.network_policy,
extra_config=spec.extra_config,
)
Expand Down
14 changes: 10 additions & 4 deletions sdk/python/akernel_sdk/_backends/openyuanrong_sdk_impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
from .._sandbox_resources import (
normalize_xpu,
storage_bytes,
validate_storage_mb,
validate_storage,
xpu_custom_resource,
)
from ..types import (
Expand Down Expand Up @@ -155,6 +155,7 @@ def build_options(
node_id: str | None,
xpu: str | None,
storage_mb: int | None,
storage_limit_mb: int | None,
network_policy: NetworkPolicy | None,
extra_config: Mapping[str, object],
) -> Any:
Expand All @@ -171,7 +172,7 @@ def build_options(
if mem_limit and mem_limit < memory:
raise ValueError("mem_limit must be 0 or greater than or equal to memory")
normalized_xpu = normalize_xpu(xpu)
validate_storage_mb(storage_mb)
validate_storage(storage_mb, storage_limit_mb)

options = yr.InvokeOptions()
# A Sandbox is driven by one sequential SDK client. Disabling ordered RPC
Expand Down Expand Up @@ -201,8 +202,13 @@ def build_options(
if normalized_xpu is not None:
resource_name, count = xpu_custom_resource(normalized_xpu)
options.custom_resources[resource_name] = count
if storage_mb is not None:
options.custom_resources["storage"] = storage_bytes(storage_mb)
storage_request_mb = storage_mb if storage_mb is not None else storage_limit_mb
if storage_request_mb is not None:
options.custom_resources["storage"] = storage_bytes(storage_request_mb)
if storage_limit_mb is not None:
options.custom_extensions["STORAGE_LIMIT"] = str(
int(storage_bytes(storage_limit_mb))
)
if network_policy is not None:
options.custom_extensions["network_policy"] = json.dumps(
network_policy.to_dict()
Expand Down
2 changes: 1 addition & 1 deletion sdk/python/akernel_sdk/_dockerfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ def _resolve_entrypoint(


_IGNORED_INSTRUCTIONS: dict[str, str] = {
"VOLUME": "not supported; use storage_mb or mounts for persistence",
"VOLUME": "not supported; use storage_mb/storage_limit_mb or mounts",
"LABEL": "not supported",
"HEALTHCHECK": "not supported",
"SHELL": "not supported",
Expand Down
34 changes: 28 additions & 6 deletions sdk/python/akernel_sdk/_sandbox_resources.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,21 +59,43 @@ def xpu_custom_resource(value: str) -> tuple[str, float]:
return f"{xpu_type.upper()}/{re.escape(model)}/count", float(count_text)


def validate_storage_mb(value: int | None) -> None:
"""Validate a writable-layer quota accepted by YuanRong's scalar wire type."""
def _validate_storage_value(name: str, value: int | None) -> None:
"""Validate one MiB storage value accepted by YuanRong's scalar wire type."""

if value is None:
return
if isinstance(value, bool) or not isinstance(value, int):
raise TypeError("storage_mb must be an integer")
raise TypeError(f"{name} must be an integer")
if value <= 0:
raise ValueError("storage_mb must be greater than 0")
raise ValueError(f"{name} must be greater than 0")
if value > MAX_STORAGE_MB:
raise ValueError(f"storage_mb must not exceed {MAX_STORAGE_MB}")
raise ValueError(f"{name} must not exceed {MAX_STORAGE_MB}")


def validate_storage_mb(value: int | None) -> None:
"""Validate a writable-layer scheduling request in MiB."""

_validate_storage_value("storage_mb", value)


def validate_storage(
storage_mb: int | None,
storage_limit_mb: int | None,
) -> None:
"""Validate writable-layer request and hard-limit values in MiB."""

validate_storage_mb(storage_mb)
_validate_storage_value("storage_limit_mb", storage_limit_mb)
if (
storage_mb is not None
and storage_limit_mb is not None
and storage_limit_mb < storage_mb
):
raise ValueError("storage_limit_mb must be greater than or equal to storage_mb")


def storage_bytes(value: int) -> float:
"""Convert a validated MiB quota to YuanRong's byte-valued scalar."""
"""Convert a validated positive MiB value to YuanRong's byte scalar."""

validate_storage_mb(value)
return float(value * _MIB)
23 changes: 18 additions & 5 deletions sdk/python/akernel_sdk/sandbox.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
from ._backends.base import BackendSession, SandboxSpec
from ._backends.registry import load_backend
from ._dockerfile_launch import DockerfileLaunch
from ._sandbox_resources import normalize_xpu, validate_storage_mb
from ._sandbox_resources import normalize_xpu, validate_storage
from .commands import CommandHandle, Commands
from .filesystem import Filesystem
from .pty import Pty
Expand Down Expand Up @@ -195,6 +195,7 @@ def __init__(
failover: bool = False,
xpu: str | None = None,
storage_mb: int | None = None,
storage_limit_mb: int | None = None,
network_policy: NetworkPolicy | None = None,
dockerfile: DockerfileLaunch | None = None,
extra_config: Mapping[str, object] | None = None,
Expand Down Expand Up @@ -227,9 +228,13 @@ def __init__(
``type:model:count`` format. Currently only exact-model NVIDIA
GPU requests are supported. The backend validates runtime
compatibility.
storage_mb: Experimental writable root filesystem quota in MiB.
When omitted, the configured default is used. Explicit quotas
are validated against the selected runtime by the backend.
storage_mb: Writable root filesystem scheduling request in MiB.
When no separate limit is given, this is also the writable
layer's hard limit. ``None`` makes no explicit storage request.
storage_limit_mb: Writable root filesystem hard limit in MiB.
It must be greater than or equal to ``storage_mb`` when both
are set. ``None`` follows ``storage_mb`` or the runtime's
configured default.
network_policy: Optional creation-time network policy. Omitting it
leaves sandbox networking unrestricted.
dockerfile: Supported Dockerfile direct-launch configuration.
Expand Down Expand Up @@ -271,7 +276,7 @@ def __init__(
if not runtime:
raise ValueError("runtime must be a non-empty string")
normalized_xpu = normalize_xpu(xpu)
validate_storage_mb(storage_mb)
validate_storage(storage_mb, storage_limit_mb)
if network_policy is not None and not isinstance(
network_policy, NetworkPolicy
):
Expand Down Expand Up @@ -347,6 +352,7 @@ def __init__(
self._memory = memory
self._xpu = normalized_xpu
self._storage_mb = storage_mb
self._storage_limit_mb = storage_limit_mb
self._id = ""

spec = SandboxSpec(
Expand All @@ -370,6 +376,7 @@ def __init__(
node_id=node_id,
xpu=normalized_xpu,
storage_mb=storage_mb,
storage_limit_mb=storage_limit_mb,
network_policy=(
None
if network_policy is None or network_policy.is_empty
Expand Down Expand Up @@ -519,6 +526,7 @@ def get_info(self) -> SandboxInfo:
image=self._image,
xpu=self._xpu,
storage_mb=self._storage_mb,
storage_limit_mb=self._storage_limit_mb,
)
info = self._session.get_info()
return SandboxInfo(
Expand All @@ -533,6 +541,11 @@ def get_info(self) -> SandboxInfo:
if info.storage_mb is not None
else self._storage_mb
),
storage_limit_mb=(
info.storage_limit_mb
if info.storage_limit_mb is not None
else self._storage_limit_mb
),
)

def kill(self) -> None:
Expand Down
1 change: 1 addition & 0 deletions sdk/python/akernel_sdk/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ class SandboxInfo:
image: str | None
xpu: str | None = None
storage_mb: int | None = None
storage_limit_mb: int | None = None


@dataclass(frozen=True)
Expand Down
Loading