[RFC] Report node resource utilization through /resource

[WenYuLuo]

Summary

Extend the existing node-resource Unix socket GET /resource response with a cached utilization object so an external scheduler can temporarily avoid nodes under resource pressure. This is an advisory protection signal; sandboxd reports observations and the scheduler owns isolation and recovery policy.

{
  "cpu": 32,
  "mem": 68719476736,
  "xpu": [],
  "storage": 17179869184,
  "features": ["storage-quota-v1"],
  "utilization": {
    "cpu": 0.92,
    "memory": 0.87,
    "pid": 0.88,
    "fd": 0.22,
    "disk": 0.75
  }
}

Motivation

Scheduler capacity and actual resource consumption answer different questions. A node can still advertise capacity while CPU, memory, task limits, file handles, or its writable-storage filesystem are nearly exhausted. The existing external resource query already provides the transport needed to expose these observations.

Contract

All five keys are always present. Numbers are fractions in [0, 1]; null means no usable observation, including the first CPU sample or an unavailable finite limit. A valid zero means zero consumption. Existing capacity fields and feature semantics stay compatible. Consumers ignore unknown JSON fields.

Sample in the existing five-second resource refresh loop and serve the latest cached observation. No extra request per metric, timestamps, freshness negotiation, or scheduling policy is required. Sampling errors affect the relevant observations and do not fail the HTTP response or sandboxd health check. Where a metric combines several sources, use the maximum valid ratio; return null only when none is usable. This is best-effort coverage of the procfs and cgroup hierarchy visible to sandboxd.

Sampling

Field	Proposed initial implementation
cpu	Maximum of host busy-time fraction from /proc/stat and finite cgroup CPU-quota consumption. Host busy excludes idle and iowait; guest time is not counted twice. Cgroup consumption is delta CPU time / elapsed time / quota cores.
memory	Maximum of (MemTotal - MemAvailable) / MemTotal and finite cgroup memory usage/limit ratios. Cgroup usage includes charged cache.
pid	Maximum pids.current / pids.max at applicable cgroup roots and visible ancestors. Counts tasks including threads. Unlimited or absent limits supply no ratio.
fd	Maximum of system allocated file handles / maximum handles from /proc/sys/fs/file-nr, and sandboxd's open descriptors / soft open-file limit from /proc/self. This first version does not scan other control-plane processes.
disk	1 - Bavail / Blocks from the same statfs(FilestoreDir) snapshot used to report storage. This is physical filesystem occupancy before storage overcommit.

Cgroup v1 and v2 are supported. Observe sandboxd's own hierarchy and the configured sandbox root when cgroup management is enabled, walking only visible ancestors. Pair each usage with the limit at that same cgroup, and do not scan individual sandbox cgroups. CPU cpuset-only saturation and host-wide task exhaustion without a finite cgroup task limit are outside the initial metric coverage. Cgroup discovery is read-only and refreshed as part of sampling.

disk specifically identifies the filesystem backing the reported storage, including loop-backed filestore mode. It is not the largest occupancy across arbitrary mounts, and it does not measure directory size, inodes, or disk I/O. Without usable filestore statistics it is null.

Consumer integration

FunctionSystem can extend its existing external collector to extract the utilization object from its current /resource request, then pass observations through its resource-update path. LocalScheduler owns configurable thresholds and consecutive-sample counters. Entry should require sustained pressure, while recovery can be more permissive (for example six high samples to enter, two low samples to recover).

The consumer can maintain CPU_PRESSURE, MEMORY_PRESSURE, PID_PRESSURE, FD_PRESSURE, and DISK_PRESSURE independently. Health isolation must remain separate from node lifecycle and manual isolation so recovery only removes the matching pressure. FunctionSystem changes are follow-up work.

Compatibility and validation

• Preserve existing capacity units, storage overcommit, and storage-quota-v1 behavior.
• Verify cached JSON over the actual Unix HTTP socket, including all five keys, zero, null, and independent sampling failures.
• Cover host CPU deltas and resets, memory availability, v1/v2 cgroup limits and ancestor scope, unlimited task limits, and FD accounting using deterministic fixtures.
• Verify storage and disk use one filesystem snapshot and that overcommit does not alter disk utilization.
• Run formatting, Go vet, package tests, and focused race tests on Linux.

Reference semantics: Linux procfs, cgroup v2, and cgroup v1.

Implementation

The initial implementation is available in PR #50, including the resource contract documentation and Linux sampling/socket tests.