feat(cloud-agent-next): hold Kilo under memory pressure and explain restarts

.specs/cloud-agent-session.md

@@ -116,6 +116,13 @@ repository.
    descendant progress MUST count for its ancestors, never unrelated roots.
    Requests waiting on the user MUST pause only the no-progress clock when no
    independent work is runnable. Resolving one request MUST NOT clear another.
+   A Kilo that stops answering while the sandbox reclaims at its memory cap
+   MUST be held instead of restarted, for at most ten minutes; the hold MUST
+   pause the no-progress clock and post a non-fatal notice. A turn failed by the
+   hang restart that ends an expired hold MUST say that the sandbox ran out of
+   memory. A turn failed by another hang restart MUST say that Kilo was not
+   responding, unless the restart was for exhausted activity-tracking capacity
+   while Kilo still answered.
    The separate 120-minute execution cap MUST include user waits. Native
    cancellation MUST be bounded and confirmed; unconfirmed cancellation MUST
    use bounded runtime recovery. Accepted Cloud messages MUST settle once;

apps/mobile/src/components/agents/safe-failure-messages.ts

@@ -0,0 +1,92 @@
+/**
+ * The reader's copy the Durable Object writes through its safe failure
+ * projection (services/cloud-agent-next/src/session/safe-failure-projection.ts
+ * and the assistant failures it re-exports from src/shared/assistant-failure.ts)
+ * plus the lines session-service.ts supplies directly. None of them is raw
+ * provider text, so the status line shows them as-is. A bounded workspace
+ * failure appends its detail to the projection line, so a message that starts
+ * with one of these plus ": " is the same copy.
+ */
+const SAFE_FAILURE_MESSAGES = new Set([
+  // Generic failure codes.
+  'Could not connect to the sandbox',
+  'Workspace setup failed',
+  'Kilo server failed to start',
+  'Agent wrapper failed to start',
+  'The message could not be delivered',
+  'Session metadata is unavailable',
+  'No model was selected',
+  'Agent wrapper disconnected',
+  'Agent wrapper made no execution progress during the watchdog window',
+  'Agent wrapper stopped responding',
+  'Agent wrapper failed before processing the message',
+  'Assistant request failed',
+  'Agent wrapper failed while processing the message',
+  'No assistant reply was produced',
+  'Assistant request failed: insufficient credits',
+  'The message was interrupted by the user',
+  'The agent container shut down',
+  'The message was interrupted',
+  'The message failed',
+  // Workspace failure subtypes.
+  'Repository clone timed out',
+  'Repository checkout timed out',
+  'Repository authentication failed',
+  'Repository request was rate limited',
+  'Repository network request failed',
+  'Repository data is corrupt',
+  'Repository checkout conflict',
+  'Requested repository branch was not found',
+  'Workspace setup failed: sandbox storage full',
+  'Session import timed out',
+  'Session import failed',
+  'Setup command timed out',
+  'Setup command failed',
+  // Classified assistant failures.
+  'Assistant request was rate limited',
+  'Assistant request failed: model not found',
+  'Assistant request was not authorized',
+  'Assistant service is unavailable',
+  'Assistant request timed out',
+  'Assistant request was invalid',
+  'The model context limit was exceeded',
+  'The model output limit was reached',
+  'The model provider blocked the response under its content policy',
+  'The model response did not match the required format',
+  // Lines session-service.ts supplies as `safeFailureMessage`.
+  'GitHub repository authentication failed. Check that the GitHub App is installed and has access to this repository.',
+  'GitHub credential service is unavailable. Please try again.',
+  'GitHub credential resolution failed. Please try again.',
+  // The SDK's autocommit status.
+  'Commit failed',
+  // Control-plane failure reasons
+  // (services/cloud-agent-next/src/control-plane/session/failure-messages.ts).
+  'Environment preparation timed out',
+  'The agent became unavailable',
+  'Sandbox billing requires additional credits',
+  'Sandbox billing is unavailable',
+  'Container limit reached (20 per personal account or 50 per organization). Stop an existing container before starting another.',
+  'Sandbox configuration is invalid or unsupported',
+  'The sandbox connection was lost',
+  'The sandbox was lost',
+  'The agent restarted',
+  'The sandbox ran out of memory',
+  'Kilo was not responding and was restarted',
+  'The turn made no progress',
+  'The turn did not complete',
+  'Prompt delivery failed',
+  'The sandbox stopped while waiting for an answer',
+  'The turn exceeded its time limit',
+]);
+
+export function isSafeFailureMessage(message: string): boolean {
+  if (SAFE_FAILURE_MESSAGES.has(message)) {
+    return true;
+  }
+  for (const safe of SAFE_FAILURE_MESSAGES) {
+    if (message.startsWith(`${safe}: `)) {
+      return true;
+    }
+  }
+  return false;
+}

apps/mobile/src/components/agents/session-terminal-error.test.ts

@@ -428,6 +428,8 @@ describe('sessionStatusErrorMessage', () => {
     ['Assistant request was rate limited'],
     ['Session metadata is unavailable'],
     ['Commit failed'],
+    ['Kilo was not responding and was restarted'],
+    ['The turn made no progress'],
     // A bounded workspace failure appends its own detail to the projection.
     ['Workspace setup failed: Devcontainer workspace preparation failed'],
   ] as const)('shows the safe projection copy for %s', raw => {

apps/mobile/src/components/agents/session-terminal-error.ts

@@ -4,6 +4,7 @@ import { i18n } from '@/i18n';
 import { type QueryErrorVariant } from '@/components/query-error';
 
 import { type MessageFailure } from './message-failure-state';
+import { isSafeFailureMessage } from './safe-failure-messages';
 
 /**
  * Terminal error class for a session startup failure. The session manager's
@@ -183,81 +184,6 @@ function errorClassForCode(code: SdkStatusMessageCode): TerminalErrorClass | und
   return ERROR_CLASS_BY_CODE[code as keyof typeof ERROR_CLASS_BY_CODE];
 }
 
-/**
- * The reader's copy the Durable Object writes through its safe failure
- * projection (services/cloud-agent-next/src/session/safe-failure-projection.ts
- * and the assistant failures it re-exports from src/shared/assistant-failure.ts)
- * plus the lines session-service.ts supplies directly. None of them is raw
- * provider text, so the status line shows them as-is. A bounded workspace
- * failure appends its detail to the projection line, so a message that starts
- * with one of these plus ": " is the same copy.
- */
-const SAFE_FAILURE_MESSAGES = new Set([
-  // Generic failure codes.
-  'Could not connect to the sandbox',
-  'Workspace setup failed',
-  'Kilo server failed to start',
-  'Agent wrapper failed to start',
-  'The message could not be delivered',
-  'Session metadata is unavailable',
-  'No model was selected',
-  'Agent wrapper disconnected',
-  'Agent wrapper made no execution progress during the watchdog window',
-  'Agent wrapper stopped responding',
-  'Agent wrapper failed before processing the message',
-  'Assistant request failed',
-  'Agent wrapper failed while processing the message',
-  'No assistant reply was produced',
-  'Assistant request failed: insufficient credits',
-  'The message was interrupted by the user',
-  'The agent container shut down',
-  'The message was interrupted',
-  'The message failed',
-  // Workspace failure subtypes.
-  'Repository clone timed out',
-  'Repository checkout timed out',
-  'Repository authentication failed',
-  'Repository request was rate limited',
-  'Repository network request failed',
-  'Repository data is corrupt',
-  'Repository checkout conflict',
-  'Requested repository branch was not found',
-  'Workspace setup failed: sandbox storage full',
-  'Session import timed out',
-  'Session import failed',
-  'Setup command timed out',
-  'Setup command failed',
-  // Classified assistant failures.
-  'Assistant request was rate limited',
-  'Assistant request failed: model not found',
-  'Assistant request was not authorized',
-  'Assistant service is unavailable',
-  'Assistant request timed out',
-  'Assistant request was invalid',
-  'The model context limit was exceeded',
-  'The model output limit was reached',
-  'The model provider blocked the response under its content policy',
-  'The model response did not match the required format',
-  // Lines session-service.ts supplies as `safeFailureMessage`.
-  'GitHub repository authentication failed. Check that the GitHub App is installed and has access to this repository.',
-  'GitHub credential service is unavailable. Please try again.',
-  'GitHub credential resolution failed. Please try again.',
-  // The SDK's autocommit status.
-  'Commit failed',
-]);
-
-function isSafeFailureMessage(message: string): boolean {
-  if (SAFE_FAILURE_MESSAGES.has(message)) {
-    return true;
-  }
-  for (const safe of SAFE_FAILURE_MESSAGES) {
-    if (message.startsWith(`${safe}: `)) {
-      return true;
-    }
-  }
-  return false;
-}
-
 /**
  * The reader's own copy for a session error in the transcript's status slot
  * (session-status-indicator.tsx). A code the SDK attaches to its own fixed copy

services/cloud-agent-next/DEBUG.md

@@ -180,7 +180,7 @@ logs/control/<sandboxId>/<allocationId>/<wrapperInstanceId>/files.tar.gz
 
 For a `no_progress` or `execution_limit` outcome, find the `session.execution` record with `phase=deadline_expired`. The native session supervisor emits it for every expired Kilo execution, including unrouted and subagent work. It records `reason`, `nativeRuntimeId` and the expired `kiloSessionId`, plus `sessionId` and `rootKiloSessionId` when the execution belongs to a routed chat. `no_progress` means 20 minutes passed without real assistant or tool progress; a silent running tool does not hold that clock, and only waits on the user pause it. `execution_limit` is the separate 120-minute wall-clock cap. The wrapper no longer reads native message parts before a deadline, so there is no last-tool or `phase=freshness` probe evidence; use the Kilo CLI log in the file archive for tool detail.
 
-`control.workload` records are per `scopeId`, so count distinct scopes to see whether the allocation was shared. `cpuUsageUsec`, `cpuThrottledUsec`, `cpuThrottleCount`, `ioReadBytes`, and `ioWriteBytes` are cumulative cgroup counters: compare consecutive samples from the same scope. Stats sample every 30 seconds, plus sooner (at most once per 10 seconds) when a throttle or memory-event counter increases or the group first crosses 90% of its memory cap (the pressure signal releases only below 80%), so consecutive samples may be farther apart than the old one-second cadence. `toolCpuUsageUsec`, `serverCpuUsageUsec`, `toolIoReadBytes`, and `toolIoWriteBytes` separate the tools subgroup from the Kilo server; they do not identify one command when several run together. `memoryMaxEvents` counts attempts to cross the memory limit and can rise without an OOM kill. `anonBytes`, `fileBytes`, and `shmemBytes` are the group `memory.stat` split, and `toolCurrentBytes`/`serverCurrentBytes` (with peaks) attribute the total to the tools subgroup or the Kilo server, so a full headroom can be traced to page cache from tool or Docker storage I/O (`file`) versus process memory (`anon`). `pressureAvailable` reports whether the kernel exposes `memory.pressure` (PSI); when it is false, rely on `memory.events` and `memory.stat`. A `health_probe_false` restart logs the latest snapshot for that runtime's own scope under the restart line's `workload` key, and a restart with the group at 95% of its cap fails the turn as `sandbox_out_of_memory` instead of `agent_restarted`. The Worker `container_launch_identity` diagnostic joins the physical allocation reference and sandbox name to Cloudflare's 64-character `containerInstanceId`, which can be queried in Cloudflare's historical Containers metrics. These counters show resource activity or contention, not useful tool progress.
+`control.workload` records are per `scopeId`, so count distinct scopes to see whether the allocation was shared. `cpuUsageUsec`, `cpuThrottledUsec`, `cpuThrottleCount`, `ioReadBytes`, and `ioWriteBytes` are cumulative cgroup counters: compare consecutive samples from the same scope. Stats sample every 30 seconds, plus sooner (at most once per 10 seconds) when a throttle or memory-event counter increases or the group first crosses 90% of its memory cap (the pressure signal releases only below 80%), so consecutive samples may be farther apart than the old one-second cadence. `toolCpuUsageUsec`, `serverCpuUsageUsec`, `toolIoReadBytes`, and `toolIoWriteBytes` separate the tools subgroup from the Kilo server; they do not identify one command when several run together. `memoryMaxEvents` counts attempts to cross the memory limit and can rise without an OOM kill. `anonBytes`, `fileBytes`, and `shmemBytes` are the group `memory.stat` split, and `toolCurrentBytes`/`serverCurrentBytes` (with peaks) attribute the total to the tools subgroup or the Kilo server, so a full headroom can be traced to page cache from tool or Docker storage I/O (`file`) versus process memory (`anon`). Per-subgroup `toolAnonBytes`/`toolFileBytes`/`toolShmemBytes` and `server*` equivalents come from each subgroup's `memory.stat`. `*FileBytes` includes shmem; without swap only the rest is reclaimable. Memory a direct Kilo child allocates before the sweep moves it into tools stays charged to the server. A `workloadPhase: "protection"` record per scope reports the server's `memory.min` (`serverMinBytes`), or `phase: "failed"` when the kernel refused it. `wrapper.lifecycle` `kilo_memory_hold_started`/`kilo_memory_hold_ended` mark a hang restart held while the workload reclaimed at its cap, with `currentBytes`, `aggregateMaxBytes`, `memoryMaxEvents`, and on end `memoryHoldOutcome` and `elapsedMs`. `pressureAvailable` reports whether the kernel exposes `memory.pressure` (PSI); when it is false, rely on `memory.events` and `memory.stat`. A `health_probe_false` restart logs the latest snapshot for that runtime's own scope under the restart line's `workload` key, and a hang restart after an expired memory hold fails the turn as `sandbox_out_of_memory`. The Worker `container_launch_identity` diagnostic joins the physical allocation reference and sandbox name to Cloudflare's 64-character `containerInstanceId`, which can be queried in Cloudflare's historical Containers metrics. These counters show resource activity or contention, not useful tool progress.
 
 List and download these JSON batches and the overwriteable `files.tar.gz` object from R2 using local tooling and the key prefix above. Each wrapper incarnation keeps one file archive; later uploads replace it. Uploaded objects remain available after the container disappears, subject to the bucket's retention policy. The legacy `getWrapperLogs` live-file reader and tarball retrieval do not read these control-plane archives.