fix(copilot): recover from a dead spawned Copilot runtime process

AGENTS.md

@@ -151,7 +151,7 @@ step-by-step checklist.
   - `base.py` - `AgentProvider` ABC defining `execute()`, `validate_connection()`, `close()`
   - `_output_shape.py` - `normalize_agent_output(content, schema)` — the single entry point providers call before `validate_output` (issue #343). It raises `ValidationError` when the parsed response is not a JSON object (a bare `42`/`null`/array), because `validate_output` would otherwise either raise `TypeError` from a membership test (numbers, booleans, null) or report a misleading "missing required field" (strings, arrays). It then applies `unwrap_scalar_wrappers`: fires only when the schema declares `string`/`number`/`boolean`, a `dict` arrived, and **exactly one** candidate slot has the expected type. Candidate slots are the field's own name plus the generic `value`/`result` keys, deduped so a field literally named `value` or `result` isn't rejected as ambiguous against itself. Two matches count as ambiguous; any other key shape is ignored. Both are left untouched (same object identity) so the caller re-prompts rather than guessing — this is what stops `{"answer": {"error": "..."}}` being laundered into an answer. Every unwrap logs a warning, naming discarded sibling keys when there are any. Kept out of `executor/output.py` on purpose — see the note there.
   - `_recovery_prompt.py` - `build_parse_recovery_prompt(...)` — the plain-text re-prompt shared by Copilot and Hermes (issue #343). Both providers correct an unusable response the same way (error + truncated response + rendered schema, with distinct schema-failure vs syntax-failure wording), and that text is covered by the provider-parity rule, so it lives in one place instead of two copies free to drift. Claude is deliberately not a caller: it re-prompts through its `emit_output` tool and never echoes the schema, so its instruction text stays in `claude.py::_build_recovery_instruction`.
-  - `copilot.py` - GitHub Copilot SDK implementation. By default spawns a nested `copilot` runtime via `CopilotClient()` (in `_build_client`, called from `_ensure_client_started`). When a runtime connection is resolved (`runtime.provider.runtime_url` or `COPILOT_PROVIDER_RUNTIME_URL`, optional `runtime_token` / `COPILOT_PROVIDER_RUNTIME_TOKEN`), it instead builds `CopilotClient(connection=RuntimeConnection.for_uri(url, connection_token=token))` to connect to an already-running `copilot --headless` process; the SDK skips spawning for URI connections and its `stop()` leaves the externally-owned server running. `_resolve_runtime_connection()` reads YAML first, then the namespaced env var (env activates on its own — the zero-YAML path for external orchestrators). Runtime transport can be combined with custom model-provider routing.
+  - `copilot.py` - GitHub Copilot SDK implementation. By default spawns a nested `copilot` runtime via `CopilotClient()` (in `_build_client`, called from `_ensure_client_started`). When a runtime connection is resolved (`runtime.provider.runtime_url` or `COPILOT_PROVIDER_RUNTIME_URL`, optional `runtime_token` / `COPILOT_PROVIDER_RUNTIME_TOKEN`), it instead builds `CopilotClient(connection=RuntimeConnection.for_uri(url, connection_token=token))` to connect to an already-running `copilot --headless` process; the SDK skips spawning for URI connections and its `stop()` leaves the externally-owned server running. `_resolve_runtime_connection()` reads YAML first, then the namespaced env var (env activates on its own — the zero-YAML path for external orchestrators). Runtime transport can be combined with custom model-provider routing. `_ensure_client_started()` also detects a spawned runtime whose child process has died (issue #483, `_runtime_is_dead()` polling the SDK's private `_cli_process` handle) and rebuilds the client under `_start_lock` (`_restart_spawned_runtime`) before returning, so the next SDK call lands on a fresh runtime; the rebuild invalidates `_client`/`_started` before constructing and starting the replacement, so a failed rebuild (e.g. OOM at spawn) cannot leave the provider believing a never-started client is started. A fixed, non-configurable cap (`_MAX_CONSECUTIVE_RUNTIME_RESTARTS`) on consecutive restarts with no intervening successful call prevents an infinite crash loop; a broken connection to an externally-owned runtime is never respawned. `_spawned_runtime_process` reads `_cli_process` (the spawned-child handle, `None` for URI and FFI connections) while `_fix_pipe_blocking_mode` reads `_process` (the transport handle — a `SocketWrapper` in TCP mode, an `_FfiProcessAdapter` with its own `poll()` in FFI mode); both are correct for their own purpose, and unifying the two reads onto `_process` would make FFI mode look like a killable child process.
   - `claude.py` - Anthropic Claude API provider using `pydantic-ai` (`AnthropicModel`) and the internal `_pydantic_ai` package (`converters`, `events`, `mcp_toolset`, `agent_builder`, `interrupt`, `retry`, `structured_output`, `usage`)
   - `claude_agent_sdk.py` - Claude Agent SDK implementation (uses `claude-agent-sdk` package)
   - `factory.py` - Provider instantiation

CHANGELOG.md

@@ -64,6 +64,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   either fail with a sharing violation, leaving a stale record behind. Both
   paths now use the same bounded retry that `write_run_record` already used
   for its own `os.replace`.
+- **The Copilot provider now recovers automatically when its nested runtime
+  process dies** (#483), instead of retrying against a dead process with
+  a misleading "Check that copilot CLI is installed and authenticated"
+  error. A dead spawned runtime is now detected via
+  `subprocess.Popen.poll()` on the SDK's own child handle and via explicit
+  recognition of `BrokenPipeError` / `ConnectionResetError` at the
+  agent-execution SDK boundary (including during idle-recovery "continue"
+  prompts, which previously burned every recovery attempt and were
+  reported as a stuck *agent* rather than a dead *process*). Recovery
+  rebuilds the SDK client the next time it is needed, so the existing
+  retry loop lands its next attempt on a fresh runtime with no change to
+  retry-loop shape; a runtime that keeps dying without a single
+  successful call in between fails fast after 2 consecutive restarts
+  (a fixed, non-configurable cap — with the default `max_attempts` of 3,
+  a single agent execution can only trigger 2 restarts on its own, so the
+  cap mainly bites across agents in the same workflow) rather than
+  looping forever, while a long-running, otherwise-healthy workflow can
+  restart it as many times as needed. A broken connection to an
+  **externally-owned** runtime (`runtime_url` /
+  `COPILOT_PROVIDER_RUNTIME_URL`) is treated differently: it is never
+  retried or respawned, since the orchestrator that owns that runtime is
+  responsible for its health checks and restarts.
 
 ## [0.1.33](https://github.com/microsoft/conductor/compare/v0.1.32...v0.1.33) - 2026-08-18
 

docs/configuration.md

@@ -339,7 +339,18 @@ conductor run review.yaml                 # connects; spawns no nested runtime
 - Closing the provider does **not** terminate the external runtime — the
   SDK only shuts down runtimes it spawned itself, so the orchestrator-owned
   server keeps running. The orchestrator is also responsible for runtime
-  health checks and restarts.
+  health checks and restarts: a **lost connection** to an external runtime
+  (a `BrokenPipeError` or `ConnectionResetError` at the SDK boundary) fails
+  the affected agent immediately (`is_retryable=false`) and is never
+  retried or respawned by Conductor. This differs from the default spawned
+  runtime, which Conductor restarts automatically after a detected crash
+  (a dead child process, or a `BrokenPipeError`/`ConnectionResetError` at
+  the SDK boundary) and retries against, up to a fixed cap of 2 consecutive
+  restarts without an intervening successful call (not configurable via
+  YAML or an environment variable). Note this covers a *lost* connection
+  only; a *failed initial connect* to an external runtime (e.g. it is not
+  reachable at all) is not classified by this mechanism and surfaces as a
+  generic SDK error instead.
 - Runtime-spawn-only options (custom CLI path, injected env, etc.) do not
   apply when connecting to an existing runtime.