fix(providers): suppress idle watchdog during in-flight Copilot tool calls (#490)

* fix(providers): suppress idle watchdog during in-flight Copilot tool calls

The Copilot SDK emits no events between tool.execution_start and
tool.execution_complete, so a stale idle clock during a long-running
tool call was indistinguishable from a genuinely stuck session,
triggering a spurious "please continue" recovery prompt mid tool-call
that could overwrite the agent's eventual structured output.

In-flight tool calls (tracked by tool_call_id) now suppress idle
recovery entirely while any remain outstanding; max_session_seconds /
max_agent_iterations remain the backstop for a genuinely wedged tool.
last_activity_ref's tool name is cleared (or rolled to another
still-in-flight tool) on tool.execution_complete instead of only ever
being set.

Adds configurable runtime.idle_timeout_seconds and
runtime.max_idle_recovery_attempts (Copilot-only) so workflows with
legitimately long tool calls can tune the watchdog.

Closes #488

* fix(providers): address PR #490 review findings on idle watchdog suppression

Blocking fixes (#488):
- Remove the "pop the oldest entry" fallback in the tool.execution_complete
  handler that could evict a different, still-running tool's active_tools
  entry on a duplicate/unmatched event, re-arming the watchdog mid-tool-call
  and reproducing #488 while appearing fixed. Replaced with a non-mutating
  debug log; max_session_seconds remains the backstop for a stale entry.
- Strengthen TestOnEventActiveTools assertions so both tests actually pin
  the fix (verified to fail against the pre-fix provider, pass post-fix).
- Add an end-to-end regression test driving a real tool.execution_start ->
  silence -> tool.execution_complete sequence through _send_and_wait,
  asserting the recovery prompt never clobbers response_content.
- Add a warn-once latch (mirroring _context_window_anomaly_warned) so the
  first occurrence of extended idle-recovery suppression during a session
  is logged at warning level (console + logger), instead of silently
  degrading a previously console-visible 90s warning into up to 31.5
  minutes of total silence.

Recommendations applied:
- Corrected the repeated false claim that the SDK "emits no events" during
  a tool call (it does not guarantee any, but tool.execution_progress /
  tool.execution_partial_result exist and are opt-in) across copilot.py,
  schema.py, docs/configuration.md, CHANGELOG.md, and the PR description;
  consolidated the rationale into one canonical docstring.
- Corrected the inaccurate claim that max_agent_iterations backstops a
  wedged tool call (its counter only advances on tool.execution_start, so
  it's frozen for the whole wedge) — max_session_seconds is the sole
  backstop.
- Added IdleRecoveryConfig.__post_init__ validation so directly-constructed
  configs (bypassing the Pydantic schema bounds) can't produce an unbounded
  busy-wait loop.
- Simplified factory.py's IdleRecoveryConfig construction to a dict-filter
  + single constructor call instead of a three-way ternary per field.
- Added ProviderCapabilities.idle_recovery (Copilot-only) with a
  workflow-level validator warning (not an error, since these are tuning
  knobs rather than safety bounds) when idle_timeout_seconds /
  max_idle_recovery_attempts are set against a provider that ignores them.
- Bounded two previously-unbounded busy-wait test loops with
  asyncio.wait_for(..., timeout=5.0).
- Added an overlapping-tool-calls end-to-end test keyed on tool_call_id
  (not tool_name), verified to reproduce the hang if the dict were
  mistakenly keyed by tool name instead.
- Documented the max_session_seconds backstop in docs/configuration.md so
  a legitimately long tool call doesn't silently exceed it unexpectedly.

Skipped: ACA forwarding of the two idle-recovery fields (larger, separate
scope spanning factory/aca/aca_runner) and AGENTS.md documentation update
(the two runtime knobs and active_tools mechanism are already documented
in the config docs and code comments; deferring to keep this diff scoped
to the review findings).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Jason Robert <jasonrobert@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: 3 people

CHANGELOG.md

@@ -22,6 +22,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   A custom `base_url` requires an explicit `api_key`: an ambient `OPENAI_API_KEY`
   is never forwarded to a non-OpenAI endpoint.
 
+- **`runtime.idle_timeout_seconds` / `runtime.max_idle_recovery_attempts`**
+  (#488) — Copilot-only knobs to tune the idle watchdog for workflows with
+  legitimately long tool calls. `idle_timeout_seconds` sets the time without
+  SDK events before a session is treated as idle (default 90s);
+  `max_idle_recovery_attempts` caps the number of "please continue" prompts
+  sent before failing (default 5; `0` fails on the first genuine idle
+  without ever injecting a prompt). See `docs/configuration.md` and
+  `docs/workflow-syntax.md`.
+
 ### Changed
 
 - The Pydantic AI dependency was narrowed from the full `pydantic-ai` package to
@@ -34,6 +43,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **The Copilot idle watchdog no longer fires during long-running tool
+  calls** (#488). The SDK does not guarantee any events during a tool
+  call — `tool.execution_progress` / `tool.execution_partial_result` exist
+  in the SDK schema but are opt-in per tool, so for most tool calls nothing
+  arrives between `tool.execution_start` and `tool.execution_complete` — so
+  a stale idle clock while a tool was still executing was previously
+  indistinguishable from a genuinely stuck session — triggering a spurious
+  "please continue" recovery prompt mid tool-call. That prompt's
+  conversational reply then overwrote the agent's eventual structured
+  output (`response_content` is last-message-wins), turning a healthy run
+  into a non-retryable failure. In-flight tool calls (tracked by
+  `tool_call_id`) now suppress idle recovery entirely while any remain
+  outstanding; `max_session_seconds` is the sole backstop for a genuinely
+  wedged tool. Recovery-prompt and
+  stuck-session messages also no longer misattribute the failure to a tool
+  that has already completed — `last_activity_ref`'s tool name is now
+  cleared (or rolled to another still-in-flight tool) on
+  `tool.execution_complete` instead of only ever being set. The first
+  occurrence of extended suppression during a session is logged at
+  `warning` level (naming the in-flight tools and the `max_session_seconds`
+  backstop); further occurrences in the same session are debug-only.
 - Retry classification now covers the `ModelHTTPError` and `ModelAPIError` types
   pydantic-ai actually raises, so `408`, `429` and `5xx` responses are retried on
   the Claude provider as well as the new OpenAI one. Previously they were treated

docs/configuration.md

@@ -17,6 +17,8 @@ workflow:
     max_tokens: 4096
     default_reasoning_effort: medium  # low | medium | high | xhigh | max (optional)
     default_context_tier: default  # default | long_context (optional, Copilot only)
+    idle_timeout_seconds: 90  # Copilot only (optional)
+    max_idle_recovery_attempts: 5  # Copilot only (optional)
     # Provider-specific settings...
 ```
 
@@ -31,6 +33,20 @@ context-window tier that every provider-backed agent inherits unless it
 declares its own `context_tier` override. See [Context Tier](#context-tier)
 for details. This is a Copilot-only capability.
 
+The `idle_timeout_seconds` and `max_idle_recovery_attempts` fields tune the
+Copilot provider's idle watchdog: when a session stops emitting SDK events
+for `idle_timeout_seconds` (default 90s), Conductor sends a "please continue"
+recovery prompt, up to `max_idle_recovery_attempts` times (default 5) before
+failing the session. A tool call that is still executing suppresses the
+watchdog — most tools emit nothing while running, so a stale idle clock
+during a long-running tool call usually means the tool is still running, not
+that the session is stuck. Suppression is bounded by `max_session_seconds`
+(default 1800s), which is still enforced while a tool is in flight and is the
+only limit that can end a genuinely hung tool call. Raise it alongside
+`idle_timeout_seconds` if your workflow has tool calls that legitimately run
+longer than 30 minutes. Both `idle_timeout_seconds` and
+`max_idle_recovery_attempts` are Copilot-only; other providers ignore them.
+
 ## Provider Selection
 
 ### Copilot Provider
@@ -47,6 +63,8 @@ workflow:
         command: npx
         args: ["-y", "open-websearch@latest"]
         tools: ["*"]
+    idle_timeout_seconds: 90
+    max_idle_recovery_attempts: 5
 ```
 
 **Features**:

docs/workflow-syntax.md

@@ -75,6 +75,14 @@ workflow:
                                       # its own `context_tier`.
                                       # See docs/configuration.md#context-tier.
 
+    idle_timeout_seconds: 90          # Optional: seconds without SDK events before a
+                                      # Copilot session is treated as idle (Copilot only).
+                                      # Default: 90. Suppressed entirely while a tool
+                                      # call is in flight.
+    max_idle_recovery_attempts: 5     # Optional: "please continue" prompts sent before
+                                      # failing an idle Copilot session (Copilot only).
+                                      # Default: 5. 0 means fail on first genuine idle.
+
     working_dir: "/path/to/cwd"       # Optional: global default working directory for LLM agents
                                       # and their MCP servers. Relative paths resolve against the
                                       # parent directory of the workflow YAML file.

plugins/conductor/skills/conductor/references/yaml-schema.md

@@ -35,6 +35,8 @@ workflow:
     timeout: float                  # Per-request timeout in seconds (optional, default: 600, copilot/claude only)
     max_agent_iterations: integer   # Max tool-use roundtrips per agent (1-500, optional)
     max_session_seconds: float      # Wall-clock timeout per agent session in seconds (optional)
+    idle_timeout_seconds: float     # Seconds without SDK events before session is idle (optional, Copilot only, default 90)
+    max_idle_recovery_attempts: integer # "please continue" prompts before failing idle session (optional, Copilot only, default 5)
     default_reasoning_effort: string # Workflow-wide reasoning/thinking effort: low, medium, high, xhigh, max (optional)
     skills: [string]                # Skills enabled for every provider-backed agent (default: [])
                                     # Each entry is a built-in NAME or a filesystem PATH.

src/conductor/config/schema.py

@@ -3419,6 +3419,28 @@ def _coerce_provider(cls, value: Any) -> Any:
     (Claude: 50, Copilot: unlimited).
     """
 
+    idle_timeout_seconds: float | None = Field(None, ge=1.0)
+    """Time without SDK events before a Copilot session is treated as idle.
+
+    Copilot provider only; other providers ignore this field. Default is
+    None, which uses the provider's built-in default (90s). A session is
+    only considered idle when no SDK events at all have arrived within the
+    window — an in-flight tool call (between ``tool.execution_start`` and
+    ``tool.execution_complete``) suppresses the check entirely, since most
+    tools emit nothing during execution (see
+    ``IdleRecoveryConfig.idle_timeout_seconds`` in ``providers/copilot.py``
+    for the full rationale).
+    """
+
+    max_idle_recovery_attempts: int | None = Field(None, ge=0)
+    """Maximum number of "please continue" prompts sent to an idle Copilot session.
+
+    Copilot provider only; other providers ignore this field. Default is
+    None, which uses the provider's built-in default (5). ``0`` means the
+    session fails on the first genuine idle timeout without ever injecting
+    a recovery prompt.
+    """
+
     default_reasoning_effort: ReasoningEffort | None = None
     """Workflow-wide default reasoning effort applied to provider-backed agents.
 

src/conductor/config/validator.py

@@ -1821,6 +1821,8 @@ def _validate_provider_capabilities(
     # no matter where future call sites are added.
     runtime_default_effort = config.workflow.runtime.default_reasoning_effort
     runtime_max_session_seconds = config.workflow.runtime.max_session_seconds
+    runtime_idle_timeout_seconds = config.workflow.runtime.idle_timeout_seconds
+    runtime_max_idle_recovery_attempts = config.workflow.runtime.max_idle_recovery_attempts
     runtime_working_dir = config.workflow.runtime.working_dir
     runtime_skills = config.workflow.runtime.skills
     skill_limits = config.workflow.runtime.skill_injection
@@ -2466,6 +2468,36 @@ def _check_agent_capabilities(
                     f"max_session_seconds."
                 )
 
+    # ----- Workflow-level: idle_recovery tuning knobs -----
+    # runtime.idle_timeout_seconds / runtime.max_idle_recovery_attempts are
+    # Copilot-only tuning knobs (#488). Unlike max_session_seconds, these are
+    # not safety bounds — a provider that ignores them just runs its own
+    # idle-detection defaults (or none at all) rather than violating an
+    # operational guarantee — so a mismatch is a warning, not an error.
+    if runtime_idle_timeout_seconds is not None or runtime_max_idle_recovery_attempts is not None:
+        providers_using_idle_recovery: dict[str, list[str]] = {}
+        for agent in all_llm_agents:
+            pname = _resolved_provider_name(agent, default_provider)
+            providers_using_idle_recovery.setdefault(pname, []).append(agent.name)
+        for pname, agent_names in providers_using_idle_recovery.items():
+            pcaps = _caps_for(pname)
+            if pcaps is not None and not pcaps.idle_recovery:
+                set_fields = [
+                    name
+                    for name, value in (
+                        ("idle_timeout_seconds", runtime_idle_timeout_seconds),
+                        ("max_idle_recovery_attempts", runtime_max_idle_recovery_attempts),
+                    )
+                    if value is not None
+                ]
+                warnings.append(
+                    f"Workflow declares 'runtime.{'/'.join(set_fields)}' but provider "
+                    f"'{pname}' does not support idle-recovery tuning "
+                    f"(capabilities.idle_recovery=False) and is used by agent(s): "
+                    f"{sorted(agent_names)!r}. The setting will be silently ignored "
+                    f"for these agents."
+                )
+
     # ----- Workflow-level: working_dir -----
     # A runtime-wide working_dir is inherited by every LLM agent that does
     # not set its own. A provider that cannot apply it would silently run

src/conductor/providers/capabilities.py

@@ -192,6 +192,15 @@ class ProviderCapabilities(BaseModel):
     session_continuity: bool = False
     """``True`` when the provider supports per-agent ``session_key``."""
 
+    idle_recovery: bool = False
+    """``True`` when the provider honors ``runtime.idle_timeout_seconds`` /
+    ``runtime.max_idle_recovery_attempts`` (#488). These are Copilot-only
+    tuning knobs for its SDK-event-driven idle watchdog; other providers
+    have no equivalent mechanism and silently ignore both fields. Unlike
+    ``max_session_seconds``, this is a tuning knob rather than a safety
+    bound, so a mismatch is a validate-time **warning**, not an error.
+    Defaults to ``False``."""
+
     max_temperature: float | None = None
     """Highest temperature the provider accepts.
 
@@ -274,6 +283,8 @@ def declared_limitations(self) -> list[str]:
             items.append("no skills support")
         if not self.session_continuity:
             items.append("no session_key continuity")
+        if not self.idle_recovery:
+            items.append("idle_timeout_seconds/max_idle_recovery_attempts ignored")
         return items