From 002b06e6ebade0e1163dedfe0baf049e17d99722 Mon Sep 17 00:00:00 2001 From: Pavel Fadeev Date: Fri, 17 Jul 2026 02:11:29 +0200 Subject: [PATCH 1/6] docs: triage Qwen worker budget failures --- README.md | 1 + docs/qwen-worker-failure-triage.md | 57 ++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+) create mode 100644 docs/qwen-worker-failure-triage.md diff --git a/README.md b/README.md index 18b16a9..1b74903 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,7 @@ Workspace-only lab helpers live under `packages/lab/`; they are not part of the - [Advisor Board agent/skill taxonomy](docs/advisor-board-agent-skill-taxonomy.md) — boundary between advisory agents, executable skills, and role Markdown. - [Advisor Board replay PoC](docs/advisor-board-poc.md) — deterministic ledger/replay gate before live board behavior. - [Orchestration README](packages/orchestration/README.md) — goal, loop, autoresearch, and lab behavior. +- [Qwen worker failure triage](docs/qwen-worker-failure-triage.md) — distinguish runner budgets from local-model health and use bounded worker chunks. - [Release guide](docs/release.md) — canonical `pi-rogue-` release process. ## Local development diff --git a/docs/qwen-worker-failure-triage.md b/docs/qwen-worker-failure-triage.md new file mode 100644 index 0000000..3bf1ce7 --- /dev/null +++ b/docs/qwen-worker-failure-triage.md @@ -0,0 +1,57 @@ +# Qwen execution-worker failure triage + +This note documents how to distinguish a local Qwen inference failure from a Pi subagent-runner budget outcome. It is an operational guardrail; Pi-Rogue does not own the llama.cpp server lifecycle or the installed `pi-subagents` runner. + +## Diagnosis checklist + +1. Confirm the configured worker target is the expected model: + + ```bash + curl -fsS http://127.0.0.1:8004/v1/models + ``` + + The response must include `qwen3.6-35b-a3b-ud-q4-k-m` and the expected context metadata. + +2. Confirm the endpoint is responsive: + + ```bash + curl -fsS http://127.0.0.1:8004/health + ``` + +3. Run a minimal generation smoke test before retrying a worker task: + + ```bash + curl -fsS --max-time 60 http://127.0.0.1:8004/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"qwen3.6-35b-a3b-ud-q4-k-m","messages":[{"role":"user","content":"Reply with exactly OK."}],"max_tokens":4,"temperature":0}' + ``` + +4. Inspect the child-run result and runner message. A message such as: + + ```text + Tool budget soft limit reached after 30 tool calls (soft 30, hard 45) + ``` + + is a **tool-call budget** outcome from `pi-subagents`, not a turns quota and not proof that Qwen or llama.cpp failed. The runner has separate tool-budget and turn-budget controls. + +5. Treat these as separate diagnoses: + + - `tool budget ...`: bound the task, checkpoint findings, and continue in a new child run. + - `turn budget ...`: the child exceeded its configured assistant-turn budget. + - endpoint/model error: investigate the local server and model identity. + - timeout/cancellation: investigate the runner deadline or explicit cancellation. + +## Safe operating pattern + +- Keep the frontier model as controller and reviewer. +- Use explicit, run-scoped worker approval; AC power alone is not consent. +- Split repository exploration into bounded chunks of roughly 20 tool calls or fewer. +- Ask each child to return a checkpoint/artifact before the soft budget and to stop browsing when its budget warning appears. +- Do not silently fall back to a paid/frontier worker when the approved local worker fails. +- Preserve the child result, budget counts, and artifact path when opening or updating a ticket. + +## Current incident + +The July 17, 2026 investigation reached the `pi-subagents` tool budget at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome; that runtime fix is tracked in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356). + +This workaround does not change model settings, context size, budgets, fallback policy, or server lifecycle. From 13c076e3278127040a098b7dba5fbf82810dae22 Mon Sep 17 00:00:00 2001 From: Pavel Fadeev Date: Fri, 17 Jul 2026 02:19:24 +0200 Subject: [PATCH 2/6] docs: clarify upstream worker runtime ownership --- docs/qwen-worker-failure-triage.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/qwen-worker-failure-triage.md b/docs/qwen-worker-failure-triage.md index 3bf1ce7..1b8844a 100644 --- a/docs/qwen-worker-failure-triage.md +++ b/docs/qwen-worker-failure-triage.md @@ -1,6 +1,6 @@ # Qwen execution-worker failure triage -This note documents how to distinguish a local Qwen inference failure from a Pi subagent-runner budget outcome. It is an operational guardrail; Pi-Rogue does not own the llama.cpp server lifecycle or the installed `pi-subagents` runner. +This note documents how to distinguish a local Qwen inference failure from a Pi subagent-runner budget outcome. It is an operational guardrail; Pi-Rogue does not own the llama.cpp server lifecycle or the installed `pi-subagents` runner. The actual machine-readable runner outcome fix belongs upstream; Pi-Rogue tracks the integration and operating contract here. ## Diagnosis checklist @@ -52,6 +52,6 @@ This note documents how to distinguish a local Qwen inference failure from a Pi ## Current incident -The July 17, 2026 investigation reached the `pi-subagents` tool budget at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome; that runtime fix is tracked in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356). +The July 17, 2026 investigation reached the `pi-subagents` tool budget at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. This workaround does not change model settings, context size, budgets, fallback policy, or server lifecycle. From 7c91af4a1533b3b8c139277820d3d10d9e38ad0c Mon Sep 17 00:00:00 2001 From: Pavel Fadeev Date: Fri, 17 Jul 2026 02:24:01 +0200 Subject: [PATCH 3/6] docs: add exact Qwen child-run evidence --- docs/qwen-worker-failure-triage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/qwen-worker-failure-triage.md b/docs/qwen-worker-failure-triage.md index 1b8844a..90e2804 100644 --- a/docs/qwen-worker-failure-triage.md +++ b/docs/qwen-worker-failure-triage.md @@ -52,6 +52,6 @@ This note documents how to distinguish a local Qwen inference failure from a Pi ## Current incident -The July 17, 2026 investigation reached the `pi-subagents` tool budget at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. +The July 17, 2026 investigation’s Qwen child ran from `2026-07-16T23:55:57.725Z` to `2026-07-17T00:04:27.544Z` UTC, used the expected `llamacpp-qwen-unsloth` model, completed 20 assistant turns and 44 `bash` tool calls, and reached the `pi-subagents` tool-budget warning at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. This workaround does not change model settings, context size, budgets, fallback policy, or server lifecycle. From 57eed67f14021fa947f5b493cd54c3e156247116 Mon Sep 17 00:00:00 2001 From: Pavel Fadeev Date: Fri, 17 Jul 2026 02:29:03 +0200 Subject: [PATCH 4/6] docs: keep raw session counts in issue evidence --- docs/qwen-worker-failure-triage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/qwen-worker-failure-triage.md b/docs/qwen-worker-failure-triage.md index 90e2804..cde687a 100644 --- a/docs/qwen-worker-failure-triage.md +++ b/docs/qwen-worker-failure-triage.md @@ -52,6 +52,6 @@ This note documents how to distinguish a local Qwen inference failure from a Pi ## Current incident -The July 17, 2026 investigation’s Qwen child ran from `2026-07-16T23:55:57.725Z` to `2026-07-17T00:04:27.544Z` UTC, used the expected `llamacpp-qwen-unsloth` model, completed 20 assistant turns and 44 `bash` tool calls, and reached the `pi-subagents` tool-budget warning at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. +The July 17, 2026 investigation recorded the expected `llamacpp-qwen-unsloth` model and the `pi-subagents` tool-budget warning at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The detailed session counts and timestamps are preserved in issue #371 rather than treated as a repository fixture. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. This workaround does not change model settings, context size, budgets, fallback policy, or server lifecycle. From b5245c734fa39a4f7daef9d684b04790cf2fda30 Mon Sep 17 00:00:00 2001 From: Pavel Fadeev Date: Fri, 17 Jul 2026 02:32:06 +0200 Subject: [PATCH 5/6] docs: preserve Qwen worker evidence manifest --- .../qwen-worker-tool-budget-2026-07-17.json | 26 +++++++++++++++++++ docs/qwen-worker-failure-triage.md | 2 +- 2 files changed, 27 insertions(+), 1 deletion(-) create mode 100644 docs/evidence/qwen-worker-tool-budget-2026-07-17.json diff --git a/docs/evidence/qwen-worker-tool-budget-2026-07-17.json b/docs/evidence/qwen-worker-tool-budget-2026-07-17.json new file mode 100644 index 0000000..e091726 --- /dev/null +++ b/docs/evidence/qwen-worker-tool-budget-2026-07-17.json @@ -0,0 +1,26 @@ +{ + "schema": "pi-rogue.qwen-worker-evidence.v1", + "incidentDate": "2026-07-17", + "source": { + "kind": "local-pi-session-jsonl", + "relativePath": "~/.pi/agent/sessions/--Users-pavel-repos-fiale-plus-pi-rogue--/2026-07-16T19-45-07-578Z_019f6c76-1bfa-7824-b10f-903741487599/c07f273d/run-0/session.jsonl", + "sha256": "d0f47a30582ed5260697ace8f4de842c1243239c3bf6988ba5f6cbe9fecadd81" + }, + "observed": { + "start": "2026-07-16T23:55:57.725Z", + "end": "2026-07-17T00:04:27.544Z", + "provider": "llamacpp-qwen-unsloth", + "model": "qwen3.6-35b-a3b-ud-q4-k-m", + "assistantTurns": 20, + "toolCalls": 44, + "toolNames": { "bash": 44 }, + "runnerWarning": "Tool budget soft limit reached after 30 tool calls (soft 30, hard 45)", + "requestedArtifactProduced": false + }, + "conclusion": { + "turnQuotaImplicated": false, + "primaryCause": "pi-subagents tool-call budget boundary", + "currentModelHealthImplicated": false, + "notes": "The local endpoint separately passed health, exact-model, and OK-generation smoke checks." + } +} diff --git a/docs/qwen-worker-failure-triage.md b/docs/qwen-worker-failure-triage.md index cde687a..0fd5722 100644 --- a/docs/qwen-worker-failure-triage.md +++ b/docs/qwen-worker-failure-triage.md @@ -52,6 +52,6 @@ This note documents how to distinguish a local Qwen inference failure from a Pi ## Current incident -The July 17, 2026 investigation recorded the expected `llamacpp-qwen-unsloth` model and the `pi-subagents` tool-budget warning at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The detailed session counts and timestamps are preserved in issue #371 rather than treated as a repository fixture. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. +The July 17, 2026 investigation recorded the expected `llamacpp-qwen-unsloth` model and the `pi-subagents` tool-budget warning at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The detailed session counts, timestamps, and source hash are preserved in the [evidence manifest](evidence/qwen-worker-tool-budget-2026-07-17.json) and issue #371. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. This workaround does not change model settings, context size, budgets, fallback policy, or server lifecycle. From 1fd2046b1bca0781f58ff3d7c046603b1e389c3e Mon Sep 17 00:00:00 2001 From: Pavel Fadeev Date: Fri, 17 Jul 2026 02:37:23 +0200 Subject: [PATCH 6/6] docs: record Qwen runner timeout as terminal outcome --- .../evidence/qwen-worker-tool-budget-2026-07-17.json | 12 +++++++++++- docs/qwen-worker-failure-triage.md | 4 ++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/evidence/qwen-worker-tool-budget-2026-07-17.json b/docs/evidence/qwen-worker-tool-budget-2026-07-17.json index e091726..6192785 100644 --- a/docs/evidence/qwen-worker-tool-budget-2026-07-17.json +++ b/docs/evidence/qwen-worker-tool-budget-2026-07-17.json @@ -6,6 +6,15 @@ "relativePath": "~/.pi/agent/sessions/--Users-pavel-repos-fiale-plus-pi-rogue--/2026-07-16T19-45-07-578Z_019f6c76-1bfa-7824-b10f-903741487599/c07f273d/run-0/session.jsonl", "sha256": "d0f47a30582ed5260697ace8f4de842c1243239c3bf6988ba5f6cbe9fecadd81" }, + "runnerMetadata": { + "kind": "local-pi-subagents-run-metadata", + "relativePath": ".pi-subagents/artifacts/c07f273d_local-worker-poc_0_meta.json", + "sha256": "0eb302a4cbc82b3b47914f3d9e0b3c9567ea10d8d98c289085c3b518431f595e", + "exitCode": 1, + "error": "Subagent timed out after 600000ms", + "durationMs": 600086, + "toolBudgetOutcome": "soft-reached (30/45 warning; hard limit not reached)" + }, "observed": { "start": "2026-07-16T23:55:57.725Z", "end": "2026-07-17T00:04:27.544Z", @@ -15,11 +24,12 @@ "toolCalls": 44, "toolNames": { "bash": 44 }, "runnerWarning": "Tool budget soft limit reached after 30 tool calls (soft 30, hard 45)", + "runnerTerminalOutcome": "Subagent timed out after 600000ms", "requestedArtifactProduced": false }, "conclusion": { "turnQuotaImplicated": false, - "primaryCause": "pi-subagents tool-call budget boundary", + "primaryCause": "pi-subagents runner timeout after soft tool-budget pressure", "currentModelHealthImplicated": false, "notes": "The local endpoint separately passed health, exact-model, and OK-generation smoke checks." } diff --git a/docs/qwen-worker-failure-triage.md b/docs/qwen-worker-failure-triage.md index 0fd5722..1e5c8ce 100644 --- a/docs/qwen-worker-failure-triage.md +++ b/docs/qwen-worker-failure-triage.md @@ -32,7 +32,7 @@ This note documents how to distinguish a local Qwen inference failure from a Pi Tool budget soft limit reached after 30 tool calls (soft 30, hard 45) ``` - is a **tool-call budget** outcome from `pi-subagents`, not a turns quota and not proof that Qwen or llama.cpp failed. The runner has separate tool-budget and turn-budget controls. + is a **soft tool-call budget warning** from `pi-subagents`, not a turns quota and not proof that Qwen or llama.cpp failed. The runner has separate tool-budget and turn-budget controls; the observed child ultimately terminated at the runner’s 600-second timeout before reaching the hard tool limit. 5. Treat these as separate diagnoses: @@ -52,6 +52,6 @@ This note documents how to distinguish a local Qwen inference failure from a Pi ## Current incident -The July 17, 2026 investigation recorded the expected `llamacpp-qwen-unsloth` model and the `pi-subagents` tool-budget warning at 30 calls (hard limit 45). The local server independently passed health, exact-model, and `OK` smoke checks. The detailed session counts, timestamps, and source hash are preserved in the [evidence manifest](evidence/qwen-worker-tool-budget-2026-07-17.json) and issue #371. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. +The July 17, 2026 investigation recorded the expected `llamacpp-qwen-unsloth` model, a `pi-subagents` soft tool-budget warning at 30 calls (hard limit 45), and a terminal runner timeout after 600 seconds. The local server independently passed health, exact-model, and `OK` smoke checks. The detailed session counts, timestamps, and source hash are preserved in the [evidence manifest](evidence/qwen-worker-tool-budget-2026-07-17.json) and issue #371. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. This workaround does not change model settings, context size, budgets, fallback policy, or server lifecycle.