diff --git a/README.md b/README.md index 18b16a9..1b74903 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,7 @@ Workspace-only lab helpers live under `packages/lab/`; they are not part of the - [Advisor Board agent/skill taxonomy](docs/advisor-board-agent-skill-taxonomy.md) — boundary between advisory agents, executable skills, and role Markdown. - [Advisor Board replay PoC](docs/advisor-board-poc.md) — deterministic ledger/replay gate before live board behavior. - [Orchestration README](packages/orchestration/README.md) — goal, loop, autoresearch, and lab behavior. +- [Qwen worker failure triage](docs/qwen-worker-failure-triage.md) — distinguish runner budgets from local-model health and use bounded worker chunks. - [Release guide](docs/release.md) — canonical `pi-rogue-` release process. ## Local development diff --git a/docs/evidence/qwen-worker-tool-budget-2026-07-17.json b/docs/evidence/qwen-worker-tool-budget-2026-07-17.json new file mode 100644 index 0000000..6192785 --- /dev/null +++ b/docs/evidence/qwen-worker-tool-budget-2026-07-17.json @@ -0,0 +1,36 @@ +{ + "schema": "pi-rogue.qwen-worker-evidence.v1", + "incidentDate": "2026-07-17", + "source": { + "kind": "local-pi-session-jsonl", + "relativePath": "~/.pi/agent/sessions/--Users-pavel-repos-fiale-plus-pi-rogue--/2026-07-16T19-45-07-578Z_019f6c76-1bfa-7824-b10f-903741487599/c07f273d/run-0/session.jsonl", + "sha256": "d0f47a30582ed5260697ace8f4de842c1243239c3bf6988ba5f6cbe9fecadd81" + }, + "runnerMetadata": { + "kind": "local-pi-subagents-run-metadata", + "relativePath": ".pi-subagents/artifacts/c07f273d_local-worker-poc_0_meta.json", + "sha256": "0eb302a4cbc82b3b47914f3d9e0b3c9567ea10d8d98c289085c3b518431f595e", + "exitCode": 1, + "error": "Subagent timed out after 600000ms", + "durationMs": 600086, + "toolBudgetOutcome": "soft-reached (30/45 warning; hard limit not reached)" + }, + "observed": { + "start": "2026-07-16T23:55:57.725Z", + "end": "2026-07-17T00:04:27.544Z", + "provider": "llamacpp-qwen-unsloth", + "model": "qwen3.6-35b-a3b-ud-q4-k-m", + "assistantTurns": 20, + "toolCalls": 44, + "toolNames": { "bash": 44 }, + "runnerWarning": "Tool budget soft limit reached after 30 tool calls (soft 30, hard 45)", + "runnerTerminalOutcome": "Subagent timed out after 600000ms", + "requestedArtifactProduced": false + }, + "conclusion": { + "turnQuotaImplicated": false, + "primaryCause": "pi-subagents runner timeout after soft tool-budget pressure", + "currentModelHealthImplicated": false, + "notes": "The local endpoint separately passed health, exact-model, and OK-generation smoke checks." + } +} diff --git a/docs/qwen-worker-failure-triage.md b/docs/qwen-worker-failure-triage.md new file mode 100644 index 0000000..1e5c8ce --- /dev/null +++ b/docs/qwen-worker-failure-triage.md @@ -0,0 +1,57 @@ +# Qwen execution-worker failure triage + +This note documents how to distinguish a local Qwen inference failure from a Pi subagent-runner budget outcome. It is an operational guardrail; Pi-Rogue does not own the llama.cpp server lifecycle or the installed `pi-subagents` runner. The actual machine-readable runner outcome fix belongs upstream; Pi-Rogue tracks the integration and operating contract here. + +## Diagnosis checklist + +1. Confirm the configured worker target is the expected model: + + ```bash + curl -fsS http://127.0.0.1:8004/v1/models + ``` + + The response must include `qwen3.6-35b-a3b-ud-q4-k-m` and the expected context metadata. + +2. Confirm the endpoint is responsive: + + ```bash + curl -fsS http://127.0.0.1:8004/health + ``` + +3. Run a minimal generation smoke test before retrying a worker task: + + ```bash + curl -fsS --max-time 60 http://127.0.0.1:8004/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"qwen3.6-35b-a3b-ud-q4-k-m","messages":[{"role":"user","content":"Reply with exactly OK."}],"max_tokens":4,"temperature":0}' + ``` + +4. Inspect the child-run result and runner message. A message such as: + + ```text + Tool budget soft limit reached after 30 tool calls (soft 30, hard 45) + ``` + + is a **soft tool-call budget warning** from `pi-subagents`, not a turns quota and not proof that Qwen or llama.cpp failed. The runner has separate tool-budget and turn-budget controls; the observed child ultimately terminated at the runner’s 600-second timeout before reaching the hard tool limit. + +5. Treat these as separate diagnoses: + + - `tool budget ...`: bound the task, checkpoint findings, and continue in a new child run. + - `turn budget ...`: the child exceeded its configured assistant-turn budget. + - endpoint/model error: investigate the local server and model identity. + - timeout/cancellation: investigate the runner deadline or explicit cancellation. + +## Safe operating pattern + +- Keep the frontier model as controller and reviewer. +- Use explicit, run-scoped worker approval; AC power alone is not consent. +- Split repository exploration into bounded chunks of roughly 20 tool calls or fewer. +- Ask each child to return a checkpoint/artifact before the soft budget and to stop browsing when its budget warning appears. +- Do not silently fall back to a paid/frontier worker when the approved local worker fails. +- Preserve the child result, budget counts, and artifact path when opening or updating a ticket. + +## Current incident + +The July 17, 2026 investigation recorded the expected `llamacpp-qwen-unsloth` model, a `pi-subagents` soft tool-budget warning at 30 calls (hard limit 45), and a terminal runner timeout after 600 seconds. The local server independently passed health, exact-model, and `OK` smoke checks. The detailed session counts, timestamps, and source hash are preserved in the [evidence manifest](evidence/qwen-worker-tool-budget-2026-07-17.json) and issue #371. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project. + +This workaround does not change model settings, context size, budgets, fallback policy, or server lifecycle.