Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ Workspace-only lab helpers live under `packages/lab/`; they are not part of the
- [Advisor Board agent/skill taxonomy](docs/advisor-board-agent-skill-taxonomy.md) — boundary between advisory agents, executable skills, and role Markdown.
- [Advisor Board replay PoC](docs/advisor-board-poc.md) — deterministic ledger/replay gate before live board behavior.
- [Orchestration README](packages/orchestration/README.md) — goal, loop, autoresearch, and lab behavior.
- [Qwen worker failure triage](docs/qwen-worker-failure-triage.md) — distinguish runner budgets from local-model health and use bounded worker chunks.
- [Release guide](docs/release.md) — canonical `pi-rogue-<semver>` release process.

## Local development
Expand Down
36 changes: 36 additions & 0 deletions docs/evidence/qwen-worker-tool-budget-2026-07-17.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"schema": "pi-rogue.qwen-worker-evidence.v1",
"incidentDate": "2026-07-17",
"source": {
"kind": "local-pi-session-jsonl",
"relativePath": "~/.pi/agent/sessions/--Users-pavel-repos-fiale-plus-pi-rogue--/2026-07-16T19-45-07-578Z_019f6c76-1bfa-7824-b10f-903741487599/c07f273d/run-0/session.jsonl",
"sha256": "d0f47a30582ed5260697ace8f4de842c1243239c3bf6988ba5f6cbe9fecadd81"
},
"runnerMetadata": {
"kind": "local-pi-subagents-run-metadata",
"relativePath": ".pi-subagents/artifacts/c07f273d_local-worker-poc_0_meta.json",
"sha256": "0eb302a4cbc82b3b47914f3d9e0b3c9567ea10d8d98c289085c3b518431f595e",
"exitCode": 1,
"error": "Subagent timed out after 600000ms",
"durationMs": 600086,
"toolBudgetOutcome": "soft-reached (30/45 warning; hard limit not reached)"
},
"observed": {
"start": "2026-07-16T23:55:57.725Z",
"end": "2026-07-17T00:04:27.544Z",
"provider": "llamacpp-qwen-unsloth",
"model": "qwen3.6-35b-a3b-ud-q4-k-m",
"assistantTurns": 20,
"toolCalls": 44,
"toolNames": { "bash": 44 },
"runnerWarning": "Tool budget soft limit reached after 30 tool calls (soft 30, hard 45)",
"runnerTerminalOutcome": "Subagent timed out after 600000ms",
"requestedArtifactProduced": false
},
"conclusion": {
"turnQuotaImplicated": false,
"primaryCause": "pi-subagents runner timeout after soft tool-budget pressure",
"currentModelHealthImplicated": false,
"notes": "The local endpoint separately passed health, exact-model, and OK-generation smoke checks."
}
}
57 changes: 57 additions & 0 deletions docs/qwen-worker-failure-triage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Qwen execution-worker failure triage

This note documents how to distinguish a local Qwen inference failure from a Pi subagent-runner budget outcome. It is an operational guardrail; Pi-Rogue does not own the llama.cpp server lifecycle or the installed `pi-subagents` runner. The actual machine-readable runner outcome fix belongs upstream; Pi-Rogue tracks the integration and operating contract here.

## Diagnosis checklist

1. Confirm the configured worker target is the expected model:

```bash
curl -fsS http://127.0.0.1:8004/v1/models
```

The response must include `qwen3.6-35b-a3b-ud-q4-k-m` and the expected context metadata.

2. Confirm the endpoint is responsive:

```bash
curl -fsS http://127.0.0.1:8004/health
```

3. Run a minimal generation smoke test before retrying a worker task:

```bash
curl -fsS --max-time 60 http://127.0.0.1:8004/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"qwen3.6-35b-a3b-ud-q4-k-m","messages":[{"role":"user","content":"Reply with exactly OK."}],"max_tokens":4,"temperature":0}'
```

4. Inspect the child-run result and runner message. A message such as:

```text
Tool budget soft limit reached after 30 tool calls (soft 30, hard 45)
```

is a **soft tool-call budget warning** from `pi-subagents`, not a turns quota and not proof that Qwen or llama.cpp failed. The runner has separate tool-budget and turn-budget controls; the observed child ultimately terminated at the runner’s 600-second timeout before reaching the hard tool limit.

5. Treat these as separate diagnoses:

- `tool budget ...`: bound the task, checkpoint findings, and continue in a new child run.
- `turn budget ...`: the child exceeded its configured assistant-turn budget.
- endpoint/model error: investigate the local server and model identity.
- timeout/cancellation: investigate the runner deadline or explicit cancellation.

## Safe operating pattern

- Keep the frontier model as controller and reviewer.
- Use explicit, run-scoped worker approval; AC power alone is not consent.
- Split repository exploration into bounded chunks of roughly 20 tool calls or fewer.
- Ask each child to return a checkpoint/artifact before the soft budget and to stop browsing when its budget warning appears.
- Do not silently fall back to a paid/frontier worker when the approved local worker fails.
- Preserve the child result, budget counts, and artifact path when opening or updating a ticket.

## Current incident

The July 17, 2026 investigation recorded the expected `llamacpp-qwen-unsloth` model, a `pi-subagents` soft tool-budget warning at 30 calls (hard limit 45), and a terminal runner timeout after 600 seconds. The local server independently passed health, exact-model, and `OK` smoke checks. The detailed session counts, timestamps, and source hash are preserved in the [evidence manifest](evidence/qwen-worker-tool-budget-2026-07-17.json) and issue #371. The incident therefore does not support a turns-quota or current llama.cpp-health diagnosis. The runner should expose a stable, resumable budget outcome. Pi-Rogue tracks the integration and operating contract in [issue #371](https://github.com/fiale-plus/pi-rogue/issues/371), related to [issue #356](https://github.com/fiale-plus/pi-rogue/issues/356); the runtime implementation belongs in the `pi-subagents` project.

This workaround does not change model settings, context size, budgets, fallback policy, or server lifecycle.
Loading