This repository was archived by the owner on Jul 3, 2026. It is now read-only.
feat: incremental analyzer graph with versioned lineage and Pi-AI provider - #4
Merged
Conversation
…vider Implements the analyzer framework on a clean foundation: - Append-only, typed-edge analysis graph (no parent_id; edges are the single source of truth). New 'revises' edge kind links version alternatives of the same logical unit. - 100% incremental via scan(): every planned unit is classified as missing|stale|current by input_hash + source-set lookup. shallow runs process missing units only; deep runs re-analyse stale units into new versions linked by revises edges, so alternatives coexist at the same level with timestamps and navigable lineage. - No crash-recovery bookkeeping: idempotency is structural (finished nodes are 'current'; unfinished units remain missing/stale). - LLM wired to Pi's AI provider system (modelRegistry.find + getApiKeyAndHeaders + @earendil-works/pi-ai complete()); never Ollama or a local server. Deterministic mock caller for tests. - Three analyzers: turn-pair-core (deterministic friction), turn-pair-llm (cheap LLM enrichment of high-signal pairs), session-overview (digest + map-reduce, emits proposals). Dependency-visibility enforcement and edge validation in the framework. - Clean v2 schema (proposals + framework tables), proposal materialiser with SHA-256 dedup, env-overridable config. Tests: 130 unit/component + 19 integration, all green; ~98% line coverage. TypeBox for all new data shapes.
This was referenced Jun 11, 2026
Remove docs/analyzer-design-c.md (the pre-implementation design draft, now partly obsolete: it specified progress cursors and crash recovery, which the implementation deliberately replaced with scan-based incrementality) and add a root-level DESIGN.md. DESIGN.md is an orientation guide for engineers and AI agents new to the repo. It describes the system's purpose and the rationale for the current architecture, and fixes a ubiquitous language for the core concepts (session, pair, analysis node, edge kinds, recipe/input hash, scan, unit status, run modes, lineage, model tiers, proposal lifecycle). It contains no code pointers by design.
…ections to match implementation
A tier (cheap/mid/expensive) is shorthand that resolves to a concrete provider/model; fold that resolved model into input_hash via a new modelBundleHash. Changing which model a tier maps to now marks affected nodes stale (deep run revises into a new version; shallow leaves them, so swaps never force surprise recomputation). Deterministic analyzers declare no model and are unaffected.
buildTurnPairs now starts a new turn at any host turn-start entry: user and bashExecution messages, plus branch_summary/custom_message entries. Compaction summaries remain boundaries, not turn starts.
--model now pins every tier to one concrete model for the invocation (applyModelOverride), instead of being recorded but ignored. The same effective tiers feed both the framework and the LLM caller, and analyzers resolve their tier from ctx.modelTiers before calling the model, so the model used and the model folded into node identity are one source of truth and cannot diverge. A pinned run produces its own nodes; rerunning under the normal mapping marks them stale. Drop the vestigial defaultModel seam.
Rework the ubiquitous language so a run's reach is a composable set of revise reasons rather than a shallow/deep depth, and split staleness by why a node is out of date: - Analyzer version is a major.minor pair the author owns; only the version is graded (major/minor). The shipped logic, default prompt, and default tier are all represented by the version. - Everything the user sets (thresholds, prompt override, tier->model mapping, model pin, and so the resolved model) is config, and config is never graded -- a different prompt or model is just different. - Identity is analyzer + version + config + source set. - A run always fills missing work; --revise widens its reach with major/minor/config in any combination. Reasons only select which stale units to touch; a selected unit is always recomputed to the current recipe in full (no intermediate state, no half-updated node).
Replace the shallow/deep run mode with a composable --revise reason set
and grade staleness by cause, per DESIGN.md.
- AnalyzerVersion is now { major, minor } (was an opaque versionId
string); versionIdOf/parseVersionId give the canonical "M.m" form
stored on runs and nodes.
- Identity drops the standalone shipped-prompt and model-bundle hashes
and folds the user's config + resolved model into one config
fingerprint: input_hash = analyzer + version + config_fingerprint +
source set. The shipped prompt is represented by the version; the
prompt bundle hash is kept only as run provenance.
- scan() grades each stale unit's reasons: a version move is major or
minor (gradeVersionMove), a differing config fingerprint is config.
run({ revise }) selects missing + stale units whose reasons intersect
the requested set (minor implies major); selected units recompute to
the current recipe.
- /prospect-analyze takes --revise major|minor|config|all (replacing
--deep); any reason re-scans all sessions, a plain fill only the
unanalysed ones.
- analysis_nodes gains config_fingerprint; run mode is recorded as a
reach label (fill or revise:...).
- New version.ts (version + reason helpers) with unit tests; new
revise-reasons component test proves major/minor/config selection;
model-identity test reframed (a model swap is the config reason).
The README still described the pre-framework design: a stub analyzer that ran an LLM over "unprocessed sessions", space-form command names (/prospect analyze), the wrong database name (sessions.db), proposal statuses new/accepted/rejected, a non-existent analyze tool action, and an install path pointing at the wrong owner. Rewrite it to match the implementation and DESIGN.md's language: - Dash-form commands (/prospect-sync, /prospect-analyze, ...) with the real flags, including --revise major|minor|config|all and the --model per-run pin. - The append-only, incrementally-recomputed analysis graph and its deterministic-first layering (turn-pair-core -> turn-pair-llm -> session-overview); turn as the unit of analysis. - Correct DB name (prospector.db), proposal statuses (open/applied/rejected/duplicate), and the prospect tool's actual actions (no analyze action). - Configuration documents the fields that actually take effect (dbPath, modelTiers) plus the test-only env overrides; install uses git:github.com/v2nic/pi-prospector. - Points at DESIGN.md as the canonical model.
There was a problem hiding this comment.
Pull request overview
This PR replaces the previous analyzer stub with a full incremental “analysis graph” framework: an append-only, content-addressed node/edge model with versioned lineage and selective recomputation, wired to Pi’s model registry/provider system for LLM calls. It introduces three analyzers (deterministic turn metrics → LLM enrichment → session overview proposals), proposal materialization/dedup, new v2 schema + queries, and updated slash commands/tooling plus extensive tests and refreshed docs.
Changes:
- Add v2 SQLite schema + analysis-graph queries (nodes/edges/runs/configs/lineage) to support append-only incremental recomputation.
- Implement the analyzer framework and bundled analyzers (turn-pair-core, turn-pair-llm, session-overview) with Pi-provider LLM integration + deterministic mock for tests.
- Update commands/config/docs/tests to operate end-to-end (sync → analyze → proposals lifecycle) with high coverage.
Reviewed changes
Copilot reviewed 63 out of 63 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/version.test.ts | Unit tests for analyzer version identity + revise-reason parsing/labeling. |
| tests/unit/turn-pair-build.test.ts | Unit tests for turn boundary construction into turn pairs. |
| tests/unit/prompt-parsing.test.ts | Unit tests for LLM prompt/response parsing helpers (JSON extraction + schemas). |
| tests/unit/pi-llm.test.ts | Unit tests for Pi LLM caller adapter and response flattening. |
| tests/unit/patterns.test.ts | Unit tests for deterministic correction/repetition detection heuristics. |
| tests/unit/model-tiers.test.ts | Unit tests for tier resolution, provider/model parsing, and run-time model pinning. |
| tests/unit/mock-llm.test.ts | Unit tests for deterministic mock LLM and throwing LLM guard. |
| tests/unit/input-hash.test.ts | Unit tests for hashing/canonical JSON/config fingerprint/uuidv7 helper behavior. |
| tests/unit/edge-kinds.test.ts | Unit tests for edge/ref vocabulary and target validation. |
| tests/unit/digest.test.ts | Unit tests for session digest building and segmentation. |
| tests/unit/config.test.ts | Unit tests for config loading + env overrides + model tiers fallback. |
| tests/component/turn-pair-core.test.ts | Component tests for deterministic turn-pair-core scoring behavior. |
| tests/component/sync.test.ts | Component sync tests refactored to shared temp DB helper; removes proposal tests moved to v2 query tests. |
| tests/component/schema.test.ts | Component tests verifying v2 schema tables/columns and constraints. |
| tests/component/revise-reasons.test.ts | Component tests ensuring revise reasons select stale units correctly. |
| tests/component/queries.test.ts | Component tests for v2 proposal queries + stats including analysis graph. |
| tests/component/proposal-materializer.test.ts | Component tests for proposal extraction, dedup, and produces-edge wiring. |
| tests/component/model-identity.test.ts | Component tests proving resolved model is part of node identity + --model override is used/recorded. |
| tests/component/helpers.ts | Shared temp DB + fixture helpers for component tests. |
| tests/component/framework.test.ts | Component tests for scan/fill idempotency, lineage via revises edges, dependency visibility, topo sort. |
| tests/component/commands.test.ts | Component tests for slash commands and prospect tool behavior under test context. |
| tests/component/analyzers-e2e.test.ts | End-to-end analyzer pipeline tests with mock LLM and map-reduce path exercise. |
| tests/component/analysis-queries.test.ts | Component tests for run lifecycle, config resolution, edges, anchored-message resolution. |
| test/integration/test-commands.ts | Integration test updated to run real pipeline with mock LLM and assert graph/proposals/lineage/lifecycle. |
| src/types.ts | Updates config/proposal/stats types to v2 proposal lifecycle and analysis graph stats. |
| src/pi-stubs.ts | Expands host-package stubs for model registry + pi-ai complete() surface used at runtime. |
| src/db/schema.ts | Introduces clean v2 schema (sessions/messages/FTS, proposals v2, analyzer registry, configs, runs, nodes, edges). |
| src/db/queries.ts | Updates session/proposal/stats queries for v2 proposals and analysis stats integration. |
| src/db/analysis-queries.ts | Adds analysis graph data-access layer (registry, configs, runs, nodes, edges, lineage, stats). |
| src/config.ts | Adds env-overridable config paths and model-tier resolution. |
| src/commands/tool.ts | Updates prospect tool to v2 proposal statuses/fields and consistent ToolResult shape. |
| src/commands/sync.ts | Types command context against stubs and uses env-overridable config. |
| src/commands/stats.ts | Updates stats output to include v2 proposal statuses + analysis graph counts/breakdown. |
| src/commands/proposals.ts | Updates proposal listing/output and accept/reject semantics for v2 statuses/fields. |
| src/commands/analyze.ts | Implements incremental analyzer framework execution with revise reasons, analyzer/session/model selection, and Pi LLM seam. |
| src/analyze/version.ts | Adds major.minor version identity, grading, revise-reason parsing/expansion, and reach labels. |
| src/analyze/types.ts | Defines framework/analyzer/node/edge/run/LLM contracts via TypeBox for data shapes. |
| src/analyze/proposal-materializer.ts | Extracts improvement proposals from nodes into proposals table with dedup and produces edges. |
| src/analyze/model-tiers.ts | Implements tier→provider/model resolution and per-run model pinning. |
| src/analyze/mock-llm.ts | Deterministic mock and throwing LLM implementations for tests. |
| src/analyze/input-hash.ts | Implements canonical JSON + hashing for source sets/config identity/input recipe and uuidv7 IDs. |
| src/analyze/edge-kinds.ts | Defines typed edge/ref vocabulary and validates allowed edge→target combinations. |
| src/analyze/defaults.ts | Registers the default bundled analyzers in dependency order. |
| src/analyze/analyzers/turn-pair-llm/prompt.ts | Defines classify prompt, JSON extraction, and tolerant response parsing. |
| src/analyze/analyzers/turn-pair-llm/index.ts | Adds LLM enrichment analyzer over high-signal turn pairs with consumes/anchors/prompt edges. |
| src/analyze/analyzers/turn-pair-llm/config.ts | Defines LLM enrichment config schema (tier/temperature/maxPairsPerSession). |
| src/analyze/analyzers/turn-pair-core/patterns.ts | Adds deterministic correction/repetition detection patterns. |
| src/analyze/analyzers/turn-pair-core/index.ts | Adds deterministic per-turn metrics analyzer producing friction scores and anchors edges. |
| src/analyze/analyzers/turn-pair-core/config.ts | Defines deterministic scoring weights/thresholds as config schema. |
| src/analyze/analyzers/turn-pair-core/build.ts | Builds turn pairs from message stream using host platform turn boundaries. |
| src/analyze/analyzers/session-overview/prompt-reduce.ts | Defines reduce-phase prompt and parsing for session summary + proposals. |
| src/analyze/analyzers/session-overview/prompt-map.ts | Defines map-phase prompt and parsing for segment summaries. |
| src/analyze/analyzers/session-overview/index.ts | Adds session overview analyzer with digest + optional map-reduce and consumes/anchors/prompt edges. |
| src/analyze/analyzers/session-overview/digest.ts | Builds structured digest text from core metrics, classifications, and compaction summaries. |
| src/analyze/analyzers/session-overview/config.ts | Defines overview analyzer config schema (tiers/thresholds/segment limits). |
| src/analyze/pi-llm.ts | Implements production LLM caller via Pi model registry + optional pi-ai dynamic import. |
| src/analyze/prompt.ts | Removes prior monolithic v1 prompt/tool-call schema (superseded by v2 analyzers). |
| src/analyze/parser.ts | Removes prior v1 response parser (superseded by v2 analyzers/prompt parsing). |
| README.md | Updates docs to describe the append-only analysis graph, commands, proposal lifecycle, and configuration. |
| AGENTS.md | Updates contributor guidance to align with the v2 architecture and test setup. |
Comments suppressed due to low confidence (1)
src/db/schema.ts:194
migrate()always drops and recreatesmessages_fts, but never rebuilds/backfills it from existingmessagesrows. If the DB already has messages (e.g. after a prior/prospect-sync), this leaves the FTS index empty until messages are re-inserted.
If you keep the DROP/CREATE approach, add an FTS5 rebuild so messages_fts is consistent immediately after migration.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ings Three correctness fixes surfaced by code review, each making the implementation match its intended design: - session-overview dropped all turn-pair-llm enrichment. The digest merged classifications via anchorUserId(), which always returned null (the promised "merge by order" fallback was never written), so sentiment/friction never reached the digest or the proposals it feeds. turn-pair-llm now records the anchor user_message_id in its node content (set from the planned unit, not the model), and the digest merges by that id. Split the pure model result (ClassifyResult) from the stored shape (TurnPairLLMProperties). - turn-pair-llm ignored its maxPairsPerSession cost guard: plan() emitted a unit per high-signal pair, so a large session could fire unbounded LLM calls. plan() now receives the resolved config (new AnalyzerPlanContext.config) and enriches at most maxPairsPerSession pairs, highest friction first (ties broken by pair order for deterministic, idempotent selection). - role spellings disagreed with the host platform. Pi emits the entry types branch_summary and custom_message (snake_case); turn-pair-core's turn boundaries already matched, but the MessageRole type, the session-overview digest, and a parser test used camelCase branchSummary, so branch summaries were silently omitted from digests. Standardised on Pi's snake_case across the type, the digest, and the test fixture. Tests: digest merges enrichment by id and includes branch_summary text; an e2e run with >cap high-signal pairs produces exactly maxPairsPerSession classifications. 161 unit/component + 19 integration green; coverage ~98.5%.
…ling Running the analyzers against real sessions through the Pi CLI surfaced two rough edges in the production LLM path: - A reasoning model (e.g. gemini-2.5-flash) spends its output-token budget on thinking and truncates the JSON answer. The truncated body then failed deep in the parser with a cryptic "Unterminated JSON object". toLLMResponse now detects stopReason "length" and throws an actionable error naming the output limit and pointing at maxTokens / a non-reasoning tier. - A throttled provider (HTTP 429) failed a whole run on the first call. The caller now passes maxRetries so pi-ai rides out transient rate limits. Also document running a single command non-interactively straight from the Pi CLI (pi -e src/index.ts -ne --no-session -p "/prospect-..."), including pointing the env overrides at a small private session subset for local iteration. Tests: pi-llm asserts the truncation error path. 162 unit/component + 19 integration green.
Running prospector from the command line previously required Pi's generic print mode (pi -e src/index.ts -p "/prospect-..."). The extension now registers its own --prospect flag so a bare invocation runs one command and exits, with no -p: pi -e ./src/index.ts --prospect sync pi -e ./src/index.ts --prospect stats pi -e ./src/index.ts --prospect "analyze --limit 3 --model openrouter/anthropic/claude-3.5-haiku" pi -e ./src/index.ts --prospect proposals A session_start hook reads the flag, dispatches to the matching command, and calls ctx.shutdown() (guarded to run once). When --prospect is absent the extension stays fully interactive. The flag value is "<command> [args]"; commands are sync | analyze | stats | proposals | accept | reject. To wire this up, each command's handler is extracted into an exported function (prospectSync/Stats/Proposals/Accept/Reject/Analyze) reused by both the slash command and the flag dispatcher, and pi-stubs gains registerFlag/getFlag/on plus ctx.shutdown. Tests: headless dispatcher unit tests (spec splitting, dispatch, empty/unknown, error propagation); commands.test mock updated for the new API. 169 unit/component + 19 integration green.
An error node's identity is now recipe + message + timestamp (plus the node id as a uniqueness nonce) instead of the recipe alone, so a failed attempt never occupies the successful result's identity. classify() sees only successful nodes at a recipe (findLatestNodeBySourceSet skips errors), so a unit whose only history is failures stays `missing` and is recomputed on the next scan. A session is retired from the unanalysed queue only when it completes with no errors, so a plain `fill` self-heals prior failures with no special retry mode. Error nodes stay append-only (never deleted). Verified on real local sessions: a bogus-model run records 9 error nodes and leaves all sessions queued; a subsequent plain fill recomputes exactly the 9 missing LLM nodes, retains the 9 error nodes, and a third fill scans nothing. 170 unit/component + 19 integration green; framework.ts and analysis-queries.ts at 100% line coverage.
The proposals listing (slash command and `--prospect proposals`) now ranks recommendations by confidence (desc, nulls last, newest-first tiebreak) and shows the confidence percentage inline. A new `--full`/`-v` flag adds the proposal's detail, evidence, and source provenance (session · analyzer · proposal id) so analyzer output can be evaluated without dropping to SQL. Status filtering (open|applied|rejected|duplicate) is unchanged and composes with the flag in any order. Adds unit tests for parseProposalsArgs and rankProposals (176 unit/component + 19 integration green).
The proposals listing now groups recommendations under a per-session header (short id · cwd · count), keeping confidence ranking both across sessions (the session with the strongest single recommendation leads) and within each group. Adds getSessionLabels query and a sessionLabel helper (cwd with $HOME collapsed to ~, falling back to project, then short id). Duplicate/overlapping recommendations are intentionally retained — grouping makes them easy for a downstream AI consumer to read per session. 178 unit/component + 19 integration green.
Make analysis identities global, reproducible, and hash-verifiable instead of DB-local. Two changes to the identity model: - Rename the recipe identity `input_hash` -> `input_key` (analyzer + version + config + source set; inputs only, never the LLM output). The config dimension now hashes the config's *content* hash, not its DB-local uuid row id, so input_key no longer depends on incidental ids. - Add `output_key = H(input_key | canonical(content))`, the content-addressed id of a node's result. Consumers (turn-pair-llm, session-overview) reference their upstream sources by output_key instead of node uuid, so a consumer's input_key transitively commits to upstream outputs. The graph is now a Merkle DAG: identical inputs+outputs reproduce identical keys on any machine, after any wipe. Proposals: rename `dedup_key` -> `input_key`, derived from the source node's output_key + ordinal (never the model's title/path/severity). Re-materialising a node is idempotent; distinct sources keep their proposals (duplicates retained, grouped per session at display). Add `prospect verify`: recompute every node's output_key from stored content and confirm it matches (tamper/corruption detection). Tests: cross-DB reproducibility (identical input_key/output_key over the same fixture in two independent DBs), verify clean+tamper+unparseable, output_key unit tests, and all renamed-field updates. 183 unit/component + 20 integration green; input-hash.ts/proposal-materializer.ts 100%, verify.ts ~99%, ~98% overall. Validated on real sessions: 155 nodes, 16 proposals, 0 errors, verify clean. Implements #5.
Reviewing a proposal previously required re-opening the raw session transcript to check the LLM's free-text evidence string. `prospect show` walks the provenance graph instead — proposal → source summary node --consumes--> the turn nodes --anchors--> their messages — and prints the verbatim anchored turns: the user text, the assistant text, and crucially each tool call WITH its arguments, plus any tool-result errors. Because the overview consumes every turn, output is focused to the high-signal turns (high friction or an LLM classification) and capped, with a remainder note. This makes human review trustable without a transcript hunt, and surfaces mechanism-level reality the text-only classifier misses: e.g. for the gh-pr-review session it shows the user shouting "PUSHED TO v2nic/..." next to the agent's actual `git push origin …` (correct) and `gh pr create --repo v2nic/…`/`--head` calls — exposing that the friction was PR-creation targeting, not the push. Also: proposal listings now print the full proposal id with a ready `prospect show <id>` hint (uuidv7 ids share long prefixes, so short prefixes are ambiguous); `show` resolves by exact id or unambiguous prefix. Adds getSessionMessageRows; registers prospect-show slash command + headless `show` action. Tests: unit (toolCallPreview, renderAnchoredTurns incl. args/ errors/cap) + component (full pipeline → resolve → provenance walk → output). 191 unit/component + 20 integration green; show.ts ~98% lines.
README: add `/prospect-show <id>` (provenance walk to verbatim anchored turns incl. tool-call arguments) and `/prospect-verify` (the latter was undocumented since it shipped); note the proposals listing now prints the full id + a `prospect show <id>` hint and accepts `--full`; add show/verify to the headless command list. DESIGN: remove the stale "Dedup key" glossary bullet — proposal identity is the content-addressed input_key (source output_key + ordinal) and duplicates are intentionally retained, so the old dedup-by-title concept no longer exists; add the `prospect show` evidence-walk to the Materialisation concept; fix one "recommendations" → "proposals" term slip. README intro: "deduplicated" → "ranked" proposals, matching the design (overlaps are retained and ranked, not collapsed).
Same-run proposal ids are uuidv7 values that share a long timestamp prefix, so a short prefix rarely resolves to one proposal. The old ambiguous-match warning printed each match as id.slice(0, 8), yielding N identical, unusable strings. Print one line per match with the shortest distinct id prefix plus the title, so the user can copy a longer id straight back into prospect show. (cherry picked from commit 1ea8689b832b80f582c5aab53986fcd581156d84)
…o classifier Issue #8: Two coupled changes to improve recall and mechanism-grounding. Change 1 — Un-gate the digest: - buildDigest() now includes a truncated verbatim user-text snippet (text=) for EVERY pair, not just those where the correction regex matched. - The regex remains a ranking/enrichment signal (note=) but no longer gates what the synthesizer can see. - Per-line truncation at 200 chars keeps the digest within mapReduceOverChars/segmentChars budgets. Change 2 — Feed tool evidence to the classifier: - PairToolCall extended with argumentsPreview (bash command or key=value summary for other tools, truncated at 300 chars). - PairToolResult extended with errorHead (first 300 chars of error text from toolResult content_text, null for non-errors). - ClassifyInput/buildClassifyPrompt extended with ToolCallEvidence[] and ToolResultEvidence[] — includes a TOOL CALLS section showing tool names, truncated arguments, and FAILED markers with error text. - CLASSIFY_PROMPT updated with guidance to prefer friction_type tool_misuse when tool evidence reveals the mechanism. Both analyzers bumped minor version: turn-pair-llm 1.0→1.1, session-overview 1.0→1.1. DESIGN.md updated with note that tool arguments and error payloads are first-class evidence, and the correction regex is a ranking signal only, never a visibility gate. Tests added: - Digest: unmatched correction still contributes text; plain pairs get text snippets; long text is truncated. - Prompt: classify prompt contains failing command for tool-error turn; shows tool calls, errors, and arguments. - Build: tool call arguments, error heads, key=value summaries, null errorHead for non-errors, long command truncation. (cherry picked from commit 6a1b020)
…ation, pre-flight gaps) Implements GitHub issue #7: a new session-anchored, deterministic (no LLM) analyzer that detects friction in the tool-call trajectory of a session. New detectors: - stuck-loop: same tool + near-identical normalized args invoked N≥3 times without an intervening success/state change - polling-loop: read-only command repeated while waiting for external state - oscillation: action followed by its inverse on the same target within a sliding window (checkout x→y→x, push→force-push, create→delete) - pre-flight gap: mutating command that fails on a missing precondition New files: - src/analyze/analyzers/tool-trajectory/{index,config,arg-parser,detectors}.ts - tests/unit/tool-trajectory.test.ts (30 tests per detector) - tests/component/tool-trajectory.test.ts (2 component tests) Changes: - DESIGN.md: define ubiquitous-language terms (trajectory signal, stuck-loop, polling-loop, oscillation, pre-flight gap); update node-kind definition - AGENTS.md: register tool-trajectory in code organization - session-overview digest: consume trajectory nodes, render signal lines - session-overview analyzer: declare tool-trajectory as dependency - defaults.ts: register tool-trajectory analyzer in dependency order - tests/unit/digest.test.ts: add trajectoryNodes to buildDigest calls All 223 unit+component tests pass. Integration tests pass. tsc clean. (cherry picked from commit b34b8d8)
…nt proposals Implement GitHub issue #9 — analyze successful sessions and use success/failure contrast (ExpeL pattern). Changes: - DESIGN.md: define reinforcement proposal, positive signal, success/failure contrast; note clean sessions are first-class; add change-checklist question for clean-session output - digest.ts: compute three positive signals (task-completed-without-correction, correction-then-clean-recovery, low-tool-failure-density); include positive_signals in header and new section in text - prompt-reduce.ts: always produce session_summary (never empty); add key_positive_signals; add reinforcement as a severity; update REDUCE_PROMPT with contrast instructions - prompt-map.ts: update MAP_PROMPT to capture positive patterns - index.ts: pass positive signals through stats and reduce prompt; bump minor version 1.0→1.1 - types.ts: add reinforcement to ProposalSeverity - proposals.ts: display reinforce label for reinforcement severity - Unit tests: positive signals in digest, parseReduceResponse with key_positive_signals, reinforcement severity - Component tests: clean-recovery yields reinforcement proposal; clean session yields non-empty overview with positive signals (cherry picked from commit 6c062d4)
…nrichment cap (cherry picked from commit eaa324f9ca380052f533fb01b3ca61c08ce24a1e)
Add a `proposal-validate` analyzer that replays each open proposal against its originating high-signal turns with a distinct validator model — once as-is, once with the candidate rule injected as a standing instruction — crediting the proposal only where the rule turns friction into no-friction. The grounded validated_score / validation_status is written back onto the proposal (symmetric to proposal materialisation), so the proposals view ranks supported → unvalidated → unsupported instead of trusting the model's self-rated confidence. - schema: `validation` node kind; proposals gain source_message_ids, validated_score, validation_status, validation_node_id (clean v2) - session-overview stays 1.1 on analyzer-graph: attach source_message_ids (high-signal turn ids, friction desc) to each proposal as its replay set - commands: `prospect validate` (+ `--prospect validate`); proposals view labels replay-validated vs model-rated and shows the with/without delta - validation nodes are content-addressed and covered by `prospect verify` - DESIGN.md/README: validation node kind, replay-validated confidence, advisory-only caveat - tests: unit (scoreReplay, tiered ranking, prompts) + component e2e proving a misattributed high-confidence proposal is marked unsupported and ranked below a replay-supported one Refs #6 (cherry picked from commit 7b418c9acc40e8d64c2b8f894fde32da3432a900)
…tion Update each analyzer's in-code description to reflect post-graft behavior (tool-action trace, tool evidence in classifier, tool-trajectory consumption, enumerate-then-propose, replay validation). Add a plain-language ## Analyzers README section with per-analyzer explanations and source-file links, refresh the how-it-works diagram to include tool-trajectory, and fix the stale --analyzer ID list.
…easoning models) Real-data validation with ollama/glm-5.1:cloud showed every session-overview (reduce) and turn-pair-llm (classify) call failing "No JSON object found": the reasoning model returned prose/markdown despite "Return ONLY a JSON object", producing zero proposals. Fix: offer a single forced-output tool to the model and instruct it to call the tool, then read the parsed tool-call arguments as the structured result. pi-ai already supports tools (TypeBox-schema parameters on the Context); the ollama provider's native path serialises them and parses tool calls. - pi-stubs: PiTool + PiContext.tools - types: LLMRequest.tool (name/description/TypeBox params), LLMResponse.structured - pi-llm: offer the tool on the context; extract toolCall arguments into `structured`; only fail on stopReason "length" when nothing parsed - classify/map/reduce: add CLASSIFY_TOOL / MAP_TOOL / REDUCE_TOOL, split parsers into object-level parseClassifyObject/parseMapObject/parseReduceObject, prefer response.structured with a text-JSON fallback for prose-answering providers - proposal-validate replay also uses the classify tool - prompts now instruct calling the tool instead of "return only JSON" - version bumps (shipped prompt changed → node identity): turn-pair-llm 1.1→1.2, session-overview 1.1→1.2, proposal-validate 1.0→1.1 Validated on 10 private sessions: 169 nodes, 27 proposals, 0 errors, prospect verify clean (was: 0 proposals, all LLM nodes errored).
scan() snapshots every unit before any insert, so an analyzer that plans two units resolving to the same source_set_hash (byte-identical turn-pairs, or identical map/reduce source sets) classifies both as missing and tries to persist two nodes with the same input_key. The first insert wins and the duplicate throws 'UNIQUE constraint failed: analysis_nodes.input_key', leaving a spurious error node and burning a redundant LLM call. Two units with the same source_set_hash are the same logical unit by the framework's own contract, so one node is correct. Dedup the todo list by input_key within a run. No node identity, content, or recipe changes (no version bump); the cross-run case already converged via findNodeByInputKey. Surfaced on the 1376-session corpus run (13 sessions, 39 error nodes).
The dedup guard matched 'WHERE input_key = ? AND status = 'open''. Once a human accepted/rejected a proposal (status leaves 'open'), a later analysis run no longer matched it and re-inserted a fresh 'open' duplicate, silently resurrecting decided proposals. A proposal's input_key is content-addressed and reproducible, so it must materialise exactly once; dedup on input_key alone and preserve the existing row + its human decision. Regression test: accept/reject a proposal, re-materialise the same source, assert one row with the decision preserved.
A human accept/reject is external input (like messages), not derived data: not an analysis_node, not part of a proposal's identity. Record it in a new append-only proposal_decisions table keyed by the proposal's content-addressed input_key (not the row id), so the decision re-attaches to the regenerated proposal after a wipe+recompute -- durable memory. acceptProposal/rejectProposal now take an optional decision payload (disposition planned|done|done_differently, rationale, actual_change, harness_ref) and write the status flip + decision row in one transaction; done_differently maps to the accepted_modified verdict. id-only calls remain backward-compatible. Adds getLatestDecision/getDecisionsForProposal/ getAllDecisions (the corpus for the future meta-analyzer).
The prospect tool's accept/reject actions gain optional rationale, disposition (planned|done|done_differently) and actual_change params so the agent can record the human's reasoning. The slash commands parse '<id> [--planned|--done|--done-differently] [rationale...]'; id-only calls stay backward-compatible. Decisions are written to proposal_decisions in the same transaction as the status flip.
prospect-proposals and prospect-show now render the latest decision for each proposal (verdict, disposition, rationale, actual_change), looked up by the content-addressed input_key. The decision is the durable record of how the human responded; the status badge stays as the lifecycle projection.
DESIGN.md: define Decision and Disposition in the ubiquitous language, add a principle section (decisions are external input keyed by input_key, append-only, never analysis nodes, never part of proposal identity, durable across recompute, the future meta-analyzer's source), note the materialise-once-per-input_key guarantee, and add the corresponding invariant. README: document the rationale/disposition options on accept/reject (commands, prospect tool, headless) and that decisions surface in proposals/show.
Corpus analysis is dominated by sequential network-bound LLM calls. Add a
bounded async pool (mapWithConcurrency) and a counting semaphore
(createSemaphore) and use them in the analyze command to:
- cap concurrent LLM calls at a hard ceiling via a global semaphore wrapped
around the LLM caller (default 10), so the limit holds regardless of
dispatch; and
- fan out the per-session loop (default 10 for LLM-bearing runs, 20 for
deterministic-only runs, since those have no provider to protect).
Both are hard-coded defaults overridable with --llm-concurrency and
--analyzer-concurrency. Concurrency is execution-only: node and proposal
identity is unaffected (no version bump). A component test proves a concurrent
run over a shared connection yields byte-identical node/proposal identities to
a sequential run and stays idempotent; unit tests cover the pool and semaphore
(ordering, limit enforcement, error propagation, slot release).
v2nic
marked this pull request as ready for review
June 16, 2026 21:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A Pi extension that mines your past Pi coding sessions for friction and emits ranked proposals to improve your
AGENTS.md, skills, and prompts. Pi is the only entry point — slash commands and a headless--prospect <cmd>flag. No standalone CLI.Analysis is an append-only, content-addressed graph over each session: deterministic metrics and LLM judgements become
analysis_nodeslinked by typed edges. Identity is computed from inputs only (input_key), results are content-addressed (output_key = H(input_key | canonicalJson(content))), and consumers reference an upstream node'soutput_key— a Merkle DAG thatprospect verifyre-derives and checks. Incrementalscanclassifies every unit as current/stale/missing; there is no crash-recovery bookkeeping (idempotency is structural). Clean v2 schema, no migrations.Pipeline
1.0, deterministic) — turn-pair friction metrics; now also extracts a per-turn tool-action trace (call names + truncated args + failed-result error heads) for downstream evidence.1.2, cheap LLM) — classifies high-signal pairs. The classifier receives the actual tool calls and error heads, so friction is attributed to the command at fault rather than paraphrased from user wording. Length-aware enrichment cap (minPairFraction+maxPairsHardCeiling) replaces the flat cap.1.0, deterministic) — orthogonal session-level analyzer detecting stuck-loops, polling-loops, oscillation, and pre-flight gaps from the tool-call stream.1.2, LLM map-reduce) — digest is un-gated (every pair carries verbatim user text; the correction regex is a ranking signal only). Emits a node even for clean sessions, computes deterministic positive signals, and synthesises via enumerate-then-propose (enumerate all friction as textual gradients, then one proposal per point). Adds areinforcementproposal severity for things done right.1.1, opt-in viaprospect validate) — offline replay-validation: re-runs each open proposal's originating turns with a distinct validator model, once as-is and once with the rule injected, scores averted friction, and writes a groundedvalidated_score/validation_statusback onto the proposal (mutable result columns — never part of any identity key). Proposals rank supported → unvalidated → unsupported and are labelledreplay-validatedvsmodel-rated.Commands
/prospect-sync,/prospect-analyze,/prospect-proposals(ranked;--full),/prospect-show <id>(provenance/evidence drill-down),/prospect-validate,/prospect-verify(content-addressed integrity),/prospect-stats, plus theprospecttool and headless--prospect <cmd>.Human decisions (durable memory)
Accept/reject now records an append-only decision in a
proposal_decisionstable keyed by the proposal's content-addressedinput_key— not its row id — so a decision re-attaches to the regenerated proposal after a wipe-and-recompute. A decision carries a verdict (accepted / rejected / accepted_modified), a disposition (planned= will do it ·done= did the recommended action ·done_differently= the idea triggered a different action), a free-text rationale, and an optional actual_change. Decisions are external input, never analysis nodes and never part of a proposal's identity; the latest decision for aninput_keyis authoritative and surfaces in/prospect-proposalsand/prospect-show. Theprospecttool'saccept/rejectand the slash commands (/prospect-accept <id> [--planned|--done|--done-differently] [rationale]) capture them; id-only calls stay backward-compatible. This decision corpus is the intended gold-label input for a future quality-improving meta-analyzer.A companion fix makes proposals materialise exactly once per
input_keyregardless of status, so a re-run can no longer resurrect a decided proposal as a freshopenrow.Concurrency
The analyze command runs sessions through a bounded worker pool and caps concurrent LLM calls with a global semaphore wrapped around the LLM caller — hard-coded defaults (10 concurrent LLM calls; 20-way session fan-out for deterministic-only runs) overridable with
--llm-concurrency/--analyzer-concurrency. Concurrency is execution-only: a component test proves a concurrent run over a shared connection yields byte-identical node/proposal identities to a sequential run and stays idempotent (no version bump). On the full 1376-session corpus this lifts throughput from ~2–6 to ~15 sessions/min. A within-run dedup byinput_keyalso removes spuriousUNIQUE constrainterror nodes when an analyzer plans two units with the same source set.Reliable structured output
Every LLM call (classify, map, reduce, replay) requests its result via a forced tool call with a TypeBox-schema parameter, not by asking the model to "return only JSON". Reasoning models (e.g. glm-5.1) routinely answer structured prompts in prose; offering a single tool and instructing the model to call it makes the output parseable across providers. The caller reads the parsed tool-call arguments (
LLMResponse.structured) with a text-JSON fallback for providers that still answer in prose. Validated on a 10-session real-data run: 0 LLM-parse errors (previously every session-overview/turn-pair-llm call failed "No JSON object found").Identity & reproducibility
input_keyis derived fromanalyzer + major.minor version + config fingerprint (config content hash + resolved models) + source-set hash. LLM output, scores, titles, and severities never feed an identity key. Deterministic node content keepsoutput_keyreproducible across machines and DB wipes;prospect verifyre-derives everyoutput_keyand reports mismatches.Testing
node:testonly — 304 unit/component pass, 0 fail (incl. the forced-tool-call structured-output path, the decision log, and concurrency-identity); integration (tmux + real Pi + mocked LLM) 20/20;tsc --noEmitclean. Fixtures are hand-written synthetic JSONL — no real session data anywhere. TypeBox for every data shape; all SQL confined todb/queries.ts/db/analysis-queries.ts.