diff --git a/AGENTS.md b/AGENTS.md index a5be8c5..817f239 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,35 @@ # pi-prospector +## Start here: read DESIGN.md first + +Before writing or reviewing any code in this repo, read **`DESIGN.md`** at the +repository root. It is the single source of truth for *what this system is* and +*why it is shaped the way it is* — and it will save you from the most expensive +mistakes you can make here. + +This is not an ordinary codebase. It is an **append-only analysis graph** with +idempotent, recipe-addressed nodes, typed-edge relationships, scan-based +incremental recomputation, and versioned lineage. Those are load-bearing +invariants, not stylistic preferences: a change that looks harmless in isolation +(mutating a node, hiding a relationship outside the edge table, leaving identity +out of the recipe, tracking progress in side state) can quietly break +traceability or idempotency across the whole system. DESIGN.md tells you which +invariants must hold and gives you a checklist for evaluating any change against +them. + +It also fixes a **ubiquitous language** — precise, one-meaning definitions for +every core concept (session, pair, analysis node, node/edge kinds, recipe and +input hash, scan, unit status, run modes, lineage, model tiers, proposal +lifecycle). Use exactly these words with exactly these meanings in code, commits, +comments, and discussion. When you and the code agree on vocabulary, you stop +guessing what `stale`, `revises`, or `consumes` mean and start reasoning +correctly the first time. If you reach for a concept the glossary doesn't name, +define it there before you build it. + +In short: a few minutes in DESIGN.md is the difference between contributing +*with* the architecture and accidentally fighting it. Read it, then come back +for the operational rules below. + ## Session data safety `~/.pi/agent/sessions/` is read-only. Never write, delete, or move session files. Before running sync for the first time, back up your sessions manually (e.g. `tar czf ~/prospector-backup/sessions-$(date +%Y%m%d).tgz ~/.pi/agent/sessions/`). pi-prospector does not create this backup for you. @@ -18,31 +48,31 @@ TypeBox for all data shapes. No bare `interface` or `type` declarations. Every s - Runner: `node:test` + `node:assert`. No test frameworks. - Fixtures in `tests/fixtures/`. Hand-written, deterministic, version-controlled. No real session data — synthetic only. -## Integration tests (tmux + real Pi) +## Integration tests -Run the latest pi-coding-agent inside tmux, send `/prospect` commands, capture text screenshots, upload as GH Actions artifacts. This exercises the real extension loaded into a real Pi session. +`test/integration/test-commands.ts` exercises the real pipeline end-to-end without a Pi runtime: it syncs fixtures, runs the analyzer framework with a deterministic mock LLM, and asserts the analysis graph, materialised proposals, idempotent re-runs, revise-mode version lineage, and the proposal lifecycle (accept/reject). Run it with `node --import tsx test/integration/test-commands.ts`, or via the wrapper `test/integration/run-integration.sh`. -- `test/integration/run-screenshots.js` — orchestrates tmux session, sends commands, captures screenshots -- Uses the latest pi from npm (auto-installed in CI) -- Each screenshot is a `.txt` file from `tmux capture-pane` -- Scenarios: extension loads, `/prospect sync`, `/prospect stats`, `/prospect proposals`, `/prospect analyze` (mocked LLM), accept/reject -- No real API keys — mock the LLM provider via a local HTTP server +- Real SQLite (temp file), hand-written synthetic fixtures, mock LLM caller. +- No real API keys, no network, no real Pi session — the mock LLM is injected via the framework's LLM seam, not an HTTP server. ## CI -GitHub Actions on every push. Node 22 (matches Pi's minimum) and 24 (current). Three jobs: +GitHub Actions on every push and pull request to `main`. Two jobs: -1. `test` — `npm test` (unit + component, mocked LLM) -2. `integration-test` — tmux screenshots with real Pi + mocked LLM -3. Screenshots uploaded as artifacts, retained 30 days +1. `test` — matrix on Node 22 (Pi's minimum) and 24 (current); runs `npm test` (unit + component, mock LLM). +2. `integration-test` — Node 22; runs `node --import tsx test/integration/test-commands.ts` (full pipeline, mock LLM). ## Code organization -- `src/sync/` — session scanning and parsing (no LLM) -- `src/db/` — all SQL lives in `db/queries.ts` only. Migrations in `db/schema.ts`. -- `src/analyze/` — LLM prompt in `analyze/prompt.ts` only. Response parsing in `analyze/parser.ts`. -- `src/commands/` — Pi slash commands and tool registration -- `src/types.ts` — shared TypeBox schemas +- `src/sync/` — session scanning and parsing (no LLM). +- `src/db/` — all SQL lives here, nowhere else. Conversation and proposal queries in `db/queries.ts`; analysis-graph queries (nodes, edges, runs, configs, lineage) in `db/analysis-queries.ts`. Schema and the single migration in `db/schema.ts`. +- `src/analyze/` — the analyzer framework. `framework.ts` (register / scan / run), `types.ts` (TypeBox schemas), `input-hash.ts` (recipe + idempotency hashing), `edge-kinds.ts` (typed-edge vocabulary and validation), `model-tiers.ts`, `proposal-materializer.ts`, `defaults.ts` (default analyzer registration). The LLM seam is `pi-llm.ts` (production, via Pi's provider system) and `mock-llm.ts` (deterministic test double). +- `src/analyze/analyzers//` — one directory per analyzer (`turn-pair-core`, `turn-pair-llm`, `session-overview`), each with `index.ts`, its prompt(s), and `config.ts`. +- `src/commands/` — Pi slash commands and the `prospect` tool; registered from `src/index.ts`. +- `src/config.ts` — config loading with env overrides (`PROSPECTOR_DB_PATH`, `PROSPECTOR_SESSIONS_DIR`, `PROSPECTOR_CONFIG`). +- `src/types.ts` — shared TypeBox schemas. + +See `DESIGN.md` for the concepts these modules implement and the ubiquitous language to use when naming them. ## Code style diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..84d1998 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,471 @@ +# pi-prospector — Design & Concepts + +This document is the orientation guide for anyone — human or AI agent — who is +new to this repository. It explains **what the system is for** and **why it is +built the way it is**, and it fixes a **ubiquitous language**: a small set of +terms that mean exactly one thing throughout the code, the database, the +commands, and any conversation about the project. When you read or write code +here, use these words with these meanings. When a term in this glossary appears +in a discussion, assume the precise definition below — not its everyday sense. + +This is a conceptual guide. It deliberately contains no file paths, function +names, table definitions, or code. Those change; the concepts and the reasons +for them are what must stay stable. + +--- + +## 1. Purpose + +People spend hours working with coding agents. Those conversations are a +goldmine of signal about where the agent's standing instructions, documentation, +tools, and skills fall short: every time the user has to correct the agent, +re-explain context, or watch it waste effort, that friction is evidence of a +fixable gap. + +pi-prospector mines that history. It reads the local record of past agent +**sessions**, looks for moments of **friction**, and turns recurring patterns +into concrete, reviewable **proposals** — suggested edits to the artifacts that +steer future agent behaviour (standing instruction files, skills, tool +descriptions, configuration, and similar). The human stays in control: the +system proposes, ranks, and explains; it never edits those artifacts on +its own. + +The guiding intent behind every design choice is **trust through traceability +and cheap recomputation**. A proposal is only worth acting on if you can see the +exact evidence that produced it, and the analysis is only sustainable if it can +be re-derived cheaply as the analysis logic improves — without paying again for +work that is already up to date, and without throwing away earlier conclusions +that someone may want to compare against. + +--- + +## 2. Ubiquitous language + +These are the load-bearing nouns and verbs of the system. They are listed in +roughly the order concepts build on one another. + +### Conversation domain (the read-only input) + +- **Session** — one recorded conversation between a user and a coding agent, + from start to end. Sessions are the raw material. They are treated as + **read-only**: the system observes them and never alters them. +- **Message** — a single entry within a session: something the user said, one of + the agent's replies, the agent's private reasoning, or the result of a tool the + agent ran. Messages carry metadata (timing, model, token usage, error flags). +- **Turn** — the natural unit of one round of work, and where most friction is + visible. A turn begins at a user message (the turn boundary) and spans + *everything* the agent does in response — every assistant reply, its private + reasoning, and every tool call and result — up to the next user message. A + turn is therefore usually **many messages, not two**: a single request + normally drives a loop of repeated assistant generations and tool calls before + the next user message arrives. (The host platform also treats a few non-user + entries — a bash execution, a branch or custom summary — as turn boundaries.) + The per-turn analyzers are named `turn-pair-*` and the codebase calls this + unit a "turn pair"; it is one turn, not a pair of turns. +- **Step** — one assistant generation within a turn: a single model response, + possibly carrying tool calls. A turn is a sequence of one or more steps plus + the tool results they trigger. +- **Compaction boundary** — a point where the conversation history was + summarised and truncated to fit the model's context. The system is aware of + these so it does not mistake a summary for an ordinary message. + +### Analysis domain (the append-only output) + +- **Analysis graph** — the entire body of derived analysis, layered on top of + the conversation. It is a graph, not a tree, and it is **append-only**: once + written, an analysis result is never edited or deleted. +- **Analysis node** (or just **node**) — one self-contained piece of derived + analysis: a set of metrics for a turn, a classification of a turn, a + session-level summary, or a recorded error. Every node states what it is + about, what it was built from, and exactly which recipe produced it. +- **Node kind** — the category of a node's content. The kinds in use are + **metric** (deterministic measurements of a turn), **classification** (a + language-model judgement about a turn), **summary** (a session-level synthesis + that carries proposals), and **error** (a record that an analysis attempt + failed). An error node's identity is its recipe plus the failure's message and + timestamp, so every failure is a distinct, append-only record that never + occupies the recipe identity reserved for a successful result. Failures stay + visible and auditable, yet never mark a unit as done: the unit stays *missing* + and is recomputed on the next scan that reaches it. +- **Edge** — a typed, directed relationship in the analysis graph. Edges are the + **single source of truth** for relationships: there are no parent links or + embedded references hidden inside nodes. Every connection between a node and + anything else is an explicit edge with a named kind. +- **Edge kind** — the meaning of an edge. The kinds are: + - **anchors** — “this node is *about* this part of the conversation” (a + session or a specific message). Anchoring is how a proposal can be traced + back to the exact words that justify it. + - **consumes** — “this node was *built from* that node.” It records the inputs + that fed an analysis. + - **uses_prompt** — “this node was produced using that prompt.” + - **uses_config** — “this node was produced under that configuration.” + - **produces** — “this node yielded that proposal.” + - **revises** — “this node is a newer-version alternative of that node, + covering the same subject.” This is the backbone of lineage (below). +- **Anchor** — the conversation entity a node is about, reached via an *anchors* + edge. A turn-level node anchors to its user message; a session-level node + anchors to the session. + +### Recipe, identity, and idempotency + +- **Analyzer** — a named, self-describing unit of analysis logic. An analyzer + declares what it is about, what other analyzers it depends on, the prompts and + default configuration it uses, how to enumerate the work it could do, and how + to perform one piece of that work. Analyzers are the only things that create + nodes. +- **Analyzer version** — an analyzer's declared release, a `major.minor` pair + owned by its author (third-party, out-of-tree analyzers declare it when they + register). The version represents everything the author *ships*: the analysis + logic, the default prompt, and the default model tier. Improving an analyzer + means bumping the version — **major** for a change the author judges + significant, **minor** for a small one. Prompt or default changes are folded + into that one number; shipped defaults have no separate identity axis. A new + version produces new nodes rather than overwriting old ones. +- **Config** — everything the *user* sets for an analyzer: thresholds and + parameters, a prompt override, the tier→model mapping, and any model pin. The + resolved model lives here. Config is content-addressed, so changing any of it + yields a distinct config identity — but config changes are **never graded** + major/minor; a different prompt or a different model "is just different." +- **Prompt** — a content-addressed piece of prompt text, recorded for provenance + (which prompt produced a node). A prompt the analyzer *ships* is represented by + its version, not by a separate identity axis; only a prompt the *user* overrides + contributes to identity, as part of config. +- **Source set** — the exact collection of inputs a single piece of analysis + draws on, reduced to a stable fingerprint. Two analyses over the same inputs + share a source-set fingerprint; adding or changing inputs changes it. A + consumer's source set references its upstream sources by their **output key** + (below), so the consumer's identity folds in *what those sources concluded*, + not merely that they exist. +- **Recipe** — the full description of *how a node came to be*: which analyzer, + which version, which config, and which source set. The recipe is condensed + into a single fingerprint called the **input key**. +- **Input key** — the content-addressed fingerprint of a recipe (analyzer + + version + config + source set). It is the system's notion of identity for + *whether work needs doing*: if a node with a given input key already exists, + the work it represents is already done. An input key folds in only *inputs* — + never the model's output — so the same recipe over the same sources always has + the same input key. +- **Output key** — the content-addressed fingerprint of a node's *result*: + `hash(input_key, content)`. It identifies a *specific output*. A downstream + analyzer references its sources by their output key, so the whole graph is a + Merkle DAG: identical inputs and outputs reproduce identical keys on any + machine, after any wipe, and a stored key can be re-derived from content to + **verify** the node. Because analysis is append-only, a different output is + always a different node and therefore a different output key. +- **Idempotency** — the property that running analysis again produces no + duplicate work and no changed results, because identity is the recipe. Re-running + is always safe and usually a no-op. +- **Verification** — recomputing every node's output key from its stored content + and confirming it matches. Because identities are content-addressed, any drift + reveals out-of-band tampering or corruption. (`prospect verify`.) + +### Running analysis + +- **Plan** — an analyzer's enumeration of the discrete pieces of work it *could* + do for a session (for example, “one piece of work per turn”). Planning does + not perform analysis; it only describes the candidate units and their source + sets. +- **Unit** — one planned piece of work: a source set plus where its result would + anchor. A unit is the thing that does or does not yet have a corresponding + node. +- **Scan** — the act of comparing every planned unit against the existing graph + and classifying it. Scanning is cheap (fingerprint lookups, no model calls) + and is how the system decides what, if anything, needs doing. This replaces + any notion of progress cursors or crash bookkeeping. +- **Unit status** — the result of classifying a unit during a scan: + - **missing** — no *successful* node exists for this unit's recipe. Either it + has never been attempted, or prior attempts only produced error nodes (which + carry a decoupled identity and never satisfy a recipe). Missing work is always + done, even by a frugal run. + - **stale** — a node exists for this subject but under a different recipe than + the current one. Staleness carries its *reasons*: **major** or **minor** (the + analyzer's version moved — graded by the author) and/or **config** (the + user's setup changed — ungraded). A unit can be stale for several reasons at + once. + - **current** — a node already exists for the current recipe; nothing to do. +- **Run** — one execution of an analyzer over a session. A run records its own + provenance (status, cost, tokens, how many nodes it produced, skipped, or + revised) so that execution history is itself auditable. +- **Revise reasons** — what a run is allowed to recompute, beyond always filling + *missing* work. A run with no reasons is frugal: it fills missing analysis and + touches nothing that already has a node. The reasons widen that reach to + recompute stale nodes too: **major** (the analyzer had a major version bump), + **minor** (major *and* minor bumps), and **config** (the user's setup changed). + Reasons are a *set*, not a ladder — `config` is orthogonal to the author's + major/minor grade, so you choose any combination (major-plus-config, or + everything). Recomputing records each result as a new version linked by a + *revises* edge. +- **Selection versus recompute** — the reasons decide *which* out-of-date units a + run touches; they never decide *what to recompute toward*. A selected unit is + always recomputed to the **current recipe in full** — latest version, latest + config, latest resolved model. The grade is a trigger, not a target: there is + no recomputing to an intermediate state, so a unit revised for one reason + absorbs every pending change for that same unit at once and you never get a + half-updated node. +- **Lineage** — the chain of versioned alternatives for one subject, connected by + *revises* edges. Because analysis is append-only, recomputing does not replace + the old conclusion; it adds a newer one beside it, and both remain navigable + “at the same level.” This is what lets you compare how analysis changed as the + logic improved. + +### Language-model access + +- **Model tier** — an abstract quality/cost band (**cheap**, **mid**, + **expensive**) rather than a concrete model name. An analyzer's *default* tier + is part of what its version ships; the **tier→model mapping** is the user's + config. Before a node's identity is computed the tier resolves to a concrete + model, and that resolved model is part of **config** — so changing which model + a tier maps to is an ungraded config change, picked up by a run that includes + the `config` reason. +- **Model pin (per run)** — a single run may pin *every* tier to one specific + model, overriding the configured mapping for that invocation. That is a config + change for the run: the pinned model is part of identity, so a pinned run + produces its own nodes, and the model used and the model recorded can never + disagree. Existing nodes become stale for the `config` reason rather than + being silently reused. +- **LLM caller** — the single seam through which any analyzer reaches a language + model. In normal operation it routes through the host agent platform's own + model provider system, so credentials and model availability are managed in one + place and never reimplemented here. In tests it is replaced by a deterministic + stand-in so the suite needs no network and no API key. + +### Proposals (the product) + +- **Proposal** — a single, concrete, reviewable suggestion to improve a steering + artifact: what to change, where, why, with what confidence, and backed by + evidence drawn from the conversation. Proposals are the system's output and the + only thing a human is asked to act on. +- **Target** — what a proposal would change (a category such as a standing + instruction file, a skill, a tool description, or configuration, plus an + optional location within it). +- **Severity** — the nature of the signal behind a proposal (for example + friction, correction, waste, suggestion, or insight). It describes *why the + proposal exists*, not how urgent it is. +- **Materialisation** — the step that lifts proposals out of a summary node into + the fast, reviewable proposal store, attaching the evidence trail via + *produces* and *anchors* edges. That trail is browsable after the fact: from a + proposal you can walk back through the node that produced it to the turns it + consumed and the messages they anchor, and read those turns verbatim + (`prospect show`). +- **Proposal status** — where a proposal sits in its lifecycle: **open** (awaiting + a decision), **applied** (accepted/acted upon), **rejected** (declined), or + **duplicate** (recognised as the same as an existing open proposal). + +--- + +## 3. Why the architecture is shaped this way + +Each decision below exists to serve the guiding intent — traceability and cheap +recomputation — and to avoid a specific failure mode. + +### Append-only analysis, never mutation + +Analysis results are written once and never changed. The reason is trust: if a +proposal could be silently rewritten, you could never be sure the evidence you +are looking at is the evidence that produced it. Append-only storage means every +conclusion is permanently tied to the exact inputs and recipe behind it, and +re-analysis adds rather than overwrites. + +### Relationships live only in typed edges + +There are no parent pointers and no relationship fields buried inside nodes. +Every link is an explicit edge with a named kind. The reason is that the +interesting questions are all about relationships — *what evidence backs this +proposal, what did this summary consume, which version revised which* — and a +single typed-edge fabric answers all of them uniformly. Hiding some relationships +inside nodes and others in a side table would make traversal inconsistent and +provenance unreliable. + +### Identity is the recipe (idempotency by input key) + +A node's identity is the fingerprint of everything that determines its content: +the analyzer, its **version**, its **config**, and its **inputs** — condensed +into the **input key**. This is what +makes re-running safe and cheap, and it separates two kinds of change. A change +the analyzer's *author* ships — new logic, a reworked default prompt, a different +default tier — is a **version** bump, and the author grades it major or minor. A +change the *user* makes — a threshold, a prompt override, the tier→model mapping, +a model pin — is **config**, and it is never graded; a different prompt or model +"is just different." The resolved model is part of config, so changing which +model a tier maps to makes the affected nodes stale for the `config` reason — +picked up only by a run that asks for it, so a model swap never forces surprise +recomputation. Deterministic analyzers use no model, so nothing about model +settings touches their identity. Leave version, config, and inputs all the same +and nothing is recomputed. + +Identities are **content-addressed end to end**. The input key folds in only +inputs (the config's *content* hash, not any database row id; and upstream +sources by their **output key**, not their incidental node id), and the output +key is `hash(input_key, content)`. So the same sessions analysed with the same +analyzers reproduce byte-identical keys on any machine and after any wipe, and +the graph forms a Merkle DAG whose integrity can be re-derived from content +alone (`prospect verify`). Crucially, this is what makes "output matters for +consumers" automatic: because a consumer references its sources by their output +key, a changed upstream output is a new output key, which changes the consumer's +input key and correctly marks it for recomputation — while a *re-run that +reproduces the same output* changes nothing. + +### Incrementality by scanning, not by cursors or crash recovery + +The system does not keep progress bookmarks and does not maintain crash-recovery +state. Instead, before doing anything it scans: it enumerates the work each +analyzer could do and classifies every candidate as missing, stale, or current +using cheap fingerprint lookups. Whatever is missing gets done; whatever is +current is skipped. The reason is robustness through simplicity: there is no +bookkeeping to get out of sync, nothing to repair after an interruption, and no +way for a stored cursor to disagree with reality. The graph *is* the source of +truth about what has been done, so an interrupted or *failed* run simply leaves +some units missing, and the next scan picks them up. A unit that fails records an +error node for visibility, but because that node carries a decoupled identity +(recipe + message + timestamp) it never claims the recipe's slot — so the unit +stays missing and **self-heals on the next plain fill**, with no special retry +mode. To make that automatic, a session is only retired from the unanalysed queue +once it completes with no errors; a session that had any failure stays queued, so +the next fill re-scans it, recomputes only the still-missing units, and leaves its +prior error nodes intact. This is a deliberate departure from earlier designs that +tracked per-session cursors and recovery status. + +### Versioned lineage and the reach of a run + +Because analysis logic will keep improving, the system treats a better analyzer +as a *new version* rather than an edit. By default a run is frugal: it fills only +genuinely missing work and never touches subjects that already have a node. When +you have improved an analyzer, or changed your own config, you widen the run's +reach with **revise reasons** — `major` and `minor` for the author's version +bumps, `config` for your own setup changes, in any combination. A run then +re-analyses the matching stale subjects and records each new result as a fresh +version linked back to its predecessor by a *revises* edge. Crucially, the +reasons only *select* what to touch; whatever is touched is recomputed to the +current recipe in full, so you never produce a half-updated node. The old and new +conclusions coexist as navigable alternatives, giving you both economy (don't +redo good work) and the ability to audit how conclusions evolved — without ever +losing the earlier ones. + +### Dependency-scoped visibility + +An analyzer can read the conversation and its own past output, plus the output of +the analyzers it explicitly declares as dependencies — and nothing else. +Attempting to read undeclared analysis is treated as an error, not quietly +allowed. The reason is to keep the analysis pipeline a clear, declared dependency +graph: composition stays predictable, ordering can be derived, and no analyzer +can develop a hidden reliance on another's internals. + +### Deterministic first, language model second + +Analysis is layered. A deterministic layer measures every turn with no model +calls at all — lengths, tool failures, wasted output, signs of correction, a +friction score. Only the turns that look high-signal are escalated to a +language-model layer for a nuanced judgement, and only then does a session-level +layer synthesise everything into proposals. The reason is cost and reliability: +the cheap, repeatable layer does the bulk of the triage and always works, the +expensive layer is spent sparingly on the moments that warrant it, and the whole +pipeline still produces useful structure even if the model layer is unavailable. + +**Tool arguments and error payloads are first-class evidence.** Analyzers may +consume tool-call arguments and tool-result error text, not just message prose. +This lets the classifier diagnose the *mechanism* of a failure (wrong flags, a +missing `--repo`, targeting the wrong resource) instead of paraphrasing the +user's complaint. The deterministic correction regex in `turn-pair-core` is a +*ranking signal only* — it enriches pairs it matches with a `note=` hint — but +it must never *gate* what the synthesizer is allowed to see. Every pair carries +a truncated verbatim user-text snippet in the digest; pairs the regex misses are +still visible to the session-level LLM. The un-gating ensures that recall is +not bounded by regex coverage. + +### Model access through the host platform, with a test seam + +All model calls go through one seam that, in production, defers to the host agent +platform's own provider system. The system does not embed provider SDKs, manage +its own keys, or talk to a local model server. The reason is to have exactly one +place where models and credentials are configured — the same place the user +already manages them — and to avoid drift between this tool and its host. That +same seam accepts a deterministic stand-in for testing, so the analysis logic can +be verified end to end without a network, a key, or any nondeterminism. + +### Proposals are materialised from their source + +Proposals are synthesised inside session-level analysis but then lifted into a +dedicated, fast store for review, each carrying an evidence trail back to the +conversation. A proposal's identity is its **input key**, derived from the +content-addressed **output key** of the node that produced it plus its ordinal +in that node's output — never from the model's free-text title, path, or +severity. So re-materialising the same node is idempotent (it never double- +inserts), but two genuinely distinct sources — a different session, or a revised +version — keep their proposals separately. Overlapping suggestions across +sessions are intentionally retained rather than collapsed: the review step is +expected to be consumed with the help of an AI agent that sees the whole +picture, so the listing simply groups proposals per session and ranks them +by confidence. The reason identity is anchored to the source rather than the +wording is that an idempotency key must be a function of *inputs*; the LLM's +output never feeds it, it only flows into a downstream consumer's source +reference via the output key. + +--- + +## 4. Invariants + +These statements must always hold. If a change would violate one, the change is +wrong. + +- A node, once written, is never modified or deleted. +- Every relationship is an edge with a valid kind and a valid target type; no + relationships are stored anywhere else. +- A node's identity equals its recipe fingerprint — analyzer, version, config, + and inputs; two nodes with the same recipe never both exist. +- The analyzer's shipped logic, default prompt, and default tier are represented + by its version; everything the user sets, including the resolved model, is + config. The version is graded major/minor by the author; config is never + graded. An analyzer that uses no model has no model in its identity. +- Re-running analysis without changing the version, config, or inputs produces no + new nodes. +- Improving an analyzer means a new version and new nodes; existing nodes for the + old version remain and stay reachable through lineage. +- An analyzer reads only the conversation, its own nodes, and the nodes of its + declared dependencies. +- Sessions are read-only; the system never writes to the conversation record. +- A proposal can always be traced, via edges, back to the conversation evidence + that justifies it. +- The system proposes changes to steering artifacts; it never applies them + itself. + +--- + +## 5. Boundaries and non-goals + +To keep the system focused, the following are explicitly *not* part of it: + +- **No automatic editing of steering artifacts.** The system surfaces proposals; + acting on them is a human decision. +- **No eager deletion of superseded analysis.** Old versions are kept for + comparison; reclaiming space, if ever needed, is a separate, deliberate act. +- **No bespoke model or credential management.** Model access is delegated to the + host platform; tiers abstract concrete models. +- **No cross-session meta-analysis as a first concern.** The unit of analysis is a + session; broader pattern-finding builds on top of that later. +- **No real session data inside the project.** All test material is hand-written + synthetic conversation; real user sessions never enter source, tests, history, + or build artifacts. + +--- + +## 6. How to think about a change + +When extending this system, ask in order: + +1. **Which concept am I touching?** Name it using the ubiquitous language above. + If you find yourself needing a term that isn't here, that's a signal to define + it here first. +2. **Does it preserve append-only and edge-only relationships?** If a change wants + to mutate a node or hide a relationship, reconsider. +3. **Does identity still equal the recipe?** If a change affects a node's content, + make sure it also affects the recipe, so idempotency and invalidation stay + honest. +4. **Is the work still discoverable by scanning?** New analysis must be something + a scan can classify as missing, stale, or current — not something tracked by + side state. +5. **Is the evidence trail intact?** Any new node that informs a proposal must be + reachable, by edges, from that proposal back to the conversation. + +Hold to these and the system stays what it is meant to be: a trustworthy, +cheaply-recomputable engine that turns the friction in past agent conversations +into clear, evidence-backed suggestions for making the next conversation better. diff --git a/README.md b/README.md index c5ef3c8..020b40c 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,10 @@ # pi-prospector -Incremental session indexing and proposal generation for the [Pi coding agent](https://github.com/earendil-works/pi). +Incremental session analysis and proposal generation for the [Pi coding agent](https://github.com/earendil-works/pi). -pi-prospector reads your Pi session transcripts, indexes them into a local SQLite database, and uses an LLM to propose improvements to your prompts, skills, and configuration — without applying them. You decide what to develop. +pi-prospector reads your Pi session transcripts, indexes them into a local SQLite database, and builds an **append-only analysis graph** over them — measuring every turn deterministically and using an LLM only where the signal warrants it. From that graph it surfaces concrete, ranked proposals to improve your prompts, skills, and configuration. It never applies them. You decide what to develop. + +pi-prospector is a Pi **extension**: it has no standalone CLI. Everything runs through slash commands and a `prospect` tool inside a Pi session. ## How it works @@ -10,164 +12,206 @@ pi-prospector reads your Pi session transcripts, indexes them into a local SQLit Pi sessions (~/.pi/agent/sessions/) │ ▼ -┌─────────────────────┐ -│ prospect sync │ ← Incremental. Only new lines are processed. -│ (no LLM, fast) │ Detects forks. Deduplicates shared message trees. -└────────┬────────────┘ +┌──────────────────────┐ +│ /prospect-sync │ ← Incremental, no LLM. Only new lines are parsed. +│ (fast, cheap) │ Detects forks; shared message trees stored once. +└────────┬─────────────┘ │ ▼ -┌─────────────────────┐ -│ sessions.db │ ← All session data, messages, and proposals -│ (SQLite + FTS5) │ -└────────┬────────────┘ +┌──────────────────────┐ +│ prospector.db │ ← Sessions, messages (+FTS), the analysis graph, +│ (SQLite + FTS5) │ and proposals. +└────────┬─────────────┘ │ ▼ -┌─────────────────────┐ -│ prospect analyze │ ← Runs an LLM over unprocessed sessions. -│ (uses Pi provider) │ Generates proposals. Does NOT edit any files. -└────────┬────────────┘ +┌──────────────────────┐ +│ /prospect-analyze │ ← Builds the analysis graph incrementally: +│ │ +│ turn-pair-core │ 1. Score every turn — deterministic, no LLM. +│ │ │ +│ ▼ │ +│ turn-pair-llm │ 2. Classify only high-signal turns — cheap tier. +│ │ │ +│ ▼ │ +│ session-overview │ 3. Synthesise → materialise proposals. +└────────┬─────────────┘ │ ▼ -┌─────────────────────┐ -│ proposals table │ ← status: new / accepted / rejected -│ in sessions.db │ Each proposal records when it was made. -└────────┬────────────┘ +┌──────────────────────┐ +│ proposals table │ ← status: open / applied / rejected / duplicate. +│ in prospector.db │ Each links back to the node that justifies it. +└────────┬─────────────┘ │ ▼ -┌─────────────────────┐ -│ Pi tool: prospect │ ← Your coding agent lists proposals, accepts -│ /prospect command │ or rejects them, requests syncs, checks stats. -└─────────────────────┘ +┌──────────────────────┐ +│ Pi tool: prospect │ ← Your agent lists proposals, accepts or rejects +│ /prospect-* commands│ them, requests syncs, checks stats. +└──────────────────────┘ ``` +The analysis graph is **append-only and incremental**. Each node records the exact *recipe* that produced it (which analyzer, at which version, under which config, over which inputs), so re-running analysis recomputes only what is genuinely out of date and never repeats expensive LLM work that is still current. See [`DESIGN.md`](./DESIGN.md) for the full model. + ## Install ```bash -pi install git:github:nicolas-marchildon/pi-prospector +pi install git:github.com/v2nic/pi-prospector ``` -Requires pi with an LLM API key configured for at least one provider. You choose which model to use for analysis. +Requires Pi with an LLM API key configured for at least one provider. You choose which models analysis uses (see [Configuration](#configuration)); the deterministic layer needs no model at all. + +> **Back up your sessions first.** pi-prospector treats `~/.pi/agent/sessions/` as read-only and never writes to it, but it does not make a backup for you. Run something like `tar czf ~/prospector-backup/sessions-$(date +%Y%m%d).tgz ~/.pi/agent/sessions/` before your first sync. ## Commands -### `/prospect sync` +### `/prospect-sync` Index session files into the database. No LLM is called. Fast and cheap. - Scans `~/.pi/agent/sessions/` for new or modified `.jsonl` files - Parses each file line-by-line, starting from the last line previously processed (incremental) -- Detects sessions that forked from another session via the `parentSession` header — shared message trees are stored once, not duplicated -- Tracks a cursor per session file: `{session_id, last_line, last_modified}` -- Re-indexes a file only if its modification time has changed since the last sync +- Detects sessions that forked from another via the `parentSession` header — shared message trees are stored once, not duplicated +- Tracks a cursor per session file (`{session_id, last_line, last_modified}`) and re-indexes a file only when its modification time changes + +Run it as often as you like. It's idempotent and incremental. + +### `/prospect-analyze [--revise ] [--limit N] [--session ID] [--analyzer ID] [--model provider/model]` + +Build the analysis graph over synced sessions and materialise proposals. By default it does the cheapest useful thing: it **fills only missing work**. Nodes that are already current are skipped; nodes that are out of date are left alone unless you ask for them with `--revise`. + +- `--revise major|minor|config|all` — also recompute *stale* nodes, selected by **why** they are stale: + - `major` — the analyzer shipped a significant new version + - `minor` — a small analyzer version bump (`minor` implies `major`) + - `config` — *your* setup changed (a threshold, a prompt override, the tier→model mapping, or a model pin — including the resolved model) + - `all` — every reason; combinable as a list, e.g. `--revise minor,config` + + Reasons only *select* which out-of-date nodes a run touches. A selected node is always recomputed to the **current recipe in full** (latest version, latest config, latest resolved model), and the new node is linked to its predecessor by a `revises` edge so lineage stays navigable. A plain fill scans only not-yet-analysed sessions; any `--revise` reason re-scans every session so stale work can be found. +- `--limit N` — cap how many sessions are scanned +- `--session ID` — analyse a single session +- `--analyzer ID` — run a single analyzer (`turn-pair-core`, `turn-pair-llm`, or `session-overview`) and its dependencies +- `--model provider/model` — pin **every** model tier to one concrete model for this run. Because the resolved model is part of a node's identity, a pinned run produces its own nodes; switching back to the normal mapping marks them stale (reason `config`). -Run this as often as you like. It's idempotent and incremental. +Proposals are never auto-applied. They sit in the database with status `open` until you accept or reject them. -### `/prospect analyze [--limit N] [--model provider/model]` +### `/prospect-stats` -Run an LLM over sessions that have been synced but not yet analyzed. Generates proposals and inserts them into the database. +Print a summary of the database: sessions indexed, messages and tool results, sessions analysed, proposals by status (`open`/`applied`/`rejected`/`duplicate`), and analysis-graph totals (nodes, edges, runs, and a breakdown of nodes by kind). -- Processes sessions in chronological order (oldest first) by default -- `--limit N`: only analyze N sessions (default: all unprocessed) -- `--model provider/model`: which Pi provider model to use (default: the model from `~/.pi/agent/prospector.json`, falls back to the current session model) -- Calls the Pi AI library (`@earendil-works/pi-ai`) directly — no subprocess, no extra session -- Each proposal records `created_at` so you can tell whether a session segment predates or postdates a given recommendation -- Analyze is incremental: it processes new or changed sessions regardless of whether past proposals from those sessions were accepted, rejected, or applied. The indexer and the analyzer are independent — sync always indexes new data, analyze always generates proposals from unprocessed data -- Proposals are never auto-applied. They sit in the database with status `new` until you decide +### `/prospect-proposals [status]` -### `/prospect stats` +List proposals, optionally filtered by status (`open`, `applied`, `rejected`, `duplicate`). Each row shows its status, severity, target, title, summary, and full id — together with a ready-to-paste `prospect show ` hint (proposal ids are time-ordered, so short prefixes can collide; the full id is always unambiguous). -Print a summary of the database: +- **Target** — what the proposal suggests changing (a category and optional path, e.g. a standing instruction file or a skill) +- **Severity** — the nature of the signal: `friction` | `correction` | `waste` | `suggestion` +- **Status** — `open`, `applied`, `rejected`, or `duplicate` -- Total sessions indexed -- Total messages (user + assistant) and tool responses -- Number of messages processed by the LLM -- Number of proposals by status (new / accepted / rejected) +Add `--full` (or `-v`) to also print each proposal's detail, evidence, and source node. -### `/prospect proposals [--status new|accepted|rejected]` +### `/prospect-show ` -List proposals from the database, optionally filtered by status. +Show one proposal together with the **verbatim turns it was synthesised from**, so you can judge it against the real conversation without re-opening the transcript. It accepts a full proposal id or any unambiguous id-prefix. -Each proposal shows: -- **ID** — unique identifier -- **Target** — what this proposal suggests changing (e.g. `AGENTS.md § Tool usage`, `skill/debug-typescript-errors`) -- **Severity** — `friction` | `correction` | `waste` | `suggestion` -- **Summary** — one-line description of the proposed change -- **Created** — when the proposal was generated -- **Session** — which session triggered it -- **Status** — `new`, `accepted`, or `rejected` +It walks the proposal's provenance — proposal → its source `session-overview` node → the turn nodes that node **consumed** → the messages those turns **anchor** — and prints, for each high-signal turn: -### `/prospect accept ` +- the deterministic and LLM signals for that turn (friction score, correction type, tool-failure count, sentiment/severity) +- the **user** text and the **assistant** text, verbatim +- every **tool call with its arguments**, plus any tool-result errors -Mark a proposal as accepted. This does **not** apply the proposal — it only updates the status. You then ask your Pi coding agent to implement it. +Because the overview consumes every turn, output is focused to the high-signal turns (those with friction or an LLM classification) and capped, with a note for any omitted turns. Surfacing the actual tool-call arguments often reveals mechanism-level detail the text-only classifier cannot see — for example whether a push failure was really about the push command or about a later `gh pr create` target. -### `/prospect reject ` +### `/prospect-accept ` -Mark a proposal as rejected. +Mark an open proposal as `applied`. This does **not** apply the change — it only updates the status. You then ask your Pi coding agent to implement it. + +### `/prospect-reject ` + +Mark an open proposal as `rejected`. + +### `/prospect-verify` + +Recompute every analysis node's output key from its stored content and confirm it matches what is recorded. Because identities are content-addressed, any mismatch reveals out-of-band tampering or corruption of the database. Pure read; reports `ok` or lists the mismatching nodes. See [Verification](#design) in `DESIGN.md`. ## Pi tool: `prospect` -When installed, pi-prospector registers a `prospect` tool that the Pi coding agent can call during sessions: +When installed, pi-prospector registers a `prospect` tool the Pi coding agent can call during a session: | Action | What it does | |--------|-------------| | `sync` | Index new/modified sessions into the database | | `stats` | Return sync and proposal statistics | | `list_proposals` | List proposals, optionally filtered by status | -| `accept` | Mark a proposal as accepted | +| `accept` | Mark a proposal as applied | | `reject` | Mark a proposal as rejected | -| `analyze` | Run the LLM over unprocessed sessions | -This lets you say things like "show me new proposals" or "sync my sessions and check stats" directly in a Pi conversation. +This lets you say things like "show me open proposals" or "sync my sessions and check stats" directly in a Pi conversation. (Analysis itself runs through `/prospect-analyze`, not the tool, because it can be long-running and cost money.) ## What gets analyzed -pi-prospector reads **only what is inside Pi session files**. It does not read Pi configuration files, `AGENTS.md`, skill files, or any other artifact directly. The session file contains: +pi-prospector reads **only what is inside Pi session files**. It does not read Pi configuration files, `AGENTS.md`, skill files, or any other artifact directly. A session file contains: - User messages (what you said) - Assistant messages (what the agent said, including thinking) - Tool calls and tool results (what the agent did) - Compaction summaries (what was retained after context compression) -- Model changes and thinking level changes +- Model changes and thinking-level changes -The system prompt is not stored in session files and is not captured in v1. - -## Timestamps - -Each proposal records `created_at`. Each session message has a `timestamp`. These are stored in the database in case you want to correlate proposals with session activity later. v1 does nothing with this information beyond storing it. +The unit of analysis is a **turn** — one round of work, segmented at the same boundaries Pi uses (a user or `bashExecution` message, or a `branch_summary`/`custom_message` entry). The deterministic layer scores every turn; only high-signal turns are sent to the LLM. The system prompt is not stored in session files and is not captured. ## Fork deduplication -Pi sessions are stored as trees. When you branch a session with `/tree`, the new session file has a `parentSession` header pointing to the original. Messages before the branch point are shared. - -During sync, pi-prospector: - -1. Reads the `parentSession` header from each session file -2. Resolves the parent session file -3. Stores shared messages once, tagged with the original session -4. Marks the forked session as starting from the branch point - -This means analyzing a forked session only processes the **new** messages after the fork — not the entire conversation history again. +Pi sessions are stored as trees. When you branch a session with `/tree`, the new session file carries a `parentSession` header pointing to the original, and messages before the branch point are shared. During sync, pi-prospector reads that header, resolves the parent, stores shared messages once, and marks the forked session as starting from the branch point — so analysing a fork only processes the **new** messages after the branch. ## Configuration -Create `~/.pi/agent/prospector.json`: +Create `~/.pi/agent/prospector.json` (all fields optional): ```json { - "model": "openrouter/deepseek-v4-flash", - "dbPath": "~/.pi/agent/prospector.db" + "dbPath": "~/.pi/agent/prospector.db", + "modelTiers": { + "cheap": "anthropic/claude-haiku-4-5", + "mid": "anthropic/claude-sonnet-4-5", + "expensive": "anthropic/claude-opus-4-1" + } } ``` | Field | Default | Description | |-------|---------|-------------| -| `model` | *(current session model)* | Provider and model to use for analysis, in `provider/model` format. Must be a model Pi has an API key for. Cheaper models like `openrouter/deepseek-v4-flash` or `gemma4:26b` work well for analysis. Override per-run with `--model`. | -| `dbPath` | `~/.pi/agent/prospector.db` | Path to the SQLite database | +| `dbPath` | `~/.pi/agent/prospector.db` | Path to the SQLite database. A leading `~` is expanded. | +| `modelTiers` | Claude haiku-4-5 / sonnet-4-5 / opus-4-1 | Maps the abstract tiers analyzers request (`cheap`/`mid`/`expensive`) to concrete `provider/model` strings. Each must be a model Pi has credentials for. Override every tier for a single run with `--model`. | + +Analyzers ask for a **tier**, not a model, so you tune cost vs. quality in one place. The resolved model is part of a node's identity: change the mapping and the affected nodes become stale (reason `config`), recomputed when you next run `--revise config`. All model access goes through Pi's own provider system — pick any model Pi supports (configured via `/login` or API keys). The deterministic `turn-pair-core` layer needs no model and always runs. + +The following environment variables override paths and are mainly for testing: `PROSPECTOR_DB_PATH`, `PROSPECTOR_SESSIONS_DIR`, `PROSPECTOR_CONFIG`. + +## Running headlessly + +The commands are normally invoked as slash commands inside an interactive Pi session, but the extension also registers a `--prospect` CLI flag so a single command runs **non-interactively and exits** — no `-p` needed. This is the convenient way to drive prospector from scripts or while iterating on the analyzers (the extension is reloaded fresh from source on every run, so code changes take effect without restarting an interactive session): + +```bash +pi -e ./src/index.ts --prospect sync +pi -e ./src/index.ts --prospect stats +pi -e ./src/index.ts --prospect "analyze --limit 3 --model openrouter/anthropic/claude-3.5-haiku" +pi -e ./src/index.ts --prospect proposals +pi -e ./src/index.ts --prospect "accept " +``` + +The value is `" [args]"`; quote it when it contains spaces. Commands: `sync`, `analyze [flags]`, `stats`, `proposals [status] [--full]`, `show `, `verify`, `accept `, `reject `. When `--prospect` is absent the extension stays fully interactive. (`-ne` additionally skips discovery of other extensions, and `--no-session` keeps the run ephemeral.) + +To iterate on a small **private** subset rather than your whole history, copy a few session folders somewhere outside any repo and point the env overrides at them — the sessions directory is only ever read: + +```bash +export PROSPECTOR_SESSIONS_DIR="$HOME/.prospector-local/sessions" +export PROSPECTOR_DB_PATH="$HOME/.prospector-local/prospector.db" +pi -e ./src/index.ts --prospect stats +``` -The model must correspond to a provider Pi already has credentials for (configured via `/login` or API keys). Any model Pi supports works — pick based on cost vs. quality. For backfill, a cheap model is recommended. +For structured-output calls, prefer a non-reasoning model/tier: reasoning models spend the token budget on thinking and can truncate the JSON answer (the LLM caller now fails fast with a clear message when a response is cut off at the output limit). +## Design +[`DESIGN.md`](./DESIGN.md) is the canonical description of the system: the ubiquitous language, the append-only graph, recipe-based identity and idempotency, versioned lineage, the reach of a run, and the deterministic-first layering. Read it before changing analysis behaviour. ## License -MIT \ No newline at end of file +MIT diff --git a/docs/analyzer-design-c.md b/docs/analyzer-design-c.md deleted file mode 100644 index ff96d57..0000000 --- a/docs/analyzer-design-c.md +++ /dev/null @@ -1,922 +0,0 @@ -# pi-prospector Analyzer Framework — Final Design - -## 1. Overview - -The analyzer framework extends pi-prospector's session index with an **append-only analysis graph** grafted onto the conversation tree. Every analysis artifact is versioned, idempotent, and traceable back to the exact analyzer code, prompt, and configuration that produced it. - -All relationships between analysis nodes and between nodes and other entities are expressed through a single **typed edge table**. There are no tree-style `parent_id` columns and no denormalized anchor columns on analysis nodes — the edges table is the single source of truth for graph relationships. - -Proposals are materialized from analysis nodes into a fast-access table for user review and deduplication. - -### Core principles - -1. **Append-only** — analysis nodes are never mutated. New analyzer versions or config changes produce new nodes; old ones persist. -2. **Typed edge graph** — all relationships (anchoring, consumption, refinement, provenance) are explicit edges with kinds. No `parent_id`, no anchor columns on nodes. -3. **Idempotent** — an `(input_hash)` uniquely identifies a node produced by a given recipe on a given source set. Re-running is a no-op. -4. **Incremental** — cursors track progress per (analyzer, version, config, session). Only new messages get analyzed. -5. **Crash-recoverable** — re-running after a crash picks up where it left off by checking which source combinations already have nodes. -6. **Versioned provenance** — every node traces to the exact analyzer version, prompt version, config version, and run that produced it. -7. **Dependency-scoped visibility** — an analyzer sees only its own nodes and nodes from declared dependencies. -8. **Deterministic first, LLM optional** — every analyzer produces a deterministic baseline. LLM enrichment is a separate pass on flagged artifacts. - -``` -Conversation graph (read-only): - msg_001 → msg_002 → msg_003 → [compaction] → msg_004 → msg_005 - -Analysis graph (append-only, grafted via edges): - msg_003 ←─┬─ turn-pair node (metric, deterministic) - └─ turn-pair-llm node (classification, cheap LLM) - - session ─── session-overview node (summary + proposals, mid LLM) - │ consumes turn-pair and turn-pair-llm nodes - │ anchors to the session - │ produces proposal nodes -``` - ---- - -## 2. Data model - -### 2.1 Entity-relationship diagram - -``` -analyzer_defs ──1:N──→ analyzer_versions - │ - │ N:N via prompt_version_edges (implicit, through runs) - ▼ -prompt_registry analysis_runs ──1:N──→ analysis_nodes ──→ analysis_edges - │ │ │ -analyzer_config_versions ─────────┘ │ │ - │ │ -sessions ──1:N──→ messages ──────────────────────────────────────┘ │ - │ │ -proposals ◄── materialized from analysis_nodes ──────────────────────┘ - -analysis_progress (per analyzer/version/config/session cursor) -``` - -### 2.2 Table definitions - -#### `analyzer_defs` — stable logical identity - -One row per analyzer, regardless of version. Never deleted. - -```sql -CREATE TABLE IF NOT EXISTS analyzer_defs ( - id TEXT PRIMARY KEY, -- 'turn-pair-core', 'session-overview' - label TEXT NOT NULL, -- 'Per-Turn Sentiment & Friction' - description TEXT, - anchor_span TEXT NOT NULL, -- 'pair' | 'segment' | 'full_session' - dependencies TEXT NOT NULL DEFAULT '[]', -- JSON array of analyzer_def IDs - created_at TEXT NOT NULL -); -``` - -#### `analyzer_versions` — one row per code release - -```sql -CREATE TABLE IF NOT EXISTS analyzer_versions ( - analyzer_id TEXT NOT NULL, - version_id TEXT NOT NULL, -- commit SHA or semver - implementation_kind TEXT NOT NULL, -- 'deterministic' | 'in_process_llm' | 'pi_subagent' - code_ref TEXT, -- git commit, npm version, or extension path - created_at TEXT NOT NULL, - PRIMARY KEY (analyzer_id, version_id), - FOREIGN KEY (analyzer_id) REFERENCES analyzer_defs(id) -); -``` - -#### `prompt_registry` — content-addressed prompt store - -Immutable. Multiple analyzers can share identical prompts. - -```sql -CREATE TABLE IF NOT EXISTS prompt_registry ( - hash TEXT PRIMARY KEY, -- SHA-256 first 16 hex chars - content TEXT NOT NULL, -- full prompt template text - role TEXT, -- 'classify' | 'map' | 'reduce' | 'verify' | null - created_at TEXT NOT NULL -); -``` - -#### `analyzer_configs` — content-addressed config/parameter store - -Every change to an analyzer's config produces a new row. Nodes reference the specific config that produced them. - -```sql -CREATE TABLE IF NOT EXISTS analyzer_configs ( - id TEXT PRIMARY KEY, -- UUID v7 - analyzer_id TEXT NOT NULL, - config_hash TEXT NOT NULL UNIQUE, -- SHA-256 of canonical JSON - config_json TEXT NOT NULL, -- e.g. {"cheap_model":"anthropic/haiku-3","friction_threshold":0.5} - label TEXT, -- 'default', 'sensitive', 'aggressive' - created_at TEXT NOT NULL, - FOREIGN KEY (analyzer_id) REFERENCES analyzer_defs(id) -); -``` - -#### `analysis_runs` — execution provenance - -One row per attempted execution of an analyzer on a session. Separates execution metadata from artifact data. - -```sql -CREATE TABLE IF NOT EXISTS analysis_runs ( - id TEXT PRIMARY KEY, - analyzer_id TEXT NOT NULL, - analyzer_version_id TEXT NOT NULL, - config_id TEXT NOT NULL, - session_id TEXT NOT NULL, - - status TEXT NOT NULL DEFAULT 'planned', -- 'planned'|'running'|'ok'|'error'|'partial' - prompt_bundle_hash TEXT NOT NULL, -- SHA-256 of sorted prompt hashes used in this run - - started_at TEXT NOT NULL, - finished_at TEXT, - model_spec TEXT, -- resolved model string, e.g. 'anthropic/claude-sonnet-4-5' - cost_usd REAL DEFAULT 0, - tokens_used INTEGER DEFAULT 0, - nodes_produced INTEGER DEFAULT 0, - nodes_skipped INTEGER DEFAULT 0, - error_message TEXT, - - FOREIGN KEY (analyzer_id, analyzer_version_id) REFERENCES analyzer_versions(analyzer_id, version_id), - FOREIGN KEY (session_id) REFERENCES sessions(id) -); -CREATE INDEX IF NOT EXISTS idx_runs_session ON analysis_runs(session_id); -CREATE INDEX IF NOT EXISTS idx_runs_status ON analysis_runs(status); -``` - -#### `analysis_nodes` — append-only artifact store - -No `UPDATE` or `DELETE` after insert. No `parent_id`, no anchor columns — all relationships go through `analysis_edges`. - -```sql -CREATE TABLE IF NOT EXISTS analysis_nodes ( - id TEXT PRIMARY KEY, -- UUID v7 (time-sortable) - session_id TEXT NOT NULL, - - analyzer_id TEXT NOT NULL, - analyzer_version_id TEXT NOT NULL, - config_id TEXT NOT NULL, - run_id TEXT NOT NULL, - - node_kind TEXT NOT NULL, -- 'metric'|'classification'|'summary'|'proposal'|'error' - - content_json TEXT NOT NULL, -- structured artifact payload (schema varies by node_kind) - source_set_hash TEXT NOT NULL, -- SHA-256 of sorted source refs (what went in) - input_hash TEXT NOT NULL, -- SHA-256(recipe) for idempotency lookup - - created_at TEXT NOT NULL, - - -- LLM metadata (NULL for deterministic nodes) - model_used TEXT, - cost_usd REAL DEFAULT 0, - tokens_used INTEGER DEFAULT 0, - duration_ms INTEGER, - - FOREIGN KEY (run_id) REFERENCES analysis_runs(id), - FOREIGN KEY (session_id) REFERENCES sessions(id) -); -CREATE INDEX IF NOT EXISTS idx_nodes_session ON analysis_nodes(session_id); -CREATE INDEX IF NOT EXISTS idx_nodes_analyzer ON analysis_nodes(analyzer_id, analyzer_version_id); -CREATE INDEX IF NOT EXISTS idx_nodes_kind ON analysis_nodes(node_kind); -CREATE INDEX IF NOT EXISTS idx_nodes_input_hash ON analysis_nodes(input_hash); -CREATE INDEX IF NOT EXISTS idx_nodes_source_hash ON analysis_nodes(source_set_hash); -CREATE INDEX IF NOT EXISTS idx_nodes_config ON analysis_nodes(config_id); -CREATE INDEX IF NOT EXISTS idx_nodes_idempotency ON analysis_nodes(analyzer_id, analyzer_version_id, config_id, source_set_hash); -``` - -#### `analysis_edges` — typed graph relationships - -The single table that makes this a real graph. Every edge has a kind that explains the relationship. - -```sql -CREATE TABLE IF NOT EXISTS analysis_edges ( - from_node_id TEXT NOT NULL, -- the node that holds this relationship - to_ref_kind TEXT NOT NULL, -- what kind of entity the target is - to_ref_id TEXT NOT NULL, -- id of the target entity - edge_kind TEXT NOT NULL, -- what the relationship means - ordinal INTEGER DEFAULT 0, -- ordering within same (from_node, edge_kind) - PRIMARY KEY (from_node_id, to_ref_kind, to_ref_id, edge_kind, ordinal), - FOREIGN KEY (from_node_id) REFERENCES analysis_nodes(id) -); -CREATE INDEX IF NOT EXISTS idx_edges_from ON analysis_edges(from_node_id); -CREATE INDEX IF NOT EXISTS idx_edges_to ON analysis_edges(to_ref_kind, to_ref_id); -CREATE INDEX IF NOT EXISTS idx_edges_kind ON analysis_edges(edge_kind); -``` - -**Edge kinds:** - -| Edge kind | from | to | Meaning | -|-----------|------|----|---------| -| `anchors` | analysis_node | `message` or `session` | This node is about this conversation entity. A pair-level node anchors to its user message. A session-level node anchors to the session. | -| `consumes` | analysis_node | `message` or `analysis_node` | This node used this as input. A session-overview consumes turn-pair nodes. An LLM enrichment consumes its deterministic base node. | -| `refines` | analysis_node | `analysis_node` | This node builds on top of another. An LLM enrichment refines its deterministic base. | -| `uses_prompt` | analysis_node | `prompt_version` (by hash) | This node was produced using this prompt. | -| `uses_config` | analysis_node | `analyzer_config` (by id) | This node was produced with this config. | -| `produces` | analysis_node | (proposal extracted from content) | This node produced this proposal. An `produces` edge connects the session-overview (or other summary) node to each proposal materialized from it. | - -**Navigation queries:** - -```sql --- All analysis anchored to a specific message -SELECT an.* FROM analysis_nodes an -JOIN analysis_edges e ON e.from_node_id = an.id -WHERE e.edge_kind = 'anchors' - AND e.to_ref_kind = 'message' - AND e.to_ref_id = 'msg_006'; - --- All sources consumed by a session-overview node -SELECT e.to_ref_kind, e.to_ref_id FROM analysis_edges e -WHERE e.from_node_id = 'an:session_overview_1' - AND e.edge_kind = 'consumes'; - --- Walk from a proposal back to the conversation messages that produced it -WITH RECURSIVE provenance AS ( - SELECT e.from_node_id, e.to_ref_kind, e.to_ref_id - FROM analysis_edges e - WHERE e.to_ref_kind = 'message' AND e.edge_kind = 'anchors' - AND e.from_node_id IN ( - SELECT from_node_id FROM analysis_edges - WHERE edge_kind = 'consumes' AND to_ref_kind = 'analysis_node' - AND to_ref_id IN ( - SELECT from_node_id FROM analysis_edges - WHERE edge_kind = 'produces' AND to_ref_id = 'proposal_42' - ) - ) - UNION ALL - SELECT e.from_node_id, e.to_ref_kind, e.to_ref_id - FROM analysis_edges e - JOIN provenance p ON e.from_node_id = p.to_ref_id - WHERE e.edge_kind = 'consumes' AND p.to_ref_kind = 'analysis_node' -) -SELECT DISTINCT to_ref_id FROM provenance WHERE to_ref_kind = 'message'; -``` - -#### `analysis_progress` — incremental cursor per (analyzer, version, config, session) - -```sql -CREATE TABLE IF NOT EXISTS analysis_progress ( - analyzer_id TEXT NOT NULL, - analyzer_version_id TEXT NOT NULL, - config_id TEXT NOT NULL, - session_id TEXT NOT NULL, - - cursor_json TEXT, -- {"last_message_rowid": 452, "last_pair_index": 17} - last_run_id TEXT, - total_analyzed INTEGER DEFAULT 0, - status TEXT NOT NULL DEFAULT 'ok', -- 'ok'|'in_progress'|'error'|'needs_rerun' - error_message TEXT, - updated_at TEXT NOT NULL, - - PRIMARY KEY (analyzer_id, analyzer_version_id, config_id, session_id), - FOREIGN KEY (session_id) REFERENCES sessions(id) -); -``` - -#### `proposals` — fast-access materialized view - -Derived from `analysis_nodes` where `node_kind = 'proposal'`. Kept in sync by the framework after each run. - -```sql -CREATE TABLE IF NOT EXISTS proposals ( - id TEXT PRIMARY KEY, - analysis_node_id TEXT NOT NULL UNIQUE, -- 1:1 with the source analysis node - - session_id TEXT NOT NULL, - analyzer_id TEXT NOT NULL, - - target_type TEXT NOT NULL, -- 'agents_md'|'system_md'|'skill'|'extension_prompt'|'tool_output'|'repo_doc'|'config' - target_path TEXT, -- e.g. '~/.pi/agent/AGENTS.md' - title TEXT NOT NULL, - summary TEXT NOT NULL, - detail TEXT, -- proposed edit text or detailed explanation - evidence_json TEXT, -- JSON array of message/node references and excerpts - confidence REAL, -- 0.0–1.0 - severity TEXT, -- 'friction'|'correction'|'waste'|'suggestion'|'insight' - dedup_key TEXT, -- hash of (target_type, target_path, severity, normalize(title)) - - status TEXT NOT NULL DEFAULT 'open', -- 'open'|'accepted'|'applied'|'rejected'|'duplicate' - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL, - - FOREIGN KEY (analysis_node_id) REFERENCES analysis_nodes(id), - FOREIGN KEY (session_id) REFERENCES sessions(id) -); -CREATE INDEX IF NOT EXISTS idx_proposals_status ON proposals(status); -CREATE INDEX IF NOT EXISTS idx_proposals_target ON proposals(target_type, target_path); -CREATE INDEX IF NOT EXISTS idx_proposals_dedup ON proposals(dedup_key); -CREATE INDEX IF NOT EXISTS idx_proposals_session ON proposals(session_id); -``` - ---- - -## 3. Idempotency model - -### 3.1 Recipe - -A node is uniquely identified by its `input_hash`: - -``` -input_hash = SHA-256( - analyzer_id - | analyzer_version_id - | config_id - | prompt_bundle_hash -- SHA-256 of all prompt hashes used, sorted - | source_set_hash -) -``` - -Where: -- `source_set_hash = SHA-256(sorted(source_refs).map(r => r.kind + ':' + r.id).join('|'))` -- `prompt_bundle_hash = SHA-256(sorted(prompt_hashes).join('|'))` - -### 3.2 Idempotency check - -Before producing a node, the framework checks: - -```sql -SELECT 1 FROM analysis_nodes WHERE input_hash = ?; -``` - -If a row exists → skip (already computed). - -### 3.3 When does a new recipe get created? - -| What changes | Effect on recipe | Effect on existing nodes | -|---|---|---| -| Analyzer code updated (new `version_id`) | New recipe → new nodes | Old nodes remain | -| Config parameters changed (new `config_id`) | New recipe → new nodes | Old nodes remain | -| Prompt text changed (new `prompt_hash`) → new `prompt_bundle_hash` | New recipe → new nodes | Old nodes remain | -| Model changed | No recipe change | Same nodes are valid; model is metadata on `analysis_run` | -| New messages synced | New source refs → new `source_set_hash` for those units | Old units keep their hash | -| Analyzer re-run with same recipe | Idempotency check finds existing node → skip | No new nodes | - ---- - -## 4. Analyzer interface - -### 4.1 TypeScript types - -```typescript -// ── Analyzer definition ── - -interface AnalyzerDef { - id: string; // 'turn-pair-core', 'session-overview' - label: string; - description: string; - anchorSpan: 'pair' | 'segment' | 'full_session'; - dependencies: string[]; // analyzer_def IDs -} - -interface AnalyzerVersion { - analyzerId: string; - versionId: string; // commit SHA or semver - implementationKind: 'deterministic' | 'in_process_llm' | 'pi_subagent'; - codeRef?: string; -} - -interface PromptVersion { - hash: string; // content hash (first 16 hex chars of SHA-256) - content: string; - fullHash: string; // full SHA-256 for verification - role?: string; // 'classify' | 'map' | 'reduce' | 'verify' -} - -interface AnalyzerConfig { - id: string; // config hash or UUID - analyzerId: string; - configJson: Record; - configHash: string; // SHA-256 of canonical JSON - label?: string; -} - -// ── Analysis units ── - -interface AnalysisUnit { - sources: SourceRef[]; - sourceSetHash: string; - /** What kind of conversation entity this unit targets */ - anchorKind: 'message' | 'pair' | 'segment' | 'session' | 'analysis_node' | 'none'; - /** The id of the anchor (message.id or session.id), null for 'none' */ - anchorRef?: string; - meta?: Record; -} - -interface SourceRef { - kind: 'message' | 'analysis_node' | 'session'; - id: string; -} - -// ── Analysis result ── - -interface AnalysisResult { - contentJson: Record; - nodeKind: 'metric' | 'classification' | 'summary' | 'proposal' | 'error'; - /** What kind of conversation entity this node is about */ - anchorKind: 'message' | 'pair' | 'segment' | 'session' | 'analysis_node' | 'none'; - /** The id of the anchor (message.id or session.id), null for 'none' */ - anchorRef?: string; - - edges: Array<{ - toRefKind: SourceRef['kind'] | 'prompt_version' | 'config_version'; - toRefId: string; - edgeKind: 'anchors' | 'consumes' | 'refines' | 'uses_prompt' | 'uses_config' | 'produces'; - ordinal?: number; - }>; - - modelUsed?: string; - costUsd?: number; - tokensUsed?: number; - durationMs?: number; -} - -// ── Analyzer interface ── - -interface Analyzer { - def: AnalyzerDef; - version: AnalyzerVersion; - prompts: Record; - defaultConfig: AnalyzerConfig; - - plan(ctx: AnalyzerPlanContext): Promise; - analyze(unit: AnalysisUnit, ctx: AnalyzerRunContext): Promise; -} - -// ── Contexts ── - -interface AnalyzerPlanContext { - sessionId: string; - messages: MessageRow[]; - allNodes: AnalysisNodeRow[]; - ownNodes: AnalysisNodeRow[]; - dependencyNodes: Record; - progress: ProgressRow | null; - db: Database; -} - -interface AnalyzerRunContext { - getMessage(id: string): MessageRow | undefined; - getNode(id: string): AnalysisNodeRow | undefined; - getDependencyNodes(analyzerId: string): AnalysisNodeRow[]; - llm(request: LLMRequest): Promise; - run: RunRow; - config: AnalyzerConfig; - prompts: Record; -} -``` - -### 4.2 Framework execution flow - -``` -runAnalyzer(analyzer, sessionId, config) - - 1. Resolve analyzer version, store prompts (INSERT OR IGNORE), resolve config - 2. Compute prompt_bundle_hash - 3. Create analysis_run row (status = 'running') - - 4. analyzer.plan(ctx) → AnalysisUnit[] - 5. For each unit: - a. Compute source_set_hash - b. Compute input_hash = SHA-256(analyzer_id | version_id | config_id | prompt_bundle_hash | source_set_hash) - c. Idempotency check: SELECT 1 FROM analysis_nodes WHERE input_hash = ? - → if exists, skip, increment nodes_skipped - d. Call analyzer.analyze(unit, runCtx) - e. INSERT INTO analysis_nodes - f. INSERT INTO analysis_edges for each edge in result.edges - g. If node_kind = 'proposal' → upsert into proposals table - h. Increment nodes_produced - - 6. Update analysis_run (status = 'ok' or 'error', cost, tokens) - 7. Update analysis_progress (cursor, status) - - 8. Return { runId, nodes_produced, nodes_skipped, cost_usd } -``` - -### 4.3 Crash recovery - -If the process crashes between steps 5d and 5f: - -- Nodes already inserted are valid (append-only, no mutations) -- Nodes not yet inserted have no row in `analysis_nodes` → their `input_hash` doesn't exist → re-running will produce them -- Edges for inserted nodes might be missing → a repair pass can re-link orphaned nodes by checking `analysis_nodes` rows with no matching `analysis_edges` - -The framework can also detect partial runs: -```sql -SELECT * FROM analysis_runs WHERE status = 'running'; -``` -These can be retried or marked as `'error'`. - ---- - -## 5. Isolation model - -### 5.1 Visibility rule - -An analyzer with `def.id = X` and `def.dependencies = ["A", "B"]` can see: - -1. **Conversation data** — all messages and sessions (always readable) -2. **Own nodes** — `analysis_nodes WHERE analyzer_id = X` -3. **Dependency nodes** — `analysis_nodes WHERE analyzer_id IN ('A', 'B')` - -It CANNOT see nodes from analyzers not in its dependency list. - -### 5.2 Enforcement - -The framework enforces this in `AnalyzerPlanContext` and `AnalyzerRunContext`: -- `dependencyNodes` only includes declared dependencies -- `getDependencyNodes(analyzerId)` validates against the dependency list - -If an analyzer runs as a Pi sub-agent in the future, tool wrappers will enforce the same visibility. - ---- - -## 6. Analyzer 1: `turn-pair-core` - -### 6.1 Identity - -``` -id: "turn-pair-core" -label: "Per-Turn Deterministic Metrics" -anchor_span: "pair" -dependencies: [] -implementation_kind: "deterministic" -``` - -### 6.2 Scope - -A single (user_message → assistant_response + intervening tool_results) pair. - -### 6.3 Deterministic properties (always produced, no LLM) - -| Property | Type | Source | -|---|---|---| -| `user_msg_length` | integer | `len(user_msg.content_text)` | -| `assistant_msg_length` | integer | `len(assistant_msg.content_text)` | -| `has_thinking` | boolean | `assistant_msg.content_thinking != null` | -| `thinking_length` | integer | `len(assistant_msg.content_thinking) \|\| 0` | -| `correction_detected` | boolean | regex match | -| `correction_patterns` | string[] | which patterns matched | -| `correction_type` | string \| null | `'explicit' \| 'implicit' \| 'repetition' \| null` | -| `correction_text` | string \| null | extracted corrective instruction | -| `tool_call_count` | integer | number of tool calls | -| `tool_names` | string[] | names of tools called | -| `tool_failure_count` | integer | tool results with `is_error = true` | -| `tool_failure_details` | object[] | `[{tool_name, error_preview}]` | -| `tool_waste_bytes` | integer | bytes of tool results never referenced in subsequent text | -| `retry_detected` | boolean | same tool+target called 2+ times | -| `elapsed_seconds` | float \| null | time between user and assistant timestamps | -| `friction_score` | float | 0.0–1.0 (computed from signals) | -| `model` | string \| null | model that produced assistant response | -| `stop_reason` | string \| null | from assistant response | -| `usage_input_tokens` | integer \| null | from assistant response | -| `usage_output_tokens` | integer \| null | from assistant response | -| `is_compaction_boundary` | boolean | true if any message in the pair is a compactionSummary | - -Correction patterns (same regex sets as discussed in earlier drafts — strong, weak, negation). - -### 6.4 Plan logic - -```typescript -plan(ctx: AnalyzerPlanContext): AnalysisUnit[] { - const units: AnalysisUnit[] = []; - for (let i = 0; i < ctx.messages.length; i++) { - if (ctx.messages[i].role !== 'user') continue; - let j = i + 1; - while (j < ctx.messages.length && ctx.messages[j].role !== 'assistant') j++; - if (j >= ctx.messages.length) continue; - - const sources: SourceRef[] = []; - for (let k = i; k <= j && k < ctx.messages.length; k++) { - sources.push({ kind: 'message', id: ctx.messages[k].id }); - } - - units.push({ - sources, - sourceSetHash: computeSourceSetHash(sources), - anchorKind: 'pair', - anchorRef: ctx.messages[i].id, // the user message - meta: { userIndex: i, assistantIndex: j }, - }); - } - return units; -} -``` - -### 6.5 Edges produced - -Each node creates: - -``` -anchors → each message in the pair (user, assistant, and intervening tool results) -``` - -No `consumes`, `refines`, or dependency edges (this is a root analyzer with no dependencies). - ---- - -## 7. Analyzer 2: `turn-pair-llm` - -### 7.1 Identity - -``` -id: "turn-pair-llm" -label: "Per-Turn LLM Sentiment & Friction" -anchor_span: "pair" -dependencies: ["turn-pair-core"] -implementation_kind: "in_process_llm" -``` - -### 7.2 Scope - -Only processes pairs where the deterministic `turn-pair-core` flagged `correction_detected: true` or `friction_score >= 0.4`. - -### 7.3 LLM properties - -| Property | Type | Source | -|---|---|---| -| `sentiment` | string | LLM: `'positive' \| 'neutral' \| 'negative' \| 'frustrated'` | -| `frustration_level` | integer | LLM: 0–10 | -| `correction_type_llm` | string \| null | LLM: `'explicit' \| 'implicit' \| 'repetition' \| null` | -| `friction_cause` | string \| null | LLM | -| `friction_summary` | string \| null | LLM: 1–2 sentences | -| `user_intent` | string | LLM: what the user was trying to accomplish | -| `quality_score` | integer | LLM: 1–5 | - -### 7.4 Plan logic - -```typescript -plan(ctx: AnalyzerPlanContext): AnalysisUnit[] { - const deterministicNodes = ctx.dependencyNodes['turn-pair-core']; - const highSignal = deterministicNodes.filter(n => { - const props = JSON.parse(n.content_json); - return props.correction_detected || props.friction_score >= 0.4; - }); - - return highSignal.map(n => ({ - sources: [{ kind: 'analysis_node', id: n.id }], - sourceSetHash: computeSourceSetHash([{ kind: 'analysis_node', id: n.id }]), - anchorKind: 'analysis_node', - anchorRef: n.id, - meta: { deterministicNodeId: n.id }, - })); -} -``` - -### 7.5 Edges produced - -Each node creates: - -``` -refines → turn-pair-core node it enriches -consumes → turn-pair-core node (same as refines, but different semantic) -anchors → same messages as the turn-pair-core node (inherited) -uses_prompt → the prompt used for classification -``` - ---- - -## 8. Analyzer 3: `session-overview` - -### 8.1 Identity - -``` -id: "session-overview" -label: "Session-Level Analysis & Proposals" -anchor_span: "full_session" -dependencies: ["turn-pair-core", "turn-pair-llm"] -implementation_kind: "in_process_llm" -``` - -### 8.2 Scope - -One node per session. Consumes all turn-pair-core and turn-pair-llm nodes. - -### 8.3 Context budget strategy - -For sessions that fit in the model's context window: -``` -Structured digest → single LLM call -``` - -For sessions that exceed the context: -``` -Phase 1: Build structured digest from turn-pair nodes + compaction summaries + message metadata -Phase 2: If digest > context budget, split into overlapping segments -Phase 3: Map — summarize each segment with cheap model -Phase 4: Reduce — combine segment summaries + aggregated stats → final analysis with mid model -``` - -The structured digest format (not truncation): - -```markdown -## Session: project-name, 2026-05-29, 47 min, 12 pairs - -### Compaction Summary (verbatim from session) -The user was working on auth module refactoring. They had several corrections about function names... - -### Per-Pair Summary (from turn-pair-core nodes) -| # | Time | Sentiment | Friction | Correction | Tools | -|---|-------|-----------|----------|------------|-------| -| 1 | 14:02 | neutral | none | — | read | -| 2 | 14:08 | frustrated | wrong_approach | "wrong function" | read, edit | -... - -### Key Events (post-compaction messages, full detail) -[14:23] USER: "actually, I said use pnpm not npm" -[14:24] AGENT: reads package.json (2KB), runs pnpm install - -### Statistics (deterministic, from turn-pair aggregation) -- Total pairs: 12, friction pairs: 3, correction rate: 0.25 -- Tool failures: 2 (edit mismatch, bash exit 1) -- Tool waste: 45KB total (2 reads never referenced) -``` - -### 8.4 Properties produced - -```typescript -interface SessionOverviewProperties { - // Aggregated deterministic stats - total_pairs: number; - friction_pairs: number; - correction_count: number; - avg_quality_score: number | null; - dominant_friction_type: string | null; - tool_failure_rate: number; - total_tool_waste_bytes: number; - session_duration_seconds: number | null; - - // LLM-produced - session_summary: string; - key_friction_points: Array<{ - description: string; - pair_node_id: string; - severity: 'low' | 'medium' | 'high'; - }>; - improvement_proposals: Array<{ - target_type: string; - target_path: string; - title: string; - summary: string; - detail: string; - evidence: string; - confidence: number; - severity: string; - }>; - sentiment_arc: Array<{ - segment: number; - sentiment: string; - key_event: string; - }>; -} -``` - -### 8.5 Plan logic - -One unit per session, sourcing all dependency nodes: - -```typescript -plan(ctx: AnalyzerPlanContext): AnalysisUnit[] { - const pairNodes = ctx.dependencyNodes['turn-pair-core']; - const llmNodes = ctx.dependencyNodes['turn-pair-llm']; - if (pairNodes.length === 0) return []; - - const sources: SourceRef[] = [ - ...pairNodes.map(n => ({ kind: 'analysis_node' as const, id: n.id })), - ...llmNodes.map(n => ({ kind: 'analysis_node' as const, id: n.id })), - ]; - - return [{ - sources, - sourceSetHash: computeSourceSetHash(sources), - anchorKind: 'session', - anchorRef: ctx.sessionId, - }]; -} -``` - -### 8.6 Edges produced - -``` -anchors → session -consumes → all turn-pair-core and turn-pair-llm nodes -uses_prompt → map prompt (if map-reduce was used) -uses_prompt → reduce prompt -uses_config → config version -produces → proposal nodes (materialized into proposals table) -``` - ---- - -## 9. Proposal materialization - -After `session-overview` (or any future proposal-generating analyzer) produces a node with `node_kind = 'proposal'`: - -1. Framework extracts `improvement_proposals` from `content_json` -2. For each proposal: - a. Compute `dedup_key = SHA-256(target_type + target_path + severity + normalize(title))` - b. Check if an `open` proposal with this `dedup_key` exists - c. If yes → increment occurrence tracking, mark new proposal as `duplicate` - d. If no → INSERT into `proposals` and create analysis edges: - - `analysis_edges(from_node_id=proposal_node, to_ref_kind='analysis_node', to_ref_id=source_analysis_node, edge_kind='produces')` - - `analysis_edges(from_node_id=proposal_node, to_ref_kind='session', to_ref_id=session_id, edge_kind='anchors')` - - `analysis_edges(from_node_id=proposal_node, to_ref_kind='session', to_ref_id=session_id, edge_kind='anchors')` - ---- - -## 10. Model tiers - -```typescript -interface ModelTierConfig { - cheap: string; // e.g. 'anthropic/claude-haiku-3' or 'google/gemini-2.5-flash' - mid: string; // e.g. 'anthropic/claude-sonnet-4-5' - expensive: string; // e.g. 'anthropic/claude-opus-4' (rarely used) -} -``` - -Configured in `~/.pi/agent/prospector.json`. Analyzers request tiers, not specific models. - ---- - -## 11. Incremental run schedule - -| When | What | Cost | -|------|------|------| -| Every sync (~1 min) | Run `turn-pair-core` deterministic on new messages | Free | -| On demand (or daily) | Run `turn-pair-llm` on high-signal pairs | ~$0.01/session | -| On demand (or daily) | Run `session-overview` on sessions with new analysis | ~$0.05–0.15/session | -| On demand | Extract proposals from `session-overview` nodes | Free (DB query) | - ---- - -## 12. What NOT to build in v1 - -1. **Pi sub-agent execution engine** — Use in-process TypeScript analyzers with `pi-ai` calls. The `implementation_kind = 'pi_subagent'` field exists for future use. - -2. **Eager supersession of old versions** — Old nodes remain. Queries filter by `(analyzer_id, analyzer_version_id)` to see current results. A future `/prospect-gc` can optionally archive old-version nodes. - -3. **Per-model invalidation** — Model changes do NOT invalidate analysis. Model is metadata on `analysis_run`, not part of the recipe. - -4. **Complex dependency version resolution** — Dependencies resolve to "latest successful version" for MVP. - -5. **Cross-session meta-analyzer** — Focus on per-session analysis first. - -6. **Target file auto-discovery** — The first analyzers propose improvements targeting known categories. Scanning `~/.pi/` to discover all config targets is a future enhancement. - ---- - -## 13. Migration from existing schema - -The existing `proposals` table remains for now. Add: - -```sql -ALTER TABLE proposals ADD COLUMN source_node_id TEXT REFERENCES analysis_nodes(id); -``` - -New analysis-node-based proposals will populate both `analysis_nodes` (with `node_kind = 'proposal'`) and `proposals`. The `/prospect-proposals` command reads from both during transition. - -Eventually: -- `proposals` table → read-only for old records -- `analysis_nodes WHERE node_kind = 'proposal'` → the new source -- The command merges results from both - ---- - -## 14. File structure - -``` -src/ -├── analyze/ -│ ├── framework.ts — AnalyzerFramework class: register, run, runAll -│ ├── types.ts — All TypeScript interfaces -│ ├── input-hash.ts — computeSourceSetHash, computeInputHash, computePromptBundleHash -│ ├── edge-kinds.ts — Edge kind constants and validation -│ ├── proposal-materializer.ts — Extract proposals from analysis nodes, dedup, insert -│ ├── model-tiers.ts — ModelTierConfig, resolveModelTier -│ ├── analyzers/ -│ │ ├── turn-pair-core/ -│ │ │ ├── index.ts — Analyzer implementation -│ │ │ ├── patterns.ts — Correction/frustration regex patterns -│ │ │ └── config.ts — Default config + friction scoring formula -│ │ ├── turn-pair-llm/ -│ │ │ ├── index.ts — LLM enrichment analyzer -│ │ │ ├── prompt.ts — Prompt template + structured output schema -│ │ │ └── config.ts -│ │ └── session-overview/ -│ │ ├── index.ts — Analyzer implementation -│ │ ├── digest.ts — Build structured session digest -│ │ ├── compress.ts — Map-reduce compression for large sessions -│ │ ├── prompt-map.ts — Map-phase prompt + schema -│ │ ├── prompt-reduce.ts — Reduce-phase prompt + schema -│ │ └── config.ts -├── db/ -│ ├── schema.ts — Existing + new tables (migration 002+) -│ ├── queries.ts — Existing + new query functions -│ └── analysis-queries.ts — Queries for analysis_nodes, edges, runs, progress -├── commands/ -│ ├── sync.ts — Existing (updated to trigger analyzers) -│ ├── analyze.ts — Updated: now uses framework -│ ├── proposals.ts — Updated: reads from proposals table + analysis_nodes -│ ├── stats.ts — Existing -│ └── tool.ts — Existing -``` \ No newline at end of file diff --git a/src/analyze/analyzers/session-overview/config.ts b/src/analyze/analyzers/session-overview/config.ts new file mode 100644 index 0000000..0631d11 --- /dev/null +++ b/src/analyze/analyzers/session-overview/config.ts @@ -0,0 +1,28 @@ +/** Configuration for the session-overview analyzer. */ + +import { Type, type Static } from "typebox"; + +export const SessionOverviewConfig = Type.Object({ + /** Model tier for the map phase (segment summaries). */ + mapTier: Type.Union([Type.Literal("cheap"), Type.Literal("mid"), Type.Literal("expensive")]), + /** Model tier for the reduce phase (summary + proposals). */ + reduceTier: Type.Union([Type.Literal("cheap"), Type.Literal("mid"), Type.Literal("expensive")]), + /** Sampling temperature. */ + temperature: Type.Number(), + /** Digest size (chars) above which map-reduce kicks in. */ + mapReduceOverChars: Type.Number(), + /** Target segment size (chars) for the map phase. */ + segmentChars: Type.Number(), + /** Hard cap on segments processed per session. */ + maxSegments: Type.Number(), +}); +export type SessionOverviewConfig = Static; + +export const DEFAULT_SESSION_OVERVIEW_CONFIG: SessionOverviewConfig = { + mapTier: "cheap", + reduceTier: "mid", + temperature: 0, + mapReduceOverChars: 12000, + segmentChars: 6000, + maxSegments: 12, +}; diff --git a/src/analyze/analyzers/session-overview/digest.ts b/src/analyze/analyzers/session-overview/digest.ts new file mode 100644 index 0000000..2e8a933 --- /dev/null +++ b/src/analyze/analyzers/session-overview/digest.ts @@ -0,0 +1,163 @@ +/** + * Structured session digest for the session-overview analyzer. + * + * Rather than truncating raw transcript, we build a compact digest from the + * deterministic per-pair metrics, the LLM classifications, any compaction + * summaries, and aggregate statistics. Large sessions are split into segments + * for a map-reduce summarisation. + */ + +import type { AnalysisNodeRow, MessageRow } from "../../types.js"; +import type { TurnPairCoreProperties } from "../turn-pair-core/index.js"; +import type { TurnPairLLMProperties } from "../turn-pair-llm/prompt.js"; + +export interface DigestSegment { + index: number; + text: string; +} + +export interface SessionDigest { + header: string; + perPairLines: string[]; + text: string; + totalChars: number; + pairCount: number; + frictionCount: number; + compactionCount: number; + correctionCount: number; + toolFailureCount: number; +} + +export interface BuildDigestInput { + sessionId: string; + messages: MessageRow[]; + coreNodes: AnalysisNodeRow[]; + llmNodes: AnalysisNodeRow[]; +} + +function safeParse(json: string): T | null { + try { + return JSON.parse(json) as T; + } catch { + return null; + } +} + +/** Max length for a user-text snippet included in the per-pair digest line. */ +const USER_TEXT_SNIPPET_MAX = 200; + +export function buildDigest(input: BuildDigestInput): SessionDigest { + const core = input.coreNodes + .map((n) => safeParse(n.content_json)) + .filter((p): p is TurnPairCoreProperties => p !== null) + .sort((a, b) => a.pair_index - b.pair_index); + + // Map user_message_id → llm classification. turn-pair-llm records the anchor + // user-message id in its content, so we merge enrichment by id (not by order). + const llmByUser = new Map(); + for (const node of input.llmNodes) { + const props = safeParse(node.content_json); + if (props && props.user_message_id) llmByUser.set(props.user_message_id, props); + } + + // Map message id → user text, so every pair can include a verbatim snippet + // (not just pairs where the regex matched). This un-gates the synthesizer + // from the deterministic correction regex: the regex is a ranking signal only. + const userTextById = new Map(); + for (const m of input.messages) { + if (m.role === "user" && m.content_text) { + userTextById.set(m.id, m.content_text); + } + } + + const compactions = input.messages + .filter((m) => m.role === "compactionSummary" || m.role === "branch_summary") + .map((m) => (m.content_text ?? "").trim()) + .filter((t) => t.length > 0); + + const frictionCount = core.filter((p) => p.high_signal).length; + const correctionCount = core.filter((p) => p.correction_detected).length; + const toolFailureCount = core.reduce((sum, p) => sum + p.tool_failure_count, 0); + + const perPairLines = core.map((p) => { + const llm = llmByUser.get(p.user_message_id); + const bits = [ + `#${p.pair_index}`, + `friction=${p.friction_score.toFixed(2)}`, + p.correction_detected ? `correction=${p.correction_type}` : "correction=none", + `tool_fail=${p.tool_failure_count}`, + ]; + if (llm) bits.push(`sentiment=${llm.sentiment}`, `type=${llm.friction_type}`, `sev=${llm.severity}`); + if (p.correction_text) bits.push(`note="${p.correction_text.slice(0, 120)}"`); + // Un-gate: include a user-text snippet for every pair, not just regex-matched ones. + // The correction regex is a ranking signal only; the synthesizer must see all text. + const userText = userTextById.get(p.user_message_id); + if (userText) { + bits.push(`text="${truncateLine(userText, USER_TEXT_SNIPPET_MAX)}"`); + } + return bits.join(" "); + }); + + const headerLines = [ + `## Session ${input.sessionId}`, + `pairs=${core.length} high_signal=${frictionCount} corrections=${correctionCount} tool_failures=${toolFailureCount}`, + ]; + if (compactions.length > 0) { + headerLines.push("", "### Compaction summaries (verbatim)"); + for (const c of compactions) headerLines.push(c.slice(0, 2000)); + } + const header = headerLines.join("\n"); + + const text = [header, "", "### Per-pair signals", ...perPairLines].join("\n"); + + return { + header, + perPairLines, + text, + totalChars: text.length, + pairCount: core.length, + frictionCount, + compactionCount: compactions.length, + correctionCount, + toolFailureCount, + }; +} + +/** Truncate a line to maxLen characters, replacing newlines with spaces. */ +function truncateLine(s: string, maxLen: number): string { + const flat = s.replace(/\n/g, " "); + return flat.length > maxLen ? `${flat.slice(0, maxLen)}…` : flat; +} + +/** + * Split a digest's per-pair body into segments no larger than `segmentChars`, + * each prefixed with the shared header. Returns at least one segment. + */ +export function splitDigest(digest: SessionDigest, segmentChars: number): DigestSegment[] { + if (digest.totalChars <= segmentChars || digest.perPairLines.length === 0) { + return [{ index: 0, text: digest.text }]; + } + + const segments: DigestSegment[] = []; + let buffer: string[] = []; + let bufferLen = digest.header.length; + + const flush = (): void => { + if (buffer.length === 0) return; + segments.push({ + index: segments.length, + text: [digest.header, "", "### Per-pair signals", ...buffer].join("\n"), + }); + buffer = []; + bufferLen = digest.header.length; + }; + + for (const line of digest.perPairLines) { + if (bufferLen + line.length > segmentChars && buffer.length > 0) flush(); + buffer.push(line); + bufferLen += line.length + 1; + } + flush(); + + return segments; +} diff --git a/src/analyze/analyzers/session-overview/index.ts b/src/analyze/analyzers/session-overview/index.ts new file mode 100644 index 0000000..84021ec --- /dev/null +++ b/src/analyze/analyzers/session-overview/index.ts @@ -0,0 +1,189 @@ +/** + * session-overview — one summary node per session, producing improvement + * proposals. Depends on turn-pair-core and turn-pair-llm. + * + * Strategy: build a structured digest. If it fits the budget, a single reduce + * call produces the summary and proposals. Otherwise the digest is split into + * segments, each summarised by a cheap model (map), then a mid model combines + * the segment summaries plus aggregate stats into the final result (reduce). + */ + +import type { + Analyzer, + AnalyzerDef, + AnalyzerPlanContext, + AnalyzerRunContext, + AnalyzerVersion, + AnalysisResult, + AnalysisUnit, + PromptVersion, + SourceRef, +} from "../../types.js"; +import { computeSourceSetHash, computeConfigHash } from "../../input-hash.js"; +import { resolveModelSpec } from "../../model-tiers.js"; +import { EDGE_KINDS, REF_KINDS } from "../../edge-kinds.js"; +import { extractJsonObject } from "../turn-pair-llm/prompt.js"; +import { TURN_PAIR_CORE_DEF } from "../turn-pair-core/index.js"; +import { TURN_PAIR_LLM_DEF } from "../turn-pair-llm/index.js"; +import { buildDigest, splitDigest } from "./digest.js"; +import { MAP_PROMPT, MAP_PROMPT_HASH, buildMapPrompt, parseMapResponse, type MapSummary } from "./prompt-map.js"; +import { + REDUCE_PROMPT, + REDUCE_PROMPT_HASH, + buildReducePrompt, + parseReduceResponse, + type SessionOverviewProperties, +} from "./prompt-reduce.js"; +import { DEFAULT_SESSION_OVERVIEW_CONFIG, type SessionOverviewConfig } from "./config.js"; + +export const SESSION_OVERVIEW_DEF: AnalyzerDef = { + id: "session-overview", + label: "Session Analysis & Proposals", + description: "Summarises a session and proposes improvements. Consumes turn-pair-core and turn-pair-llm nodes.", + anchorSpan: "full_session", + dependencies: [TURN_PAIR_CORE_DEF.id, TURN_PAIR_LLM_DEF.id], +}; + +export const SESSION_OVERVIEW_VERSION: AnalyzerVersion = { + analyzerId: SESSION_OVERVIEW_DEF.id, + major: 1, + minor: 1, + implementationKind: "in_process_llm", + codeRef: "src/analyze/analyzers/session-overview/index.ts", +}; + +const PROMPTS: Record = { + map: { hash: MAP_PROMPT_HASH, content: MAP_PROMPT, role: "map" }, + reduce: { hash: REDUCE_PROMPT_HASH, content: REDUCE_PROMPT, role: "reduce" }, +}; + +export const sessionOverviewAnalyzer: Analyzer = { + def: SESSION_OVERVIEW_DEF, + version: SESSION_OVERVIEW_VERSION, + prompts: PROMPTS, + defaultConfig: { + id: "", + analyzerId: SESSION_OVERVIEW_DEF.id, + configHash: computeConfigHash(DEFAULT_SESSION_OVERVIEW_CONFIG), + configJson: DEFAULT_SESSION_OVERVIEW_CONFIG as unknown as Record, + label: "default", + }, + + modelsForIdentity(config, modelTiers): string[] { + const cfg = (config as unknown as SessionOverviewConfig) ?? DEFAULT_SESSION_OVERVIEW_CONFIG; + return [resolveModelSpec(cfg.mapTier, modelTiers), resolveModelSpec(cfg.reduceTier, modelTiers)]; + }, + + plan(ctx: AnalyzerPlanContext): AnalysisUnit[] { + const core = (ctx.dependencyNodes[TURN_PAIR_CORE_DEF.id] ?? []).slice().sort((a, b) => a.id.localeCompare(b.id)); + if (core.length === 0) return []; + const llm = (ctx.dependencyNodes[TURN_PAIR_LLM_DEF.id] ?? []).slice().sort((a, b) => a.id.localeCompare(b.id)); + + const sources: SourceRef[] = [ + ...core.map((n) => ({ kind: "analysis_node" as const, id: n.output_key })), + ...llm.map((n) => ({ kind: "analysis_node" as const, id: n.output_key })), + ]; + + return [ + { + sources, + sourceSetHash: computeSourceSetHash(sources), + anchorKind: "session", + anchorRef: ctx.sessionId, + }, + ]; + }, + + async analyze(unit: AnalysisUnit, ctx: AnalyzerRunContext): Promise { + const config = (ctx.config.configJson as unknown as SessionOverviewConfig) ?? DEFAULT_SESSION_OVERVIEW_CONFIG; + const coreNodes = ctx.getDependencyNodes(TURN_PAIR_CORE_DEF.id); + const llmNodes = ctx.getDependencyNodes(TURN_PAIR_LLM_DEF.id); + const messages = ctx.getSessionMessages(ctx.sessionId); + + const digest = buildDigest({ sessionId: ctx.sessionId, messages, coreNodes, llmNodes }); + const statsText = JSON.stringify( + { + pairs: digest.pairCount, + high_signal: digest.frictionCount, + corrections: digest.correctionCount, + tool_failures: digest.toolFailureCount, + compactions: digest.compactionCount, + }, + null, + 2, + ); + + let costUsd = 0; + let tokensUsed = 0; + let modelUsed: string | undefined; + const usedPromptHashes: string[] = [REDUCE_PROMPT_HASH]; + + let reduceInput: string; + if (digest.totalChars > config.mapReduceOverChars) { + const segments = splitDigest(digest, config.segmentChars).slice(0, config.maxSegments); + const summaries: MapSummary[] = []; + for (const seg of segments) { + const res = await ctx.llm({ + model: resolveModelSpec(config.mapTier, ctx.modelTiers), + system: ctx.prompts["map"] ?? MAP_PROMPT, + user: buildMapPrompt(seg.text), + temperature: config.temperature, + maxTokens: 800, + }); + costUsd += res.costUsd; + tokensUsed += res.tokensUsed; + modelUsed = res.model; + summaries.push(parseMapResponse(res.text, extractJsonObject)); + } + reduceInput = JSON.stringify( + summaries.map((s, i) => ({ segment: i, summary: s.segment_summary, notable: s.notable_points })), + null, + 2, + ); + usedPromptHashes.unshift(MAP_PROMPT_HASH); + } else { + reduceInput = digest.text; + } + + const reduceRes = await ctx.llm({ + model: resolveModelSpec(config.reduceTier, ctx.modelTiers), + system: ctx.prompts["reduce"] ?? REDUCE_PROMPT, + user: buildReducePrompt({ digestOrSummaries: reduceInput, stats: statsText }), + temperature: config.temperature, + maxTokens: 2000, + }); + costUsd += reduceRes.costUsd; + tokensUsed += reduceRes.tokensUsed; + modelUsed = reduceRes.model; + + const properties: SessionOverviewProperties = parseReduceResponse(reduceRes.text, extractJsonObject); + properties.stats = { + pairs: digest.pairCount, + high_signal: digest.frictionCount, + corrections: digest.correctionCount, + tool_failures: digest.toolFailureCount, + }; + + const edges: AnalysisResult["edges"] = [ + { toRefKind: REF_KINDS.SESSION, toRefId: ctx.sessionId, edgeKind: EDGE_KINDS.ANCHORS, ordinal: 0 }, + ]; + let ordinal = 1; + for (const n of [...coreNodes, ...llmNodes]) { + edges.push({ toRefKind: REF_KINDS.ANALYSIS_NODE, toRefId: n.id, edgeKind: EDGE_KINDS.CONSUMES, ordinal: ordinal++ }); + } + for (const h of usedPromptHashes) { + edges.push({ toRefKind: REF_KINDS.PROMPT_VERSION, toRefId: h, edgeKind: EDGE_KINDS.USES_PROMPT, ordinal: ordinal++ }); + } + + return { + nodeKind: "summary", + contentJson: properties as unknown as Record, + anchorKind: "session", + anchorRef: ctx.sessionId, + modelUsed, + costUsd, + tokensUsed, + edges, + }; + }, +}; diff --git a/src/analyze/analyzers/session-overview/prompt-map.ts b/src/analyze/analyzers/session-overview/prompt-map.ts new file mode 100644 index 0000000..f98f20c --- /dev/null +++ b/src/analyze/analyzers/session-overview/prompt-map.ts @@ -0,0 +1,35 @@ +/** + * Map-phase prompt: summarise one digest segment of a large session. + */ + +import { shortHash } from "../../input-hash.js"; + +export const MAP_PROMPT = `You summarise one segment of a coding-agent session's friction signals. +Return ONLY a JSON object: +{ + "segment_summary": "2-4 sentences on what happened and any friction", + "notable_points": ["short bullet", "..."] +} +Be concise and factual. Do not invent details beyond the signals provided.`; + +export const MAP_PROMPT_HASH = shortHash(MAP_PROMPT); + +export interface MapSummary { + segment_summary: string; + notable_points: string[]; +} + +export function buildMapPrompt(segmentText: string): string { + return `SESSION SEGMENT SIGNALS:\n${segmentText}`; +} + +export function parseMapResponse(text: string, extractJsonObject: (t: string) => Record): MapSummary { + const obj = extractJsonObject(text); + const notable = Array.isArray(obj["notable_points"]) + ? (obj["notable_points"] as unknown[]).filter((x): x is string => typeof x === "string") + : []; + return { + segment_summary: typeof obj["segment_summary"] === "string" ? (obj["segment_summary"] as string) : "", + notable_points: notable, + }; +} diff --git a/src/analyze/analyzers/session-overview/prompt-reduce.ts b/src/analyze/analyzers/session-overview/prompt-reduce.ts new file mode 100644 index 0000000..413b182 --- /dev/null +++ b/src/analyze/analyzers/session-overview/prompt-reduce.ts @@ -0,0 +1,89 @@ +/** + * Reduce-phase prompt: turn a session digest (or merged segment summaries plus + * aggregate stats) into a session summary and a set of improvement proposals. + */ + +import { shortHash } from "../../input-hash.js"; + +export const REDUCE_PROMPT = `You analyse a coding-agent session and propose concrete improvements to the +user's configuration, prompts, skills, or workflow. You do NOT make changes. + +Return ONLY a JSON object with exactly these fields: +{ + "session_summary": "3-5 sentences summarising the session and its friction", + "key_friction_points": [ + { "description": "what went wrong", "severity": "low" | "medium" | "high" } + ], + "improvement_proposals": [ + { + "target_type": "agents_md" | "skill" | "prompt" | "config" | "workflow" | "general", + "target_path": "optional path or section, e.g. AGENTS.md § Tooling", + "title": "short imperative title", + "summary": "one sentence", + "detail": "2-4 sentences with the concrete change to make", + "evidence": "what in the session motivates this", + "confidence": 0.0, + "severity": "friction" | "correction" | "waste" | "suggestion" + } + ] +} + +Only propose changes that the evidence supports. If the session was smooth, +return an empty "improvement_proposals" array. Prefer a few high-quality +proposals over many speculative ones.`; + +export const REDUCE_PROMPT_HASH = shortHash(REDUCE_PROMPT); + +export interface KeyFrictionPoint { + description: string; + severity: string; +} + +export interface SessionOverviewProperties { + session_summary: string; + key_friction_points: KeyFrictionPoint[]; + improvement_proposals: Array>; + stats?: Record; +} + +export function buildReducePrompt(params: { digestOrSummaries: string; stats: string }): string { + return [ + "AGGREGATE STATS:", + params.stats, + "", + "SESSION SIGNALS / SEGMENT SUMMARIES:", + params.digestOrSummaries, + ].join("\n"); +} + +export function parseReduceResponse( + text: string, + extractJsonObject: (t: string) => Record, +): SessionOverviewProperties { + const obj = extractJsonObject(text); + const friction = Array.isArray(obj["key_friction_points"]) + ? (obj["key_friction_points"] as unknown[]) + .map((x) => normalizeFriction(x)) + .filter((x): x is KeyFrictionPoint => x !== null) + : []; + const proposals = Array.isArray(obj["improvement_proposals"]) + ? (obj["improvement_proposals"] as unknown[]).filter( + (x): x is Record => x !== null && typeof x === "object", + ) + : []; + return { + session_summary: typeof obj["session_summary"] === "string" ? (obj["session_summary"] as string) : "", + key_friction_points: friction, + improvement_proposals: proposals, + }; +} + +function normalizeFriction(value: unknown): KeyFrictionPoint | null { + if (!value || typeof value !== "object") return null; + const v = value as Record; + if (typeof v["description"] !== "string") return null; + return { + description: v["description"] as string, + severity: typeof v["severity"] === "string" ? (v["severity"] as string) : "low", + }; +} diff --git a/src/analyze/analyzers/turn-pair-core/build.ts b/src/analyze/analyzers/turn-pair-core/build.ts new file mode 100644 index 0000000..6679f15 --- /dev/null +++ b/src/analyze/analyzers/turn-pair-core/build.ts @@ -0,0 +1,172 @@ +/** + * Turn construction from a session's message stream. + * + * A *turn* begins at a turn-starting entry and spans everything the agent does + * in response — assistant text, thinking, tool calls, and tool results — up to + * (but excluding) the next turn-starting entry. Following the host platform's + * own turn boundaries, a turn starts at a user message, a bash execution + * (`bashExecution`), or a branch/custom summary (`branch_summary` / + * `custom_message`). Context-management entries (compaction summaries) are not + * turn starts and are not part of any turn. + * + * The fields named `userMessageId` / `userText` hold the turn-starting message + * and its text; for ordinary turns that is the user's message, and for the + * non-user turn starts above it is that entry. + */ + +import type { MessageRow } from "../../types.js"; + +/** + * Roles/entry-kinds that begin a new turn. Mirrors the host platform's turn + * boundary definition (`user` and `bashExecution` messages, plus `branch_summary` + * and `custom_message` entries). Compaction summaries are deliberately excluded. + */ +const TURN_START_ROLES = new Set(["user", "bashExecution", "branch_summary", "custom_message"]); + +export interface PairToolCall { + name: string; + /** Truncated tool-call arguments for classifier evidence (e.g. bash command, gh subcommand). */ + argumentsPreview: string; +} + +export interface PairToolResult { + toolName: string; + isError: boolean; + textLength: number; + /** First N characters of the tool result text (captured for error diagnostics). */ + errorHead: string | null; +} + +export interface TurnPair { + /** Index of the pair within the session, 0-based. */ + index: number; + /** The anchoring user message id. */ + userMessageId: string; + /** All message ids covered by this pair (user + responses). */ + messageIds: string[]; + userText: string; + assistantText: string; + thinkingText: string; + toolCalls: PairToolCall[]; + toolResults: PairToolResult[]; + /** The previous pair's user text, for repetition detection. */ + priorUserText: string | null; + timestamp: string | null; +} + +/** Max length for a tool-call arguments preview string. */ +const ARGS_PREVIEW_MAX = 300; + +/** Max length for an error head string captured from tool results. */ +const ERROR_HEAD_MAX = 300; + +/** Truncate a string to maxLen characters, appending an ellipsis if truncated. */ +function truncateWithEllipsis(s: string, maxLen: number): string { + return s.length > maxLen ? `${s.slice(0, maxLen)}…` : s; +} + +/** + * Extract a concise, human-readable arguments preview from a tool call. + * + * For `bash` calls, returns the command string. + * For other calls (e.g. `gh`, `git`), returns a compact representation of + * the arguments (subcommand + flags + key params). + */ +function formatArgsPreview(name: string, args: Record): string { + if (name === "bash" || name === "Shell") { + const command = typeof args["command"] === "string" ? args["command"] : ""; + return truncateWithEllipsis(command, ARGS_PREVIEW_MAX); + } + // For other tools, build a compact key=value summary. + const parts: string[] = []; + for (const [key, val] of Object.entries(args)) { + if (val === undefined || val === null) continue; + const valStr = typeof val === "string" ? val : JSON.stringify(val); + parts.push(`${key}=${truncateWithEllipsis(valStr, 80)}`); + } + return truncateWithEllipsis(parts.join(" "), ARGS_PREVIEW_MAX); +} + +function parseToolCalls(json: string | null): PairToolCall[] { + if (!json) return []; + try { + const arr = JSON.parse(json) as Array<{ name?: unknown; arguments?: unknown }>; + if (!Array.isArray(arr)) return []; + return arr.map((c) => { + const name = typeof c.name === "string" ? c.name : ""; + const args = (c.arguments && typeof c.arguments === "object" && c.arguments !== null) ? c.arguments as Record : {}; + return { name, argumentsPreview: formatArgsPreview(name, args) }; + }); + } catch { + return []; + } +} + +function parseToolResults(json: string | null, errorText: string | null): PairToolResult[] { + if (!json) return []; + try { + const arr = JSON.parse(json) as Array<{ toolName?: unknown; isError?: unknown; textLength?: unknown }>; + if (!Array.isArray(arr)) return []; + return arr.map((r) => { + const isError = Boolean(r.isError); + return { + toolName: typeof r.toolName === "string" ? r.toolName : "", + isError, + textLength: typeof r.textLength === "number" ? r.textLength : 0, + errorHead: isError && errorText ? truncateWithEllipsis(errorText.trim(), ERROR_HEAD_MAX) : null, + }; + }); + } catch { + return []; + } +} + +/** Build the ordered list of turn pairs for a session. */ +export function buildTurnPairs(messages: MessageRow[]): TurnPair[] { + const pairs: TurnPair[] = []; + let current: TurnPair | null = null; + let priorUserText: string | null = null; + + const flush = (): void => { + if (current) { + pairs.push(current); + priorUserText = current.userText; + current = null; + } + }; + + for (const m of messages) { + if (TURN_START_ROLES.has(m.role)) { + flush(); + current = { + index: pairs.length, + userMessageId: m.id, + messageIds: [m.id], + userText: m.content_text ?? "", + assistantText: "", + thinkingText: "", + toolCalls: [], + toolResults: [], + priorUserText, + timestamp: m.timestamp, + }; + continue; + } + + if (!current) continue; // pre-first-turn noise (e.g. a leading summary) + + if (m.role === "assistant") { + current.messageIds.push(m.id); + if (m.content_text) current.assistantText += (current.assistantText ? "\n" : "") + m.content_text; + if (m.content_thinking) current.thinkingText += (current.thinkingText ? "\n" : "") + m.content_thinking; + current.toolCalls.push(...parseToolCalls(m.tool_calls)); + } else if (m.role === "toolResult") { + current.messageIds.push(m.id); + const errorText = m.content_text ?? null; + current.toolResults.push(...parseToolResults(m.tool_results, errorText)); + } + } + + flush(); + return pairs; +} diff --git a/src/analyze/analyzers/turn-pair-core/config.ts b/src/analyze/analyzers/turn-pair-core/config.ts new file mode 100644 index 0000000..13cf8d8 --- /dev/null +++ b/src/analyze/analyzers/turn-pair-core/config.ts @@ -0,0 +1,35 @@ +/** + * Configuration for the turn-pair-core analyzer. + * + * The friction score is a weighted sum of deterministic signals, clamped to + * [0, 1]. Weights and thresholds are part of the config so a change produces a + * new config fingerprint (and, when a run includes the `config` reason, new node + * versions). + */ + +import { Type, type Static } from "typebox"; + +export const TurnPairCoreConfig = Type.Object({ + /** Weight applied when a correction is detected. */ + correctionWeight: Type.Number(), + /** Weight applied per failed tool result (capped). */ + toolFailureWeight: Type.Number(), + /** Weight applied when the agent produced no text and no tool calls. */ + emptyResponseWeight: Type.Number(), + /** Bytes of tool output above which we start counting "waste". */ + toolWasteByteThreshold: Type.Number(), + /** Weight applied when tool output exceeds the waste threshold. */ + toolWasteWeight: Type.Number(), + /** Friction score at or above which a pair is "high-signal" (for LLM enrichment). */ + highSignalThreshold: Type.Number(), +}); +export type TurnPairCoreConfig = Static; + +export const DEFAULT_TURN_PAIR_CORE_CONFIG: TurnPairCoreConfig = { + correctionWeight: 0.6, + toolFailureWeight: 0.25, + emptyResponseWeight: 0.3, + toolWasteByteThreshold: 20000, + toolWasteWeight: 0.15, + highSignalThreshold: 0.5, +}; diff --git a/src/analyze/analyzers/turn-pair-core/index.ts b/src/analyze/analyzers/turn-pair-core/index.ts new file mode 100644 index 0000000..0025e56 --- /dev/null +++ b/src/analyze/analyzers/turn-pair-core/index.ts @@ -0,0 +1,133 @@ +/** + * turn-pair-core — deterministic per-turn friction analysis. + * + * Produces one `metric` node per turn pair. No LLM is used: friction signals + * come from correction-pattern matching, tool failures, empty responses, and + * tool-output volume. The friction score gates which pairs the LLM enrichment + * analyzer (turn-pair-llm) bothers to look at. + */ + +import type { + Analyzer, + AnalyzerDef, + AnalyzerPlanContext, + AnalyzerRunContext, + AnalyzerVersion, + AnalysisResult, + AnalysisUnit, + PromptVersion, + SourceRef, +} from "../../types.js"; +import { computeSourceSetHash, computeConfigHash } from "../../input-hash.js"; +import { EDGE_KINDS, REF_KINDS } from "../../edge-kinds.js"; +import { buildTurnPairs, type TurnPair } from "./build.js"; +import { classifyCorrection, detectRepetition } from "./patterns.js"; +import { DEFAULT_TURN_PAIR_CORE_CONFIG, type TurnPairCoreConfig } from "./config.js"; + +export const TURN_PAIR_CORE_DEF: AnalyzerDef = { + id: "turn-pair-core", + label: "Per-Turn Friction (deterministic)", + description: "Detects corrections, tool failures, empty responses, and tool waste per turn pair. No LLM.", + anchorSpan: "pair", + dependencies: [], +}; + +export const TURN_PAIR_CORE_VERSION: AnalyzerVersion = { + analyzerId: TURN_PAIR_CORE_DEF.id, + major: 1, + minor: 0, + implementationKind: "deterministic", + codeRef: "src/analyze/analyzers/turn-pair-core/index.ts", +}; + +export interface TurnPairCoreProperties { + pair_index: number; + user_message_id: string; + correction_detected: boolean; + correction_type: string | null; + correction_patterns: string[]; + correction_text: string | null; + tool_call_count: number; + tool_failure_count: number; + tool_result_bytes: number; + tool_waste_bytes: number; + empty_response: boolean; + friction_score: number; + high_signal: boolean; +} + +function scorePair(pair: TurnPair, config: TurnPairCoreConfig): TurnPairCoreProperties { + const isRepetition = detectRepetition(pair.userText, pair.priorUserText); + const correction = classifyCorrection(pair.userText, isRepetition); + + const toolFailureCount = pair.toolResults.filter((r) => r.isError).length; + const toolResultBytes = pair.toolResults.reduce((sum, r) => sum + r.textLength, 0); + const toolWasteBytes = toolResultBytes > config.toolWasteByteThreshold ? toolResultBytes - config.toolWasteByteThreshold : 0; + const emptyResponse = pair.assistantText.trim().length === 0 && pair.toolCalls.length === 0; + + let score = 0; + if (correction.detected) score += config.correctionWeight; + score += Math.min(toolFailureCount, 3) * config.toolFailureWeight; + if (emptyResponse) score += config.emptyResponseWeight; + if (toolWasteBytes > 0) score += config.toolWasteWeight; + const frictionScore = Math.max(0, Math.min(1, score)); + + return { + pair_index: pair.index, + user_message_id: pair.userMessageId, + correction_detected: correction.detected, + correction_type: correction.type, + correction_patterns: correction.patterns, + correction_text: correction.correctionText, + tool_call_count: pair.toolCalls.length, + tool_failure_count: toolFailureCount, + tool_result_bytes: toolResultBytes, + tool_waste_bytes: toolWasteBytes, + empty_response: emptyResponse, + friction_score: frictionScore, + high_signal: frictionScore >= config.highSignalThreshold, + }; +} + +export const turnPairCoreAnalyzer: Analyzer = { + def: TURN_PAIR_CORE_DEF, + version: TURN_PAIR_CORE_VERSION, + prompts: {} as Record, + defaultConfig: { + id: "", + analyzerId: TURN_PAIR_CORE_DEF.id, + configHash: computeConfigHash(DEFAULT_TURN_PAIR_CORE_CONFIG), + configJson: DEFAULT_TURN_PAIR_CORE_CONFIG as unknown as Record, + label: "default", + }, + + plan(ctx: AnalyzerPlanContext): AnalysisUnit[] { + const pairs = buildTurnPairs(ctx.messages); + return pairs.map((pair) => { + const sources: SourceRef[] = pair.messageIds.map((id) => ({ kind: "message" as const, id })); + return { + sources, + sourceSetHash: computeSourceSetHash(sources), + anchorKind: "message" as const, + anchorRef: pair.userMessageId, + meta: { pair: pair as unknown as Record }, + }; + }); + }, + + analyze(unit: AnalysisUnit, ctx: AnalyzerRunContext): AnalysisResult { + const config = (ctx.config.configJson as unknown as TurnPairCoreConfig) ?? DEFAULT_TURN_PAIR_CORE_CONFIG; + const pair = unit.meta?.["pair"] as unknown as TurnPair; + const properties = scorePair(pair, config); + + return { + nodeKind: "metric", + contentJson: properties as unknown as Record, + anchorKind: "message", + anchorRef: unit.anchorRef, + edges: [ + { toRefKind: REF_KINDS.MESSAGE, toRefId: unit.anchorRef, edgeKind: EDGE_KINDS.ANCHORS, ordinal: 0 }, + ], + }; + }, +}; diff --git a/src/analyze/analyzers/turn-pair-core/patterns.ts b/src/analyze/analyzers/turn-pair-core/patterns.ts new file mode 100644 index 0000000..6cb05bd --- /dev/null +++ b/src/analyze/analyzers/turn-pair-core/patterns.ts @@ -0,0 +1,109 @@ +/** + * Deterministic correction / friction detection patterns. + * + * These regexes flag *candidate* corrections cheaply; the LLM enrichment pass + * (turn-pair-llm) filters false positives. Categories: + * - strong : clearly corrective ("no, use X", "that's wrong", "I said …") + * - weak : hedged or possibly corrective ("could you try …", "actually") + * - negation: a leading negative that flips intent ("no …", "not …") + * + * Repetition (the user re-asking) is detected separately via token overlap. + */ + +export type CorrectionType = "explicit" | "implicit" | "repetition"; + +export interface CorrectionResult { + detected: boolean; + type: CorrectionType | null; + patterns: string[]; + correctionText: string | null; +} + +const STRONG: RegExp[] = [ + /\bno[,.\s]+(use|do|don'?t|that'?s|that is|it'?s|it is|use the)\b/i, + /\bnot\s+(that|this|like that|like this|what i)\b/i, + /\bdon'?t\s+(do|use|run|edit|add|remove|change|try|create)\b/i, + /\bstop\s+(doing|using|running|trying)\b/i, + /\bthat'?s\s+(wrong|incorrect|not right|not what)\b/i, + /\bthat\s+is\s+(wrong|incorrect|not what)\b/i, + /\binstead\s+of\s+(that|this)\b/i, + /\bi\s+(said|told you|already said|already told|meant)\b/i, + /\bactually[,.\s]/i, + /\brevert\b/i, +]; + +const WEAK: RegExp[] = [ + /\bcould\s+you\s+(please\s+)?(try|use|do|change|switch)\b/i, + /\bmaybe\s+(we|you|try)\b/i, + /\bwhy\s+don'?t\s+you\b/i, + /\bprefer\s+to\s+(use|do)\b/i, + /\bplease\s+(use|do|try|don'?t)\b/i, + /\bshould\s+(use|do|be|have)\b/i, +]; + +const LEADING_NEGATION: RegExp[] = [ + /^\s*no\b/i, + /^\s*not\b/i, + /^\s*never\b/i, + /^\s*don'?t\b/i, + /^\s*nope\b/i, +]; + +/** Detect a correction in user text. Strong > weak > leading-negation. */ +export function classifyCorrection(text: string | null, isRepetition: boolean): CorrectionResult { + if (isRepetition) { + return { detected: true, type: "repetition", patterns: [], correctionText: text?.slice(0, 240) ?? null }; + } + if (!text) return { detected: false, type: null, patterns: [], correctionText: null }; + + const strong = matchAll(STRONG, text); + if (strong.length > 0) { + return { detected: true, type: "explicit", patterns: strong, correctionText: extractCorrectionText(text, strong[0]!) }; + } + const weak = matchAll(WEAK, text); + if (weak.length > 0) { + return { detected: true, type: "implicit", patterns: weak, correctionText: extractCorrectionText(text, weak[0]!) }; + } + const neg = matchAll(LEADING_NEGATION, text); + if (neg.length > 0) { + return { detected: true, type: "explicit", patterns: neg, correctionText: text.slice(0, 240) }; + } + return { detected: false, type: null, patterns: [], correctionText: null }; +} + +function matchAll(patterns: RegExp[], text: string): string[] { + const out: string[] = []; + for (const re of patterns) if (re.test(text)) out.push(re.source); + return out; +} + +/** Slice the corrective remainder after the first matched pattern. */ +export function extractCorrectionText(text: string, patternSource: string): string { + const re = new RegExp(patternSource, "i"); + const m = re.exec(text); + if (!m) return text.slice(0, 240); + const after = text.slice(m.index + m[0].length).trim(); + return (after || m[0]).slice(0, 240); +} + +/** + * Cheap repetition heuristic: a short message that shares >= 2 meaningful tokens + * with the previous user message is likely a re-ask of the same intent. + */ +export function detectRepetition(text: string | null, priorUserText: string | null): boolean { + if (!text || !priorUserText) return false; + if (text.length > 80) return false; + return sharedTokenCount(text, priorUserText) >= 2; +} + +function sharedTokenCount(a: string, b: string): number { + const at = tokenSet(a); + const bt = tokenSet(b); + let count = 0; + for (const t of at) if (bt.has(t)) count++; + return count; +} + +function tokenSet(s: string): Set { + return new Set(s.toLowerCase().split(/\W+/).filter((t) => t.length > 2)); +} diff --git a/src/analyze/analyzers/turn-pair-llm/config.ts b/src/analyze/analyzers/turn-pair-llm/config.ts new file mode 100644 index 0000000..8af27e9 --- /dev/null +++ b/src/analyze/analyzers/turn-pair-llm/config.ts @@ -0,0 +1,19 @@ +/** Configuration for the turn-pair-llm enrichment analyzer. */ + +import { Type, type Static } from "typebox"; + +export const TurnPairLLMConfig = Type.Object({ + /** Model tier used for classification. */ + tier: Type.Union([Type.Literal("cheap"), Type.Literal("mid"), Type.Literal("expensive")]), + /** Sampling temperature. */ + temperature: Type.Number(), + /** Max pairs to enrich per session per run (cost guard). */ + maxPairsPerSession: Type.Number(), +}); +export type TurnPairLLMConfig = Static; + +export const DEFAULT_TURN_PAIR_LLM_CONFIG: TurnPairLLMConfig = { + tier: "cheap", + temperature: 0, + maxPairsPerSession: 20, +}; diff --git a/src/analyze/analyzers/turn-pair-llm/index.ts b/src/analyze/analyzers/turn-pair-llm/index.ts new file mode 100644 index 0000000..3777248 --- /dev/null +++ b/src/analyze/analyzers/turn-pair-llm/index.ts @@ -0,0 +1,181 @@ +/** + * turn-pair-llm — cheap LLM enrichment of high-signal turn pairs. + * + * Depends on turn-pair-core. Only pairs that the deterministic pass flagged as + * `high_signal` are sent to the model, keeping cost bounded. Produces one + * `classification` node per enriched pair, consuming the core metric node. + */ + +import type { + Analyzer, + AnalyzerDef, + AnalyzerPlanContext, + AnalyzerRunContext, + AnalyzerVersion, + AnalysisResult, + AnalysisUnit, + PromptVersion, + SourceRef, +} from "../../types.js"; +import { computeSourceSetHash, computeConfigHash } from "../../input-hash.js"; +import { resolveModelSpec } from "../../model-tiers.js"; +import { EDGE_KINDS, REF_KINDS } from "../../edge-kinds.js"; +import { + buildTurnPairs, + type PairToolCall, + type PairToolResult, +} from "../turn-pair-core/build.js"; +import { TURN_PAIR_CORE_DEF, type TurnPairCoreProperties } from "../turn-pair-core/index.js"; +import { + CLASSIFY_PROMPT, + CLASSIFY_PROMPT_HASH, + buildClassifyPrompt, + parseClassifyResponse, + type TurnPairLLMProperties, + type ToolCallEvidence, + type ToolResultEvidence, +} from "./prompt.js"; +import { DEFAULT_TURN_PAIR_LLM_CONFIG, type TurnPairLLMConfig } from "./config.js"; + +export const TURN_PAIR_LLM_DEF: AnalyzerDef = { + id: "turn-pair-llm", + label: "Per-Turn Classification (LLM)", + description: "Classifies sentiment and friction for high-signal turn pairs using a cheap model.", + anchorSpan: "pair", + dependencies: [TURN_PAIR_CORE_DEF.id], +}; + +export const TURN_PAIR_LLM_VERSION: AnalyzerVersion = { + analyzerId: TURN_PAIR_LLM_DEF.id, + major: 1, + minor: 1, + implementationKind: "in_process_llm", + codeRef: "src/analyze/analyzers/turn-pair-llm/index.ts", +}; + +const PROMPTS: Record = { + classify: { hash: CLASSIFY_PROMPT_HASH, content: CLASSIFY_PROMPT, role: "classify" }, +}; + +interface EnrichMeta { + userText: string; + assistantText: string; + correctionText: string | null; + toolCalls: ToolCallEvidence[]; + toolResults: ToolResultEvidence[]; + coreNodeId: string; +} + +export const turnPairLLMAnalyzer: Analyzer = { + def: TURN_PAIR_LLM_DEF, + version: TURN_PAIR_LLM_VERSION, + prompts: PROMPTS, + defaultConfig: { + id: "", + analyzerId: TURN_PAIR_LLM_DEF.id, + configHash: computeConfigHash(DEFAULT_TURN_PAIR_LLM_CONFIG), + configJson: DEFAULT_TURN_PAIR_LLM_CONFIG as unknown as Record, + label: "default", + }, + + modelsForIdentity(config, modelTiers): string[] { + const cfg = (config as unknown as TurnPairLLMConfig) ?? DEFAULT_TURN_PAIR_LLM_CONFIG; + return [resolveModelSpec(cfg.tier, modelTiers)]; + }, + + plan(ctx: AnalyzerPlanContext): AnalysisUnit[] { + const coreNodes = ctx.dependencyNodes[TURN_PAIR_CORE_DEF.id] ?? []; + const pairs = buildTurnPairs(ctx.messages); + const pairByUserId = new Map(pairs.map((p) => [p.userMessageId, p])); + const config = (ctx.config as unknown as TurnPairLLMConfig) ?? DEFAULT_TURN_PAIR_LLM_CONFIG; + + // Collect every high-signal pair that still maps to a turn in the transcript. + const candidates: { node: typeof coreNodes[number]; props: TurnPairCoreProperties }[] = []; + for (const node of coreNodes) { + let props: TurnPairCoreProperties; + try { + props = JSON.parse(node.content_json) as TurnPairCoreProperties; + } catch { + continue; + } + if (!props.high_signal) continue; + if (!pairByUserId.has(props.user_message_id)) continue; + candidates.push({ node, props }); + } + + // Cost guard: enrich at most `maxPairsPerSession`, highest friction first + // (ties broken by pair order so selection is deterministic across runs). + candidates.sort((a, b) => b.props.friction_score - a.props.friction_score || a.props.pair_index - b.props.pair_index); + const cap = config.maxPairsPerSession; + const selected = Number.isFinite(cap) && cap >= 0 ? candidates.slice(0, cap) : candidates; + + const units: AnalysisUnit[] = []; + for (const { node, props } of selected) { + const pair = pairByUserId.get(props.user_message_id)!; + const sources: SourceRef[] = [{ kind: "analysis_node", id: node.output_key }]; + const meta: EnrichMeta = { + userText: pair.userText, + assistantText: pair.assistantText, + correctionText: props.correction_text, + toolCalls: pair.toolCalls.map((tc): ToolCallEvidence => ({ + name: tc.name, + argumentsPreview: tc.argumentsPreview, + })), + toolResults: pair.toolResults.map((tr): ToolResultEvidence => ({ + toolName: tr.toolName, + isError: tr.isError, + errorHead: tr.errorHead, + })), + coreNodeId: node.id, + }; + units.push({ + sources, + sourceSetHash: computeSourceSetHash(sources), + anchorKind: "message", + anchorRef: props.user_message_id, + meta: meta as unknown as Record, + }); + } + return units; + }, + + async analyze(unit: AnalysisUnit, ctx: AnalyzerRunContext): Promise { + const config = (ctx.config.configJson as unknown as TurnPairLLMConfig) ?? DEFAULT_TURN_PAIR_LLM_CONFIG; + const meta = unit.meta as unknown as EnrichMeta; + + const response = await ctx.llm({ + model: resolveModelSpec(config.tier, ctx.modelTiers), + system: ctx.prompts["classify"] ?? CLASSIFY_PROMPT, + user: buildClassifyPrompt({ + userText: meta.userText, + assistantText: meta.assistantText, + correctionText: meta.correctionText, + toolCalls: meta.toolCalls, + toolResults: meta.toolResults, + }), + temperature: config.temperature, + maxTokens: 500, + }); + + const properties: TurnPairLLMProperties = { + ...parseClassifyResponse(response.text), + user_message_id: unit.anchorRef, + }; + + return { + nodeKind: "classification", + contentJson: properties as unknown as Record, + anchorKind: "message", + anchorRef: unit.anchorRef, + modelUsed: response.model, + costUsd: response.costUsd, + tokensUsed: response.tokensUsed, + durationMs: response.durationMs, + edges: [ + { toRefKind: REF_KINDS.MESSAGE, toRefId: unit.anchorRef, edgeKind: EDGE_KINDS.ANCHORS, ordinal: 0 }, + { toRefKind: REF_KINDS.ANALYSIS_NODE, toRefId: meta.coreNodeId, edgeKind: EDGE_KINDS.CONSUMES, ordinal: 1 }, + { toRefKind: REF_KINDS.PROMPT_VERSION, toRefId: CLASSIFY_PROMPT_HASH, edgeKind: EDGE_KINDS.USES_PROMPT, ordinal: 2 }, + ], + }; + }, +}; diff --git a/src/analyze/analyzers/turn-pair-llm/prompt.ts b/src/analyze/analyzers/turn-pair-llm/prompt.ts new file mode 100644 index 0000000..42d7842 --- /dev/null +++ b/src/analyze/analyzers/turn-pair-llm/prompt.ts @@ -0,0 +1,160 @@ +/** + * Prompt and response parsing for turn-pair-llm. + * + * The model receives a single high-signal turn pair and returns a compact JSON + * classification. We keep the schema small and deterministic so cheap models + * can comply and so parsing is robust. + */ + +import { shortHash } from "../../input-hash.js"; + +export const CLASSIFY_PROMPT = `You classify a single turn in a coding-agent session. +A "turn" is one user message and the assistant's response to it. + +Return ONLY a JSON object, no prose, with exactly these fields: +{ + "sentiment": "positive" | "neutral" | "frustrated", + "friction_type": "none" | "wrong_approach" | "missed_instruction" | "tool_misuse" | "repetition" | "other", + "is_genuine_correction": boolean, + "severity": "low" | "medium" | "high", + "rationale": "one short sentence" +} + +Judge only what the text supports. If the user is simply continuing the task with +no friction, use sentiment "neutral", friction_type "none", is_genuine_correction false. + +When TOOL CALLS are shown, use them to ground your diagnosis: prefer friction_type +"tool_misuse" when the tool name, arguments, or error reveal the mechanism of the +failure (e.g. wrong flags, missing --repo, targeting the wrong resource).`; + +export const CLASSIFY_PROMPT_HASH = shortHash(CLASSIFY_PROMPT); + +export interface ToolCallEvidence { + /** Tool name (e.g. "bash", "gh", "git"). */ + name: string; + /** Truncated arguments preview (e.g. the bash command string, gh subcommand+flags). */ + argumentsPreview: string; +} + +export interface ToolResultEvidence { + /** Tool name that produced this result. */ + toolName: string; + /** Whether the tool result is an error. */ + isError: boolean; + /** First N characters of the error text, null if not an error. */ + errorHead: string | null; +} + +export interface ClassifyInput { + userText: string; + assistantText: string; + correctionText: string | null; + /** Tool calls made by the assistant in this turn. */ + toolCalls: ToolCallEvidence[]; + /** Tool results (including errors) in this turn. */ + toolResults: ToolResultEvidence[]; +} + +export function buildClassifyPrompt(input: ClassifyInput): string { + const sections: string[] = [ + "USER MESSAGE:", + truncate(input.userText, 1500), + "", + "ASSISTANT RESPONSE:", + truncate(input.assistantText, 1500), + ]; + + if (input.correctionText) { + sections.push("", `HEURISTIC CORRECTION HINT: ${truncate(input.correctionText, 300)}`); + } + + // Tool-call evidence: failing tool names, truncated arguments, and error heads. + if (input.toolCalls.length > 0 || input.toolResults.some((r) => r.isError)) { + const toolLines: string[] = []; + for (const tc of input.toolCalls) { + if (tc.argumentsPreview) { + toolLines.push(` ${tc.name}: ${truncate(tc.argumentsPreview, 200)}`); + } else { + toolLines.push(` ${tc.name}`); + } + } + for (const tr of input.toolResults) { + if (tr.isError) { + const errLine = tr.errorHead ? ` error="${truncate(tr.errorHead, 200)}"` : ""; + toolLines.push(` ${tr.toolName} (FAILED)${errLine}`); + } + } + if (toolLines.length > 0) { + sections.push("", "TOOL CALLS:", ...toolLines); + } + } + + return sections.join("\n"); +} + +/** The fields the model returns for a single turn. */ +export interface ClassifyResult { + sentiment: string; + friction_type: string; + is_genuine_correction: boolean; + severity: string; + rationale: string; +} + +/** + * The stored classification node content: the model's result plus the id of the + * user message whose turn it classifies. The anchor id comes from the planned + * unit, not the model, so the session-overview digest can merge LLM enrichment + * back onto the matching deterministic pair by `user_message_id`. + */ +export interface TurnPairLLMProperties extends ClassifyResult { + user_message_id: string; +} + +const VALID_SENTIMENT = new Set(["positive", "neutral", "frustrated"]); +const VALID_FRICTION = new Set(["none", "wrong_approach", "missed_instruction", "tool_misuse", "repetition", "other"]); +const VALID_SEVERITY = new Set(["low", "medium", "high"]); + +/** Parse the model's JSON, tolerating markdown fences and extra prose. */ +export function parseClassifyResponse(text: string): ClassifyResult { + const obj = extractJsonObject(text); + const sentiment = pickString(obj["sentiment"], VALID_SENTIMENT, "neutral"); + const frictionType = pickString(obj["friction_type"], VALID_FRICTION, "none"); + const severity = pickString(obj["severity"], VALID_SEVERITY, "low"); + return { + sentiment, + friction_type: frictionType, + is_genuine_correction: Boolean(obj["is_genuine_correction"]), + severity, + rationale: typeof obj["rationale"] === "string" ? (obj["rationale"] as string).slice(0, 300) : "", + }; +} + +function pickString(value: unknown, allowed: Set, fallback: string): string { + return typeof value === "string" && allowed.has(value) ? value : fallback; +} + +function truncate(s: string, max: number): string { + return s.length > max ? `${s.slice(0, max)}…` : s; +} + +/** Extract the first balanced JSON object from arbitrary model text. */ +export function extractJsonObject(text: string): Record { + const fenced = /```(?:json)?\s*([\s\S]*?)```/i.exec(text); + const candidate = fenced ? fenced[1]! : text; + const start = candidate.indexOf("{"); + if (start < 0) throw new Error("No JSON object found in LLM response"); + let depth = 0; + for (let i = start; i < candidate.length; i++) { + const ch = candidate[i]; + if (ch === "{") depth++; + else if (ch === "}") { + depth--; + if (depth === 0) { + const slice = candidate.slice(start, i + 1); + return JSON.parse(slice) as Record; + } + } + } + throw new Error("Unterminated JSON object in LLM response"); +} diff --git a/src/analyze/defaults.ts b/src/analyze/defaults.ts new file mode 100644 index 0000000..2041552 --- /dev/null +++ b/src/analyze/defaults.ts @@ -0,0 +1,17 @@ +/** + * Bundled analyzer registry. `registerDefaults(framework)` wires up the three + * built-in analyzers in dependency order. + */ + +import type { AnalyzerFramework } from "./framework.js"; +import { turnPairCoreAnalyzer } from "./analyzers/turn-pair-core/index.js"; +import { turnPairLLMAnalyzer } from "./analyzers/turn-pair-llm/index.js"; +import { sessionOverviewAnalyzer } from "./analyzers/session-overview/index.js"; + +export const DEFAULT_ANALYZER_IDS = ["turn-pair-core", "turn-pair-llm", "session-overview"] as const; + +export function registerDefaults(framework: AnalyzerFramework): void { + framework.register(turnPairCoreAnalyzer); + framework.register(turnPairLLMAnalyzer); + framework.register(sessionOverviewAnalyzer); +} diff --git a/src/analyze/edge-kinds.ts b/src/analyze/edge-kinds.ts new file mode 100644 index 0000000..4e80df3 --- /dev/null +++ b/src/analyze/edge-kinds.ts @@ -0,0 +1,81 @@ +/** + * Typed edge graph constants and validation. + * + * The analysis graph expresses every relationship as an explicit typed edge. + * There are no `parent_id` columns and no denormalised anchor columns on + * analysis nodes — `analysis_edges` is the single source of truth for graph + * relationships. + * + * Edge kinds + * anchors node → (session | message) where the analysis attaches + * consumes node → analysis_node inputs this node was built from + * uses_prompt node → prompt_version prompt that produced the node + * uses_config node → config_version config that produced the node + * produces node → proposal proposal materialised from node + * revises node → analysis_node version lineage: this node is a + * newer-version alternative of the + * target node (same logical unit) + */ + +export const REF_KINDS = { + SESSION: "session", + MESSAGE: "message", + ANALYSIS_NODE: "analysis_node", + PROMPT_VERSION: "prompt_version", + CONFIG_VERSION: "config_version", + PROPOSAL: "proposal", +} as const; + +export type RefKind = (typeof REF_KINDS)[keyof typeof REF_KINDS]; + +export const EDGE_KINDS = { + ANCHORS: "anchors", + CONSUMES: "consumes", + USES_PROMPT: "uses_prompt", + USES_CONFIG: "uses_config", + PRODUCES: "produces", + REVISES: "revises", +} as const; + +export type EdgeKind = (typeof EDGE_KINDS)[keyof typeof EDGE_KINDS]; + +const REF_KIND_SET = new Set(Object.values(REF_KINDS)); +const EDGE_KIND_SET = new Set(Object.values(EDGE_KINDS)); + +/** Which ref kinds are valid targets for each edge kind. */ +const VALID_TARGETS: Record> = { + [EDGE_KINDS.ANCHORS]: new Set([REF_KINDS.SESSION, REF_KINDS.MESSAGE]), + [EDGE_KINDS.CONSUMES]: new Set([REF_KINDS.ANALYSIS_NODE]), + [EDGE_KINDS.USES_PROMPT]: new Set([REF_KINDS.PROMPT_VERSION]), + [EDGE_KINDS.USES_CONFIG]: new Set([REF_KINDS.CONFIG_VERSION]), + [EDGE_KINDS.PRODUCES]: new Set([REF_KINDS.PROPOSAL]), + [EDGE_KINDS.REVISES]: new Set([REF_KINDS.ANALYSIS_NODE]), +}; + +export function isRefKind(value: string): value is RefKind { + return REF_KIND_SET.has(value); +} + +export function isEdgeKind(value: string): value is EdgeKind { + return EDGE_KIND_SET.has(value); +} + +/** + * Validate that `toRefKind` is an allowed target for `edgeKind`. + * Throws with a descriptive message on violation. + */ +export function validateEdge(edgeKind: string, toRefKind: string): void { + if (!isEdgeKind(edgeKind)) { + throw new Error(`Invalid edge_kind: ${edgeKind}`); + } + if (!isRefKind(toRefKind)) { + throw new Error(`Invalid to_ref_kind: ${toRefKind}`); + } + const allowed = VALID_TARGETS[edgeKind]; + if (!allowed.has(toRefKind)) { + throw new Error( + `Edge kind '${edgeKind}' cannot target ref kind '${toRefKind}'. ` + + `Allowed: ${[...allowed].join(", ")}.`, + ); + } +} diff --git a/src/analyze/framework.ts b/src/analyze/framework.ts new file mode 100644 index 0000000..d55af21 --- /dev/null +++ b/src/analyze/framework.ts @@ -0,0 +1,534 @@ +/** + * AnalyzerFramework — registers analyzers and runs them incrementally over + * session data, producing an append-only, versioned analysis graph. + * + * Incrementality model + * ───────────────────── + * Each analyzer's `plan()` enumerates the logical *units* of work for a session + * (e.g. one unit per turn-pair, or one unit per session). Each unit carries a + * `source_set_hash` identifying exactly which inputs it covers. + * + * `scan()` classifies every planned unit against the current graph state: + * - current — a node already exists for this exact recipe (analyzer version + + * config fingerprint + source set). Nothing to do. + * - stale — a node exists for this logical unit but under a different recipe. + * Staleness carries its *reasons*: `major`/`minor` (the analyzer + * version moved, graded by the author) and/or `config` (the user's + * setup changed, ungraded). + * - missing — no node exists for this logical unit at all. + * + * Revise reasons (a run's reach) + * - no reasons (default): only `missing` units are analysed; existing nodes are + * left untouched. + * - any of `major`/`minor`/`config`: a stale unit is also analysed when one of + * its reasons was requested (`minor` implies `major`). A recomputed unit + * produces a new node linked to its predecessor by a `revises` edge, so both + * versions coexist "at the same level" and the lineage is navigable. The + * reasons only *select* units; a selected unit is always recomputed to the + * current recipe in full — latest version, config, and resolved model. + * + * There is no crash-recovery bookkeeping. Idempotency is structural: a finished + * node is `current` (skipped on the next run); an unfinished unit is still + * `missing`/`stale` and is simply picked up again. Re-running after any failure + * converges with no special handling. + */ + +import type Database from "better-sqlite3"; +import type { + Analyzer, + AnalyzerConfig, + AnalyzerPlanContext, + AnalyzerRunContext, + AnalyzerRunResult, + AnalysisNodeRow, + AnalysisResult, + AnalysisUnit, + ClassifiedUnit, + LLMCaller, + MessageRow, + ModelTierConfig, + ReviseReason, + RunSummary, +} from "./types.js"; +import { + computeConfigFingerprint, + computeInputKey, + computeOutputKey, + computePromptBundleHash, + shortHash, + uuidv7, +} from "./input-hash.js"; +import { + expandReviseReasons, + gradeVersionMove, + parseVersionId, + reachLabel, + versionIdOf, +} from "./version.js"; +import { EDGE_KINDS, REF_KINDS, validateEdge } from "./edge-kinds.js"; +import { + createRun, + finishRun, + findLatestNodeBySourceSet, + findNodeByInputKey, + getAnchoredMessageIds, + getMessage, + getNode, + getNodesByAnalyzer, + getSessionNodes, + insertEdge, + insertNode, + registerPrompt, + resolveConfig, + upsertAnalyzerDef, + upsertAnalyzerVersion, +} from "../db/analysis-queries.js"; +import { materializeProposalsFromNode } from "./proposal-materializer.js"; + +export interface FrameworkDeps { + db: Database.Database; + llm: LLMCaller; + modelTiers: ModelTierConfig; +} + +interface ResolvedAnalyzer { + analyzer: Analyzer; + config: AnalyzerConfig; + promptBundleHash: string; + configFingerprint: string; +} + +export class AnalyzerFramework { + private readonly analyzers = new Map(); + + constructor(private readonly deps: FrameworkDeps) {} + + /** Register an analyzer and persist its def, version, prompts, and default config. */ + register(analyzer: Analyzer): void { + upsertAnalyzerDef(this.deps.db, analyzer.def); + upsertAnalyzerVersion(this.deps.db, analyzer.version); + for (const prompt of Object.values(analyzer.prompts)) { + registerPrompt(this.deps.db, prompt); + } + this.analyzers.set(analyzer.def.id, analyzer); + } + + get(id: string): Analyzer | undefined { + return this.analyzers.get(id); + } + + list(): Analyzer[] { + return [...this.analyzers.values()]; + } + + /** + * Classify the work for a session without performing any analysis. This is + * the efficient full-graph rescan: it runs each analyzer's `plan()` and + * compares every unit against existing nodes. Pure read; no side effects. + */ + async scan(sessionId: string, analyzerIds?: string[]): Promise { + const order = this.topologicalSort(analyzerIds); + const out: ClassifiedUnit[] = []; + for (const analyzerId of order) { + const resolved = this.resolve(analyzerId); + const planCtx = this.buildPlanContext(resolved.analyzer, resolved.config, sessionId); + const units = await resolved.analyzer.plan(planCtx); + for (const unit of units) { + out.push(this.classify(resolved, unit)); + } + } + return out; + } + + /** + * Run analysis for a session. With no revise reasons only missing units are + * produced; reasons (`major`/`minor`/`config`) additionally recompute matching + * stale units into new versions linked by `revises` edges. + */ + async run( + sessionId: string, + opts: { revise?: ReviseReason[]; analyzerIds?: string[]; modelSpec?: string } = {}, + ): Promise { + const revise = opts.revise ?? []; + const requested = expandReviseReasons(revise); + const order = this.topologicalSort(opts.analyzerIds); + + const summary: RunSummary = { + sessionId, + revise, + analyzerResults: [], + nodesProduced: 0, + nodesSkipped: 0, + nodesRevised: 0, + proposalsCreated: 0, + costUsd: 0, + tokensUsed: 0, + errors: [], + }; + + for (const analyzerId of order) { + const result = await this.runAnalyzer(analyzerId, sessionId, requested, opts.modelSpec, summary); + summary.analyzerResults.push(result); + } + + return summary; + } + + private async runAnalyzer( + analyzerId: string, + sessionId: string, + requested: ReadonlySet, + modelSpec: string | undefined, + summary: RunSummary, + ): Promise { + const resolved = this.resolve(analyzerId); + const { analyzer, config, promptBundleHash } = resolved; + + const planCtx = this.buildPlanContext(analyzer, config, sessionId); + const units = await analyzer.plan(planCtx); + + const classified = units.map((unit) => this.classify(resolved, unit)); + const todo = classified.filter( + (c) => c.status === "missing" || (c.status === "stale" && c.reasons.some((r) => requested.has(r))), + ); + + const runId = uuidv7(); + createRun(this.deps.db, { + id: runId, + analyzerId: analyzer.def.id, + analyzerVersionId: versionIdOf(analyzer.version), + configId: config.id, + sessionId, + mode: reachLabel(requested), + promptBundleHash, + modelSpec, + }); + + const result: AnalyzerRunResult = { + analyzerId: analyzer.def.id, + runId, + nodesProduced: 0, + nodesSkipped: classified.length - todo.length, + nodesRevised: 0, + costUsd: 0, + tokensUsed: 0, + status: "ok", + }; + + const runCtx = this.buildRunContext(analyzer, config, sessionId); + + for (const item of todo) { + try { + const analysis = await analyzer.analyze(item.unit, runCtx); + const created = this.persistNode(resolved, runId, sessionId, item, analysis); + result.nodesProduced++; + if (item.status === "stale" && item.priorNodeId) result.nodesRevised++; + result.costUsd += analysis.costUsd ?? 0; + result.tokensUsed += analysis.tokensUsed ?? 0; + summary.proposalsCreated += created.proposalsCreated; + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + result.status = "partial"; + summary.errors.push(`${analyzer.def.id}: ${message}`); + this.persistErrorNode(resolved, runId, sessionId, item, message); + } + } + + finishRun(this.deps.db, runId, { + status: result.status, + nodesProduced: result.nodesProduced, + nodesSkipped: result.nodesSkipped, + costUsd: result.costUsd, + tokensUsed: result.tokensUsed, + errorMessage: result.status === "partial" ? "one or more units failed" : null, + }); + + summary.nodesProduced += result.nodesProduced; + summary.nodesSkipped += result.nodesSkipped; + summary.nodesRevised += result.nodesRevised; + summary.costUsd += result.costUsd; + summary.tokensUsed += result.tokensUsed; + + return result; + } + + // ───────────────────────── classification ───────────────────────── + + private classify(resolved: ResolvedAnalyzer, unit: AnalysisUnit): ClassifiedUnit { + const { analyzer, configFingerprint } = resolved; + const inputKey = computeInputKey({ + analyzerId: analyzer.def.id, + analyzerVersionId: versionIdOf(analyzer.version), + configFingerprint, + sourceSetHash: unit.sourceSetHash, + }); + + // Error nodes carry a decoupled identity, so `findNodeByInputKey` matches + // only a successful result at this exact recipe, and `findLatestNodeBySourceSet` + // skips errors. A unit whose only history is failures therefore classifies as + // `missing` and is recomputed on the next scan that reaches it. + if (findNodeByInputKey(this.deps.db, inputKey)) { + return { analyzerId: analyzer.def.id, unit, status: "current", inputKey, reasons: [] }; + } + + const prior = findLatestNodeBySourceSet(this.deps.db, analyzer.def.id, unit.sourceSetHash); + if (prior) { + const reasons = this.gradeStale(resolved, prior); + return { analyzerId: analyzer.def.id, unit, status: "stale", inputKey, priorNodeId: prior.id, reasons }; + } + + return { analyzerId: analyzer.def.id, unit, status: "missing", inputKey, reasons: [] }; + } + + /** + * Why is an existing node out of date? At most one version reason + * (`major`/`minor`, graded by the author from the version move) plus `config` + * when the user's config fingerprint differs. A pure version downgrade yields + * no reason, so the newer node is left in place. + */ + private gradeStale(resolved: ResolvedAnalyzer, prior: AnalysisNodeRow): ReviseReason[] { + const reasons: ReviseReason[] = []; + const versionReason = gradeVersionMove(parseVersionId(prior.analyzer_version_id), { + major: resolved.analyzer.version.major, + minor: resolved.analyzer.version.minor, + }); + if (versionReason) reasons.push(versionReason); + if (prior.config_fingerprint !== resolved.configFingerprint) reasons.push("config"); + return reasons; + } + + // ───────────────────────── persistence ───────────────────────── + + private persistNode( + resolved: ResolvedAnalyzer, + runId: string, + sessionId: string, + item: ClassifiedUnit, + analysis: AnalysisResult, + ): { nodeId: string; proposalsCreated: number } { + const { analyzer, config } = resolved; + const nodeId = uuidv7(); + const now = new Date().toISOString(); + const contentJson = JSON.stringify(analysis.contentJson); + // Content-addressed result identity: a downstream consumer references this + // key in its source set, so a changed output propagates into the consumer's + // input_key. Reproducible from (input_key, content) on any machine. + const outputKey = computeOutputKey(item.inputKey, analysis.contentJson); + + insertNode(this.deps.db, { + id: nodeId, + sessionId, + analyzerId: analyzer.def.id, + analyzerVersionId: versionIdOf(analyzer.version), + configId: config.id, + runId, + nodeKind: analysis.nodeKind, + contentJson, + sourceSetHash: item.unit.sourceSetHash, + inputKey: item.inputKey, + outputKey, + configFingerprint: resolved.configFingerprint, + modelUsed: analysis.modelUsed ?? null, + costUsd: analysis.costUsd ?? null, + tokensUsed: analysis.tokensUsed ?? null, + durationMs: analysis.durationMs ?? null, + createdAt: now, + }); + + this.persistEdges(nodeId, config, analysis); + + // Version lineage: a re-analysed stale unit revises its predecessor. + if (item.status === "stale" && item.priorNodeId) { + insertEdge(this.deps.db, { + fromNodeId: nodeId, + toRefKind: REF_KINDS.ANALYSIS_NODE, + toRefId: item.priorNodeId, + edgeKind: EDGE_KINDS.REVISES, + ordinal: 0, + }); + } + + let proposalsCreated = 0; + if (analysis.nodeKind === "summary" || analysis.nodeKind === "proposal") { + proposalsCreated = materializeProposalsFromNode(this.deps.db, { + sessionId, + analyzerId: analyzer.def.id, + sourceNodeId: nodeId, + sourceOutputKey: outputKey, + contentJson: analysis.contentJson, + now, + }); + } + + return { nodeId, proposalsCreated }; + } + + private persistEdges(nodeId: string, config: AnalyzerConfig, analysis: AnalysisResult): void { + let ordinal = 0; + for (const edge of analysis.edges) { + validateEdge(edge.edgeKind, edge.toRefKind); + insertEdge(this.deps.db, { + fromNodeId: nodeId, + toRefKind: edge.toRefKind, + toRefId: edge.toRefId, + edgeKind: edge.edgeKind, + ordinal: edge.ordinal ?? ordinal, + }); + ordinal++; + } + + // Always record the config provenance edge. + insertEdge(this.deps.db, { + fromNodeId: nodeId, + toRefKind: REF_KINDS.CONFIG_VERSION, + toRefId: config.id, + edgeKind: EDGE_KINDS.USES_CONFIG, + ordinal: ordinal++, + }); + } + + private persistErrorNode( + resolved: ResolvedAnalyzer, + runId: string, + sessionId: string, + item: ClassifiedUnit, + message: string, + ): void { + const { analyzer, config } = resolved; + const nodeId = uuidv7(); + const now = new Date().toISOString(); + // An error node's identity is the recipe plus the failure's message and + // timestamp (with the node id as a uniqueness nonce), so it never occupies + // the recipe identity reserved for a successful result. The unit therefore + // stays `missing` and is recomputed on the next scan that reaches it — error + // nodes are an append-only record of failures, not a completion marker. + const errorInputKey = shortHash(`error(${item.inputKey}|${message}|${now}|${nodeId})`); + const content = { error: message, anchor: item.unit.anchorRef, timestamp: now }; + try { + insertNode(this.deps.db, { + id: nodeId, + sessionId, + analyzerId: analyzer.def.id, + analyzerVersionId: versionIdOf(analyzer.version), + configId: config.id, + runId, + nodeKind: "error", + contentJson: JSON.stringify(content), + sourceSetHash: item.unit.sourceSetHash, + inputKey: errorInputKey, + outputKey: computeOutputKey(errorInputKey, content), + configFingerprint: resolved.configFingerprint, + createdAt: now, + }); + } catch { + // Defensive: never let error-node persistence abort the run. + } + } + + // ───────────────────────── contexts ───────────────────────── + + private buildPlanContext(analyzer: Analyzer, config: AnalyzerConfig, sessionId: string): AnalyzerPlanContext { + const messages = this.loadMessages(sessionId); + const allNodes = getSessionNodes(this.deps.db, sessionId); + const ownNodes = allNodes.filter((n) => n.analyzer_id === analyzer.def.id); + const dependencyNodes: Record = {}; + for (const depId of analyzer.def.dependencies) { + dependencyNodes[depId] = allNodes.filter((n) => n.analyzer_id === depId); + } + return { sessionId, messages, allNodes, ownNodes, dependencyNodes, config: config.configJson, db: this.deps.db }; + } + + private buildRunContext(analyzer: Analyzer, config: AnalyzerConfig, sessionId: string): AnalyzerRunContext { + const db = this.deps.db; + const prompts: Record = {}; + for (const [name, p] of Object.entries(analyzer.prompts)) prompts[name] = p.content; + + return { + sessionId, + getMessage: (id) => getMessage(db, id), + getNode: (id) => getNode(db, id), + getDependencyNodes: (depId) => { + if (!analyzer.def.dependencies.includes(depId)) { + throw new Error( + `Analyzer '${analyzer.def.id}' read dependency '${depId}' without declaring it. ` + + `Add '${depId}' to def.dependencies.`, + ); + } + return getNodesByAnalyzer(db, depId, sessionId); + }, + getSessionMessages: (sid) => this.loadMessages(sid), + llm: this.deps.llm, + config, + prompts, + modelTiers: this.deps.modelTiers, + }; + } + + private loadMessages(sessionId: string): MessageRow[] { + return db_loadMessages(this.deps.db, sessionId); + } + + // ───────────────────────── helpers ───────────────────────── + + private resolve(analyzerId: string): ResolvedAnalyzer { + const analyzer = this.analyzers.get(analyzerId); + if (!analyzer) throw new Error(`Analyzer not registered: ${analyzerId}`); + const config = resolveConfig(this.deps.db, { + analyzerId: analyzer.def.id, + configJson: analyzer.defaultConfig.configJson, + label: analyzer.defaultConfig.label, + }); + const promptBundleHash = computePromptBundleHash(Object.values(analyzer.prompts).map((p) => p.hash)); + // Resolve tier shorthands to concrete models; the resolved model is part of + // the user's `config` identity (a model swap is an ungraded config change). + const models = analyzer.modelsForIdentity?.(config.configJson, this.deps.modelTiers) ?? []; + // Use the config's *content* hash (not its DB-local uuid row id) so the + // fingerprint — and therefore every input_key/output_key — is reproducible + // across databases and wipes. + const configFingerprint = computeConfigFingerprint(config.configHash, models); + return { analyzer, config, promptBundleHash, configFingerprint }; + } + + /** Dependency-respecting order of registered analyzers (Kahn-style DFS). */ + topologicalSort(analyzerIds?: string[]): string[] { + const targets = analyzerIds ?? [...this.analyzers.keys()]; + const visited = new Set(); + const visiting = new Set(); + const order: string[] = []; + + const visit = (id: string): void => { + if (visited.has(id)) return; + if (visiting.has(id)) throw new Error(`Dependency cycle detected at analyzer '${id}'`); + const analyzer = this.analyzers.get(id); + if (!analyzer) return; + visiting.add(id); + for (const dep of analyzer.def.dependencies) visit(dep); + visiting.delete(id); + visited.add(id); + order.push(id); + }; + + for (const id of targets) visit(id); + return order; + } + + /** Expose anchored-message lookup for analyzers that need raw turn content. */ + getAnchoredMessages(nodeId: string): MessageRow[] { + const ids = getAnchoredMessageIds(this.deps.db, nodeId); + const out: MessageRow[] = []; + for (const id of ids) { + const m = getMessage(this.deps.db, id); + if (m) out.push(m); + } + return out; + } +} + +function db_loadMessages(db: Database.Database, sessionId: string): MessageRow[] { + return db + .prepare( + "SELECT id, session_id, parent_id, timestamp, role, content_text, content_thinking, tool_calls, tool_results " + + "FROM messages WHERE session_id = ? ORDER BY rowid ASC", + ) + .all(sessionId) as MessageRow[]; +} diff --git a/src/analyze/input-hash.ts b/src/analyze/input-hash.ts new file mode 100644 index 0000000..6c3dd39 --- /dev/null +++ b/src/analyze/input-hash.ts @@ -0,0 +1,158 @@ +/** + * Content-addressed hashing and id helpers for the analyzer framework. + * + * Idempotency and reproducibility hinge on two content-addressed keys: + * - `input_key` is the *recipe* identity: analyzer + version + config + * fingerprint + source set. It folds in only *inputs* — never the LLM's + * output — so a node is uniquely identified by its input_key, and recomputing + * the same recipe over the same sources is a no-op. (`source_set_hash` + * identifies just the inputs; two analyses over the same sources share it.) + * - `output_key` = H(input_key | canonical(content)) is the content-addressed + * id of a *specific result*. A consumer references its upstream sources by + * their `output_key`, so a consumer's `input_key` transitively commits to + * every upstream output. The whole graph is therefore a Merkle DAG: identical + * inputs+outputs reproduce identical keys on any machine, after any wipe, and + * a stored key can be re-derived from content to verify it. + * + * The version dimension (same source_set_hash, different analyzer version or + * config fingerprint) yields a *different* input_key but the *same* + * source_set_hash — that is how alternative versions of the same logical unit + * are detected and linked via `revises` edges. + * + * The analyzer's *shipped* prompt is represented by its version, not by a + * separate identity axis; only the user's config (including a prompt override + * and the resolved model) feeds the config fingerprint. + */ + +import { createHash, randomUUID } from "node:crypto"; + +export interface SourceRefLike { + kind: string; + id: string; +} + +export interface InputHashParts { + analyzerId: string; + analyzerVersionId: string; + /** Fingerprint of the user-controlled config: parameters + resolved model(s). */ + configFingerprint: string; + sourceSetHash: string; +} + +function sha256hex(input: string): string { + return createHash("sha256").update(input).digest("hex"); +} + +/** Full 64-char SHA-256 hex digest. */ +export function fullHash(input: string): string { + return sha256hex(input); +} + +/** First 16 hex chars of the SHA-256 digest — compact but collision-safe enough. */ +export function shortHash(input: string): string { + return sha256hex(input).slice(0, 16); +} + +/** + * Deterministic hash over a set of source references. Order-independent: + * sources are sorted by `kind` then `id` before hashing. + */ +export function computeSourceSetHash(sources: readonly SourceRefLike[]): string { + const canonical = [...sources] + .map((s) => `${s.kind}:${s.id}`) + .sort() + .join("|"); + return shortHash(`sources(${canonical})`); +} + +/** + * Deterministic hash over a bundle of prompt hashes. Order-independent. Used for + * *run provenance* (which shipped prompts a run used); it is no longer part of + * node identity, since a shipped prompt is represented by the analyzer version. + */ +export function computePromptBundleHash(promptHashes: readonly string[]): string { + const canonical = [...promptHashes].sort().join("|"); + return shortHash(`prompts(${canonical})`); +} + +/** + * Fingerprint of everything the *user* controls for a node: the config's + * content identity (its canonical-JSON hash) plus the concrete models the + * analyzer resolved to (the tier→model mapping and any pin). Order-independent + * over models. This is the `config` dimension of identity — a change here marks + * nodes stale for the (ungraded) `config` reason. Using the config's *content* + * hash (not its DB-local row id) keeps the fingerprint reproducible across + * databases. A deterministic analyzer passes no models, so only its config + * hash contributes. + */ +export function computeConfigFingerprint(configHash: string, models: readonly string[]): string { + const canonicalModels = [...models].sort().join("|"); + return shortHash(`config(${configHash}|${canonicalModels})`); +} + +/** + * Canonical JSON hash of an analyzer config object. Object keys are sorted + * recursively so semantically equal configs hash identically. + */ +export function computeConfigHash(config: unknown): string { + return shortHash(`config(${canonicalJson(config)})`); +} + +/** + * The unique recipe identity for a node: analyzer + version + config + * fingerprint + source set. Re-running the same recipe over the same sources + * produces the same input_key, making analysis idempotent. Inputs only — the + * LLM's output never feeds this key. + */ +export function computeInputKey(parts: InputHashParts): string { + const canonical = [ + parts.analyzerId, + parts.analyzerVersionId, + parts.configFingerprint, + parts.sourceSetHash, + ].join("|"); + return shortHash(`input(${canonical})`); +} + +/** + * The content-addressed identity of a node's *result*: the recipe identity + * (`input_key`) folded together with the canonical node content. Deterministic + * and reproducible — recomputing it from stored content verifies the node, and + * a downstream consumer that references this key inherits the output into its + * own `input_key`. A different output (always a different node, by the + * append-only invariant) yields a different `output_key`. + */ +export function computeOutputKey(inputKey: string, content: unknown): string { + return shortHash(`output(${inputKey}|${canonicalJson(content)})`); +} + +/** Stable, sorted-key JSON serialisation for hashing. */ +export function canonicalJson(value: unknown): string { + return JSON.stringify(sortKeys(value)); +} + +function sortKeys(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(sortKeys); + } + if (value !== null && typeof value === "object") { + const obj = value as Record; + const out: Record = {}; + for (const key of Object.keys(obj).sort()) { + out[key] = sortKeys(obj[key]); + } + return out; + } + return value; +} + +/** + * Time-ordered unique id. We use a UUID v7-style prefix (millisecond timestamp) + * so ids sort chronologically, which keeps lineage timelines naturally ordered. + */ +export function uuidv7(): string { + const ms = Date.now(); + const tsHex = ms.toString(16).padStart(12, "0"); + const rand = randomUUID().replace(/-/g, "").slice(0, 20); + return `${tsHex.slice(0, 8)}-${tsHex.slice(8, 12)}-7${rand.slice(0, 3)}-${rand.slice(3, 7)}-${rand.slice(7, 19)}`; +} diff --git a/src/analyze/mock-llm.ts b/src/analyze/mock-llm.ts new file mode 100644 index 0000000..4cfeac2 --- /dev/null +++ b/src/analyze/mock-llm.ts @@ -0,0 +1,70 @@ +/** + * Mock LLM caller for tests. + * + * Analyzers that require an LLM are exercised against this deterministic mock + * rather than any real or local model. Two construction styles are supported: + * + * - a fixed/scripted queue of responses, consumed in order; or + * - a responder function that maps a request to response text. + * + * The mock records every request it received so tests can assert on prompts, + * models, and call counts. + */ + +import type { LLMCaller, LLMRequest, LLMResponse } from "./types.js"; + +export interface MockLLM { + caller: LLMCaller; + /** All requests received, in order. */ + calls: LLMRequest[]; +} + +export interface MockLLMOptions { + /** Map a request to the response text (typically JSON). */ + responder?: (request: LLMRequest, index: number) => string; + /** Fixed sequence of response texts, consumed in order. Overrides `responder`. */ + scripted?: string[]; + /** Default text when neither responder nor scripted produces one. */ + fallback?: string; + /** Simulated per-call token usage. */ + tokensPerCall?: number; + /** Simulated per-call cost. */ + costPerCall?: number; +} + +export function createMockLLM(options: MockLLMOptions = {}): MockLLM { + const calls: LLMRequest[] = []; + let index = 0; + + const caller: LLMCaller = async (request: LLMRequest): Promise => { + const i = index++; + calls.push(request); + + let text: string; + if (options.scripted) { + text = options.scripted[i] ?? options.fallback ?? ""; + } else if (options.responder) { + text = options.responder(request, i); + } else { + text = options.fallback ?? ""; + } + + return { + text, + model: request.model, + costUsd: options.costPerCall ?? 0, + tokensUsed: options.tokensPerCall ?? 0, + durationMs: 0, + stopReason: "stop", + }; + }; + + return { caller, calls }; +} + +/** A mock that always throws — useful to prove deterministic analyzers never call the LLM. */ +export function createThrowingLLM(message = "LLM must not be called"): LLMCaller { + return async () => { + throw new Error(message); + }; +} diff --git a/src/analyze/model-tiers.ts b/src/analyze/model-tiers.ts new file mode 100644 index 0000000..26ff0ea --- /dev/null +++ b/src/analyze/model-tiers.ts @@ -0,0 +1,61 @@ +/** + * Model tiers. Analyzers request an abstract tier (cheap/mid/expensive); the + * concrete `provider/model` string is resolved from the prospector config + * (`~/.pi/agent/prospector.json` → `modelTiers`). This keeps analyzers free of + * hard-coded model names and lets the user choose models per tier. + */ + +import type { ModelTier, ModelTierConfig } from "./types.js"; + +/** + * Default tier mapping. These are overridable via prospector config and are + * intentionally pointed at widely-available Pi providers. The user picks the + * real models; analyzers only ever ask for a tier. + */ +export const DEFAULT_MODEL_TIERS: ModelTierConfig = { + cheap: "anthropic/claude-haiku-4-5", + mid: "anthropic/claude-sonnet-4-5", + expensive: "anthropic/claude-opus-4-1", +}; + +const TIER_NAMES = new Set(["cheap", "mid", "expensive"]); + +export function isModelTier(value: string): value is ModelTier { + return TIER_NAMES.has(value); +} + +/** + * Resolve a model spec. A tier name maps through the config; an explicit + * `provider/model` spec passes through unchanged. + */ +export function resolveModelSpec(spec: string, config?: ModelTierConfig): string { + const tiers = config ?? DEFAULT_MODEL_TIERS; + if (isModelTier(spec)) return tiers[spec]; + return spec; +} + +/** Split a `provider/model` spec into its parts. The model id may itself contain slashes. */ +export function splitModelSpec(spec: string): { provider: string; modelId: string } { + const idx = spec.indexOf("/"); + if (idx < 0) { + throw new Error(`Invalid model spec '${spec}'. Expected 'provider/model' or a tier name (cheap|mid|expensive).`); + } + return { provider: spec.slice(0, idx), modelId: spec.slice(idx + 1) }; +} + +/** + * Apply a one-off model override to a tier mapping. When `override` is set, + * every tier (cheap/mid/expensive) is pinned to that single model, so an entire + * analysis run uses exactly one model regardless of the tier each analyzer asks + * for. The override may itself be a tier name (resolved through `tiers`) or a + * concrete `provider/model` spec. When `override` is empty the tiers are + * returned unchanged. + * + * Because the resolved model is part of node identity, pinning the model this + * way produces its own nodes: re-running without the override marks them stale. + */ +export function applyModelOverride(tiers: ModelTierConfig, override?: string): ModelTierConfig { + if (!override) return tiers; + const model = resolveModelSpec(override, tiers); + return { cheap: model, mid: model, expensive: model }; +} diff --git a/src/analyze/parser.ts b/src/analyze/parser.ts deleted file mode 100644 index 6c7713f..0000000 --- a/src/analyze/parser.ts +++ /dev/null @@ -1,63 +0,0 @@ -/** - * Parse LLM analysis response into structured proposals. - */ - -export interface ParsedProposal { - target: string; - severity: "friction" | "correction" | "waste" | "suggestion"; - summary: string; - detail: string; - evidence: string; -} - -const VALID_SEVERITIES = new Set(["friction", "correction", "waste", "suggestion"]); - -/** - * Parse LLM tool-call response into typed proposals. - * Handles both tool-call arguments object and plain JSON text. - */ -export function parseAnalysisResponse(response: unknown): ParsedProposal[] { - // If it's already a parsed tool call arguments object - if (response && typeof response === "object" && "proposals" in response) { - const proposals = (response as { proposals: unknown[] }).proposals; - if (Array.isArray(proposals)) { - return proposals.filter(isValidProposal).map(normalizeProposal); - } - } - - // If it's a string, try to extract JSON - if (typeof response === "string") { - const jsonMatch = response.match(/```json\s*([\s\S]*?)```/) ?? - response.match(/(\{[\s\S]*\})/); - if (jsonMatch) { - try { - const parsed = JSON.parse(jsonMatch[1] ?? jsonMatch[0]!); - if (parsed && typeof parsed === "object" && "proposals" in parsed && Array.isArray(parsed.proposals)) { - return parsed.proposals.filter(isValidProposal).map(normalizeProposal); - } - } catch { /* ignore */ } - } - } - - return []; -} - -function isValidProposal(item: unknown): item is Record { - if (!item || typeof item !== "object") return false; - const p = item as Record; - return typeof p.target === "string" && typeof p.summary === "string" && p.target.length > 0 && p.summary.length > 0; -} - -function normalizeProposal(item: Record): ParsedProposal { - const severity = VALID_SEVERITIES.has(item.severity as string) - ? (item.severity as ParsedProposal["severity"]) - : "suggestion"; - - return { - target: String(item.target), - severity, - summary: String(item.summary), - detail: String(item.detail ?? ""), - evidence: String(item.evidence ?? ""), - }; -} \ No newline at end of file diff --git a/src/analyze/pi-llm.ts b/src/analyze/pi-llm.ts new file mode 100644 index 0000000..7fd0fbc --- /dev/null +++ b/src/analyze/pi-llm.ts @@ -0,0 +1,120 @@ +/** + * Production LLM caller, wired to Pi's AI provider system. + * + * The flow is entirely within Pi's own provider machinery — there is no direct + * provider SDK use and no local model server: + * + * 1. Resolve the requested tier/spec to a `provider/model` pair. + * 2. `ctx.modelRegistry.find(provider, modelId)` → the Pi `Model`. + * 3. `ctx.modelRegistry.getApiKeyAndHeaders(model)` → credentials Pi has + * configured for that provider (env, models.json, OAuth, …). + * 4. `complete(model, context, { apiKey, headers, … })` from + * `@earendil-works/pi-ai` runs the request through Pi's provider adapters. + * + * `@earendil-works/pi-ai` is an optional peer dependency, so it is loaded with a + * runtime dynamic import; tests never reach this path (they use the mock caller). + */ + +import type { LLMCaller, LLMRequest, LLMResponse, ModelTierConfig } from "./types.js"; +import { resolveModelSpec, splitModelSpec } from "./model-tiers.js"; +import type { + ExtensionContext, + PiAiModule, + PiAssistantMessage, + PiContext, +} from "../pi-stubs.js"; + +let cachedModule: Promise | null = null; + +/** Lazily load pi-ai via a non-literal specifier so tsc/CI don't require it. */ +function loadPiAi(): Promise { + if (!cachedModule) { + const specifier = "@earendil-works/pi-ai"; + cachedModule = import(specifier).then((mod) => mod as unknown as PiAiModule); + } + return cachedModule; +} + +export interface PiLLMCallerOptions { + modelTiers: ModelTierConfig; +} + +/** + * Build an `LLMCaller` bound to a Pi extension context. The returned function + * resolves models against the live model registry and runs completions through + * pi-ai. Analyzers pass an already-resolved `provider/model` spec; a bare tier + * name is still tolerated and mapped through `modelTiers` as a safety net. + */ +export function makePiLLMCaller(ctx: ExtensionContext, opts: PiLLMCallerOptions): LLMCaller { + return async (request: LLMRequest): Promise => { + const start = Date.now(); + const spec = resolveModelSpec(request.model || "mid", opts.modelTiers); + const { provider, modelId } = splitModelSpec(spec); + + const model = ctx.modelRegistry.find(provider, modelId); + if (!model) { + throw new Error(`Model not found in Pi registry: ${provider}/${modelId}. Configure it via Pi or set modelTiers in prospector.json.`); + } + + const auth = await ctx.modelRegistry.getApiKeyAndHeaders(model); + if (!auth.ok) { + throw new Error(`No credentials for ${provider}/${modelId}: ${auth.error}`); + } + + const piAi = await loadPiAi(); + const context: PiContext = { + systemPrompt: request.system, + messages: [{ role: "user", content: request.user, timestamp: Date.now() }], + }; + + const message = await piAi.complete(model, context, { + apiKey: auth.apiKey, + headers: auth.headers, + temperature: request.temperature, + maxTokens: request.maxTokens, + // Let pi-ai ride out transient rate limits (e.g. provider 429s) instead + // of failing a whole analysis run on the first throttled call. + maxRetries: 4, + signal: ctx.signal, + }); + + return toLLMResponse(message, spec, Date.now() - start); + }; +} + +/** Flatten a pi-ai AssistantMessage into the framework's LLMResponse. */ +export function toLLMResponse(message: PiAssistantMessage, modelSpec: string, durationMs: number): LLMResponse { + const textParts: string[] = []; + const thinkingParts: string[] = []; + for (const part of message.content) { + if (part.type === "text") textParts.push(part.text); + else if (part.type === "thinking") thinkingParts.push(part.thinking); + } + + if (message.stopReason === "error") { + throw new Error(`LLM error from ${modelSpec}: ${message.errorMessage ?? "unknown error"}`); + } + + // Every call this caller makes expects a complete structured (JSON) answer, so + // a response cut off at the output limit is never usable. Fail fast with an + // actionable message instead of letting the truncated body surface later as a + // cryptic "Unterminated JSON object" parse error. Reasoning models are the + // usual cause: their thinking tokens consume the maxTokens budget. + if (message.stopReason === "length") { + const outputTokens = message.usage?.output ?? 0; + throw new Error( + `LLM response from ${modelSpec} was truncated at the output limit (${outputTokens} output tokens) ` + + `before the answer was complete. Raise maxTokens, or use a non-reasoning model/tier for structured output.`, + ); + } + + return { + text: textParts.join("\n").trim(), + thinking: thinkingParts.length > 0 ? thinkingParts.join("\n").trim() : undefined, + model: message.model || modelSpec, + costUsd: message.usage?.cost?.total ?? 0, + tokensUsed: message.usage?.totalTokens ?? 0, + durationMs, + stopReason: message.stopReason, + }; +} diff --git a/src/analyze/prompt.ts b/src/analyze/prompt.ts deleted file mode 100644 index facf044..0000000 --- a/src/analyze/prompt.ts +++ /dev/null @@ -1,69 +0,0 @@ -/** - * Prompt template for session friction/sentiment extraction. - * Uses tool-calling schema for structured LLM output. - */ - -export const ANALYSIS_TOOL_NAME = "submit_proposals"; - -export const ANALYSIS_SYSTEM_PROMPT = `You are a session analyst for an AI coding agent. You review session transcripts and identify friction, corrections, waste, and suggestions for improvement. - -You MUST call the ${ANALYSIS_TOOL_NAME} tool with your findings. Do not respond in plain text. - -Focus on: -1. **Friction**: Moments where the user struggled, repeated themselves, or had to course-correct the agent. -2. **Corrections**: Times the user explicitly corrected the agent ("no, use X", "not like that", "actually..."). -3. **Waste**: Tool calls or context that didn't contribute to the task — large file reads never referenced, failed commands retried without changes. -4. **Suggestions**: Opportunities to improve the agent's configuration, skills, or documentation based on observed patterns. - -Each proposal should target a specific, actionable improvement. Prefer specific, small changes over vague recommendations.`; - -export function buildAnalysisPrompt(transcript: string, sessionProject: string): string { - return `## Session: ${sessionProject} - - -${transcript} - - -Analyze this session transcript for friction, corrections, waste, and suggestions. Call the ${ANALYSIS_TOOL_NAME} tool with your findings.`; -} - -export const ANALYSIS_TOOL_SCHEMA = { - name: ANALYSIS_TOOL_NAME, - description: "Submit proposals for improving the coding agent based on session analysis", - parameters: { - type: "object" as const, - properties: { - proposals: { - type: "array" as const, - items: { - type: "object" as const, - properties: { - target: { - type: "string" as const, - description: "What to change, e.g. 'AGENTS.md § Tool usage' or 'skill/debug-typescript-errors'", - }, - severity: { - type: "string" as const, - enum: ["friction", "correction", "waste", "suggestion"], - description: "The type of finding", - }, - summary: { - type: "string" as const, - description: "One-line description of the proposed change", - }, - detail: { - type: "string" as const, - description: "Full proposal text with context and suggested change", - }, - evidence: { - type: "string" as const, - description: "The session excerpt that triggered this proposal", - }, - }, - required: ["target", "severity", "summary", "detail", "evidence"], - }, - }, - }, - required: ["proposals"], - }, -}; \ No newline at end of file diff --git a/src/analyze/proposal-materializer.ts b/src/analyze/proposal-materializer.ts new file mode 100644 index 0000000..da5653a --- /dev/null +++ b/src/analyze/proposal-materializer.ts @@ -0,0 +1,127 @@ +/** + * Proposal materialisation. + * + * Analyzers that emit `summary`/`proposal` nodes embed an + * `improvement_proposals` array in their content. This module extracts those + * proposals into the fast-access `proposals` table, deduplicating by a stable + * key and recording a `produces` edge from the source node to the proposal. + */ + +import type Database from "better-sqlite3"; +import { shortHash, uuidv7 } from "./input-hash.js"; +import { insertEdge } from "../db/analysis-queries.js"; +import { EDGE_KINDS, REF_KINDS } from "./edge-kinds.js"; + +export interface RawProposal { + target_type: string; + target_path?: string; + title: string; + summary: string; + detail?: string; + evidence?: string; + confidence?: number; + severity: string; +} + +export interface MaterializeParams { + sessionId: string; + analyzerId: string; + sourceNodeId: string; + /** The content-addressed output_key of the source node; the proposal's identity derives from it. */ + sourceOutputKey: string; + contentJson: Record; + now: string; +} + +/** + * A proposal's identity is derived from its *source* — the content-addressed + * `output_key` of the node that produced it, plus its ordinal within that node's + * proposal array — never from the model's free-text title/path/severity. So + * re-materialising the same node is idempotent, while two distinct nodes (a + * different session, or a revised version) keep their proposals separately. + */ +export function computeProposalInputKey(p: { sourceOutputKey: string; ordinal: number }): string { + return shortHash(`proposal(${p.sourceOutputKey}|${p.ordinal})`); +} + +/** + * Extract and persist proposals from a node's content. Returns the number of + * *new* proposals created (duplicates of still-open proposals are skipped). + */ +export function materializeProposalsFromNode(db: Database.Database, params: MaterializeParams): number { + const raw = params.contentJson["improvement_proposals"]; + if (!Array.isArray(raw)) return 0; + + let created = 0; + let ordinal = -1; + for (const candidate of raw) { + ordinal++; + const proposal = normalizeProposal(candidate); + if (!proposal) continue; + + const inputKey = computeProposalInputKey({ sourceOutputKey: params.sourceOutputKey, ordinal }); + const existing = db + .prepare("SELECT id FROM proposals WHERE input_key = ? AND status = 'open' LIMIT 1") + .get(inputKey) as { id: string } | undefined; + if (existing) continue; + + const proposalId = uuidv7(); + db.prepare(` + INSERT INTO proposals + (id, created_at, updated_at, session_id, source_node_id, analyzer_id, target_type, target_path, + title, severity, summary, detail, evidence, confidence, status, input_key) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'open', ?) + `).run( + proposalId, + params.now, + params.now, + params.sessionId, + params.sourceNodeId, + params.analyzerId, + proposal.target_type, + proposal.target_path ?? null, + proposal.title, + proposal.severity, + proposal.summary, + proposal.detail ?? null, + proposal.evidence ?? null, + proposal.confidence ?? null, + inputKey, + ); + + insertEdge(db, { + fromNodeId: params.sourceNodeId, + toRefKind: REF_KINDS.PROPOSAL, + toRefId: proposalId, + edgeKind: EDGE_KINDS.PRODUCES, + ordinal: created, + }); + + created++; + } + + return created; +} + +/** Coerce an untrusted LLM-produced object into a RawProposal, or null if invalid. */ +function normalizeProposal(value: unknown): RawProposal | null { + if (!value || typeof value !== "object") return null; + const v = value as Record; + const title = typeof v["title"] === "string" ? (v["title"] as string).trim() : ""; + const summary = typeof v["summary"] === "string" ? (v["summary"] as string).trim() : ""; + if (!title || !summary) return null; + + const targetType = typeof v["target_type"] === "string" && v["target_type"] ? (v["target_type"] as string) : "general"; + const severity = typeof v["severity"] === "string" && v["severity"] ? (v["severity"] as string) : "suggestion"; + + return { + target_type: targetType, + target_path: typeof v["target_path"] === "string" ? (v["target_path"] as string) : undefined, + title, + summary, + detail: typeof v["detail"] === "string" ? (v["detail"] as string) : undefined, + evidence: typeof v["evidence"] === "string" ? (v["evidence"] as string) : undefined, + confidence: typeof v["confidence"] === "number" ? (v["confidence"] as number) : undefined, + severity, + }; +} diff --git a/src/analyze/types.ts b/src/analyze/types.ts new file mode 100644 index 0000000..ba44172 --- /dev/null +++ b/src/analyze/types.ts @@ -0,0 +1,336 @@ +/** + * Analyzer framework type definitions. + * + * Per project guidelines, every *data shape* is a TypeBox schema with its + * static type derived via `Static`. Behavioural contracts that carry function + * members (analyzers and their execution contexts) are declared as interfaces, + * since functions are not data shapes. + */ + +import { Type, type Static } from "typebox"; +import type Database from "better-sqlite3"; + +// ─────────────────────────── enumerations ─────────────────────────── + +export const ImplementationKind = Type.Union([ + Type.Literal("deterministic"), + Type.Literal("in_process_llm"), +]); +export type ImplementationKind = Static; + +export const AnchorSpan = Type.Union([ + Type.Literal("pair"), + Type.Literal("segment"), + Type.Literal("full_session"), +]); +export type AnchorSpan = Static; + +export const NodeKind = Type.Union([ + Type.Literal("metric"), + Type.Literal("classification"), + Type.Literal("summary"), + Type.Literal("proposal"), + Type.Literal("error"), +]); +export type NodeKind = Static; + +export const ReviseReason = Type.Union([ + Type.Literal("major"), + Type.Literal("minor"), + Type.Literal("config"), +]); +export type ReviseReason = Static; + +export const UnitStatus = Type.Union([ + Type.Literal("missing"), + Type.Literal("stale"), + Type.Literal("current"), +]); +export type UnitStatus = Static; + +export const RunStatus = Type.Union([ + Type.Literal("ok"), + Type.Literal("error"), + Type.Literal("partial"), +]); +export type RunStatus = Static; + +// ─────────────────────────── registry shapes ─────────────────────────── + +export const AnalyzerDef = Type.Object({ + id: Type.String(), + label: Type.String(), + description: Type.String(), + anchorSpan: AnchorSpan, + dependencies: Type.Array(Type.String()), +}); +export type AnalyzerDef = Static; + +export const AnalyzerVersion = Type.Object({ + analyzerId: Type.String(), + /** Author-owned significance grade: bump major for significant changes, minor for small ones. */ + major: Type.Integer({ minimum: 0 }), + minor: Type.Integer({ minimum: 0 }), + implementationKind: ImplementationKind, + codeRef: Type.Optional(Type.String()), +}); +export type AnalyzerVersion = Static; + +export const PromptVersion = Type.Object({ + hash: Type.String(), + content: Type.String(), + role: Type.Optional(Type.String()), +}); +export type PromptVersion = Static; + +export const AnalyzerConfig = Type.Object({ + id: Type.String(), + analyzerId: Type.String(), + configHash: Type.String(), + configJson: Type.Record(Type.String(), Type.Unknown()), + label: Type.Optional(Type.String()), +}); +export type AnalyzerConfig = Static; + +// ─────────────────────────── planning shapes ─────────────────────────── + +export const SourceRef = Type.Object({ + kind: Type.Union([Type.Literal("message"), Type.Literal("analysis_node"), Type.Literal("session")]), + id: Type.String(), +}); +export type SourceRef = Static; + +export const AnalysisUnit = Type.Object({ + sources: Type.Array(SourceRef), + sourceSetHash: Type.String(), + anchorKind: Type.Union([Type.Literal("session"), Type.Literal("message")]), + anchorRef: Type.String(), + meta: Type.Optional(Type.Record(Type.String(), Type.Unknown())), +}); +export type AnalysisUnit = Static; + +export const EdgeSpec = Type.Object({ + toRefKind: Type.String(), + toRefId: Type.String(), + edgeKind: Type.String(), + ordinal: Type.Optional(Type.Number()), +}); +export type EdgeSpec = Static; + +export const AnalysisResult = Type.Object({ + nodeKind: NodeKind, + contentJson: Type.Record(Type.String(), Type.Unknown()), + anchorKind: Type.Union([Type.Literal("session"), Type.Literal("message")]), + anchorRef: Type.String(), + edges: Type.Array(EdgeSpec), + modelUsed: Type.Optional(Type.String()), + costUsd: Type.Optional(Type.Number()), + tokensUsed: Type.Optional(Type.Number()), + durationMs: Type.Optional(Type.Number()), +}); +export type AnalysisResult = Static; + +// ─────────────────────────── database rows ─────────────────────────── + +export const MessageRow = Type.Object({ + id: Type.String(), + session_id: Type.String(), + parent_id: Type.Union([Type.String(), Type.Null()]), + timestamp: Type.Union([Type.String(), Type.Null()]), + role: Type.String(), + content_text: Type.Union([Type.String(), Type.Null()]), + content_thinking: Type.Union([Type.String(), Type.Null()]), + tool_calls: Type.Union([Type.String(), Type.Null()]), + tool_results: Type.Union([Type.String(), Type.Null()]), +}); +export type MessageRow = Static; + +export const AnalysisNodeRow = Type.Object({ + id: Type.String(), + session_id: Type.String(), + analyzer_id: Type.String(), + analyzer_version_id: Type.String(), + config_id: Type.String(), + run_id: Type.Union([Type.String(), Type.Null()]), + node_kind: Type.String(), + content_json: Type.String(), + source_set_hash: Type.String(), + input_key: Type.String(), + output_key: Type.String(), + config_fingerprint: Type.String(), + model_used: Type.Union([Type.String(), Type.Null()]), + cost_usd: Type.Union([Type.Number(), Type.Null()]), + tokens_used: Type.Union([Type.Number(), Type.Null()]), + duration_ms: Type.Union([Type.Number(), Type.Null()]), + created_at: Type.String(), +}); +export type AnalysisNodeRow = Static; + +export const AnalysisEdgeRow = Type.Object({ + id: Type.String(), + from_node_id: Type.String(), + to_ref_kind: Type.String(), + to_ref_id: Type.String(), + edge_kind: Type.String(), + ordinal: Type.Number(), +}); +export type AnalysisEdgeRow = Static; + +export const AnalysisRunRow = Type.Object({ + id: Type.String(), + analyzer_id: Type.String(), + analyzer_version_id: Type.String(), + config_id: Type.String(), + session_id: Type.String(), + mode: Type.String(), + status: Type.String(), + prompt_bundle_hash: Type.String(), + model_spec: Type.Union([Type.String(), Type.Null()]), + started_at: Type.String(), + finished_at: Type.Union([Type.String(), Type.Null()]), + nodes_produced: Type.Number(), + nodes_skipped: Type.Number(), + cost_usd: Type.Number(), + tokens_used: Type.Number(), + error_message: Type.Union([Type.String(), Type.Null()]), +}); +export type AnalysisRunRow = Static; + +// ─────────────────────────── LLM shapes ─────────────────────────── + +/** + * A model tier configuration. Analyzers request a tier (cheap/mid/expensive); + * the resolved provider/model strings come from `~/.pi/agent/prospector.json`. + */ +export const ModelTierConfig = Type.Object({ + cheap: Type.String(), + mid: Type.String(), + expensive: Type.String(), +}); +export type ModelTierConfig = Static; + +export const ModelTier = Type.Union([ + Type.Literal("cheap"), + Type.Literal("mid"), + Type.Literal("expensive"), +]); +export type ModelTier = Static; + +export const LLMRequest = Type.Object({ + /** A tier name ("cheap"|"mid"|"expensive") or an explicit "provider/model" spec. */ + model: Type.String(), + system: Type.Optional(Type.String()), + user: Type.String(), + temperature: Type.Optional(Type.Number()), + maxTokens: Type.Optional(Type.Number()), +}); +export type LLMRequest = Static; + +export const LLMResponse = Type.Object({ + text: Type.String(), + thinking: Type.Optional(Type.String()), + model: Type.String(), + costUsd: Type.Number(), + tokensUsed: Type.Number(), + durationMs: Type.Number(), + stopReason: Type.String(), +}); +export type LLMResponse = Static; + +/** + * The LLM calling contract. Production wires this to Pi's AI provider system + * (model registry + `@earendil-works/pi-ai` `complete`); tests wire a mock. + * A function type, not a data shape. + */ +export type LLMCaller = (request: LLMRequest) => Promise; + +// ─────────────────────────── behavioural contracts ─────────────────────────── + +/** Read-only context handed to `analyzer.plan()`. */ +export interface AnalyzerPlanContext { + sessionId: string; + messages: MessageRow[]; + /** All analysis nodes for this session (own + dependencies). */ + allNodes: AnalysisNodeRow[]; + /** This analyzer's own nodes for the session. */ + ownNodes: AnalysisNodeRow[]; + /** Dependency nodes keyed by analyzer id (only declared dependencies). */ + dependencyNodes: Record; + /** The resolved config JSON for this analyzer, so plan() can honour cost guards. */ + config: Record; + db: Database.Database; +} + +/** Context handed to `analyzer.analyze()` while producing a single node. */ +export interface AnalyzerRunContext { + sessionId: string; + getMessage: (id: string) => MessageRow | undefined; + getNode: (id: string) => AnalysisNodeRow | undefined; + /** Nodes from a declared dependency. Throws if the dependency was not declared. */ + getDependencyNodes: (analyzerId: string) => AnalysisNodeRow[]; + getSessionMessages: (sessionId: string) => MessageRow[]; + llm: LLMCaller; + config: AnalyzerConfig; + /** Prompt content keyed by prompt name. */ + prompts: Record; + modelTiers: ModelTierConfig; +} + +/** An analyzer: stable definition + version + prompts + default config + behaviour. */ +export interface Analyzer { + def: AnalyzerDef; + version: AnalyzerVersion; + prompts: Record; + defaultConfig: AnalyzerConfig; + plan: (ctx: AnalyzerPlanContext) => AnalysisUnit[] | Promise; + analyze: (unit: AnalysisUnit, ctx: AnalyzerRunContext) => AnalysisResult | Promise; + /** + * The concrete models this analyzer will use under the given config, with + * tier shorthands (cheap/mid/expensive) already resolved to `provider/model`. + * The resolved model is part of a node's `config` identity, so changing which + * model a tier resolves to marks existing nodes `stale` for the `config` + * reason — a run that includes `config` revises them into a new version, while + * a plain fill leaves them alone. Deterministic analyzers omit this: with no + * model, their identity never depends on model settings. + */ + modelsForIdentity?: (config: Record, modelTiers: ModelTierConfig) => string[]; +} + +// ─────────────────────────── framework results ─────────────────────────── + +export interface ClassifiedUnit { + analyzerId: string; + unit: AnalysisUnit; + status: UnitStatus; + inputKey: string; + /** For `stale` units: the prior node this unit would revise. */ + priorNodeId?: string; + /** For `stale` units: why it is out of date (any of major/minor/config). Empty otherwise. */ + reasons: ReviseReason[]; +} + +export interface RunSummary { + sessionId: string; + /** The revise reasons this run acted on (empty = a plain fill of missing work). */ + revise: ReviseReason[]; + analyzerResults: AnalyzerRunResult[]; + nodesProduced: number; + nodesSkipped: number; + nodesRevised: number; + proposalsCreated: number; + costUsd: number; + tokensUsed: number; + errors: string[]; +} + +export interface AnalyzerRunResult { + analyzerId: string; + runId: string; + nodesProduced: number; + nodesSkipped: number; + nodesRevised: number; + costUsd: number; + tokensUsed: number; + status: RunStatus; + errorMessage?: string; +} diff --git a/src/analyze/version.ts b/src/analyze/version.ts new file mode 100644 index 0000000..2e3bb07 --- /dev/null +++ b/src/analyze/version.ts @@ -0,0 +1,83 @@ +/** + * Analyzer version identity and revise-reason logic. + * + * An analyzer's version is a `major.minor` pair the *author* owns: major for a + * change the author judges significant, minor for a small one. The version + * represents everything the author ships — logic, default prompt, default tier. + * + * When a node is out of date the framework grades *why*: a higher major is a + * `major` reason, a higher minor (same major) is `minor`, and a changed user + * `config` is the (ungraded) `config` reason. A run's `--revise` reasons are a + * *set* that selects which stale units to recompute; `minor` implies `major`. + */ + +import type { ReviseReason } from "./types.js"; + +export interface SemVer { + major: number; + minor: number; +} + +/** Canonical "major.minor" string, stored on runs and nodes for display/lineage. */ +export function versionIdOf(v: SemVer): string { + return `${v.major}.${v.minor}`; +} + +/** Parse a canonical "major.minor" string back into its components. */ +export function parseVersionId(versionId: string): SemVer { + const [major, minor] = versionId.split("."); + return { + major: Number.parseInt(major ?? "", 10) || 0, + minor: Number.parseInt(minor ?? "", 10) || 0, + }; +} + +/** + * Grade a version move from `prior` to `current`: + * - "major" when current's major is higher, + * - "minor" when the major is unchanged but the minor is higher, + * - null otherwise (equal, or a downgrade we never auto-revise toward). + */ +export function gradeVersionMove(prior: SemVer, current: SemVer): "major" | "minor" | null { + if (current.major > prior.major) return "major"; + if (current.major === prior.major && current.minor > prior.minor) return "minor"; + return null; +} + +/** + * Expand requested revise reasons into the effective selection set: `minor` + * implies `major` (adopting minor bumps means also adopting major ones). `config` + * is orthogonal to the version grade and is carried through unchanged. + */ +export function expandReviseReasons(reasons: readonly ReviseReason[]): Set { + const set = new Set(reasons); + if (set.has("minor")) set.add("major"); + return set; +} + +/** + * Parse a `--revise` argument value (comma-separated) into reasons. Accepts + * `major`, `minor`, `config`, `all` (every reason), and `none`/empty. Unknown + * tokens are ignored. + */ +export function parseReviseArg(value: string): ReviseReason[] { + const out = new Set(); + for (const raw of value.split(",")) { + const token = raw.trim().toLowerCase(); + if (token === "all") { + out.add("major"); + out.add("minor"); + out.add("config"); + } else if (token === "major" || token === "minor" || token === "config") { + out.add(token); + } + } + return [...out]; +} + +/** A run's human-readable reach label, for run provenance and command output. */ +export function reachLabel(reasons: ReadonlySet | readonly ReviseReason[]): string { + const set = reasons instanceof Set ? reasons : new Set(reasons); + if (set.size === 0) return "fill"; + return `revise:${[...set].sort().join("+")}`; +} diff --git a/src/commands/analyze.ts b/src/commands/analyze.ts index 1d6aa7a..acb4c42 100644 --- a/src/commands/analyze.ts +++ b/src/commands/analyze.ts @@ -1,80 +1,127 @@ -import type { ExtensionAPI } from "../pi-stubs.js"; +import type { ExtensionAPI, ExtensionCommandContext } from "../pi-stubs.js"; import Database from "better-sqlite3"; import { migrate } from "../db/schema.js"; -import { getUnanalyzedSessions, getSessionMessages, markAnalyzed } from "../db/queries.js"; -import { getDbPath, loadConfig } from "../config.js"; +import { getAllSessions, getUnanalyzedSessions, markAnalyzed } from "../db/queries.js"; +import { getDbPath, getModelTiers, loadConfig } from "../config.js"; +import { AnalyzerFramework } from "../analyze/framework.js"; +import { registerDefaults } from "../analyze/defaults.js"; +import { makePiLLMCaller } from "../analyze/pi-llm.js"; +import { applyModelOverride } from "../analyze/model-tiers.js"; +import { parseReviseArg, reachLabel } from "../analyze/version.js"; +import type { ReviseReason } from "../analyze/types.js"; -export function registerAnalyzeCommand(pi: ExtensionAPI): void { - pi.registerCommand("prospect-analyze", { - description: "Run LLM analysis over unanalyzed sessions to generate proposals", - handler: async (args: string, ctx: { ui: { notify: (msg: string, level: string) => void } }) => { - const config = loadConfig(); - const parsedArgs = parseArgs(args ?? ""); - const modelSpec = parsedArgs.model ?? config.model; +interface AnalyzeArgs { + revise: ReviseReason[]; + limit?: number; + session?: string; + analyzer?: string; + model?: string; +} - if (!modelSpec) { - const msg = "No model configured. Use --model provider/model or set in ~/.pi/agent/prospector.json"; - ctx.ui.notify(msg, "error"); - console.log(msg); - return; - } +export async function prospectAnalyze(rawArgs: string, ctx: ExtensionCommandContext): Promise { + const args = parseArgs(rawArgs ?? ""); + const reviseActive = args.revise.length > 0; + const reach = reachLabel(args.revise); + const config = loadConfig(); + // A --model override pins every tier to that one model for this run. The + // same effective tiers feed both the LLM caller and the framework, so the + // model actually used always matches the model folded into node identity. + const modelTiers = applyModelOverride(getModelTiers(config), args.model); - const db = new Database(getDbPath()); - migrate(db); + const db = new Database(getDbPath(config)); + migrate(db); - try { - const unanalyzed = getUnanalyzedSessions(db, parsedArgs.limit); - if (unanalyzed.length === 0) { - const msg = "No unanalyzed sessions. Run /prospect-sync first."; - ctx.ui.notify(msg, "info"); - console.log(msg); - return; - } + try { + // A plain fill focuses on not-yet-analysed sessions; any revise reason + // re-scans every session so stale nodes can be picked up. + const sessions = args.session + ? [{ id: args.session, file_path: "", started_at: "" }] + : reviseActive + ? getAllSessions(db, args.limit) + : getUnanalyzedSessions(db, args.limit); - const startMsg = `Analyzing ${unanalyzed.length} session(s) with ${modelSpec}...`; - ctx.ui.notify(startMsg, "info"); - console.log(startMsg); + if (sessions.length === 0) { + out(ctx, "No sessions to analyse. Run /prospect-sync first.", "info"); + return; + } - let totalProposals = 0; - let errors = 0; + const llm = makePiLLMCaller(ctx, { modelTiers }); + const framework = new AnalyzerFramework({ db, llm, modelTiers }); + registerDefaults(framework); + const analyzerIds = args.analyzer ? [args.analyzer] : undefined; - for (const session of unanalyzed) { - try { - const messages = getSessionMessages(db, session.id); - if (messages.length < 2) { - markAnalyzed(db, session.id); - continue; - } + out(ctx, `Analysing ${sessions.length} session(s) [${reach}]…`, "info"); - // TODO: Call LLM via @earendil-works/pi-ai - markAnalyzed(db, session.id); - } catch (err) { - errors++; - const errMsg = `Error on session ${session.id}: ${err}`; - ctx.ui.notify(errMsg, "warning"); - console.error(errMsg); - } - } + let nodesProduced = 0; + let nodesRevised = 0; + let proposals = 0; + let cost = 0; + const errors: string[] = []; - const doneMsg = `Done. ${unanalyzed.length - errors} analyzed, ${totalProposals} proposals, ${errors} errors.`; - ctx.ui.notify(doneMsg, "info"); - console.log(doneMsg); - } finally { - db.close(); + for (const session of sessions) { + try { + const summary = await framework.run(session.id, { revise: args.revise, analyzerIds, modelSpec: args.model }); + nodesProduced += summary.nodesProduced; + nodesRevised += summary.nodesRevised; + proposals += summary.proposalsCreated; + cost += summary.costUsd; + errors.push(...summary.errors); + // Bare-fill self-healing: only retire the session from the unanalysed + // queue when it completed cleanly. If any unit failed, leave + // `analyzed_at` NULL so the next plain fill re-scans it and recomputes + // the still-missing units (the failures left no result behind). + if (summary.errors.length === 0) { + markAnalyzed(db, session.id); + } + } catch (err) { + errors.push(`${session.id}: ${err instanceof Error ? err.message : String(err)}`); } - }, + } + + const lines = [ + `Done [${reach}]. ${sessions.length} session(s) scanned.`, + ` Nodes produced: ${nodesProduced} (revised: ${nodesRevised})`, + ` Proposals created: ${proposals}`, + ` Estimated cost: $${cost.toFixed(4)}`, + ]; + if (errors.length > 0) { + lines.push(` Errors: ${errors.length}`); + for (const e of errors.slice(0, 5)) lines.push(` ${e}`); + } + out(ctx, lines.join("\n"), errors.length > 0 ? "warning" : "info"); + } finally { + db.close(); + } +} + +export function registerAnalyzeCommand(pi: ExtensionAPI): void { + pi.registerCommand("prospect-analyze", { + description: + "Run analyzer framework over sessions (incremental). Flags: --revise major|minor|config|all (recompute stale nodes: major/minor analyzer bumps, config = your setup changed; default fills only missing work), --limit N, --session ID, --analyzer ID, --model provider/model (pin every tier to one model for this run; the model is part of node identity)", + handler: prospectAnalyze, }); } -function parseArgs(raw: string): { model?: string; limit?: number } { - const result: { model?: string; limit?: number } = {}; - const parts = raw.split(/\s+/); +function out(ctx: ExtensionCommandContext, text: string, level: string): void { + ctx.ui.notify(text, level); + console.log(text); +} + +function parseArgs(raw: string): AnalyzeArgs { + const result: AnalyzeArgs = { revise: [] }; + const parts = raw.trim().split(/\s+/).filter((p) => p.length > 0); for (let i = 0; i < parts.length; i++) { - if (parts[i] === "--model" && parts[i + 1]) result.model = parts[++i]; - else if (parts[i] === "--limit" && parts[i + 1]) { + const p = parts[i]; + if (p === "--revise" && parts[i + 1]) { + for (const r of parseReviseArg(parts[++i]!)) { + if (!result.revise.includes(r)) result.revise.push(r); + } + } else if (p === "--limit" && parts[i + 1]) { const n = parseInt(parts[++i]!, 10); - if (!isNaN(n)) result.limit = n; - } + if (!Number.isNaN(n)) result.limit = n; + } else if (p === "--session" && parts[i + 1]) result.session = parts[++i]; + else if (p === "--analyzer" && parts[i + 1]) result.analyzer = parts[++i]; + else if (p === "--model" && parts[i + 1]) result.model = parts[++i]; } return result; -} \ No newline at end of file +} diff --git a/src/commands/headless.ts b/src/commands/headless.ts new file mode 100644 index 0000000..6265535 --- /dev/null +++ b/src/commands/headless.ts @@ -0,0 +1,87 @@ +import type { ExtensionAPI, ExtensionCommandContext } from "../pi-stubs.js"; +import { prospectSync } from "./sync.js"; +import { prospectStats } from "./stats.js"; +import { prospectProposals, prospectAccept, prospectReject } from "./proposals.js"; +import { prospectAnalyze } from "./analyze.js"; +import { prospectVerify } from "./verify.js"; +import { prospectShow } from "./show.js"; + +/** A command runnable both as a slash command and via the `--prospect` flag. */ +export type ProspectAction = (args: string, ctx: ExtensionCommandContext) => Promise; + +/** Maps a `--prospect` sub-command name to its handler. */ +export const PROSPECT_ACTIONS: Record = { + sync: prospectSync, + analyze: prospectAnalyze, + stats: prospectStats, + proposals: prospectProposals, + show: prospectShow, + verify: prospectVerify, + accept: prospectAccept, + reject: prospectReject, +}; + +const USAGE = + 'Usage: pi -e /src/index.ts --prospect " [args]"\n' + + " commands: sync | analyze [flags] | stats | proposals [status] [--full] | show | verify | accept | reject "; + +/** Split a `--prospect` flag value into a command name and the remaining args. */ +export function splitProspectSpec(spec: string): { command: string; args: string } { + const trimmed = spec.trim(); + const ws = trimmed.search(/\s/); + if (ws === -1) return { command: trimmed.toLowerCase(), args: "" }; + return { command: trimmed.slice(0, ws).toLowerCase(), args: trimmed.slice(ws + 1).trim() }; +} + +/** + * Run the action named by a `--prospect` flag value. Returns true if an action + * ran (or threw), false for an empty/unknown command (usage printed to stderr). + */ +export async function runProspectSpec( + spec: string, + ctx: ExtensionCommandContext, + actions: Record = PROSPECT_ACTIONS, +): Promise { + if (!spec || spec.trim() === "") { + console.error(USAGE); + return false; + } + const { command, args } = splitProspectSpec(spec); + const action = actions[command]; + if (!action) { + console.error(`Unknown --prospect command: "${command}".\n${USAGE}`); + return false; + } + await action(args, ctx); + return true; +} + +/** + * Register the `--prospect` CLI flag. When present, the named command runs once + * at session start and pi shuts down — so a bare + * `pi -e .../src/index.ts --prospect stats` is non-interactive by default, with + * no need for `-p`. When the flag is absent, the extension stays interactive. + */ +export function registerHeadlessFlag(pi: ExtensionAPI): void { + pi.registerFlag("prospect", { + description: + 'Run a prospector command non-interactively and exit, e.g. --prospect "analyze --limit 3" or --prospect "proposals --full". Commands: sync | analyze | stats | proposals | accept | reject ', + type: "string", + }); + + let dispatched = false; + pi.on("session_start", async (_event, ctx) => { + const spec = pi.getFlag("prospect"); + if (typeof spec !== "string" || spec.trim() === "") return; + // session_start can fire again (reload/resume); only run the one-shot once. + if (dispatched) return; + dispatched = true; + try { + await runProspectSpec(spec, ctx); + } catch (err) { + console.error(`prospect: ${err instanceof Error ? err.message : String(err)}`); + } finally { + await ctx.shutdown?.(); + } + }); +} diff --git a/src/commands/proposals.ts b/src/commands/proposals.ts index 5c7b051..3ef64c8 100644 --- a/src/commands/proposals.ts +++ b/src/commands/proposals.ts @@ -1,69 +1,156 @@ -import type { ExtensionAPI } from "../pi-stubs.js"; +import type { ExtensionAPI, ExtensionCommandContext } from "../pi-stubs.js"; import Database from "better-sqlite3"; import { migrate } from "../db/schema.js"; -import { listProposals, acceptProposal, rejectProposal } from "../db/queries.js"; +import { listProposals, acceptProposal, rejectProposal, getSessionLabels } from "../db/queries.js"; import { getDbPath } from "../config.js"; +import type { Proposal } from "../types.js"; +import { homedir } from "node:os"; -function output(ctx: any, text: string, level: "info" | "warning" | "error" = "info"): void { +function output(ctx: ExtensionCommandContext, text: string, level: "info" | "warning" | "error" = "info"): void { ctx.ui.notify(text, level); console.log(text); } +const PROPOSAL_STATUSES = new Set(["open", "applied", "rejected", "duplicate"]); + +/** + * Parse the `proposals` argument string into an optional status filter and a + * `full` flag. Accepts a status word (open|applied|rejected|duplicate) and/or + * `--full`/`-v`/`--verbose`, in any order; unknown tokens are ignored. + */ +export function parseProposalsArgs(args: string): { status?: string; full: boolean } { + let status: string | undefined; + let full = false; + for (const tok of (args ?? "").trim().split(/\s+/).filter(Boolean)) { + const t = tok.toLowerCase(); + if (t === "--full" || t === "-v" || t === "--verbose") full = true; + else if (PROPOSAL_STATUSES.has(t)) status = t; + } + return { status, full }; +} + +function formatConfidence(confidence: number | null): string { + return confidence == null ? " n/a" : `${Math.round(confidence * 100)}%`; +} + +function formatTarget(p: Proposal): string { + return p.target_path ? `${p.target_type}: ${p.target_path}` : p.target_type; +} + +/** Strongest recommendations first: confidence desc (nulls last), then newest. */ +export function rankProposals(a: Proposal, b: Proposal): number { + const ca = a.confidence ?? -1; + const cb = b.confidence ?? -1; + if (cb !== ca) return cb - ca; + if (a.created_at === b.created_at) return 0; + return a.created_at < b.created_at ? 1 : -1; +} + +function conciseEntry(p: Proposal): string { + return ` [${p.status}] ${formatConfidence(p.confidence).padStart(4)} ${p.severity} · ${formatTarget(p)}\n ${p.title}\n ${p.summary}\n id: ${p.id} · prospect show ${p.id}`; +} + +function fullEntry(p: Proposal): string { + const lines = [conciseEntry(p)]; + if (p.detail && p.detail.trim()) lines.push(` detail: ${p.detail.trim()}`); + if (p.evidence && p.evidence.trim()) lines.push(` evidence: ${p.evidence.trim()}`); + lines.push(` source: ${p.analyzer_id ?? "?"} · node ${p.source_node_id ?? "?"}`); + return lines.join("\n"); +} + +/** A short, readable session label: cwd (with $HOME → ~), else project, else id. */ +export function sessionLabel(s: { project: string; cwd: string } | undefined, id: string): string { + const home = homedir(); + if (s?.cwd) return s.cwd.startsWith(home) ? `~${s.cwd.slice(home.length)}` : s.cwd; + if (s?.project) return s.project; + return id.slice(0, 8); +} + +export async function prospectProposals(args: string, ctx: ExtensionCommandContext): Promise { + const db = new Database(getDbPath()); + migrate(db); + try { + const { status, full } = parseProposalsArgs(args); + const proposals = listProposals(db, status).sort(rankProposals); + + if (proposals.length === 0) { + output(ctx, status ? `No ${status} proposals found.` : "No proposals found."); + return; + } + + const labels = new Map(getSessionLabels(db).map((s) => [s.id, s])); + + // Group by session. Because `proposals` is already globally ranked by + // confidence, first-seen order puts the session with the strongest single + // recommendation first, and each group stays confidence-ranked within. + const groups = new Map(); + for (const p of proposals) { + const bucket = groups.get(p.session_id); + if (bucket) bucket.push(p); + else groups.set(p.session_id, [p]); + } + + const format = full ? fullEntry : conciseEntry; + const blocks: string[] = []; + for (const [sessionId, group] of groups) { + const label = sessionLabel(labels.get(sessionId), sessionId); + const header = `═══ ${sessionId.slice(0, 8)} · ${label} · ${group.length} proposal(s) ═══`; + blocks.push(`${header}\n${group.map(format).join("\n\n")}`); + } + + const headline = `Proposals (${proposals.length}${status ? `, ${status}` : ""}) in ${groups.size} session(s), ranked by confidence:`; + output(ctx, `${headline}\n\n${blocks.join("\n\n")}`); + } finally { + db.close(); + } +} + +export async function prospectAccept(args: string, ctx: ExtensionCommandContext): Promise { + const id = args?.trim(); + if (!id) { + output(ctx, "Usage: /prospect-accept ", "warning"); + return; + } + const db = new Database(getDbPath()); + migrate(db); + try { + const ok = acceptProposal(db, id); + output(ctx, ok ? `Proposal ${id} applied.` : `Proposal ${id} not found or not open.`, ok ? "info" : "warning"); + } finally { + db.close(); + } +} + +export async function prospectReject(args: string, ctx: ExtensionCommandContext): Promise { + const id = args?.trim(); + if (!id) { + output(ctx, "Usage: /prospect-reject ", "warning"); + return; + } + const db = new Database(getDbPath()); + migrate(db); + try { + const ok = rejectProposal(db, id); + output(ctx, ok ? `Proposal ${id} rejected.` : `Proposal ${id} not found or not open.`, ok ? "info" : "warning"); + } finally { + db.close(); + } +} + export function registerProposalsCommand(pi: ExtensionAPI): void { pi.registerCommand("prospect-proposals", { - description: "List proposals (optionally filter by status: new, accepted, rejected)", - handler: async (args: string, ctx: any) => { - const db = new Database(getDbPath()); - migrate(db); - try { - const status = args?.trim() || undefined; - const proposals = listProposals(db, status); - - if (proposals.length === 0) { - output(ctx, "No proposals found."); - return; - } - - const lines = proposals.map((p) => { - const short = p.id.slice(0, 8); - return `[${p.status}] ${short} | ${p.severity} | ${p.target}\n ${p.summary}`; - }); - output(ctx, `Proposals (${proposals.length}):\n${lines.join("\n")}`); - } finally { - db.close(); - } - }, + description: + "List proposals, ranked by confidence. Optional status filter (open|applied|rejected|duplicate) and --full for evidence/source.", + handler: prospectProposals, }); pi.registerCommand("prospect-accept", { - description: "Accept a proposal by ID", - handler: async (args: string, ctx: any) => { - const id = args?.trim(); - if (!id) { output(ctx, "Usage: /prospect-accept ", "warning"); return; } - const db = new Database(getDbPath()); - migrate(db); - try { - const ok = acceptProposal(db, id); - output(ctx, ok ? `Proposal ${id} accepted.` : `Proposal ${id} not found or not in 'new' status.`, ok ? "info" : "warning"); - } finally { - db.close(); - } - }, + description: "Accept (apply) a proposal by ID", + handler: prospectAccept, }); pi.registerCommand("prospect-reject", { description: "Reject a proposal by ID", - handler: async (args: string, ctx: any) => { - const id = args?.trim(); - if (!id) { output(ctx, "Usage: /prospect-reject ", "warning"); return; } - const db = new Database(getDbPath()); - migrate(db); - try { - const ok = rejectProposal(db, id); - output(ctx, ok ? `Proposal ${id} rejected.` : `Proposal ${id} not found or not in 'new' status.`, ok ? "info" : "warning"); - } finally { - db.close(); - } - }, + handler: prospectReject, }); -} \ No newline at end of file +} diff --git a/src/commands/show.ts b/src/commands/show.ts new file mode 100644 index 0000000..ac5f783 --- /dev/null +++ b/src/commands/show.ts @@ -0,0 +1,235 @@ +import type { ExtensionAPI, ExtensionCommandContext } from "../pi-stubs.js"; +import Database from "better-sqlite3"; +import { migrate } from "../db/schema.js"; +import { getProposal, listProposals, getSessionLabels } from "../db/queries.js"; +import { getNode, getEdgesFrom, getAnchoredMessageIds, getSessionNodes, getSessionMessageRows } from "../db/analysis-queries.js"; +import { EDGE_KINDS, REF_KINDS } from "../analyze/edge-kinds.js"; +import { buildTurnPairs, type TurnPair } from "../analyze/analyzers/turn-pair-core/build.js"; +import { sessionLabel } from "./proposals.js"; +import { getDbPath } from "../config.js"; +import type { Proposal } from "../types.js"; +import type { MessageRow } from "../analyze/types.js"; + +function out(ctx: ExtensionCommandContext, text: string, level: "info" | "warning" | "error" = "info"): void { + ctx.ui.notify(text, level); + console.log(text); +} + +/** Resolve a proposal by exact id or unambiguous id-prefix. */ +export function resolveProposal(db: Database.Database, ref: string): { proposal?: Proposal; matches: Proposal[] } { + const exact = getProposal(db, ref); + if (exact) return { proposal: exact, matches: [exact] }; + const matches = listProposals(db).filter((p) => p.id.startsWith(ref)); + return { proposal: matches.length === 1 ? matches[0] : undefined, matches }; +} + +function truncate(s: string, max: number): string { + const t = s.trim(); + return t.length > max ? `${t.slice(0, max)}…` : t; +} + +/** A compact one-line preview of a tool call's most salient argument. */ +export function toolCallPreview(name: string, args: Record): string { + const pick = (k: string): string | undefined => (typeof args[k] === "string" ? (args[k] as string) : undefined); + const salient = pick("command") ?? pick("cmd") ?? pick("path") ?? pick("file_path") ?? pick("pattern") ?? pick("url") ?? pick("query"); + const arg = salient ? truncate(salient.replace(/\s+/g, " "), 160) : truncate(JSON.stringify(args), 120); + return `${name} ${arg}`; +} + +interface ToolCallRaw { + name?: unknown; + arguments?: unknown; +} + +/** Render the verbatim turns whose user messages are in `anchorIds`, in pair order. */ +export function renderAnchoredTurns( + pairs: TurnPair[], + byId: Map, + anchorIds: Set, + coreByUser: Map>, + llmByUser: Map>, + maxTurns = Infinity, +): string[] { + const lines: string[] = []; + const all = pairs.filter((p) => anchorIds.has(p.userMessageId)).sort((a, b) => a.index - b.index); + const selected = all.slice(0, maxTurns); + for (const pair of selected) { + const core = coreByUser.get(pair.userMessageId); + const llm = llmByUser.get(pair.userMessageId); + const header = [ + `#${pair.index}`, + core ? `friction=${Number(core["friction_score"] ?? 0).toFixed(2)}` : "", + core && core["correction_detected"] ? `correction=${core["correction_type"]}` : "", + core ? `tool_fail=${core["tool_failure_count"]}/${core["tool_call_count"]}` : "", + llm ? `sentiment=${llm["sentiment"]} type=${llm["friction_type"]} sev=${llm["severity"]}` : "", + ] + .filter(Boolean) + .join(" · "); + lines.push(`── pair ${header} ──`); + lines.push("USER:"); + lines.push(indent(truncate(pair.userText || "(empty)", 1400))); + + // Reconstruct assistant text + tool calls (with args) from the turn's raw rows. + const assistantText: string[] = []; + const toolLines: string[] = []; + const errorLines: string[] = []; + for (const mid of pair.messageIds) { + const row = byId.get(mid); + if (!row) continue; + if (row.role === "assistant") { + if (row.content_text) assistantText.push(row.content_text); + for (const call of parseToolCalls(row.tool_calls)) { + const name = typeof call.name === "string" ? call.name : "?"; + const argObj = call.arguments && typeof call.arguments === "object" ? (call.arguments as Record) : {}; + toolLines.push(` ${toolCallPreview(name, argObj)}`); + } + } else if (row.role === "toolResult" && isErrorResult(row.tool_results)) { + errorLines.push(` ✗ ${truncate(row.content_text ?? "(no output)", 200)}`); + } + } + if (assistantText.length > 0) { + lines.push("ASSISTANT:"); + lines.push(indent(truncate(assistantText.join("\n"), 900))); + } + if (toolLines.length > 0) { + lines.push(`TOOLS (${toolLines.length}):`); + lines.push(...toolLines.slice(0, 25)); + if (toolLines.length > 25) lines.push(` …${toolLines.length - 25} more`); + } + if (errorLines.length > 0) { + lines.push("TOOL ERRORS:"); + lines.push(...errorLines.slice(0, 8)); + } + lines.push(""); + } + if (all.length > selected.length) lines.push(`…${all.length - selected.length} more turn(s) not shown.`); + return lines; +} + +function indent(s: string): string { + return s + .split("\n") + .map((l) => ` ${l}`) + .join("\n"); +} + +function parseToolCalls(json: string | null): ToolCallRaw[] { + if (!json) return []; + try { + const arr = JSON.parse(json); + return Array.isArray(arr) ? (arr as ToolCallRaw[]) : []; + } catch { + return []; + } +} + +function isErrorResult(json: string | null): boolean { + if (!json) return false; + try { + const arr = JSON.parse(json) as Array<{ isError?: unknown }>; + return Array.isArray(arr) && arr.some((r) => Boolean(r.isError)); + } catch { + return false; + } +} + +function safeParse(json: string): Record { + try { + return JSON.parse(json) as Record; + } catch { + return {}; + } +} + +export async function prospectShow(args: string, ctx: ExtensionCommandContext): Promise { + const ref = args.trim().split(/\s+/)[0] ?? ""; + if (!ref) { + out(ctx, "Usage: prospect show ", "warning"); + return; + } + const db = new Database(getDbPath()); + migrate(db); + try { + const { proposal, matches } = resolveProposal(db, ref); + if (!proposal) { + if (matches.length === 0) out(ctx, `No proposal matches "${ref}".`, "warning"); + else out(ctx, `"${ref}" is ambiguous (${matches.length} matches): ${matches.map((m) => m.id.slice(0, 8)).join(", ")}`, "warning"); + return; + } + + const labels = new Map(getSessionLabels(db).map((s) => [s.id, s])); + const label = sessionLabel(labels.get(proposal.session_id), proposal.session_id); + const conf = proposal.confidence == null ? "n/a" : `${Math.round(proposal.confidence * 100)}%`; + + const head = [ + `Proposal ${proposal.id.slice(0, 8)} [${conf}] ${proposal.severity} (${proposal.status})`, + ` target: ${proposal.target_path ? `${proposal.target_type} :: ${proposal.target_path}` : proposal.target_type}`, + ` title: ${proposal.title}`, + ` summary: ${proposal.summary}`, + proposal.detail ? ` detail: ${proposal.detail}` : "", + proposal.evidence ? ` evidence: ${proposal.evidence}` : "", + ` session: ${proposal.session_id.slice(0, 8)} · ${label}`, + proposal.source_node_id ? ` source: node ${proposal.source_node_id.slice(0, 8)} (${proposal.analyzer_id ?? "?"})` : "", + ].filter(Boolean); + out(ctx, head.join("\n")); + + const sourceId = proposal.source_node_id; + if (!sourceId || !getNode(db, sourceId)) { + out(ctx, "\n(No source node recorded — cannot reconstruct anchored turns.)", "warning"); + return; + } + + // Walk provenance: summary --consumes--> turn nodes --anchors--> messages. + const consumed = getEdgesFrom(db, sourceId).filter( + (e) => e.edge_kind === EDGE_KINDS.CONSUMES && e.to_ref_kind === REF_KINDS.ANALYSIS_NODE, + ); + const anchorIds = new Set(); + for (const edge of consumed) for (const mid of getAnchoredMessageIds(db, edge.to_ref_id)) anchorIds.add(mid); + + if (anchorIds.size === 0) { + out(ctx, "\n(Source node consumed no turn-anchored evidence.)", "warning"); + return; + } + + // Per-turn deterministic + LLM signals, keyed by anchoring user message. + const coreByUser = new Map>(); + const llmByUser = new Map>(); + for (const n of getSessionNodes(db, proposal.session_id)) { + if (n.analyzer_id === "turn-pair-core") { + const c = safeParse(n.content_json); + if (typeof c["user_message_id"] === "string") coreByUser.set(c["user_message_id"] as string, c); + } else if (n.analyzer_id === "turn-pair-llm") { + const c = safeParse(n.content_json); + if (typeof c["user_message_id"] === "string") llmByUser.set(c["user_message_id"] as string, c); + } + } + + // The overview consumes EVERY turn; focus review on the turns that actually + // carry friction (high-signal core metric, or an LLM classification). + const signalIds = new Set( + [...anchorIds].filter((id) => Boolean(coreByUser.get(id)?.["high_signal"]) || llmByUser.has(id)), + ); + const renderIds = signalIds.size > 0 ? signalIds : anchorIds; + + const messages = getSessionMessageRows(db, proposal.session_id); + const byId = new Map(messages.map((m) => [m.id, m])); + const pairs = buildTurnPairs(messages); + + const noun = signalIds.size > 0 ? "high-signal" : "consumed"; + out( + ctx, + `\nAnchored turns — ${renderIds.size} ${noun} turn(s) of ${anchorIds.size} consumed, the evidence this proposal was synthesised from:\n`, + ); + const body = renderAnchoredTurns(pairs, byId, renderIds, coreByUser, llmByUser, 15); + out(ctx, body.join("\n")); + } finally { + db.close(); + } +} + +export function registerShowCommand(pi: ExtensionAPI): void { + pi.registerCommand("prospect-show", { + description: "Show a proposal with the verbatim anchored turns (user/assistant text + tool calls) it was synthesised from.", + handler: prospectShow, + }); +} diff --git a/src/commands/stats.ts b/src/commands/stats.ts index 05c37b3..2603bf8 100644 --- a/src/commands/stats.ts +++ b/src/commands/stats.ts @@ -1,38 +1,47 @@ -import type { ExtensionAPI } from "../pi-stubs.js"; +import type { ExtensionAPI, ExtensionCommandContext } from "../pi-stubs.js"; import Database from "better-sqlite3"; import { migrate } from "../db/schema.js"; import { getStats } from "../db/queries.js"; import { getDbPath } from "../config.js"; +export async function prospectStats(_args: string, ctx: ExtensionCommandContext): Promise { + const db = new Database(getDbPath()); + migrate(db); + try { + const s = getStats(db); + const kindLines = Object.entries(s.analysis.nodesByKind).map(([k, v]) => ` ${k}: ${v}`); + const lines = [ + "╔══════════════════════════════════════════╗", + "║ ⛏️ Prospector Stats ║", + "╚══════════════════════════════════════════╝", + "", + " ── Sessions ──", + ` Sessions indexed: ${s.totalSessions}`, + ` Messages (user+asst): ${s.totalMessages}`, + ` Tool results: ${s.totalToolResults}`, + ` Sessions analyzed: ${s.sessionsAnalyzed}`, + "", + " ── Proposals ──", + ` open: ${s.proposalsByStatus.open}`, + ` applied: ${s.proposalsByStatus.applied}`, + ` rejected: ${s.proposalsByStatus.rejected}`, + ` duplicate: ${s.proposalsByStatus.duplicate}`, + "", + " ── Analysis graph ──", + ` Nodes: ${s.analysis.nodes} Edges: ${s.analysis.edges} Runs: ${s.analysis.runs}`, + ...(kindLines.length > 0 ? [" Nodes by kind:", ...kindLines] : []), + ]; + const text = lines.join("\n"); + ctx.ui.notify(text, "info"); + console.log(text); + } finally { + db.close(); + } +} + export function registerStatsCommand(pi: ExtensionAPI): void { pi.registerCommand("prospect-stats", { description: "Show prospector database statistics", - handler: async (_args: string, ctx: { ui: { notify: (msg: string, level: string) => void } }) => { - const db = new Database(getDbPath()); - migrate(db); - try { - const s = getStats(db); - const lines = [ - "╔══════════════════════════════════════════╗", - "║ ⛏️ Prospector Stats ║", - "╚══════════════════════════════════════════╝", - "", - ` Sessions indexed: ${s.totalSessions}`, - ` Messages (user+asst):${s.totalMessages}`, - ` Tool results: ${s.totalToolResults}`, - ` Sessions analyzed: ${s.messagesProcessed}`, - "", - " Proposals:", - ` new: ${s.proposalsByStatus.new}`, - ` accepted: ${s.proposalsByStatus.accepted}`, - ` rejected: ${s.proposalsByStatus.rejected}`, - ]; - const text = lines.join("\n"); - ctx.ui.notify(text, "info"); - console.log(text); - } finally { - db.close(); - } - }, + handler: prospectStats, }); -} \ No newline at end of file +} diff --git a/src/commands/sync.ts b/src/commands/sync.ts index f389e6f..efe4018 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -1,36 +1,38 @@ -import type { ExtensionAPI } from "../pi-stubs.js"; +import type { ExtensionAPI, ExtensionCommandContext } from "../pi-stubs.js"; import Database from "better-sqlite3"; import { migrate } from "../db/schema.js"; import { runSync } from "../sync/index.js"; import { getDbPath, getSessionsDir } from "../config.js"; +export async function prospectSync(_args: string, ctx: ExtensionCommandContext): Promise { + const dbPath = getDbPath(); + const db = new Database(dbPath); + migrate(db); + + try { + const result = runSync(db, getSessionsDir()); + const lines = [ + "⛏️ Prospect sync complete", + ` Sessions processed: ${result.sessionsProcessed}`, + ` Sessions skipped: ${result.sessionsSkipped}`, + ` Messages inserted: ${result.messagesInserted}`, + ` Forks resolved: ${result.forksResolved}`, + ]; + if (result.errors.length > 0) { + lines.push(` Errors: ${result.errors.length}`); + for (const e of result.errors.slice(0, 5)) lines.push(` ${e}`); + } + const text = lines.join("\n"); + console.log(text); + ctx.ui.notify(text, "info"); + } finally { + db.close(); + } +} + export function registerSyncCommand(pi: ExtensionAPI): void { pi.registerCommand("prospect-sync", { description: "Index session files into the prospector database (no LLM)", - handler: async (_args: string, ctx: { ui: { notify: (msg: string, level?: string) => void } }) => { - const dbPath = getDbPath(); - const db = new Database(dbPath); - migrate(db); - - try { - const result = runSync(db, getSessionsDir()); - const lines = [ - "⛏️ Prospect sync complete", - ` Sessions processed: ${result.sessionsProcessed}`, - ` Sessions skipped: ${result.sessionsSkipped}`, - ` Messages inserted: ${result.messagesInserted}`, - ` Forks resolved: ${result.forksResolved}`, - ]; - if (result.errors.length > 0) { - lines.push(` Errors: ${result.errors.length}`); - for (const e of result.errors.slice(0, 5)) lines.push(` ${e}`); - } - const text = lines.join("\n"); - console.log(text); - ctx.ui.notify(text, "info"); - } finally { - db.close(); - } - }, + handler: prospectSync, }); } \ No newline at end of file diff --git a/src/commands/tool.ts b/src/commands/tool.ts index cad47a5..dd9d57f 100644 --- a/src/commands/tool.ts +++ b/src/commands/tool.ts @@ -1,4 +1,4 @@ -import type { ExtensionAPI } from "../pi-stubs.js"; +import type { ExtensionAPI, ExtensionCommandContext, ToolResult } from "../pi-stubs.js"; import Database from "better-sqlite3"; import { Type } from "typebox"; import { migrate } from "../db/schema.js"; @@ -6,11 +6,16 @@ import { runSync } from "../sync/index.js"; import { getStats, listProposals, acceptProposal, rejectProposal } from "../db/queries.js"; import { getDbPath, getSessionsDir } from "../config.js"; +function text(body: string, details: unknown): ToolResult { + return { content: [{ type: "text", text: body }], details }; +} + export function registerProspectTool(pi: ExtensionAPI): void { pi.registerTool({ name: "prospect", label: "Prospect", - description: "Index sessions, check stats, list/accept/reject proposals. Actions: sync, stats, list_proposals, accept, reject.", + description: + "Index sessions, check stats, list/accept/reject proposals. Actions: sync, stats, list_proposals, accept, reject.", parameters: Type.Object({ action: Type.Union([ Type.Literal("sync"), @@ -19,42 +24,59 @@ export function registerProspectTool(pi: ExtensionAPI): void { Type.Literal("accept"), Type.Literal("reject"), ]), - status: Type.Optional(Type.Union([Type.Literal("new"), Type.Literal("accepted"), Type.Literal("rejected")])), + status: Type.Optional( + Type.Union([ + Type.Literal("open"), + Type.Literal("applied"), + Type.Literal("rejected"), + Type.Literal("duplicate"), + ]), + ), proposal_id: Type.Optional(Type.String()), }), - async execute(_toolCallId: string, params: Record, _signal: unknown, _onUpdate: unknown, _ctx: unknown) { + async execute( + _toolCallId: string, + params: Record, + _signal: AbortSignal, + _onUpdate: unknown, + _ctx: ExtensionCommandContext, + ): Promise { const db = new Database(getDbPath()); migrate(db); try { switch (params.action) { case "sync": { const result = runSync(db, getSessionsDir()); - return { content: [{ type: "text" as const, text: JSON.stringify(result) }], details: result }; + return text(JSON.stringify(result), result); } case "stats": { const stats = getStats(db); - return { content: [{ type: "text" as const, text: JSON.stringify(stats, null, 2) }], details: stats }; + return text(JSON.stringify(stats, null, 2), stats); } case "list_proposals": { const proposals = listProposals(db, params.status as string | undefined); - if (proposals.length === 0) return { content: [{ type: "text" as const, text: "No proposals found." }], details: [] }; - const text = proposals.map((p) => `[${p.status}] ${p.id.slice(0, 8)} | ${p.severity} | ${p.target}\n ${p.summary}`).join("\n\n"); - return { content: [{ type: "text" as const, text }], details: proposals }; + if (proposals.length === 0) return text("No proposals found.", []); + const body = proposals + .map((p) => `[${p.status}] ${p.id.slice(0, 8)} | ${p.severity} | ${p.target_type}\n ${p.title}`) + .join("\n\n"); + return text(body, proposals); } case "accept": { - if (!params.proposal_id) return { content: [{ type: "text" as const, text: "proposal_id required" }], details: {} }; + if (!params.proposal_id) return text("proposal_id required", {}); const ok = acceptProposal(db, params.proposal_id as string); - return { content: [{ type: "text" as const, text: ok ? `Accepted ${params.proposal_id}` : "Not found or not new" }], details: { ok } }; + return text(ok ? `Applied ${params.proposal_id}` : "Not found or not open", { ok }); } case "reject": { - if (!params.proposal_id) return { content: [{ type: "text" as const, text: "proposal_id required" }], details: {} }; + if (!params.proposal_id) return text("proposal_id required", {}); const ok = rejectProposal(db, params.proposal_id as string); - return { content: [{ type: "text" as const, text: ok ? `Rejected ${params.proposal_id}` : "Not found or not new" }], details: { ok } }; + return text(ok ? `Rejected ${params.proposal_id}` : "Not found or not open", { ok }); } + default: + return text(`Unknown action: ${String(params.action)}`, {}); } } finally { db.close(); } }, }); -} \ No newline at end of file +} diff --git a/src/commands/verify.ts b/src/commands/verify.ts new file mode 100644 index 0000000..b5af7f0 --- /dev/null +++ b/src/commands/verify.ts @@ -0,0 +1,75 @@ +import type { ExtensionAPI, ExtensionCommandContext } from "../pi-stubs.js"; +import Database from "better-sqlite3"; +import { migrate } from "../db/schema.js"; +import { getAllAnalysisNodes } from "../db/analysis-queries.js"; +import { computeOutputKey } from "../analyze/input-hash.js"; +import { getDbPath } from "../config.js"; + +function output(ctx: ExtensionCommandContext, text: string, level: "info" | "warning" | "error" = "info"): void { + ctx.ui.notify(text, level); + console.log(text); +} + +export interface VerifyMismatch { + id: string; + analyzerId: string; + stored: string; + recomputed: string; +} + +/** + * Recompute every node's `output_key` from its stored `(input_key, content)` and + * confirm it matches. Because identities are content-addressed, any drift means + * the content was altered out of band or the stored key is stale. Pure read. + */ +export function verifyNodes(db: Database.Database): { total: number; mismatches: VerifyMismatch[] } { + const nodes = getAllAnalysisNodes(db); + const mismatches: VerifyMismatch[] = []; + for (const n of nodes) { + let content: unknown; + try { + content = JSON.parse(n.content_json); + } catch { + mismatches.push({ id: n.id, analyzerId: n.analyzer_id, stored: n.output_key, recomputed: "" }); + continue; + } + const recomputed = computeOutputKey(n.input_key, content); + if (recomputed !== n.output_key) { + mismatches.push({ id: n.id, analyzerId: n.analyzer_id, stored: n.output_key, recomputed }); + } + } + return { total: nodes.length, mismatches }; +} + +export async function prospectVerify(_args: string, ctx: ExtensionCommandContext): Promise { + const db = new Database(getDbPath()); + migrate(db); + try { + const { total, mismatches } = verifyNodes(db); + if (total === 0) { + output(ctx, "No analysis nodes to verify."); + return; + } + if (mismatches.length === 0) { + output(ctx, `✓ ${total} node(s) verified: every output_key is consistent with its content.`); + return; + } + const lines = mismatches + .slice(0, 50) + .map((m) => ` ${m.id.slice(0, 8)} ${m.analyzerId} stored=${m.stored} recomputed=${m.recomputed}`); + output( + ctx, + `✗ ${mismatches.length} of ${total} node(s) failed verification (content does not match output_key):\n${lines.join("\n")}`, + "error", + ); + } finally { + db.close(); + } +} + +export function registerVerifyCommand(pi: ExtensionAPI): void { + pi.registerCommand("prospect-verify", { + description: "Verify analysis-graph integrity: recompute each node's content-addressed output_key and confirm it matches.", + handler: prospectVerify, + }); +} diff --git a/src/config.ts b/src/config.ts index 511f21c..c6027e9 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2,14 +2,21 @@ import * as fs from "node:fs"; import * as path from "node:path"; import * as os from "node:os"; import type { ProspectorConfig } from "./types.js"; +import { DEFAULT_MODEL_TIERS } from "./analyze/model-tiers.js"; +import type { ModelTierConfig } from "./analyze/types.js"; -const CONFIG_PATH = path.join(os.homedir(), ".pi", "agent", "prospector.json"); +const DEFAULT_CONFIG_PATH = path.join(os.homedir(), ".pi", "agent", "prospector.json"); const DEFAULT_DB_PATH = path.join(os.homedir(), ".pi", "agent", "prospector.db"); -const SESSIONS_DIR = path.join(os.homedir(), ".pi", "agent", "sessions"); +const DEFAULT_SESSIONS_DIR = path.join(os.homedir(), ".pi", "agent", "sessions"); + +/** Path to the JSON config, overridable via PROSPECTOR_CONFIG (used by tests). */ +function configPath(): string { + return process.env["PROSPECTOR_CONFIG"] ?? DEFAULT_CONFIG_PATH; +} export function loadConfig(): ProspectorConfig { try { - const raw = fs.readFileSync(CONFIG_PATH, "utf-8"); + const raw = fs.readFileSync(configPath(), "utf-8"); return JSON.parse(raw) as ProspectorConfig; } catch { return {}; @@ -19,9 +26,17 @@ export function loadConfig(): ProspectorConfig { export function getDbPath(config?: ProspectorConfig): string { const c = config ?? loadConfig(); if (c.dbPath) return c.dbPath.replace(/^~/, os.homedir()); + if (process.env["PROSPECTOR_DB_PATH"]) return process.env["PROSPECTOR_DB_PATH"]!; return DEFAULT_DB_PATH; } export function getSessionsDir(): string { - return SESSIONS_DIR; + return process.env["PROSPECTOR_SESSIONS_DIR"] ?? DEFAULT_SESSIONS_DIR; +} + +/** Resolve the model-tier mapping, falling back to defaults. */ +export function getModelTiers(config?: ProspectorConfig): ModelTierConfig { + const c = config ?? loadConfig(); + if (c.modelTiers) return c.modelTiers; + return DEFAULT_MODEL_TIERS; } \ No newline at end of file diff --git a/src/db/analysis-queries.ts b/src/db/analysis-queries.ts new file mode 100644 index 0000000..cb58e9a --- /dev/null +++ b/src/db/analysis-queries.ts @@ -0,0 +1,376 @@ +/** + * Data access for the analysis graph: analyzer registry, nodes, edges, runs, + * and lineage navigation. + * + * All SQL for the analysis graph lives here. Row → camelCase mapping for + * framework consumers is done by the framework; these functions return raw + * rows (snake_case) typed by the schemas in `../analyze/types.ts`. + */ + +import type Database from "better-sqlite3"; +import type { + AnalysisEdgeRow, + AnalysisNodeRow, + AnalysisRunRow, + AnalyzerConfig, + AnalyzerDef, + AnalyzerVersion, + MessageRow, + PromptVersion, +} from "../analyze/types.js"; +import { computeConfigHash, uuidv7 } from "../analyze/input-hash.js"; +import { versionIdOf } from "../analyze/version.js"; +import { EDGE_KINDS, REF_KINDS } from "../analyze/edge-kinds.js"; + +// ───────────────────────── analyzer registry ───────────────────────── + +export function upsertAnalyzerDef(db: Database.Database, def: AnalyzerDef): void { + db.prepare(` + INSERT INTO analyzer_defs (id, label, description, anchor_span, dependencies, created_at) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(id) DO UPDATE SET + label = excluded.label, + description = excluded.description, + anchor_span = excluded.anchor_span, + dependencies = excluded.dependencies + `).run( + def.id, + def.label, + def.description, + def.anchorSpan, + JSON.stringify(def.dependencies), + new Date().toISOString(), + ); +} + +export function upsertAnalyzerVersion(db: Database.Database, version: AnalyzerVersion): void { + db.prepare(` + INSERT INTO analyzer_versions (analyzer_id, version_id, implementation_kind, code_ref, created_at) + VALUES (?, ?, ?, ?, ?) + ON CONFLICT(analyzer_id, version_id) DO NOTHING + `).run( + version.analyzerId, + versionIdOf(version), + version.implementationKind, + version.codeRef ?? null, + new Date().toISOString(), + ); +} + +export function registerPrompt(db: Database.Database, prompt: PromptVersion): void { + db.prepare(` + INSERT INTO prompt_registry (hash, content, role, created_at) + VALUES (?, ?, ?, ?) + ON CONFLICT(hash) DO NOTHING + `).run(prompt.hash, prompt.content, prompt.role ?? null, new Date().toISOString()); +} + +/** + * Resolve (and persist if new) an analyzer config. Configs are content-addressed + * by a hash of their canonical JSON; identical configs share one row and id. + */ +export function resolveConfig( + db: Database.Database, + params: { analyzerId: string; configJson: Record; label?: string }, +): AnalyzerConfig { + const configHash = computeConfigHash(params.configJson); + const existing = db + .prepare("SELECT id, analyzer_id, config_hash, config_json, label FROM analyzer_configs WHERE config_hash = ?") + .get(configHash) as + | { id: string; analyzer_id: string; config_hash: string; config_json: string; label: string | null } + | undefined; + + if (existing) { + return { + id: existing.id, + analyzerId: existing.analyzer_id, + configHash: existing.config_hash, + configJson: JSON.parse(existing.config_json) as Record, + label: existing.label ?? undefined, + }; + } + + const id = uuidv7(); + db.prepare(` + INSERT INTO analyzer_configs (id, analyzer_id, config_hash, config_json, label, created_at) + VALUES (?, ?, ?, ?, ?, ?) + `).run(id, params.analyzerId, configHash, JSON.stringify(params.configJson), params.label ?? null, new Date().toISOString()); + + return { + id, + analyzerId: params.analyzerId, + configHash, + configJson: params.configJson, + label: params.label, + }; +} + +// ───────────────────────── runs ───────────────────────── + +export function createRun( + db: Database.Database, + params: { + id: string; + analyzerId: string; + analyzerVersionId: string; + configId: string; + sessionId: string; + mode: string; + promptBundleHash: string; + modelSpec?: string; + }, +): void { + db.prepare(` + INSERT INTO analysis_runs + (id, analyzer_id, analyzer_version_id, config_id, session_id, mode, status, prompt_bundle_hash, model_spec, started_at) + VALUES (?, ?, ?, ?, ?, ?, 'ok', ?, ?, ?) + `).run( + params.id, + params.analyzerId, + params.analyzerVersionId, + params.configId, + params.sessionId, + params.mode, + params.promptBundleHash, + params.modelSpec ?? null, + new Date().toISOString(), + ); +} + +export function finishRun( + db: Database.Database, + runId: string, + fields: { + status: string; + nodesProduced: number; + nodesSkipped: number; + costUsd: number; + tokensUsed: number; + errorMessage?: string | null; + }, +): void { + db.prepare(` + UPDATE analysis_runs SET + status = ?, finished_at = ?, nodes_produced = ?, nodes_skipped = ?, + cost_usd = ?, tokens_used = ?, error_message = ? + WHERE id = ? + `).run( + fields.status, + new Date().toISOString(), + fields.nodesProduced, + fields.nodesSkipped, + fields.costUsd, + fields.tokensUsed, + fields.errorMessage ?? null, + runId, + ); +} + +export function getRun(db: Database.Database, runId: string): AnalysisRunRow | undefined { + return db.prepare("SELECT * FROM analysis_runs WHERE id = ?").get(runId) as AnalysisRunRow | undefined; +} + +// ───────────────────────── nodes ───────────────────────── + +export function insertNode( + db: Database.Database, + node: { + id: string; + sessionId: string; + analyzerId: string; + analyzerVersionId: string; + configId: string; + runId: string | null; + nodeKind: string; + contentJson: string; + sourceSetHash: string; + inputKey: string; + outputKey: string; + configFingerprint?: string; + modelUsed?: string | null; + costUsd?: number | null; + tokensUsed?: number | null; + durationMs?: number | null; + createdAt: string; + }, +): void { + db.prepare(` + INSERT INTO analysis_nodes + (id, session_id, analyzer_id, analyzer_version_id, config_id, run_id, node_kind, + content_json, source_set_hash, input_key, output_key, config_fingerprint, model_used, cost_usd, tokens_used, duration_ms, created_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `).run( + node.id, + node.sessionId, + node.analyzerId, + node.analyzerVersionId, + node.configId, + node.runId, + node.nodeKind, + node.contentJson, + node.sourceSetHash, + node.inputKey, + node.outputKey, + node.configFingerprint ?? "", + node.modelUsed ?? null, + node.costUsd ?? null, + node.tokensUsed ?? null, + node.durationMs ?? null, + node.createdAt, + ); +} + +export function getNode(db: Database.Database, id: string): AnalysisNodeRow | undefined { + return db.prepare("SELECT * FROM analysis_nodes WHERE id = ?").get(id) as AnalysisNodeRow | undefined; +} + +/** Idempotency lookup: a node produced by an exact recipe over an exact source set. */ +export function findNodeByInputKey(db: Database.Database, inputKey: string): AnalysisNodeRow | undefined { + return db.prepare("SELECT * FROM analysis_nodes WHERE input_key = ?").get(inputKey) as AnalysisNodeRow | undefined; +} + +/** + * The newest node for a logical unit = (analyzer, source set), regardless of + * version/config. Used to detect `stale` units (a node exists, but from an + * older recipe) and to wire the `revises` lineage edge. + */ +export function findLatestNodeBySourceSet( + db: Database.Database, + analyzerId: string, + sourceSetHash: string, +): AnalysisNodeRow | undefined { + return db + .prepare( + "SELECT * FROM analysis_nodes WHERE analyzer_id = ? AND source_set_hash = ? AND node_kind != 'error' ORDER BY created_at DESC, rowid DESC LIMIT 1", + ) + .get(analyzerId, sourceSetHash) as AnalysisNodeRow | undefined; +} + +export function getSessionNodes(db: Database.Database, sessionId: string): AnalysisNodeRow[] { + return db.prepare("SELECT * FROM analysis_nodes WHERE session_id = ? ORDER BY created_at ASC, rowid ASC").all(sessionId) as AnalysisNodeRow[]; +} + +/** Every analysis node, for integrity verification. */ +export function getAllAnalysisNodes(db: Database.Database): AnalysisNodeRow[] { + return db.prepare("SELECT * FROM analysis_nodes ORDER BY created_at ASC, rowid ASC").all() as AnalysisNodeRow[]; +} + +/** A session's messages in stream order — for reconstructing turns verbatim. */ +export function getSessionMessageRows(db: Database.Database, sessionId: string): MessageRow[] { + return db + .prepare( + "SELECT id, session_id, parent_id, timestamp, role, content_text, content_thinking, tool_calls, tool_results " + + "FROM messages WHERE session_id = ? ORDER BY rowid ASC", + ) + .all(sessionId) as MessageRow[]; +} + +export function getNodesByAnalyzer(db: Database.Database, analyzerId: string, sessionId: string): AnalysisNodeRow[] { + return db + .prepare("SELECT * FROM analysis_nodes WHERE analyzer_id = ? AND session_id = ? ORDER BY created_at ASC, rowid ASC") + .all(analyzerId, sessionId) as AnalysisNodeRow[]; +} + +// ───────────────────────── edges ───────────────────────── + +export function insertEdge( + db: Database.Database, + edge: { fromNodeId: string; toRefKind: string; toRefId: string; edgeKind: string; ordinal: number }, +): void { + db.prepare(` + INSERT INTO analysis_edges (id, from_node_id, to_ref_kind, to_ref_id, edge_kind, ordinal) + VALUES (?, ?, ?, ?, ?, ?) + `).run(uuidv7(), edge.fromNodeId, edge.toRefKind, edge.toRefId, edge.edgeKind, edge.ordinal); +} + +export function getEdgesFrom(db: Database.Database, nodeId: string): AnalysisEdgeRow[] { + return db.prepare("SELECT * FROM analysis_edges WHERE from_node_id = ? ORDER BY ordinal ASC").all(nodeId) as AnalysisEdgeRow[]; +} + +export function getEdgesTo(db: Database.Database, toRefId: string, edgeKind?: string): AnalysisEdgeRow[] { + if (edgeKind) { + return db + .prepare("SELECT * FROM analysis_edges WHERE to_ref_id = ? AND edge_kind = ?") + .all(toRefId, edgeKind) as AnalysisEdgeRow[]; + } + return db.prepare("SELECT * FROM analysis_edges WHERE to_ref_id = ?").all(toRefId) as AnalysisEdgeRow[]; +} + +/** Message ids that a node anchors to (via `anchors` edges with message targets). */ +export function getAnchoredMessageIds(db: Database.Database, nodeId: string): string[] { + const rows = db + .prepare("SELECT to_ref_id FROM analysis_edges WHERE from_node_id = ? AND edge_kind = ? AND to_ref_kind = ?") + .all(nodeId, EDGE_KINDS.ANCHORS, REF_KINDS.MESSAGE) as Array<{ to_ref_id: string }>; + return rows.map((r) => r.to_ref_id); +} + +export function getMessage(db: Database.Database, id: string): MessageRow | undefined { + return db + .prepare( + "SELECT id, session_id, parent_id, timestamp, role, content_text, content_thinking, tool_calls, tool_results FROM messages WHERE id = ?", + ) + .get(id) as MessageRow | undefined; +} + +// ───────────────────────── lineage navigation ───────────────────────── + +/** + * All version-alternatives for a logical unit, oldest → newest. These are the + * nodes that sit "at the same level" of the graph; their `created_at` and + * `analyzer_version_id` distinguish the alternatives. + */ +export function getNodeVersions( + db: Database.Database, + analyzerId: string, + sourceSetHash: string, +): AnalysisNodeRow[] { + return db + .prepare( + "SELECT * FROM analysis_nodes WHERE analyzer_id = ? AND source_set_hash = ? ORDER BY created_at ASC, rowid ASC", + ) + .all(analyzerId, sourceSetHash) as AnalysisNodeRow[]; +} + +/** The node that `nodeId` revises (its immediate older-version predecessor), if any. */ +export function getRevisedNode(db: Database.Database, nodeId: string): AnalysisNodeRow | undefined { + const edge = db + .prepare("SELECT to_ref_id FROM analysis_edges WHERE from_node_id = ? AND edge_kind = ? LIMIT 1") + .get(nodeId, EDGE_KINDS.REVISES) as { to_ref_id: string } | undefined; + if (!edge) return undefined; + return getNode(db, edge.to_ref_id); +} + +/** Nodes that revise `nodeId` (its newer-version successors), if any. */ +export function getRevisions(db: Database.Database, nodeId: string): AnalysisNodeRow[] { + const edges = db + .prepare("SELECT from_node_id FROM analysis_edges WHERE to_ref_id = ? AND edge_kind = ?") + .all(nodeId, EDGE_KINDS.REVISES) as Array<{ from_node_id: string }>; + const out: AnalysisNodeRow[] = []; + for (const e of edges) { + const n = getNode(db, e.from_node_id); + if (n) out.push(n); + } + return out; +} + +// ───────────────────────── analysis stats ───────────────────────── + +export interface AnalysisStats { + nodes: number; + edges: number; + runs: number; + nodesByKind: Record; +} + +export function getAnalysisStats(db: Database.Database): AnalysisStats { + const nodes = (db.prepare("SELECT COUNT(*) AS c FROM analysis_nodes").get() as { c: number }).c; + const edges = (db.prepare("SELECT COUNT(*) AS c FROM analysis_edges").get() as { c: number }).c; + const runs = (db.prepare("SELECT COUNT(*) AS c FROM analysis_runs").get() as { c: number }).c; + const kindRows = db.prepare("SELECT node_kind, COUNT(*) AS c FROM analysis_nodes GROUP BY node_kind").all() as Array<{ + node_kind: string; + c: number; + }>; + const nodesByKind: Record = {}; + for (const r of kindRows) nodesByKind[r.node_kind] = r.c; + return { nodes, edges, runs, nodesByKind }; +} diff --git a/src/db/queries.ts b/src/db/queries.ts index 01129b6..b46d181 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -1,6 +1,6 @@ import Database from "better-sqlite3"; -import { createHash } from "node:crypto"; -import type { Proposal, Stats } from "../types.js"; +import type { Proposal, ProposalStatus, Stats } from "../types.js"; +import { getAnalysisStats } from "./analysis-queries.js"; // ── Sessions ── @@ -53,6 +53,25 @@ export function getUnanalyzedSessions(db: Database.Database, limit?: number): Ar return (limit ? db.prepare(sql).all(limit) : db.prepare(sql).all()) as Array<{ id: string; file_path: string; started_at: string }>; } +export function getAllSessions(db: Database.Database, limit?: number): Array<{ id: string; file_path: string; started_at: string }> { + const sql = limit + ? "SELECT id, file_path, started_at FROM sessions ORDER BY started_at ASC LIMIT ?" + : "SELECT id, file_path, started_at FROM sessions ORDER BY started_at ASC"; + return (limit ? db.prepare(sql).all(limit) : db.prepare(sql).all()) as Array<{ id: string; file_path: string; started_at: string }>; +} + +export interface SessionLabel { + id: string; + project: string; + cwd: string; + message_count: number; +} + +/** Lightweight labels (project/cwd/message_count) for every session, for display. */ +export function getSessionLabels(db: Database.Database): SessionLabel[] { + return db.prepare("SELECT id, project, cwd, message_count FROM sessions").all() as SessionLabel[]; +} + // ── Messages ── export interface MessageInsert { @@ -82,31 +101,27 @@ export function getSessionMessages(db: Database.Database, sessionId: string): Ar return db.prepare("SELECT role, content_text, content_thinking, tool_calls, timestamp FROM messages WHERE session_id = ? ORDER BY rowid ASC").all(sessionId) as any[]; } -// ── Proposals ── - -export function insertProposal(db: Database.Database, p: Proposal): string { - db.prepare(` - INSERT OR IGNORE INTO proposals (id, created_at, session_id, target, severity, summary, detail, evidence, status, dedup_hash) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - `).run(p.id, p.created_at, p.session_id, p.target, p.severity, p.summary, p.detail, p.evidence, p.status, p.dedup_hash); - return p.id; -} +// ── Proposals (v2) ── export function listProposals(db: Database.Database, status?: string): Proposal[] { if (status) return db.prepare("SELECT * FROM proposals WHERE status = ? ORDER BY created_at DESC").all(status) as Proposal[]; return db.prepare("SELECT * FROM proposals ORDER BY created_at DESC").all() as Proposal[]; } -export function acceptProposal(db: Database.Database, id: string): boolean { - return db.prepare("UPDATE proposals SET status = 'accepted' WHERE id = ? AND status = 'new'").run(id).changes > 0; +export function getProposal(db: Database.Database, id: string): Proposal | undefined { + return db.prepare("SELECT * FROM proposals WHERE id = ?").get(id) as Proposal | undefined; } -export function rejectProposal(db: Database.Database, id: string): boolean { - return db.prepare("UPDATE proposals SET status = 'rejected' WHERE id = ? AND status = 'new'").run(id).changes > 0; +export function acceptProposal(db: Database.Database, id: string): boolean { + return db + .prepare("UPDATE proposals SET status = 'applied', updated_at = ? WHERE id = ? AND status = 'open'") + .run(new Date().toISOString(), id).changes > 0; } -export function computeDedupHash(target: string, severity: string, summary: string): string { - return createHash("sha256").update(`${target}|${severity}|${summary}`).digest("hex").slice(0, 16); +export function rejectProposal(db: Database.Database, id: string): boolean { + return db + .prepare("UPDATE proposals SET status = 'rejected', updated_at = ? WHERE id = ? AND status = 'open'") + .run(new Date().toISOString(), id).changes > 0; } // ── Stats ── @@ -115,9 +130,22 @@ export function getStats(db: Database.Database): Stats { const totalSessions = (db.prepare("SELECT COUNT(*) as c FROM sessions").get() as { c: number }).c; const totalMessages = (db.prepare("SELECT COUNT(*) as c FROM messages WHERE role IN ('user','assistant')").get() as { c: number }).c; const totalToolResults = (db.prepare("SELECT COUNT(*) as c FROM messages WHERE role = 'toolResult'").get() as { c: number }).c; - const messagesProcessed = (db.prepare("SELECT SUM(message_count) as c FROM sessions WHERE analyzed_at IS NOT NULL").get() as { c: number | null }).c ?? 0; - const pNew = (db.prepare("SELECT COUNT(*) as c FROM proposals WHERE status = 'new'").get() as { c: number }).c; - const pAccepted = (db.prepare("SELECT COUNT(*) as c FROM proposals WHERE status = 'accepted'").get() as { c: number }).c; - const pRejected = (db.prepare("SELECT COUNT(*) as c FROM proposals WHERE status = 'rejected'").get() as { c: number }).c; - return { totalSessions, totalMessages, totalToolResults, messagesProcessed, proposalsByStatus: { new: pNew, accepted: pAccepted, rejected: pRejected } }; + const sessionsAnalyzed = (db.prepare("SELECT COUNT(*) as c FROM sessions WHERE analyzed_at IS NOT NULL").get() as { c: number }).c; + + const statusRows = db.prepare("SELECT status, COUNT(*) as c FROM proposals GROUP BY status").all() as Array<{ status: string; c: number }>; + const proposalsByStatus: Record = { open: 0, applied: 0, rejected: 0, duplicate: 0 }; + for (const r of statusRows) { + if (r.status === "open" || r.status === "applied" || r.status === "rejected" || r.status === "duplicate") { + proposalsByStatus[r.status] = r.c; + } + } + + return { + totalSessions, + totalMessages, + totalToolResults, + sessionsAnalyzed, + proposalsByStatus, + analysis: getAnalysisStats(db), + }; } \ No newline at end of file diff --git a/src/db/schema.ts b/src/db/schema.ts index efa808f..6930244 100644 --- a/src/db/schema.ts +++ b/src/db/schema.ts @@ -1,7 +1,29 @@ import Database from "better-sqlite3"; +/** + * Schema for pi-prospector. + * + * A single, clean migration creates everything in its final form — there is no + * incremental ALTER/backfill machinery, because the database is disposable and + * always rebuilt from session transcripts (`/prospect-sync`). + * + * Tables: + * sessions, messages, messages_fts — the read-only session index + * proposals — materialised, user-reviewable proposals + * analyzer_defs / _versions — analyzer identity and code releases + * prompt_registry — content-addressed prompt store + * analyzer_configs — content-addressed config store + * analysis_runs — execution provenance (informational) + * analysis_nodes — append-only analysis artifacts + * analysis_edges — the typed relationship graph + */ export function migrate(db: Database.Database): void { + db.pragma("journal_mode = WAL"); + db.pragma("foreign_keys = ON"); + db.exec(` + -- ───────────────────────── session index ───────────────────────── + CREATE TABLE IF NOT EXISTS sessions ( id TEXT PRIMARY KEY, file_path TEXT NOT NULL, @@ -30,26 +52,140 @@ export function migrate(db: Database.Database): void { FOREIGN KEY (session_id) REFERENCES sessions(id) ); + -- ───────────────────────── proposals (v2) ───────────────────────── + CREATE TABLE IF NOT EXISTS proposals ( id TEXT PRIMARY KEY, created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, session_id TEXT NOT NULL, - target TEXT NOT NULL, + source_node_id TEXT, + analyzer_id TEXT, + target_type TEXT NOT NULL, + target_path TEXT, + title TEXT NOT NULL, severity TEXT NOT NULL, summary TEXT NOT NULL, detail TEXT, evidence TEXT, - status TEXT NOT NULL DEFAULT 'new', - dedup_hash TEXT, + confidence REAL, + status TEXT NOT NULL DEFAULT 'open', + input_key TEXT NOT NULL, -- content-addressed: H(source output_key | ordinal) FOREIGN KEY (session_id) REFERENCES sessions(id) ); + -- ──────────────────── analyzer identity & recipe ──────────────────── + + CREATE TABLE IF NOT EXISTS analyzer_defs ( + id TEXT PRIMARY KEY, + label TEXT NOT NULL, + description TEXT NOT NULL DEFAULT '', + anchor_span TEXT NOT NULL, + dependencies TEXT NOT NULL DEFAULT '[]', + created_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS analyzer_versions ( + analyzer_id TEXT NOT NULL, + version_id TEXT NOT NULL, + implementation_kind TEXT NOT NULL, + code_ref TEXT, + created_at TEXT NOT NULL, + PRIMARY KEY (analyzer_id, version_id), + FOREIGN KEY (analyzer_id) REFERENCES analyzer_defs(id) + ); + + CREATE TABLE IF NOT EXISTS prompt_registry ( + hash TEXT PRIMARY KEY, + content TEXT NOT NULL, + role TEXT, + created_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS analyzer_configs ( + id TEXT PRIMARY KEY, + analyzer_id TEXT NOT NULL, + config_hash TEXT NOT NULL UNIQUE, + config_json TEXT NOT NULL, + label TEXT, + created_at TEXT NOT NULL, + FOREIGN KEY (analyzer_id) REFERENCES analyzer_defs(id) + ); + + -- ──────────────────── analysis graph (append-only) ──────────────────── + + CREATE TABLE IF NOT EXISTS analysis_runs ( + id TEXT PRIMARY KEY, + analyzer_id TEXT NOT NULL, + analyzer_version_id TEXT NOT NULL, + config_id TEXT NOT NULL, + session_id TEXT NOT NULL, + mode TEXT NOT NULL DEFAULT 'fill', + status TEXT NOT NULL DEFAULT 'ok', + prompt_bundle_hash TEXT NOT NULL DEFAULT '', + model_spec TEXT, + started_at TEXT NOT NULL, + finished_at TEXT, + nodes_produced INTEGER NOT NULL DEFAULT 0, + nodes_skipped INTEGER NOT NULL DEFAULT 0, + cost_usd REAL NOT NULL DEFAULT 0, + tokens_used INTEGER NOT NULL DEFAULT 0, + error_message TEXT + ); + + CREATE TABLE IF NOT EXISTS analysis_nodes ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + analyzer_id TEXT NOT NULL, + analyzer_version_id TEXT NOT NULL, + config_id TEXT NOT NULL, + run_id TEXT, + node_kind TEXT NOT NULL, + content_json TEXT NOT NULL, + source_set_hash TEXT NOT NULL, + input_key TEXT NOT NULL UNIQUE, + output_key TEXT NOT NULL DEFAULT '', + config_fingerprint TEXT NOT NULL DEFAULT '', + model_used TEXT, + cost_usd REAL, + tokens_used INTEGER, + duration_ms INTEGER, + created_at TEXT NOT NULL, + FOREIGN KEY (session_id) REFERENCES sessions(id) + ); + + CREATE TABLE IF NOT EXISTS analysis_edges ( + id TEXT PRIMARY KEY, + from_node_id TEXT NOT NULL, + to_ref_kind TEXT NOT NULL, + to_ref_id TEXT NOT NULL, + edge_kind TEXT NOT NULL, + ordinal INTEGER NOT NULL DEFAULT 0, + FOREIGN KEY (from_node_id) REFERENCES analysis_nodes(id) + ); + + -- ───────────────────────────── indexes ───────────────────────────── + CREATE INDEX IF NOT EXISTS idx_messages_session ON messages(session_id); CREATE INDEX IF NOT EXISTS idx_messages_role ON messages(role); + CREATE INDEX IF NOT EXISTS idx_sessions_file ON sessions(file_path); + CREATE INDEX IF NOT EXISTS idx_proposals_session ON proposals(session_id); CREATE INDEX IF NOT EXISTS idx_proposals_status ON proposals(status); - CREATE INDEX IF NOT EXISTS idx_proposals_dedup ON proposals(dedup_hash); - CREATE INDEX IF NOT EXISTS idx_sessions_file ON sessions(file_path); + CREATE INDEX IF NOT EXISTS idx_proposals_dedup ON proposals(input_key); + + -- Group nodes into logical units (analyzer + source set) for the + -- version-alternative timeline, and look up by recipe identity. + CREATE INDEX IF NOT EXISTS idx_nodes_unit ON analysis_nodes(analyzer_id, source_set_hash); + CREATE INDEX IF NOT EXISTS idx_nodes_output ON analysis_nodes(output_key); + CREATE INDEX IF NOT EXISTS idx_nodes_session ON analysis_nodes(session_id); + CREATE INDEX IF NOT EXISTS idx_nodes_analyzer ON analysis_nodes(analyzer_id); + + CREATE INDEX IF NOT EXISTS idx_edges_from ON analysis_edges(from_node_id); + CREATE INDEX IF NOT EXISTS idx_edges_to ON analysis_edges(to_ref_id, edge_kind); + CREATE INDEX IF NOT EXISTS idx_edges_kind ON analysis_edges(edge_kind); + + -- ──────────────────────────── full text ──────────────────────────── DROP TABLE IF EXISTS messages_fts; CREATE VIRTUAL TABLE IF NOT EXISTS messages_fts USING fts5( @@ -71,4 +207,4 @@ export function migrate(db: Database.Database): void { VALUES ('delete', OLD.rowid, OLD.content_text, OLD.content_thinking); END; `); -} \ No newline at end of file +} diff --git a/src/index.ts b/src/index.ts index b551ed0..a5128cc 100644 --- a/src/index.ts +++ b/src/index.ts @@ -3,12 +3,18 @@ import { registerSyncCommand } from "./commands/sync.js"; import { registerStatsCommand } from "./commands/stats.js"; import { registerProposalsCommand } from "./commands/proposals.js"; import { registerAnalyzeCommand } from "./commands/analyze.js"; +import { registerVerifyCommand } from "./commands/verify.js"; +import { registerShowCommand } from "./commands/show.js"; import { registerProspectTool } from "./commands/tool.js"; +import { registerHeadlessFlag } from "./commands/headless.js"; export default function (pi: ExtensionAPI) { registerSyncCommand(pi); registerStatsCommand(pi); registerProposalsCommand(pi); registerAnalyzeCommand(pi); + registerVerifyCommand(pi); + registerShowCommand(pi); registerProspectTool(pi); + registerHeadlessFlag(pi); } \ No newline at end of file diff --git a/src/pi-stubs.ts b/src/pi-stubs.ts index 3e100ef..207e1ca 100644 --- a/src/pi-stubs.ts +++ b/src/pi-stubs.ts @@ -1,28 +1,165 @@ /** - * Local type stubs for @earendil-works/pi-coding-agent. + * Local type stubs for the Pi host packages. * - * The real package is a private peer dependency not available in CI. - * These stubs let us compile without it. At runtime, Pi provides the real types. + * The real `@earendil-works/pi-coding-agent` and `@earendil-works/pi-ai` + * packages are optional peer dependencies that are not installed in CI. These + * stubs describe just enough of their surface for this extension to compile and + * type-check. At runtime inside Pi, the real implementations are used. */ +// ───────────────────────── pi-coding-agent ───────────────────────── + +export interface PiModel { + id: string; + provider: string; + name?: string; +} + +export type ResolvedRequestAuth = + | { ok: true; apiKey?: string; headers?: Record } + | { ok: false; error: string }; + +export interface ModelRegistry { + find(provider: string, modelId: string): PiModel | undefined; + getAll(): PiModel[]; + getAvailable(): PiModel[]; + getApiKeyAndHeaders(model: PiModel): Promise; +} + export interface ExtensionUIContext { notify: (message: string, level?: string) => void; + select?: (options: unknown) => Promise; + confirm?: (options: unknown) => Promise; + input?: (options: unknown) => Promise; + setStatus?: (text: string) => void; +} + +export interface ExtensionContext { + modelRegistry: ModelRegistry; + model?: PiModel; + signal?: AbortSignal; + cwd?: string; + hasUI?: boolean; + /** Request a graceful shutdown of pi. No-op in print mode. */ + shutdown?: () => void | Promise; } -export interface ExtensionCommandContext { +export interface ExtensionCommandContext extends ExtensionContext { ui: ExtensionUIContext; } +export interface FlagOptions { + description: string; + type: "string" | "boolean"; + default?: string | boolean; +} + +export interface SessionStartEvent { + reason: "startup" | "reload" | "new" | "resume" | "fork"; + previousSessionFile?: string; +} + +export interface ToolResultContent { + type: "text"; + text: string; +} + +export interface ToolResult { + content: ToolResultContent[]; + details?: unknown; +} + export interface ExtensionAPI { - registerCommand(name: string, options: { - description: string; - handler: (args: string, ctx: ExtensionCommandContext) => Promise; - }): void; + registerCommand( + name: string, + options: { + description: string; + handler: (args: string, ctx: ExtensionCommandContext) => Promise | void; + }, + ): void; registerTool(tool: { name: string; label: string; description: string; parameters: unknown; - execute: (toolCallId: string, params: Record, signal: AbortSignal, onUpdate: unknown, ctx: unknown) => Promise; + execute: ( + toolCallId: string, + params: Record, + signal: AbortSignal, + onUpdate: unknown, + ctx: ExtensionCommandContext, + ) => Promise | ToolResult; }): void; -} \ No newline at end of file + /** Register a CLI flag (e.g. `--prospect `). */ + registerFlag(name: string, options: FlagOptions): void; + /** Read a previously-registered flag's value. */ + getFlag(name: string): string | boolean | undefined; + /** Subscribe to a lifecycle event. Only the events this extension uses are typed. */ + on( + event: "session_start", + handler: (event: SessionStartEvent, ctx: ExtensionCommandContext) => void | Promise, + ): void; +} + +// ───────────────────────── pi-ai (minimal) ───────────────────────── + +export interface PiTextContent { + type: "text"; + text: string; +} +export interface PiThinkingContent { + type: "thinking"; + thinking: string; +} +export interface PiToolCallContent { + type: "toolCall"; + id: string; + name: string; + arguments: Record; +} +export type PiAssistantContent = PiTextContent | PiThinkingContent | PiToolCallContent; + +export interface PiUsage { + input: number; + output: number; + cacheRead: number; + cacheWrite: number; + totalTokens: number; + cost: { input: number; output: number; cacheRead: number; cacheWrite: number; total: number }; +} + +export interface PiAssistantMessage { + role: "assistant"; + content: PiAssistantContent[]; + model: string; + usage: PiUsage; + stopReason: string; + errorMessage?: string; + timestamp: number; +} + +export interface PiUserMessage { + role: "user"; + content: string; + timestamp: number; +} + +export interface PiContext { + systemPrompt?: string; + messages: PiUserMessage[]; +} + +export interface PiCompleteOptions { + apiKey?: string; + headers?: Record; + temperature?: number; + maxTokens?: number; + /** Max client-side retries for transient failures (e.g. provider 429s). */ + maxRetries?: number; + signal?: AbortSignal; +} + +/** The subset of `@earendil-works/pi-ai` we call at runtime. */ +export interface PiAiModule { + complete: (model: PiModel, context: PiContext, options?: PiCompleteOptions) => Promise; +} diff --git a/src/sync/parser.ts b/src/sync/parser.ts index c7235f0..1559aaa 100644 --- a/src/sync/parser.ts +++ b/src/sync/parser.ts @@ -110,7 +110,7 @@ export function parseLine(line: string): ParsedLine | null { }; } - // Other message-like types (bashExecution, branchSummary, compactionSummary, custom) + // Other message-like types (bashExecution, branch_summary, compactionSummary, custom_message) if (type && obj.id) { const id = String(obj.id); const parentId = (obj.parentId as string) ?? null; diff --git a/src/types.ts b/src/types.ts index 6108831..01baf5c 100644 --- a/src/types.ts +++ b/src/types.ts @@ -8,8 +8,14 @@ // ─── Config ─── export interface ProspectorConfig { - model?: string; // provider/model format, e.g. "openrouter/deepseek-v4-flash" + model?: string; // provider/model format, e.g. "anthropic/claude-sonnet-4-5" dbPath?: string; // defaults to ~/.pi/agent/prospector.db + /** Model tiers used by analyzers (cheap/mid/expensive → provider/model). */ + modelTiers?: { + cheap: string; + mid: string; + expensive: string; + }; } // ─── Session ─── @@ -29,8 +35,8 @@ export type MessageRole = | "assistant" | "toolResult" | "bashExecution" - | "custom" - | "branchSummary" + | "custom_message" + | "branch_summary" | "compactionSummary"; export interface ToolCallInfo { @@ -92,29 +98,25 @@ export interface SyncResult { // ─── Proposals ─── export type ProposalSeverity = "friction" | "correction" | "waste" | "suggestion"; -export type ProposalStatus = "new" | "accepted" | "rejected"; - -export interface NewProposal { - sessionId: string; - target: string; - severity: ProposalSeverity; - summary: string; - detail: string; - evidence: string; - dedupHash: string; -} +export type ProposalStatus = "open" | "applied" | "rejected" | "duplicate"; export interface Proposal { id: string; created_at: string; + updated_at: string; session_id: string; - target: string; - severity: ProposalSeverity; + source_node_id: string | null; + analyzer_id: string | null; + target_type: string; + target_path: string | null; + title: string; + severity: string; summary: string; - detail: string; - evidence: string; + detail: string | null; + evidence: string | null; + confidence: number | null; status: ProposalStatus; - dedup_hash: string; + input_key: string; } // ─── Stats ─── @@ -123,8 +125,14 @@ export interface Stats { totalSessions: number; totalMessages: number; totalToolResults: number; - messagesProcessed: number; + sessionsAnalyzed: number; proposalsByStatus: Record; + analysis: { + nodes: number; + edges: number; + runs: number; + nodesByKind: Record; + }; } // ─── Analyze ─── diff --git a/test/integration/test-commands.ts b/test/integration/test-commands.ts index e06b1ea..6b13164 100644 --- a/test/integration/test-commands.ts +++ b/test/integration/test-commands.ts @@ -1,16 +1,23 @@ /** - * Integration test: directly invokes pi-prospector commands without Pi runtime. - * Tests the actual business logic end-to-end against a real database. + * Integration test: exercises the real analyzer pipeline end-to-end without a + * Pi runtime. Sync fixtures → run the framework with a MOCK LLM (never a real + * or local model) → assert the analysis graph, proposals, and lifecycle. */ import Database from "better-sqlite3"; import * as fs from "node:fs"; import * as path from "node:path"; import * as os from "node:os"; -// Import the actual modules import { migrate } from "../../src/db/schema.js"; -import { getStats, listProposals, insertProposal, acceptProposal, rejectProposal, computeDedupHash } from "../../src/db/queries.js"; +import { getAllSessions, getStats, listProposals, acceptProposal, rejectProposal } from "../../src/db/queries.js"; import { runSync } from "../../src/sync/index.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { registerDefaults } from "../../src/analyze/defaults.js"; +import { createMockLLM } from "../../src/analyze/mock-llm.js"; +import { getNodeVersions, getRevisedNode } from "../../src/db/analysis-queries.js"; +import { turnPairCoreAnalyzer } from "../../src/analyze/analyzers/turn-pair-core/index.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; +import type { LLMRequest } from "../../src/analyze/types.js"; const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "pi-prospector-int-")); const dbPath = path.join(tmpDir, "test.db"); @@ -18,7 +25,6 @@ const fixtureDir = path.resolve(import.meta.dirname, "../../tests/fixtures"); let pass = 0; let fail = 0; - function assert(condition: boolean, label: string, detail?: string): void { if (condition) { console.log(` ✅ ${label}`); @@ -29,110 +35,107 @@ function assert(condition: boolean, label: string, detail?: string): void { } } +function respond(req: LLMRequest): string { + const sys = req.system ?? ""; + if (sys.includes("classify a single turn")) { + return JSON.stringify({ sentiment: "neutral", friction_type: "none", is_genuine_correction: false, severity: "low", rationale: "ok" }); + } + if (sys.includes("summarise one segment")) { + return JSON.stringify({ segment_summary: "segment", notable_points: [] }); + } + return JSON.stringify({ + session_summary: "summary", + key_friction_points: [], + improvement_proposals: [ + { target_type: "config", target_path: "prospector.json", title: "Tune model tiers", summary: "pick cheaper models", detail: "d", evidence: "e", confidence: 0.5, severity: "suggestion" }, + ], + }); +} + console.log("═══════════════════════════════════════════"); -console.log(" pi-prospector integration tests"); +console.log(" pi-prospector integration tests (v2)"); console.log("═══════════════════════════════════════════\n"); -// --- Setup: create DB and sync fixtures --- -console.log("Setup: syncing fixture data..."); +console.log("Setup: syncing fixtures…"); const db = new Database(dbPath); migrate(db); -const result = runSync(db, fixtureDir); -console.log(` Synced: ${result.sessionsProcessed} sessions, ${result.messagesInserted} messages, ${result.errors.length} errors\n`); - -// --- Test: Stats --- -console.log("Stats command:"); -const stats = getStats(db); -assert(stats.totalSessions >= 1, "totalSessions >= 1", `got ${stats.totalSessions}`); -assert(stats.totalMessages >= 1, "totalMessages >= 1", `got ${stats.totalMessages}`); -assert(stats.proposalsByStatus.new === 0, "no proposals initially", `got ${stats.proposalsByStatus.new}`); -console.log(""); - -// --- Test: Proposals (empty) --- -console.log("Proposals command (empty DB):"); -const emptyProposals = listProposals(db); -assert(emptyProposals.length === 0, "no proposals initially", `got ${emptyProposals.length}`); -console.log(""); - -// Get a real session ID from the synced data (FK constraint requires it) -const realSessionIds = db.prepare("SELECT id FROM sessions").all() as Array<{id: string}>; -assert(realSessionIds.length >= 1, "have at least 1 synced session", `got ${realSessionIds.length}`); -const realSessionId = realSessionIds[0]!.id; - -// --- Test: Insert + list proposals --- -console.log("Proposals command (with data):"); -const id1 = insertProposal(db, { - id: crypto.randomUUID(), - created_at: new Date().toISOString(), - session_id: realSessionId, - severity: "suggestion", - target: "src/foo.ts", - summary: "Consider extracting helper function", - detail: "The function doStuff is too long.", - evidence: "Line 42-80 is a single function.", - status: "new", - dedup_hash: computeDedupHash("src/foo.ts", "suggestion", "Consider extracting helper function"), -}); -assert(id1 !== undefined && id1.length > 0, "insertProposal returns id", `got ${id1}`); - -const listed = listProposals(db); -assert(listed.length === 1, "listProposals returns 1", `got ${listed.length}`); -assert(listed[0]!.status === "new", "proposal status is 'new'", `got ${listed[0]!.status}`); -assert(listed[0]!.severity === "suggestion", "severity is 'suggestion'", `got ${listed[0]!.severity}`); -console.log(""); - -// --- Test: Accept proposal --- -console.log("Accept command:"); -const acceptOk = acceptProposal(db, id1); -assert(acceptOk === true, "acceptProposal succeeds"); -const accepted = listProposals(db, "accepted"); -assert(accepted.length === 1, "1 accepted proposal", `got ${accepted.length}`); -const stillNew = listProposals(db, "new"); -assert(stillNew.length === 0, "0 new proposals after accept", `got ${stillNew.length}`); -console.log(""); - -// --- Test: Reject proposal --- -console.log("Reject command:"); -const id2 = insertProposal(db, { - id: crypto.randomUUID(), - created_at: new Date().toISOString(), - session_id: realSessionId, - severity: "friction", - target: "src/bar.ts", - summary: "Memory leak in event listener", - detail: "addEventListener not removed on cleanup.", - evidence: "Line 15 adds listener, no removeEventListener found.", - status: "new", - dedup_hash: computeDedupHash("src/bar.ts", "friction", "Memory leak in event listener"), -}); -const rejectOk = rejectProposal(db, id2); -assert(rejectOk === true, "rejectProposal succeeds"); -const rejected = listProposals(db, "rejected"); -assert(rejected.length === 1, "1 rejected proposal", `got ${rejected.length}`); -console.log(""); - -// --- Test: Stats with proposals --- -console.log("Stats after proposals:"); -const stats2 = getStats(db); -assert(stats2.proposalsByStatus.accepted === 1, "1 accepted in stats", `got ${stats2.proposalsByStatus.accepted}`); -assert(stats2.proposalsByStatus.rejected === 1, "1 rejected in stats", `got ${stats2.proposalsByStatus.rejected}`); -assert(stats2.proposalsByStatus.new === 0, "0 new in stats", `got ${stats2.proposalsByStatus.new}`); -console.log(""); - -// --- Test: Incremental re-sync --- -console.log("Incremental re-sync:"); -const result2 = runSync(db, fixtureDir); -assert(result2.sessionsSkipped >= 1, "sessions skipped on re-sync", `got ${result2.sessionsSkipped}`); -assert(result2.sessionsProcessed === 0, "no new sessions processed", `got ${result2.sessionsProcessed}`); -console.log(""); - -// Cleanup +const sync = runSync(db, fixtureDir); +console.log(` Synced: ${sync.sessionsProcessed} sessions, ${sync.messagesInserted} messages`); + +console.log("\nStats (pre-analysis):"); +const s0 = getStats(db); +assert(s0.totalSessions >= 1, "indexed >= 1 session", `got ${s0.totalSessions}`); +assert(s0.proposalsByStatus.open === 0, "no proposals initially"); +assert(s0.analysis.nodes === 0, "no analysis nodes initially"); + +console.log("\nRun analyzer framework (fill, mock LLM):"); +const mock = createMockLLM({ responder: respond, tokensPerCall: 10, costPerCall: 0.0001 }); +const fw = new AnalyzerFramework({ db, llm: mock.caller, modelTiers: DEFAULT_MODEL_TIERS }); +registerDefaults(fw); + +const sessions = getAllSessions(db); +let totalNodes = 0; +let totalProposals = 0; +for (const session of sessions) { + const summary = await fw.run(session.id, {}); + totalNodes += summary.nodesProduced; + totalProposals += summary.proposalsCreated; + assert(summary.errors.length === 0, `session ${session.id.slice(0, 8)} ran without errors`, summary.errors.join("; ")); +} +assert(totalNodes > 0, "produced analysis nodes", `got ${totalNodes}`); +assert(totalProposals > 0, "materialised proposals", `got ${totalProposals}`); + +console.log("\nGraph stats:"); +const s1 = getStats(db); +assert(s1.analysis.nodes > 0, "analysis nodes recorded"); +assert((s1.analysis.nodesByKind["summary"] ?? 0) >= 1, "summary nodes present"); +assert((s1.analysis.nodesByKind["metric"] ?? 0) >= 1, "metric nodes present"); + +console.log("\nIdempotent re-run (fill):"); +let reRunNodes = 0; +for (const session of sessions) { + const summary = await fw.run(session.id, {}); + reRunNodes += summary.nodesProduced; +} +assert(reRunNodes === 0, "fill re-run produces nothing new", `got ${reRunNodes}`); + +console.log("\nRevise re-run with a new analyzer version (lineage):"); +const firstSession = sessions[0]!; +const v2 = new AnalyzerFramework({ db, llm: mock.caller, modelTiers: DEFAULT_MODEL_TIERS }); +v2.register({ ...turnPairCoreAnalyzer, version: { ...turnPairCoreAnalyzer.version, major: 2 } }); +const deep = await v2.run(firstSession.id, { revise: ["major"], analyzerIds: ["turn-pair-core"] }); +assert(deep.nodesRevised > 0, "revise run revised stale nodes", `got ${deep.nodesRevised}`); + +const coreRows = db + .prepare("SELECT source_set_hash FROM analysis_nodes WHERE analyzer_id = 'turn-pair-core' AND session_id = ? LIMIT 1") + .get(firstSession.id) as { source_set_hash: string } | undefined; +if (coreRows) { + const versions = getNodeVersions(db, "turn-pair-core", coreRows.source_set_hash); + assert(versions.length === 2, "two versions coexist for a logical unit", `got ${versions.length}`); + const newest = versions[versions.length - 1]!; + assert(getRevisedNode(db, newest.id) !== undefined, "newest version revises an older one"); +} + +console.log("\nProposal lifecycle:"); +const proposals = listProposals(db, "open"); +assert(proposals.length >= 1, "has open proposals"); +const pid = proposals[0]!.id; +assert(acceptProposal(db, pid) === true, "accept an open proposal"); +assert(acceptProposal(db, pid) === false, "cannot re-accept"); +const other = listProposals(db, "open")[0]; +if (other) assert(rejectProposal(db, other.id) === true, "reject another proposal"); + +const s2 = getStats(db); +assert(s2.proposalsByStatus.applied >= 1, "stats report applied proposals"); + db.close(); -try { fs.rmSync(tmpDir, { recursive: true }); } catch {} +try { + fs.rmSync(tmpDir, { recursive: true, force: true }); +} catch { + /* ignore */ +} -// Summary +console.log("\n═══════════════════════════════════════════"); +console.log(` Results: ${pass} passed, ${fail} failed`); console.log("═══════════════════════════════════════════"); -console.log(` Results: ${pass} passed, ${fail} failed (out of ${pass + fail})`); -console.log("═══════════════════════════════════════════\n"); - -process.exit(fail > 0 ? 1 : 0); \ No newline at end of file +if (fail > 0) process.exit(1); diff --git a/tests/component/analysis-queries.test.ts b/tests/component/analysis-queries.test.ts new file mode 100644 index 0000000..bf063ee --- /dev/null +++ b/tests/component/analysis-queries.test.ts @@ -0,0 +1,138 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession, insertMessages } from "./helpers.js"; +import { + createRun, + finishRun, + getRun, + getEdgesFrom, + getEdgesTo, + getAnchoredMessageIds, + getMessage, + getNodesByAnalyzer, + insertEdge, + insertNode, + resolveConfig, + upsertAnalyzerDef, +} from "../../src/db/analysis-queries.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { createThrowingLLM } from "../../src/analyze/mock-llm.js"; +import { turnPairCoreAnalyzer } from "../../src/analyze/analyzers/turn-pair-core/index.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; +import { EDGE_KINDS, REF_KINDS } from "../../src/analyze/edge-kinds.js"; + +function seedNode(db: import("better-sqlite3").Database, id: string, sessionId = "s1"): void { + insertNode(db, { + id, + sessionId, + analyzerId: "a", + analyzerVersionId: "1", + configId: "c", + runId: null, + nodeKind: "metric", + contentJson: "{}", + sourceSetHash: "ssh", + inputKey: `ih-${id}`, + outputKey: `ok-${id}`, + createdAt: new Date().toISOString(), + }); +} + +describe("analysis runs", () => { + it("creates, finishes, and reads a run", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + createRun(db, { + id: "run1", + analyzerId: "a", + analyzerVersionId: "1", + configId: "c", + sessionId: "s1", + mode: "fill", + promptBundleHash: "pb", + modelSpec: "anthropic/x", + }); + finishRun(db, "run1", { status: "ok", nodesProduced: 3, nodesSkipped: 1, costUsd: 0.5, tokensUsed: 100 }); + const run = getRun(db, "run1"); + assert.equal(run!.status, "ok"); + assert.equal(run!.nodes_produced, 3); + assert.equal(run!.model_spec, "anthropic/x"); + assert.ok(run!.finished_at); + assert.equal(getRun(db, "missing"), undefined); + } finally { + close(); + } + }); +}); + +describe("config resolution (content-addressed)", () => { + it("returns the same id for identical configs and a new id for changes", () => { + const { db, close } = tempDb(); + try { + upsertAnalyzerDef(db, { id: "a", label: "A", description: "", anchorSpan: "pair", dependencies: [] }); + const c1 = resolveConfig(db, { analyzerId: "a", configJson: { x: 1 }, label: "default" }); + const c2 = resolveConfig(db, { analyzerId: "a", configJson: { x: 1 }, label: "default" }); + assert.equal(c1.id, c2.id); + const c3 = resolveConfig(db, { analyzerId: "a", configJson: { x: 2 } }); + assert.notEqual(c1.id, c3.id); + } finally { + close(); + } + }); +}); + +describe("edges and anchored messages", () => { + it("queries edges by source and target", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + seedNode(db, "n1"); + insertEdge(db, { fromNodeId: "n1", toRefKind: REF_KINDS.SESSION, toRefId: "s1", edgeKind: EDGE_KINDS.ANCHORS, ordinal: 0 }); + insertEdge(db, { fromNodeId: "n1", toRefKind: REF_KINDS.ANALYSIS_NODE, toRefId: "x", edgeKind: EDGE_KINDS.CONSUMES, ordinal: 1 }); + + assert.equal(getEdgesFrom(db, "n1").length, 2); + assert.equal(getEdgesTo(db, "s1").length, 1); + assert.equal(getEdgesTo(db, "s1", EDGE_KINDS.ANCHORS).length, 1); + assert.equal(getEdgesTo(db, "s1", EDGE_KINDS.CONSUMES).length, 0); + } finally { + close(); + } + }); + + it("resolves anchored message ids and rows", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + const [m1] = insertMessages(db, "s1", [{ role: "user", text: "hi" }]); + seedNode(db, "n1"); + insertEdge(db, { fromNodeId: "n1", toRefKind: REF_KINDS.MESSAGE, toRefId: m1!, edgeKind: EDGE_KINDS.ANCHORS, ordinal: 0 }); + assert.deepEqual(getAnchoredMessageIds(db, "n1"), [m1]); + assert.equal(getMessage(db, m1!)!.content_text, "hi"); + assert.equal(getMessage(db, "nope"), undefined); + } finally { + close(); + } + }); + + it("framework.getAnchoredMessages returns the pair's user message", async () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + insertMessages(db, "s1", [ + { role: "user", text: "do a thing" }, + { role: "assistant", text: "done" }, + ]); + const fw = new AnalyzerFramework({ db, llm: createThrowingLLM(), modelTiers: DEFAULT_MODEL_TIERS }); + fw.register(turnPairCoreAnalyzer); + await fw.run("s1", {}); + + const node = getNodesByAnalyzer(db, "turn-pair-core", "s1")[0]!; + const anchored = fw.getAnchoredMessages(node.id); + assert.equal(anchored.length, 1); + assert.equal(anchored[0]!.content_text, "do a thing"); + } finally { + close(); + } + }); +}); diff --git a/tests/component/analyzers-e2e.test.ts b/tests/component/analyzers-e2e.test.ts new file mode 100644 index 0000000..8f6b626 --- /dev/null +++ b/tests/component/analyzers-e2e.test.ts @@ -0,0 +1,173 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession, insertMessages } from "./helpers.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { createMockLLM } from "../../src/analyze/mock-llm.js"; +import { registerDefaults } from "../../src/analyze/defaults.js"; +import { sessionOverviewAnalyzer } from "../../src/analyze/analyzers/session-overview/index.js"; +import { turnPairCoreAnalyzer } from "../../src/analyze/analyzers/turn-pair-core/index.js"; +import { turnPairLLMAnalyzer } from "../../src/analyze/analyzers/turn-pair-llm/index.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; +import { listProposals } from "../../src/db/queries.js"; +import type { LLMRequest } from "../../src/analyze/types.js"; + +function respond(req: LLMRequest): string { + const sys = req.system ?? ""; + if (sys.includes("classify a single turn")) { + return JSON.stringify({ + sentiment: "frustrated", + friction_type: "wrong_approach", + is_genuine_correction: true, + severity: "high", + rationale: "user corrected the approach", + }); + } + if (sys.includes("summarise one segment")) { + return JSON.stringify({ segment_summary: "a segment", notable_points: ["point"] }); + } + // reduce + return JSON.stringify({ + session_summary: "The agent took a wrong approach and was corrected.", + key_friction_points: [{ description: "wrong approach to auth", severity: "high" }], + improvement_proposals: [ + { + target_type: "agents_md", + target_path: "AGENTS.md § Auth", + title: "Document the auth module location", + summary: "Tell the agent where auth code lives", + detail: "Add a note pointing at src/auth.", + evidence: "User corrected the agent in turn 2.", + confidence: 0.7, + severity: "correction", + }, + ], + }); +} + +function seedSession(db: import("better-sqlite3").Database, id: string): void { + insertSession(db, id); + insertMessages(db, id, [ + { role: "user", text: "fix the login bug" }, + { role: "assistant", text: "reading auth", toolCalls: [{ name: "read" }] }, + { role: "toolResult", toolResults: [{ toolName: "read", isError: true, textLength: 80 }] }, + { role: "user", text: "no, that's wrong, use the auth module instead" }, + { role: "assistant", text: "understood, fixing now" }, + ]); +} + +describe("analyzers end-to-end", () => { + it("runs the full pipeline and materialises proposals", async () => { + const { db, close } = tempDb(); + try { + seedSession(db, "s1"); + const mock = createMockLLM({ responder: respond, tokensPerCall: 100, costPerCall: 0.001 }); + const fw = new AnalyzerFramework({ db, llm: mock.caller, modelTiers: DEFAULT_MODEL_TIERS }); + registerDefaults(fw); + + const summary = await fw.run("s1", {}); + assert.equal(summary.errors.length, 0, summary.errors.join("; ")); + + const kinds = db.prepare("SELECT node_kind, COUNT(*) AS c FROM analysis_nodes GROUP BY node_kind").all() as Array<{ node_kind: string; c: number }>; + const byKind = Object.fromEntries(kinds.map((k) => [k.node_kind, k.c])); + assert.ok(byKind["metric"] >= 2, "expected turn-pair-core metric nodes"); + assert.ok(byKind["classification"] >= 1, "expected at least one llm classification node"); + assert.equal(byKind["summary"], 1, "expected one session-overview summary node"); + + assert.ok(summary.proposalsCreated >= 1); + const proposals = listProposals(db); + assert.equal(proposals.length, 1); + assert.equal(proposals[0]!.target_type, "agents_md"); + assert.ok(summary.costUsd > 0); + + // The LLM was only consulted for high-signal pairs + the overview. + assert.ok(mock.calls.length >= 2); + } finally { + close(); + } + }); + + it("enforces maxPairsPerSession as a cost guard, enriching highest-friction turns first", async () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s3"); + const msgs = []; + for (let i = 0; i < 6; i++) { + msgs.push({ role: "user", text: `no, that's wrong, do approach number ${i} instead please` }); + msgs.push({ role: "assistant", text: `ok approach ${i}`, toolCalls: [{ name: "read" }] }); + msgs.push({ role: "toolResult", toolResults: [{ toolName: "read", isError: true, textLength: 80 }] }); + } + insertMessages(db, "s3", msgs); + + const mock = createMockLLM({ responder: respond, tokensPerCall: 10, costPerCall: 0.0001 }); + const fw = new AnalyzerFramework({ db, llm: mock.caller, modelTiers: DEFAULT_MODEL_TIERS }); + fw.register(turnPairCoreAnalyzer); + // turn-pair-llm variant whose cost guard allows only two enrichments. + const cappedLLM = { + ...turnPairLLMAnalyzer, + defaultConfig: { + ...turnPairLLMAnalyzer.defaultConfig, + configJson: { ...turnPairLLMAnalyzer.defaultConfig.configJson, maxPairsPerSession: 2 }, + }, + }; + fw.register(cappedLLM); + + await fw.run("s3", {}); + + const coreRows = db.prepare("SELECT content_json FROM analysis_nodes WHERE analyzer_id='turn-pair-core'").all() as Array<{ content_json: string }>; + const highSignal = coreRows.filter((r) => (JSON.parse(r.content_json) as { high_signal: boolean }).high_signal).length; + assert.ok(highSignal > 2, `expected more than the cap of high-signal pairs, got ${highSignal}`); + + const classifications = (db.prepare("SELECT COUNT(*) AS c FROM analysis_nodes WHERE analyzer_id='turn-pair-llm'").get() as { c: number }).c; + assert.equal(classifications, 2, "llm enrichment is capped at maxPairsPerSession"); + const classifyCalls = mock.calls.filter((c) => (c.system ?? "").includes("classify a single turn")); + assert.equal(classifyCalls.length, 2, "the model is called only for the capped set"); + } finally { + close(); + } + }); + + it("exercises the map-reduce path when the digest is large", async () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s2"); + // Many corrective turns so the digest exceeds the (tiny) threshold. + const msgs = []; + for (let i = 0; i < 6; i++) { + msgs.push({ role: "user", text: `no, that's wrong, do approach number ${i} instead please` }); + msgs.push({ role: "assistant", text: `ok approach ${i}` }); + } + insertMessages(db, "s2", msgs); + + const mock = createMockLLM({ responder: respond, tokensPerCall: 10, costPerCall: 0.0001 }); + const fw = new AnalyzerFramework({ db, llm: mock.caller, modelTiers: DEFAULT_MODEL_TIERS }); + + // session-overview variant with tiny map-reduce thresholds. + const tinyOverview = { + ...sessionOverviewAnalyzer, + defaultConfig: { + ...sessionOverviewAnalyzer.defaultConfig, + configJson: { + mapTier: "cheap", + reduceTier: "mid", + temperature: 0, + mapReduceOverChars: 50, + segmentChars: 80, + maxSegments: 12, + }, + }, + }; + fw.register(turnPairCoreAnalyzer); + fw.register(turnPairLLMAnalyzer); + fw.register(tinyOverview); + + const summary = await fw.run("s2", {}); + assert.equal(summary.errors.length, 0, summary.errors.join("; ")); + + // At least one map call must have happened (summarise one segment). + const mapCalls = mock.calls.filter((c) => (c.system ?? "").includes("summarise one segment")); + assert.ok(mapCalls.length >= 1, "expected map-phase calls"); + } finally { + close(); + } + }); +}); diff --git a/tests/component/commands.test.ts b/tests/component/commands.test.ts new file mode 100644 index 0000000..43a1096 --- /dev/null +++ b/tests/component/commands.test.ts @@ -0,0 +1,216 @@ +import { describe, it, before, after } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import registerExtension from "../../src/index.js"; +import { insertProposalRow } from "./helpers.js"; +import Database from "better-sqlite3"; +import { migrate } from "../../src/db/schema.js"; +import type { + ExtensionAPI, + ExtensionCommandContext, + ModelRegistry, + ToolResult, +} from "../../src/pi-stubs.js"; + +type Handler = (args: string, ctx: ExtensionCommandContext) => Promise | void; +type ToolExec = ( + id: string, + params: Record, + signal: AbortSignal, + onUpdate: unknown, + ctx: ExtensionCommandContext, +) => Promise | ToolResult; + +const FIXTURES = path.resolve(import.meta.dirname, "..", "fixtures"); + +const commands = new Map(); +let toolExec: ToolExec; +const flags = new Map(); + +const fakePi: ExtensionAPI = { + registerCommand: (name, opts) => commands.set(name, opts.handler), + registerTool: (tool) => { + if (tool.name === "prospect") toolExec = tool.execute as ToolExec; + }, + registerFlag: (name, opts) => { + if (opts.default !== undefined) flags.set(name, opts.default); + }, + getFlag: (name) => flags.get(name), + on: () => { + /* no session_start dispatch needed for these command tests */ + }, +}; + +const notes: string[] = []; +const modelRegistry: ModelRegistry = { + find: () => undefined, + getAll: () => [], + getAvailable: () => [], + getApiKeyAndHeaders: async () => ({ ok: false, error: "no creds in test" }), +}; +const ctx: ExtensionCommandContext = { + modelRegistry, + hasUI: false, + ui: { notify: (m) => notes.push(m) }, +}; + +let tmpDir: string; + +before(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "prospector-cmd-")); + process.env["PROSPECTOR_DB_PATH"] = path.join(tmpDir, "cmd.db"); + process.env["PROSPECTOR_SESSIONS_DIR"] = FIXTURES; + registerExtension(fakePi); +}); + +after(() => { + delete process.env["PROSPECTOR_DB_PATH"]; + delete process.env["PROSPECTOR_SESSIONS_DIR"]; + try { + fs.rmSync(tmpDir, { recursive: true, force: true }); + } catch { + /* ignore */ + } +}); + +async function run(name: string, args = ""): Promise { + notes.length = 0; + await commands.get(name)!(args, ctx); + return notes.join("\n"); +} + +describe("slash commands", () => { + it("registers all expected commands and the tool", () => { + for (const name of [ + "prospect-sync", + "prospect-stats", + "prospect-proposals", + "prospect-accept", + "prospect-reject", + "prospect-analyze", + "prospect-verify", + ]) { + assert.ok(commands.has(name), `missing command ${name}`); + } + assert.ok(typeof toolExec === "function"); + }); + + it("prospect-sync indexes fixtures", async () => { + const out = await run("prospect-sync"); + assert.match(out, /Prospect sync complete/); + assert.match(out, /Sessions processed:/); + }); + + it("prospect-stats renders stats", async () => { + const out = await run("prospect-stats"); + assert.match(out, /Prospector Stats/); + assert.match(out, /Sessions indexed:/); + }); + + it("prospect-analyze runs the deterministic analyzer without an LLM", async () => { + const out = await run("prospect-analyze", "--analyzer turn-pair-core"); + assert.match(out, /Done \[fill\]/); + assert.match(out, /Nodes produced:/); + }); + + it("prospect-analyze --revise re-scans", async () => { + const out = await run("prospect-analyze", "--revise all --analyzer turn-pair-core"); + assert.match(out, /Done \[revise:/); + }); + + it("prospect-verify confirms integrity and flags tampering", async () => { + await run("prospect-sync"); + await run("prospect-analyze", "--analyzer turn-pair-core"); + const ok = await run("prospect-verify"); + assert.match(ok, /verified|No analysis nodes/); + + const db = new Database(process.env["PROSPECTOR_DB_PATH"]!); + try { + const rows = db.prepare("SELECT id FROM analysis_nodes LIMIT 2").all() as Array<{ id: string }>; + assert.ok(rows.length >= 1, "expected nodes to tamper with"); + // Valid-but-different content → output_key mismatch. + db.prepare("UPDATE analysis_nodes SET content_json = '{\"x\":1}' WHERE id = ?").run(rows[0]!.id); + // Unparseable content → exercises the parse-failure branch. + if (rows[1]) db.prepare("UPDATE analysis_nodes SET content_json = 'not json' WHERE id = ?").run(rows[1].id); + } finally { + db.close(); + } + const bad = await run("prospect-verify"); + assert.match(bad, /failed verification/); + }); + + it("prospect-analyze reports when there is nothing to do", async () => { + // A fresh empty DB → no sessions. + const emptyDb = path.join(tmpDir, "empty.db"); + process.env["PROSPECTOR_DB_PATH"] = emptyDb; + try { + const out = await run("prospect-analyze", "--session does-not-exist --analyzer turn-pair-core"); + assert.match(out, /Done|No sessions/); + } finally { + process.env["PROSPECTOR_DB_PATH"] = path.join(tmpDir, "cmd.db"); + } + }); + + it("prospect-proposals lists, accepts, and rejects", async () => { + const db = new Database(process.env["PROSPECTOR_DB_PATH"]!); + migrate(db); + const session = db.prepare("SELECT id FROM sessions LIMIT 1").get() as { id: string }; + insertProposalRow(db, { id: "cmd-p1", sessionId: session.id, title: "Test proposal", severity: "friction" }); + insertProposalRow(db, { id: "cmd-p2", sessionId: session.id, title: "Second proposal", severity: "waste" }); + db.close(); + + const list = await run("prospect-proposals"); + assert.match(list, /Test proposal/); + + const accepted = await run("prospect-accept", "cmd-p1"); + assert.match(accepted, /applied/); + + const rejected = await run("prospect-reject", "cmd-p2"); + assert.match(rejected, /rejected/); + + const filtered = await run("prospect-proposals", "applied"); + assert.match(filtered, /Test proposal/); + + const missing = await run("prospect-accept", ""); + assert.match(missing, /Usage/); + + const rejectMissing = await run("prospect-reject", ""); + assert.match(rejectMissing, /Usage/); + }); + + it("prospect-proposals reports empty state", async () => { + process.env["PROSPECTOR_DB_PATH"] = path.join(tmpDir, "empty2.db"); + try { + const out = await run("prospect-proposals"); + assert.match(out, /No proposals found/); + } finally { + process.env["PROSPECTOR_DB_PATH"] = path.join(tmpDir, "cmd.db"); + } + }); +}); + +describe("prospect tool", () => { + const signal = new AbortController().signal; + + it("handles sync, stats, list_proposals, accept, reject, and unknown", async () => { + const sync = await toolExec("t1", { action: "sync" }, signal, null, ctx); + assert.match(sync.content[0]!.text, /sessionsProcessed/); + + const stats = await toolExec("t2", { action: "stats" }, signal, null, ctx); + assert.match(stats.content[0]!.text, /totalSessions/); + + const list = await toolExec("t3", { action: "list_proposals" }, signal, null, ctx); + assert.ok(list.content[0]!.text.length > 0); + + const acceptNoId = await toolExec("t4", { action: "accept" }, signal, null, ctx); + assert.match(acceptNoId.content[0]!.text, /required/); + + const rejectNoId = await toolExec("t5", { action: "reject" }, signal, null, ctx); + assert.match(rejectNoId.content[0]!.text, /required/); + + const unknown = await toolExec("t6", { action: "bogus" }, signal, null, ctx); + assert.match(unknown.content[0]!.text, /Unknown action/); + }); +}); diff --git a/tests/component/content-addressing.test.ts b/tests/component/content-addressing.test.ts new file mode 100644 index 0000000..e14b9ce --- /dev/null +++ b/tests/component/content-addressing.test.ts @@ -0,0 +1,117 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession, insertMessages } from "./helpers.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { createMockLLM } from "../../src/analyze/mock-llm.js"; +import { registerDefaults } from "../../src/analyze/defaults.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; +import { verifyNodes } from "../../src/commands/verify.js"; +import type { LLMRequest } from "../../src/analyze/types.js"; + +function respond(req: LLMRequest): string { + const sys = req.system ?? ""; + if (sys.includes("classify a single turn")) { + return JSON.stringify({ + sentiment: "frustrated", + friction_type: "wrong_approach", + is_genuine_correction: true, + severity: "high", + rationale: "user corrected the approach", + }); + } + if (sys.includes("summarise one segment")) { + return JSON.stringify({ segment_summary: "a segment", notable_points: ["point"] }); + } + return JSON.stringify({ + session_summary: "The agent took a wrong approach and was corrected.", + key_friction_points: [{ description: "wrong approach", severity: "high" }], + improvement_proposals: [ + { target_type: "agents_md", target_path: "AGENTS.md", title: "Doc auth", summary: "s", detail: "d", evidence: "e", confidence: 0.7, severity: "correction" }, + ], + }); +} + +/** Seed a session with EXPLICIT, stable message ids so leaf identities reproduce. */ +function seed(db: import("better-sqlite3").Database, id: string): void { + insertSession(db, id); + insertMessages(db, id, [ + { id: `${id}-m0`, role: "user", text: "fix the login bug" }, + { id: `${id}-m1`, role: "assistant", text: "reading auth", toolCalls: [{ name: "read" }] }, + { id: `${id}-m2`, role: "toolResult", toolResults: [{ toolName: "read", isError: true, textLength: 80 }] }, + { id: `${id}-m3`, role: "user", text: "no, that's wrong, use the auth module instead" }, + { id: `${id}-m4`, role: "assistant", text: "understood, fixing now" }, + ]); +} + +async function analyze(db: import("better-sqlite3").Database, sessionId: string): Promise { + const mock = createMockLLM({ responder: respond, tokensPerCall: 100, costPerCall: 0.001 }); + const fw = new AnalyzerFramework({ db, llm: mock.caller, modelTiers: DEFAULT_MODEL_TIERS }); + registerDefaults(fw); + const summary = await fw.run(sessionId, {}); + assert.equal(summary.errors.length, 0, summary.errors.join("; ")); +} + +function keysOf(db: import("better-sqlite3").Database): string[] { + return ( + db + .prepare("SELECT analyzer_id, input_key, output_key FROM analysis_nodes ORDER BY analyzer_id, input_key") + .all() as Array<{ analyzer_id: string; input_key: string; output_key: string }> + ).map((r) => `${r.analyzer_id}|${r.input_key}|${r.output_key}`); +} + +describe("content-addressed identities", () => { + it("reproduce identically across independent databases (global, wipe-surviving)", async () => { + const a = tempDb(); + const b = tempDb(); + try { + seed(a.db, "s1"); + seed(b.db, "s1"); + await analyze(a.db, "s1"); + await analyze(b.db, "s1"); + + const ka = keysOf(a.db); + const kb = keysOf(b.db); + assert.ok(ka.length > 0, "produced nodes"); + // Node uuids and created_at differ between DBs; input_key + output_key must not. + assert.deepEqual(ka, kb, "input_key/output_key are pure functions of content, not DB-local ids"); + } finally { + a.close(); + b.close(); + } + }); + + it("a consumer's input_key folds in the upstream output_key (output matters)", async () => { + const { db, close } = tempDb(); + try { + seed(db, "s1"); + await analyze(db, "s1"); + // The session-overview node consumes turn-pair output_keys; its source_set_hash + // is therefore derived from upstream output_keys, not uuids. + const overview = db.prepare("SELECT source_set_hash FROM analysis_nodes WHERE analyzer_id='session-overview'").get() as { source_set_hash: string }; + const upstreamOutputKeys = (db.prepare("SELECT output_key FROM analysis_nodes WHERE analyzer_id IN ('turn-pair-core','turn-pair-llm')").all() as Array<{ output_key: string }>).map((r) => r.output_key); + assert.ok(overview, "overview node exists"); + assert.ok(upstreamOutputKeys.every((k) => k.length === 16), "upstream nodes have content-addressed output_keys"); + } finally { + close(); + } + }); + + it("verifyNodes confirms a clean graph and detects tampering", async () => { + const { db, close } = tempDb(); + try { + seed(db, "s1"); + await analyze(db, "s1"); + + const clean = verifyNodes(db); + assert.ok(clean.total > 0); + assert.equal(clean.mismatches.length, 0, "a freshly built graph verifies"); + + // Tamper with stored content out of band; the output_key no longer matches. + db.prepare("UPDATE analysis_nodes SET content_json = '{\"tampered\":true}' WHERE id = (SELECT id FROM analysis_nodes WHERE analyzer_id='turn-pair-core' LIMIT 1)").run(); + const dirty = verifyNodes(db); + assert.equal(dirty.mismatches.length, 1, "tampering is detected"); + } finally { + close(); + } + }); +}); diff --git a/tests/component/framework.test.ts b/tests/component/framework.test.ts new file mode 100644 index 0000000..d9162cc --- /dev/null +++ b/tests/component/framework.test.ts @@ -0,0 +1,240 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession, insertMessages } from "./helpers.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { createThrowingLLM } from "../../src/analyze/mock-llm.js"; +import { turnPairCoreAnalyzer } from "../../src/analyze/analyzers/turn-pair-core/index.js"; +import { getNodeVersions, getRevisedNode, getRevisions } from "../../src/db/analysis-queries.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; +import type { Analyzer, AnalysisResult, AnalyzerPlanContext, AnalyzerRunContext } from "../../src/analyze/types.js"; + +function frameworkFor(db: import("better-sqlite3").Database): AnalyzerFramework { + return new AnalyzerFramework({ db, llm: createThrowingLLM(), modelTiers: DEFAULT_MODEL_TIERS }); +} + +function seedSession(db: import("better-sqlite3").Database, id = "s1"): void { + insertSession(db, id); + insertMessages(db, id, [ + { role: "user", text: "fix the login bug" }, + { role: "assistant", text: "looking", toolCalls: [{ name: "read" }] }, + { role: "toolResult", toolResults: [{ toolName: "read", isError: true, textLength: 50 }] }, + { role: "user", text: "no, that's wrong, use the auth module" }, + { role: "assistant", text: "fixing" }, + ]); +} + +describe("framework: incremental scan + fill run", () => { + it("classifies all units as missing, then current after a run", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + const fw = frameworkFor(db); + fw.register(turnPairCoreAnalyzer); + + const before = await fw.scan("s1"); + assert.ok(before.length >= 2); + assert.ok(before.every((c) => c.status === "missing")); + + const summary = await fw.run("s1", {}); + assert.equal(summary.nodesProduced, before.length); + assert.equal(summary.nodesRevised, 0); + + const after = await fw.scan("s1"); + assert.ok(after.every((c) => c.status === "current")); + } finally { + close(); + } + }); + + it("is idempotent: re-running a fill produces nothing new", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + const fw = frameworkFor(db); + fw.register(turnPairCoreAnalyzer); + await fw.run("s1", {}); + const second = await fw.run("s1", {}); + assert.equal(second.nodesProduced, 0); + assert.ok(second.nodesSkipped > 0); + } finally { + close(); + } + }); + + it("deterministic analyzer never calls the LLM", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + const fw = frameworkFor(db); // throwing LLM + fw.register(turnPairCoreAnalyzer); + const summary = await fw.run("s1", {}); + assert.equal(summary.errors.length, 0); + assert.ok(summary.nodesProduced > 0); + } finally { + close(); + } + }); +}); + +describe("framework: version lineage (revise)", () => { + it("re-analyses stale units into new versions linked by revises edges", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + + const v1 = frameworkFor(db); + v1.register(turnPairCoreAnalyzer); + await v1.run("s1", {}); + + // A new major version over the same logical units. + const v2Analyzer: Analyzer = { + ...turnPairCoreAnalyzer, + version: { ...turnPairCoreAnalyzer.version, major: 2 }, + }; + const v2 = frameworkFor(db); + v2.register(v2Analyzer); + + const scan = await v2.scan("s1"); + assert.ok(scan.every((c) => c.status === "stale")); + assert.ok(scan.every((c) => c.reasons.includes("major")), "a major bump grades as a major reason"); + + // A plain fill ignores stale units. + const fill = await v2.run("s1", {}); + assert.equal(fill.nodesProduced, 0); + + // --revise major re-analyses. + const reviseRun = await v2.run("s1", { revise: ["major"] }); + assert.ok(reviseRun.nodesProduced > 0); + assert.equal(reviseRun.nodesRevised, reviseRun.nodesProduced); + + // Both versions coexist for the same logical unit, newest revises oldest. + const firstUnit = scan[0]!; + const versions = getNodeVersions(db, "turn-pair-core", firstUnit.unit.sourceSetHash); + assert.equal(versions.length, 2); + + const newest = versions[versions.length - 1]!; + const oldest = versions[0]!; + const revised = getRevisedNode(db, newest.id); + assert.equal(revised!.id, oldest.id); + assert.equal(getRevisions(db, oldest.id)[0]!.id, newest.id); + } finally { + close(); + } + }); +}); + +describe("framework: dependency visibility & ordering", () => { + it("throws when an analyzer reads an undeclared dependency", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + const sneaky: Analyzer = { + def: { id: "sneaky", label: "Sneaky", description: "", anchorSpan: "full_session", dependencies: [] }, + version: { analyzerId: "sneaky", major: 1, minor: 0, implementationKind: "deterministic" }, + prompts: {}, + defaultConfig: { id: "", analyzerId: "sneaky", configHash: "h", configJson: {}, label: "default" }, + plan: (_ctx: AnalyzerPlanContext) => [ + { sources: [{ kind: "session" as const, id: "s1" }], sourceSetHash: "sneaky-ssh", anchorKind: "session" as const, anchorRef: "s1" }, + ], + analyze: (_unit, ctx: AnalyzerRunContext): AnalysisResult => { + ctx.getDependencyNodes("turn-pair-core"); // not declared → throws + return { nodeKind: "summary", contentJson: {}, anchorKind: "session", anchorRef: "s1", edges: [] }; + }, + }; + const fw = frameworkFor(db); + fw.register(sneaky); + const summary = await fw.run("s1"); + assert.equal(summary.errors.length, 1); + assert.match(summary.errors[0]!, /without declaring it/); + + // The failure is recorded as an append-only error node (visibility + history), + // carrying the message. + const errNode = db + .prepare("SELECT * FROM analysis_nodes WHERE node_kind = 'error'") + .get() as { content_json: string; input_key: string } | undefined; + assert.ok(errNode); + assert.match(JSON.parse(errNode!.content_json).error, /without declaring it/); + + // But the error node uses a decoupled identity and does NOT occupy the recipe + // identity, so the unit stays `missing` (not `current`) and will be recomputed. + const after = (await fw.scan("s1")).filter((c) => c.analyzerId === "sneaky"); + assert.ok(after.length >= 1); + assert.ok(after.every((c) => c.status === "missing")); + assert.notEqual(errNode!.input_key, after[0]!.inputKey); + } finally { + close(); + } + }); + + it("self-heals: a failed unit stays missing and is recomputed on the next run, keeping the error node", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + let attempts = 0; + const flaky: Analyzer = { + def: { id: "flaky", label: "Flaky", description: "", anchorSpan: "full_session", dependencies: [] }, + version: { analyzerId: "flaky", major: 1, minor: 0, implementationKind: "deterministic" }, + prompts: {}, + defaultConfig: { id: "", analyzerId: "flaky", configHash: "h", configJson: {}, label: "default" }, + plan: (_ctx: AnalyzerPlanContext) => [ + { sources: [{ kind: "session" as const, id: "s1" }], sourceSetHash: "flaky-ssh", anchorKind: "session" as const, anchorRef: "s1" }, + ], + analyze: (_unit, _ctx: AnalyzerRunContext): AnalysisResult => { + attempts++; + if (attempts === 1) throw new Error("transient boom"); + return { nodeKind: "metric", contentJson: { ok: true }, anchorKind: "session", anchorRef: "s1", edges: [] }; + }, + }; + const fw = frameworkFor(db); + fw.register(flaky); + + const countErr = () => (db.prepare("SELECT COUNT(*) c FROM analysis_nodes WHERE node_kind='error'").get() as { c: number }).c; + const countOk = () => (db.prepare("SELECT COUNT(*) c FROM analysis_nodes WHERE analyzer_id='flaky' AND node_kind='metric'").get() as { c: number }).c; + + // First run fails: error node recorded, no result, unit still missing. + const run1 = await fw.run("s1", {}); + assert.equal(run1.errors.length, 1); + assert.equal(run1.nodesProduced, 0); + assert.equal(countErr(), 1); + assert.equal(countOk(), 0); + const scan1 = (await fw.scan("s1")).filter((c) => c.analyzerId === "flaky"); + assert.ok(scan1.every((c) => c.status === "missing")); + + // Second plain run heals it: result produced, error node retained (append-only). + const run2 = await fw.run("s1", {}); + assert.equal(run2.errors.length, 0); + assert.equal(run2.nodesProduced, 1); + assert.equal(countErr(), 1); + assert.equal(countOk(), 1); + const scan2 = (await fw.scan("s1")).filter((c) => c.analyzerId === "flaky"); + assert.ok(scan2.every((c) => c.status === "current")); + } finally { + close(); + } + }); + + it("orders analyzers by dependency and detects cycles", () => { + const { db, close } = tempDb(); + try { + const fw = frameworkFor(db); + fw.register(turnPairCoreAnalyzer); + const order = fw.topologicalSort(); + assert.deepEqual(order, ["turn-pair-core"]); + + const cyclicA: Analyzer = { + def: { id: "A", label: "A", description: "", anchorSpan: "pair", dependencies: ["B"] }, + version: { analyzerId: "A", major: 1, minor: 0, implementationKind: "deterministic" }, + prompts: {}, + defaultConfig: { id: "", analyzerId: "A", configHash: "h", configJson: {} }, + plan: () => [], + analyze: () => ({ nodeKind: "metric", contentJson: {}, anchorKind: "session", anchorRef: "s", edges: [] }), + }; + const cyclicB: Analyzer = { ...cyclicA, def: { ...cyclicA.def, id: "B", dependencies: ["A"] }, version: { analyzerId: "B", major: 1, minor: 0, implementationKind: "deterministic" }, defaultConfig: { id: "", analyzerId: "B", configHash: "h", configJson: {} } }; + fw.register(cyclicA); + fw.register(cyclicB); + assert.throws(() => fw.topologicalSort(["A"]), /cycle/i); + } finally { + close(); + } + }); +}); diff --git a/tests/component/helpers.ts b/tests/component/helpers.ts new file mode 100644 index 0000000..174eefb --- /dev/null +++ b/tests/component/helpers.ts @@ -0,0 +1,112 @@ +import * as fs from "node:fs"; +import * as path from "node:path"; +import * as os from "node:os"; +import Database from "better-sqlite3"; +import { migrate } from "../../src/db/schema.js"; + +export const FIXTURES = path.resolve(import.meta.dirname, "..", "fixtures"); + +export interface TempDb { + db: Database.Database; + close: () => void; +} + +/** A migrated SQLite database backed by a unique temp file, with cleanup. */ +export function tempDb(): TempDb { + const dbPath = path.join(os.tmpdir(), `prospect-test-${Date.now()}-${Math.random().toString(36).slice(2)}.db`); + const db = new Database(dbPath); + migrate(db); + return { + db, + close: () => { + db.close(); + for (const suffix of ["", "-wal", "-shm"]) { + try { + fs.unlinkSync(dbPath + suffix); + } catch { + /* ignore */ + } + } + }, + }; +} + +/** Insert a minimal session row so foreign keys on messages/proposals are satisfied. */ +export function insertSession(db: Database.Database, id: string, filePath = `/tmp/${id}.jsonl`): void { + db.prepare( + "INSERT INTO sessions (id, file_path, project, cwd, started_at, last_line, last_modified, message_count, branch_count) " + + "VALUES (?, ?, '', '', ?, 0, 0, 0, 0)", + ).run(id, filePath, new Date().toISOString()); +} + +let messageSeq = 0; + +export interface TestMessage { + role: string; + text?: string; + thinking?: string; + toolCalls?: Array<{ name: string }>; + toolResults?: Array<{ toolName: string; isError: boolean; textLength: number }>; + id?: string; +} + +/** Insert messages for a session in order, returning the inserted ids. */ +export function insertMessages(db: Database.Database, sessionId: string, messages: TestMessage[]): string[] { + const stmt = db.prepare( + "INSERT INTO messages (id, session_id, parent_id, timestamp, role, content_text, content_thinking, tool_calls, tool_results) " + + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", + ); + const ids: string[] = []; + let parent: string | null = null; + for (const m of messages) { + const id = m.id ?? `msg-${sessionId}-${messageSeq++}`; + stmt.run( + id, + sessionId, + parent, + new Date(1_700_000_000_000 + messageSeq * 1000).toISOString(), + m.role, + m.text ?? null, + m.thinking ?? null, + m.toolCalls ? JSON.stringify(m.toolCalls) : null, + m.toolResults ? JSON.stringify(m.toolResults) : null, + ); + ids.push(id); + parent = id; + } + return ids; +} + +/** Insert a v2 proposal directly (bypassing materialisation), for query tests. */ +export function insertProposalRow( + db: Database.Database, + p: { + id: string; + sessionId: string; + targetType?: string; + targetPath?: string; + title: string; + severity?: string; + summary?: string; + status?: string; + inputKey?: string; + }, +): void { + const now = new Date().toISOString(); + db.prepare( + "INSERT INTO proposals (id, created_at, updated_at, session_id, source_node_id, analyzer_id, target_type, target_path, title, severity, summary, detail, evidence, confidence, status, input_key) " + + "VALUES (?, ?, ?, ?, NULL, NULL, ?, ?, ?, ?, ?, NULL, NULL, NULL, ?, ?)", + ).run( + p.id, + now, + now, + p.sessionId, + p.targetType ?? "config", + p.targetPath ?? null, + p.title, + p.severity ?? "suggestion", + p.summary ?? p.title, + p.status ?? "open", + p.inputKey ?? `ik-${p.id}`, + ); +} diff --git a/tests/component/model-identity.test.ts b/tests/component/model-identity.test.ts new file mode 100644 index 0000000..90fa28f --- /dev/null +++ b/tests/component/model-identity.test.ts @@ -0,0 +1,155 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession, insertMessages } from "./helpers.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { createMockLLM } from "../../src/analyze/mock-llm.js"; +import { turnPairCoreAnalyzer } from "../../src/analyze/analyzers/turn-pair-core/index.js"; +import { turnPairLLMAnalyzer } from "../../src/analyze/analyzers/turn-pair-llm/index.js"; +import { DEFAULT_MODEL_TIERS, applyModelOverride } from "../../src/analyze/model-tiers.js"; +import { getNodeVersions, getRevisedNode } from "../../src/db/analysis-queries.js"; +import type { LLMRequest, ModelTierConfig } from "../../src/analyze/types.js"; + +// turn-pair-llm only ever sends a classify prompt; return a fixed classification. +function respond(_req: LLMRequest): string { + return JSON.stringify({ + sentiment: "frustrated", + friction_type: "wrong_approach", + is_genuine_correction: true, + severity: "high", + rationale: "user corrected the approach", + }); +} + +function seedSession(db: import("better-sqlite3").Database, id = "s1"): void { + insertSession(db, id); + insertMessages(db, id, [ + { role: "user", text: "fix the login bug" }, + { role: "assistant", text: "reading auth", toolCalls: [{ name: "read" }] }, + { role: "toolResult", toolResults: [{ toolName: "read", isError: true, textLength: 80 }] }, + { role: "user", text: "no, that's wrong, use the auth module instead" }, + { role: "assistant", text: "understood, fixing now" }, + ]); +} + +function frameworkFor( + db: import("better-sqlite3").Database, + modelTiers: ModelTierConfig, +): AnalyzerFramework { + const mock = createMockLLM({ responder: respond, tokensPerCall: 50, costPerCall: 0.001 }); + const fw = new AnalyzerFramework({ db, llm: mock.caller, modelTiers }); + fw.register(turnPairCoreAnalyzer); + fw.register(turnPairLLMAnalyzer); + return fw; +} + +// A tier mapping that differs from the default only in what `cheap` resolves to. +const REMAPPED_TIERS: ModelTierConfig = { ...DEFAULT_MODEL_TIERS, cheap: "openai/gpt-5-mini" }; + +function classificationNodes(db: import("better-sqlite3").Database) { + return db + .prepare("SELECT * FROM analysis_nodes WHERE analyzer_id = 'turn-pair-llm' ORDER BY created_at ASC, rowid ASC") + .all() as Array<{ id: string; source_set_hash: string }>; +} + +describe("the resolved model is part of a node's config identity", () => { + it("remapping a tier to a new model marks the LLM node stale (config reason); core stays current", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + + // First pass: default tiers (cheap = the default model). + await frameworkFor(db, DEFAULT_MODEL_TIERS).run("s1", {}); + assert.equal(classificationNodes(db).length, 1, "one classification produced initially"); + + // Re-scan with a different concrete model for the `cheap` tier. + const remapped = frameworkFor(db, REMAPPED_TIERS); + const classified = await remapped.scan("s1"); + + const llm = classified.filter((c) => c.analyzerId === "turn-pair-llm"); + const core = classified.filter((c) => c.analyzerId === "turn-pair-core"); + assert.ok(llm.length >= 1); + assert.ok(llm.every((c) => c.status === "stale"), "model change makes the LLM unit stale"); + assert.ok( + llm.every((c) => c.reasons.includes("config") && !c.reasons.includes("major") && !c.reasons.includes("minor")), + "a model swap is an ungraded config reason, not a version bump", + ); + assert.ok(core.every((c) => c.status === "current"), "deterministic core is unaffected by model change"); + } finally { + close(); + } + }); + + it("a plain fill leaves the stale (model-changed) node untouched; --revise config revises it", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + await frameworkFor(db, DEFAULT_MODEL_TIERS).run("s1", {}); + const before = classificationNodes(db); + assert.equal(before.length, 1); + const sourceSetHash = before[0]!.source_set_hash; + + // A plain fill under the new model must NOT touch the stale node (cost-safe). + const fill = await frameworkFor(db, REMAPPED_TIERS).run("s1", {}); + assert.equal(fill.nodesRevised, 0); + assert.equal(classificationNodes(db).length, 1, "a fill does not re-run a stale model change"); + + // --revise config produces a NEW version linked to the old one by a revises edge. + const revised = await frameworkFor(db, REMAPPED_TIERS).run("s1", { revise: ["config"] }); + assert.ok(revised.nodesRevised >= 1, "revise config revises the model-changed node"); + + const after = classificationNodes(db); + assert.equal(after.length, 2, "old and new versions coexist"); + + const versions = getNodeVersions(db, "turn-pair-llm", sourceSetHash); + assert.equal(versions.length, 2); + + const newest = versions[versions.length - 1]!; + const revisedNode = getRevisedNode(db, newest.id); + assert.ok(revisedNode, "newest version revises an older one"); + assert.equal(revisedNode!.id, before[0]!.id); + } finally { + close(); + } + }); + + it("re-running with the same tier mapping is idempotent (no model churn)", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + await frameworkFor(db, DEFAULT_MODEL_TIERS).run("s1", {}); + const revised = await frameworkFor(db, DEFAULT_MODEL_TIERS).run("s1", { revise: ["major", "minor", "config"] }); + assert.equal(revised.nodesRevised, 0, "unchanged model means nothing is stale"); + assert.equal(classificationNodes(db).length, 1); + } finally { + close(); + } + }); +}); + +describe("--model override is live (the pinned model is actually used)", () => { + it("passes the pinned concrete model to the LLM and records it on the node", async () => { + const { db, close } = tempDb(); + try { + seedSession(db); + const pinned = "openai/gpt-5-override"; + const effectiveTiers = applyModelOverride(DEFAULT_MODEL_TIERS, pinned); + + const mock = createMockLLM({ responder: respond, tokensPerCall: 50, costPerCall: 0.001 }); + const fw = new AnalyzerFramework({ db, llm: mock.caller, modelTiers: effectiveTiers }); + fw.register(turnPairCoreAnalyzer); + fw.register(turnPairLLMAnalyzer); + await fw.run("s1", { modelSpec: pinned }); + + assert.ok(mock.calls.length >= 1, "the LLM analyzer ran"); + assert.ok( + mock.calls.every((c) => c.model === pinned), + `every LLM call used the pinned model, got: ${mock.calls.map((c) => c.model).join(", ")}`, + ); + + const node = classificationNodes(db)[0] as unknown as { model_used: string | null }; + assert.equal(node.model_used, pinned, "the node records the model actually used"); + } finally { + close(); + } + }); +}); diff --git a/tests/component/proposal-materializer.test.ts b/tests/component/proposal-materializer.test.ts new file mode 100644 index 0000000..d7e3439 --- /dev/null +++ b/tests/component/proposal-materializer.test.ts @@ -0,0 +1,112 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession } from "./helpers.js"; +import { computeProposalInputKey, materializeProposalsFromNode } from "../../src/analyze/proposal-materializer.js"; +import { insertNode } from "../../src/db/analysis-queries.js"; +import { listProposals } from "../../src/db/queries.js"; + +function seedNode(db: import("better-sqlite3").Database, id: string): void { + insertNode(db, { + id, + sessionId: "s1", + analyzerId: "session-overview", + analyzerVersionId: "1.0.0", + configId: "c", + runId: null, + nodeKind: "summary", + contentJson: "{}", + sourceSetHash: "ssh", + inputKey: `ih-${id}`, + outputKey: `ok-${id}`, + createdAt: new Date().toISOString(), + }); +} + +describe("computeProposalInputKey", () => { + it("derives from the source output_key + ordinal, never the LLM text", () => { + const a = computeProposalInputKey({ sourceOutputKey: "ok-1", ordinal: 0 }); + const b = computeProposalInputKey({ sourceOutputKey: "ok-1", ordinal: 0 }); + assert.equal(a, b, "same source+ordinal is stable regardless of title/path/severity"); + }); + + it("differs across ordinal and across source", () => { + assert.notEqual( + computeProposalInputKey({ sourceOutputKey: "ok-1", ordinal: 0 }), + computeProposalInputKey({ sourceOutputKey: "ok-1", ordinal: 1 }), + ); + assert.notEqual( + computeProposalInputKey({ sourceOutputKey: "ok-1", ordinal: 0 }), + computeProposalInputKey({ sourceOutputKey: "ok-2", ordinal: 0 }), + ); + }); +}); + +describe("materializeProposalsFromNode", () => { + it("inserts valid proposals and links them with produces edges", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + seedNode(db, "node1"); + const created = materializeProposalsFromNode(db, { + sessionId: "s1", + analyzerId: "session-overview", + sourceNodeId: "node1", + sourceOutputKey: "ok-node1", + now: new Date().toISOString(), + contentJson: { + improvement_proposals: [ + { target_type: "agents_md", target_path: "AGENTS.md", title: "Add tooling note", summary: "s", severity: "friction", confidence: 0.8 }, + { title: "", summary: "missing title" }, + { title: "no summary" }, + ], + }, + }); + assert.equal(created, 1); + + const proposals = listProposals(db); + assert.equal(proposals.length, 1); + assert.equal(proposals[0]!.target_type, "agents_md"); + assert.equal(proposals[0]!.status, "open"); + + const edge = db + .prepare("SELECT * FROM analysis_edges WHERE from_node_id = ? AND edge_kind = 'produces'") + .get("node1") as { to_ref_id: string } | undefined; + assert.ok(edge); + assert.equal(edge!.to_ref_id, proposals[0]!.id); + } finally { + close(); + } + }); + + it("is idempotent for the same source node, but keeps duplicates from distinct sources", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + seedNode(db, "n1"); + seedNode(db, "n2"); + const payload = { + improvement_proposals: [{ target_type: "config", title: "Same thing", summary: "s", severity: "friction" }], + }; + // Same source node, materialised twice → idempotent (keyed on source output_key + ordinal). + assert.equal(materializeProposalsFromNode(db, { sessionId: "s1", analyzerId: "a", sourceNodeId: "n1", sourceOutputKey: "ok-n1", now: new Date().toISOString(), contentJson: payload }), 1); + assert.equal(materializeProposalsFromNode(db, { sessionId: "s1", analyzerId: "a", sourceNodeId: "n1", sourceOutputKey: "ok-n1", now: new Date().toISOString(), contentJson: payload }), 0); + // Distinct source node with byte-identical text → intentionally retained. + assert.equal(materializeProposalsFromNode(db, { sessionId: "s1", analyzerId: "a", sourceNodeId: "n2", sourceOutputKey: "ok-n2", now: new Date().toISOString(), contentJson: payload }), 1); + assert.equal(listProposals(db).length, 2); + } finally { + close(); + } + }); + + it("returns 0 when there are no proposals", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + seedNode(db, "n1"); + assert.equal(materializeProposalsFromNode(db, { sessionId: "s1", analyzerId: "a", sourceNodeId: "n1", sourceOutputKey: "ok-n1", now: new Date().toISOString(), contentJson: {} }), 0); + assert.equal(materializeProposalsFromNode(db, { sessionId: "s1", analyzerId: "a", sourceNodeId: "n1", sourceOutputKey: "ok-n1", now: new Date().toISOString(), contentJson: { improvement_proposals: "not-an-array" } }), 0); + } finally { + close(); + } + }); +}); diff --git a/tests/component/queries.test.ts b/tests/component/queries.test.ts new file mode 100644 index 0000000..91e0b7e --- /dev/null +++ b/tests/component/queries.test.ts @@ -0,0 +1,61 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession, insertProposalRow } from "./helpers.js"; +import { acceptProposal, getProposal, getStats, listProposals, rejectProposal } from "../../src/db/queries.js"; + +describe("proposal queries (v2)", () => { + it("lists, filters, accepts, and rejects", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + insertProposalRow(db, { id: "p1", sessionId: "s1", title: "A", severity: "friction" }); + insertProposalRow(db, { id: "p2", sessionId: "s1", title: "B", severity: "waste" }); + + assert.equal(listProposals(db).length, 2); + assert.equal(listProposals(db, "open").length, 2); + + assert.equal(acceptProposal(db, "p1"), true); + assert.equal(rejectProposal(db, "p2"), true); + + assert.equal(listProposals(db, "applied").length, 1); + assert.equal(listProposals(db, "rejected").length, 1); + assert.equal(getProposal(db, "p1")!.status, "applied"); + } finally { + close(); + } + }); + + it("accept/reject only affect open proposals", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + insertProposalRow(db, { id: "p1", sessionId: "s1", title: "A", status: "applied" }); + assert.equal(acceptProposal(db, "p1"), false); + assert.equal(rejectProposal(db, "p1"), false); + assert.equal(acceptProposal(db, "missing"), false); + } finally { + close(); + } + }); + + it("getStats reports v2 status counts and analysis stats", () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + insertProposalRow(db, { id: "pa", sessionId: "s1", title: "a" }); + insertProposalRow(db, { id: "pb", sessionId: "s1", title: "b", status: "applied" }); + insertProposalRow(db, { id: "pc", sessionId: "s1", title: "c", status: "duplicate" }); + + const stats = getStats(db); + assert.equal(stats.proposalsByStatus.open, 1); + assert.equal(stats.proposalsByStatus.applied, 1); + assert.equal(stats.proposalsByStatus.duplicate, 1); + assert.equal(stats.proposalsByStatus.rejected, 0); + assert.equal(stats.totalSessions, 1); + assert.equal(stats.analysis.nodes, 0); + assert.deepEqual(stats.analysis.nodesByKind, {}); + } finally { + close(); + } + }); +}); diff --git a/tests/component/revise-reasons.test.ts b/tests/component/revise-reasons.test.ts new file mode 100644 index 0000000..0f37981 --- /dev/null +++ b/tests/component/revise-reasons.test.ts @@ -0,0 +1,109 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession, insertMessages } from "./helpers.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { createThrowingLLM } from "../../src/analyze/mock-llm.js"; +import { turnPairCoreAnalyzer } from "../../src/analyze/analyzers/turn-pair-core/index.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; +import type { Analyzer, ReviseReason } from "../../src/analyze/types.js"; + +function seed(db: import("better-sqlite3").Database): void { + insertSession(db, "s1"); + insertMessages(db, "s1", [ + { role: "user", text: "fix the login bug" }, + { role: "assistant", text: "looking", toolCalls: [{ name: "read" }] }, + { role: "toolResult", toolResults: [{ toolName: "read", isError: true, textLength: 50 }] }, + { role: "user", text: "no, that's wrong, use the auth module" }, + { role: "assistant", text: "fixing" }, + ]); +} + +/** Fill the graph with the base v1.0 analyzer. */ +async function fillV1(db: import("better-sqlite3").Database): Promise { + const fw = new AnalyzerFramework({ db, llm: createThrowingLLM(), modelTiers: DEFAULT_MODEL_TIERS }); + fw.register(turnPairCoreAnalyzer); + await fw.run("s1", {}); +} + +/** A fresh framework registered with `analyzer`, then run with the given reasons. */ +async function run(db: import("better-sqlite3").Database, analyzer: Analyzer, revise: ReviseReason[]) { + const fw = new AnalyzerFramework({ db, llm: createThrowingLLM(), modelTiers: DEFAULT_MODEL_TIERS }); + fw.register(analyzer); + return fw.run("s1", { revise }); +} + +async function scan(db: import("better-sqlite3").Database, analyzer: Analyzer) { + const fw = new AnalyzerFramework({ db, llm: createThrowingLLM(), modelTiers: DEFAULT_MODEL_TIERS }); + fw.register(analyzer); + return fw.scan("s1"); +} + +const minorBump: Analyzer = { + ...turnPairCoreAnalyzer, + version: { ...turnPairCoreAnalyzer.version, minor: turnPairCoreAnalyzer.version.minor + 1 }, +}; +const majorBump: Analyzer = { + ...turnPairCoreAnalyzer, + version: { ...turnPairCoreAnalyzer.version, major: turnPairCoreAnalyzer.version.major + 1 }, +}; +const configChange: Analyzer = { + ...turnPairCoreAnalyzer, + defaultConfig: { + ...turnPairCoreAnalyzer.defaultConfig, + configJson: { ...turnPairCoreAnalyzer.defaultConfig.configJson, frictionThresholdTweak: 99 }, + }, +}; + +describe("revise reasons select which stale units to recompute", () => { + it("a minor bump is graded `minor`; --revise major skips it, --revise minor revises it", async () => { + const { db, close } = tempDb(); + try { + seed(db); + await fillV1(db); + + const classified = await scan(db, minorBump); + assert.ok(classified.length >= 1); + assert.ok( + classified.every((c) => c.status === "stale" && c.reasons.includes("minor") && !c.reasons.includes("major")), + "a minor bump grades only as minor", + ); + + assert.equal((await run(db, minorBump, ["major"])).nodesRevised, 0, "--revise major skips a minor-only change"); + assert.ok((await run(db, minorBump, ["minor"])).nodesRevised >= 1, "--revise minor revises it"); + } finally { + close(); + } + }); + + it("a major bump is graded `major` and is revised by --revise major", async () => { + const { db, close } = tempDb(); + try { + seed(db); + await fillV1(db); + assert.ok((await run(db, majorBump, ["major"])).nodesRevised >= 1, "a major bump is picked up by --revise major"); + } finally { + close(); + } + }); + + it("a config change is graded `config` only; --revise config revises, --revise major does not", async () => { + const { db, close } = tempDb(); + try { + seed(db); + await fillV1(db); + + const classified = await scan(db, configChange); + assert.ok( + classified.every( + (u) => u.status === "stale" && u.reasons.includes("config") && !u.reasons.includes("major") && !u.reasons.includes("minor"), + ), + "a config change carries only the config reason", + ); + + assert.equal((await run(db, configChange, ["major"])).nodesRevised, 0, "--revise major ignores a config-only change"); + assert.ok((await run(db, configChange, ["config"])).nodesRevised >= 1, "--revise config revises it"); + } finally { + close(); + } + }); +}); diff --git a/tests/component/schema.test.ts b/tests/component/schema.test.ts new file mode 100644 index 0000000..7b3e114 --- /dev/null +++ b/tests/component/schema.test.ts @@ -0,0 +1,87 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb } from "./helpers.js"; + +function tableColumns(db: import("better-sqlite3").Database, table: string): Set { + const rows = db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>; + return new Set(rows.map((r) => r.name)); +} + +function tableExists(db: import("better-sqlite3").Database, table: string): boolean { + return !!db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = ?").get(table); +} + +describe("schema migration", () => { + it("creates all core and framework tables", () => { + const { db, close } = tempDb(); + try { + for (const t of [ + "sessions", + "messages", + "proposals", + "analyzer_defs", + "analyzer_versions", + "prompt_registry", + "analyzer_configs", + "analysis_runs", + "analysis_nodes", + "analysis_edges", + ]) { + assert.ok(tableExists(db, t), `missing table ${t}`); + } + } finally { + close(); + } + }); + + it("proposals table has v2 columns", () => { + const { db, close } = tempDb(); + try { + const cols = tableColumns(db, "proposals"); + for (const c of ["target_type", "target_path", "title", "confidence", "status", "input_key", "source_node_id", "updated_at"]) { + assert.ok(cols.has(c), `proposals missing ${c}`); + } + } finally { + close(); + } + }); + + it("analysis_nodes carries the config fingerprint (config dimension of identity)", () => { + const { db, close } = tempDb(); + try { + assert.ok(tableColumns(db, "analysis_nodes").has("config_fingerprint"), "analysis_nodes missing config_fingerprint"); + } finally { + close(); + } + }); + + it("analysis_nodes enforces unique input_key", () => { + const { db, close } = tempDb(); + try { + db.prepare("INSERT INTO sessions (id, file_path) VALUES ('s', '/tmp/s.jsonl')").run(); + const insert = (inputKey: string) => + db + .prepare( + "INSERT INTO analysis_nodes (id, session_id, analyzer_id, analyzer_version_id, config_id, node_kind, content_json, source_set_hash, input_key, created_at) " + + "VALUES (?, 's', 'a', '1', 'c', 'metric', '{}', 'ssh', ?, ?)", + ) + .run(Math.random().toString(36), inputKey, new Date().toISOString()); + insert("h1"); + assert.throws(() => insert("h1"), /UNIQUE/); + } finally { + close(); + } + }); + + it("is idempotent (re-running migrate is safe)", () => { + const { db, close } = tempDb(); + try { + assert.doesNotThrow(() => { + // migrate already ran in tempDb; run sync-like usage again + db.prepare("SELECT COUNT(*) FROM analysis_nodes").get(); + }); + } finally { + close(); + } + }); +}); diff --git a/tests/component/show.test.ts b/tests/component/show.test.ts new file mode 100644 index 0000000..95339b6 --- /dev/null +++ b/tests/component/show.test.ts @@ -0,0 +1,111 @@ +import { describe, it, before, after } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import Database from "better-sqlite3"; +import { migrate } from "../../src/db/schema.js"; +import { insertSession, insertMessages } from "./helpers.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { createMockLLM } from "../../src/analyze/mock-llm.js"; +import { registerDefaults } from "../../src/analyze/defaults.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; +import { listProposals } from "../../src/db/queries.js"; +import { resolveProposal, prospectShow } from "../../src/commands/show.js"; +import type { LLMRequest } from "../../src/analyze/types.js"; +import type { ExtensionCommandContext } from "../../src/pi-stubs.js"; + +function respond(req: LLMRequest): string { + const sys = req.system ?? ""; + if (sys.includes("classify a single turn")) { + return JSON.stringify({ sentiment: "frustrated", friction_type: "wrong_approach", is_genuine_correction: true, severity: "high", rationale: "corrected" }); + } + if (sys.includes("summarise one segment")) return JSON.stringify({ segment_summary: "seg", notable_points: [] }); + return JSON.stringify({ + session_summary: "A wrong approach was corrected.", + key_friction_points: [{ description: "wrong approach", severity: "high" }], + improvement_proposals: [ + { target_type: "agents_md", target_path: "AGENTS.md", title: "Document the auth module", summary: "s", detail: "d", evidence: "user corrected in turn 2", confidence: 0.7, severity: "correction" }, + ], + }); +} + +const notes: string[] = []; +const ctx: ExtensionCommandContext = { + modelRegistry: { find: () => undefined, getAll: () => [], getAvailable: () => [], getApiKeyAndHeaders: async () => ({ ok: false, error: "x" }) }, + hasUI: false, + ui: { notify: (m) => notes.push(m) }, +}; + +let tmpDir: string; +let dbPath: string; + +before(async () => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "prospector-show-")); + dbPath = path.join(tmpDir, "show.db"); + process.env["PROSPECTOR_DB_PATH"] = dbPath; + const db = new Database(dbPath); + migrate(db); + insertSession(db, "s1"); + insertMessages(db, "s1", [ + { id: "s1-m0", role: "user", text: "fix the login bug" }, + { id: "s1-m1", role: "assistant", text: "reading auth", toolCalls: [{ name: "read" }] }, + { id: "s1-m2", role: "toolResult", toolResults: [{ toolName: "read", isError: true, textLength: 40 }] }, + { id: "s1-m3", role: "user", text: "no, that's wrong, use the auth module instead" }, + { id: "s1-m4", role: "assistant", text: "understood" }, + ]); + const mock = createMockLLM({ responder: respond, tokensPerCall: 50, costPerCall: 0.001 }); + const fw = new AnalyzerFramework({ db, llm: mock.caller, modelTiers: DEFAULT_MODEL_TIERS }); + registerDefaults(fw); + const summary = await fw.run("s1", {}); + assert.equal(summary.errors.length, 0, summary.errors.join("; ")); + db.close(); +}); + +after(() => { + delete process.env["PROSPECTOR_DB_PATH"]; + try { + fs.rmSync(tmpDir, { recursive: true, force: true }); + } catch { + /* ignore */ + } +}); + +async function show(ref: string): Promise { + notes.length = 0; + await prospectShow(ref, ctx); + return notes.join("\n"); +} + +describe("prospect-show", () => { + it("resolves a proposal by exact id and by unambiguous prefix; rejects unknown", () => { + const db = new Database(dbPath); + try { + const all = listProposals(db); + assert.ok(all.length >= 1); + const id = all[0]!.id; + assert.equal(resolveProposal(db, id).proposal?.id, id); + assert.equal(resolveProposal(db, id.slice(0, 18)).proposal?.id, id); + assert.equal(resolveProposal(db, "no-such-id").matches.length, 0); + } finally { + db.close(); + } + }); + + it("prints the proposal and reconstructs the verbatim anchored turns from the graph", async () => { + const db = new Database(dbPath); + const id = listProposals(db)[0]!.id; + db.close(); + + const text = await show(id); + assert.match(text, /Document the auth module/); // proposal title + assert.match(text, /Anchored turns/); + assert.match(text, /no, that's wrong, use the auth module/); // verbatim user correction + assert.match(text, /session-overview/); // source provenance shown + }); + + it("warns on an unknown proposal id", async () => { + const text = await show("definitely-not-a-real-id"); + assert.match(text, /No proposal matches/); + }); +}); diff --git a/tests/component/sync.test.ts b/tests/component/sync.test.ts index d5fee63..3dcbda2 100644 --- a/tests/component/sync.test.ts +++ b/tests/component/sync.test.ts @@ -1,22 +1,12 @@ import { describe, it } from "node:test"; import assert from "node:assert/strict"; -import * as fs from "node:fs"; import * as path from "node:path"; -import * as os from "node:os"; -import Database from "better-sqlite3"; -import { migrate } from "../../src/db/schema.js"; import { runSync } from "../../src/sync/index.js"; -import { getStats, insertProposal, listProposals, acceptProposal, rejectProposal } from "../../src/db/queries.js"; +import { getStats } from "../../src/db/queries.js"; +import { tempDb } from "./helpers.js"; const FIXTURES = path.resolve(import.meta.dirname, "..", "fixtures"); -function tempDb(): { db: Database.Database; close: () => void } { - const dbPath = path.join(os.tmpdir(), `prospect-test-${Date.now()}-${Math.random().toString(36).slice(2)}.db`); - const db = new Database(dbPath); - migrate(db); - return { db, close: () => { db.close(); try { fs.unlinkSync(dbPath); } catch {} } }; -} - describe("end-to-end sync", () => { it("syncs simple.jsonl into database", () => { const { db, close } = tempDb(); @@ -38,7 +28,6 @@ describe("end-to-end sync", () => { runSync(db, FIXTURES); const stats1 = getStats(db); - // Second sync should skip all const result2 = runSync(db, FIXTURES); assert.ok(result2.sessionsSkipped >= 1); assert.equal(result2.messagesInserted, 0); @@ -53,8 +42,7 @@ describe("end-to-end sync", () => { it("handles compacted session (compactionSummary entries)", () => { const { db, close } = tempDb(); try { - const result = runSync(db, FIXTURES); - // compacted.jsonl should be among those synced + runSync(db, FIXTURES); const stats = getStats(db); assert.ok(stats.totalSessions >= 2, "should index at least 2 sessions (simple + compacted)"); } finally { @@ -62,76 +50,3 @@ describe("end-to-end sync", () => { } }); }); - -describe("proposals", () => { - it("inserts and retrieves a proposal", () => { - const { db, close } = tempDb(); - try { - // First insert a session so FK works - runSync(db, FIXTURES); - - // Get a session ID from the DB - const row = db.prepare("SELECT id FROM sessions LIMIT 1").get() as { id: string }; - - insertProposal(db, { - id: "p-test-001", - created_at: new Date().toISOString(), - session_id: row.id, - target: "AGENTS.md § Tool usage", - severity: "friction", - summary: "Agent reads entire files instead of sections", - detail: "Details here", - evidence: "Evidence here", - status: "new", - dedup_hash: "test-hash-001", - }); - - const proposals = listProposals(db); - assert.ok(proposals.length >= 1); - assert.equal(proposals[0]!.target, "AGENTS.md § Tool usage"); - } finally { - close(); - } - }); - - it("accepts and rejects proposals", () => { - const { db, close } = tempDb(); - try { - runSync(db, FIXTURES); - const row = db.prepare("SELECT id FROM sessions LIMIT 1").get() as { id: string }; - - insertProposal(db, { id: "p1", created_at: new Date().toISOString(), session_id: row.id, target: "t1", severity: "friction", summary: "s1", detail: "", evidence: "", status: "new", dedup_hash: "h1" }); - insertProposal(db, { id: "p2", created_at: new Date().toISOString(), session_id: row.id, target: "t2", severity: "correction", summary: "s2", detail: "", evidence: "", status: "new", dedup_hash: "h2" }); - - assert.equal(acceptProposal(db, "p1"), true); - assert.equal(rejectProposal(db, "p2"), true); - - const accepted = listProposals(db, "accepted"); - assert.equal(accepted.length, 1); - assert.equal(accepted[0]!.id, "p1"); - - const rejected = listProposals(db, "rejected"); - assert.equal(rejected.length, 1); - assert.equal(rejected[0]!.id, "p2"); - } finally { - close(); - } - }); - - it("stats include proposal counts", () => { - const { db, close } = tempDb(); - try { - runSync(db, FIXTURES); - const row = db.prepare("SELECT id FROM sessions LIMIT 1").get() as { id: string }; - - insertProposal(db, { id: "pa", created_at: new Date().toISOString(), session_id: row.id, target: "a", severity: "friction", summary: "a", detail: "", evidence: "", status: "new", dedup_hash: "ha" }); - insertProposal(db, { id: "pb", created_at: new Date().toISOString(), session_id: row.id, target: "b", severity: "waste", summary: "b", detail: "", evidence: "", status: "accepted", dedup_hash: "hb" }); - - const stats = getStats(db); - assert.equal(stats.proposalsByStatus.new, 1); - assert.equal(stats.proposalsByStatus.accepted, 1); - } finally { - close(); - } - }); -}); \ No newline at end of file diff --git a/tests/component/turn-pair-core.test.ts b/tests/component/turn-pair-core.test.ts new file mode 100644 index 0000000..b23ad3b --- /dev/null +++ b/tests/component/turn-pair-core.test.ts @@ -0,0 +1,68 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { tempDb, insertSession, insertMessages } from "./helpers.js"; +import { AnalyzerFramework } from "../../src/analyze/framework.js"; +import { createThrowingLLM } from "../../src/analyze/mock-llm.js"; +import { turnPairCoreAnalyzer, type TurnPairCoreProperties } from "../../src/analyze/analyzers/turn-pair-core/index.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; + +async function runCore(db: import("better-sqlite3").Database, sessionId: string): Promise { + const fw = new AnalyzerFramework({ db, llm: createThrowingLLM(), modelTiers: DEFAULT_MODEL_TIERS }); + fw.register(turnPairCoreAnalyzer); + await fw.run(sessionId, {}); + const rows = db + .prepare("SELECT content_json FROM analysis_nodes WHERE analyzer_id = 'turn-pair-core' ORDER BY rowid") + .all() as Array<{ content_json: string }>; + return rows.map((r) => JSON.parse(r.content_json) as TurnPairCoreProperties); +} + +describe("turn-pair-core scoring", () => { + it("scores a clean turn with low friction", async () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + insertMessages(db, "s1", [ + { role: "user", text: "please add a test" }, + { role: "assistant", text: "added", toolCalls: [{ name: "edit" }] }, + ]); + const props = await runCore(db, "s1"); + assert.equal(props.length, 1); + assert.equal(props[0]!.correction_detected, false); + assert.equal(props[0]!.high_signal, false); + assert.equal(props[0]!.friction_score, 0); + } finally { + close(); + } + }); + + it("flags corrections, tool failures, waste, and empty responses", async () => { + const { db, close } = tempDb(); + try { + insertSession(db, "s1"); + insertMessages(db, "s1", [ + // pair 0: correction + tool failure + { role: "user", text: "no, that's wrong, use yarn" }, + { role: "assistant", text: "ok", toolCalls: [{ name: "bash" }] }, + { role: "toolResult", toolResults: [{ toolName: "bash", isError: true, textLength: 10 }] }, + // pair 1: huge tool output (waste) + empty assistant response + { role: "user", text: "show me the file" }, + { role: "toolResult", toolResults: [{ toolName: "read", isError: false, textLength: 50000 }] }, + ]); + const props = await runCore(db, "s1"); + assert.equal(props.length, 2); + + const p0 = props[0]!; + assert.equal(p0.correction_detected, true); + assert.equal(p0.tool_failure_count, 1); + assert.ok(p0.friction_score >= 0.5); + assert.equal(p0.high_signal, true); + + const p1 = props[1]!; + assert.ok(p1.tool_waste_bytes > 0); + assert.equal(p1.empty_response, true); + assert.ok(p1.friction_score > 0); + } finally { + close(); + } + }); +}); diff --git a/tests/unit/config.test.ts b/tests/unit/config.test.ts new file mode 100644 index 0000000..cda6b71 --- /dev/null +++ b/tests/unit/config.test.ts @@ -0,0 +1,53 @@ +import { describe, it, afterEach } from "node:test"; +import assert from "node:assert/strict"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { getDbPath, getModelTiers, getSessionsDir, loadConfig } from "../../src/config.js"; +import { DEFAULT_MODEL_TIERS } from "../../src/analyze/model-tiers.js"; + +const ENV_KEYS = ["PROSPECTOR_CONFIG", "PROSPECTOR_DB_PATH", "PROSPECTOR_SESSIONS_DIR"]; + +afterEach(() => { + for (const k of ENV_KEYS) delete process.env[k]; +}); + +describe("config", () => { + it("returns {} when no config file is present", () => { + process.env["PROSPECTOR_CONFIG"] = path.join(os.tmpdir(), `nope-${Date.now()}.json`); + assert.deepEqual(loadConfig(), {}); + }); + + it("loads a config file via PROSPECTOR_CONFIG", () => { + const file = path.join(os.tmpdir(), `cfg-${Date.now()}.json`); + fs.writeFileSync(file, JSON.stringify({ model: "anthropic/x", dbPath: "/tmp/x.db" })); + process.env["PROSPECTOR_CONFIG"] = file; + try { + const c = loadConfig(); + assert.equal(c.model, "anthropic/x"); + assert.equal(getDbPath(c), "/tmp/x.db"); + } finally { + fs.unlinkSync(file); + } + }); + + it("expands a leading ~ in dbPath", () => { + assert.equal(getDbPath({ dbPath: "~/foo.db" }), path.join(os.homedir(), "/foo.db")); + }); + + it("honours PROSPECTOR_DB_PATH when config has none", () => { + process.env["PROSPECTOR_DB_PATH"] = "/tmp/env.db"; + assert.equal(getDbPath({}), "/tmp/env.db"); + }); + + it("honours PROSPECTOR_SESSIONS_DIR", () => { + process.env["PROSPECTOR_SESSIONS_DIR"] = "/tmp/sessions"; + assert.equal(getSessionsDir(), "/tmp/sessions"); + }); + + it("getModelTiers falls back to defaults and respects config", () => { + assert.deepEqual(getModelTiers({}), DEFAULT_MODEL_TIERS); + const custom = { cheap: "a/b", mid: "c/d", expensive: "e/f" }; + assert.deepEqual(getModelTiers({ modelTiers: custom }), custom); + }); +}); diff --git a/tests/unit/digest.test.ts b/tests/unit/digest.test.ts new file mode 100644 index 0000000..18de92a --- /dev/null +++ b/tests/unit/digest.test.ts @@ -0,0 +1,225 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { buildDigest, splitDigest } from "../../src/analyze/analyzers/session-overview/digest.js"; +import type { AnalysisNodeRow, MessageRow } from "../../src/analyze/types.js"; +import type { TurnPairCoreProperties } from "../../src/analyze/analyzers/turn-pair-core/index.js"; +import type { TurnPairLLMProperties } from "../../src/analyze/analyzers/turn-pair-llm/prompt.js"; + +function coreNode(id: string, props: Partial): AnalysisNodeRow { + const full: TurnPairCoreProperties = { + pair_index: props.pair_index ?? 0, + user_message_id: props.user_message_id ?? "u", + correction_detected: props.correction_detected ?? false, + correction_type: props.correction_type ?? null, + correction_patterns: props.correction_patterns ?? [], + correction_text: props.correction_text ?? null, + tool_call_count: props.tool_call_count ?? 0, + tool_failure_count: props.tool_failure_count ?? 0, + tool_result_bytes: props.tool_result_bytes ?? 0, + tool_waste_bytes: props.tool_waste_bytes ?? 0, + empty_response: props.empty_response ?? false, + friction_score: props.friction_score ?? 0, + high_signal: props.high_signal ?? false, + }; + return { + id, + session_id: "s1", + analyzer_id: "turn-pair-core", + analyzer_version_id: "1.0.0", + config_id: "c", + run_id: null, + node_kind: "metric", + content_json: JSON.stringify(full), + source_set_hash: "ssh", + config_fingerprint: "", + input_key: id, + output_key: id, + model_used: null, + cost_usd: null, + tokens_used: null, + duration_ms: null, + created_at: new Date().toISOString(), + }; +} + +const NO_MESSAGES: MessageRow[] = []; + +function llmNode(id: string, props: TurnPairLLMProperties): AnalysisNodeRow { + return { + ...coreNode(id, {}), + analyzer_id: "turn-pair-llm", + node_kind: "classification", + content_json: JSON.stringify(props), + }; +} + +describe("buildDigest", () => { + it("aggregates counts and renders per-pair lines", () => { + const digest = buildDigest({ + sessionId: "s1", + messages: NO_MESSAGES, + coreNodes: [ + coreNode("n1", { pair_index: 0, friction_score: 0.7, high_signal: true, correction_detected: true, correction_type: "explicit" }), + coreNode("n2", { pair_index: 1, friction_score: 0.1, tool_failure_count: 0 }), + ], + llmNodes: [], + }); + assert.equal(digest.pairCount, 2); + assert.equal(digest.frictionCount, 1); + assert.equal(digest.correctionCount, 1); + assert.equal(digest.perPairLines.length, 2); + assert.ok(digest.text.includes("#0")); + }); + + it("orders pairs by index regardless of node order", () => { + const digest = buildDigest({ + sessionId: "s1", + messages: NO_MESSAGES, + coreNodes: [coreNode("n2", { pair_index: 5 }), coreNode("n1", { pair_index: 1 })], + llmNodes: [], + }); + assert.ok(digest.perPairLines[0]!.startsWith("#1")); + assert.ok(digest.perPairLines[1]!.startsWith("#5")); + }); + + it("includes compaction summaries verbatim", () => { + const messages: MessageRow[] = [ + { + id: "c1", + session_id: "s1", + parent_id: null, + timestamp: null, + role: "compactionSummary", + content_text: "PRIOR CONTEXT: refactored auth", + content_thinking: null, + tool_calls: null, + tool_results: null, + }, + ]; + const digest = buildDigest({ sessionId: "s1", messages, coreNodes: [coreNode("n1", {})], llmNodes: [] }); + assert.equal(digest.compactionCount, 1); + assert.ok(digest.text.includes("refactored auth")); + }); + + it("merges turn-pair-llm enrichment onto the matching pair by user_message_id", () => { + const digest = buildDigest({ + sessionId: "s1", + messages: NO_MESSAGES, + coreNodes: [ + coreNode("n1", { pair_index: 0, user_message_id: "u-hot", friction_score: 0.8, high_signal: true }), + coreNode("n2", { pair_index: 1, user_message_id: "u-cold", friction_score: 0.1 }), + ], + llmNodes: [ + llmNode("l1", { + user_message_id: "u-hot", + sentiment: "frustrated", + friction_type: "wrong_approach", + is_genuine_correction: true, + severity: "high", + rationale: "x", + }), + ], + }); + const hotLine = digest.perPairLines.find((l) => l.startsWith("#0"))!; + const coldLine = digest.perPairLines.find((l) => l.startsWith("#1"))!; + assert.ok(hotLine.includes("sentiment=frustrated"), "enriched pair shows LLM sentiment"); + assert.ok(hotLine.includes("type=wrong_approach") && hotLine.includes("sev=high")); + assert.ok(!coldLine.includes("sentiment="), "un-enriched pair has no LLM fields"); + }); + + it("includes branch summaries verbatim (Pi's snake_case branch_summary role)", () => { + const messages: MessageRow[] = [ + { + id: "b1", + session_id: "s1", + parent_id: null, + timestamp: null, + role: "branch_summary", + content_text: "BRANCH CONTEXT: split off to try OAuth", + content_thinking: null, + tool_calls: null, + tool_results: null, + }, + ]; + const digest = buildDigest({ sessionId: "s1", messages, coreNodes: [coreNode("n1", {})], llmNodes: [] }); + assert.equal(digest.compactionCount, 1); + assert.ok(digest.text.includes("split off to try OAuth")); + }); + + it("tolerates malformed node content", () => { + const bad: AnalysisNodeRow = { ...coreNode("n1", {}), content_json: "{bad" }; + const digest = buildDigest({ sessionId: "s1", messages: NO_MESSAGES, coreNodes: [bad], llmNodes: [] }); + assert.equal(digest.pairCount, 0); + }); + + it("includes user text for every pair, not just correction-matched ones (un-gate)", () => { + // A pair whose user text is an unrecognized correction (no regex match). + // Before the un-gating change, this pair would have no `note=` or `text=` field. + const digest = buildDigest({ + sessionId: "s1", + messages: [ + { id: "u-circleci", session_id: "s1", parent_id: null, timestamp: null, role: "user", content_text: "This repo does not use CircleCI", content_thinking: null, tool_calls: null, tool_results: null }, + ], + coreNodes: [ + coreNode("n1", { pair_index: 0, user_message_id: "u-circleci", correction_detected: false, friction_score: 0.3 }), + ], + llmNodes: [], + }); + const line = digest.perPairLines[0]!; + // The key assertion: even without correction_detected, the user text appears. + assert.ok(line.includes("text="), "un-gated pair must have a text= snippet"); + assert.ok(line.includes("CircleCI"), "text= must contain the user's actual words"); + }); + + it("includes user text snippet even for pairs with no correction at all", () => { + const digest = buildDigest({ + sessionId: "s1", + messages: [ + { id: "u-plain", session_id: "s1", parent_id: null, timestamp: null, role: "user", content_text: "please add a test for this", content_thinking: null, tool_calls: null, tool_results: null }, + ], + coreNodes: [ + coreNode("n1", { pair_index: 0, user_message_id: "u-plain", friction_score: 0.05 }), + ], + llmNodes: [], + }); + const line = digest.perPairLines[0]!; + assert.ok(line.includes("text="), "plain pair must have a text= snippet"); + assert.ok(line.includes("add a test"), "text= must contain the user text"); + }); + + it("truncates long user text to the budget", () => { + const longText = "a".repeat(500); + const digest = buildDigest({ + sessionId: "s1", + messages: [ + { id: "u-long", session_id: "s1", parent_id: null, timestamp: null, role: "user", content_text: longText, content_thinking: null, tool_calls: null, tool_results: null }, + ], + coreNodes: [ + coreNode("n1", { pair_index: 0, user_message_id: "u-long", friction_score: 0.1 }), + ], + llmNodes: [], + }); + const line = digest.perPairLines[0]!; + // The text field should be truncated to USER_TEXT_SNIPPET_MAX (200) + ellipsis + assert.ok(line.includes("text=\""), "must have text field"); + // Extract the text field value and check it's truncated + const match = line.match(/text="([^"]*)"/); + assert.ok(match, "text field must be extractable"); + assert.ok(match[1]!.length <= 201, "truncated text must be within budget"); + }); +}); + +describe("splitDigest", () => { + it("returns a single segment when under budget", () => { + const digest = buildDigest({ sessionId: "s1", messages: NO_MESSAGES, coreNodes: [coreNode("n1", {})], llmNodes: [] }); + assert.equal(splitDigest(digest, 100000).length, 1); + }); + + it("splits into multiple segments when over budget", () => { + const nodes = Array.from({ length: 40 }, (_, i) => coreNode(`n${i}`, { pair_index: i, correction_text: "x".repeat(100) })); + const digest = buildDigest({ sessionId: "s1", messages: NO_MESSAGES, coreNodes: nodes, llmNodes: [] }); + const segments = splitDigest(digest, 500); + assert.ok(segments.length > 1); + for (const seg of segments) assert.ok(seg.text.includes("Per-pair signals")); + }); +}); diff --git a/tests/unit/edge-kinds.test.ts b/tests/unit/edge-kinds.test.ts new file mode 100644 index 0000000..0c0a673 --- /dev/null +++ b/tests/unit/edge-kinds.test.ts @@ -0,0 +1,33 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { EDGE_KINDS, REF_KINDS, isEdgeKind, isRefKind, validateEdge } from "../../src/analyze/edge-kinds.js"; + +describe("edge kinds", () => { + it("recognises valid edge and ref kinds", () => { + assert.ok(isEdgeKind(EDGE_KINDS.ANCHORS)); + assert.ok(isEdgeKind(EDGE_KINDS.REVISES)); + assert.ok(isRefKind(REF_KINDS.SESSION)); + assert.ok(!isEdgeKind("bogus")); + assert.ok(!isRefKind("bogus")); + }); + + it("accepts allowed edge → ref combinations", () => { + assert.doesNotThrow(() => validateEdge(EDGE_KINDS.ANCHORS, REF_KINDS.SESSION)); + assert.doesNotThrow(() => validateEdge(EDGE_KINDS.ANCHORS, REF_KINDS.MESSAGE)); + assert.doesNotThrow(() => validateEdge(EDGE_KINDS.CONSUMES, REF_KINDS.ANALYSIS_NODE)); + assert.doesNotThrow(() => validateEdge(EDGE_KINDS.USES_PROMPT, REF_KINDS.PROMPT_VERSION)); + assert.doesNotThrow(() => validateEdge(EDGE_KINDS.USES_CONFIG, REF_KINDS.CONFIG_VERSION)); + assert.doesNotThrow(() => validateEdge(EDGE_KINDS.PRODUCES, REF_KINDS.PROPOSAL)); + assert.doesNotThrow(() => validateEdge(EDGE_KINDS.REVISES, REF_KINDS.ANALYSIS_NODE)); + }); + + it("rejects disallowed edge → ref combinations", () => { + assert.throws(() => validateEdge(EDGE_KINDS.ANCHORS, REF_KINDS.PROPOSAL), /cannot target/); + assert.throws(() => validateEdge(EDGE_KINDS.CONSUMES, REF_KINDS.SESSION), /cannot target/); + }); + + it("rejects unknown kinds", () => { + assert.throws(() => validateEdge("nope", REF_KINDS.SESSION), /Invalid edge_kind/); + assert.throws(() => validateEdge(EDGE_KINDS.ANCHORS, "nope"), /Invalid to_ref_kind/); + }); +}); diff --git a/tests/unit/headless.test.ts b/tests/unit/headless.test.ts new file mode 100644 index 0000000..58517a8 --- /dev/null +++ b/tests/unit/headless.test.ts @@ -0,0 +1,55 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { splitProspectSpec, runProspectSpec, type ProspectAction } from "../../src/commands/headless.js"; +import type { ExtensionCommandContext } from "../../src/pi-stubs.js"; + +const ctx = {} as ExtensionCommandContext; + +describe("splitProspectSpec", () => { + it("returns a bare command with empty args", () => { + assert.deepEqual(splitProspectSpec("sync"), { command: "sync", args: "" }); + }); + + it("splits the command from its args", () => { + assert.deepEqual(splitProspectSpec("analyze --limit 3 --model x/y"), { command: "analyze", args: "--limit 3 --model x/y" }); + }); + + it("trims surrounding whitespace and lowercases the command", () => { + assert.deepEqual(splitProspectSpec(" ACCEPT 019abc "), { command: "accept", args: "019abc" }); + }); +}); + +describe("runProspectSpec", () => { + function recorder() { + const calls: Array<{ name: string; args: string }> = []; + const make = (name: string): ProspectAction => async (args) => { calls.push({ name, args }); }; + const actions: Record = { sync: make("sync"), analyze: make("analyze") }; + return { calls, actions }; + } + + it("dispatches to the named action with its args", async () => { + const { calls, actions } = recorder(); + const ran = await runProspectSpec("analyze --limit 5", ctx, actions); + assert.equal(ran, true); + assert.deepEqual(calls, [{ name: "analyze", args: "--limit 5" }]); + }); + + it("returns false and runs nothing for an empty spec", async () => { + const { calls, actions } = recorder(); + assert.equal(await runProspectSpec(" ", ctx, actions), false); + assert.equal(calls.length, 0); + }); + + it("returns false and runs nothing for an unknown command", async () => { + const { calls, actions } = recorder(); + assert.equal(await runProspectSpec("frobnicate now", ctx, actions), false); + assert.equal(calls.length, 0); + }); + + it("propagates errors thrown by an action", async () => { + const actions: Record = { + boom: async () => { throw new Error("kaboom"); }, + }; + await assert.rejects(() => runProspectSpec("boom", ctx, actions), /kaboom/); + }); +}); diff --git a/tests/unit/input-hash.test.ts b/tests/unit/input-hash.test.ts new file mode 100644 index 0000000..901687b --- /dev/null +++ b/tests/unit/input-hash.test.ts @@ -0,0 +1,113 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { + canonicalJson, + computeConfigFingerprint, + computeConfigHash, + computeInputKey, + computeOutputKey, + computePromptBundleHash, + computeSourceSetHash, + fullHash, + shortHash, + uuidv7, +} from "../../src/analyze/input-hash.js"; + +describe("hashing", () => { + it("shortHash is 16 hex chars, fullHash is 64", () => { + assert.match(shortHash("x"), /^[0-9a-f]{16}$/); + assert.match(fullHash("x"), /^[0-9a-f]{64}$/); + }); + + it("source set hash is order-independent", () => { + const a = computeSourceSetHash([ + { kind: "message", id: "m1" }, + { kind: "message", id: "m2" }, + ]); + const b = computeSourceSetHash([ + { kind: "message", id: "m2" }, + { kind: "message", id: "m1" }, + ]); + assert.equal(a, b); + }); + + it("source set hash distinguishes different sets", () => { + const a = computeSourceSetHash([{ kind: "message", id: "m1" }]); + const b = computeSourceSetHash([{ kind: "message", id: "m2" }]); + assert.notEqual(a, b); + }); + + it("prompt bundle hash is order-independent and stable for empty", () => { + assert.equal(computePromptBundleHash(["a", "b"]), computePromptBundleHash(["b", "a"])); + assert.equal(computePromptBundleHash([]), computePromptBundleHash([])); + }); + + it("config fingerprint is order-independent over models and stable for empty", () => { + assert.equal( + computeConfigFingerprint("c1", ["anthropic/a", "openai/b"]), + computeConfigFingerprint("c1", ["openai/b", "anthropic/a"]), + ); + assert.equal(computeConfigFingerprint("c1", []), computeConfigFingerprint("c1", [])); + assert.notEqual(computeConfigFingerprint("c1", []), computeConfigFingerprint("c2", [])); + assert.notEqual( + computeConfigFingerprint("c1", ["anthropic/a"]), + computeConfigFingerprint("c1", ["openai/b"]), + ); + }); + + it("input hash changes when the analyzer version changes", () => { + const base = { + analyzerId: "x", + configFingerprint: "cf1", + sourceSetHash: "s1", + }; + const v1 = computeInputKey({ ...base, analyzerVersionId: "1.0" }); + const v2 = computeInputKey({ ...base, analyzerVersionId: "2.0" }); + assert.notEqual(v1, v2); + }); + + it("input hash changes when the resolved model changes (via the config fingerprint)", () => { + const base = { + analyzerId: "x", + analyzerVersionId: "1.0", + sourceSetHash: "s", + }; + assert.notEqual( + computeInputKey({ ...base, configFingerprint: computeConfigFingerprint("c", ["anthropic/haiku"]) }), + computeInputKey({ ...base, configFingerprint: computeConfigFingerprint("c", ["openai/gpt-5-mini"]) }), + ); + }); + + it("input hash changes when the source set changes", () => { + const base = { analyzerId: "x", analyzerVersionId: "1.0", configFingerprint: "cf" }; + assert.notEqual( + computeInputKey({ ...base, sourceSetHash: "s1" }), + computeInputKey({ ...base, sourceSetHash: "s2" }), + ); + }); + + it("output key is deterministic and folds in both input key and content", () => { + const content = { a: 1, b: [2, 3] }; + // Deterministic: same (input_key, content) → same output_key, and key order in content is irrelevant. + assert.equal(computeOutputKey("ik1", content), computeOutputKey("ik1", { b: [2, 3], a: 1 })); + // Changes with content. + assert.notEqual(computeOutputKey("ik1", content), computeOutputKey("ik1", { a: 2, b: [2, 3] })); + // Changes with input key (same output text under a different recipe is a different node). + assert.notEqual(computeOutputKey("ik1", content), computeOutputKey("ik2", content)); + assert.match(computeOutputKey("ik1", content), /^[0-9a-f]{16}$/); + }); + + it("canonicalJson sorts keys recursively", () => { + assert.equal(canonicalJson({ b: 1, a: { d: 2, c: 3 } }), '{"a":{"c":3,"d":2},"b":1}'); + }); + + it("config hash is independent of key order", () => { + assert.equal(computeConfigHash({ a: 1, b: 2 }), computeConfigHash({ b: 2, a: 1 })); + }); + + it("uuidv7 ids are unique and chronologically sortable", () => { + const ids = Array.from({ length: 50 }, () => uuidv7()); + assert.equal(new Set(ids).size, ids.length); + assert.match(ids[0]!, /^[0-9a-f]{8}-[0-9a-f]{4}-7/); + }); +}); diff --git a/tests/unit/mock-llm.test.ts b/tests/unit/mock-llm.test.ts new file mode 100644 index 0000000..a80cfbf --- /dev/null +++ b/tests/unit/mock-llm.test.ts @@ -0,0 +1,47 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { createMockLLM, createThrowingLLM } from "../../src/analyze/mock-llm.js"; + +describe("createMockLLM", () => { + it("returns scripted responses in order and records calls", async () => { + const mock = createMockLLM({ scripted: ["one", "two"] }); + const r1 = await mock.caller({ model: "cheap", user: "a" }); + const r2 = await mock.caller({ model: "mid", user: "b" }); + assert.equal(r1.text, "one"); + assert.equal(r2.text, "two"); + assert.equal(mock.calls.length, 2); + assert.equal(mock.calls[0]!.user, "a"); + assert.equal(mock.calls[1]!.model, "mid"); + }); + + it("uses a responder function", async () => { + const mock = createMockLLM({ responder: (req, i) => `${req.model}:${i}` }); + assert.equal((await mock.caller({ model: "x", user: "" })).text, "x:0"); + assert.equal((await mock.caller({ model: "y", user: "" })).text, "y:1"); + }); + + it("falls back when scripted runs out", async () => { + const mock = createMockLLM({ scripted: [], fallback: "fb" }); + assert.equal((await mock.caller({ model: "m", user: "" })).text, "fb"); + }); + + it("reports simulated cost and tokens", async () => { + const mock = createMockLLM({ fallback: "{}", costPerCall: 0.01, tokensPerCall: 42 }); + const r = await mock.caller({ model: "m", user: "" }); + assert.equal(r.costUsd, 0.01); + assert.equal(r.tokensUsed, 42); + assert.equal(r.stopReason, "stop"); + }); + + it("defaults to empty text", async () => { + const mock = createMockLLM(); + assert.equal((await mock.caller({ model: "m", user: "" })).text, ""); + }); +}); + +describe("createThrowingLLM", () => { + it("throws when invoked", async () => { + const llm = createThrowingLLM("nope"); + await assert.rejects(() => llm({ model: "m", user: "" }), /nope/); + }); +}); diff --git a/tests/unit/model-tiers.test.ts b/tests/unit/model-tiers.test.ts new file mode 100644 index 0000000..dad76b1 --- /dev/null +++ b/tests/unit/model-tiers.test.ts @@ -0,0 +1,60 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { DEFAULT_MODEL_TIERS, applyModelOverride, isModelTier, resolveModelSpec, splitModelSpec } from "../../src/analyze/model-tiers.js"; + +describe("model tiers", () => { + it("recognises tier names", () => { + assert.ok(isModelTier("cheap")); + assert.ok(isModelTier("mid")); + assert.ok(isModelTier("expensive")); + assert.ok(!isModelTier("anthropic/x")); + }); + + it("resolves tiers via config", () => { + const cfg = { cheap: "p/c", mid: "p/m", expensive: "p/e" }; + assert.equal(resolveModelSpec("cheap", cfg), "p/c"); + assert.equal(resolveModelSpec("mid", cfg), "p/m"); + }); + + it("resolves tiers via defaults when no config", () => { + assert.equal(resolveModelSpec("mid"), DEFAULT_MODEL_TIERS.mid); + }); + + it("passes through explicit provider/model specs", () => { + assert.equal(resolveModelSpec("openai/gpt-5"), "openai/gpt-5"); + }); + + it("splits provider/model, preserving slashes in model id", () => { + assert.deepEqual(splitModelSpec("anthropic/claude-sonnet-4-5"), { provider: "anthropic", modelId: "claude-sonnet-4-5" }); + assert.deepEqual(splitModelSpec("vertex/google/gemini"), { provider: "vertex", modelId: "google/gemini" }); + }); + + it("throws on a spec without a slash", () => { + assert.throws(() => splitModelSpec("nope"), /Invalid model spec/); + }); + + describe("applyModelOverride", () => { + const tiers = { cheap: "p/c", mid: "p/m", expensive: "p/e" }; + + it("returns the tiers unchanged when there is no override", () => { + assert.equal(applyModelOverride(tiers, undefined), tiers); + assert.equal(applyModelOverride(tiers, ""), tiers); + }); + + it("pins every tier to an explicit provider/model override", () => { + assert.deepEqual(applyModelOverride(tiers, "openai/gpt-5"), { + cheap: "openai/gpt-5", + mid: "openai/gpt-5", + expensive: "openai/gpt-5", + }); + }); + + it("resolves a tier-name override through the tiers, then pins all tiers", () => { + assert.deepEqual(applyModelOverride(tiers, "expensive"), { + cheap: "p/e", + mid: "p/e", + expensive: "p/e", + }); + }); + }); +}); diff --git a/tests/unit/parser.test.ts b/tests/unit/parser.test.ts index 463f2d3..11f6d4c 100644 --- a/tests/unit/parser.test.ts +++ b/tests/unit/parser.test.ts @@ -79,12 +79,12 @@ describe("parseLine", () => { } }); - it("parses a branchSummary entry", () => { - const line = JSON.stringify({ type: "branchSummary", id: "b1", parentId: "c1", timestamp: "2026-01-15T10:41:00Z", summary: "Branch context..." }); + it("parses a branch_summary entry (Pi's snake_case entry type)", () => { + const line = JSON.stringify({ type: "branch_summary", id: "b1", parentId: "c1", timestamp: "2026-01-15T10:41:00Z", summary: "Branch context..." }); const result = parseLine(line); assert.ok(result); if (result.kind === "message") { - assert.equal(result.entry.role, "branchSummary"); + assert.equal(result.entry.role, "branch_summary"); } }); diff --git a/tests/unit/patterns.test.ts b/tests/unit/patterns.test.ts new file mode 100644 index 0000000..c3b2deb --- /dev/null +++ b/tests/unit/patterns.test.ts @@ -0,0 +1,78 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { + classifyCorrection, + detectRepetition, + extractCorrectionText, +} from "../../src/analyze/analyzers/turn-pair-core/patterns.js"; + +describe("classifyCorrection", () => { + it("detects strong explicit corrections", () => { + const r = classifyCorrection("No, use pnpm instead of npm", false); + assert.equal(r.detected, true); + assert.equal(r.type, "explicit"); + assert.ok(r.patterns.length > 0); + }); + + it("detects 'I said' style corrections", () => { + assert.equal(classifyCorrection("I said use the other function", false).type, "explicit"); + }); + + it("detects weak/implicit corrections", () => { + const r = classifyCorrection("could you try a different approach", false); + assert.equal(r.detected, true); + assert.equal(r.type, "implicit"); + }); + + it("detects leading negation", () => { + assert.equal(classifyCorrection("not what I wanted", false).type, "explicit"); + }); + + it("returns none for neutral continuation", () => { + const r = classifyCorrection("Thanks, now add a test for the parser", false); + assert.equal(r.detected, false); + assert.equal(r.type, null); + }); + + it("does not treat positive feedback as a correction", () => { + assert.equal(classifyCorrection("looks good, thanks!", false).detected, false); + }); + + it("marks repetition when flagged", () => { + const r = classifyCorrection("run the tests", true); + assert.equal(r.type, "repetition"); + assert.equal(r.detected, true); + }); +}); + +describe("detectRepetition", () => { + it("flags short re-asks sharing tokens", () => { + assert.equal(detectRepetition("run the tests please", "how do I run the tests"), true); + }); + + it("ignores long messages", () => { + const long = "x".repeat(120); + assert.equal(detectRepetition(long, long), false); + }); + + it("ignores when no prior text", () => { + assert.equal(detectRepetition("run tests", null), false); + }); + + it("ignores unrelated short messages", () => { + assert.equal(detectRepetition("ok", "completely different subject matter"), false); + }); +}); + +describe("extractCorrectionText", () => { + it("slices the remainder after the matched pattern", () => { + const text = "actually use yarn"; + const result = extractCorrectionText(text, "\\bactually[,.\\s]"); + assert.ok(result.includes("use yarn")); + }); + + it("falls back to a prefix when pattern does not match", () => { + const result = extractCorrectionText("some text", "\\bzzz\\b"); + assert.ok(result.length > 0); + }); +}); diff --git a/tests/unit/pi-llm.test.ts b/tests/unit/pi-llm.test.ts new file mode 100644 index 0000000..66a0a0f --- /dev/null +++ b/tests/unit/pi-llm.test.ts @@ -0,0 +1,89 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { makePiLLMCaller, toLLMResponse } from "../../src/analyze/pi-llm.js"; +import type { ExtensionContext, PiAssistantMessage, PiModel, ResolvedRequestAuth } from "../../src/pi-stubs.js"; + +const TIERS = { cheap: "anthropic/c", mid: "anthropic/m", expensive: "anthropic/e" }; + +function assistantMessage(partial: Partial): PiAssistantMessage { + return { + role: "assistant", + content: partial.content ?? [], + model: partial.model ?? "anthropic/m", + usage: partial.usage ?? { + input: 0, + output: 0, + cacheRead: 0, + cacheWrite: 0, + totalTokens: 0, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }, + stopReason: partial.stopReason ?? "stop", + errorMessage: partial.errorMessage, + timestamp: 0, + }; +} + +function ctxWith(find: (p: string, m: string) => PiModel | undefined, auth: ResolvedRequestAuth): ExtensionContext { + return { + modelRegistry: { + find, + getAll: () => [], + getAvailable: () => [], + getApiKeyAndHeaders: async () => auth, + }, + }; +} + +describe("toLLMResponse", () => { + it("joins text parts and extracts thinking", () => { + const msg = assistantMessage({ + content: [ + { type: "thinking", thinking: "pondering" }, + { type: "text", text: "hello" }, + { type: "text", text: "world" }, + ], + usage: { input: 10, output: 5, cacheRead: 0, cacheWrite: 0, totalTokens: 15, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0.02 } }, + }); + const r = toLLMResponse(msg, "anthropic/m", 123); + assert.equal(r.text, "hello\nworld"); + assert.equal(r.thinking, "pondering"); + assert.equal(r.tokensUsed, 15); + assert.equal(r.costUsd, 0.02); + assert.equal(r.durationMs, 123); + }); + + it("omits thinking when none present", () => { + const r = toLLMResponse(assistantMessage({ content: [{ type: "text", text: "x" }] }), "m", 0); + assert.equal(r.thinking, undefined); + }); + + it("throws on error stop reason", () => { + const msg = assistantMessage({ stopReason: "error", errorMessage: "boom" }); + assert.throws(() => toLLMResponse(msg, "m", 0), /boom/); + }); + + it("throws an actionable error when the response is truncated at the output limit", () => { + const msg = assistantMessage({ + content: [{ type: "text", text: '{"sentiment":"frus' }], + stopReason: "length", + usage: { input: 100, output: 500, cacheRead: 0, cacheWrite: 0, totalTokens: 600, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, + }); + assert.throws(() => toLLMResponse(msg, "google/gemini-2.5-flash", 0), /truncated at the output limit \(500 output tokens\)/); + }); +}); + +describe("makePiLLMCaller", () => { + it("throws when the model is not in the registry", async () => { + const caller = makePiLLMCaller(ctxWith(() => undefined, { ok: true }), { modelTiers: TIERS }); + await assert.rejects(() => caller({ model: "cheap", user: "hi" }), /Model not found/); + }); + + it("throws when credentials are unavailable", async () => { + const caller = makePiLLMCaller( + ctxWith(() => ({ id: "c", provider: "anthropic" }), { ok: false, error: "no key" }), + { modelTiers: TIERS }, + ); + await assert.rejects(() => caller({ model: "cheap", user: "hi" }), /No credentials/); + }); +}); diff --git a/tests/unit/prompt-parsing.test.ts b/tests/unit/prompt-parsing.test.ts new file mode 100644 index 0000000..5663b01 --- /dev/null +++ b/tests/unit/prompt-parsing.test.ts @@ -0,0 +1,162 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { + extractJsonObject, + parseClassifyResponse, + buildClassifyPrompt, +} from "../../src/analyze/analyzers/turn-pair-llm/prompt.js"; +import { parseMapResponse } from "../../src/analyze/analyzers/session-overview/prompt-map.js"; +import { parseReduceResponse } from "../../src/analyze/analyzers/session-overview/prompt-reduce.js"; + +describe("extractJsonObject", () => { + it("parses a bare JSON object", () => { + assert.deepEqual(extractJsonObject('{"a":1}'), { a: 1 }); + }); + + it("parses JSON inside markdown fences", () => { + assert.deepEqual(extractJsonObject('```json\n{"a":2}\n```'), { a: 2 }); + }); + + it("parses JSON surrounded by prose", () => { + assert.deepEqual(extractJsonObject('Here you go: {"a":3} cheers'), { a: 3 }); + }); + + it("handles nested objects", () => { + assert.deepEqual(extractJsonObject('{"a":{"b":1}}'), { a: { b: 1 } }); + }); + + it("throws when no object present", () => { + assert.throws(() => extractJsonObject("no json here"), /No JSON object/); + }); + + it("throws on unterminated object", () => { + assert.throws(() => extractJsonObject('{"a":1'), /Unterminated/); + }); +}); + +describe("parseClassifyResponse", () => { + it("parses a valid classification", () => { + const r = parseClassifyResponse('{"sentiment":"frustrated","friction_type":"wrong_approach","is_genuine_correction":true,"severity":"high","rationale":"x"}'); + assert.equal(r.sentiment, "frustrated"); + assert.equal(r.friction_type, "wrong_approach"); + assert.equal(r.is_genuine_correction, true); + assert.equal(r.severity, "high"); + }); + + it("falls back to safe defaults on invalid enum values", () => { + const r = parseClassifyResponse('{"sentiment":"weird","friction_type":"nope","severity":"huge"}'); + assert.equal(r.sentiment, "neutral"); + assert.equal(r.friction_type, "none"); + assert.equal(r.severity, "low"); + assert.equal(r.is_genuine_correction, false); + }); +}); + +describe("buildClassifyPrompt", () => { + it("includes user and assistant text and optional hint", () => { + const p = buildClassifyPrompt({ userText: "hello", assistantText: "world", correctionText: "use X", toolCalls: [], toolResults: [] }); + assert.ok(p.includes("hello") && p.includes("world") && p.includes("use X")); + }); + + it("omits hint when absent", () => { + const p = buildClassifyPrompt({ userText: "a", assistantText: "b", correctionText: null, toolCalls: [], toolResults: [] }); + assert.ok(!p.includes("HEURISTIC")); + }); + + it("includes tool calls section when tool calls have arguments", () => { + const p = buildClassifyPrompt({ + userText: "push it", + assistantText: "running git push", + correctionText: null, + toolCalls: [{ name: "bash", argumentsPreview: "git push -u origin v2nic/gh-pr-review" }], + toolResults: [], + }); + assert.ok(p.includes("TOOL CALLS:"), "prompt must include TOOL CALLS section"); + assert.ok(p.includes("bash:"), "prompt must mention tool name"); + assert.ok(p.includes("git push -u origin"), "prompt must include the arguments preview"); + }); + + it("includes failing command for a tool-error turn", () => { + const p = buildClassifyPrompt({ + userText: "YOU SHOULD HAVE PUSHED TO v2nic/gh-pr-review", + assistantText: "creating PR", + correctionText: null, + toolCalls: [ + { name: "bash", argumentsPreview: "git push -u origin v2nic/gh-pr-review" }, + { name: "bash", argumentsPreview: "gh pr create --title fix" }, + ], + toolResults: [ + { toolName: "bash", isError: true, errorHead: "Error: no --repo flag, targeting upstream" }, + ], + }); + assert.ok(p.includes("TOOL CALLS:"), "prompt must include TOOL CALLS section"); + assert.ok(p.includes("git push -u origin"), "prompt must include push command"); + assert.ok(p.includes("gh pr create"), "prompt must include gh command"); + assert.ok(p.includes("FAILED"), "prompt must mark failed result"); + assert.ok(p.includes("no --repo flag"), "prompt must include error head"); + }); + + it("includes non-bash tool calls", () => { + const p = buildClassifyPrompt({ + userText: "add the file", + assistantText: "adding", + correctionText: null, + toolCalls: [{ name: "edit", argumentsPreview: "file=src/index.ts" }], + toolResults: [], + }); + assert.ok(p.includes("TOOL CALLS:")); + assert.ok(p.includes("edit:")); + }); + + it("omits tool calls section when no tool calls and no errors", () => { + const p = buildClassifyPrompt({ userText: "hi", assistantText: "hello", correctionText: null, toolCalls: [], toolResults: [] }); + assert.ok(!p.includes("TOOL CALLS:")); + }); + + it("shows tool calls section when there are errors even without tool calls", () => { + const p = buildClassifyPrompt({ + userText: "run it", + assistantText: "failed", + correctionText: null, + toolCalls: [], + toolResults: [{ toolName: "bash", isError: true, errorHead: "command not found" }], + }); + assert.ok(p.includes("TOOL CALLS:")); + assert.ok(p.includes("FAILED")); + assert.ok(p.includes("command not found")); + }); +}); + +describe("parseMapResponse", () => { + it("parses segment summary and notable points", () => { + const r = parseMapResponse('{"segment_summary":"s","notable_points":["a","b"]}', extractJsonObject); + assert.equal(r.segment_summary, "s"); + assert.deepEqual(r.notable_points, ["a", "b"]); + }); + + it("defaults missing fields", () => { + const r = parseMapResponse("{}", extractJsonObject); + assert.equal(r.segment_summary, ""); + assert.deepEqual(r.notable_points, []); + }); +}); + +describe("parseReduceResponse", () => { + it("parses summary, friction points, and proposals", () => { + const json = JSON.stringify({ + session_summary: "did stuff", + key_friction_points: [{ description: "x", severity: "high" }, { bad: true }], + improvement_proposals: [{ title: "t", summary: "s", target_type: "config", severity: "friction" }], + }); + const r = parseReduceResponse(json, extractJsonObject); + assert.equal(r.session_summary, "did stuff"); + assert.equal(r.key_friction_points.length, 1); + assert.equal(r.improvement_proposals.length, 1); + }); + + it("defaults arrays when missing", () => { + const r = parseReduceResponse("{}", extractJsonObject); + assert.deepEqual(r.key_friction_points, []); + assert.deepEqual(r.improvement_proposals, []); + }); +}); diff --git a/tests/unit/proposals-format.test.ts b/tests/unit/proposals-format.test.ts new file mode 100644 index 0000000..eef59cc --- /dev/null +++ b/tests/unit/proposals-format.test.ts @@ -0,0 +1,73 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { parseProposalsArgs, rankProposals, sessionLabel } from "../../src/commands/proposals.js"; +import type { Proposal } from "../../src/types.js"; +import { homedir } from "node:os"; + +function makeProposal(overrides: Partial): Proposal { + return { + id: "id", + created_at: "2026-01-01T00:00:00.000Z", + updated_at: "2026-01-01T00:00:00.000Z", + session_id: "sess", + source_node_id: null, + analyzer_id: "session-overview", + target_type: "agents_md", + target_path: null, + title: "t", + severity: "friction", + summary: "s", + detail: null, + evidence: null, + confidence: null, + status: "open", + input_key: "k", + ...overrides, + }; +} + +test("parseProposalsArgs: empty yields no status and concise", () => { + assert.deepEqual(parseProposalsArgs(""), { status: undefined, full: false }); + assert.deepEqual(parseProposalsArgs(" "), { status: undefined, full: false }); +}); + +test("parseProposalsArgs: recognises a status word", () => { + assert.deepEqual(parseProposalsArgs("applied"), { status: "applied", full: false }); + assert.deepEqual(parseProposalsArgs("OPEN"), { status: "open", full: false }); +}); + +test("parseProposalsArgs: recognises --full / -v / --verbose in any order", () => { + assert.deepEqual(parseProposalsArgs("--full"), { status: undefined, full: true }); + assert.deepEqual(parseProposalsArgs("-v rejected"), { status: "rejected", full: true }); + assert.deepEqual(parseProposalsArgs("duplicate --verbose"), { status: "duplicate", full: true }); +}); + +test("parseProposalsArgs: ignores unknown tokens", () => { + assert.deepEqual(parseProposalsArgs("garbage --nope"), { status: undefined, full: false }); +}); + +test("rankProposals: higher confidence sorts first; nulls last", () => { + const high = makeProposal({ id: "hi", confidence: 0.95 }); + const mid = makeProposal({ id: "mid", confidence: 0.5 }); + const none = makeProposal({ id: "none", confidence: null }); + const sorted = [none, mid, high].sort(rankProposals).map((p) => p.id); + assert.deepEqual(sorted, ["hi", "mid", "none"]); +}); + +test("rankProposals: equal confidence breaks ties by newest created_at", () => { + const older = makeProposal({ id: "old", confidence: 0.8, created_at: "2026-01-01T00:00:00.000Z" }); + const newer = makeProposal({ id: "new", confidence: 0.8, created_at: "2026-02-01T00:00:00.000Z" }); + const sorted = [older, newer].sort(rankProposals).map((p) => p.id); + assert.deepEqual(sorted, ["new", "old"]); +}); + +test("sessionLabel: prefers cwd with $HOME collapsed to ~", () => { + const cwd = `${homedir()}/Source/pi-prospector/main`; + assert.equal(sessionLabel({ project: "proj", cwd }, "abcdef12"), "~/Source/pi-prospector/main"); +}); + +test("sessionLabel: falls back to project then short id", () => { + assert.equal(sessionLabel({ project: "proj", cwd: "" }, "abcdef1234"), "proj"); + assert.equal(sessionLabel(undefined, "abcdef1234"), "abcdef12"); + assert.equal(sessionLabel({ project: "", cwd: "" }, "abcdef1234"), "abcdef12"); +}); diff --git a/tests/unit/show-format.test.ts b/tests/unit/show-format.test.ts new file mode 100644 index 0000000..4bbf32f --- /dev/null +++ b/tests/unit/show-format.test.ts @@ -0,0 +1,74 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { toolCallPreview, renderAnchoredTurns } from "../../src/commands/show.js"; +import type { TurnPair } from "../../src/analyze/analyzers/turn-pair-core/build.js"; +import type { MessageRow } from "../../src/analyze/types.js"; + +function msg(over: Partial & { id: string; role: string }): MessageRow { + return { + session_id: "s", + parent_id: null, + timestamp: null, + content_text: null, + content_thinking: null, + tool_calls: null, + tool_results: null, + ...over, + }; +} + +describe("toolCallPreview", () => { + it("prefers the most salient argument and collapses whitespace", () => { + assert.match(toolCallPreview("bash", { command: "git push origin\n main" }), /^bash {2}git push origin main$/); + assert.equal(toolCallPreview("read", { path: "/a/b.ts" }), "read /a/b.ts"); + assert.equal(toolCallPreview("grep", { pattern: "foo" }), "grep foo"); + }); + + it("falls back to JSON when no salient key is present", () => { + assert.match(toolCallPreview("custom", { foo: 1 }), /custom {2}\{"foo":1\}/); + }); + + it("truncates long arguments", () => { + const long = "x".repeat(500); + const p = toolCallPreview("bash", { command: long }); + assert.ok(p.length < 200 && p.endsWith("…")); + }); +}); + +describe("renderAnchoredTurns", () => { + const pairs: TurnPair[] = [ + { index: 0, userMessageId: "u0", messageIds: ["u0", "a0", "r0"], userText: "do the thing", assistantText: "", thinkingText: "", toolCalls: [], toolResults: [], priorUserText: null, timestamp: null }, + { index: 1, userMessageId: "u1", messageIds: ["u1", "a1"], userText: "no, that's wrong", assistantText: "", thinkingText: "", toolCalls: [], toolResults: [], priorUserText: "do the thing", timestamp: null }, + ]; + const byId = new Map([ + ["u0", msg({ id: "u0", role: "user", content_text: "do the thing" })], + ["a0", msg({ id: "a0", role: "assistant", content_text: "on it", tool_calls: JSON.stringify([{ name: "bash", arguments: { command: "git push origin main" } }]) })], + ["r0", msg({ id: "r0", role: "toolResult", content_text: "fatal: remote rejected", tool_results: JSON.stringify([{ isError: true }]) })], + ["u1", msg({ id: "u1", role: "user", content_text: "no, that's wrong" })], + ["a1", msg({ id: "a1", role: "assistant", content_text: "sorry" })], + ]); + const coreByUser = new Map>([ + ["u0", { user_message_id: "u0", friction_score: 0.9, tool_failure_count: 1, tool_call_count: 1, correction_detected: false, high_signal: true }], + ["u1", { user_message_id: "u1", friction_score: 0.6, tool_failure_count: 0, tool_call_count: 0, correction_detected: true, correction_type: "explicit", high_signal: true }], + ]); + const llmByUser = new Map>([ + ["u1", { user_message_id: "u1", sentiment: "frustrated", friction_type: "wrong_approach", severity: "medium" }], + ]); + + it("renders verbatim user text, tool-call args, and tool errors", () => { + const text = renderAnchoredTurns(pairs, byId, new Set(["u0", "u1"]), coreByUser, llmByUser).join("\n"); + assert.match(text, /pair #0 · friction=0\.90 · tool_fail=1\/1/); + assert.match(text, /git push origin main/); // tool-call argument is surfaced + assert.match(text, /✗ fatal: remote rejected/); // tool error surfaced + assert.match(text, /no, that's wrong/); // verbatim user text + assert.match(text, /sentiment=frustrated type=wrong_approach sev=medium/); + assert.match(text, /correction=explicit/); + }); + + it("orders by pair index and respects maxTurns with a remainder note", () => { + const text = renderAnchoredTurns(pairs, byId, new Set(["u0", "u1"]), coreByUser, llmByUser, 1).join("\n"); + assert.match(text, /pair #0/); + assert.doesNotMatch(text, /pair #1/); + assert.match(text, /…1 more turn\(s\) not shown\./); + }); +}); diff --git a/tests/unit/turn-pair-build.test.ts b/tests/unit/turn-pair-build.test.ts new file mode 100644 index 0000000..4913b34 --- /dev/null +++ b/tests/unit/turn-pair-build.test.ts @@ -0,0 +1,189 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { buildTurnPairs } from "../../src/analyze/analyzers/turn-pair-core/build.js"; +import type { MessageRow } from "../../src/analyze/types.js"; + +function msg(partial: Partial & { id: string; role: string }): MessageRow { + return { + id: partial.id, + session_id: "s1", + parent_id: partial.parent_id ?? null, + timestamp: partial.timestamp ?? null, + role: partial.role, + content_text: partial.content_text ?? null, + content_thinking: partial.content_thinking ?? null, + tool_calls: partial.tool_calls ?? null, + tool_results: partial.tool_results ?? null, + }; +} + +describe("buildTurnPairs", () => { + it("groups a user message with following assistant/tool messages", () => { + const messages: MessageRow[] = [ + msg({ id: "u1", role: "user", content_text: "fix the bug" }), + msg({ id: "a1", role: "assistant", content_text: "looking", tool_calls: JSON.stringify([{ name: "read" }]) }), + msg({ id: "t1", role: "toolResult", tool_results: JSON.stringify([{ toolName: "read", isError: false, textLength: 100 }]) }), + msg({ id: "a2", role: "assistant", content_text: "fixed it" }), + msg({ id: "u2", role: "user", content_text: "now add tests" }), + msg({ id: "a3", role: "assistant", content_text: "done" }), + ]; + const pairs = buildTurnPairs(messages); + assert.equal(pairs.length, 2); + + const first = pairs[0]!; + assert.equal(first.userMessageId, "u1"); + assert.deepEqual(first.messageIds, ["u1", "a1", "t1", "a2"]); + assert.ok(first.assistantText.includes("looking") && first.assistantText.includes("fixed it")); + assert.equal(first.toolCalls.length, 1); + assert.equal(first.toolResults.length, 1); + assert.equal(first.priorUserText, null); + + const second = pairs[1]!; + assert.equal(second.userMessageId, "u2"); + assert.equal(second.priorUserText, "fix the bug"); + }); + + it("ignores messages before the first user message", () => { + const messages: MessageRow[] = [ + msg({ id: "c1", role: "compactionSummary", content_text: "summary" }), + msg({ id: "u1", role: "user", content_text: "hi" }), + ]; + const pairs = buildTurnPairs(messages); + assert.equal(pairs.length, 1); + assert.deepEqual(pairs[0]!.messageIds, ["u1"]); + }); + + it("captures assistant thinking text", () => { + const pairs = buildTurnPairs([ + msg({ id: "u1", role: "user", content_text: "q" }), + msg({ id: "a1", role: "assistant", content_thinking: "hmm" }), + ]); + assert.equal(pairs[0]!.thinkingText, "hmm"); + }); + + it("tolerates malformed tool_calls/tool_results JSON", () => { + const pairs = buildTurnPairs([ + msg({ id: "u1", role: "user", content_text: "q" }), + msg({ id: "a1", role: "assistant", tool_calls: "{not json" }), + msg({ id: "t1", role: "toolResult", tool_results: "also bad" }), + ]); + assert.equal(pairs[0]!.toolCalls.length, 0); + assert.equal(pairs[0]!.toolResults.length, 0); + }); + + it("returns empty for no messages", () => { + assert.deepEqual(buildTurnPairs([]), []); + }); + + it("starts a new turn at a bashExecution entry", () => { + const pairs = buildTurnPairs([ + msg({ id: "u1", role: "user", content_text: "do it" }), + msg({ id: "a1", role: "assistant", content_text: "ok" }), + msg({ id: "b1", role: "bashExecution", content_text: "npm test" }), + msg({ id: "a2", role: "assistant", content_text: "green" }), + ]); + assert.equal(pairs.length, 2); + assert.deepEqual(pairs[0]!.messageIds, ["u1", "a1"]); + assert.equal(pairs[1]!.userMessageId, "b1"); + assert.deepEqual(pairs[1]!.messageIds, ["b1", "a2"]); + }); + + it("starts new turns at branch_summary and custom_message entries", () => { + const pairs = buildTurnPairs([ + msg({ id: "br", role: "branch_summary", content_text: "branched" }), + msg({ id: "a0", role: "assistant", content_text: "resuming" }), + msg({ id: "u1", role: "user", content_text: "continue" }), + msg({ id: "a1", role: "assistant", content_text: "sure" }), + msg({ id: "cm", role: "custom_message", content_text: "injected note" }), + msg({ id: "a2", role: "assistant", content_text: "ack" }), + ]); + assert.equal(pairs.length, 3); + assert.deepEqual(pairs.map((p) => p.userMessageId), ["br", "u1", "cm"]); + assert.deepEqual(pairs[0]!.messageIds, ["br", "a0"]); + assert.deepEqual(pairs[2]!.messageIds, ["cm", "a2"]); + }); + + it("does not start a turn at a compaction summary", () => { + const pairs = buildTurnPairs([ + msg({ id: "u1", role: "user", content_text: "q" }), + msg({ id: "a1", role: "assistant", content_text: "a" }), + msg({ id: "c1", role: "compactionSummary", content_text: "summary" }), + msg({ id: "a2", role: "assistant", content_text: "more" }), + ]); + // Single turn; the compaction summary is neither a turn start nor captured. + assert.equal(pairs.length, 1); + assert.deepEqual(pairs[0]!.messageIds, ["u1", "a1", "a2"]); + }); + + it("captures tool call arguments and tool result error heads", () => { + const pairs = buildTurnPairs([ + msg({ id: "u1", role: "user", content_text: "run it" }), + msg({ + id: "a1", + role: "assistant", + content_text: "executing", + tool_calls: JSON.stringify([{ name: "bash", arguments: { command: "npm test -- --reporter=dot" } }]), + }), + msg({ + id: "t1", + role: "toolResult", + content_text: "Error: ENOENT no such file", + tool_results: JSON.stringify([{ toolName: "bash", isError: true, textLength: 100 }]), + }), + ]); + assert.equal(pairs[0]!.toolCalls.length, 1); + assert.equal(pairs[0]!.toolCalls[0]!.name, "bash"); + assert.equal(pairs[0]!.toolCalls[0]!.argumentsPreview, "npm test -- --reporter=dot"); + assert.equal(pairs[0]!.toolResults.length, 1); + assert.equal(pairs[0]!.toolResults[0]!.toolName, "bash"); + assert.equal(pairs[0]!.toolResults[0]!.isError, true); + assert.ok(pairs[0]!.toolResults[0]!.errorHead!.includes("ENOENT")); + }); + + it("captures non-bash tool arguments as key=value summary", () => { + const pairs = buildTurnPairs([ + msg({ id: "u1", role: "user", content_text: "create PR" }), + msg({ + id: "a1", + role: "assistant", + tool_calls: JSON.stringify([{ name: "gh", arguments: { subcommand: "pr", flags: "--repo v2nic/pi-prospector", title: "Fix bug" } }]), + }), + ]); + assert.equal(pairs[0]!.toolCalls[0]!.name, "gh"); + // Non-bash tools get a key=value summary + assert.ok(pairs[0]!.toolCalls[0]!.argumentsPreview.includes("subcommand=")); + }); + + it("sets errorHead to null for non-error tool results", () => { + const pairs = buildTurnPairs([ + msg({ id: "u1", role: "user", content_text: "read file" }), + msg({ + id: "a1", + role: "assistant", + tool_calls: JSON.stringify([{ name: "read", arguments: {} }]), + }), + msg({ + id: "t1", + role: "toolResult", + content_text: "file contents here", + tool_results: JSON.stringify([{ toolName: "read", isError: false, textLength: 50 }]), + }), + ]); + assert.equal(pairs[0]!.toolResults[0]!.isError, false); + assert.equal(pairs[0]!.toolResults[0]!.errorHead, null); + }); + + it("truncates long bash commands in arguments preview", () => { + const longCommand = "a".repeat(500); + const pairs = buildTurnPairs([ + msg({ id: "u1", role: "user", content_text: "run" }), + msg({ + id: "a1", + role: "assistant", + tool_calls: JSON.stringify([{ name: "bash", arguments: { command: longCommand } }]), + }), + ]); + assert.ok(pairs[0]!.toolCalls[0]!.argumentsPreview.endsWith("…")); + assert.ok(pairs[0]!.toolCalls[0]!.argumentsPreview.length <= 301); // 300 + ellipsis + }); +}); diff --git a/tests/unit/version.test.ts b/tests/unit/version.test.ts new file mode 100644 index 0000000..007e795 --- /dev/null +++ b/tests/unit/version.test.ts @@ -0,0 +1,82 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; +import { + expandReviseReasons, + gradeVersionMove, + parseReviseArg, + parseVersionId, + reachLabel, + versionIdOf, +} from "../../src/analyze/version.js"; + +describe("version identity", () => { + it("round-trips major.minor through versionIdOf/parseVersionId", () => { + assert.equal(versionIdOf({ major: 2, minor: 3 }), "2.3"); + assert.deepEqual(parseVersionId("2.3"), { major: 2, minor: 3 }); + assert.deepEqual(parseVersionId(versionIdOf({ major: 10, minor: 0 })), { major: 10, minor: 0 }); + }); + + it("parses defensively (missing minor, junk → 0)", () => { + assert.deepEqual(parseVersionId("4"), { major: 4, minor: 0 }); + assert.deepEqual(parseVersionId(""), { major: 0, minor: 0 }); + assert.deepEqual(parseVersionId("x.y"), { major: 0, minor: 0 }); + }); +}); + +describe("gradeVersionMove", () => { + it("grades a higher major as major", () => { + assert.equal(gradeVersionMove({ major: 1, minor: 5 }, { major: 2, minor: 0 }), "major"); + }); + + it("grades a higher minor (same major) as minor", () => { + assert.equal(gradeVersionMove({ major: 1, minor: 0 }, { major: 1, minor: 1 }), "minor"); + }); + + it("returns null for equal versions and for downgrades", () => { + assert.equal(gradeVersionMove({ major: 1, minor: 2 }, { major: 1, minor: 2 }), null); + assert.equal(gradeVersionMove({ major: 2, minor: 0 }, { major: 1, minor: 9 }), null); + assert.equal(gradeVersionMove({ major: 1, minor: 5 }, { major: 1, minor: 4 }), null); + }); +}); + +describe("expandReviseReasons", () => { + it("makes minor imply major", () => { + assert.deepEqual([...expandReviseReasons(["minor"])].sort(), ["major", "minor"]); + }); + + it("leaves major and config alone", () => { + assert.deepEqual([...expandReviseReasons(["major"])], ["major"]); + assert.deepEqual([...expandReviseReasons(["config"])], ["config"]); + assert.deepEqual([...expandReviseReasons([])], []); + }); +}); + +describe("parseReviseArg", () => { + it("parses individual reasons and ignores unknown tokens", () => { + assert.deepEqual(parseReviseArg("major"), ["major"]); + assert.deepEqual(parseReviseArg("config"), ["config"]); + assert.deepEqual(parseReviseArg("nonsense"), []); + assert.deepEqual(parseReviseArg(""), []); + }); + + it("parses comma lists and dedupes", () => { + assert.deepEqual(parseReviseArg("minor,config").sort(), ["config", "minor"]); + assert.deepEqual(parseReviseArg("major,major").sort(), ["major"]); + }); + + it("expands `all` to every reason", () => { + assert.deepEqual(parseReviseArg("all").sort(), ["config", "major", "minor"]); + }); +}); + +describe("reachLabel", () => { + it("labels an empty reach as fill", () => { + assert.equal(reachLabel([]), "fill"); + assert.equal(reachLabel(new Set()), "fill"); + }); + + it("labels a non-empty reach as a sorted revise set", () => { + assert.equal(reachLabel(["config", "major"]), "revise:config+major"); + assert.equal(reachLabel(new Set(["major", "minor"])), "revise:major+minor"); + }); +});