Skip to content
This repository was archived by the owner on Jul 3, 2026. It is now read-only.

trajectory: add deterministic tool-trajectory analyzer - #14

Closed
v2nic wants to merge 21 commits into
mainfrom
recall-7-trajectory
Closed

trajectory: add deterministic tool-trajectory analyzer#14
v2nic wants to merge 21 commits into
mainfrom
recall-7-trajectory

Conversation

@v2nic

@v2nic v2nic commented Jun 12, 2026

Copy link
Copy Markdown
Owner

Summary

Implements GitHub issue #7: a new session-anchored, deterministic (no LLM) analyzer that detects friction in the tool-call trajectory of a session — retry-without-change, polling loops, do→undo→redo oscillation, and pre-flight gaps.

New detectors

Pattern Description
stuck-loop Same tool + near-identical normalised args invoked N≥3 times without an intervening success/state change
polling-loop Read-only command repeated while waiting for external state (e.g. gh pr view 29 ×5)
oscillation Action followed by its inverse on the same target within a sliding window (checkout x→y→x, push→force-push)
pre-flight gap Mutating command that fails on a missing precondition (mv into non-existent dir)

New files

  • src/analyze/analyzers/tool-trajectory/{index,config,arg-parser,detectors}.ts — the analyzer and its detectors
  • tests/unit/tool-trajectory.test.ts — 30 unit tests per detector
  • tests/component/tool-trajectory.test.ts — 2 component tests (polite error-free thrash session yields ≥1 trajectory signal)

Changes

  • DESIGN.md: Define ubiquitous-language terms (trajectory signal, stuck-loop, polling-loop, oscillation, pre-flight gap); update node-kind definition
  • AGENTS.md: Register tool-trajectory in code organization
  • session-overview digest: Consume trajectory nodes, render signal lines
  • session-overview analyzer: Declare tool-trajectory as dependency
  • defaults.ts: Register tool-trajectory analyzer in dependency order
  • tests/unit/digest.test.ts: Add trajectoryNodes to buildDigest calls

Test results

All 223 unit+component tests pass. Integration tests pass. tsc --noEmit clean.

Closes #7

v2nic added 21 commits June 10, 2026 22:28
@v2nic
feat: incremental analyzer graph with versioned lineage and Pi-AI pro…
c26546f 
…vider

Implements the analyzer framework on a clean foundation:

- Append-only, typed-edge analysis graph (no parent_id; edges are the
  single source of truth). New 'revises' edge kind links version
  alternatives of the same logical unit.
- 100% incremental via scan(): every planned unit is classified as
  missing|stale|current by input_hash + source-set lookup. shallow runs
  process missing units only; deep runs re-analyse stale units into new
  versions linked by revises edges, so alternatives coexist at the same
  level with timestamps and navigable lineage.
- No crash-recovery bookkeeping: idempotency is structural (finished
  nodes are 'current'; unfinished units remain missing/stale).
- LLM wired to Pi's AI provider system (modelRegistry.find +
  getApiKeyAndHeaders + @earendil-works/pi-ai complete()); never Ollama
  or a local server. Deterministic mock caller for tests.
- Three analyzers: turn-pair-core (deterministic friction), turn-pair-llm
  (cheap LLM enrichment of high-signal pairs), session-overview (digest +
  map-reduce, emits proposals). Dependency-visibility enforcement and
  edge validation in the framework.
- Clean v2 schema (proposals + framework tables), proposal materialiser
  with SHA-256 dedup, env-overridable config.

Tests: 130 unit/component + 19 integration, all green; ~98% line
coverage. TypeBox for all new data shapes.
@v2nic
docs: replace analyzer-design-c with root DESIGN.md
521a821 
Remove docs/analyzer-design-c.md (the pre-implementation design draft,
now partly obsolete: it specified progress cursors and crash recovery,
which the implementation deliberately replaced with scan-based
incrementality) and add a root-level DESIGN.md.

DESIGN.md is an orientation guide for engineers and AI agents new to the
repo. It describes the system's purpose and the rationale for the
current architecture, and fixes a ubiquitous language for the core
concepts (session, pair, analysis node, edge kinds, recipe/input hash,
scan, unit status, run modes, lineage, model tiers, proposal lifecycle).
It contains no code pointers by design.
@v2nic
docs: point AGENTS.md at DESIGN.md as required first reading
192b77f
@v2nic
docs: refresh AGENTS.md code-organization, integration-test, and CI s…
23a9eee 
…ections to match implementation
@v2nic
docs: define turn (Pi-aligned) and fix the pair framing in DESIGN.md
e22c5b7
@v2nic
analyze: make resolved model part of node identity
845d59e 
A tier (cheap/mid/expensive) is shorthand that resolves to a concrete
provider/model; fold that resolved model into input_hash via a new
modelBundleHash. Changing which model a tier maps to now marks affected
nodes stale (deep run revises into a new version; shallow leaves them,
so swaps never force surprise recomputation). Deterministic analyzers
declare no model and are unaffected.
@v2nic
analyze: align turn boundaries with host platform
3a624dc 
buildTurnPairs now starts a new turn at any host turn-start entry:
user and bashExecution messages, plus branch_summary/custom_message
entries. Compaction summaries remain boundaries, not turn starts.
@v2nic
analyze: bring --model to life as a per-run model pin
754c25b 
--model now pins every tier to one concrete model for the invocation
(applyModelOverride), instead of being recorded but ignored. The same
effective tiers feed both the framework and the LLM caller, and analyzers
resolve their tier from ctx.modelTiers before calling the model, so the
model used and the model folded into node identity are one source of
truth and cannot diverge. A pinned run produces its own nodes; rerunning
under the normal mapping marks them stale. Drop the vestigial
defaultModel seam.
@v2nic
design: replace shallow/deep with revise reasons and graded versions
b05e042 
Rework the ubiquitous language so a run's reach is a composable set of
revise reasons rather than a shallow/deep depth, and split staleness by
why a node is out of date:

- Analyzer version is a major.minor pair the author owns; only the
  version is graded (major/minor). The shipped logic, default prompt,
  and default tier are all represented by the version.
- Everything the user sets (thresholds, prompt override, tier->model
  mapping, model pin, and so the resolved model) is config, and config
  is never graded -- a different prompt or model is just different.
- Identity is analyzer + version + config + source set.
- A run always fills missing work; --revise widens its reach with
  major/minor/config in any combination. Reasons only select which
  stale units to touch; a selected unit is always recomputed to the
  current recipe in full (no intermediate state, no half-updated node).
@v2nic
analyze: implement revise reasons and graded analyzer versions
e751d7b 
Replace the shallow/deep run mode with a composable --revise reason set
and grade staleness by cause, per DESIGN.md.

- AnalyzerVersion is now { major, minor } (was an opaque versionId
  string); versionIdOf/parseVersionId give the canonical "M.m" form
  stored on runs and nodes.
- Identity drops the standalone shipped-prompt and model-bundle hashes
  and folds the user's config + resolved model into one config
  fingerprint: input_hash = analyzer + version + config_fingerprint +
  source set. The shipped prompt is represented by the version; the
  prompt bundle hash is kept only as run provenance.
- scan() grades each stale unit's reasons: a version move is major or
  minor (gradeVersionMove), a differing config fingerprint is config.
  run({ revise }) selects missing + stale units whose reasons intersect
  the requested set (minor implies major); selected units recompute to
  the current recipe.
- /prospect-analyze takes --revise major|minor|config|all (replacing
  --deep); any reason re-scans all sessions, a plain fill only the
  unanalysed ones.
- analysis_nodes gains config_fingerprint; run mode is recorded as a
  reach label (fill or revise:...).
- New version.ts (version + reason helpers) with unit tests; new
  revise-reasons component test proves major/minor/config selection;
  model-identity test reframed (a model swap is the config reason).
@v2nic
docs: rewrite README for the analyzer-graph extension
82053fe 
The README still described the pre-framework design: a stub analyzer
that ran an LLM over "unprocessed sessions", space-form command names
(/prospect analyze), the wrong database name (sessions.db), proposal
statuses new/accepted/rejected, a non-existent analyze tool action, and
an install path pointing at the wrong owner.

Rewrite it to match the implementation and DESIGN.md's language:

- Dash-form commands (/prospect-sync, /prospect-analyze, ...) with the
  real flags, including --revise major|minor|config|all and the
  --model per-run pin.
- The append-only, incrementally-recomputed analysis graph and its
  deterministic-first layering (turn-pair-core -> turn-pair-llm ->
  session-overview); turn as the unit of analysis.
- Correct DB name (prospector.db), proposal statuses
  (open/applied/rejected/duplicate), and the prospect tool's actual
  actions (no analyze action).
- Configuration documents the fields that actually take effect
  (dbPath, modelTiers) plus the test-only env overrides; install uses
  git:github.com/v2nic/pi-prospector.
- Points at DESIGN.md as the canonical model.
@v2nic
analyze: fix LLM-enrichment merge, enforce pair cap, align role spell…
dd70391 
…ings

Three correctness fixes surfaced by code review, each making the
implementation match its intended design:

- session-overview dropped all turn-pair-llm enrichment. The digest
  merged classifications via anchorUserId(), which always returned null
  (the promised "merge by order" fallback was never written), so
  sentiment/friction never reached the digest or the proposals it feeds.
  turn-pair-llm now records the anchor user_message_id in its node
  content (set from the planned unit, not the model), and the digest
  merges by that id. Split the pure model result (ClassifyResult) from
  the stored shape (TurnPairLLMProperties).

- turn-pair-llm ignored its maxPairsPerSession cost guard: plan() emitted
  a unit per high-signal pair, so a large session could fire unbounded
  LLM calls. plan() now receives the resolved config (new
  AnalyzerPlanContext.config) and enriches at most maxPairsPerSession
  pairs, highest friction first (ties broken by pair order for
  deterministic, idempotent selection).

- role spellings disagreed with the host platform. Pi emits the entry
  types branch_summary and custom_message (snake_case); turn-pair-core's
  turn boundaries already matched, but the MessageRole type, the
  session-overview digest, and a parser test used camelCase
  branchSummary, so branch summaries were silently omitted from digests.
  Standardised on Pi's snake_case across the type, the digest, and the
  test fixture.

Tests: digest merges enrichment by id and includes branch_summary text;
an e2e run with >cap high-signal pairs produces exactly maxPairsPerSession
classifications. 161 unit/component + 19 integration green; coverage ~98.5%.
@v2nic
analyze: fail fast on truncated LLM responses; retry transient thrott…
fec941f 
…ling

Running the analyzers against real sessions through the Pi CLI surfaced two
rough edges in the production LLM path:

- A reasoning model (e.g. gemini-2.5-flash) spends its output-token budget on
  thinking and truncates the JSON answer. The truncated body then failed deep
  in the parser with a cryptic "Unterminated JSON object". toLLMResponse now
  detects stopReason "length" and throws an actionable error naming the output
  limit and pointing at maxTokens / a non-reasoning tier.

- A throttled provider (HTTP 429) failed a whole run on the first call. The
  caller now passes maxRetries so pi-ai rides out transient rate limits.

Also document running a single command non-interactively straight from the Pi
CLI (pi -e src/index.ts -ne --no-session -p "/prospect-..."), including pointing
the env overrides at a small private session subset for local iteration.

Tests: pi-llm asserts the truncation error path. 162 unit/component + 19
integration green.
@v2nic
commands: add --prospect CLI flag for non-interactive runs
beacb71 
Running prospector from the command line previously required Pi's generic
print mode (pi -e src/index.ts -p "/prospect-..."). The extension now registers
its own --prospect flag so a bare invocation runs one command and exits, with
no -p:

  pi -e ./src/index.ts --prospect sync
  pi -e ./src/index.ts --prospect stats
  pi -e ./src/index.ts --prospect "analyze --limit 3 --model openrouter/anthropic/claude-3.5-haiku"
  pi -e ./src/index.ts --prospect proposals

A session_start hook reads the flag, dispatches to the matching command, and
calls ctx.shutdown() (guarded to run once). When --prospect is absent the
extension stays fully interactive. The flag value is "<command> [args]";
commands are sync | analyze | stats | proposals | accept | reject.

To wire this up, each command's handler is extracted into an exported function
(prospectSync/Stats/Proposals/Accept/Reject/Analyze) reused by both the slash
command and the flag dispatcher, and pi-stubs gains registerFlag/getFlag/on plus
ctx.shutdown.

Tests: headless dispatcher unit tests (spec splitting, dispatch, empty/unknown,
error propagation); commands.test mock updated for the new API. 169
unit/component + 19 integration green.
@v2nic
analyze: self-heal failed units via decoupled error-node identity
815ba0c 
An error node's identity is now recipe + message + timestamp (plus the
node id as a uniqueness nonce) instead of the recipe alone, so a failed
attempt never occupies the successful result's identity. classify() sees
only successful nodes at a recipe (findLatestNodeBySourceSet skips errors),
so a unit whose only history is failures stays `missing` and is recomputed
on the next scan. A session is retired from the unanalysed queue only when
it completes with no errors, so a plain `fill` self-heals prior failures
with no special retry mode. Error nodes stay append-only (never deleted).

Verified on real local sessions: a bogus-model run records 9 error nodes
and leaves all sessions queued; a subsequent plain fill recomputes exactly
the 9 missing LLM nodes, retains the 9 error nodes, and a third fill scans
nothing. 170 unit/component + 19 integration green; framework.ts and
analysis-queries.ts at 100% line coverage.
@v2nic
proposals: rank by confidence and add --full evidence/source view
960f58e 
The proposals listing (slash command and `--prospect proposals`) now ranks
recommendations by confidence (desc, nulls last, newest-first tiebreak) and
shows the confidence percentage inline. A new `--full`/`-v` flag adds the
proposal's detail, evidence, and source provenance (session · analyzer ·
proposal id) so analyzer output can be evaluated without dropping to SQL.
Status filtering (open|applied|rejected|duplicate) is unchanged and composes
with the flag in any order.

Adds unit tests for parseProposalsArgs and rankProposals (176 unit/component
+ 19 integration green).
@v2nic
proposals: group recommendations by session
a506b81 
The proposals listing now groups recommendations under a per-session header
(short id · cwd · count), keeping confidence ranking both across sessions
(the session with the strongest single recommendation leads) and within each
group. Adds getSessionLabels query and a sessionLabel helper (cwd with $HOME
collapsed to ~, falling back to project, then short id). Duplicate/overlapping
recommendations are intentionally retained — grouping makes them easy for a
downstream AI consumer to read per session. 178 unit/component + 19
integration green.
@v2nic
analyze: content-address identities (input_key + output_key), add verify
5ce8fa0 
Make analysis identities global, reproducible, and hash-verifiable instead of
DB-local. Two changes to the identity model:

- Rename the recipe identity `input_hash` -> `input_key` (analyzer + version +
  config + source set; inputs only, never the LLM output). The config dimension
  now hashes the config's *content* hash, not its DB-local uuid row id, so
  input_key no longer depends on incidental ids.
- Add `output_key = H(input_key | canonical(content))`, the content-addressed id
  of a node's result. Consumers (turn-pair-llm, session-overview) reference their
  upstream sources by output_key instead of node uuid, so a consumer's input_key
  transitively commits to upstream outputs. The graph is now a Merkle DAG:
  identical inputs+outputs reproduce identical keys on any machine, after any
  wipe.

Proposals: rename `dedup_key` -> `input_key`, derived from the source node's
output_key + ordinal (never the model's title/path/severity). Re-materialising a
node is idempotent; distinct sources keep their proposals (duplicates retained,
grouped per session at display).

Add `prospect verify`: recompute every node's output_key from stored content and
confirm it matches (tamper/corruption detection).

Tests: cross-DB reproducibility (identical input_key/output_key over the same
fixture in two independent DBs), verify clean+tamper+unparseable, output_key
unit tests, and all renamed-field updates. 183 unit/component + 20 integration
green; input-hash.ts/proposal-materializer.ts 100%, verify.ts ~99%, ~98% overall.
Validated on real sessions: 155 nodes, 16 proposals, 0 errors, verify clean.

Implements #5.
@v2nic
commands: add prospect show <id> evidence drill-down
e49e57f 
Reviewing a proposal previously required re-opening the raw session
transcript to check the LLM's free-text evidence string. `prospect show`
walks the provenance graph instead — proposal → source summary node
--consumes--> the turn nodes --anchors--> their messages — and prints the
verbatim anchored turns: the user text, the assistant text, and crucially
each tool call WITH its arguments, plus any tool-result errors. Because the
overview consumes every turn, output is focused to the high-signal turns
(high friction or an LLM classification) and capped, with a remainder note.

This makes human review trustable without a transcript hunt, and surfaces
mechanism-level reality the text-only classifier misses: e.g. for the
gh-pr-review session it shows the user shouting "PUSHED TO v2nic/..." next
to the agent's actual `git push origin …` (correct) and
`gh pr create --repo v2nic/…`/`--head` calls — exposing that the friction
was PR-creation targeting, not the push.

Also: proposal listings now print the full proposal id with a ready
`prospect show <id>` hint (uuidv7 ids share long prefixes, so short
prefixes are ambiguous); `show` resolves by exact id or unambiguous prefix.

Adds getSessionMessageRows; registers prospect-show slash command + headless
`show` action. Tests: unit (toolCallPreview, renderAnchoredTurns incl. args/
errors/cap) + component (full pipeline → resolve → provenance walk → output).
191 unit/component + 20 integration green; show.ts ~98% lines.
@v2nic
docs: document prospect show + verify; fix stale dedup/term slips
f8f7ae2 
README: add `/prospect-show <id>` (provenance walk to verbatim anchored
turns incl. tool-call arguments) and `/prospect-verify` (the latter was
undocumented since it shipped); note the proposals listing now prints the
full id + a `prospect show <id>` hint and accepts `--full`; add show/verify
to the headless command list.

DESIGN: remove the stale "Dedup key" glossary bullet — proposal identity is
the content-addressed input_key (source output_key + ordinal) and duplicates
are intentionally retained, so the old dedup-by-title concept no longer
exists; add the `prospect show` evidence-walk to the Materialisation concept;
fix one "recommendations" → "proposals" term slip.

README intro: "deduplicated" → "ranked" proposals, matching the design
(overlaps are retained and ranked, not collapsed).
@v2nic
trajectory: add deterministic tool-trajectory analyzer (loops, oscill…
b34b8d8 
…ation, pre-flight gaps)

Implements GitHub issue #7: a new session-anchored, deterministic (no LLM)
analyzer that detects friction in the tool-call trajectory of a session.

New detectors:
- stuck-loop: same tool + near-identical normalized args invoked N≥3 times
  without an intervening success/state change
- polling-loop: read-only command repeated while waiting for external state
- oscillation: action followed by its inverse on the same target within a
  sliding window (checkout x→y→x, push→force-push, create→delete)
- pre-flight gap: mutating command that fails on a missing precondition

New files:
- src/analyze/analyzers/tool-trajectory/{index,config,arg-parser,detectors}.ts
- tests/unit/tool-trajectory.test.ts (30 tests per detector)
- tests/component/tool-trajectory.test.ts (2 component tests)

Changes:
- DESIGN.md: define ubiquitous-language terms (trajectory signal, stuck-loop,
  polling-loop, oscillation, pre-flight gap); update node-kind definition
- AGENTS.md: register tool-trajectory in code organization
- session-overview digest: consume trajectory nodes, render signal lines
- session-overview analyzer: declare tool-trajectory as dependency
- defaults.ts: register tool-trajectory analyzer in dependency order
- tests/unit/digest.test.ts: add trajectoryNodes to buildDigest calls

All 223 unit+component tests pass. Integration tests pass. tsc clean.
@v2nic v2nic closed this Jun 12, 2026
@v2nic
v2nic deleted the recall-7-trajectory branch June 12, 2026 03:13
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Recall: add a deterministic tool-trajectory analyzer (loops, oscillation, pre-flight gaps)

1 participant

@v2nic