Skip to content
This repository was archived by the owner on Jul 3, 2026. It is now read-only.

Implement analyzer framework (by Minimax M3) - #3

Closed
v2nic wants to merge 5 commits into
mainfrom
feature/analyzer-design-c
Closed

Implement analyzer framework (by Minimax M3)#3
v2nic wants to merge 5 commits into
mainfrom
feature/analyzer-design-c

Conversation

@v2nic

@v2nic v2nic commented Jun 2, 2026

Copy link
Copy Markdown
Owner

Summary

Implements the analyzer framework described in docs/analyzer-design-c.md:
an append-only analysis graph grafted onto the conversation tree, with
typed edges, idempotent recipe hashing, dependency-scoped visibility,
and materialization of proposals from analysis_nodes.

The framework replaces the previous single-prompt LLM stub in
src/commands/analyze.ts with a real pipeline:

turn-pair-core (deterministic)
↓ high-signal pairs only
turn-pair-llm (cheap LLM classify)

session-overview (map-reduce, cheap→mid)

All three run over the existing SQLite index. Re-runs are
idempotent: a node's input_hash is (analyzer, version, config, prompts, source_set). Changing any of those produces a new node;
old nodes are preserved.

Data model

Adds migration 002 in src/db/schema.ts:

  • analyzer_defs, analyzer_versions, prompt_registry,
    analyzer_configs — registry
  • analysis_runs, analysis_progress — execution + incremental
    cursor
  • analysis_nodes (append-only, no parent_id, no anchor cols)
  • analysis_edges — typed relationships (anchors, consumes,
    refines, uses_prompt, uses_config, produces)
  • Extends messages.meta_json for assistant model/usage/stopReason
  • Extends proposals with analyzer_id, target_type,
    target_path, title, analysis_node_id, confidence,
    dedup_key, updated_at

The framework enforces (analyzer_id, config_hash) uniqueness so two
analyzers can share a default config without colliding.

Analyzers

turn-pair-core (deterministic)

19 properties per pair: lengths, thinking, correction detection
(strong/weak/negation regex sets), tool call/failure/retry counts,
tool waste bytes (results not referenced in the assistant text),
elapsed seconds, friction score, model/usage/stopReason,
compaction-boundary flag. No LLM.

turn-pair-llm (in-process LLM, cheap tier)

Filters turn-pair-core dependency nodes to
correction_detected OR friction_score >= 0.4. Calls a cheap
model with a structured classify prompt. Emits sentiment,
frustration level, correction type, friction cause, user intent,
quality score. Refines + consumes the deterministic node.

session-overview (in-process LLM, map→reduce)

Builds a structured digest from messages + pair nodes. If the
digest fits in use_map_reduce_over_chars, runs a single reduce
call. Otherwise splits into segments, runs map on each (cheap
model), then reduce (mid model). Produces a session summary, key
friction points, sentiment arc, and a list of improvement
proposals. The framework materializes each proposal into a
proposals row with a 1:1 analysis_nodes row of node_kind = 'proposal'. Edges: produces (overview→proposal), anchors
(proposal→session), consumes (overview→dependency nodes),
uses_prompt (overview→map+reduce hashes).

Commands

  • /prospect analyze [--analyzer ID] [--limit N] runs the three
    default analyzers in order. --analyzer runs a single one.
  • /prospect proposals now shows target_type:target_path and
    the proposal's title from the new schema.
  • /prospect accept|reject works on both legacy new and
    framework open statuses.
  • /prospect stats reports analyzer-framework health
    (registered analyzers, nodes by analyzer/kind, successful runs).
  • prospect tool gains an analyze action callable from a Pi
    session.

Tests

115 tests, all green:

  • tests/unit/framework-hash.test.ts — hashing, edge kinds
  • tests/unit/turn-pair-patterns.test.ts — correction detection,
    friction score
  • tests/unit/turn-pair-builder.test.ts — all 19 properties
  • tests/unit/turn-pair-llm.test.ts — prompt building, response
    parsing (defensive, clamps, rejects invalid enums)
  • tests/unit/session-overview.test.ts — digest, split, map/reduce
    prompt parsing
  • tests/component/framework.test.ts — registration,
    idempotency, source-set changes, dependency visibility, error
    nodes, dedup, crash recovery
  • tests/component/turn-pair-llm.test.ts, session-overview.test.ts
    — end-to-end with stub LLM
  • tests/component/e2e.test.ts — sync real fixtures + run the
    full pipeline

Out of scope (per §12 of the design)

  • Pi sub-agent execution engine
  • Eager supersession of old nodes
  • Cross-session meta-analyzers
  • Target file auto-discovery

Notes

  • The legacy proposals schema is preserved; both
    proposals and analysis_nodes WHERE node_kind = 'proposal'
    are populated for new materializations.
  • Idempotency has been verified by re-running the same analyzer
    over the same session and observing nodesSkipped increment
    while nodesProduced stays at 0.

v2nic added 5 commits June 2, 2026 11:11
Implements the analyzer framework per docs/analyzer-design-c.md:
- Append-only analysis_nodes and typed analysis_edges
- analyzer_defs, analyzer_versions, prompt_registry, analyzer_configs
- analysis_runs and analysis_progress for incremental cursors
- Idempotent input_hash from (analyzer, version, config, prompts, source_set)
- Dependency-scoped visibility enforced in run context
- Proposal materializer with dedup on (target_type, target_path, severity, normalized title)
- Migration 002 adds the new tables and extends messages/proposals
- Configured (analyzer_id, config_hash) uniqueness
- Edge-kind validation: anchors, consumes, refines, uses_prompt, uses_config, produces
- Default analyzer registry: turn-pair-core, turn-pair-llm, session-overview
turn-pair-core: deterministic per-pair metrics
- 19 properties from §6.3: lengths, correction detection, tool stats,
  friction score, model/usage capture, compaction boundary
- Friction score uses weighted sum of binary signals (correction,
  tool_failure threshold, retry, thinking, compaction)
- Step function on tool failures (>= max_tool_failures => full weight)
- Anchors to every message in the pair (user, assistant, tool results)

turn-pair-llm: LLM enrichment for high-signal pairs
- Filters dependency nodes to correction_detected OR friction_score >= 0.4
- Calls cheap model with structured classify prompt
- Refines + consumes the deterministic node; uses_prompt edges
- Captures LLM cost/tokens on run and node rows

session-overview: session-level analysis with map-reduce
- Builds structured digest from messages + pair nodes
- Splits into segments when digest > use_map_reduce_over_chars
- Map phase on each segment (cheap model), reduce phase on the
  combined summaries (mid model)
- Produces improvement_proposals materialized into the proposals table
- consumes both dependency analyzers; uses_prompt edges for map+reduce
….meta_json

The deterministic turn-pair-core analyzer needs the assistant's model,
usage, and stop_reason to populate per-pair metrics. The JSONL parser
now extracts these from the assistant message envelope and the sync
loop writes them as a JSON meta_json column on the messages table.
- analyze command runs the framework's three default analyzers
  in order over each unanalyzed session; supports --analyzer and
  --limit; reports node and proposal counts
- proposals command now reads the enriched view (target_type,
  target_path, title); accept/reject also work on 'open' status
- stats command shows analyzer-framework health (registered
  analyzers, node counts per analyzer, successful runs)
- tool gains an 'analyze' action callable from a Pi agent
- index.ts installs the default LLM caller (delegates to pi.ai)
- drop src/analyze/prompt.ts and parser.ts: their job moved to
  the per-analyzer prompt modules with proper schemas
Unit tests:
- framework-hash: shortHash, fullHash, source_set_hash, prompt
  bundle hash, input hash, edge-kind validation
- turn-pair-patterns: detectCorrection, detectAllCorrectionPatterns,
  detectRepetition, extractCorrectionText, computeFrictionScore
- turn-pair-builder: buildTurnPairNode length/thinking/correction/
  tool calls/failures/retry/model/elapsed/waste/compaction
- turn-pair-llm: buildTurnPairLlmPrompt, parseTurnPairLlmResponse
- session-overview: buildDigest, splitDigest, parseMapResponse,
  parseReduceResponse, buildMapPrompt, buildReducePrompt

Component tests:
- framework: registration, idempotent re-run, source-set changes,
  LLM cost capture, error nodes, proposal materialization, dedup,
  dependency visibility, crash recovery (stale running runs)
- turn-pair-llm: end-to-end enrichment of high-signal pairs
- session-overview: end-to-end with materialized proposals
- e2e: real fixture sync + full framework flow, verifies
  meta_json is captured on assistant messages
@v2nic v2nic changed the title Implement analyzer framework per docs/analyzer-design-c.md Implement analyzer framework (by Minimax M3) Jun 5, 2026
@v2nic

v2nic commented Jun 11, 2026

Copy link
Copy Markdown
Owner Author

Closing in favour of #4 (analyzer-graph), which supersedes this branch.

Why

This is the best-engineered of the earlier analyzer attempts — clean v2 schema, dependency-visibility enforcement, edge/ref validation, granular commits, 115 green tests — and #4 deliberately keeps that structure. But it does not work in production, and the gap is structural rather than a tuning issue:

  • The production LLM caller targets an API that does not exist. makePiLLMCaller calls pi.ai.complete(...). Pi exposes no pi.ai.complete — the LLM path is modelRegistry.findgetApiKeyAndHeaders@earendil-works/pi-ai complete(). Optional-chaining on the missing method yields undefined, the surrounding catch returns empty text, and the parsers receive nothing.
  • The green tests pass because they bypass the broken path. They inject a stubbed caller via setDefaultLLMCaller, so the suite proves the analyzers and framework are correct but never exercises the real provider call. In a real Pi session this branch silently produces no proposals.

What replaces it

#4 carries over the strengths of this branch (typed-edge append-only graph, idempotent input_hash, visibility enforcement, edge validation, clean v2 schema, three-analyzer pipeline) and fixes the wiring:

  • LLM is wired to Pi's real AI provider system (modelRegistry.findgetApiKeyAndHeaders@earendil-works/pi-ai complete()), with a deterministic mock caller for tests.
  • Adds 100% incremental scan() with shallow/deep modes and versioned node lineage via revises edges, so re-analysis with a newer analyzer produces navigable alternatives instead of overwriting.
  • An end-to-end test asserts a proposal is materialised through the full pipeline — the one guarantee that was missing here.
  • 130 unit/component + 19 integration assertions green; ~98% line coverage; clean tsc.

No commits are lost — the branch feature/analyzer-design-c and ref remain available for reference. Closing rather than merging.

@v2nic v2nic closed this Jun 11, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant