This repository was archived by the owner on Jul 3, 2026. It is now read-only.
Implement analyzer framework (by Minimax M3) - #3
Closed
v2nic wants to merge 5 commits into
Closed
Conversation
Implements the analyzer framework per docs/analyzer-design-c.md: - Append-only analysis_nodes and typed analysis_edges - analyzer_defs, analyzer_versions, prompt_registry, analyzer_configs - analysis_runs and analysis_progress for incremental cursors - Idempotent input_hash from (analyzer, version, config, prompts, source_set) - Dependency-scoped visibility enforced in run context - Proposal materializer with dedup on (target_type, target_path, severity, normalized title) - Migration 002 adds the new tables and extends messages/proposals - Configured (analyzer_id, config_hash) uniqueness - Edge-kind validation: anchors, consumes, refines, uses_prompt, uses_config, produces - Default analyzer registry: turn-pair-core, turn-pair-llm, session-overview
turn-pair-core: deterministic per-pair metrics - 19 properties from §6.3: lengths, correction detection, tool stats, friction score, model/usage capture, compaction boundary - Friction score uses weighted sum of binary signals (correction, tool_failure threshold, retry, thinking, compaction) - Step function on tool failures (>= max_tool_failures => full weight) - Anchors to every message in the pair (user, assistant, tool results) turn-pair-llm: LLM enrichment for high-signal pairs - Filters dependency nodes to correction_detected OR friction_score >= 0.4 - Calls cheap model with structured classify prompt - Refines + consumes the deterministic node; uses_prompt edges - Captures LLM cost/tokens on run and node rows session-overview: session-level analysis with map-reduce - Builds structured digest from messages + pair nodes - Splits into segments when digest > use_map_reduce_over_chars - Map phase on each segment (cheap model), reduce phase on the combined summaries (mid model) - Produces improvement_proposals materialized into the proposals table - consumes both dependency analyzers; uses_prompt edges for map+reduce
….meta_json The deterministic turn-pair-core analyzer needs the assistant's model, usage, and stop_reason to populate per-pair metrics. The JSONL parser now extracts these from the assistant message envelope and the sync loop writes them as a JSON meta_json column on the messages table.
- analyze command runs the framework's three default analyzers in order over each unanalyzed session; supports --analyzer and --limit; reports node and proposal counts - proposals command now reads the enriched view (target_type, target_path, title); accept/reject also work on 'open' status - stats command shows analyzer-framework health (registered analyzers, node counts per analyzer, successful runs) - tool gains an 'analyze' action callable from a Pi agent - index.ts installs the default LLM caller (delegates to pi.ai) - drop src/analyze/prompt.ts and parser.ts: their job moved to the per-analyzer prompt modules with proper schemas
Unit tests: - framework-hash: shortHash, fullHash, source_set_hash, prompt bundle hash, input hash, edge-kind validation - turn-pair-patterns: detectCorrection, detectAllCorrectionPatterns, detectRepetition, extractCorrectionText, computeFrictionScore - turn-pair-builder: buildTurnPairNode length/thinking/correction/ tool calls/failures/retry/model/elapsed/waste/compaction - turn-pair-llm: buildTurnPairLlmPrompt, parseTurnPairLlmResponse - session-overview: buildDigest, splitDigest, parseMapResponse, parseReduceResponse, buildMapPrompt, buildReducePrompt Component tests: - framework: registration, idempotent re-run, source-set changes, LLM cost capture, error nodes, proposal materialization, dedup, dependency visibility, crash recovery (stale running runs) - turn-pair-llm: end-to-end enrichment of high-signal pairs - session-overview: end-to-end with materialized proposals - e2e: real fixture sync + full framework flow, verifies meta_json is captured on assistant messages
Owner
Author
|
Closing in favour of #4 ( WhyThis is the best-engineered of the earlier analyzer attempts — clean v2 schema, dependency-visibility enforcement, edge/ref validation, granular commits, 115 green tests — and #4 deliberately keeps that structure. But it does not work in production, and the gap is structural rather than a tuning issue:
What replaces it#4 carries over the strengths of this branch (typed-edge append-only graph, idempotent
No commits are lost — the branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the analyzer framework described in
docs/analyzer-design-c.md:an append-only analysis graph grafted onto the conversation tree, with
typed edges, idempotent recipe hashing, dependency-scoped visibility,
and materialization of
proposalsfromanalysis_nodes.The framework replaces the previous single-prompt LLM stub in
src/commands/analyze.tswith a real pipeline:turn-pair-core (deterministic)
↓ high-signal pairs only
turn-pair-llm (cheap LLM classify)
↓
session-overview (map-reduce, cheap→mid)
All three run over the existing SQLite index. Re-runs are
idempotent: a node's
input_hashis(analyzer, version, config, prompts, source_set). Changing any of those produces a new node;old nodes are preserved.
Data model
Adds migration 002 in
src/db/schema.ts:analyzer_defs,analyzer_versions,prompt_registry,analyzer_configs— registryanalysis_runs,analysis_progress— execution + incrementalcursor
analysis_nodes(append-only, noparent_id, no anchor cols)analysis_edges— typed relationships (anchors, consumes,refines, uses_prompt, uses_config, produces)
messages.meta_jsonfor assistant model/usage/stopReasonproposalswithanalyzer_id,target_type,target_path,title,analysis_node_id,confidence,dedup_key,updated_atThe framework enforces (analyzer_id, config_hash) uniqueness so two
analyzers can share a default config without colliding.
Analyzers
turn-pair-core(deterministic)19 properties per pair: lengths, thinking, correction detection
(strong/weak/negation regex sets), tool call/failure/retry counts,
tool waste bytes (results not referenced in the assistant text),
elapsed seconds, friction score, model/usage/stopReason,
compaction-boundary flag. No LLM.
turn-pair-llm(in-process LLM,cheaptier)Filters
turn-pair-coredependency nodes tocorrection_detectedORfriction_score >= 0.4. Calls a cheapmodel with a structured classify prompt. Emits sentiment,
frustration level, correction type, friction cause, user intent,
quality score. Refines + consumes the deterministic node.
session-overview(in-process LLM, map→reduce)Builds a structured digest from messages + pair nodes. If the
digest fits in
use_map_reduce_over_chars, runs a single reducecall. Otherwise splits into segments, runs map on each (cheap
model), then reduce (mid model). Produces a session summary, key
friction points, sentiment arc, and a list of improvement
proposals. The framework materializes each proposal into a
proposalsrow with a 1:1analysis_nodesrow ofnode_kind = 'proposal'. Edges:produces(overview→proposal),anchors(proposal→session),
consumes(overview→dependency nodes),uses_prompt(overview→map+reduce hashes).Commands
/prospect analyze [--analyzer ID] [--limit N]runs the threedefault analyzers in order.
--analyzerruns a single one./prospect proposalsnow showstarget_type:target_pathandthe proposal's
titlefrom the new schema./prospect accept|rejectworks on both legacynewandframework
openstatuses./prospect statsreports analyzer-framework health(registered analyzers, nodes by analyzer/kind, successful runs).
prospecttool gains ananalyzeaction callable from a Pisession.
Tests
115 tests, all green:
tests/unit/framework-hash.test.ts— hashing, edge kindstests/unit/turn-pair-patterns.test.ts— correction detection,friction score
tests/unit/turn-pair-builder.test.ts— all 19 propertiestests/unit/turn-pair-llm.test.ts— prompt building, responseparsing (defensive, clamps, rejects invalid enums)
tests/unit/session-overview.test.ts— digest, split, map/reduceprompt parsing
tests/component/framework.test.ts— registration,idempotency, source-set changes, dependency visibility, error
nodes, dedup, crash recovery
tests/component/turn-pair-llm.test.ts,session-overview.test.ts— end-to-end with stub LLM
tests/component/e2e.test.ts— sync real fixtures + run thefull pipeline
Out of scope (per §12 of the design)
Notes
proposalsschema is preserved; bothproposalsandanalysis_nodes WHERE node_kind = 'proposal'are populated for new materializations.
over the same session and observing
nodesSkippedincrementwhile
nodesProducedstays at 0.