feat(lexicon): learn multi-word phrases, not just single tokens

[elecnix]

Closes #40

Problem

Running the lexicon over a real corpus made the limitation concrete: a French-speaking user's frustration went undetected, even though the vocabulary was nominated correctly. The terms were judged individually, and individually they are all neutral:

term	verdict	but in context
laisse	neutral	laisse tomber — "forget it"
tomber	neutral	⬆
trop	neutral	trop lent — "too slow"
lent	neutral	⬆

Every one of those verdicts is correct for the token in isolation. The signal simply does not live in any single token. Not French-specific — never mind, forget it, come on, not again all share the shape.

Why not a prompt fix

The term prompt deliberately asks about a token's habitual usage with no session context, because a term node's recipe is the term alone. Feeding it an example sentence would make identity dishonest — the recipe would claim "just the word" while the verdict turned on one session's sentence. That constraint stays; phrases become first-class subjects instead.

What made this small

A phrase is a corpus-wide string, so its source ref stays term-kind (words joined by a space). It inherits the existing corpus-wide cache, the existing revises lineage, and the existing one-node-per-(turn, signal) additivity — no new machinery.

• Phrases are extracted as adjacent bigrams only — a width-2 sliding window over the observed token stream, never a permutation set. A segment of n tokens yields n−1 candidates, not n(n−1); order comes from the text, so laisse tomber is a candidate and tomber laisse is not.
• Built within a sentence only. fix it. laisse tomber must not yield it laisse — that is a seam, not speech. tokenizeSegments splits on .!?;:\n while keeping commas inside, so putain, c'est faux stays one segment.
• turn-frustration matches over those same segments, inheriting the Unicode-correct, substring-proof guarantees.

Second commit: the cap was wrong, and so was the obvious fix

Reviewing how ranking worked exposed a flaw in the existing per-session cap, and investigating it showed the tempting fix to be worse.

The cap bought nothing. It applied at nomination, which is deterministic and graph-blind, so the same ranked list is re-offered every session and ordinary words occupied every slot permanently. Measured: 92% of nomination slots went to terms already judged, and 62% of sessions hit the cap with vocabulary left unoffered.

Budgeting the unjudged entries breaks idempotency. It makes planning a function of graph state, so each re-run frees the previous run's slots:

run 1: nodesProduced=61   LLM calls=60
run 2: nodesProduced=60   LLM calls=120
run 3: nodesProduced=60   LLM calls=180
run 4: nodesProduced=60   LLM calls=240

That violates "re-running analysis without changing the version, config, or inputs produces no new nodes." Rejected; a test now pins idempotency across repeated passes.

A recurrence floor optimises for the wrong tail. It would cut phrases 190k → 34k, but the idioms this feature exists to catch are hapaxes:

phrase	sessions
in the, of the, do not	400–700
laisse tomber	1
trop lent	1

It would discard the signal and keep the noise.

What actually bounds spend is the corpus cache, which needs no cap: a session only ever pays for vocabulary no earlier session used, so total spend is the corpus's distinct vocabulary, once. The ceilings remain as node-size bounds, raised high enough not to bind (largest real session nominated 990 terms).

Measured cost over 1,533 sessions: 14,321 distinct words + 190,125 distinct phrases ≈ 204k one-time cheap-tier adjudications (~$7.80), then near-zero as the corpus saturates. Shipping uncapped is a deliberate recall-over-cost choice.

Versions

lexicon-candidates and turn-frustration → 1.1 (behaviour); frustration-lexicon → 1.1 (prompt now covers phrases). Existing single-word verdicts stay valid and are re-judged only by an explicit --revise minor — upgrading invalidates nothing.

Test plan

• npm test — 427 passing. New unit tests for tokenizeSegments / rankPhrases / matchPhrases (boundary handling, locale-independent ordering, no cross-sentence bigrams); component tests for the end-to-end case (every component word neutral, turn still flagged via the phrase, phrase cached corpus-wide); and cost tests pinning idempotency and the "pays only for what is new" property.
• node --import tsx test/integration/test-commands.ts — 21 passing.
• npx tsc --noEmit clean.
• Verified against the real 1,533-session corpus for the measurements quoted above.

One test-helper fix was needed: classifyCallsFor matched TERM: putain by substring, so TERM: putain c'est counted as a second adjudication of the single word. Now matched exactly.

🤖 Generated with Claude Code