Frustration lexicon: learn multi-word phrases, not just single tokens

[elecnix]

Problem

The learned frustration lexicon added in #39 judges one token at a time. Running it over a real corpus made the limitation concrete: a French-speaking user's frustration was not detected at all, despite the vocabulary being nominated correctly.

The terms were judged individually, and individually they are all neutral:

term	verdict	but in context
laisse	neutral / und	laisse tomber — "forget it", clear disengagement
tomber	neutral / fr	⬆
trop	neutral / fr	trop lent — "too slow", dissatisfaction
lent	neutral / und	⬆

Each judgement is correct for the token in isolation. The signal simply does not live in any single token — it lives in the bigram.

This is not French-specific. English has the same shape: never mind, forget it, come on, what the hell, not again, same thing. Single-token judgement structurally cannot see any of them.

Why it is not a prompt fix

The token prompt in frustration-lexicon/prompt.ts deliberately asks about a token's habitual usage with no session context, because a term node's recipe is the term alone. Adding an example sentence would make identity dishonest — the recipe would claim "just the word" while the verdict actually turned on one session's sentence, and the first session to nominate the word would silently fix the verdict for everyone. That constraint is correct and should stay.

The fix is to make phrases first-class subjects, not to weaken term identity.

Sketch

Everything needed is already in place — a phrase is just another corpus-keyed subject:

• lexicon-candidates additionally nominates bigrams (and possibly trigrams) from adjacent tokens in the same message, ranked by frequency, under their own cap.
• frustration-lexicon judges a phrase exactly as it judges a term. The source ref stays {kind: "term", id: "laisse tomber"} — the existing term source kind already carries arbitrary normalised strings, so a phrase gets the same corpus-wide cache and the same revises lineage for free.
• turn-frustration matches phrases over the ordered token stream (it already tokenises; matching an n-gram is a windowed compare) and emits the same one-node-per-(turn, signal) hit, so additivity is preserved.

Open questions worth settling before building:

• Combinatorics. Bigrams are far more numerous than unigrams and most are junk. Frequency ranking plus a per-session cap probably suffices, but it needs measuring against a real corpus before the cap is chosen — the cache is permanent, so junk judged once is junk stored forever.
• Overlap. When both putain and putain c'est faux match a turn, do we emit both hits, or prefer the longest match? Emitting both is simpler and stays additive; the weight calculation should probably not double-count.
• Do phrases need their own analyzer version axis, or is bumping lexicon-candidates enough? (Nomination changing is what alters what gets judged.)

Value

Likely the single highest-recall improvement available to the lexicon. Multi-word expressions are how a lot of natural frustration is actually phrased, in every language — and the corpus run shows we currently score all of it at zero.

Follow-up to #39.

[elecnix]

Reopening: the implementation in #45 was reverted in #48.

Per-pair LLM adjudication of every adjacent word pair does not work. Measured over the real corpus, phrases were 84% of adjudications and 75% of hits, and were overwhelmingly noise. Prompt tightening plus a deterministic redundancy guard cut the volume (flag rate 6.66% → 1.65%) but not the kind — surviving entries were accepted or, after another, an explicit, aach wbaagb.

The need is still real: laisse tomber and trop lent are genuine signals whose component words are each neutral, and single-word judgement cannot reach them.

Constraints a future attempt should respect, from the measurements here:

• Candidate generation is the problem, not the judge. 190,125 distinct bigrams from 1,533 sessions, 128,645 occurring in exactly one session. No judge finds the handful of real idioms in that.
• A recurrence floor does not work either. laisse tomber and trop lent each appear in exactly one session, while the bigrams recurring across hundreds are in the, of the, do not. Filtering by frequency keeps the noise and discards the signal.
• Redundancy with word-level signals must be excluded — most flagged pairs merely contained an already-flagged word.

Plausible directions: a curated idiom seed list per language (small, high precision, no discovery cost), or statistical collocation measures (PMI / log-likelihood) to select genuine fixed expressions before any model sees them.