Skip to content

Shared lexicon (c): one corporate-suffix token base + composer, preserving the four vocabularies' memberships #917

Description

@s-annam

Part (c) of #653. Behaviour-preserving consolidation with one deliberate distinction to protect.

Problem

Corporate-suffix vocabulary (Inc, LLC, Ltd, GmbH, …) exists as four separate sets across the parser. The #641 fix — "the captured token carries a trailing period" — had to be diagnosed and applied per copy, which is the tell.

Implementation plan

  1. Create one token base under src/lib/heuristics/extract/ holding the suffix tokens as data.
  2. Add a composer that generates each of the four vocabularies from it, with an allowTrailingDot option so the [parser] Pass B single-token city rule cleaves a company whose last word precedes a state tail (Palo Alto Networks, CA) #641 behaviour is expressed once and applied where it belongs.
  3. Repoint the four sites at the composer.

Constraint — do not collapse the sets

Membership stays per-set human judgement. The tail-deferral vocabulary at src/lib/heuristics/extract/experience-disambiguate.ts:188-199 is deliberately broader than COMPANY_SUFFIX_RE: it includes tokens like Media and Partners. Adding those to the strict set would flip looksLikeTitle false on a real title such as "Media Director".

So this issue shares the token base and the composition mechanics, not the memberships. Each generated set keeps its own explicit token list and a docblock saying what it is for and why it differs.

Acceptance criteria

Parent: #653. Roadmap item 5(c) of #646.

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvementEnhancing existing functionalityrefactorCode restructuring without behavior change

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions