Add tokenization exceptions and detect some false positive EOS#192

Merged

percevalw merged 2 commits intomasterfrom fix-tokenizer

Mar 7, 2023

+73-14

Commits on Mar 1, 2023

feat: add tokenization exceptions and detect some false positive end of sent periods
percevalw
committed
feat: update adicap patterns to match dot formats as in #158
percevalw
committed