Fix incorrect char_interval for non-ASCII text due to tokenizer merging (Fixes #334)#350

ambicuity wants to merge 2 commits intogoogle:maingoogle/langextract:mainfrom

ambicuity:fix/issue-334-regex-tokenizerambicuity/langextract:fix/issue-334-regex-tokenizerCopy head branch name to clipboard