One language can be written in multiple scripts, so it will be detected as a different
ScriptLanguage(language + script)
Does not have any models, just matches the alphabet. Not recommended to use as a standalone detector, it's more like a word separator + language prefilter for an actual language detector (Langram).
Splits text (iterator CharIndices) to words, and detects ScriptLanguages (language + script) of words by used letters (chars).
To split text to the iterator of WordLang:
let word_iter = words::from_ch_ind::<Vec<char>>(text.char_indices());If you don't need individual words, but just want to analyze a full text:
let (all_words, all_langs) = fulltext_filter_with_margin_sorted::<Vec<char>, 95>(text.char_indices());It will give you all Words (Vec<Word<Vec<char>>>) of text and Vec<(ScriptLanguage, u32)> filtered with a less then 5% margin for an error.
Instead of Vec<char> you can use other types of words.
Look at the alphabets.rs to understand what languages have already defined alphabets. Some of them need validation.
Warning: can return words with chars from the Unicode private area (for example Lingala, Nuer or Yoruba languages), because of char normalization (composition with Inherited), and there are no such chars defined in Unicode.