-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expand 4cat to other languages (Finnish) #470
base: master
Are you sure you want to change the base?
Conversation
…ors. Missing: filter
@ErikBorra @stijn-uva Would be lovely if you could check and merge this soonish! :) Please let me know if there are any issues with the code, happy to patch this PR. |
@sal-uva you removed spacy last year. I cannot recall or find what the specific issue was and only vaguely remember some dependency problems. Do you recall? |
There were some dependency problems (iirc it didn't like our version of NumPy) and because of its resource-heavy nature was only suited for small-scale datasets for (among others) entity recognition, which can now be done much better with LLMs. But if it's handy for Finnish and it doesn't cause any dependency errors for you, all good to add back in! |
I think we did not encounter issues, but happy to test more if that helps - any hints on what to test out? |
These are the changes proposed in this pull request:
Wordlists added:
Processors changed:
In the setup, SpaCy is included again to allow for lemmatization on Finnish text.