Hello, this repository is what I believe I can upload having in mind copyrights of project that used Plano Nacional de Leitura (PNL) to make a text complexity metric for Portuguese. PNL is the Portuguese Ministry of Education list of books to be read by students of different school grades.
NB: Since 2020 (I believe) the recommendations are no longer always given by grade, at times they are by "skill level", but I used past years grade recommendations.
1 - I bought a series of books for which I had an associated schooling grade of the PNL. 2 - I parsed them with several OCR tools into .txt UTF-8 (no BOM) files. 3 - I read the books into Python and obtained so text complexity features. 4 - I trained a Machine Learning model to get the year of grade.
1 - Notes on the original data.ipynb || Is a file in which I give some tips on how I imported the data. Naturally I cannot upload the books. 2 - Making the Model.ipynb || Based on a DataFrame that was saved in .xlsx I show how I trained the model. Here you have basis for working with GridSearch, XGBoost, etc... The trained model is a regressor, meaning it provides a number back. xg_reg_ai.dat || Are the pickle files of the trained model. PT_Get_Text.py and Prever_Ano.py || Are the files imported to extract the used features from texts and to predict the year. 3 - Example of Use to Predict.ipynb || A file in which I provide use cases of the above. Presentation_IH.pptx || I first presented this project as a final project of Ironhack DataAnalatyics bootcamp. This file is the presentation I used when delivering the 8 mins presentation.
Here under are the most relevant tools used or adapted. As is:
- NLPy_Port [NLTK based] Mostly used to get POS parts.
- Spacy. Mostly used for stemming and lemmatizing.
Adapted:
- PE2LGP. Just adapted one function. This project is awesome.
- Separasilabas. Did some minor changes to make it more accurate for Portuguese (given it was done for Spanish)
I read a lot of papers, but the most relevant read was a Master thesis from IST: Classificador de textos para o ensino de português como segunda língua, by: Pedro dos Santos Lopes Curto, November 2014.