1.phase_1 -test multi.ipynb
This code uses multiprocess technique to process files. It is recommended to run files seperately, because it stores data in the memory first, which means the memory might be full and the program crashs if too many files have been run. Here we run 1k files at a time.
2.Deal_with_the_rest_of_data.ipynb
Some files cannot be processed normally due to some reasons. This code splits and reruns those files.
3.Wash.ipynb
While we process a image file, we create a pdf version behind it and extract the texts. However, if we run our code for this image file at the second time, the code will read both of the image file and pdf version of it. This can waste a huge amount of time. Therefore, it is recommended to remove all extra files from original folders. This code removes all 'converted.pdf' and 'extractable.pdf' files from folders.
4.Sample_data.ipynb
This code works for the response we received from UNGC. Basically it extracts all targeting corp_id, and match them with overall files to gather basic file information, and then move to extract texts.
5.phase_1 -test multi(final update).ipynb
This code is a modified version of phase_1 -test multi.ipynb, which adds a funtion to clean the text.
6.2018_COP_phase_1_result.xlsx
This is the big table generated by phase_1 -test multi(final update).ipynb
7.PHASE_3.ipynb
This code is used to do the sentiment analysis.