NLP_Homework_2_NLTK install -lexdiv_diff score_Olanipekun.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lexical Diversity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Introductory Examples for the NLTK Book ***\n",
      "Loading text1, ..., text9 and sent1, ..., sent9\n",
      "Type the name of the text or sentence to view it.\n",
      "Type: 'texts()' or 'sents()' to list the materials.\n",
      "text1: Moby Dick by Herman Melville 1851\n",
      "text2: Sense and Sensibility by Jane Austen 1811\n",
      "text3: The Book of Genesis\n",
      "text4: Inaugural Address Corpus\n",
      "text5: Chat Corpus\n",
      "text6: Monty Python and the Holy Grail\n",
      "text7: Wall Street Journal\n",
      "text8: Personals Corpus\n",
      "text9: The Man Who Was Thursday by G . K . Chesterton 1908\n"
     ]
    }
   ],
   "source": [
    "from nltk.book import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Displaying 21 of 21 matches:\n",
      "[ The Man Who Was Thursday by G . K . Chesterton 1908 ] To Ed\n",
      "l applause , the difficult part of Thursday , has died quite suddenly . Conseq\n",
      "st a settled thing that I am to be Thursday .\" \" My dear fellow .\" said Syme h\n",
      "then -- oh , the wild joy of being Thursday !\" And he clasped his hands . Syme\n",
      "idor . CHAPTER III THE MAN WHO WAS THURSDAY BEFORE one of the fresh faces coul\n",
      "mpany present the man who shall be Thursday . If any comrade suggests a name I\n",
      "ve that Comrade Gregory be elected Thursday ,\" and sat lumberingly down again \n",
      "esolute , forcible , and efficient Thursday ( hear , hear ).\" \" Comrade Gregor\n",
      "hat Comrade Gregory is unfit to be Thursday for all his amiable qualities . He\n",
      "able qualities . He is unfit to be Thursday because of his amiable qualities .\n",
      " as ready to stand for the post of Thursday , a roar of excitement and assent \n",
      " heavier voice , \" And you are not Thursday .\" \" Comrades ,\" cried Gregory , i\n",
      "ade Syme be elected to the post of Thursday on the General Council .\" The roar\n",
      "rvice , was elected to the post of Thursday on the General Council of the Anar\n",
      "ck and revolver , the duly elected Thursday of the Central Council of Anarchis\n",
      " it was a ritual . Perhaps the new Thursday was always chased along Cheapside \n",
      "ents . \" You ' re to be dressed as Thursday , sir ,\" said the valet somewhat a\n",
      "et somewhat affably . \" Dressed as Thursday !\" said Syme in meditation . \" It \n",
      " ,\" said the other eagerly , \" the Thursday costume is quite warm , sir . It f\n",
      " why I should be particularly like Thursday in a green frock spotted all over \n",
      "ou were to hell . I know how you , Thursday , crossed swords with King Satan ,\n"
     ]
    }
   ],
   "source": [
    "text9.concordance(\"Thursday\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "69213"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#length: words and punctuation. these are the tokens\n",
    "len(text9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A token is the technical name for a sequence of characters — such as hairy, his, or :) — that we want to treat as a group. When we count the number of tokens in a text, say, the phrase to be or not to be, we are counting occurrences of these sequences. Thus, in our example phrase there are two occurrences of to, two of be, and one each of or and not. But there are only four distinct vocabulary items in this phrase. How many distinct words does the book of Genesis contain? To work this out in Python, we have to pose the question slightly differently. The vocabulary of a text is just the set of tokens that it uses, since in a set, all duplicates are collapsed together. In Python we can obtain the vocabulary items of text3 with the command: set(text3). When you do this, many screens of words will fly past. Now try the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['!',\n",
       " '!\"',\n",
       " \"!'\",\n",
       " '!--',\n",
       " '\"',\n",
       " '\"\\'',\n",
       " '\"--',\n",
       " '\";',\n",
       " \"'\",\n",
       " \"'--\",\n",
       " \"'.\",\n",
       " '\\'.\"',\n",
       " \"';\",\n",
       " \"'?\",\n",
       " '\\'?\"',\n",
       " '(',\n",
       " ')',\n",
       " '),',\n",
       " ').',\n",
       " ').\"',\n",
       " '*',\n",
       " ',',\n",
       " ',\"',\n",
       " \",'\",\n",
       " ',\\'\"',\n",
       " '-',\n",
       " '--',\n",
       " '--\"',\n",
       " \"--'\",\n",
       " '.',\n",
       " '.\"',\n",
       " '.\"\\'',\n",
       " \".'\",\n",
       " '.\\'\"',\n",
       " '.--',\n",
       " '1350',\n",
       " '1908',\n",
       " '217',\n",
       " '45',\n",
       " '7',\n",
       " ':',\n",
       " ':--',\n",
       " ';',\n",
       " '?',\n",
       " '?\"',\n",
       " \"?'\",\n",
       " '?)',\n",
       " 'A',\n",
       " 'ACCUSER',\n",
       " 'ACROSS',\n",
       " 'ANARCHY',\n",
       " 'AS',\n",
       " 'AT',\n",
       " 'About',\n",
       " 'Above',\n",
       " 'Absurd',\n",
       " 'Accepting',\n",
       " 'Acheron',\n",
       " 'Across',\n",
       " 'Acting',\n",
       " 'Adam',\n",
       " 'Admiral',\n",
       " 'Aesop',\n",
       " 'African',\n",
       " 'After',\n",
       " 'Afterwards',\n",
       " 'Again',\n",
       " 'Against',\n",
       " 'Ages',\n",
       " 'Ah',\n",
       " 'Albany',\n",
       " 'Albert',\n",
       " 'Alhambra',\n",
       " 'Alice',\n",
       " 'All',\n",
       " 'Ally',\n",
       " 'Almighty',\n",
       " 'Almost',\n",
       " 'Along',\n",
       " 'Alpine',\n",
       " 'Also',\n",
       " 'Am',\n",
       " 'America',\n",
       " 'American',\n",
       " 'An',\n",
       " 'Anarch',\n",
       " 'Anarchist',\n",
       " 'Anarchists',\n",
       " 'Anarchy',\n",
       " 'And',\n",
       " 'Anne',\n",
       " 'Another',\n",
       " 'Answer',\n",
       " 'Anyhow',\n",
       " 'Archdeacon',\n",
       " 'Are',\n",
       " 'Argus',\n",
       " 'Aristocrats',\n",
       " 'Armageddon',\n",
       " 'As',\n",
       " 'Asiatic',\n",
       " 'Asked',\n",
       " 'Assyrian',\n",
       " 'At',\n",
       " 'Attila',\n",
       " 'B',\n",
       " 'BEFORE',\n",
       " 'BURGUNDY',\n",
       " 'Baal',\n",
       " 'Babel',\n",
       " 'Bad',\n",
       " 'Bagdad',\n",
       " 'Baker',\n",
       " 'Bank',\n",
       " 'Bannockburn',\n",
       " 'Barnum',\n",
       " 'Baron',\n",
       " 'Battersea',\n",
       " 'Battle',\n",
       " 'Be',\n",
       " 'Because',\n",
       " 'Before',\n",
       " 'Beg',\n",
       " 'Being',\n",
       " 'Bellegarde',\n",
       " 'Bentley',\n",
       " 'Besides',\n",
       " 'Better',\n",
       " 'Between',\n",
       " 'Beyond',\n",
       " 'Bible',\n",
       " 'Bibles',\n",
       " 'Biffin',\n",
       " 'Bigamists',\n",
       " 'Blessed',\n",
       " 'Blood',\n",
       " 'Bloody',\n",
       " 'Board',\n",
       " 'Bogy',\n",
       " 'Bohemian',\n",
       " 'Both',\n",
       " 'Bounder',\n",
       " 'Bradshaw',\n",
       " 'Breakfast',\n",
       " 'Brighton',\n",
       " 'British',\n",
       " 'Brother',\n",
       " 'Bruce',\n",
       " 'Buck',\n",
       " 'Buddhism',\n",
       " 'Buddhist',\n",
       " 'Buildings',\n",
       " 'Bull',\n",
       " 'Bulwer',\n",
       " 'Burgundy',\n",
       " 'Burnt',\n",
       " 'But',\n",
       " 'Buttons',\n",
       " 'By',\n",
       " 'Byron',\n",
       " 'C',\n",
       " 'CHAPTER',\n",
       " 'CHASE',\n",
       " 'CONDUCT',\n",
       " 'CRIMINALS',\n",
       " 'Caesar',\n",
       " 'Calais',\n",
       " 'Call',\n",
       " 'Can',\n",
       " 'Candidates',\n",
       " 'Cannot',\n",
       " 'Capital',\n",
       " 'Captain',\n",
       " 'Carnation',\n",
       " 'Catacombs',\n",
       " 'Catch',\n",
       " 'Cathedral',\n",
       " 'Catholic',\n",
       " 'Catholics',\n",
       " 'Central',\n",
       " 'Certainly',\n",
       " 'Chairman',\n",
       " 'Chamberlain',\n",
       " 'Chamberlains',\n",
       " 'Channel',\n",
       " 'Chaos',\n",
       " 'Charing',\n",
       " 'Cheapside',\n",
       " 'Chesterton',\n",
       " 'Children',\n",
       " 'Chinaman',\n",
       " 'Chinese',\n",
       " 'Chiswick',\n",
       " 'Chretiens',\n",
       " 'Christendom',\n",
       " 'Christian',\n",
       " 'Christians',\n",
       " 'Christmas',\n",
       " 'Church',\n",
       " 'Circus',\n",
       " 'City',\n",
       " 'Clapham',\n",
       " 'Clapping',\n",
       " 'Clashing',\n",
       " 'Clean',\n",
       " 'Clerihew',\n",
       " 'Clever',\n",
       " 'Close',\n",
       " 'Clothes',\n",
       " 'Cluny',\n",
       " 'Cockney',\n",
       " 'Cold',\n",
       " 'Colney',\n",
       " 'Colonel',\n",
       " 'Colt',\n",
       " 'Come',\n",
       " 'Common',\n",
       " 'Compared',\n",
       " 'Comrade',\n",
       " 'Comrades',\n",
       " 'Concealment',\n",
       " 'Confound',\n",
       " 'Consequently',\n",
       " 'Constitution',\n",
       " 'Corps',\n",
       " 'Could',\n",
       " 'Council',\n",
       " 'Couples',\n",
       " 'Court',\n",
       " 'Covent',\n",
       " 'Creation',\n",
       " 'Cross',\n",
       " 'Crown',\n",
       " 'Cruelty',\n",
       " 'Crusade',\n",
       " 'Cups',\n",
       " 'Czar',\n",
       " 'D',\n",
       " 'DE',\n",
       " 'DETECTIVE',\n",
       " 'DUEL',\n",
       " 'Damn',\n",
       " 'Dawn',\n",
       " 'Days',\n",
       " 'De',\n",
       " 'Dead',\n",
       " 'Death',\n",
       " 'Delightful',\n",
       " 'Depend',\n",
       " 'Detective',\n",
       " 'Devil',\n",
       " 'Dickens',\n",
       " 'Did',\n",
       " 'Directly',\n",
       " 'Disguised',\n",
       " 'Disgusting',\n",
       " 'Do',\n",
       " 'Dock',\n",
       " 'Doctor',\n",
       " 'Does',\n",
       " 'Don',\n",
       " 'Dosition',\n",
       " 'Dover',\n",
       " 'Down',\n",
       " 'Dr',\n",
       " 'Dressed',\n",
       " 'Ducroix',\n",
       " 'Dunciad',\n",
       " 'Dunedin',\n",
       " 'Dynamite',\n",
       " 'Dynamiter',\n",
       " 'Dynamiters',\n",
       " 'EARTH',\n",
       " 'EXPLAINS',\n",
       " 'EXPOSURE',\n",
       " 'Each',\n",
       " 'Earl',\n",
       " 'Early',\n",
       " 'East',\n",
       " 'Eden',\n",
       " 'Edgware',\n",
       " 'Edmund',\n",
       " 'Egyptian',\n",
       " 'Eh',\n",
       " 'Either',\n",
       " 'Elizabethan',\n",
       " 'Embankment',\n",
       " 'Emperors',\n",
       " 'Empire',\n",
       " 'End',\n",
       " 'Energy',\n",
       " 'Engage',\n",
       " 'England',\n",
       " 'English',\n",
       " 'Erie',\n",
       " 'Et',\n",
       " 'Europe',\n",
       " 'European',\n",
       " 'Eustache',\n",
       " 'Even',\n",
       " 'Ever',\n",
       " 'Every',\n",
       " 'Everyone',\n",
       " 'Everything',\n",
       " 'Evidently',\n",
       " 'Exactly',\n",
       " 'Excuse',\n",
       " 'Exhibition',\n",
       " 'Extraordinary',\n",
       " 'FEAR',\n",
       " 'FEAST',\n",
       " 'FRIEND',\n",
       " 'Family',\n",
       " 'Far',\n",
       " 'Father',\n",
       " 'February',\n",
       " 'Fight',\n",
       " 'Finally',\n",
       " 'Finish',\n",
       " 'First',\n",
       " 'Fleet',\n",
       " 'Fly',\n",
       " 'Fools',\n",
       " 'For',\n",
       " 'Foremost',\n",
       " 'Four',\n",
       " 'France',\n",
       " 'French',\n",
       " 'Frenchman',\n",
       " 'Frenchmen',\n",
       " 'Friday',\n",
       " 'From',\n",
       " 'Funny',\n",
       " 'G',\n",
       " 'GABRIEL',\n",
       " 'Gabriel',\n",
       " 'Garden',\n",
       " 'Gardens',\n",
       " 'Gemini',\n",
       " 'Gendarmerie',\n",
       " 'General',\n",
       " 'Genesis',\n",
       " 'Gentleman',\n",
       " 'Gentlemen',\n",
       " 'George',\n",
       " 'German',\n",
       " 'Germany',\n",
       " 'Get',\n",
       " 'Getting',\n",
       " 'Give',\n",
       " 'Given',\n",
       " 'Glumpe',\n",
       " 'Go',\n",
       " 'God',\n",
       " 'Godforsaken',\n",
       " 'Gogol',\n",
       " 'Going',\n",
       " 'Good',\n",
       " 'Got',\n",
       " 'Government',\n",
       " 'Governments',\n",
       " 'Great',\n",
       " 'Greek',\n",
       " 'Green',\n",
       " 'Gregory',\n",
       " 'Grey',\n",
       " 'Grub',\n",
       " 'Guinea',\n",
       " 'Had',\n",
       " 'Hairy',\n",
       " 'Half',\n",
       " 'Hall',\n",
       " 'Hanwell',\n",
       " 'Harrow',\n",
       " 'Hartle',\n",
       " 'Has',\n",
       " 'Hatch',\n",
       " 'Have',\n",
       " 'Haven',\n",
       " 'Having',\n",
       " 'He',\n",
       " 'Healy',\n",
       " 'Heavy',\n",
       " 'Hell',\n",
       " 'Hence',\n",
       " 'Here',\n",
       " 'Herod',\n",
       " 'High',\n",
       " 'Hill',\n",
       " 'Him',\n",
       " 'His',\n",
       " 'Hitherto',\n",
       " 'Hold',\n",
       " 'Holiday',\n",
       " 'Homeric',\n",
       " 'Honesty',\n",
       " 'Honour',\n",
       " 'Hoping',\n",
       " 'Horatius',\n",
       " 'Horrible',\n",
       " 'Horses',\n",
       " 'How',\n",
       " 'However',\n",
       " 'Humanitarian',\n",
       " 'Humanity',\n",
       " 'Hurt',\n",
       " 'I',\n",
       " 'II',\n",
       " 'III',\n",
       " 'IN',\n",
       " 'IV',\n",
       " 'IX',\n",
       " 'Idiots',\n",
       " 'If',\n",
       " 'Impressionism',\n",
       " 'In',\n",
       " 'Indeed',\n",
       " 'India',\n",
       " 'Inside',\n",
       " 'Inspector',\n",
       " 'Insulted',\n",
       " 'Into',\n",
       " 'Ireland',\n",
       " 'Is',\n",
       " 'It',\n",
       " 'Its',\n",
       " 'Jabberwock',\n",
       " 'Jack',\n",
       " 'Jacobins',\n",
       " 'Jericho',\n",
       " 'Jerusalem',\n",
       " 'Jew',\n",
       " 'Jewels',\n",
       " 'Join',\n",
       " 'Joseph',\n",
       " 'Judas',\n",
       " 'Just',\n",
       " 'K',\n",
       " 'Keep',\n",
       " 'Kensington',\n",
       " 'Kill',\n",
       " 'King',\n",
       " 'LITTLE',\n",
       " 'Lancy',\n",
       " 'Lane',\n",
       " 'Last',\n",
       " 'Law',\n",
       " 'Le',\n",
       " 'Leave',\n",
       " 'Left',\n",
       " 'Legion',\n",
       " 'Leicester',\n",
       " 'Let',\n",
       " 'Life',\n",
       " 'Light',\n",
       " 'Like',\n",
       " 'Lion',\n",
       " 'Listen',\n",
       " 'Little',\n",
       " 'Lo',\n",
       " 'London',\n",
       " 'Look',\n",
       " 'Lord',\n",
       " 'Lucian',\n",
       " 'Ludgate',\n",
       " 'Lush',\n",
       " 'Lust',\n",
       " 'Lytton',\n",
       " 'MAN',\n",
       " 'Macon',\n",
       " 'Man',\n",
       " 'Mansoul',\n",
       " 'Many',\n",
       " 'Marat',\n",
       " 'Marquis',\n",
       " 'Martin',\n",
       " 'May',\n",
       " 'Mayor',\n",
       " 'Mean',\n",
       " 'Meanwhile',\n",
       " 'Mediterranean',\n",
       " 'Memnon',\n",
       " 'Men',\n",
       " 'Mental',\n",
       " 'Mere',\n",
       " 'Middle',\n",
       " 'Might',\n",
       " 'Millionaires',\n",
       " 'Milton',\n",
       " 'Mind',\n",
       " 'Miss',\n",
       " 'Moderate',\n",
       " 'Mohammedan',\n",
       " 'Monday',\n",
       " 'Monsieur',\n",
       " 'Montaigne',\n",
       " 'Moorish',\n",
       " 'More',\n",
       " 'Most',\n",
       " 'Mr',\n",
       " 'Much',\n",
       " 'Murderers',\n",
       " 'Musee',\n",
       " 'Museum',\n",
       " 'Must',\n",
       " 'My',\n",
       " 'NEXT',\n",
       " 'Naples',\n",
       " 'Napoleon',\n",
       " 'Naturally',\n",
       " 'Nature',\n",
       " 'Need',\n",
       " 'Nevertheless',\n",
       " 'New',\n",
       " 'Next',\n",
       " 'Nietzsche',\n",
       " 'Nihilist',\n",
       " 'No',\n",
       " 'Nobody',\n",
       " 'Nonconformists',\n",
       " 'Nonsence',\n",
       " 'Nonsense',\n",
       " 'Nor',\n",
       " 'Norman',\n",
       " 'North',\n",
       " 'Nose',\n",
       " 'Not',\n",
       " 'Nothing',\n",
       " 'Now',\n",
       " 'OF',\n",
       " 'Of',\n",
       " 'Oh',\n",
       " 'On',\n",
       " 'Once',\n",
       " 'One',\n",
       " 'Only',\n",
       " 'Or',\n",
       " 'Ordinarily',\n",
       " 'Originally',\n",
       " 'Our',\n",
       " 'Over',\n",
       " 'PARK',\n",
       " 'PHILOSOPHERS',\n",
       " 'POETS',\n",
       " 'POLICE',\n",
       " 'PRESIDENT',\n",
       " 'PROFESSOR',\n",
       " 'PURSUIT',\n",
       " 'Paddington',\n",
       " 'Pagens',\n",
       " 'Pan',\n",
       " 'Panic',\n",
       " 'Pantheist',\n",
       " 'Paradise',\n",
       " 'Paris',\n",
       " 'Park',\n",
       " 'Parkers',\n",
       " 'Partly',\n",
       " 'Pathetic',\n",
       " 'Paul',\n",
       " 'Paumanok',\n",
       " 'Peabody',\n",
       " 'Perfectly',\n",
       " 'Perhaps',\n",
       " 'Perjury',\n",
       " 'Permit',\n",
       " 'Persian',\n",
       " 'Peste',\n",
       " 'Piccadilly',\n",
       " 'Pinckwerts',\n",
       " 'Place',\n",
       " 'Please',\n",
       " 'Poe',\n",
       " 'Poland',\n",
       " 'Pole',\n",
       " 'Police',\n",
       " 'Polish',\n",
       " 'Pommery',\n",
       " 'Pope',\n",
       " 'Pray',\n",
       " 'Prepare',\n",
       " 'President',\n",
       " 'Prey',\n",
       " 'Priests',\n",
       " 'Professor',\n",
       " 'Project',\n",
       " 'Proverb',\n",
       " 'Pull',\n",
       " 'Puritan',\n",
       " 'Put',\n",
       " 'Queen',\n",
       " 'Quite',\n",
       " 'Railway',\n",
       " 'Raphaelite',\n",
       " 'Ratcliffe',\n",
       " 'Really',\n",
       " 'Red',\n",
       " 'Refreshments',\n",
       " 'Rembrandt',\n",
       " 'Renaissance',\n",
       " 'Renard',\n",
       " 'Republic',\n",
       " 'Resuming',\n",
       " 'Revolt',\n",
       " 'Revolution',\n",
       " 'Right',\n",
       " 'Rights',\n",
       " 'Road',\n",
       " 'Roared',\n",
       " 'Robespierre',\n",
       " 'Roland',\n",
       " 'Roman',\n",
       " 'Romans',\n",
       " 'Rosamond',\n",
       " 'Round',\n",
       " 'Russia',\n",
       " 'Russian',\n",
       " 'Rustic',\n",
       " 'SAFFRON',\n",
       " 'SECRET',\n",
       " 'SIT',\n",
       " 'SIX',\n",
       " 'SNOWDROP',\n",
       " 'SPECTACLES',\n",
       " 'SUCH',\n",
       " 'SYME',\n",
       " 'Sabbatarian',\n",
       " 'Sabbath',\n",
       " 'Saffron',\n",
       " 'Saint',\n",
       " 'Samoa',\n",
       " 'Saracenic',\n",
       " 'Satan',\n",
       " 'Satanic',\n",
       " 'Saturday',\n",
       " 'Saumur',\n",
       " 'Save',\n",
       " 'Savoy',\n",
       " 'Scarcely',\n",
       " 'School',\n",
       " 'Schools',\n",
       " 'Science',\n",
       " 'Scotch',\n",
       " 'Scotland',\n",
       " 'Secret',\n",
       " 'Secretary',\n",
       " 'Seeking',\n",
       " 'Serious',\n",
       " 'Service',\n",
       " 'Seven',\n",
       " 'Shall',\n",
       " 'Shame',\n",
       " 'She',\n",
       " 'Shops',\n",
       " 'Show',\n",
       " 'Sicily',\n",
       " 'Since',\n",
       " 'Sir',\n",
       " 'Skye',\n",
       " 'Slav',\n",
       " 'Sloane',\n",
       " 'Sloper',\n",
       " 'Snow',\n",
       " 'Snowdrop',\n",
       " 'So',\n",
       " 'Soho',\n",
       " 'Soleil',\n",
       " 'Some',\n",
       " 'Something',\n",
       " 'Sometimes',\n",
       " 'Somewhat',\n",
       " 'Somewhere',\n",
       " 'Soon',\n",
       " 'South',\n",
       " 'Southend',\n",
       " 'Spanish',\n",
       " 'Sporting',\n",
       " 'Square',\n",
       " 'St',\n",
       " 'Stare',\n",
       " 'State',\n",
       " 'Still',\n",
       " 'Stop',\n",
       " 'Strangers',\n",
       " 'Strasbourg',\n",
       " 'Street',\n",
       " 'Strike',\n",
       " 'Such',\n",
       " 'Suddenly',\n",
       " 'Suffice',\n",
       " 'Sunday',\n",
       " 'Superbly',\n",
       " 'Superstition',\n",
       " 'Suppose',\n",
       " 'Supreme',\n",
       " 'Surely',\n",
       " 'Surrey',\n",
       " 'Sussex',\n",
       " 'Swiftly',\n",
       " 'Swords',\n",
       " 'Syme',\n",
       " 'Symes',\n",
       " 'TALE',\n",
       " 'THE',\n",
       " 'THURSDAY',\n",
       " 'TWO',\n",
       " 'Take',\n",
       " 'Taking',\n",
       " 'Talk',\n",
       " 'Tall',\n",
       " 'Telephone',\n",
       " 'Tell',\n",
       " 'Ten',\n",
       " 'Thames',\n",
       " 'Thank',\n",
       " 'That',\n",
       " 'The',\n",
       " 'Their',\n",
       " 'Then',\n",
       " 'Theoretically',\n",
       " 'There',\n",
       " 'Therefore',\n",
       " 'These',\n",
       " 'They',\n",
       " 'Thieves',\n",
       " 'Thing',\n",
       " 'Things',\n",
       " 'This',\n",
       " 'Those',\n",
       " 'Though',\n",
       " 'Three',\n",
       " 'Through',\n",
       " 'Throughout',\n",
       " 'Thursday',\n",
       " 'Thursdays',\n",
       " 'Thus',\n",
       " 'Thy',\n",
       " 'Tim',\n",
       " 'Timbuctoo',\n",
       " 'Time',\n",
       " 'Times',\n",
       " 'To',\n",
       " 'Too',\n",
       " 'Tower',\n",
       " 'Traffic',\n",
       " 'Tree',\n",
       " 'Truth',\n",
       " 'Try',\n",
       " 'Tuesday',\n",
       " 'Tupper',\n",
       " 'Turning',\n",
       " 'Tusitala',\n",
       " 'Twenty',\n",
       " 'Twice',\n",
       " 'Two',\n",
       " 'UNACCOUNTABLE',\n",
       " 'URGING',\n",
       " 'Under',\n",
       " 'Underground',\n",
       " 'Unless',\n",
       " 'Up',\n",
       " 'Upon',\n",
       " 'Us',\n",
       " 'Utterly',\n",
       " 'V',\n",
       " 'VI',\n",
       " 'VII',\n",
       " 'VIII',\n",
       " 'Vampire',\n",
       " 'Ven',\n",
       " 'Very',\n",
       " 'Victoria',\n",
       " 'Virtue',\n",
       " 'Vulgar',\n",
       " 'WAS',\n",
       " 'WHEN',\n",
       " 'WHO',\n",
       " 'WORMS',\n",
       " 'Wagner',\n",
       " 'Wake',\n",
       " 'Walking',\n",
       " 'Was',\n",
       " 'Wasn',\n",
       " 'We',\n",
       " 'Weak',\n",
       " 'Wednesday',\n",
       " 'Well',\n",
       " 'West',\n",
       " 'Westminster',\n",
       " 'What',\n",
       " 'Whatever',\n",
       " 'Wheel',\n",
       " 'When',\n",
       " 'Whenever',\n",
       " 'Where',\n",
       " 'Whether',\n",
       " 'Which',\n",
       " 'While',\n",
       " 'Whistler',\n",
       " 'Who',\n",
       " 'Whom',\n",
       " 'Why',\n",
       " 'Wilks',\n",
       " 'Will',\n",
       " 'Wishing',\n",
       " 'With',\n",
       " 'Witherspoon',\n",
       " 'Without',\n",
       " 'Wonderland',\n",
       " 'Worms',\n",
       " 'Would',\n",
       " 'Wrong',\n",
       " 'Wrongs',\n",
       " 'X',\n",
       " 'XI',\n",
       " 'XII',\n",
       " 'XIII',\n",
       " 'XIV',\n",
       " 'XV',\n",
       " 'Yard',\n",
       " 'Yea',\n",
       " 'Yes',\n",
       " 'Yet',\n",
       " 'You',\n",
       " 'Young',\n",
       " 'Your',\n",
       " 'Zoo',\n",
       " 'Zoological',\n",
       " 'Zso',\n",
       " 'Zumpt',\n",
       " '[',\n",
       " ']',\n",
       " 'a',\n",
       " 'abandoned',\n",
       " 'abandonment',\n",
       " 'aberrations',\n",
       " 'abiding',\n",
       " 'able',\n",
       " 'abnegations',\n",
       " 'abnormal',\n",
       " 'abnormally',\n",
       " 'abolish',\n",
       " 'abolished',\n",
       " 'abolishes',\n",
       " 'about',\n",
       " 'above',\n",
       " 'abrupt',\n",
       " 'abruptly',\n",
       " 'abruptness',\n",
       " 'absence',\n",
       " 'absent',\n",
       " 'absently',\n",
       " 'absentminded',\n",
       " 'absinth',\n",
       " 'absolute',\n",
       " 'absolutely',\n",
       " 'abstinence',\n",
       " 'abstract',\n",
       " 'abstractedly',\n",
       " 'abstraction',\n",
       " 'absurd',\n",
       " 'absurdity',\n",
       " 'abused',\n",
       " 'abyss',\n",
       " 'abysses',\n",
       " 'accent',\n",
       " 'accept',\n",
       " 'acceptance',\n",
       " 'accepted',\n",
       " 'accident',\n",
       " 'accidental',\n",
       " 'accidentally',\n",
       " 'accidents',\n",
       " 'acclaim',\n",
       " 'accordingly',\n",
       " 'accuracy',\n",
       " 'accursed',\n",
       " 'accuse',\n",
       " 'accused',\n",
       " 'accuser',\n",
       " 'accuses',\n",
       " 'acknowledged',\n",
       " 'acquaintance',\n",
       " 'acquainted',\n",
       " 'across',\n",
       " 'act',\n",
       " 'acted',\n",
       " 'acting',\n",
       " 'action',\n",
       " 'actions',\n",
       " 'actor',\n",
       " 'acts',\n",
       " 'actual',\n",
       " 'actuality',\n",
       " 'actually',\n",
       " 'acumen',\n",
       " 'adamantine',\n",
       " 'added',\n",
       " 'addition',\n",
       " 'additional',\n",
       " 'address',\n",
       " 'addressed',\n",
       " 'addresses',\n",
       " 'addressing',\n",
       " 'admirable',\n",
       " 'admiration',\n",
       " 'admire',\n",
       " 'admired',\n",
       " 'admirers',\n",
       " 'admit',\n",
       " 'adopt',\n",
       " 'adopted',\n",
       " 'adorable',\n",
       " 'advance',\n",
       " 'advancing',\n",
       " 'advantage',\n",
       " 'advantages',\n",
       " 'adventure',\n",
       " 'adventurer',\n",
       " 'adventurers',\n",
       " 'adventures',\n",
       " 'advertisement',\n",
       " 'advice',\n",
       " 'advise',\n",
       " 'advised',\n",
       " 'aesthete',\n",
       " 'affably',\n",
       " 'affair',\n",
       " 'affected',\n",
       " 'afford',\n",
       " 'afraid',\n",
       " 'after',\n",
       " 'afterglow',\n",
       " 'afternoon',\n",
       " 'afterwards',\n",
       " 'again',\n",
       " 'against',\n",
       " 'age',\n",
       " 'aged',\n",
       " 'agent',\n",
       " 'aggressive',\n",
       " 'ago',\n",
       " 'agonies',\n",
       " 'agonised',\n",
       " 'agony',\n",
       " 'agree',\n",
       " 'agreed',\n",
       " 'ahead',\n",
       " 'ailment',\n",
       " 'aim',\n",
       " 'aimed',\n",
       " 'aimless',\n",
       " 'aims',\n",
       " 'air',\n",
       " 'airless',\n",
       " 'airs',\n",
       " 'airy',\n",
       " 'alarm',\n",
       " 'alarmed',\n",
       " 'ale',\n",
       " 'alien',\n",
       " 'aliens',\n",
       " 'alighted',\n",
       " 'alive',\n",
       " 'all',\n",
       " 'alleged',\n",
       " 'allegiance',\n",
       " 'allegoric',\n",
       " 'allegory',\n",
       " 'alley',\n",
       " 'allies',\n",
       " 'allowed',\n",
       " 'allusion',\n",
       " 'allusions',\n",
       " 'ally',\n",
       " 'almond',\n",
       " 'almost',\n",
       " 'alone',\n",
       " 'along',\n",
       " 'alongside',\n",
       " 'aloud',\n",
       " 'alphabet',\n",
       " 'alphabetical',\n",
       " 'already',\n",
       " 'also',\n",
       " 'alter',\n",
       " 'alteration',\n",
       " 'altered',\n",
       " 'alternate',\n",
       " 'alternative',\n",
       " 'although',\n",
       " 'altogether',\n",
       " 'always',\n",
       " 'am',\n",
       " 'amain',\n",
       " 'amazement',\n",
       " 'ambassador',\n",
       " 'ambassadors',\n",
       " 'ambiguous',\n",
       " 'ambling',\n",
       " ...]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#vocabulary\n",
    "sorted(set(text9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6807"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(text9))\n",
    "#we can see that Although it has 69,213 tokens, this book has only 6,807 distinct words, or \"word types.\" \n",
    "#A word type is the form or spelling of the word independently of its specific occurrences in a text — \n",
    "#that is, the word considered as a unique item of vocabulary. \n",
    "#Our count of 6,807 items will include punctuation symbols, \n",
    "#so we will generally call these unique items types instead of word types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's calculate a measure of the lexical richness of the text. The next example shows us that the number of distinct words is just 9.8% of the total number of words, or equivalently that each word is used 16 times on average (remember if you're using Python 2, to start with from __future__ import division)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0983485761345412"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(text9)) / len(text9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can count how often a word occurs in a text, and compute what percentage of the text is taken up by a specific word:\n",
    "You may want to repeat such calculations on several texts, but it is tedious to keep retyping the formula. Instead, you can come up with your own name for a task, like \"lexical_diversity\" or \"percentage\", and associate it with a block of code. Now you only have to type a short name instead of one or more complete lines of Python code, and you can re-use it as often as you like. The block of code that does a task for us is called a function, and we define a short name for our function with the keyword def. The next example shows how to define two new functions, lexical_diversity() and percentage():\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lexical_diversity(text):\n",
    "   return len(set(text)) / len(text) \n",
    "\n",
    "def vocab_size(text):\n",
    "    return len(set(word.lower() for word in text)) # this accounts for case\n",
    "    #return len(set(text))\n",
    "\n",
    "def percentage(count, total):\n",
    "    return 100 * count / total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt2 = \"We can count how Often often Apple a Word occurs in A text of Of Apple apple mon tue wed\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(word.lower() for word in txt2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22, 88)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(txt2)), len(txt2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'default_wt' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-14-f29f809f5a95>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mupper\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdefault_wt\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtxt2\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mupper\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mupper\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mword\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mword\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mtxt2\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mupper\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'default_wt' is not defined"
     ]
    }
   ],
   "source": [
    "upper = default_wt(txt2)\n",
    "len(set(upper)), len(upper), len(set(word.lower() for word in txt2)), upper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lexical_diversity(text9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size(text9)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Homework 1\n",
    "2.\tFollow the instructions in chapter 1 of Bird-Klein for implementing a “lexical diversity” scoring routine.\n",
    "\n",
    "3.\tGo to http://www.gutenberg.org/wiki/Children%27s_Instructional_Books_(Bookshelf), and obtain three texts (of different grade levels) from the “Graded Readers” section. Report the lexical diversity score of each. Explain whether the result was surprising.\n",
    "\n",
    "4.\tAlso compare the vocabulary size of the same three texts. Explain whether the result was surprising.  \n",
    "\n",
    "5.\tWrite a paragraph arguing whether vocabulary size and lexical diversity in combination could be a better measure of text difficulty (or reading level) than either measure is by itself.\n",
    "\n",
    "Approach:\n",
    "webscrape the text from the gutenberg website.: successful\n",
    "Remove html tags: successfull\n",
    "remove title to use only main text: unsuccessfull.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Token:a sequence of characters — such as hairy, his, or :) — that we want to treat as a group.They are words and punctuation symbols."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Custom function for computing the lexical diversity\n",
    "def lexical_diversity(text):\n",
    "   return len(set(text)) / len(text) \n",
    "\n",
    "def percentage(count, total):\n",
    "    return 100 * count / total"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Texts from Gutenberg project\n",
    "- Issues with the archive portal did not allow me to download more books.\n",
    "- Other texts from the Gutenberg will added from the downloaded Gutenberg library of nltk.\n",
    "\n",
    "### McGuffey's Eclectic Reader\n",
    "According to Wikipedea, McGuffey Readers were a series of graded primers for grade levels 1-6. They were widely used as textbooks in American schools from the mid-19th century to the early-20th century, and are still used today in homeschooling.\n",
    "\n",
    "\n",
    "### Text preprocessing\n",
    "Title of text: McGuffey's Fifth Eclectic Reader by William Holmes McGuffey\n",
    "- html tags were removed\n",
    "- text was subset for the main text\n",
    "    - so Preface and 'End of Project.....' were removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Path to the McGuffey's First Eclectic Reader for Grade level 1\n",
    "data = requests.get('https://web.archive.org/web/20210212173048/http://www.gutenberg.org/cache/epub/14640/pg14640.txt')\n",
    "eclectic_1_a = data.content\n",
    "print(eclectic_1_a[:5300])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(eclectic_1_a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Path to the McGuffey's Fifth Eclectic for grade level 5\n",
    "data2 = requests.get('https://www.gutenberg.org/cache/epub/15040/pg15040.txt')\n",
    "eclectic_5_b = data2.content\n",
    "print(eclectic_5_b[:2300])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Path to the McGuffey's Third Eclectic reader for grade level 3\n",
    "data3 = requests.get('https://www.gutenberg.org/cache/epub/14766/pg14766.txt')\n",
    "eclectic_3a = data3.content\n",
    "print(eclectic_3a[:2300])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Removing the html tags\n",
    "import re\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def strip_html_tags(text):\n",
    "    soup = BeautifulSoup(text, \"html.parser\")\n",
    "    [s.extract() for s in soup(['iframe', 'script'])]\n",
    "    stripped_text = soup.get_text()\n",
    "    stripped_text = re.sub(r'[\\r|\\n|\\r\\n]+', '\\n', stripped_text)\n",
    "    return stripped_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_special_characters(text):\n",
    "    pattern = r'[^a-zA-z\\s]'\n",
    "    text = re.sub(pattern, '', text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The text is in form of individual characters not in words like the contemporaries are.\n",
    "- So I will word tokenize the text to show the tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_wt = nltk.word_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eclectic_1b = strip_html_tags(eclectic_1_a)    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eclectic_1b_ = remove_special_characters(eclectic_1b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eclectic_1b_[:1500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#word tokenize\n",
    "clean_eclectic_1b_word= default_wt(eclectic_1b_)\n",
    "len(clean_eclectic_1b_word), len(eclectic_1b_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eclectic_3b = strip_html_tags(eclectic_3a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eclectic_3b_ = remove_special_characters(eclectic_3b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_eclectic_3b_word= default_wt(eclectic_3b_)\n",
    "len(clean_eclectic_3b_word), len(eclectic_3b_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eclectic_5b = strip_html_tags(eclectic_5_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eclectic_5b_ = remove_special_characters(eclectic_5b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#word tokenize\n",
    "clean_eclectic_5b_word= default_wt(eclectic_5b_)\n",
    "len(clean_eclectic_5b_word), len(eclectic_5b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove stop words \n",
    "Not on this one given that stop words may be part of the vocabulary "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#stopword_list = nltk.corpus.stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#def remove_stopwords(text):\n",
    "#     stopwords = nltk.corpus.stopwords.words('english')\n",
    "#     content = [w for w in text if w.lower() not in stopwords]\n",
    "#     return content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#eclectic_1_word_clean = remove_stopwords(clean_eclectic_1b_word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#len(eclectic_1_word_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#eclectic_3_word_clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subset the cleaned corpus for the main text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_eclectic_1b_word.index(\"Cincinnati\"), clean_eclectic_1b_word[804:808]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_eclectic_1b_word.index(\"formats\"), clean_eclectic_1b_word[7777:7780]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eclectic_1_Main = clean_eclectic_1b_word[808:7777]\n",
    "eclectic_1_Main[100:200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_eclectic_3b_word.index(\"Apostrophe\"), clean_eclectic_3b_word[1488:1527]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean_eclectic_3b_word.index(\"Produced\")#,\n",
    "clean_eclectic_3b_word[26561:26600]\n",
    "eclectic_3_Main = clean_eclectic_3b_word[1527:26562]\n",
    "eclectic_3_Main[:300]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Fifth read\n",
    "#clean_eclectic_5b_word.index('redistribution'), \n",
    "clean_eclectic_5b_word[1602:1610]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_eclectic_5b_word.index('redistribution'), clean_eclectic_5b_word[96712:96720]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean_eclectic_3b_word.index(\"Produced\")#,\n",
    "eclectic_5_Main = clean_eclectic_5b_word[1602:96712]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(eclectic_5_Main)\n",
    "eclectic_5_Main[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Selecting texts from gutenberg library for comparison\n",
    "#from nltk.corpus import gutenberg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#nltk.corpus.gutenberg.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#carroll = nltk.Text(nltk.corpus.gutenberg.words('carroll-alice.txt'))\n",
    "#milton = nltk.Text(nltk.corpus.gutenberg.words('milton-paradise.txt'))\n",
    "#bryant = nltk.Text(nltk.corpus.gutenberg.words('bryant-stories.txt'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_list = [\n",
    "    (\"McGuffey's First Eclectic\", eclectic_1_Main), \n",
    "    (\"McGuffey's Third Eclectic\", eclectic_3_Main), \n",
    "    (\"McGuffey's Fifth Eclectic\", eclectic_5_Main)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, text in text_list:\n",
    "    lex_div = lexical_diversity(text)\n",
    "    vocab = vocab_size(text) #Accounts for lower case\n",
    "    print('for ',name + \"'s text, the lexical diversity is\", lex_div, 'and vocabulary size is', vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- It is an eye opener that an almost 100,000 text token size only contains about 14% distinct words.\n",
    "\n",
    "- This preliminary study shows that the vocabulary size increased considerably with higher grade level (Grade 1 = 6,969, Grade 5 = 95,110)\n",
    "\n",
    "- However, from the output, the lexical richness depletes from First Eclectic Reader (Grade 1) to the Fifth Eclectic Reader (Grade 5) \n",
    "    -I was expecting the lexical diversity to increase with grade level but the contrast is the case in this study.\n",
    "\n",
    "- Clearly, there is considerable disparity in the lexical diversity as seen in the outputs above. This disparity is much more meaningful when paired with the vocabulary size of the respective text. \n",
    "        -This perspective can be seeing in the comparison of the Third Eclectic Reader and the Fifth Eclectic reader with relatively lexical diversity for widely different vocabulary size (about 25,035 versus 95,110). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Scoring and Normalization\n",
    "\n",
    "#### Min-Max Normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a function for this. This is call a Min-Max Scaler. Also available in Sci-Kit learn\n",
    "def MinMaxScale(size):\n",
    "    size = np.asarray(size, dtype=np.float32)\n",
    "    maxmin = max(size) - min (size)\n",
    "    return list((size-min(size))/maxmin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the MinMaxScale, the lowest number will set to 0 while the highest number will be set to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size_normal = MinMaxScale([vocab_size(text) for name, text in text_list])\n",
    "vocab_size_normal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Score Long words\n",
    " - Normalizing the occurence of long words in the vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us start with looking at the usefulness finding the 'Long' words of a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def long_word_vocab_size(text, length):\n",
    "    return len(set(word.lower() for word in text if len(word) > length))\n",
    "\n",
    "def long_word_vocab_words(text, length):\n",
    "    return sorted(set(word.lower() for word in text if len(word) > length))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using length = 15 returned 0 counts, therefore my long words default threshold to 8, but can be changed to anything.\n",
    "Setting the longwords threshold to 6 produced reasonable vocabulary. I figure the text will not contain a lot of real long words given that the text is for young adults trying to learn to read.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Testing the custome function\n",
    "long_word_vocab_size(eclectic_3_Main, 6), long_word_vocab_words(eclectic_3_Main, 6)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Which words typify these texts?. \n",
    "- The list of frequently occurring long words commonly give indication of the contextual meaning of a text. \n",
    "- We can design a custom function for that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vocab_context(text, length, count):\n",
    "    fdist5 = FreqDist(text)\n",
    "    return sorted(w for w in set(text) if len(w) > length and fdist5[w] > count)\n",
    "    #return (w for w in long_word_vocab_size(text, 6) and fdist5[w] > 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_context(eclectic_3_Main, 6, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Populate a dictionary with words that typify each text.\n",
    "voca_cont= {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, text in text_list:\n",
    "    voca_context = vocab_context(text, 6, 7)\n",
    "    voca_cont[name] = voca_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let s see what the First Eclectic \n",
    "voca_cont[\"McGuffey's First Eclectic\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above, the First Electic is mostly about Children and Grandma. A family friendly text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[long_word_vocab_size(text, 6) for name, text in text_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "norm_long_word_vocab = MinMaxScale([long_word_vocab_size(text, 6) for name, text in text_list])\n",
    "norm_long_word_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.\t“text difficulty score” by combining the lexical diversity score from homework 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lexical_diversity_scores = [lexical_diversity(text) for name, text in text_list]  # save lexical diversity to list\n",
    "lexical_diversity_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_scores = np.asarray(vocab_size_normal) + np.asarray(norm_long_word_vocab) + np.asarray(lexical_diversity_scores)\n",
    "text_difficulty_score = list(total_scores / 3)\n",
    "text_difficulty_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#for name, text in text_list:\n",
    "#    for score in text_difficulty_score:\n",
    "#        print('for ',name + \"'s text, the text difficulty score is \", score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"for McGuffey's First Eclectic's text\", \"the text difficulty score is \", text_difficulty_score[0])\n",
    "print(\"for McGuffey's Third Eclectic's text\", \"the text difficulty score is \", text_difficulty_score[1])\n",
    "print(\"for McGuffey's Fifth Eclectic's text\", \"the text difficulty score is \", text_difficulty_score[2])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare with previously obtained Lexical diversity and vocabulary size "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, text in text_list:\n",
    "    lex_div = lexical_diversity(text)\n",
    "    vocab = vocab_size(text) #Accounts for lower case\n",
    "    print('for ',name + \"'s text, the lexical diversity is\", lex_div, 'and vocabulary size is', vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comment\n",
    "\n",
    "As a reminder, the Fifth McGuffey's Eclectic text is set to be more 'difficult' than the First. The 'levels' (i.e. difficulty level) increase for each reading Grade.\n",
    "The lexical diversity measure obtained without any normalization was not descriptive enough to show the level of difficulty.\n",
    "    As can be seeing we cannot tell a difference between the Fifth and the First Eclectic.\n",
    "    \n",
    "However, normalized vocabulary seem to better reflect that the Fifth Eclectic (0.71) is more difficult than the First Eclectic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}