NLP_Home work 3_Olanipekun.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c18ecd84",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import numpy as np\n",
    "from string import punctuation\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.porter import PorterStemmer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91774d5a",
   "metadata": {},
   "source": [
    "- What is the edit distance between your nickname and your given name?.\n",
    "- What is the percentage string match between your nickname and your given name?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "51cd4424",
   "metadata": {},
   "outputs": [],
   "source": [
    "firstname = 'John'\n",
    "nickname = \"Dolomite\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "956ec68d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Levenshtein edit-distance between John and Dolomite is 7\n"
     ]
    }
   ],
   "source": [
    "print(\"The Levenshtein edit-distance between {} and {} is {}\". \n",
    "      format(firstname, nickname,nltk.edit_distance(firstname.lower(),nickname.lower(), transpositions = True)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "85555b52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The computed percentage match between John and Dolomite is 25.0 and the matching string/s is/are: ['o']\n"
     ]
    }
   ],
   "source": [
    "# Percentage String Match\n",
    "common_list = [string for string in firstname if string in nickname]\n",
    "print(\"The computed percentage match between {} and {} is {} and the matching string/s is/are: {}\". \n",
    "      format(firstname, nickname,len(common_list)/len(firstname)*100, common_list))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f054e08a",
   "metadata": {},
   "source": [
    "### 2.\n",
    "Find a friend (or family member or classmate) who you know has read a certain book. Without your friend knowing, copy the first two sentences of that book. Now rewrite the words from those sentences, excluding stop words. Now tell your friend to guess which book the words are from by reading them just that list of words. Did you friend correctly guess the book on the first try? What did he or she guess? Explain why you think you friend either was or was not able to guess the book from hearing the list of words.\n",
    "\n",
    "-My friend recently read Hyperion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5624a27",
   "metadata": {},
   "source": [
    "#### Book: Hyperion, p1 by Dan Simmons:\n",
    "- First two sentences: \n",
    "    'On the eve of Armageddon, with the entire galaxy at war, seven pilgrims set forth on a voyage to the Time Tombs. They seek the answers to the unsolved riddles of their lives and what they find may hold the key to the deliverance of humanity itself.'\n",
    "    \n",
    "Source:\n",
    "https://onlinereadfreenovel.com/dan-simmons/37212-hyperion.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1b8a4001",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperion_2 = \"\"\"On the eve of Armageddon, with the entire galaxy at war, seven pilgrims set forth on a voyage to the Time Tombs. They seek the answers to the unsolved riddles of their lives and what they find may hold the key to the deliverance of humanity itself.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c2335cb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# my standard practice is to remove special characters\n",
    "import re\n",
    "def remove_special_characters(text):\n",
    "    pattern = r'[^a-zA-z\\s]'\n",
    "    text = re.sub(pattern, '', text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fad670ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'On the eve of Armageddon with the entire galaxy at war seven pilgrims set forth on a voyage to the Time Tombs They seek the answers to the unsolved riddles of their lives and what they find may hold the key to the deliverance of humanity itself'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hyper_clean = remove_special_characters(hyperion_2)\n",
    "hyper_clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "954de965",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['eve', 'armageddon', 'entire', 'galaxy', 'war', 'seven', 'pilgrims', 'set', 'forth', 'voyage', 'time', 'tombs', 'seek', 'answers', 'unsolved', 'riddles', 'lives', 'find', 'may', 'hold', 'key', 'deliverance', 'humanity']\n"
     ]
    }
   ],
   "source": [
    "import string\n",
    "\n",
    "tokens = [word for word in hyper_clean if word not in string.punctuation]\n",
    "tokens = hyper_clean.lower().split(' ')\n",
    "tokens = [word for word in tokens if word not in set(stopwords.words('english'))]\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "111dbb66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['eve', 'Armageddon', 'entire', 'galaxy', 'war', 'seven', 'pilgrims', 'set', 'forth', 'voyage', 'Time', 'Tombs', 'seek', 'answers', 'unsolved', 'riddles', 'lives', 'find', 'may', 'hold', 'key', 'deliverance', 'humanity']\n"
     ]
    }
   ],
   "source": [
    "#almost the same result as before but I like the below one better.\n",
    "\n",
    "default_wt = nltk.word_tokenize\n",
    "hyper_tokens = [word for word in default_wt(hyper_clean) if word.lower() not in set(stopwords.words('english'))]\n",
    "\n",
    "#hyper_token = default_wt(hyper_clean)\n",
    "print(hyper_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ae32ddf",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0300426f",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = nltk.Text(hyper_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8eb48506",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lives find may hold key deliverance humanity find may hold key\n",
      "deliverance humanity answers unsolved riddles lives find may hold key\n",
      "deliverance humanity seven pilgrims set forth voyage Time Tombs seek\n",
      "answers unsolved riddles lives find may hold key deliverance humanity\n",
      "entire galaxy war seven pilgrims set forth voyage Time Tombs seek\n",
      "answers unsolved riddles lives find may hold key deliverance humanity\n",
      "hold key deliverance humanity galaxy war seven pilgrims set forth\n",
      "voyage Time Tombs seek answers unsolved riddles lives find may hold\n",
      "key deliverance humanity Tombs seek answers unsolved riddles lives\n",
      "find may hold key deliverance humanity war seven\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building ngram index...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'lives find may hold key deliverance humanity find may hold key\\ndeliverance humanity answers unsolved riddles lives find may hold key\\ndeliverance humanity seven pilgrims set forth voyage Time Tombs seek\\nanswers unsolved riddles lives find may hold key deliverance humanity\\nentire galaxy war seven pilgrims set forth voyage Time Tombs seek\\nanswers unsolved riddles lives find may hold key deliverance humanity\\nhold key deliverance humanity galaxy war seven pilgrims set forth\\nvoyage Time Tombs seek answers unsolved riddles lives find may hold\\nkey deliverance humanity Tombs seek answers unsolved riddles lives\\nfind may hold key deliverance humanity war seven'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Let us generate text using the token list\n",
    "#successfully generated random text as stated in Chapter 1 of nltk book but it just repeats the tokenized words.\n",
    "gene_text = text.generate()\n",
    "gene_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd8b652c",
   "metadata": {},
   "source": [
    "#### Comment on Question2:\n",
    "My friend immediately recognized that those words came from the Hyperion book series that he read."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04b9dd12",
   "metadata": {},
   "source": [
    "3.\tRun one of the stemmers available in Python. Run the same two sentences from question 2 above through the stemmer and show the results. How many of the outputted stems are valid morphological roots of the corresponding words? Express this answer as a percentage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7381227e",
   "metadata": {},
   "source": [
    "##### Stemming\n",
    "- Morphemes are the smallest independent uniy in any language. \n",
    "    - they consist of stems and affixes that are attached to the word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c9cc7cfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "from nltk.stem import SnowballStemmer\n",
    "ps = PorterStemmer()\n",
    "ss = SnowballStemmer(\"english\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9a1ad219",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['eve', 'armageddon', 'entir', 'galaxi', 'war', 'seven', 'pilgrim', 'set', 'forth', 'voyag', 'time', 'tomb', 'seek', 'answer', 'unsolv', 'riddl', 'live', 'find', 'may', 'hold', 'key', 'deliver', 'human']\n"
     ]
    }
   ],
   "source": [
    "#Stem list_ porter\n",
    "hyper_stem = [ps.stem(word) for word in hyper_tokens]\n",
    "print(hyper_stem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6b5d43e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['eve', 'armageddon', 'entir', 'galaxi', 'war', 'seven', 'pilgrim', 'set', 'forth', 'voyag', 'time', 'tomb', 'seek', 'answer', 'unsolv', 'riddl', 'live', 'find', 'may', 'hold', 'key', 'deliver', 'human']\n"
     ]
    }
   ],
   "source": [
    "#Stem list_Snowball\n",
    "hyper_stem_ss = [ss.stem(word) for word in hyper_tokens]\n",
    "print(hyper_stem_ss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e33abd7",
   "metadata": {},
   "source": [
    "#### Lemmatization\n",
    "The process of lemmatization is very similar to stemming, where we remove word affixes\n",
    "to get to a base form (i.e. root) of the word. However in this case, this base form is also known\n",
    "as the root word but not the root stem. The difference between the two is that the root\n",
    "stem may not always be a lexicographically correct word, i.e., it may not be present in\n",
    "the dictionary but the root word, also known as the lemma, will always be present in the\n",
    "dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6c4cbf05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('eve', 'JJ'), ('Armageddon', 'NNP'), ('entire', 'JJ'), ('galaxy', 'NN'), ('war', 'NN'), ('seven', 'CD'), ('pilgrims', 'NNS'), ('set', 'VBD'), ('forth', 'JJ'), ('voyage', 'NN'), ('Time', 'NNP'), ('Tombs', 'NNP'), ('seek', 'JJ'), ('answers', 'NNS'), ('unsolved', 'VBD'), ('riddles', 'NNS'), ('lives', 'NNS'), ('find', 'VBP'), ('may', 'MD'), ('hold', 'VB'), ('key', 'JJ'), ('deliverance', 'NN'), ('humanity', 'NN')]\n"
     ]
    }
   ],
   "source": [
    "#POS tag the tokens\n",
    "hyper_pos = nltk.pos_tag(hyper_tokens)\n",
    "print(hyper_pos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f6a313a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Using the Wordnet Lemmatizer.\n",
    "\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "wnl = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "319064c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Lemmatization only takes POS tags that are available in Wordnet. \n",
    "#Therefore I converted the nltk POS labels to their respective wordnet variant using the custom function below.\n",
    "#https://stackoverflow.com/questions/61982023/using-wordnetlemmatizer-lemmatize-with-pos-tags-throws-keyerror\n",
    "\n",
    "from nltk.corpus import wordnet\n",
    "\n",
    "def get_wordnet_pos(treebank_tag):\n",
    "    if treebank_tag.startswith('J'):\n",
    "        return wordnet.ADJ\n",
    "    elif treebank_tag.startswith('V'):\n",
    "        return wordnet.VERB\n",
    "    elif treebank_tag.startswith('N'):\n",
    "        return wordnet.NOUN\n",
    "    elif treebank_tag.startswith('R'):\n",
    "        return wordnet.ADV\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f2fa42cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#one word ('seven') returned 'None' pos label, \n",
    "#so I tweaked my custom function to convert the label to 'Noun' in this case.\n",
    "def get_wordnet_pos_edit(treebank_tag):\n",
    "    if treebank_tag.startswith('J'):\n",
    "        return wordnet.ADJ\n",
    "    elif treebank_tag.startswith('V'):\n",
    "        return wordnet.VERB\n",
    "    elif treebank_tag.startswith('N'):\n",
    "        return wordnet.NOUN\n",
    "    elif treebank_tag.startswith('R'):\n",
    "        return wordnet.ADV\n",
    "    else:\n",
    "        return wordnet.NOUN #Just to ensure every word has a suitable tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "dee341db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenized Hyperion with wordnet POS tags: \n",
      " [('eve', 'a'), ('Armageddon', 'n'), ('entire', 'a'), ('galaxy', 'n'), ('war', 'n'), ('seven', 'n'), ('pilgrims', 'n'), ('set', 'v'), ('forth', 'a'), ('voyage', 'n'), ('Time', 'n'), ('Tombs', 'n'), ('seek', 'a'), ('answers', 'n'), ('unsolved', 'v'), ('riddles', 'n'), ('lives', 'n'), ('find', 'v'), ('may', 'n'), ('hold', 'v'), ('key', 'a'), ('deliverance', 'n'), ('humanity', 'n')]\n"
     ]
    }
   ],
   "source": [
    "#Accummulates the list of words and their wordnet POS tags\n",
    "#The list will be parsed into the lemmatization function.\n",
    "\n",
    "hyper_prelem = [(word, get_wordnet_pos_edit(tag)) for word, tag in hyper_pos]\n",
    "print(\"Tokenized Hyperion with wordnet POS tags: \\n\", hyper_prelem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0f88939d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['eve', 'Armageddon', 'entire', 'galaxy', 'war', 'seven', 'pilgrim', 'set', 'forth', 'voyage', 'Time', 'Tombs', 'seek', 'answer', 'unsolved', 'riddle', 'life', 'find', 'may', 'hold', 'key', 'deliverance', 'humanity']\n"
     ]
    }
   ],
   "source": [
    "#Accummulates the list of lemm\n",
    "\n",
    "hyper_lem = [wnl.lemmatize(word, tag) for word, tag in hyper_prelem]\n",
    "print(hyper_lem)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db7564dd",
   "metadata": {},
   "source": [
    "##### Valid roots\n",
    "Given that stemming often does not return valid morphological roots, I compared the stem list to lemm list (valid root). \n",
    "- i.e. what percentage of lemm is stem? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "4824452d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The computed percentage match between stem and lemm is 52% and the matching root words are:\n",
      " ['eve', 'war', 'seven', 'pilgrim', 'set', 'forth', 'seek', 'answer', 'find', 'may', 'hold', 'key']\n"
     ]
    }
   ],
   "source": [
    "root_list = [root for root in hyper_stem_ss if root in hyper_lem]\n",
    "print(\"The computed percentage match between {} and {} is {}% and the matching root words are:\\n {}\". \n",
    "      format('stem', 'lemm',round(len(root_list)/len(hyper_lem)*100), root_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0b90416",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}