07. Text Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topics\n",
    "#### Task\n",
    "\n",
    "1. Tokenization  \n",
    "2. Stopword Removal  \n",
    "3. N- Grams  \n",
    "4. Stemming  \n",
    "5. Word Sense Disambiguation  \n",
    "6. Count Vectorizer\n",
    "7. TF-IDF(TfidfVectorizer)\n",
    "8. HashingVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Tokenization\n",
    "Task of breaking a text into pieces called as token\n",
    "\n",
    "<img src=\"Image/token.png\" width=\"400\" />\n",
    "\n",
    "- Word Tokenization\n",
    "- Sentence Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I believe this would help the reader understand how tokenization works.', 'as well as realize its importance.']\n",
      "['I', 'believe', 'this', 'would', 'help', 'the', 'reader', 'understand', 'how', 'tokenization', 'works', '.', 'as', 'well', 'as', 'realize', 'its', 'importance', '.']\n",
      "[['I', 'believe', 'this', 'would', 'help', 'the', 'reader', 'understand', 'how', 'tokenization', 'works', '.'], ['as', 'well', 'as', 'realize', 'its', 'importance', '.']]\n"
     ]
    }
   ],
   "source": [
    "#Tokenization\n",
    "import nltk\n",
    "#nltk.download('punkt')\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "\n",
    "text = \"I believe this would help the reader understand how tokenization works. as well as realize its importance.\"\n",
    "sents = (sent_tokenize(text)) \n",
    "print(sents)\n",
    "print(word_tokenize(text))\n",
    "words = [word_tokenize(sent) for sent in sents]\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Stopword Removal\n",
    "- Stopwords are the English words which does not add much meaning to a sentence. \n",
    "- They can safely be ignored without sacrificing the meaning of the sentence.  \n",
    "A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore.\n",
    "<img src=\"Image/stop1.jpg\" width=\"500\" />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n",
      "['I', 'believe', 'would', 'help', 'reader', 'understand', 'tokenization', 'works', 'well', 'realize', 'importance', 'text']\n"
     ]
    }
   ],
   "source": [
    "#Removing Stopwords\n",
    "#Import libraries\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "from string import punctuation\n",
    "\n",
    "text = \"I believe this would help the reader understand how tokenization \\\n",
    "        works. as well as realize its importance (text) .\"\n",
    "        \n",
    "print(punctuation)\n",
    "\n",
    "custom_list = set(stopwords.words('english')+list(punctuation))\n",
    "\n",
    "word_list = [word for word in word_tokenize(text) if word not in custom_list]\n",
    "print(word_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. N-Grams\n",
    "An n-gram is a contiguous sequence of n items from a given sample of text or speech.\n",
    "\n",
    "<img src=\"Image/n-grams.jpg\" width=\"300\" />\n",
    "\n",
    "- While typing we get suggestion\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_items([(('I', 'believe'), 1), (('believe', 'would'), 1), (('would', 'help'), 1), (('help', 'reader'), 1), (('reader', 'understand'), 1), (('understand', 'tokenization'), 1), (('tokenization', 'works'), 1), (('works', 'well'), 1), (('well', 'realize'), 1), (('realize', 'importance'), 1), (('importance', 'text'), 1)])\n"
     ]
    }
   ],
   "source": [
    "#N-grams\n",
    "from nltk.collocations import BigramCollocationFinder\n",
    "\n",
    "word_list = ['I', 'believe', 'would', 'help', 'reader', 'understand', \\\n",
    "             'tokenization', 'works', 'well', 'realize', 'importance', 'text']\n",
    "\n",
    "finde = BigramCollocationFinder.from_words(word_list)\n",
    "print(finde.ngram_fd.items())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Stemming\n",
    "Stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base or root form\n",
    "\n",
    "<img src=\"Image/stem.jpg\" width=\"300\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['it', 'is', 'import', 'to', 'by', 'very', 'python', 'whil', 'you', 'ar', 'python', 'with', 'python', '.', 'al', 'python', 'hav', 'python', 'poor', 'at', 'least', 'ont', '.']\n"
     ]
    }
   ],
   "source": [
    "#Stemming\n",
    "#Import Libraries\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.stem.lancaster import LancasterStemmer\n",
    "\n",
    "l_s = LancasterStemmer()\n",
    "new_text = \"It is important to by very pythonly while you are pythoning\\\n",
    "             with python. All pythoners have pythoned poorly at least once.\"\n",
    "             \n",
    "stem_lan =  [l_s.stem(word) for word in word_tokenize(new_text)] \n",
    "print(stem_lan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('It', 'PRP'),\n",
       " ('is', 'VBZ'),\n",
       " ('important', 'JJ'),\n",
       " ('to', 'TO'),\n",
       " ('by', 'IN'),\n",
       " ('very', 'RB'),\n",
       " ('pythonly', 'RB'),\n",
       " ('while', 'IN'),\n",
       " ('you', 'PRP'),\n",
       " ('are', 'VBP'),\n",
       " ('pythoning', 'VBG'),\n",
       " ('with', 'IN'),\n",
       " ('python', 'NN'),\n",
       " ('.', '.'),\n",
       " ('All', 'DT'),\n",
       " ('pythoners', 'NNS'),\n",
       " ('have', 'VBP'),\n",
       " ('pythoned', 'VBN'),\n",
       " ('poorly', 'RB'),\n",
       " ('at', 'IN'),\n",
       " ('least', 'JJS'),\n",
       " ('once', 'RB'),\n",
       " ('.', '.')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Part of Speech\n",
    "#nltk.download('averaged_perceptron_tagger')\n",
    "nltk.pos_tag(word_tokenize(new_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Word Sense Disambiguation\n",
    "WSD is identifying which sense of a word (i.e. meaning) is used in a sentence, when the word has multiple meanings. \n",
    "<img src=\"Image/mouse.png\" width=\"300\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Synset('mouse.n.01') any of numerous small rodents typically resembling diminutive rats having pointed snouts and small ears on elongated bodies with slender usually hairless tails\n",
      "Synset('shiner.n.01') a swollen bruise caused by a blow to the eye\n",
      "Synset('mouse.n.03') person who is quiet or timid\n",
      "Synset('mouse.n.04') a hand-operated electronic device that controls the coordinates of a cursor on your computer screen as you move it around on a pad; on the bottom of the device is a ball that rolls on the surface of the pad\n",
      "Synset('sneak.v.01') to go stealthily or furtively\n",
      "Synset('mouse.v.02') manipulate the mouse of a computer\n"
     ]
    }
   ],
   "source": [
    "#Word Sense Disambiguation\n",
    "#Import Libraries\n",
    "#nltk.download('wordnet')\n",
    "from nltk.corpus import wordnet\n",
    "for ss in wordnet.synsets('mouse'):\n",
    "    print(ss, ss.definition())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Synset('bass.n.07') the member with the lowest range of a family of musical instruments\n",
      "Synset('sea_bass.n.01') the lean flesh of a saltwater fish of the family Serranidae\n",
      "Synset('mouse.n.04') a hand-operated electronic device that controls the coordinates of a cursor on your computer screen as you move it around on a pad; on the bottom of the device is a ball that rolls on the surface of the pad\n"
     ]
    }
   ],
   "source": [
    "from nltk.wsd import lesk\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "context_1 = lesk(word_tokenize(\"Sing in a lower tone, along with the bass\"), \"bass\")\n",
    "print(context_1, context_1.definition())\n",
    "\n",
    "context_2 = lesk(word_tokenize(\"The sea bass really very hard to catch\"), \"bass\")\n",
    "print(context_2, context_2.definition())\n",
    "\n",
    "context_3 = lesk(word_tokenize(\"My mouse is not working, need to change it\"), \"mouse\")\n",
    "print(context_3, context_3.definition())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. CountVectorizer\n",
    "- provides a simple way to both tokenize a collection of text documents and build a vocabulary of known words, but also to encode new documents using that vocabulary.\n",
    "- the same vectorizer can be used on documents that contain words not included in the vocabulary. These words are ignored and no count is given in the resulting vector.\n",
    "- Issue: Appearance of “the”\n",
    "- Each column represents one word, count refers to frequency of the word\n",
    "- Sequence of words are not maintained"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>This is the first document from heaven</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>but the second document is from mars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>And this is the third one from nowhere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Is this the first document from nowhere?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       Text\n",
       "0    This is the first document from heaven\n",
       "1      but the second document is from mars\n",
       "2    And this is the third one from nowhere\n",
       "3  Is this the first document from nowhere?"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "corpus = [\n",
    "     'This is the first document from heaven',\n",
    "     'but the second document is from mars',\n",
    "     'And this is the third one from nowhere',\n",
    "     'Is this the first document from nowhere?',\n",
    "]\n",
    "df = pd.DataFrame({'Text':corpus})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['and', 'but', 'document', 'first', 'from', 'heaven', 'is', 'mars', 'nowhere', 'one', 'second', 'the', 'third', 'this']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "count_v = CountVectorizer()\n",
    "X = count_v.fit_transform(df.Text).toarray()\n",
    "print(count_v.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 1 1 1 1 1 0 0 0 0 1 0 1]\n",
      " [0 1 1 0 1 0 1 1 0 0 1 1 0 0]\n",
      " [1 0 0 0 1 0 1 0 1 1 0 1 1 1]\n",
      " [0 0 1 1 1 0 1 0 1 0 0 1 0 1]]\n",
      "{'this': 13, 'is': 6, 'the': 11, 'first': 3, 'document': 2, 'from': 4, 'heaven': 5, 'but': 1, 'second': 10, 'mars': 7, 'and': 0, 'third': 12, 'one': 9, 'nowhere': 8}\n"
     ]
    }
   ],
   "source": [
    "print(X)\n",
    "print(count_v.vocabulary_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 1 1 1 1 0 0 0 0 0 1]\n",
      " [0 1 1 0 1 0 1 0 0 1 0 0]\n",
      " [1 0 0 0 1 0 0 1 1 0 1 1]\n",
      " [0 0 1 1 1 0 0 1 0 0 0 1]]\n",
      "{'this': 11, 'first': 3, 'document': 2, 'from': 4, 'heaven': 5, 'but': 1, 'second': 9, 'mars': 6, 'and': 0, 'third': 10, 'one': 8, 'nowhere': 7}\n"
     ]
    }
   ],
   "source": [
    "#Removing Stopwords\n",
    "count_v = CountVectorizer(stop_words=['the','is'])\n",
    "print(count_v.fit_transform(df.Text).toarray())\n",
    "print(count_v.vocabulary_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0],\n",
       "       [0, 1, 0],\n",
       "       [0, 0, 1],\n",
       "       [0, 0, 1]], dtype=int64)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_v = CountVectorizer(vocabulary=['heaven','mars','nowhere'])\n",
    "count_v.fit_transform(df.Text).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 0 0 1 1 0 1 1 1 1 0 0 1 1 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 1 1 0]\n",
      " [0 0 1 1 1 0 1 0 0 1 0 1 0 0 1 1 0 0 1 0 0 0 1 1 1 0 1 0 0 0 0 0 0]\n",
      " [1 1 0 0 0 0 0 0 0 1 0 0 1 0 1 0 1 0 0 1 1 1 0 0 1 0 0 1 1 1 1 1 0]\n",
      " [0 0 0 0 1 1 0 1 1 1 0 0 1 0 1 0 0 1 0 1 0 0 0 0 1 1 0 0 0 0 1 0 1]]\n",
      "{'this': 30, 'is': 14, 'the': 24, 'first': 7, 'document': 4, 'from': 9, 'heaven': 13, 'this is': 31, 'is the': 16, 'the first': 25, 'first document': 8, 'document from': 5, 'from heaven': 10, 'but': 2, 'second': 22, 'mars': 18, 'but the': 3, 'the second': 26, 'second document': 23, 'document is': 6, 'is from': 15, 'from mars': 11, 'and': 0, 'third': 28, 'one': 20, 'nowhere': 19, 'and this': 1, 'the third': 27, 'third one': 29, 'one from': 21, 'from nowhere': 12, 'is this': 17, 'this the': 32}\n"
     ]
    }
   ],
   "source": [
    "#n-grams\n",
    "count_v = CountVectorizer(ngram_range=[1,2])\n",
    "print(count_v.fit_transform(df.Text).toarray())\n",
    "print(count_v.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. TF-IDF (TfidfVectorizer)\n",
    "TF-IDF are word frequency scores that try to highlight words that are more interesting, e.g. frequent in a document but not across documents.\n",
    "- The importance is in scale of 0 & 1\n",
    "\n",
    "<b>Term Frequency:</b> This summarizes how often a given word appears within a document.  \n",
    "<b>Inverse Document Frequency:</b> This downscales words that appear a lot across documents.\n",
    "  \n",
    "Adv:  \n",
    "- Feature vector much more tractable in size  \n",
    "- Frequency and relevance captured  \n",
    "\n",
    "DisAdv:  \n",
    "- Context still not captured  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'this': 13, 'is': 6, 'the': 11, 'first': 3, 'document': 2, 'from': 4, 'heaven': 5, 'but': 1, 'second': 10, 'mars': 7, 'and': 0, 'third': 12, 'one': 9, 'nowhere': 8}\n",
      "[1.91629073 1.91629073 1.22314355 1.51082562 1.         1.91629073\n",
      " 1.         1.91629073 1.51082562 1.91629073 1.91629073 1.\n",
      " 1.91629073 1.22314355]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import pandas as pd\n",
    "corpus = [\n",
    "     'This is the first document from heaven',\n",
    "     'but the second document is from mars',\n",
    "     'And this is the third one from nowhere',\n",
    "     'Is this the first document from nowhere?',\n",
    "]\n",
    "\n",
    "vectorizer = TfidfVectorizer()\n",
    "vectorizer.fit(corpus)\n",
    "print(vectorizer.vocabulary_)\n",
    "print(vectorizer.idf_)\n",
    "#df = pd.DataFrame({'Text':corpus})\n",
    "#tfidf_v = TfidfVectorizer(stop_words='english')\n",
    "#tfidf_v.fit_transform(df.Text).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 14)\n",
      "[[0.         0.         0.35387458 0.43710551 0.28931566 0.55441292\n",
      "  0.28931566 0.         0.         0.         0.         0.28931566\n",
      "  0.         0.35387458]]\n"
     ]
    }
   ],
   "source": [
    "#tfidf_v.get_feature_names()\n",
    "# encode document\n",
    "vector = vectorizer.transform([corpus[0]])\n",
    "# summarize encoded vector\n",
    "print(vector.shape)\n",
    "print(vector.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. HashingVectorizer\n",
    "- Issue with Counts and frequencies – vocabulary can become very large\n",
    "- Work around is to use a one way hash of words to convert them to integers\n",
    "- No vocabulary is required and you can choose an arbitrary-long fixed length vector\n",
    "- downside - no way to convert the encoding back to a word\n",
    "\n",
    "Step1:\n",
    "<img src=\"Image/hash11.JPG\" width=\"400\" />\n",
    "Step2:\n",
    "<img src=\"Image/hash12.JPG\" width=\"400\" />\n",
    "Step3:\n",
    "<img src=\"Image/hash13.JPG\" width=\"400\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2., 1., 0., 0., 1., 1., 2., 0.],\n",
       "       [2., 0., 0., 1., 1., 1., 2., 0.],\n",
       "       [0., 0., 0., 0., 2., 3., 3., 0.],\n",
       "       [2., 0., 0., 0., 1., 1., 3., 0.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "import pandas as pd\n",
    "corpus = [\n",
    "     'This is the first document from heaven',\n",
    "     'but the second document is from mars',\n",
    "     'And this is the third one from nowhere',\n",
    "     'Is this the first document from nowhere?',\n",
    "]\n",
    "df = pd.DataFrame({'Text':corpus})\n",
    "hash_v = HashingVectorizer(n_features=8,norm=None,alternate_sign=False)\n",
    "hash_v.fit_transform(df.Text).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}