week_4/word-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings: Hands On\n",
    "\n",
    "In previous lecture notebooks you saw all the steps needed to train the CBOW model. This notebook will walk you through how to extract the word embedding vectors from a model.\n",
    "\n",
    "Let's dive into it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from utils2 import get_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before moving on, you will be provided with some variables needed for further procedures, which should be familiar by now. Also a trained CBOW model will be simulated, the corresponding weights and biases are provided: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the tokenized version of the corpus\n",
    "words = ['i', 'am', 'happy', 'because', 'i', 'am', 'learning']\n",
    "\n",
    "# Define V. Remember this is the size of the vocabulary\n",
    "V = 5\n",
    "\n",
    "# Get 'word2Ind' and 'Ind2word' dictionaries for the tokenized corpus\n",
    "word2Ind, Ind2word = get_dict(words)\n",
    "\n",
    "\n",
    "# Define first matrix of weights\n",
    "W1 = np.array([[ 0.41687358,  0.08854191, -0.23495225,  0.28320538,  0.41800106],\n",
    "               [ 0.32735501,  0.22795148, -0.23951958,  0.4117634 , -0.23924344],\n",
    "               [ 0.26637602, -0.23846886, -0.37770863, -0.11399446,  0.34008124]])\n",
    "\n",
    "# Define second matrix of weights\n",
    "W2 = np.array([[-0.22182064, -0.43008631,  0.13310965],\n",
    "               [ 0.08476603,  0.08123194,  0.1772054 ],\n",
    "               [ 0.1871551 , -0.06107263, -0.1790735 ],\n",
    "               [ 0.07055222, -0.02015138,  0.36107434],\n",
    "               [ 0.33480474, -0.39423389, -0.43959196]])\n",
    "\n",
    "# Define first vector of biases\n",
    "b1 = np.array([[ 0.09688219],\n",
    "               [ 0.29239497],\n",
    "               [-0.27364426]])\n",
    "\n",
    "# Define second vector of biases\n",
    "b2 = np.array([[ 0.0352008 ],\n",
    "               [-0.36393384],\n",
    "               [-0.12775555],\n",
    "               [-0.34802326],\n",
    "               [-0.07017815]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Extracting word embedding vectors\n",
    "\n",
    "Once you have finished training the neural network, you have three options to get word embedding vectors for the words of your vocabulary, based on the weight matrices $\\mathbf{W_1}$ and/or $\\mathbf{W_2}$.\n",
    "\n",
    "### Option 1: extract embedding vectors from $\\mathbf{W_1}$\n",
    "\n",
    "The first option is to take the columns of $\\mathbf{W_1}$ as the embedding vectors of the words of the vocabulary, using the same order of the words as for the input and output vectors.\n",
    "\n",
    "> Note: in this practice notebooks the values of the word embedding vectors are meaningless after a single iteration with just one training example, but here's how you would proceed after the training process is complete.\n",
    "\n",
    "For example $\\mathbf{W_1}$ is this matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.41687358,  0.08854191, -0.23495225,  0.28320538,  0.41800106],\n",
       "       [ 0.32735501,  0.22795148, -0.23951958,  0.4117634 , -0.23924344],\n",
       "       [ 0.26637602, -0.23846886, -0.37770863, -0.11399446,  0.34008124]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print W1\n",
    "W1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first column, which is a 3-element vector, is the embedding vector of the first word of your vocabulary. The second column is the word embedding vector for the second word, and so on.\n",
    "\n",
    "The first, second, etc. words are ordered as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "am\n",
      "because\n",
      "happy\n",
      "i\n",
      "learning\n"
     ]
    }
   ],
   "source": [
    "# Print corresponding word for each index within vocabulary's range\n",
    "for i in range(V):\n",
    "    print(Ind2word[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the word embedding vectors corresponding to each word are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "am: [0.41687358 0.32735501 0.26637602]\n",
      "because: [ 0.08854191  0.22795148 -0.23846886]\n",
      "happy: [-0.23495225 -0.23951958 -0.37770863]\n",
      "i: [ 0.28320538  0.4117634  -0.11399446]\n",
      "learning: [ 0.41800106 -0.23924344  0.34008124]\n"
     ]
    }
   ],
   "source": [
    "# Loop through each word of the vocabulary\n",
    "for word in word2Ind:\n",
    "    # Extract the column corresponding to the index of the word in the vocabulary\n",
    "    word_embedding_vector = W1[:, word2Ind[word]]\n",
    "    # Print word alongside word embedding vector\n",
    "    print(f'{word}: {word_embedding_vector}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option 2: extract embedding vectors from $\\mathbf{W_2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second option is to take $\\mathbf{W_2}$ transposed, and take its columns as the word embedding vectors just like you did for $\\mathbf{W_1}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.22182064,  0.08476603,  0.1871551 ,  0.07055222,  0.33480474],\n",
       "       [-0.43008631,  0.08123194, -0.06107263, -0.02015138, -0.39423389],\n",
       "       [ 0.13310965,  0.1772054 , -0.1790735 ,  0.36107434, -0.43959196]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print transposed W2\n",
    "W2.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "am: [-0.22182064 -0.43008631  0.13310965]\n",
      "because: [0.08476603 0.08123194 0.1772054 ]\n",
      "happy: [ 0.1871551  -0.06107263 -0.1790735 ]\n",
      "i: [ 0.07055222 -0.02015138  0.36107434]\n",
      "learning: [ 0.33480474 -0.39423389 -0.43959196]\n"
     ]
    }
   ],
   "source": [
    "# Loop through each word of the vocabulary\n",
    "for word in word2Ind:\n",
    "    # Extract the column corresponding to the index of the word in the vocabulary\n",
    "    word_embedding_vector = W2.T[:, word2Ind[word]]\n",
    "    # Print word alongside word embedding vector\n",
    "    print(f'{word}: {word_embedding_vector}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option 3: extract embedding vectors from $\\mathbf{W_1}$ and $\\mathbf{W_2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The third option, which is the one you will use in this week's assignment, uses the average of $\\mathbf{W_1}$ and $\\mathbf{W_2^\\top}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculate the average of $\\mathbf{W_1}$ and $\\mathbf{W_2^\\top}$, and store the result in `W3`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.09752647,  0.08665397, -0.02389858,  0.1768788 ,  0.3764029 ],\n",
       "       [-0.05136565,  0.15459171, -0.15029611,  0.19580601, -0.31673866],\n",
       "       [ 0.19974284, -0.03063173, -0.27839106,  0.12353994, -0.04975536]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute W3 as the average of W1 and W2 transposed\n",
    "W3 = (W1 + W2.T) / 2\n",
    "\n",
    "# Print W3\n",
    "W3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected output:\n",
    "\n",
    "    array([[ 0.09752647,  0.08665397, -0.02389858,  0.1768788 ,  0.3764029 ],\n",
    "           [-0.05136565,  0.15459171, -0.15029611,  0.19580601, -0.31673866],\n",
    "           [ 0.19974284, -0.03063173, -0.27839106,  0.12353994, -0.04975536]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extracting the word embedding vectors works just like the two previous options, by taking the columns of the matrix you've just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "am: [ 0.09752647 -0.05136565  0.19974284]\n",
      "because: [ 0.08665397  0.15459171 -0.03063173]\n",
      "happy: [-0.02389858 -0.15029611 -0.27839106]\n",
      "i: [0.1768788  0.19580601 0.12353994]\n",
      "learning: [ 0.3764029  -0.31673866 -0.04975536]\n"
     ]
    }
   ],
   "source": [
    "# Loop through each word of the vocabulary\n",
    "for word in word2Ind:\n",
    "    # Extract the column corresponding to the index of the word in the vocabulary\n",
    "    word_embedding_vector = W3[:, word2Ind[word]]\n",
    "    # Print word alongside word embedding vector\n",
    "    print(f'{word}: {word_embedding_vector}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you know 3 different options to get the word embedding vectors from a model! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How this practice relates to and differs from the upcoming graded assignment\n",
    "\n",
    "- After extracting the word embedding vectors, you will use principal component analysis (PCA) to visualize the vectors, which will enable you to perform an intrinsic evaluation of the quality of the vectors, as explained in the lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations on finishing all lecture notebooks for this week!** \n",
    "\n",
    "You're now ready to take on this week's assignment!\n",
    "\n",
    "**Keep it up!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}