summarizer_amazon_reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-tk5ROC6mw0f"
   },
   "source": [
    "# Text Summarization of Amazon reviews\n",
    "\n",
    "In this notebook I will write summaries with the help of my Seq2Seq model in Summarizer.py.\n",
    "\n",
    "The model works impressively well in the end!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "B_ULQF2smw0h"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "from collections import Counter\n",
    "\n",
    "import Summarizer\n",
    "import summarizer_data_utils\n",
    "import summarizer_model_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.8.0\n"
     ]
    }
   ],
   "source": [
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-XBIbvs4mw0m"
   },
   "source": [
    "## The data\n",
    "\n",
    "\n",
    "The data we will be using with is a dataset from Kaggle, the Amazon Fine Food Reviews dataset.  \n",
    "It contains, as the name suggests, 570.000 reviews of fine foods from Amazon and summaries of those reviews. \n",
    "Our aim is to input a review (Text column) and automatically create a summary (Summary colum) for it.\n",
    "\n",
    "\n",
    "https://www.kaggle.com/snap/amazon-fine-food-reviews/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8NYVncFKmw0n"
   },
   "source": [
    "### Reading and exploring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4894,
     "status": "ok",
     "timestamp": 1526227108183,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "2OiSxpApmw0o",
    "outputId": "98a255ad-248e-4f1e-b43f-1c8d384fd453"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(568454, 10)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load csv file using pandas.\n",
    "file_path = './Reviews.csv'\n",
    "data = pd.read_csv(file_path)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "rElWMbT2mw0t",
    "outputId": "08f4b9ee-8c78-4f3b-c986-f7f3dd2ff248"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>ProductId</th>\n",
       "      <th>UserId</th>\n",
       "      <th>ProfileName</th>\n",
       "      <th>HelpfulnessNumerator</th>\n",
       "      <th>HelpfulnessDenominator</th>\n",
       "      <th>Score</th>\n",
       "      <th>Time</th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>B001E4KFG0</td>\n",
       "      <td>A3SGXH7AUHU8GW</td>\n",
       "      <td>delmartian</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1303862400</td>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>B00813GRG4</td>\n",
       "      <td>A1D87F6ZCVE5NK</td>\n",
       "      <td>dll pa</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1346976000</td>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>B000LQOCH0</td>\n",
       "      <td>ABXLMWJIXXAIN</td>\n",
       "      <td>Natalia Corres \"Natalia Corres\"</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1219017600</td>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>B000UA0QIQ</td>\n",
       "      <td>A395BORC6FGVXV</td>\n",
       "      <td>Karl</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1307923200</td>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>B006K2ZZ7K</td>\n",
       "      <td>A1UQRSCLF8GW1T</td>\n",
       "      <td>Michael D. Bigham \"M. Wassir\"</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1350777600</td>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id   ProductId          UserId                      ProfileName  \\\n",
       "0   1  B001E4KFG0  A3SGXH7AUHU8GW                       delmartian   \n",
       "1   2  B00813GRG4  A1D87F6ZCVE5NK                           dll pa   \n",
       "2   3  B000LQOCH0   ABXLMWJIXXAIN  Natalia Corres \"Natalia Corres\"   \n",
       "3   4  B000UA0QIQ  A395BORC6FGVXV                             Karl   \n",
       "4   5  B006K2ZZ7K  A1UQRSCLF8GW1T    Michael D. Bigham \"M. Wassir\"   \n",
       "\n",
       "   HelpfulnessNumerator  HelpfulnessDenominator  Score        Time  \\\n",
       "0                     1                       1      5  1303862400   \n",
       "1                     0                       0      1  1346976000   \n",
       "2                     1                       1      4  1219017600   \n",
       "3                     3                       3      2  1307923200   \n",
       "4                     0                       0      5  1350777600   \n",
       "\n",
       "                 Summary                                               Text  \n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...  \n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...  \n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...  \n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...  \n",
       "4            Great taffy  Great taffy at a great price.  There was a wid...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we will only use the last two columns Summary (target) and Text (input).\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1055,
     "status": "ok",
     "timestamp": 1526227113630,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "N9bHztjpmw0x",
    "outputId": "aa4ce93d-efde-4ebb-bc3e-7b03d477a5e4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                         0\n",
       "ProductId                  0\n",
       "UserId                     0\n",
       "ProfileName               16\n",
       "HelpfulnessNumerator       0\n",
       "HelpfulnessDenominator     0\n",
       "Score                      0\n",
       "Time                       0\n",
       "Summary                   26\n",
       "Text                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check for missings --> got some in summary drop those. \n",
    "# 26 are missing, so we will drop those!\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "boMCgsgTmw00"
   },
   "outputs": [],
   "source": [
    "# drop row, if values in Summary is missing. \n",
    "data.dropna(subset=['Summary'],inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 734,
     "status": "ok",
     "timestamp": 1526227125421,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "ESv4XLgQmw03",
    "outputId": "0ca5d3f7-7efc-46dc-bf8e-bb5803c6376c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Summary                                               Text\n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...\n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...\n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...\n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...\n",
       "4            Great taffy  Great taffy at a great price.  There was a wid..."
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# only summary and text are useful for us.\n",
    "data = data[['Summary', 'Text']]\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "BjmIGbXtmw08"
   },
   "outputs": [],
   "source": [
    "# we will not use all of them, only short ones and ones of similar size. \n",
    "# choosing the ones that are of similar length makes it easier for the model to learn.\n",
    "raw_texts = []\n",
    "raw_summaries = []\n",
    "\n",
    "for text, summary in zip(data.Text, data.Summary):\n",
    "    if 100< len(text) < 150:\n",
    "        raw_texts.append(text)\n",
    "        raw_summaries.append(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1560,
     "status": "ok",
     "timestamp": 1526227148045,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "t5JBoyqKmw0_",
    "outputId": "150af52c-a0af-4eec-f0ff-399ec33e434d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(78863, 78863)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(raw_texts), len(raw_summaries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "tMsDeec4mw1F",
    "outputId": "de46147e-d2c4-40e5-9a0a-22f27ac360fd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text:\n",
      " Great taffy at a great price.  There was a wide assortment of yummy taffy.  Delivery was very quick.  If your a taffy lover, this is a deal.\n",
      "Summary:\n",
      " Great taffy \n",
      "\n",
      "\n",
      "Text:\n",
      " This taffy is so good.  It is very soft and chewy.  The flavors are amazing.  I would definitely recommend you buying it.  Very satisfying!!\n",
      "Summary:\n",
      " Wonderful, tasty taffy \n",
      "\n",
      "\n",
      "Text:\n",
      " Right now I'm mostly just sprouting this so my cats can eat the grass. They love it. I rotate it around with Wheatgrass and Rye too\n",
      "Summary:\n",
      " Yay Barley \n",
      "\n",
      "\n",
      "Text:\n",
      " This is a very healthy dog food. Good for their digestion. Also good for small puppies. My dog eats her required amount at every feeding.\n",
      "Summary:\n",
      " Healthy Dog Food \n",
      "\n",
      "\n",
      "Text:\n",
      " The Strawberry Twizzlers are my guilty pleasure - yummy. Six pounds will be around for a while with my son and I.\n",
      "Summary:\n",
      " Strawberry Twizzlers - Yummy \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for t, s in zip(raw_texts[:5], raw_summaries[:5]):\n",
    "    print('Text:\\n', t)\n",
    "    print('Summary:\\n', s, '\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y-CyKX1gmw1J"
   },
   "source": [
    "### Clean and prepare the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 47800,
     "status": "ok",
     "timestamp": 1526227313932,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "4CLoTqyzmw1K",
    "outputId": "2fcb8327-e714-48e2-fca1-6bf39b8e6411"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing Time:  36.91733002662659\n"
     ]
    }
   ],
   "source": [
    "# the function gives us the option to keep_most of the characters inisde the texts and summaries, meaning\n",
    "# punctuation, question marks, slashes...\n",
    "# or we can set it to False, meaning we only want to keep letters and numbers like here.\n",
    "processed_texts, processed_summaries, words_counted = summarizer_data_utils.preprocess_texts_and_summaries(\n",
    "    raw_texts,\n",
    "    raw_summaries,\n",
    "    keep_most=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "yquphHYJmw1R",
    "outputId": "cd9ad917-33da-4294-eb41-f3b0abb967e9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text\n",
      ": ['great', 'taffy', 'at', 'a', 'great', 'price', 'there', 'was', 'a', 'wide', 'assortment', 'of', 'yummy', 'taffy', 'delivery', 'was', 'very', 'quick', 'if', 'your', 'a', 'taffy', 'lover', 'this', 'is', 'a', 'deal'] \n",
      "\n",
      "Summary:\n",
      " ['great', 'taffy'] \n",
      "\n",
      "\n",
      "\n",
      "Text\n",
      ": ['this', 'taffy', 'is', 'so', 'good', 'it', 'is', 'very', 'soft', 'and', 'chewy', 'the', 'flavors', 'are', 'amazing', 'i', 'would', 'definitely', 'recommend', 'you', 'buying', 'it', 'very', 'satisfying'] \n",
      "\n",
      "Summary:\n",
      " ['wonderful', 'tasty', 'taffy'] \n",
      "\n",
      "\n",
      "\n",
      "Text\n",
      ": ['right', 'now', 'i', 'm', 'mostly', 'just', 'sprouting', 'this', 'so', 'my', 'cats', 'can', 'eat', 'the', 'grass', 'they', 'love', 'it', 'i', 'rotate', 'it', 'around', 'with', 'wheatgrass', 'and', 'rye', 'too'] \n",
      "\n",
      "Summary:\n",
      " ['yay', 'barley'] \n",
      "\n",
      "\n",
      "\n",
      "Text\n",
      ": ['this', 'is', 'a', 'very', 'healthy', 'dog', 'food', 'good', 'for', 'their', 'digestion', 'also', 'good', 'for', 'small', 'puppies', 'my', 'dog', 'eats', 'her', 'required', 'amount', 'at', 'every', 'feeding'] \n",
      "\n",
      "Summary:\n",
      " ['healthy', 'dog', 'food'] \n",
      "\n",
      "\n",
      "\n",
      "Text\n",
      ": ['the', 'strawberry', 'twizzlers', 'are', 'my', 'guilty', 'pleasure', 'yummy', 'six', 'pounds', 'will', 'be', 'around', 'for', 'a', 'while', 'with', 'my', 'son', 'and', 'i'] \n",
      "\n",
      "Summary:\n",
      " ['strawberry', 'twizzlers', 'yummy'] \n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for t,s in zip(processed_texts[:5], processed_summaries[:5]):\n",
    "    print('Text\\n:', t, '\\n')\n",
    "    print('Summary:\\n', s, '\\n\\n\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "34eUnqVQmw1c"
   },
   "source": [
    "### Create lookup dicts\n",
    "\n",
    "We cannot feed our network actual words, but numbers. So we first have to create our lookup dicts, where each words gets and int value (high or low, depending on its frequency in our corpus). Those help us to later convert the texts into numbers.\n",
    "\n",
    "We also add special tokens. EndOfSentence and StartOfSentence are crucial for the Seq2Seq model we later use.\n",
    "Pad token, because all summaries and texts in a batch need to have the same length, pad token helps us do that.\n",
    "\n",
    "So we need 2 lookup dicts:\n",
    " - From word to index \n",
    " - from index to word. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 716,
     "status": "ok",
     "timestamp": 1526227336251,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "zwqbTP8jmw1d",
    "outputId": "3788ccc5-bec0-4d32-8885-0698b45d7164"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25067 25067 0\n"
     ]
    }
   ],
   "source": [
    "specials = [\"<EOS>\", \"<SOS>\",\"<PAD>\",\"<UNK>\"]\n",
    "word2ind, ind2word,  missing_words = summarizer_data_utils.create_word_inds_dicts(words_counted,\n",
    "                                                                       specials = specials)\n",
    "print(len(word2ind), len(ind2word), len(missing_words))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EVZ1Qmk9mw1j"
   },
   "source": [
    "### Pretrained embeddings\n",
    "\n",
    "Optionally we can use pretrained word embeddings. Those have proved to increase training speed and accuracy.\n",
    "Here I used two different options. Either we use glove embeddings or embeddings from tf_hub.\n",
    "The ones from tf_hub worked better, so we use those. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# glove_embeddings_path = './glove.6B.300d.txt'\n",
    "# embedding_matrix_save_path = './embeddings/my_embedding_github.npy'\n",
    "# emb = summarizer_data_utils.create_and_save_embedding_matrix(word2ind,\n",
    "#                                                              glove_embeddings_path,\n",
    "#                                                              embedding_matrix_save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 61662,
     "status": "ok",
     "timestamp": 1526227413054,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "ObE6ggfAmw1o",
    "outputId": "e12b1f22-0fad-4a3d-e934-5fc480b83788"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using /tmp/tfhub_modules to cache modules.\n",
      "INFO:tensorflow:Downloading TF-Hub Module 'https://tfhub.dev/google/Wiki-words-250/1'.\n",
      "INFO:tensorflow:Downloaded TF-Hub Module 'https://tfhub.dev/google/Wiki-words-250/1'.\n",
      "INFO:tensorflow:Initialize variable module/embeddings/part_0:0,module/embeddings/part_1:0 from checkpoint b'/tmp/tfhub_modules/1e7e72950396d90315d3d9c57eddf5da44d4dca2/variables/variables' with embeddings\n"
     ]
    }
   ],
   "source": [
    "# the embeddings from tf_hub. \n",
    "# embed = hub.Module(\"https://tfhub.dev/google/nnlm-en-dim128/1\")\n",
    "embed = hub.Module(\"https://tfhub.dev/google/Wiki-words-250/1\")\n",
    "emb = embed([key for key in word2ind.keys()])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    sess.run(tf.tables_initializer())\n",
    "    embedding = sess.run(emb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 687,
     "status": "ok",
     "timestamp": 1526227413774,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "ayXi9D7Umw1u",
    "outputId": "7b5b5522-8c21-4c70-a5be-46f6ed2cdbd2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25067, 250)"
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "QoGa9EWdmw11"
   },
   "outputs": [],
   "source": [
    "np.save('./tf_hub_embedding.npy', embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QV1HB3zzmw12"
   },
   "source": [
    "### Convert text and summaries\n",
    "\n",
    "As I said before we cannot feed the words directly to our network, we have to convert them to numbers first of all. This is what we do here. And we also append the SOS and EOS tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "NjudfxFPmw13"
   },
   "outputs": [],
   "source": [
    "# converts words in texts and summaries to indices\n",
    "# it looks like we have to set eos here to False\n",
    "converted_texts, unknown_words_in_texts = summarizer_data_utils.convert_to_inds(processed_texts,\n",
    "                                                                                word2ind,\n",
    "                                                                                eos = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "1dFsLoAqmw16"
   },
   "outputs": [],
   "source": [
    "converted_summaries, unknown_words_in_summaries = summarizer_data_utils.convert_to_inds(processed_summaries,\n",
    "                                                                                        word2ind,\n",
    "                                                                                        eos = True,\n",
    "                                                                                        sos = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 476
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2143,
     "status": "ok",
     "timestamp": 1526227545460,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "ghATcyE4mw2A",
    "outputId": "06d3f934-7c97-49e2-c358-21bd7552689d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[12,\n",
       " 1727,\n",
       " 47,\n",
       " 8,\n",
       " 12,\n",
       " 45,\n",
       " 130,\n",
       " 29,\n",
       " 8,\n",
       " 2728,\n",
       " 1159,\n",
       " 15,\n",
       " 106,\n",
       " 1727,\n",
       " 322,\n",
       " 29,\n",
       " 25,\n",
       " 249,\n",
       " 62,\n",
       " 101,\n",
       " 8,\n",
       " 1727,\n",
       " 662,\n",
       " 9,\n",
       " 10,\n",
       " 8,\n",
       " 193]"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "converted_texts[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1124,
     "status": "ok",
     "timestamp": 1526227550694,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "pSTzMURHmw2E",
    "outputId": "4dc8b405-f811-49f2-b7b3-83495dbf7640"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['great', 'taffy', 'at', 'a', 'great', 'price', 'there', 'was', 'a', 'wide', 'assortment', 'of', 'yummy', 'taffy', 'delivery', 'was', 'very', 'quick', 'if', 'your', 'a', 'taffy', 'lover', 'this', 'is', 'a', 'deal'] ['<SOS>', 'great', 'taffy', '<EOS>']\n"
     ]
    }
   ],
   "source": [
    "# seems to have worked well. \n",
    "print( summarizer_data_utils.convert_inds_to_text(converted_texts[0], ind2word),\n",
    "       summarizer_data_utils.convert_inds_to_text(converted_summaries[0], ind2word))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a8b9Nd0zmw2H"
   },
   "source": [
    "## The model\n",
    "\n",
    "Now we can build and train our model. First we define the hyperparameters we want to use. Then we create our Summarizer and call the function .build_graph(), which as the name suggests, builds the computation graph. \n",
    "Then we can train the model using .train()\n",
    "\n",
    "After training we can try our model using .infer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L2z9xOKzmw2I"
   },
   "source": [
    "### Training\n",
    "\n",
    "We can optionally use a cyclic learning rate, which we do here. \n",
    "I trained the model for 20 epochs and the loss was low then, but we could train it longer and would probably get better results.\n",
    "\n",
    "Unfortunately I do not have the resources to find the perfect (or right) hyperparameters, but these do pretty well. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "tEItjpP4mw2J"
   },
   "outputs": [],
   "source": [
    "# model hyperparametes\n",
    "num_layers_encoder = 4\n",
    "num_layers_decoder = 4\n",
    "rnn_size_encoder = 512\n",
    "rnn_size_decoder = 512\n",
    "\n",
    "batch_size = 256\n",
    "epochs = 200\n",
    "clip = 5\n",
    "keep_probability = 0.5\n",
    "learning_rate = 0.0005\n",
    "max_lr=0.005\n",
    "learning_rate_decay_steps = 700\n",
    "learning_rate_decay = 0.90\n",
    "\n",
    "\n",
    "pretrained_embeddings_path = './tf_hub_embedding.npy'\n",
    "summary_dir = os.path.join('./tensorboard', str('Nn_' + str(rnn_size_encoder) + '_Lr_' + str(learning_rate)))\n",
    "\n",
    "\n",
    "use_cyclic_lr = True\n",
    "inference_targets=True\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1464,
     "status": "ok",
     "timestamp": 1526234914336,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "u8lJ_OI5mw2Q",
    "outputId": "1c06bc51-01eb-4a68-b4ca-38d56a4a2a76"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "78862"
      ]
     },
     "execution_count": 59,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(converted_summaries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1026,
     "status": "ok",
     "timestamp": 1526234915582,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "w_VDuiHyQK84",
    "outputId": "9bc17a2e-837b-41bd-a40d-0f116e143d8f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70976"
      ]
     },
     "execution_count": 60,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "round(78862*0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 85881
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8531236,
     "status": "error",
     "timestamp": 1526243447242,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "E0BX6Z7Kmw2T",
    "outputId": "e734411e-2fbc-4960-e2aa-fb98108c4576"
   },
   "outputs": [],
   "source": [
    "# build graph and train the model \n",
    "summarizer_model_utils.reset_graph()\n",
    "summarizer = Summarizer.Summarizer(word2ind,\n",
    "                                   ind2word,\n",
    "                                   save_path='./models/amazon/my_model',\n",
    "                                   mode='TRAIN',\n",
    "                                   num_layers_encoder = num_layers_encoder,\n",
    "                                   num_layers_decoder = num_layers_decoder,\n",
    "                                   rnn_size_encoder = rnn_size_encoder,\n",
    "                                   rnn_size_decoder = rnn_size_decoder,\n",
    "                                   batch_size = batch_size,\n",
    "                                   clip = clip,\n",
    "                                   keep_probability = keep_probability,\n",
    "                                   learning_rate = learning_rate,\n",
    "                                   max_lr=max_lr,\n",
    "                                   learning_rate_decay_steps = learning_rate_decay_steps,\n",
    "                                   learning_rate_decay = learning_rate_decay,\n",
    "                                   epochs = epochs,\n",
    "                                   pretrained_embeddings_path = pretrained_embeddings_path,\n",
    "                                   use_cyclic_lr = use_cyclic_lr,\n",
    "                                   summary_dir = summary_dir)           \n",
    "\n",
    "summarizer.build_graph()\n",
    "summarizer.train(converted_texts[:70976], \n",
    "                 converted_summaries[:70976],\n",
    "                 validation_inputs=converted_texts[70976:],\n",
    "                 validation_targets=converted_summaries[70976:])\n",
    "\n",
    "\n",
    "# hidden training output.\n",
    "# both train and validation loss decrease nicely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U5Hqzvocmw2W"
   },
   "source": [
    "### Inference\n",
    "Now we can use our trained model to create summaries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4607,
     "status": "ok",
     "timestamp": 1526243454761,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "ljN9a1hemw2Y",
    "outputId": "f60102af-44f0-4c45-8ba3-2548e5af0a4c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded pretrained embeddings.\n",
      "Graph built.\n",
      "INFO:tensorflow:Restoring parameters from ./models/amazon/my_model\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "summarizer_model_utils.reset_graph()\n",
    "summarizer = Summarizer.Summarizer(word2ind,\n",
    "                                   ind2word,\n",
    "                                   './models/amazon/my_model',\n",
    "                                   'INFER',\n",
    "                                   num_layers_encoder = num_layers_encoder,\n",
    "                                   num_layers_decoder = num_layers_decoder,\n",
    "                                   batch_size = len(converted_texts[:50]),\n",
    "                                   clip = clip,\n",
    "                                   keep_probability = 1.0,\n",
    "                                   learning_rate = 0.0,\n",
    "                                   beam_width = 5,\n",
    "                                   rnn_size_encoder = rnn_size_encoder,\n",
    "                                   rnn_size_decoder = rnn_size_decoder,\n",
    "                                   inference_targets = True,\n",
    "                                   pretrained_embeddings_path = pretrained_embeddings_path)\n",
    "\n",
    "summarizer.build_graph()\n",
    "preds = summarizer.infer(converted_texts[:50],\n",
    "                         restore_path =  './models/amazon/my_model',\n",
    "                         targets = converted_summaries[:50])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 11917
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 833,
     "status": "ok",
     "timestamp": 1526243456128,
     "user": {
      "displayName": "Thomas Schmied",
      "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128",
      "userId": "102636220151368904258"
     },
     "user_tz": -120
    },
    "id": "JtB2kNIWmw2j",
    "outputId": "b2b34d18-062a-4ea0-e48f-29ed6c3fe123",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "great taffy at a great price there was a wide assortment of yummy taffy delivery was very quick if your a taffy lover this is a deal\n",
      "\n",
      "Actual Summary:\n",
      "great taffy\n",
      "\n",
      "Created Summary:\n",
      "taffy taffy taffy\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this taffy is so good it is very soft and chewy the flavors are amazing i would definitely recommend you buying it very satisfying\n",
      "\n",
      "Actual Summary:\n",
      "wonderful tasty taffy\n",
      "\n",
      "Created Summary:\n",
      "taffy taffy\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "right now i m mostly just sprouting this so my cats can eat the grass they love it i rotate it around with wheatgrass and rye too\n",
      "\n",
      "Actual Summary:\n",
      "yay barley\n",
      "\n",
      "Created Summary:\n",
      "yay barley\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this is a very healthy dog food good for their digestion also good for small puppies my dog eats her required amount at every feeding\n",
      "\n",
      "Actual Summary:\n",
      "healthy dog food\n",
      "\n",
      "Created Summary:\n",
      "great\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "the strawberry twizzlers are my guilty pleasure yummy six pounds will be around for a while with my son and i\n",
      "\n",
      "Actual Summary:\n",
      "strawberry twizzlers yummy\n",
      "\n",
      "Created Summary:\n",
      "twizzlers twizzlers\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i am very satisfied with my twizzler purchase i shared these with others and we have all enjoyed them i will definitely be ordering more\n",
      "\n",
      "Actual Summary:\n",
      "love it\n",
      "\n",
      "Created Summary:\n",
      "love it\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "candy was delivered very fast and was purchased at a reasonable price i was home bound and unable to get to a store so this was perfect for me\n",
      "\n",
      "Actual Summary:\n",
      "home delivered twizlers\n",
      "\n",
      "Created Summary:\n",
      "delivered delivered twizlers\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i can remember buying this candy as a kid and the quality hasn t dropped in all these years still a superb product you won t be disappointed with\n",
      "\n",
      "Actual Summary:\n",
      "delicious product\n",
      "\n",
      "Created Summary:\n",
      "yummy yummy\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this offer is a great price and a great taste thanks amazon for selling this product staral\n",
      "\n",
      "Actual Summary:\n",
      "this is my taste\n",
      "\n",
      "Created Summary:\n",
      "scary name\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "the flavors are good however i do not see any differce between this and oaker oats brand they are both mushy\n",
      "\n",
      "Actual Summary:\n",
      "mushy\n",
      "\n",
      "Created Summary:\n",
      "not\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "halloween is over but i sent a bag to my daughters class for her share the chocolate was fresh and enjoyed by many\n",
      "\n",
      "Actual Summary:\n",
      "great deal\n",
      "\n",
      "Created Summary:\n",
      "great\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "no tea flavor at all just whole brunch of artifial flavors it is not returnable i wasted 20 bucks\n",
      "\n",
      "Actual Summary:\n",
      "no tea flavor\n",
      "\n",
      "Created Summary:\n",
      "great tea tea\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "the taste was great but the berries had melted may order again in winter if you order in cold weather you should enjoy flavor\n",
      "\n",
      "Actual Summary:\n",
      "order only in cold weather\n",
      "\n",
      "Created Summary:\n",
      "melted melted\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "great gift for all ages i purchased these giant canes before and the recipients loved them so much they kept them and would not eat them\n",
      "\n",
      "Actual Summary:\n",
      "great\n",
      "\n",
      "Created Summary:\n",
      "great\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this is great dog food my dog has severs allergies and this brand is the only one that we can feed him\n",
      "\n",
      "Actual Summary:\n",
      "great dog food\n",
      "\n",
      "Created Summary:\n",
      "great dog food\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this is the same food we get at pet store but it s delivered to my door and for the same price or slightly less\n",
      "\n",
      "Actual Summary:\n",
      "so convenient\n",
      "\n",
      "Created Summary:\n",
      "healthy healthy food food\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "arrived slightly thawed my parents wouldn t accept it however the company was very helpful and issued a full refund\n",
      "\n",
      "Actual Summary:\n",
      "great support\n",
      "\n",
      "Created Summary:\n",
      "great support\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this has to be one of the best teas i have ever tasted it s clean bright fresh great delivery again quality just try it\n",
      "\n",
      "Actual Summary:\n",
      "the best tea ever freah bright clean\n",
      "\n",
      "Created Summary:\n",
      "exceptional\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i love this tea it helps curb my eating during the day my mom and i have given it all friends to try\n",
      "\n",
      "Actual Summary:\n",
      "wonderful tea\n",
      "\n",
      "Created Summary:\n",
      "wonderful tea\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i have bought allot of different flavors and this happens to be one of my favorites and will be getting more soon\n",
      "\n",
      "Actual Summary:\n",
      "great flavor\n",
      "\n",
      "Created Summary:\n",
      "great flavor\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i keep trying other brands cheaper brands stupid me this ginger is soooo worth the money tender moist and never a let down\n",
      "\n",
      "Actual Summary:\n",
      "simply the best\n",
      "\n",
      "Created Summary:\n",
      "simply far\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "works with chicken fish beef or pork fast easy and makes it taste excellent plus buying in bulk is more than 50 off from box stores\n",
      "\n",
      "Actual Summary:\n",
      "awesome stuff\n",
      "\n",
      "Created Summary:\n",
      "great stuff\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "got this for my brother who is on jorge cruise diet and decided to try one for myself it actually tastes pretty good\n",
      "\n",
      "Actual Summary:\n",
      "tastes good\n",
      "\n",
      "Created Summary:\n",
      "very good\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "perfect size sea salt for the table or the picnic basket we love it shakes well no clumping and flows freely\n",
      "\n",
      "Actual Summary:\n",
      "great tasting sea salt with iodine\n",
      "\n",
      "Created Summary:\n",
      "great and and\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "these are the best tasting tuna pack they make in my opinion make a great on the go snack and really satisfying with the tomato\n",
      "\n",
      "Actual Summary:\n",
      "tasty\n",
      "\n",
      "Created Summary:\n",
      "tasty thins thins\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this is an great product the taste is great and it works exactly as described superb natural sleep aid amazing\n",
      "\n",
      "Actual Summary:\n",
      "marley s mellow mood lite half tea half lemonade\n",
      "\n",
      "Created Summary:\n",
      "great\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "we had trouble finding this locally delivery was fast no more hunting up and down the flour aisle at our local grocery stores\n",
      "\n",
      "Actual Summary:\n",
      "yum falafel\n",
      "\n",
      "Created Summary:\n",
      "great falafel\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i like these better than the regular altoids but they re even more costly so you d better like them\n",
      "\n",
      "Actual Summary:\n",
      "nice little mints but pricey\n",
      "\n",
      "Created Summary:\n",
      "yummy\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "be careful not to eat too many of them in one day as one of the chemicals in it is the same as that of a laxative lololol\n",
      "\n",
      "Actual Summary:\n",
      "sugarfree\n",
      "\n",
      "Created Summary:\n",
      "sugarfree\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "these mints are really strong and have a great taste they are also a convenient size to carry in a pocket or purse\n",
      "\n",
      "Actual Summary:\n",
      "tasty\n",
      "\n",
      "Created Summary:\n",
      "good good\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "received as shown my mother in law likes these and walgreens quit carrying them it was great i could get them on amazon\n",
      "\n",
      "Actual Summary:\n",
      "altoids\n",
      "\n",
      "Created Summary:\n",
      "altoids variety\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "fresh scent and taste easy to eat one tin in few hours i use them as favors and everyone loved them\n",
      "\n",
      "Actual Summary:\n",
      "fresh\n",
      "\n",
      "Created Summary:\n",
      "fresh\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "hey the description says 360 grams that is roughly 13 ounces at under 4 00 per can no way that is the approximate price for a 100 gram can\n",
      "\n",
      "Actual Summary:\n",
      "price can not be correct\n",
      "\n",
      "Created Summary:\n",
      "it at at way way\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this is an excellent tea one of the best i have ever had it is especially great when you prepare it with a samovar\n",
      "\n",
      "Actual Summary:\n",
      "best tea ever\n",
      "\n",
      "Created Summary:\n",
      "great tea tea\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "herbal additives in this blend destroy real tea taste it is only for people who like herbal taste i don t\n",
      "\n",
      "Actual Summary:\n",
      "not a real tea\n",
      "\n",
      "Created Summary:\n",
      "strong strong tea\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "took me one or two to get used to the pickle taste but now aim hooked and have to keep a bottle on hand\n",
      "\n",
      "Actual Summary:\n",
      "mcclures bloody mary mix\n",
      "\n",
      "Created Summary:\n",
      "mcclures bloody mary\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "fresh a great way to get a little chocolate in my life without a million calories they taste just like chocolate pudding\n",
      "\n",
      "Actual Summary:\n",
      "omg best chocolate jelly belly\n",
      "\n",
      "Created Summary:\n",
      "omg\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i have tried it out despite the other review and i found that this is exactly the same tea i buy in either russian or italian groceries\n",
      "\n",
      "Actual Summary:\n",
      "this is what you get in the store\n",
      "\n",
      "Created Summary:\n",
      "the the same\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "ahmad tea is an excellent looseleaf tea eith hot or for making iced tea great flavor with no lingering aftertaste\n",
      "\n",
      "Actual Summary:\n",
      "ahmad tea\n",
      "\n",
      "Created Summary:\n",
      "ahmad tea\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "if used on a daily basis you will keep your gums healty and between your teeth clean and free of plack\n",
      "\n",
      "Actual Summary:\n",
      "keeps you out of the dentest chair\n",
      "\n",
      "Created Summary:\n",
      "fast fast\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "five minutes in one tentacle was bitten off ball inside cracked in half not durable enough to be a dog toy disappointed so is the dog\n",
      "\n",
      "Actual Summary:\n",
      "sad outcome\n",
      "\n",
      "Created Summary:\n",
      "sad outcome\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "love this sugar i also get muscavado sugar and they are both great to use in place of regular white sugar recommend\n",
      "\n",
      "Actual Summary:\n",
      "sugar in the raw\n",
      "\n",
      "Created Summary:\n",
      "sugar sugar\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "product is exactly as advertised what a savings at least half the retail price easy to store because of the individual packaging\n",
      "\n",
      "Actual Summary:\n",
      "sugar in the raw\n",
      "\n",
      "Created Summary:\n",
      "buy buy\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i only use raw sugar it did seem a little smaller than the normal crystals but it is still good will buy again\n",
      "\n",
      "Actual Summary:\n",
      "good product\n",
      "\n",
      "Created Summary:\n",
      "sugar sugar\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i really love this sugar and price wise it s much cheaper to buy it here on amazon compared to anywhere else\n",
      "\n",
      "Actual Summary:\n",
      "great\n",
      "\n",
      "Created Summary:\n",
      "great\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "this is one of the best tasting crisps i have ever eaten i only buy it on special occassions because i know i ll eat the whole bag\n",
      "\n",
      "Actual Summary:\n",
      "excellent taste\n",
      "\n",
      "Created Summary:\n",
      "great\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i bought these for my husband and he said they are the best energy shots out there he takes one in the mornings and works hard all day good stuff\n",
      "\n",
      "Actual Summary:\n",
      "great energy\n",
      "\n",
      "Created Summary:\n",
      "great tasting\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "you have to try the vanilla tootsie rolls the regular chocolate ones are good but these are the best ever\n",
      "\n",
      "Actual Summary:\n",
      "vanilla tootsie rolls\n",
      "\n",
      "Created Summary:\n",
      "these tootsie\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "i wouldn t even think of buying this product unless i know the price per unit how can i compare other items online or in a store\n",
      "\n",
      "Actual Summary:\n",
      "what quantity is it\n",
      "\n",
      "Created Summary:\n",
      "did did\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " ----------------------------------------------------------------------------------------------------\n",
      "Actual Text:\n",
      "it is hot i love it tasty and a little sweet i usually scoff at salsas labeled as hot but this one lives up to its billing\n",
      "\n",
      "Actual Summary:\n",
      "hot\n",
      "\n",
      "Created Summary:\n",
      "hot hot\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# show results\n",
    "summarizer_model_utils.sample_results(preds,\n",
    "                                      ind2word,\n",
    "                                      word2ind,\n",
    "                                      converted_summaries[:50],\n",
    "                                      converted_texts[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j_bcG5CPmw2m"
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "Generally I am really impressed by how well the model works. \n",
    "We only used a limited amount of data, trained it for a limited amount of time and used nearly random hyperparameters and it still delivers good results. \n",
    "\n",
    "However, we are clearly overfitting the training data and the model does not perfectly generalize.\n",
    "Sometimes the summaries the model creates are good, sometimes bad, sometimes they are better than the original ones and sometimes they are just really funny.\n",
    "\n",
    "\n",
    "Therefore it would be really interesting to scale it up and see how it performs. \n",
    "\n",
    "To sum up, I am impressed by seq2seq models, they perform great on many different tasks and I look foward to exploring more possible applications. \n",
    "(speech recognition...)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "summarizer_amazon_reviews.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}