deep-elon-tweet-generator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Elon Musk-Like Tweet Generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's load the dataset we will use to create a language model. \n",
    "\n",
    "The data is compiled of tweets from Elon Musk.\n",
    "\n",
    "Twitter data (2010 to 2018):\n",
    "- 6,094 total tweets\n",
    "    - 5,249 tweets\n",
    "    - 845 re-tweets\n",
    "\n",
    "The Twitter data used to train the data was compiled from 3 Elon Musk Twitter datasets from 2010 to 2018 and duplicates were removed:\n",
    "\n",
    "1. [2010-06-04 to 2017-04-05](https://data.world/adamhelsinger/elon-musk-tweets-until-4-6-17) from data.world\n",
    "2. [2017-04-06 to 2017-09-21](https://www.kaggle.com/kulgen/elon-musks-tweets) from kaggle\n",
    "3. [2017-12-17 tp 2018-12-11](https://data.world/barbaramaseda/elon-musk-tweet) from data.world"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Please ignore prior tweets, as that was someon...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Went to Iceland on Sat to ride bumper cars on ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I made the volume on the Model S http://t.co/w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Great Voltaire quote, arguably better than Twa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>That was a total non sequitur btw</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text\n",
       "0  Please ignore prior tweets, as that was someon...\n",
       "1  Went to Iceland on Sat to ride bumper cars on ...\n",
       "2  I made the volume on the Model S http://t.co/w...\n",
       "3  Great Voltaire quote, arguably better than Twa...\n",
       "4                  That was a total non sequitur btw"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('/storage/deep-elon-tweet-generator/musk-tweets-2010-to-2018.csv');\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training data contains 1 tweet per row and has no labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Went to Iceland on Sat to ride bumper cars on ice!  No, not the country, Vlad's rink in Van Nuys. Awesome family fun :) http://t.co/rBQXJ9IT\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's look at a single tweet\n",
    "df['text'][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6094"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of total tweets in training data (2010-2018)\n",
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# batch size\n",
    "bs = 48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create langugage model using our twitter data\n",
    "data_lm = (TextList.from_df(df,Path('/storage/deep-elon-tweet-generator/'), col=['text']) \n",
    "          #We randomly split and keep 10% (~609 tweets) for validation\n",
    "           .random_split_by_pct(0.1)\n",
    "          #We want to do a language model so we label accordingly\n",
    "           .label_for_lm()\n",
    "          .databunch(bs=bs))\n",
    "data_lm.save('tmp_lm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step of processing we make tweets go through is to split the raw sentences into words, or more exactly tokens. The easiest way to do this would be to split the string on spaces, but we can be smarter:\n",
    "\n",
    "- we need to take care of punctuation\n",
    "- some words are contractions of two different words, like isn't or don't\n",
    "- we may need to clean some parts of our texts, if there's HTML code for instance\n",
    "\n",
    "To see what the tokenizer had done behind the scenes, let's have a look at a few tweets in a batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = TextLMDataBunch.load(Path('/storage/deep-elon-tweet-generator/'), 'tmp_lm', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>  <col width='5%'>  <col width='95%'>  <tr>\n",
       "    <th>idx</th>\n",
       "    <th>text</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>xxbos xxmaj the xxmaj model s beta xxunk car just passed xxunk miles on a single battery pack ! xxbos xxmaj model s xxunk into semi truck xxunk and lifts it off the ground . xxmaj driver xxunk away . https : / / t.co / xxunk xxbos xxup rt @teslamotors : xxmaj xxunk xxmaj xxunk street view from a right hand drive xxmaj model xxup s. http : /</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>https : / / t.co / xxunk xxbos xxunk i want to know who is running the xxmaj xxunk xxunk ! xxmaj mad xxunk ... xxbos xxunk xxmaj thanks xxbos xxup rt @john_gardi : @elonmusk xxmaj you should point out this campaign was n't started by xxup xxunk but by concerned citizens that want a fair deal & & a xxunk \\ xe2 \\ x80 \\ xa6 xxbos @redletterdave xxmaj</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>xxmaj the cat videos are awesome xxbos xxmaj worth reading xxmaj the xxmaj machine xxmaj stops , an old story by xxup xxunk xxup xxunk xxmaj xxunk https : / / t.co / xxunk xxbos xxmaj expect to reach xxunk xxunk regarding last flight by end of week . xxmaj will brief key customers & & xxup xxunk , then post on our website . xxbos xxup rt @electrekco :</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>xxunk xxbos xxup rt @spacex : xxmaj early this morning , @spacex engineers xxunk # xxmaj dragon into the xxmaj port of xxup la http : / / t.co / xxunk xxbos xxunk xxmaj wow , that 's great ! xxbos xxmaj owner video of xxmaj autopilot steering to avoid collision with a truck \\n https : / / t.co / xxunk xxbos xxup rt @arstechnica : xxmaj xxunk \\</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>xxunk xxunk has details https : / / t.co / xxunk ? xxbos xxmaj about to give a talk at the xxmaj xxunk https : / / t.co / xxunk xxbos xxmaj we did n't even have kids back then . xxmaj just little xxunk . xxbos xxmaj xxunk judge xxunk auto dealers ' demand to kill our little xxmaj tesla store . xxmaj yay , justice xxunk ! http</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>xxunk xxunk xxmaj thanks for offering to help . i will check with my team . xxbos xxup xxunk xxmaj you 're right . xxmaj xxunk is a no - win situation . xxmaj xxunk is going to hate what you say no matter what that is . xxbos lol https : / / t.co / xxunk xxbos xxmaj first flight of 10 story xxunk xxmaj grasshopper rocket using closed</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>sales xxunk . xxbos xxup rt @spacex : xxmaj falcon 9 and xxmaj telstar 19 xxup vantage went vertical this afternoon on xxmaj pad 40 in xxmaj florida . xxmaj weather is 60 % favorable for the four - hour launch window , which opens xxmaj sunday , xxmaj july 22 at xxunk a.m. xxup edt , xxunk xxup utc . https : / / t.co / gtc39ubc7z https :</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>like xxmaj blackrock keep up to 50 % of short interest revenue , but xxunk almost none of xxunk decline , as they 're just \" passive \" managers . xxmaj blackrock made $ xxunk xxbos https : / / t.co / xxunk https : / / t.co / xxunk xxbos @djsnm xxmaj pump is single xxunk . xxmaj some landing systems are not xxunk , as landing is considered</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>a boat for short periods of time . xxmaj thrust via wheel xxunk . xxbos xxup rt @teslarati : spacex 's xxmaj mr xxmaj steven xxunk in high - speed test at sea with upgraded net ahead of fairing catch xxunk https : / / t.co / xxunk xxbos xxunk xxmaj early next year . xxmaj to start with , we 're making the xxunk xxmaj model 3 first ,</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>what is more xxunk than xxunk xxunk xxunk ? xxmaj xxunk ' the xxunk ... http : / / t.co / xxunk xxbos @samabuelsamid @teslamotors i 've written two forward - looking master plans , but have n't told the xxmaj tesla history of ho ? https : / / t.co / xxunk xxbos xxup rt @spacex : xxmaj close , but no xxunk . xxmaj this time . https</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch(rows=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The texts are truncated at 100 tokens for more readability. We can see that it did more than just split on space and punctuation symbols: \n",
    "- the \"'s\" are grouped together in one token\n",
    "- the contractions are separated like his: \"did\", \"n't\"\n",
    "- content has been cleaned for any HTML symbol and lower cased\n",
    "- there are several special tokens (all those that begin by xx), to replace unknown tokens (see below) or to introduce different text fields (here we only have one)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numericalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have extracted tokens from our texts, we convert to integers by creating a list of all the words used. We only keep the ones that appear at list twice with a maximum vocabulary size of 60,000 (by default) and replace the ones that don't make the cut by the unknown token `UNK`.\n",
    "\n",
    "The correspondance from ids tokens is stored in the `vocab` attribute of our datasets, in a dictionary called `itos` (for int to string)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['xxunk',\n",
       " 'xxpad',\n",
       " 'xxbos',\n",
       " 'xxfld',\n",
       " 'xxmaj',\n",
       " 'xxup',\n",
       " 'xxrep',\n",
       " 'xxwrep',\n",
       " '/',\n",
       " '.',\n",
       " ':',\n",
       " ',',\n",
       " 'the',\n",
       " 'to',\n",
       " 't.co']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the most common dictionary terms\n",
    "data_lm.vocab.itos[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're not going to train a model that generates language from scratch. \n",
    "\n",
    "Instead, we'll use a model pretrained on a bigger dataset (a cleaned subset of wikipeia called [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset)). That model has been trained to guess what the next word, its input being all the previous words. It has a recurrent structure and a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.\n",
    "\n",
    "We are going to use that 'knowledge' of the English language to build our tweet generator, but first, we need to fine-tune the pretrained model with Elon's tweets. Because Twitter language/style isn't the same as the English of Wikipedia, we'll need to adjust a little bit the parameters of our model. Plus there might be some words extremely common in that dataset that were barely present in Wikipedia, and therefore might no be part of the vocabulary the model was trained on.\n",
    "\n",
    "Note that language models can use a lot of GPU, so you may need to decrease batchsize here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop_mult is a parameter that controls the % of drop-out used\n",
    "learn = language_model_learner(data_lm, pretrained_model=URLs.WT103, drop_mult=0.30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "# use learning rate finder to identify a good learning rate to use\n",
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3Xl8XHW9//HXJ5nse5t0S9qm6UqBrimlLWUHwQUsWgVBoKBeFsHlonCvv3v16gUV3AAFRKCKoF4poILIpiwt0EJKN9rSfUm6Jm2WNmnW+f7+mKmEmGbpzMlJMu/n4zGPzpz5zszn2zPJO9+zfI855xARETlecX4XICIifZuCREREIqIgERGRiChIREQkIgoSERGJiIJEREQioiAREZGIKEhERCQiChIREYlIwO8Cuis3N9cVFhb6XYaISJ+yfPnyCudcnhfv3eeCpLCwkJKSEr/LEBHpU8xsh1fvrU1bIiISEQWJiIhEREEiIiIRUZCIiEhEFCQiIhIRBYmIiEREQSIiIhGJmSApP9TA/z67jorDDX6XIiLSr8RMkLy19QAL39zOGXe+wk9e3EBNfZPfJYmI9AsxEyQXTR7Gi187nTPHD+Kef2zm9Dtf4Vevb6WpJeh3aSIifVrMBAnA6Lx0fnH5NJ758mlMKsjm9ufWc/lDyyg/pM1dIiLHK6aC5KiTC7J49JpTuPvSKawuq+IT9y5hxc5Kv8sSEemTYjJIjrp4Sj5PXT+HhIDx2V8u5fdv78Q553dZIiJ9ivW1X5zFxcUu2rP/VtU1cvMfVvL6xnKGZCYzY9QAZhTmMKNwAOMHZxAXZ1H9PBGRnmZmy51zxV68d5+bRt4L2amJLLx6BouWl7Jk8wHe2XaQZ1btBiA/O4WLpgzjk1PyGT8ko1vvu72ilmXbDjA0K4XhA1LJz04hMRDTg0AR6Yc0ImmHc46yyiO8tfUAf129hyWbK2gJOiYMyeBT0wqYNy2f3PSkY76+JehY+MY27nphAw3NHxwVZgYD05JIjDcC8XEE4o20xACnjc3l/ImDmVyQrdGPiHjCyxGJgqQLyg818NfVu3l65W5WlVYRiDPOPWEwn5lRwNyxeSTEfzDK2FZRyzeeWEXJjkrOPWEQt3xkPNV1TZRWHqH0YB37D9XT2OxoCQZpCjoqDjWwfEclzUHHoIwk5o7No6G5hb3V9eyprqf8cAOFA1OZPjKHaSNymD4yh1G5aZh1HDhNLUHe3HKAv67eTUsQbjxrNEV56V7/V7XLOcfWilre3VHJuzurWLGzkoO1jQzNSmZIVjJDs1IYlp3MsOyU0C0rhfTkAFvLD7Nh7yE27T/Mnup6JuVnMWdMLhOGHP/mxqq6Rl5at48X1u7jQG0DIwakMnJAKsMHpDJmUDoTh2WSFIjv9H1ago6mliAJ8XHEt6rlcEMzuyqPsKuqjj3V9aQkxDMoI5lBmUnkpSdhBrWNLdQ1NHO4oZnUxAD5OSmkJ2njgHhLQdKKH0HS2sZ9h/jjO6U8vWIXB2obMYOc1ETy0pMYmJ7IuzsrSYyP49ufOJFLpuV3+gsfoLquiVc27OeldftYuvUAGcmBf/6CHZCWyOb9h1mxs5Ka+mYABmeGAmfu2FxOG5NLRnIC+2rq2VtTz+6qI7y15QDPr91LVV0TGUkBgs7R0BzkczNHcPM5YzscTXVHU0uQFTureHNLBcGgIyctkQHhW8XhBt7bVcPa3dWs3V3DoXDtGckBpgzPZkhmMvsONbCn6gh7q+s51NB8zM9JDMSRl57ErqojAAxMS2T2mFwunTGc2aMHHvP/OBh07K4+wub9h9m8/zCvbSznrS0HaA46hmUlM3JgGjsP1rGn+ghB98FnTcrPYvrIHEYMTKX04BG2V9Sy/UAtu6uO0NAcpKkl+M/2AHEWel2cGXWNLcf1f5mdmkB+dgrjB2dw+rjQuh0YpfUkAgqSD/E7SI5qbA7yyob9rN1dQ8XhBioONVB+uIH87BT+38cmMiQrOaqfFww6tpQfpmRHJUs2V/DG5gqq6kJn55tB69WYlhjPeRMH8/FJw5g7LpeaI83c8/dN/O7tnSQH4phfPJy8jCRSEuJJTYwnJTGeODPi44w4M5xzVNQ2Ul4TGhEdrG0kJSGezJQEMpMTSE6IY2VpNW9tqaC2sYWjv8fbfpWSAnGcMDSTE4dlMqkgi2kjchidl97uaKKmvok9VaEg3FV1hJr6Jopy0xk3OJ2RA9OIjzP2VtfzRrjvr24s52BtIyfnZ/FvZxRxwYlDcMDK0ioWbyxnyeYK3t976EO/2AsHpnLBSUO58KQhTCrI+mcANTYH2VV1hPV7anh3RyXLd1by3q5qmlocCfHGiAGpjMpNIz87heTEeBLj4/45Ejk6MmlsDtIcdOSmJ5Gfk0J+dmiUdaSxhfJDDew/1PDP85XSkwKkJoX+72sbWigLj2DKKo+wuqyag+E/UE7Oz+K8EwZz2cwRUQt/iV0KklZ6S5D4rSXoeG9XNUs2V9DYHGRYdjJDslIYlpXM8AGpJCf86+aZLeWHuev5Dby8fh/Nwc7X+9F9OgPSEmhoDlJzpIma+mZago6RA1M5bUwuc8fmMWv0QNKTAlTVNVJZ18iBw41kpyYyOi+NQLw3BxfUN7Xw9Ipd/Or1rWytqGVoVjKH65s51NBMnMHk4dlMGZ7NmEHpjMlLZ8yg9G79hV/f1MKB2kaGZCZ/aNOV14JBx3u7q3ltQzmvbixn+Y5KEgNxfGpaPteeVsSYQf5snpS+T0HSioIkcs45GluCHGlsoa6xhSNNLQSDjhbnCIaPDchND22iahsEzjnqm4KkJHa+H6EntAQdL63bxx9LShmcmczpY3OZPTqXrNQEv0uLis37D/PIG9t4cnkZDc1B5o7N5ewJg5g7NpfReeld2nQqAgqSD1GQSCw6cLiBx5bu5KkVZew4UAfA0KxkzhiXx4I5o7p9aLrEHgVJKwoSiXWlB+tYvKmCJZvLeXVDOXWNLXz05CF8+ayxTByW6Xd50kspSFpRkIh8oLK2kUfe2Mav39jOoYZmzps4mK+cM5aT8rP8Lk16GS+DxLPTrM1svJmtbHWrMbOvtmljZnaPmW02s9VmNs2rekT6o5y0RP79/PEsue1svnbuOJZtPcDH713Clx4tYe3uar/LkxjRIyMSM4sHdgEznXM7Wi3/KHAT8FFgJnC3c25mR++lEYnIsdXUN/HrN7bz0OKt1NQ3c/7EwXzrYycwcmCa36WJz/rkiKSNc4AtrUMk7GLgUReyFMg2s6E9VJNIv5OZnMDN54xl8a2hEcpbWw5w4d2L+d0yzWwt3umpILkU+H07y/OB0laPy8LLPsTMvmRmJWZWUl5e7lGJIv1HVkoCXzl3LC987XSmjsjmP59ewzW/fof9NfV+lyb9kOdBYmaJwEXAE+093c6yf/mzyTn3oHOu2DlXnJeXF+0SRfqtYdkp/PaamXz7ExN5c8sBPvKz17n/1S3sqT7id2nSj/TEiORC4F3n3L52nisDhrd6XADs7oGaRGJGXJyxYM4o/nrzXMYPyeCHz7/P7B/8g8sfWsqi5WUcOc75wUSO6okguYz2N2sB/AW4Mnz01qlAtXNuTw/UJBJzxgxK5w9fmsWrt5zJzWePpfTgEW55YhUfu3cx6/fU+F2e9GGeHrVlZqmE9oEUOeeqw8uuA3DOPWCh+R1+DlwA1AELnHMdHpKlo7ZEosM5x2sby/nGotXUHGniOxedyKUzhmvalX5KJyS2oiARia7yQw18/Y8rWbypgounDOP2eSfr+ij9UH84/FdEeqm8jCR+s+AUbjl/HM+s2s0l973B7irtjJeuU5CICHFxxpfPHstvr53Jnqp65t33But2a7+JdI2CRET+ac6YXJ64fhZxZnzml2+xeJPO25LOKUhE5EMmDMnkqRtmU5CTwoKF77BoeZnfJUkvpyARkX8xNCuFP143i5lFA7jliVX85KWNmmJFjklBIiLtykxOYOHVpzB/egH3/H0TX/2/lTQ06+RF+Vc6xk9EjikxEMedn55EYW4ad72wgd1VR/jl54sZkJbod2nSi2hEIiIdMjNuPGsM9142lVVl1Vxy3xtsKT/sd1nSiyhIRKRLPjF5GL//4kwO1Tcz7xdv8ObmCr9Lkl5CQSIiXTZ95AD+dOMcBmcmc+Ujb/OHt3f6XZL0AgoSEemW4QNSefKG2cwek8ttT63h+8+t1xFdMU5BIiLdlpmcwCNXFXPFqSP45etbeWjxNr9LEh/pqC0ROS6B+Di+d/FJVNY28f2/rWfckAzOGKcLz8UijUhE5LiZGXfNn8T4IZl8+XfvslVHc8UkBYmIRCQ1McCvrpxOQnwcX3i0hJr6Jr9Lkh6mIBGRiBXkpHL/5dPYeaCOr/x+BS1B7XyPJQoSEYmKmUUD+c5FJ/LKhnIeeG2L3+VID1KQiEjUXD5zBBdNHsZPXtrI8h0H/S5HeoiCRESixsy4fd5J5GencPPvV1Jdp/0lsUBBIiJRlZGcwL2XTWVfTT23PrlaJyvGAAWJiETd5OHZ3HrBBJ5fu5fHlmkalf5OQSIinrj2tFGcOT6P7z27jo37DvldjnhIQSIinoiLM340fzLpSQG+sWi1DgnuxxQkIuKZ3PQkvnPRiawqreKRJZqPq79SkIiIpz4xaSjnTRzMj17cwLaKWr/LEQ8oSETEU2bG/37yJBIDcdz65GqC2sTV7yhIRMRzgzOT+a+PTeTtbQd5fNkOv8uRKFOQiEiPmF9cwNyxufzgb+9TVlnndzkSRQoSEekRZsYd807GAf/1p/d0omI/oiARkR4zfEAq/37+eF7ZUM6zq/f4XY5EiYJERHrU1bMLmVSQxf88s1ZzcfUTChIR6VHxccb3LzmZyrom7nhuvd/lSBR4GiRmlm1mi8zsfTNbb2az2jyfZWbPmNkqM1trZgu8rEdEeocTh2Xxhbmj+L+SUt7acsDvciRCXo9I7gaed85NACYDbf/8uBFY55ybDJwJ/NjMEj2uSUR6ga+eM47hA1L41tNrqG9q8bsciYBnQWJmmcDpwMMAzrlG51xVm2YOyDAzA9KBg0CzVzWJSO+RkhjP7Z88ma0VtTz4+la/y5EIeDkiKQLKgYVmtsLMHjKztDZtfg6cAOwG1gBfcc4F276RmX3JzErMrKS8vNzDkkWkJ50+Lo+PnDiYB1/fysHaRr/LkePkZZAEgGnA/c65qUAtcFubNh8BVgLDgCnAz8MjmQ9xzj3onCt2zhXn5eV5WLKI9LRbzh9PXWMz972y2e9S5Dh5GSRlQJlzbln48SJCwdLaAuApF7IZ2AZM8LAmEellxg7O4JJpBTy6dAe7q474XY4cB8+CxDm3Fyg1s/HhRecA69o02xlejpkNBsYD2lgqEmO+eu5YcHD3y5v8LkWOg9dHbd0EPG5mqwlturrDzK4zs+vCz38PmG1ma4C/A7c65yo8rklEepmCnFQuP3UETywvZfP+w36XI91kfW2+m+LiYldSUuJ3GSISZRWHGzjjzlc4Y3we910+3e9y+h0zW+6cK/bivXVmu4j0CrnpSVw7t4jn1uxldVnbMwWkN1OQiEiv8cW5o8hJTeCuFzb4XYp0g4JERHqNjOQErj9zNIs3VbBsq6ZO6SsUJCLSq1w5q5BBGUn86MUNumZJH6EgEZFeJTkhnpvOHsM72yt5baNmsugLFCQi0ut8dsYICnJS+PGLGzUq6QMUJCLS6yQG4vjKOWNZs6uaF9bu9bsc6YSCRER6pXlT8ynKS+PHL26kJahRSW+mIBGRXikQH8fXzxvHpv2H+cuqXX6XIx1QkIhIr/XRk4ZywtBM7v37ZoIalfRaChIR6bXi4owbzhzN1opaXlq/z+9y5BgUJCLSq1140hAKclJ0FcVeTEEiIr1aID6OL84tYvmOSkq2H/S7HGmHgkREer35xQXkpCbwwGsalfRGChIR6fVSEwN8flYhL6/fp+uV9EIKEhHpE66aNZKkQBy/0r6SXkdBIiJ9wsD0JD5TPJynV+xif0293+VIKwoSEekzvjB3FM3BIAvf3O53KdKKgkRE+oyRA9O44KQhPL50B0caW/wuR8IUJCLSp1w1q5Ca+maeWb3b71IkTEEiIn3KKaMGMG5wOo8t3eF3KRKmIBGRPsXMuOLUkawuq2ZVaZXf5QgKEhHpg+ZNzSc1MV6jkl5CQSIifU5GcgLzpubzl1W7qapr9LucmNelIDGz0WaWFL5/ppndbGbZ3pYmInJsV5w6kobmIIuWl/ldSszr6ojkSaDFzMYADwOjgN95VpWISCdOGJrJjMIcHlu6Q9cq8VlXgyTonGsG5gE/c859DRjqXVkiIp274tSRbD9Qx5LNFX6XEtO6GiRNZnYZcBXwbHhZgjcliYh0zQUnDWFgWiK/1U53X3U1SBYAs4DbnXPbzGwU8Jh3ZYmIdC4pEM9nZwzn7+v3UVZZ53c5MatLQeKcW+ecu9k593szywEynHM/8Lg2EZFOXX7qSAAeX7bT50piV1eP2nrVzDLNbACwClhoZj/xtjQRkc7lZ6dw3sTB/OHtndQ3af4tP3R101aWc64GuARY6JybDpzrXVkiIl131exCKuuaeHb1Hr9LiUldDZKAmQ0FPsMHO9s7ZWbZZrbIzN43s/VmNqudNmea2UozW2tmr3X1vUVEjppVNJBxg9P5zZvbcU6HAve0rgbJd4EXgC3OuXfMrAjY1IXX3Q0875ybAEwG1rd+MnxS433ARc65E4H5Xa5cRCTMzLhyViFrdlWzQvNv9biu7mx/wjk3yTl3ffjxVufcpzp6jZllAqcTOoER51yjc67tGv4c8JRzbme4zf7udkBEBELzb2UkBXhUF73qcV3d2V5gZk+b2X4z22dmT5pZQScvKwLKCe2YX2FmD5lZWps244Cc8M785WZ25XH0QUSEtKQAny4u4K9r9rD/kC7F25O6umlrIfAXYBiQDzwTXtaRADANuN85NxWoBW5rp8104GPAR4D/MrNxbd/IzL5kZiVmVlJeXt7FkkUk1lw5q5CmFscf3i71u5SY0tUgyXPOLXTONYdvvwbyOnlNGVDmnFsWfryIULC0bfO8c67WOVcBvE5oX8qHOOcedM4VO+eK8/I6+1gRiVWjctM4Y1wejy/bQVNL0O9yYkZXg6TCzK4ws/jw7QrgQEcvcM7tBUrNbHx40TnAujbN/gzMNbOAmaUCM2mzQ15EpDuunl3IvpoG/vbeXr9LiRldDZJrCB36uxfYA3ya0LQpnbkJeNzMVgNTgDvM7Dozuw7AObceeB5YDbwNPOSce697XRAR+cAZ4/Ioyk3jkSXb/C4lZgS60ih8VNVFrZeZ2VeBn3XyupVAcZvFD7RpcxdwV1fqEBHpTFyccdXsQr79l7Ws2FnJ1BE5fpfU70VyhcSvR60KEZEo+tT0AjKSAix8Y7vfpcSESILEolaFiEgUpScF+OyM4Ty3Zg97q3UosNciCRLNQyAivdZVswsJOsdvl273u5R+r8MgMbNDZlbTzu0QoXNKRER6peEDUjn3hMH8bplmBfZah0HinMtwzmW2c8twznVpR72IiF8WzBlFZV0Tf1qxy+9S+rVINm2JiPRqpxYN4IShmSx8Q7MCe0lBIiL9lplxzZxCNuw7xKsbNL2SVxQkItKvXTwln/zsFH768kaNSjyiIBGRfi0xEMfN54xhdVk1/3hfV6rwgoJERPq9S6YVMGJAqkYlHlGQiEi/lxAfx01nj+G9XTW8tG6f3+X0OwoSEYkJ86bmUzgwlZ++vIlgUKOSaFKQiEhMCMTHcfM5Y1m/p4YX12mK+WhSkIhIzLho8jCK8tL46UsalUSTgkREYkYgPo6vnDOWDfsOaVQSRQoSEYkpH580jOEDUnjw9a1+l9JvKEhEJKbExxlfOK2Id3dWsXzHQb/L6RcUJCISc+YXF5CVkqBRSZQoSEQk5qQmBvj8qSN5cd0+tlXU+l1On6cgEZGYdOXskSTExfHwEo1KIqUgEZGYNCgjmXlT83mipIwDhxv8LqdPU5CISMz6wtxRNDQHeWzpTr9L6dMUJCISs8YOzuDsCYN49K3tuhxvBBQkIhLTvji3iAO1jSxaXuZ3KX2WgkREYtqpRQOYXJDFrxZvpUXTphwXBYmIxDQz4/ozR7PjQB1/e2+P3+X0SQoSEYl5508cQlFeGve/ukUXvjoOChIRiXlxccZ1p49m7e4aFm+q8LucPkdBIiICXDx1GIMzk7j/1S1+l9LnKEhERICkQDxfOK2It7YeYGVpld/l9CkKEhGRsMtmjiAzOcADGpV0i4JERCQsPSnAVbMLeWHdXjbvP+x3OX2Gp0FiZtlmtsjM3jez9WY26xjtZphZi5l92st6REQ6c/XsQpICcdpX0g1ej0juBp53zk0AJgPr2zYws3jgh8ALHtciItKpgelJfO6Ukfxp5S52HNAU813hWZCYWSZwOvAwgHOu0TnX3h6sm4Angf1e1SIi0h3XnVFEIM74xSub/S6lT/ByRFIElAMLzWyFmT1kZmmtG5hZPjAPeMDDOkREumVQZjKXnTKCp97dRenBOr/L6fW8DJIAMA243zk3FagFbmvT5mfArc65DqfdNLMvmVmJmZWUl5d7U62ISCvXnTGaONOopCu8DJIyoMw5tyz8eBGhYGmtGPiDmW0HPg3cZ2afbPtGzrkHnXPFzrnivLw8D0sWEQkZkpXMpacMZ9HyMsoqNSrpiGdB4pzbC5Sa2fjwonOAdW3ajHLOFTrnCgkFzQ3OuT95VZOISHdcf2ZoVHKfjuDqkNdHbd0EPG5mq4EpwB1mdp2ZXefx54qIRGxoVgrziwt4oqSUXVVH/C6n1/I0SJxzK8ObpCY55z7pnKt0zj3gnPuXnevOuaudc4u8rEdEpLtuOGsMAPe/qn0lx6Iz20VEOpCfncL84uH88Z0ydmtU0i4FiYhIJ244czQOp7Pdj0FBIiLSiYKcVOYXD+f/3inVqKQdChIRkS44Oiq5T/tK/oWCRESkCzQqOTYFiYhIF90YPoJLo5IPU5CIiHRRfnYKn9Go5F8oSEREuuHoeSWag+sDChIRkW44Oir5Y0mpZgYOU5CIiHTTl88eg5lx7z82+V1Kr6AgERHppqFZKXzulBE8+e4utlfoKooKEhGR43DDWaNJiDfu/rtGJQoSEZHjMCgjmStnFfKnlbvYvP+Q3+X4SkEiInKc/u30IlIS4vnZy7E9KlGQiIgcp4HpSSyYU8izq/fw/t4av8vxjYJERCQCX5xbREZSgJ++tNHvUnyjIBERiUB2aiLXnDaKF9buY93u2ByVKEhERCJ0zWmjyEgOcE+MHsGlIBERiVBWSgIL5ozi+bV7Wb8n9kYlChIRkSi4ds4oMpICMXm2u4JERCQKslITuHpOIc+t2cuGvbF1XomCREQkSq49bRTpSQHuibFRiYJERCRKslMTuWr2SJ5bs4dN+2JnVKIgERGJoi+cVkRqQnxMzcGlIBERiaKctEQWzBnFs6v38NaWA36X0yMUJCIiUXbjWWMYOTCVW59cTV1js9/leE5BIiISZSmJ8dz5qUnsPFjHnc9v8LsczylIREQ8MLNoIFfNGslv3trOO9sP+l2OpxQkIiIe+eYFEyjISeGbi1ZzpLHF73I8oyAREfFIWlKAH14yiW0Vtfzkpf67iUtBIiLiodljcrl85ggeWrKNNWXVfpfjCQWJiIjHbr1wAgPTkvivP79HMOj8LifqFCQiIh7LTE7gPy6cwMrSKha9W+Z3OVHnaZCYWbaZLTKz981svZnNavP85Wa2Onx708wme1mPiIhf5k3NZ/rIHH74t/eprmvyu5yo8npEcjfwvHNuAjAZWN/m+W3AGc65ScD3gAc9rkdExBdxccZ3Lz6RyrrGfrfj3bMgMbNM4HTgYQDnXKNzrqp1G+fcm865yvDDpUCBV/WIiPjtxGFZXHHqSH67dAdrd/efHe9ejkiKgHJgoZmtMLOHzCytg/bXAn9r7wkz+5KZlZhZSXl5uRe1ioj0iH8/bzzZqYl8+89r+82Ody+DJABMA+53zk0FaoHb2mtoZmcRCpJb23veOfegc67YOVecl5fnVb0iIp7LSk3gtgsmULKjkp+9vNHvcqLCyyApA8qcc8vCjxcRCpYPMbNJwEPAxc652JgqU0Ri2vziAj5TXMA9/9jMEyWlfpcTMc+CxDm3Fyg1s/HhRecA61q3MbMRwFPA551z/SOaRUQ6YWbcPu9k5owZyH88tYY3N1f4XVJEvD5q6ybgcTNbDUwB7jCz68zsuvDz/w0MBO4zs5VmVuJxPSIivUJCfBz3XT6dUblp/Ntjy9m8v+9eUdGc61s7e4qLi11JifJGRPqHsso6PvmLN0lOiOPpG+aQl5HkyeeY2XLnXLEX760z20VEfFSQk8rDVxVz4HAjC379Nocb+t6FsBQkIiI+mzw8m/sun8b6PYe4/rHlNDYH/S6pWxQkIiK9wFkTBvGDS05m8aYKvrloVZ86xyTgdwEiIhIyv3g4+w81cNcLGxiUmcx/fvQEv0vqEgWJiEgvcsOZo9lXU8+Dr2+lJei47cIJJMT37o1HChIRkV7EzPj2J07EgIeXbGNVaRU//9w0hmQl+13aMfXumBMRiUHxccb/XHwSd186hXV7avj4vYt79UmLChIRkV7q4in5/PnGOWSnJnLFw8t4ZMk2v0tql4JERKQXGzs4gz/fOIeLJg9jVF5HE6j7R/tIRER6ubSkAD+7dKrfZRyTRiQiIhIRBYmIiEREQSIiIhFRkIiISEQUJCIiEhEFiYiIRERBIiIiEVGQiIhIRPrcpXbNrBzY0WZxFlDdybKOHh+933pZLnC8k9u0V0932nS3P53dj6QvndXaWZv+tG660pe2y7xcN/qedby8r37PjvVcpOsmzTmX12nlx8M51+dvwIOdLevo8dH7bZaVRLOe7rTpbn86ux9JXyLtT39aN13pS0+uG33P+uf3rDeum85u/WXT1jNdWNbR42eO0Saa9XSnTXf705X7kYikP/1p3XSlL22Xeblu9D3reHlf/Z4d6zk/102H+tymrZ5iZiXOuWK/64iG/tQX6F/9UV96r/7UH6/70l9GJF540O/EtGh6AAAHTklEQVQCoqg/9QX6V3/Ul96rP/XH075oRCIiIhHRiERERCLS74PEzB4xs/1m9t5xvHa6ma0xs81mdo+ZWavnbjKzDWa21szujG7VHdYU9f6Y2XfMbJeZrQzfPhr9ytutx5N1E37+FjNzZpYbvYo7rcmLdfM9M1sdXi8vmtmw6Ffebj1e9OUuM3s/3J+nzSw7+pUfsyYv+jM//PMfNDPP96VE0odjvN9VZrYpfLuq1fIOf7ba5eUhYb3hBpwOTAPeO47Xvg3MAgz4G3BhePlZwMtAUvjxoD7en+8At/SHdRN+bjjwAqHzjXL7cn+AzFZtbgYe6MN9OR8IhO//EPhhH183JwDjgVeB4t7ah3B9hW2WDQC2hv/NCd/P6ai/Hd36/YjEOfc6cLD1MjMbbWbPm9lyM1tsZhPavs7MhhL6IX7Lhf53HwU+GX76euAHzrmG8Gfs97YXH/CoP77wsC8/Bb4J9OgOQC/645yradU0jR7qk0d9edE51xxuuhQo8LYXH/CoP+udcxt6ov7w5x1XH47hI8BLzrmDzrlK4CXgguP9PdHvg+QYHgRucs5NB24B7munTT5Q1upxWXgZwDhgrpktM7PXzGyGp9V2LtL+AHw5vMnhETPL8a7UTkXUFzO7CNjlnFvldaFdFPG6MbPbzawUuBz4bw9r7Uw0vmdHXUPor10/RbM/fulKH9qTD5S2eny0X8fV35i7ZruZpQOzgSdabfpLaq9pO8uO/jUYIDQcPBWYAfzRzIrCCd6jotSf+4HvhR9/D/gxoR/0HhVpX8wsFfgWoU0ovovSusE59y3gW2b2H8CXgW9HudRORasv4ff6FtAMPB7NGrsjmv3xS0d9MLMFwFfCy8YAz5lZI7DNOTePY/fruPobc0FCaBRW5Zyb0nqhmcUDy8MP/0Lol2vroXcBsDt8vwx4Khwcb5tZkNBcNuVeFn4MEffHObev1et+BTzrZcEdiLQvo4FRwKrwD1YB8K6ZneKc2+tx7e2Jxnettd8Bf8WHICFKfQnv1P04cI4ff3i1Eu1144d2+wDgnFsILAQws1eBq51z21s1KQPObPW4gNC+lDKOp79e7yDqDTegkFY7qIA3gfnh+wZMPsbr3iE06ji60+mj4eXXAd8N3x9HaIhofbg/Q1u1+Rrwh77alzZtttODO9s9WjdjW7W5CVjUh/tyAbAOyOvJdeL1d40e2tl+vH3g2DvbtxHaspITvj+gK/1tty4/VmgPf3l+D+wBmgil7bWE/mp9HlgV/mL/9zFeWwy8B2wBfs4HJ3AmAo+Fn3sXOLuP9+e3wBpgNaG/wob21b60abOdnj1qy4t182R4+WpC8ybl9+G+bCb0R9fK8K1HjkDzsD/zwu/VAOwDXuiNfaCdIAkvvya8TjYDCzrrb0c3ndkuIiIRidWjtkREJEoUJCIiEhEFiYiIRERBIiIiEVGQiIhIRBQk0i+Y2eEe/ryHzGxilN6rxUKz+75nZs90NiuumWWb2Q3R+GyRaNDhv9IvmNlh51x6FN8v4D6YYNBTrWs3s98AG51zt3fQvhB41jl3Uk/UJ9IZjUik3zKzPDN70szeCd/mhJefYmZvmtmK8L/jw8uvNrMnzOwZ4EUzO9PMXjWzRRa6jsbjR6/NEF5eHL5/ODyx4iozW2pmg8PLR4cfv2Nm3+3iqOktPpiAMt3M/m5m71ro+hAXh9v8ABgdHsXcFW77jfDnrDaz/4nif6NIpxQk0p/dDfzUOTcD+BTwUHj5+8DpzrmphGbTvaPVa2YBVznnzg4/ngp8FZgIFAFz2vmcNGCpc24y8DrwxVaff3f48zudryg8z9M5hGYXAKgH5jnnphG6Bs6Pw0F2G7DFOTfFOfcNMzsfGAucAkwBppvZ6Z19nki0xOKkjRI7zgUmtpoZNdPMMoAs4DdmNpbQzKYJrV7zknOu9TUf3nbOlQGY2UpCcx0tafM5jXww0eVy4Lzw/Vl8cC2H3wE/OkadKa3eezmha0NAaK6jO8KhECQ0UhnczuvPD99WhB+nEwqW14/xeSJRpSCR/iwOmOWcO9J6oZndC7zinJsX3t/waquna9u8R0Or+y20/zPT5D7Y2XisNh054pybYmZZhALpRuAeQtcfyQOmO+eazGw7kNzO6w34vnPul938XJGo0KYt6c9eJHT9DgDM7Oh021nArvD9qz38/KWENqkBXNpZY+dcNaHL6d5iZgmE6twfDpGzgJHhpoeAjFYvfQG4Jnx9Csws38wGRakPIp1SkEh/kWpmZa1uXyf0S7k4vAN6HaHp/wHuBL5vZm8A8R7W9FXg62b2NjAUqO7sBc65FYRmcr2U0IWfis2shNDo5P1wmwPAG+HDhe9yzr1IaNPZW2a2BljEh4NGxFM6/FfEI+ErNh5xzjkzuxS4zDl3cWevE+lrtI9ExDvTgZ+Hj7SqwofLF4v0BI1IREQkItpHIiIiEVGQiIhIRBQkIiISEQWJiIhEREEiIiIRUZCIiEhE/j/msnoDNELuQgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:52 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>6.673347</th>\n",
       "    <th>5.848070</th>\n",
       "    <th>0.201297</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>5.525470</th>\n",
       "    <th>4.180490</th>\n",
       "    <th>0.287321</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>4.683749</th>\n",
       "    <th>3.861969</th>\n",
       "    <th>0.326021</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>4.202083</th>\n",
       "    <th>3.738675</th>\n",
       "    <th>0.335610</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>3.903042</th>\n",
       "    <th>3.669463</th>\n",
       "    <th>0.338576</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>6</th>\n",
       "    <th>3.682685</th>\n",
       "    <th>3.621780</th>\n",
       "    <th>0.345820</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>7</th>\n",
       "    <th>3.530087</th>\n",
       "    <th>3.596351</th>\n",
       "    <th>0.348234</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>8</th>\n",
       "    <th>3.420229</th>\n",
       "    <th>3.582476</th>\n",
       "    <th>0.347647</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>9</th>\n",
       "    <th>3.348133</th>\n",
       "    <th>3.582009</th>\n",
       "    <th>0.347682</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>10</th>\n",
       "    <th>3.302161</th>\n",
       "    <th>3.582578</th>\n",
       "    <th>0.347544</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# as a rule of thumb, review the plot above and choose the learning rate with the steepest slope to fit the model\n",
    "learn.fit_one_cycle(10, 1e-2, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('tweet_head')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('tweet_head');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete the fine-tuning, we can then unfeeze and launch a new training to fine-tune all layers of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:26 <p><table style='width:300px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>3.271033</th>\n",
       "    <th>3.578148</th>\n",
       "    <th>0.350580</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>3.268870</th>\n",
       "    <th>3.575207</th>\n",
       "    <th>0.350235</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>3.271037</th>\n",
       "    <th>3.573545</th>\n",
       "    <th>0.348027</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>3.255284</th>\n",
       "    <th>3.572066</th>\n",
       "    <th>0.348924</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>3.244405</th>\n",
       "    <th>3.569673</th>\n",
       "    <th>0.348846</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(5, 1e-3, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('tweet_fine_tuned')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LanguageLearner(data=TextLMDataBunch;\n",
       "\n",
       "Train: LabelList\n",
       "y: LMLabel (5485 items)\n",
       "[Category 0, Category 0, Category 0, Category 0, Category 0]...\n",
       "Path: /storage/deep-elon-tweet-generator\n",
       "x: LMTextList (5485 items)\n",
       "[Text xxbos xxmaj please xxunk prior tweets , as that was someone pretending to be me :) xxmaj this is actually me ., Text xxbos xxmaj went to xxmaj xxunk on xxmaj sat to ride bumper cars on ice ! xxmaj no , not the country , xxmaj xxunk 's xxunk in xxmaj van xxmaj xxunk . xxmaj awesome family fun :) http : / / t.co / xxunk, Text xxbos i made the volume on the xxmaj model s http : / / t.co / xxunk m go to 11 . xxmaj now i just need to work in a xxunk xxmaj xxunk ..., Text xxbos xxmaj great xxmaj xxunk quote , arguably better than xxmaj xxunk . xxmaj hearing news of his own death , xxmaj xxunk xxunk the reports were true , only xxunk ., Text xxbos xxmaj that was a total non xxunk btw]...\n",
       "Path: /storage/deep-elon-tweet-generator;\n",
       "\n",
       "Valid: LabelList\n",
       "y: LMLabel (609 items)\n",
       "[Category 0, Category 0, Category 0, Category 0, Category 0]...\n",
       "Path: /storage/deep-elon-tweet-generator\n",
       "x: LMTextList (609 items)\n",
       "[Text xxbos xxmaj very funny post of a xxmaj tesla owner test driving a gasoline car http : / / t.co / xxunk, Text xxbos @danfromtheweb xxmaj ok, Text xxbos xxunk xxunk xxmaj sounds like the xxmaj tesla xxmaj powerwall ( consumer ) and xxmaj powerpack ( utility / commercial ) might be useful, Text xxbos xxunk ca n't always xxunk xxunk and xxunk, Text xxbos xxmaj now with actual picture :) http : / / t.co / xxunk]...\n",
       "Path: /storage/deep-elon-tweet-generator;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): RNNCore(\n",
       "    (encoder): Embedding(3685, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(3685, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(400, 1150)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(1150, 1150)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(1150, 400)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=3685, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=<fastai.layers.FlattenedLoss object at 0x7f4d300a9668>, metrics=[<function accuracy at 0x7f4d31ce3620>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/storage/deep-elon-tweet-generator'), model_dir='models', callback_fns=[<class 'fastai.basic_train.Recorder'>], callbacks=[RNNTrainer(learn=LanguageLearner(data=TextLMDataBunch;\n",
       "\n",
       "Train: LabelList\n",
       "y: LMLabel (5485 items)\n",
       "[Category 0, Category 0, Category 0, Category 0, Category 0]...\n",
       "Path: /storage/deep-elon-tweet-generator\n",
       "x: LMTextList (5485 items)\n",
       "[Text xxbos xxmaj please xxunk prior tweets , as that was someone pretending to be me :) xxmaj this is actually me ., Text xxbos xxmaj went to xxmaj xxunk on xxmaj sat to ride bumper cars on ice ! xxmaj no , not the country , xxmaj xxunk 's xxunk in xxmaj van xxmaj xxunk . xxmaj awesome family fun :) http : / / t.co / xxunk, Text xxbos i made the volume on the xxmaj model s http : / / t.co / xxunk m go to 11 . xxmaj now i just need to work in a xxunk xxmaj xxunk ..., Text xxbos xxmaj great xxmaj xxunk quote , arguably better than xxmaj xxunk . xxmaj hearing news of his own death , xxmaj xxunk xxunk the reports were true , only xxunk ., Text xxbos xxmaj that was a total non xxunk btw]...\n",
       "Path: /storage/deep-elon-tweet-generator;\n",
       "\n",
       "Valid: LabelList\n",
       "y: LMLabel (609 items)\n",
       "[Category 0, Category 0, Category 0, Category 0, Category 0]...\n",
       "Path: /storage/deep-elon-tweet-generator\n",
       "x: LMTextList (609 items)\n",
       "[Text xxbos xxmaj very funny post of a xxmaj tesla owner test driving a gasoline car http : / / t.co / xxunk, Text xxbos @danfromtheweb xxmaj ok, Text xxbos xxunk xxunk xxmaj sounds like the xxmaj tesla xxmaj powerwall ( consumer ) and xxmaj powerpack ( utility / commercial ) might be useful, Text xxbos xxunk ca n't always xxunk xxunk and xxunk, Text xxbos xxmaj now with actual picture :) http : / / t.co / xxunk]...\n",
       "Path: /storage/deep-elon-tweet-generator;\n",
       "\n",
       "Test: None, model=SequentialRNN(\n",
       "  (0): RNNCore(\n",
       "    (encoder): Embedding(3685, 400, padding_idx=1)\n",
       "    (encoder_dp): EmbeddingDropout(\n",
       "      (emb): Embedding(3685, 400, padding_idx=1)\n",
       "    )\n",
       "    (rnns): ModuleList(\n",
       "      (0): WeightDropout(\n",
       "        (module): LSTM(400, 1150)\n",
       "      )\n",
       "      (1): WeightDropout(\n",
       "        (module): LSTM(1150, 1150)\n",
       "      )\n",
       "      (2): WeightDropout(\n",
       "        (module): LSTM(1150, 400)\n",
       "      )\n",
       "    )\n",
       "    (input_dp): RNNDropout()\n",
       "    (hidden_dps): ModuleList(\n",
       "      (0): RNNDropout()\n",
       "      (1): RNNDropout()\n",
       "      (2): RNNDropout()\n",
       "    )\n",
       "  )\n",
       "  (1): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=3685, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       "), opt_func=functools.partial(<class 'torch.optim.adam.Adam'>, betas=(0.9, 0.99)), loss_func=<fastai.layers.FlattenedLoss object at 0x7f4d300a9668>, metrics=[<function accuracy at 0x7f4d31ce3620>], true_wd=True, bn_wd=True, wd=0.01, train_bn=True, path=PosixPath('/storage/deep-elon-tweet-generator'), model_dir='models', callback_fns=[<class 'fastai.basic_train.Recorder'>], callbacks=[...], layer_groups=[Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1150)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1150, 1150)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1150, 400)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): Embedding(3685, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(3685, 400, padding_idx=1)\n",
       "  )\n",
       "  (2): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=3685, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")]), bptt=70, alpha=2.0, beta=1.0, adjust=False)], layer_groups=[Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(400, 1150)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1150, 1150)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): WeightDropout(\n",
       "    (module): LSTM(1150, 400)\n",
       "  )\n",
       "  (1): RNNDropout()\n",
       "), Sequential(\n",
       "  (0): Embedding(3685, 400, padding_idx=1)\n",
       "  (1): EmbeddingDropout(\n",
       "    (emb): Embedding(3685, 400, padding_idx=1)\n",
       "  )\n",
       "  (2): LinearDecoder(\n",
       "    (decoder): Linear(in_features=400, out_features=3685, bias=True)\n",
       "    (output_dp): RNNDropout()\n",
       "  )\n",
       ")])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.load('tweet_fine_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tweet Generation Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# adjust below parameters to test inference (generated tweets)\n",
    "\n",
    "TEXT = \"Robots will\"\n",
    "N_WORDS = 20\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Robots will own a three - character xxup bfr spacecraft companies . this meeting will be that following as no launching xxbos\n",
      "Robots will act on their later designs . one of the most prominent designs that telstar 1 remains with the credit of\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS) for _ in range(N_SENTENCES)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}