5.2 Recurrent Neural Networks in tf and keras.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN in ``tf`` \n",
    "\n",
    "LSTM model in ``tf``:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/tf-lstm.png?raw=true![image.png](attachment:image.png)\" alt=\"\" style=\"width: 600px;\"/> \n",
    "Source: https://cloud.google.com/blog/big-data/2017/01/learn-tensorflow-and-deep-learning-without-a-phd\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GRU model in ``tf``:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/tf-gru.png?raw=true\" alt=\"\" style=\"width: 600px;\"/> \n",
    "Source: https://cloud.google.com/blog/big-data/2017/01/learn-tensorflow-and-deep-learning-without-a-phd\n",
    "<center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning to add in ``tf``\n",
    "\n",
    "Source: http://projects.rajivshah.com/blog/2016/04/05/rnn_addition/ \n",
    "\n",
    "The objective of this code developed by Rajiv Shah is to train a RNN for adding a sequence of integers. In this case we must process all the sequence in order to produce a result. We will use the ``tf`` implementation of the ``seq2seq`` architecure, originally proposed by Sutskever, Vinyals and Le in 2014. \n",
    "\n",
    "The architecture diagram from their paper is:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/seq2seq.png?raw=true\" alt=\"\" style=\"width: 800px;\"/> \n",
    "Source: https://arxiv.org/abs/1409.3215\n",
    "<center>\n",
    "\n",
    "Rectangles are recurrent layers. Encoder receives ``[A, B, C]`` sequence as inputs. We don't care about encoder outputs, only about the hidden state it accumulates while reading the sequence. \n",
    "\n",
    "After input sequence ends, encoder passes its final state to decoder, which receives ``[<EOS>, W, X, Y, Z]`` and is trained to output ``[W, X, Y, Z, <EOS>]``. \n",
    "\n",
    "``tf`` implementation of the ``seq2seq`` allows sequences of different lenghts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sequence-to-sequence, or *Seq2Seq*, is a relatively new paradigm made up of two recurrent networks: an *encoder* which takes the model's input sequence as input and encodes it into a fixed-size *context vector*, and a *decoder*, which uses the context vector from above as a *seed* from which to generate an output sequence.\n",
    "\n",
    "For this reason, Seq2Seq models are often referred to as *encoder-decoder* models.\n",
    "\n",
    "The encoder network's job is to read the input sequence and generate a fixed-dimensional context vector $C$ for the sequence. \n",
    "\n",
    "To do so, the encoder will use a recurrent neural network cell -- usually an LSTM -- to read the input tokens one at a time. The final hidden state of the cell will then become $C$. \n",
    "\n",
    "However, because it's so difficult to compress an arbitrary-length sequence into a single fixed-size vector, the encoder will usually consist of *stacked* LSTMs. The *final* layer's LSTM hidden state will be used as $C$.\n",
    "\n",
    "The decoder is also an LSTM network, but its usage is a little more complex than the encoder network. Essentially, we'd like to use it as a language model that's \"aware\" of the words that it's generated so far *and* of the input. \n",
    "\n",
    "To that end, we'll keep the \"stacked\" LSTM architecture from the encoder, but we'll initialize the hidden state of our first layer with the context vector from above; the decoder will literally use the context of the input to generate an output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When both input sequences and output sequences have the same length, you can implement such models simply with a Keras LSTM or GRU layer (or stack thereof).\n",
    "\n",
    "One caveat of this approach is that it assumes that it is possible to generate ``target[...t]`` given ``input[...t]``. That works in some cases (e.g. adding strings of digits) but does not work for most use cases. In the general case, information about the entire input sequence is necessary in order to start generating the target sequence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating data...\n",
      "Total addition questions: 50000\n",
      "642+1   643 \n"
     ]
    }
   ],
   "source": [
    "'''An implementation of sequence to sequence learning for performing addition\n",
    "Input: \"535+61 \"\n",
    "Output:\"596 \"\n",
    "Padding is handled by using a repeated sentinel character (space)\n",
    "'''\n",
    "\n",
    "from __future__ import print_function\n",
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "import numpy as np\n",
    "from six.moves import range\n",
    "\n",
    "\n",
    "class CharacterTable(object):\n",
    "    \"\"\"Given a set of characters:\n",
    "    + Encode them to a one hot integer representation\n",
    "    + Decode the one hot integer representation to their character output\n",
    "    + Decode a vector of probabilities to their character output\n",
    "    \"\"\"\n",
    "    def __init__(self, chars):\n",
    "        \"\"\"Initialize character table.\n",
    "        # Arguments\n",
    "            chars: Characters that can appear in the input.\n",
    "        \"\"\"\n",
    "        self.chars = sorted(set(chars))\n",
    "        self.char_indices = dict((c, i) for i, c in enumerate(self.chars))\n",
    "        self.indices_char = dict((i, c) for i, c in enumerate(self.chars))\n",
    "\n",
    "    def encode(self, C, num_rows):\n",
    "        \"\"\"One hot encode given string C.\n",
    "        # Arguments\n",
    "            num_rows: Number of rows in the returned one hot encoding. This is\n",
    "                used to keep the # of rows for each data the same.\n",
    "        \"\"\"\n",
    "        x = np.zeros((num_rows, len(self.chars)))\n",
    "        for i, c in enumerate(C):\n",
    "            x[i, self.char_indices[c]] = 1\n",
    "        return x\n",
    "\n",
    "    def decode(self, x, calc_argmax=True):\n",
    "        if calc_argmax:\n",
    "            x = x.argmax(axis=-1)\n",
    "        return ''.join(self.indices_char[x] for x in x)\n",
    "\n",
    "\n",
    "class colors:\n",
    "    ok = '\\033[92m'\n",
    "    fail = '\\033[91m'\n",
    "    close = '\\033[0m'\n",
    "\n",
    "# Parameters for the model and dataset.\n",
    "TRAINING_SIZE = 50000\n",
    "DIGITS = 3\n",
    "REVERSE = True\n",
    "\n",
    "# Maximum length of input is 'int + int' (e.g., '345+678'). Maximum length of\n",
    "# int is DIGITS.\n",
    "MAXLEN = DIGITS + 1 + DIGITS\n",
    "\n",
    "# All the numbers, plus sign and space for padding.\n",
    "chars = '0123456789+ '\n",
    "ctable = CharacterTable(chars)\n",
    "\n",
    "questions = []\n",
    "expected = []\n",
    "seen = set()\n",
    "print('Generating data...')\n",
    "while len(questions) < TRAINING_SIZE:\n",
    "    f = lambda: int(''.join(np.random.choice(list('0123456789'))\n",
    "                    for i in range(np.random.randint(1, DIGITS + 1))))\n",
    "    a, b = f(), f()\n",
    "    # Skip any addition questions we've already seen\n",
    "    # Also skip any such that x+Y == Y+x (hence the sorting).\n",
    "    key = tuple(sorted((a, b)))\n",
    "    if key in seen:\n",
    "        continue\n",
    "    seen.add(key)\n",
    "    # Pad the data with spaces such that it is always MAXLEN.\n",
    "    q = '{}+{}'.format(a, b)\n",
    "    query = q + ' ' * (MAXLEN - len(q))\n",
    "    ans = str(a + b)\n",
    "    # Answers can be of maximum size DIGITS + 1.\n",
    "    ans += ' ' * (DIGITS + 1 - len(ans))\n",
    "    questions.append(query)\n",
    "    expected.append(ans)\n",
    "print('Total addition questions:', len(questions))\n",
    "print(questions[3], expected[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vectorization...\n",
      "Training Data:\n",
      "(45000, 7, 12)\n",
      "(45000, 4, 12)\n",
      "Validation Data:\n",
      "(5000, 7, 12)\n",
      "(5000, 4, 12)\n",
      "Build model...\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_3 (LSTM)                (None, 128)               72192     \n",
      "_________________________________________________________________\n",
      "repeat_vector_2 (RepeatVecto (None, 4, 128)            0         \n",
      "_________________________________________________________________\n",
      "lstm_4 (LSTM)                (None, 4, 128)            131584    \n",
      "_________________________________________________________________\n",
      "time_distributed_2 (TimeDist (None, 4, 12)             1548      \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 4, 12)             0         \n",
      "=================================================================\n",
      "Total params: 205,324\n",
      "Trainable params: 205,324\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 1\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      "45000/45000 [==============================] - 36s 811us/step - loss: 1.8935 - acc: 0.3212 - val_loss: 1.7999 - val_acc: 0.3377\n",
      "Q   206+3 T 605  \u001b[91m☒\u001b[0m 22  \n",
      "Q   5+426 T 629  \u001b[91m☒\u001b[0m 22  \n",
      "Q   9+951 T 168  \u001b[91m☒\u001b[0m 100 \n",
      "Q  49+532 T 329  \u001b[91m☒\u001b[0m 121 \n",
      "Q  63+476 T 710  \u001b[91m☒\u001b[0m 121 \n",
      "Q  996+26 T 761  \u001b[91m☒\u001b[0m 100 \n",
      "Q 569+667 T 1731 \u001b[91m☒\u001b[0m 1116\n",
      "Q 295+051 T 742  \u001b[91m☒\u001b[0m 101 \n",
      "Q  86+646 T 714  \u001b[91m☒\u001b[0m 121 \n",
      "Q  45+443 T 398  \u001b[91m☒\u001b[0m 121 \n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 2\n",
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/1\n",
      " 9088/45000 [=====>........................] - ETA: 24s - loss: 1.7922 - acc: 0.3376"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-13-e1e303710aa3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     72\u001b[0m               \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mBATCH_SIZE\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     73\u001b[0m               \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 74\u001b[0;31m               validation_data=(x_val, y_val))\n\u001b[0m\u001b[1;32m     75\u001b[0m     \u001b[0;31m# Select 10 samples from the validation set at random so we can visualize\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     76\u001b[0m     \u001b[0;31m# errors.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/models.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m    963\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    964\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 965\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m    966\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    967\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m   1667\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1668\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1669\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m   1670\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1671\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36m_fit_loop\u001b[0;34m(self, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)\u001b[0m\n\u001b[1;32m   1204\u001b[0m                         \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1205\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1206\u001b[0;31m                     \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1207\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1208\u001b[0m                         \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2473\u001b[0m         \u001b[0msession\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2474\u001b[0m         updated = session.run(fetches=fetches, feed_dict=feed_dict,\n\u001b[0;32m-> 2475\u001b[0;31m                               **self.session_kwargs)\n\u001b[0m\u001b[1;32m   2476\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mupdated\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2477\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "print('Vectorization...')\n",
    "x = np.zeros((len(questions), MAXLEN, len(chars)), dtype=np.bool)\n",
    "y = np.zeros((len(questions), DIGITS + 1, len(chars)), dtype=np.bool)\n",
    "for i, sentence in enumerate(questions):\n",
    "    x[i] = ctable.encode(sentence, MAXLEN)\n",
    "for i, sentence in enumerate(expected):\n",
    "    y[i] = ctable.encode(sentence, DIGITS + 1)\n",
    "\n",
    "# Shuffle (x, y) \n",
    "indices = np.arange(len(y))\n",
    "np.random.shuffle(indices)\n",
    "x = x[indices]\n",
    "y = y[indices]\n",
    "\n",
    "# Explicitly set apart 10% for validation data that we never train over.\n",
    "split_at = len(x) - len(x) // 10\n",
    "(x_train, x_val) = x[:split_at], x[split_at:]\n",
    "(y_train, y_val) = y[:split_at], y[split_at:]\n",
    "\n",
    "print('Training Data:')\n",
    "print(x_train.shape)\n",
    "print(y_train.shape)\n",
    "\n",
    "print('Validation Data:')\n",
    "print(x_val.shape)\n",
    "print(y_val.shape)\n",
    "\n",
    "# Try replacing GRU, or SimpleRNN.\n",
    "RNN = layers.LSTM\n",
    "HIDDEN_SIZE = 128\n",
    "BATCH_SIZE = 128\n",
    "LAYERS = 1\n",
    "\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "\n",
    "# \"Encode\" the input sequence using an RNN, producing an output of HIDDEN_SIZE.\n",
    "# Note: In a situation where your input sequences have a variable length,\n",
    "# use input_shape=(None, num_feature).\n",
    "# Returns the last output in the output sequence\n",
    "model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))\n",
    "\n",
    "# As the decoder RNN's input, repeatedly provide with the last hidden state of\n",
    "# RNN for each time step. Repeat 'DIGITS + 1' times as that's the maximum\n",
    "# length of output, e.g., when DIGITS=3, max output is 999+999=1998.\n",
    "model.add(layers.RepeatVector(DIGITS + 1))\n",
    "\n",
    "# The decoder RNN could be multiple layers stacked or a single layer.\n",
    "for _ in range(LAYERS):\n",
    "    # By setting return_sequences to True, return not only the last output but\n",
    "    # all the outputs so far in the form of (num_samples, timesteps,\n",
    "    # output_dim). This is necessary as TimeDistributed in the below expects\n",
    "    # the first dimension to be the timesteps.\n",
    "    model.add(RNN(HIDDEN_SIZE, return_sequences=True))\n",
    "\n",
    "# Apply a dense layer to the every temporal slice of an input. For each of step\n",
    "# of the output sequence, decide which character should be chosen.\n",
    "model.add(layers.TimeDistributed(layers.Dense(len(chars))))\n",
    "model.add(layers.Activation('softmax'))\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "model.summary()\n",
    "\n",
    "# Train the model each generation and show predictions against the validation\n",
    "# dataset.\n",
    "for iteration in range(1, 200):\n",
    "    print()\n",
    "    print('-' * 50)\n",
    "    print('Iteration', iteration)\n",
    "    model.fit(x_train, y_train,\n",
    "              batch_size=BATCH_SIZE,\n",
    "              epochs=1,\n",
    "              validation_data=(x_val, y_val))\n",
    "    # Select 10 samples from the validation set at random so we can visualize\n",
    "    # errors.\n",
    "    for i in range(10):\n",
    "        ind = np.random.randint(0, len(x_val))\n",
    "        rowx, rowy = x_val[np.array([ind])], y_val[np.array([ind])]\n",
    "        preds = model.predict_classes(rowx, verbose=0)\n",
    "        q = ctable.decode(rowx[0])\n",
    "        correct = ctable.decode(rowy[0])\n",
    "        guess = ctable.decode(preds[0], calc_argmax=False)\n",
    "        print('Q', q[::-1] if REVERSE else q, end=' ')\n",
    "        print('T', correct, end=' ')\n",
    "        if correct == guess:\n",
    "            print(colors.ok + '☑' + colors.close, end=' ')\n",
    "        else:\n",
    "            print(colors.fail + '☒' + colors.close, end=' ')\n",
    "        print(guess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example\n",
    "\n",
    "A Recurrent Neural Network (LSTM) implementation example using TensorFlow library.\n",
    "\n",
    "This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)\n",
    "Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf\n",
    "\n",
    "Author: Aymeric Damien\n",
    "Project: https://github.com/aymericdamien/TensorFlow-Examples/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.contrib import rnn\n",
    "\n",
    "# Import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To classify images using a recurrent neural network, we consider every image row as a sequence of pixels. \n",
    "\n",
    "Because MNIST image shape is $28 \\times 28$ px, we will then handle 28 sequences of 28 steps for every sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training Parameters\n",
    "learning_rate = 0.001\n",
    "training_steps = 10000\n",
    "batch_size = 128\n",
    "display_step = 200\n",
    "\n",
    "# Network Parameters\n",
    "num_input = 28 # MNIST data input (img shape: 28*28)\n",
    "timesteps = 28 # timesteps\n",
    "num_hidden = 128 # hidden layer num of features\n",
    "num_classes = 10 # MNIST total classes (0-9 digits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tf Graph input\n",
    "X = tf.placeholder(\"float\", [None, timesteps, num_input])\n",
    "Y = tf.placeholder(\"float\", [None, num_classes])\n",
    "\n",
    "# Define weights\n",
    "weights = {\n",
    "    'out': tf.Variable(tf.random_normal([num_hidden, num_classes]))\n",
    "}\n",
    "biases = {\n",
    "    'out': tf.Variable(tf.random_normal([num_classes]))\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-17-3f42a567fcfd>:24: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See tf.nn.softmax_cross_entropy_with_logits_v2.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def RNN(x, weights, biases):\n",
    "\n",
    "    # Prepare data shape to match `rnn` function requirements\n",
    "    # Current data input shape: (batch_size, timesteps, n_input)\n",
    "    # Required shape: 'timesteps' tensors list of shape (batch_size, n_input)\n",
    "\n",
    "    # Unstack to get a list of 'timesteps' tensors of shape (batch_size, n_input)\n",
    "    x = tf.unstack(x, timesteps, 1)\n",
    "\n",
    "    # Define a lstm cell with tensorflow\n",
    "    lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
    "\n",
    "    # Get lstm cell output\n",
    "    outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)\n",
    "\n",
    "    # Linear activation, using rnn inner loop last output\n",
    "    return tf.matmul(outputs[-1], weights['out']) + biases['out']\n",
    "\n",
    "logits = RNN(X, weights, biases)\n",
    "prediction = tf.nn.softmax(logits)\n",
    "\n",
    "# Define loss and optimizer\n",
    "loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n",
    "    logits=logits, labels=Y))\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "train_op = optimizer.minimize(loss_op)\n",
    "\n",
    "# Evaluate model (with test logits, for dropout to be disabled)\n",
    "correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
    "\n",
    "# Initialize the variables (i.e. assign their default value)\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 1, Minibatch Loss= 2.5969, Training Accuracy= 0.102\n",
      "Step 200, Minibatch Loss= 2.0649, Training Accuracy= 0.344\n",
      "Step 400, Minibatch Loss= 2.0016, Training Accuracy= 0.289\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-49b87c1a1010>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m         \u001b[0mbatch_x\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch_x\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimesteps\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_input\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0;31m# Run optimization op (backprop)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m         \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_op\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbatch_x\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbatch_y\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mdisplay_step\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m             \u001b[0;31m# Calculate batch loss and accuracy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "\n",
    "    # Run the initializer\n",
    "    sess.run(init)\n",
    "\n",
    "    for step in range(1, training_steps+1):\n",
    "        batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
    "        # Reshape data to get 28 seq of 28 elements\n",
    "        batch_x = batch_x.reshape((batch_size, timesteps, num_input))\n",
    "        # Run optimization op (backprop)\n",
    "        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n",
    "        if step % display_step == 0 or step == 1:\n",
    "            # Calculate batch loss and accuracy\n",
    "            loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n",
    "                                                                 Y: batch_y})\n",
    "            print(\"Step \" + str(step) + \", Minibatch Loss= \" + \\\n",
    "                  \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n",
    "                  \"{:.3f}\".format(acc))\n",
    "\n",
    "    print(\"Optimization Finished!\")\n",
    "\n",
    "    # Calculate accuracy for 128 mnist test images\n",
    "    test_len = 128\n",
    "    test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))\n",
    "    test_label = mnist.test.labels[:test_len]\n",
    "    print(\"Testing Accuracy:\", \\\n",
    "        sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bidirectional LSTM\n",
    "\n",
    "So  far,  we  have  focused  on  RNNs  that  look  into  the  past    to predict future values in the sequence, but not to  to make predictions based on future values by reading throught the sequence backwards?\n",
    "\n",
    "**Bi-directional  deep  neural networs**,  at each time-step, $t$,  maintain two hidden layers, one for the left-to-right propagation and another for the right-to-left  propagation (hence, consuming twice  as  much  memory  space).   \n",
    "\n",
    "The final classification result, $\\hat{y}$, is generated through combining the score results produced by both RNN hidden layers.\n",
    "\n",
    "<img src = \"https://github.com/DataScienceUB/DeepLearningfromScratch2018/blob/master/images/t9.png?raw=true\"  width = \"600\">\n",
    "\n",
    "The equations are (arrows are for designing left-to-right and right-to-left tensors):\n",
    "\n",
    "$$\n",
    "\t\\overrightarrow{h}_t = f(\\overrightarrow{W} x_t + \\overrightarrow{V} \\overrightarrow{h}_{t-1} + \\overrightarrow{b})\n",
    "$$\n",
    "$$\n",
    "\t\\overleftarrow{h}_t = f(\\overleftarrow{W} x_t + \\overleftarrow{V} \\overleftarrow{h}_{t+1} + \\overleftarrow{b})\n",
    "$$\n",
    "$$\n",
    "\t\\hat{y}_t = g(U h_t + c) = g(U [\\overrightarrow{h}_t; \\overleftarrow{h}_t] + c)\n",
    "$$\n",
    "\n",
    "$[\\overrightarrow{h}_t; \\overleftarrow{h}_t]$ summarizes the past and future of a single element of the sequence.\n",
    "\n",
    "Biderectional RNNs can be stacked as usual!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Name generation with LSTM\n",
    "\n",
    "We are going to train RNN \"character-level\" language models. \n",
    "\n",
    "That is, we’ll give the RNN a huge chunk of text and ask it to model the probability distribution of the next character in the sequence given a sequence of previous characters. This will then allow us to generate new text one character at a time.\n",
    "\n",
    "We will encode each character into a vector using ``1-of-k`` encoding (i.e. all zero except for a single one at the index of the character in the vocabulary), and feed them into the RNN one at a time. \n",
    "\n",
    "At test time, we will feed a character into the RNN and get a distribution over what characters are likely to come next. We sample from this distribution, and feed it right back in to get the next letter. Repeat this process and you’re sampling text!\n",
    "\n",
    "We can also play with the temperature of the Softmax during sampling. Decreasing the temperature from 1 to some lower number (e.g. 0.5) makes the RNN more confident, but also more conservative in its samples. Conversely, higher temperatures will give more diversity but at cost of more mistakes.\n",
    "\n",
    "> Remider: **Softmax**\n",
    ">$$\n",
    "\tP(y = j | \\mathbf{x}) = \\frac{\\exp(\\mathbf{x}^T \\mathbf{w}_j)/\\tau}{\\sum_{k=1}^K \\exp(\\mathbf{x}^T \\mathbf{w}_k)/\\tau}\n",
    "$$\n",
    "> $\\tau$ is called *temperature*. For high temperatures ( $ \\tau \\to \\infty $ ), all $y$ have nearly the same probability. For a low temperature ( $ \\tau \\to 0^{+}$), the probability of the most probable $y$ tends to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to process sequences of symbols with RNN we need to represent these symbols by numbers.\n",
    "\n",
    "Let's suppose we have $|V|$ different symbols. The most simple representation is the **one-hot vector**: Represent every word as an $\\mathbb{R}^{|V|\\times1}$ vector with all $0$s and one $1$ at the index of that word in the sorted english language. Symbol vectors in this type of encoding would appear as the following:\n",
    "\n",
    "$$w^{s_1} = \\left[ \\begin{array}{c} 1 \\\\ 0 \\\\ 0 \\\\ \\vdots \\\\ 0 \\end{array} \\right], w^{s_2} = \\left[ \\begin{array}{c} 0 \\\\ 1 \\\\ 0 \\\\ \\vdots \\\\ 0 \\end{array} \\right], w^{s_3} = \\left[ \\begin{array}{c} 0 \\\\ 0 \\\\ 1 \\\\ \\vdots \\\\ 0 \\end{array} \\right], \\cdots \n",
    "w^{s_{|V|}} = \\left[ \\begin{array}{c} 0 \\\\ 0 \\\\ 0 \\\\ \\vdots \\\\ 1 \\end{array} \\right] $$\n",
    "\n",
    "We represent each symbol as a completely independent entity. This symbol representation does not give us directly any notion of similarity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train our model we need text to learn from a large dataset of names. Fortunately we don’t need any labels to train a language model, just raw text.\n",
    "\n",
    "Places names: You can download 52,700 Catalan names from a dataset available on http://territori.gencat.cat/ca/01_departament/11_normativa_i_documentacio/03_documentacio/02_territori_i_mobilitat/cartografia/nomenclator_oficial_de_toponimia_de_catalunya/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-06-13 18:10:32--  https://raw.githubusercontent.com/DataScienceUB/DeepLearningfromScratch2018/master/files/womennamesbarcelona.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.132.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.132.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 47544 (46K) [text/plain]\n",
      "Saving to: 'STDOUT'\n",
      "\n",
      "-                   100%[===================>]  46.43K  --.-KB/s    in 0.03s   \n",
      "\n",
      "2018-06-13 18:10:33 (1.30 MB/s) - written to stdout [47544/47544]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -O - 'https://raw.githubusercontent.com/DataScienceUB/DeepLearningfromScratch2018/master/files/womennamesbarcelona.txt' > womennamesbarcelona.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus length: 47527\n",
      "total chars: 30\n",
      "nb sequences: 15836\n",
      "Vectorization...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation\n",
    "from keras.layers import LSTM\n",
    "from keras.optimizers import RMSprop\n",
    "import numpy as np\n",
    "import random\n",
    "import sys\n",
    "\n",
    "import codecs\n",
    "f = codecs.open('womennamesbarcelona.txt', \"r\", \"utf-8\")\n",
    "string = f.read()\n",
    "string.encode('utf-8')\n",
    "text = string.lower()\n",
    "\n",
    "# text = text.replace(\"\\n\", \" \")\n",
    "    \n",
    "print('corpus length:', len(text))\n",
    "\n",
    "chars = sorted(list(set(text)))\n",
    "print('total chars:', len(chars))\n",
    "char_indices = dict((c, i) for i, c in enumerate(chars))\n",
    "indices_char = dict((i, c) for i, c in enumerate(chars))\n",
    "\n",
    "# cut the text in semi-redundant sequences of maxlen characters\n",
    "maxlen = 20\n",
    "step = 3\n",
    "sentences = []\n",
    "next_chars = []\n",
    "for i in range(0, len(text) - maxlen, step):\n",
    "    sentences.append(text[i: i + maxlen])\n",
    "    next_chars.append(text[i + maxlen])\n",
    "print('nb sequences:', len(sentences))\n",
    "\n",
    "print('Vectorization...')\n",
    "X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)\n",
    "y = np.zeros((len(sentences), len(chars)), dtype=np.bool)\n",
    "for i, sentence in enumerate(sentences):\n",
    "    for t, char in enumerate(sentence):\n",
    "        X[i, t, char_indices[char]] = 1\n",
    "    y[i, char_indices[next_chars[i]]] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build model...\n"
     ]
    }
   ],
   "source": [
    "# build the model\n",
    "\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(LSTM(64, \n",
    "               input_shape=(20, 30), \n",
    "               dropout=0.2, \n",
    "               recurrent_dropout=0.2))\n",
    "\n",
    "model.add(Dense(len(chars)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "optimizer = RMSprop(lr=0.01)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest way to use the Keras LSTM model to make predictions is to first start off with a seed sequence as input, generate the next character then update the seed sequence to add the generated character on the end and trim off the first character. \n",
    "\n",
    "This process is repeated for as long as we want to predict new characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "--------------------------------------------------\n",
      "Iteration 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/keras/models.py:944: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.\n",
      "  warnings.warn('The `nb_epoch` argument in `fit` '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 6s 372us/step - loss: 2.9291\n",
      "----- Generating with seed: \"elena gabriela elena\"\n",
      "\n",
      "----- diversity: 0.5\n",
      " eria\n",
      "a\n",
      "çeria arña\n",
      "maria\n",
      "pene\n",
      "a\n",
      "qjara\n",
      "acia\n",
      "anañja\n",
      "\n",
      "\n",
      "----- diversity: 1.0\n",
      "yxa sçebln\n",
      "slariana\n",
      "sfrj eanenh\n",
      "ada\n",
      "eezqyyazuba\n",
      "hj\n",
      "\n",
      "----- diversity: 1.2\n",
      "\n",
      "w oaqdqqiy\n",
      "xtonzoza\n",
      "velia wbsz \n",
      "ç\n",
      "icgckionç nivar\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 2\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 6s 372us/step - loss: 2.6633\n",
      "----- Generating with seed: \" maria alba marina a\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "rina\n",
      "waria\n",
      "ryska\n",
      "saria gelina\n",
      "aria licia\n",
      "vinelna\n",
      "ñ\n",
      "\n",
      "----- diversity: 1.0\n",
      "irian\n",
      "tea\n",
      "hvia kazgnae\n",
      "exjza zñib\n",
      "xlaç\n",
      "drgaeçxa\n",
      "ñx\n",
      "\n",
      "----- diversity: 1.2\n",
      "a\n",
      "sgninnñrkalaed\n",
      "ytñpiyfknçgina\n",
      "rlydz g\n",
      " umefsa\n",
      "ni\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 3\n",
      "Epoch 1/1\n",
      "15836/15836 [==============================] - 7s 413us/step - loss: 2.6141\n",
      "----- Generating with seed: \"des carmen montserra\"\n",
      "\n",
      "----- diversity: 0.5\n",
      "\n",
      "rkelia\n",
      "anzisa\n",
      "saria\n",
      "laciea\n",
      "ana\n",
      "arenia\n",
      "agatia\n",
      "esa \n",
      "\n",
      "----- diversity: 1.0\n",
      "gñuhika o\n",
      "e pfrawfa\n",
      "skviawsaban\n",
      "narfa anaeqcaduina\n",
      "\n",
      "----- diversity: 1.2\n",
      "\n",
      "dzfiyensrçwangghañygsvprtakullazw\n",
      "\n",
      "vwitejzlparraw\n",
      "\n",
      "--------------------------------------------------\n",
      "Iteration 4\n",
      "Epoch 1/1\n",
      " 6912/15836 [============>.................] - ETA: 4s - loss: 2.5541"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-7-cfcd02269284>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     13\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'-'\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m50\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Iteration'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0miteration\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m     \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m256\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnb_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m     \u001b[0mstart_index\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mmaxlen\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/models.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m    963\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    964\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 965\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m    966\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    967\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m   1667\u001b[0m                               \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1668\u001b[0m                               \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1669\u001b[0;31m                               validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m   1670\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1671\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/engine/training.py\u001b[0m in \u001b[0;36m_fit_loop\u001b[0;34m(self, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)\u001b[0m\n\u001b[1;32m   1204\u001b[0m                         \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1205\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1206\u001b[0;31m                     \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1207\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1208\u001b[0m                         \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2473\u001b[0m         \u001b[0msession\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_session\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2474\u001b[0m         updated = session.run(fetches=fetches, feed_dict=feed_dict,\n\u001b[0;32m-> 2475\u001b[0;31m                               **self.session_kwargs)\n\u001b[0m\u001b[1;32m   2476\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mupdated\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2477\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    893\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    894\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 895\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    896\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    897\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1126\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1128\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1129\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1342\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1343\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[0;32m-> 1344\u001b[0;31m                            options, run_metadata)\n\u001b[0m\u001b[1;32m   1345\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1346\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1348\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1349\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1350\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1351\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1352\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1327\u001b[0m           return tf_session.TF_Run(session, options,\n\u001b[1;32m   1328\u001b[0m                                    \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1329\u001b[0;31m                                    status, run_metadata)\n\u001b[0m\u001b[1;32m   1330\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1331\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "def sample(preds, temperature=1.0):\n",
    "    # helper function to sample an index from a probability array\n",
    "    preds = np.asarray(preds).astype('float64')\n",
    "    preds = np.log(preds) / temperature\n",
    "    exp_preds = np.exp(preds)\n",
    "    preds = exp_preds / np.sum(exp_preds)\n",
    "    probas = np.random.multinomial(1, preds, 1)\n",
    "    return np.argmax(probas)\n",
    "\n",
    "# train the model, output generated text after each iteration\n",
    "for iteration in range(1, 60):\n",
    "    print()\n",
    "    print('-' * 50)\n",
    "    print('Iteration', iteration)\n",
    "    model.fit(X, y, batch_size=256, epochs=1)\n",
    "\n",
    "    start_index = random.randint(0, len(text) - maxlen - 1)\n",
    "    generated = ''\n",
    "    sentence = text[start_index: start_index + maxlen]\n",
    "    generated += sentence\n",
    "    print('----- Generating with seed: \"' + sentence.replace(\"\\n\", \" \") + '\"')\n",
    "        \n",
    "    for diversity in [0.5, 1.0, 1.2]:\n",
    "        print()\n",
    "        print('----- diversity:', diversity)\n",
    "        for i in range(50):\n",
    "            \n",
    "            x = np.zeros((1, maxlen, len(chars)))\n",
    "            for t, char in enumerate(sentence):\n",
    "                x[0, t, char_indices[char]] = 1.\n",
    "\n",
    "            preds = model.predict(x, verbose=0)[0]\n",
    "            next_index = sample(preds, diversity)\n",
    "            next_char = indices_char[next_index]\n",
    "\n",
    "            generated += next_char\n",
    "            sentence = sentence[1:] + next_char\n",
    "\n",
    "            sys.stdout.write(next_char)\n",
    "            sys.stdout.flush()\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'lianda' in text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In the case of places, after several hours you can generate names such as:\n",
    "\n",
    "+ Alzinetes, torrent de les\n",
    "+ Alzinetes, vall de les\n",
    "+ **Alzinó, Mas d'**\n",
    "+ Alzinosa, collada de l'\n",
    "+ Alzinosa, font de l'\n",
    "\n",
    "-\n",
    "\n",
    "+ Benavent, roc de\n",
    "+ Benaviure, Cal\n",
    "+ **Benca**\n",
    "+ Bendiners, pla de\n",
    "+ Benedi, roc del\n",
    "\n",
    "-\n",
    "\n",
    "+ Fiola, la\n",
    "+ Fiola, puig de la\n",
    "+ **Fiper, Granja del**\n",
    "+ Firassa, Finca\n",
    "+ Firell\n",
    "\n",
    "-\n",
    "\n",
    "+ Regueret, lo\n",
    "+ Regueret, lo\n",
    "+ **Regueró**\n",
    "+ Reguerols, els\n",
    "+ Reguerons, els\n",
    "\n",
    "-\n",
    "\n",
    "+ Vallverdú, Mas de\n",
    "+ Vallverdú, serrat de\n",
    "+ **Vallvicamanyà**\n",
    "+ Vallvidrera\n",
    "+ Vallvidrera, riera de\n",
    "\n",
    "-\n",
    "\n",
    "+ Terraubella, Corral de\n",
    "+ Terraubes\n",
    "+ **Terravanca**\n",
    "+ Terrer Nou, Can\n",
    "+ Terrer Roig, lo\n",
    "\n",
    "where names in **bold** are generated and other names are the nearest neighbours (in the training dataset) of the generated name."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}