NER_LSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Recognize named entities on Twitter with LSTMs\n",
    "\n",
    "A recurrent neural network is used to solve Named Entity Recognition (NER) problem (recognizing named entities from Twitter). NER is a common task in natural language processing systems. It serves for extraction such entities from the text as persons, organizations, locations, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the Twitter Named Entity Recognition corpus\n",
    "\n",
    "We will work with a corpus, which contains twits with NE tags. Every line of a file contains a pair of a token (word/punctuation symbol) and a tag, separated by a whitespace. Different tweets are separated by an empty line.\n",
    "\n",
    "The function *read_data* reads a corpus from the *file_path* and returns two lists: one with tokens and one with the corresponding tags. This function also replaces a user's nickname to `<USR>` token and any URL to `<URL>` token (thinking that a URL and a nickname are just strings which start with *http://* or *https://* in case of URLs and a *@* symbol for nicknames)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_data(file_path):\n",
    "    tokens = []\n",
    "    tags = []\n",
    "    \n",
    "    tweet_tokens = []\n",
    "    tweet_tags = []\n",
    "    for line in open(file_path, encoding='utf-8'):\n",
    "        line = line.strip()\n",
    "        if not line:\n",
    "            if tweet_tokens:\n",
    "                tokens.append(tweet_tokens)\n",
    "                tags.append(tweet_tags)\n",
    "            tweet_tokens = []\n",
    "            tweet_tags = []\n",
    "        else:\n",
    "            token, tag = line.split()\n",
    "            # Replace all urls with <URL> token\n",
    "            # Replace all users with <USR> token\n",
    "\n",
    "            \n",
    "            if(token.startswith('@')):\n",
    "                token = '<USR>'\n",
    "            elif(token.startswith('http://') or token.startswith('https://')):\n",
    "                token = '<URL>'\n",
    "            tweet_tokens.append(token)\n",
    "            tweet_tags.append(tag)\n",
    "            \n",
    "    return tokens, tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, load three separate parts of the dataset:\n",
    " - *train* data for training the model;\n",
    " - *validation* data for evaluation and hyperparameters tuning;\n",
    " - *test* data for final evaluation of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_tokens, train_tags = read_data('data/train.txt')\n",
    "validation_tokens, validation_tags = read_data('data/validation.txt')\n",
    "test_tokens, test_tags = read_data('data/test.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should always understand what kind of data you deal with. For this purpose, you can print the data running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RT\tO\n",
      "<USR>\tO\n",
      ":\tO\n",
      "Online\tO\n",
      "ticket\tO\n",
      "sales\tO\n",
      "for\tO\n",
      "Ghostland\tB-musicartist\n",
      "Observatory\tI-musicartist\n",
      "extended\tO\n",
      "until\tO\n",
      "6\tO\n",
      "PM\tO\n",
      "EST\tO\n",
      "due\tO\n",
      "to\tO\n",
      "high\tO\n",
      "demand\tO\n",
      ".\tO\n",
      "Get\tO\n",
      "them\tO\n",
      "before\tO\n",
      "they\tO\n",
      "sell\tO\n",
      "out\tO\n",
      "...\tO\n",
      "\n",
      "Apple\tB-product\n",
      "MacBook\tI-product\n",
      "Pro\tI-product\n",
      "A1278\tI-product\n",
      "13.3\tI-product\n",
      "\"\tI-product\n",
      "Laptop\tI-product\n",
      "-\tI-product\n",
      "MD101LL/A\tI-product\n",
      "(\tO\n",
      "June\tO\n",
      ",\tO\n",
      "2012\tO\n",
      ")\tO\n",
      "-\tO\n",
      "Full\tO\n",
      "read\tO\n",
      "by\tO\n",
      "eBay\tB-company\n",
      "<URL>\tO\n",
      "<URL>\tO\n",
      "\n",
      "Happy\tO\n",
      "Birthday\tO\n",
      "<USR>\tO\n",
      "!\tO\n",
      "May\tO\n",
      "Allah\tB-person\n",
      "s.w.t\tO\n",
      "bless\tO\n",
      "you\tO\n",
      "with\tO\n",
      "goodness\tO\n",
      "and\tO\n",
      "happiness\tO\n",
      ".\tO\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(3):\n",
    "    for token, tag in zip(train_tokens[i], train_tags[i]):\n",
    "        print('%s\\t%s' % (token, tag))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare dictionaries\n",
    "\n",
    "To train a neural network, we will use two mappings: \n",
    "- {token}$\\to${token id}: address the row in embeddings matrix for the current token;\n",
    "- {tag}$\\to${tag id}: one-hot ground truth probability distribution vectors for computing the loss at the output of the network.\n",
    "\n",
    "Now you need to implement the function *build_dict* which will return {token or tag}$\\to${index} and vice versa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_dict(tokens_or_tags, special_tokens):\n",
    "    \"\"\"\n",
    "        tokens_or_tags: a list of lists of tokens or tags\n",
    "        special_tokens: some special tokens\n",
    "    \"\"\"\n",
    "    # Create a dictionary with default value 0\n",
    "    tok2idx = defaultdict(lambda: 0)\n",
    "    idx2tok = []\n",
    "    \n",
    "    # Create mappings from tokens to indices and vice versa\n",
    "    # Add special tokens to dictionaries\n",
    "    # The first special token must have index 0\n",
    "    \n",
    "    \n",
    "    idx = 0\n",
    "    for token in special_tokens:\n",
    "        idx2tok.append(token)\n",
    "        tok2idx[token] = idx\n",
    "        idx += 1\n",
    "\n",
    "    for token_list in tokens_or_tags:\n",
    "        for token in token_list:\n",
    "            if token not in tok2idx:\n",
    "                idx2tok.append(token)\n",
    "                tok2idx[token] = idx\n",
    "                idx += 1\n",
    "    \n",
    "    return tok2idx, idx2tok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After implementing the function *build_dict* you can make dictionaries for tokens and tags. Special tokens in our case will be:\n",
    " - `<UNK>` token for out of vocabulary tokens;\n",
    " - `<PAD>` token for padding sentence to the same length when we create batches of sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "special_tokens = ['<UNK>', '<PAD>']\n",
    "special_tags = ['O']\n",
    "\n",
    "# Create dictionaries \n",
    "token2idx, idx2token = build_dict(train_tokens + validation_tokens, special_tokens)\n",
    "tag2idx, idx2tag = build_dict(train_tags, special_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next additional functions will help you to create the mapping between tokens and ids for a sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def words2idxs(tokens_list):\n",
    "    return [token2idx[word] for word in tokens_list]\n",
    "\n",
    "def tags2idxs(tags_list):\n",
    "    return [tag2idx[tag] for tag in tags_list]\n",
    "\n",
    "def idxs2words(idxs):\n",
    "    return [idx2token[idx] for idx in idxs]\n",
    "\n",
    "def idxs2tags(idxs):\n",
    "    return [idx2tag[idx] for idx in idxs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate batches\n",
    "\n",
    "Neural Networks are usually trained with batches. It means that weight updates of the network are based on several sequences at every single time. The tricky part is that all sequences within a batch need to have the same length. So we will pad them with a special `<PAD>` token. It is also a good practice to provide RNN with sequence lengths, so it can skip computations for padding parts. We provide the batching function *batches_generator* readily available for you to save time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def batches_generator(batch_size, tokens, tags,\n",
    "                      shuffle=True, allow_smaller_last_batch=True):\n",
    "    \"\"\"Generates padded batches of tokens and tags.\"\"\"\n",
    "    \n",
    "    n_samples = len(tokens)\n",
    "    if shuffle:\n",
    "        order = np.random.permutation(n_samples)\n",
    "    else:\n",
    "        order = np.arange(n_samples)\n",
    "\n",
    "    n_batches = n_samples // batch_size\n",
    "    if allow_smaller_last_batch and n_samples % batch_size:\n",
    "        n_batches += 1\n",
    "\n",
    "    for k in range(n_batches):\n",
    "        batch_start = k * batch_size\n",
    "        batch_end = min((k + 1) * batch_size, n_samples)\n",
    "        current_batch_size = batch_end - batch_start\n",
    "        x_list = []\n",
    "        y_list = []\n",
    "        max_len_token = 0\n",
    "        for idx in order[batch_start: batch_end]:\n",
    "            x_list.append(words2idxs(tokens[idx]))\n",
    "            y_list.append(tags2idxs(tags[idx]))\n",
    "            max_len_token = max(max_len_token, len(tags[idx]))\n",
    "            \n",
    "        # Fill in the data into numpy nd-arrays filled with padding indices.\n",
    "        x = np.ones([current_batch_size, max_len_token], dtype=np.int32) * token2idx['<PAD>']\n",
    "        y = np.ones([current_batch_size, max_len_token], dtype=np.int32) * tag2idx['O']\n",
    "        lengths = np.zeros(current_batch_size, dtype=np.int32)\n",
    "        for n in range(current_batch_size):\n",
    "            utt_len = len(x_list[n])\n",
    "            x[n, :utt_len] = x_list[n]\n",
    "            lengths[n] = utt_len\n",
    "            y[n, :utt_len] = y_list[n]\n",
    "        yield x, y, lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a recurrent neural network\n",
    "\n",
    "This is the most important part of the assignment. Here we will specify the network architecture based on TensorFlow building blocks. It's fun and easy as a lego constructor! We will create an LSTM network which will produce probability distribution over tags for each token in a sentence. To take into account both right and left contexts of the token, we will use Bi-Directional LSTM (Bi-LSTM). Dense layer will be used on top to perform tag classification.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BiLSTMModel():\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "First, we need to create [placeholders](https://www.tensorflow.org/versions/master/api_docs/python/tf/placeholder) to specify what data we are going to feed into the network during the execution time.  For this task we will need the following placeholders:\n",
    " - *input_batch* — sequences of words (the shape equals to [batch_size, sequence_len]);\n",
    " - *ground_truth_tags* — sequences of tags (the shape equals to [batch_size, sequence_len]);\n",
    " - *lengths* — lengths of not padded sequences (the shape equals to [batch_size]);\n",
    " - *dropout_ph* — dropout keep probability; this placeholder has a predefined value 1;\n",
    " - *learning_rate_ph* — learning rate; we need this placeholder because we want to change the value during training.\n",
    "\n",
    "It could be noticed that we use *None* in the shapes in the declaration, which means that data of any size can be feeded. \n",
    "\n",
    "You need to complete the function *declare_placeholders*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def declare_placeholders(self):\n",
    "    \"\"\"Specifies placeholders for the model.\"\"\"\n",
    "\n",
    "    # Placeholders for input and ground truth output.\n",
    "    self.input_batch = tf.placeholder(dtype=tf.int32, shape=[None, None], name='input_batch') \n",
    "    self.ground_truth_tags = tf.placeholder(dtype=tf.int32, shape=[None, None], name='ground_truth_tags')\n",
    "  \n",
    "    # Placeholder for lengths of the sequences.\n",
    "    self.lengths = tf.placeholder(dtype=tf.int32, shape=[None], name='lengths')\n",
    "    \n",
    "    # Placeholder for a dropout keep probability. If we don't feed\n",
    "    # a value for this placeholder, it will be equal to 1.0.\n",
    "    self.dropout_ph = tf.placeholder_with_default(tf.cast(1.0, tf.float32), shape=[])\n",
    "    \n",
    "    # Placeholder for a learning rate (tf.float32).\n",
    "    self.learning_rate_ph = tf.placeholder_with_default(1e4, shape=[]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "BiLSTMModel.__declare_placeholders = classmethod(declare_placeholders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now, let us specify the layers of the neural network. First, we need to perform some preparatory steps: \n",
    " \n",
    "- Create embeddings matrix with [tf.Variable](https://www.tensorflow.org/api_docs/python/tf/Variable). Specify its name (*embeddings_matrix*), type  (*tf.float32*), and initialize with random values.\n",
    "- Create forward and backward LSTM cells. TensorFlow provides a number of [RNN cells](https://www.tensorflow.org/api_guides/python/contrib.rnn#Core_RNN_Cells_for_use_with_TensorFlow_s_core_RNN_methods) ready for you. We suggest that you use *BasicLSTMCell*, but you can also experiment with other types, e.g. GRU cells. [This](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) blogpost could be interesting if you want to learn more about the differences.\n",
    "- Wrap your cells with [DropoutWrapper](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/DropoutWrapper). Dropout is an important regularization technique for neural networks. Specify all keep probabilities using the dropout placeholder that we created before.\n",
    " \n",
    "After that, you can build the computation graph that transforms an input_batch:\n",
    "\n",
    "- [Look up](https://www.tensorflow.org/api_docs/python/tf/nn/embedding_lookup) embeddings for an *input_batch* in the prepared *embedding_matrix*.\n",
    "- Pass the embeddings through [Bidirectional Dynamic RNN](https://www.tensorflow.org/api_docs/python/tf/nn/bidirectional_dynamic_rnn) with the specified forward and backward cells. Use the lengths placeholder here to avoid computations for padding tokens inside the RNN.\n",
    "- Create a dense layer on top. Its output will be used directly in loss function.  \n",
    " \n",
    "Fill in the code below. In case you need to debug something, the easiest way is to check that tensor shapes of each step match the expected ones. \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_layers(self, vocabulary_size, embedding_dim, n_hidden_rnn, n_tags):\n",
    "    \"\"\"Specifies bi-LSTM architecture and computes logits for inputs.\"\"\"\n",
    "    \n",
    "    # Create embedding variable (tf.Variable) with dtype tf.float32\n",
    "    initial_embedding_matrix = np.random.randn(vocabulary_size, embedding_dim) / np.sqrt(embedding_dim)\n",
    "    embedding_matrix_variable = tf.Variable(initial_embedding_matrix, name='embeddings_matrix', dtype=tf.float32)\n",
    "    \n",
    "    # Create RNN cells (for example, tf.nn.rnn_cell.BasicLSTMCell) with n_hidden_rnn number of units \n",
    "    # and dropout (tf.nn.rnn_cell.DropoutWrapper), initializing all *_keep_prob with dropout placeholder.\n",
    "    forward_cell = tf.nn.rnn_cell.DropoutWrapper(\n",
    "        tf.nn.rnn_cell.BasicLSTMCell(num_units=n_hidden_rnn, forget_bias=3.0),\n",
    "        input_keep_prob=self.dropout_ph,\n",
    "        output_keep_prob=self.dropout_ph,\n",
    "        state_keep_prob=self.dropout_ph\n",
    "    )\n",
    "    backward_cell = tf.nn.rnn_cell.DropoutWrapper(\n",
    "        tf.nn.rnn_cell.BasicLSTMCell(num_units=n_hidden_rnn, forget_bias=3.0),\n",
    "        input_keep_prob=self.dropout_ph,\n",
    "        output_keep_prob=self.dropout_ph,\n",
    "        state_keep_prob=self.dropout_ph\n",
    "    )\n",
    "\n",
    "    # Look up embeddings for self.input_batch (tf.nn.embedding_lookup).\n",
    "    # Shape: [batch_size, sequence_len, embedding_dim].\n",
    "    embeddings = tf.nn.embedding_lookup(embedding_matrix_variable, self.input_batch)\n",
    "    \n",
    "    # Pass them through Bidirectional Dynamic RNN (tf.nn.bidirectional_dynamic_rnn).\n",
    "    # Shape: [batch_size, sequence_len, 2 * n_hidden_rnn]. \n",
    "    # Also don't forget to initialize sequence_length as self.lengths and dtype as tf.float32.\n",
    "    (rnn_output_fw, rnn_output_bw), _ = tf.nn.bidirectional_dynamic_rnn(\n",
    "        cell_fw= forward_cell, cell_bw= backward_cell,\n",
    "        dtype=tf.float32,\n",
    "        inputs=embeddings,\n",
    "        sequence_length=self.lengths\n",
    "    )\n",
    "    rnn_output = tf.concat([rnn_output_fw, rnn_output_bw], axis=2)\n",
    "\n",
    "    # Dense layer on top.\n",
    "    # Shape: [batch_size, sequence_len, n_tags].   \n",
    "    self.logits = tf.layers.dense(rnn_output, n_tags, activation=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "BiLSTMModel.__build_layers = classmethod(build_layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the actual predictions of the neural network, you need to apply [softmax](https://www.tensorflow.org/api_docs/python/tf/nn/softmax) to the last layer and find the most probable tags with [argmax](https://www.tensorflow.org/api_docs/python/tf/argmax)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_predictions(self):\n",
    "    \"\"\"Transforms logits to probabilities and finds the most probable tags.\"\"\"\n",
    "    \n",
    "    # Create softmax (tf.nn.softmax) function\n",
    "    softmax_output = tf.nn.softmax(self.logits)\n",
    "    \n",
    "    # Use argmax (tf.argmax) to get the most probable tags\n",
    "    # Don't forget to set axis=-1\n",
    "    # otherwise argmax will be calculated in a wrong way\n",
    "    self.predictions = tf.argmax(softmax_output, axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "BiLSTMModel.__compute_predictions = classmethod(compute_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "During training we do not need predictions of the network, but we need a loss function. We will use [cross-entropy loss](http://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#cross-entropy), efficiently implemented in TF as \n",
    "[cross entropy with logits](https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits). Note that it should be applied to logits of the model (not to softmax probabilities!). Also note,  that we do not want to take into account loss terms coming from `<PAD>` tokens. So we need to mask them out, before computing [mean](https://www.tensorflow.org/api_docs/python/tf/reduce_mean)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_loss(self, n_tags, PAD_index):\n",
    "    \"\"\"Computes masked cross-entopy loss with logits.\"\"\"\n",
    "    \n",
    "    # Create cross entropy function function (tf.nn.softmax_cross_entropy_with_logits)\n",
    "    ground_truth_tags_one_hot = tf.one_hot(self.ground_truth_tags, n_tags)\n",
    "    loss_tensor = tf.nn.softmax_cross_entropy_with_logits(labels=ground_truth_tags_one_hot, logits=self.logits)\n",
    "    \n",
    "    # Create loss function which doesn't operate with <PAD> tokens (tf.reduce_mean)\n",
    "    mask = tf.cast(tf.not_equal(loss_tensor, PAD_index), tf.float32)\n",
    "    self.loss =  tf.reduce_mean(tf.reduce_sum(tf.multiply(loss_tensor, mask), axis=-1) / tf.reduce_sum(mask, axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "BiLSTMModel.__compute_loss = classmethod(compute_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing to specify is how we want to optimize the loss. \n",
    "We suggest that you use [Adam](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer) optimizer with a learning rate from the corresponding placeholder. \n",
    "You will also need to apply [clipping](https://www.tensorflow.org/versions/r0.12/api_docs/python/train/gradient_clipping) to eliminate exploding gradients. It can be easily done with [clip_by_norm](https://www.tensorflow.org/api_docs/python/tf/clip_by_norm) function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perform_optimization(self):\n",
    "    \"\"\"Specifies the optimizer and train_op for the model.\"\"\"\n",
    "    \n",
    "    # Create an optimizer (tf.train.AdamOptimizer)\n",
    "    self.optimizer = tf.train.AdamOptimizer(self.learning_rate_ph)\n",
    "    self.grads_and_vars = self.optimizer.compute_gradients(self.loss)\n",
    "    \n",
    "    # Gradient clipping (tf.clip_by_norm) for self.grads_and_vars\n",
    "    # Pay attention that you need to apply this operation only for gradients \n",
    "    # because self.grads_and_vars contains also variables.\n",
    "    # list comprehension might be useful in this case.\n",
    "    clip_norm = tf.cast(1.0, tf.float32)\n",
    "    self.grads_and_vars = [(tf.clip_by_norm(grad, clip_norm), var) for grad, var in self.grads_and_vars]\n",
    "    \n",
    "    self.train_op = self.optimizer.apply_gradients(self.grads_and_vars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "BiLSTMModel.__perform_optimization = classmethod(perform_optimization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "You have specified all the parts of your network. You may have noticed, that we didn't deal with any real data yet, so what you have written is just recipes on how the network should function.\n",
    "Now we will put them to the constructor of our Bi-LSTM class to use it in the next section. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_model(self, vocabulary_size, n_tags, embedding_dim, n_hidden_rnn, PAD_index):\n",
    "    self.__declare_placeholders()\n",
    "    self.__build_layers(vocabulary_size, embedding_dim, n_hidden_rnn, n_tags)\n",
    "    self.__compute_predictions()\n",
    "    self.__compute_loss(n_tags, PAD_index)\n",
    "    self.__perform_optimization()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "BiLSTMModel.__init__ = classmethod(init_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the network and predict tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "[Session.run](https://www.tensorflow.org/api_docs/python/tf/Session#run) is a point which initiates computations in the graph that we have defined. To train the network, we need to compute *self.train_op*, which was declared in *perform_optimization*. To predict tags, we just need to compute *self.predictions*. Anyway, we need to feed actual data through the placeholders that we defined before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_on_batch(self, session, x_batch, y_batch, lengths, learning_rate, dropout_keep_probability):\n",
    "    feed_dict = {self.input_batch: x_batch,\n",
    "                 self.ground_truth_tags: y_batch,\n",
    "                 self.learning_rate_ph: learning_rate,\n",
    "                 self.dropout_ph: dropout_keep_probability,\n",
    "                 self.lengths: lengths}\n",
    "    \n",
    "    session.run(self.train_op, feed_dict=feed_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "BiLSTMModel.train_on_batch = classmethod(train_on_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement the function *predict_for_batch* by initializing *feed_dict* with input *x_batch* and *lengths* and running the *session* for *self.predictions*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_for_batch(self, session, x_batch, lengths):\n",
    "    \n",
    "    predictions = session.run(self.predictions, feed_dict={self.input_batch:x_batch, self.lengths:lengths})\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "BiLSTMModel.predict_for_batch = classmethod(predict_for_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We finished with necessary methods of our BiLSTMModel model and almost ready to start experimenting.\n",
    "\n",
    "### Evaluation \n",
    "To simplify the evaluation process we provide two functions for you:\n",
    " - *predict_tags*: uses a model to get predictions and transforms indices to tokens and tags;\n",
    " - *eval_conll*: calculates precision, recall and F1 for the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "from evaluation import precision_recall_f1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_tags(model, session, token_idxs_batch, lengths):\n",
    "    \"\"\"Performs predictions and transforms indices to tokens and tags.\"\"\"\n",
    "    \n",
    "    tag_idxs_batch = model.predict_for_batch(session, token_idxs_batch, lengths)\n",
    "    \n",
    "    tags_batch, tokens_batch = [], []\n",
    "    for tag_idxs, token_idxs in zip(tag_idxs_batch, token_idxs_batch):\n",
    "        tags, tokens = [], []\n",
    "        for tag_idx, token_idx in zip(tag_idxs, token_idxs):\n",
    "            tags.append(idx2tag[tag_idx])\n",
    "            tokens.append(idx2token[token_idx])\n",
    "        tags_batch.append(tags)\n",
    "        tokens_batch.append(tokens)\n",
    "    return tags_batch, tokens_batch\n",
    "    \n",
    "    \n",
    "def eval_conll(model, session, tokens, tags, short_report=True):\n",
    "    \"\"\"Computes NER quality measures using CONLL shared task script.\"\"\"\n",
    "    \n",
    "    y_true, y_pred = [], []\n",
    "    for x_batch, y_batch, lengths in batches_generator(1, tokens, tags):\n",
    "        tags_batch, tokens_batch = predict_tags(model, session, x_batch, lengths)\n",
    "        if len(x_batch[0]) != len(tags_batch[0]):\n",
    "            raise Exception(\"Incorrect length of prediction for the input, \"\n",
    "                            \"expected length: %i, got: %i\" % (len(x_batch[0]), len(tags_batch[0])))\n",
    "        predicted_tags = []\n",
    "        ground_truth_tags = []\n",
    "        for gt_tag_idx, pred_tag, token in zip(y_batch[0], tags_batch[0], tokens_batch[0]): \n",
    "            if token != '<PAD>':\n",
    "                ground_truth_tags.append(idx2tag[gt_tag_idx])\n",
    "                predicted_tags.append(pred_tag)\n",
    "\n",
    "        # We extend every prediction and ground truth sequence with 'O' tag\n",
    "        # to indicate a possible end of entity.\n",
    "        y_true.extend(ground_truth_tags + ['O'])\n",
    "        y_pred.extend(predicted_tags + ['O'])\n",
    "        \n",
    "    results = precision_recall_f1(y_true, y_pred, print_results=True, short_report=short_report)\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create *BiLSTMModel* model with the following parameters:\n",
    " - *vocabulary_size* — number of tokens;\n",
    " - *n_tags* — number of tags;\n",
    " - *embedding_dim* — dimension of embeddings, recommended value: 200;\n",
    " - *n_hidden_rnn* — size of hidden layers for RNN, recommended value: 200;\n",
    " - *PAD_index* — an index of the padding token (`<PAD>`).\n",
    "\n",
    "Set hyperparameters. You might want to start with the following recommended values:\n",
    "- *batch_size*: 32;\n",
    "- 4 epochs;\n",
    "- starting value of *learning_rate*: 0.005\n",
    "- *learning_rate_decay*: a square root of 2;\n",
    "- *dropout_keep_probability*: try several values: 0.1, 0.5, 0.9.\n",
    "\n",
    "However, feel free to conduct more experiments to tune hyperparameters and earn extra points for the assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20505"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(idx2token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(idx2tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "model = BiLSTMModel(20505, 21, 200, 200, token2idx['<PAD>'])\n",
    "\n",
    "batch_size = 32\n",
    "n_epochs = 4\n",
    "learning_rate = 0.005\n",
    "learning_rate_decay = 1.414\n",
    "dropout_keep_probability = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you got an error *\"Tensor conversion requested dtype float64 for Tensor with dtype float32\"* in this point, check if there are variables without dtype initialised. Set the value of dtype equals to *tf.float32* for such variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, run the training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training... \n",
      "\n",
      "-------------------- Epoch 1 of 4 --------------------\n",
      "Train data evaluation:\n",
      "processed 105778 tokens with 4489 phrases; found: 53922 phrases; correct: 137.\n",
      "\n",
      "precision:  0.25%; recall:  3.05%; F1:  0.47\n",
      "\n",
      "Validation data evaluation:\n",
      "processed 12836 tokens with 537 phrases; found: 6745 phrases; correct: 14.\n",
      "\n",
      "precision:  0.21%; recall:  2.61%; F1:  0.38\n",
      "\n",
      "-------------------- Epoch 2 of 4 --------------------\n",
      "Train data evaluation:\n",
      "processed 105778 tokens with 4489 phrases; found: 1811 phrases; correct: 411.\n",
      "\n",
      "precision:  22.69%; recall:  9.16%; F1:  13.05\n",
      "\n",
      "Validation data evaluation:\n",
      "processed 12836 tokens with 537 phrases; found: 165 phrases; correct: 37.\n",
      "\n",
      "precision:  22.42%; recall:  6.89%; F1:  10.54\n",
      "\n",
      "-------------------- Epoch 3 of 4 --------------------\n",
      "Train data evaluation:\n",
      "processed 105778 tokens with 4489 phrases; found: 4766 phrases; correct: 1764.\n",
      "\n",
      "precision:  37.01%; recall:  39.30%; F1:  38.12\n",
      "\n",
      "Validation data evaluation:\n",
      "processed 12836 tokens with 537 phrases; found: 343 phrases; correct: 126.\n",
      "\n",
      "precision:  36.73%; recall:  23.46%; F1:  28.64\n",
      "\n",
      "-------------------- Epoch 4 of 4 --------------------\n",
      "Train data evaluation:\n",
      "processed 105778 tokens with 4489 phrases; found: 4662 phrases; correct: 2785.\n",
      "\n",
      "precision:  59.74%; recall:  62.04%; F1:  60.87\n",
      "\n",
      "Validation data evaluation:\n",
      "processed 12836 tokens with 537 phrases; found: 360 phrases; correct: 162.\n",
      "\n",
      "precision:  45.00%; recall:  30.17%; F1:  36.12\n",
      "\n",
      "...training finished.\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "print('Start training... \\n')\n",
    "for epoch in range(n_epochs):\n",
    "    # For each epoch evaluate the model on train and validation data\n",
    "    print('-' * 20 + ' Epoch {} '.format(epoch+1) + 'of {} '.format(n_epochs) + '-' * 20)\n",
    "    print('Train data evaluation:')\n",
    "    eval_conll(model, sess, train_tokens, train_tags, short_report=True)\n",
    "    print('Validation data evaluation:')\n",
    "    eval_conll(model, sess, validation_tokens, validation_tags, short_report=True)\n",
    "    \n",
    "    # Train the model\n",
    "    for x_batch, y_batch, lengths in batches_generator(batch_size, train_tokens, train_tags):\n",
    "        model.train_on_batch(sess, x_batch, y_batch, lengths, learning_rate, dropout_keep_probability)\n",
    "        \n",
    "    # Decaying the learning rate\n",
    "    learning_rate = learning_rate / learning_rate_decay\n",
    "    \n",
    "print('...training finished.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Full quality reports for the final model on train, validation, and test sets. To give you a hint whether you have implemented everything correctly, you might expect F-score about 40% on the validation set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------- Train set quality: --------------------\n",
      "processed 105778 tokens with 4489 phrases; found: 4687 phrases; correct: 3437.\n",
      "\n",
      "precision:  73.33%; recall:  76.56%; F1:  74.91\n",
      "\n",
      "\t     company: precision:   81.22%; recall:   84.76%; F1:   82.95; predicted:   671\n",
      "\n",
      "\t    facility: precision:   62.73%; recall:   76.11%; F1:   68.78; predicted:   381\n",
      "\n",
      "\t     geo-loc: precision:   82.10%; recall:   92.57%; F1:   87.02; predicted:  1123\n",
      "\n",
      "\t       movie: precision:    0.00%; recall:    0.00%; F1:    0.00; predicted:     3\n",
      "\n",
      "\t musicartist: precision:   49.61%; recall:   54.31%; F1:   51.85; predicted:   254\n",
      "\n",
      "\t       other: precision:   64.44%; recall:   80.18%; F1:   71.45; predicted:   942\n",
      "\n",
      "\t      person: precision:   85.65%; recall:   87.58%; F1:   86.61; predicted:   906\n",
      "\n",
      "\t     product: precision:   52.35%; recall:   59.43%; F1:   55.67; predicted:   361\n",
      "\n",
      "\t  sportsteam: precision:   75.00%; recall:   15.21%; F1:   25.29; predicted:    44\n",
      "\n",
      "\t      tvshow: precision:    0.00%; recall:    0.00%; F1:    0.00; predicted:     2\n",
      "\n",
      "-------------------- Validation set quality: --------------------\n",
      "processed 12836 tokens with 537 phrases; found: 374 phrases; correct: 179.\n",
      "\n",
      "precision:  47.86%; recall:  33.33%; F1:  39.30\n",
      "\n",
      "\t     company: precision:   66.25%; recall:   50.96%; F1:   57.61; predicted:    80\n",
      "\n",
      "\t    facility: precision:   32.26%; recall:   29.41%; F1:   30.77; predicted:    31\n",
      "\n",
      "\t     geo-loc: precision:   65.52%; recall:   50.44%; F1:   57.00; predicted:    87\n",
      "\n",
      "\t       movie: precision:    0.00%; recall:    0.00%; F1:    0.00; predicted:     0\n",
      "\n",
      "\t musicartist: precision:   23.08%; recall:   10.71%; F1:   14.63; predicted:    13\n",
      "\n",
      "\t       other: precision:   30.34%; recall:   33.33%; F1:   31.76; predicted:    89\n",
      "\n",
      "\t      person: precision:   54.17%; recall:   23.21%; F1:   32.50; predicted:    48\n",
      "\n",
      "\t     product: precision:   11.54%; recall:    8.82%; F1:   10.00; predicted:    26\n",
      "\n",
      "\t  sportsteam: precision:    0.00%; recall:    0.00%; F1:    0.00; predicted:     0\n",
      "\n",
      "\t      tvshow: precision:    0.00%; recall:    0.00%; F1:    0.00; predicted:     0\n",
      "\n",
      "-------------------- Test set quality: --------------------\n",
      "processed 13258 tokens with 604 phrases; found: 452 phrases; correct: 215.\n",
      "\n",
      "precision:  47.57%; recall:  35.60%; F1:  40.72\n",
      "\n",
      "\t     company: precision:   66.67%; recall:   42.86%; F1:   52.17; predicted:    54\n",
      "\n",
      "\t    facility: precision:   43.59%; recall:   36.17%; F1:   39.53; predicted:    39\n",
      "\n",
      "\t     geo-loc: precision:   71.05%; recall:   49.09%; F1:   58.06; predicted:   114\n",
      "\n",
      "\t       movie: precision:    0.00%; recall:    0.00%; F1:    0.00; predicted:     0\n",
      "\n",
      "\t musicartist: precision:   15.38%; recall:    7.41%; F1:   10.00; predicted:    13\n",
      "\n",
      "\t       other: precision:   28.15%; recall:   36.89%; F1:   31.93; predicted:   135\n",
      "\n",
      "\t      person: precision:   56.06%; recall:   35.58%; F1:   43.53; predicted:    66\n",
      "\n",
      "\t     product: precision:    7.69%; recall:    7.14%; F1:    7.41; predicted:    26\n",
      "\n",
      "\t  sportsteam: precision:   40.00%; recall:    6.45%; F1:   11.11; predicted:     5\n",
      "\n",
      "\t      tvshow: precision:    0.00%; recall:    0.00%; F1:    0.00; predicted:     0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('-' * 20 + ' Train set quality: ' + '-' * 20)\n",
    "train_results = eval_conll(model, sess, train_tokens, train_tags, short_report=False)\n",
    "\n",
    "print('-' * 20 + ' Validation set quality: ' + '-' * 20)\n",
    "validation_results = eval_conll(model, sess, validation_tokens, validation_tags, short_report=False)\n",
    "\n",
    "print('-' * 20 + ' Test set quality: ' + '-' * 20)\n",
    "test_results = eval_conll(model, sess, test_tokens, test_tags, short_report=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions\n",
    "\n",
    "Nowadays, Bi-LSTM is one of the state of the art approaches for solving NER problem and it outperforms other classical methods. Despite the fact that we used small training corpora (in comparison with usual sizes of corpora in Deep Learning), our results are quite good. In addition, in this task there are many possible named entities and for some of them we have only several dozens of trainig examples, which is definately small. However, the implemented model outperforms classical CRFs for this task. Even better results could be obtained by some combinations of several types of methods, e.g. see [this](https://arxiv.org/abs/1603.01354) paper if you are interested."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}