ch7_image_caption.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ch7_image_caption.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lblogan14/deep_learning_for_computer_vision/blob/master/ch7_image_caption.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "nRtRC3BYjT_N",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Captioning images is to detect the objects and come up with a text caption for the image. Also called **Image to Text Translation**."
      ]
    },
    {
      "metadata": {
        "id": "pLYpPaWPjpHi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Natural Language Processing for Image Captioning\n",
        "As natural language has to be generated from the image, getting familiar with\n",
        "natural language processing (NLP) becomes important. One form of natural language is text. The text is a sequence of\n",
        "words or characters. The atomic element of text is called token, which is a\n",
        "sequence of characters. A character is an atomic element of text.\n",
        "\n",
        "In order to process any natural language in the form of text, the text has to be\n",
        "preprocessed by removing punctuation, brackets and so on. Then, the text has to\n",
        "be tokenized into words by separating them into spaces. Then, the words have to\n",
        "be converted to vectors."
      ]
    },
    {
      "metadata": {
        "id": "kjTz3kTMkIAT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Express words in vector form\n",
        "Words expressed in vector form can help perform arithmetic operations on\n",
        "themselves. The vector has to be compact, with less dimension. Synonyms\n",
        "should have similar vectors and antonyms should have a different vector.\n",
        "![alt text](https://github.com/lblogan14/deep_learning_for_computer_vision/blob/master/notes_images/ch7/word_vector.JPG?raw=true)\n",
        "\n",
        "This vector arithmetic enables comparison in semantic space between different\n",
        "entities."
      ]
    },
    {
      "metadata": {
        "id": "Bsp3M4ewkqQr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Convert words to vectors\n",
        "The words can be converted to vectors by training a model on a large text\n",
        "corpus. The model is trained such that given a word, the model can predict\n",
        "nearby words. The words are first one-hot encoded followed by hidden layers\n",
        "before predicting the one-hot encoding of nearby words. Training this way will\n",
        "create a compact representation of words.\n",
        "\n",
        "There are two common methods to obtain the word context:\n",
        "* **Skip-gram**: Given a single word, try to predict a few words which are close to\n",
        "* **Continuous Bag of Words (CBOW)**: Reverse of skip-gram by predicting a word given a group of words\n",
        "![alt text](https://github.com/lblogan14/deep_learning_for_computer_vision/blob/master/notes_images/ch7/skip_cbow.JPG?raw=true)\n",
        "\n",
        "The words are converted to vectors in an embedding space which is used for training."
      ]
    },
    {
      "metadata": {
        "id": "jC9AH_u3lu5i",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Train an embedding\n",
        "A model is trained with the embedding:\n",
        "\n",
        "![alt text](https://github.com/lblogan14/deep_learning_for_computer_vision/blob/master/notes_images/ch7/embedding.JPG?raw=true)\n",
        "\n",
        "The target word is predicted based on context\n",
        "or history. The prediction is based on the Softmax classifier. The hidden layer\n",
        "learns the embedding as a compact representation."
      ]
    },
    {
      "metadata": {
        "id": "_NajjRPlM4lz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Dense Captioning\n",
        "Separate captions are generated for objects and actions in the image as shown below:\n",
        "![alt text](https://github.com/lblogan14/deep_learning_for_computer_vision/blob/master/notes_images/ch7/dense_caption.JPG?raw=true)\n",
        "\n",
        "The architecture is a combination of Faster-RCNN and LSTM. The\n",
        "region is generated producing the object detection results, and the visual features\n",
        "of the regions are used to generate the captions.\n",
        "\n",
        "![alt text](https://github.com/lblogan14/deep_learning_for_computer_vision/blob/master/notes_images/ch7/dense_caption_archi.JPG?raw=true)"
      ]
    },
    {
      "metadata": {
        "id": "oU9hw3QbN_zd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Attention Network for Captioning\n",
        "The **attention mechanism**\n",
        "gives more weight to certain regions of the image than others. Attention also\n",
        "enables visualization, showing us where the model is focusing when it generates\n",
        "the next word.\n",
        "\n",
        "First, CNN features are extracted from the image. Then, RNN with attention is\n",
        "applied to the image from which the words are generated.\n",
        "\n",
        "![alt text](https://github.com/lblogan14/deep_learning_for_computer_vision/blob/master/notes_images/ch7/attention_network.JPG?raw=true)\n"
      ]
    },
    {
      "metadata": {
        "id": "kf4wpDKXOrAg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##Tensorflow Implementation"
      ]
    },
    {
      "metadata": {
        "id": "LlydmExsOvwD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6df95a88-a4f6-48c8-bc4b-7cdffad72689"
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "from keras.layers.recurrent import *"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "t086J1t9O1NL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "training = True\n",
        "sequence_leagth = 0\n",
        "vocabulary_size = 0\n",
        "input_tensor = 0\n",
        "input_shape = 0\n",
        "embedding_dimension = 0\n",
        "dropout_prob = 0.3\n",
        "previous_words = 0\n",
        "height = 0\n",
        "shape = 0\n",
        "cnn_features = 0\n",
        "depth = 0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DnpetQDHPEl6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "vgg_model = tf.keras.applications.vgg16.VGG16(weights='imagenet',\n",
        "                                              include_top=False,\n",
        "                                              input_tensor=input_tensor,\n",
        "                                              input_shape=input_shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3RnaYBm3PRTy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "word_embedding = tf.keras.layers.Embedding(vocabulary_size,\n",
        "                                           embedding_dimension,\n",
        "                                           input_length=sequence_length)\n",
        "embedding = word_embedding(previous_words)\n",
        "embedding = tf.keras.layers.Activation('relu')(embedding)\n",
        "embedding = tf.keras.layers.Dropout(dropout_prob)(embedding)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_Lw5v6vIP68z",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cnn_features_flattened = tf.keras.layers.Reshape((height * height, shape))(cnn_features)\n",
        "net = tf.keras.layers.GlobalAveragePooling1D()(cnn_features_flattened)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gS26OJvIQJLr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "net = tf.keras.layers.Dense(embedding_dimension, activation='relu')(net)\n",
        "net = tf.keras.layers.Dropout(dropout_prob)(net)\n",
        "net = tf.keras.layers.RepeatVector(sequence_length)(net)\n",
        "net = tf.keras.layers.concatenate()([net, embbedding])\n",
        "net = tf.keras.layers.Dropout(dropout_prob)(net)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "peJTsbEAQVQl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class LSTM_sent(Recurrent):\n",
        "  def __init__(self, output_dim, init='glorot_uniform',\n",
        "               inner_init='orthogonal', forget_bias_init='one',\n",
        "               activation='tanh', inner_activation='hard_sigmoid',\n",
        "               W_regularizer=None, U_regularizer=None, b_regularizer=None,\n",
        "               dropout_W=0., dropout_U=0., sentinel=True, **kwargs):\n",
        "    self.output_dim = output_dim\n",
        "    self.init = initializations.get(init)\n",
        "    self.inner_init = initializations.get(inner_init)\n",
        "    self.forget_bias_init = initializations.get(forget_bias_init)\n",
        "    self.activation = activations.get(activation)\n",
        "    self.inner_activation = activations.get(inner_activation)\n",
        "    self.W_regularizer = regularizers.get(W_regularizer)\n",
        "    self.U_regularizer = regularizers.get(U_regularizer)\n",
        "    self.b_regularizer = regularizers.get(b_regularizer)\n",
        "    self.dropout_W, self.dropout_U = dropout_W, dropout_U\n",
        "    self.sentinel = sentinel\n",
        "    if self.dropout_W or self.dropout_U:\n",
        "      self.uses_learning_phase = True\n",
        "    super(LSTM_sent, self).__init__(**kwargs)\n",
        "    \n",
        "  def build(self, input_shape):\n",
        "    self.input_spec = [InputSpec(shape=input_shape)]\n",
        "    input_dim = input_shape[2]\n",
        "    self.input_dim = input_dim\n",
        "\n",
        "    if self.stateful:\n",
        "      self.reset_states()\n",
        "    else:\n",
        "      if self.sentinel:\n",
        "        self.states = [None, None]\n",
        "      else:\n",
        "        self.states = [None]\n",
        "\n",
        "      self.W_i = self.init((input_dim, self.output_dim),\n",
        "                             name='{}_W_i'.format(self.name))\n",
        "      self.U_i = self.inner_init((self.output_dim, self.output_dim),\n",
        "                                   name='{}_U_i'.format(self.name))\n",
        "      self.b_i = K.zeros((self.output_dim,), name='{}_b_i'.format(self.name))\n",
        "\n",
        "      self.W_f = self.init((input_dim, self.output_dim),\n",
        "                             name='{}_W_f'.format(self.name))\n",
        "      self.U_f = self.inner_init((self.output_dim, self.output_dim),\n",
        "                                   name='{}_U_f'.format(self.name))\n",
        "      self.b_f = self.forget_bias_init((self.output_dim,),\n",
        "                                         name='{}_b_f'.format(self.name))\n",
        "\n",
        "      self.W_c = self.init((input_dim, self.output_dim),\n",
        "                             name='{}_W_c'.format(self.name))\n",
        "      self.U_c = self.inner_init((self.output_dim, self.output_dim),\n",
        "                                   name='{}_U_c'.format(self.name))\n",
        "      self.b_c = K.zeros((self.output_dim,), name='{}_b_c'.format(self.name))\n",
        "\n",
        "      self.W_o = self.init((input_dim, self.output_dim),\n",
        "                             name='{}_W_o'.format(self.name))\n",
        "      self.U_o = self.inner_init((self.output_dim, self.output_dim),\n",
        "                                   name='{}_U_o'.format(self.name))\n",
        "      self.b_o = K.zeros((self.output_dim,), name='{}_b_o'.format(self.name))\n",
        "\n",
        "      if self.sentinel:\n",
        "        # sentinel gate\n",
        "        self.W_g = self.init((input_dim, self.output_dim),\n",
        "                                 name='{}_W_g'.format(self.name))\n",
        "        self.U_g = self.inner_init((self.output_dim, self.output_dim),\n",
        "                                       name='{}_U_g'.format(self.name))\n",
        "        self.b_g = K.zeros((self.output_dim,), name='{}_b_g'.format(self.name))\n",
        "\n",
        "\n",
        "        self.trainable_weights = [self.W_i, self.U_i, self.b_i,\n",
        "                                  self.W_c, self.U_c, self.b_c,\n",
        "                                  self.W_f, self.U_f, self.b_f,\n",
        "                                  self.W_o, self.U_o, self.b_o,\n",
        "                                  self.W_g, self.U_g, self.b_g]\n",
        "      else:\n",
        "        self.trainable_weights = [self.W_i, self.U_i, self.b_i,\n",
        "                                  self.W_c, self.U_c, self.b_c,\n",
        "                                  self.W_f, self.U_f, self.b_f,\n",
        "                                  self.W_o, self.U_o, self.b_o]\n",
        "\n",
        "      if self.initial_weights is not None:\n",
        "        self.set_weights(self.initial_weights)\n",
        "        del self.initial_weights\n",
        "\n",
        "  def reset_states(self):\n",
        "    assert self.stateful, 'Layer must be stateful.'\n",
        "    input_shape = self.input_spec[0].shape\n",
        "    if not input_shape[0]:\n",
        "      raise Exception('If a RNN is stateful, a complete ' +\n",
        "                            'input_shape must be provided (including batch size).')\n",
        "    if hasattr(self, 'states'):\n",
        "      K.set_value(self.states[0],\n",
        "                        np.zeros((input_shape[0], self.output_dim)))\n",
        "      K.set_value(self.states[1],\n",
        "                        np.zeros((input_shape[0], self.output_dim)))\n",
        "    else:\n",
        "      self.states = [K.zeros((input_shape[0], self.output_dim)),\n",
        "                           K.zeros((input_shape[0], self.output_dim))]\n",
        "\n",
        "  def preprocess_input(self, x, train=False):\n",
        "    if self.consume_less == 'cpu':\n",
        "      if train and (0 < self.dropout_W < 1):\n",
        "        dropout = self.dropout_W\n",
        "      else:\n",
        "        dropout = 0\n",
        "      input_shape = self.input_spec[0].shape\n",
        "      input_dim = input_shape[2]\n",
        "      timesteps = input_shape[1]\n",
        "\n",
        "      x_i = time_distributed_dense(x, self.W_i, self.b_i, dropout,\n",
        "                                         input_dim, self.output_dim, timesteps)\n",
        "      x_f = time_distributed_dense(x, self.W_f, self.b_f, dropout,\n",
        "                                         input_dim, self.output_dim, timesteps)\n",
        "      x_c = time_distributed_dense(x, self.W_c, self.b_c, dropout,\n",
        "                                         input_dim, self.output_dim, timesteps)\n",
        "      x_o = time_distributed_dense(x, self.W_o, self.b_o, dropout,\n",
        "                                         input_dim, self.output_dim, timesteps)\n",
        "      if self.sentinel:\n",
        "        x_g = time_distributed_dense(x, self.W_g, self.b_g, dropout,\n",
        "                                             input_dim, self.output_dim, timesteps)\n",
        "        return K.concatenate([x_i, x_f, x_c, x_o,x_g], axis=2)\n",
        "      else:\n",
        "        return K.concatenate([x_i, x_f, x_c, x_o], axis=2)\n",
        "\n",
        "    else:\n",
        "      return x\n",
        "\n",
        "  def step(self, x, states):\n",
        "    h_tm1 = states[0]\n",
        "    c_tm1 = states[1]\n",
        "    B_U = states[2]\n",
        "    B_W = states[3]\n",
        "\n",
        "    if self.consume_less == 'cpu':\n",
        "      x_i = x[:, :self.output_dim]\n",
        "      x_f = x[:, self.output_dim: 2 * self.output_dim]\n",
        "      x_c = x[:, 2 * self.output_dim: 3 * self.output_dim]\n",
        "      x_o = x[:, 3 * self.output_dim: 4 * self.output_dim]\n",
        "      if self.sentinel:\n",
        "        x_g = x[:, 4 * self.output_dim:]\n",
        "    else:\n",
        "      x_i = K.dot(x, self.W_i) + self.b_i\n",
        "      x_f = K.dot(x * B_W[1], self.W_f) + self.b_f\n",
        "      x_c = K.dot(x * B_W[2], self.W_c) + self.b_c\n",
        "      x_o = K.dot(x * B_W[3], self.W_o) + self.b_o\n",
        "      if self.sentinel:\n",
        "        x_g = K.dot(x * B_W[4], self.W_g) + self.b_g\n",
        "\n",
        "    i = self.inner_activation(x_i + K.dot(h_tm1 * B_U[0], self.U_i))\n",
        "    f = self.inner_activation(x_f + K.dot(h_tm1 * B_U[1], self.U_f))\n",
        "    c = f * c_tm1 + i * self.activation(x_c + K.dot(h_tm1 * B_U[2], self.U_c))\n",
        "    o = self.inner_activation(x_o + K.dot(h_tm1 * B_U[3], self.U_o))\n",
        "    h = o * self.activation(c)\n",
        "    if self.sentinel:\n",
        "      g = self.inner_activation(x_g + K.dot(h_tm1 * B_U[4], self.U_g))\n",
        "      s = g * self.activation(c)\n",
        "      return [h,s], [h, c]\n",
        "    else:\n",
        "      return h, [h, c]\n",
        "\n",
        "  def get_constants(self, x):\n",
        "    constants = []\n",
        "    if self.sentinel:\n",
        "      Ngate = 5\n",
        "    else:\n",
        "      Ngate = 4\n",
        "    if 0 < self.dropout_U < 1:\n",
        "      ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))\n",
        "      ones = K.concatenate([ones] * self.output_dim, 1)\n",
        "      B_U = [K.dropout(ones, self.dropout_U) for _ in range(Ngate)]\n",
        "      constants.append(B_U)\n",
        "    else:\n",
        "      constants.append([K.cast_to_floatx(1.) for _ in range(Ngate)])\n",
        "\n",
        "    if self.consume_less == 'cpu' and 0 < self.dropout_W < 1:\n",
        "      input_shape = self.input_spec[0].shape\n",
        "      input_dim = input_shape[-1]\n",
        "      ones = K.ones_like(K.reshape(x[:, 0, 0], (-1, 1)))\n",
        "      ones = K.concatenate([ones] * input_dim, 1)\n",
        "      B_W = [K.dropout(ones, self.dropout_W) for _ in range(Ngate)]\n",
        "      constants.append(B_W)\n",
        "    else:\n",
        "      constants.append([K.cast_to_floatx(1.) for _ in range(Ngate)])\n",
        "    return constants\n",
        "\n",
        "\n",
        "  def get_output_shape_for(self, input_shape):\n",
        "    if isinstance(input_shape, list) and len(input_shape) > 1:\n",
        "      input_shape = input_shape[0]\n",
        "    if self.return_sequences:\n",
        "      output_shape = (input_shape[0], input_shape[1], self.output_dim)\n",
        "    else:\n",
        "      output_shape = (input_shape[0], self.output_dim)\n",
        "    #the hidden state and the sentinel have the same shape\n",
        "    if self.sentinel:\n",
        "      return [output_shape, output_shape]\n",
        "    else:\n",
        "      return output_shape\n",
        "\n",
        "  def compute_mask(self, input, mask):\n",
        "    if self.return_sequences:\n",
        "      if self.sentinel:\n",
        "        return [mask, mask]\n",
        "      else :\n",
        "        return mask\n",
        "    else:\n",
        "      if self.sentinel:\n",
        "        return [None, None]\n",
        "      else:\n",
        "        return None\n",
        "\n",
        "  def call(self, x, mask=None):\n",
        "    input_shape = self.input_spec[0].shape\n",
        "    if K._BACKEND == 'tensorflow':\n",
        "      if not input_shape[1]:\n",
        "        raise Exception('When using TensorFlow, you should define '\n",
        "                        'explicitly the number of timesteps of '\n",
        "                        'your sequences.\\n'\n",
        "                        'If your first layer is an Embedding, '\n",
        "                        'make sure to pass it an \"input_length\" '\n",
        "                        'argument. Otherwise, make sure '\n",
        "                        'the first layer has '\n",
        "                        'an \"input_shape\" or \"batch_input_shape\" '\n",
        "                        'argument, including the time axis. '\n",
        "                        'Found input shape at layer ' + self.name +\n",
        "                        ': ' + str(input_shape))\n",
        "    if self.stateful:\n",
        "      initial_states = self.states\n",
        "    else:\n",
        "      initial_states = self.get_initial_states(x)\n",
        "    constants = self.get_constants(x)\n",
        "    preprocessed_input = self.preprocess_input(x)\n",
        "\n",
        "    last_output, outputs, states = K.rnn(self.step, preprocessed_input,\n",
        "                                             initial_states,\n",
        "                                             go_backwards=self.go_backwards,\n",
        "                                             mask=mask,\n",
        "                                             constants=constants,\n",
        "                                             unroll=self.unroll,\n",
        "                                             input_length=input_shape[1])\n",
        "\n",
        "\n",
        "    if self.stateful:\n",
        "      self.updates = []\n",
        "      for i in range(len(states)):\n",
        "        self.updates.append((self.states[i], states[i]))\n",
        "    if self.sentinel:\n",
        "      outputs = K.permute_dimensions(outputs, [0,2,1,3])\n",
        "      if self.return_sequences:\n",
        "        return [outputs[0], outputs[1]]\n",
        "      else:\n",
        "        return [last_output[0],last_output[1]]\n",
        "    else:\n",
        "     if self.return_sequences:\n",
        "       return outputs\n",
        "     else:\n",
        "       return last_output\n",
        "\n",
        "\n",
        "  def get_config(self):\n",
        "    config = {\"output_dim\": self.output_dim,\n",
        "              \"init\": self.init.__name__,\n",
        "              \"inner_init\": self.inner_init.__name__,\n",
        "              \"forget_bias_init\": self.forget_bias_init.__name__,\n",
        "              \"activation\": self.activation.__name__,\n",
        "              \"inner_activation\": self.inner_activation.__name__,\n",
        "              \"W_regularizer\": self.W_regularizer.get_config() if self.W_regularizer else None,\n",
        "              \"U_regularizer\": self.U_regularizer.get_config() if self.U_regularizer else None,\n",
        "              \"b_regularizer\": self.b_regularizer.get_config() if self.b_regularizer else None,\n",
        "              \"dropout_W\": self.dropout_W,\n",
        "              \"dropout_U\": self.dropout_U,\n",
        "              \"sentinel\": self.sentinel}\n",
        "    base_config = super(LSTM_sent, self).get_config()\n",
        "    return dict(list(base_config.items()) + list(config.items()))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "q23Fhe6PS_Im",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "lstm_ = LSTM_sent(output_dim = args_dict.lstm_dim,\n",
        "                  return_sequences=True,stateful=True,\n",
        "                  dropout_W = dropout_prob,\n",
        "                  dropout_U = dropout_prob,\n",
        "                  sentinel=True,name='hs')\n",
        "h, s = lstm_(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OTd67aJ_TBc5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "num_vfeats = wh * wh\n",
        "num_vfeats = num_vfeats + 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gMcjXULLTC3J",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "h_out_linear = tf.keras.layers.Convolution1D(\n",
        "    depth, 1, activation='tanh', border_mode='same')(h)\n",
        "h_out_linear = tf.keras.layers.Dropout(\n",
        "    dropout_prob)(h_out_linear)\n",
        "h_out_embed = tf.keras.layers.Convolution1D(\n",
        "    embedding_dimension, 1, border_mode='same')(h_out_linear)\n",
        "z_h_embed = tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.RepeatVector(num_vfeats))(h_out_embed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "E4jee1QWTEnV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "Vi = tf.keras.layers.Convolution1D(\n",
        "    depth, 1, border_mode='same', activation='relu')(V)\n",
        "\n",
        "Vi = tf.keras.layers.Dropout(dropout_prob)(Vi)\n",
        "Vi_emb = tf.keras.layers.Convolution1D(\n",
        "    embedding_dimension, 1, border_mode='same', activation='relu')(Vi)\n",
        "\n",
        "z_v_linear = tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.RepeatVector(sequence_length))(Vi)\n",
        "z_v_embed = tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.RepeatVector(sequence_length))(Vi_emb)\n",
        "\n",
        "z_v_linear = tf.keras.layers.Permute((2, 1, 3))(z_v_linear)\n",
        "z_v_embed = tf.keras.layers.Permute((2, 1, 3))(z_v_embed)\n",
        "\n",
        "fake_feat = tf.keras.layers.Convolution1D(\n",
        "    depth, 1, activation='relu', border_mode='same')(s)\n",
        "fake_feat = tf.keras.layers.Dropout(dropout_prob)(fake_feat)\n",
        "\n",
        "fake_feat_embed = tf.keras.layers.Convolution1D(\n",
        "    embedding_dimension, 1, border_mode='same')(fake_feat)\n",
        "z_s_linear = tf.keras.layers.Reshape((sequence_length, 1, depth))(fake_feat)\n",
        "z_s_embed = tf.keras.layers.Reshape(\n",
        "    (sequence_length, 1, embedding_dimension))(fake_feat_embed)\n",
        "\n",
        "z_v_linear = tf.keras.layers.concatenate(axis=-2)([z_v_linear, z_s_linear])\n",
        "z_v_embed = tf.keras.layers.concatenate(axis=-2)([z_v_embed, z_s_embed])\n",
        "\n",
        "z = tf.keras.layers.Merge(mode='sum')([z_h_embed,z_v_embed])\n",
        "z = tf.keras.layers.Dropout(dropout_prob)(z)\n",
        "z = tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.Activation('tanh'))(z)\n",
        "attention= tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.Convolution1D(1, 1, border_mode='same'))(z)\n",
        "\n",
        "attention = tf.keras.layers.Reshape((sequence_length, num_vfeats))(attention)\n",
        "attention = tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.Activation('softmax'))(attention)\n",
        "attention = tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.RepeatVector(depth))(attention)\n",
        "attention = tf.keras.layers.Permute((1,3,2))(attention)\n",
        "w_Vi = tf.keras.layers.Add()([attention,z_v_linear])\n",
        "sumpool = tf.keras.layers.Lambda(lambda x: K.sum(x, axis=-2),\n",
        "               output_shape=(depth,))\n",
        "c_vec = tf.keras.layers.TimeDistributed(sumpool)(w_Vi)\n",
        "atten_out = tf.keras.layers.Merge(mode='sum')([h_out_linear,c_vec])\n",
        "h = tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.Dense(embedding_dimension,activation='tanh'))(atten_out)\n",
        "h = tf.keras.layers.Dropout(dropout_prob)(h)\n",
        "\n",
        "predictions = tf.keras.layers.TimeDistributed(\n",
        "    tf.keras.layers.Dense(vocabulary_size, activation='softmax'))(h)\n",
        "\n",
        "model = tf.keras.models.Model(input=[cnn_features, prev_words], output=predictions)\n",
        "opt = get_opt(args_dict)\n",
        "\n",
        "\n",
        "in_im = tf.keras.layers.Input(\n",
        "    batch_shape=(args_dict.bs, args_dict.imsize, args_dict.imsize, 3), name='image')\n",
        "\n",
        "wh = vgg_model.output_shape[1]\n",
        "dim = vgg_model.output_shape[3]\n",
        "\n",
        "if not args_dict.cnn_train:\n",
        "    for i,layer in enumerate(convnet.layers):\n",
        "        if i > args_dict.finetune_start_layer:\n",
        "            layer.trainable = False\n",
        "\n",
        "imfeats = vgg_model(in_im)\n",
        "cnn_features = tf.keras.layers.Input(batch_shape=(args_dict.bs, wh, wh, dim))\n",
        "prev_words = tf.keras.layers.Input(batch_shape=(args_dict.bs, sequence_length))\n",
        "lang_model = language_model(args_dict, wh, dim, cnn_features, prev_words)\n",
        "\n",
        "out = lang_model([imfeats,prev_words])\n",
        "\n",
        "model = tf.keras.models.Model(input=[in_im, prev_words], output=out)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}