Densenet_on_CIFAR10.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Densenet-on-CIFAR10.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "K70hAckqg0EA",
        "colab_type": "code",
        "outputId": "d1955d92-d9fc-4d22-df0d-510238376899",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# https://keras.io/\n",
        "!pip install -q keras\n",
        "import keras\n",
        "\n",
        "import scipy\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import keras"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wVIx_KIigxPV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import keras\n",
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "import h5py\n",
        "from keras.datasets import cifar10\n",
        "from keras.models import Model, Sequential\n",
        "from keras.layers import Dense, Dropout, Flatten, Input, AveragePooling2D, merge, Activation\n",
        "from keras.layers import Conv2D, MaxPooling2D, BatchNormalization\n",
        "from keras.layers import Concatenate\n",
        "from keras.optimizers import Adam\n",
        "from keras.callbacks import LearningRateScheduler"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qTFJhpFzMnps",
        "colab_type": "text"
      },
      "source": [
        "## **Here we have implemented densnet model for cifar10 database using only 50 epochs**\n",
        "\n",
        "We perform the learning task of densenet in two steps. First we take images of size 32X32 and downsize them to 16x16 and train the network for **20 epochs** defined by model **pre_model28_l** with parameters count **2,062,538**. \n",
        "Then we save these weights change the output layer of the previous model and make a new model **model28_l** with parameters count **2,072,618** and train it for actual images of size 32x32 for **30 epochs**. This sort of transfer-learning should facilitate the learning of network quite easier/faster.\n",
        "\n",
        "**Key techniques used are:**\n",
        "**1. Image Augmentation to avoid overfitting to the training data like **\n",
        "        rotation_range=12,  # randomly rotate images in the range (degrees, 0 to 180)\n",
        "        zoom_range = 0.15, # Randomly zoom image \n",
        "        width_shift_range=0.15,  # randomly shift images horizontally (fraction of total width)\n",
        "        height_shift_range=0.15,  # randomly shift images vertically (fraction of total height)\n",
        "        horizontal_flip=True,  # randomly flip images \n",
        "        \n",
        "**2. learning-rate scheduling for damping the ocsillations in validation accuracy defined by** \n",
        "        lr_schedule (for pre_model28_l) and \n",
        "        lr_schedule_1(for model28_l)\n",
        "        \n",
        " **Results of this experiments**\n",
        " Training accuracy: 93.91% and validation accuracy: 90.98%"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YZznMOAiPNrm",
        "colab_type": "text"
      },
      "source": [
        "# **Let us begin:**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OA3V6ceePGqF",
        "colab_type": "text"
      },
      "source": [
        "**lr_schedule** for **pre_model28_l**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aSL8yUrkbPe2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def lr_schedule(epoch):\n",
        "    lrate = 0.001\n",
        "    if epoch > 15:\n",
        "        lrate = 0.0001\n",
        "         \n",
        "    return lrate"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tlOMJmkMPM4l",
        "colab_type": "text"
      },
      "source": [
        "**lr_schedule_1** for **model28_l**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VKbSSPUCYvsh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def lr_schedule_1(epoch):\n",
        "    if epoch > 26:\n",
        "        lrate = 0.0001\n",
        "    elif epoch > 23:\n",
        "        lrate = 0.0002\n",
        "    elif epoch > 18:\n",
        "        lrate = 0.0003\n",
        "    elif epoch > 13:\n",
        "        lrate = 0.0005\n",
        "    else:\n",
        "        lrate = 0.001\n",
        "         \n",
        "    return lrate"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UNHw6luQg3gc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# this part will prevent tensorflow to allocate all the avaliable GPU Memory\n",
        "# backend\n",
        "import tensorflow as tf\n",
        "from keras import backend as k\n",
        "\n",
        "# Don't pre-allocate memory; allocate as-needed\n",
        "config = tf.ConfigProto()\n",
        "config.gpu_options.allow_growth = True\n",
        "\n",
        "# Create a session with the above options specified.\n",
        "k.tensorflow_backend.set_session(tf.Session(config=config))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3YYrZOxqPv7x",
        "colab_type": "text"
      },
      "source": [
        "# **Hyperparameters**\n",
        "**l=20, num_filter=32, compression=0.5 and drop_out=0.1**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dsO_yGxcg5D8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Hyperparameters\n",
        "batch_size = 64\n",
        "num_classes = 10\n",
        "epochs = 50\n",
        "l = 20\n",
        "num_filter = 32\n",
        "compression = 0.5\n",
        "dropout_rate = 0.1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xsTjDJBHQC9H",
        "colab_type": "text"
      },
      "source": [
        "**load cifar10 dataset**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mB7o3zu1g6eT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Load CIFAR10 Data\n",
        "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
        "img_height, img_width, channel = x_train.shape[1],x_train.shape[2],x_train.shape[3]\n",
        "\n",
        "# convert to one hot encoing \n",
        "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = keras.utils.to_categorical(y_test, num_classes)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1DNMmH_K1F5b",
        "colab_type": "text"
      },
      "source": [
        "## **RESIZING IMAGES FOR PRE-TRAINING**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PE9uhHa1x2Bj",
        "colab_type": "code",
        "outputId": "d54d89d6-d432-4948-c7d0-8f59fd9eb06b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "x_train.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(50000, 32, 32, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NVlr1K2i00-N",
        "colab_type": "code",
        "outputId": "7ab2a46a-dca3-431c-f5c2-ba4d6d3eb9e6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "x_train_resized = np.array([scipy.misc.imresize(image, (16, 16),mode='RGB') for image in x_train])\n",
        "x_train_resized.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
            "  if issubdtype(ts, int):\n",
            "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
            "  elif issubdtype(type(size), float):\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(50000, 16, 16, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZBlVN32yHL2P",
        "colab_type": "code",
        "outputId": "732023e3-94c7-4ba7-c8f5-e4c83cd8a12a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "x_test.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, 32, 32, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A2y5c1bi1a4h",
        "colab_type": "code",
        "outputId": "8088f846-a0bf-4891-e2fa-fbfaeb8c245d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        }
      },
      "source": [
        "\n",
        "x_test_resized = np.array([scipy.misc.imresize(image, (16, 16),mode='RGB') for image in x_test])\n",
        "x_test_resized.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
            "  if issubdtype(ts, int):\n",
            "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
            "  elif issubdtype(type(size), float):\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, 16, 16, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e7niw6qgFyDn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "img_height, img_width, channel = x_train_resized.shape[1],x_train_resized.shape[2],x_train_resized.shape[3] ###used for pre-training the model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nSYYkKfbQI53",
        "colab_type": "text"
      },
      "source": [
        "# **Define data augmentation using ImageDataGenerator facility provided by keras**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qBnl6TkJMKcB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# With data augmentation to prevent overfitting \n",
        "\n",
        "datagen = ImageDataGenerator(\n",
        "        featurewise_center=False,  # set input mean to 0 over the dataset\n",
        "        samplewise_center=False,  # set each sample mean to 0\n",
        "        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n",
        "        samplewise_std_normalization=False,  # divide each input by its std\n",
        "        zca_whitening=False,  # apply ZCA whitening\n",
        "        rotation_range=12,  # randomly rotate images in the range (degrees, 0 to 180)\n",
        "        zoom_range = 0.15, # Randomly zoom image \n",
        "        width_shift_range=0.15,  # randomly shift images horizontally (fraction of total width)\n",
        "        height_shift_range=0.15,  # randomly shift images vertically (fraction of total height)\n",
        "        horizontal_flip=True,  # randomly flip images\n",
        "        vertical_flip=False)  # randomly flip images"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eYAS3TGxQRSA",
        "colab_type": "text"
      },
      "source": [
        "## **Denseblock, add_transition, output layer for the densenet given to us**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ee-sge5Kg7vr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Dense Block\n",
        "def add_denseblock(input, num_filter = 12, dropout_rate = 0.2):\n",
        "    global compression\n",
        "    temp = input\n",
        "    for _ in range(l):\n",
        "        BatchNorm = BatchNormalization(name='Batch'+str(block)+str(_))(temp)\n",
        "        relu = Activation('relu',name='Activation'+str(block)+str(_))(BatchNorm)\n",
        "        Conv2D_3_3 = Conv2D(int(num_filter*compression), (3,3), use_bias=False ,padding='same',name='Conv'+str(block)+str(_))(relu)\n",
        "        if dropout_rate>0:\n",
        "          Conv2D_3_3 = Dropout(dropout_rate,name='Dropout'+str(block)+str(_))(Conv2D_3_3)\n",
        "        concat = Concatenate(axis=-1,name='Concat'+str(block)+str(_))([temp,Conv2D_3_3])\n",
        "        \n",
        "        temp = concat\n",
        "        \n",
        "    return temp"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OOP6IPsGhBwb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def add_transition(input, num_filter = 12, dropout_rate = 0.2):\n",
        "    global compression\n",
        "    BatchNorm = BatchNormalization(name='Batch'+'_transition'+str(block))(input)\n",
        "    relu = Activation('relu',name='Activation'+'_transition'+str(block))(BatchNorm)\n",
        "    Conv2D_BottleNeck = Conv2D(int(num_filter*compression), (1,1), use_bias=False ,padding='same',name='Conv'+'_transition'+str(block))(relu)\n",
        "    if dropout_rate>0:\n",
        "      Conv2D_BottleNeck = Dropout(dropout_rate,name='Dropout'+'_transition'+str(block))(Conv2D_BottleNeck)\n",
        "    avg = AveragePooling2D(pool_size=(2,2),name='AvgPooling'+'_transition'+str(block))(Conv2D_BottleNeck)\n",
        "    \n",
        "    return avg"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0RaKFpubhDIC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def output_layer(input):\n",
        "    global compression\n",
        "    BatchNorm = BatchNormalization(name='Batch_output_layer')(input)\n",
        "    relu = Activation('relu',name='Activation_output_layer')(BatchNorm)\n",
        "    AvgPooling = AveragePooling2D(pool_size=(2,2),name='AvgPooling_output_layer')(relu)\n",
        "    flat = Flatten(name='Flattern_output_layer')(AvgPooling)\n",
        "    output = Dense(num_classes, activation='softmax',name='Dense_output_layer')(flat)\n",
        "    \n",
        "    return output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YBOnTKxWQeNT",
        "colab_type": "text"
      },
      "source": [
        "# **Defining pre_model28_l for input size 16x16**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "anPCpQWhhGb7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "dropout_rate = 0.1\n",
        "input = Input(shape=(img_height, img_width, channel,),name='input_layer')\n",
        "First_Conv2D = Conv2D(num_filter, (3,3), use_bias=False ,padding='same',name='Conv2D_1st')(input)\n",
        "block=1\n",
        "First_Block = add_denseblock(First_Conv2D, num_filter, dropout_rate)\n",
        "First_Transition = add_transition(First_Block, num_filter, dropout_rate)\n",
        "block=2\n",
        "Second_Block = add_denseblock(First_Transition, num_filter, dropout_rate)\n",
        "Second_Transition = add_transition(Second_Block, num_filter, dropout_rate)\n",
        "block=3\n",
        "Third_Block = add_denseblock(Second_Transition, num_filter, dropout_rate)\n",
        "Third_Transition = add_transition(Third_Block, num_filter, dropout_rate)\n",
        "block=4\n",
        "Last_Block = add_denseblock(Third_Transition,  num_filter, dropout_rate)\n",
        "output = output_layer(Last_Block)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1kFh7pdxhNtT",
        "colab_type": "code",
        "outputId": "c20ac08f-bc24-43d3-ac6c-f55e370d5110",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 15658
        }
      },
      "source": [
        "pre_model28_l = Model(inputs=[input], outputs=[output])\n",
        "pre_model28_l.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_layer (InputLayer)        (None, 16, 16, 3)    0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Conv2D_1st (Conv2D)             (None, 16, 16, 32)   864         input_layer[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Batch10 (BatchNormalization)    (None, 16, 16, 32)   128         Conv2D_1st[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Activation10 (Activation)       (None, 16, 16, 32)   0           Batch10[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv10 (Conv2D)                 (None, 16, 16, 16)   4608        Activation10[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout10 (Dropout)             (None, 16, 16, 16)   0           Conv10[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat10 (Concatenate)          (None, 16, 16, 48)   0           Conv2D_1st[0][0]                 \n",
            "                                                                 Dropout10[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch11 (BatchNormalization)    (None, 16, 16, 48)   192         Concat10[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation11 (Activation)       (None, 16, 16, 48)   0           Batch11[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv11 (Conv2D)                 (None, 16, 16, 16)   6912        Activation11[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout11 (Dropout)             (None, 16, 16, 16)   0           Conv11[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat11 (Concatenate)          (None, 16, 16, 64)   0           Concat10[0][0]                   \n",
            "                                                                 Dropout11[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch12 (BatchNormalization)    (None, 16, 16, 64)   256         Concat11[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation12 (Activation)       (None, 16, 16, 64)   0           Batch12[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv12 (Conv2D)                 (None, 16, 16, 16)   9216        Activation12[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout12 (Dropout)             (None, 16, 16, 16)   0           Conv12[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat12 (Concatenate)          (None, 16, 16, 80)   0           Concat11[0][0]                   \n",
            "                                                                 Dropout12[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch13 (BatchNormalization)    (None, 16, 16, 80)   320         Concat12[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation13 (Activation)       (None, 16, 16, 80)   0           Batch13[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv13 (Conv2D)                 (None, 16, 16, 16)   11520       Activation13[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout13 (Dropout)             (None, 16, 16, 16)   0           Conv13[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat13 (Concatenate)          (None, 16, 16, 96)   0           Concat12[0][0]                   \n",
            "                                                                 Dropout13[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch14 (BatchNormalization)    (None, 16, 16, 96)   384         Concat13[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation14 (Activation)       (None, 16, 16, 96)   0           Batch14[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv14 (Conv2D)                 (None, 16, 16, 16)   13824       Activation14[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout14 (Dropout)             (None, 16, 16, 16)   0           Conv14[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat14 (Concatenate)          (None, 16, 16, 112)  0           Concat13[0][0]                   \n",
            "                                                                 Dropout14[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch15 (BatchNormalization)    (None, 16, 16, 112)  448         Concat14[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation15 (Activation)       (None, 16, 16, 112)  0           Batch15[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv15 (Conv2D)                 (None, 16, 16, 16)   16128       Activation15[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout15 (Dropout)             (None, 16, 16, 16)   0           Conv15[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat15 (Concatenate)          (None, 16, 16, 128)  0           Concat14[0][0]                   \n",
            "                                                                 Dropout15[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch16 (BatchNormalization)    (None, 16, 16, 128)  512         Concat15[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation16 (Activation)       (None, 16, 16, 128)  0           Batch16[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv16 (Conv2D)                 (None, 16, 16, 16)   18432       Activation16[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout16 (Dropout)             (None, 16, 16, 16)   0           Conv16[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat16 (Concatenate)          (None, 16, 16, 144)  0           Concat15[0][0]                   \n",
            "                                                                 Dropout16[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch17 (BatchNormalization)    (None, 16, 16, 144)  576         Concat16[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation17 (Activation)       (None, 16, 16, 144)  0           Batch17[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv17 (Conv2D)                 (None, 16, 16, 16)   20736       Activation17[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout17 (Dropout)             (None, 16, 16, 16)   0           Conv17[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat17 (Concatenate)          (None, 16, 16, 160)  0           Concat16[0][0]                   \n",
            "                                                                 Dropout17[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch18 (BatchNormalization)    (None, 16, 16, 160)  640         Concat17[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation18 (Activation)       (None, 16, 16, 160)  0           Batch18[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv18 (Conv2D)                 (None, 16, 16, 16)   23040       Activation18[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout18 (Dropout)             (None, 16, 16, 16)   0           Conv18[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat18 (Concatenate)          (None, 16, 16, 176)  0           Concat17[0][0]                   \n",
            "                                                                 Dropout18[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch19 (BatchNormalization)    (None, 16, 16, 176)  704         Concat18[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation19 (Activation)       (None, 16, 16, 176)  0           Batch19[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv19 (Conv2D)                 (None, 16, 16, 16)   25344       Activation19[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout19 (Dropout)             (None, 16, 16, 16)   0           Conv19[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat19 (Concatenate)          (None, 16, 16, 192)  0           Concat18[0][0]                   \n",
            "                                                                 Dropout19[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch110 (BatchNormalization)   (None, 16, 16, 192)  768         Concat19[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation110 (Activation)      (None, 16, 16, 192)  0           Batch110[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv110 (Conv2D)                (None, 16, 16, 16)   27648       Activation110[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout110 (Dropout)            (None, 16, 16, 16)   0           Conv110[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat110 (Concatenate)         (None, 16, 16, 208)  0           Concat19[0][0]                   \n",
            "                                                                 Dropout110[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch111 (BatchNormalization)   (None, 16, 16, 208)  832         Concat110[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation111 (Activation)      (None, 16, 16, 208)  0           Batch111[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv111 (Conv2D)                (None, 16, 16, 16)   29952       Activation111[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout111 (Dropout)            (None, 16, 16, 16)   0           Conv111[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat111 (Concatenate)         (None, 16, 16, 224)  0           Concat110[0][0]                  \n",
            "                                                                 Dropout111[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch112 (BatchNormalization)   (None, 16, 16, 224)  896         Concat111[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation112 (Activation)      (None, 16, 16, 224)  0           Batch112[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv112 (Conv2D)                (None, 16, 16, 16)   32256       Activation112[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout112 (Dropout)            (None, 16, 16, 16)   0           Conv112[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat112 (Concatenate)         (None, 16, 16, 240)  0           Concat111[0][0]                  \n",
            "                                                                 Dropout112[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch113 (BatchNormalization)   (None, 16, 16, 240)  960         Concat112[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation113 (Activation)      (None, 16, 16, 240)  0           Batch113[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv113 (Conv2D)                (None, 16, 16, 16)   34560       Activation113[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout113 (Dropout)            (None, 16, 16, 16)   0           Conv113[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat113 (Concatenate)         (None, 16, 16, 256)  0           Concat112[0][0]                  \n",
            "                                                                 Dropout113[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch114 (BatchNormalization)   (None, 16, 16, 256)  1024        Concat113[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation114 (Activation)      (None, 16, 16, 256)  0           Batch114[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv114 (Conv2D)                (None, 16, 16, 16)   36864       Activation114[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout114 (Dropout)            (None, 16, 16, 16)   0           Conv114[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat114 (Concatenate)         (None, 16, 16, 272)  0           Concat113[0][0]                  \n",
            "                                                                 Dropout114[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch115 (BatchNormalization)   (None, 16, 16, 272)  1088        Concat114[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation115 (Activation)      (None, 16, 16, 272)  0           Batch115[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv115 (Conv2D)                (None, 16, 16, 16)   39168       Activation115[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout115 (Dropout)            (None, 16, 16, 16)   0           Conv115[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat115 (Concatenate)         (None, 16, 16, 288)  0           Concat114[0][0]                  \n",
            "                                                                 Dropout115[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch116 (BatchNormalization)   (None, 16, 16, 288)  1152        Concat115[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation116 (Activation)      (None, 16, 16, 288)  0           Batch116[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv116 (Conv2D)                (None, 16, 16, 16)   41472       Activation116[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout116 (Dropout)            (None, 16, 16, 16)   0           Conv116[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat116 (Concatenate)         (None, 16, 16, 304)  0           Concat115[0][0]                  \n",
            "                                                                 Dropout116[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch117 (BatchNormalization)   (None, 16, 16, 304)  1216        Concat116[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation117 (Activation)      (None, 16, 16, 304)  0           Batch117[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv117 (Conv2D)                (None, 16, 16, 16)   43776       Activation117[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout117 (Dropout)            (None, 16, 16, 16)   0           Conv117[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat117 (Concatenate)         (None, 16, 16, 320)  0           Concat116[0][0]                  \n",
            "                                                                 Dropout117[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch118 (BatchNormalization)   (None, 16, 16, 320)  1280        Concat117[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation118 (Activation)      (None, 16, 16, 320)  0           Batch118[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv118 (Conv2D)                (None, 16, 16, 16)   46080       Activation118[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout118 (Dropout)            (None, 16, 16, 16)   0           Conv118[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat118 (Concatenate)         (None, 16, 16, 336)  0           Concat117[0][0]                  \n",
            "                                                                 Dropout118[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch119 (BatchNormalization)   (None, 16, 16, 336)  1344        Concat118[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation119 (Activation)      (None, 16, 16, 336)  0           Batch119[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv119 (Conv2D)                (None, 16, 16, 16)   48384       Activation119[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout119 (Dropout)            (None, 16, 16, 16)   0           Conv119[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat119 (Concatenate)         (None, 16, 16, 352)  0           Concat118[0][0]                  \n",
            "                                                                 Dropout119[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch_transition1 (BatchNormali (None, 16, 16, 352)  1408        Concat119[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation_transition1 (Activat (None, 16, 16, 352)  0           Batch_transition1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Conv_transition1 (Conv2D)       (None, 16, 16, 16)   5632        Activation_transition1[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Dropout_transition1 (Dropout)   (None, 16, 16, 16)   0           Conv_transition1[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "AvgPooling_transition1 (Average (None, 8, 8, 16)     0           Dropout_transition1[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "Batch20 (BatchNormalization)    (None, 8, 8, 16)     64          AvgPooling_transition1[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Activation20 (Activation)       (None, 8, 8, 16)     0           Batch20[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv20 (Conv2D)                 (None, 8, 8, 16)     2304        Activation20[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout20 (Dropout)             (None, 8, 8, 16)     0           Conv20[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat20 (Concatenate)          (None, 8, 8, 32)     0           AvgPooling_transition1[0][0]     \n",
            "                                                                 Dropout20[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch21 (BatchNormalization)    (None, 8, 8, 32)     128         Concat20[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation21 (Activation)       (None, 8, 8, 32)     0           Batch21[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv21 (Conv2D)                 (None, 8, 8, 16)     4608        Activation21[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout21 (Dropout)             (None, 8, 8, 16)     0           Conv21[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat21 (Concatenate)          (None, 8, 8, 48)     0           Concat20[0][0]                   \n",
            "                                                                 Dropout21[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch22 (BatchNormalization)    (None, 8, 8, 48)     192         Concat21[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation22 (Activation)       (None, 8, 8, 48)     0           Batch22[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv22 (Conv2D)                 (None, 8, 8, 16)     6912        Activation22[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout22 (Dropout)             (None, 8, 8, 16)     0           Conv22[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat22 (Concatenate)          (None, 8, 8, 64)     0           Concat21[0][0]                   \n",
            "                                                                 Dropout22[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch23 (BatchNormalization)    (None, 8, 8, 64)     256         Concat22[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation23 (Activation)       (None, 8, 8, 64)     0           Batch23[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv23 (Conv2D)                 (None, 8, 8, 16)     9216        Activation23[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout23 (Dropout)             (None, 8, 8, 16)     0           Conv23[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat23 (Concatenate)          (None, 8, 8, 80)     0           Concat22[0][0]                   \n",
            "                                                                 Dropout23[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch24 (BatchNormalization)    (None, 8, 8, 80)     320         Concat23[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation24 (Activation)       (None, 8, 8, 80)     0           Batch24[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv24 (Conv2D)                 (None, 8, 8, 16)     11520       Activation24[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout24 (Dropout)             (None, 8, 8, 16)     0           Conv24[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat24 (Concatenate)          (None, 8, 8, 96)     0           Concat23[0][0]                   \n",
            "                                                                 Dropout24[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch25 (BatchNormalization)    (None, 8, 8, 96)     384         Concat24[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation25 (Activation)       (None, 8, 8, 96)     0           Batch25[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv25 (Conv2D)                 (None, 8, 8, 16)     13824       Activation25[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout25 (Dropout)             (None, 8, 8, 16)     0           Conv25[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat25 (Concatenate)          (None, 8, 8, 112)    0           Concat24[0][0]                   \n",
            "                                                                 Dropout25[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch26 (BatchNormalization)    (None, 8, 8, 112)    448         Concat25[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation26 (Activation)       (None, 8, 8, 112)    0           Batch26[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv26 (Conv2D)                 (None, 8, 8, 16)     16128       Activation26[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout26 (Dropout)             (None, 8, 8, 16)     0           Conv26[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat26 (Concatenate)          (None, 8, 8, 128)    0           Concat25[0][0]                   \n",
            "                                                                 Dropout26[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch27 (BatchNormalization)    (None, 8, 8, 128)    512         Concat26[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation27 (Activation)       (None, 8, 8, 128)    0           Batch27[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv27 (Conv2D)                 (None, 8, 8, 16)     18432       Activation27[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout27 (Dropout)             (None, 8, 8, 16)     0           Conv27[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat27 (Concatenate)          (None, 8, 8, 144)    0           Concat26[0][0]                   \n",
            "                                                                 Dropout27[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch28 (BatchNormalization)    (None, 8, 8, 144)    576         Concat27[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation28 (Activation)       (None, 8, 8, 144)    0           Batch28[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv28 (Conv2D)                 (None, 8, 8, 16)     20736       Activation28[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout28 (Dropout)             (None, 8, 8, 16)     0           Conv28[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat28 (Concatenate)          (None, 8, 8, 160)    0           Concat27[0][0]                   \n",
            "                                                                 Dropout28[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch29 (BatchNormalization)    (None, 8, 8, 160)    640         Concat28[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation29 (Activation)       (None, 8, 8, 160)    0           Batch29[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv29 (Conv2D)                 (None, 8, 8, 16)     23040       Activation29[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout29 (Dropout)             (None, 8, 8, 16)     0           Conv29[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat29 (Concatenate)          (None, 8, 8, 176)    0           Concat28[0][0]                   \n",
            "                                                                 Dropout29[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch210 (BatchNormalization)   (None, 8, 8, 176)    704         Concat29[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation210 (Activation)      (None, 8, 8, 176)    0           Batch210[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv210 (Conv2D)                (None, 8, 8, 16)     25344       Activation210[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout210 (Dropout)            (None, 8, 8, 16)     0           Conv210[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat210 (Concatenate)         (None, 8, 8, 192)    0           Concat29[0][0]                   \n",
            "                                                                 Dropout210[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch211 (BatchNormalization)   (None, 8, 8, 192)    768         Concat210[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation211 (Activation)      (None, 8, 8, 192)    0           Batch211[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv211 (Conv2D)                (None, 8, 8, 16)     27648       Activation211[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout211 (Dropout)            (None, 8, 8, 16)     0           Conv211[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat211 (Concatenate)         (None, 8, 8, 208)    0           Concat210[0][0]                  \n",
            "                                                                 Dropout211[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch212 (BatchNormalization)   (None, 8, 8, 208)    832         Concat211[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation212 (Activation)      (None, 8, 8, 208)    0           Batch212[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv212 (Conv2D)                (None, 8, 8, 16)     29952       Activation212[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout212 (Dropout)            (None, 8, 8, 16)     0           Conv212[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat212 (Concatenate)         (None, 8, 8, 224)    0           Concat211[0][0]                  \n",
            "                                                                 Dropout212[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch213 (BatchNormalization)   (None, 8, 8, 224)    896         Concat212[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation213 (Activation)      (None, 8, 8, 224)    0           Batch213[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv213 (Conv2D)                (None, 8, 8, 16)     32256       Activation213[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout213 (Dropout)            (None, 8, 8, 16)     0           Conv213[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat213 (Concatenate)         (None, 8, 8, 240)    0           Concat212[0][0]                  \n",
            "                                                                 Dropout213[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch214 (BatchNormalization)   (None, 8, 8, 240)    960         Concat213[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation214 (Activation)      (None, 8, 8, 240)    0           Batch214[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv214 (Conv2D)                (None, 8, 8, 16)     34560       Activation214[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout214 (Dropout)            (None, 8, 8, 16)     0           Conv214[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat214 (Concatenate)         (None, 8, 8, 256)    0           Concat213[0][0]                  \n",
            "                                                                 Dropout214[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch215 (BatchNormalization)   (None, 8, 8, 256)    1024        Concat214[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation215 (Activation)      (None, 8, 8, 256)    0           Batch215[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv215 (Conv2D)                (None, 8, 8, 16)     36864       Activation215[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout215 (Dropout)            (None, 8, 8, 16)     0           Conv215[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat215 (Concatenate)         (None, 8, 8, 272)    0           Concat214[0][0]                  \n",
            "                                                                 Dropout215[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch216 (BatchNormalization)   (None, 8, 8, 272)    1088        Concat215[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation216 (Activation)      (None, 8, 8, 272)    0           Batch216[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv216 (Conv2D)                (None, 8, 8, 16)     39168       Activation216[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout216 (Dropout)            (None, 8, 8, 16)     0           Conv216[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat216 (Concatenate)         (None, 8, 8, 288)    0           Concat215[0][0]                  \n",
            "                                                                 Dropout216[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch217 (BatchNormalization)   (None, 8, 8, 288)    1152        Concat216[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation217 (Activation)      (None, 8, 8, 288)    0           Batch217[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv217 (Conv2D)                (None, 8, 8, 16)     41472       Activation217[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout217 (Dropout)            (None, 8, 8, 16)     0           Conv217[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat217 (Concatenate)         (None, 8, 8, 304)    0           Concat216[0][0]                  \n",
            "                                                                 Dropout217[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch218 (BatchNormalization)   (None, 8, 8, 304)    1216        Concat217[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation218 (Activation)      (None, 8, 8, 304)    0           Batch218[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv218 (Conv2D)                (None, 8, 8, 16)     43776       Activation218[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout218 (Dropout)            (None, 8, 8, 16)     0           Conv218[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat218 (Concatenate)         (None, 8, 8, 320)    0           Concat217[0][0]                  \n",
            "                                                                 Dropout218[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch219 (BatchNormalization)   (None, 8, 8, 320)    1280        Concat218[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation219 (Activation)      (None, 8, 8, 320)    0           Batch219[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv219 (Conv2D)                (None, 8, 8, 16)     46080       Activation219[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout219 (Dropout)            (None, 8, 8, 16)     0           Conv219[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat219 (Concatenate)         (None, 8, 8, 336)    0           Concat218[0][0]                  \n",
            "                                                                 Dropout219[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch_transition2 (BatchNormali (None, 8, 8, 336)    1344        Concat219[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation_transition2 (Activat (None, 8, 8, 336)    0           Batch_transition2[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Conv_transition2 (Conv2D)       (None, 8, 8, 16)     5376        Activation_transition2[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Dropout_transition2 (Dropout)   (None, 8, 8, 16)     0           Conv_transition2[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "AvgPooling_transition2 (Average (None, 4, 4, 16)     0           Dropout_transition2[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "Batch30 (BatchNormalization)    (None, 4, 4, 16)     64          AvgPooling_transition2[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Activation30 (Activation)       (None, 4, 4, 16)     0           Batch30[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv30 (Conv2D)                 (None, 4, 4, 16)     2304        Activation30[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout30 (Dropout)             (None, 4, 4, 16)     0           Conv30[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat30 (Concatenate)          (None, 4, 4, 32)     0           AvgPooling_transition2[0][0]     \n",
            "                                                                 Dropout30[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch31 (BatchNormalization)    (None, 4, 4, 32)     128         Concat30[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation31 (Activation)       (None, 4, 4, 32)     0           Batch31[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv31 (Conv2D)                 (None, 4, 4, 16)     4608        Activation31[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout31 (Dropout)             (None, 4, 4, 16)     0           Conv31[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat31 (Concatenate)          (None, 4, 4, 48)     0           Concat30[0][0]                   \n",
            "                                                                 Dropout31[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch32 (BatchNormalization)    (None, 4, 4, 48)     192         Concat31[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation32 (Activation)       (None, 4, 4, 48)     0           Batch32[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv32 (Conv2D)                 (None, 4, 4, 16)     6912        Activation32[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout32 (Dropout)             (None, 4, 4, 16)     0           Conv32[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat32 (Concatenate)          (None, 4, 4, 64)     0           Concat31[0][0]                   \n",
            "                                                                 Dropout32[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch33 (BatchNormalization)    (None, 4, 4, 64)     256         Concat32[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation33 (Activation)       (None, 4, 4, 64)     0           Batch33[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv33 (Conv2D)                 (None, 4, 4, 16)     9216        Activation33[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout33 (Dropout)             (None, 4, 4, 16)     0           Conv33[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat33 (Concatenate)          (None, 4, 4, 80)     0           Concat32[0][0]                   \n",
            "                                                                 Dropout33[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch34 (BatchNormalization)    (None, 4, 4, 80)     320         Concat33[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation34 (Activation)       (None, 4, 4, 80)     0           Batch34[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv34 (Conv2D)                 (None, 4, 4, 16)     11520       Activation34[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout34 (Dropout)             (None, 4, 4, 16)     0           Conv34[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat34 (Concatenate)          (None, 4, 4, 96)     0           Concat33[0][0]                   \n",
            "                                                                 Dropout34[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch35 (BatchNormalization)    (None, 4, 4, 96)     384         Concat34[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation35 (Activation)       (None, 4, 4, 96)     0           Batch35[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv35 (Conv2D)                 (None, 4, 4, 16)     13824       Activation35[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout35 (Dropout)             (None, 4, 4, 16)     0           Conv35[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat35 (Concatenate)          (None, 4, 4, 112)    0           Concat34[0][0]                   \n",
            "                                                                 Dropout35[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch36 (BatchNormalization)    (None, 4, 4, 112)    448         Concat35[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation36 (Activation)       (None, 4, 4, 112)    0           Batch36[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv36 (Conv2D)                 (None, 4, 4, 16)     16128       Activation36[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout36 (Dropout)             (None, 4, 4, 16)     0           Conv36[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat36 (Concatenate)          (None, 4, 4, 128)    0           Concat35[0][0]                   \n",
            "                                                                 Dropout36[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch37 (BatchNormalization)    (None, 4, 4, 128)    512         Concat36[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation37 (Activation)       (None, 4, 4, 128)    0           Batch37[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv37 (Conv2D)                 (None, 4, 4, 16)     18432       Activation37[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout37 (Dropout)             (None, 4, 4, 16)     0           Conv37[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat37 (Concatenate)          (None, 4, 4, 144)    0           Concat36[0][0]                   \n",
            "                                                                 Dropout37[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch38 (BatchNormalization)    (None, 4, 4, 144)    576         Concat37[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation38 (Activation)       (None, 4, 4, 144)    0           Batch38[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv38 (Conv2D)                 (None, 4, 4, 16)     20736       Activation38[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout38 (Dropout)             (None, 4, 4, 16)     0           Conv38[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat38 (Concatenate)          (None, 4, 4, 160)    0           Concat37[0][0]                   \n",
            "                                                                 Dropout38[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch39 (BatchNormalization)    (None, 4, 4, 160)    640         Concat38[0][0]                   \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "__________________________________________________________________________________________________\n",
            "Activation39 (Activation)       (None, 4, 4, 160)    0           Batch39[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv39 (Conv2D)                 (None, 4, 4, 16)     23040       Activation39[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout39 (Dropout)             (None, 4, 4, 16)     0           Conv39[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat39 (Concatenate)          (None, 4, 4, 176)    0           Concat38[0][0]                   \n",
            "                                                                 Dropout39[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch310 (BatchNormalization)   (None, 4, 4, 176)    704         Concat39[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation310 (Activation)      (None, 4, 4, 176)    0           Batch310[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv310 (Conv2D)                (None, 4, 4, 16)     25344       Activation310[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout310 (Dropout)            (None, 4, 4, 16)     0           Conv310[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat310 (Concatenate)         (None, 4, 4, 192)    0           Concat39[0][0]                   \n",
            "                                                                 Dropout310[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch311 (BatchNormalization)   (None, 4, 4, 192)    768         Concat310[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation311 (Activation)      (None, 4, 4, 192)    0           Batch311[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv311 (Conv2D)                (None, 4, 4, 16)     27648       Activation311[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout311 (Dropout)            (None, 4, 4, 16)     0           Conv311[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat311 (Concatenate)         (None, 4, 4, 208)    0           Concat310[0][0]                  \n",
            "                                                                 Dropout311[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch312 (BatchNormalization)   (None, 4, 4, 208)    832         Concat311[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation312 (Activation)      (None, 4, 4, 208)    0           Batch312[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv312 (Conv2D)                (None, 4, 4, 16)     29952       Activation312[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout312 (Dropout)            (None, 4, 4, 16)     0           Conv312[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat312 (Concatenate)         (None, 4, 4, 224)    0           Concat311[0][0]                  \n",
            "                                                                 Dropout312[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch313 (BatchNormalization)   (None, 4, 4, 224)    896         Concat312[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation313 (Activation)      (None, 4, 4, 224)    0           Batch313[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv313 (Conv2D)                (None, 4, 4, 16)     32256       Activation313[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout313 (Dropout)            (None, 4, 4, 16)     0           Conv313[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat313 (Concatenate)         (None, 4, 4, 240)    0           Concat312[0][0]                  \n",
            "                                                                 Dropout313[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch314 (BatchNormalization)   (None, 4, 4, 240)    960         Concat313[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation314 (Activation)      (None, 4, 4, 240)    0           Batch314[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv314 (Conv2D)                (None, 4, 4, 16)     34560       Activation314[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout314 (Dropout)            (None, 4, 4, 16)     0           Conv314[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat314 (Concatenate)         (None, 4, 4, 256)    0           Concat313[0][0]                  \n",
            "                                                                 Dropout314[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch315 (BatchNormalization)   (None, 4, 4, 256)    1024        Concat314[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation315 (Activation)      (None, 4, 4, 256)    0           Batch315[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv315 (Conv2D)                (None, 4, 4, 16)     36864       Activation315[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout315 (Dropout)            (None, 4, 4, 16)     0           Conv315[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat315 (Concatenate)         (None, 4, 4, 272)    0           Concat314[0][0]                  \n",
            "                                                                 Dropout315[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch316 (BatchNormalization)   (None, 4, 4, 272)    1088        Concat315[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation316 (Activation)      (None, 4, 4, 272)    0           Batch316[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv316 (Conv2D)                (None, 4, 4, 16)     39168       Activation316[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout316 (Dropout)            (None, 4, 4, 16)     0           Conv316[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat316 (Concatenate)         (None, 4, 4, 288)    0           Concat315[0][0]                  \n",
            "                                                                 Dropout316[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch317 (BatchNormalization)   (None, 4, 4, 288)    1152        Concat316[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation317 (Activation)      (None, 4, 4, 288)    0           Batch317[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv317 (Conv2D)                (None, 4, 4, 16)     41472       Activation317[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout317 (Dropout)            (None, 4, 4, 16)     0           Conv317[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat317 (Concatenate)         (None, 4, 4, 304)    0           Concat316[0][0]                  \n",
            "                                                                 Dropout317[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch318 (BatchNormalization)   (None, 4, 4, 304)    1216        Concat317[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation318 (Activation)      (None, 4, 4, 304)    0           Batch318[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv318 (Conv2D)                (None, 4, 4, 16)     43776       Activation318[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout318 (Dropout)            (None, 4, 4, 16)     0           Conv318[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat318 (Concatenate)         (None, 4, 4, 320)    0           Concat317[0][0]                  \n",
            "                                                                 Dropout318[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch319 (BatchNormalization)   (None, 4, 4, 320)    1280        Concat318[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation319 (Activation)      (None, 4, 4, 320)    0           Batch319[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv319 (Conv2D)                (None, 4, 4, 16)     46080       Activation319[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout319 (Dropout)            (None, 4, 4, 16)     0           Conv319[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat319 (Concatenate)         (None, 4, 4, 336)    0           Concat318[0][0]                  \n",
            "                                                                 Dropout319[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch_transition3 (BatchNormali (None, 4, 4, 336)    1344        Concat319[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation_transition3 (Activat (None, 4, 4, 336)    0           Batch_transition3[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Conv_transition3 (Conv2D)       (None, 4, 4, 16)     5376        Activation_transition3[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Dropout_transition3 (Dropout)   (None, 4, 4, 16)     0           Conv_transition3[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "AvgPooling_transition3 (Average (None, 2, 2, 16)     0           Dropout_transition3[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "Batch40 (BatchNormalization)    (None, 2, 2, 16)     64          AvgPooling_transition3[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Activation40 (Activation)       (None, 2, 2, 16)     0           Batch40[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv40 (Conv2D)                 (None, 2, 2, 16)     2304        Activation40[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout40 (Dropout)             (None, 2, 2, 16)     0           Conv40[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat40 (Concatenate)          (None, 2, 2, 32)     0           AvgPooling_transition3[0][0]     \n",
            "                                                                 Dropout40[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch41 (BatchNormalization)    (None, 2, 2, 32)     128         Concat40[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation41 (Activation)       (None, 2, 2, 32)     0           Batch41[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv41 (Conv2D)                 (None, 2, 2, 16)     4608        Activation41[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout41 (Dropout)             (None, 2, 2, 16)     0           Conv41[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat41 (Concatenate)          (None, 2, 2, 48)     0           Concat40[0][0]                   \n",
            "                                                                 Dropout41[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch42 (BatchNormalization)    (None, 2, 2, 48)     192         Concat41[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation42 (Activation)       (None, 2, 2, 48)     0           Batch42[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv42 (Conv2D)                 (None, 2, 2, 16)     6912        Activation42[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout42 (Dropout)             (None, 2, 2, 16)     0           Conv42[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat42 (Concatenate)          (None, 2, 2, 64)     0           Concat41[0][0]                   \n",
            "                                                                 Dropout42[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch43 (BatchNormalization)    (None, 2, 2, 64)     256         Concat42[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation43 (Activation)       (None, 2, 2, 64)     0           Batch43[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv43 (Conv2D)                 (None, 2, 2, 16)     9216        Activation43[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout43 (Dropout)             (None, 2, 2, 16)     0           Conv43[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat43 (Concatenate)          (None, 2, 2, 80)     0           Concat42[0][0]                   \n",
            "                                                                 Dropout43[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch44 (BatchNormalization)    (None, 2, 2, 80)     320         Concat43[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation44 (Activation)       (None, 2, 2, 80)     0           Batch44[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv44 (Conv2D)                 (None, 2, 2, 16)     11520       Activation44[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout44 (Dropout)             (None, 2, 2, 16)     0           Conv44[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat44 (Concatenate)          (None, 2, 2, 96)     0           Concat43[0][0]                   \n",
            "                                                                 Dropout44[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch45 (BatchNormalization)    (None, 2, 2, 96)     384         Concat44[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation45 (Activation)       (None, 2, 2, 96)     0           Batch45[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv45 (Conv2D)                 (None, 2, 2, 16)     13824       Activation45[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout45 (Dropout)             (None, 2, 2, 16)     0           Conv45[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat45 (Concatenate)          (None, 2, 2, 112)    0           Concat44[0][0]                   \n",
            "                                                                 Dropout45[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch46 (BatchNormalization)    (None, 2, 2, 112)    448         Concat45[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation46 (Activation)       (None, 2, 2, 112)    0           Batch46[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv46 (Conv2D)                 (None, 2, 2, 16)     16128       Activation46[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout46 (Dropout)             (None, 2, 2, 16)     0           Conv46[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat46 (Concatenate)          (None, 2, 2, 128)    0           Concat45[0][0]                   \n",
            "                                                                 Dropout46[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch47 (BatchNormalization)    (None, 2, 2, 128)    512         Concat46[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation47 (Activation)       (None, 2, 2, 128)    0           Batch47[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv47 (Conv2D)                 (None, 2, 2, 16)     18432       Activation47[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout47 (Dropout)             (None, 2, 2, 16)     0           Conv47[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat47 (Concatenate)          (None, 2, 2, 144)    0           Concat46[0][0]                   \n",
            "                                                                 Dropout47[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch48 (BatchNormalization)    (None, 2, 2, 144)    576         Concat47[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation48 (Activation)       (None, 2, 2, 144)    0           Batch48[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv48 (Conv2D)                 (None, 2, 2, 16)     20736       Activation48[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout48 (Dropout)             (None, 2, 2, 16)     0           Conv48[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat48 (Concatenate)          (None, 2, 2, 160)    0           Concat47[0][0]                   \n",
            "                                                                 Dropout48[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch49 (BatchNormalization)    (None, 2, 2, 160)    640         Concat48[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation49 (Activation)       (None, 2, 2, 160)    0           Batch49[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv49 (Conv2D)                 (None, 2, 2, 16)     23040       Activation49[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout49 (Dropout)             (None, 2, 2, 16)     0           Conv49[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat49 (Concatenate)          (None, 2, 2, 176)    0           Concat48[0][0]                   \n",
            "                                                                 Dropout49[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch410 (BatchNormalization)   (None, 2, 2, 176)    704         Concat49[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation410 (Activation)      (None, 2, 2, 176)    0           Batch410[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv410 (Conv2D)                (None, 2, 2, 16)     25344       Activation410[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout410 (Dropout)            (None, 2, 2, 16)     0           Conv410[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat410 (Concatenate)         (None, 2, 2, 192)    0           Concat49[0][0]                   \n",
            "                                                                 Dropout410[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch411 (BatchNormalization)   (None, 2, 2, 192)    768         Concat410[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation411 (Activation)      (None, 2, 2, 192)    0           Batch411[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv411 (Conv2D)                (None, 2, 2, 16)     27648       Activation411[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout411 (Dropout)            (None, 2, 2, 16)     0           Conv411[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat411 (Concatenate)         (None, 2, 2, 208)    0           Concat410[0][0]                  \n",
            "                                                                 Dropout411[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch412 (BatchNormalization)   (None, 2, 2, 208)    832         Concat411[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation412 (Activation)      (None, 2, 2, 208)    0           Batch412[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv412 (Conv2D)                (None, 2, 2, 16)     29952       Activation412[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout412 (Dropout)            (None, 2, 2, 16)     0           Conv412[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat412 (Concatenate)         (None, 2, 2, 224)    0           Concat411[0][0]                  \n",
            "                                                                 Dropout412[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch413 (BatchNormalization)   (None, 2, 2, 224)    896         Concat412[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation413 (Activation)      (None, 2, 2, 224)    0           Batch413[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv413 (Conv2D)                (None, 2, 2, 16)     32256       Activation413[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout413 (Dropout)            (None, 2, 2, 16)     0           Conv413[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat413 (Concatenate)         (None, 2, 2, 240)    0           Concat412[0][0]                  \n",
            "                                                                 Dropout413[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch414 (BatchNormalization)   (None, 2, 2, 240)    960         Concat413[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation414 (Activation)      (None, 2, 2, 240)    0           Batch414[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv414 (Conv2D)                (None, 2, 2, 16)     34560       Activation414[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout414 (Dropout)            (None, 2, 2, 16)     0           Conv414[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat414 (Concatenate)         (None, 2, 2, 256)    0           Concat413[0][0]                  \n",
            "                                                                 Dropout414[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch415 (BatchNormalization)   (None, 2, 2, 256)    1024        Concat414[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation415 (Activation)      (None, 2, 2, 256)    0           Batch415[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv415 (Conv2D)                (None, 2, 2, 16)     36864       Activation415[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout415 (Dropout)            (None, 2, 2, 16)     0           Conv415[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat415 (Concatenate)         (None, 2, 2, 272)    0           Concat414[0][0]                  \n",
            "                                                                 Dropout415[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch416 (BatchNormalization)   (None, 2, 2, 272)    1088        Concat415[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation416 (Activation)      (None, 2, 2, 272)    0           Batch416[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv416 (Conv2D)                (None, 2, 2, 16)     39168       Activation416[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout416 (Dropout)            (None, 2, 2, 16)     0           Conv416[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat416 (Concatenate)         (None, 2, 2, 288)    0           Concat415[0][0]                  \n",
            "                                                                 Dropout416[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch417 (BatchNormalization)   (None, 2, 2, 288)    1152        Concat416[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation417 (Activation)      (None, 2, 2, 288)    0           Batch417[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv417 (Conv2D)                (None, 2, 2, 16)     41472       Activation417[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout417 (Dropout)            (None, 2, 2, 16)     0           Conv417[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat417 (Concatenate)         (None, 2, 2, 304)    0           Concat416[0][0]                  \n",
            "                                                                 Dropout417[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch418 (BatchNormalization)   (None, 2, 2, 304)    1216        Concat417[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation418 (Activation)      (None, 2, 2, 304)    0           Batch418[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv418 (Conv2D)                (None, 2, 2, 16)     43776       Activation418[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout418 (Dropout)            (None, 2, 2, 16)     0           Conv418[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat418 (Concatenate)         (None, 2, 2, 320)    0           Concat417[0][0]                  \n",
            "                                                                 Dropout418[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch419 (BatchNormalization)   (None, 2, 2, 320)    1280        Concat418[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation419 (Activation)      (None, 2, 2, 320)    0           Batch419[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv419 (Conv2D)                (None, 2, 2, 16)     46080       Activation419[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout419 (Dropout)            (None, 2, 2, 16)     0           Conv419[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat419 (Concatenate)         (None, 2, 2, 336)    0           Concat418[0][0]                  \n",
            "                                                                 Dropout419[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch_output_layer (BatchNormal (None, 2, 2, 336)    1344        Concat419[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation_output_layer (Activa (None, 2, 2, 336)    0           Batch_output_layer[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "AvgPooling_output_layer (Averag (None, 1, 1, 336)    0           Activation_output_layer[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Flattern_output_layer (Flatten) (None, 336)          0           AvgPooling_output_layer[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Dense_output_layer (Dense)      (None, 10)           3370        Flattern_output_layer[0][0]      \n",
            "==================================================================================================\n",
            "Total params: 2,062,538\n",
            "Trainable params: 2,032,298\n",
            "Non-trainable params: 30,240\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dU3boX7KQnOh",
        "colab_type": "text"
      },
      "source": [
        "**Observe the definition of lr_changes which is a keras.callbacks.LearningRateSchedular allowing to change learning_rate at the given epoch**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b4XOsW3ahSkL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "weight_checksaver = keras.callbacks.ModelCheckpoint('cifar10_weights.{epoch:02d}-{val_loss:.3f}.hdf5',\n",
        "                                                  monitor='val_loss', verbose=1, save_weights_only=True, period=5, mode = 'auto')\n",
        "lr_changes = keras.callbacks.LearningRateScheduler(lr_schedule, verbose=1)\n",
        "\n",
        "#define optimizer\n",
        "opt_adam = keras.optimizers.Adam(lr=0.001,decay=1e-8)\n",
        "\n",
        "# determine Loss function using defined Optimizer\n",
        "pre_model28_l.compile(loss='categorical_crossentropy',\n",
        "              optimizer=opt_adam,\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O-6Dod4FQ6dN",
        "colab_type": "text"
      },
      "source": [
        "**After compiling the model we start learning by using fit_generator facility of keras**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I68l1PDKB6oU",
        "colab_type": "code",
        "outputId": "a4cc797a-3f8a-48f7-8c9a-62a76aaec0b5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1532
        }
      },
      "source": [
        "#Total 20 epochs with Increasing Batch size\n",
        "#No. of epochs=  epcohs - initial_epochs\n",
        "\n",
        "#batch_size  epcohs  initial_epochs  # epochs run\n",
        "# 16         2          0             2        \n",
        "# 32         5          1             3      \n",
        "# 48         10         3             5\n",
        "# 64         20         10            10\n",
        "#---------------------------------------------------\n",
        "#Total Epochs                 =    20\n",
        "\n",
        "\n",
        "\n",
        "epochs=2 #epochs run=2\n",
        "batch_size = 16\n",
        "history1=pre_model28_l.fit_generator(datagen.flow(x_train_resized,y_train,batch_size=batch_size),\n",
        "                         steps_per_epoch=len(x_train_resized)/batch_size,\n",
        "                         epochs=epochs,\n",
        "                         initial_epoch=0,\n",
        "                         verbose=1,\n",
        "                         callbacks=[weight_checksaver,lr_changes],\n",
        "                         validation_data=(x_test_resized,y_test)\n",
        "                        )\n",
        "\n",
        "epochs=5 #epochs run=3\n",
        "batch_size = 32\n",
        "history2=pre_model28_l.fit_generator(datagen.flow(x_train_resized,y_train,batch_size=batch_size),\n",
        "                         steps_per_epoch=len(x_train_resized)/batch_size,\n",
        "                         epochs=epochs,\n",
        "                         initial_epoch=2,\n",
        "                         verbose=1,\n",
        "                         callbacks=[weight_checksaver,lr_changes],\n",
        "                         validation_data=(x_test_resized,y_test)\n",
        "                        )\n",
        "\n",
        "epochs=10 #epochs run=5\n",
        "batch_size = 48\n",
        "history3=pre_model28_l.fit_generator(datagen.flow(x_train_resized,y_train,batch_size=batch_size),\n",
        "                         steps_per_epoch=len(x_train_resized)/batch_size,\n",
        "                         epochs=epochs,\n",
        "                         initial_epoch=5,\n",
        "                         verbose=1,\n",
        "                         callbacks=[weight_checksaver,lr_changes],\n",
        "                         validation_data=(x_test_resized,y_test)\n",
        "                        )\n",
        "\n",
        "epochs=20 #epochs run=10\n",
        "batch_size = 64\n",
        "history4=pre_model28_l.fit_generator(datagen.flow(x_train_resized,y_train,batch_size=batch_size),\n",
        "                         steps_per_epoch=len(x_train_resized)/batch_size,\n",
        "                         epochs=epochs,\n",
        "                         initial_epoch=10,\n",
        "                         verbose=1,\n",
        "                         callbacks=[weight_checksaver,lr_changes],\n",
        "                         validation_data=(x_test_resized,y_test)\n",
        "                        )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Variable *= will be deprecated. Use variable.assign_mul if you want assignment to the variable value or 'x = x * y' if you want a new python Tensor object.\n",
            "Epoch 1/2\n",
            "\n",
            "Epoch 00001: LearningRateScheduler reducing learning rate to 0.001.\n",
            " 293/3125 [=>............................] - ETA: 17:13 - loss: 2.0831 - acc: 0.2191"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "3124/3125 [============================>.] - ETA: 0s - loss: 1.7876 - acc: 0.3387"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r3125/3125 [==============================] - 979s 313ms/step - loss: 1.7877 - acc: 0.3387 - val_loss: 1.6219 - val_acc: 0.4107\n",
            "Epoch 2/2\n",
            "\n",
            "Epoch 00002: LearningRateScheduler reducing learning rate to 0.001.\n",
            " 267/3125 [=>............................] - ETA: 14:00 - loss: 1.5730 - acc: 0.4232"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "3124/3125 [============================>.] - ETA: 0s - loss: 1.4979 - acc: 0.4569"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r3125/3125 [==============================] - 954s 305ms/step - loss: 1.4979 - acc: 0.4569 - val_loss: 1.3765 - val_acc: 0.5214\n",
            "Epoch 3/5\n",
            "\n",
            "Epoch 00003: LearningRateScheduler reducing learning rate to 0.001.\n",
            " 268/1562 [====>.........................] - ETA: 6:53 - loss: 1.3023 - acc: 0.5274"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1562/1562 [============================>.] - ETA: 0s - loss: 1.2480 - acc: 0.5514"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r1563/1562 [==============================] - 523s 335ms/step - loss: 1.2478 - acc: 0.5515 - val_loss: 1.6044 - val_acc: 0.5042\n",
            "Epoch 4/5\n",
            "\n",
            "Epoch 00004: LearningRateScheduler reducing learning rate to 0.001.\n",
            " 268/1562 [====>.........................] - ETA: 6:58 - loss: 1.1956 - acc: 0.5668"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1562/1562 [============================>.] - ETA: 0s - loss: 1.1598 - acc: 0.5882"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r1563/1562 [==============================] - 528s 338ms/step - loss: 1.1599 - acc: 0.5882 - val_loss: 1.2561 - val_acc: 0.5825\n",
            "Epoch 5/5\n",
            "\n",
            "Epoch 00005: LearningRateScheduler reducing learning rate to 0.001.\n",
            " 268/1562 [====>.........................] - ETA: 6:58 - loss: 1.0855 - acc: 0.6133"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1562/1562 [============================>.] - ETA: 0s - loss: 1.0819 - acc: 0.6180"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r1563/1562 [==============================] - 541s 346ms/step - loss: 1.0819 - acc: 0.6180 - val_loss: 1.2288 - val_acc: 0.6033\n",
            "\n",
            "Epoch 00005: saving model to cifar10_weights.05-1.229.hdf5\n",
            "Epoch 6/10\n",
            "\n",
            "Epoch 00006: LearningRateScheduler reducing learning rate to 0.001.\n",
            " 229/1041 [=====>........................] - ETA: 4:46 - loss: 0.9917 - acc: 0.6527"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1042/1041 [==============================] - 389s 373ms/step - loss: 0.9725 - acc: 0.6581 - val_loss: 1.1369 - val_acc: 0.6255\n",
            "Epoch 7/10\n",
            "\n",
            "Epoch 00007: LearningRateScheduler reducing learning rate to 0.001.\n",
            " 111/1041 [==>...........................] - ETA: 5:17 - loss: 0.9409 - acc: 0.6706"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1042/1041 [==============================] - 384s 369ms/step - loss: 0.9304 - acc: 0.6731 - val_loss: 1.1178 - val_acc: 0.6343\n",
            "Epoch 8/10\n",
            "\n",
            "Epoch 00008: LearningRateScheduler reducing learning rate to 0.001.\n",
            "  77/1041 [=>............................] - ETA: 5:36 - loss: 0.8878 - acc: 0.6883"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1042/1041 [==============================] - 387s 372ms/step - loss: 0.8955 - acc: 0.6846 - val_loss: 1.1545 - val_acc: 0.6313\n",
            "Epoch 9/10\n",
            "\n",
            "Epoch 00009: LearningRateScheduler reducing learning rate to 0.001.\n",
            "  68/1041 [>.............................] - ETA: 5:28 - loss: 0.8573 - acc: 0.6900"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1042/1041 [==============================] - 388s 372ms/step - loss: 0.8593 - acc: 0.6980 - val_loss: 0.9153 - val_acc: 0.6878\n",
            "Epoch 10/10\n",
            "\n",
            "Epoch 00010: LearningRateScheduler reducing learning rate to 0.001.\n",
            "  65/1041 [>.............................] - ETA: 5:42 - loss: 0.8511 - acc: 0.7077"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1042/1041 [==============================] - 392s 376ms/step - loss: 0.8323 - acc: 0.7087 - val_loss: 0.8486 - val_acc: 0.7084\n",
            "\n",
            "Epoch 00010: saving model to cifar10_weights.10-0.849.hdf5\n",
            "Epoch 11/20\n",
            "\n",
            "Epoch 00011: LearningRateScheduler reducing learning rate to 0.001.\n",
            " 53/781 [=>............................] - ETA: 4:53 - loss: 0.7926 - acc: 0.7208"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 315s 403ms/step - loss: 0.7695 - acc: 0.7312 - val_loss: 0.7842 - val_acc: 0.7384\n",
            "Epoch 12/20\n",
            "\n",
            "Epoch 00012: LearningRateScheduler reducing learning rate to 0.001.\n",
            "142/781 [====>.........................] - ETA: 4:03 - loss: 0.7435 - acc: 0.7388"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 313s 401ms/step - loss: 0.7514 - acc: 0.7384 - val_loss: 0.9132 - val_acc: 0.7117\n",
            "Epoch 13/20\n",
            "\n",
            "Epoch 00013: LearningRateScheduler reducing learning rate to 0.001.\n",
            "167/781 [=====>........................] - ETA: 3:54 - loss: 0.7492 - acc: 0.7378"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 314s 401ms/step - loss: 0.7352 - acc: 0.7428 - val_loss: 0.9332 - val_acc: 0.6940\n",
            "Epoch 14/20\n",
            "\n",
            "Epoch 00014: LearningRateScheduler reducing learning rate to 0.001.\n",
            "174/781 [=====>........................] - ETA: 3:51 - loss: 0.7198 - acc: 0.7489"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 313s 401ms/step - loss: 0.7162 - acc: 0.7524 - val_loss: 0.8652 - val_acc: 0.7307\n",
            "Epoch 15/20\n",
            "\n",
            "Epoch 00015: LearningRateScheduler reducing learning rate to 0.001.\n",
            "176/781 [=====>........................] - ETA: 3:50 - loss: 0.6906 - acc: 0.7583"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 313s 401ms/step - loss: 0.7007 - acc: 0.7573 - val_loss: 1.0898 - val_acc: 0.6750\n",
            "\n",
            "Epoch 00015: saving model to cifar10_weights.15-1.090.hdf5\n",
            "Epoch 16/20\n",
            "\n",
            "Epoch 00016: LearningRateScheduler reducing learning rate to 0.001.\n",
            "145/781 [====>.........................] - ETA: 4:01 - loss: 0.6661 - acc: 0.7663"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 313s 401ms/step - loss: 0.6884 - acc: 0.7602 - val_loss: 1.1739 - val_acc: 0.6740\n",
            "Epoch 17/20\n",
            "\n",
            "Epoch 00017: LearningRateScheduler reducing learning rate to 0.0001.\n",
            "167/781 [=====>........................] - ETA: 3:53 - loss: 0.6381 - acc: 0.7761"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 313s 401ms/step - loss: 0.6117 - acc: 0.7865 - val_loss: 0.6528 - val_acc: 0.7840\n",
            "Epoch 18/20\n",
            "\n",
            "Epoch 00018: LearningRateScheduler reducing learning rate to 0.0001.\n",
            "173/781 [=====>........................] - ETA: 3:51 - loss: 0.6009 - acc: 0.7903"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 314s 401ms/step - loss: 0.5932 - acc: 0.7946 - val_loss: 0.6721 - val_acc: 0.7807\n",
            "Epoch 19/20\n",
            "\n",
            "Epoch 00019: LearningRateScheduler reducing learning rate to 0.0001.\n",
            "175/781 [=====>........................] - ETA: 3:50 - loss: 0.5906 - acc: 0.7963"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 313s 401ms/step - loss: 0.5786 - acc: 0.7983 - val_loss: 0.6190 - val_acc: 0.7935\n",
            "Epoch 20/20\n",
            "\n",
            "Epoch 00020: LearningRateScheduler reducing learning rate to 0.0001.\n",
            "175/781 [=====>........................] - ETA: 3:50 - loss: 0.5732 - acc: 0.7978"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 313s 400ms/step - loss: 0.5768 - acc: 0.7981 - val_loss: 0.6633 - val_acc: 0.7807\n",
            "\n",
            "Epoch 00020: saving model to cifar10_weights.20-0.663.hdf5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C755J2brRNmh",
        "colab_type": "text"
      },
      "source": [
        "## **For 20 epochs run on downsized images (16x16) our model has learnt good weights and given us training accuracy of 79.81% and validation accuracy of 78.07%**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-9lY8m18j8Ct",
        "colab_type": "code",
        "outputId": "21dd4d6f-87fc-4652-877f-1d80170e9dce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Save the trained weights in to .h5 format\n",
        "pre_model28_l.save_weights(\"pre_model28_l_weights.h5\")\n",
        "print(\"Saved model to disk\")\n",
        "\n",
        "from google.colab import files\n",
        "\n",
        "files.download('pre_model28_l_weights.h5')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saved model to disk\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwyoQFpIRd40",
        "colab_type": "text"
      },
      "source": [
        "**Just checking out weights at a particular layer**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ARpD-6f7kF_U",
        "colab_type": "code",
        "outputId": "c92b12e9-5e7e-4ca5-8ee3-51829eb7e6c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "len(pre_model28_l.layers[2].get_weights())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zahW3890nkOw",
        "colab_type": "code",
        "outputId": "85e680cc-ec23-4528-bd79-9b47582d7ae1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        }
      },
      "source": [
        "pre_model28_l.layers[2].get_weights()[0]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.8468731 , 0.6476582 , 0.7165509 , 1.3465536 , 0.57304126,\n",
              "       0.8646428 , 1.1702656 , 0.8359851 , 0.8980804 , 1.0968549 ,\n",
              "       0.9116674 , 0.9466966 , 0.7896094 , 0.8556654 , 0.8701283 ,\n",
              "       0.60723805, 0.56897074, 0.6901295 , 0.9055469 , 1.0864186 ,\n",
              "       1.4888134 , 1.265754  , 0.7794853 , 0.7476797 , 0.8843203 ,\n",
              "       1.1372865 , 0.92509574, 0.76956964, 0.9714205 , 0.73980266,\n",
              "       0.90889096, 1.1885608 ], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MjEajxuFki2I",
        "colab_type": "text"
      },
      "source": [
        "#**TRAINING MODEL ON ORIGINAL IMAGES**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8chF-Dw-vakA",
        "colab_type": "code",
        "outputId": "424c6f88-8264-4b8f-c52f-82313d5dbd85",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "print(num_filter)\n",
        "print(l)\n",
        "print(batch_size)\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "32\n",
            "20\n",
            "64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "64i1yxr_RmUU",
        "colab_type": "text"
      },
      "source": [
        "### **Since we are changing the input size back to 32x32 for final training we need to change the output layer Dense connections and so we define \"our_output_layer\"**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C57GI1Q_kF1d",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def our_output_layer(input):\n",
        "    global compression\n",
        "    BatchNorm = BatchNormalization(name='Batch_output_layer')(input)\n",
        "    relu = Activation('relu',name='Activation_output_layer')(BatchNorm)\n",
        "    AvgPooling = AveragePooling2D(pool_size=(2,2),name='AvgPooling_output_layer')(relu)\n",
        "    flat = Flatten(name='Flattern_output_layer')(AvgPooling)\n",
        "    output = Dense(num_classes, activation='softmax',name='our_Dense_output_layer')(flat)  #Name of Dense layer is different than output_layer's Dense layer\n",
        "    \n",
        "    return output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uCqcwW8PR0-b",
        "colab_type": "text"
      },
      "source": [
        "# **model28_l is defined** \n",
        "### for original images 32x32"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PLHk8Ye0kFmU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Everthing same except Dense layer name\n",
        "img_height, img_width, channel = x_train.shape[1],x_train.shape[2],x_train.shape[3] \n",
        "dropout_rate = 0.1\n",
        "\n",
        "input = Input(shape=(img_height, img_width, channel,),name='input_layer')\n",
        "First_Conv2D = Conv2D(num_filter, (3,3), use_bias=False ,padding='same',name='Conv2D_1st')(input)\n",
        "\n",
        "block=1\n",
        "First_Block = add_denseblock(First_Conv2D, num_filter, dropout_rate)\n",
        "First_Transition = add_transition(First_Block, num_filter, dropout_rate)\n",
        "\n",
        "block=2\n",
        "Second_Block = add_denseblock(First_Transition, num_filter, dropout_rate)\n",
        "Second_Transition = add_transition(Second_Block, num_filter, dropout_rate)\n",
        "\n",
        "block=3\n",
        "Third_Block = add_denseblock(Second_Transition, num_filter, dropout_rate)\n",
        "Third_Transition = add_transition(Third_Block, num_filter, dropout_rate)\n",
        "\n",
        "block=4\n",
        "Last_Block = add_denseblock(Third_Transition,  num_filter, dropout_rate)\n",
        "output = our_output_layer(Last_Block)       #our_output_layer:- dense layer with different name\n",
        "\n",
        "  \n",
        "model28_l = Model(inputs=[input], outputs=[output])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vCHzVXE4kFS0",
        "colab_type": "code",
        "outputId": "9921bfe9-e1f7-4ca7-ba83-681af03b3943",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 15658
        }
      },
      "source": [
        "model28_l.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_layer (InputLayer)        (None, 32, 32, 3)    0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Conv2D_1st (Conv2D)             (None, 32, 32, 32)   864         input_layer[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Batch10 (BatchNormalization)    (None, 32, 32, 32)   128         Conv2D_1st[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Activation10 (Activation)       (None, 32, 32, 32)   0           Batch10[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv10 (Conv2D)                 (None, 32, 32, 16)   4608        Activation10[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout10 (Dropout)             (None, 32, 32, 16)   0           Conv10[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat10 (Concatenate)          (None, 32, 32, 48)   0           Conv2D_1st[0][0]                 \n",
            "                                                                 Dropout10[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch11 (BatchNormalization)    (None, 32, 32, 48)   192         Concat10[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation11 (Activation)       (None, 32, 32, 48)   0           Batch11[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv11 (Conv2D)                 (None, 32, 32, 16)   6912        Activation11[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout11 (Dropout)             (None, 32, 32, 16)   0           Conv11[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat11 (Concatenate)          (None, 32, 32, 64)   0           Concat10[0][0]                   \n",
            "                                                                 Dropout11[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch12 (BatchNormalization)    (None, 32, 32, 64)   256         Concat11[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation12 (Activation)       (None, 32, 32, 64)   0           Batch12[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv12 (Conv2D)                 (None, 32, 32, 16)   9216        Activation12[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout12 (Dropout)             (None, 32, 32, 16)   0           Conv12[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat12 (Concatenate)          (None, 32, 32, 80)   0           Concat11[0][0]                   \n",
            "                                                                 Dropout12[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch13 (BatchNormalization)    (None, 32, 32, 80)   320         Concat12[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation13 (Activation)       (None, 32, 32, 80)   0           Batch13[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv13 (Conv2D)                 (None, 32, 32, 16)   11520       Activation13[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout13 (Dropout)             (None, 32, 32, 16)   0           Conv13[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat13 (Concatenate)          (None, 32, 32, 96)   0           Concat12[0][0]                   \n",
            "                                                                 Dropout13[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch14 (BatchNormalization)    (None, 32, 32, 96)   384         Concat13[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation14 (Activation)       (None, 32, 32, 96)   0           Batch14[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv14 (Conv2D)                 (None, 32, 32, 16)   13824       Activation14[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout14 (Dropout)             (None, 32, 32, 16)   0           Conv14[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat14 (Concatenate)          (None, 32, 32, 112)  0           Concat13[0][0]                   \n",
            "                                                                 Dropout14[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch15 (BatchNormalization)    (None, 32, 32, 112)  448         Concat14[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation15 (Activation)       (None, 32, 32, 112)  0           Batch15[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv15 (Conv2D)                 (None, 32, 32, 16)   16128       Activation15[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout15 (Dropout)             (None, 32, 32, 16)   0           Conv15[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat15 (Concatenate)          (None, 32, 32, 128)  0           Concat14[0][0]                   \n",
            "                                                                 Dropout15[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch16 (BatchNormalization)    (None, 32, 32, 128)  512         Concat15[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation16 (Activation)       (None, 32, 32, 128)  0           Batch16[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv16 (Conv2D)                 (None, 32, 32, 16)   18432       Activation16[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout16 (Dropout)             (None, 32, 32, 16)   0           Conv16[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat16 (Concatenate)          (None, 32, 32, 144)  0           Concat15[0][0]                   \n",
            "                                                                 Dropout16[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch17 (BatchNormalization)    (None, 32, 32, 144)  576         Concat16[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation17 (Activation)       (None, 32, 32, 144)  0           Batch17[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv17 (Conv2D)                 (None, 32, 32, 16)   20736       Activation17[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout17 (Dropout)             (None, 32, 32, 16)   0           Conv17[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat17 (Concatenate)          (None, 32, 32, 160)  0           Concat16[0][0]                   \n",
            "                                                                 Dropout17[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch18 (BatchNormalization)    (None, 32, 32, 160)  640         Concat17[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation18 (Activation)       (None, 32, 32, 160)  0           Batch18[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv18 (Conv2D)                 (None, 32, 32, 16)   23040       Activation18[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout18 (Dropout)             (None, 32, 32, 16)   0           Conv18[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat18 (Concatenate)          (None, 32, 32, 176)  0           Concat17[0][0]                   \n",
            "                                                                 Dropout18[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch19 (BatchNormalization)    (None, 32, 32, 176)  704         Concat18[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation19 (Activation)       (None, 32, 32, 176)  0           Batch19[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv19 (Conv2D)                 (None, 32, 32, 16)   25344       Activation19[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout19 (Dropout)             (None, 32, 32, 16)   0           Conv19[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat19 (Concatenate)          (None, 32, 32, 192)  0           Concat18[0][0]                   \n",
            "                                                                 Dropout19[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch110 (BatchNormalization)   (None, 32, 32, 192)  768         Concat19[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation110 (Activation)      (None, 32, 32, 192)  0           Batch110[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv110 (Conv2D)                (None, 32, 32, 16)   27648       Activation110[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout110 (Dropout)            (None, 32, 32, 16)   0           Conv110[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat110 (Concatenate)         (None, 32, 32, 208)  0           Concat19[0][0]                   \n",
            "                                                                 Dropout110[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch111 (BatchNormalization)   (None, 32, 32, 208)  832         Concat110[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation111 (Activation)      (None, 32, 32, 208)  0           Batch111[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv111 (Conv2D)                (None, 32, 32, 16)   29952       Activation111[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout111 (Dropout)            (None, 32, 32, 16)   0           Conv111[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat111 (Concatenate)         (None, 32, 32, 224)  0           Concat110[0][0]                  \n",
            "                                                                 Dropout111[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch112 (BatchNormalization)   (None, 32, 32, 224)  896         Concat111[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation112 (Activation)      (None, 32, 32, 224)  0           Batch112[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv112 (Conv2D)                (None, 32, 32, 16)   32256       Activation112[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout112 (Dropout)            (None, 32, 32, 16)   0           Conv112[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat112 (Concatenate)         (None, 32, 32, 240)  0           Concat111[0][0]                  \n",
            "                                                                 Dropout112[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch113 (BatchNormalization)   (None, 32, 32, 240)  960         Concat112[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation113 (Activation)      (None, 32, 32, 240)  0           Batch113[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv113 (Conv2D)                (None, 32, 32, 16)   34560       Activation113[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout113 (Dropout)            (None, 32, 32, 16)   0           Conv113[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat113 (Concatenate)         (None, 32, 32, 256)  0           Concat112[0][0]                  \n",
            "                                                                 Dropout113[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch114 (BatchNormalization)   (None, 32, 32, 256)  1024        Concat113[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation114 (Activation)      (None, 32, 32, 256)  0           Batch114[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv114 (Conv2D)                (None, 32, 32, 16)   36864       Activation114[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout114 (Dropout)            (None, 32, 32, 16)   0           Conv114[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat114 (Concatenate)         (None, 32, 32, 272)  0           Concat113[0][0]                  \n",
            "                                                                 Dropout114[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch115 (BatchNormalization)   (None, 32, 32, 272)  1088        Concat114[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation115 (Activation)      (None, 32, 32, 272)  0           Batch115[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv115 (Conv2D)                (None, 32, 32, 16)   39168       Activation115[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout115 (Dropout)            (None, 32, 32, 16)   0           Conv115[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat115 (Concatenate)         (None, 32, 32, 288)  0           Concat114[0][0]                  \n",
            "                                                                 Dropout115[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch116 (BatchNormalization)   (None, 32, 32, 288)  1152        Concat115[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation116 (Activation)      (None, 32, 32, 288)  0           Batch116[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv116 (Conv2D)                (None, 32, 32, 16)   41472       Activation116[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout116 (Dropout)            (None, 32, 32, 16)   0           Conv116[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat116 (Concatenate)         (None, 32, 32, 304)  0           Concat115[0][0]                  \n",
            "                                                                 Dropout116[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch117 (BatchNormalization)   (None, 32, 32, 304)  1216        Concat116[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation117 (Activation)      (None, 32, 32, 304)  0           Batch117[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv117 (Conv2D)                (None, 32, 32, 16)   43776       Activation117[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout117 (Dropout)            (None, 32, 32, 16)   0           Conv117[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat117 (Concatenate)         (None, 32, 32, 320)  0           Concat116[0][0]                  \n",
            "                                                                 Dropout117[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch118 (BatchNormalization)   (None, 32, 32, 320)  1280        Concat117[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation118 (Activation)      (None, 32, 32, 320)  0           Batch118[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv118 (Conv2D)                (None, 32, 32, 16)   46080       Activation118[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout118 (Dropout)            (None, 32, 32, 16)   0           Conv118[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat118 (Concatenate)         (None, 32, 32, 336)  0           Concat117[0][0]                  \n",
            "                                                                 Dropout118[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch119 (BatchNormalization)   (None, 32, 32, 336)  1344        Concat118[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation119 (Activation)      (None, 32, 32, 336)  0           Batch119[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv119 (Conv2D)                (None, 32, 32, 16)   48384       Activation119[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout119 (Dropout)            (None, 32, 32, 16)   0           Conv119[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat119 (Concatenate)         (None, 32, 32, 352)  0           Concat118[0][0]                  \n",
            "                                                                 Dropout119[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch_transition1 (BatchNormali (None, 32, 32, 352)  1408        Concat119[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation_transition1 (Activat (None, 32, 32, 352)  0           Batch_transition1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Conv_transition1 (Conv2D)       (None, 32, 32, 16)   5632        Activation_transition1[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Dropout_transition1 (Dropout)   (None, 32, 32, 16)   0           Conv_transition1[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "AvgPooling_transition1 (Average (None, 16, 16, 16)   0           Dropout_transition1[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "Batch20 (BatchNormalization)    (None, 16, 16, 16)   64          AvgPooling_transition1[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Activation20 (Activation)       (None, 16, 16, 16)   0           Batch20[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv20 (Conv2D)                 (None, 16, 16, 16)   2304        Activation20[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout20 (Dropout)             (None, 16, 16, 16)   0           Conv20[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat20 (Concatenate)          (None, 16, 16, 32)   0           AvgPooling_transition1[0][0]     \n",
            "                                                                 Dropout20[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch21 (BatchNormalization)    (None, 16, 16, 32)   128         Concat20[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation21 (Activation)       (None, 16, 16, 32)   0           Batch21[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv21 (Conv2D)                 (None, 16, 16, 16)   4608        Activation21[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout21 (Dropout)             (None, 16, 16, 16)   0           Conv21[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat21 (Concatenate)          (None, 16, 16, 48)   0           Concat20[0][0]                   \n",
            "                                                                 Dropout21[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch22 (BatchNormalization)    (None, 16, 16, 48)   192         Concat21[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation22 (Activation)       (None, 16, 16, 48)   0           Batch22[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv22 (Conv2D)                 (None, 16, 16, 16)   6912        Activation22[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout22 (Dropout)             (None, 16, 16, 16)   0           Conv22[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat22 (Concatenate)          (None, 16, 16, 64)   0           Concat21[0][0]                   \n",
            "                                                                 Dropout22[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch23 (BatchNormalization)    (None, 16, 16, 64)   256         Concat22[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation23 (Activation)       (None, 16, 16, 64)   0           Batch23[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv23 (Conv2D)                 (None, 16, 16, 16)   9216        Activation23[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout23 (Dropout)             (None, 16, 16, 16)   0           Conv23[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat23 (Concatenate)          (None, 16, 16, 80)   0           Concat22[0][0]                   \n",
            "                                                                 Dropout23[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch24 (BatchNormalization)    (None, 16, 16, 80)   320         Concat23[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation24 (Activation)       (None, 16, 16, 80)   0           Batch24[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv24 (Conv2D)                 (None, 16, 16, 16)   11520       Activation24[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout24 (Dropout)             (None, 16, 16, 16)   0           Conv24[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat24 (Concatenate)          (None, 16, 16, 96)   0           Concat23[0][0]                   \n",
            "                                                                 Dropout24[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch25 (BatchNormalization)    (None, 16, 16, 96)   384         Concat24[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation25 (Activation)       (None, 16, 16, 96)   0           Batch25[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv25 (Conv2D)                 (None, 16, 16, 16)   13824       Activation25[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout25 (Dropout)             (None, 16, 16, 16)   0           Conv25[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat25 (Concatenate)          (None, 16, 16, 112)  0           Concat24[0][0]                   \n",
            "                                                                 Dropout25[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch26 (BatchNormalization)    (None, 16, 16, 112)  448         Concat25[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation26 (Activation)       (None, 16, 16, 112)  0           Batch26[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv26 (Conv2D)                 (None, 16, 16, 16)   16128       Activation26[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout26 (Dropout)             (None, 16, 16, 16)   0           Conv26[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat26 (Concatenate)          (None, 16, 16, 128)  0           Concat25[0][0]                   \n",
            "                                                                 Dropout26[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch27 (BatchNormalization)    (None, 16, 16, 128)  512         Concat26[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation27 (Activation)       (None, 16, 16, 128)  0           Batch27[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv27 (Conv2D)                 (None, 16, 16, 16)   18432       Activation27[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout27 (Dropout)             (None, 16, 16, 16)   0           Conv27[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat27 (Concatenate)          (None, 16, 16, 144)  0           Concat26[0][0]                   \n",
            "                                                                 Dropout27[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch28 (BatchNormalization)    (None, 16, 16, 144)  576         Concat27[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation28 (Activation)       (None, 16, 16, 144)  0           Batch28[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv28 (Conv2D)                 (None, 16, 16, 16)   20736       Activation28[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout28 (Dropout)             (None, 16, 16, 16)   0           Conv28[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat28 (Concatenate)          (None, 16, 16, 160)  0           Concat27[0][0]                   \n",
            "                                                                 Dropout28[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch29 (BatchNormalization)    (None, 16, 16, 160)  640         Concat28[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation29 (Activation)       (None, 16, 16, 160)  0           Batch29[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv29 (Conv2D)                 (None, 16, 16, 16)   23040       Activation29[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout29 (Dropout)             (None, 16, 16, 16)   0           Conv29[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat29 (Concatenate)          (None, 16, 16, 176)  0           Concat28[0][0]                   \n",
            "                                                                 Dropout29[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch210 (BatchNormalization)   (None, 16, 16, 176)  704         Concat29[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation210 (Activation)      (None, 16, 16, 176)  0           Batch210[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv210 (Conv2D)                (None, 16, 16, 16)   25344       Activation210[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout210 (Dropout)            (None, 16, 16, 16)   0           Conv210[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat210 (Concatenate)         (None, 16, 16, 192)  0           Concat29[0][0]                   \n",
            "                                                                 Dropout210[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch211 (BatchNormalization)   (None, 16, 16, 192)  768         Concat210[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation211 (Activation)      (None, 16, 16, 192)  0           Batch211[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv211 (Conv2D)                (None, 16, 16, 16)   27648       Activation211[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout211 (Dropout)            (None, 16, 16, 16)   0           Conv211[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat211 (Concatenate)         (None, 16, 16, 208)  0           Concat210[0][0]                  \n",
            "                                                                 Dropout211[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch212 (BatchNormalization)   (None, 16, 16, 208)  832         Concat211[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation212 (Activation)      (None, 16, 16, 208)  0           Batch212[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv212 (Conv2D)                (None, 16, 16, 16)   29952       Activation212[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout212 (Dropout)            (None, 16, 16, 16)   0           Conv212[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat212 (Concatenate)         (None, 16, 16, 224)  0           Concat211[0][0]                  \n",
            "                                                                 Dropout212[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch213 (BatchNormalization)   (None, 16, 16, 224)  896         Concat212[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation213 (Activation)      (None, 16, 16, 224)  0           Batch213[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv213 (Conv2D)                (None, 16, 16, 16)   32256       Activation213[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout213 (Dropout)            (None, 16, 16, 16)   0           Conv213[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat213 (Concatenate)         (None, 16, 16, 240)  0           Concat212[0][0]                  \n",
            "                                                                 Dropout213[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch214 (BatchNormalization)   (None, 16, 16, 240)  960         Concat213[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation214 (Activation)      (None, 16, 16, 240)  0           Batch214[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv214 (Conv2D)                (None, 16, 16, 16)   34560       Activation214[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout214 (Dropout)            (None, 16, 16, 16)   0           Conv214[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat214 (Concatenate)         (None, 16, 16, 256)  0           Concat213[0][0]                  \n",
            "                                                                 Dropout214[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch215 (BatchNormalization)   (None, 16, 16, 256)  1024        Concat214[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation215 (Activation)      (None, 16, 16, 256)  0           Batch215[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv215 (Conv2D)                (None, 16, 16, 16)   36864       Activation215[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout215 (Dropout)            (None, 16, 16, 16)   0           Conv215[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat215 (Concatenate)         (None, 16, 16, 272)  0           Concat214[0][0]                  \n",
            "                                                                 Dropout215[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch216 (BatchNormalization)   (None, 16, 16, 272)  1088        Concat215[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation216 (Activation)      (None, 16, 16, 272)  0           Batch216[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv216 (Conv2D)                (None, 16, 16, 16)   39168       Activation216[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout216 (Dropout)            (None, 16, 16, 16)   0           Conv216[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat216 (Concatenate)         (None, 16, 16, 288)  0           Concat215[0][0]                  \n",
            "                                                                 Dropout216[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch217 (BatchNormalization)   (None, 16, 16, 288)  1152        Concat216[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation217 (Activation)      (None, 16, 16, 288)  0           Batch217[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv217 (Conv2D)                (None, 16, 16, 16)   41472       Activation217[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout217 (Dropout)            (None, 16, 16, 16)   0           Conv217[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat217 (Concatenate)         (None, 16, 16, 304)  0           Concat216[0][0]                  \n",
            "                                                                 Dropout217[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch218 (BatchNormalization)   (None, 16, 16, 304)  1216        Concat217[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation218 (Activation)      (None, 16, 16, 304)  0           Batch218[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv218 (Conv2D)                (None, 16, 16, 16)   43776       Activation218[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout218 (Dropout)            (None, 16, 16, 16)   0           Conv218[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat218 (Concatenate)         (None, 16, 16, 320)  0           Concat217[0][0]                  \n",
            "                                                                 Dropout218[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch219 (BatchNormalization)   (None, 16, 16, 320)  1280        Concat218[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation219 (Activation)      (None, 16, 16, 320)  0           Batch219[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv219 (Conv2D)                (None, 16, 16, 16)   46080       Activation219[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout219 (Dropout)            (None, 16, 16, 16)   0           Conv219[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat219 (Concatenate)         (None, 16, 16, 336)  0           Concat218[0][0]                  \n",
            "                                                                 Dropout219[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch_transition2 (BatchNormali (None, 16, 16, 336)  1344        Concat219[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation_transition2 (Activat (None, 16, 16, 336)  0           Batch_transition2[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Conv_transition2 (Conv2D)       (None, 16, 16, 16)   5376        Activation_transition2[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Dropout_transition2 (Dropout)   (None, 16, 16, 16)   0           Conv_transition2[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "AvgPooling_transition2 (Average (None, 8, 8, 16)     0           Dropout_transition2[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "Batch30 (BatchNormalization)    (None, 8, 8, 16)     64          AvgPooling_transition2[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Activation30 (Activation)       (None, 8, 8, 16)     0           Batch30[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv30 (Conv2D)                 (None, 8, 8, 16)     2304        Activation30[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout30 (Dropout)             (None, 8, 8, 16)     0           Conv30[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat30 (Concatenate)          (None, 8, 8, 32)     0           AvgPooling_transition2[0][0]     \n",
            "                                                                 Dropout30[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch31 (BatchNormalization)    (None, 8, 8, 32)     128         Concat30[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation31 (Activation)       (None, 8, 8, 32)     0           Batch31[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv31 (Conv2D)                 (None, 8, 8, 16)     4608        Activation31[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout31 (Dropout)             (None, 8, 8, 16)     0           Conv31[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat31 (Concatenate)          (None, 8, 8, 48)     0           Concat30[0][0]                   \n",
            "                                                                 Dropout31[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch32 (BatchNormalization)    (None, 8, 8, 48)     192         Concat31[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation32 (Activation)       (None, 8, 8, 48)     0           Batch32[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv32 (Conv2D)                 (None, 8, 8, 16)     6912        Activation32[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout32 (Dropout)             (None, 8, 8, 16)     0           Conv32[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat32 (Concatenate)          (None, 8, 8, 64)     0           Concat31[0][0]                   \n",
            "                                                                 Dropout32[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch33 (BatchNormalization)    (None, 8, 8, 64)     256         Concat32[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation33 (Activation)       (None, 8, 8, 64)     0           Batch33[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv33 (Conv2D)                 (None, 8, 8, 16)     9216        Activation33[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout33 (Dropout)             (None, 8, 8, 16)     0           Conv33[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat33 (Concatenate)          (None, 8, 8, 80)     0           Concat32[0][0]                   \n",
            "                                                                 Dropout33[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch34 (BatchNormalization)    (None, 8, 8, 80)     320         Concat33[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation34 (Activation)       (None, 8, 8, 80)     0           Batch34[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv34 (Conv2D)                 (None, 8, 8, 16)     11520       Activation34[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout34 (Dropout)             (None, 8, 8, 16)     0           Conv34[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat34 (Concatenate)          (None, 8, 8, 96)     0           Concat33[0][0]                   \n",
            "                                                                 Dropout34[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch35 (BatchNormalization)    (None, 8, 8, 96)     384         Concat34[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation35 (Activation)       (None, 8, 8, 96)     0           Batch35[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv35 (Conv2D)                 (None, 8, 8, 16)     13824       Activation35[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout35 (Dropout)             (None, 8, 8, 16)     0           Conv35[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat35 (Concatenate)          (None, 8, 8, 112)    0           Concat34[0][0]                   \n",
            "                                                                 Dropout35[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch36 (BatchNormalization)    (None, 8, 8, 112)    448         Concat35[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation36 (Activation)       (None, 8, 8, 112)    0           Batch36[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv36 (Conv2D)                 (None, 8, 8, 16)     16128       Activation36[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout36 (Dropout)             (None, 8, 8, 16)     0           Conv36[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat36 (Concatenate)          (None, 8, 8, 128)    0           Concat35[0][0]                   \n",
            "                                                                 Dropout36[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch37 (BatchNormalization)    (None, 8, 8, 128)    512         Concat36[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation37 (Activation)       (None, 8, 8, 128)    0           Batch37[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv37 (Conv2D)                 (None, 8, 8, 16)     18432       Activation37[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout37 (Dropout)             (None, 8, 8, 16)     0           Conv37[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat37 (Concatenate)          (None, 8, 8, 144)    0           Concat36[0][0]                   \n",
            "                                                                 Dropout37[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch38 (BatchNormalization)    (None, 8, 8, 144)    576         Concat37[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation38 (Activation)       (None, 8, 8, 144)    0           Batch38[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv38 (Conv2D)                 (None, 8, 8, 16)     20736       Activation38[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout38 (Dropout)             (None, 8, 8, 16)     0           Conv38[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat38 (Concatenate)          (None, 8, 8, 160)    0           Concat37[0][0]                   \n",
            "                                                                 Dropout38[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch39 (BatchNormalization)    (None, 8, 8, 160)    640         Concat38[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation39 (Activation)       (None, 8, 8, 160)    0           Batch39[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv39 (Conv2D)                 (None, 8, 8, 16)     23040       Activation39[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout39 (Dropout)             (None, 8, 8, 16)     0           Conv39[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat39 (Concatenate)          (None, 8, 8, 176)    0           Concat38[0][0]                   \n",
            "                                                                 Dropout39[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch310 (BatchNormalization)   (None, 8, 8, 176)    704         Concat39[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation310 (Activation)      (None, 8, 8, 176)    0           Batch310[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv310 (Conv2D)                (None, 8, 8, 16)     25344       Activation310[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout310 (Dropout)            (None, 8, 8, 16)     0           Conv310[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat310 (Concatenate)         (None, 8, 8, 192)    0           Concat39[0][0]                   \n",
            "                                                                 Dropout310[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch311 (BatchNormalization)   (None, 8, 8, 192)    768         Concat310[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation311 (Activation)      (None, 8, 8, 192)    0           Batch311[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv311 (Conv2D)                (None, 8, 8, 16)     27648       Activation311[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout311 (Dropout)            (None, 8, 8, 16)     0           Conv311[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat311 (Concatenate)         (None, 8, 8, 208)    0           Concat310[0][0]                  \n",
            "                                                                 Dropout311[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch312 (BatchNormalization)   (None, 8, 8, 208)    832         Concat311[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation312 (Activation)      (None, 8, 8, 208)    0           Batch312[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv312 (Conv2D)                (None, 8, 8, 16)     29952       Activation312[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout312 (Dropout)            (None, 8, 8, 16)     0           Conv312[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat312 (Concatenate)         (None, 8, 8, 224)    0           Concat311[0][0]                  \n",
            "                                                                 Dropout312[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch313 (BatchNormalization)   (None, 8, 8, 224)    896         Concat312[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation313 (Activation)      (None, 8, 8, 224)    0           Batch313[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv313 (Conv2D)                (None, 8, 8, 16)     32256       Activation313[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout313 (Dropout)            (None, 8, 8, 16)     0           Conv313[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat313 (Concatenate)         (None, 8, 8, 240)    0           Concat312[0][0]                  \n",
            "                                                                 Dropout313[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch314 (BatchNormalization)   (None, 8, 8, 240)    960         Concat313[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation314 (Activation)      (None, 8, 8, 240)    0           Batch314[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv314 (Conv2D)                (None, 8, 8, 16)     34560       Activation314[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout314 (Dropout)            (None, 8, 8, 16)     0           Conv314[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat314 (Concatenate)         (None, 8, 8, 256)    0           Concat313[0][0]                  \n",
            "                                                                 Dropout314[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch315 (BatchNormalization)   (None, 8, 8, 256)    1024        Concat314[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation315 (Activation)      (None, 8, 8, 256)    0           Batch315[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv315 (Conv2D)                (None, 8, 8, 16)     36864       Activation315[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout315 (Dropout)            (None, 8, 8, 16)     0           Conv315[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat315 (Concatenate)         (None, 8, 8, 272)    0           Concat314[0][0]                  \n",
            "                                                                 Dropout315[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch316 (BatchNormalization)   (None, 8, 8, 272)    1088        Concat315[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation316 (Activation)      (None, 8, 8, 272)    0           Batch316[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv316 (Conv2D)                (None, 8, 8, 16)     39168       Activation316[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout316 (Dropout)            (None, 8, 8, 16)     0           Conv316[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat316 (Concatenate)         (None, 8, 8, 288)    0           Concat315[0][0]                  \n",
            "                                                                 Dropout316[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch317 (BatchNormalization)   (None, 8, 8, 288)    1152        Concat316[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation317 (Activation)      (None, 8, 8, 288)    0           Batch317[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv317 (Conv2D)                (None, 8, 8, 16)     41472       Activation317[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout317 (Dropout)            (None, 8, 8, 16)     0           Conv317[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat317 (Concatenate)         (None, 8, 8, 304)    0           Concat316[0][0]                  \n",
            "                                                                 Dropout317[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch318 (BatchNormalization)   (None, 8, 8, 304)    1216        Concat317[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation318 (Activation)      (None, 8, 8, 304)    0           Batch318[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv318 (Conv2D)                (None, 8, 8, 16)     43776       Activation318[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout318 (Dropout)            (None, 8, 8, 16)     0           Conv318[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat318 (Concatenate)         (None, 8, 8, 320)    0           Concat317[0][0]                  \n",
            "                                                                 Dropout318[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch319 (BatchNormalization)   (None, 8, 8, 320)    1280        Concat318[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation319 (Activation)      (None, 8, 8, 320)    0           Batch319[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv319 (Conv2D)                (None, 8, 8, 16)     46080       Activation319[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout319 (Dropout)            (None, 8, 8, 16)     0           Conv319[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat319 (Concatenate)         (None, 8, 8, 336)    0           Concat318[0][0]                  \n",
            "                                                                 Dropout319[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch_transition3 (BatchNormali (None, 8, 8, 336)    1344        Concat319[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation_transition3 (Activat (None, 8, 8, 336)    0           Batch_transition3[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Conv_transition3 (Conv2D)       (None, 8, 8, 16)     5376        Activation_transition3[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Dropout_transition3 (Dropout)   (None, 8, 8, 16)     0           Conv_transition3[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "AvgPooling_transition3 (Average (None, 4, 4, 16)     0           Dropout_transition3[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "Batch40 (BatchNormalization)    (None, 4, 4, 16)     64          AvgPooling_transition3[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Activation40 (Activation)       (None, 4, 4, 16)     0           Batch40[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv40 (Conv2D)                 (None, 4, 4, 16)     2304        Activation40[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout40 (Dropout)             (None, 4, 4, 16)     0           Conv40[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat40 (Concatenate)          (None, 4, 4, 32)     0           AvgPooling_transition3[0][0]     \n",
            "                                                                 Dropout40[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch41 (BatchNormalization)    (None, 4, 4, 32)     128         Concat40[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation41 (Activation)       (None, 4, 4, 32)     0           Batch41[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv41 (Conv2D)                 (None, 4, 4, 16)     4608        Activation41[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout41 (Dropout)             (None, 4, 4, 16)     0           Conv41[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat41 (Concatenate)          (None, 4, 4, 48)     0           Concat40[0][0]                   \n",
            "                                                                 Dropout41[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch42 (BatchNormalization)    (None, 4, 4, 48)     192         Concat41[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation42 (Activation)       (None, 4, 4, 48)     0           Batch42[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv42 (Conv2D)                 (None, 4, 4, 16)     6912        Activation42[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout42 (Dropout)             (None, 4, 4, 16)     0           Conv42[0][0]                     \n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Concat42 (Concatenate)          (None, 4, 4, 64)     0           Concat41[0][0]                   \n",
            "                                                                 Dropout42[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch43 (BatchNormalization)    (None, 4, 4, 64)     256         Concat42[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation43 (Activation)       (None, 4, 4, 64)     0           Batch43[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv43 (Conv2D)                 (None, 4, 4, 16)     9216        Activation43[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout43 (Dropout)             (None, 4, 4, 16)     0           Conv43[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat43 (Concatenate)          (None, 4, 4, 80)     0           Concat42[0][0]                   \n",
            "                                                                 Dropout43[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch44 (BatchNormalization)    (None, 4, 4, 80)     320         Concat43[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation44 (Activation)       (None, 4, 4, 80)     0           Batch44[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv44 (Conv2D)                 (None, 4, 4, 16)     11520       Activation44[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout44 (Dropout)             (None, 4, 4, 16)     0           Conv44[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat44 (Concatenate)          (None, 4, 4, 96)     0           Concat43[0][0]                   \n",
            "                                                                 Dropout44[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch45 (BatchNormalization)    (None, 4, 4, 96)     384         Concat44[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation45 (Activation)       (None, 4, 4, 96)     0           Batch45[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv45 (Conv2D)                 (None, 4, 4, 16)     13824       Activation45[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout45 (Dropout)             (None, 4, 4, 16)     0           Conv45[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat45 (Concatenate)          (None, 4, 4, 112)    0           Concat44[0][0]                   \n",
            "                                                                 Dropout45[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch46 (BatchNormalization)    (None, 4, 4, 112)    448         Concat45[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation46 (Activation)       (None, 4, 4, 112)    0           Batch46[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv46 (Conv2D)                 (None, 4, 4, 16)     16128       Activation46[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout46 (Dropout)             (None, 4, 4, 16)     0           Conv46[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat46 (Concatenate)          (None, 4, 4, 128)    0           Concat45[0][0]                   \n",
            "                                                                 Dropout46[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch47 (BatchNormalization)    (None, 4, 4, 128)    512         Concat46[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation47 (Activation)       (None, 4, 4, 128)    0           Batch47[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv47 (Conv2D)                 (None, 4, 4, 16)     18432       Activation47[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout47 (Dropout)             (None, 4, 4, 16)     0           Conv47[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat47 (Concatenate)          (None, 4, 4, 144)    0           Concat46[0][0]                   \n",
            "                                                                 Dropout47[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch48 (BatchNormalization)    (None, 4, 4, 144)    576         Concat47[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation48 (Activation)       (None, 4, 4, 144)    0           Batch48[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv48 (Conv2D)                 (None, 4, 4, 16)     20736       Activation48[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout48 (Dropout)             (None, 4, 4, 16)     0           Conv48[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat48 (Concatenate)          (None, 4, 4, 160)    0           Concat47[0][0]                   \n",
            "                                                                 Dropout48[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch49 (BatchNormalization)    (None, 4, 4, 160)    640         Concat48[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation49 (Activation)       (None, 4, 4, 160)    0           Batch49[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Conv49 (Conv2D)                 (None, 4, 4, 16)     23040       Activation49[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "Dropout49 (Dropout)             (None, 4, 4, 16)     0           Conv49[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "Concat49 (Concatenate)          (None, 4, 4, 176)    0           Concat48[0][0]                   \n",
            "                                                                 Dropout49[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Batch410 (BatchNormalization)   (None, 4, 4, 176)    704         Concat49[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Activation410 (Activation)      (None, 4, 4, 176)    0           Batch410[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv410 (Conv2D)                (None, 4, 4, 16)     25344       Activation410[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout410 (Dropout)            (None, 4, 4, 16)     0           Conv410[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat410 (Concatenate)         (None, 4, 4, 192)    0           Concat49[0][0]                   \n",
            "                                                                 Dropout410[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch411 (BatchNormalization)   (None, 4, 4, 192)    768         Concat410[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation411 (Activation)      (None, 4, 4, 192)    0           Batch411[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv411 (Conv2D)                (None, 4, 4, 16)     27648       Activation411[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout411 (Dropout)            (None, 4, 4, 16)     0           Conv411[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat411 (Concatenate)         (None, 4, 4, 208)    0           Concat410[0][0]                  \n",
            "                                                                 Dropout411[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch412 (BatchNormalization)   (None, 4, 4, 208)    832         Concat411[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation412 (Activation)      (None, 4, 4, 208)    0           Batch412[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv412 (Conv2D)                (None, 4, 4, 16)     29952       Activation412[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout412 (Dropout)            (None, 4, 4, 16)     0           Conv412[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat412 (Concatenate)         (None, 4, 4, 224)    0           Concat411[0][0]                  \n",
            "                                                                 Dropout412[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch413 (BatchNormalization)   (None, 4, 4, 224)    896         Concat412[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation413 (Activation)      (None, 4, 4, 224)    0           Batch413[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv413 (Conv2D)                (None, 4, 4, 16)     32256       Activation413[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout413 (Dropout)            (None, 4, 4, 16)     0           Conv413[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat413 (Concatenate)         (None, 4, 4, 240)    0           Concat412[0][0]                  \n",
            "                                                                 Dropout413[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch414 (BatchNormalization)   (None, 4, 4, 240)    960         Concat413[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation414 (Activation)      (None, 4, 4, 240)    0           Batch414[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv414 (Conv2D)                (None, 4, 4, 16)     34560       Activation414[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout414 (Dropout)            (None, 4, 4, 16)     0           Conv414[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat414 (Concatenate)         (None, 4, 4, 256)    0           Concat413[0][0]                  \n",
            "                                                                 Dropout414[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch415 (BatchNormalization)   (None, 4, 4, 256)    1024        Concat414[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation415 (Activation)      (None, 4, 4, 256)    0           Batch415[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv415 (Conv2D)                (None, 4, 4, 16)     36864       Activation415[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout415 (Dropout)            (None, 4, 4, 16)     0           Conv415[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat415 (Concatenate)         (None, 4, 4, 272)    0           Concat414[0][0]                  \n",
            "                                                                 Dropout415[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch416 (BatchNormalization)   (None, 4, 4, 272)    1088        Concat415[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation416 (Activation)      (None, 4, 4, 272)    0           Batch416[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv416 (Conv2D)                (None, 4, 4, 16)     39168       Activation416[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout416 (Dropout)            (None, 4, 4, 16)     0           Conv416[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat416 (Concatenate)         (None, 4, 4, 288)    0           Concat415[0][0]                  \n",
            "                                                                 Dropout416[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch417 (BatchNormalization)   (None, 4, 4, 288)    1152        Concat416[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation417 (Activation)      (None, 4, 4, 288)    0           Batch417[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv417 (Conv2D)                (None, 4, 4, 16)     41472       Activation417[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout417 (Dropout)            (None, 4, 4, 16)     0           Conv417[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat417 (Concatenate)         (None, 4, 4, 304)    0           Concat416[0][0]                  \n",
            "                                                                 Dropout417[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch418 (BatchNormalization)   (None, 4, 4, 304)    1216        Concat417[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation418 (Activation)      (None, 4, 4, 304)    0           Batch418[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv418 (Conv2D)                (None, 4, 4, 16)     43776       Activation418[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout418 (Dropout)            (None, 4, 4, 16)     0           Conv418[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat418 (Concatenate)         (None, 4, 4, 320)    0           Concat417[0][0]                  \n",
            "                                                                 Dropout418[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch419 (BatchNormalization)   (None, 4, 4, 320)    1280        Concat418[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation419 (Activation)      (None, 4, 4, 320)    0           Batch419[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "Conv419 (Conv2D)                (None, 4, 4, 16)     46080       Activation419[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Dropout419 (Dropout)            (None, 4, 4, 16)     0           Conv419[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "Concat419 (Concatenate)         (None, 4, 4, 336)    0           Concat418[0][0]                  \n",
            "                                                                 Dropout419[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "Batch_output_layer (BatchNormal (None, 4, 4, 336)    1344        Concat419[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "Activation_output_layer (Activa (None, 4, 4, 336)    0           Batch_output_layer[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "AvgPooling_output_layer (Averag (None, 2, 2, 336)    0           Activation_output_layer[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Flattern_output_layer (Flatten) (None, 1344)         0           AvgPooling_output_layer[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "our_Dense_output_layer (Dense)  (None, 10)           13450       Flattern_output_layer[0][0]      \n",
            "==================================================================================================\n",
            "Total params: 2,072,618\n",
            "Trainable params: 2,042,378\n",
            "Non-trainable params: 30,240\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WokfjtkbQTqZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! pip install pydrive\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive \n",
        "from google.colab import auth \n",
        "from oauth2client.client import GoogleCredentials\n",
        "# 1. Authenticate and create the PyDrive client.\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OlGGHMeWQid8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "downloaded = drive.CreateFile({'id': '1-BtbHl6BE6p1mJ8DvkvwUfOE1so9ZeJ3'})\n",
        "downloaded.GetContentFile('pre_model28_l_weights.h5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MVKbPHezSCEe",
        "colab_type": "text"
      },
      "source": [
        "## **Now we load weights learnt from pre_model28_l into our new model28_l except for the last layer (output layer of densenet).**\n",
        "by_name=True will help for that purpose"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SNHxfF2WoJnD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model28_l.load_weights('pre_model28_l_weights.h5',by_name=True)  # Load weights for all layers except Dense layer"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3XD0Plsmo660",
        "colab_type": "code",
        "outputId": "c4168ab2-5a52-422b-bc4d-e890a70f9de9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        }
      },
      "source": [
        "model28_l.layers[2].get_weights()[0]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.8468731 , 0.6476582 , 0.7165509 , 1.3465536 , 0.57304126,\n",
              "       0.8646428 , 1.1702656 , 0.8359851 , 0.8980804 , 1.0968549 ,\n",
              "       0.9116674 , 0.9466966 , 0.7896094 , 0.8556654 , 0.8701283 ,\n",
              "       0.60723805, 0.56897074, 0.6901295 , 0.9055469 , 1.0864186 ,\n",
              "       1.4888134 , 1.265754  , 0.7794853 , 0.7476797 , 0.8843203 ,\n",
              "       1.1372865 , 0.92509574, 0.76956964, 0.9714205 , 0.73980266,\n",
              "       0.90889096, 1.1885608 ], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkqkDwioSiJm",
        "colab_type": "text"
      },
      "source": [
        "## **Note the definition of lr_changes_1 which uses lr_schedule_1 to change the learning rate as we have defined. Now model28_l is compiled**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l2eBrfgso61D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# determine Loss function and Optimizer\n",
        "our_weight_checksaver = keras.callbacks.ModelCheckpoint('our_cifar10_weights.{epoch:02d}-{val_loss:.3f}.hdf5',\n",
        "                                                  monitor='val_loss', verbose=1, save_weights_only=True, period=3, mode = 'auto')\n",
        "\n",
        "lr_changes_1 = keras.callbacks.LearningRateScheduler(lr_schedule_1, verbose=1)\n",
        "\n",
        "#define optimizer\n",
        "opt_adam = keras.optimizers.Adam(lr=0.001,decay=1e-8)\n",
        "\n",
        "model28_l.compile(loss='categorical_crossentropy',\n",
        "              optimizer=opt_adam,\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4jlB8ZmQSuSi",
        "colab_type": "text"
      },
      "source": [
        "## **Final run for 30 epochs**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uCR3F6D8o6vQ",
        "colab_type": "code",
        "outputId": "761671f5-e66c-40b9-866a-b1e48be5aa93",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2389
        }
      },
      "source": [
        "epochs=30\n",
        "batch_size=64\n",
        "our_history=model28_l.fit_generator(datagen.flow(x_train,y_train,batch_size=batch_size),\n",
        "                      steps_per_epoch=len(x_train)/batch_size,\n",
        "                      epochs=epochs, \n",
        "                      verbose=1,\n",
        "                      callbacks=[our_weight_checksaver,lr_changes_1],\n",
        "                      validation_data=(x_test,y_test)\n",
        "                     )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/30\n",
            "\n",
            "Epoch 00001: LearningRateScheduler reducing learning rate to 0.001.\n",
            "619/781 [======================>.......] - ETA: 2:21 - loss: 0.6842 - acc: 0.7666"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 724s 926ms/step - loss: 0.6632 - acc: 0.7733 - val_loss: 1.1940 - val_acc: 0.6922\n",
            "Epoch 2/30\n",
            "\n",
            "Epoch 00002: LearningRateScheduler reducing learning rate to 0.001.\n",
            "302/781 [==========>...................] - ETA: 6:38 - loss: 0.5504 - acc: 0.8096"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.5371 - acc: 0.8156 - val_loss: 0.6171 - val_acc: 0.8074\n",
            "Epoch 3/30\n",
            "\n",
            "Epoch 00003: LearningRateScheduler reducing learning rate to 0.001.\n",
            "212/781 [=======>......................] - ETA: 7:52 - loss: 0.4880 - acc: 0.8342"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 892ms/step - loss: 0.4923 - acc: 0.8304 - val_loss: 0.5908 - val_acc: 0.8185\n",
            "\n",
            "Epoch 00003: saving model to our_cifar10_weights.03-0.591.hdf5\n",
            "Epoch 4/30\n",
            "\n",
            "Epoch 00004: LearningRateScheduler reducing learning rate to 0.001.\n",
            "152/781 [====>.........................] - ETA: 8:43 - loss: 0.4560 - acc: 0.8414"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 891ms/step - loss: 0.4636 - acc: 0.8396 - val_loss: 0.8003 - val_acc: 0.7760\n",
            "Epoch 5/30\n",
            "\n",
            "Epoch 00005: LearningRateScheduler reducing learning rate to 0.001.\n",
            "170/781 [=====>........................] - ETA: 8:29 - loss: 0.4536 - acc: 0.8451"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.4444 - acc: 0.8485 - val_loss: 0.6623 - val_acc: 0.8086\n",
            "Epoch 6/30\n",
            "\n",
            "Epoch 00006: LearningRateScheduler reducing learning rate to 0.001.\n",
            "175/781 [=====>........................] - ETA: 8:24 - loss: 0.4201 - acc: 0.8546"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.4179 - acc: 0.8555 - val_loss: 0.6020 - val_acc: 0.8301\n",
            "\n",
            "Epoch 00006: saving model to our_cifar10_weights.06-0.602.hdf5\n",
            "Epoch 7/30\n",
            "\n",
            "Epoch 00007: LearningRateScheduler reducing learning rate to 0.001.\n",
            "143/781 [====>.........................] - ETA: 8:51 - loss: 0.3923 - acc: 0.8649"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 696s 891ms/step - loss: 0.4047 - acc: 0.8605 - val_loss: 0.6173 - val_acc: 0.8180\n",
            "Epoch 8/30\n",
            "\n",
            "Epoch 00008: LearningRateScheduler reducing learning rate to 0.001.\n",
            "167/781 [=====>........................] - ETA: 8:31 - loss: 0.3845 - acc: 0.8675"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 892ms/step - loss: 0.3905 - acc: 0.8659 - val_loss: 0.6391 - val_acc: 0.8104\n",
            "Epoch 9/30\n",
            "\n",
            "Epoch 00009: LearningRateScheduler reducing learning rate to 0.001.\n",
            "174/781 [=====>........................] - ETA: 8:23 - loss: 0.3721 - acc: 0.8739"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 892ms/step - loss: 0.3792 - acc: 0.8700 - val_loss: 0.5404 - val_acc: 0.8401\n",
            "\n",
            "Epoch 00009: saving model to our_cifar10_weights.09-0.540.hdf5\n",
            "Epoch 10/30\n",
            "\n",
            "Epoch 00010: LearningRateScheduler reducing learning rate to 0.001.\n",
            "142/781 [====>.........................] - ETA: 8:51 - loss: 0.3481 - acc: 0.8802"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.3687 - acc: 0.8731 - val_loss: 0.5249 - val_acc: 0.8513\n",
            "Epoch 11/30\n",
            "\n",
            "Epoch 00011: LearningRateScheduler reducing learning rate to 0.001.\n",
            "166/781 [=====>........................] - ETA: 8:32 - loss: 0.3358 - acc: 0.8817"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.3562 - acc: 0.8763 - val_loss: 0.6325 - val_acc: 0.8237\n",
            "Epoch 12/30\n",
            "\n",
            "Epoch 00012: LearningRateScheduler reducing learning rate to 0.001.\n",
            "173/781 [=====>........................] - ETA: 8:25 - loss: 0.3270 - acc: 0.8852"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.3376 - acc: 0.8830 - val_loss: 0.5852 - val_acc: 0.8356\n",
            "\n",
            "Epoch 00012: saving model to our_cifar10_weights.12-0.585.hdf5\n",
            "Epoch 13/30\n",
            "\n",
            "Epoch 00013: LearningRateScheduler reducing learning rate to 0.001.\n",
            "142/781 [====>.........................] - ETA: 8:52 - loss: 0.3469 - acc: 0.8775"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 892ms/step - loss: 0.3368 - acc: 0.8824 - val_loss: 0.5142 - val_acc: 0.8524\n",
            "Epoch 14/30\n",
            "\n",
            "Epoch 00014: LearningRateScheduler reducing learning rate to 0.001.\n",
            "166/781 [=====>........................] - ETA: 8:32 - loss: 0.3116 - acc: 0.8940"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.3221 - acc: 0.8897 - val_loss: 0.4929 - val_acc: 0.8572\n",
            "Epoch 15/30\n",
            "\n",
            "Epoch 00015: LearningRateScheduler reducing learning rate to 0.0005.\n",
            "172/781 [=====>........................] - ETA: 8:25 - loss: 0.2837 - acc: 0.9024"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 892ms/step - loss: 0.2761 - acc: 0.9030 - val_loss: 0.4970 - val_acc: 0.8643\n",
            "\n",
            "Epoch 00015: saving model to our_cifar10_weights.15-0.497.hdf5\n",
            "Epoch 16/30\n",
            "\n",
            "Epoch 00016: LearningRateScheduler reducing learning rate to 0.0005.\n",
            "142/781 [====>.........................] - ETA: 8:50 - loss: 0.2563 - acc: 0.9094"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.2644 - acc: 0.9084 - val_loss: 0.4361 - val_acc: 0.8758\n",
            "Epoch 17/30\n",
            "\n",
            "Epoch 00017: LearningRateScheduler reducing learning rate to 0.0005.\n",
            "166/781 [=====>........................] - ETA: 8:32 - loss: 0.2511 - acc: 0.9121"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 893ms/step - loss: 0.2558 - acc: 0.9095 - val_loss: 0.4047 - val_acc: 0.8829\n",
            "Epoch 18/30\n",
            "\n",
            "Epoch 00018: LearningRateScheduler reducing learning rate to 0.0005.\n",
            "172/781 [=====>........................] - ETA: 8:26 - loss: 0.2520 - acc: 0.9125"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 695s 888ms/step - loss: 0.2531 - acc: 0.9124 - val_loss: 0.3385 - val_acc: 0.9004\n",
            "\n",
            "Epoch 00018: saving model to our_cifar10_weights.18-0.339.hdf5\n",
            "Epoch 19/30\n",
            "\n",
            "Epoch 00019: LearningRateScheduler reducing learning rate to 0.0005.\n",
            "142/781 [====>.........................] - ETA: 8:50 - loss: 0.2450 - acc: 0.9119"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 694s 888ms/step - loss: 0.2491 - acc: 0.9129 - val_loss: 0.4665 - val_acc: 0.8706\n",
            "Epoch 20/30\n",
            "\n",
            "Epoch 00020: LearningRateScheduler reducing learning rate to 0.0003.\n",
            "166/781 [=====>........................] - ETA: 8:29 - loss: 0.2302 - acc: 0.9210"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 695s 889ms/step - loss: 0.2249 - acc: 0.9215 - val_loss: 0.4573 - val_acc: 0.8724\n",
            "Epoch 21/30\n",
            "\n",
            "Epoch 00021: LearningRateScheduler reducing learning rate to 0.0003.\n",
            "172/781 [=====>........................] - ETA: 8:27 - loss: 0.2213 - acc: 0.9194"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 891ms/step - loss: 0.2166 - acc: 0.9241 - val_loss: 0.3755 - val_acc: 0.8942\n",
            "\n",
            "Epoch 00021: saving model to our_cifar10_weights.21-0.375.hdf5\n",
            "Epoch 22/30\n",
            "\n",
            "Epoch 00022: LearningRateScheduler reducing learning rate to 0.0003.\n",
            "142/781 [====>.........................] - ETA: 8:51 - loss: 0.2096 - acc: 0.9283"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 696s 890ms/step - loss: 0.2122 - acc: 0.9257 - val_loss: 0.3331 - val_acc: 0.9013\n",
            "Epoch 23/30\n",
            "\n",
            "Epoch 00023: LearningRateScheduler reducing learning rate to 0.0003.\n",
            "166/781 [=====>........................] - ETA: 8:29 - loss: 0.2062 - acc: 0.9264"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 891ms/step - loss: 0.2086 - acc: 0.9263 - val_loss: 0.3419 - val_acc: 0.9000\n",
            "Epoch 24/30\n",
            "\n",
            "Epoch 00024: LearningRateScheduler reducing learning rate to 0.0003.\n",
            "172/781 [=====>........................] - ETA: 8:26 - loss: 0.1862 - acc: 0.9326"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 892ms/step - loss: 0.2008 - acc: 0.9283 - val_loss: 0.3377 - val_acc: 0.9038\n",
            "\n",
            "Epoch 00024: saving model to our_cifar10_weights.24-0.338.hdf5\n",
            "Epoch 25/30\n",
            "\n",
            "Epoch 00025: LearningRateScheduler reducing learning rate to 0.0002.\n",
            "142/781 [====>.........................] - ETA: 8:52 - loss: 0.1869 - acc: 0.9325"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 893ms/step - loss: 0.1937 - acc: 0.9316 - val_loss: 0.3192 - val_acc: 0.9101\n",
            "Epoch 26/30\n",
            "\n",
            "Epoch 00026: LearningRateScheduler reducing learning rate to 0.0002.\n",
            "166/781 [=====>........................] - ETA: 8:32 - loss: 0.1833 - acc: 0.9361"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 893ms/step - loss: 0.1865 - acc: 0.9362 - val_loss: 0.3529 - val_acc: 0.9041\n",
            "Epoch 27/30\n",
            "\n",
            "Epoch 00027: LearningRateScheduler reducing learning rate to 0.0002.\n",
            "172/781 [=====>........................] - ETA: 8:26 - loss: 0.1778 - acc: 0.9368"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 893ms/step - loss: 0.1861 - acc: 0.9345 - val_loss: 0.3578 - val_acc: 0.8997\n",
            "\n",
            "Epoch 00027: saving model to our_cifar10_weights.27-0.358.hdf5\n",
            "Epoch 28/30\n",
            "\n",
            "Epoch 00028: LearningRateScheduler reducing learning rate to 0.0001.\n",
            "142/781 [====>.........................] - ETA: 8:52 - loss: 0.1734 - acc: 0.9400"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 699s 893ms/step - loss: 0.1772 - acc: 0.9384 - val_loss: 0.3224 - val_acc: 0.9093\n",
            "Epoch 29/30\n",
            "\n",
            "Epoch 00029: LearningRateScheduler reducing learning rate to 0.0001.\n",
            "166/781 [=====>........................] - ETA: 8:31 - loss: 0.1615 - acc: 0.9419"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 697s 891ms/step - loss: 0.1690 - acc: 0.9398 - val_loss: 0.3067 - val_acc: 0.9110\n",
            "Epoch 30/30\n",
            "\n",
            "Epoch 00030: LearningRateScheduler reducing learning rate to 0.0001.\n",
            "172/781 [=====>........................] - ETA: 8:27 - loss: 0.1731 - acc: 0.9370"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "782/781 [==============================] - 698s 892ms/step - loss: 0.1723 - acc: 0.9391 - val_loss: 0.3216 - val_acc: 0.9098\n",
            "\n",
            "Epoch 00030: saving model to our_cifar10_weights.30-0.322.hdf5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C-mubfveS44m",
        "colab_type": "text"
      },
      "source": [
        "### **Training accuracy: 93.91% and validation accuracy: 90.98% at the end of 30th epoch**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZcWydmIVhZGr",
        "colab_type": "code",
        "outputId": "305bf56d-9db6-43b4-ba1b-04bdf13fcb45",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "# Test the model\n",
        "score = model28_l.evaluate(x_test, y_test, verbose=1)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "10000/10000 [==============================] - 55s 6ms/step\n",
            "Test loss: 0.32159562615156173\n",
            "Test accuracy: 0.9098\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Og56VCRh5j8V",
        "colab_type": "code",
        "outputId": "fdee2e40-2830-4188-965c-61c164d20c54",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "# Save the trained weights in to .h5 format\n",
        "model28_l.save_weights(\"model28_l_weights_final.h5\")\n",
        "print(\"Saved weights to disk\")\n",
        "model28_l.save('model28_l.h5')\n",
        "print(\"Saved model to disk\")\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saved weights to disk\n",
            "Saved model to disk\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QK44NOhO1-KC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "files.download('model28_l_weights_final.h5')\n",
        "files.download('model28_l.h5')\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4nPYSF9i2nLW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model28_l_weights = model28_l.get_weights()\n",
        "np.save(\"model28_l_final_weights_array\", model28_l_weights)\n",
        "files.download('model28_l_final_weights_array.npy')"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}