hands-on/060_mnist_cnn_courageous.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xX05QJuPqxDU"
   },
   "source": [
    "# MNIST: convolutional neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FeBuk0jpqxDX"
   },
   "source": [
    "We define and train a convolutional neural network (CNN) model to recognize handwritten digits.  Although CNNs can be used in many contexts, they are probably most often used in the context of image processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dsSGP2BzqxDY"
   },
   "source": [
    "## Required imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vntuf5EzqxDZ"
   },
   "outputs": [],
   "source": [
    "import keras\n",
    "from keras import backend as K\n",
    "from keras.datasets import mnist\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import FunctionTransformer, OneHotEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hah5lmFZqxDh"
   },
   "source": [
    "## Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "42NZmGBzqxDh"
   },
   "source": [
    "Depending on the backend keras uses, the input format slightly differs. Either the number of color channels is the first, or the last dimension.  Our input data is grayscale, so there is only a single color channel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TsnEMyYyqxDi"
   },
   "outputs": [],
   "source": [
    "img_channels, img_rows, img_cols = 1, 28, 28\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    shape_ord = (img_channels, img_rows, img_cols)\n",
    "else:  # channel_last\n",
    "    shape_ord = (img_rows, img_cols, img_channels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ylfYzQarqxDn"
   },
   "source": [
    "Load the data again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "11Jz611EqxDo",
    "outputId": "54e8ead8-841e-4efb-8158-0b951252efc5"
   },
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LBUQOrUAqxDr"
   },
   "source": [
    "### Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cg4Mp1wpqxDs"
   },
   "source": [
    "First, reshape the input data to take into account the color channels of the images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xYA9075OqxD6"
   },
   "source": [
    "Most learning algorithms work better when the input data is normalized to the interval $[0, 1]$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "csQt9IsVqxD7"
   },
   "source": [
    "The output can be transformed using the pipeline defined for the multilayer network and can be imported from the `mnist_utils` module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DnsXw5Z3qxEK"
   },
   "source": [
    "### Validation data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "W6AX8PY2qxEL"
   },
   "source": [
    "In order to make this reproducible, we have to seed the random number generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1Q8hDaQTqxEM"
   },
   "outputs": [],
   "source": [
    "np.random.seed(1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "azWUAholqxEP"
   },
   "source": [
    "During the training, we will require a validation set, so we split the training data into two sets, one for actual training, the other for validation.  Note, we don't touch the test data set at all during the training process.  The default is using 75 % of the data for training, 25 % for validation. This function will also shuffle the data set prior to splitting (hence seeding the random number generator)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "W464pK-sqxEQ"
   },
   "outputs": [],
   "source": [
    "x_train, x_val, y_train, y_val = train_test_split(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "05cdUzOMqxES",
    "outputId": "5f2caffc-5f3b-4611-e537-4dfa5f728a87"
   },
   "outputs": [],
   "source": [
    "x_train.shape, x_val.shape, x_train.dtype, y_train.shape, y_val.shape, y_train.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LMyrPzoHqxEY"
   },
   "source": [
    "### Verification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EzRTwDngqxEZ"
   },
   "source": [
    "Since we've done quite some transformations, let's verify whether we didn't mess up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 248
    },
    "colab_type": "code",
    "id": "v_Ysyp8wqxEa",
    "outputId": "e4c9069f-05c0-4910-9cc2-054ed147d96d"
   },
   "outputs": [],
   "source": [
    "frame = plt.gca()\n",
    "frame.axes.get_xaxis().set_visible(False)\n",
    "frame.axes.get_yaxis().set_visible(False)\n",
    "plt.imshow(x_train[0, :, :, 0], cmap='gray');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "u7P_nqh1qxEd",
    "outputId": "3c56d844-372a-4545-bc23-e9f7254eb037"
   },
   "outputs": [],
   "source": [
    "print(y_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VhxpRIIFqxEh"
   },
   "source": [
    "## Simple CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i5xRRWjfqxEi"
   },
   "source": [
    "We start with a very straightforward network with a single convolutional layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_99OaC-iqxEi"
   },
   "source": [
    "### Required imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pXEYDCjhqxEj"
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Activation, Conv2D, Dense, Dropout, Flatten\n",
    "from keras.optimizers import SGD\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CVq32ELKqxEr"
   },
   "source": [
    "Seed TensorFlow's random number generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AaZS1YfNqxEs"
   },
   "outputs": [],
   "source": [
    "tf.set_random_seed(40580)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IOlV7_JWqxEv"
   },
   "source": [
    "### Model definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UT_I9oGsqxEv"
   },
   "source": [
    "The network has 32 filters, each 3 $\\times$ 3 pixels in its convolutional layer, and a ReLU activation function. For the second, dense layer, the output of the convolutional layer is first flattened. The dense layer's activation function is SoftMax to determine the category for the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 88
    },
    "colab_type": "code",
    "id": "FipbpBIzqxEw",
    "outputId": "1f315761-95cc-4fc3-fea9-821a232e70c1"
   },
   "outputs": [],
   "source": [
    "nr_filters = 32\n",
    "nr_classes = 10\n",
    "conv_x_size, conv_y_size = 3, 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "id": "MV5FFC-IqxE0",
    "outputId": "1b4d9328-30c7-4e5b-af40-b17298cbf8dc"
   },
   "outputs": [],
   "source": [
    "cnn_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HES92ClZqxE6"
   },
   "outputs": [],
   "source": [
    "cnn_hist = cnn_model.compile(loss='categorical_crossentropy',\n",
    "                             optimizer=SGD(lr=0.1, decay=1.0e-6, \n",
    "                                           momentum=0.9, nesterov=True),\n",
    "                             metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4pVlMkXYqxE9"
   },
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 445
    },
    "colab_type": "code",
    "id": "0Xwz6yHPqxE-",
    "outputId": "e9e0d701-b892-4457-9f51-f787ba746ffb",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "cnn_hist = cnn_model.fit(x_train, y_train, batch_size=32, epochs=10,\n",
    "                         verbose=1, validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eZtRIW2qqxFB"
   },
   "outputs": [],
   "source": [
    "def plot_history(network_history):\n",
    "    plt.figure()\n",
    "    plt.xlabel('Epochs')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.plot(network_history.history['loss'])\n",
    "    plt.plot(network_history.history['val_loss'])\n",
    "    plt.legend(['Training', 'Validation'])\n",
    "    plt.figure()\n",
    "    plt.xlabel('Epochs')\n",
    "    plt.ylabel('Accuracy')\n",
    "    plt.plot(network_history.history['acc'])\n",
    "    plt.plot(network_history.history['val_acc'])\n",
    "    plt.legend(['Training', 'Validation'], loc='lower right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 549
    },
    "colab_type": "code",
    "id": "BjTXFIJ2qxFE",
    "outputId": "dfc8ddcd-038e-418a-abbf-6702bc92b467"
   },
   "outputs": [],
   "source": [
    "plot_history(cnn_hist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "fkJuL7MoqxFG",
    "outputId": "f6fabc84-85d3-434d-8e8d-a0947e5e2e4c"
   },
   "outputs": [],
   "source": [
    "cnn_model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pAvvRVrDqxFK"
   },
   "source": [
    "Although the test accuracy is quite acceptable, the training accuracy is a lot higher, so most likely there is some overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ooIrcGprqxFL"
   },
   "outputs": [],
   "source": [
    "cnn_model.save('mnist_cnn.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WEQaS5VrqxFN"
   },
   "source": [
    "## More layers & dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JXwuZrIgqxFO"
   },
   "source": [
    "This model has an additional hidden layer to improve accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "colab_type": "code",
    "id": "kqufPbwhqxFR",
    "outputId": "fbe13086-19e1-42df-b364-b66398d25fea"
   },
   "outputs": [],
   "source": [
    "cnn_do_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O-96Tt2-qxFU"
   },
   "outputs": [],
   "source": [
    "cnn_do_model.compile(loss='categorical_crossentropy',\n",
    "                     optimizer=SGD(),\n",
    "                     metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "IkvW4CAMqxFX",
    "outputId": "e1bdb278-6f08-4bb0-c66d-01a1da6b8946"
   },
   "outputs": [],
   "source": [
    "cnn_do_hist = cnn_do_model.fit(x_train, y_train, batch_size=32, epochs=10,\n",
    "                               verbose=1, validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 549
    },
    "colab_type": "code",
    "id": "DDRwiauZqxFe",
    "outputId": "90d889e0-dee5-43bb-a4d1-ee5f9fc4cd7f"
   },
   "outputs": [],
   "source": [
    "plot_history(cnn_do_hist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "k3-V9sFuqxFh",
    "outputId": "49a6da9e-5859-4db6-a23b-f342680f695c",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cnn_do_model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nwud0IKlqxFk"
   },
   "source": [
    "Now the accuracies for training, validation and test data are more in line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4feXt0MbqxFk"
   },
   "outputs": [],
   "source": [
    "cnn_do_model.save('mnist_cnn_do.h5')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "060_mnist_cnn_complete.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}