convolutional-neural-network.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-requirements\n",
    "\n",
    "1. Load the data from: <https://www.kaggle.com/datasets/kmirfan/micro-expressions> (last visit 10.12.2022)\n",
    "2. paste the train data into `data/train`\n",
    "3. paste the test data into `data/test`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Packages\n",
    "Here the libraries are imported. Important is that tensorflow with keras is used for this project to train and test the\n",
    "modle. With matplotlib some statistic will be shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "\n",
    "import numpy as np\n",
    "from keras.preprocessing import image"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configuration\n",
    "Here the shape of the images (in our cas 80x80), the number of epochs, the number of classes and the model folder are configured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SHAPE = 80\n",
    "EPOCHS = 5 \n",
    "NUM_CLASSES = 7\n",
    "\n",
    "MODEL_FOLDER = \"./data/trained_models/\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data generation\n",
    "For the data generation the ImageDataGenerator of tensorflow is used. It can be configured from which directory the\n",
    "files are loaded and if there should be any variation like a rotation or not. In this project the pictures are already\n",
    "labled by being sorted into the right folder. As a result, the ImageDataGenerator sets the lable to the foldername of\n",
    "the image. Here two generators are use for loading the test and training data provided by\n",
    "<https://www.kaggle.com/datasets/kmirfan/micro-expressions>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the training image generator\n",
    "TRAINING_DIR = \"./data/train/\"\n",
    "training_datagen = ImageDataGenerator(\n",
    "        rescale = 1./255,\n",
    "        rotation_range=20,\n",
    "        width_shift_range=0.2,\n",
    "        height_shift_range=0.2,\n",
    "        shear_range=0.2,\n",
    "        zoom_range=0.2,\n",
    "        horizontal_flip=True,\n",
    "        fill_mode='nearest'\n",
    "    )\n",
    "\n",
    "# Configure the validation image generator\n",
    "VALIDATION_DIR = \"./data/test/\"\n",
    "validation_datagen = ImageDataGenerator(rescale = 1./255)\n",
    "\n",
    "# Set the image size and batch size for training and validation\n",
    "# generators \n",
    "train_generator = training_datagen.flow_from_directory(\n",
    "    TRAINING_DIR,\n",
    "    target_size=(SHAPE,SHAPE),\n",
    "    class_mode='categorical',\n",
    "    batch_size=64\n",
    ")\n",
    "\n",
    "validation_generator = validation_datagen.flow_from_directory(\n",
    "    VALIDATION_DIR,\n",
    "    target_size=(SHAPE,SHAPE),\n",
    "    class_mode='categorical',\n",
    "    batch_size=64\n",
    ")\n",
    "\n",
    "class_names = list(validation_generator.class_indices.keys())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image informations\n",
    "Here the labels (class names) are printed to check the import of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(class_names)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model creation\n",
    "In this section the model is created. This model uses following combination:\n",
    "![Architecture of the model](pictures/architecture_model.png)\n",
    "\n",
    "Afterwards the summary of the model is provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    # Note the input shape is the desired size of the image SHAPExSHAPE with\n",
    "    # 3 bytes color\n",
    "    # This is the first convolution\n",
    "    tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(SHAPE, SHAPE, 3)),\n",
    "    tf.keras.layers.MaxPooling2D(2, 2),\n",
    "    # The second convolution\n",
    "    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # The third convolution\n",
    "    tf.keras.layers.Conv2D(128 , (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # The fourth convolution\n",
    "    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(2,2),\n",
    "    # Flatten the results to feed into a DNN\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dropout(0.5),\n",
    "    # 512 neuron hidden layer\n",
    "    tf.keras.layers.Dense(512, activation='relu'),\n",
    "    tf.keras.layers.Dense(7, activation=tf.nn.softmax)\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training\n",
    "Now the model needs to be trained for later use. By default the adam is used as optimizer and per epoch 100 steps will\n",
    "be done. After finishing the training, the model is saved to the specified location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss = 'categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(train_generator, epochs=EPOCHS, steps_per_epoch=100, validation_data = validation_generator, verbose = 1, validation_steps=3)\n",
    "\n",
    "model.save(MODEL_FOLDER + \"emotion.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# History\n",
    "After the training of the model, it is interesting to analyze the training history of the model. With its help the model\n",
    "can be optimized to overfitt not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc = history.history['accuracy']\n",
    "val_acc = history.history['val_accuracy']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(acc))\n",
    "\n",
    "plt.plot(epochs, acc, 'r', label='Training accuracy')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation accuracy')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend(loc=0)\n",
    "plt.figure()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validation\n",
    "Validate created models by running them through some probability tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load model from file\n",
    "model = tf.keras.models.load_model(MODEL_FOLDER + \"emotion_larger_adam_variation.h5\")\n",
    "# Define class names for separate execution\n",
    "class_names = ['anger', 'disgust', 'fear', 'happiness', 'neutral', 'sadness', 'surprise']\n",
    "\n",
    "# paths of the pictures loaded from the uploaded folder\n",
    "uploaded = []\n",
    "# labels bases on the folder name\n",
    "labels = []\n",
    "\n",
    "# Load the uploaded images of the different folders\n",
    "for class_name in class_names:\n",
    "    path = os.path.join('./data/uploaded/', class_name)\n",
    "    for img in os.listdir(path):\n",
    "        if (img == '.gitkeep'):\n",
    "            continue\n",
    "        uploaded.append(os.path.join(path, img))\n",
    "        labels.append(class_name)\n",
    "\n",
    "# Transform label from string to integer identifier\n",
    "for i in range(len(labels)):\n",
    "    labels[i] = class_names.index(labels[i])\n",
    "\n",
    "# Images for predictions\n",
    "images_predict = []\n",
    "# Images to showcase\n",
    "images = []\n",
    "\n",
    "# Create the images for prediction from the paths\n",
    "for img in uploaded:\n",
    "    new_image = tf.keras.utils.load_img(img, target_size=(SHAPE, SHAPE))\n",
    "    new_image_batch = tf.expand_dims(tf.keras.utils.img_to_array(new_image), 0)\n",
    "    images_predict.append(new_image_batch)\n",
    "    images.append(new_image)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculation of test accuracy\n",
    "This section calculates the test accuracy and training accuracy for the provided/loaded model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_loss, test_acc = model.evaluate(validation_generator)\n",
    "print('\\nTest accuracy:', test_acc)\n",
    "print('\\nTest loss:', test_loss)\n",
    "\n",
    "train_loss, train_acc = model.evaluate(train_generator)\n",
    "print('\\nTraining accuracy:', train_acc)\n",
    "print('\\nTraining loss:', train_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show textured predictions of one image\n",
    "predictions_self = model.predict(images_predict[6])\n",
    "print(predictions_self[0])\n",
    "\n",
    "score = predictions_self[0]\n",
    "print(\n",
    "    \"This image most likely belongs to {} with a {:.2f} percent confidence.\"\n",
    "    .format(class_names[np.argmax(score)], 100 * np.max(score))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another version of displaying the predictions in text\n",
    "probability_model = tf.keras.Sequential([model])\n",
    "\n",
    "predictions = probability_model.predict(images_predict[0])\n",
    "\n",
    "print(predictions[0])\n",
    "print(np.argmax(predictions[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image prediction plotting\n",
    "Based on: https://www.tensorflow.org/tutorials/keras/classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image plot creation of specific image\n",
    "def plot_image(predictions_array, true_label, img):\n",
    "  plt.grid(False)\n",
    "  plt.xticks([])\n",
    "  plt.yticks([])\n",
    "\n",
    "  plt.imshow(img, cmap=plt.cm.binary)\n",
    "\n",
    "  predicted_label = np.argmax(predictions_array)\n",
    "  if predicted_label == true_label:\n",
    "    color = 'blue'\n",
    "  else:\n",
    "    color = 'red'\n",
    "\n",
    "  plt.xlabel(\"{} {:2.0f}% ({})\".format(class_names[predicted_label],\n",
    "                                100*np.max(predictions_array),\n",
    "                                class_names[true_label],\n",
    "                                color=color)\n",
    "                                )\n",
    "\n",
    "# Plot creation to display multiple image prediction\n",
    "def plot_value_array(predictions_array, true_label):\n",
    "  plt.grid(False)\n",
    "  plt.xticks(range(7))\n",
    "  plt.yticks([])\n",
    "  thisplot = plt.bar(range(7), predictions_array, color=\"#777777\")\n",
    "  plt.ylim([0, 1])\n",
    "  predicted_label = np.argmax(predictions_array)\n",
    "\n",
    "  thisplot[predicted_label].set_color('red')\n",
    "  thisplot[true_label].set_color('blue')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot single image prediction\n",
    "i = 0\n",
    "plt.figure(figsize=(6,3))\n",
    "plt.subplot(1,2,1)\n",
    "plot_image(predictions[i], labels[i], images[i])\n",
    "plt.subplot(1,2,2)\n",
    "plot_value_array(predictions[i],  labels[i])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Image validation matrix\n",
    "Here the uploaded images will be drawn in a matrix.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the first X test images, their predicted labels, and the true labels.\n",
    "# Color correct predictions in blue and incorrect predictions in red.\n",
    "num_rows = 5\n",
    "num_cols = 3\n",
    "num_images = len(images)\n",
    "plt.figure(figsize=(2*2*num_cols, 2*num_rows))\n",
    "for i in range(num_images):\n",
    "  predictions = probability_model.predict(images_predict[i])\n",
    "  print(predictions[0])\n",
    "  plt.subplot(num_rows, 2*num_cols, 2*i+1)\n",
    "  plot_image(predictions[0], labels[i], images[i])\n",
    "  plt.subplot(num_rows, 2*num_cols, 2*i+2)\n",
    "  plot_value_array(predictions[0], labels[i])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8 | packaged by conda-forge | (main, Nov  4 2022, 13:42:51) [MSC v.1916 64 bit (AMD64)]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "5457cff4e4e0348cc317dd2b7808ef7c0c674a3b4613bc74cfa22be171ec1d78"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}