SVM-and-PCA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Μαυραπίδης_1115201700082.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyOqMVebGowN3zvBK3BvKovh",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/NikosMav/SVM-and-PCA/blob/main/SVM-and-PCA.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uLSMk-BPHXCE"
      },
      "source": [
        "**Μηχανική Μάθηση (ΕΠ08)**\n",
        "\n",
        "**3η Προγραμματιστική Άσκηση**\n",
        "\n",
        "> **Φοιτητής**: Νικόλαος Μαυραπίδης(sdi1700082)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8S480JaIHbhS",
        "outputId": "d471fe62-2618-45dd-821c-862c7d25a5e6"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V5LwlHrMHYdS"
      },
      "source": [
        "# Importing Libraries and Preparing the Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CG_KU5_X8STA"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn import datasets,svm,metrics\n",
        "import zipfile\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.decomposition import PCA\n",
        "import time"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jRJoZk9yHt5K"
      },
      "source": [
        "Read the files..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kvCvtdALHwnh"
      },
      "source": [
        "# Unzip file \n",
        "zip = zipfile.ZipFile('/content/drive/MyDrive/Colab Notebooks/Μηχανική Μάθηση/Project3/archive.zip')\n",
        "zip.extractall()\n",
        "\n",
        "# Read csvs\n",
        "df_train = pd.read_csv(\"/content/mnist_test.csv\")\n",
        "df_test = pd.read_csv(\"/content/mnist_train.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V233DotSIYBx",
        "outputId": "c6d30f15-2365-49e0-e1ec-65d8cfc17e9f"
      },
      "source": [
        "print(df_train.shape)\n",
        "print(df_test.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10000, 785)\n",
            "(60000, 785)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RFwJdEO_JcZo"
      },
      "source": [
        "# Split Datasets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cNxefNTwJa2R"
      },
      "source": [
        "# 1st column for labels, rest for the train dataset\n",
        "train_data=df_train.iloc[:,1:]\n",
        "train_labels=df_train.iloc[:,0]\n",
        "\n",
        "# Same for test dataset\n",
        "test_data=df_test.iloc[:,1:]\n",
        "test_labels=df_test.iloc[:,0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VuMqvgJ3R0Eb",
        "outputId": "5edc0e69-d785-4249-f670-95fa94647b69"
      },
      "source": [
        "print(train_data.shape)\n",
        "print(train_labels.shape)\n",
        "\n",
        "print(test_data.shape)\n",
        "print(test_labels.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10000, 784)\n",
            "(10000,)\n",
            "(60000, 784)\n",
            "(60000,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZLez0FejLAwf"
      },
      "source": [
        "Normalize datasets..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lYBKgVATK-lo"
      },
      "source": [
        "train_data=train_data/255\n",
        "test_data=test_data/255"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RDwgspcjPReH"
      },
      "source": [
        "# Hyperparameter Tuning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Esn7Kfy6PWkd",
        "outputId": "2a43abd4-b40d-4b8d-cd4b-cbc23e58ee52"
      },
      "source": [
        "# Parameters dictionary\n",
        "parameters = {'kernel':('linear', 'rbf'), 'C':[50, 100, 200], 'gamma':[0.01, 0.05, 0.1]}\n",
        "svc = svm.SVC()\n",
        "# Exhaustive search over specified parameter values\n",
        "clf = GridSearchCV(svc, parameters, refit=True, verbose=3)\n",
        "# fitting the model for grid search\n",
        "clf.fit(train_data,train_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fitting 5 folds for each of 18 candidates, totalling 90 fits\n",
            "[CV] C=50, gamma=0.01, kernel=linear .................................\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.895, total=  13.8s\n",
            "[CV] C=50, gamma=0.01, kernel=linear .................................\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:   13.8s remaining:    0.0s\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.895, total=  13.8s\n",
            "[CV] C=50, gamma=0.01, kernel=linear .................................\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:   27.6s remaining:    0.0s\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.918, total=  14.7s\n",
            "[CV] C=50, gamma=0.01, kernel=linear .................................\n",
            "[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.933, total=  15.2s\n",
            "[CV] C=50, gamma=0.01, kernel=linear .................................\n",
            "[CV] ..... C=50, gamma=0.01, kernel=linear, score=0.934, total=  15.4s\n",
            "[CV] C=50, gamma=0.01, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.945, total=  20.7s\n",
            "[CV] C=50, gamma=0.01, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.947, total=  20.4s\n",
            "[CV] C=50, gamma=0.01, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.960, total=  21.2s\n",
            "[CV] C=50, gamma=0.01, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.976, total=  22.3s\n",
            "[CV] C=50, gamma=0.01, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.01, kernel=rbf, score=0.980, total=  22.1s\n",
            "[CV] C=50, gamma=0.05, kernel=linear .................................\n",
            "[CV] ..... C=50, gamma=0.05, kernel=linear, score=0.895, total=  13.8s\n",
            "[CV] C=50, gamma=0.05, kernel=linear .................................\n",
            "[CV] ..... C=50, gamma=0.05, kernel=linear, score=0.895, total=  13.6s\n",
            "[CV] C=50, gamma=0.05, kernel=linear .................................\n",
            "[CV] ..... C=50, gamma=0.05, kernel=linear, score=0.918, total=  14.1s\n",
            "[CV] C=50, gamma=0.05, kernel=linear .................................\n",
            "[CV] ..... C=50, gamma=0.05, kernel=linear, score=0.933, total=  14.8s\n",
            "[CV] C=50, gamma=0.05, kernel=linear .................................\n",
            "[CV] ..... C=50, gamma=0.05, kernel=linear, score=0.934, total=  14.8s\n",
            "[CV] C=50, gamma=0.05, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.05, kernel=rbf, score=0.951, total= 1.4min\n",
            "[CV] C=50, gamma=0.05, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.05, kernel=rbf, score=0.953, total= 1.4min\n",
            "[CV] C=50, gamma=0.05, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.05, kernel=rbf, score=0.966, total= 1.4min\n",
            "[CV] C=50, gamma=0.05, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.05, kernel=rbf, score=0.974, total= 1.4min\n",
            "[CV] C=50, gamma=0.05, kernel=rbf ....................................\n",
            "[CV] ........ C=50, gamma=0.05, kernel=rbf, score=0.974, total= 1.4min\n",
            "[CV] C=50, gamma=0.1, kernel=linear ..................................\n",
            "[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.895, total=  13.7s\n",
            "[CV] C=50, gamma=0.1, kernel=linear ..................................\n",
            "[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.895, total=  13.4s\n",
            "[CV] C=50, gamma=0.1, kernel=linear ..................................\n",
            "[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.918, total=  14.1s\n",
            "[CV] C=50, gamma=0.1, kernel=linear ..................................\n",
            "[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.933, total=  14.9s\n",
            "[CV] C=50, gamma=0.1, kernel=linear ..................................\n",
            "[CV] ...... C=50, gamma=0.1, kernel=linear, score=0.934, total=  14.7s\n",
            "[CV] C=50, gamma=0.1, kernel=rbf .....................................\n",
            "[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.839, total= 2.2min\n",
            "[CV] C=50, gamma=0.1, kernel=rbf .....................................\n",
            "[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.835, total= 2.2min\n",
            "[CV] C=50, gamma=0.1, kernel=rbf .....................................\n",
            "[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.884, total= 2.2min\n",
            "[CV] C=50, gamma=0.1, kernel=rbf .....................................\n",
            "[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.902, total= 2.2min\n",
            "[CV] C=50, gamma=0.1, kernel=rbf .....................................\n",
            "[CV] ......... C=50, gamma=0.1, kernel=rbf, score=0.877, total= 2.2min\n",
            "[CV] C=100, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.01, kernel=linear, score=0.895, total=  13.7s\n",
            "[CV] C=100, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.01, kernel=linear, score=0.895, total=  13.7s\n",
            "[CV] C=100, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.01, kernel=linear, score=0.918, total=  14.2s\n",
            "[CV] C=100, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.01, kernel=linear, score=0.933, total=  14.8s\n",
            "[CV] C=100, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.01, kernel=linear, score=0.934, total=  14.7s\n",
            "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.945, total=  20.5s\n",
            "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.947, total=  20.3s\n",
            "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.960, total=  21.1s\n",
            "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.976, total=  21.7s\n",
            "[CV] C=100, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.01, kernel=rbf, score=0.980, total=  21.7s\n",
            "[CV] C=100, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.05, kernel=linear, score=0.895, total=  13.9s\n",
            "[CV] C=100, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.05, kernel=linear, score=0.895, total=  13.4s\n",
            "[CV] C=100, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.05, kernel=linear, score=0.918, total=  14.0s\n",
            "[CV] C=100, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.05, kernel=linear, score=0.933, total=  14.8s\n",
            "[CV] C=100, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=100, gamma=0.05, kernel=linear, score=0.934, total=  14.7s\n",
            "[CV] C=100, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.05, kernel=rbf, score=0.951, total= 1.4min\n",
            "[CV] C=100, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.05, kernel=rbf, score=0.953, total= 1.4min\n",
            "[CV] C=100, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.05, kernel=rbf, score=0.966, total= 1.5min\n",
            "[CV] C=100, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.05, kernel=rbf, score=0.974, total= 1.5min\n",
            "[CV] C=100, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=100, gamma=0.05, kernel=rbf, score=0.974, total= 1.4min\n",
            "[CV] C=100, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=100, gamma=0.1, kernel=linear, score=0.895, total=  13.6s\n",
            "[CV] C=100, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=100, gamma=0.1, kernel=linear, score=0.895, total=  13.5s\n",
            "[CV] C=100, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=100, gamma=0.1, kernel=linear, score=0.918, total=  14.3s\n",
            "[CV] C=100, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=100, gamma=0.1, kernel=linear, score=0.933, total=  15.0s\n",
            "[CV] C=100, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=100, gamma=0.1, kernel=linear, score=0.934, total=  14.8s\n",
            "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.839, total= 2.2min\n",
            "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.835, total= 2.2min\n",
            "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.884, total= 2.2min\n",
            "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.902, total= 2.2min\n",
            "[CV] C=100, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=100, gamma=0.1, kernel=rbf, score=0.877, total= 2.2min\n",
            "[CV] C=200, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.01, kernel=linear, score=0.895, total=  13.6s\n",
            "[CV] C=200, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.01, kernel=linear, score=0.895, total=  13.6s\n",
            "[CV] C=200, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.01, kernel=linear, score=0.918, total=  14.1s\n",
            "[CV] C=200, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.01, kernel=linear, score=0.933, total=  15.0s\n",
            "[CV] C=200, gamma=0.01, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.01, kernel=linear, score=0.934, total=  14.8s\n",
            "[CV] C=200, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.01, kernel=rbf, score=0.945, total=  20.4s\n",
            "[CV] C=200, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.01, kernel=rbf, score=0.947, total=  20.2s\n",
            "[CV] C=200, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.01, kernel=rbf, score=0.960, total=  21.2s\n",
            "[CV] C=200, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.01, kernel=rbf, score=0.976, total=  22.0s\n",
            "[CV] C=200, gamma=0.01, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.01, kernel=rbf, score=0.980, total=  21.5s\n",
            "[CV] C=200, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.05, kernel=linear, score=0.895, total=  13.6s\n",
            "[CV] C=200, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.05, kernel=linear, score=0.895, total=  13.4s\n",
            "[CV] C=200, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.05, kernel=linear, score=0.918, total=  14.3s\n",
            "[CV] C=200, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.05, kernel=linear, score=0.933, total=  15.0s\n",
            "[CV] C=200, gamma=0.05, kernel=linear ................................\n",
            "[CV] .... C=200, gamma=0.05, kernel=linear, score=0.934, total=  14.6s\n",
            "[CV] C=200, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.05, kernel=rbf, score=0.951, total= 1.4min\n",
            "[CV] C=200, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.05, kernel=rbf, score=0.953, total= 1.4min\n",
            "[CV] C=200, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.05, kernel=rbf, score=0.966, total= 1.4min\n",
            "[CV] C=200, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.05, kernel=rbf, score=0.974, total= 1.4min\n",
            "[CV] C=200, gamma=0.05, kernel=rbf ...................................\n",
            "[CV] ....... C=200, gamma=0.05, kernel=rbf, score=0.974, total= 1.4min\n",
            "[CV] C=200, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=200, gamma=0.1, kernel=linear, score=0.895, total=  13.6s\n",
            "[CV] C=200, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=200, gamma=0.1, kernel=linear, score=0.895, total=  13.4s\n",
            "[CV] C=200, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=200, gamma=0.1, kernel=linear, score=0.918, total=  14.1s\n",
            "[CV] C=200, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=200, gamma=0.1, kernel=linear, score=0.933, total=  14.8s\n",
            "[CV] C=200, gamma=0.1, kernel=linear .................................\n",
            "[CV] ..... C=200, gamma=0.1, kernel=linear, score=0.934, total=  14.6s\n",
            "[CV] C=200, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=200, gamma=0.1, kernel=rbf, score=0.839, total= 2.2min\n",
            "[CV] C=200, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=200, gamma=0.1, kernel=rbf, score=0.835, total= 2.2min\n",
            "[CV] C=200, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=200, gamma=0.1, kernel=rbf, score=0.884, total= 2.2min\n",
            "[CV] C=200, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=200, gamma=0.1, kernel=rbf, score=0.902, total= 2.2min\n",
            "[CV] C=200, gamma=0.1, kernel=rbf ....................................\n",
            "[CV] ........ C=200, gamma=0.1, kernel=rbf, score=0.877, total= 2.2min\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[Parallel(n_jobs=1)]: Done  90 out of  90 | elapsed: 69.9min finished\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=None, error_score=nan,\n",
              "             estimator=SVC(C=1.0, break_ties=False, cache_size=200,\n",
              "                           class_weight=None, coef0=0.0,\n",
              "                           decision_function_shape='ovr', degree=3,\n",
              "                           gamma='scale', kernel='rbf', max_iter=-1,\n",
              "                           probability=False, random_state=None, shrinking=True,\n",
              "                           tol=0.001, verbose=False),\n",
              "             iid='deprecated', n_jobs=None,\n",
              "             param_grid={'C': [50, 100, 200], 'gamma': [0.01, 0.05, 0.1],\n",
              "                         'kernel': ('linear', 'rbf')},\n",
              "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
              "             scoring=None, verbose=3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "10F0ABd4SMGr",
        "outputId": "49776de7-5b43-4bd7-d9a9-bab186d785ce"
      },
      "source": [
        "# print best parameter after tuning\n",
        "print(\"Best parameters: \", clf.best_params_)\n",
        "  \n",
        "# print how our model looks after hyper-parameter tuning\n",
        "print(\"Model: \", clf.best_estimator_)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Best parameters:  {'C': 50, 'gamma': 0.05, 'kernel': 'rbf'}\n",
            "Model:  SVC(C=50, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,\n",
            "    decision_function_shape='ovr', degree=3, gamma=0.05, kernel='rbf',\n",
            "    max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
            "    tol=0.001, verbose=False)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xcjqHqEXuWvI"
      },
      "source": [
        "We extracted the best parameters for SVM classifier. Now lets test them and see what scores we achieve."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bNcM9A0zLOVX"
      },
      "source": [
        "# SVM Classifier"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C30XScU6LFz3",
        "outputId": "000060fa-0452-4749-9666-0e5aa653f2ba"
      },
      "source": [
        "# Using previous parameters...\n",
        "classifier=svm.SVC(C=50,kernel='rbf',gamma=0.05)\n",
        "classifier.fit(train_data,train_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SVC(C=50, break_ties=False, cache_size=200, class_weight=None, coef0=0.0,\n",
              "    decision_function_shape='ovr', degree=3, gamma=0.05, kernel='rbf',\n",
              "    max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
              "    tol=0.001, verbose=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iy73qFUALl4Y"
      },
      "source": [
        "predicted=classifier.predict(test_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "16__xn4ERP_s"
      },
      "source": [
        "Accuracy, f1-score and other metrics...  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Md3lUQUJLqR_",
        "outputId": "f2f80bc1-84ad-4156-9581-ba3de3c6f4e1"
      },
      "source": [
        "print(\"Classification report for classifier :\\n%s\\n\" % (metrics.classification_report(test_labels, predicted)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Classification report for classifier :\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.98      0.98      0.98      5923\n",
            "           1       0.98      0.98      0.98      6742\n",
            "           2       0.92      0.97      0.94      5958\n",
            "           3       0.97      0.94      0.95      6131\n",
            "           4       0.96      0.97      0.97      5842\n",
            "           5       0.96      0.97      0.96      5421\n",
            "           6       0.98      0.98      0.98      5918\n",
            "           7       0.97      0.96      0.97      6265\n",
            "           8       0.95      0.95      0.95      5851\n",
            "           9       0.96      0.93      0.95      5949\n",
            "\n",
            "    accuracy                           0.96     60000\n",
            "   macro avg       0.96      0.96      0.96     60000\n",
            "weighted avg       0.96      0.96      0.96     60000\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mDC-v7Ivd4Zm"
      },
      "source": [
        "# PCA Process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "apDXnsccd2-v"
      },
      "source": [
        "def get_pca(x_train, x_test, n_components):\n",
        "    pca = PCA(n_components=n_components, svd_solver='full')\n",
        "    pca.fit(x_train)\n",
        "    x_train = pca.transform(x_train)\n",
        "    x_test = pca.transform(x_test)\n",
        "    print(x_train.shape, x_test.shape)\n",
        "    return x_train, x_test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AzY6W4KFvi-t"
      },
      "source": [
        "Get reduced train and test datasets with 0.95 percentage of sustained covariance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dF0RIwSqvPxH",
        "outputId": "f804178d-b7a9-4a41-b1a6-bee90263d5bc"
      },
      "source": [
        "temp_train_data = train_data\n",
        "temp_test_data = test_data\n",
        "temp_train_data, temp_test_data = get_pca(temp_train_data, temp_test_data, 0.95)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10000, 149) (60000, 149)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eIGgQUbKvpuc"
      },
      "source": [
        "And repeat the classification process again"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TKsupT0Ovn9P"
      },
      "source": [
        "# Count time\n",
        "start = time.time()\n",
        "\n",
        "# Using previous parameters...\n",
        "classifier=svm.SVC(C=50,kernel='rbf',gamma=0.05)\n",
        "classifier.fit(temp_train_data,train_labels)\n",
        "\n",
        "# Make predictions\n",
        "predicted=classifier.predict(temp_test_data)\n",
        "\n",
        "end = time.time()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f7wkj0Krv2oM",
        "outputId": "8790ed53-c6ae-463c-88ff-887f35266f1c"
      },
      "source": [
        "print(\"Classification report for classifier after dimensionality reduction :\\n%s\\n\" % (metrics.classification_report(test_labels, predicted)))\n",
        "\n",
        "print(\"Time in sec: \", end - start)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Classification report for classifier after dimensionality reduction :\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.98      0.98      0.98      5923\n",
            "           1       0.98      0.98      0.98      6742\n",
            "           2       0.94      0.97      0.95      5958\n",
            "           3       0.97      0.94      0.96      6131\n",
            "           4       0.96      0.97      0.97      5842\n",
            "           5       0.96      0.97      0.96      5421\n",
            "           6       0.98      0.98      0.98      5918\n",
            "           7       0.97      0.96      0.97      6265\n",
            "           8       0.95      0.95      0.95      5851\n",
            "           9       0.96      0.94      0.95      5949\n",
            "\n",
            "    accuracy                           0.97     60000\n",
            "   macro avg       0.97      0.97      0.97     60000\n",
            "weighted avg       0.97      0.97      0.97     60000\n",
            "\n",
            "\n",
            "Time in sec:  93.02520632743835\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E-KYjlAsmnr9"
      },
      "source": [
        "Get reduced train and test datasets with 0.75 percentage of sustained covariance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j1e3w0fYmpvk",
        "outputId": "44da3a98-17fe-489b-84de-e861f2aac9b8"
      },
      "source": [
        "temp_train_data = train_data\n",
        "temp_test_data = test_data\n",
        "temp_train_data, temp_test_data = get_pca(temp_train_data, temp_test_data, 0.75)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10000, 32) (60000, 32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jm9DrbjeoANy"
      },
      "source": [
        "# Count time\n",
        "start = time.time()\n",
        "\n",
        "# Using previous parameters...\n",
        "classifier=svm.SVC(C=50,kernel='rbf',gamma=0.05)\n",
        "classifier.fit(temp_train_data,train_labels)\n",
        "\n",
        "# Make predictions\n",
        "predicted=classifier.predict(temp_test_data)\n",
        "\n",
        "end = time.time()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XAV6_MMAoDQz",
        "outputId": "2633ea83-df46-417c-a317-5930d1a46cfc"
      },
      "source": [
        "print(\"Classification report for classifier after dimensionality reduction :\\n%s\\n\" % (metrics.classification_report(test_labels, predicted)))\n",
        "\n",
        "print(\"Time in sec: \", end - start)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Classification report for classifier after dimensionality reduction :\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.98      0.98      0.98      5923\n",
            "           1       0.98      0.98      0.98      6742\n",
            "           2       0.95      0.97      0.96      5958\n",
            "           3       0.97      0.95      0.96      6131\n",
            "           4       0.96      0.97      0.97      5842\n",
            "           5       0.96      0.97      0.96      5421\n",
            "           6       0.97      0.98      0.98      5918\n",
            "           7       0.97      0.97      0.97      6265\n",
            "           8       0.96      0.96      0.96      5851\n",
            "           9       0.96      0.94      0.95      5949\n",
            "\n",
            "    accuracy                           0.97     60000\n",
            "   macro avg       0.97      0.97      0.97     60000\n",
            "weighted avg       0.97      0.97      0.97     60000\n",
            "\n",
            "\n",
            "Time in sec:  18.408257722854614\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U43uSrpvoLBh"
      },
      "source": [
        "Get reduced train and test datasets with 0.5 percentage of sustained covariance"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mw5A7xK5oPEm",
        "outputId": "d8f1c49a-7336-42bb-9420-e4ec919bc6d7"
      },
      "source": [
        "temp_train_data = train_data\n",
        "temp_test_data = test_data\n",
        "temp_train_data, temp_test_data = get_pca(temp_train_data, temp_test_data, 0.5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(10000, 11) (60000, 11)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Over7nxUoUz6"
      },
      "source": [
        "# Count time\n",
        "start = time.time()\n",
        "\n",
        "# Using previous parameters...\n",
        "classifier=svm.SVC(C=50,kernel='rbf',gamma=0.05)\n",
        "classifier.fit(temp_train_data,train_labels)\n",
        "\n",
        "# Make predictions\n",
        "predicted=classifier.predict(temp_test_data)\n",
        "\n",
        "end = time.time()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u8VZRwluoXw8",
        "outputId": "01b5bdc0-3070-4027-f999-6b134a6d8995"
      },
      "source": [
        "print(\"Classification report for classifier after dimensionality reduction :\\n%s\\n\" % (metrics.classification_report(test_labels, predicted)))\n",
        "\n",
        "print(\"Time in sec: \", end - start)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Classification report for classifier after dimensionality reduction :\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.95      0.97      0.96      5923\n",
            "           1       0.97      0.98      0.97      6742\n",
            "           2       0.92      0.95      0.93      5958\n",
            "           3       0.91      0.88      0.90      6131\n",
            "           4       0.91      0.92      0.91      5842\n",
            "           5       0.91      0.90      0.91      5421\n",
            "           6       0.95      0.96      0.96      5918\n",
            "           7       0.94      0.93      0.94      6265\n",
            "           8       0.89      0.89      0.89      5851\n",
            "           9       0.88      0.87      0.88      5949\n",
            "\n",
            "    accuracy                           0.93     60000\n",
            "   macro avg       0.92      0.92      0.92     60000\n",
            "weighted avg       0.93      0.93      0.93     60000\n",
            "\n",
            "\n",
            "Time in sec:  7.537610054016113\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4_MRRSLVox-q"
      },
      "source": [
        "**Observation**: As we can see the best percentage is ~$0.75$% of sustained covariance, taking into consideration the accuracy and other metric scores and the time it took for the model fitting and predictions. "
      ]
    }
  ]
}