05. Artificial Neural Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Agenda\n",
    "- Deep Learning\n",
    "- Artificial Neural Network(ANN)\n",
    "- How Neural Network Works\n",
    "- Mathematics Behind NN\n",
    "- Activation Function\n",
    "- Bias Node\n",
    "- Forward Propagation\n",
    "- Back Propagation\n",
    "- Learning Rate\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Deep Learning?\n",
    "\n",
    "- Suppose we have n=100 features  \n",
    "$ x_1^2 + x_1x_2 + x_1x_3 + x_1x_4 + x_1x_5+   .....x_{99}x_{100} + x_{100}^2$\n",
    "- For Non-Linear Logistic Regression, a quadratic function of order 2 will have features = n*(n+1)/2 = 5050 \n",
    "- For order 3 it will be = n*(n+1)*(n+2)/6 = 1,71,700\n",
    "  \n",
    "  \n",
    "Suppose We have an Image of pixel size 100 * 100  \n",
    "Total pixels = 100 * 100 = 10,000 pixels, n = 10,000 (30,000 for RGB )  \n",
    "n = 10,000 for order 2 quadratic feature will have total 100 Million features (900 Million feature for RGB)\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning\n",
    "- Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. \n",
    "- Learning can be supervised, semi-supervised or unsupervised\n",
    "- The “deep” in deep learning refers to the depth of the network.\n",
    "\n",
    "<img src=\"Image/deep.png\" width=\"400\" />\n",
    "\n",
    "#### Why Deep Learning?\n",
    "\n",
    "<img src=\"Image/perf.png\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Artificial Neural Network\n",
    "- Dendrites are the structures on the neuron that receive electrical messages, to process these signals, and to transfer the information to the soma of the neuron\n",
    "- Axons: primary transmission lines of the nervous system\n",
    "\n",
    "\n",
    "<img src=\"Image/neuron.jpg\" width=\"400\" />\n",
    "<br>\n",
    "<br>  \n",
    "<br>  \n",
    "<br>\n",
    "<br>\n",
    "  \n",
    "  \n",
    "  \n",
    "  \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How Neural Network Works?\n",
    "<img src=\"Image/neuron-3.png\" width=\"300\" />\n",
    "<br>\n",
    "<br>  \n",
    "<br>  \n",
    "<br>\n",
    "<br>\n",
    "<img src=\"Image/Basic_Neural.png\" width=\"500\" />\n",
    "<br>\n",
    "<br>  \n",
    "<br>  \n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Neural Network with Hidden Layer\n",
    "\n",
    "- Between Input and Output layer\n",
    "- Allow for the function of a neural network to be broken down into specific transformations of the data\n",
    "- Each hidden layer function is having specific task to produce a defined output\n",
    "\n",
    "<img src=\"Image/house_neural.png\" width=\"500\" />\n",
    "<br>\n",
    "<br>  \n",
    "<br>  \n",
    "<br>\n",
    "<br>\n",
    "<img src=\"Image/house_hidden2.png\" width=\"500\" />\n",
    "<br>\n",
    "<br>  \n",
    "<br>  \n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation Function\n",
    "\n",
    "- These type of functions are attached to each neuron in the network, and determines whether it should be activated or not, based on whether each neuron’s input is relevant for the model’s prediction.\n",
    "- It helps to standardize the output of each neuron.\n",
    "- E.g: Threshold, Sigmoid, Relu(Rectifier), Softmax\n",
    "\n",
    "<img src=\"Image/activation.png\" width=\"500\" />\n",
    "\n",
    "#### Whats the diff between Step function, Linear function and Sigmoid function?  \n",
    "\n",
    "\n",
    "Linear Function:\n",
    "- Using Linear function only will make the output layer to be a linear function as well so we can't map a non-linear dataset\n",
    "\n",
    "Step Function: \n",
    "- we define threshold values and have discrete output values\n",
    "- if(z > threshold) — “activate” the node (value 1)\n",
    "- if(z < threshold) — don’t “activate” the node (value 0)\n",
    "- So, we have value either 0 or 1\n",
    "- issue here is that it is possible multiple output classes/nodes to be activated (to have the value 1). So we are not able to properly classify/decide.\n",
    "\n",
    "Sigmoid Function:  \n",
    "\n",
    "$ \\theta(x) = \\frac {1} {1 + e^{-x}} $\n",
    "\n",
    "- It is a non-linear function\n",
    "- Value range is (0,1)\n",
    "- classify values either 1 or 0\n",
    "\n",
    "<img src=\"Image/sigmoid1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "#### Threshold Function\n",
    "\n",
    "$ \\theta(x): $   \n",
    "=0 if x < 0  \n",
    "=1 if x > 0 \n",
    "\n",
    "<img src=\"Image/threshold1.png\" width=\"300\" />\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rectifier Function\n",
    "\n",
    "$ \\theta(x) = max(x,0) $\n",
    "\n",
    "<img src=\"Image/rectifier1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bias\n",
    "- Its a constant which helps the model in a way that it can fit best for the given data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Forward Propagation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How do Neural network Learn?\n",
    "\n",
    "- Cost reduces with adjustment in weight(w)\n",
    "<img src=\"Image/neural_learn.png\">\n",
    "\n",
    "## Back Propagation\n",
    "\n",
    "- Error propagates from right to left and update the weights according to how much they are responsible for the error.\n",
    "- Determining how changing the weights impact the overall cost in the neural network.\n",
    "\n",
    "## Learning Rate\n",
    "\n",
    "The learning rate decides by how much we update the weights\n",
    "\n",
    "## Perceptron\n",
    "A collection of neurons, along with a set of input nodes connected to the inputs via weighted edges, is a perceptron, the simplest neural network.\n",
    "\n",
    "<img src=\"Image/neuron-3.png\" width=\"300\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network Examples:\n",
    "\n",
    "$ x_1,x_2 \\epsilon (0,1)$  \n",
    "y = x1 AND x2\n",
    "<img src=\"Image/ex-nn.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### AND Gate\n",
    "$h_\\theta(x) = g(-50+30x_1+30x_2)$  \n",
    "\n",
    "<img src=\"Image/AND1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### OR Gate\n",
    "$h_\\theta(x) = g(-20+30x_1+30x_2)$  \n",
    "\n",
    "<img src=\"Image/OR.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-50, 10, 30]\n"
     ]
    }
   ],
   "source": [
    "import numpy, random, os\n",
    "lr = 1 #learning rate\n",
    "bias = 1 #value of bias\n",
    "weights = [-50, 10, 30]\n",
    "print(weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-24, 30, 39]\n",
      "1\n",
      "0\n",
      "1 or 0 is :  1\n"
     ]
    }
   ],
   "source": [
    "#Training the model\n",
    "def perceptron(x_1, x_2, output) :\n",
    "    outputP = bias*weights[0]+x_1*weights[1]+x_2*weights[2]\n",
    "    if outputP > 4.6 : #activation function (here Heaviside)\n",
    "        outputP = 1\n",
    "    else :\n",
    "        outputP = 0\n",
    "    #print(output,outputP)    \n",
    "    error = (output-outputP)**2\n",
    "    #print(error)\n",
    "    weights[0] += error * bias * lr\n",
    "    weights[1] += error * x_1 * lr\n",
    "    weights[2] += error * x_2 * lr\n",
    "\n",
    "    #print (weights)\n",
    "\n",
    "#Making the prediction\n",
    "def predict(x_1, x_2):\n",
    "    outputP = bias*weights[0] + x*weights[1] + y*weights[2] \n",
    "    if outputP > 4.6: #activation function\n",
    "        outputP = 1\n",
    "    else :\n",
    "        \n",
    "        outputP = 0\n",
    "    return outputP\n",
    "    \n",
    "for i in range(50) :\n",
    "    #print(\"Running loop i=%s\"%i)\n",
    "    perceptron(1,1,1) #True or true\n",
    "    perceptron(1,0,1) #True or false\n",
    "    perceptron(0,1,1) #False or true\n",
    "    perceptron(0,0,0) #False or false\n",
    "print(weights)     \n",
    "x = int(input())\n",
    "y = int(input())\n",
    "output_predict = predict(x, y )\n",
    "print(x, \"or\", y, \"is : \", output_predict)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent\n",
    "\n",
    "Two Types:\n",
    "1. Batch Gradient Descent\n",
    "2. Stochastic Gradient Descent\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Artificial Neural Network with Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "537/537 [==============================] - 2s 3ms/step - loss: 0.6890 - acc: 0.6518\n",
      "Epoch 2/100\n",
      "537/537 [==============================] - 0s 309us/step - loss: 0.6660 - acc: 0.6499\n",
      "Epoch 3/100\n",
      "537/537 [==============================] - 0s 268us/step - loss: 0.5985 - acc: 0.6499\n",
      "Epoch 4/100\n",
      "537/537 [==============================] - 0s 356us/step - loss: 0.5344 - acc: 0.6499\n",
      "Epoch 5/100\n",
      "537/537 [==============================] - 0s 348us/step - loss: 0.5105 - acc: 0.6499\n",
      "Epoch 6/100\n",
      "537/537 [==============================] - 0s 305us/step - loss: 0.5019 - acc: 0.6499\n",
      "Epoch 7/100\n",
      "537/537 [==============================] - 0s 354us/step - loss: 0.4952 - acc: 0.6499\n",
      "Epoch 8/100\n",
      "537/537 [==============================] - 0s 263us/step - loss: 0.4905 - acc: 0.6499\n",
      "Epoch 9/100\n",
      "537/537 [==============================] - 0s 279us/step - loss: 0.4860 - acc: 0.7095\n",
      "Epoch 10/100\n",
      "537/537 [==============================] - 0s 251us/step - loss: 0.4824 - acc: 0.7803\n",
      "Epoch 11/100\n",
      "537/537 [==============================] - 0s 311us/step - loss: 0.4791 - acc: 0.7803\n",
      "Epoch 12/100\n",
      "537/537 [==============================] - 0s 307us/step - loss: 0.4764 - acc: 0.7803\n",
      "Epoch 13/100\n",
      "537/537 [==============================] - 0s 264us/step - loss: 0.4729 - acc: 0.7858\n",
      "Epoch 14/100\n",
      "537/537 [==============================] - 0s 264us/step - loss: 0.4701 - acc: 0.7765\n",
      "Epoch 15/100\n",
      "537/537 [==============================] - 0s 261us/step - loss: 0.4681 - acc: 0.7803\n",
      "Epoch 16/100\n",
      "537/537 [==============================] - 0s 264us/step - loss: 0.4658 - acc: 0.7840\n",
      "Epoch 17/100\n",
      "537/537 [==============================] - 0s 259us/step - loss: 0.4638 - acc: 0.7858\n",
      "Epoch 18/100\n",
      "537/537 [==============================] - 0s 324us/step - loss: 0.4620 - acc: 0.7896\n",
      "Epoch 19/100\n",
      "537/537 [==============================] - 0s 277us/step - loss: 0.4608 - acc: 0.7896\n",
      "Epoch 20/100\n",
      "537/537 [==============================] - 0s 244us/step - loss: 0.4588 - acc: 0.7896\n",
      "Epoch 21/100\n",
      "537/537 [==============================] - 0s 257us/step - loss: 0.4573 - acc: 0.7933 0s - loss: 0.4825 - acc: 0.77\n",
      "Epoch 22/100\n",
      "537/537 [==============================] - 0s 237us/step - loss: 0.4559 - acc: 0.7952\n",
      "Epoch 23/100\n",
      "537/537 [==============================] - 0s 270us/step - loss: 0.4542 - acc: 0.7952\n",
      "Epoch 24/100\n",
      "537/537 [==============================] - 0s 266us/step - loss: 0.4530 - acc: 0.7970\n",
      "Epoch 25/100\n",
      "537/537 [==============================] - 0s 250us/step - loss: 0.4514 - acc: 0.8007\n",
      "Epoch 26/100\n",
      "537/537 [==============================] - 0s 289us/step - loss: 0.4504 - acc: 0.7970\n",
      "Epoch 27/100\n",
      "537/537 [==============================] - 0s 294us/step - loss: 0.4486 - acc: 0.8007\n",
      "Epoch 28/100\n",
      "537/537 [==============================] - 0s 250us/step - loss: 0.4472 - acc: 0.8007\n",
      "Epoch 29/100\n",
      "537/537 [==============================] - 0s 268us/step - loss: 0.4458 - acc: 0.8045\n",
      "Epoch 30/100\n",
      "537/537 [==============================] - 0s 255us/step - loss: 0.4445 - acc: 0.8007\n",
      "Epoch 31/100\n",
      "537/537 [==============================] - 0s 264us/step - loss: 0.4437 - acc: 0.8026\n",
      "Epoch 32/100\n",
      "537/537 [==============================] - 0s 251us/step - loss: 0.4417 - acc: 0.8045\n",
      "Epoch 33/100\n",
      "537/537 [==============================] - 0s 283us/step - loss: 0.4403 - acc: 0.8007\n",
      "Epoch 34/100\n",
      "537/537 [==============================] - 0s 292us/step - loss: 0.4388 - acc: 0.8045\n",
      "Epoch 35/100\n",
      "537/537 [==============================] - 0s 272us/step - loss: 0.4373 - acc: 0.8045\n",
      "Epoch 36/100\n",
      "537/537 [==============================] - 0s 218us/step - loss: 0.4362 - acc: 0.8045\n",
      "Epoch 37/100\n",
      "537/537 [==============================] - 0s 242us/step - loss: 0.4355 - acc: 0.8026\n",
      "Epoch 38/100\n",
      "537/537 [==============================] - 0s 242us/step - loss: 0.4336 - acc: 0.8045\n",
      "Epoch 39/100\n",
      "537/537 [==============================] - 0s 266us/step - loss: 0.4328 - acc: 0.8045\n",
      "Epoch 40/100\n",
      "537/537 [==============================] - 0s 246us/step - loss: 0.4314 - acc: 0.8045\n",
      "Epoch 41/100\n",
      "537/537 [==============================] - 0s 281us/step - loss: 0.4308 - acc: 0.8063\n",
      "Epoch 42/100\n",
      "537/537 [==============================] - 0s 354us/step - loss: 0.4289 - acc: 0.8082\n",
      "Epoch 43/100\n",
      "537/537 [==============================] - 0s 285us/step - loss: 0.4279 - acc: 0.8119\n",
      "Epoch 44/100\n",
      "537/537 [==============================] - 0s 246us/step - loss: 0.4273 - acc: 0.8101\n",
      "Epoch 45/100\n",
      "537/537 [==============================] - 0s 259us/step - loss: 0.4260 - acc: 0.8156\n",
      "Epoch 46/100\n",
      "537/537 [==============================] - 0s 281us/step - loss: 0.4249 - acc: 0.8138\n",
      "Epoch 47/100\n",
      "537/537 [==============================] - 0s 248us/step - loss: 0.4238 - acc: 0.8119\n",
      "Epoch 48/100\n",
      "537/537 [==============================] - 0s 289us/step - loss: 0.4227 - acc: 0.8156\n",
      "Epoch 49/100\n",
      "537/537 [==============================] - 0s 264us/step - loss: 0.4221 - acc: 0.8101\n",
      "Epoch 50/100\n",
      "537/537 [==============================] - 0s 231us/step - loss: 0.4209 - acc: 0.8156\n",
      "Epoch 51/100\n",
      "537/537 [==============================] - 0s 281us/step - loss: 0.4200 - acc: 0.8156\n",
      "Epoch 52/100\n",
      "537/537 [==============================] - 0s 315us/step - loss: 0.4189 - acc: 0.8138\n",
      "Epoch 53/100\n",
      "537/537 [==============================] - 0s 343us/step - loss: 0.4179 - acc: 0.8138\n",
      "Epoch 54/100\n",
      "537/537 [==============================] - 0s 266us/step - loss: 0.4173 - acc: 0.8119\n",
      "Epoch 55/100\n",
      "537/537 [==============================] - 0s 285us/step - loss: 0.4156 - acc: 0.8138\n",
      "Epoch 56/100\n",
      "537/537 [==============================] - 0s 244us/step - loss: 0.4154 - acc: 0.8138\n",
      "Epoch 57/100\n",
      "537/537 [==============================] - 0s 287us/step - loss: 0.4146 - acc: 0.8156\n",
      "Epoch 58/100\n",
      "537/537 [==============================] - 0s 311us/step - loss: 0.4131 - acc: 0.8119\n",
      "Epoch 59/100\n",
      "537/537 [==============================] - 0s 430us/step - loss: 0.4128 - acc: 0.8101\n",
      "Epoch 60/100\n",
      "537/537 [==============================] - 0s 287us/step - loss: 0.4115 - acc: 0.8101 0s - loss: 0.4199 - acc: 0.80\n",
      "Epoch 61/100\n",
      "537/537 [==============================] - 0s 296us/step - loss: 0.4106 - acc: 0.8119\n",
      "Epoch 62/100\n",
      "537/537 [==============================] - 0s 380us/step - loss: 0.4099 - acc: 0.8156\n",
      "Epoch 63/100\n",
      "537/537 [==============================] - 0s 387us/step - loss: 0.4096 - acc: 0.8063\n",
      "Epoch 64/100\n",
      "537/537 [==============================] - 0s 289us/step - loss: 0.4084 - acc: 0.8101\n",
      "Epoch 65/100\n",
      "537/537 [==============================] - 0s 432us/step - loss: 0.4073 - acc: 0.8119\n",
      "Epoch 66/100\n",
      "537/537 [==============================] - 0s 309us/step - loss: 0.4071 - acc: 0.8119\n",
      "Epoch 67/100\n",
      "537/537 [==============================] - 0s 359us/step - loss: 0.4060 - acc: 0.8082\n",
      "Epoch 68/100\n",
      "537/537 [==============================] - 0s 337us/step - loss: 0.4052 - acc: 0.8119\n",
      "Epoch 69/100\n",
      "537/537 [==============================] - 0s 328us/step - loss: 0.4055 - acc: 0.8138\n",
      "Epoch 70/100\n",
      "537/537 [==============================] - 0s 328us/step - loss: 0.4041 - acc: 0.8119\n",
      "Epoch 71/100\n",
      "537/537 [==============================] - 0s 345us/step - loss: 0.4030 - acc: 0.8138\n",
      "Epoch 72/100\n",
      "537/537 [==============================] - 0s 412us/step - loss: 0.4026 - acc: 0.8138 0s - loss: 0.4022 - acc: 0.8\n",
      "Epoch 73/100\n",
      "537/537 [==============================] - 0s 291us/step - loss: 0.4021 - acc: 0.8063\n",
      "Epoch 74/100\n",
      "537/537 [==============================] - 0s 328us/step - loss: 0.4012 - acc: 0.8138\n",
      "Epoch 75/100\n",
      "537/537 [==============================] - 0s 259us/step - loss: 0.4004 - acc: 0.8156\n",
      "Epoch 76/100\n",
      "537/537 [==============================] - 0s 385us/step - loss: 0.4001 - acc: 0.8082\n",
      "Epoch 77/100\n",
      "537/537 [==============================] - 0s 304us/step - loss: 0.3991 - acc: 0.8101\n",
      "Epoch 78/100\n",
      "537/537 [==============================] - 0s 268us/step - loss: 0.3985 - acc: 0.8101\n",
      "Epoch 79/100\n",
      "537/537 [==============================] - 0s 287us/step - loss: 0.3980 - acc: 0.8156\n",
      "Epoch 80/100\n",
      "537/537 [==============================] - 0s 287us/step - loss: 0.3977 - acc: 0.8101\n",
      "Epoch 81/100\n",
      "537/537 [==============================] - 0s 305us/step - loss: 0.3966 - acc: 0.8119\n",
      "Epoch 82/100\n",
      "537/537 [==============================] - 0s 291us/step - loss: 0.3961 - acc: 0.8119\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 83/100\n",
      "537/537 [==============================] - 0s 341us/step - loss: 0.3958 - acc: 0.8101\n",
      "Epoch 84/100\n",
      "537/537 [==============================] - 0s 250us/step - loss: 0.3951 - acc: 0.8175\n",
      "Epoch 85/100\n",
      "537/537 [==============================] - 0s 248us/step - loss: 0.3942 - acc: 0.8119\n",
      "Epoch 86/100\n",
      "537/537 [==============================] - 0s 276us/step - loss: 0.3936 - acc: 0.8138\n",
      "Epoch 87/100\n",
      "537/537 [==============================] - 0s 266us/step - loss: 0.3933 - acc: 0.8175\n",
      "Epoch 88/100\n",
      "537/537 [==============================] - 0s 281us/step - loss: 0.3923 - acc: 0.8138\n",
      "Epoch 89/100\n",
      "537/537 [==============================] - 0s 344us/step - loss: 0.3919 - acc: 0.8138\n",
      "Epoch 90/100\n",
      "537/537 [==============================] - 0s 270us/step - loss: 0.3915 - acc: 0.8119\n",
      "Epoch 91/100\n",
      "537/537 [==============================] - 0s 244us/step - loss: 0.3910 - acc: 0.8119\n",
      "Epoch 92/100\n",
      "537/537 [==============================] - 0s 277us/step - loss: 0.3901 - acc: 0.8101\n",
      "Epoch 93/100\n",
      "537/537 [==============================] - 0s 335us/step - loss: 0.3903 - acc: 0.8101\n",
      "Epoch 94/100\n",
      "537/537 [==============================] - 0s 240us/step - loss: 0.3897 - acc: 0.8138\n",
      "Epoch 95/100\n",
      "537/537 [==============================] - 0s 250us/step - loss: 0.3893 - acc: 0.8119\n",
      "Epoch 96/100\n",
      "537/537 [==============================] - 0s 233us/step - loss: 0.3883 - acc: 0.8156\n",
      "Epoch 97/100\n",
      "537/537 [==============================] - 0s 274us/step - loss: 0.3879 - acc: 0.8101\n",
      "Epoch 98/100\n",
      "537/537 [==============================] - 0s 227us/step - loss: 0.3882 - acc: 0.8156\n",
      "Epoch 99/100\n",
      "537/537 [==============================] - 0s 246us/step - loss: 0.3876 - acc: 0.8156\n",
      "Epoch 100/100\n",
      "537/537 [==============================] - 0s 261us/step - loss: 0.3873 - acc: 0.8194\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x24fc6259160>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Import Libraries\n",
    "import pandas as pd                 # pandas is a dataframe library\n",
    "import matplotlib.pyplot as plt      # matplotlib.pyplot plots data\n",
    "\n",
    "#Read the data\n",
    "df = pd.read_csv(\"Data/Deep_Learning/pima-data.csv\")\n",
    "\n",
    "#Check the Correlation\n",
    "#df.corr()\n",
    "#Delete the correlated feature\n",
    "del df['skin']\n",
    "\n",
    "#Data Molding\n",
    "diabetes_map = {True : 1, False : 0}\n",
    "df['diabetes'] = df['diabetes'].map(diabetes_map)\n",
    "\n",
    "#Splitting the data\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#This will copy all columns from 0 to 7(8 - second place counts from 1)\n",
    "X = df.iloc[:, 0:8]\n",
    "y = df.iloc[:, 8]\n",
    "\n",
    "split_test_size = 0.30\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=split_test_size, random_state=42) \n",
    "\n",
    "#Imputing\n",
    "#from sklearn.impute import SimpleImputer \n",
    "from sklearn.preprocessing import Imputer\n",
    "\n",
    "#Impute with mean all 0 readings\n",
    "fill_0 = Imputer(missing_values=0, strategy=\"mean\")\n",
    "\n",
    "X_train = fill_0.fit_transform(X_train)\n",
    "X_test = fill_0.transform(X_test)\n",
    "\n",
    "# Feature Scaling\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "sc = StandardScaler()\n",
    "X_train = sc.fit_transform(X_train)\n",
    "X_test = sc.transform(X_test)\n",
    "\n",
    "\n",
    "# Importing the Keras libraries and packages\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.utils.vis_utils import plot_model\n",
    "from keras.regularizers import l2\n",
    "\n",
    "# Initialising the ANN\n",
    "classifier = Sequential()\n",
    "\n",
    "# Adding the input layer and the first hidden layer, kernel_regularizer parameter is optional\n",
    "classifier.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu', input_dim = 8, kernel_regularizer=l2(0.001)))\n",
    "\n",
    "# Adding the second hidden layer\n",
    "classifier.add(Dense(units = 5, kernel_initializer = 'uniform', activation = 'relu', kernel_regularizer=l2(0.001)))\n",
    "\n",
    "# Adding the output layer\n",
    "classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))\n",
    "\n",
    "# Compiling the ANN\n",
    "classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])\n",
    "\n",
    "# Fitting the ANN to the Training set\n",
    "classifier.fit(X_train, y_train, batch_size = 10, epochs = 100)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[117  34]\n",
      " [ 27  53]]\n"
     ]
    }
   ],
   "source": [
    "# Predicting the Test set results\n",
    "y_pred = classifier.predict(X_test)\n",
    "y_pred = (y_pred > 0.5)\n",
    "\n",
    "# Making the Confusion Matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(y_test, y_pred)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}