notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Networks notes for Self Driving Car\n",
    "\n",
    "## Neuron\n",
    "\n",
    "We started by a neuron, where the neuron has a list of _n_ inputs, each with it's weight, a bias and an output:\n",
    "\n",
    "$$ o = \\sum_{i=1}^n {w_i*x_i} + b $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Layer\n",
    "\n",
    "Then we moved to the *Layer* concept: a Layer is a level of _k_ neurons all with the same inputs, so\n",
    "_n_ inputs and _k_ outputs\n",
    "\n",
    "Inputs nodes are modeled as a _1_ by _n_ matrix\n",
    "\n",
    "$$ X = \\left[x_{1} \\ldots x_{n} \\right] $$\n",
    "\n",
    "Weights are a matrix: a row for for every input and columns for every output.\n",
    "\n",
    "$$ W = \\left[\\begin{matrix}\n",
    "    w_{1,1} & \\ldots & w_{1,k}\\\\\n",
    "    \\vdots & \\ddots & \\vdots \\\\\n",
    "    w_{n,1} & \\ldots & w_{n,k}\\\\\n",
    "\\end{matrix}\\right]$$\n",
    "\n",
    "the biases are again a vector, with one value per output:\n",
    "\n",
    "$$ b = \\left[ b_1 \\dots b_k \\right] $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batches\n",
    "\n",
    "But normally we deal with _batches_ of inputs, to improve parallelism.\n",
    "We deal with batches of size _m_ (trying to comfortably fit in memory)\n",
    "\n",
    "So _X_ with _m_ examples of _n_ inputs is\n",
    "$$ X = \\left[\\begin{matrix}\n",
    "    x_{1,1} & \\ldots & x_{1,n} \\\\\n",
    "    \\vdots & \\ddots & \\vdots \\\\\n",
    "    x_{m,1} & \\ldots & x_{m,n}\n",
    "\\end{matrix}\\right]$$\n",
    "\n",
    "_W_ and _b_ stay the same, they are applied to all the batch.\n",
    "\n",
    "The batches-layer equation is now:\n",
    "$$ Z = XW + b $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "57a94135-c8a0-41ea-b18c-76a018d3a5a3"
    }
   },
   "source": [
    "### Sigmoid\n",
    "\n",
    "Sigmoid is a function to make discontinuity derivable:\n",
    "$$\\sigma(x) = \\frac{1}{1+e^{-x}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "d9a52550-fc2a-49a1-9270-360e9da8c069"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x7f45f280f9b0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgkAAAFyCAYAAAB/b0lnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XucXdP9//HXR8S1FdpoQt2pW/2opCVUUVTc7y2hKKW0\n6tum396rpXxbpV+iWkSrxK1DiDIRX1F3JXFJSrV1KYIqQRKCJMhl/f5YZ2oyzkzmnLnsM+e8no/H\nfszMPnvv85mczJz3rLX2WpFSQpIkqa2lii5AkiTVJkOCJEkqy5AgSZLKMiRIkqSyDAmSJKksQ4Ik\nSSrLkCBJksoyJEiSpLIMCZIkqSxDgtSLImKHiFgUEdsXXUtHIuLOiLijE8d1+vuJiE9GxL0R8VZE\nLIyIzbun2u4VEc9GxMVF1yHVAkOC1EpEfK30pjepwvMiIo6IiMkRMTMi3oiIJyLi0ojYus3hfWEu\n9AQsquDYDkXE0sC1wCrAN4HDgeeqrq6LImKbiDg5IlYq8/Ai+sZrJPW4pYsuQKoxhwLTgK0iYr2U\n0jOdPO/XwNeA64ErgAXARsDuwNPA/QAppbsiYvmU0rvdXnn3+lw3X299YC3gyymlS7r52tXYFvgJ\ncAnwRpvHNqLzAUmqa4YEqSQi1iW/eewP/BY4DDitE+d9BPgqcGFK6attHh4ZEQNb7+gDAYGU0oJu\nvuSg0sfZ3XzdakV7D6SU5vdmIVIts7tBes9hwCxgArlp/LBOnrcu+U3nvnIPppRmtHzeXh9+RJwQ\nEU9HxNxSl8V2pXEBt5c59/OlpvIXSt0a10TEByNimYg4JyJejog3I+LiiOjf5nn6RcSPI+KpiHg7\nIqZFxM8iYpk2xy323KV9H42I60tjCl6OiLOBZengDbd03iXAneQm/GtL38Pt7T1Paf+YiJjW6uu1\nS+d9KyKObVX/AxHxyTLnbxQRYyPildK/6eMR8T+lx04Gziwd+mzpugsjYq3S4+8bkxAR65b+nWdG\nxJyImBQRe7Q5pvXr86OI+FdEzIuIWyNi/Y7+jaRaZUuC9J5DgXEppQUR0QQcHxFDU0pTlnBeS9/6\n5yPi2pTSvCUcv1h/d0R8ldxdcRdwNrAOudviNeBfZc7/ATAXOB3YADgRmE9uIl8ZOBkYBhwJPAP8\nT6tzfw8cAYwF/hfYunS9jYEDO6hxOeB2YA3gV8BL5HEFO7U9tozRwAvAj0rnPgi8XO552jx/uccO\nAz5QumYCvgeMK3UNLSzVujlwD/AOcCH59Vkf2As4CbgO2BA4BPgGMLN07Vfb+d4/AkwClivVP4v8\nb9scEQemlG5oU+P3gYXAL4EBpRqvALZp53uValdKyc2t4TdgKPlN9rOt9j0PnN3J88eQ3xhmAuOA\nbwEblTluh9Jx25e+7k9+c5oELNXquMNL9dze5txFwCNAv1b7ryxd88Y2z3Uv8EyrrzcvnT+6zXFn\nls7fodW+O9o89zdKxxzQat9ywJOtv58O/n1aaj+gzf7FnqfV/kva1L526fxXgJVa7d+79Px7tNp3\nF/A68NEO6vnv0nlrlXlsGnBxq69HlY7dptW+FcljTZ4u8z3+rc3rc2Lp/E2L/n/u5lbpZneDlB0G\nTCc3i7e4GjgkIjpsTgdIKX0J+Dr5L/f9yH9FPlZqal69g1M/CXwY+F1KqfVguT+QWxLKuTSV/mou\nub/0se1te/cDa0ZEy8/5HuS/kke1Oe4scpfBnh3UuTvwUkrpupYdKaW3yWM3etNVKaXWAw3vIde+\nHkBp/MdngN+nlP7dTc+5O/BASuk/d7yklOaQv/d1ImLTNsdf3Ob1WaxGqS8xJKjhld5EDyb/Vbte\nRKxf6kN+ABgM7NyZ66SULkgpfQoYCOwL3ERujm/q4LS1yW/cT7e51kLg2XbOadsFMbuD/UuRm7xb\nnmsR8FSb53qZ/Jf32kuo86ky+5/o4JyesNj3mFJ6vfTpKqWPLW/Ef+/G51yb8t/nY60eb63t69AS\n9lZB6mMckyDlN/LVyH3UI9o8lsitDLd29mIppdeAG4EbI09ItH1ErJlSKje+oBoLK9zftiWkluYA\naK+Wfu3s7+z3WKS+UKPUKbYkSPBF8kC6g8psVwH7R8SyVV77odLH1dp5/Dnym8cGrXdGRD/yAMbu\n9Bz5Z/5jbZ7rI+QBjx1NbtQy+K+tjbtY02ul526ro1aNjrTMa7HZEo6rJCg9R547oa1NWj0u1SVD\nghpaadT+/sD4lNIfU0rXtd6A3wArAft0cI1BEbFJmf39gV0o08TfykPkwY7Htho7ADm4dHfz9E3k\nQPLNNvv/m/ymOWEJ564eEf+5AyIiVgCO7WJNTwMbR8SHW113C+DT1Vws5dtN7waOjog1Ozh0Tulj\nuYDS1k3kybX+M3NmRKwIfAWYllL6RzW1Sn2B3Q1qdPsCHwSa23l8Mvnug8OAa9o5Zg3ggdL9/reR\nB0B+hNx1sTkwKqU0q9Xx/2l2TinNj4hTgHOBOyJiLLkF4ShysOjsX7ydGVz514i4FPhKRKxCvgtg\na/ItkdellO7q4PTfkQdmXl6al6DlFsg5HZzTGReT7wS5JSJ+T5506TjyHQLlpkzujP8iDxacGhG/\nJd+tsC75DogtS8dMIf+b/TwiriLfQtqcyt+++gvya3lzRJxLvgXyS+TWjgOqrFHqEwwJanSHkucc\nKDvmIKWUImICcGhErFIab9DWE+RbBPcgz7w4CHib/EZ3THr/NMSLvfGnlM4r3UDx3+S7Ih4lt1z8\nqnSdds/txP62vkz+6/1L5LswpgM/A07t6JoppXkRsRN5Poevk//NrgBuLm2d8b4aU0qPR8Thpec/\nC/gHuRXlMKDtolHtzZ2w2P5SGBpGni3zePKtms+R71ZpOeahiDip9PhwcqvquuTbXtte75WI2AY4\no/S9Lwf8FdgrpdT2e+/q6yPVlEjJ/7tSrSnddvkqeXKn44quR1JjqnhMQkR8JiKaI+LfpSlI2+2r\nbXXOjhExpTSN6pMRcWR15Ur1p51BkUcCHyLflilJhahm4OKKwMPkFe86s0TsOuTbwW4DtiA3oV4U\nEd29ypzUVw2LiKkR8YOI+EpEXEgeA/BX8hoSklSILnU3RMQiYL+UUnuDvoiIM4DdU0qbt9rXBAxI\nKe3R3nlSo4iItcnheSty60HLIlM/SK0Wh5Kk3tYbAxeH8f5BYRN5/9SwUkNKKT1HHkQoSTWlN0LC\nYN5b8a3Fy8BKEbFsSumdtieU7pkeTp6Wtu3obkmS1L7lyLdST0wpzVzCsR2q1Vsgh5NXtpMkSdU5\njLxYXNV6IyRMJ9833tog4I1yrQglzwJcccUVbLLJ+yayUx80cuRIRo2yh6le+HrWh5TgjTfgu98d\nybHHjmLWLJg5E2bPhjffbH+bO7dz1192WVhuubz177/4tswysPTS79+/9NL5sZbPl1oK+vVb/GO5\nfe091vrziMU36J59Sy1V+bnltH6svc/Lfd12/zPPPMZ3vvNFaH+RuE7rjZAwibzUamu7lva3522A\nTTbZhCFDhvRUXepFAwYM8LWsI76etS8lmDEDnnsub88//97n//oXTJ8Or7wC8+cDDOChh/Lrueyy\nMHAgrLxy3lZZBdZa672vW/attBKsuCKssML7P66wAiy/fH7zVO+bOvU/n3a5u77ikFCas3wD3psG\ndr3SXOuzUkr/iojTgdVTSi1zIYwGTijd5XAxedndg8iz00mSumD+fHjySXj88cW3J57If/W3WH55\nWHvtvA0ZAoMHw6BBeRs1Ci65JH++0kod/7WrxlJNS8InyRO8tExdelZp/6XA0eSBiv9ZWCWl9GxE\n7Em+m+G/gBeAL6eUOr30riQJ3n0X/vY3mDIl/7U4dSo88gi8U+q4XWUV2GQT2Hxz+MIXYP313wsG\nH/5w+2/+l10GG27Ye9+H+o6KQ0JpEZh2G5FSSkeV2Xc3MLTS55KkRjZnDkyaBHffnbfJk3MgWGop\n2HTT3CJw6KGw5ZY5HAwcaCuAulet3t2gOjNixIiiS1A38vXsGSnBP/4BEybAjTfmgLBgQW4F2H57\n+MUvYNiw3FKwwgrd97y+nmpPTS7wFBFDgClTpkxxcJSkurZoEfz5z3DNNTkYPPtsHj+wyy6w226w\nww65lcBBgOqsqVOnMnToUIChKaWpSzq+I7YkSFIBHn4YrrwSrroKXngB1lwT9tkH9twTdtwxBwWp\naIYESeolc+fmUDB6NDz4YB5D8IUvwIgRsO22thao9hgSJKmHvfACnH02XHxxnrxo+HD44x9zq0H/\n/kVXJ7XPkCBJPeTJJ+GMM+Dyy/NEQ8cfD8cdB+uuW3RlUucYEiSpmz3/PJx0ElxxRZ6g6Gc/y+Fg\npZWKrkyqjCFBkrrJ66/D6afDr36Vpy/+zW/g6KPz2gVSX2RIkKQuSilPa/yd7+TJjr7/ffj2t+ED\nHyi6MqlrDAmS1AVPPpm7Eu68Ew4/PI9BWG21oquSuoc33EhSFRYuzF0Lm2+exyD86U95DQQDguqJ\nLQmSVKHnnoMvfhHuuy93MfzkJ907TbJUKwwJklSBa6+FY46BAQPgrrtgu+2KrkjqOXY3SFInLFwI\n3/sefP7zsOuueYlmA4LqnS0JkrQEs2bBIYfAbbfBWWfByJEuyazGYEiQpA5Mm5ZXY5w5Mw9O3Gmn\noiuSeo8hQZLaMWVKXl/hgx+EyZNhgw2KrkjqXY5JkKQybr8ddtgB1l4738VgQFAjMiRIUhu33JJb\nELbbLoeFVVctuiKpGIYESWrlpptgn31g553h+uvz6o1SozIkSFLJrbfC/vvngYrXXefCTJIhQZKA\n+++H/fbLLQhjx8IyyxRdkVQ8Q4Kkhvf3v8Mee8AnPpFnVDQgSJkhQVJDe+ml3L3w0Y/C+PGuwSC1\nZkiQ1LDmzctdDIsWwf/9H6yyStEVSbXFyZQkNaSU4Kij4NFH4Z57ckuCpMUZEiQ1pJ/9DK6+Gq65\nBoYOLboaqTbZ3SCp4fzpT/CTn8DJJ8NBBxVdjVS7DAmSGsq//w2HHQaf+1wOCpLaZ0iQ1DDmz4eD\nD863OF5xBSzlb0CpQ45JkNQwTjklT5p0112uxyB1hiFBUkO49174xS/gtNNg222LrkbqG2xsk1T3\n3nwTjjgChg2D732v6GqkvsOWBEl171vfgldeyXc19OtXdDVS32FIkFTXbroJLroob+utV3Q1Ut9i\nd4OkuvXWW/DVr8Kuu8LRRxddjdT3GBIk1a2TToIZM2D0aIgouhqp77G7QVJdeuABOPdc+OUvYd11\ni65G6ptsSZBUdxYsgGOPhS23hG98o+hqpL7LlgRJdeeCC/Lqjg8+CEv7W06qmi0JkurKq6/mNRmO\nOcbVHaWuMiRIqis//jGklJeCltQ1NsRJqht/+Qv89rcwapRrM0jdwZYESXUhJfjmN2GTTeBrXyu6\nGqk+2JIgqS5MmAB33w3/93/Qv3/R1Uj1wZYESX3ewoXw/e/DZz8Lw4cXXY1UP2xJkNTnXX45/P3v\ncPHFzqwodSdbEiT1aW+/nW95POgg2GqroquR6oshQVKfdv758OKL3vIo9QRDgqQ+a84c+MUv4Kij\nYMMNi65Gqj+GBEl91gUXwGuvwY9+VHQlUn2qKiRExAkRMS0i5kXE5Ij41BKOPywiHo6IORHxYkT8\nPiI+VF3JkgRz5+YVHr/0JVhnnaKrkepTxSEhIg4GzgJOBrYEHgEmRsTAdo7/NHAp8DtgU+AgYCvg\nt1XWLEmMHg2zZsEPf1h0JVL9qqYlYSRwYUrpspTS48DxwFzg6HaOHwZMSymdl1J6LqV0H3AhOShI\nUsXmzoUzz4Qjj4R11y26Gql+VRQSIqI/MBS4rWVfSikBtwLbtHPaJGDNiNi9dI1BwOeBCdUULEm/\n+x3MmGErgtTTKm1JGAj0A15us/9lYHC5E0otB18Ero6Id4GXgNeAr1f43JLE/Plw9tlw6KGw3npF\nVyPVtx6fcTEiNgV+BZwC3AKsBvwvucvhmI7OHTlyJAMGDFhs34gRIxgxYkSP1Cqp9l19NTz/PHzn\nO0VXIhWvqamJpqamxfbNnj27264fubegkwfn7oa5wIEppeZW+8cAA1JK+5c55zJguZTSF1rt+zRw\nD7BaSqltqwQRMQSYMmXKFIYMGVLBtyOpnqUEW2wBa6wBN91UdDVSbZo6dSpDhw4FGJpSmtqVa1XU\n3ZBSmg9MAXZu2RcRUfr6vnZOWwFY0GbfIiABzrIuqdNuvhkefRS+972iK5EaQzV3N5wNHBsRR0TE\nxsBochAYAxARp0fEpa2OHw8cGBHHR8S6pVaEXwH3p5Smd618SY3kzDPz+gzbb190JVJjqHhMQkpp\nbGlOhFOBQcDDwPCU0qulQwYDa7Y6/tKI+ABwAnkswuvkuyO+38XaJTWQhx6CO++Ea65xpUept1Q1\ncDGldD5wfjuPHVVm33nAedU8lyQB/OpXeU6E/d838klST3HtBkk1b/r0fFfD178O/foVXY3UOAwJ\nkmre6NGwzDJwdHvzukrqEYYESTXtnXdySDjySFh55aKrkRqLIUFSTRs7Fl5+GU48sehKpMZjSJBU\ns1KCc8+F4cNh442LrkZqPD0+LbMkVeuhh/I2fnzRlUiNyZYESTXrwgthrbVg992LrkRqTIYESTXp\njTegqQmOOcbbHqWiGBIk1aQrr8x3Nnjbo1QcQ4KkmpNS7mrYay/46EeLrkZqXIYESTXnwQfhkUfg\nuOOKrkRqbIYESTXnwgth7bVh112LrkRqbIYESTVl9my46ioHLEq1wJAgqaY4YFGqHYYESTXlootg\n771h9dWLrkSSIUFSzfjrX+Evf4Gjjiq6EklgSJBUQy69FFZd1RkWpVphSJBUExYsyOMRDjsM+vcv\nuhpJYEiQVCMmTsxLQh95ZNGVSGphSJBUE8aMgc03h098ouhKJLUwJEgq3KxZ0NxsK4JUawwJkgp3\n9dWwcGEejyCpdhgSJBXu0kvzHQ2DBhVdiaTWli66AEmN7Ykn4P774Zpriq5EUlu2JEgq1GWXwcor\n51kWJdUWQ4KkwqSUF3P6/Odh2WWLrkZSW4YESYV54AF45hkYMaLoSiSVY0iQVJg//AFWWw22377o\nSiSVY0iQVIiFC2HsWDj4YOjXr+hqJJVjSJBUiDvvhOnT7WqQapkhQVIhmppg/fXhU58quhJJ7TEk\nSOp177wD48bBIYdARNHVSGqPIUFSr5s4EV5/HQ49tOhKJHXEkCCp1zU15RUfN9206EokdcSQIKlX\nzZmTV3x0wKJU+wwJknrVjTfC3Ln51kdJtc2QIKlXXXstfPKTsO66RVciaUkMCZJ6zZw5cNNNcNBB\nRVciqTMMCZJ6zc03566GAw8suhJJnWFIkNRrrr0WPvEJ2GCDoiuR1BmGBEm9Yt68PGjRrgap7zAk\nSOoVt9wCb71lSJD6EkOCpF5x7bWw2Waw0UZFVyKpswwJknrcO+/kCZRsRZD6FkOCpB53663wxhuG\nBKmvMSRI6nHXXgsbb+xaDVJfY0iQ1KPefReuvz63IrgstNS3GBIk9ag77sjLQtvVIPU9hgRJPera\na/PkSZtvXnQlkiplSJDUYxYuzF0NBx5oV4PUFxkSJPWYyZNhxgzYd9+iK5FUjapCQkScEBHTImJe\nREyOiE8t4fhlIuJnEfFsRLwdEc9ExJeqqlhSn9HcDB/5CGy1VdGVSKrG0pWeEBEHA2cBXwEeAEYC\nEyNiw5TSjHZOuwZYFTgKeBpYDVsxpLrX3Ax77QX9+hVdiaRqVPNGPRK4MKV0WUrpceB4YC5wdLmD\nI2I34DPAHimlO1JKz6eU7k8pTaq6akk178kn4fHHYZ99iq5EUrUqCgkR0R8YCtzWsi+llIBbgW3a\nOW1v4CHgexHxQkQ8ERG/jIjlqqxZUh8wfjwstxzsskvRlUiqVqXdDQOBfsDLbfa/DLS3bMt65JaE\nt4H9Ste4APgQ8OUKn19SH9HcnAPCiisWXYmkalU8JqEKSwGLgENTSm8BRMS3gGsi4msppXfaO3Hk\nyJEMGDBgsX0jRoxgxIgRPVmvpC6aORP+/GcYPbroSqT61tTURFNT02L7Zs+e3W3XrzQkzAAWAoPa\n7B8ETG/nnJeAf7cEhJLHgADWIA9kLGvUqFEMGTKkwhIlFe2mm2DRojxoUVLPKfeH89SpUxk6dGi3\nXL+iMQkppfnAFGDnln0REaWv72vntHuB1SNihVb7NiK3LrxQUbWS+oTm5nzb42qrFV2JpK6o5u6G\ns4FjI+KIiNgYGA2sAIwBiIjTI+LSVsf/AZgJXBIRm0TE9sCZwO876mqQ1De98w7cfLN3NUj1oOIx\nCSmlsRExEDiV3M3wMDA8pfRq6ZDBwJqtjp8TEZ8Dfg08SA4MVwM/7mLtkmrQnXfCW28ZEqR6UNXA\nxZTS+cD57Tx2VJl9TwLDq3kuSX1LczOssw5stlnRlUjqKmc9lNRtUsohYZ99XNBJqgeGBEnd5uGH\n4YUX7GqQ6oUhQVK3aW6GAQNg++2LrkRSdzAkSOo2zc2w++7Qv3/RlUjqDoYESd3ihRdg6lS7GqR6\nYkiQ1C3Gj4ell4bddiu6EkndxZAgqVs0N+exCKusUnQlkrqLIUFSl735Jtx+u10NUr0xJEjqsltu\ngXffhb33LroSSd3JkCCpy5qb8wyL661XdCWSupMhQVKXLFgAEybY1SDVI0OCpC6ZNAlmzoR99y26\nEkndzZAgqUuam2HwYPjkJ4uuRFJ3MyRI6pLm5jxgcSl/m0h1xx9rSVV74gl48knHI0j1ypAgqWrN\nzbD88rDzzkVXIqknGBIkVa25GXbdNQcFSfXHkCCpKq++CvfdZ1eDVM8MCZKqctNNkBLsuWfRlUjq\nKYYESVVpboZhw2DQoKIrkdRTDAmSKvb22zBxol0NUr0zJEiq2B13wJw5hgSp3hkSJFWsuRnWXx82\n2aToSiT1JEOCpIqklEPCPvtARNHVSOpJhgRJFZk6FV580a4GqREYEiRVpLkZVlkFPv3poiuR1NMM\nCZIq0twMe+wB/fsXXYmknmZIkNRpzz8PDz9sV4PUKAwJkjpt/PjcgjB8eNGVSOoNhgRJndbcDDvu\nCAMGFF2JpN5gSJDUKW+8kSdRsqtBahyGBEmdMnEizJ8Pe+9ddCWSeoshQVKnNDfDFlvA2msXXYmk\n3mJIkLRECxbAhAl2NUiNxpAgaYnuvRdee82QIDUaQ4KkJWpuhtVXhyFDiq5EUm8yJEjqUEpwww15\nwOJS/saQGoo/8pI69Pjj8PTTdjVIjciQIKlDzc2wwgqw005FVyKptxkSJHWouTlPw7zcckVXIqm3\nGRIkteuVV2DSJLsapEZlSJDUrgkT8sc99yy2DknFMCRIatcNN8A228CqqxZdiaQiGBIklTV3Ltxy\nC+y7b9GVSCqKIUFSWbfeCvPmGRKkRmZIkFTW9dfDxhvDRhsVXYmkohgSJL3PwoVw4422IkiNzpAg\n6X0mTYJXXzUkSI3OkCDpfW64AQYNgq23LroSSUUyJEhajAs6SWrhrwBJi3n8cfjnP+1qkFRlSIiI\nEyJiWkTMi4jJEfGpTp736YiYHxFTq3leST3v+uvzgk4771x0JZKKVnFIiIiDgbOAk4EtgUeAiREx\ncAnnDQAuBW6tok5JveSGG2C33WD55YuuRFLRqmlJGAlcmFK6LKX0OHA8MBc4egnnjQauBCZX8ZyS\nesFLL8H999vVICmrKCRERH9gKHBby76UUiK3DmzTwXlHAesCP62uTEm9Yfx46NfPBZ0kZUtXePxA\noB/wcpv9LwNl52WLiI8BPwe2SyktioiKi5TUO264AbbbDj784aIrkVQLKg0JFYmIpchdDCenlJ5u\n2d3Z80eOHMmAAQMW2zdixAhGjBjRfUVKAuCtt+C22+D004uuRFJnNTU10dTUtNi+2bNnd9v1I/cW\ndPLg3N0wFzgwpdTcav8YYEBKaf82xw8AXgMW8F44WKr0+QJg15TSnWWeZwgwZcqUKQwZMqSS70dS\nlcaNg4MOgqefhvXWK7oaSdWaOnUqQ4cOBRiaUurS3YQVjUlIKc0HpgD/uTkqcv/BzsB9ZU55A9gM\n+ASwRWkbDTxe+vz+qqqW1O2uvx4228yAIOk91XQ3nA2MiYgpwAPkux1WAMYARMTpwOoppSNLgxr/\n0frkiHgFeDul9FhXCpfUfebPhwkT4IQTiq5EUi2pOCSklMaW5kQ4FRgEPAwMTym9WjpkMLBm95Uo\nqafdcw+89pq3PkpaXFUDF1NK5wPnt/PYUUs496d4K6RUU8aNg7XWgtyNKUmZazdIDW7RIvjjH+GA\nA8A7lCW1ZkiQGtykSXmmxQMPLLoSSbXGkCA1uHHjYPBg2HbboiuRVGsMCVIDSwmuuw723x+W8reB\npDb8tSA1sClT4Lnn7GqQVJ4hQWpg48bldRp22KHoSiTVIkOC1KBSyiFh331h6R5dxUVSX2VIkBrU\n3/4G//ynXQ2S2mdIkBrUuHGw0kqw885LPlZSYzIkSA1q3DjYe29YdtmiK5FUqwwJUgN68snc3WBX\ng6SOGBKkBjRuHKywAgwfXnQlkmqZIUFqQOPGwR575KAgSe0xJEgN5qmn8iRKBx1UdCWSap0hQWow\nV18NK64Ie+1VdCWSap0hQWowV10F++yTg4IkdcSQIDWQv/8939VwyCFFVyKpLzAkSA3k6qthwADv\napDUOYYEqUGklLsaDjjACZQkdY4hQWoQf/lLXqvBrgZJnWVIkBrEVVfBwIGw005FVyKprzAkSA1g\n0aI8HuHzn3dZaEmdZ0iQGsDkyfD883Y1SKqMIUFqAFddBauvDtttV3QlkvoSQ4JU5+bPh6YmGDEC\nlvInXlIF/JUh1bmbb4YZM+CII4quRFJfY0iQ6tzll8Pmm+dNkiphSJDq2OuvQ3OzrQiSqmNIkOrY\nNdfkMQmHHlp0JZL6IkOCVMcuuww+9zlYbbWiK5HUFzmtilSnnnkG/vxnuOKKoiuR1FfZkiDVqSuu\ngA98APbbr+hKJPVVhgSpDqWU72o48EBYccWiq5HUVxkSpDo0eTI89RQcfnjRlUjqywwJUh36/e9h\nrbVgxx1gADlfAAAPVklEQVSLrkRSX2ZIkOrMm2/mtRqOPhr69Su6Gkl9mSFBqjNXXQXz5uWQIEld\nYUiQ6szvfge77QZrrll0JZL6OudJkOrII4/Agw/CddcVXYmkemBLglRHLroIBg2CvfYquhJJ9cCQ\nINWJefPyBEpf+hL07190NZLqgSFBqhPjxuVVH485puhKJNULQ4JUJy66CD77Wdhgg6IrkVQvHLgo\n1YHHH4e77oIrryy6Ekn1xJYEqQ6cdx585CN5rQZJ6i6GBKmPe+MNGDMGvvIVWHbZoquRVE8MCVIf\nd/nl+c6G444ruhJJ9caQIPVhKcFvfgP77w9rrFF0NZLqjQMXpT7s9tvzoMXRo4uuRFI9siVB6sN+\n8xvYbDPYfvuiK5FUj2xJkPqop56CG27IrQgRRVcjqR5V1ZIQESdExLSImBcRkyPiUx0cu39E3BIR\nr0TE7Ii4LyJ2rb5kSQDnnAMDB8LhhxddiaR6VXFIiIiDgbOAk4EtgUeAiRExsJ1TtgduAXYHhgB3\nAOMjYouqKpbEzJlw8cVwwgmw/PJFVyOpXlXTkjASuDCldFlK6XHgeGAucHS5g1NKI1NK/5tSmpJS\nejql9CPgn8DeVVctNbjRo/OdDV/7WtGVSKpnFYWEiOgPDAVua9mXUkrArcA2nbxGAB8EZlXy3JKy\nt9+GX/86r/a46qpFVyOpnlXakjAQ6Ae83Gb/y8DgTl7jO8CKwNgKn1sSeX2GV16BkSOLrkRSvevV\nuxsi4lDgx8A+KaUZSzp+5MiRDBgwYLF9I0aMYMSIET1UoVTbFi6EM8+EffaBDTcsuhpJRWtqaqKp\nqWmxfbNnz+6260fuLejkwbm7YS5wYEqpudX+McCAlNL+HZx7CHARcFBK6eYlPM8QYMqUKVMYMmRI\np+uT6t3VV8Mhh8ADD8Cn2r2nSFIjmzp1KkOHDgUYmlKa2pVrVdTdkFKaD0wBdm7ZVxpjsDNwX3vn\nRcQI4PfAIUsKCJLKW7QI/ud/YPhwA4Kk3lFNd8PZwJiImAI8QL7bYQVgDEBEnA6snlI6svT1oaXH\n/gt4MCIGla4zL6X0RpeqlxrI+PHwt7/BBRcUXYmkRlFxSEgpjS3NiXAqMAh4GBieUnq1dMhgYM1W\npxxLHux4XmlrcSnt3DYpaXEp5VaEHXaA7bYruhpJjaKqgYsppfOB89t57Kg2X3+2mueQ9J6JE+Gh\nh+DWW4uuRFIjcYEnqcalBD/+MWyzDey0U9HVSGokLvAk1bjrr8+tCHfc4UJOknqXLQlSDVu4EE46\nCT73Odhxx6KrkdRobEmQatiVV8I//gFjxhRdiaRGZEuCVKPefRdOPhn23995ESQVw5YEqUZdcAE8\n/zzceGPRlUhqVLYkSDVo5kw45RQ45hj4+MeLrkZSozIkSDXopz/NgxZPPbXoSiQ1MrsbpBrz2GNw\n/vnw85/DoEFLPl6SeootCVKN+fa3Ya214BvfKLoSSY3OlgSphjQ3w003wbXXwrLLFl2NpEZnS4JU\nI+bMgRNPhN12gwMOKLoaSbIlQaoZp54Kr7wCt9/u9MuSaoMtCVINePRROPvsPAXz+usXXY0kZYYE\nqWALF8Lxx8MGG+RBi5JUK+xukAp2zjkwaRLcdZeDFSXVFlsSpAI99hj86EfwzW/CZz5TdDWStDhD\nglSQBQvgyCNh7bXhZz8ruhpJej+7G6SCnHEGTJkC994Lyy9fdDWS9H62JEgFuPfevAz0978Pw4YV\nXY0klWdIkHrZrFkwYgRsvXVeyEmSapUhQepFKcGXvwxvvQVNTbC0HX6Sapi/oqRedM45cP318Mc/\n5kWcJKmW2ZIg9ZJbb82TJf33f8N++xVdjSQtmSFB6gVPPw1f+ALssku+q0GS+gJDgtTD3nwztxx8\n+MNw1VXQr1/RFUlS5zgmQepB774LBx4Izz+fp15eZZWiK5KkzjMkSD2k5U6Gu+6Cm2+GTTctuiJJ\nqowhQeohP/gBXHFF7mL47GeLrkaSKueYBKkHnHZaHqA4ahQcfHDR1UhSdQwJUjc7/XT4yU/yok3f\n/GbR1UhS9QwJUjc64wz44Q/zdMs//GHR1UhS1zgmQeoGKeXFms48My/c9JOfFF2RJHWdIUHqogUL\n4Ljj4OKL87TL3/hG0RVJUvcwJEhd8MYbcOihMHFivpPhsMOKrkiSuo8hQarSU0/BPvvAiy/ChAmw\n665FVyRJ3cuBi1IVbrkFtt4aFi6E++83IEiqT4YEqQILFuS7FoYPzyFh8mTYaKOiq5KknmF3g9RJ\n06bB4YfnYHDGGXnZ56WM2ZLqmL/ipCVYtAjOOw/+3/+DF16Au++G737XgCCp/vlrTurAY4/BLrvA\n17+eWxEefRS23bboqiSpdxgSpDJmz4ZvfQs23xz+9S/405/gggvggx8sujJJ6j2OSZBamT8fxoyB\nk06COXPyQk0jR8KyyxZdmST1PlsSJPJdC5ddBptsAl/5Cuy8Mzz+eJ5q2YAgqVEZEtTQ3n47T6f8\n8Y/DkUfm7oVHHoE//AHWWKPo6iSpWHY3qCG99FIeYzB6NMyYAXvvDU1NMGRI0ZVJUu0wJKhhLFgA\nN98Ml1wC48fDMsvA0UfDiSfCxz5WdHWSVHsMCaprixbBpElw3XVw5ZXw8suwxRbwy1/m7oWVVy66\nQkmqXYYE1Z1334U77oA//hFuuAGmT4dBg+Dgg+Goo+ATnyi6QknqGwwJ6vMWLYKHH4bbbsvbPffA\n3Lmw7rp5GecDDoBhw6Bfv6IrlaS+xbsb1Cuampq67VozZ+axBT/9Key5J6y6KgwdCiefDCnBKafk\n0PD003DWWfDpTxsQult3vp4qnq+n2lNVSIiIEyJiWkTMi4jJEfGpJRy/Y0RMiYi3I+LJiDiyunLV\nV1XzS2jhQnjyydxtcNppcMgheYDhwIGw++5w7rm5FeHEE+HOO+G112DiRPjOd/K4g4ju/z6U+aZS\nX3w91Z6Kuxsi4mDgLOArwAPASGBiRGyYUppR5vh1gBuB84FDgV2AiyLixZTSn6ovXfXgnXfguefy\nCovTpsEzz+SP//xnnszonXfycR/6UJ7LYI898hLNW20F669vEJCknlTNmISRwIUppcsAIuJ4YE/g\naODMMsd/FXgmpfTd0tdPRMR2pesYEurQokV57YNZs3LXwEsvwbPP5u6Al156/5ZSPq9fP1h77TyW\nYOut8+2JH/943gYNMhBIUm+rKCRERH9gKPDzln0ppRQRtwLbtHPaMODWNvsmAqMqeW71nJTymgXv\nvJMH/M2ZA2+9lT+2bK2/bvl89uzcxD9r1uIfX3/9vTf+1mbOhNVWy9uQIfnjWmvBeuvlYLDGGrC0\nQ2klqWZU+it5INAPeLnN/peBjdo5Z3A7x68UEcumlN4pc85yAGPHPsakSYu/4aT03tetP2/5utwx\nlTze3rW7+7kXLcp97q0/tt7KPdZ2X3uPLViQt3ffzW/+LVt7Xy9YUOYVaMfyy7+3rbgiDBgAK60E\na64Jm2763tct24ABeQzBaafN5pxzprZ73Vmz8qa+Yfbs2Uyd2v7rqb7F17O+PPbYYy2fLtfVa9Xq\n323rAJxxxhcLLkNtzZuXt2oMHTq0e4tRoXw964uvZ11aB7ivKxeoNCTMABYCg9rsHwRMb+ec6e0c\n/0Y7rQiQuyMOA54F3q6wRkmSGtly5IAwsasXqigkpJTmR8QUYGegGSAiovT1ue2cNgnYvc2+XUv7\n23uemcAfKqlNkiT9R5daEFpUM0/C2cCxEXFERGwMjAZWAMYARMTpEXFpq+NHA+tFxBkRsVFEfA04\nqHQdSZJUoyoek5BSGhsRA4FTyd0GDwPDU0qvlg4ZDKzZ6vhnI2JP8t0M/wW8AHw5pdT2jgdJklRD\nIpW7V02SJDU8126QJEllGRIkSVJZNRUSIuKHEXFvRMyJiLJT60TEmhExoXTM9Ig4MyJq6vtQ+yLi\n2YhY1GpbGBHfXfKZqhWVLvCm2hQRJ7f5WVwUEf8oui51TkR8JiKaI+LfpddunzLHnBoRL0bE3Ij4\nU0RsUOnz1Nqba39gLHBBuQdLYeAm8oDLYcCRwJfIgyjVNyTgJPKg18HAasCvC61IndZqgbeTgS2B\nR8gLvA0stDBV62+897M4GNiu2HJUgRXJNw58jfx7dTER8T3g6+TFGLcC5pB/Vpep5ElqcuBiaSnp\nUSmlD7XZvzt5fobVWlacjIjjgF8Aq6aUKphgWEWIiGnk17a9eTVUwyJiMnB/Sukbpa8D+Bdwbkqp\n3AJvqlERcTKwb0ppSNG1qGsiYhGwX0qpudW+F4FfppRGlb5eibwkwpEppbGdvXattSQsyTDg0TZL\nUk8EBgAfL6YkVeH7ETEjIqZGxLcjol/RBWnJWi3wdlvLvpT/yuhogTfVto+VmqufjogrImLNJZ+i\nWhcR65Jbhlr/rL4B3E+FP6u1unZDe9pbLKrlsUd6txxV4VfAVGAWsC25FWgw8O0ii1KnVLPAm2rX\nZHJ37RPkbr9TgLsjYrOU0pwC61LXDSZ3QZT7WR1cyYV6vCWhNANj28ExbQeubdjTdajnVPIap5TO\nSSndnVL6W0rpt8C3gBNLf6VK6iUppYkppXGln8U/AXsAqwBfKLg01ZDeaEn4X+CSJRzzTCevNR1o\nO5J6UKvHVIyuvMYPkP8frgP8sxtrUverZoE39REppdkR8SRQ8Qh41ZzpQJB/Nlu3JgwC/lLJhXo8\nJJQWa5rZTZebBPwwIga2GpewKzAb8NadgnTxNd4SWAS80n0VqSdUucCb+oiI+AA5IFxWdC3qmpTS\ntIiYTv7Z/Cv8Z+Di1sB5lVyrpsYklAbNfAhYG+gXEVuUHnqq1Ed2CzkMXF66vWM14DTgNyml+UXU\nrM6LiGHk/6R3AG+SxyScDVyeUppdZG3qtLOBMaWw8AAwklYLvKnviIhfAuOB54CPAj8F5gNNRdal\nzomIFcmhLkq71iu9Z85KKf0LOAc4KSKeAp4lv1e+ANxQ0fPU0i2QEXEJcESZhz6bUrq7dMya5HkU\ndiTf9zkG+EFKaVEvlakqRcSWwPnkQW7LAtPIf7WMMuT1HaWVXL/Lewu8nZhSeqjYqlSpiGgCPgN8\nGHgV+DPwo5TStEILU6dExA7kP7javolfmlI6unTMKeR5ElYG7gFOSCk9VdHz1FJIkCRJtaOvzZMg\nSZJ6iSFBkiSVZUiQJEllGRIkSVJZhgRJklSWIUGSJJVlSJAkSWUZEiRJUlmGBEmSVJYhQZIklWVI\nkCRJZf1/WOoSREV8cXcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45f793d940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "\n",
    "x = np.linspace(-10, 10, 500)\n",
    "plt.plot(x, 1 / (1 + np.e ** (-x)))\n",
    "plt.title('A Sigmoid function')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "a35d0282-7ee3-489b-aa16-349f4364e28d"
    }
   },
   "source": [
    "## MSE mean squared error\n",
    "\n",
    "$$ C(w, b) = \\frac{1}{ m } \\sum_x ( y(x) - a)^2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross entropy\n",
    "\n",
    "$$D(S, L) = -\\sum{L_i log(S_i)} $$\n",
    "S= Softmax ($\\frac{e^x}{\\sum e^x}$), L=Onehot encoded labels\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.356675\n"
     ]
    }
   ],
   "source": [
    "# A tensorflow example: compute cross entropy\n",
    "import tensorflow as tf\n",
    "\n",
    "softmax_data = [0.7, 0.2, 0.1]\n",
    "one_hot_data = [1.0, 0.0, 0.0]\n",
    "\n",
    "softmax = tf.placeholder(tf.float32)\n",
    "one_hot = tf.placeholder(tf.float32)\n",
    "\n",
    "cross_entropy = - tf.reduce_sum( tf.mul(one_hot, tf.log(softmax)) )\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    output = sess.run(cross_entropy, feed_dict={softmax:softmax_data, one_hot:one_hot_data})\n",
    "    print (output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "1ca316c9-4c0b-48a1-8450-26dd7bf9457c"
    }
   },
   "source": [
    "\n",
    "## Gradient descend\n",
    "\n",
    "Given the gradient of a function (it's derivative), we change the future $x$ generation as\n",
    "$$ x = x - gradient * learning\\_rate $$\n",
    "\n",
    "## Stochastic gradient descend (SDG)\n",
    "\n",
    "In practice, because computing the gradient is expensive (the derivative),\n",
    "we adjust the network using instead a small subset of the inputs, and calculating the gradient on them... but many more times.\n",
    "\n",
    "### Momentum and learning rate decay\n",
    "Because the approximation is usually quite bad, we can keep a running average of the direction of the approximated gradient descend, and use that to move out input (momentum).\n",
    "\n",
    "Another trick is to reduce the learning rate going on with the generations.\n",
    "\n",
    "ADAGRAD is a modification of SDG that does momentum and LRD for us"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Avoid overfitting\n",
    "\n",
    "* Use larger datasets\n",
    "* Validate the data with an unused subset of your data\n",
    "* **Dropout**: randomly drop half of the signals and boost the others\n",
    "      keep_prob = tf.placeholder(tf.float32)\n",
    "      tf.nn.dropout(hidden_layer, keep_prob)\n",
    "  remember to use dropout in the training phase, not during validation\n",
    "\n",
    "## Optimization suggestions\n",
    "\n",
    "Try to keep variables with 0 mean ($X_i = 0$) and variance 1 ($X_i = X_j$)\n",
    "\n",
    "For Weights it means choose them from standard distribution (tf, a small $\\sigma$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Network (CNN)\n",
    "\n",
    "It works by taking the input matrix and filter it in smaller *patches*, each moving of an amount called *strive*.\n",
    "\n",
    "Every patch is connected with the next level, a layer wide *k*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CCN dimensionality\n",
    "\n",
    "Given our input layer has a volume of W\n",
    "\n",
    "Our filter has a volume F = ($height * width * depth$)\n",
    "with a stride of S, and a padding of P\n",
    "\n",
    "The following formula gives us the volume of the next layer:\n",
    "$$ (W−F+2P)/S+1 $$\n",
    "(with padding='VALID', with padding='SAME', it simplify to W/S)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "The input is 4D:\n",
    "\n",
    "`X = [batch_size, width, height, depth]`\n",
    "\n",
    "for every level we train a 4D array \n",
    "\n",
    "`W = [filterWidth, filter_height, input_depth, output_depth]`\n",
    "\n",
    "and a bias 1D\n",
    "\n",
    "`B = [output_depth]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polling\n",
    "With this technique instead of taking all the values of a patch we take just the max (or average, depending)\n",
    "In this way we avoid overfitting (reducing the number of parameters). Recently Dropout is more used, as it's more efficient for avoiding overfitting.\n",
    "\n",
    "## 1x1 convolutions\n",
    "Put in front of the patch to create little inexpensive neural networks: more deep and with more parameters"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:drive]",
   "language": "python",
   "name": "conda-env-drive-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3.0
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "nbpresent": {
   "slides": {},
   "themes": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}