4_train_a_classifier.ipynb

{
 "metadata": {
  "language": "lua",
  "name": "",
  "signature": "sha256:a9b8f0b26989edb5dc2cf6a5abbce7a4e80cf4c6e9897b6cd208f084e7665ed5"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You have experience with the Clement Farabet starter code. That is a great starting point.\n",
      "\n",
      "Here's the goal of this tutorial: instead of presenting you with a full working system, let's build it from scratch. We won't use an external package like optim, rather do the weight updates manually so you really control every aspect of what is going on.\n",
      "\n",
      "Also, I'll put everything in global variables which are visible inside functions, and the functions don't take any arguments. Don't do this in real life.\n",
      "\n",
      "The goal is that after this tutorial you'll be confident using Clement's code and changing any element of it."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Implement a full training procedure from scratch"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We will train a small 3-layer neural network, named \"net\", to reproduce the outputs from a larger random neural network (called \"oracle\", you should assume this is a fully trained net that is doing some task really well). I mostly do this because then I don't have to download an actual dataset (can just generate data on the fly). At the other hand, the fact that this works at all (approximating a deep net with a much smaller & shallower net) is pretty interesting and was the subject of a NIPS publication: http://papers.nips.cc/paper/5484-do-deep-nets-really-need-to-be-deep.pdf\n",
      "\n",
      "For an intuitive explanation, Hinton talked about this as \"Dark Knowledge\", see google or for example here: http://fastml.com/geoff-hintons-dark-knowledge/"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "require 'nn' -- no cuda today\n",
      "epochs = 20\n",
      "batchesPerEpoch = 1000\n",
      "testBatches = 400\n",
      "batchSize = 64\n",
      "lrate = 0.01\n",
      "wdecay= 0.00001\n",
      "inpsize = 50\n",
      "outsize = 10\n",
      "hiddens = {40,40}\n",
      "labelNoiseEpsilon = 0.0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Construct a neural net with fully connected layers, with the desired structure you can infer from the parameters (inpsize->hiddens[1]->hiddens[2]->outsize). Use ReLU nonlinearities.\n",
      "\n",
      "The outputs should be log probabilities. The criterion is DistKLDivCriterion, because we will reduce the KL divergence between the learned network and the oracle."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "net = nn.Sequential()\n",
      "-- TODO\n",
      "crit = nn.DistKLDivCriterion()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "How many parameters does the model have in total? Use the function that flattens the weight and bias into one vector."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here is the oracle:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "oracle = nn.Sequential()\n",
      "oracle:add(nn.Linear(inpsize,50))\n",
      "oracle:add(nn.ReLU())\n",
      "oracle:add(nn.Linear(50,100))\n",
      "oracle:add(nn.ReLU())\n",
      "oracle:add(nn.Linear(100,100))\n",
      "oracle:add(nn.ReLU())\n",
      "oracle:add(nn.Linear(100,outsize))\n",
      "oracle:add(nn.SoftMax())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note: the oracle produces probabilities (NOT log probabilities). This is just because that's what DistKLDivCriterion wants. Always give DistKLDivCriterion what she wants."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "How many parameters does the oracle have? What ratio of parameters between the network we'll train and the oracle?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Construct a training set:\n",
      "+ X = randn tensor of size ( (batchesPerEpoch\\*batchSize,inpsize)) and \n",
      "+ Y = calculate the oracle's outputs. MAKE A COPY OF THE OUTPUT! Add some small uniform noise with zero mean and range labelNoiseEpsilon (defined above)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Implement a function evaluate() doing the following (given in not-even-pseudo-code):"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "function evaluate()\n",
      "    local loss = 0\n",
      "    local clfErr = 0\n",
      "    local Xeval, Yeval, Ypred, predClass, oracleClass\n",
      "    -- for each batch:\n",
      "    -- Make Xeval with torch.randn, size (batchSize, inpsize).\n",
      "    -- Compute the true probs from the oracle\n",
      "    -- Compute the predictions from the net\n",
      "    -- Compute the loss from the criterion\n",
      "    -- Get the percentage misclassified (use torch.max and torch.ne)\n",
      "    return loss, clfErr\n",
      "end"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Write the training code. \n",
      "+ Outer loop: epoch. \n",
      "    - ix <- Generate random permutation w torch.randperm\n",
      "    - Inner loop: per-batch.\n",
      "        * Select training batch from X & Y using unmodified X,Y and \n",
      "        * zeroGradParameters\n",
      "        * forward\n",
      "        * backward\n",
      "        * apply weight decay by subtracting the current weights with factor wdecay\n",
      "        * make a gradient step\n",
      "        * extra: add 0.9 momentum like Krizhevsky 2012\n",
      "    - Evaluate, print result.\n",
      "    - If evaluation loss didn't decrease: divide lr by 4.\n",
      "    - collect garbage\n",
      "+ Extra: use https://github.com/tomsercu/tprof to profile your code."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "losses, errs = {}, {}\n",
      "netPar, netGrad = net:getParameters()\n",
      "Xb = torch.Tensor(batchSize, inpsize)\n",
      "Yb = torch.Tensor(batchSize, outsize)\n",
      "for epoch = 1, epochs do\n",
      "    -- TODO magic\n",
      "    loss, err = evaluate()\n",
      "    print(epoch .. \"\\t\" .. loss .. \"\\t\" .. err)\n",
      "    losses[#losses+1] = loss\n",
      "    errs[#errs+1] = err\n",
      "end"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "What's next? Try to play with the label noise and see the impact on performance. Try adding dropout. Try increasing the dataset size (batchesPerEpoch). When adding dropout, make sure you turn it off during evaluation.\n",
      "\n",
      "Do it again: implement a neural network to learn a regression task.\n",
      "Inspiration can be found at \n",
      "https://github.com/Atcold/Machine-learning-with-Torch\n",
      "It's basically the same as what you just did but with a different loss function (squared error)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Similar to this tutorial, Soumith has this great tutorial online that constructs a CIFAR classifier from scratch and trains it: http://nbviewer.ipython.org/github/soumith/nextml/blob/master/05-train-convnet-on-cifar.ipynb"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}