SOL3_fullNetwork.ipynb

{
 "metadata": {
  "language": "lua",
  "name": "",
  "signature": "sha256:55740e94f9cd077c92ccd694b4acfd9c013fc4ae49f23faf47494dc44425d432"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You have explored simple layers and the two most popular criterions. Now let's put things together and do one forward & backward pass through a network."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "require 'nn'; torch.manualSeed(1238)\n",
      "input = torch.randn(16, 50) --batchsize, inputsize\n",
      "labels = torch.rand(16):mul(3):add(1):floor() -- 3 class labels"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Make a network with a linear layer (50->h), a ReLU, another linear layer with output size 3, and a logsoftmax."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "model = nn.Sequential()\n",
      "-- model:add(...)\n",
      "crit = nn.ClassNLLCriterion()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "H=5\n",
      "model = nn.Sequential()\n",
      "model:add(nn.Linear(50,H))\n",
      "model:add(nn.ReLU())\n",
      "model:add(nn.Linear(H,3))\n",
      "model:add(nn.LogSoftMax())\n",
      "print(model:__tostring())\n",
      "crit = nn.ClassNLLCriterion()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 58,
       "text": [
        "nn.Sequential {\n",
        "  [input -> (1) -> (2) -> (3) -> (4) -> output]\n",
        "  (1): nn.Linear\n",
        "  (2): nn.ReLU\n",
        "  (3): nn.Linear\n",
        "  (4): nn.LogSoftMax\n",
        "}\t\n"
       ]
      }
     ],
     "prompt_number": 58
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Do a forward and backward pass. Set the gradients to zero first (they get initialized on a chunk of memory so will contain garbage). Forward through the network. Check that the output looks like log probabilities. Calculate & print the NLL loss. Calculate the gradient wrt the log probabilities. Backprop this gradient through the network. Done"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "model:zeroGradParameters()\n",
      "logprobs = model:forward(input)\n",
      "loss = crit:forward(logprobs, labels)\n",
      "print(loss)\n",
      "gradModel = crit:backward(logprobs, labels)\n",
      "model:backward(input, gradModel)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 59,
       "text": [
        "1.098967644844\t\n"
       ]
      }
     ],
     "prompt_number": 59
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(model:get(4).gradInput)\n",
      "-- note how the correct class has a negative gradient, the others have a positive gradient.\n",
      "-- explain this."
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 60,
       "text": [
        "0.01 *\n",
        "  2.3184 -4.1781  1.8597\n",
        "  2.0801  2.0480 -4.1281\n",
        "  2.2018 -4.2360  2.0342\n",
        " -4.1352  1.9999  2.1353\n",
        "  2.4446 -4.9845  2.5400\n",
        "  2.1475  1.9339 -4.0813\n",
        "  2.3350  1.5159 -3.8509\n",
        " -4.0759  1.9785  2.0974\n",
        " -3.4036  1.5557  1.8479\n",
        " -3.7601  1.4270  2.3332\n",
        " -4.1025  1.9339  2.1687\n",
        "  2.2842  2.0140 -4.2982\n",
        "  2.5025  2.1803 -4.6827\n",
        " -4.1783  2.4144  1.7639\n",
        "  2.4317 -3.9486  1.5170\n",
        "  2.1112  2.0341 -4.1453\n",
        "[torch.DoubleTensor of dimension 16x3]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 60
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Look at the gradWeight and gradInput of the second linear layer. Now zero the gradWeight & gradBias and calculate the gradient again by backpropping the gradInput of the layer above (using as input: the output of the layer below). This is what sequential does: https://github.com/torch/nn/blob/master/Sequential.lua"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(model:get(3).gradWeight)\n",
      "print(model:get(3).gradInput)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 63,
       "text": [
        "0.01 *\n",
        "  2.8435  2.8336  2.7869  0.1387 -2.2476\n",
        " -3.2253  4.4082 -0.9398  1.0173  1.9105\n",
        "  0.3819 -7.2418 -1.8472 -1.1560  0.3371\n",
        "[torch.DoubleTensor of dimension 3x5]\n",
        "\n",
        "0.01 *\n",
        " -0.7775 -1.5580  3.1117 -2.0005  1.9035\n",
        "  1.8345  0.7686 -1.6457  2.5383 -0.1279\n",
        " -0.8555 -1.5799  3.1604 -2.1003  1.8926\n",
        " -0.9948  0.7412 -1.3762 -0.5076 -1.6684\n",
        " -1.0728 -1.8593  3.7244 -2.5422  2.1904\n",
        "  1.8151  0.7260 -1.5609  2.4856 -0.0749\n",
        "  1.7172  0.5701 -1.2479  2.2640  0.1044\n",
        " -0.9774  0.7333 -1.3621 -0.4949 -1.6468\n",
        " -0.8583  0.5763 -1.0635 -0.4854 -1.3446\n",
        " -1.0757  0.5276 -0.9516 -0.7543 -1.3931\n",
        " -1.0089  0.7166 -1.3269 -0.5411 -1.6393\n",
        "  1.9118  0.7562 -1.6273  2.6117 -0.0660\n",
        "  2.0831  0.8187 -1.7627  2.8417 -0.0641\n",
        " -0.8333  0.8960 -1.6919 -0.2186 -1.8105\n",
        " -0.6261 -1.4721  2.9318 -1.7741  1.8591\n",
        "  1.8424  0.7634 -1.6362  2.5430 -0.1157\n",
        "[torch.DoubleTensor of dimension 16x5]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 63
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "model:get(3):zeroGradParameters()\n",
      "model:get(3).gradInput:zero()\n",
      "model:get(3):backward(model:get(2).output, model:get(4).gradInput)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 71
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(model:get(3).gradWeight)\n",
      "print(model:get(3).gradInput)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 72,
       "text": [
        "0.01 *\n",
        "  2.8435  2.8336  2.7869  0.1387 -2.2476\n",
        " -3.2253  4.4082 -0.9398  1.0173  1.9105\n",
        "  0.3819 -7.2418 -1.8472 -1.1560  0.3371\n",
        "[torch.DoubleTensor of dimension 3x5]\n",
        "\n",
        "0.01 *\n",
        " -0.7775 -1.5580  3.1117 -2.0005  1.9035\n",
        "  1.8345  0.7686 -1.6457  2.5383 -0.1279\n",
        " -0.8555 -1.5799  3.1604 -2.1003  1.8926\n",
        " -0.9948  0.7412 -1.3762 -0.5076 -1.6684\n",
        " -1.0728 -1.8593  3.7244 -2.5422  2.1904\n",
        "  1.8151  0.7260 -1.5609  2.4856 -0.0749\n",
        "  1.7172  0.5701 -1.2479  2.2640  0.1044\n",
        " -0.9774  0.7333 -1.3621 -0.4949 -1.6468\n",
        " -0.8583  0.5763 -1.0635 -0.4854 -1.3446\n",
        " -1.0757  0.5276 -0.9516 -0.7543 -1.3931\n",
        " -1.0089  0.7166 -1.3269 -0.5411 -1.6393\n",
        "  1.9118  0.7562 -1.6273  2.6117 -0.0660\n",
        "  2.0831  0.8187 -1.7627  2.8417 -0.0641\n",
        " -0.8333  0.8960 -1.6919 -0.2186 -1.8105\n",
        " -0.6261 -1.4721  2.9318 -1.7741  1.8591\n",
        "  1.8424  0.7634 -1.6362  2.5430 -0.1157\n",
        "[torch.DoubleTensor of dimension 16x5]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 72
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "See also soumith's tutorial from nextml:\n",
      "\n",
      "http://nbviewer.ipython.org/github/soumith/nextml/blob/master/04-neural-networks-basics.ipynb"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}