2.Plain_Pytorch_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Pytorch](images/pytorch_logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression in plain Pytorch\n",
    "Let's rebuild our regression, but this time we get to see how Pytorch simplifies the code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set seed\n",
    "seed = 42\n",
    "torch.manual_seed(seed);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "We are using the built-in sklearn dataset Boston House Prices.\n",
    "\n",
    "Our goal is to predict the median price of a home in a given town from a number of features, such as Crime Rate, Property Tax Rate, amount of Industry etc.\n",
    "\n",
    "It's generally a good idea to scale our data, so we use Sklearn's MinMax scaler to scale our values between 0 and 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load our dataset\n",
    "boston = load_boston()\n",
    "train_x, test_x, train_y, test_y = train_test_split(boston.data, boston.target, random_state=seed)\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "train_x = torch.tensor(scaler.fit_transform(train_x), dtype=torch.float)\n",
    "test_x = torch.tensor(scaler.transform(test_x), dtype=torch.float)\n",
    "train_y = torch.tensor(train_y, dtype=torch.float).view(-1, 1)\n",
    "test_y = torch.tensor(test_y, dtype=torch.float).view(-1, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup parameters\n",
    "\n",
    "We have some hyperparameters to set, as well as some numbers we need to know upfront.\n",
    "\n",
    "`layer_size` --> We need to know how many input variables there are, so we can create an equivalent number of weights\n",
    "\n",
    "`lr` --> Aka learning rate.\n",
    "When we take a step in our gradient descent, we multiply by this factor, so we don't take too big or too large a step. \n",
    "\n",
    "`epochs` --> How many times should we keep stepping?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_size = train_x.shape[1]\n",
    "lr = 0.05\n",
    "epochs = 700"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize weights and bias\n",
    "\n",
    "We need one weight to multiply each feature with - we are learning what these should be, so we start them as a random number. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize weights\n",
    "w = torch.randn(layer_size, 1, requires_grad=True, dtype=torch.float)\n",
    "b = torch.zeros(1, requires_grad=True, dtype=torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Loss Function\n",
    "\n",
    "Just like before, we want to use mean squared error to say how bad or good our line is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define loss function\n",
    "def mean_squared_error(y_hat, y):\n",
    "    return ((y_hat - y) ** 2).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 Train Loss: 507.965576171875 Test Loss: 223.5579071044922\n",
      "Epoch: 10 Train Loss: 99.24380493164062 Test Loss: 78.80070495605469\n",
      "Epoch: 20 Train Loss: 77.88609313964844 Test Loss: 60.800838470458984\n",
      "Epoch: 30 Train Loss: 65.93444061279297 Test Loss: 51.3888053894043\n",
      "Epoch: 40 Train Loss: 58.795101165771484 Test Loss: 46.30845260620117\n",
      "Epoch: 50 Train Loss: 54.17264175415039 Test Loss: 43.3354606628418\n",
      "Epoch: 60 Train Loss: 50.90646743774414 Test Loss: 41.38874053955078\n",
      "Epoch: 70 Train Loss: 48.40536880493164 Test Loss: 39.94911193847656\n",
      "Epoch: 80 Train Loss: 46.364295959472656 Test Loss: 38.77071762084961\n",
      "Epoch: 90 Train Loss: 44.622459411621094 Test Loss: 37.739166259765625\n",
      "Epoch: 100 Train Loss: 43.09225082397461 Test Loss: 36.802215576171875\n",
      "Epoch: 110 Train Loss: 41.72350311279297 Test Loss: 35.93635559082031\n",
      "Epoch: 120 Train Loss: 40.48552322387695 Test Loss: 35.130828857421875\n",
      "Epoch: 130 Train Loss: 39.35799789428711 Test Loss: 34.380210876464844\n",
      "Epoch: 140 Train Loss: 38.326358795166016 Test Loss: 33.68110656738281\n",
      "Epoch: 150 Train Loss: 37.37943649291992 Test Loss: 33.030757904052734\n",
      "Epoch: 160 Train Loss: 36.50814437866211 Test Loss: 32.42655563354492\n",
      "Epoch: 170 Train Loss: 35.70485305786133 Test Loss: 31.86577033996582\n",
      "Epoch: 180 Train Loss: 34.96299743652344 Test Loss: 31.345691680908203\n",
      "Epoch: 190 Train Loss: 34.276798248291016 Test Loss: 30.86356544494629\n",
      "Epoch: 200 Train Loss: 33.64120101928711 Test Loss: 30.416736602783203\n",
      "Epoch: 210 Train Loss: 33.05164337158203 Test Loss: 30.002620697021484\n",
      "Epoch: 220 Train Loss: 32.50410079956055 Test Loss: 29.618770599365234\n",
      "Epoch: 230 Train Loss: 31.99493980407715 Test Loss: 29.262897491455078\n",
      "Epoch: 240 Train Loss: 31.52094078063965 Test Loss: 28.932865142822266\n",
      "Epoch: 250 Train Loss: 31.079137802124023 Test Loss: 28.626663208007812\n",
      "Epoch: 260 Train Loss: 30.66690444946289 Test Loss: 28.34246253967285\n",
      "Epoch: 270 Train Loss: 30.28186798095703 Test Loss: 28.078580856323242\n",
      "Epoch: 280 Train Loss: 29.92184829711914 Test Loss: 27.833446502685547\n",
      "Epoch: 290 Train Loss: 29.58488655090332 Test Loss: 27.60562515258789\n",
      "Epoch: 300 Train Loss: 29.269222259521484 Test Loss: 27.393802642822266\n",
      "Epoch: 310 Train Loss: 28.973236083984375 Test Loss: 27.19677734375\n",
      "Epoch: 320 Train Loss: 28.695457458496094 Test Loss: 27.013444900512695\n",
      "Epoch: 330 Train Loss: 28.434520721435547 Test Loss: 26.84275245666504\n",
      "Epoch: 340 Train Loss: 28.189228057861328 Test Loss: 26.683801651000977\n",
      "Epoch: 350 Train Loss: 27.958433151245117 Test Loss: 26.53569793701172\n",
      "Epoch: 360 Train Loss: 27.74113655090332 Test Loss: 26.397672653198242\n",
      "Epoch: 370 Train Loss: 27.536361694335938 Test Loss: 26.268970489501953\n",
      "Epoch: 380 Train Loss: 27.343257904052734 Test Loss: 26.14892578125\n",
      "Epoch: 390 Train Loss: 27.161029815673828 Test Loss: 26.03692626953125\n",
      "Epoch: 400 Train Loss: 26.988943099975586 Test Loss: 25.93238067626953\n",
      "Epoch: 410 Train Loss: 26.826309204101562 Test Loss: 25.834754943847656\n",
      "Epoch: 420 Train Loss: 26.672513961791992 Test Loss: 25.743572235107422\n",
      "Epoch: 430 Train Loss: 26.526988983154297 Test Loss: 25.658374786376953\n",
      "Epoch: 440 Train Loss: 26.389171600341797 Test Loss: 25.578725814819336\n",
      "Epoch: 450 Train Loss: 26.25858497619629 Test Loss: 25.504241943359375\n",
      "Epoch: 460 Train Loss: 26.1347713470459 Test Loss: 25.434572219848633\n",
      "Epoch: 470 Train Loss: 26.01729965209961 Test Loss: 25.369359970092773\n",
      "Epoch: 480 Train Loss: 25.90578269958496 Test Loss: 25.308307647705078\n",
      "Epoch: 490 Train Loss: 25.799848556518555 Test Loss: 25.25112533569336\n",
      "Epoch: 500 Train Loss: 25.69915771484375 Test Loss: 25.197532653808594\n",
      "Epoch: 510 Train Loss: 25.603368759155273 Test Loss: 25.147281646728516\n",
      "Epoch: 520 Train Loss: 25.51222038269043 Test Loss: 25.10013771057129\n",
      "Epoch: 530 Train Loss: 25.425405502319336 Test Loss: 25.05589485168457\n",
      "Epoch: 540 Train Loss: 25.342695236206055 Test Loss: 25.014345169067383\n",
      "Epoch: 550 Train Loss: 25.263832092285156 Test Loss: 24.975296020507812\n",
      "Epoch: 560 Train Loss: 25.188600540161133 Test Loss: 24.938568115234375\n",
      "Epoch: 570 Train Loss: 25.11677360534668 Test Loss: 24.904008865356445\n",
      "Epoch: 580 Train Loss: 25.04816246032715 Test Loss: 24.871463775634766\n",
      "Epoch: 590 Train Loss: 24.98259925842285 Test Loss: 24.840789794921875\n",
      "Epoch: 600 Train Loss: 24.919891357421875 Test Loss: 24.811851501464844\n",
      "Epoch: 610 Train Loss: 24.85987663269043 Test Loss: 24.784534454345703\n",
      "Epoch: 620 Train Loss: 24.802425384521484 Test Loss: 24.75871467590332\n",
      "Epoch: 630 Train Loss: 24.747385025024414 Test Loss: 24.73428726196289\n",
      "Epoch: 640 Train Loss: 24.69462013244629 Test Loss: 24.711153030395508\n",
      "Epoch: 650 Train Loss: 24.64399528503418 Test Loss: 24.68921661376953\n",
      "Epoch: 660 Train Loss: 24.59541130065918 Test Loss: 24.668394088745117\n",
      "Epoch: 670 Train Loss: 24.54875373840332 Test Loss: 24.648616790771484\n",
      "Epoch: 680 Train Loss: 24.50391387939453 Test Loss: 24.62977409362793\n",
      "Epoch: 690 Train Loss: 24.46080780029297 Test Loss: 24.611827850341797\n"
     ]
    }
   ],
   "source": [
    "# Training loop\n",
    "for epoch in range(epochs):\n",
    "    # Forward pass\n",
    "    pred = train_x @ w + b\n",
    "\n",
    "    loss = mean_squared_error(pred, train_y)\n",
    "\n",
    "    # Backpropagation\n",
    "    loss.backward() # The magic bit!\n",
    "    with torch.no_grad():\n",
    "        w -= w.grad * lr\n",
    "        b -= b.grad * lr\n",
    "        w.grad.zero_() # Gotta reset the gradients to zero, so they don't accumulate\n",
    "        b.grad.zero_()\n",
    "        \n",
    "        # Validate model\n",
    "        val_pred = test_x @ w + b\n",
    "        val_loss = mean_squared_error(val_pred, test_y)\n",
    "        if epoch % 10 == 0:\n",
    "            print(f\"Epoch: {epoch} Train Loss: {loss.item()} Test Loss: {val_loss.item()}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}