update week 36

Author: mhjensen

doc/LectureNotes/_build/.doctrees/environment.pickle

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "1b638450",
+   "id": "1b941c35",
    "metadata": {
     "editable": true
    },
@@ -14,7 +14,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "60060788",
+   "id": "dc05b096",
    "metadata": {
     "editable": true
    },
@@ -27,7 +27,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7cdd88e4",
+   "id": "2cf07405",
    "metadata": {
     "editable": true
    },
@@ -37,7 +37,7 @@
     "After having completed these exercises you will have:\n",
     "1. Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression\n",
     "\n",
-    "2. Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach\n",
+    "2. Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach\n",
     "\n",
     "3. Explore the role of the learning rate in the gradient descent approach and the hyperparameter $\\lambda$ in Ridge regression\n",
     "\n",
@@ -46,7 +46,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "0328fa2a",
+   "id": "3c139edb",
    "metadata": {
     "editable": true
    },
@@ -58,7 +58,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "ac760265",
+   "id": "aad4cfac",
    "metadata": {
     "editable": true
    },
@@ -70,7 +70,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8d4b0753",
+   "id": "6682282f",
    "metadata": {
     "editable": true
    },
@@ -83,7 +83,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4517d311",
+   "id": "89e2f4c4",
    "metadata": {
     "editable": true
    },
@@ -99,7 +99,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "da1834a9",
+   "id": "b06d4e53",
    "metadata": {
     "editable": true
    },
@@ -120,7 +120,7 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "590b2fb0",
+   "id": "63796480",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -140,7 +140,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "24dd92fc",
+   "id": "80748600",
    "metadata": {
     "editable": true
    },
@@ -156,7 +156,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4ba80e16",
+   "id": "92751e5f",
    "metadata": {
     "editable": true
    },
@@ -168,18 +168,18 @@
   },
   {
    "cell_type": "markdown",
-   "id": "be65f56f",
+   "id": "aedfbd7a",
    "metadata": {
     "editable": true
    },
    "source": [
-    "## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
+    "## Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "c265677e",
+   "id": "5d1288fa",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -200,7 +200,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b989efb9",
+   "id": "628f5e89",
    "metadata": {
     "editable": true
    },
@@ -214,7 +214,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "17b08af7",
+   "id": "f115ba4e",
    "metadata": {
     "editable": true
    },
@@ -226,7 +226,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6acd4708",
+   "id": "a9b5189c",
    "metadata": {
     "editable": true
    },
@@ -238,7 +238,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "50579422",
+   "id": "a3969ff6",
    "metadata": {
     "editable": true
    },
@@ -258,7 +258,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "c57cc917",
+   "id": "34d87303",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -277,13 +277,14 @@
     "cost_history = np.zeros(num_iters)\n",
     "\n",
     "# Gradient descent loop\n",
-    "m = n_samples  # number of examples\n",
+    "m = n_samples  # number of data points\n",
     "for t in range(num_iters):\n",
     "    # Compute prediction error\n",
     "    error = X_norm.dot(theta) - y_centered \n",
     "    # Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring\n",
     "    cost_OLS = ?\n",
     "    cost_Ridge = ?\n",
+    "    # You could add a history for both methods (optional)\n",
     "    cost_history[t] = ?\n",
     "    # Compute gradients for OSL and Ridge\n",
     "    grad_OLS = ?\n",
@@ -301,7 +302,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e2654903",
+   "id": "989f70bb",
    "metadata": {
     "editable": true
    },
@@ -313,19 +314,19 @@
   },
   {
    "cell_type": "markdown",
-   "id": "7e7e7de6",
+   "id": "370b2dad",
    "metadata": {
     "editable": true
    },
    "source": [
     "### 4b)\n",
     "\n",
-    "Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
+    "Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\\theta$ values. How would you define a stopping criterion?"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "75542708",
+   "id": "ef197cd7",
    "metadata": {
     "editable": true
    },
@@ -351,7 +352,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "id": "06077986",
+   "id": "4ccc2f65",
    "metadata": {
     "collapsed": false,
     "editable": true
@@ -380,7 +381,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "86d46505",
+   "id": "00e279ef",
    "metadata": {
     "editable": true
    },
@@ -394,7 +395,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1d2f848",
+   "id": "c910b3f4",
    "metadata": {
     "editable": true
    },
@@ -406,7 +407,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f1c91e84",
+   "id": "89e6e040",
    "metadata": {
     "editable": true
    },

doc/LectureNotes/_build/.doctrees/exercisesweek37.doctree

@@ -389,7 +389,7 @@ <h2> Contents </h2>
 </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
 </ul>
@@ -422,7 +422,7 @@ <h2>Learning goals<a class="headerlink" href="#learning-goals" title="Link to th
 <p>After having completed these exercises you will have:</p>
 <ol class="arabic simple">
 <li><p>Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression</p></li>
-<li><p>Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach</p></li>
+<li><p>Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach</p></li>
 <li><p>Explore the role of the learning rate in the gradient descent approach and the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span> in Ridge regression</p></li>
 <li><p>Scale the data properly</p></li>
 </ol>
@@ -482,8 +482,8 @@ <h3>1a)<a class="headerlink" href="#a" title="Link to this heading">#</a></h3>
 <h2>Exercise 2, calculate the gradients<a class="headerlink" href="#exercise-2-calculate-the-gradients" title="Link to this heading">#</a></h2>
 <p>Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function.</p>
 </section>
-<section id="exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
-<h2>Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
+<section id="exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
+<h2>Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Set regularization parameter, either a single value or a vector of values
@@ -537,13 +537,14 @@ <h2>Exercise 4, Implementing the simplest form for gradient descent<a class="hea
 cost_history = np.zeros(num_iters)
 
 # Gradient descent loop
-m = n_samples  # number of examples
+m = n_samples  # number of data points
 for t in range(num_iters):
     # Compute prediction error
     error = X_norm.dot(theta) - y_centered 
     # Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring
     cost_OLS = ?
     cost_Ridge = ?
+    # You could add a history for both methods (optional)
     cost_history[t] = ?
     # Compute gradients for OSL and Ridge
     grad_OLS = ?
@@ -567,7 +568,7 @@ <h3>4a)<a class="headerlink" href="#id2" title="Link to this heading">#</a></h3>
 </section>
 <section id="id3">
 <h3>4b)<a class="headerlink" href="#id3" title="Link to this heading">#</a></h3>
-<p>Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?</p>
+<p>Try to add a stopping parameter as function of the number iterations and the difference between the new and old <span class="math notranslate nohighlight">\(\theta\)</span> values. How would you define a stopping criterion?</p>
 </section>
 </section>
 <section id="exercise-5-ridge-regression-and-a-new-synthetic-dataset">
@@ -697,7 +698,7 @@ <h2>Exercise 5, Ridge regression and a new Synthetic Dataset<a class="headerlink
 </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
 </ul>