update

Author: mhjensen

doc/LectureNotes/_build/.doctrees/environment.pickle

@@ -419,7 +419,6 @@ <h2> Contents </h2>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-hessian-matrix">The Hessian matrix</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-program">Simple program</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">Gradient Descent Example</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#and-a-corresponding-example-using-scikit-learn">And a corresponding example using <strong>scikit-learn</strong></a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#gradient-descent-and-ridge">Gradient descent and Ridge</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-hessian-matrix-for-ridge-regression">The Hessian matrix for Ridge Regression</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression</a></li>
@@ -631,7 +630,7 @@ <h2>Fixing the singularity<a class="headerlink" href="#fixing-the-singularity" t
 \]</div>
 <p>has linearly dependent column vectors, we will not be able to compute the inverse
 of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> and we cannot find the parameters (estimators) <span class="math notranslate nohighlight">\(\theta_i\)</span>.
-The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})^{-1}\)</span> exits.
+The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})^{-1}\)</span> exists.
 This is more likely to happen when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is high-dimensional. In this case it is likely to encounter a situation where
 the regression parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> cannot be estimated.</p>
 <p>A cheap  <em>ad hoc</em> approach is  simply to add a small diagonal component to the matrix to invert, that is we change</p>
@@ -1239,12 +1238,12 @@ <h2>Deriving the  Lasso Regression Equations<a class="headerlink" href="#derivin
 <p>and reordering we have</p>
 <div class="math notranslate nohighlight">
 \[
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=\boldsymbol{X}^T\boldsymbol{y}.
 \]</div>
 <p>We can redefine <span class="math notranslate nohighlight">\(\lambda\)</span> to absorb the constant <span class="math notranslate nohighlight">\(n/2\)</span> and we rewrite the last equation as</p>
 <div class="math notranslate nohighlight">
 \[
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=\boldsymbol{X}^T\boldsymbol{y}.
 \]</div>
 <p>This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms.We will discuss how to code the above methods using gradient descent methods.</p>
 </section>
@@ -1643,31 +1642,6 @@ <h2>Gradient Descent Example<a class="headerlink" href="#id1" title="Link to thi
 </div>
 </div>
 </section>
-<section id="and-a-corresponding-example-using-scikit-learn">
-<h2>And a corresponding example using <strong>scikit-learn</strong><a class="headerlink" href="#and-a-corresponding-example-using-scikit-learn" title="Link to this heading">#</a></h2>
-<div class="cell docutils container">
-<div class="cell_input docutils container">
-<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Importing various packages
-from random import random, seed
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.linear_model import SGDRegressor
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)
-print(theta_linreg)
-sgdreg = SGDRegressor(max_iter = 50, penalty=None, eta0=0.1)
-sgdreg.fit(x,y.ravel())
-print(sgdreg.intercept_, sgdreg.coef_)
-</pre></div>
-</div>
-</div>
-</div>
-</section>
 <section id="gradient-descent-and-ridge">
 <h2>Gradient descent and Ridge<a class="headerlink" href="#gradient-descent-and-ridge" title="Link to this heading">#</a></h2>
 <p>We have also discussed Ridge regression where the loss function contains a regularized term given by the <span class="math notranslate nohighlight">\(L_2\)</span> norm of <span class="math notranslate nohighlight">\(\theta\)</span>,</p>
@@ -2488,7 +2462,6 @@ <h2>Another Example, now with a polynomial fit<a class="headerlink" href="#anoth
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-hessian-matrix">The Hessian matrix</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-program">Simple program</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">Gradient Descent Example</a></li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#and-a-corresponding-example-using-scikit-learn">And a corresponding example using <strong>scikit-learn</strong></a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#gradient-descent-and-ridge">Gradient descent and Ridge</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-hessian-matrix-for-ridge-regression">The Hessian matrix for Ridge Regression</a></li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#program-example-for-gradient-descent-with-ridge-regression">Program example for gradient descent with Ridge Regression</a></li>