update

Author: mhjensen

doc/pub/week35/html/._week35-bs002.html

@@ -323,13 +323,12 @@
 <!-- !split -->
 <h2 id="reminder-from-last-week" class="anchor">Reminder from last week </h2>
 
-<p>We need first a reminder from last week about linear regression. </p>
+<p>We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters  \( \boldsymbol{\theta} \) and our first encounter is ordinary least squares.</p>
 
-<p>Fitting a continuous function with linear parameterization in terms of the parameters  \( \boldsymbol{\beta} \).</p>
 <ul>
-<li> Method of choice for fitting a continuous function!</li>
+<li> It is the method of choice for fitting a continuous function</li>
 <li> Gives an excellent introduction to central Machine Learning features with <b>understandable pedagogical</b> links to other methods like <b>Neural Networks</b>, <b>Support Vector Machines</b> etc</li>
-<li> Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)</li>
+<li> Analytical expression for the fitting parameters \( \boldsymbol{\theta} \)</li>
 <li> Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more</li>
 <li> Analytical relation with probabilistic interpretations</li> 
 <li> Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics</li>

doc/pub/week35/html/._week35-bs003.html

@@ -346,19 +346,19 @@ <h2 id="the-equations-for-ordinary-least-squares" class="anchor">The equations f
 <p>In linear regression we approximate the unknown function with another
 continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
 some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
+\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
 </p>
 
 <p>Last week we introduced the so-called design matrix in order to define
 the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
-\( \boldsymbol{\beta} \) as
+\( \boldsymbol{\theta} \) as
 </p>
 
 $$
-\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
 $$
 
-<p>and in order to find the optimal parameters \( \beta_i \) we defined a function which
+<p>and in order to find the optimal parameters \( \theta_i \) we defined a function which
 gives a measure of the spread between the values \( y_i \) (which
 represent the output values we want to reproduce) and the parametrized
 values \( \tilde{y}_i \), namely the so-called cost/loss function.

doc/pub/week35/html/._week35-bs004.html

@@ -325,12 +325,12 @@ <h2 id="the-cost-loss-function" class="anchor">The cost/loss function </h2>
 
 <p>We used the mean squared error to define the way we measure the quality of our model</p>
 $$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
 $$
 
 <p>or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as</p>
 $$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
 $$
 
 <p>This function represents one of many possible ways to define the so-called cost function.</p>
@@ -340,10 +340,10 @@ <h2 id="the-cost-loss-function" class="anchor">The cost/loss function </h2>
 </p>
 
 $$
-C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
+C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
 $$
 
-<p>since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. </p>
+<p>since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out. </p>
 
 <p>
 <!-- navigation buttons at the bottom of the page -->

doc/pub/week35/html/._week35-bs005.html

@@ -325,14 +325,14 @@ <h2 id="interpretations-and-optimizing-our-parameters" class="anchor">Interpreta
 
 <p>The function </p>
 $$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
 $$
 
 <p>can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. 
 When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
 </p>
 $$
-y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
+y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
 $$
 
 <p>where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
@@ -345,25 +345,25 @@ <h2 id="interpretations-and-optimizing-our-parameters" class="anchor">Interpreta
 will treat \( y_i \) as our exact value for the response variable.
 </p>
 
-<p>In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem</p>
+<p>In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem</p>
 $$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
 $$
 
 <p>In practical terms it means we will require</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, 
+\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0, 
 $$
 
 <p>which results in</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, 
+\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0, 
 $$
 
 <p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).  
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).  
 $$
 
 

doc/pub/week35/html/._week35-bs006.html

@@ -327,17 +327,17 @@ <h2 id="interpretations-and-optimizing-our-parameters" class="anchor">Interpreta
 <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
 <p>We can rewrite, see the derivations below, </p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),  
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),  
 $$
 
 <p>as</p>
 $$
-\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},  
+\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},  
 $$
 
 <p>and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution</p>
 $$
-\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
 $$
 
 <p>We note also that since our design matrix is defined as \( \boldsymbol{X}\in

doc/pub/week35/html/._week35-bs013.html

@@ -325,46 +325,46 @@ <h2 id="the-mean-squared-error-and-its-derivative" class="anchor">The mean squar
 
 <p>We defined earlier a possible cost function using the mean squared error</p>
 $$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
 $$
 
 <p>or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector</p>
 $$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
 $$
 
-<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
-We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
+<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
+We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
 </p>
 
 <p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
 $$
-\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
 $$
 
-<p>which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as</p>
+<p>which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as</p>
 $$
-C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
+C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
 $$
 
 <p>with partial derivative</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}},
 $$
 
 <p>and using that</p>
 $$
-\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
+\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}}=-\boldsymbol{X},
 $$
 
 <p>where we used the result from example two above. Inserting the last expression we obtain</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
 $$
 
 <p>or as</p>
 $$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
 $$
 
 

doc/pub/week35/html/._week35-bs015.html

@@ -325,13 +325,13 @@ <h2 id="meet-the-hessian-matrix" class="anchor">Meet the Hessian Matrix </h2>
 
 <p>A very important matrix we will meet again and again in machine
 learning is the Hessian.  It is given by the second derivative of the
-cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
+cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
 expression for derivatives of vectors and matrices, we find that the
 second derivative of the mean squared error as cost function is,
 </p>
 
 $$
-\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.  
+\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.  
 $$
 
 <p>The Hessian matrix plays an important role and is defined here as</p>
@@ -342,7 +342,7 @@ <h2 id="meet-the-hessian-matrix" class="anchor">Meet the Hessian Matrix </h2>
 
 <p>For ordinary least squares, it is inversely proportional (derivation
 next week) with the variance of the optimal parameters
-\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
+\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
 (aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
 important role in optmization algorithms and Principal Component
 Analysis as a way to reduce the dimensionality of a machine learning/data analysis

doc/pub/week35/html/._week35-bs016.html

@@ -328,20 +328,20 @@ <h2 id="interpretations-and-optimizing-our-parameters" class="anchor">Interpreta
 <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
 <p>The residuals \( \boldsymbol{\epsilon} \) are in turn given by</p>
 $$
-\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
 $$
 
 <p>and with </p>
 $$
-\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, 
+\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0, 
 $$
 
 <p>we have</p>
 $$
-\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, 
+\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0, 
 $$
 
-<p>meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals.  </p>
+<p>meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals.  </p>
 </div>
 </div>
 

doc/pub/week35/html/._week35-bs017.html

@@ -328,10 +328,10 @@ <h2 id="example-relevant-for-the-exercises" class="anchor">Example relevant for
 We assume our data can represented by a fourth-order polynomial. For the $i$th component we have 
 </p>
 $$
-\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
+\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
 $$
 
-<p>we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an 
+<p>we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an 
 \( n\times p \) matrix \( \boldsymbol{X} \).
 </p>
 

doc/pub/week35/html/._week35-bs018.html

@@ -323,7 +323,7 @@
 <!-- !split -->
 <h2 id="own-code-for-ordinary-least-squares" class="anchor">Own code for Ordinary Least Squares </h2>
 
-<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
+<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
 
 <!-- code=python (!bc pycod) typeset with pygments style "default" -->
 <div class="cell border-box-sizing code_cell rendered">