Skip to content

Commit f202dd1

Browse files
mhjensenmhjensen
committed
updated exercises week37
1 parent 454f165 commit f202dd1

File tree

3 files changed

+41
-56
lines changed

3 files changed

+41
-56
lines changed

‎doc/LectureNotes/exercisesweek37.ipynb‎

Lines changed: 32 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"cells": [
33
{
44
"cell_type": "markdown",
5-
"id": "1b941c35",
5+
"id": "8e6632a0",
66
"metadata": {
77
"editable": true
88
},
@@ -14,7 +14,7 @@
1414
},
1515
{
1616
"cell_type": "markdown",
17-
"id": "dc05b096",
17+
"id": "82705c4f",
1818
"metadata": {
1919
"editable": true
2020
},
@@ -27,7 +27,7 @@
2727
},
2828
{
2929
"cell_type": "markdown",
30-
"id": "2cf07405",
30+
"id": "921bf331",
3131
"metadata": {
3232
"editable": true
3333
},
@@ -46,7 +46,7 @@
4646
},
4747
{
4848
"cell_type": "markdown",
49-
"id": "3c139edb",
49+
"id": "adff65d5",
5050
"metadata": {
5151
"editable": true
5252
},
@@ -58,7 +58,7 @@
5858
},
5959
{
6060
"cell_type": "markdown",
61-
"id": "aad4cfac",
61+
"id": "70418b3d",
6262
"metadata": {
6363
"editable": true
6464
},
@@ -70,7 +70,7 @@
7070
},
7171
{
7272
"cell_type": "markdown",
73-
"id": "6682282f",
73+
"id": "11a3cf73",
7474
"metadata": {
7575
"editable": true
7676
},
@@ -83,7 +83,7 @@
8383
},
8484
{
8585
"cell_type": "markdown",
86-
"id": "89e2f4c4",
86+
"id": "04a06b51",
8787
"metadata": {
8888
"editable": true
8989
},
@@ -99,7 +99,7 @@
9999
},
100100
{
101101
"cell_type": "markdown",
102-
"id": "b06d4e53",
102+
"id": "408db3d9",
103103
"metadata": {
104104
"editable": true
105105
},
@@ -120,7 +120,7 @@
120120
{
121121
"cell_type": "code",
122122
"execution_count": 1,
123-
"id": "63796480",
123+
"id": "37fb732c",
124124
"metadata": {
125125
"collapsed": false,
126126
"editable": true
@@ -140,12 +140,12 @@
140140
},
141141
{
142142
"cell_type": "markdown",
143-
"id": "80748600",
143+
"id": "d861e1e3",
144144
"metadata": {
145145
"editable": true
146146
},
147147
"source": [
148-
"Fill in the necessary details.\n",
148+
"Fill in the necessary details. Do we need to center the $y$-values? \n",
149149
"\n",
150150
"After this preprocessing, each column of $\\boldsymbol{X}_{\\mathrm{norm}}$ has mean zero and standard deviation $1$\n",
151151
"and $\\boldsymbol{y}_{\\mathrm{centered}}$ has mean 0. This makes the optimization landscape\n",
@@ -156,7 +156,7 @@
156156
},
157157
{
158158
"cell_type": "markdown",
159-
"id": "92751e5f",
159+
"id": "b3e774d0",
160160
"metadata": {
161161
"editable": true
162162
},
@@ -168,7 +168,7 @@
168168
},
169169
{
170170
"cell_type": "markdown",
171-
"id": "aedfbd7a",
171+
"id": "d5dc7708",
172172
"metadata": {
173173
"editable": true
174174
},
@@ -179,15 +179,17 @@
179179
{
180180
"cell_type": "code",
181181
"execution_count": 2,
182-
"id": "5d1288fa",
182+
"id": "4c9c86ac",
183183
"metadata": {
184184
"collapsed": false,
185185
"editable": true
186186
},
187187
"outputs": [],
188188
"source": [
189189
"# Set regularization parameter, either a single value or a vector of values\n",
190-
"lambda = ?\n",
190+
"# Note that lambda is a python keyword. The lambda keyword is used to create small, single-expression functions without a formal name. These are often called \"anonymous functions\" or \"lambda functions.\"\n",
191+
"lam = ?\n",
192+
"\n",
191193
"\n",
192194
"# Analytical form for OLS and Ridge solution: theta_Ridge = (X^T X + lambda * I)^{-1} X^T y and theta_OLS = (X^T X)^{-1} X^T y\n",
193195
"I = np.eye(n_features)\n",
@@ -200,7 +202,7 @@
200202
},
201203
{
202204
"cell_type": "markdown",
203-
"id": "628f5e89",
205+
"id": "eeae00fd",
204206
"metadata": {
205207
"editable": true
206208
},
@@ -214,7 +216,7 @@
214216
},
215217
{
216218
"cell_type": "markdown",
217-
"id": "f115ba4e",
219+
"id": "e1c215d5",
218220
"metadata": {
219221
"editable": true
220222
},
@@ -226,7 +228,7 @@
226228
},
227229
{
228230
"cell_type": "markdown",
229-
"id": "a9b5189c",
231+
"id": "587dd3dc",
230232
"metadata": {
231233
"editable": true
232234
},
@@ -238,7 +240,7 @@
238240
},
239241
{
240242
"cell_type": "markdown",
241-
"id": "a3969ff6",
243+
"id": "bfa34697",
242244
"metadata": {
243245
"editable": true
244246
},
@@ -258,7 +260,7 @@
258260
{
259261
"cell_type": "code",
260262
"execution_count": 3,
261-
"id": "34d87303",
263+
"id": "49245f55",
262264
"metadata": {
263265
"collapsed": false,
264266
"editable": true
@@ -273,19 +275,8 @@
273275
"# Initialize weights for gradient descent\n",
274276
"theta = np.zeros(n_features)\n",
275277
"\n",
276-
"# Arrays to store history for plotting\n",
277-
"cost_history = np.zeros(num_iters)\n",
278-
"\n",
279278
"# Gradient descent loop\n",
280-
"m = n_samples # number of data points\n",
281279
"for t in range(num_iters):\n",
282-
" # Compute prediction error\n",
283-
" error = X_norm.dot(theta) - y_centered \n",
284-
" # Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring\n",
285-
" cost_OLS = ?\n",
286-
" cost_Ridge = ?\n",
287-
" # You could add a history for both methods (optional)\n",
288-
" cost_history[t] = ?\n",
289280
" # Compute gradients for OSL and Ridge\n",
290281
" grad_OLS = ?\n",
291282
" grad_Ridge = ?\n",
@@ -302,31 +293,33 @@
302293
},
303294
{
304295
"cell_type": "markdown",
305-
"id": "989f70bb",
296+
"id": "f3f43f2c",
306297
"metadata": {
307298
"editable": true
308299
},
309300
"source": [
310301
"### 4a)\n",
311302
"\n",
312-
"Discuss the results as function of the learning rate parameters and the number of iterations."
303+
"Write first a gradient descent code for OLS only using the above template.\n",
304+
"Discuss the results as function of the learning rate parameters and the number of iterations"
313305
]
314306
},
315307
{
316308
"cell_type": "markdown",
317-
"id": "370b2dad",
309+
"id": "9ba303be",
318310
"metadata": {
319311
"editable": true
320312
},
321313
"source": [
322314
"### 4b)\n",
323315
"\n",
316+
"Write then a similar code for Ridge regression using the above template.\n",
324317
"Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\\theta$ values. How would you define a stopping criterion?"
325318
]
326319
},
327320
{
328321
"cell_type": "markdown",
329-
"id": "ef197cd7",
322+
"id": "78362c6c",
330323
"metadata": {
331324
"editable": true
332325
},
@@ -352,7 +345,7 @@
352345
{
353346
"cell_type": "code",
354347
"execution_count": 4,
355-
"id": "4ccc2f65",
348+
"id": "8be1cebe",
356349
"metadata": {
357350
"collapsed": false,
358351
"editable": true
@@ -381,7 +374,7 @@
381374
},
382375
{
383376
"cell_type": "markdown",
384-
"id": "00e279ef",
377+
"id": "e2693666",
385378
"metadata": {
386379
"editable": true
387380
},
@@ -395,7 +388,7 @@
395388
},
396389
{
397390
"cell_type": "markdown",
398-
"id": "c910b3f4",
391+
"id": "bc954d12",
399392
"metadata": {
400393
"editable": true
401394
},
@@ -407,7 +400,7 @@
407400
},
408401
{
409402
"cell_type": "markdown",
410-
"id": "89e6e040",
403+
"id": "6534b610",
411404
"metadata": {
412405
"editable": true
413406
},

‎doc/src/week37/exercisesweek37.do.txt‎

Lines changed: 7 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ y_mean = ?
5959
y_centered = ?
6060
!ec
6161

62-
Fill in the necessary details.
62+
Fill in the necessary details. Do we need to center the $y$-values?
6363

6464
After this preprocessing, each column of $\bm{X}_{\mathrm{norm}}$ has mean zero and standard deviation $1$
6565
and $\bm{y}_{\mathrm{centered}}$ has mean 0. This makes the optimization landscape
@@ -76,7 +76,9 @@ Find the gradients for OLS and Ridge regression using the mean-squared error as
7676

7777
!bc pycod
7878
# Set regularization parameter, either a single value or a vector of values
79-
lambda = ?
79+
# Note that lambda is a python keyword. The lambda keyword is used to create small, single-expression functions without a formal name. These are often called "anonymous functions" or "lambda functions."
80+
lam = ?
81+
8082

8183
# Analytical form for OLS and Ridge solution: theta_Ridge = (X^T X + lambda * I)^{-1} X^T y and theta_OLS = (X^T X)^{-1} X^T y
8284
I = np.eye(n_features)
@@ -120,19 +122,8 @@ num_iters = 1000
120122
# Initialize weights for gradient descent
121123
theta = np.zeros(n_features)
122124

123-
# Arrays to store history for plotting
124-
cost_history = np.zeros(num_iters)
125-
126125
# Gradient descent loop
127-
m = n_samples # number of data points
128126
for t in range(num_iters):
129-
# Compute prediction error
130-
error = X_norm.dot(theta) - y_centered
131-
# Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring
132-
cost_OLS = ?
133-
cost_Ridge = ?
134-
# You could add a history for both methods (optional)
135-
cost_history[t] = ?
136127
# Compute gradients for OSL and Ridge
137128
grad_OLS = ?
138129
grad_Ridge = ?
@@ -148,9 +139,11 @@ print("Gradient Descent Ridge coefficients:", theta_gdRidge)
148139
!ec
149140

150141
=== 4a) ===
151-
Discuss the results as function of the learning rate parameters and the number of iterations.
142+
Write first a gradient descent code for OLS only using the above template.
143+
Discuss the results as function of the learning rate parameters and the number of iterations
152144

153145
=== 4b) ===
146+
Write then a similar code for Ridge regression using the above template.
154147
Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\theta$ values. How would you define a stopping criterion?
155148

156149

‎doc/src/week37/week37.do.txt‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1556,9 +1556,8 @@ print(theta)
15561556
===== Material for the lab sessions =====
15571557

15581558

1559-
1560-
!bblock Material for the lab sessions on Tuesday and Wednesday
1561-
o Exercise set for week 37
1559+
!bblock
1560+
o Exercise set for week 37 and reminder on scaling (from lab sessions of week 35)
15621561
o Work on project 1
15631562
# * "Video of exercise sessions week 37":"https://youtu.be/bK4AEcTu-oM"
15641563
* For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.

0 commit comments

Comments
 (0)