-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
46bcf99
commit a234b93
Showing
1 changed file
with
334 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,334 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "Name: **Sakshi Kakde** \n", | ||
| "UID: **117472448**" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "# Homework 5: Convex Functions \n", | ||
| "\n", | ||
| "Use either the slides, or Boyd $S$ 3.2 as a reference.\n" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "# Problem 1\n", | ||
| "For this problem, you may use any of the results/rules from the lecture slides.\n", | ||
| "\n", | ||
| "### (a) Consider the set of probability functions defined on $\\mathbb{R}$, given by $\\{f| f\\ge 0, \\text{and} \\int f = 1 \\}.$ Is this a convex set?" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "*Your work here* \n", | ||
| "Let $f_1$ and $f_2$ be two points in set $f$. Then, \n", | ||
| "1) $f_1 \\ge 0$ and $f_2 \\ge 0$ \n", | ||
| "2) $\\int f_1 = 1$ and $\\int f_2 = 1$ \n", | ||
| "\n", | ||
| "The line connecting $f_1$ and $f_2$ is $\\lambda f_1 + (1 - \\lambda)f_2$ for $ 0 \\le \\lambda \\le 1$. \n", | ||
| " \n", | ||
| "Let $f_3 = \\lambda f_1 + (1 - \\lambda)f_2$ for some $\\lambda$. \n", | ||
| "\n", | ||
| "Using the above statements, $f_3 \\ge 0$. \n", | ||
| "\n", | ||
| " $\\int f_3 = \\int \\lambda f_1 + (1 - \\lambda)f_2 $ \n", | ||
| "\n", | ||
| " $\\int f_3 = \\lambda \\int f_1 + (1 - \\lambda) \\int f_2 = \\lambda + 1 - \\lambda = 1$ \n", | ||
| "\n", | ||
| "Since $f_3$ satisfies the conditions, it lies within the set $f$. Therefore, $f$ is a convex set. " | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "### (b) Consider the set of $n\\times n$ low-rank matrices $\\{A\\in\\mathbb{R}^{n\\times n} | \\, \\text{rank}(A)\\le k \\},$ where $0<k<n$. Is this set convex? Why?" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "*Your work here* \n", | ||
| "\n", | ||
| "Let $S$ be a set of all matrices satisfying $\\{A\\in\\mathbb{R}^{n\\times n} | \\, \\text{rank}(A)\\le k \\},$ where $0<k<n$.\n", | ||
| "\n", | ||
| "Let $A_1$ and $A_2$ be an element of set S. Therefore, ${rank}(A_1)\\le k1 \\}$ and ${rank}(A_2)\\le k2 \\}$ where $0 < k1, k2 < n$.\n", | ||
| "\n", | ||
| "Let $A_3$ lie on the line joining $A_1$ and $A_2$. Therefore, $A_3 = \\lambda A_1 + (1 - \\lambda)A_2$ for $ 0 \\le \\lambda \\le 1$. \n", | ||
| "\n", | ||
| "For non-zero $\\lambda$, the $rank(A_3)$ will not depend on the $rank(A_1)$ and $rank(A_2)$. \n", | ||
| "\n", | ||
| "For example, if \n", | ||
| "$A_1 = \\begin{matrix}\n", | ||
| "1 & 2 & 3\\\\\n", | ||
| "4 & 5 & 6 \\\\\n", | ||
| "0 & 0 & 0\n", | ||
| "\\end{matrix}$ and \n", | ||
| "$A_2 = \\begin{matrix}\n", | ||
| "0 & 0 & 0\\\\\n", | ||
| "4 & 5 & 6 \\\\\n", | ||
| "7 & 8 & 9\n", | ||
| "\\end{matrix}$\n", | ||
| "The rank of both is 2. However, the $rank(A_1 + A_2)$ is 3. Therefore, this is not a convex set." | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "### (c) Suppose $\\{A_i\\}$ are matrices of identical size. Define the set $$\\{X \\,\\,|\\,\\, \\text{trace}(A_i^T X A_i) \\le 0, \\,\\, \\forall i \\}.$$\n", | ||
| "Is this set of matrices convex?" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "*Your work here*\n", | ||
| "\n", | ||
| "1) $A^T X A$ is a linear operation on X.\n", | ||
| "\n", | ||
| "2) $trace(A^T X A)$ is also a linear operator on X, which can be expressed as $trace(Mx)$\n", | ||
| "\n", | ||
| "We can express this as an intersection of two convex sets: All *X* and *X* that satisfy $\\{X | {trace}(M_i X) \\le 0, \\forall i \\}$. Therefore, it is a convex set." | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "### (d) Let $C \\subset \\mathbb{R}^n$ be a compact (closed and bounded), non-empty convex set. For a scalar $x,$ let C(x) be the set of all points in $C$ whose first coordinate is closer to x than any other point. More precisely: $C(x)=\\{a\\in C \\, |\\, \\forall b \\in C , |a_1-x|\\le |b_1-x| \\}.$ Is $C(x)$ a convex set for any choice of $x$? Why?" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "*Your work here*\n", | ||
| "\n", | ||
| "Let, $x_1, x_2 \\in C(x)$ \\\n", | ||
| "For $C(x)$ to be convex, there has to be a $x_3 \\in C(x)$, such that: $x_3 = \\lambda x_1 + (1-\\lambda)x_2$, where $\\lambda \\in [0,1]$\n", | ||
| "\n", | ||
| "Now, $ |x_{31} - x| = | \\lambda x_{11} + (1-\\lambda)x_{21} - x | $ \n", | ||
| "\n", | ||
| "$ = | \\lambda x_{11} + (1-\\lambda)x_{21} - x + \\lambda x - \\lambda x| = | \\lambda x_{11} - \\lambda x + (1-\\lambda)x_{21} - x + \\lambda x| = | \\lambda (x_{11} - x) + (1-\\lambda)x_{21} - (1-\\lambda)x |$ \n", | ||
| "\n", | ||
| "$ = | \\lambda (x_{11} - x) + (1-\\lambda)(x_{21} - x)| \\le | \\lambda (x_{11} - x)| + |(1-\\lambda)(x_{21} - x)| \\le \\le \\lambda|(b_1 - x)| + (1-\\lambda)|(b1 - x)| \\le |(b1 - x)| $\n", | ||
| "\n", | ||
| "Satisfies the condition. Therefore is a convex set.\n", | ||
| "\n", | ||
| "\n" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "### (e) Let $\\|\\cdot\\|$ be the vector 2-norm. Is the set $C = \\{(x,y)\\, |\\,\\, \\|x\\| \\le y \\}$ convex? Is it a cone? Why? What about the set $C = \\{(x,y)\\, |\\,\\, \\|x\\|^2 \\le y \\}?$ Is this convex? Is this a cone? Here, $x$ is a vector and $y$ is a scalar." | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "*Your work here* \n", | ||
| "\n", | ||
| "1) \n", | ||
| "a) Convexity : \n", | ||
| "Let $(x_1, y)$ and $(x_2, y)$ be two points in $C$. Therefore, $\\,\\, \\|x_1\\| \\le y$ and $\\,\\, \\|x_2\\| \\le y$. For any $\\lambda$, $0 \\le \\lambda \\le 1$, $x_3$ is the point lying on the line joining $x_1$ and $x_2$. Therefore, $x_3 = \\lambda x_1 + (1 - \\lambda) x_2$. \n", | ||
| "$\\,\\, \\|x_3\\| = \\,\\, \\|\\lambda x_1 + (1 - \\lambda) x_2\\| \\le \\,\\, \\|\\lambda x_1\\| + \\,\\, \\|(1 - \\lambda) x_2\\| \\le \\,\\, \\|\\lambda\\|y + \\,\\, \\|(1 - \\lambda)\\|y \\le y$ \n", | ||
| "Therefore, the set is convex. \n", | ||
| "\n", | ||
| "b) Cone : \n", | ||
| "If $X \\in C$, then $aX \\in C$ for a positive constant $a$. \n", | ||
| "Let $X = (x, y) \\in C$. Then $\\,\\, \\|x\\| \\le y$ \n", | ||
| "$aX = (ax, ay)$\n", | ||
| "$\\,\\, \\|ax\\| \\le \\,\\, \\|a\\|\\,\\, \\|x\\| \\le \\,\\, \\|a\\|\\,\\, \\|y\\| \\le a\\,\\, \\|y\\|$ \n", | ||
| "Therefore, $aX \\in C$. Hence, it is a cone. \n", | ||
| "\n", | ||
| "2) \n", | ||
| "a) Convexity : \n", | ||
| "Let $(x_1, y)$ and $(x_2, y)$ be two points in $C$. Therefore, $\\,\\, \\|x_1\\| ^ 2 \\le y$ and $\\,\\, \\|x_2\\|^2 \\le y$. For any $\\lambda$, $0 \\le \\lambda \\le 1$, $x_3$ is the point lying on the line joining $x_1$ and $x_2$. Therefore, $x_3 = \\lambda x_1 + (1 - \\lambda) x_2$. \n", | ||
| "$\\,\\, \\|x_3\\|^2 = \n", | ||
| "\\,\\, \\|\\lambda x_1 + (1 - \\lambda) x_2\\|^2 $ \n", | ||
| "$\\le \n", | ||
| "\\,\\, \\|\\lambda x_1\\|^2 + \\,\\, \\|(1 - \\lambda) x_2\\|^2 + 2 \\,\\, \\|\\lambda x_1\\|\\,\\, \\|(1 - \\lambda) x_2\\| $ \n", | ||
| "$\\le \n", | ||
| "\\lambda^2\\,\\, \\|x_1\\|^2 + (1 - \\lambda)^2 \\,\\, \\|x_2\\|^2 + 2 \\lambda (1 - \\lambda) \\,\\, \\|x_1\\|\\,\\, \\|x_2\\|$ \n", | ||
| "$\\le \n", | ||
| "\\lambda^2 y + (1 - \\lambda)^2 y + 2 \\lambda (1 - \\lambda) y$ \n", | ||
| "\n", | ||
| "$\\le \n", | ||
| "\\lambda^2 y + (1 + \\lambda^2 - 2\\lambda) y + (2 \\lambda - 2\\lambda^2) y$ \n", | ||
| "\n", | ||
| "$\\le \n", | ||
| "\\lambda^2 y + (1 + \\lambda^2 - 2\\lambda) y + (2 \\lambda - 2\\lambda^2) y$ \n", | ||
| "\n", | ||
| "$= \n", | ||
| "y$ \n", | ||
| "Therefore, it is a convex function.\n", | ||
| "\n", | ||
| " \n", | ||
| "b) Cone : \n", | ||
| "If $X \\in C$, then $aX \\in C$ for a positive constant $a$. \n", | ||
| "Let $X = (x, y) \\in C$. Then $\\,\\, \\|x\\|^2 \\le y$ \n", | ||
| "$aX = (ax, ay)$\n", | ||
| "$\\,\\, \\|ax\\|^2 \\le \\,\\, \\|a\\|^2\\,\\, \\|x\\|^2 \\le \\,\\, \\|a\\|^2\\,\\, \\|y\\| \\nleq ay$ \n", | ||
| "\n", | ||
| "Therefore, $aX \\notin C$. Hence, it is not a cone. \n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "\n", | ||
| "\n" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "## Problem 2\n", | ||
| "### (a) Suppose the function $f(x,y)$ is convex. Prove that \n", | ||
| " $$g(x) = \\min_y f(x,y)$$\n", | ||
| " is also convex.\n" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "*Your work here* \n", | ||
| "For a function to be convex, it must satisfy Jensen's inequality. Let $x_1$ and $x_2$ be two points. Let $x_3 = \\lambda x_1 + (1 - \\lambda) x_2$. Then \n", | ||
| "\n", | ||
| "if $g(\\lambda x_1 + (1 - \\lambda) x_2) \\le \\lambda g(x_1) + (1 - \\lambda)g(x_2) $, it is convex. \n", | ||
| "\n", | ||
| "$ g(\\lambda x_1 + (1 - \\lambda) x_2) = \\min_y f(\\lambda x_1 + (1 - \\lambda) x_2,y)$ \n", | ||
| "\n", | ||
| "Given that $f(x, y) is convex$. Therefore, by Jensen's inequality, \n", | ||
| "\n", | ||
| "$f(\\lambda x_1 + (1 - \\lambda) x_2,y) \\le \\lambda f(x_1, y) + (1 - \\lambda)f(x_2, y) $\n", | ||
| "\n", | ||
| "Therefore, \n", | ||
| "\n", | ||
| "$ g(\\lambda x_1 + (1 - \\lambda) x_2) \\le \\min_y (\\lambda f(x_1, y) + (1 - \\lambda)f(x_2, y)) $ \n", | ||
| "\n", | ||
| "$ \\le \\lambda \\min_y (f(x_1, y)) + (1 - \\lambda)\\min_y (f(x_2, y))$ \n", | ||
| "\n", | ||
| "$ \\le \\lambda g(x_1) + (1 - \\lambda) g(x_2)$\n", | ||
| "\n", | ||
| "Therefore, is convex.\n" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "### (b) Suppose $f$ is strongly convex. Show that the gradient of $f$ is strongly monotone, i.e., there is a constant $m>0$ with \n", | ||
| " $$ \\langle x-y,\\nabla f(x) - \\nabla f(y) \\rangle \\ge m\\|x-y\\|^2.$$\n" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "*Your work here* \n", | ||
| "If $f$ is strongly convex then $\\nabla^2f(x) \\ge mI$ $\\forall m > 0,$ and $I$ is identity matrix. \n", | ||
| "From the Taylor series expansion of $f$, we have:\n", | ||
| "\n", | ||
| "$f(y) \\ge f(x) + \\nabla f(x)^T(y-x) + \\frac{1}{2}(y-x)^T\\nabla^2f(x)(y-x)$\n", | ||
| "\n", | ||
| "Similarly we can also have:\n", | ||
| "\n", | ||
| "$f(x) \\ge f(y) + \\nabla f(y)^T(x-y) + \\frac{1}{2}(x-y)^T\\nabla^2f(y)(x-y)$ \n", | ||
| "\n", | ||
| "Adding both \n", | ||
| "\n", | ||
| "$f(y) + f(x) \\ge f(x) + \\nabla f(x)^T(y-x) + \\frac{1}{2}(y-x)^T\\nabla^2f(x)(y-x) + f(y) + \\nabla f(y)^T(x-y) + \\frac{1}{2}(x-y)^T\\nabla^2f(y)(x-y)$\n", | ||
| "\n", | ||
| "$0 \\ge \\nabla f(x)^T(y-x) - \\nabla f(y)^T(y-x) + \\frac{1}{2}(y-x)^T\\nabla^2f(x)(y-x) + \\frac{1}{2}(y-x)^T\\nabla^2f(y)(y-x)$\n", | ||
| "\n", | ||
| "$0 \\ge -(\\nabla f(x) - \\nabla f(y))^T(x-y) + \\frac{1}{2}(y-x)^T\\nabla^2f(x)(y-x) + \\frac{1}{2}(y-x)^T\\nabla^2f(y)(y-x)$\n", | ||
| "\n", | ||
| "$(\\nabla f(x) - \\nabla f(y))^T(x-y) \\ge \\frac{1}{2}(y-x)^T\\nabla^2f(x)(y-x) + \\frac{1}{2}(y-x)^T\\nabla^2f(y)(y-x)$\n", | ||
| "\n", | ||
| "$(\\nabla f(x) - \\nabla f(y))^T(x-y) \\ge \\frac{1}{2}(y-x)^Tm(y-x) + \\frac{1}{2}(y-x)^Tm(y-x)$ \n", | ||
| "\n", | ||
| "$(\\nabla f(x) - \\nabla f(y))^T(x-y) \\ge m(x - y)^T (x - y)$\n", | ||
| "\n", | ||
| "$ \\langle x-y,\\nabla f(x) - \\nabla f(y) \\rangle \\ge m\\|x-y\\|^2.$\n" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "# Problem 3\n", | ||
| " Suppose that $g(x)$ is convex and $h(x)$ is concave (i.e. $-h(x)$ is convex). Suppose we restrict both functions into a closed, convex set $C$ such that both $g(x)$ and $h(x)$ are always positive when $x \\in C.$ Prove that the function\n", | ||
| " $f(x) = g(x)/h(x)$ \n", | ||
| " is quasi-convex. (note: It follows that every local minimum is also global)\n", | ||
| "\n" | ||
| ], | ||
| "metadata": {} | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "source": [ | ||
| "*Your work here* \n", | ||
| "If $f(x)$ is quasi-convex, then its domain($dom(f)$) and all\n", | ||
| "its sublevel sets $S_\\alpha = \\{x\\in dom(f) |\\ f(x) \\leq \\alpha \\}$ are convex, where $\\alpha \\in \\mathbb{R}$ \n", | ||
| "\n", | ||
| "We have $x \\in C$. We have $f(x) = \\frac{g(x)}{h(x)}$. Therefore, $dom(f(x)) \\in C$. Given that $C$ is convex. Hence $dom(f(x))$ is convex.\n", | ||
| "\n", | ||
| "We can have two cases, $\\alpha \\ge 0$ and $\\alpha < 0$. \n", | ||
| "\n", | ||
| "For $\\alpha < 0$: \n", | ||
| "$f(x) = \\frac{g(x)}{h(x)}$. But, it is given that $g(x)$ and $h(x)$ are positive. Therefore, $f(x)$ can never be negative. Thus, $S_\\alpha$ is empty for $\\alpha < 0$. An empty set is a convex set. Thus, $S_\\alpha$ is convex for $\\alpha < 0$. \n", | ||
| "\n", | ||
| "For $\\alpha \\ge 0$: \n", | ||
| "$f(x) = \\frac{g(x)}{h(x)} \\le \\alpha$ \n", | ||
| "$g(x) \\le \\alpha h(x)$ \n", | ||
| "$g(x) - \\alpha h(x) \\le 0$ \n", | ||
| "$g(x)$ and $-h(x)$ are convex and $\\alpha$ is positive. Therefore, $g(x) - \\alpha h(x)$ is convex. Any convex function has convex sublevel sets. Therefore, Thus, $S_\\alpha$ is convex for $\\alpha \\ge 0$. \n", | ||
| "\n", | ||
| "Since $S_\\alpha$ is convex, $f(x)$ is quasi-convex." | ||
| ], | ||
| "metadata": {} | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.7.6" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |