3.0_Tensorflow_Basic_Concepts.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3.0_Tensorflow_Basic_Concepts.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 2",
      "language": "python",
      "name": "python2"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "1T3YlQlAKasq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Start with TensorFlow"
      ]
    },
    {
      "metadata": {
        "id": "UKS4TJuQKasr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**What is TensorFlow?** TensorFlow is a deep learning library recently open-sourced by Google. It is a programming system in which you represent computations as **graphs**. Nodes in the graph are called ops (short for operations). An op takes zero or more **Tensors**, performs some computation, and produces zero or more Tensors. TensorFlow provides primitives for defining functions on tensors and automatically computing their derivatives. \n",
        "\n",
        "\n",
        "Formally, tensors are mulinear maps from vectors spaces to the real numbers. A Tensor can be represented as a multi-dimensional array of numbers. For example, you can represent a mini-batch of images as a 4-D array of floating point numbers with dimensions [batch, height, width, channels].\n",
        "\n",
        "Every operation of machine learning can be writen as a function on tensors\n",
        "\n",
        "**Why TensorFlow?**\n",
        "TensorFlow is quite similar to Numpy, the most important difference is that numpy doesn't offer methods to create tensors as a functions and automatically compute derivatives (also, numpy does not provide GPU support)\n",
        "\n",
        "**TensorFlow Mechanics:**\n",
        "* Prepare the Data\n",
        "    * Inputs and Placeholders\n",
        "* Build the Graph\n",
        "    * Inference\n",
        "    * Loss\n",
        "    * Training\n",
        "* Train The model\n",
        "    * The Session\n",
        "    * The Graph\n",
        "    * Train loop\n",
        "* Evaluate the model"
      ]
    },
    {
      "metadata": {
        "id": "yJ1z5E9eKass",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Simple Numpy Recap"
      ]
    },
    {
      "metadata": {
        "id": "Ui1C31FNKast",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "14a82060-2d82-4960-b2e1-5925386f5305",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877235105,
          "user_tz": -120,
          "elapsed": 1003,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "a = np.zeros((2,2)); b = np.ones((2,2))\n",
        "print np.sum(b,axis=1)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[2. 2.]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "DU7hAMOWKasy",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c3d88a88-f00d-4988-cf3a-4b53fda67b26",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877235677,
          "user_tz": -120,
          "elapsed": 515,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print a.shape"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(2, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "LKBX4lB3Kas0",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7ab9ce74-9044-49f3-e356-f5ba46c88149",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877236213,
          "user_tz": -120,
          "elapsed": 483,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print np.reshape(a,(1,4))"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0. 0. 0. 0.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Y1xzMctdKas3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Let's do the same with TensorFlow"
      ]
    },
    {
      "metadata": {
        "id": "CjQmqu0rKas3",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "317effb4-52a2-4d3e-a9bc-57f702eb9ded",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877237278,
          "user_tz": -120,
          "elapsed": 1016,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vvGYU2xHKas6",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "7eee7f8d-ec39-4b0d-b842-7077bc86686d",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877237830,
          "user_tz": -120,
          "elapsed": 493,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "sess = tf.InteractiveSession()"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IEUpYm9eKas8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "791b1c02-dec7-4bc1-ecbd-339159061778",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877238370,
          "user_tz": -120,
          "elapsed": 471,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "a = tf.zeros((2,2)); b = tf.ones((2,2))"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lXOzvHKYKas-",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c7ad517e-f500-45ea-936e-8a910cc45d7c",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877238938,
          "user_tz": -120,
          "elapsed": 503,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "tf.reduce_sum(b, reduction_indices=1).eval()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2., 2.], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "metadata": {
        "id": "FiVt3L6kKatB",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "613f3de3-1803-4098-f814-c8b099d7a0ee",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877239863,
          "user_tz": -120,
          "elapsed": 870,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "a.get_shape()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([Dimension(2), Dimension(2)])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "metadata": {
        "id": "s2nFBXH8KatD",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3795c8d2-a271-4252-da50-5db58ca83c9a",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877240529,
          "user_tz": -120,
          "elapsed": 609,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "tf.reshape(a, (1, 4)).eval()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0., 0., 0., 0.]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "dDJdOAReKatG",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "08b0c1b6-1860-4559-d4b8-9e4bb44a1d73",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877241050,
          "user_tz": -120,
          "elapsed": 464,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "a = np.zeros((2,2))\n",
        "ta = tf.zeros((2,2))\n",
        "print a\n",
        "print ta"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0. 0.]\n",
            " [0. 0.]]\n",
            "Tensor(\"zeros_1:0\", shape=(2, 2), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "KVA0jfOiKatI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "TensorFlow computations define a **computation graph** that has no numerical value until it is evaluated!"
      ]
    },
    {
      "metadata": {
        "id": "SoZ-rKFlKatK",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "23794e31-0b75-4e4f-fec3-a597033410e0",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877241594,
          "user_tz": -120,
          "elapsed": 474,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print ta.eval()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0. 0.]\n",
            " [0. 0.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "kkVpaRnmKatO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Placeholders \n",
        "TensorFlow provides a placeholder operation that must be fed with data on execution. It will be used to load input data to the model.\n",
        "```python\n",
        "x = tf.placeholder(tf.float32, shape=(2, 2))\n",
        "y = tf.matmul(x, x)\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  print(sess.run(y))  # ERROR: will fail because x was not fed.\n",
        "\n",
        "  rand_array = np.random.rand(2, 2) # we should get data from some training data\n",
        "  print(sess.run(y, feed_dict={x: rand_array}))  # Will succeed.\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "2DlZ5CimKatO",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "6ae0731d-a1cb-485c-dbef-d1983822b1ee",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877242131,
          "user_tz": -120,
          "elapsed": 486,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "x = tf.placeholder(tf.float32, shape=(2, 2))\n",
        "y = tf.matmul(x, x)\n",
        "\n",
        "with tf.Session() as sess:\n",
        "    rand_array = np.random.rand(2, 2) # we should get data from some training data\n",
        "    print(sess.run(y, feed_dict={x: rand_array}))  "
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0.33912265 0.84068996]\n",
            " [0.36637893 0.92535   ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "uSGNoMZGKatQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## ** Session **\n",
        "A Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated. A session may own resources, such as variables, queues, and readers. It is important to release these resources when they are no longer required. \n",
        "\n",
        "Three basic ways to work: \n",
        "\n",
        "1) Using the Session object:\n",
        "```python\n",
        "a = tf.constant(5.0)\n",
        "b = tf.constant(6.0)\n",
        "c = a * b\n",
        "sess = tf.Session()\n",
        "print sess.run(c)\n",
        "sess.close()\n",
        "```\n",
        "\n",
        "2) Using the context manager:\n",
        "```python\n",
        "a = tf.constant(5.0)\n",
        "b = tf.constant(6.0)\n",
        "c = a * b\n",
        "with tf.Session() as sess:\n",
        "  print(c.eval())\n",
        "```\n",
        "\n",
        "3) Using Interactive Session:\n",
        "```python\n",
        "sess = tf.InteractiveSession()\n",
        "a = tf.constant(5.0)\n",
        "b = tf.constant(6.0)\n",
        "c = a * b\n",
        "#We can just use 'c.eval()' without passing 'sess'\n",
        "print(c.eval())\n",
        "sess.close()\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "ldRLT6v-KatQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Variables \n",
        "When you train a model, you use variables to hold and update parameters. Variables are in-memory buffers containing tensors. A tensorFlow variable does not exist until you initialize it, so they must be explicitly initialized and can be saved to disk during and after training. You can later restore saved values to exercise or analyse the model.\n"
      ]
    },
    {
      "metadata": {
        "id": "IfoQbbMFKatR",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "8c4adc93-f2f6-4320-ff66-d6d91916f2a6",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877242825,
          "user_tz": -120,
          "elapsed": 601,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "W1 = tf.ones((2,2))\n",
        "W2 = tf.Variable(tf.zeros((2,2)),name = \"weights\")\n",
        "\n",
        "with tf.Session() as sess:\n",
        "    print(sess.run(W1))\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    print(sess.run(W2))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[1. 1.]\n",
            " [1. 1.]]\n",
            "[[0. 0.]\n",
            " [0. 0.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EocelWWzKatU",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "9fa431a9-0ea1-41db-d3e2-2f13490589b2",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877243376,
          "user_tz": -120,
          "elapsed": 508,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "W = tf.Variable(tf.zeros((2,2)),name = \"weights\")\n",
        "R = tf.Variable(tf.random_normal((2,2)),name = \"random_weights\")\n",
        "\n",
        "with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    print(sess.run(W))\n",
        "    print(sess.run(R))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0. 0.]\n",
            " [0. 0.]]\n",
            "[[-0.17557839  0.9507274 ]\n",
            " [-0.4273468  -0.16205686]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Y3Yxyc4eKatY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Our first TensorFlow program"
      ]
    },
    {
      "metadata": {
        "id": "ZKpjh4-QKatY",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7a3c05b2-2531-4fef-8311-91a5c4ea1fac",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877243905,
          "user_tz": -120,
          "elapsed": 477,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "sess = tf.Session()\n",
        "x = tf.placeholder(tf.float32, shape=(1,3))\n",
        "w = tf.Variable(tf.random_normal([3,3]))\n",
        "y = tf.matmul(x, w)\n",
        "\n",
        "out = tf.nn.relu(y)\n",
        "\n",
        "sess = tf.Session()\n",
        "sess.run(tf.global_variables_initializer())\n",
        "print sess.run(out,feed_dict={x:np.array([[1.0,2.0,3.0]])})\n",
        "\n",
        "sess.close()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0. 0. 0.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "DgKKOIBCKatb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Let's try to do a simple counter"
      ]
    },
    {
      "metadata": {
        "id": "7f9-KHCRKatc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "94bc72f7-31ed-4375-f1a1-afaa2114be9e",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877244442,
          "user_tz": -120,
          "elapsed": 495,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Python code\n",
        "state = 0\n",
        "for _ in range(3):\n",
        "    state +=1\n",
        "    print state"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1\n",
            "2\n",
            "3\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bxX8FfNlKatf",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "5ae9c28e-65a4-45fb-ac25-15bb8e8bbad3",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1528877244968,
          "user_tz": -120,
          "elapsed": 483,
          "user": {
            "displayName": "",
            "photoUrl": "",
            "userId": ""
          }
        }
      },
      "cell_type": "code",
      "source": [
        "# Your tensorflow code"
      ],
      "execution_count": 17,
      "outputs": []
    }
  ]
}