Copy_of_Deep_Learning_Notes.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Deep Learning Notes.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sudarshanswain/Siebel_code/blob/master/Copy_of_Deep_Learning_Notes.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ujELF2qppkH",
        "colab_type": "text"
      },
      "source": [
        "#Day 1 - 31 August"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UscalBu-tFXf",
        "colab_type": "text"
      },
      "source": [
        "# Maintainers \n",
        "# SPOC \n",
        "~ 3-4 \n",
        "\n",
        "-- "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tTSV3iBcrJ8t",
        "colab_type": "text"
      },
      "source": [
        "## Structure \n",
        "     -- Open Questions ( Discussions on 10+ yrs exp job profile ) \n",
        "     -- Unanswered Question (  )\n",
        "     -- Answered Question \n",
        "     -- Links to Resource \n",
        "     -- Topic Covered : \n",
        "        -- \n",
        "    -- Hands-on (Assignments) \n",
        "        - Individual Notebook links \n",
        "        - Ashsih, \n",
        "        - Sachin, \n",
        "        \n",
        "\n",
        "    -- Github/Extra Codes \n",
        "\n",
        "      "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ojrVS9Hzj651",
        "colab_type": "text"
      },
      "source": [
        " -- Links shared by Lavi to code using colab notebook\n",
        "       https://colab.research.google.com\n",
        " -- Link shared by Sonya for the sample notes preparation from previous batch          https://colab.research.google.com/drive/1ByHZ8Qzd05_GbkM5U7ZgVxF1whwSwZo6#scrollTo=PhiLqiyj_0DT\n",
        "    \n",
        "\n",
        "     *   Concept Discussed on H   C    A\n",
        "     *   H- Human mind is thinking\n",
        "     *   C - How normal computational process happens\n",
        "     *   A - How ML/DL(AI) algos are learning from human and implementing them when comes to scaling / huge data\n",
        "\n",
        " -- Interview Question - Unanswered\n",
        "   What is the default number of iteration happens in boosting algo? Can we control the number of iteration -- Amazon Interview Question\n",
        "(XGBoost)\n",
        "   -- from lavi to All Participants to go through the podcosts on regular basis\n",
        "https://www.youtube.com/user/lexfridman/videos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmLTfZaNpvA_",
        "colab_type": "text"
      },
      "source": [
        "#Day 2 - 1 st September"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PkMwUhA_px6P",
        "colab_type": "text"
      },
      "source": [
        "   -- Inputs on notes preparation\n",
        "\n",
        "   -- Monthly meeting to share some topic and support each other\n",
        "\n",
        "   -- People within same City to meet twice in a month \n",
        "      1. Discuss the Issues during session / topics , doubt clearance\n",
        "      2. Networking \n",
        "\n",
        "   -- Regarding Lex Fridman PodCosts, go through the below 2 videos \n",
        "      Elan Musk, Yann LeCun\n",
        "\n",
        "\n",
        "   -- from lavi to All Participants:\n",
        "      https://pyvideo.org/pydata-london-2019/a-new-approach-to-dataframes-and-pipelines.html\n",
        "   -- from lavi to All Participants:\n",
        "      https://github.com/vaexio/vaex\n",
        "\n",
        "   -- Lavi Suggested to  create a calender for pyvideo within 1 or 2 weeks\n",
        "\n",
        "   -- from lavi to All Participants:\n",
        "      https://colab.research.google.com/drive/1sxmUaW_gKTjVRkOW711_l-YZFvxjym9H\n",
        "\n",
        "   -- Topics Discussed\n",
        "   H - how human is solving ? Generalization\n",
        "      \n",
        "   C - Computational without complexity  (Scaling), how complex it is to       use category 'C' to identify images and come up with the result\n",
        "      \n",
        "   A - When huge automation involved, AI is involved, DL will be at the backend. ML at the lower level\n",
        "      (Generalization and Scaling)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4WII0lP5ht6M",
        "colab_type": "text"
      },
      "source": [
        "  -- from lavi to All Participants:\n",
        "       https://deepmind.com/blog/article/alphago-zero-starting-scratch\n",
        "\n",
        "  -- from Akhtar Rizvi to All Participants:\n",
        "       https://www.tylervigen.com/spurious-correlations\n",
        "\n",
        "  -- Research topics suggested by Lavi\n",
        "       FinTech, EdTech, LogisTech, Hospitality\n",
        "\n",
        "  --  om Venkataraju to All Participants:\n",
        "       https://drive.google.com/open?id=1ByHZ8Qzd05_GbkM5U7ZgVxF1whwSwZo6\n",
        "\n",
        "  --  from Akhtar Rizvi to All Participants:\n",
        "       www.datarobot.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tx3Ox8Yj7dGO",
        "colab_type": "text"
      },
      "source": [
        "  # **Week 3 - 7/8 th September**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N5mS5a438HL6",
        "colab_type": "text"
      },
      "source": [
        "## Day 4 - 7th September\n",
        "links shared by Lavi  - 7th September \n",
        "  \n",
        "  https://colab.research.google.com/drive/11o8ArSSvVnpAUc8G_Mzu1d4uhKi6ksI9\n",
        "  https://colab.research.google.com/drive/191IyzQ7SJdMrcPmemScFuElYOq7s4xYJ#scrollTo=uCPLJ521KUVj\n",
        "  \n",
        "  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ywBu7i_8cPCh",
        "colab_type": "text"
      },
      "source": [
        "### Walkthrough of PPT\n",
        "\n",
        "How DL algos work?\n",
        "  -- adding weights and bias to each layer\n",
        "  -- Minimize the diff between actual Vs Predicted\n",
        "  -- can loss fn be defined by user\n",
        "  -- loss fn is cost of error can be calculated to find the loss\n",
        "\n",
        "  -- Loss fn will be row level \n",
        "  -- cost fn at summary level\n",
        "\n",
        "  -- loss Vs weights - parabolic\n",
        "\n",
        "  -- keep on minimizing the loss fn in iterative manner such that weights associated are also changing in order to minimize the loss fn\n",
        "\n",
        "\n",
        "  -- Data Perspective  - We try to plot the Cost Vs Weights graph which is parabolic\n",
        "  -- Gradient Descent  \n",
        "\n",
        "1.   Gradient - Differentiation at the point\n",
        "2.   Descent - Depression  (direction to proceed)\n",
        "\n",
        "\n",
        "  -- iteration to find x \n",
        "         x1 = x0 + n(d(f(x0)/d(x))  ; n- learning rate\n",
        "\n",
        "  Linear Regression - 3 types\n",
        " --Ordinary least square method - to minimize predicted Vs actual loss\n",
        " \n",
        " --Matrix Multiplication - linear equation to find X value\n",
        " \n",
        " --Gradient descent - find the weights and bias value\n",
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pWQ3VrAMiTcG",
        "colab_type": "text"
      },
      "source": [
        " -- Tensor flow setups hands on demonstrated\n",
        "\n",
        " -- Select Runtime, Change runtime type python3 GPU\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wC6xnKaWcwtf",
        "colab_type": "text"
      },
      "source": [
        "## Day 5 - 8th September\n",
        "\n",
        "Links shared by Lavi - 8th September\n",
        "\n",
        "\n",
        "TensorFlow Federated \n",
        "https://www.tensorflow.org/federated\n",
        "\n",
        "- Swiggy works on Federated learning \n",
        "- Decentralised way of learnings \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xnAOCCsBdJBO",
        "colab_type": "text"
      },
      "source": [
        "Notebook on Colab for practice - Boston Housing Prices Normalized : https://colab.research.google.com/drive/1sxmUaW_gKTjVRkOW711_l-YZFvxjym9H"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LcCuShzWeW5I",
        "colab_type": "text"
      },
      "source": [
        "Python file on Colab that was used in the class :\n",
        "**Building a Linear Regression Model**\n",
        "Tensorflow_Boston_Housing_Prices_Normalized.ipynb\n",
        "https://colab.research.google.com/drive/191IyzQ7SJdMrcPmemScFuElYOq7s4xYJ\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HMIXEhq3Ylj1",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "# **Week 4 - 14/15 th September**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gZL50zrWYsEu",
        "colab_type": "text"
      },
      "source": [
        "## Day 6 - 14th September\n",
        "\n",
        "#### Classification\n",
        "\n",
        "\n",
        "Revision on Boston Pricing - Using Tensor Flow\n",
        "\n",
        "Learing rate - how do we decide the learning rate \n",
        "\n",
        "we are not decidnig ...we keep on changing the parameters of LR and epochs to get the minimum loss and in the process we will arrive at a better combination \n",
        "\n",
        "from lavi to All Participants:\n",
        "http://hyperopt.github.io/hyperopt/\n",
        "\n",
        "#### what are different ways of doing hyperparameter optimization - Interview question\n",
        "---- 1 .Gridsearch\n",
        "\n",
        "---- 2. hyperopt using Bayesian optimization technique\n",
        "\n",
        "L1, L2 - Regularization to investigate the parameter does not lead to loss\n",
        "\n",
        "\n",
        "Epoch network is set of data to train the model  - like CV, variations like k-fold, stratified k-fold\n",
        "\n",
        "varations will be there , how to finailize ?\n",
        "average the distribution to arrive at accuracy in regressive models\n",
        "\n",
        "--- Early stopping  -  Telling model to stop earlier without considering the Epoch value as there is no significant improvement in the reduction of loss fn\n",
        "\n",
        "DL uses Multi model with ensampling technique in multi layers, \n",
        "\n",
        "stacking - output of one model to input of other model, goes from complicate to easy model\n",
        "\n",
        "In Multineuron n/w the Activation Functions uses sigmoid function in the neural network models where x1 is sent as i/p to first layer which is x1w1+b \n",
        "First layer will have i/p like x1w1+b, x1W2+b, x1w3+b etc\n",
        "\n",
        "common activation fun - logistic , in other words sigmoid functions - they are always non-linear\n",
        "\n",
        "#### Keywords / Terms to remember\n",
        "* Number of layers \n",
        "\n",
        "* Number of neurons in each hidden layer\n",
        "\n",
        "* Activation function - sigmoid function\n",
        "\n",
        "* Epochs - passes we let our data make an optimized algo\n",
        "\n",
        "* Batch size - set of data to be passed to Epoch (10 in 1000)\n",
        "\n",
        "* Optimization  algorithm - Gradient Descent\n",
        "\n",
        "* Hidden Layer\n",
        "\n",
        "\n",
        "Epoch -> How many passes should I make for my data to reach an optimization\n",
        " algorithm /  how many times data can be exposed\n",
        "\n",
        "from Padmasri to All Participants:\n",
        "https://www.datarobot.com/wiki/deep-learning/\n",
        "\n",
        "\n",
        "iteration 1 - randomly take  weights and bias\n",
        "\n",
        "iteration 2 - calculate the w2, and b2 to minimize the loss applying Gradient descent algo\n",
        "\n",
        "iteration 3 - the selection of weight and bias (w3, b3) considering w2 and b2 to reduce the loss till we find the value of Wi and Bi where it reaches 0 or local minima \n",
        "and so on, \n",
        "\n",
        "##### Please refer the below link for further iteration wise details \n",
        "\n",
        "From lavi to All Participants:\n",
        "https://colab.research.google.com/drive/1felNAT_gtYIjak77KSsMf-zZIJ4OozHb\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9tkK9gdIbfDN",
        "colab_type": "text"
      },
      "source": [
        "## Day 7 - 15th September\n",
        "\n",
        "#### Classification - Tensor Flow\n",
        "\n",
        "Loss function - func which shows the % of far away from the accuracy\n",
        "optimization function - func which uses to optimize the parameters of the loss func\n",
        "Epochs - its the number of passes that we allow the data to go through iteration\n",
        "\n",
        "\n",
        "Iteration 1 - Randomly calculates the weight and bias, i.e w1, b1 to find loss\n",
        "\n",
        "Iteration 2 - Use w1 , b1 to minimize the loss, in other words it picks up the earlier predicted value and adjust the weights and bias to be able to minimize the loss func\n",
        "\n",
        "and so on, \n",
        "our agenda is to minimize the loss at every iteration.\n",
        "\n",
        "Learning rate - the rate how it proceeds to reach local minima\n",
        "\n",
        "Based on initial random w1, b1 we will reach local minima using Gradient descent. Can we check with another random value and try to attempt another local minima.\n",
        "\n",
        "\n",
        "Stochastic Gradient Descent - picks diff values \n",
        "Batch Size also will have same effect\n",
        "\n",
        "Dense Layer = (Activation Function +Neurons +Hidden Layer) or something different\n",
        "\n",
        "Each Neuron is an activation function\n",
        "\n",
        "HL - combination of neurons, where each neuron performs an activation function \n",
        "\n",
        "#### Interview Question\n",
        "\n",
        "Why is it important to have Randomness ?\n",
        "\n",
        "Randomness is better than our own biased values.\n",
        "\n",
        "All ML algos use random seed to display the same output each time.\n",
        "\n",
        "#### Questions Raised : \n",
        "\n",
        "* one neuron of hidden layer can have more than one activation func? no. single activation func\n",
        "\n",
        "* Is there any criteria to define no of hidden layer - depends on use case\n",
        "\n",
        "* I guess activation and optimizer will be applied together for each epoch - yes \n",
        "\n",
        "* Is every bit of input used to get a bit of hidden layer?- yes, tightly connected layer\n",
        "\n",
        "* Is the random selection of w and b done in each hidden layer or once in the first hidden layer? - no, only in first layer\n",
        "\n",
        "* here we have 5 model at the output level, do we calc average -- 5 clauses not models\n",
        "\n",
        "#### Difference between ML and DL \n",
        "\n",
        "* DL - Each layer will do feature selection on its own.\n",
        "\n",
        "* ML - Feature Engineering to be done by us\n",
        "\n",
        "In DL, each Hidden Layer performs feature extraction \n",
        "\n",
        "classification - needs probability application, linear algo can't give capping range.\n",
        "\n",
        "#### softmax function is used for multiclassification (like sigmoid fn)\n",
        "\n",
        "\n",
        "Loss fn - in linear classification \n",
        "\n",
        "Actual value - log(predited)\n",
        "\n",
        "Cross entrophy model -  log value should be minimum, i.e. smaller the loss , model is doing better\n",
        "\n",
        "e=1;randomness = max\n",
        "e=0;randomness = min\n",
        "\n",
        "* Entropy - degree of randomness, \n",
        "* Cross Entrophy - compare across the models\n",
        "\n",
        "use case - MNIST dataset to identify the handwritten digits and predict them\n",
        "\n",
        "Image is represented by 28x28 pixels\n",
        "\n",
        "fully connected n/w - all neuraons are connected with each other\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://colab.research.google.com/drive/1h6VWSH0YahNKdjwoE5ZGJ6ytbNRO6dEA\n",
        "\n",
        "default graph - workflow\n",
        "\n",
        "what is lazy execution - session run need to be done in TF1.14 and TF1.16\n",
        "\n",
        "#### Interview Question\n",
        "\n",
        "What is the diff between newer and older version?\n",
        "\n",
        "Tensor Flow - 2 onwards approach will be diff to include both lazy and interpretor-driven execution \n",
        "\n",
        "0-255 digits could be considered as categorical for ont-hot encoding.\n",
        "\n",
        "DL Classification MNIST NB:\n",
        "https://colab.research.google.com/drive/1h6VWSH0YahNKdjwoE5ZGJ6ytbNRO6dEA"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hU7RCYgd1Q2Q",
        "colab_type": "text"
      },
      "source": [
        "## Day 8 - 21st September\n",
        "\n",
        "\n",
        "Discussed on the project that already Lavi is working, Anyone interested to join, can communicate to DSSupport .\n",
        "\n",
        "##### Interview tips - advised to apply and appear for interviews\n",
        "\n",
        "\n",
        "\n",
        "1.   Advised to appear for interview even before the completion of course\n",
        "2.   Resume preparation with DS projects - Algorithms, Process, ongoing projects / course\n",
        "3.   Apply through LinkedIn, StackOverflow, Indeed\n",
        "\n",
        "https://stackoverflow.com/jobs\n",
        "\n",
        "\n",
        "##### Topic Covered : Classification Neural Network - Runthrough\n",
        "\n",
        "* how do we decide the no of layers to be considered?\n",
        "its based on hyperparameters\n",
        "\n",
        "* hidden layer have same activation fn,\n",
        "but output layer might have different activation fn\n",
        "\n",
        "* Total no. of layers in any neural network does not include o/p layer\n",
        "\n",
        "* Adding more layers may / may not give optimization for the model to learn\n",
        "\n",
        "* Is the loss fn at every layer?\n",
        "only at the end - output layer\n",
        "\n",
        "\n",
        "** Logging - During runtime, log into will get saved on specific folder\n",
        "By default /content will be default folder (colab)\n",
        "\n",
        "\n",
        "!mkdir sample -- create a newfolder\n",
        "\n",
        "logs_path = '/content/sample'\n",
        "\n",
        "* ckpt - chk point\n",
        "even during training something happened , we can start from the chkpoint\n",
        "\n",
        "* Based on batchsize, we send the data to Epoch, instead of sending complete data\n",
        "\n",
        "for each_Epoch on totalEPochs:\n",
        "    for each_batch_of_data in total_Batch:\n",
        "        Calculate weight and loss\n",
        "        \n",
        "* K, L, M , N are the number of hidden layers with neurons\n",
        "\n",
        "* placeholders - to define the data type with size, initialized during runtime\n",
        "\n",
        "* NN starts with denser number of neurons and towards the end, it is less\n",
        "\n",
        "* Each neuron needs W, B, X and also an activation fn\n",
        "\n",
        "* According to the coding, \n",
        "* TF - though Python based, uses the concept of C++ / Java for inititalization\n",
        "\n",
        "* Keras- It's Python and does initialize the data strcture dynamically\n",
        "(Pytorch - same as Keras)\n",
        "##### Definition of Layers, Loss, GD \n",
        "* input Layers\n",
        "* Hidden Layers\n",
        "* Output Layer \n",
        "* Loss Function  -  Sigma Yi.log (Yi)\n",
        "* Gradent Descent Optimizer  - uses learing rate to minimize the cross entrophy defined in Loss Function\n",
        "\n",
        "\n",
        "###### google tpot gitgub\n",
        "\n",
        "GCP, Azure, AWS provide AutoML options\n",
        "\n",
        "https://www.neuralink.com\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "op-Ba3mb-Qcg",
        "colab_type": "text"
      },
      "source": [
        "## Day 9 - 22nd September\n",
        "\n",
        "##### DL Concepts , classification using DNN -Execution of graph\n",
        "\n",
        "\n",
        "##### MNIST Classification Data - Walkthrough\n",
        "\n",
        "##### Revision\n",
        "\n",
        "* HL - layers between i/p and o/p\n",
        "\n",
        "* HL - comprises of X, W, B and Activation function\n",
        "\n",
        "* use of activation fn  - neutralizes the input values between 0 and 1\n",
        "\n",
        "* user defined- number of HL and the no of neurons\n",
        "\n",
        "* softmax - in the o/p layer does both loss fn and  multiclassification process\n",
        "\n",
        "* learning rate - step to take to reach the local minima by reducing the error\n",
        "\n",
        "\n",
        "Padmasri https://colab.research.google.com/drive/1PHYNOE4fUyjdNdebUyrjhM-MW7zUs44P\n",
        "\n",
        "* Dont we use LRFinder method ? \n",
        "\n",
        "* To explore keras with gridsearch use below links\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://www.kaggle.com/shujunge/gridsearchcv-with-keras\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/\n",
        "\n",
        "\n",
        "from Venkataraju to All Participants:\n",
        "https://colab.research.google.com/drive/1PHYNOE4fUyjdNdebUyrjhM-MW7zUs44P\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://colab.research.google.com/drive/1it23ZfK1QJdam4_fCowc9Ok75BC6bGL6\n",
        "\n",
        "\n",
        "##### Assignment, 5 values of LR and Batch size to check the accuracy\n",
        "\n",
        "Event file created whenever the graph executed\n",
        "\n",
        "* Gradient Descent  graph,   Cost(y) Vs Weights(x)\n",
        "\n",
        "* Regression y = wx+b  (Loss fn : RMSE, MSE) sigma(p-a)^2\n",
        "\n",
        "* Classification y = SoftMax( Wx + B )  \n",
        "Loss fun : SM - cross entrophy -( -Sigma y * log y )\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "from Sandeep to All Participants: (Boston)\n",
        "\n",
        "https://colab.research.google.com/drive/1felNAT_gtYIjak77KSsMf-zZIJ4OozHb\n",
        "\n",
        "###### Revision on Boston\n",
        "\n",
        "\n",
        "Following is our Graph just we did --> https://drive.google.com/file/d/1AuW9HuHVYvUiU092UlaNkuRuhaOu7IbV/view?usp=sharing \n",
        "\n",
        "* Tensorboard is used to plot the graph\n",
        "* Fully connected - every neuron is connected to other neurons directly or indirectly\n",
        "\n",
        "* Activation fn are more important element of NN , \n",
        "\n",
        " -- they normalize the data \n",
        " \n",
        " -- they help to learn and extract the inherant data such that they can used to do classification at the output layer\n",
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-BBopHWBLhzR",
        "colab_type": "text"
      },
      "source": [
        "## Day 10 - 28-Sep-19\n",
        "\n",
        "* why is it that no of neurons is not same across input,hidden and output layers? - Comprehensive in learning\n",
        "* RNN would be better choice for speech recognition\n",
        "* Regressive output layer will have 2 neurons\n",
        "\n",
        "* DL - how would we know the hyperparameter setting for any topic given ?\n",
        "Ideally we do the test scenario and do experiment.\n",
        "based on the use case, we choose the parameters.\n",
        "will check the research paper happened in that area, then will optimize the parameters, will check the blogs\n",
        "\n",
        "#### Walkthrough of Code(Execute the graph)\n",
        "\n",
        "\n",
        "* Regarding mnist image data, how to view input data files which are in ubyte format.\n",
        " --  to see pixel value trainX[0]\n",
        " \n",
        " -- to view the images \n",
        " \n",
        " from lavi to All Participants:\n",
        "http://yann.lecun.com/exdb/mnist/\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://pypi.org/project/Pillow/\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://stackoverflow.com/questions/42353676/display-mnist-image-using-matplotlib\n",
        "\n",
        "* Activation fn - neutralizes the input values between some range\n",
        "\n",
        "#### Sigmoid Activation\n",
        "1. sigmoid fn - 1/(1+e^-x) \n",
        "2. ranges between 0 and 1  \n",
        "3. advantage - simple algo\n",
        "4. disadvantage - it has vanishing gradient issue\n",
        "\n",
        "#### Vanishing Gradient - While training, the gradient based optimizers, will make the values of parameters in the network (W, B) close to zero quickly. \n",
        "\n",
        "* Why is it issue it reaches 0 - It may not learn larger data values\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://medium.com/@anishsingh20/the-vanishing-gradient-problem-48ae7f501257\n",
        "\n",
        "#### TANH Hyperbolic Tangent Function\n",
        "1. TanH - -1 and +1\n",
        "2. it tries to avoid the faster convergence of 0 and capping is set to -1\n",
        "3. Disadv - still have vanishing gradient issue\n",
        "\n",
        "#### ReLu - Rectified Linear Unit\n",
        "1. we go for Relu to avoid the issues in Sigmoid and TanH\n",
        "all negative values becomes zero, +ve as it is,\n",
        "range is 0 to x, x>0\n",
        "f(x) = max(0,x)  \n",
        "\n",
        "2. No vanishing gradient\n",
        "3. Accuracy is better and  it is faster also since no convergence\n",
        "4. 6 times faster than Tanh\n",
        "5. In some cases, -ve values are important and ReLu considers them as 0 which results in dead neuron\n",
        "\n",
        "#### Leaky ReLu - Leaky Rectified Linear Unit\n",
        "\n",
        "1. All positive values as it is, negative input will have small value\n",
        "2. Avoids dead neuron issue\n",
        "3. not all negative values, few values are normalized\n",
        "4. slower than ReLu , faster than TanH\n",
        "\n",
        "we are close to f(y)=y for both -ve & +ve range, then why don't use f(y)=y itself for both ranges ? :-)\n",
        "\n",
        "f(y) = ay for -ve\n",
        "\n",
        "f(y) = y\n",
        "\n",
        "\n",
        "from akram ahmad to All Participants:\n",
        "https://towardsdatascience.com/how-to-train-neural-network-faster-with-optimizers-d297730b3713\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://research.fb.com/category/natural-language-processing-and-speech/\n",
        "\n",
        "\n",
        "Formulas used during ML https://drive.google.com/file/d/0B0RLknmL54khQlhGUzFUWEtncTA/view\n",
        "\n",
        "\n",
        "#### Optimizers\n",
        "\n",
        "##### GD - Disadv - Choosing the learning rate with momentum, may reach local minima but not global minima\n",
        "\n",
        "##### SGD - variance of GD, randomly picks up the learning rate. Accelerates the SGD in the direction with less oscillations\n",
        "\n",
        "##### Nesterov Accelerated Gradient (NAG, variant of GD) - Capping the learning rate, in case very close to minima reducing the momentum so that randomness will be less in case not to miss out the minima , if it is further to minima, randomness will be more. \n",
        "* How does it know the closeness - steepness of the differentiation curve\n",
        "i.e high steepness - randonmess will be high \n",
        "\n",
        "##### Ada Grad - In Adagrad we adopt the learning rate to the parameters, Adagrad eliminates the need to manually tune the learning rate.\n",
        "\n",
        "##### AdaDelta - Adadelta is an extension of Adagrad and it also tries to reduce Adagrad’s aggressive, monotonically reducing the decaying learning rate\n",
        "\n",
        "####NOTE : If learing rate is figured, SD is the best among all algos\n",
        "\n",
        "##### Adam - Adaptive Moment Optimizer  - Initially takes random and learns from the previous iteration. Thus decaying learning rate is avoided and converges very fastly. Adam implements the exponential moving average of the gradients to scale the learning rate instead of a simple average as in Adagrad. It keeps an exponentially decaying average of past gradients\n",
        "\n",
        "\n",
        "\n",
        "* Normally Activation fn and optimizer combination would be ReLu and Adam\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jacnLIzV-bIi",
        "colab_type": "text"
      },
      "source": [
        "## DAY 11 - 29-Sep-19\n",
        "\n",
        "---\n",
        "#### Revision\n",
        "\n",
        "* AdaGrad - figures the next ieration  (Adaptive learning rate)\n",
        "\n",
        "* Adam - it combines adaptive with stochastic gradient able to get global minima\n",
        "\n",
        "* which one better - Adagrad and Adam\n",
        "\n",
        "In general ReLu and Adam is better, but for use case, we need to do some acdemic research then decide on the AF and Optimizer combo\n",
        "\n",
        "#### Interview Questions\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://colab.research.google.com/drive/1PtT0BrlWuieoFOvru5xHOEZDkSQSzOc-\n",
        "\n",
        "* ckpt - tf.model\n",
        "\n",
        "CNN - special neural n/w to handle images (Convolution Neural Network)Keras\n",
        "\n",
        "* How to represent color Vs grayscale image \n",
        "\n",
        "   * 3 dimensional tensor is used for color image representation\n",
        "   \n",
        "* Components of CNN\n",
        "    * Input\n",
        "    * Convolution\n",
        "    * Pooling\n",
        "    * Flatten\n",
        "    * FC Layer\n",
        "    * Output \n",
        "\n",
        "##### CNNs (convolutional neural networks) essentially have three parts, convolution layers, pooling layers, and fully-connected layers. It usually takes a 2D (sometimes more dimensions) matrix and outputs a result.\n",
        "\n",
        "* Convolution starts at the top left and takes a small window with a certain width and height and performs an operation on that, the operation is usually a matrix multiplication where the matrix to multiply by is decided via gradient descent to get the best final results. It then moves according to a stride parameter. It does this all the way across the image and outputs a new image.\n",
        "\n",
        "* Pooling is similar in the sense that it breaks the image down using small windows; (average, max, or min) where higher dimensional value is reduced into lower dimensional value\n",
        "\n",
        "* No activation fn is used in the max-pooling layer.\n",
        "\n",
        "* After a set amount of convolutions and pooling, the final output is put through a fully connected layer, which is a conventional feed forward neural network to output a result.\n",
        "\n",
        "##### Interview Question - Do you think CNN can be used other than image processing? \n",
        "* yes, can be used for NLP\n",
        "\n",
        "* Text data processing\n",
        "* Streaming continuous data\n",
        "\n",
        "Each neuron is represented as pixel.\n",
        "\n",
        "Parameters in convolution layer\n",
        "\n",
        "* Filter - is a Matrix, at the specific point of time convolues the input image and produces compressed image\n",
        "* Filter size - dimension of the filter (e.g. 3X3)\n",
        "* Filter stride - The amount by which the filter shifts is the stride.\n",
        "For dense image, stride rate should be less (e.g. 2X2)\n",
        "* Padding - Padding is to add extra pixels outside the image. And zero padding means every pixel value that you add is zero.\n",
        "\n",
        "* Pooling layer - used to reduce the spatial dimension and has 3 diff types of performing the redution process\n",
        "  * Max, Min, Avg\n",
        "\n",
        "* Flatten layer - Pooling layer's multidimensional output is flattend into single dimension which is the input for FC layer\n",
        "\n",
        "* Fully connected Layer - Each neuron is connected to every other neuron. Activation fn ReLu or sigmoid is used.\n",
        "\n",
        "* Output Layer - Softmax(multiclassification) is used for loss functionality in output layer. \n",
        "\n",
        "* VGGNet, Resnet are some samples for comples CNN architecture.\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://openai.com/blog/better-language-models/\n",
        "\n",
        "from lavi to All Participants:\n",
        " https://www.youtube.com/watch?v=AgkfIQ4IGaM\n",
        "\n",
        "written by ML\n",
        "openai.com\n",
        "\n",
        "Boston Robotics\n",
        "\n",
        "https://www.youtube.com/watch?v=Jy9-aGMB_TE&feature=youtu.be\n",
        "\n",
        "https://colab.research.google.com/drive/1sxmUaW_gKTjVRkOW711_l-YZFvxjym9H#scrollTo=hU7RCYgd1Q2Q\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7yVCwD3aKiCF",
        "colab_type": "text"
      },
      "source": [
        "## Day 12- 05-OCT-19\n",
        "#### Classification of MNIST algorithm Hands On Part 1 - Keras \n",
        "\n",
        "#### Revision on CNN\n",
        "\n",
        "Conv layer --> Pooling --> Flatten --> FC Layer --> output layer\n",
        "\n",
        "\n",
        "1.   convolution layer\n",
        "\n",
        "* Convolution layer - Analyse the feature and compress the image\n",
        "In 28X28 image, we take filter size at any given moment, consider 3X3 matrix which moves across the image pixelwise and produces compressed image. \n",
        "\n",
        "* If the matrix multiplication of filter and portion of input image represented in pixels gives the output which has got important features, the activation fn will label with important value.\n",
        "\n",
        "#### How to define fliter value\n",
        "\n",
        "* CNN does Automatically\n",
        "\n",
        "* Once you decide/define the filter size(may be 5X5), we randomly initialize the weight of the filter and allow back propagation algorithm to learn weights automatically.\n",
        "\n",
        "* CNN work as trainable feature extractor, for more detailed content based on larger number of pixels shall require bigger filter sizes. But for cases where localized differences are to receive greater attention, smaller filter sizes are required.\n",
        "\n",
        "* Stride can/can't be used to overlap the filter area of image.\n",
        "\n",
        "#### How does the stride value affects the feature selection, meaning the larger or smaller stride will affect the  features? is it larger or smaller\n",
        "\n",
        "* larger stride will produce more compression \n",
        "\n",
        "#### Why padding is used?\n",
        "\n",
        "* unimportant area of image, like boundary\n",
        "* all input images won't have same resolution, so make sure the resolution are similar. Like if the image is lesser than 720p, padding should be done to match with the 720p by adding zeros.\n",
        "\n",
        "\n",
        "2.   Pooling\n",
        "\n",
        "* pooling is the second level compression where the different options are available max, min or average pooling\n",
        "\n",
        "* conv layer output is taken and compressed to produce the output in pooling layer\n",
        "\n",
        "* filter value (identity matrix of size) are taken to compress the image further\n",
        "\n",
        "\n",
        "3.   Flattening\n",
        "\n",
        "* Next layer FC need the input in a flattened way. Flattening into single dimension\n",
        "\n",
        "4.   FC \n",
        "\n",
        "*  where the input from the other layers is flattened and sent so as the transform the output into the number of classes as desired by the network.\n",
        "* Primarily, the fully connected layer does categorization whereas convolution layers are filters. They transform the data into something easier to categorize. \n",
        "* The filter value, weights and bias are adjusted in this layer based on the activation fn to minimize the loss function.\n",
        "\n",
        "5.  Output Layer\n",
        "\n",
        "* This layer contains softmax activation function, which outputs a probability (a number ranging from 0-1) for each of the classification, which is the model prediction.\n",
        "\n",
        "### The Overall Training Process\n",
        "   1 Initialize all filters values, parameters and weights with random values.\n",
        "   2 Apply Convolution modules followed by the Fully-Connected Layer(s) and obtain final class probabilities.\n",
        "   3 Calculate the total output error. (At first this will be high because we started with random values)\n",
        "   4 Use backpropagation and optimizer fn like ReLu to update filter values and weights so that the output error is minimized. \n",
        "   5 Note that the parameters like filter size and number of filters do not get updated – they are to be specified at the beginning of the training process.\n",
        "   6 Steps 2-4 are continued in a loop until the output error reaches the desired value.\n",
        "\n",
        "\n",
        "Alex Net, VGGNet - CNN samples \n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://arxiv.org/abs/1808.05377\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://colab.research.google.com/drive/1B5j5XJ3BsXPw3VgMcnfkZYxLbDlF8u0Q\n",
        "\n",
        "#### Keras \n",
        "\n",
        "Conv2D - Grey scale image\n",
        "Conv3D - Color Image\n",
        "\n",
        "Where do we use activation fn in conv layer?\n",
        "\n",
        "During Matrix multiplication (process), to get the compressed value, the activation fn is used\n",
        "\n",
        "##### Drop-Out\n",
        "\n",
        "* Anything is not activated frequently is considered to be dropped out,\n",
        "close to 0 always\n",
        "\n",
        "\n",
        "model.add(Dropout(0.25)) , here we are drpping only 25% .How to decide this value.Cant we drop all such neurns which are not adding value to output?\n",
        "\n",
        "* the number denotes the threshold of the bandwidth defined \n",
        "\n",
        "0.25 - maximum threshold of neurons to be dropped( should be less than 0.25)\n",
        "\n",
        "model.compile - optimizer, loss fn, metrix\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://colab.research.google.com/drive/1qoyoen50MGYUthi99sUrCTzw3oZL2An-\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IMXUDG3ZJeUF",
        "colab_type": "text"
      },
      "source": [
        "## Day  13 - 06-OCT-19 \n",
        "#### Classification of MNIST algorithm Hands On Part 2 - Keras \n",
        "\n",
        "#### Revision Questions\n",
        "\n",
        "* model.save('keras_cnn.h5') - model can be  saved in binary format\n",
        "\n",
        "* model.load - to load the saved model \n",
        "\n",
        "* model.predict - use the loaded model to predict \n",
        "\n",
        "* Drop out - Dropping unwanted , drop out increases the accuracy ? Not always\n",
        "\n",
        "* Drop out can be added into any layer which could be considered as filter of noise\n",
        "\n",
        "* Training time will be same but prediction time may vary according to change in parameters.\n",
        "\n",
        "* Total parameters changes after the drop out.\n",
        "\n",
        "* Find the optimal parameters to avoid overfitting and underfitting\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://keras.io/layers/convolutional/\n",
        "\n",
        "* Drop-out is similar to L1, L2 regularization where certain parameters are set to 0\n",
        "\n",
        "##### NAS - Grid search with Keras (Neural architecture search n/w)\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/\n",
        "\n",
        "* CNN keras access\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://colab.research.google.com/drive/10zt7Zt2kgrDHzeU3H4RYYJTSuBa1z6DK\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "## NLP\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://drive.google.com/file/d/1L7SJdvTK9dOcv1CAHphE7ZjKFKz8gj48/view?usp=sharing\n",
        "\n",
        "* sequential data  - presence of each point is related to the previous and future point\n",
        "\n",
        "--- language\n",
        "\n",
        "* Sequential Data - Video, stock price, Route map source to destination, Vital signs of patients, weather forecast, speech to text, chat data\n",
        "\n",
        "* NLP and RNN both have sequential data\n",
        "\n",
        "* Word Embedding - classification of text, sentiment analysis \n",
        "\n",
        "* Supervised & Un-Supervised\n",
        "\n",
        "#### Text Classification\n",
        "\n",
        "* Sentiment Analysis\n",
        "* Indent classification\n",
        "* Document classification (research, business functional/technical)\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://docs.google.com/spreadsheets/d/11yFyDOrCz-pUTY1xunsKmEDuIT85l9qoKHlgGnwoLyw/edit?usp=sharing\n",
        "\n",
        "\n",
        "#### Transformer models\n",
        "* SQUAD 2.2 - Stanford Question Answering Dataset\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMuE82wUgYiv",
        "colab_type": "text"
      },
      "source": [
        "## Day  14 - 12-OCT-19 \n",
        "#### NLP  -  Hands On and Word Embedding\n",
        "\n",
        "##### Revision CNN\n",
        "\n",
        "* I st Layer - convulution Layer - Main Function of Filter in Convolution layer - Extract the important features of the input. We define filter size, filter value are defined using random seed and due course of the training, the algorithm changes filter values\n",
        "* II nd Layer - Pooling - Second layer for extracting important features.\n",
        "* Parameters to be optimized - stride, fliter size, padding\n",
        "* NOTE : Pooling layer has two param  only filter size and stride\n",
        "* Padding - Higher resolution image boundary will be removed and lower iage boundary will have zeros added to standardize the image resolution\n",
        "\n",
        "* Filter size (5,5), stride (1,1) which stride the compression will be maximum? Higher the stride, higher the compression\n",
        "* How do you control overfitting and underfitting in convultion layer?\n",
        " Adjusting the stride values, optimal value will avoid both the scenarios to give unbiased output.\n",
        " * Lesser stride --> more accuracy\n",
        "\n",
        "* III rd Layer - Flatten Layer\n",
        "* IV Layer - FC (Dense) Layer\n",
        "* V Layer - Flatten Layer \n",
        "* VI Layer - Output Layer \n",
        "\n",
        "* More the dense, more training will happen.\n",
        "* Drop out removes neurons that are less significant during training\n",
        "* CNN can have combination of the above layers in differenet numbers, sometimes 2 conv layer etc\n",
        "* How to identify the network based on domain like medical domain - U-Net\n",
        "* Even image is rotated to 90 or 180 degree, will model identify the image?\n",
        "Augmentation - we have to train with rotated image for the model to identify\n",
        "* So Conv layer should always be associated with Dense layer in classification - yes\n",
        "\n",
        "#### NLP\n",
        "\n",
        "* CNN is also used in Text analytics\n",
        "* Speech - RNN / Transformer models are used\n",
        "\n",
        "#### Word Embeddings - words are decoded to numbers\n",
        "* Frequency based Embeddings\n",
        "* Prediction based Embeddings\n",
        "\n",
        "\n",
        "* count vector - representation of text in vectorized format \n",
        "\n",
        "3 possibilities of representation\n",
        "\n",
        "* doc to term \n",
        "* doc to doc \n",
        "* term to term \n",
        "\n",
        "TF-IDF - Instead of writing the count of words, we find the importance of words in a particular doc vs weightage of word  across all doc or vocab\n",
        "\n",
        "TF = Freq. of word in a doc / Total no. of words in a doc\n",
        "\n",
        "DF = word in no. of docs / totl no of docs\n",
        "\n",
        "IDF = Log(no. of docs / word in no. of docs)\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://docs.google.com/spreadsheets/d/11yFyDOrCz-pUTY1xunsKmEDuIT85l9qoKHlgGnwoLyw/edit?usp=sharing\n",
        "\n",
        "\n",
        "#### TFIDF sample code\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html\n",
        "\n",
        "\n",
        "#### access to NB created by Lavi \n",
        "\n",
        "from lavi to All Participants:\n",
        "https://colab.research.google.com/drive/1wx2lHBnydIQ4KJOBwycBZju_MvNRUbcq\n",
        "\n",
        "* TFVectorizer takes list as input\n",
        "\n",
        "* txt or docx files added to corpus as list of docs, please follow the link below to read glob to read the txt files\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://stackoverflow.com/questions/35672809/how-to-read-a-list-of-txt-files-in-a-folder-in-python\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iDse-Ph3cZPa",
        "colab_type": "text"
      },
      "source": [
        "## Day  15 - 19-OCT-19 \n",
        "#### NLP  -  Hands On \n",
        "\n",
        "##### Revision\n",
        "\n",
        "* Why do we represent language like number? - text is represented in vectorized format for machine to understand\n",
        "\n",
        "* Text related problems like sentiment analysis, text classification, indent classification can be handled in NLP\n",
        "\n",
        "* TF - Freq. of word in a doc (Local)\n",
        "* DF - Freq. of word across the docs / total no. of docs\n",
        "\n",
        "* intent classification - specifically for chatbot, the intent to do something, like to book flight tickets\n",
        "\n",
        "#### Frequncy based representation\n",
        "\n",
        "* Dictionary - unique words in the data\n",
        "\n",
        "* ngram - combination of adjacent words\n",
        "\n",
        "* System logs - activity info of the system , in this case, the dict will have details related to iP, timed out, connectivity, other info.\n",
        "\n",
        "\n",
        "like bigram and trigram (single, 2 or 3 words) - combo of 2 or 3 words, eg Ram is \n",
        "\n",
        "   -  Purpose of ngram -- to find the sentiment analysis\n",
        "   1.   when we take word to word mapping , Contextual info is missing \n",
        "   2.   to reduce the number of dimensions\n",
        "   \n",
        "#### Prediction based representation\n",
        "\n",
        "* What is the probability of word in a sentence ?\n",
        "\n",
        "##### Word2Vec - Semantic pattern of words\n",
        "\n",
        "* It can be obtained using two methods (both involving Neural Networks): Skip Gram and Common Bag Of Words (CBOW) CBOW Model: This method takes the context of each word as the input and tries to predict the word corresponding to the context.\n",
        "\n",
        "Window size - similar to ngram analysis\n",
        "\n",
        "* What info hid layer store?\n",
        "W, Bias, X , inputs and activation fn - actiation fn of prev layer along with weights,bias and inputs\n",
        "\n",
        "\n",
        "* Predicting a word from sentence or Predicting sentence from word \n",
        "\n",
        "Word Embeddings - hidden layer of N dimesion is info of model , registry of all patterns of given language on a trained huge corpus of data.\n",
        "\n",
        "* Continuous bag of words  - context of sentence and try to predict a word\n",
        "* Skip-gram (oppsite to CBOW) - have a word and tries to understand the sentence \n",
        "\n",
        "* if N is the total vocab, the sentences are input, the output is N \n",
        "\n",
        "* GLOVE (Global Vectors for Word Representation ) - google has done the training on the whole corpus of input.\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://nlp.stanford.edu/projects/glove/\n",
        "\n",
        "* Fasttext - FB has trained on most famous languages on the world\n",
        "\n",
        "fasttext - from lavi to All Participants:\n",
        "https://fasttext.cc/docs/en/crawl-vectors.html\n",
        "\n",
        "* The text file, has two versions\n",
        "\n",
        "    - regular version\n",
        "    - compressed version - lesser dimension\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://gab41.lab41.org/three-things-we-learned-about-applying-word-vectors-to-computer-logs-c199070f390b#.bk8wnk7pr\n",
        "\n",
        "* Interview Question - Wealth of information of text data available in each and every domain, how can you handle? we can handle the data using word embedding\n",
        "\n",
        "\n",
        "* To know the New Embeddings on the AI world\n",
        "     -- Medium for daily updates in AI\n",
        "\n",
        "#### HandsOn - To predict movie ratings (Word2Vec_gensim)\n",
        "\n",
        "Gensim - Library \n",
        "\n",
        "Iterations - Epoch\n",
        "\n",
        "Assignment -  convert the org data into meaningful vector (Json, txt format can be used)\n",
        "\n",
        "GPT - can we create story telling with the data? no. only model building\n",
        "\n",
        "##### Link to train text data \n",
        "\n",
        "from Venkataraju to All Participants:\n",
        "url = 'https://raw.githubusercontent.com/MatthieuBizien/Bag-popcorn/master/unlabeledTrainData.tsv'\n",
        "\n",
        "* Additional info on transformer model and making DL project agile requested\n",
        "* ML + DNN and ML + NLP combination use cases requested"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wt7sxHpbSr1i",
        "colab_type": "text"
      },
      "source": [
        "## Day  16 - 20-OCT-19 \n",
        "#### NLP  -  Hands On Keras Tokenizer\n",
        "\n",
        "Link to Data \n",
        "https://raw.githubusercontent.com/MatthieuBizien/Bag-popcorn/master/unlabeledTrainData.tsv\n",
        "\n",
        "\n",
        "df = pd.read_csv('https://raw.githubusercontent.com/MatthieuBizien/Bag-popcorn/master/labeledTrainData.tsv',header=0, delimiter=\"\\t\", quoting=3)\n",
        "\n",
        "#### Text Analytics/NLP \n",
        "\n",
        "* Text Classification \n",
        "      \n",
        "* Information Retreival \n",
        "      \n",
        "* Etc \n",
        "\n",
        "~ Let's say, data not tagged; \n",
        "\n",
        "   - Any business rule, that can create a tag \n",
        "      - If business rules are working fine, once they are figured out, why do model?\n",
        "      - IF something can be solved with automation rules (code logic), why to do AI?? Question to be asked! \n",
        "         - If rules are exact ; no need of AI\n",
        "         - if rules are not there and it might change in future (Dynamic); then do AI \n",
        "   - if not, do tagging through prodi.gy tool , then model \n",
        "   - If Data tagging need to be done,\n",
        "       - Get Rules\n",
        "       - Get Actual classes\n",
        "       - Amount of data that requires tagging\n",
        "           - For each rule, atleast have 100 variations\n",
        "           - For each class, have atleast 300 variations\n",
        "   \n",
        "* Text Classification (Supervised Learning)\n",
        "\n",
        "     - Chatbots\n",
        "     - Intents  (\"Book ticket\", \"Cancel Ticket\")\n",
        "     - Entities  (\"<from_location >\")\n",
        "     - Intent Classification ; Book a ticket from A to B --> Intent;\"Book Ticket\"\n",
        "     - Sentiment Analysis \n",
        "     - Document Classification \n",
        "     - Custom Entity Prediction; Book a ticket from A to B ---> Extract \"A\" & \"B\"; to_location - ; from_location        \n",
        "     \n",
        "* There are 3 ways to vectorizer \n",
        "     - Count Vectoriser or Tfidf Vectoriser   (Sklearn, Keras)\n",
        "     - Take standard English Embedding  (Glove, FasTtext)\n",
        "     - Take custom trained Word Embedding ( on your data)   (Genism )  \n",
        "     \n",
        "* Once you vectorise, you have 2 methods to do prediction \n",
        "\n",
        "     - assuming, you already have a class --> Supervised \n",
        "     - Prirority FLow of ALgo ; Transformer Model ; ML MOdels (GridSearch) ; All Remaining DL Model (AutoDL; Ludwig)\n",
        "     - Hyperparameter Tuning ; Do gridsearch like Logic \n",
        "     - Do, normal ML Modeling ( To start; try SVM & XGBoost, Baseline, GridSearch   \n",
        "     - Do Neural Network Modelling\n",
        "         - Simple FC Model\n",
        "         - Do CNN based model\n",
        "         - Do RNN\n",
        "         - Do other networks as well\n",
        "         - Any transformer model (Bert, Elno, Alberta, Xlnet)\n",
        "     - Do autoDL (Choose network you want to try, let autoDL figure out the best combo)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://uber.github.io/ludwig/examples/#text-classification\n",
        "\n",
        "\n",
        "* prodi.gy tool --- to tag the data\n",
        "\n",
        "https://prodi.gy\n",
        "\n",
        "\n",
        "* When image classification use case, each frame is dependent on previous frame , we use RNN otherwise CNN and its variations\n",
        "\n",
        "##### Link to Hands-on data file\n",
        "\n",
        "from lavi to All Participants:\n",
        "https://raw.githubusercontent.com/MatthieuBizien/Bag-popcorn/master/labeledTrainData.tsv\n",
        "\n",
        "\n",
        "* Ludwig is autoDL model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S2RjFrWvs1-J",
        "colab_type": "text"
      },
      "source": [
        "# Meetup - Gurgaon - 14 Sept \n",
        "\n",
        "--- THings discussed \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SAnaGLQIs7KG",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jrdrZoBicvaF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CeaJgFOgdGrY",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    }
  ]
}