09. Ensemble Method Bagging and Boosting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensemble Method\n",
    "* Main cause of error while learning are due to noise, bias and variance  \n",
    "* minimize above factors  \n",
    "* group of weak learners combined to form a strong learner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two Types:  \n",
    "Homogeneous Algorithm – Bagging, Boosting  \n",
    "Heterogeneous Algorithm - Stacking  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models are different based on four criteria:\n",
    "1. Difference in population\n",
    "2. Difference in hypothesis\n",
    "3. Difference in modeling technique\n",
    "4. Difference in initial seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bagging**\n",
    "- Goal is to minimize variance. \n",
    "- It creates several subsets of data chosen randomly with replacement. Each subset data is used to train. Average of all predictions are taken  \n",
    "- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Boosting**  \n",
    "- Goal is to minimize variance\n",
    "- Aims at fitting sequentially weak learners in a adaptive way. Each model learns from the mistake of previous model and minimize error.  \n",
    "\n",
    "- High bias model are computationally less expensive to fit. Once weak learners are chosen, there are two way they can be sequentially fitted.  \n",
    "- Two Important algo: Adaboost and Gradient Boosting  \n",
    "- These two algo differ on how they create and aggregate the weak learners during the sequential process.  \n",
    "- Adaboost: updates the weights attached to each of the training dataset observations  \n",
    "- Gradient Boosting: updates the value of these observations  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"Image/ensemble.JPG\" width=\"800\" />"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaboost \n",
    "- Increases the accuracy by giving more weightage to the target which is misclassified\n",
    "- Next sample – repeat same\n",
    "- Weak learners combine to form a strong learners\n",
    "- it minimizes the exponential loss function that can make the algorithm sensitive to the outliers\n",
    "\n",
    "## Gradient Boost:\n",
    "- increases the accuracy by minimizing the loss function and keep this as target for next decision tree building\n",
    "- any differentiable loss function can be utilized\n",
    "- robust to outliers than AdaBoost\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}