3 Fitted values and residuals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitted values and residuals\n",
    "***\n",
    "## Fitted values\n",
    "Each observation in a time series can be forecast using all previous observations. We call these fitted values and they are denoted by $\\hat{y}_{t|t-1}$, meaning the forecast of $y_{t}$ based on observations $y_{,},...,y_{t-1}$ . We use these so often, we sometimes drop part of the subscript and just write $\\hat{y}_{t}$ instead of $\\hat{y}_{t|t-1}$. Fitted values almost always involve one-step forecasts.\n",
    "\n",
    "Actually, fitted values are often not true forecasts because any parameters involved in the forecasting method are estimated using all available observations in the time series, including future observations. For example, if we use the mean method, the fitted values are given by:\n",
    "\n",
    "\\begin{gather*} \n",
    "\\hat{y}_{t}=\\hat{c}\n",
    "\\end{gather*}\n",
    "\n",
    "where $\\hat{c}$ is the average computed over all available observations, including those at times after $t$. Similarly, for the drift method, the drift parameter is estimated using all available observations. In this case, the fitted values are given by\n",
    "\n",
    "\\begin{gather*} \n",
    "\\hat{y}_{t}={y}_{t-1}\\hat{c}\n",
    "\\end{gather*}\n",
    "\n",
    "Where $\\hat{c}=(y_{T}-y_{1})/(T-1)$ In both cases, there is a parameter to be estimated from the data. The “hat” above the $c$ reminds us that this is an estimate. When the estimate of $c$ involves observations after time $t$, the fitted values are not true forecasts. On the other hand, naïve or seasonal naïve forecasts do not involve any parameters, and so fitted values are true forecasts in such cases.\n",
    "\n",
    "## Residuals\n",
    "\n",
    "The “residuals” in a time series model are what is left over after fitting a model. The residuals are equal to the difference between the observations and the corresponding fitted values:\n",
    "\n",
    "\\begin{gather*} \n",
    "e_{t}=y_{t}-\\hat{y}_{t*}\n",
    "\\end{gather*}\n",
    "\n",
    "If a transformation has been used in the model, then it is often useful to look at residuals on the transformed scale. We call these **“innovation residuals”**. For example, suppose we modelled the logarithms of the data, $w_{t}=log(y_{t})$. Then the innovation residuals are given by $w_{t}-\\hat{w}_{t}$ whereas the regular residuals are given by $y_{t}-\\hat{y}_{t*}$. If no transformation has been used then the innovation residuals are identical to the regular residuals, and in such cases we will simply call them “residuals”.\n",
    "\n",
    "The fitted values and residuals from a model can be obtained using the `augment()` function. In the beer production example in Section 2, we saved the fitted models as `beer_fit`. So we can simply apply `augment()` to this object to compute the fitted values and residuals for all models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import the libraries that we are going to use for the analysis:\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from statsforecast import StatsForecast\n",
    "from statsforecast.models import __all__\n",
    "from statsforecast.models import HistoricAverage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a dataframe from a csv file:\n",
    "df = pd.read_csv(\"Assets/aus-production.csv\", sep=\";\")\n",
    "\n",
    "# Create a dataframe with beer production: \n",
    "beer = df[[\"Quarter\",\"Beer\"]]\n",
    "\n",
    "#Inferior limit:\n",
    "beer_mask=beer['Quarter']>=\"1992 Q1\"\n",
    "filtered_beer = beer[beer_mask]\n",
    "\n",
    "#Superior limit:\n",
    "beer_mask=filtered_beer['Quarter']<=\"2006 Q4\"\n",
    "beer = filtered_beer[beer_mask]\n",
    "\n",
    "# Create an array with the observed values\n",
    "y_beer = beer[\"Beer\"].values\n",
    "\n",
    "# Mean method:\n",
    "model = HistoricAverage()\n",
    "model = model.fit(y=y_beer)\n",
    "mean = model.predict(h=14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "The function augment() return a DataFrame with five columns (model, time_var, obs_values, fitted_values and residuals) and has 3 arguments:\n",
    "\n",
    "    * model_name: String variable. It´s reffer to what kind of model are we using (Mean, Naive, SNaive, ETS, etc.).\n",
    "\n",
    "    * time_var: Column of the input dataframe or numpy array. It´s reffer to the time index of the observed values.\n",
    "\n",
    "    * obs_values: Column of the input dataframe or numpy array. It´s reffer to the the observed values.\n",
    "\n",
    "'''\n",
    "\n",
    "def augment(model_name, time_var, obs_values):\n",
    "\n",
    "    type_model_list = []\n",
    "    for n in range(len(obs_values)):\n",
    "        type_model_list.append(model_name)\n",
    "        n += 1\n",
    "    fitted_values = model.predict_in_sample()\n",
    "    residuals = obs_values - fitted_values[\"mean\"]\n",
    "\n",
    "    augment_df = pd.DataFrame({'model':type_model_list,\n",
    "                        'time_var':time_var,\n",
    "                        'obs_values':obs_values,\n",
    "                        'fitted_values':fitted_values[\"mean\"],\n",
    "                        'residuals':residuals})\n",
    "\n",
    "    return(augment_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>time_var</th>\n",
       "      <th>obs_values</th>\n",
       "      <th>fitted_values</th>\n",
       "      <th>residuals</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>Mean</td>\n",
       "      <td>2005 Q4</td>\n",
       "      <td>482</td>\n",
       "      <td>436.450012</td>\n",
       "      <td>45.549988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>Mean</td>\n",
       "      <td>2006 Q1</td>\n",
       "      <td>438</td>\n",
       "      <td>436.450012</td>\n",
       "      <td>1.549988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>201</th>\n",
       "      <td>Mean</td>\n",
       "      <td>2006 Q2</td>\n",
       "      <td>386</td>\n",
       "      <td>436.450012</td>\n",
       "      <td>-50.450012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>Mean</td>\n",
       "      <td>2006 Q3</td>\n",
       "      <td>405</td>\n",
       "      <td>436.450012</td>\n",
       "      <td>-31.450012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>Mean</td>\n",
       "      <td>2006 Q4</td>\n",
       "      <td>491</td>\n",
       "      <td>436.450012</td>\n",
       "      <td>54.549988</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    model time_var  obs_values  fitted_values  residuals\n",
       "199  Mean  2005 Q4         482     436.450012  45.549988\n",
       "200  Mean  2006 Q1         438     436.450012   1.549988\n",
       "201  Mean  2006 Q2         386     436.450012 -50.450012\n",
       "202  Mean  2006 Q3         405     436.450012 -31.450012\n",
       "203  Mean  2006 Q4         491     436.450012  54.549988"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# augment(model_name, time_var, obs_values)\n",
    "augment_df = augment(\"Mean\",beer[\"Quarter\"],beer[\"Beer\"])\n",
    "\n",
    "augment_df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two new columns added to the original data:\n",
    "\n",
    "+ `fitted_values` contains the fitted values;\n",
    "+ `residuals` contains the residuals;\n",
    "\n",
    "If patterns are observable in the residuals, the model can probably be improved. We will look at some tools for exploring patterns in residuals in the next section."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.7 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "b1896fd924129e7fd2d4d004268bb9f63a53b23ae7875be394e789ef8193cc30"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}