3_Machine_Learning_and_Deep_Learning_Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import Model\n",
    "from tensorflow.keras.layers import Input, Dense, Dropout, BatchNormalization\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.model_selection import train_test_split, RandomizedSearchCV\n",
    "from sklearn.linear_model import Ridge, Lasso\n",
    "from xgboost import XGBRegressor\n",
    "\n",
    "from nltk.tokenize import word_tokenize\n",
    "import gensim.models\n",
    "\n",
    "from scipy.sparse import hstack\n",
    "from sklearn.metrics import mean_squared_log_error\n",
    "\n",
    "from prettytable import PrettyTable\n",
    "\n",
    "import joblib\n",
    "import gc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the featurized data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train, df_cv = joblib.load('df_train_cv_after_featurizations.joblib')\n",
    "y_train = np.array(df_train['price'].values).reshape(-1, 1)\n",
    "y_cv    = np.array(df_cv['price'].values).reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np.log1p(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The target variable was converted to its log form so that we can directly optimize for Mean Squared Error (MSE) instead of Mean Squared Log Error (MSLE). This was easy since most models optimize for MSE by default for Regression tasks while not all support direct optimization for MSLE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Defining the loss metric for evaluation: Root mean squared log error (RMSLE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rmsle(y_true, y_pred):\n",
    "    '''\n",
    "    This function take the tuple of true class labels and the predicted class labels as input and gives the Root mean squared\n",
    "    log error between these as the output\n",
    "    '''\n",
    "    return np.sqrt(mean_squared_log_error(y_true, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Baseline Model\n",
    "We build a baseline model so as to get an upper bound to the error with a very simple model that predicts the mean price of the training data for each CV data point. In this way, we'll know, how much better, each of our model performs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline Model CV RMSLE =  0.746\n"
     ]
    }
   ],
   "source": [
    "y_pred_baseline = np.expm1(np.repeat(np.mean(y_train), len(y_cv)))\n",
    "print(\"Baseline Model CV RMSLE = \", np.round(rmsle(y_cv, y_pred_baseline), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Since, text is a crucial part in deciding the price of the product (intuition), we will featurize it in form of sparse as well as dense vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TF-IDF Vectorization for Text Data (Sparse Vectors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I tried different number of MaxFeatures and found out that using 100,000 features is appropriate as it captures substantial amount of information <br>\n",
    "Bi-grams were used for text field which contained item description with the intuition that there were brands like \"Michael Kors\" and since brands played an important role in predicting the price (observed in EDA), it would be better if this information is captured in a better way through bi-grams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Token pattern was changed to '\\w+' because in normal TF-IDF vectorization, punctuations are treated as token separators which may not always be the case in this case study since there are reviews which contain alphabets, numbers, periods, underscores (for brand names) and hence '\\w+' token pattern is more appropriate for this case study."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tfidf_encoder(train_data, test_data, N_GRAMS = 1):\n",
    "    '''\n",
    "    This function returns the TF-IDF encoding of the text\n",
    "    \n",
    "    Input ->\n",
    "    \n",
    "        train_data       : Text (string or list of strings or Pandas Series with elements as strings)\n",
    "        test_data        : Text (string or list of strings or Pandas Series with elements as strings)\n",
    "        N_GRAMS(int)     : Upper bound of the n_grams to be considered while vectorizing the data using TF-IDF encoder\n",
    "                           For eg., If the n_grams = 2, then both unigrams and bi-grams will be used while vectorizing\n",
    "                           the text data. Default value is kept as 1, which means only uni-grams will be generated if this\n",
    "                           argument is not supplied explicitly while calling this function\n",
    "    \n",
    "    Output -> Tuple of TF-IDF vectors of \"train_data\" and \"test_data\" computed using sklearn's Tfidfvectorizer() \n",
    "    \n",
    "    Task   -> Given a text (string), return the TF-IDF vectors for that text\n",
    "              The vectorizer is fitted on the train_data and used to tranform both the train data and the test data\n",
    "    '''\n",
    "    vectorizer = TfidfVectorizer(max_features = 100000,\n",
    "                                 ngram_range = (1, N_GRAMS),\n",
    "                                 strip_accents = 'unicode',\n",
    "                                 analyzer = 'word',\n",
    "                                 token_pattern = r'\\w+')\n",
    "    \n",
    "    train_tdidf = vectorizer.fit_transform(train_data)\n",
    "    test_tfidf =  vectorizer.transform(test_data)\n",
    "    return (train_tdidf, test_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_tr_name, X_cv_name = tfidf_encoder(df_train['name'], df_cv['name'], N_GRAMS = 1)\n",
    "X_tr_text, X_cv_text = tfidf_encoder(df_train['text'], df_cv['text'], N_GRAMS = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1111234, 85394)\n",
      "(1111234, 100000)\n",
      "(370424, 85394)\n",
      "(370424, 100000)\n"
     ]
    }
   ],
   "source": [
    "print(X_tr_name.shape)\n",
    "print(X_tr_text.shape)\n",
    "print(X_cv_name.shape)\n",
    "print(X_cv_text.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One Hot Encoding for Shipping and item_condition_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_hot_encoder(train_data, cv_data):\n",
    "    '''\n",
    "    This function returns the One Hot Encoded vectors for the given train and CV data\n",
    "    Input ->\n",
    "        train_data : Training data to be one hot encoded (List of integers/strings or a Pandas Series)\n",
    "        cv_data    : Cross Validation data to be one hot encoded (List of integers/strings or a Pandas Series)\n",
    "    Output -> Tuple of One hot encoded vectors of training and CV data\n",
    "    Task   -> This function converts the raw values (integers/strings) into one hot encoded vectors using\n",
    "              sklearn's OneHotEncoder()\n",
    "    '''\n",
    "    ohe_encoder = OneHotEncoder()\n",
    "    train_ohe = ohe_encoder.fit_transform(train_data)\n",
    "    cv_ohe = ohe_encoder.transform(cv_data)\n",
    "    return train_ohe, cv_ohe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_tr_shipping, X_cv_shipping = one_hot_encoder(np.reshape(df_train['shipping'].values, (-1, 1)), np.reshape(df_cv['shipping'].values, (-1, 1)))\n",
    "X_tr_item_condition, X_cv_item_condition = one_hot_encoder(np.reshape(df_train['item_condition_id'].values, (-1, 1)), np.reshape(df_cv['item_condition_id'].values, (-1, 1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1111234, 2)\n",
      "(370424, 2)\n",
      "(1111234, 5)\n",
      "(370424, 5)\n"
     ]
    }
   ],
   "source": [
    "print(X_tr_shipping.shape)\n",
    "print(X_cv_shipping.shape)\n",
    "print(X_tr_item_condition.shape)\n",
    "print(X_cv_item_condition.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Combining all the features to create the training data matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = hstack((X_tr_name,\n",
    "                  X_tr_text,\n",
    "                  X_tr_shipping,\n",
    "                  X_tr_item_condition)).tocsr().astype('float32')\n",
    "\n",
    "X_cv   = hstack((X_cv_name,\n",
    "                 X_cv_text,\n",
    "                 X_cv_shipping,\n",
    "                 X_cv_item_condition)).tocsr().astype('float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1111234, 185401)\n",
      "(370424, 185401)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_cv.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 1: Ridge Regression (Linear Regression + L2 Regularization)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since, Linear regression does not work well with correlated features, hence, we'll use only one of the price statistics features for linear regression as they're highly correlated with each other. We choose median of the log price since it had the highest correlation with the target variable (as observed in EDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ridge_model(X_train, y_train, parameter):\n",
    "    '''\n",
    "    Task -> This function fits the Ridge Model (Linear Regression + L2 Regularization) using sklearn's Ridge() Model \n",
    "            on the training data(X_train and y_train) and returns the fitted model\n",
    "    \n",
    "    Input ->\n",
    "        \n",
    "        X_train:   Training data matrix (Numpy/Scipy Array)\n",
    "        y_train:   Class labels of the training data matrix in the form of a 1-D numpy array/Pandas Series\n",
    "        parameter: Hyperparameter \"alpha\" of Ridge() model of sklearn.linear_model\n",
    "    \n",
    "    Output -> Ridge model fitted on X_train and y_train\n",
    "    '''\n",
    "    # Solver was chosen to be 'lsqr' as it converged very quickly\n",
    "    \n",
    "    model = Ridge(solver = 'lsqr', fit_intercept=False, alpha = parameter, random_state = 11)\n",
    "    model.fit(X_train, y_train)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since, we took log of original y_train, we do the inverse process so as to get back the original y_train which \n",
    "# we will use for calculating training error\n",
    "\n",
    "y_train_original = np.expm1(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSLE for alpha =  1e-05  is  0.4349957935621989\n",
      "CV RMSLE for alpha =  1e-05  is  0.4551584494167266\n",
      "--------------------------------------------------------------\n",
      "Train RMSLE for alpha =  0.0001  is  0.4349957935639252\n",
      "CV RMSLE for alpha =  0.0001  is  0.4551583628082112\n",
      "--------------------------------------------------------------\n",
      "Train RMSLE for alpha =  0.001  is  0.43499579374758734\n",
      "CV RMSLE for alpha =  0.001  is  0.45515749692449653\n",
      "--------------------------------------------------------------\n",
      "Train RMSLE for alpha =  0.01  is  0.4349958122058277\n",
      "CV RMSLE for alpha =  0.01  is  0.4551488582121009\n",
      "--------------------------------------------------------------\n",
      "Train RMSLE for alpha =  0.1  is  0.43499764082049175\n",
      "CV RMSLE for alpha =  0.1  is  0.45506446447269716\n",
      "--------------------------------------------------------------\n",
      "Train RMSLE for alpha =  1  is  0.4367416695328943\n",
      "CV RMSLE for alpha =  1  is  0.455041713196296\n",
      "--------------------------------------------------------------\n",
      "Train RMSLE for alpha =  10  is  0.4514105776046936\n",
      "CV RMSLE for alpha =  10  is  0.46209915210229974\n",
      "--------------------------------------------------------------\n",
      "Train RMSLE for alpha =  100  is  0.49220474841346973\n",
      "CV RMSLE for alpha =  100  is  0.49439543405185965\n",
      "--------------------------------------------------------------\n",
      "Train RMSLE for alpha =  1000  is  0.564571929308979\n",
      "CV RMSLE for alpha =  1000  is  0.5642618608630365\n",
      "--------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Here, we find the best hyperparameter alpha for the Ridge model by evaluating the performance on both train as well as CV data\n",
    "using various values of hyperparameter alpha\n",
    "'''\n",
    "for alpha in [10**x for x in range(-5, 4)]:\n",
    "    clf = ridge_model(X_train, y_train, alpha)\n",
    "    y_pred_train = clf.predict(X_train)[:, 0]\n",
    "    y_pred_cv = clf.predict(X_cv)[:, 0]\n",
    "    preds_ridge_train = np.expm1(y_pred_train.reshape(-1, 1))[:, 0]\n",
    "    preds_ridge_cv = np.expm1(y_pred_cv.reshape(-1, 1))[:, 0]\n",
    "    print(\"Train RMSLE for alpha = \", alpha, \" is \", rmsle(y_train_original, preds_ridge_train))\n",
    "    print(\"CV RMSLE for alpha = \", alpha, \" is \", rmsle(np.reshape(y_cv, (-1, 1)), preds_ridge_cv))\n",
    "    del clf\n",
    "    gc.collect()\n",
    "    print(\"--------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating performance on Cross Validation Data for best hyperparameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_alpha = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.455041713196296"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = ridge_model(X_train, y_train, best_alpha)\n",
    "y_pred = clf.predict(X_cv)[:, 0]\n",
    "preds_ridge = np.expm1(y_pred.reshape(-1, 1))[:, 0]\n",
    "rmsle(y_cv, preds_ridge)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 2: Lasso Regression (Linear Regression + L1 Regularization)\n",
    "Since we have very sparse data, it is logical to train a Lasso model so that inherent feature selection happens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lasso_model(X_train, y_train, parameter):\n",
    "    '''\n",
    "    Task -> This function fits the Lasso Model (Linear Regression + L1 Regularization) using sklearn's Lasso() Model \n",
    "            on the training data(X_train and y_train) and returns the fitted model\n",
    "    \n",
    "    Input ->\n",
    "        \n",
    "        X_train:   Training data matrix (Numpy/Scipy Array)\n",
    "        y_train:   Class labels of the training data matrix in the form of a 1-D numpy array/Pandas Series\n",
    "        parameter: Hyperparameter \"alpha\" of Lasso() model of sklearn.linear_model\n",
    "    \n",
    "    Output -> Lasso model fitted on X_train and y_train\n",
    "    '''\n",
    "    model = Lasso(alpha = parameter,\n",
    "                  normalize = True,\n",
    "                  max_iter=2000,\n",
    "                  random_state = 11)\n",
    "    model.fit(X_train, y_train)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5min 26s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "clf = lasso_model(X_train, y_train, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating performance on Cross Validation Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.74622476"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = clf.predict(X_cv)\n",
    "preds_lasso = np.expm1(y_pred.reshape(-1, 1))[:, 0]\n",
    "rmsle(y_cv, preds_lasso)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems like the model did not converge within 2000 iterations and hence it is not feasible to use it since it will take a lot of time to train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 3: XGBoost on TF-IDF Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_cv = np.log1p(y_cv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since XGBoost supported direct optimization of squared log error, hence we used original y_train for training here\n",
    "xgb_reg = XGBRegressor(random_state = 21,\n",
    "                       n_jobs = -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using RandomSearch to find the best set of hyperparameters on XGBoost model on TF-IDF features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'n_estimators' : [500, 1000, 1500],\n",
    "          'max_depth' : [5, 7],\n",
    "          'subsample' : [0.7, 0.9],\n",
    "          'colsample_bytree' : [0.7, 0.9]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'subsample': 0.9,\n",
       " 'n_estimators': 1500,\n",
       " 'max_depth': 7,\n",
       " 'colsample_bytree': 0.7}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = RandomizedSearchCV(xgb_reg,\n",
    "                         params,\n",
    "                         cv=3,\n",
    "                         n_jobs=-1,\n",
    "                         random_state=21,\n",
    "                         verbose=2)\n",
    "\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2552079203446068"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSLE =  0.4766871070500511\n",
      "CV RMSLE =  0.490089474076724\n"
     ]
    }
   ],
   "source": [
    "best_est = clf.best_estimator_\n",
    "y_pred_train = best_est.predict(X_train)\n",
    "'''\n",
    "Below statement was added since RMSLE's definition does not allow negative numbers and it makes logical sense also since product\n",
    "price can never be negative. However, to be on the safe side, we make this ammendment\n",
    "'''\n",
    "y_pred_train[y_pred_train < 0] = 0\n",
    "y_pred_cv = best_est.predict(X_cv)\n",
    "y_pred_cv[y_pred_cv < 0] = 0\n",
    "print(\"Training RMSLE = \", rmsle(y_train, y_pred_train))\n",
    "print(\"CV RMSLE = \", rmsle(y_cv, y_pred_cv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training time for XGBoost was significantly high but still the RSMLE is higher than simple linear regression model <br> This could be because we used high number of features for XGBoost, which is a tree based ensemble and does not work well with high dimensional features <br> Now, we will use dense Word2Vec representations for vectorizing text fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 4: XGBoost on Word2Vec Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dense Vectorizations (Word2Vec)\n",
    "Here, we use 300-dimensional Word2Vec dense vectors trained on Google News"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Here, we tokenize the 'name' and 'text' fields of the train and CV data using NLTK's work_tokenize() function\n",
    "'''\n",
    "train_name = [0]*len(df_train)\n",
    "for index, review in enumerate(df_train['name'].values):\n",
    "    train_name[index] = word_tokenize(review)\n",
    "\n",
    "cv_name = [0]*len(df_cv)\n",
    "for index, review in enumerate(df_cv['name'].values):\n",
    "    cv_name[index] = word_tokenize(review)\n",
    "    \n",
    "train_text = [0]*len(df_train)\n",
    "for index, review in enumerate(df_train['text'].values):\n",
    "    train_text[index] = word_tokenize(review)\n",
    "    \n",
    "cv_text = [0]*len(df_cv)\n",
    "for index, review in enumerate(df_cv['text'].values):\n",
    "    cv_text[index] = word_tokenize(review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Word2Vec model is used to generate 300 - dimensional dense vectors for the train and text data\n",
    "# Here, we load the pre-trained model trained on Google News data, downloaded from https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz\n",
    "\n",
    "model = gensim.models.KeyedVectors.load_word2vec_format(r'./GoogleNews-vectors-negative300.bin', binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Code inspired from: https://github.com/sdimi/average-word2vec/blob/master/avg_word2vec_from_documents.py\n",
    "zeros_vector = np.zeros(300)\n",
    "\n",
    "def document_vector(word2vec_model, doc):\n",
    "    '''\n",
    "    Task -> This function finds the average word vector representation for all the words given in the \"doc\"\n",
    "            This is done by performing a lookup for the words in the doc which are also present in the word2vec_model's\n",
    "            vocabulary and then calculating the mean/average vector for all the words in the doc which are also present in the\n",
    "            w2v model's vocabulary. If none of the words in the doc are present in the w2v model's vocabulary, we return a 300\n",
    "            dimensional vector of all zeroes\n",
    "            \n",
    "    Input ->\n",
    "            word2vec_model: Pre-trained word2vec model trained on Google News data (300-dimensional dense vectors)\n",
    "            doc           : (string) The document/text for which we have to calculate the average word2vec\n",
    "    \n",
    "    Output ->\n",
    "            300 dimensional average dense word vector of \"doc\" or a same sized vector of all zeroes if none of the word\n",
    "            in doc is present in the word2vec_model's vocabulary    \n",
    "    '''\n",
    "    doc = [word for word in doc if word in word2vec_model.vocab]\n",
    "    if len(doc) != 0:\n",
    "        return np.mean(word2vec_model[doc], axis=0)\n",
    "    else:\n",
    "        return zeros_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Here, we calculate the average word-vectors for each of the document in the 'name' and 'text' fields of the training and CV\n",
    "data using the document_vector() function defined above\n",
    "'''\n",
    "avg_word2vec_train_name = np.zeros((len(train_name), 300))\n",
    "for index, doc in enumerate(train_name): \n",
    "    avg_word2vec_train_name[index] = document_vector(model, doc)\n",
    "\n",
    "avg_word2vec_cv_name = np.zeros((len(cv_name), 300))\n",
    "for index, doc in enumerate(cv_name): \n",
    "    avg_word2vec_cv_name[index] = document_vector(model, doc)\n",
    "\n",
    "avg_word2vec_train_text = np.zeros((len(train_text), 300))\n",
    "for index, doc in enumerate(train_text): \n",
    "    avg_word2vec_train_text[index] = document_vector(model, doc)\n",
    "\n",
    "avg_word2vec_cv_text = np.zeros((len(cv_text), 300))\n",
    "for index, doc in enumerate(cv_text):\n",
    "    avg_word2vec_cv_text[index] = document_vector(model, doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One Hot Encoding for Shipping and item_condition_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_tr_shipping, X_cv_shipping = one_hot_encoder(np.reshape(df_train['shipping'].values, (-1, 1)), np.reshape(df_cv['shipping'].values, (-1, 1)))\n",
    "X_tr_item_condition, X_cv_item_condition = one_hot_encoder(np.reshape(df_train['item_condition_id'].values, (-1, 1)), np.reshape(df_cv['item_condition_id'].values, (-1, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Combining all the features to create the training data matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "All the data is converted to 'float32' type for efficient RAM utilisation\n",
    "'''\n",
    "avg_word2vec_train_name = avg_word2vec_train_name.astype('float32')\n",
    "avg_word2vec_train_text = avg_word2vec_train_name.astype('float32')\n",
    "avg_word2vec_cv_name = avg_word2vec_cv_name.astype('float32')\n",
    "avg_word2vec_cv_text = avg_word2vec_cv_text.astype('float32')\n",
    "\n",
    "X_tr_shipping = X_tr_shipping.astype('float32')\n",
    "X_cv_shipping = X_cv_shipping.astype('float32')\n",
    "\n",
    "X_tr_item_condition = X_tr_item_condition.astype('float32')\n",
    "X_cv_item_condition = X_cv_item_condition.astype('float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Converting all data to dense type as these are not sparse vectors\n",
    "'''\n",
    "X_tr_shipping = X_tr_shipping.todense()\n",
    "X_tr_item_condition = X_tr_item_condition.todense()\n",
    "X_cv_shipping = X_cv_shipping.todense()\n",
    "X_cv_item_condition = X_cv_item_condition.todense()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Combining the dense features to form the training and cross validation data matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = np.concatenate((avg_word2vec_train_name,\n",
    "                          avg_word2vec_train_text,\n",
    "                          X_tr_shipping,\n",
    "                          X_tr_item_condition), axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_cv= np.concatenate((avg_word2vec_cv_name,\n",
    "                      avg_word2vec_cv_text,\n",
    "                      X_cv_shipping,\n",
    "                      X_cv_item_condition), axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 4.1: XGboost with 500 trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSLE =  0.53467884038316\n",
      "CV RMSLE =  0.5667811273104297\n"
     ]
    }
   ],
   "source": [
    "xgb_reg = XGBRegressor(n_estimators = 500,\n",
    "                       random_state = 21,\n",
    "                       n_jobs = -1)\n",
    "\n",
    "xgb_reg.fit(X_train, y_train)\n",
    "\n",
    "y_pred_train = xgb_reg.predict(X_train)\n",
    "y_pred_train[y_pred_train < 0] = 0\n",
    "y_pred_cv = xgb_reg.predict(X_cv)\n",
    "y_pred_cv[y_pred_cv < 0] = 0\n",
    "print(\"Training RMSLE = \", rmsle(y_train, y_pred_train))\n",
    "print(\"CV RMSLE = \", rmsle(y_cv, y_pred_cv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 4.2: XGboost with 1000 trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSLE =  0.5048164553553274\n",
      "CV RMSLE =  0.5566548151608218\n"
     ]
    }
   ],
   "source": [
    "xgb_reg = XGBRegressor(n_estimators = 1000,\n",
    "                       random_state = 21,\n",
    "                       n_jobs = -1)\n",
    "\n",
    "xgb_reg.fit(X_train, y_train)\n",
    "\n",
    "y_pred_train = xgb_reg.predict(X_train)\n",
    "y_pred_train[y_pred_train < 0] = 0\n",
    "y_pred_cv = xgb_reg.predict(X_cv)\n",
    "y_pred_cv[y_pred_cv < 0] = 0\n",
    "print(\"Training RMSLE = \", rmsle(y_train, y_pred_train))\n",
    "print(\"CV RMSLE = \", rmsle(y_cv, y_pred_cv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 4.3: XGboost with 1500 trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training RMSLE =  0.48509613301263727\n",
      "CV RMSLE =  0.5519472286014295\n"
     ]
    }
   ],
   "source": [
    "xgb_reg = XGBRegressor(n_estimators = 1500,\n",
    "                       random_state = 21,\n",
    "                       n_jobs = -1)\n",
    "\n",
    "xgb_reg.fit(X_train, y_train)\n",
    "\n",
    "y_pred_train = xgb_reg.predict(X_train)\n",
    "y_pred_train[y_pred_train < 0] = 0\n",
    "y_pred_cv = xgb_reg.predict(X_cv)\n",
    "y_pred_cv[y_pred_cv < 0] = 0\n",
    "print(\"Training RMSLE = \", rmsle(y_train, y_pred_train))\n",
    "print(\"CV RMSLE = \", rmsle(y_cv, y_pred_cv))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### None of the boosting models performed well on Dense vector representations\n",
    "- This maybe because most of the words in the given text corpus are not present in the Word2Vec model's vocabulary and therefore, we have to use a vector of all zeros place of those. <br>\n",
    "- Exhaustive GridSearch/RandomSearch was not performed due to memory constraints as the dense vector representations consumed very high RAM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 5: MLP on TF-IDF Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For reproducible results\n",
    "tf.random.set_seed(21)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Since, we took log of the original y_train and y_cv (so that we could directly optimize for MSE), we convert it back\n",
    "to the original format so that this can be used to evaluate model's performance on training data\n",
    "'''\n",
    "y_train_original = np.expm1(y_train.reshape(-1, 1))\n",
    "y_cv_original    = np.expm1(y_cv.reshape(-1, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mlp_model_1(train_shape):\n",
    "    '''\n",
    "    Task  -> This function builds the architecture of an MLP model with the input dimensions as \"train_shape\"\n",
    "             The architecture of the model is as follows:\n",
    "             Input Layer -> Dense (256) -> Dense (128) -> Dense (1) -> Output Layer\n",
    "             The activation function is kept as ReLu for the hidden layers and linear activation (f(x) = x) for the output layer\n",
    "    \n",
    "    Input  -> train_shape: Input shape (dimensions) of the data which will be fed to the MLP\n",
    "    \n",
    "    Output -> Builded MLP Model\n",
    "    '''\n",
    "    model_input = Input(shape=(train_shape,), dtype='float32', sparse=True)\n",
    "    out = Dense(256, activation='relu')(model_input)\n",
    "    out = Dense(128, activation='relu')(out)\n",
    "    model_out = Dense(1)(out)\n",
    "    model = Model(model_input, model_out)\n",
    "    return model\n",
    "    \n",
    "def mlp_model_2(train_shape):\n",
    "    '''\n",
    "    Task  -> This function builds the architecture of an MLP model with the input dimensions as \"train_shape\"\n",
    "             The architecture of the model is as follows:\n",
    "             Input Layer -> Dense (1024) -> Dense (512) -> Dense (256) -> Dense (128) -> Dense (64) -> Dense (32) -> Dense (1)\n",
    "             -> Output Layer\n",
    "             The activation function is kept as ReLu for the hidden layers and linear activation (f(x) = x) for the output layer\n",
    "    \n",
    "    Input  -> train_shape: Input shape (dimensions) of the data which will be fed to the MLP\n",
    "    \n",
    "    Output -> Builded MLP Model\n",
    "    \n",
    "    '''\n",
    "    model_input = Input(shape=(train_shape,), dtype='float32', sparse=True)\n",
    "    out = Dense(1024, activation='relu')(model_input)\n",
    "    out = Dense(512, activation='relu')(out)\n",
    "    out = Dense(256, activation='relu')(out)\n",
    "    out = Dense(128, activation='relu')(out)\n",
    "    out = Dense(64, activation='relu')(out)\n",
    "    out = Dense(32, activation='relu')(out)\n",
    "    out = Dense(1)(out)\n",
    "    model = Model(model_input, out)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Each of these models were trained for only 3 epochs since they heavily overfitted within that only <br> The doubling of batch size after each epoch was a hack that worked brilliantly (inspired by the winners' solution to this problem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training MLP Model  - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4341/4341 [==============================] - 31s 7ms/step - loss: 0.3510\n",
      "2171/2171 [==============================] - 18s 8ms/step - loss: 0.2058\n",
      "1086/1086 [==============================] - 11s 10ms/step - loss: 0.1114\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "We build the MLP Model - 1 using the above function mlp_model_1(), then compile it using the Adam Optimizer, optimizing for\n",
    "the mean squared error. Finally, we fit the model on the training data for 3 epochs doubling the batch size after each epoch\n",
    "with an initial batch size of 256\n",
    "'''\n",
    "\n",
    "mlp1 = mlp_model_1(X_train.shape[1])\n",
    "mlp1.compile(optimizer='adam', loss='mean_squared_error')\n",
    "mlp1.fit(X_train, y_train, batch_size = 256, epochs = 1, verbose = 1)\n",
    "mlp1.fit(X_train, y_train, batch_size = 512, epochs = 1, verbose = 1)\n",
    "mlp1.fit(X_train, y_train, batch_size = 1024, epochs = 1, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predictions from MLP Model - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSLE for MLP model 1 is:  0.18930484114517668\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "We obtain the predictions from the above trained model on the training data and calculate the train RMSLE\n",
    "'''\n",
    "\n",
    "y_pred_train = mlp1.predict(X_train)[:, 0]\n",
    "y_pred_train_mlp_1 = np.expm1(y_pred_train.reshape(-1, 1))[:, 0]\n",
    "print(\"Train RMSLE for MLP model 1 is: \", rmsle(y_train_original, y_pred_train_mlp_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV RMSLE for MLP model 1 is:  0.4119441261549299\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "We obtain the predictions from the above trained model on the CV data and calculate the CV RMSLE\n",
    "'''\n",
    "y_pred_cv = mlp1.predict(X_cv)[:, 0]\n",
    "y_pred_cv_mlp_1 = np.expm1(y_pred_cv.reshape(-1, 1))[:, 0]\n",
    "print(\"CV RMSLE for MLP model 1 is: \", rmsle(y_cv_original, y_pred_cv_mlp_1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training MLP Model  - 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4341/4341 [==============================] - 85s 20ms/step - loss: 0.3514\n",
      "2171/2171 [==============================] - 47s 22ms/step - loss: 0.1900\n",
      "1086/1086 [==============================] - 28s 26ms/step - loss: 0.0854\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "We build the MLP Model - 2 using the above function mlp_model_2(), then compile it using the Adam Optimizer, optimizing for\n",
    "the mean squared error. Finally, we fit the model on the training data for 3 epochs doubling the batch size after each epoch\n",
    "with an initial batch size of 256\n",
    "'''\n",
    "\n",
    "mlp2 = mlp_model_2(X_train.shape[1])\n",
    "mlp2.compile(optimizer='adam', loss='mean_squared_error')\n",
    "mlp2.fit(X_train, y_train, batch_size = 256, epochs = 1, verbose = 1)\n",
    "mlp2.fit(X_train, y_train, batch_size = 512, epochs = 1, verbose = 1)\n",
    "mlp2.fit(X_train, y_train, batch_size = 1024, epochs = 1, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predictions from MLP Model - 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSLE for MLP model 2 is:  0.15686919200114158\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "We obtain the predictions from the above trained model on the training data and calculate the train RMSLE\n",
    "'''\n",
    "y_pred_train = mlp2.predict(X_train)[:, 0]\n",
    "y_pred_train_mlp_2 = np.expm1(y_scaler.inverse_transform(y_pred_train.reshape(-1, 1))[:, 0])\n",
    "print(\"Train RMSLE for MLP model 2 is: \", rmsle(y_train_original, y_pred_train_mlp_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV RMSLE for MLP model 2 is:  0.40526546757131204\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "We obtain the predictions from the above trained model on the CV data and calculate the CV RMSLE\n",
    "'''\n",
    "y_pred_cv = mlp2.predict(X_cv)[:, 0]\n",
    "y_pred_cv_mlp_2 = np.expm1(y_pred_cv.reshape(-1, 1))[:, 0]\n",
    "print(\"CV RMSLE for MLP model 2 is: \", rmsle(y_cv_original, y_pred_cv_mlp_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#This code is inspired from: https://github.com/debayanmitra1993-data/Mercari-Price-Recommendation/blob/master/kaggle_sub.py\n",
    "\n",
    "def ensemble_generator(mlp1_preds, mlp2_preds):\n",
    "    '''\n",
    "    \n",
    "    Task   -> This function calculates the best ensemble we can generate from predictions of MLP model 1 and MLP model 2.\n",
    "              This is done by assigning a weight of 'w' to the first model's predictions and a weight of '(1-w)' to the second\n",
    "              model's predictions. This is like weighted averaging of the two models to generate a better model.\n",
    "              We select w using hyperparameter search, with, w in range [0, 0.02, 0.04, 0.06, ..., 1)\n",
    "              Best w is selected which gives the lowest RMSLE on the CV data\n",
    "              The best w along with the final weighted predictions are returned as an output\n",
    "    \n",
    "    Input  -> Tuple: (MLP_Model_1_Predictions, MLP_Model_2_Predictions)\n",
    "    \n",
    "    Output -> Tuple: (best_w, final_weighted_predictions using averaging from best_w)\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    weights = np.arange(0, 1, 0.02)\n",
    "    scores = []\n",
    "    \n",
    "    for w in weights:\n",
    "        preds_f = (w*mlp1_preds) + (1-w)*(mlp2_preds)\n",
    "        scores.append(rmsle(y_cv, preds_f))\n",
    "    \n",
    "    min_rmsle_index = np.argmin(scores)\n",
    "    \n",
    "    w_min_rmsle = weights[min_rmsle_index]\n",
    "\n",
    "    preds_final = (w_min_rmsle*mlp1_preds) + (1-w_min_rmsle)*(mlp2_preds)\n",
    "    \n",
    "    return w_min_rmsle, preds_final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating performance of the Ensemble model on training and CV data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best w = 0.4\n",
      "CV RMSLE for ensemble of MLP-1 and MLP-2 is: 0.3986\n"
     ]
    }
   ],
   "source": [
    "w_best, final_predictions_cv = ensemble_generator(y_pred_cv_mlp_1, y_pred_cv_mlp_2)\n",
    "print(\"Best w =\", w_best)\n",
    "print(\"CV RMSLE for ensemble of MLP-1 and MLP-2 is:\", np.round(rmsle(y_cv_original, final_predictions_cv), 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSLE for ensemble of MLP-1 and MLP-2 is: 0.1511\n"
     ]
    }
   ],
   "source": [
    "final_predictions_train = w_best*y_pred_train_mlp_1 + (1-w_best)*y_pred_train_mlp_2\n",
    "print(\"Train RMSLE for ensemble of MLP-1 and MLP-2 is:\", np.round(rmsle(y_train_original, final_predictions_train), 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the model's performance on unseen testing data (on Kaggle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test = pd.read_csv('test_stg2.tsv', sep='\\t')\n",
    "test_ids = df_test['test_id'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Handling Null Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test.fillna('', inplace=True)\n",
    "df_test['item_description']  = df_test['item_description'].str.replace('^no description yet$', '', regex=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Combining text fields together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test['name'] = df_test['name'] + \" \" + df_test['brand_name']\n",
    "df_test['text'] = df_test['item_description'] + \" \" + df_test['name'] + \" \" + df_test['category_name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning Text Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ref: AAIC Notebook for Donors' Choose\n",
    "def decontracted(sent):\n",
    "    '''\n",
    "    Task:   This Function changes common short forms like can't, won't to can not, will not resp. (Decontraction)\n",
    "            This is done to ensure uniformity in the whole text\n",
    "    Input:  Raw Text\n",
    "    Output: Decontracted Text\n",
    "    '''\n",
    "    sent = re.sub(r\"aren\\'t\", \"are not\", sent)\n",
    "    sent = re.sub(r\"didn\\'t\", \"did not\", sent)\n",
    "    sent = re.sub(r\"can\\'t\", \"can not\", sent)\n",
    "    sent = re.sub(r\"couldn\\'t\", \"could not\", sent)\n",
    "    sent = re.sub(r\"won\\'t\", \"would not\", sent)\n",
    "    sent = re.sub(r\"wouldn\\'t\", \"would not\", sent)\n",
    "    sent = re.sub(r\"haven\\'t\", \"have not\", sent)\n",
    "    sent = re.sub(r\"shouldn\\'t\", \"should not\", sent)\n",
    "    sent = re.sub(r\"doesn\\'t\", \"does not\", sent)\n",
    "    sent = re.sub(r\"don\\'t\", \"do not\", sent)\n",
    "    sent = re.sub(r\"didn\\'t\", \"did not\", sent)\n",
    "    sent = re.sub(r\"mustn\\'t\", \"must not\", sent)\n",
    "    sent = re.sub(r\"needn\\'t\", \"need not\", sent)\n",
    "    \n",
    "    return sent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test['name'] = df_test['name'].apply(lambda x : decontracted(x))\n",
    "df_test['text'] = df_test['text'].apply(lambda x : decontracted(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Defining some special regexes which would be used in the function text_preprocessing() to clean the text\n",
    "regex_special_chars = re.compile('[^A-Za-z0-9.]+')\n",
    "regex_decimal_digits = re.compile('(?<!\\d)\\.(?!\\d)')\n",
    "regex_white_space = re.compile(r'\\s+')           \n",
    "    \n",
    "#Creating a slightly modified list of stopwords which does not contain \"no\", \"nor\" or \"not\"\n",
    "stop_words = set(stopwords.words(\"english\")) - {\"no\", \"nor\", \"not\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_preprocessing(sent):\n",
    "    '''\n",
    "    Input  -> Raw text (string)\n",
    "    Output -> Cleaned Text (string)\n",
    "    Task   -> The objective of this function is to clean the text and make it suitable for Bag of Words/TF-IDF vectorization\n",
    "              This includes removal of new lines, special characters, emojis etc.\n",
    "    \n",
    "    '''\n",
    "    #Removing special characters such as carriage return and newline character\n",
    "    sent = sent.replace('\\\\r', ' ')\n",
    "    sent = sent.replace('\\\\n', ' ')\n",
    "\n",
    "    #Removing all special characters except the period\n",
    "    sent = regex_special_chars.sub(' ', sent)\n",
    "    \n",
    "    #Removing periods which are not either followed or preceeded by a digit\n",
    "    #Ref: https://stackoverflow.com/questions/6599646/remove-decimal-point-when-not-between-two-digits\n",
    "    \n",
    "    sent = regex_decimal_digits.sub(' ', sent)\n",
    "    \n",
    "    #Converting multiple white spaces to single white space\n",
    "    sent = regex_white_space.sub(' ', sent)\n",
    "    \n",
    "    #Removing space at starting and ending and converting to lower case\n",
    "    sent = sent.strip().lower()\n",
    "    \n",
    "    # Lemmatizing the text: Lemmetization in NLP means to convert similar words to the same word while taking care of grammar\n",
    "    sent_list = sent.split()\n",
    "    lem = WordNetLemmatizer()\n",
    "    text = [lem.lemmatize(word) for word in sent_list if word not in stop_words] \n",
    "    sent = \" \".join(text)\n",
    "    \n",
    "    return sent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test['name'] = df_test['name'].apply(lambda x : text_preprocessing(x))\n",
    "df_test['text'] = df_test['text'].apply(lambda x : text_preprocessing(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TF-IDF Encoding the text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tfidf_encoder_evaluate(train_data, test_data, N_GRAMS = 1):\n",
    "    '''\n",
    "    \n",
    "    This function returns the TF-IDF encoding of the test data\n",
    "    \n",
    "    Input ->\n",
    "    \n",
    "        train_data       : Text (string or list of strings or Pandas Series with elements as strings)\n",
    "        test_data        : Text (string or list of strings or Pandas Series with elements as strings)\n",
    "        n_grams(int)     : Upper bound of the n_grams to be considered while vectorizing the data using TF-IDF encoder\n",
    "                           For eg., If the n_grams = 2, then both unigrams and bi-grams will be used while vectorizing\n",
    "                           the text data. Default value is kept as 1, which means only uni-grams will be generated if this\n",
    "                           argument is not supplied explicitly while calling this function\n",
    "        MaxFeatures(int) : Maximum number of features that the TF-IDF vectorizer will consider while vectorizing the text data\n",
    "                           provided as input in the \"train_data\" and \"test_data\" arguments. Default value in kept as 10,000\n",
    "    \n",
    "    Output -> Tuple of TF-IDF vectors of \"test_data\" computed using sklearn's Tfidfvectorizer() with parameters\n",
    "              as \"n_grams\" and \"MaxFeatures\"\n",
    "    \n",
    "    Task   -> Given a text (string), return the TF-IDF vectors for that text\n",
    "              The vectorizer is fitted on the train_data and used to tranform the test data\n",
    "    \n",
    "    '''\n",
    "    vectorizer = TfidfVectorizer(max_features = 100000,\n",
    "                                 ngram_range = (1, N_GRAMS),\n",
    "                                 strip_accents = 'unicode',\n",
    "                                 analyzer = 'word',\n",
    "                                 token_pattern = r'\\w+')\n",
    "    \n",
    "    vectorizer.fit(train_data)\n",
    "    \n",
    "    test_tfidf = vectorizer.transform(test_data)\n",
    "    \n",
    "    del vectorizer\n",
    "    gc.collect()\n",
    "    \n",
    "    return test_tfidf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test_name = tfidf_encoder_evaluate(df_train['name'], df_test['name'], N_GRAMS = 1)\n",
    "X_test_text = tfidf_encoder_evaluate(df_train['text'], df_test['text'], N_GRAMS = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One Hot Encoding for Shipping and item_condition_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_hot_encoder_evaluate(train_data, test_data):\n",
    "    '''\n",
    "    This function returns the One Hot Encoded vectors for the given train and test data\n",
    "    Input ->\n",
    "        train_data : Training data to be fitted on (List of integers/strings or a Pandas Series)\n",
    "        test_data  : Testing data to be one hot encoded (List of integers/strings or a Pandas Series)\n",
    "    Output -> Tuple of One hot encoded vectors of testing data\n",
    "    Task   -> This function converts the raw values (integers/strings) into one hot encoded vectors using\n",
    "              sklearn's OneHotEncoder()\n",
    "    '''\n",
    "    ohe_encoder = OneHotEncoder()\n",
    "    ohe_encoder.fit(train_data)\n",
    "    test_ohe = ohe_encoder.transform(test_data)\n",
    "    return test_ohe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test_shipping       = one_hot_encoder_evaluate(np.reshape(df_train['shipping'].values, (-1, 1)),\n",
    "                                                 np.reshape(df_test['shipping'].values, (-1, 1)))\n",
    "\n",
    "X_test_item_condition = one_hot_encoder_evaluate(np.reshape(df_train['item_condition_id'].values, (-1, 1)),\n",
    "                                                 np.reshape(df_test['item_condition_id'].values, (-1, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Combining all the features to create the test data matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = hstack((X_test_name,\n",
    "                 X_test_text,\n",
    "                 X_test_shipping,\n",
    "                 X_test_item_condition)).tocsr().astype('float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3460725, 190470)\n"
     ]
    }
   ],
   "source": [
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting the final outputs using MLP Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "We obtain the predictions from the above trained model on the Test data\n",
    "'''\n",
    "y_pred_test       = mlp1.predict(X_test)[:, 0]\n",
    "y_pred_test_mlp_1 = np.expm1(y_pred_test.reshape(-1, 1))[:, 0]\n",
    "\n",
    "y_pred_test       = mlp2.predict(X_test)[:, 0]\n",
    "y_pred_test_mlp_2 = np.expm1(y_pred_test.reshape(-1, 1))[:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "w_best = 0.4\n",
    "final_predictions_test = w_best*y_pred_test_mlp_1 + (1-w_best)*y_pred_train_mlp_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Storing the result to a csv file to submit to Kaggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission = pd.DataFrame({'test_id' : test_ids, 'price' : final_predictions_test})\n",
    "submission.to_csv(\"submission.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test_id</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>6.283154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>9.482358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>50.164560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>12.036359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>8.741852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>7.547048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>8.493185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>25.180970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>62.259933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>11.921490</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   test_id      price\n",
       "0        0   6.283154\n",
       "1        1   9.482358\n",
       "2        2  50.164560\n",
       "3        3  12.036359\n",
       "4        4   8.741852\n",
       "5        5   7.547048\n",
       "6        6   8.493185\n",
       "7        7  25.180970\n",
       "8        8  62.259933\n",
       "9        9  11.921490"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------------------+---------------------------+-------+\n",
      "|             Model              |       Vectorization       | RMSLE |\n",
      "+--------------------------------+---------------------------+-------+\n",
      "|        Ridge Regression        |   TF-IDF Sparse vectors   | 0.455 |\n",
      "|        Lasso Regression        |   TF-IDF Sparse vectors   | 0.746 |\n",
      "|            XGBoost             |   TF-IDF Sparse vectors   | 0.490 |\n",
      "|            XGBoost             | Average W2V dense vectors | 0.552 |\n",
      "|         MLP Model - 1          |   TF-IDF Sparse vectors   | 0.412 |\n",
      "|         MLP Model - 2          |   TF-IDF Sparse vectors   | 0.405 |\n",
      "| Ensemble Model (MLP-1 + MLP-2) |   TF-IDF Sparse vectors   | 0.398 |\n",
      "+--------------------------------+---------------------------+-------+\n"
     ]
    }
   ],
   "source": [
    "x = PrettyTable()\n",
    "\n",
    "x.field_names = [\"Model\", \"Vectorization\", \"RMSLE\"]\n",
    "\n",
    "x.add_row([\"Ridge Regression\", \"TF-IDF Sparse vectors\", \"0.455\"])\n",
    "x.add_row([\"Lasso Regression\", \"TF-IDF Sparse vectors\", \"0.746\"])\n",
    "x.add_row([\"XGBoost\", \"TF-IDF Sparse vectors\", \"0.490\"])\n",
    "x.add_row([\"XGBoost\", \"Average W2V dense vectors\", \"0.552\"])\n",
    "x.add_row([\"MLP Model - 1\", \"TF-IDF Sparse vectors\", \"0.412\"])\n",
    "x.add_row([\"MLP Model - 2\", \"TF-IDF Sparse vectors\", \"0.405\"])\n",
    "x.add_row([\"Ensemble Model (MLP-1 + MLP-2)\", \"TF-IDF Sparse vectors\", \"0.398\"])\n",
    "\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. The best model we got was an ensemble of MLP Model 1 and MLP Model 2. It had a train RMSLE of 0.151 and a <b> CV RMSLE of 0.398 </b> <br> <br>\n",
    "2. Rest all the models could not capture the information present in the data and thus had a relatively higher bias but reduced variance, while, the MLP models, being Deep Neural Networks, were able to capture the information in the data and had very less bias but a high variance. <br> <br>\n",
    "3. New features like historical price statistics which were engineered during Feature Engineering phase were not finally used as  these caused significant decrease in model's performance, probably, because these were causing overiftting due to being very focused towards the training data <br> <br>\n",
    "4. Sparse vecrorizations (TF-IDF) were used instead of dense vectorizations (Word2Vec) as most of the words (~70%) in the given text corpus were not present in the vocabulary of Word2Vec model trained Google News dataset and hence the information present in the corpus was not utilized at its full potential. <br> <br>\n",
    "5. Ridge Regression model is a very simple model, still, it performed quite well due to the presence of high number of features. Lasso Regression typically works better in case of sparse vectorization, but, it failed to converge in this case and hence could not be used. XGBoost models are complex tree based ensembles and hence these don't perform well with high dimensional data. Finally, the MLP models performed very well on the test data, but, as these are deep neural networks, these are prone to overfitting. <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Kaggle Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final RMSLE obtained on the unseen test data of 3.5 million rows on Kaggle was 0.405, which resulted in <b> 16th position out of 2380 participants </b>, that is, in <b>top 1% of the final results.</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Kaggle_Submission](https://i.imgur.com/BS8zTqz.png)\n",
    "![Kaggle_Leaderboard](https://i.imgur.com/7m7jXB9.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}