apache_ecommerce.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f40dac6b",
   "metadata": {},
   "source": [
    "# Ecommerce Sales Prediction using Apache Spark\n",
    "#### Contributor(s): Yash Sethia, Ritesh Kumar, Shubham \n",
    "![Snapshot](https://www.edureka.co/blog/wp-content/uploads/2018/09/Picture5-2.png)\n",
    "----------\n",
    "### About Apache Spark\n",
    "\n",
    "*Apache Spark*, written in Scala, is a general-purpose distributed data processing engine. Or in other words: load big data, do computations on it in a distributed way, and then store it. <br/> <br/>\n",
    "Apache Spark contains libraries for data analysis, machine learning, graph analysis, and streaming live data. Spark is generally faster than *Hadoop*. \n",
    "This is because Hadoop writes intermediate results to disk (that is, lots of I/O operations) whereas Spark tries to keep intermediate results in memory \n",
    "(that is, in-memory computation) whenever possible. Moreover, Spark offers lazy evaluation of operations and optimizes them just before the final result; \n",
    "Sparks maintains a series of transformations that are to be performed without actually performing those operations unless we try to obtain the results. \n",
    "This way, Spark is able to find the best path looking at overall transformations required (for example, reducing two separate steps of adding number 5 and\n",
    "20 to each element of the dataset into just a single step of adding 25 to each element of the dataset, or not actually doing operations on part of the dataset\n",
    "which will eventually will be filtered out in the final result). \n",
    "<br/><br/>\n",
    "This makes Spark one of the most popular tools for big data analytics currently.\n",
    "\n",
    "PySpark is an interface for Apache Spark in Python. It not only allows you to write Spark applications using Python APIs, but also provides the PySpark shell for interactively analyzing your data in a distributed environment.\n",
    "\n",
    "In this Notebook, we have the data of an Ecommerce website with the following fields:\n",
    "Definitions:\n",
    "* Email ID of the user\n",
    "* Address of the user\n",
    "* Average Session Length of the user\n",
    "* Time spent by the user on the app\n",
    "* Time spent by the user on the website\n",
    "* Length of the membership of the user\n",
    "* Yearly amount spent by the user\n",
    "\n",
    "We used Pyspark API of Apache Spark to handle this data to bring out meaningful inferences and try to predict the Amount spent by users using this data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e6b9972",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyspark\n",
    "!pip install findspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2bd57728",
   "metadata": {},
   "outputs": [],
   "source": [
    "import findspark\n",
    "findspark.init()\n",
    "import pyspark\n",
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.getOrCreate()\n",
    "from pyspark.sql.functions import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "28bbcbd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = ['./Ecommerce_Customers.csv',\n",
    "        './Ecommerce_Customers1.csv',\n",
    "        './Ecommerce_Customers2.csv'\n",
    "       ]\n",
    " \n",
    "dataset = spark.read.options(inferSchema = True, header=True).csv(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f0e5801",
   "metadata": {},
   "source": [
    "## Data Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a87a9885",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Email: string (nullable = true)\n",
      " |-- Address: string (nullable = true)\n",
      " |-- Avg Session Length: double (nullable = true)\n",
      " |-- Time on App: double (nullable = true)\n",
      " |-- Time on Website: double (nullable = true)\n",
      " |-- Length of Membership: double (nullable = true)\n",
      " |-- Yearly Amount Spent: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dataset.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b00588a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+------------------+-----------+---------------+--------------------+-------------------+\n",
      "|               Email|             Address|Avg Session Length|Time on App|Time on Website|Length of Membership|Yearly Amount Spent|\n",
      "+--------------------+--------------------+------------------+-----------+---------------+--------------------+-------------------+\n",
      "|    ryan36@gmail.com|5379 Rhonda Prair...|       32.22729914|13.72862718|    37.99702801|         4.802630631|        613.5993234|\n",
      "|    chris67@ryan.biz|716 Bush Greens A...|       32.78977262|11.67006592|    37.40874848|         3.414688423|        483.1597208|\n",
      "|   katie25@gmail.com|6861 Lopez Fork A...|       32.77260993|13.27631301|    36.60077705|         3.462298847|        540.2634004|\n",
      "|      kyang@diaz.org|223 Love Trail Su...|       34.37425805|15.12699429|    37.15762409|         5.377593584|        765.5184619|\n",
      "|tadams@contreras....|049 Matthew Terra...|       33.07871721|12.69578975|    35.35844431|         4.001786346|        553.6015347|\n",
      "+--------------------+--------------------+------------------+-----------+---------------+--------------------+-------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dataset.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cfc3947f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1f01a15d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Email</th>\n",
       "      <th>Address</th>\n",
       "      <th>Avg Session Length</th>\n",
       "      <th>Time on App</th>\n",
       "      <th>Time on Website</th>\n",
       "      <th>Length of Membership</th>\n",
       "      <th>Yearly Amount Spent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ryan36@gmail.com</td>\n",
       "      <td>5379 Rhonda Prairie Apt. 696Brownland, WA 87027</td>\n",
       "      <td>32.227299</td>\n",
       "      <td>13.728627</td>\n",
       "      <td>37.997028</td>\n",
       "      <td>4.802631</td>\n",
       "      <td>613.599323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chris67@ryan.biz</td>\n",
       "      <td>716 Bush Greens Apt. 098Trevorton, MA 17817-8000</td>\n",
       "      <td>32.789773</td>\n",
       "      <td>11.670066</td>\n",
       "      <td>37.408748</td>\n",
       "      <td>3.414688</td>\n",
       "      <td>483.159721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>katie25@gmail.com</td>\n",
       "      <td>6861 Lopez Fork Apt. 114South Jamiechester, DE...</td>\n",
       "      <td>32.772610</td>\n",
       "      <td>13.276313</td>\n",
       "      <td>36.600777</td>\n",
       "      <td>3.462299</td>\n",
       "      <td>540.263400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>kyang@diaz.org</td>\n",
       "      <td>223 Love Trail Suite 831Port Jeffrey, IN 46849</td>\n",
       "      <td>34.374258</td>\n",
       "      <td>15.126994</td>\n",
       "      <td>37.157624</td>\n",
       "      <td>5.377594</td>\n",
       "      <td>765.518462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>tadams@contreras.info</td>\n",
       "      <td>049 Matthew TerraceLake Matthew, MS 20210</td>\n",
       "      <td>33.078717</td>\n",
       "      <td>12.695790</td>\n",
       "      <td>35.358444</td>\n",
       "      <td>4.001786</td>\n",
       "      <td>553.601535</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Email                                            Address  \\\n",
       "0       ryan36@gmail.com    5379 Rhonda Prairie Apt. 696Brownland, WA 87027   \n",
       "1       chris67@ryan.biz   716 Bush Greens Apt. 098Trevorton, MA 17817-8000   \n",
       "2      katie25@gmail.com  6861 Lopez Fork Apt. 114South Jamiechester, DE...   \n",
       "3         kyang@diaz.org     223 Love Trail Suite 831Port Jeffrey, IN 46849   \n",
       "4  tadams@contreras.info          049 Matthew TerraceLake Matthew, MS 20210   \n",
       "\n",
       "   Avg Session Length  Time on App  Time on Website  Length of Membership  \\\n",
       "0           32.227299    13.728627        37.997028              4.802631   \n",
       "1           32.789773    11.670066        37.408748              3.414688   \n",
       "2           32.772610    13.276313        36.600777              3.462299   \n",
       "3           34.374258    15.126994        37.157624              5.377594   \n",
       "4           33.078717    12.695790        35.358444              4.001786   \n",
       "\n",
       "   Yearly Amount Spent  \n",
       "0           613.599323  \n",
       "1           483.159721  \n",
       "2           540.263400  \n",
       "3           765.518462  \n",
       "4           553.601535  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.limit(5).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1f279594",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Email</th>\n",
       "      <th>Address</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ryan36@gmail.com</td>\n",
       "      <td>5379 Rhonda Prairie Apt. 696Brownland, WA 87027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chris67@ryan.biz</td>\n",
       "      <td>716 Bush Greens Apt. 098Trevorton, MA 17817-8000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>katie25@gmail.com</td>\n",
       "      <td>6861 Lopez Fork Apt. 114South Jamiechester, DE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>kyang@diaz.org</td>\n",
       "      <td>223 Love Trail Suite 831Port Jeffrey, IN 46849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>tadams@contreras.info</td>\n",
       "      <td>049 Matthew TerraceLake Matthew, MS 20210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>craigcastro@burgess.com</td>\n",
       "      <td>65407 Warner Forges Suite 071New Lori, AZ 3013...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ellen24@anthony.com</td>\n",
       "      <td>PSC 3408, Box 3353APO AE 05113-1257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>davisrobert@hicks-smith.com</td>\n",
       "      <td>95967 Pitts BurgsWest Sarah, MH 27817-5147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>walkerlaura@peterson-yates.com</td>\n",
       "      <td>89154 Jones Stream Suite 311East Katie, NC 727...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>marc54@hotmail.com</td>\n",
       "      <td>Unit 6174 Box 1949DPO AP 27478</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            Email  \\\n",
       "0                ryan36@gmail.com   \n",
       "1                chris67@ryan.biz   \n",
       "2               katie25@gmail.com   \n",
       "3                  kyang@diaz.org   \n",
       "4           tadams@contreras.info   \n",
       "5         craigcastro@burgess.com   \n",
       "6             ellen24@anthony.com   \n",
       "7     davisrobert@hicks-smith.com   \n",
       "8  walkerlaura@peterson-yates.com   \n",
       "9              marc54@hotmail.com   \n",
       "\n",
       "                                             Address  \n",
       "0    5379 Rhonda Prairie Apt. 696Brownland, WA 87027  \n",
       "1   716 Bush Greens Apt. 098Trevorton, MA 17817-8000  \n",
       "2  6861 Lopez Fork Apt. 114South Jamiechester, DE...  \n",
       "3     223 Love Trail Suite 831Port Jeffrey, IN 46849  \n",
       "4          049 Matthew TerraceLake Matthew, MS 20210  \n",
       "5  65407 Warner Forges Suite 071New Lori, AZ 3013...  \n",
       "6                PSC 3408, Box 3353APO AE 05113-1257  \n",
       "7         95967 Pitts BurgsWest Sarah, MH 27817-5147  \n",
       "8  89154 Jones Stream Suite 311East Katie, NC 727...  \n",
       "9                     Unit 6174 Box 1949DPO AP 27478  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = dataset.select(\"Email\", \"Address\")\n",
    "ds.limit(10).toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9f29efe",
   "metadata": {},
   "source": [
    "## Descriptive Analytics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0a0fd188",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Email</th>\n",
       "      <th>Address</th>\n",
       "      <th>Avg Session Length</th>\n",
       "      <th>Time on App</th>\n",
       "      <th>Time on Website</th>\n",
       "      <th>Length of Membership</th>\n",
       "      <th>Yearly Amount Spent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>kyang@diaz.org</td>\n",
       "      <td>223 Love Trail Suite 831Port Jeffrey, IN 46849</td>\n",
       "      <td>34.374258</td>\n",
       "      <td>15.126994</td>\n",
       "      <td>37.157624</td>\n",
       "      <td>5.377594</td>\n",
       "      <td>765.518462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>asilva@yahoo.com</td>\n",
       "      <td>USNV JohnsonFPO AP 19026</td>\n",
       "      <td>34.603311</td>\n",
       "      <td>12.207298</td>\n",
       "      <td>33.913847</td>\n",
       "      <td>6.922689</td>\n",
       "      <td>744.221867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>william82@gmail.com</td>\n",
       "      <td>11143 Park SquaresSamanthatown, UT 97073</td>\n",
       "      <td>33.256335</td>\n",
       "      <td>13.858062</td>\n",
       "      <td>37.780265</td>\n",
       "      <td>5.976768</td>\n",
       "      <td>725.584814</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>jeffrey54@mcdonald-williams.com</td>\n",
       "      <td>297 Francis ValleySouth Lindsey, NY 13669-5367</td>\n",
       "      <td>34.967610</td>\n",
       "      <td>13.919494</td>\n",
       "      <td>37.952013</td>\n",
       "      <td>5.066697</td>\n",
       "      <td>712.396327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>rhonda01@gmail.com</td>\n",
       "      <td>939 Watson RunStaceyberg, VT 58376-0454</td>\n",
       "      <td>34.385820</td>\n",
       "      <td>12.729720</td>\n",
       "      <td>36.232110</td>\n",
       "      <td>5.705941</td>\n",
       "      <td>708.935185</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Email  \\\n",
       "0                   kyang@diaz.org   \n",
       "1                 asilva@yahoo.com   \n",
       "2              william82@gmail.com   \n",
       "3  jeffrey54@mcdonald-williams.com   \n",
       "4               rhonda01@gmail.com   \n",
       "\n",
       "                                          Address  Avg Session Length  \\\n",
       "0  223 Love Trail Suite 831Port Jeffrey, IN 46849           34.374258   \n",
       "1                        USNV JohnsonFPO AP 19026           34.603311   \n",
       "2        11143 Park SquaresSamanthatown, UT 97073           33.256335   \n",
       "3  297 Francis ValleySouth Lindsey, NY 13669-5367           34.967610   \n",
       "4         939 Watson RunStaceyberg, VT 58376-0454           34.385820   \n",
       "\n",
       "   Time on App  Time on Website  Length of Membership  Yearly Amount Spent  \n",
       "0    15.126994        37.157624              5.377594           765.518462  \n",
       "1    12.207298        33.913847              6.922689           744.221867  \n",
       "2    13.858062        37.780265              5.976768           725.584814  \n",
       "3    13.919494        37.952013              5.066697           712.396327  \n",
       "4    12.729720        36.232110              5.705941           708.935185  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Arraged according to Amount Spent\n",
    "dataset.orderBy('Yearly Amount Spent', ascending=False).limit(5).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "68456bbe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Email</th>\n",
       "      <th>Address</th>\n",
       "      <th>Avg Session Length</th>\n",
       "      <th>Time on App</th>\n",
       "      <th>Time on Website</th>\n",
       "      <th>Length of Membership</th>\n",
       "      <th>Yearly Amount Spent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>asilva@yahoo.com</td>\n",
       "      <td>USNV JohnsonFPO AP 19026</td>\n",
       "      <td>34.603311</td>\n",
       "      <td>12.207298</td>\n",
       "      <td>33.913847</td>\n",
       "      <td>6.922689</td>\n",
       "      <td>744.221867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>alicia85@lee.com</td>\n",
       "      <td>14220 Carla Flat Suite 521Lake Matthew, DE 06183</td>\n",
       "      <td>32.887105</td>\n",
       "      <td>12.387184</td>\n",
       "      <td>37.431159</td>\n",
       "      <td>6.401229</td>\n",
       "      <td>684.163431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>waltonkaren@gmail.com</td>\n",
       "      <td>355 Villegas Isle Apt. 070West Jenniferview, N...</td>\n",
       "      <td>35.742670</td>\n",
       "      <td>10.889828</td>\n",
       "      <td>35.565436</td>\n",
       "      <td>6.115199</td>\n",
       "      <td>669.987141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ebrown@osborne.com</td>\n",
       "      <td>4291 Nichols Fork Apt. 562Thomaschester, SC 08...</td>\n",
       "      <td>31.945396</td>\n",
       "      <td>12.965761</td>\n",
       "      <td>36.966389</td>\n",
       "      <td>6.076654</td>\n",
       "      <td>657.019924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>william82@gmail.com</td>\n",
       "      <td>11143 Park SquaresSamanthatown, UT 97073</td>\n",
       "      <td>33.256335</td>\n",
       "      <td>13.858062</td>\n",
       "      <td>37.780265</td>\n",
       "      <td>5.976768</td>\n",
       "      <td>725.584814</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Email                                            Address  \\\n",
       "0       asilva@yahoo.com                           USNV JohnsonFPO AP 19026   \n",
       "1       alicia85@lee.com   14220 Carla Flat Suite 521Lake Matthew, DE 06183   \n",
       "2  waltonkaren@gmail.com  355 Villegas Isle Apt. 070West Jenniferview, N...   \n",
       "3     ebrown@osborne.com  4291 Nichols Fork Apt. 562Thomaschester, SC 08...   \n",
       "4    william82@gmail.com           11143 Park SquaresSamanthatown, UT 97073   \n",
       "\n",
       "   Avg Session Length  Time on App  Time on Website  Length of Membership  \\\n",
       "0           34.603311    12.207298        33.913847              6.922689   \n",
       "1           32.887105    12.387184        37.431159              6.401229   \n",
       "2           35.742670    10.889828        35.565436              6.115199   \n",
       "3           31.945396    12.965761        36.966389              6.076654   \n",
       "4           33.256335    13.858062        37.780265              5.976768   \n",
       "\n",
       "   Yearly Amount Spent  \n",
       "0           744.221867  \n",
       "1           684.163431  \n",
       "2           669.987141  \n",
       "3           657.019924  \n",
       "4           725.584814  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Arraged according to Length of membership\n",
    "dataset.orderBy('Length of Membership', ascending=False).limit(5).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "eb3b913c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----------------+--------------------+\n",
      "|summary|            Email|             Address|\n",
      "+-------+-----------------+--------------------+\n",
      "|  count|              500|                 500|\n",
      "|   mean|             null|                null|\n",
      "| stddev|             null|                null|\n",
      "|    min|aaron04@yahoo.com|0001 Mack MillNor...|\n",
      "|    max|zscott@wright.com|Unit 7502 Box 834...|\n",
      "+-------+-----------------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dataset.describe('Email', 'Address').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f498f8ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------------+\n",
      "|avg(Length of Membership)|\n",
      "+-------------------------+\n",
      "|        3.533461555930001|\n",
      "+-------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dataset.agg({'Length of Membership':'mean'}).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4fb6ee2e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------+\n",
      "|avg(Yearly Amount Spent)|\n",
      "+------------------------+\n",
      "|       499.3140382607999|\n",
      "+------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dataset.agg({'Yearly Amount Spent':'mean'}).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "733e9710",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>summary</th>\n",
       "      <th>Email</th>\n",
       "      <th>Address</th>\n",
       "      <th>Avg Session Length</th>\n",
       "      <th>Time on App</th>\n",
       "      <th>Time on Website</th>\n",
       "      <th>Length of Membership</th>\n",
       "      <th>Yearly Amount Spent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>count</td>\n",
       "      <td>500</td>\n",
       "      <td>500</td>\n",
       "      <td>500</td>\n",
       "      <td>500</td>\n",
       "      <td>500</td>\n",
       "      <td>500</td>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>33.053193518240015</td>\n",
       "      <td>12.052487936927994</td>\n",
       "      <td>37.06044542108</td>\n",
       "      <td>3.533461555930001</td>\n",
       "      <td>499.3140382607999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>stddev</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>0.9925631111602917</td>\n",
       "      <td>0.9942156084624618</td>\n",
       "      <td>1.0104889068105996</td>\n",
       "      <td>0.9992775024367541</td>\n",
       "      <td>79.31478155115916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>min</td>\n",
       "      <td>aaron04@yahoo.com</td>\n",
       "      <td>0001 Mack MillNorth Jennifer, NE 42021-5936</td>\n",
       "      <td>29.53242897</td>\n",
       "      <td>8.508152176</td>\n",
       "      <td>33.91384725</td>\n",
       "      <td>0.26990109</td>\n",
       "      <td>256.6705823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>25%</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>32.33889932</td>\n",
       "      <td>11.38677555</td>\n",
       "      <td>36.33952101</td>\n",
       "      <td>2.926940235</td>\n",
       "      <td>444.9665517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>50%</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>33.07871721</td>\n",
       "      <td>11.98204499</td>\n",
       "      <td>37.06708997</td>\n",
       "      <td>3.53286158</td>\n",
       "      <td>498.6355985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>75%</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>33.71065306</td>\n",
       "      <td>12.75207661</td>\n",
       "      <td>37.71598618</td>\n",
       "      <td>4.125584363</td>\n",
       "      <td>549.1315733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>max</td>\n",
       "      <td>zscott@wright.com</td>\n",
       "      <td>Unit 7502 Box 8345DPO AE 53747</td>\n",
       "      <td>36.13966249</td>\n",
       "      <td>15.12699429</td>\n",
       "      <td>40.00518164</td>\n",
       "      <td>6.922689335</td>\n",
       "      <td>765.5184619</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  summary              Email                                      Address  \\\n",
       "0   count                500                                          500   \n",
       "1    mean               None                                         None   \n",
       "2  stddev               None                                         None   \n",
       "3     min  aaron04@yahoo.com  0001 Mack MillNorth Jennifer, NE 42021-5936   \n",
       "4     25%               None                                         None   \n",
       "5     50%               None                                         None   \n",
       "6     75%               None                                         None   \n",
       "7     max  zscott@wright.com               Unit 7502 Box 8345DPO AE 53747   \n",
       "\n",
       "   Avg Session Length         Time on App     Time on Website  \\\n",
       "0                 500                 500                 500   \n",
       "1  33.053193518240015  12.052487936927994      37.06044542108   \n",
       "2  0.9925631111602917  0.9942156084624618  1.0104889068105996   \n",
       "3         29.53242897         8.508152176         33.91384725   \n",
       "4         32.33889932         11.38677555         36.33952101   \n",
       "5         33.07871721         11.98204499         37.06708997   \n",
       "6         33.71065306         12.75207661         37.71598618   \n",
       "7         36.13966249         15.12699429         40.00518164   \n",
       "\n",
       "  Length of Membership Yearly Amount Spent  \n",
       "0                  500                 500  \n",
       "1    3.533461555930001   499.3140382607999  \n",
       "2   0.9992775024367541   79.31478155115916  \n",
       "3           0.26990109         256.6705823  \n",
       "4          2.926940235         444.9665517  \n",
       "5           3.53286158         498.6355985  \n",
       "6          4.125584363         549.1315733  \n",
       "7          6.922689335         765.5184619  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.summary().toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c873c020",
   "metadata": {},
   "source": [
    "## Linear Regression\n",
    "From the above data, we have seen that even after ordering data on the basis of various attributes, there is no direct inference about how Yearly amount spent by user varies with these attributes. So we can use the Linear Regression predictor of Pyspark to predict the Yearly amount spent by using the following attributes as independent features:\n",
    "* Average Session Length\n",
    "* Time spent on the App\n",
    "* Time spent on the Website\n",
    "* Length of Membership"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "022ab9e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Email</th>\n",
       "      <th>Address</th>\n",
       "      <th>Avg Session Length</th>\n",
       "      <th>Time on App</th>\n",
       "      <th>Time on Website</th>\n",
       "      <th>Length of Membership</th>\n",
       "      <th>Yearly Amount Spent</th>\n",
       "      <th>Independent Features</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ryan36@gmail.com</td>\n",
       "      <td>5379 Rhonda Prairie Apt. 696Brownland, WA 87027</td>\n",
       "      <td>32.227299</td>\n",
       "      <td>13.728627</td>\n",
       "      <td>37.997028</td>\n",
       "      <td>4.802631</td>\n",
       "      <td>613.599323</td>\n",
       "      <td>[32.22729914, 13.72862718, 37.99702801, 4.8026...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chris67@ryan.biz</td>\n",
       "      <td>716 Bush Greens Apt. 098Trevorton, MA 17817-8000</td>\n",
       "      <td>32.789773</td>\n",
       "      <td>11.670066</td>\n",
       "      <td>37.408748</td>\n",
       "      <td>3.414688</td>\n",
       "      <td>483.159721</td>\n",
       "      <td>[32.78977262, 11.67006592, 37.40874848, 3.4146...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>katie25@gmail.com</td>\n",
       "      <td>6861 Lopez Fork Apt. 114South Jamiechester, DE...</td>\n",
       "      <td>32.772610</td>\n",
       "      <td>13.276313</td>\n",
       "      <td>36.600777</td>\n",
       "      <td>3.462299</td>\n",
       "      <td>540.263400</td>\n",
       "      <td>[32.77260993, 13.27631301, 36.60077705, 3.4622...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>kyang@diaz.org</td>\n",
       "      <td>223 Love Trail Suite 831Port Jeffrey, IN 46849</td>\n",
       "      <td>34.374258</td>\n",
       "      <td>15.126994</td>\n",
       "      <td>37.157624</td>\n",
       "      <td>5.377594</td>\n",
       "      <td>765.518462</td>\n",
       "      <td>[34.37425805, 15.12699429, 37.15762409, 5.3775...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>tadams@contreras.info</td>\n",
       "      <td>049 Matthew TerraceLake Matthew, MS 20210</td>\n",
       "      <td>33.078717</td>\n",
       "      <td>12.695790</td>\n",
       "      <td>35.358444</td>\n",
       "      <td>4.001786</td>\n",
       "      <td>553.601535</td>\n",
       "      <td>[33.07871721, 12.69578975, 35.35844431, 4.0017...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>495</th>\n",
       "      <td>lewisjessica@craig-evans.com</td>\n",
       "      <td>4483 Jones Motorway Suite 872Lake Jamiefurt, U...</td>\n",
       "      <td>33.237660</td>\n",
       "      <td>13.566160</td>\n",
       "      <td>36.417985</td>\n",
       "      <td>3.746573</td>\n",
       "      <td>573.847438</td>\n",
       "      <td>[33.23765998, 13.56615961, 36.4179848, 3.74657...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>496</th>\n",
       "      <td>katrina56@gmail.com</td>\n",
       "      <td>172 Owen Divide Suite 497West Richard, CA 19320</td>\n",
       "      <td>34.702529</td>\n",
       "      <td>11.695736</td>\n",
       "      <td>37.190268</td>\n",
       "      <td>3.576526</td>\n",
       "      <td>529.049004</td>\n",
       "      <td>[34.70252897, 11.69573629, 37.19026771, 3.5765...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>497</th>\n",
       "      <td>dale88@hotmail.com</td>\n",
       "      <td>0787 Andrews Ranch Apt. 633South Chadburgh, TN...</td>\n",
       "      <td>32.646777</td>\n",
       "      <td>11.499409</td>\n",
       "      <td>38.332576</td>\n",
       "      <td>4.958264</td>\n",
       "      <td>551.620146</td>\n",
       "      <td>[32.64677668, 11.49940906, 38.33257633, 4.9582...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>498</th>\n",
       "      <td>cwilson@hotmail.com</td>\n",
       "      <td>680 Jennifer Lodge Apt. 808Brendachester, TX 0...</td>\n",
       "      <td>33.322501</td>\n",
       "      <td>12.391423</td>\n",
       "      <td>36.840086</td>\n",
       "      <td>2.336485</td>\n",
       "      <td>456.469510</td>\n",
       "      <td>[33.32250105, 12.39142299, 36.84008573, 2.3364...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>hannahwilson@davidson.com</td>\n",
       "      <td>49791 Rachel Heights Apt. 898East Drewborough,...</td>\n",
       "      <td>33.715981</td>\n",
       "      <td>12.418808</td>\n",
       "      <td>35.771016</td>\n",
       "      <td>2.735160</td>\n",
       "      <td>497.778642</td>\n",
       "      <td>[33.71598092, 12.41880832, 35.77101619, 2.7351...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>500 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                            Email  \\\n",
       "0                ryan36@gmail.com   \n",
       "1                chris67@ryan.biz   \n",
       "2               katie25@gmail.com   \n",
       "3                  kyang@diaz.org   \n",
       "4           tadams@contreras.info   \n",
       "..                            ...   \n",
       "495  lewisjessica@craig-evans.com   \n",
       "496           katrina56@gmail.com   \n",
       "497            dale88@hotmail.com   \n",
       "498           cwilson@hotmail.com   \n",
       "499     hannahwilson@davidson.com   \n",
       "\n",
       "                                               Address  Avg Session Length  \\\n",
       "0      5379 Rhonda Prairie Apt. 696Brownland, WA 87027           32.227299   \n",
       "1     716 Bush Greens Apt. 098Trevorton, MA 17817-8000           32.789773   \n",
       "2    6861 Lopez Fork Apt. 114South Jamiechester, DE...           32.772610   \n",
       "3       223 Love Trail Suite 831Port Jeffrey, IN 46849           34.374258   \n",
       "4            049 Matthew TerraceLake Matthew, MS 20210           33.078717   \n",
       "..                                                 ...                 ...   \n",
       "495  4483 Jones Motorway Suite 872Lake Jamiefurt, U...           33.237660   \n",
       "496    172 Owen Divide Suite 497West Richard, CA 19320           34.702529   \n",
       "497  0787 Andrews Ranch Apt. 633South Chadburgh, TN...           32.646777   \n",
       "498  680 Jennifer Lodge Apt. 808Brendachester, TX 0...           33.322501   \n",
       "499  49791 Rachel Heights Apt. 898East Drewborough,...           33.715981   \n",
       "\n",
       "     Time on App  Time on Website  Length of Membership  Yearly Amount Spent  \\\n",
       "0      13.728627        37.997028              4.802631           613.599323   \n",
       "1      11.670066        37.408748              3.414688           483.159721   \n",
       "2      13.276313        36.600777              3.462299           540.263400   \n",
       "3      15.126994        37.157624              5.377594           765.518462   \n",
       "4      12.695790        35.358444              4.001786           553.601535   \n",
       "..           ...              ...                   ...                  ...   \n",
       "495    13.566160        36.417985              3.746573           573.847438   \n",
       "496    11.695736        37.190268              3.576526           529.049004   \n",
       "497    11.499409        38.332576              4.958264           551.620146   \n",
       "498    12.391423        36.840086              2.336485           456.469510   \n",
       "499    12.418808        35.771016              2.735160           497.778642   \n",
       "\n",
       "                                  Independent Features  \n",
       "0    [32.22729914, 13.72862718, 37.99702801, 4.8026...  \n",
       "1    [32.78977262, 11.67006592, 37.40874848, 3.4146...  \n",
       "2    [32.77260993, 13.27631301, 36.60077705, 3.4622...  \n",
       "3    [34.37425805, 15.12699429, 37.15762409, 5.3775...  \n",
       "4    [33.07871721, 12.69578975, 35.35844431, 4.0017...  \n",
       "..                                                 ...  \n",
       "495  [33.23765998, 13.56615961, 36.4179848, 3.74657...  \n",
       "496  [34.70252897, 11.69573629, 37.19026771, 3.5765...  \n",
       "497  [32.64677668, 11.49940906, 38.33257633, 4.9582...  \n",
       "498  [33.32250105, 12.39142299, 36.84008573, 2.3364...  \n",
       "499  [33.71598092, 12.41880832, 35.77101619, 2.7351...  \n",
       "\n",
       "[500 rows x 8 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Linear Regression on the data to predict the yearly amount spent\n",
    "\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "\n",
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "\n",
    "featureassembler=VectorAssembler(inputCols=[\"Avg Session Length\",\"Time on App\",\"Time on Website\",\"Length of Membership\"],outputCol=\"Independent Features\")\n",
    "\n",
    "output=featureassembler.transform(dataset)\n",
    "\n",
    "output.toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "18007b6b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|Independent Features|\n",
      "+--------------------+\n",
      "|[32.22729914,13.7...|\n",
      "|[32.78977262,11.6...|\n",
      "|[32.77260993,13.2...|\n",
      "|[34.37425805,15.1...|\n",
      "|[33.07871721,12.6...|\n",
      "|[32.8052204,11.83...|\n",
      "|[32.43075793,11.3...|\n",
      "|[32.17909997,11.1...|\n",
      "|[33.15417579,11.8...|\n",
      "|[34.33589584,12.2...|\n",
      "|[32.38625186,10.6...|\n",
      "|[32.80869759,12.8...|\n",
      "|[33.87974497,13.5...|\n",
      "|[32.04983939,12.2...|\n",
      "|[33.55520742,11.5...|\n",
      "|[33.14207933,11.4...|\n",
      "|[32.59718266,10.8...|\n",
      "|[33.16713688,11.9...|\n",
      "|[31.51473786,12.5...|\n",
      "|[34.59402115,10.9...|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.select(\"Independent Features\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6a86e379",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Email',\n",
       " 'Address',\n",
       " 'Avg Session Length',\n",
       " 'Time on App',\n",
       " 'Time on Website',\n",
       " 'Length of Membership',\n",
       " 'Yearly Amount Spent',\n",
       " 'Independent Features']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9cdedf00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Independent Features</th>\n",
       "      <th>Yearly Amount Spent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[32.22729914, 13.72862718, 37.99702801, 4.8026...</td>\n",
       "      <td>613.599323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[32.78977262, 11.67006592, 37.40874848, 3.4146...</td>\n",
       "      <td>483.159721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[32.77260993, 13.27631301, 36.60077705, 3.4622...</td>\n",
       "      <td>540.263400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[34.37425805, 15.12699429, 37.15762409, 5.3775...</td>\n",
       "      <td>765.518462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[33.07871721, 12.69578975, 35.35844431, 4.0017...</td>\n",
       "      <td>553.601535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>495</th>\n",
       "      <td>[33.23765998, 13.56615961, 36.4179848, 3.74657...</td>\n",
       "      <td>573.847438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>496</th>\n",
       "      <td>[34.70252897, 11.69573629, 37.19026771, 3.5765...</td>\n",
       "      <td>529.049004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>497</th>\n",
       "      <td>[32.64677668, 11.49940906, 38.33257633, 4.9582...</td>\n",
       "      <td>551.620146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>498</th>\n",
       "      <td>[33.32250105, 12.39142299, 36.84008573, 2.3364...</td>\n",
       "      <td>456.469510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>[33.71598092, 12.41880832, 35.77101619, 2.7351...</td>\n",
       "      <td>497.778642</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>500 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  Independent Features  Yearly Amount Spent\n",
       "0    [32.22729914, 13.72862718, 37.99702801, 4.8026...           613.599323\n",
       "1    [32.78977262, 11.67006592, 37.40874848, 3.4146...           483.159721\n",
       "2    [32.77260993, 13.27631301, 36.60077705, 3.4622...           540.263400\n",
       "3    [34.37425805, 15.12699429, 37.15762409, 5.3775...           765.518462\n",
       "4    [33.07871721, 12.69578975, 35.35844431, 4.0017...           553.601535\n",
       "..                                                 ...                  ...\n",
       "495  [33.23765998, 13.56615961, 36.4179848, 3.74657...           573.847438\n",
       "496  [34.70252897, 11.69573629, 37.19026771, 3.5765...           529.049004\n",
       "497  [32.64677668, 11.49940906, 38.33257633, 4.9582...           551.620146\n",
       "498  [33.32250105, 12.39142299, 36.84008573, 2.3364...           456.469510\n",
       "499  [33.71598092, 12.41880832, 35.77101619, 2.7351...           497.778642\n",
       "\n",
       "[500 rows x 2 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "finalized_data=output.select(\"Independent Features\",\"Yearly Amount Spent\")\n",
    "finalized_data.toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ebbb70fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data,test_data=finalized_data.randomSplit([0.75,0.25])\n",
    "\n",
    "regressor=LinearRegression(featuresCol='Independent Features', labelCol='Yearly Amount Spent')\n",
    "regressor=regressor.fit(train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c3fd988",
   "metadata": {},
   "source": [
    "### Coefficients and Intercept\n",
    "Linear Regression predicts the Yearly Amount spent (say Y) as a function of independent features (say X1, X2, X3, X4). So linear regression predicts Y as:\n",
    "\n",
    "Y = W0 + W1 x X1 + W2 x X2 + W3 x X3 + W4 x X4\n",
    "\n",
    "where, W1, W2, W3, W4 are coefficients or weights for each independent feature and W0 = intercept. <br/>\n",
    "We can show these values as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4d11620f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DenseVector([25.6933, 38.582, 0.2568, 61.3066])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regressor.coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f46afe32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1040.8083731912807"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regressor.intercept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "73b4f48a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\SHUBHAM\\anaconda3\\lib\\site-packages\\pyspark\\sql\\context.py:125: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------------------+------------------+\n",
      "|Independent Features|Yearly Amount Spent|        prediction|\n",
      "+--------------------+-------------------+------------------+\n",
      "|[30.83643267,13.1...|        467.5019004| 472.2180354539273|\n",
      "|[31.04722214,11.1...|        392.4973992| 388.2973841854264|\n",
      "|[31.12397435,12.3...|        486.9470538|  508.826799247724|\n",
      "|[31.12809005,13.2...|        557.2526867| 564.5117028397945|\n",
      "|[31.42522688,13.2...|        530.7667187| 534.7921525532133|\n",
      "|[31.44597248,12.8...|        484.8769649| 482.1866211449312|\n",
      "|[31.65480968,13.0...|        475.2634237|468.87996503054774|\n",
      "|[31.73663569,10.7...|        496.9334463|494.93608937420777|\n",
      "|[31.76561882,12.4...|        496.5540816|501.25400841604596|\n",
      "|[31.80930032,11.6...|        536.7718994|  548.136300180279|\n",
      "|[31.81248256,10.8...|         392.810345|397.36457109201115|\n",
      "|[31.81861657,11.2...|        446.4186734|449.27719256392515|\n",
      "|[31.82934646,11.2...|         385.152338|384.85552230709163|\n",
      "|[31.86274111,14.0...|        556.2981412| 558.2199932060018|\n",
      "|[31.90962683,11.3...|        563.4460357| 551.9863593532875|\n",
      "|[31.94801742,13.0...|        461.9208769| 456.9324568805712|\n",
      "|[31.95490386,10.9...|        439.9978799| 432.1756531389883|\n",
      "|[32.04780098,12.7...|        513.4505712| 508.3159556923656|\n",
      "|[32.11640012,12.3...|        463.4849954| 461.0075452009103|\n",
      "|[32.1223648,11.43...|        528.9336186| 532.5385170644304|\n",
      "|[32.20079864,12.2...|        478.8853913|473.34420402376895|\n",
      "|[32.21552742,12.2...|         438.417742| 446.1880769850327|\n",
      "|[32.23014912,11.0...|        517.1651356| 514.3074444482065|\n",
      "|[32.2559012,10.48...|        479.7319376| 478.6431875897463|\n",
      "|[32.27844335,12.5...|         488.270298| 497.7841588865433|\n",
      "|[32.29464155,12.4...|        586.1558702| 590.3446099127702|\n",
      "|[32.37798966,11.9...|        408.2169018| 436.0131861997502|\n",
      "|[32.38103459,12.4...|        532.7248055| 546.2257296355576|\n",
      "|[32.38473266,10.8...|        465.8893127| 464.5551313621022|\n",
      "|[32.39742194,12.0...|        483.7965221|481.38332934990353|\n",
      "|[32.40173183,12.0...|        506.5473071|505.42590730788766|\n",
      "|[32.4071483,13.80...|        662.9610878|  643.201039873345|\n",
      "|[32.43083858,13.8...|        557.6082621| 569.4081216396382|\n",
      "|[32.43977025,12.4...|        573.3062223| 583.6721972958806|\n",
      "|[32.44952156,13.4...|        503.9783791| 502.0416951888494|\n",
      "|[32.46121245,13.2...|        543.3401663| 553.2633195045055|\n",
      "|[32.49541877,12.9...|        368.6547849| 377.8800712788411|\n",
      "|[32.53082944,12.3...|        420.9161595|422.68065331367757|\n",
      "|[32.55949313,11.7...|        468.6684656| 456.5471660119688|\n",
      "|[32.58249357,11.7...|        376.4968407|392.52057256195826|\n",
      "+--------------------+-------------------+------------------+\n",
      "only showing top 40 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pred_results=regressor.evaluate(test_data)\n",
    "pred_results.predictions.show(40)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc14c107",
   "metadata": {},
   "source": [
    "## Performance Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "21bbf7c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root Mean Squared Error on training data: 9.750906\n",
      "R Squared (R2) on training data: 0.986095\n"
     ]
    }
   ],
   "source": [
    "trainingSummary = regressor.summary\n",
    "print(\"Root Mean Squared Error on training data: %f\" % trainingSummary.rootMeanSquaredError)\n",
    "print(\"R Squared (R2) on training data: %f\" % trainingSummary.r2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e9dbcb",
   "metadata": {},
   "source": [
    "Root Mean Sqaured Error (RMSE) measures the differences between predicted values by the model and the actual values. However, RMSE alone is meaningless until we compare with the actual “Yearly Amount Spent” value, such as mean, min and max. After such comparison, our RMSE looks pretty good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "1317d4a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                377|\n",
      "|   mean| 500.22762147745334|\n",
      "| stddev|   82.7999884747694|\n",
      "|    min|        256.6705823|\n",
      "|    max|        765.5184619|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b54c0da7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------+-------------------+--------------------+\n",
      "|       prediction|Yearly Amount Spent|Independent features|\n",
      "+-----------------+-------------------+--------------------+\n",
      "|472.2180354539273|        467.5019004|[30.83643267,13.1...|\n",
      "|388.2973841854264|        392.4973992|[31.04722214,11.1...|\n",
      "| 508.826799247724|        486.9470538|[31.12397435,12.3...|\n",
      "|564.5117028397945|        557.2526867|[31.12809005,13.2...|\n",
      "|534.7921525532133|        530.7667187|[31.42522688,13.2...|\n",
      "+-----------------+-------------------+--------------------+\n",
      "only showing top 5 rows\n",
      "\n",
      "R Squared (R2) on test data = 0.975913\n"
     ]
    }
   ],
   "source": [
    "lr_predictions = regressor.transform(test_data)\n",
    "lr_predictions.select(\"prediction\",\"Yearly Amount Spent\",\"Independent features\").show(5)\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "lr_evaluator = RegressionEvaluator(predictionCol=\"prediction\", labelCol=\"Yearly Amount Spent\",metricName=\"r2\")\n",
    "r2_test = lr_evaluator.evaluate(lr_predictions)\n",
    "print(\"R Squared (R2) on test data = %g\" % r2_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a05a5a3",
   "metadata": {},
   "source": [
    "R squared at **0.975913** indicates that in our model, approximate **97%** of the variability in “Yearly Amount Spent” can be explained using the regressor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e50527f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}