session-2-preprocessing-data/preprocessing_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing and Walkthrough\n",
    "\n",
    "Aniruddh K B and Rachit Yagnik\n",
    "\n",
    "## Acknowledgements: \n",
    "\n",
    "This content borrows heavily from last year's material prepared by the TAs. The TAs at that time were:\n",
    "\n",
    "Vibhav Agarwal (IMT2016),\n",
    "Tejas Kotha (IMT2016),\n",
    "Tanmay Jain (IMT2016),\n",
    "Shreyas Gupta (IMT2016),\n",
    "Saurabh Jain (IMT2016),\n",
    "Divyanshu Khandelwal (IMT2016),\n",
    "Arjun Verma (IMT2017),\n",
    "Amitesh Anand (MT2019),\n",
    "Mohd Zahid Faiz (MT2019),\n",
    "Tushar Anil Masane (MT2019)\n",
    "\n",
    "\n",
    "# 1. Preprocessing\n",
    "\n",
    "So far, we have been considering data to be in the form of neatly packaged matrices of floating-point numbers. As you might imagine, real life is never quite so simple. \n",
    "\n",
    "There are numerous issues which might crop up with data, but we will be addressing only a few of them here. The aim is to give you a little bit of information that will be helpful for the first assignment.\n",
    "\n",
    "**Important note: When working with tabular data, ALWAYS, ALWAYS merge your training and test data, perform the preprocessing steps on this combined data, and then split the data. NEVER preprocess them separately!**\n",
    "\n",
    "## 1.1. Imports and introduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "main_df = pd.read_csv(\"datasets/preprocessing/to_work_on.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>706</td>\n",
       "      <td>150</td>\n",
       "      <td>856</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2008</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>978</td>\n",
       "      <td>284</td>\n",
       "      <td>1262</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>6</td>\n",
       "      <td>TA</td>\n",
       "      <td>0</td>\n",
       "      <td>2007</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>486</td>\n",
       "      <td>434</td>\n",
       "      <td>920</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>6</td>\n",
       "      <td>TA</td>\n",
       "      <td>0</td>\n",
       "      <td>2008</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>216</td>\n",
       "      <td>540</td>\n",
       "      <td>756</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>7</td>\n",
       "      <td>Gd</td>\n",
       "      <td>0</td>\n",
       "      <td>2006</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>655</td>\n",
       "      <td>490</td>\n",
       "      <td>1145</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>9</td>\n",
       "      <td>TA</td>\n",
       "      <td>0</td>\n",
       "      <td>2008</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Alley LotShape  BsmtFinSF1  \\\n",
       "0   1          60       RL         65.0     8450   NaN      Reg         706   \n",
       "1   2          20       RL         80.0     9600   NaN      Reg         978   \n",
       "2   3          60       RL         68.0    11250   NaN      IR1         486   \n",
       "3   4          70       RL         60.0     9550   NaN      IR1         216   \n",
       "4   5          60       RL         84.0    14260   NaN      IR1         655   \n",
       "\n",
       "   BsmtUnfSF  TotalBsmtSF Heating CentralAir Electrical  TotRmsAbvGrd  \\\n",
       "0        150          856    GasA          Y      SBrkr             8   \n",
       "1        284         1262    GasA          Y      SBrkr             6   \n",
       "2        434          920    GasA          Y      SBrkr             6   \n",
       "3        540          756    GasA          Y      SBrkr             7   \n",
       "4        490         1145    GasA          Y      SBrkr             9   \n",
       "\n",
       "  FireplaceQu  ScreenPorch  YrSold  SalePrice  \n",
       "0         NaN            0    2008     208500  \n",
       "1          TA            0    2007     181500  \n",
       "2          TA            0    2008     223500  \n",
       "3          Gd            0    2006     140000  \n",
       "4          TA            0    2008     250000  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1201.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "      <td>1460.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>730.500000</td>\n",
       "      <td>56.897260</td>\n",
       "      <td>70.049958</td>\n",
       "      <td>10516.828082</td>\n",
       "      <td>443.639726</td>\n",
       "      <td>567.240411</td>\n",
       "      <td>1057.429452</td>\n",
       "      <td>6.517808</td>\n",
       "      <td>15.060959</td>\n",
       "      <td>2007.815753</td>\n",
       "      <td>180921.195890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>421.610009</td>\n",
       "      <td>42.300571</td>\n",
       "      <td>24.284752</td>\n",
       "      <td>9981.264932</td>\n",
       "      <td>456.098091</td>\n",
       "      <td>441.866955</td>\n",
       "      <td>438.705324</td>\n",
       "      <td>1.625393</td>\n",
       "      <td>55.757415</td>\n",
       "      <td>1.328095</td>\n",
       "      <td>79442.502883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>1300.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2006.000000</td>\n",
       "      <td>34900.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>365.750000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>7553.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>223.000000</td>\n",
       "      <td>795.750000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2007.000000</td>\n",
       "      <td>129975.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>730.500000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>69.000000</td>\n",
       "      <td>9478.500000</td>\n",
       "      <td>383.500000</td>\n",
       "      <td>477.500000</td>\n",
       "      <td>991.500000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2008.000000</td>\n",
       "      <td>163000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1095.250000</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>11601.500000</td>\n",
       "      <td>712.250000</td>\n",
       "      <td>808.000000</td>\n",
       "      <td>1298.250000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2009.000000</td>\n",
       "      <td>214000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1460.000000</td>\n",
       "      <td>190.000000</td>\n",
       "      <td>313.000000</td>\n",
       "      <td>215245.000000</td>\n",
       "      <td>5644.000000</td>\n",
       "      <td>2336.000000</td>\n",
       "      <td>6110.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>480.000000</td>\n",
       "      <td>2010.000000</td>\n",
       "      <td>755000.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Id   MSSubClass  LotFrontage        LotArea   BsmtFinSF1  \\\n",
       "count  1460.000000  1460.000000  1201.000000    1460.000000  1460.000000   \n",
       "mean    730.500000    56.897260    70.049958   10516.828082   443.639726   \n",
       "std     421.610009    42.300571    24.284752    9981.264932   456.098091   \n",
       "min       1.000000    20.000000    21.000000    1300.000000     0.000000   \n",
       "25%     365.750000    20.000000    59.000000    7553.500000     0.000000   \n",
       "50%     730.500000    50.000000    69.000000    9478.500000   383.500000   \n",
       "75%    1095.250000    70.000000    80.000000   11601.500000   712.250000   \n",
       "max    1460.000000   190.000000   313.000000  215245.000000  5644.000000   \n",
       "\n",
       "         BsmtUnfSF  TotalBsmtSF  TotRmsAbvGrd  ScreenPorch       YrSold  \\\n",
       "count  1460.000000  1460.000000   1460.000000  1460.000000  1460.000000   \n",
       "mean    567.240411  1057.429452      6.517808    15.060959  2007.815753   \n",
       "std     441.866955   438.705324      1.625393    55.757415     1.328095   \n",
       "min       0.000000     0.000000      2.000000     0.000000  2006.000000   \n",
       "25%     223.000000   795.750000      5.000000     0.000000  2007.000000   \n",
       "50%     477.500000   991.500000      6.000000     0.000000  2008.000000   \n",
       "75%     808.000000  1298.250000      7.000000     0.000000  2009.000000   \n",
       "max    2336.000000  6110.000000     14.000000   480.000000  2010.000000   \n",
       "\n",
       "           SalePrice  \n",
       "count    1460.000000  \n",
       "mean   180921.195890  \n",
       "std     79442.502883  \n",
       "min     34900.000000  \n",
       "25%    129975.000000  \n",
       "50%    163000.000000  \n",
       "75%    214000.000000  \n",
       "max    755000.000000  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two important things should jump out at you: \n",
    "\n",
    "1. We have missing values. \n",
    "2. The data isn't only numeric.\n",
    "\n",
    "\n",
    "## 1.2. Dealing with missing values \n",
    "\n",
    "1. Drop the rows with null values.\n",
    "\n",
    "This can be done if you have plenty of good data available and the loss of a few rows doesn't impact the training much.\n",
    "\n",
    "2. Drop the columns with null values.\n",
    "\n",
    "This should be done if there is a lot of data missing from a particular column and no suitable default value can be inferred for the missing values.\n",
    "\n",
    "3. Replace the null values with the mean, median, mode or a default value. \n",
    "\n",
    "This should be done if there is not much data missing in a column, the column is important for training, OR if a sensible default value exists.\n",
    "\n",
    "4. Use sophisticated methods to infer the data (clustering or supervised learning).\n",
    "\n",
    "We won't be covering this here, but the main idea is to think of the data with missing values as a target column (y) and to use the remaining features (X) to somehow predict the missing values. This method is accurate, but difficult. It also needs at least _some_ useful data to be present for the missing column, and for the missing column to somehow be related to the other columns.\n",
    "\n",
    "**Much of machine learning is more art than science** -- including this step here. If you are doubtful of the effect of an individual choice, and computation/time is not a concern, you can always try both approaches and see what happens.\n",
    "\n",
    "Now, we will demonstrate the first three approaches.\n",
    "\n",
    "Note: In some datasets, a missing value may be denoted with \"?\". Check those out too, using df[column].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                 0\n",
       "MSSubClass         0\n",
       "MSZoning           0\n",
       "LotFrontage      259\n",
       "LotArea            0\n",
       "Alley           1369\n",
       "LotShape           0\n",
       "BsmtFinSF1         0\n",
       "BsmtUnfSF          0\n",
       "TotalBsmtSF        0\n",
       "Heating            0\n",
       "CentralAir         0\n",
       "Electrical         1\n",
       "TotRmsAbvGrd       0\n",
       "FireplaceQu      690\n",
       "ScreenPorch        0\n",
       "YrSold             0\n",
       "SalePrice          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.1 Deleting rows\n",
    "\n",
    "\"Electrical\" has just one missing value. Since we have enough data, removing 1 row probably won't hurt the process. Let's do that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1379</th>\n",
       "      <td>1380</td>\n",
       "      <td>80</td>\n",
       "      <td>RL</td>\n",
       "      <td>73.0</td>\n",
       "      <td>9735</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>0</td>\n",
       "      <td>384</td>\n",
       "      <td>384</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Y</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2008</td>\n",
       "      <td>167500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Id  MSSubClass MSZoning  LotFrontage  LotArea Alley LotShape  \\\n",
       "1379  1380          80       RL         73.0     9735   NaN      Reg   \n",
       "\n",
       "      BsmtFinSF1  BsmtUnfSF  TotalBsmtSF Heating CentralAir Electrical  \\\n",
       "1379           0        384          384    GasA          Y        NaN   \n",
       "\n",
       "      TotRmsAbvGrd FireplaceQu  ScreenPorch  YrSold  SalePrice  \n",
       "1379             7         NaN            0    2008     167500  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.loc[main_df[\"Electrical\"].isna(), :] #Displays the row which is null in \"Electrical\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([1379], dtype='int64')"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.index[main_df[\"Electrical\"].isna()] \n",
    "# Returns all indices with \"na\" in \"Electrical\" column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df.drop(axis=\"rows\", labels=main_df.index[main_df[\"Electrical\"].isna()], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                 0\n",
       "MSSubClass         0\n",
       "MSZoning           0\n",
       "LotFrontage      259\n",
       "LotArea            0\n",
       "Alley           1368\n",
       "LotShape           0\n",
       "BsmtFinSF1         0\n",
       "BsmtUnfSF          0\n",
       "TotalBsmtSF        0\n",
       "Heating            0\n",
       "CentralAir         0\n",
       "Electrical         0\n",
       "TotRmsAbvGrd       0\n",
       "FireplaceQu      689\n",
       "ScreenPorch        0\n",
       "YrSold             0\n",
       "SalePrice          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The row with one missing value in Electrical is gone.\n",
    "\n",
    "### 1.2.2. Dropping problematic columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Grvl    50\n",
       "Pave    41\n",
       "Name: Alley, dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"Alley\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we have two choices: We replace with a default value (say, \"noAlley\"), or we drop the column entirely. The choice is yours. For demonstration's sake, I will drop this column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df.drop(axis=\"columns\", labels=\"Alley\", inplace=True) \n",
    "#Think about why we're using inplace=True everywhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                0\n",
       "MSSubClass        0\n",
       "MSZoning          0\n",
       "LotFrontage     259\n",
       "LotArea           0\n",
       "LotShape          0\n",
       "BsmtFinSF1        0\n",
       "BsmtUnfSF         0\n",
       "TotalBsmtSF       0\n",
       "Heating           0\n",
       "CentralAir        0\n",
       "Electrical        0\n",
       "TotRmsAbvGrd      0\n",
       "FireplaceQu     689\n",
       "ScreenPorch       0\n",
       "YrSold            0\n",
       "SalePrice         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.3. Replacing with mean/median/mode/default value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([nan, 'TA', 'Gd', 'Fa', 'Ex', 'Po'], dtype=object)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"FireplaceQu\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gd    380\n",
       "TA    313\n",
       "Fa     33\n",
       "Ex     24\n",
       "Po     20\n",
       "Name: FireplaceQu, dtype: int64"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"FireplaceQu\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's where domain knowledge comes in. Maybe a NA means there is no fireplace? We could fill with a new category \"noFireplace\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df[\"FireplaceQu\"].fillna(\"noFireplace\", inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "noFireplace    689\n",
       "Gd             380\n",
       "TA             313\n",
       "Fa              33\n",
       "Ex              24\n",
       "Po              20\n",
       "Name: FireplaceQu, dtype: int64"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"FireplaceQu\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                0\n",
       "MSSubClass        0\n",
       "MSZoning          0\n",
       "LotFrontage     259\n",
       "LotArea           0\n",
       "LotShape          0\n",
       "BsmtFinSF1        0\n",
       "BsmtUnfSF         0\n",
       "TotalBsmtSF       0\n",
       "Heating           0\n",
       "CentralAir        0\n",
       "Electrical        0\n",
       "TotRmsAbvGrd      0\n",
       "FireplaceQu       0\n",
       "ScreenPorch       0\n",
       "YrSold            0\n",
       "SalePrice         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60.0     143\n",
       "70.0      70\n",
       "80.0      69\n",
       "50.0      57\n",
       "75.0      53\n",
       "        ... \n",
       "137.0      1\n",
       "141.0      1\n",
       "38.0       1\n",
       "140.0      1\n",
       "46.0       1\n",
       "Name: LotFrontage, Length: 110, dtype: int64"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"LotFrontage\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1200.000000\n",
       "mean       70.047500\n",
       "std        24.294727\n",
       "min        21.000000\n",
       "25%        59.000000\n",
       "50%        69.000000\n",
       "75%        80.000000\n",
       "max       313.000000\n",
       "Name: LotFrontage, dtype: float64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"LotFrontage\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a continuous variable. The mean and the median (50 %ile) are pretty close, and the mode is not very far from there either. We could use any of the three to fill this column. \n",
    "\n",
    "If there are a small or moderate number of NANs and the data is categorical in nature, the mode is a nice value to fill."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df[\"LotFrontage\"].fillna(main_df[\"LotFrontage\"].median(), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id              0\n",
       "MSSubClass      0\n",
       "MSZoning        0\n",
       "LotFrontage     0\n",
       "LotArea         0\n",
       "LotShape        0\n",
       "BsmtFinSF1      0\n",
       "BsmtUnfSF       0\n",
       "TotalBsmtSF     0\n",
       "Heating         0\n",
       "CentralAir      0\n",
       "Electrical      0\n",
       "TotRmsAbvGrd    0\n",
       "FireplaceQu     0\n",
       "ScreenPorch     0\n",
       "YrSold          0\n",
       "SalePrice       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3. Dealing with categorical and non-numeric data\n",
    "\n",
    "In most cases (where the column is useful for our purposes) non-numeric data is *categorical* in nature. What this means is that the column has a small number of unique values or **categories**. It could be something like a yes/no question, or more complex than that. \n",
    "\n",
    "Suppose we have a column \"colour\" where there are three categories -- red, blue and green. \n",
    "\n",
    "One thing we could do is assign \"red\" -> 0, \"blue\" -> 1, \"green\" -> 2 and so on. This is called **label encoding**.\n",
    "\n",
    "Another option is to have a dedicated column each for red, blue and green. When the original column is \"red\", make the entry for the red column 1 and the other two columns zero. And similar for blue and green. This is called **one-hot encoding**. \n",
    "\n",
    "### 1.3.1. Label encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['RL', 'RM', 'C (all)', 'FV', 'RH'], dtype=object)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"MSZoning\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    RL\n",
       "1    RL\n",
       "2    RL\n",
       "3    RL\n",
       "4    RL\n",
       "Name: MSZoning, dtype: object"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_to_label_encode = main_df[\"MSZoning\"]\n",
    "column_to_label_encode.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3, 3, 3, ..., 3, 3, 3])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_encoder = LabelEncoder()\n",
    "label_encoded_column = label_encoder.fit_transform(column_to_label_encode)\n",
    "label_encoded_column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we did here: \n",
    "\n",
    "1. Take the column you wish to label encode. \n",
    "2. Make a label encoder.\n",
    "3. Pass the column to the label encoder. \n",
    "\n",
    "Note: It would be better to create separate label encoders for different columns if you wish to use the label encoder on test data later on as well (when reusing on test data, use the transform method, not fit_transform).\n",
    "\n",
    "Note2: You can only pass an individual column to the SKLearn Label Encoder.\n",
    "\n",
    "Now, you can replace the column in the original data. \n",
    "\n",
    "However, label encoding has many issues -- even though it is very space-efficient.\n",
    "\n",
    "1. It creates an implicit ordering within the data which is sometimes undesirable. For instance, if red -> 0, blue-> 1, green -> 3 , does that mean that red < blue < green ? No. \n",
    "2. What about the values in between the labels? For instance, what of the value 1.5 ? That value has no meaning in our encoding -- but the model doesn't know that! \n",
    "\n",
    "It is for these reasons that we usually prefer one-hot encoding*, UNLESS we have too many columns or too many clases within a particular column. \n",
    "\n",
    "\n",
    "*Footnote (can ignore for now): Tree based methods like decision trees, random forests, and gradient boosted trees work well with label encoding, and in some cases, better than with one-hot encoding.\n",
    "\n",
    "### 1.3.2. One-hot encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names_to_one_hot = [\"LotShape\", \"Heating\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df = pd.get_dummies(main_df, columns=column_names_to_one_hot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>...</th>\n",
       "      <th>LotShape_IR1</th>\n",
       "      <th>LotShape_IR2</th>\n",
       "      <th>LotShape_IR3</th>\n",
       "      <th>LotShape_Reg</th>\n",
       "      <th>Heating_Floor</th>\n",
       "      <th>Heating_GasA</th>\n",
       "      <th>Heating_GasW</th>\n",
       "      <th>Heating_Grav</th>\n",
       "      <th>Heating_OthW</th>\n",
       "      <th>Heating_Wall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>706</td>\n",
       "      <td>150</td>\n",
       "      <td>856</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>978</td>\n",
       "      <td>284</td>\n",
       "      <td>1262</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>486</td>\n",
       "      <td>434</td>\n",
       "      <td>920</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>216</td>\n",
       "      <td>540</td>\n",
       "      <td>756</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>655</td>\n",
       "      <td>490</td>\n",
       "      <td>1145</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>50</td>\n",
       "      <td>RL</td>\n",
       "      <td>85.0</td>\n",
       "      <td>14115</td>\n",
       "      <td>732</td>\n",
       "      <td>64</td>\n",
       "      <td>796</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>75.0</td>\n",
       "      <td>10084</td>\n",
       "      <td>1369</td>\n",
       "      <td>317</td>\n",
       "      <td>1686</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>10382</td>\n",
       "      <td>859</td>\n",
       "      <td>216</td>\n",
       "      <td>1107</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>50</td>\n",
       "      <td>RM</td>\n",
       "      <td>51.0</td>\n",
       "      <td>6120</td>\n",
       "      <td>0</td>\n",
       "      <td>952</td>\n",
       "      <td>952</td>\n",
       "      <td>Y</td>\n",
       "      <td>FuseF</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>190</td>\n",
       "      <td>RL</td>\n",
       "      <td>50.0</td>\n",
       "      <td>7420</td>\n",
       "      <td>851</td>\n",
       "      <td>140</td>\n",
       "      <td>991</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>70.0</td>\n",
       "      <td>11200</td>\n",
       "      <td>906</td>\n",
       "      <td>134</td>\n",
       "      <td>1040</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>85.0</td>\n",
       "      <td>11924</td>\n",
       "      <td>998</td>\n",
       "      <td>177</td>\n",
       "      <td>1175</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>12968</td>\n",
       "      <td>737</td>\n",
       "      <td>175</td>\n",
       "      <td>912</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>91.0</td>\n",
       "      <td>10652</td>\n",
       "      <td>0</td>\n",
       "      <td>1494</td>\n",
       "      <td>1494</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>10920</td>\n",
       "      <td>733</td>\n",
       "      <td>520</td>\n",
       "      <td>1253</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>45</td>\n",
       "      <td>RM</td>\n",
       "      <td>51.0</td>\n",
       "      <td>6120</td>\n",
       "      <td>0</td>\n",
       "      <td>832</td>\n",
       "      <td>832</td>\n",
       "      <td>Y</td>\n",
       "      <td>FuseA</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>11241</td>\n",
       "      <td>578</td>\n",
       "      <td>426</td>\n",
       "      <td>1004</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>90</td>\n",
       "      <td>RL</td>\n",
       "      <td>72.0</td>\n",
       "      <td>10791</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>66.0</td>\n",
       "      <td>13695</td>\n",
       "      <td>646</td>\n",
       "      <td>468</td>\n",
       "      <td>1114</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>70.0</td>\n",
       "      <td>7560</td>\n",
       "      <td>504</td>\n",
       "      <td>525</td>\n",
       "      <td>1029</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>101.0</td>\n",
       "      <td>14215</td>\n",
       "      <td>0</td>\n",
       "      <td>1158</td>\n",
       "      <td>1158</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Id  MSSubClass MSZoning  LotFrontage  LotArea  BsmtFinSF1  BsmtUnfSF  \\\n",
       "0    1          60       RL         65.0     8450         706        150   \n",
       "1    2          20       RL         80.0     9600         978        284   \n",
       "2    3          60       RL         68.0    11250         486        434   \n",
       "3    4          70       RL         60.0     9550         216        540   \n",
       "4    5          60       RL         84.0    14260         655        490   \n",
       "5    6          50       RL         85.0    14115         732         64   \n",
       "6    7          20       RL         75.0    10084        1369        317   \n",
       "7    8          60       RL         69.0    10382         859        216   \n",
       "8    9          50       RM         51.0     6120           0        952   \n",
       "9   10         190       RL         50.0     7420         851        140   \n",
       "10  11          20       RL         70.0    11200         906        134   \n",
       "11  12          60       RL         85.0    11924         998        177   \n",
       "12  13          20       RL         69.0    12968         737        175   \n",
       "13  14          20       RL         91.0    10652           0       1494   \n",
       "14  15          20       RL         69.0    10920         733        520   \n",
       "15  16          45       RM         51.0     6120           0        832   \n",
       "16  17          20       RL         69.0    11241         578        426   \n",
       "17  18          90       RL         72.0    10791           0          0   \n",
       "18  19          20       RL         66.0    13695         646        468   \n",
       "19  20          20       RL         70.0     7560         504        525   \n",
       "20  21          60       RL        101.0    14215           0       1158   \n",
       "\n",
       "    TotalBsmtSF CentralAir Electrical  ...  LotShape_IR1 LotShape_IR2  \\\n",
       "0           856          Y      SBrkr  ...             0            0   \n",
       "1          1262          Y      SBrkr  ...             0            0   \n",
       "2           920          Y      SBrkr  ...             1            0   \n",
       "3           756          Y      SBrkr  ...             1            0   \n",
       "4          1145          Y      SBrkr  ...             1            0   \n",
       "5           796          Y      SBrkr  ...             1            0   \n",
       "6          1686          Y      SBrkr  ...             0            0   \n",
       "7          1107          Y      SBrkr  ...             1            0   \n",
       "8           952          Y      FuseF  ...             0            0   \n",
       "9           991          Y      SBrkr  ...             0            0   \n",
       "10         1040          Y      SBrkr  ...             0            0   \n",
       "11         1175          Y      SBrkr  ...             1            0   \n",
       "12          912          Y      SBrkr  ...             0            1   \n",
       "13         1494          Y      SBrkr  ...             1            0   \n",
       "14         1253          Y      SBrkr  ...             1            0   \n",
       "15          832          Y      FuseA  ...             0            0   \n",
       "16         1004          Y      SBrkr  ...             1            0   \n",
       "17            0          Y      SBrkr  ...             0            0   \n",
       "18         1114          Y      SBrkr  ...             0            0   \n",
       "19         1029          Y      SBrkr  ...             0            0   \n",
       "20         1158          Y      SBrkr  ...             1            0   \n",
       "\n",
       "    LotShape_IR3  LotShape_Reg  Heating_Floor  Heating_GasA  Heating_GasW  \\\n",
       "0              0             1              0             1             0   \n",
       "1              0             1              0             1             0   \n",
       "2              0             0              0             1             0   \n",
       "3              0             0              0             1             0   \n",
       "4              0             0              0             1             0   \n",
       "5              0             0              0             1             0   \n",
       "6              0             1              0             1             0   \n",
       "7              0             0              0             1             0   \n",
       "8              0             1              0             1             0   \n",
       "9              0             1              0             1             0   \n",
       "10             0             1              0             1             0   \n",
       "11             0             0              0             1             0   \n",
       "12             0             0              0             1             0   \n",
       "13             0             0              0             1             0   \n",
       "14             0             0              0             1             0   \n",
       "15             0             1              0             1             0   \n",
       "16             0             0              0             1             0   \n",
       "17             0             1              0             1             0   \n",
       "18             0             1              0             1             0   \n",
       "19             0             1              0             1             0   \n",
       "20             0             0              0             1             0   \n",
       "\n",
       "    Heating_Grav  Heating_OthW  Heating_Wall  \n",
       "0              0             0             0  \n",
       "1              0             0             0  \n",
       "2              0             0             0  \n",
       "3              0             0             0  \n",
       "4              0             0             0  \n",
       "5              0             0             0  \n",
       "6              0             0             0  \n",
       "7              0             0             0  \n",
       "8              0             0             0  \n",
       "9              0             0             0  \n",
       "10             0             0             0  \n",
       "11             0             0             0  \n",
       "12             0             0             0  \n",
       "13             0             0             0  \n",
       "14             0             0             0  \n",
       "15             0             0             0  \n",
       "16             0             0             0  \n",
       "17             0             0             0  \n",
       "18             0             0             0  \n",
       "19             0             0             0  \n",
       "20             0             0             0  \n",
       "\n",
       "[21 rows x 25 columns]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.loc[:20,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Id', 'MSSubClass', 'MSZoning', 'LotFrontage', 'LotArea', 'BsmtFinSF1',\n",
       "       'BsmtUnfSF', 'TotalBsmtSF', 'CentralAir', 'Electrical', 'TotRmsAbvGrd',\n",
       "       'FireplaceQu', 'ScreenPorch', 'YrSold', 'SalePrice', 'LotShape_IR1',\n",
       "       'LotShape_IR2', 'LotShape_IR3', 'LotShape_Reg', 'Heating_Floor',\n",
       "       'Heating_GasA', 'Heating_GasW', 'Heating_Grav', 'Heating_OthW',\n",
       "       'Heating_Wall'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4. Duplicate rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([], dtype='int64')"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.index[main_df.duplicated()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no duplicate rows here. Though, strictly speaking, we should do this after removing any serial-number columns.\n",
    "\n",
    "\n",
    "Let's say, for the sake of the argument, that we introduce some duplicate columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df = main_df.append(main_df.iloc[:5,:]) \n",
    "# We're copying the first five rows again to the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df.drop(axis=\"rows\", labels=main_df.index[main_df.duplicated()], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.duplicated().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.5. Train-test split\n",
    "The reasons for train-test split should be discussed by the TA. TLDR: To prevent overfitting, you train using a training dataset and compare against a test dataset which the model has not seen during training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "train_df, test_df = train_test_split(main_df, train_size=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>...</th>\n",
       "      <th>LotShape_IR1</th>\n",
       "      <th>LotShape_IR2</th>\n",
       "      <th>LotShape_IR3</th>\n",
       "      <th>LotShape_Reg</th>\n",
       "      <th>Heating_Floor</th>\n",
       "      <th>Heating_GasA</th>\n",
       "      <th>Heating_GasW</th>\n",
       "      <th>Heating_Grav</th>\n",
       "      <th>Heating_OthW</th>\n",
       "      <th>Heating_Wall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>456</th>\n",
       "      <td>457</td>\n",
       "      <td>70</td>\n",
       "      <td>RM</td>\n",
       "      <td>34.0</td>\n",
       "      <td>4571</td>\n",
       "      <td>0</td>\n",
       "      <td>624</td>\n",
       "      <td>624</td>\n",
       "      <td>N</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>868</th>\n",
       "      <td>869</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>14762</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346</th>\n",
       "      <td>1347</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>20781</td>\n",
       "      <td>297</td>\n",
       "      <td>1203</td>\n",
       "      <td>1568</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>78</td>\n",
       "      <td>50</td>\n",
       "      <td>RM</td>\n",
       "      <td>50.0</td>\n",
       "      <td>8635</td>\n",
       "      <td>336</td>\n",
       "      <td>295</td>\n",
       "      <td>672</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>523</th>\n",
       "      <td>524</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>130.0</td>\n",
       "      <td>40094</td>\n",
       "      <td>2260</td>\n",
       "      <td>878</td>\n",
       "      <td>3138</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1302</th>\n",
       "      <td>1303</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>92.0</td>\n",
       "      <td>10120</td>\n",
       "      <td>740</td>\n",
       "      <td>425</td>\n",
       "      <td>1165</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342</th>\n",
       "      <td>343</td>\n",
       "      <td>90</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>8544</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>N</td>\n",
       "      <td>FuseA</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1363</th>\n",
       "      <td>1364</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>73.0</td>\n",
       "      <td>8499</td>\n",
       "      <td>0</td>\n",
       "      <td>616</td>\n",
       "      <td>616</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1429</th>\n",
       "      <td>1430</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>12546</td>\n",
       "      <td>678</td>\n",
       "      <td>762</td>\n",
       "      <td>1440</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1350</th>\n",
       "      <td>1351</td>\n",
       "      <td>90</td>\n",
       "      <td>RL</td>\n",
       "      <td>91.0</td>\n",
       "      <td>11643</td>\n",
       "      <td>500</td>\n",
       "      <td>748</td>\n",
       "      <td>1248</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1308 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        Id  MSSubClass MSZoning  LotFrontage  LotArea  BsmtFinSF1  BsmtUnfSF  \\\n",
       "456    457          70       RM         34.0     4571           0        624   \n",
       "868    869          60       RL         69.0    14762           0          0   \n",
       "1346  1347          20       RL         69.0    20781         297       1203   \n",
       "77      78          50       RM         50.0     8635         336        295   \n",
       "523    524          60       RL        130.0    40094        2260        878   \n",
       "...    ...         ...      ...          ...      ...         ...        ...   \n",
       "1302  1303          60       RL         92.0    10120         740        425   \n",
       "342    343          90       RL         69.0     8544           0          0   \n",
       "1363  1364          60       RL         73.0     8499           0        616   \n",
       "1429  1430          20       RL         69.0    12546         678        762   \n",
       "1350  1351          90       RL         91.0    11643         500        748   \n",
       "\n",
       "      TotalBsmtSF CentralAir Electrical  ...  LotShape_IR1 LotShape_IR2  \\\n",
       "456           624          N      SBrkr  ...             0            0   \n",
       "868             0          Y      SBrkr  ...             0            1   \n",
       "1346         1568          Y      SBrkr  ...             0            1   \n",
       "77            672          Y      SBrkr  ...             0            0   \n",
       "523          3138          Y      SBrkr  ...             1            0   \n",
       "...           ...        ...        ...  ...           ...          ...   \n",
       "1302         1165          Y      SBrkr  ...             0            0   \n",
       "342             0          N      FuseA  ...             0            0   \n",
       "1363          616          Y      SBrkr  ...             1            0   \n",
       "1429         1440          Y      SBrkr  ...             1            0   \n",
       "1350         1248          Y      SBrkr  ...             0            0   \n",
       "\n",
       "      LotShape_IR3  LotShape_Reg  Heating_Floor  Heating_GasA  Heating_GasW  \\\n",
       "456              0             1              0             1             0   \n",
       "868              0             0              0             1             0   \n",
       "1346             0             0              0             1             0   \n",
       "77               0             1              0             1             0   \n",
       "523              0             0              0             1             0   \n",
       "...            ...           ...            ...           ...           ...   \n",
       "1302             0             1              0             1             0   \n",
       "342              0             1              0             0             0   \n",
       "1363             0             0              0             1             0   \n",
       "1429             0             0              0             1             0   \n",
       "1350             0             1              0             1             0   \n",
       "\n",
       "      Heating_Grav  Heating_OthW  Heating_Wall  \n",
       "456              0             0             0  \n",
       "868              0             0             0  \n",
       "1346             0             0             0  \n",
       "77               0             0             0  \n",
       "523              0             0             0  \n",
       "...            ...           ...           ...  \n",
       "1302             0             0             0  \n",
       "342              0             0             1  \n",
       "1363             0             0             0  \n",
       "1429             0             0             0  \n",
       "1350             0             0             0  \n",
       "\n",
       "[1308 rows x 25 columns]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 392,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>...</th>\n",
       "      <th>LotShape_IR1</th>\n",
       "      <th>LotShape_IR2</th>\n",
       "      <th>LotShape_IR3</th>\n",
       "      <th>LotShape_Reg</th>\n",
       "      <th>Heating_Floor</th>\n",
       "      <th>Heating_GasA</th>\n",
       "      <th>Heating_GasW</th>\n",
       "      <th>Heating_Grav</th>\n",
       "      <th>Heating_OthW</th>\n",
       "      <th>Heating_Wall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>832</th>\n",
       "      <td>833</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>44.0</td>\n",
       "      <td>9548</td>\n",
       "      <td>483</td>\n",
       "      <td>458</td>\n",
       "      <td>941</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185</th>\n",
       "      <td>186</td>\n",
       "      <td>75</td>\n",
       "      <td>RM</td>\n",
       "      <td>90.0</td>\n",
       "      <td>22950</td>\n",
       "      <td>0</td>\n",
       "      <td>1107</td>\n",
       "      <td>1107</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>542</th>\n",
       "      <td>543</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>78.0</td>\n",
       "      <td>10140</td>\n",
       "      <td>144</td>\n",
       "      <td>379</td>\n",
       "      <td>1650</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>361</th>\n",
       "      <td>362</td>\n",
       "      <td>50</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>9144</td>\n",
       "      <td>399</td>\n",
       "      <td>484</td>\n",
       "      <td>883</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1062</th>\n",
       "      <td>1063</td>\n",
       "      <td>190</td>\n",
       "      <td>RM</td>\n",
       "      <td>85.0</td>\n",
       "      <td>13600</td>\n",
       "      <td>0</td>\n",
       "      <td>662</td>\n",
       "      <td>662</td>\n",
       "      <td>N</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1424</th>\n",
       "      <td>1425</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>69.0</td>\n",
       "      <td>9503</td>\n",
       "      <td>457</td>\n",
       "      <td>193</td>\n",
       "      <td>1024</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>788</th>\n",
       "      <td>789</td>\n",
       "      <td>20</td>\n",
       "      <td>RM</td>\n",
       "      <td>50.0</td>\n",
       "      <td>6000</td>\n",
       "      <td>0</td>\n",
       "      <td>901</td>\n",
       "      <td>901</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>373</th>\n",
       "      <td>374</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>79.0</td>\n",
       "      <td>10634</td>\n",
       "      <td>428</td>\n",
       "      <td>0</td>\n",
       "      <td>608</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>104</th>\n",
       "      <td>105</td>\n",
       "      <td>50</td>\n",
       "      <td>RM</td>\n",
       "      <td>69.0</td>\n",
       "      <td>7758</td>\n",
       "      <td>224</td>\n",
       "      <td>816</td>\n",
       "      <td>1040</td>\n",
       "      <td>Y</td>\n",
       "      <td>FuseF</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>299</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>90.0</td>\n",
       "      <td>11700</td>\n",
       "      <td>384</td>\n",
       "      <td>143</td>\n",
       "      <td>702</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>146 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        Id  MSSubClass MSZoning  LotFrontage  LotArea  BsmtFinSF1  BsmtUnfSF  \\\n",
       "832    833          60       RL         44.0     9548         483        458   \n",
       "185    186          75       RM         90.0    22950           0       1107   \n",
       "542    543          20       RL         78.0    10140         144        379   \n",
       "361    362          50       RL         69.0     9144         399        484   \n",
       "1062  1063         190       RM         85.0    13600           0        662   \n",
       "...    ...         ...      ...          ...      ...         ...        ...   \n",
       "1424  1425          20       RL         69.0     9503         457        193   \n",
       "788    789          20       RM         50.0     6000           0        901   \n",
       "373    374          20       RL         79.0    10634         428          0   \n",
       "104    105          50       RM         69.0     7758         224        816   \n",
       "298    299          60       RL         90.0    11700         384        143   \n",
       "\n",
       "      TotalBsmtSF CentralAir Electrical  ...  LotShape_IR1 LotShape_IR2  \\\n",
       "832           941          Y      SBrkr  ...             1            0   \n",
       "185          1107          Y      SBrkr  ...             0            1   \n",
       "542          1650          Y      SBrkr  ...             0            0   \n",
       "361           883          Y      SBrkr  ...             0            0   \n",
       "1062          662          N      SBrkr  ...             0            0   \n",
       "...           ...        ...        ...  ...           ...          ...   \n",
       "1424         1024          Y      SBrkr  ...             0            0   \n",
       "788           901          Y      SBrkr  ...             0            0   \n",
       "373           608          Y      SBrkr  ...             0            0   \n",
       "104          1040          Y      FuseF  ...             0            0   \n",
       "298           702          Y      SBrkr  ...             0            0   \n",
       "\n",
       "      LotShape_IR3  LotShape_Reg  Heating_Floor  Heating_GasA  Heating_GasW  \\\n",
       "832              0             0              0             1             0   \n",
       "185              0             0              0             1             0   \n",
       "542              0             1              0             1             0   \n",
       "361              0             1              0             1             0   \n",
       "1062             0             1              0             1             0   \n",
       "...            ...           ...            ...           ...           ...   \n",
       "1424             0             1              0             1             0   \n",
       "788              0             1              0             1             0   \n",
       "373              0             1              0             1             0   \n",
       "104              0             1              0             1             0   \n",
       "298              0             1              0             1             0   \n",
       "\n",
       "      Heating_Grav  Heating_OthW  Heating_Wall  \n",
       "832              0             0             0  \n",
       "185              0             0             0  \n",
       "542              0             0             0  \n",
       "361              0             0             0  \n",
       "1062             0             0             0  \n",
       "...            ...           ...           ...  \n",
       "1424             0             0             0  \n",
       "788              0             0             0  \n",
       "373              0             0             0  \n",
       "104              0             0             0  \n",
       "298              0             0             0  \n",
       "\n",
       "[146 rows x 25 columns]"
      ]
     },
     "execution_count": 392,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. An example linear regression problem, from beginning to end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Imports and intro \n",
    "\n",
    "Just reimporting stuff, so that it's clear what all we need: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd \n",
    "import matplotlib.pyplot as plt \n",
    "from sklearn.linear_model import LinearRegression \n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df = pd.read_csv(\"./datasets/cardekho/car_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Car_Name</th>\n",
       "      <th>Year</th>\n",
       "      <th>Selling_Price</th>\n",
       "      <th>Present_Price</th>\n",
       "      <th>Kms_Driven</th>\n",
       "      <th>Fuel_Type</th>\n",
       "      <th>Seller_Type</th>\n",
       "      <th>Transmission</th>\n",
       "      <th>Owner</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ritz</td>\n",
       "      <td>2014</td>\n",
       "      <td>3.35</td>\n",
       "      <td>5.59</td>\n",
       "      <td>27000</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>sx4</td>\n",
       "      <td>2013</td>\n",
       "      <td>4.75</td>\n",
       "      <td>9.54</td>\n",
       "      <td>43000</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ciaz</td>\n",
       "      <td>2017</td>\n",
       "      <td>7.25</td>\n",
       "      <td>9.85</td>\n",
       "      <td>6900</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>wagon r</td>\n",
       "      <td>2011</td>\n",
       "      <td>2.85</td>\n",
       "      <td>4.15</td>\n",
       "      <td>5200</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>swift</td>\n",
       "      <td>2014</td>\n",
       "      <td>4.60</td>\n",
       "      <td>6.87</td>\n",
       "      <td>42450</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Car_Name  Year  Selling_Price  Present_Price  Kms_Driven Fuel_Type  \\\n",
       "0     ritz  2014           3.35           5.59       27000    Petrol   \n",
       "1      sx4  2013           4.75           9.54       43000    Diesel   \n",
       "2     ciaz  2017           7.25           9.85        6900    Petrol   \n",
       "3  wagon r  2011           2.85           4.15        5200    Petrol   \n",
       "4    swift  2014           4.60           6.87       42450    Diesel   \n",
       "\n",
       "  Seller_Type Transmission  Owner  \n",
       "0      Dealer       Manual      0  \n",
       "1      Dealer       Manual      0  \n",
       "2      Dealer       Manual      0  \n",
       "3      Dealer       Manual      0  \n",
       "4      Dealer       Manual      0  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main objective is to predict the Selling_Price , given the rest of the data available to us. A detailed description of the data is available at https://www.kaggle.com/nehalbirla/vehicle-dataset-from-cardekho "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Checking for missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Car_Name         0\n",
       "Year             0\n",
       "Selling_Price    0\n",
       "Present_Price    0\n",
       "Kms_Driven       0\n",
       "Fuel_Type        0\n",
       "Seller_Type      0\n",
       "Transmission     0\n",
       "Owner            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No missing values. But wait..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Car_Name         0\n",
       "Year             0\n",
       "Selling_Price    0\n",
       "Present_Price    0\n",
       "Kms_Driven       0\n",
       "Fuel_Type        0\n",
       "Seller_Type      0\n",
       "Transmission     0\n",
       "Owner            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(main_df == \"?\").sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, I guess there are no missing values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. Categorical --> Numeric\n",
    "\n",
    "First, we must evaluate the columns and see if one-hot encodings are viable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "301"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(main_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Car_Name          98\n",
       "Year              16\n",
       "Selling_Price    156\n",
       "Present_Price    147\n",
       "Kms_Driven       206\n",
       "Fuel_Type          3\n",
       "Seller_Type        2\n",
       "Transmission       2\n",
       "Owner              3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Car_Name, Fuel_Type, Seller_Type and Transmission can be thought of as categorical. The problem is, Car_Name has way too many unique values to be useful. Since we are restricted to linear regression, it doesn't make sense to use this column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df.drop(axis=\"columns\", labels=\"Car_Name\", inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Selling_Price</th>\n",
       "      <th>Present_Price</th>\n",
       "      <th>Kms_Driven</th>\n",
       "      <th>Fuel_Type</th>\n",
       "      <th>Seller_Type</th>\n",
       "      <th>Transmission</th>\n",
       "      <th>Owner</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014</td>\n",
       "      <td>3.35</td>\n",
       "      <td>5.59</td>\n",
       "      <td>27000</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2013</td>\n",
       "      <td>4.75</td>\n",
       "      <td>9.54</td>\n",
       "      <td>43000</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017</td>\n",
       "      <td>7.25</td>\n",
       "      <td>9.85</td>\n",
       "      <td>6900</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011</td>\n",
       "      <td>2.85</td>\n",
       "      <td>4.15</td>\n",
       "      <td>5200</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014</td>\n",
       "      <td>4.60</td>\n",
       "      <td>6.87</td>\n",
       "      <td>42450</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>Dealer</td>\n",
       "      <td>Manual</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year  Selling_Price  Present_Price  Kms_Driven Fuel_Type Seller_Type  \\\n",
       "0  2014           3.35           5.59       27000    Petrol      Dealer   \n",
       "1  2013           4.75           9.54       43000    Diesel      Dealer   \n",
       "2  2017           7.25           9.85        6900    Petrol      Dealer   \n",
       "3  2011           2.85           4.15        5200    Petrol      Dealer   \n",
       "4  2014           4.60           6.87       42450    Diesel      Dealer   \n",
       "\n",
       "  Transmission  Owner  \n",
       "0       Manual      0  \n",
       "1       Manual      0  \n",
       "2       Manual      0  \n",
       "3       Manual      0  \n",
       "4       Manual      0  "
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names_to_one_hot = [\"Fuel_Type\", \"Seller_Type\", \"Transmission\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df = pd.get_dummies(main_df, columns=column_names_to_one_hot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Year</th>\n",
       "      <th>Selling_Price</th>\n",
       "      <th>Present_Price</th>\n",
       "      <th>Kms_Driven</th>\n",
       "      <th>Owner</th>\n",
       "      <th>Fuel_Type_CNG</th>\n",
       "      <th>Fuel_Type_Diesel</th>\n",
       "      <th>Fuel_Type_Petrol</th>\n",
       "      <th>Seller_Type_Dealer</th>\n",
       "      <th>Seller_Type_Individual</th>\n",
       "      <th>Transmission_Automatic</th>\n",
       "      <th>Transmission_Manual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014</td>\n",
       "      <td>3.35</td>\n",
       "      <td>5.59</td>\n",
       "      <td>27000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2013</td>\n",
       "      <td>4.75</td>\n",
       "      <td>9.54</td>\n",
       "      <td>43000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017</td>\n",
       "      <td>7.25</td>\n",
       "      <td>9.85</td>\n",
       "      <td>6900</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011</td>\n",
       "      <td>2.85</td>\n",
       "      <td>4.15</td>\n",
       "      <td>5200</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014</td>\n",
       "      <td>4.60</td>\n",
       "      <td>6.87</td>\n",
       "      <td>42450</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Year  Selling_Price  Present_Price  Kms_Driven  Owner  Fuel_Type_CNG  \\\n",
       "0  2014           3.35           5.59       27000      0              0   \n",
       "1  2013           4.75           9.54       43000      0              0   \n",
       "2  2017           7.25           9.85        6900      0              0   \n",
       "3  2011           2.85           4.15        5200      0              0   \n",
       "4  2014           4.60           6.87       42450      0              0   \n",
       "\n",
       "   Fuel_Type_Diesel  Fuel_Type_Petrol  Seller_Type_Dealer  \\\n",
       "0                 0                 1                   1   \n",
       "1                 1                 0                   1   \n",
       "2                 0                 1                   1   \n",
       "3                 0                 1                   1   \n",
       "4                 1                 0                   1   \n",
       "\n",
       "   Seller_Type_Individual  Transmission_Automatic  Transmission_Manual  \n",
       "0                       0                       0                    1  \n",
       "1                       0                       0                    1  \n",
       "2                       0                       0                    1  \n",
       "3                       0                       0                    1  \n",
       "4                       0                       0                    1  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Year', 'Selling_Price', 'Present_Price', 'Kms_Driven', 'Owner',\n",
       "       'Fuel_Type_CNG', 'Fuel_Type_Diesel', 'Fuel_Type_Petrol',\n",
       "       'Seller_Type_Dealer', 'Seller_Type_Individual',\n",
       "       'Transmission_Automatic', 'Transmission_Manual'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4. Duplicate rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.duplicated().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df.drop(axis=\"rows\", labels=main_df.index[main_df.duplicated()], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.duplicated().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5. Training and train-test-split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = main_df.drop(axis=\"columns\", labels=\"Selling_Price\").to_numpy().astype(np.float64)\n",
    "y = main_df[\"Selling_Price\"].to_numpy().astype(np.float64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_X, test_X, train_y, test_y = train_test_split(X, y, train_size=0.6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the above step fails, it means that there is some data that is still non-numeric. This could be because we haven't dealt with all missing data, some column that we thought is numeric is \"object\" for some reason, or that we have forgotten to encode a column.\n",
    "\n",
    "Ignore for now: This is also the point at which we would do a train-dev-test split of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model = LinearRegression()\n",
    "lr_model.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_yhat = lr_model.predict(train_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.8809341565548916"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_squared_error(train_y, train_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.1000522641196337"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_absolute_error(train_y, train_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_yhat = lr_model.predict(test_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.6479426411830636"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_squared_error(test_y, test_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2116391117252334"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_absolute_error(test_y, test_yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, on average, the model is off by a lakh or so in the predictions. Not too bad, considering that we have dropped one of the most important columns for the regression -- the model of the car itself! For instance, Toyotas are famous for having a high resale value due to reliability and quality, while some German makes are infamous for being expensive to maintain in their later years, thus commanding a lower price in the resale market. \n",
    "\n",
    "Try to fiddle with the hyperparameters of the model https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html , and see if you can extract any information from the car model column -- maybe you could just extract the company, and drop the specific model? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. An example classification problem from start to end\n",
    "\n",
    "Here, we use data taken from ECG and bloodwork of patients to predict whether they have heart disease or not. To know more about the feature columns, read the .names file in the datasets/cleveland_heart_disease folder.\n",
    "\n",
    "Data taken from the UCI ML database.\n",
    "\n",
    "## 3.1. Imports and introduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "from sklearn.linear_model import LogisticRegression \n",
    "from sklearn.metrics import accuracy_score, f1_score \n",
    "from sklearn.naive_bayes import GaussianNB, BernoulliNB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df = pd.read_csv(\"./datasets/cleveland_heart_disease/heart_disease.data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>63.0</th>\n",
       "      <th>1.0</th>\n",
       "      <th>1.0.1</th>\n",
       "      <th>145.0</th>\n",
       "      <th>233.0</th>\n",
       "      <th>1.0.2</th>\n",
       "      <th>2.0</th>\n",
       "      <th>150.0</th>\n",
       "      <th>0.0</th>\n",
       "      <th>2.3</th>\n",
       "      <th>3.0</th>\n",
       "      <th>0.0.1</th>\n",
       "      <th>6.0</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>41.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>56.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>236.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>178.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   63.0  1.0  1.0.1  145.0  233.0  1.0.2  2.0  150.0  0.0  2.3  3.0 0.0.1  \\\n",
       "0  67.0  1.0    4.0  160.0  286.0    0.0  2.0  108.0  1.0  1.5  2.0   3.0   \n",
       "1  67.0  1.0    4.0  120.0  229.0    0.0  2.0  129.0  1.0  2.6  2.0   2.0   \n",
       "2  37.0  1.0    3.0  130.0  250.0    0.0  0.0  187.0  0.0  3.5  3.0   0.0   \n",
       "3  41.0  0.0    2.0  130.0  204.0    0.0  2.0  172.0  0.0  1.4  1.0   0.0   \n",
       "4  56.0  1.0    2.0  120.0  236.0    0.0  0.0  178.0  0.0  0.8  1.0   0.0   \n",
       "\n",
       "   6.0  0  \n",
       "0  3.0  2  \n",
       "1  7.0  1  \n",
       "2  3.0  0  \n",
       "3  3.0  0  \n",
       "4  3.0  0  "
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops! Pandas thought that the first row was the header. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df = pd.read_csv(\"./datasets/cleveland_heart_disease/heart_disease.data\", header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1    2      3      4    5    6      7    8    9    10   11   12  13\n",
       "0  63.0  1.0  1.0  145.0  233.0  1.0  2.0  150.0  0.0  2.3  3.0  0.0  6.0   0\n",
       "1  67.0  1.0  4.0  160.0  286.0  0.0  2.0  108.0  1.0  1.5  2.0  3.0  3.0   2\n",
       "2  67.0  1.0  4.0  120.0  229.0  0.0  2.0  129.0  1.0  2.6  2.0  2.0  7.0   1\n",
       "3  37.0  1.0  3.0  130.0  250.0  0.0  0.0  187.0  0.0  3.5  3.0  0.0  3.0   0\n",
       "4  41.0  0.0  2.0  130.0  204.0  0.0  2.0  172.0  0.0  1.4  1.0  0.0  3.0   0"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Better, but we need to rename the columns. See the .names file in the datasets/cleveland_heart_disease folder. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "names_mapper = {\n",
    "    0:\"age\",\n",
    "    1:\"sex\",\n",
    "    2:\"cp\", \n",
    "    3:\"trestbps\", \n",
    "    4:\"chol\", \n",
    "    5:\"fbs\", \n",
    "    6:\"restecg\", \n",
    "    7:\"thalach\", \n",
    "    8:\"exang\", \n",
    "    9:\"oldpeak\", \n",
    "    10:\"slope\", \n",
    "    11:\"ca\", \n",
    "    12:\"thal\", \n",
    "    13:\"target\"\n",
    "}\n",
    "\n",
    "main_df.rename(axis=\"columns\", mapper=names_mapper, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    age  sex   cp  trestbps   chol  fbs  restecg  thalach  exang  oldpeak  \\\n",
       "0  63.0  1.0  1.0     145.0  233.0  1.0      2.0    150.0    0.0      2.3   \n",
       "1  67.0  1.0  4.0     160.0  286.0  0.0      2.0    108.0    1.0      1.5   \n",
       "2  67.0  1.0  4.0     120.0  229.0  0.0      2.0    129.0    1.0      2.6   \n",
       "3  37.0  1.0  3.0     130.0  250.0  0.0      0.0    187.0    0.0      3.5   \n",
       "4  41.0  0.0  2.0     130.0  204.0  0.0      2.0    172.0    0.0      1.4   \n",
       "\n",
       "   slope   ca thal  target  \n",
       "0    3.0  0.0  6.0       0  \n",
       "1    2.0  3.0  3.0       2  \n",
       "2    2.0  2.0  7.0       1  \n",
       "3    3.0  0.0  3.0       0  \n",
       "4    1.0  0.0  3.0       0  "
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>54.438944</td>\n",
       "      <td>0.679868</td>\n",
       "      <td>3.158416</td>\n",
       "      <td>131.689769</td>\n",
       "      <td>246.693069</td>\n",
       "      <td>0.148515</td>\n",
       "      <td>0.990099</td>\n",
       "      <td>149.607261</td>\n",
       "      <td>0.326733</td>\n",
       "      <td>1.039604</td>\n",
       "      <td>1.600660</td>\n",
       "      <td>0.937294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.038662</td>\n",
       "      <td>0.467299</td>\n",
       "      <td>0.960126</td>\n",
       "      <td>17.599748</td>\n",
       "      <td>51.776918</td>\n",
       "      <td>0.356198</td>\n",
       "      <td>0.994971</td>\n",
       "      <td>22.875003</td>\n",
       "      <td>0.469794</td>\n",
       "      <td>1.161075</td>\n",
       "      <td>0.616226</td>\n",
       "      <td>1.228536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>94.000000</td>\n",
       "      <td>126.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>48.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>211.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>133.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>56.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>241.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>153.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>61.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>275.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>166.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>77.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>564.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>202.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              age         sex          cp    trestbps        chol         fbs  \\\n",
       "count  303.000000  303.000000  303.000000  303.000000  303.000000  303.000000   \n",
       "mean    54.438944    0.679868    3.158416  131.689769  246.693069    0.148515   \n",
       "std      9.038662    0.467299    0.960126   17.599748   51.776918    0.356198   \n",
       "min     29.000000    0.000000    1.000000   94.000000  126.000000    0.000000   \n",
       "25%     48.000000    0.000000    3.000000  120.000000  211.000000    0.000000   \n",
       "50%     56.000000    1.000000    3.000000  130.000000  241.000000    0.000000   \n",
       "75%     61.000000    1.000000    4.000000  140.000000  275.000000    0.000000   \n",
       "max     77.000000    1.000000    4.000000  200.000000  564.000000    1.000000   \n",
       "\n",
       "          restecg     thalach       exang     oldpeak       slope      target  \n",
       "count  303.000000  303.000000  303.000000  303.000000  303.000000  303.000000  \n",
       "mean     0.990099  149.607261    0.326733    1.039604    1.600660    0.937294  \n",
       "std      0.994971   22.875003    0.469794    1.161075    0.616226    1.228536  \n",
       "min      0.000000   71.000000    0.000000    0.000000    1.000000    0.000000  \n",
       "25%      0.000000  133.500000    0.000000    0.000000    1.000000    0.000000  \n",
       "50%      1.000000  153.000000    0.000000    0.800000    2.000000    0.000000  \n",
       "75%      2.000000  166.000000    1.000000    1.600000    2.000000    2.000000  \n",
       "max      2.000000  202.000000    1.000000    6.200000    3.000000    4.000000  "
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. Missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age         0\n",
       "sex         0\n",
       "cp          0\n",
       "trestbps    0\n",
       "chol        0\n",
       "fbs         0\n",
       "restecg     0\n",
       "thalach     0\n",
       "exang       0\n",
       "oldpeak     0\n",
       "slope       0\n",
       "ca          0\n",
       "thal        0\n",
       "target      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 426,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age         0\n",
       "sex         0\n",
       "cp          0\n",
       "trestbps    0\n",
       "chol        0\n",
       "fbs         0\n",
       "restecg     0\n",
       "thalach     0\n",
       "exang       0\n",
       "oldpeak     0\n",
       "slope       0\n",
       "ca          4\n",
       "thal        2\n",
       "target      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 426,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(main_df == \"?\").sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have missing values in \"ca\" and \"thal\" columns. The \"ca\" column seems to be categorical with four categories -- but the categories have some natural ordering. \n",
    "The \"thal\" column is also categorical with three categories -- with no ordering in the categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     303\n",
       "unique      5\n",
       "top       0.0\n",
       "freq      176\n",
       "Name: ca, dtype: object"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"ca\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0    176\n",
       "1.0     65\n",
       "2.0     38\n",
       "3.0     20\n",
       "?        4\n",
       "Name: ca, dtype: int64"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"ca\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By and large, most people have ca=0, so we can make that replacement. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df[\"ca\"].replace(\"?\", 0, inplace=True)\n",
    "main_df[\"ca\"] = pd.to_numeric(main_df[\"ca\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    303.000000\n",
       "mean       0.663366\n",
       "std        0.934375\n",
       "min        0.000000\n",
       "25%        0.000000\n",
       "50%        0.000000\n",
       "75%        1.000000\n",
       "max        3.000000\n",
       "Name: ca, dtype: float64"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"ca\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0    180\n",
       "1.0     65\n",
       "2.0     38\n",
       "3.0     20\n",
       "Name: ca, dtype: int64"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"ca\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0    166\n",
       "7.0    117\n",
       "6.0     18\n",
       "?        2\n",
       "Name: thal, dtype: int64"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"thal\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the majority class is 3.0, so we can safely replace the missing values with 3.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df[\"thal\"].replace(\"?\", 3.0, inplace=True)\n",
    "main_df[\"thal\"] = pd.to_numeric(main_df[\"thal\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0    168\n",
       "7.0    117\n",
       "6.0     18\n",
       "Name: thal, dtype: int64"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"thal\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    303.000000\n",
       "mean       4.722772\n",
       "std        1.938383\n",
       "min        3.000000\n",
       "25%        3.000000\n",
       "50%        3.000000\n",
       "75%        7.000000\n",
       "max        7.000000\n",
       "Name: thal, dtype: float64"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"thal\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Categorical -> numeric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "column_names_to_one_hot = [ \n",
    "    \"cp\",\n",
    "    \"restecg\", \n",
    "    \"slope\", \n",
    "    \"thal\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, we have to convert the target to 1/0 form since we won't be predicting the severity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df[\"target\"]  = (main_df[\"target\"] > 0).astype(\"float\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      0.0\n",
       "1      1.0\n",
       "2      1.0\n",
       "3      0.0\n",
       "4      0.0\n",
       "      ... \n",
       "298    1.0\n",
       "299    1.0\n",
       "300    1.0\n",
       "301    1.0\n",
       "302    0.0\n",
       "Name: target, Length: 303, dtype: float64"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df[\"target\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "main_df = pd.get_dummies(main_df, columns=column_names_to_one_hot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>ca</th>\n",
       "      <th>target</th>\n",
       "      <th>...</th>\n",
       "      <th>cp_4.0</th>\n",
       "      <th>restecg_0.0</th>\n",
       "      <th>restecg_1.0</th>\n",
       "      <th>restecg_2.0</th>\n",
       "      <th>slope_1.0</th>\n",
       "      <th>slope_2.0</th>\n",
       "      <th>slope_3.0</th>\n",
       "      <th>thal_3.0</th>\n",
       "      <th>thal_6.0</th>\n",
       "      <th>thal_7.0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>233.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>160.0</td>\n",
       "      <td>286.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    age  sex  trestbps   chol  fbs  thalach  exang  oldpeak   ca  target  ...  \\\n",
       "0  63.0  1.0     145.0  233.0  1.0    150.0    0.0      2.3  0.0     0.0  ...   \n",
       "1  67.0  1.0     160.0  286.0  0.0    108.0    1.0      1.5  3.0     1.0  ...   \n",
       "2  67.0  1.0     120.0  229.0  0.0    129.0    1.0      2.6  2.0     1.0  ...   \n",
       "3  37.0  1.0     130.0  250.0  0.0    187.0    0.0      3.5  0.0     0.0  ...   \n",
       "4  41.0  0.0     130.0  204.0  0.0    172.0    0.0      1.4  0.0     0.0  ...   \n",
       "\n",
       "   cp_4.0  restecg_0.0  restecg_1.0  restecg_2.0  slope_1.0  slope_2.0  \\\n",
       "0       0            0            0            1          0          0   \n",
       "1       1            0            0            1          0          1   \n",
       "2       1            0            0            1          0          1   \n",
       "3       0            1            0            0          0          0   \n",
       "4       0            0            0            1          1          0   \n",
       "\n",
       "   slope_3.0  thal_3.0  thal_6.0  thal_7.0  \n",
       "0          1         0         1         0  \n",
       "1          0         1         0         0  \n",
       "2          0         0         0         1  \n",
       "3          1         1         0         0  \n",
       "4          0         1         0         0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['age', 'sex', 'trestbps', 'chol', 'fbs', 'thalach', 'exang', 'oldpeak',\n",
       "       'ca', 'target', 'cp_1.0', 'cp_2.0', 'cp_3.0', 'cp_4.0', 'restecg_0.0',\n",
       "       'restecg_1.0', 'restecg_2.0', 'slope_1.0', 'slope_2.0', 'slope_3.0',\n",
       "       'thal_3.0', 'thal_6.0', 'thal_7.0'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = main_df[\"target\"].to_numpy().astype(np.float64)\n",
    "X = main_df.drop(columns=\"target\").to_numpy().astype(np.float64)\n",
    "\n",
    "\n",
    "train_X, test_X, train_y, test_y = train_test_split(X, y, train_size=0.6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4. Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "logistic_regression_model = LogisticRegression(max_iter=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\user\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python37\\site-packages\\sklearn\\linear_model\\_logistic.py:818: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG,\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(max_iter=1000)"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logistic_regression_model.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_yhat = logistic_regression_model.predict(train_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.856353591160221"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(train_y, train_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8311688311688312"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(train_y, train_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_yhat = logistic_regression_model.predict(test_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8524590163934426 0.8363636363636364\n"
     ]
    }
   ],
   "source": [
    "print(accuracy_score(test_y, test_yhat), f1_score(test_y, test_yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad for a first attempt. You can read about the F1 score here. My main motivation for F1 score as opposed to just accuracy is this -- if a dataset is imbalanced and y = 0 in 90% of the cases, then a model that only predicts 0 for any input will have a 90% accuracy! \n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html \n",
    "\n",
    "https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9 \n",
    "\n",
    "https://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5. Naive Bayes (a couple of flavours)\n",
    "\n",
    "We will be trying two types of Naive Bayes here -- Gaussian, and Bernoulli. The method is pretty much the same, the only difference is that in Gaussian Naive Bayes, we assume that every feature in X is sampled from a Gaussian, while in Bernoulli Naive Bayes, we assume that every feature in X is sampled from a Bernoulli distribution.\n",
    "\n",
    "### 3.5.1 Gaussian "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GaussianNB()"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gnb = GaussianNB()\n",
    "gnb.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_yhat = gnb.predict(train_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7734806629834254"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(train_y, train_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7050359712230215"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(train_y, train_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8360655737704918 0.7959183673469388\n"
     ]
    }
   ],
   "source": [
    "test_yhat = gnb.predict(test_X)\n",
    "print(accuracy_score(test_y, test_yhat), f1_score(test_y, test_yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.2 Bernoulli"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BernoulliNB()"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bnb = BernoulliNB()\n",
    "bnb.fit(train_X, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "yhat = bnb.predict(train_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7734806629834254"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(train_y, train_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7050359712230215"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(train_y, train_yhat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8360655737704918 0.8214285714285714\n"
     ]
    }
   ],
   "source": [
    "test_yhat = bnb.predict(test_X)\n",
    "print(accuracy_score(test_y, test_yhat), f1_score(test_y, test_yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.6. Advanced topic (optional)\n",
    "\n",
    "Here, we have a mixture of continuous and categorical data. Therefore, a good option would be: \n",
    "\n",
    "1. Label encode the categorical columns. \n",
    "2. Perform Gaussian NB using just the continuous columns and MultinomialNB using the label-encoded categorical columns. \n",
    "3. Combine these two models to obtain the final prediction as given here: https://medium.com/analytics-vidhya/naive-bayes-for-mixed-typed-data-in-scikit-learn-fb6843e241f0 \n",
    "\n",
    "We can discuss this further in #doubts or in the TA sessions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.9 64-bit (windows store)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "a3228c686c13c3891ac7b9546204ca4390d65863877b8863b460a3da758c70c9"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}