GP_LCCM_Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GP-LCCM Example\n",
    "The purpose of this notebook is to demonstrate how the Gaussian Process - Latent Class Choice Model (GP-LCCM) code from Sfeir et al. (2021) could be used.\n",
    "\n",
    "The application is based on the \"London Dataset\" from Hillel et al.(2018) which is available online as a supplementary material to the paper.\n",
    "\n",
    "The dataset contains individual trip diaries of the London Travel Demand Survey (LTDS) from April 2012 to March 2015 with their corresponding modes alternatives extracted from a Google directions application programming interface (API) and corresponding estimates of car operating costs and public transport fares. The dataset consists of 81,086 trips, four modes (walke, cycle, public transport, and drive), and different trip purposes (e.g. Home-Based Work, Home-Based Education, etc.).\n",
    "\n",
    "The required steps to build a travel mode choie model and run the GP-LCCM code successfully are listed below.\n",
    "\n",
    "##### References\n",
    "Sfeir, G., Rodrigues, F., Abou-Zeid, M. (2022). “Gaussian Process Latent Class Choice Models.” Transportation Research Part C: Emerging Technologies, 136, 103552. https://doi.org/10.1016/j.trc.2022.103552\n",
    "\n",
    "Hillel, T., Elshafie, M.Z.E.B., Jin, Y. (2018). \"Recreating passenger mode choice-sets for transport simulation: A case study of London, UK,\" in: Proceedings of the Institution of Civil Engineers - Smart Infrastructure and Construction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Import the GP-LCCM Package and All other Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import GP_LCCM\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pylogit as pl\n",
    "import warnings\n",
    "from collections import OrderedDict\n",
    "from sklearn import preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Load the Datasets\n",
    "The GP-LCCM consists of two sub-models, a class memberhsip model and a class-specific choice model:\n",
    "\n",
    "- The class membership model is defined as a Gaussian Process to cluster individuals probabilistically to K clusters (latent classes)\n",
    "\n",
    "- Conditioned on the class membership of an individual, the class-specific choice model estimates the probability of choosing a specific alternative\n",
    "\n",
    "The class-specific choice model needs the data to be in \"long\" format while the class membership model requires a \"wide\" format.\n",
    "\n",
    "For more details regarding how to convert your data from wide to long format, you can refer to the following link: https://github.com/timothyb0912/pylogit/blob/master/examples/notebooks/Main%20PyLogit%20Example.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the \"london\" dataset, which is available in wide format\n",
    "wide_london = pd.read_csv('dataset.csv')\n",
    "\n",
    "# the .copy() returns a copy of the dataset and ensures that modifications are made to a copy and not the original data\n",
    "wide_london = wide_london.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trip_id</th>\n",
       "      <th>household_id</th>\n",
       "      <th>person_n</th>\n",
       "      <th>trip_n</th>\n",
       "      <th>travel_mode</th>\n",
       "      <th>purpose</th>\n",
       "      <th>fueltype</th>\n",
       "      <th>faretype</th>\n",
       "      <th>bus_scale</th>\n",
       "      <th>survey_year</th>\n",
       "      <th>...</th>\n",
       "      <th>dur_pt_int_total</th>\n",
       "      <th>dur_pt_int_waiting</th>\n",
       "      <th>dur_pt_int_walking</th>\n",
       "      <th>pt_n_interchanges</th>\n",
       "      <th>dur_driving</th>\n",
       "      <th>cost_transit</th>\n",
       "      <th>cost_driving_total</th>\n",
       "      <th>cost_driving_fuel</th>\n",
       "      <th>cost_driving_con_charge</th>\n",
       "      <th>driving_traffic_percent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>drive</td>\n",
       "      <td>HBO</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>full</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.052222</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.111702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>drive</td>\n",
       "      <td>HBO</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>full</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.059444</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.112150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>drive</td>\n",
       "      <td>HBO</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>full</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.236667</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.203052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>drive</td>\n",
       "      <td>HBO</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>full</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.233333</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.78</td>\n",
       "      <td>0.78</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.160714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>drive</td>\n",
       "      <td>HBO</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>dis</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.229167</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.78</td>\n",
       "      <td>0.78</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.130909</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   trip_id  household_id  person_n  trip_n travel_mode purpose    fueltype  \\\n",
       "0        0             0         0       0       drive     HBO  Petrol_Car   \n",
       "1        1             0         0       1       drive     HBO  Petrol_Car   \n",
       "2        2             0         0       2       drive     HBO  Petrol_Car   \n",
       "3        3             0         0       3       drive     HBO  Petrol_Car   \n",
       "4        4             0         1       2       drive     HBO  Petrol_Car   \n",
       "\n",
       "  faretype  bus_scale  survey_year  ...  dur_pt_int_total  dur_pt_int_waiting  \\\n",
       "0     full        1.0            1  ...               0.0                 0.0   \n",
       "1     full        1.0            1  ...               0.0                 0.0   \n",
       "2     full        1.0            1  ...               0.0                 0.0   \n",
       "3     full        1.0            1  ...               0.0                 0.0   \n",
       "4      dis        1.0            1  ...               0.0                 0.0   \n",
       "\n",
       "   dur_pt_int_walking  pt_n_interchanges  dur_driving  cost_transit  \\\n",
       "0                 0.0                  0     0.052222           1.5   \n",
       "1                 0.0                  0     0.059444           1.5   \n",
       "2                 0.0                  0     0.236667           1.5   \n",
       "3                 0.0                  0     0.233333           1.5   \n",
       "4                 0.0                  0     0.229167           1.5   \n",
       "\n",
       "   cost_driving_total  cost_driving_fuel  cost_driving_con_charge  \\\n",
       "0                0.14               0.14                      0.0   \n",
       "1                0.15               0.15                      0.0   \n",
       "2                0.79               0.79                      0.0   \n",
       "3                0.78               0.78                      0.0   \n",
       "4                0.78               0.78                      0.0   \n",
       "\n",
       "   driving_traffic_percent  \n",
       "0                 0.111702  \n",
       "1                 0.112150  \n",
       "2                 0.203052  \n",
       "3                 0.160714  \n",
       "4                 0.130909  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wide_london.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 6 columns need to be added to the original dataset:\n",
    "# person_id: a unique id for each person regardless of the household id (starts from 1)\n",
    "# travel_mode_nb: 1 for walk, 2 for cycle, 3 for pt, and 4 for drive\n",
    "# walk_av, cycle_av, pt_av, drive_av: to determine the availability of each mode (1 if available and 0 otherwise).\n",
    "# For simplicity, we assume that all modes are available for all individuals\n",
    "\n",
    "# Create unique person_id\n",
    "wide_london['person_id'] = wide_london.groupby(['household_id', 'person_n']).ngroup() + 1\n",
    "\n",
    "# travel_mode_nb: 1 for walk, 2 for cycle, 3 for pt, and 4 for drive\n",
    "# walk_av, cycle_av, pt_av, drive_av: to determine the availability of each mode (1 if available and 0 otherwise).\n",
    "# For simplicity, we assume that all modes are available for all individuals\n",
    "\n",
    "# Define mapping\n",
    "mode_mapping = {'walk': 1, 'cycle': 2, 'pt': 3, 'drive': 4}\n",
    "\n",
    "# Apply mapping\n",
    "wide_london['travel_mode_nb'] = wide_london['travel_mode'].map(mode_mapping)\n",
    "\n",
    "wide_london['walk_av'] = 1\n",
    "wide_london['cycle_av'] = 1\n",
    "wide_london['pt_av'] = 1\n",
    "wide_london['drive_av'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for this application we will use a sample of 1000 individuals\n",
    "wide_london = wide_london[wide_london['person_id'] < 1000]\n",
    "\n",
    "# reset index \n",
    "wide_london.reset_index(inplace=True, drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>trip_id</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>household_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>person_n</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trip_n</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_mode</th>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purpose</th>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fueltype</th>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>faretype</th>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>dis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bus_scale</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>survey_year</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_year</th>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_month</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_date</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day_of_week</th>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>start_time_linear</th>\n",
       "      <td>10.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>17.35</td>\n",
       "      <td>19.583333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>driving_license</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car_ownership</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>distance</th>\n",
       "      <td>778</td>\n",
       "      <td>778</td>\n",
       "      <td>4562</td>\n",
       "      <td>4562</td>\n",
       "      <td>4562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_walking</th>\n",
       "      <td>0.218056</td>\n",
       "      <td>0.188889</td>\n",
       "      <td>1.037778</td>\n",
       "      <td>1.039444</td>\n",
       "      <td>1.037778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_cycling</th>\n",
       "      <td>0.092222</td>\n",
       "      <td>0.061944</td>\n",
       "      <td>0.355833</td>\n",
       "      <td>0.384444</td>\n",
       "      <td>0.355833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_pt_total</th>\n",
       "      <td>0.149444</td>\n",
       "      <td>0.168056</td>\n",
       "      <td>0.412222</td>\n",
       "      <td>0.463611</td>\n",
       "      <td>0.391389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_pt_access</th>\n",
       "      <td>0.134444</td>\n",
       "      <td>0.109444</td>\n",
       "      <td>0.203056</td>\n",
       "      <td>0.205556</td>\n",
       "      <td>0.203056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_pt_rail</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_pt_bus</th>\n",
       "      <td>0.016667</td>\n",
       "      <td>0.055556</td>\n",
       "      <td>0.210278</td>\n",
       "      <td>0.258611</td>\n",
       "      <td>0.189444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_pt_int_total</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_pt_int_waiting</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_pt_int_walking</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pt_n_interchanges</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dur_driving</th>\n",
       "      <td>0.052222</td>\n",
       "      <td>0.059444</td>\n",
       "      <td>0.236667</td>\n",
       "      <td>0.233333</td>\n",
       "      <td>0.229167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cost_transit</th>\n",
       "      <td>1.5</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cost_driving_total</th>\n",
       "      <td>0.14</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.78</td>\n",
       "      <td>0.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cost_driving_fuel</th>\n",
       "      <td>0.14</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.79</td>\n",
       "      <td>0.78</td>\n",
       "      <td>0.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cost_driving_con_charge</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>driving_traffic_percent</th>\n",
       "      <td>0.111702</td>\n",
       "      <td>0.11215</td>\n",
       "      <td>0.203052</td>\n",
       "      <td>0.160714</td>\n",
       "      <td>0.130909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>person_id</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_mode_nb</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>walk_av</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cycle_av</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pt_av</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>drive_av</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  0           1           2           3  \\\n",
       "trip_id                           0           1           2           3   \n",
       "household_id                      0           0           0           0   \n",
       "person_n                          0           0           0           0   \n",
       "trip_n                            0           1           2           3   \n",
       "travel_mode                   drive       drive       drive       drive   \n",
       "purpose                         HBO         HBO         HBO         HBO   \n",
       "fueltype                 Petrol_Car  Petrol_Car  Petrol_Car  Petrol_Car   \n",
       "faretype                       full        full        full        full   \n",
       "bus_scale                       1.0         1.0         1.0         1.0   \n",
       "survey_year                       1           1           1           1   \n",
       "travel_year                    2012        2012        2012        2012   \n",
       "travel_month                      4           4           4           4   \n",
       "travel_date                       1           1           1           1   \n",
       "day_of_week                       7           7           7           7   \n",
       "start_time_linear              10.0        13.0        17.0       17.35   \n",
       "age                              56          56          56          56   \n",
       "female                            1           1           1           1   \n",
       "driving_license                   1           1           1           1   \n",
       "car_ownership                     2           2           2           2   \n",
       "distance                        778         778        4562        4562   \n",
       "dur_walking                0.218056    0.188889    1.037778    1.039444   \n",
       "dur_cycling                0.092222    0.061944    0.355833    0.384444   \n",
       "dur_pt_total               0.149444    0.168056    0.412222    0.463611   \n",
       "dur_pt_access              0.134444    0.109444    0.203056    0.205556   \n",
       "dur_pt_rail                     0.0         0.0         0.0         0.0   \n",
       "dur_pt_bus                 0.016667    0.055556    0.210278    0.258611   \n",
       "dur_pt_int_total                0.0         0.0         0.0         0.0   \n",
       "dur_pt_int_waiting              0.0         0.0         0.0         0.0   \n",
       "dur_pt_int_walking              0.0         0.0         0.0         0.0   \n",
       "pt_n_interchanges                 0           0           0           0   \n",
       "dur_driving                0.052222    0.059444    0.236667    0.233333   \n",
       "cost_transit                    1.5         1.5         1.5         1.5   \n",
       "cost_driving_total             0.14        0.15        0.79        0.78   \n",
       "cost_driving_fuel              0.14        0.15        0.79        0.78   \n",
       "cost_driving_con_charge         0.0         0.0         0.0         0.0   \n",
       "driving_traffic_percent    0.111702     0.11215    0.203052    0.160714   \n",
       "person_id                         1           1           1           1   \n",
       "travel_mode_nb                    4           4           4           4   \n",
       "walk_av                           1           1           1           1   \n",
       "cycle_av                          1           1           1           1   \n",
       "pt_av                             1           1           1           1   \n",
       "drive_av                          1           1           1           1   \n",
       "\n",
       "                                  4  \n",
       "trip_id                           4  \n",
       "household_id                      0  \n",
       "person_n                          1  \n",
       "trip_n                            2  \n",
       "travel_mode                   drive  \n",
       "purpose                         HBO  \n",
       "fueltype                 Petrol_Car  \n",
       "faretype                        dis  \n",
       "bus_scale                       1.0  \n",
       "survey_year                       1  \n",
       "travel_year                    2012  \n",
       "travel_month                      4  \n",
       "travel_date                       1  \n",
       "day_of_week                       7  \n",
       "start_time_linear         19.583333  \n",
       "age                              54  \n",
       "female                            0  \n",
       "driving_license                   1  \n",
       "car_ownership                     2  \n",
       "distance                       4562  \n",
       "dur_walking                1.037778  \n",
       "dur_cycling                0.355833  \n",
       "dur_pt_total               0.391389  \n",
       "dur_pt_access              0.203056  \n",
       "dur_pt_rail                     0.0  \n",
       "dur_pt_bus                 0.189444  \n",
       "dur_pt_int_total                0.0  \n",
       "dur_pt_int_waiting              0.0  \n",
       "dur_pt_int_walking              0.0  \n",
       "pt_n_interchanges                 0  \n",
       "dur_driving                0.229167  \n",
       "cost_transit                    1.5  \n",
       "cost_driving_total             0.78  \n",
       "cost_driving_fuel              0.78  \n",
       "cost_driving_con_charge         0.0  \n",
       "driving_traffic_percent    0.130909  \n",
       "person_id                         2  \n",
       "travel_mode_nb                    4  \n",
       "walk_av                           1  \n",
       "cycle_av                          1  \n",
       "pt_av                             1  \n",
       "drive_av                          1  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the first 5 rows of the data\n",
    "wide_london.head(5).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Convert the data from wide to long format\n",
    "We will rely in this section on the pylogit package which was developed to estimate many discrete choice models such as multinomial logit models, mixed logit and nested logit https://pypi.org/project/pylogit/. The package can be also used to convert the data from wide to long format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['trip_id', 'household_id', 'person_n', 'trip_n', 'travel_mode',\n",
       "       'purpose', 'fueltype', 'faretype', 'bus_scale', 'survey_year',\n",
       "       'travel_year', 'travel_month', 'travel_date', 'day_of_week',\n",
       "       'start_time_linear', 'age', 'female', 'driving_license',\n",
       "       'car_ownership', 'distance', 'dur_walking', 'dur_cycling',\n",
       "       'dur_pt_total', 'dur_pt_access', 'dur_pt_rail', 'dur_pt_bus',\n",
       "       'dur_pt_int_total', 'dur_pt_int_waiting', 'dur_pt_int_walking',\n",
       "       'pt_n_interchanges', 'dur_driving', 'cost_transit',\n",
       "       'cost_driving_total', 'cost_driving_fuel', 'cost_driving_con_charge',\n",
       "       'driving_traffic_percent', 'person_id', 'travel_mode_nb', 'walk_av',\n",
       "       'cycle_av', 'pt_av', 'drive_av'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the columns of the london dataset\n",
    "wide_london.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the list of individual specific variables (variables that are specific to an individual \n",
    "# and do not vary across alternatives)\n",
    "ind_variables = ['trip_id', 'household_id', 'person_n', 'trip_n', 'travel_mode',\n",
    "       'purpose', 'fueltype', 'faretype', 'bus_scale', 'survey_year',\n",
    "       'travel_year', 'travel_month', 'travel_date', 'day_of_week',\n",
    "       'start_time_linear', 'age', 'female', 'driving_license',\n",
    "       'car_ownership', 'person_id']\n",
    "\n",
    "\n",
    "# Specify the variables that vary across individuals and some or all alternatives\n",
    "# The keys are the column names that will be used in the long format dataframe\n",
    "# The values are dictionaries whose key-value pairs are the alternative id and\n",
    "# the column name of the corresponding column that encodes that variable for\n",
    "# the given alternative. Examples below.\n",
    "alt_varying_variables = {u'travel_time': dict([(1, 'dur_walking'),\n",
    "                                               (2, 'dur_cycling'),\n",
    "                                               (3, 'dur_pt_total'),\n",
    "                                               (4, 'dur_driving')]),\n",
    "                        u'travel_cost': dict([(3, 'cost_transit'),\n",
    "                                              (4, 'cost_driving_total')])}\n",
    "\n",
    "# Specify the availability variables\n",
    "# Note that the keys of the dictionary are the alternative id's.\n",
    "# The values are the columns denoting the availability for the\n",
    "# given mode in the dataset.\n",
    "availability_variables = {1: 'walk_av',\n",
    "                          2: 'cycle_av',\n",
    "                          3: 'pt_av',\n",
    "                          4: 'drive_av'}\n",
    "\n",
    "\n",
    "# Determine the columns for: alternative ids, the observation ids and the choice\n",
    "# The 'custom_alt_id' is the name of a column to be created in the long-format data\n",
    "# It will identify the alternative associated with each row.\n",
    "custom_alt_id = \"mode_id\"\n",
    "\n",
    "# Create a custom id column that ignores the fact that this is a \n",
    "# panel/repeated-observations dataset. Note the +1 ensures the id's start at one.\n",
    "obs_id_column = \"custom_id\"\n",
    "wide_london[obs_id_column] = np.arange(wide_london.shape[0],\n",
    "                                            dtype=int) + 1\n",
    "\n",
    "\n",
    "# Create a variable recording the choice column\n",
    "choice_column = \"travel_mode_nb\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/GEOSAF/Desktop/BiogemeEnv/lib/python3.12/site-packages/pylogit/choice_tools.py:454: UserWarning: Note, there are 43 variables in wide_data but the inputs ind_vars, alt_specific_vars, and subset_specific_vars only account for 32 variables.\n",
      "  warnings.warn(msg.format(num_dataframe_vars, data_title) +\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>custom_id</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mode_id</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_mode_nb</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trip_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>household_id</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>person_n</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trip_n</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_mode</th>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "      <td>drive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purpose</th>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "      <td>HBO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fueltype</th>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "      <td>Petrol_Car</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>faretype</th>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "      <td>full</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bus_scale</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>survey_year</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_year</th>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_month</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_date</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day_of_week</th>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>start_time_linear</th>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>driving_license</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>car_ownership</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>person_id</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_time</th>\n",
       "      <td>0.218056</td>\n",
       "      <td>0.092222</td>\n",
       "      <td>0.149444</td>\n",
       "      <td>0.052222</td>\n",
       "      <td>0.188889</td>\n",
       "      <td>0.061944</td>\n",
       "      <td>0.168056</td>\n",
       "      <td>0.059444</td>\n",
       "      <td>1.037778</td>\n",
       "      <td>0.355833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>travel_cost</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            0           1           2           3           4  \\\n",
       "custom_id                   1           1           1           1           2   \n",
       "mode_id                     1           2           3           4           1   \n",
       "travel_mode_nb              0           0           0           1           0   \n",
       "trip_id                     0           0           0           0           1   \n",
       "household_id                0           0           0           0           0   \n",
       "person_n                    0           0           0           0           0   \n",
       "trip_n                      0           0           0           0           1   \n",
       "travel_mode             drive       drive       drive       drive       drive   \n",
       "purpose                   HBO         HBO         HBO         HBO         HBO   \n",
       "fueltype           Petrol_Car  Petrol_Car  Petrol_Car  Petrol_Car  Petrol_Car   \n",
       "faretype                 full        full        full        full        full   \n",
       "bus_scale                 1.0         1.0         1.0         1.0         1.0   \n",
       "survey_year                 1           1           1           1           1   \n",
       "travel_year              2012        2012        2012        2012        2012   \n",
       "travel_month                4           4           4           4           4   \n",
       "travel_date                 1           1           1           1           1   \n",
       "day_of_week                 7           7           7           7           7   \n",
       "start_time_linear        10.0        10.0        10.0        10.0        13.0   \n",
       "age                        56          56          56          56          56   \n",
       "female                      1           1           1           1           1   \n",
       "driving_license             1           1           1           1           1   \n",
       "car_ownership               2           2           2           2           2   \n",
       "person_id                   1           1           1           1           1   \n",
       "travel_time          0.218056    0.092222    0.149444    0.052222    0.188889   \n",
       "travel_cost               0.0         0.0         1.5        0.14         0.0   \n",
       "\n",
       "                            5           6           7           8           9  \n",
       "custom_id                   2           2           2           3           3  \n",
       "mode_id                     2           3           4           1           2  \n",
       "travel_mode_nb              0           0           1           0           0  \n",
       "trip_id                     1           1           1           2           2  \n",
       "household_id                0           0           0           0           0  \n",
       "person_n                    0           0           0           0           0  \n",
       "trip_n                      1           1           1           2           2  \n",
       "travel_mode             drive       drive       drive       drive       drive  \n",
       "purpose                   HBO         HBO         HBO         HBO         HBO  \n",
       "fueltype           Petrol_Car  Petrol_Car  Petrol_Car  Petrol_Car  Petrol_Car  \n",
       "faretype                 full        full        full        full        full  \n",
       "bus_scale                 1.0         1.0         1.0         1.0         1.0  \n",
       "survey_year                 1           1           1           1           1  \n",
       "travel_year              2012        2012        2012        2012        2012  \n",
       "travel_month                4           4           4           4           4  \n",
       "travel_date                 1           1           1           1           1  \n",
       "day_of_week                 7           7           7           7           7  \n",
       "start_time_linear        13.0        13.0        13.0        17.0        17.0  \n",
       "age                        56          56          56          56          56  \n",
       "female                      1           1           1           1           1  \n",
       "driving_license             1           1           1           1           1  \n",
       "car_ownership               2           2           2           2           2  \n",
       "person_id                   1           1           1           1           1  \n",
       "travel_time          0.061944    0.168056    0.059444    1.037778    0.355833  \n",
       "travel_cost               0.0         1.5        0.15         0.0         0.0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform the conversion to long-format\n",
    "long_london = pl.convert_wide_to_long(wide_london, \n",
    "                                           ind_variables, \n",
    "                                           alt_varying_variables, \n",
    "                                           availability_variables, \n",
    "                                           obs_id_column, \n",
    "                                           choice_column,\n",
    "                                           new_alt_id_name=custom_alt_id)\n",
    "# Look at the resulting long-format dataframe\n",
    "long_london.head(10).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Model Specification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4.1: Class Membership"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the number of clusters (latent classes) to 2\n",
    "n_classes = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create dummy variables for car ownership\n",
    "wide_london['car_ownership0'] = (wide_london['car_ownership'] == 0).astype(int)\n",
    "wide_london['car_ownership1'] = (wide_london['car_ownership'] == 1).astype(int)\n",
    "wide_london['car_ownership2'] = (wide_london['car_ownership'] == 2).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/m2/kskgzsyn6t9b8xsjgl711fv00000gp/T/ipykernel_7174/1145831177.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_Mem2.drop_duplicates(subset =\"person_id\",keep= 'first', inplace = True)\n"
     ]
    }
   ],
   "source": [
    "# the class membership model is defined as a Gaussian Process\n",
    "\n",
    "# select the variables for the class membership model\n",
    "X_Mem2 = wide_london[['person_id','age','female','driving_license','car_ownership1','car_ownership2']]\n",
    "\n",
    "# remove duplicates\n",
    "X_Mem2.drop_duplicates(subset =\"person_id\",keep= 'first', inplace = True)\n",
    "\n",
    "X_Mem = X_Mem2[['age','female','driving_license','car_ownership1','car_ownership2']]\n",
    "\n",
    "# it is a good practice to standardize the variables for the Gaussian Process Model\n",
    "Standardized_X_Mem = preprocessing.scale(X_Mem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different kernel or combination of kernel functions can be used for the Gaussian Process model. In this exmaple we use the following kernel function: DotProduct(sigma_0=1)\n",
    "\n",
    "The kernel function can be changed from the GP_LCCM.py file under: #### kernel function.\n",
    "\n",
    "For more details about the different kernel functions that can be used, you can refer to Section 1.7.5 on the following link:\n",
    "https://scikit-learn.org/stable/modules/gaussian_process.html#kernels-for-gaussian-processes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4.2: Class-Specific Choice Model\n",
    "This section is based on the latent class choice model (lccm) package. For more details on how to specify the class-specific choice model, you can check the lccm package on the following link: https://github.com/ferasz/LCCM\n",
    "\n",
    "You can use alternative specific or generic speficifation for the parameters. \n",
    "We will use in this example both types to demonstrate how each one can be used:\n",
    "\n",
    "- intercept: alternative specific constants included in all the utilities except the one related to walk, hence the one braket with the numbers of the corresponding alternatives [2,3,4] (walk is used as a base)\n",
    "- travel time parameters for both classes are specified as alternative specific and included in the utilities of all modes, hence the one braket with the numbers of the corresponding alternatives [1,2,3,4]\n",
    "- travel cost parameters for both classes are specified as generic and included only in the utilities of drive and pt, hence the two brakets and the numbers of the alternatives [[3,4]]\n",
    "\n",
    "\n",
    "The specification of the two classes is as follows:\n",
    "##### Latent Class 1:\n",
    "\n",
    "$V_{walk} =        \\beta_{tt, class1} * TravelTime_{walk}  $  \n",
    "\n",
    "$V_{cycle} = ASC-cycle_{class1} + \\beta_{tt, class1} * TravelTime_{cycle}$\n",
    "\n",
    "$V_{pt} = ASC-pt_{class1} + \\beta_{tt, class1} * TravelTime_{pt} + \\beta_{cost, class1} * TravelCost_{pt} $\n",
    "\n",
    "$V_{drive} = ASC-drive_{class1} + \\beta_{tt, class1} * TravelTime_{drive} + \\beta_{cost, class1} * TravelCost_{drive} $\n",
    "\n",
    "##### Latent Class 2:\n",
    "\n",
    "$V_{walk} =        \\beta_{tt, class2} * TravelTime_{walk}  $  \n",
    "\n",
    "$V_{cycle} = ASC-cycle_{class2} + \\beta_{tt, class2} * TravelTime_{cycle}$\n",
    "\n",
    "$V_{pt} = ASC-pt_{class2} + \\beta_{tt, class2} * TravelTime_{pt} + \\beta_{cost, class2} * TravelCost_{pt} $\n",
    "\n",
    "$V_{drive} = ASC-drive_{class2} + \\beta_{tt, class2} * TravelTime_{drive} + \\beta_{cost, class2} * TravelCost_{drive} $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scale travel time from hours to minutes\n",
    "long_london['travel_time'] = long_london['travel_time'] * 60\n",
    "\n",
    "# add a column of ones named 'intercept'. This will be used later to define the Alternative-Specific Constants (ASCs).\n",
    "long_london['intercept']=np.ones(long_london.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['custom_id', 'mode_id', 'travel_mode_nb', 'trip_id', 'household_id',\n",
       "       'person_n', 'trip_n', 'travel_mode', 'purpose', 'fueltype', 'faretype',\n",
       "       'bus_scale', 'survey_year', 'travel_year', 'travel_month',\n",
       "       'travel_date', 'day_of_week', 'start_time_linear', 'age', 'female',\n",
       "       'driving_license', 'car_ownership', 'person_id', 'travel_time',\n",
       "       'travel_cost', 'intercept'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "long_london.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: Specification and variable names must be in lists of ordered dictionaries.\n",
    "# class_specific_specs defines the variables names to be used in the specification of the \n",
    "# class specific choice model of each class.\n",
    "# class_specific_labels defines the names associated with each of the variables which\n",
    "# will be displayed in the output tables after estimation.\n",
    "\n",
    "\n",
    "class_specific_specs = [OrderedDict([('intercept',[2,3,4]),\n",
    "                                     ('travel_time',[1,2,3,4]),\n",
    "                                     ('travel_cost',[[3,4]])]),\n",
    "                        OrderedDict([('intercept',[2,3,4]),\n",
    "                                     ('travel_time',[1,2,3,4]),\n",
    "                                     ('travel_cost',[[3,4]])])]\n",
    "                       \n",
    "class_specific_labels = [OrderedDict([('ASC',['ASC(cycle)',\n",
    "                                             'ASC(pt)',\n",
    "                                             'ASC(drive)',]),\n",
    "                                      ('travel_time',['Travel Time(walk)',\n",
    "                                                      'Travel Time(cycle)',\n",
    "                                                      'Travel Time(pt)',\n",
    "                                                      'Travel Time(drive)']),\n",
    "                                      ('travel_cost',['Travel Cost'])]),\n",
    "                         OrderedDict([('ASC',['ASC(cycle)',\n",
    "                                             'ASC(pt)',\n",
    "                                             'ASC(drive)',]),\n",
    "                                      ('travel_time',['Travel Time(walk)',\n",
    "                                                      'Travel Time(cycle)',\n",
    "                                                      'Travel Time(pt)',\n",
    "                                                      'Travel Time(drive)']),\n",
    "                                      ('travel_cost',['Travel Cost'])])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# starting values for the Class-Specific Choice Model paramerters\n",
    "paramClassSpec = []\n",
    "paramClassSpec.append(np.array([0,0,0,0,0,0,0,0])) # for the first class\n",
    "paramClassSpec.append(np.array([0,0,0,0,0,0,0,0])) # for the second class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Model Estimation and Results\n",
    "\n",
    "To train/estimate the model, the following must be included in the GP_LCCM.lccm_fit function:\n",
    "- data: data used for the class-speficic choice model in long format\n",
    "- X: variables used for the class membership model in wide format\n",
    "\n",
    "For prediction/test, the following must be included in the GP_LCCM.lccm_fit function:\n",
    "- dataTest = data for the class-speficic choice model in long format\n",
    "- XTest = variables used for the class membership model in wide format\n",
    "\n",
    "Note that after training your model you can test it on a different dataset (test dataset). For this example, we use the same one that was used in training. However, you can split the data into train and test datasets(e.g. 80%-20%), estimate the model on the train dataset (data, X) and test the model on the test dataset (dataTest, XTest). The model will generate the prediction accuracy in terms of LL at the end of the notebook (Predicted Log-Likelihood).\n",
    "\n",
    "The remaining variables that must be included in the GP_LCCM.lccm_fit function are:\n",
    "- prediction_test: 'Yes' if you would like to test the model on a test dataset and 'No' otherwise\n",
    "- ind_id_col: name of column identifying the individual for each row of data \n",
    "- obs_id_col: name of column identifying the observation (choice scenario)\n",
    "- choice_col: name of column identifying whether the alternative corresponding by a row was chosen or not\n",
    "- n_classes: number of latent classes (clusters)\n",
    "- paramClassSpec: starting values for the class-specific choice models. If paramClassSpec is None, the model will generate the starting values randomly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing data\n",
      "Initializing EM Algorithm...\n",
      "\n",
      "<Tue, 11 Feb 2025 14:35:41> Iteration 0: -3119.0130\n",
      "<Tue, 11 Feb 2025 14:35:42> Iteration 1: -2156.8601\n",
      "<Tue, 11 Feb 2025 14:35:43> Iteration 2: -2080.6024\n",
      "<Tue, 11 Feb 2025 14:35:44> Iteration 3: -2008.1784\n",
      "<Tue, 11 Feb 2025 14:35:45> Iteration 4: -1914.8490\n",
      "<Tue, 11 Feb 2025 14:35:45> Iteration 5: -1816.6073\n",
      "<Tue, 11 Feb 2025 14:35:45> Iteration 6: -1733.7329\n",
      "<Tue, 11 Feb 2025 14:35:46> Iteration 7: -1667.7187\n",
      "<Tue, 11 Feb 2025 14:35:48> Iteration 8: -1631.7268\n",
      "<Tue, 11 Feb 2025 14:35:48> Iteration 9: -1618.8634\n",
      "<Tue, 11 Feb 2025 14:35:48> Iteration 10: -1616.2394\n",
      "<Tue, 11 Feb 2025 14:35:50> Iteration 11: -1613.8252\n",
      "<Tue, 11 Feb 2025 14:35:50> Iteration 12: -1611.8262\n",
      "<Tue, 11 Feb 2025 14:35:51> Iteration 13: -1610.3688\n",
      "<Tue, 11 Feb 2025 14:35:52> Iteration 14: -1609.3526\n",
      "<Tue, 11 Feb 2025 14:35:53> Iteration 15: -1608.6399\n",
      "<Tue, 11 Feb 2025 14:35:53> Iteration 16: -1608.0656\n",
      "<Tue, 11 Feb 2025 14:35:54> Iteration 17: -1607.8427\n",
      "<Tue, 11 Feb 2025 14:35:55> Iteration 18: -1607.5910\n",
      "<Tue, 11 Feb 2025 14:35:55> Iteration 19: -1607.4764\n",
      "<Tue, 11 Feb 2025 14:35:56> Iteration 20: -1607.5780\n",
      "<Tue, 11 Feb 2025 14:35:56> Iteration 21: -1607.5371\n",
      "<Tue, 11 Feb 2025 14:35:57> Iteration 22: -1607.4999\n",
      "<Tue, 11 Feb 2025 14:35:58> Iteration 23: -1607.4747\n",
      "<Tue, 11 Feb 2025 14:35:58> Iteration 24: -1607.4850\n",
      "<Tue, 11 Feb 2025 14:35:59> Iteration 25: -1607.5247\n",
      "<Tue, 11 Feb 2025 14:35:59> Iteration 26: -1607.5216\n",
      "<Tue, 11 Feb 2025 14:36:00> Iteration 27: -1607.5205\n",
      "<Tue, 11 Feb 2025 14:36:00> Iteration 28: -1607.5200\n",
      "<Tue, 11 Feb 2025 14:36:01> Iteration 29: -1607.5198\n",
      "\n",
      "Enumerating choices for the sample\n",
      "\n",
      "\n",
      "Number of Observations:                             2571\n",
      "Null Log-Likelihood:                            -3573.52\n",
      "Fitted Log-Likelihood:                          -1582.11\n",
      "Rho-Squared:                                        0.56\n",
      "Estimation time (minutes):                          0.35\n",
      "\n",
      "\n",
      "Normalized Log-Likelihood:                      -1607.52\n",
      "\n",
      "\n",
      "\n",
      "Class-Specific Choice Model:\n",
      "-----------------------------------------------------------------------------------------\n",
      "Number of Parameters:                                 16\n",
      "\n",
      "Class 1 Model: \n",
      "-----------------------------------------------------------------------------------------\n",
      "Variables                                     parameters    std_err     t_stat    p_value\n",
      "-----------------------------------------------------------------------------------------\n",
      "ASC(cycle)                                       -5.1488     0.2937   -17.5322     0.0000\n",
      "ASC(pt)                                          -1.9645     0.1491   -13.1777     0.0000\n",
      "ASC(drive)                                       -4.8136     0.3900   -12.3411     0.0000\n",
      "Travel Time(walk)                                -0.1282     0.0033   -38.6646     0.0000\n",
      "Travel Time(cycle)                               -0.0872     0.0120    -7.2413     0.0000\n",
      "Travel Time(pt)                                  -0.0476     0.0075    -6.3930     0.0000\n",
      "Travel Time(drive)                               -0.1187     0.0242    -4.9020     0.0000\n",
      "Travel Cost                                      -0.4727     0.0759    -6.2244     0.0000\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "Class 2 Model: \n",
      "-----------------------------------------------------------------------------------------\n",
      "Variables                                     parameters    std_err     t_stat    p_value\n",
      "-----------------------------------------------------------------------------------------\n",
      "ASC(cycle)                                       -6.8228     1.0170    -6.7086     0.0000\n",
      "ASC(pt)                                          -4.1427     0.2730   -15.1751     0.0000\n",
      "ASC(drive)                                       -1.6031     0.1243   -12.8985     0.0000\n",
      "Travel Time(walk)                                -0.1934     0.0070   -27.6468     0.0000\n",
      "Travel Time(cycle)                               -0.1055     0.0618    -1.7076     0.0877\n",
      "Travel Time(pt)                                  -0.0632     0.0066    -9.5974     0.0000\n",
      "Travel Time(drive)                               -0.0994     0.0068   -14.5234     0.0000\n",
      "Travel Cost                                      -0.1598     0.0348    -4.5966     0.0000\n",
      "-----------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------\n",
      "Predicted Log-Likelihood:                       -1607.52\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# define a location to store the resutls \n",
    "outputFilePath = '.../Results/'\n",
    "\n",
    "# run the model \n",
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter('ignore')\n",
    "    \n",
    "    \n",
    "    GP_LCCM.lccm_fit(data = long_london,\n",
    "                             X = Standardized_X_Mem,\n",
    "                             dataTest = long_london,\n",
    "                             XTest = Standardized_X_Mem,\n",
    "                             prediction_test = 'Yes',\n",
    "                             ind_id_col = 'person_id', \n",
    "                             obs_id_col = 'custom_id',\n",
    "                             alt_id_col = 'mode_id',\n",
    "                             choice_col = 'travel_mode_nb', \n",
    "                             n_classes = n_classes,\n",
    "                             reg_covar = 1e-6,\n",
    "                             tol = 1e-3,\n",
    "                             max_iter = 100,\n",
    "                             class_specific_specs = class_specific_specs,\n",
    "                             class_specific_labels = class_specific_labels,\n",
    "                             #paramClassSpec = paramClassSpec,\n",
    "                             outputFilePath = outputFilePath,\n",
    "                             outputFileName = 'ModelResults')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DotProduct(sigma_0=1)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# initial kernel function\n",
    "GP_LCCM.gpc_model.kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DotProduct(sigma_0=0.406)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# estimated kernel function\n",
    "GP_LCCM.gpc_model.kernel_"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "BiogemeEnv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}