notebook_examples/xgboost-with-rapids.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "@notebook{xgboost-with-rapids.ipynb,\n",
    "    title: XGBoost with RAPIDS,\n",
    "    summary: Compare training time between CPU and GPU trained models using XGBoost,\n",
    "    developed_on: rapids2110_p37_gpu_v1,\n",
    "    keywords: xgboost, rapids, gpu, machine learning, classification,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "\n",
    "!pip install -U oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2020, 2022 Oracle, Inc. All rights reserved. Licensed under the [Universal Permissive License v 1.0](https://oss.oracle.com/licenses/upl).\n",
    "\n",
    "---\n",
    "\n",
    "# <font color=\"red\">XGBoost with RAPIDS</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=\"teal\">Oracle Cloud Infrastructure Data Science Service.</font></p>\n",
    "\n",
    "---\n",
    "\n",
    "# Overview:\n",
    "\n",
    "The purpose of this notebook is to compare the speedup of the training time between an XGBoost model trained on GPUs versus the same model trained on CPUs.\n",
    "\n",
    "Compatible with: [NVIDIA RAPIDS 21.10](https://docs.oracle.com/en-us/iaas/data-science/using/conda-rapids-fam.htm) for GPU on Python 3.7 (version 1.0)\n",
    "\n",
    "---\n",
    "\n",
    "## Contents:\n",
    "\n",
    "- <a href='#synthetic-data'>Creating a Synthetic Dataset</a>    \n",
    "    - <a href='#split'>Splitting the Dataset into a Training and a Validation Sample</a>\n",
    "    - <a href='#convert'>Converting NumPy Arrays to XGBoost DMatrix Data Format</a>\n",
    "- <a href='#assign'>Assigning Values to XGBoost Hyperparameteers</a>\n",
    "- <a href='#cpu'>Training an XGBoost Model using CPUs</a>\n",
    "- <a href='#gpu'>Training an XGBoost Model using GPUs</a>\n",
    "- <a href='#conclusion'>Conclusion</a>\n",
    "\n",
    "---\n",
    "\n",
    "**Important:**\n",
    "\n",
    "Placeholder text for required values are surrounded by angle brackets that must be removed when adding the indicated content. For example, when adding a database name to `database_name = \"<database_name>\"` would become `database_name = \"production\"`.\n",
    "\n",
    "---\n",
    "\n",
    "Datasets are provided as a convenience.  Datasets are considered third-party content and are not considered materials \n",
    "under your agreement with Oracle.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:41.067879Z",
     "start_time": "2018-11-06T21:03:40.256654Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='synthetic-data'></a>\n",
    "## Creating a Synthetic Dataset\n",
    "\n",
    "The first step is to create synthetic training and validation datasets. The features will be tabular with `n_rows` and `n_columns` in the training dataset, where each value is either of type `np.float32` if the data is numerical or `np.uint8` if the data is categorical. Both numerical and categorical data can also be combined; for this experiment, we have ignored this combination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Synthetic dataset size (rows and column counts):\n",
    "n_rows = int(1e5)\n",
    "n_columns = int(100)\n",
    "n_categories = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to create the synthetic dataset:\n",
    "def simulate_data(m, n, k=2, numerical=False):\n",
    "    if numerical:\n",
    "        features = np.random.rand(m, n)\n",
    "    else:\n",
    "        features = np.random.randint(2, size=(m, n))\n",
    "    labels = np.random.randint(k, size=m)\n",
    "    return np.c_[labels, features].astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "dataset = simulate_data(n_rows, n_columns, n_categories)\n",
    "print(dataset.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='split'></a>\n",
    "### Splitting the Dataset into a Training and a Validation Sample\n",
    "\n",
    "We'll split our dataset into a 80% training dataset and a 20% validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# identify shape and indices\n",
    "n_rows, n_columns = dataset.shape\n",
    "train_size = 0.80\n",
    "train_index = int(n_rows * train_size)\n",
    "\n",
    "# split X, y\n",
    "X, y = dataset[:, 1:], dataset[:, 0]\n",
    "del dataset\n",
    "\n",
    "# split train data\n",
    "X_train, y_train = X[:train_index, :], y[:train_index]\n",
    "\n",
    "# split validation data\n",
    "X_validation, y_validation = X[train_index:, :], y[train_index:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the dimensions and proportions of our training and validation dataets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check dimensions\n",
    "print(\n",
    "    \"X_train: \", X_train.shape, X_train.dtype, \"y_train: \", y_train.shape, y_train.dtype\n",
    ")\n",
    "print(\n",
    "    \"X_validation\",\n",
    "    X_validation.shape,\n",
    "    X_validation.dtype,\n",
    "    \"y_validation: \",\n",
    "    y_validation.shape,\n",
    "    y_validation.dtype,\n",
    ")\n",
    "\n",
    "# check the proportions\n",
    "total = X_train.shape[0] + X_validation.shape[0]\n",
    "print(\"X_train proportion:\", X_train.shape[0] / total)\n",
    "print(\"X_validation proportion:\", X_validation.shape[0] / total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='convert'></a>\n",
    "### Converting NumPy Arrays to XGBoost DMatrix Data Format \n",
    "\n",
    "The next step is to convert the `X_train`, `y_train`, `X_validation`, `y_validation` numpy arrays to the data matrix (`DMatrix`) format that XGBoost supports. We can instantiate an object of the `xgboost.DMatrix` by passing in the feature matrix as the first argument followed by the label vector using the `label=` keyword argument. \n",
    "\n",
    "To learn more about XGBoost's support for data structures see the XGBoost documentation: \n",
    "https://xgboost.readthedocs.io/en/latest/python/python_intro.html#data-interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:55.278322Z",
     "start_time": "2018-11-06T21:03:54.059643Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# converting both training and validation datasets:\n",
    "dtrain = xgb.DMatrix(X_train, label=y_train)\n",
    "dvalidation = xgb.DMatrix(X_validation, label=y_validation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='assign'></a>\n",
    "## Assigning Values to XGBoost Hyperparameteers \n",
    "\n",
    "There are a number of parameters that can be set before XGBoost can be run. \n",
    "\n",
    "* General parameters relate to which booster we are using to do boosting, commonly tree or linear model\n",
    "* Booster parameters depend on which booster you have chosen\n",
    "* Learning task parameters decide on the learning scenario. For example, regression tasks may use different parameters with ranking tasks.\n",
    "\n",
    "For more information on the configurable parameters within the XGBoost module, see the documentation here:\n",
    "\n",
    "\n",
    "https://xgboost.readthedocs.io/en/latest/parameter.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:57.443698Z",
     "start_time": "2018-11-06T21:03:57.438288Z"
    }
   },
   "outputs": [],
   "source": [
    "# instantiate params\n",
    "params = {}\n",
    "\n",
    "# general params\n",
    "general_params = {\"silent\": 1}\n",
    "params.update(general_params)\n",
    "\n",
    "# learning task params\n",
    "learning_task_params = {\"eval_metric\": \"auc\", \"objective\": \"binary:logistic\"}\n",
    "params.update(learning_task_params)\n",
    "print(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='cpu'></a>\n",
    "## Training an XGBoost Model using CPUs\n",
    "\n",
    "Now it's time to train the model! You can use the `xgb.train` function and pass in the parameters, training dataset, the number of boosting iterations, and the list of items to be evaluated during training. For more information on the parameters that can be passed into `xgb.train`, check out the documentation:\n",
    "\n",
    "https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.train\n",
    "\n",
    "In the cell below, replace the value of `n_gpus` with 0. This will trigger moddel training on CPUs. Note XGBoost supports multithreading out-of-the-box. You will notice that all the CPUs of your VM are utilized. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# booster params.\n",
    "n_gpus = 0\n",
    "booster_params = {}\n",
    "\n",
    "if n_gpus != 0:\n",
    "    booster_params[\"tree_method\"] = \"gpu_hist\"\n",
    "    booster_params[\"n_gpus\"] = n_gpus\n",
    "params.update(booster_params)\n",
    "print(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model training settings\n",
    "evallist = [(dvalidation, \"validation\"), (dtrain, \"train\")]\n",
    "\n",
    "num_round = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:04:50.201308Z",
     "start_time": "2018-11-06T21:04:00.363740Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "bst = xgb.train(params, dtrain, num_round, evallist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='gpu'></a>\n",
    "## Training an XGBoost Model using GPUs\n",
    "\n",
    "Let's now repeat the same training step but this time we will use GPUs to train the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GPUs available? (1/0)\n",
    "n_gpus = 1\n",
    "booster_params = {}\n",
    "\n",
    "if n_gpus != 0:\n",
    "    booster_params[\"tree_method\"] = \"gpu_hist\"\n",
    "    booster_params[\"n_gpus\"] = n_gpus\n",
    "params.update(booster_params)\n",
    "print(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "bst = xgb.train(params, dtrain, num_round, evallist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was fast! You can typically get speedups of 50x or more depending on the VM shape and the generation of GPUs (e.g. P100 vs V100) you are using. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='conclusion'></a>\n",
    "## Conclusion\n",
    "\n",
    "We compared the performance of an xgboost model trained on CPUs vs GPUs. For additional resources, \n",
    "\n",
    "<a id='ref'></a>\n",
    "# References\n",
    "\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)\n",
    "- [NVIDIA RAPIDS](http://rapids.ai)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}