penguins/cohort.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "00b7ba7b-433c-463c-8e5e-8b975a5be463",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Machine Learning In Production Using SageMaker\n",
    "\n",
    "This notebook creates a [SageMaker Pipeline](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_building_pipeline.html) to build an end-to-end Machine Learning system to solve the problem of classifying penguin species. With a SageMaker Pipeline, you can create, automate, and manage end-to-end Machine Learning workflows at scale.\n",
    "\n",
    "You can find more information about Amazon SageMaker in the [Amazon SageMaker Developer Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html). The [AWS Machine Learning Blog](https://aws.amazon.com/blogs/machine-learning/) is an excellent source to stay up-to-date with SageMaker.\n",
    "\n",
    "This example uses the [Penguins dataset](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data), the [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html) library, and the [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/). \n",
    "\n",
    "<img src='https://imgur.com/orZWHly.png' alt='Penguins dataset' width=\"800\">\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "1. [Exploratory Data Analysis](#Exploratory-Data-Analysis) \n",
    "2. [Training Pipeline](#Training-Pipeline)\n",
    "   * [Preprocessing](#Preprocessing)\n",
    "   * [Training](#Training)   \n",
    "   * [Tuning](#Tuning)\n",
    "   * [Evaluation](#Evaluation)\n",
    "   * [Inference Pipeline](#Inference-Pipeline)\n",
    "   * [Data Quality Baseline](#Data-Quality-Baseline)\n",
    "   * [Model Quality Baseline](#Model-Quality-Baseline)\n",
    "   * [Registration](#Registration)\n",
    "   * [Condition Step](#Condition-Step)\n",
    "   * [Pipeline](#Pipeline)\n",
    "   * [Quality Baselines](#Quality-Baselines)\n",
    "3. [Model Deployment](#Model-Deployment)\n",
    "   * [Lambda](#Lambda)\n",
    "   * [EventBridge](#EventBridge)\n",
    "   * [Predictions](#Predictions)\n",
    "   * [Data Capture](#Data-Capture)\n",
    "   * [Clean up](#Clean-up)\n",
    "4. [Model Monitoring](#Model-Monitoring)\n",
    "   * [Fake Traffic](#Fake-Traffic)\n",
    "   * [Fake Labels](#Fake-Labels)\n",
    "   * [Monitoring Jobs](#Monitoring-Jobs)\n",
    "\n",
    "This notebook is part of the [Machine Learning School](https://www.ml.school) program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "212a0538-62b5-4875-a59c-1a4e1a833a07",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n",
      "The dotenv extension is already loaded. To reload it, use:\n",
      "  %reload_ext dotenv\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%load_ext dotenv\n",
    "%dotenv\n",
    "\n",
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "CODE_FOLDER = Path(\"code\")\n",
    "CODE_FOLDER.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "sys.path.append(f\"./{CODE_FOLDER}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58994d91",
   "metadata": {},
   "source": [
    "To run this notebook in **Local Mode** set the `LOCAL_MODE` constant to `True`. \n",
    "\n",
    "To build our system using SageMaker, we need access to `ml.m5.xlarge` instances. By default, the quota on a new AWS account is zero, so you need to request a quota increase. You can do that under Service Quotas > AWS Services > Amazon SageMaker. Find `ml.m5.xlarge` and request a quota increase for processing jobs, training jobs, transform jobs, and endpoint usage. In the meantime, you can use `ml.t3.large` as a substitute.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "c1f894e2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:Couldn't call 'get_role' to get Role ARN from role name svpino to get Role path.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import logging\n",
    "\n",
    "# By default, The SageMaker SDK logs events related to the default\n",
    "# configuration using the INFO level. To prevent these from spoiling\n",
    "# the output of this notebook cells, we can change the logging\n",
    "# level to ERROR instead.\n",
    "logging.getLogger(\"sagemaker.config\").setLevel(logging.ERROR)\n",
    "\n",
    "import sagemaker\n",
    "from sagemaker.workflow.pipeline_context import PipelineSession, LocalPipelineSession\n",
    "\n",
    "LOCAL_MODE = False\n",
    "BUCKET = os.environ[\"BUCKET\"]\n",
    "\n",
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except ValueError:\n",
    "    role = os.environ[\"ROLE\"]\n",
    "\n",
    "# If you are running this notebook on an ARM64 machine, you will need\n",
    "# to build a custom Docker image using the setup notebook. This is\n",
    "# because SageMaker doesn't provide a TensorFlow image that supports \n",
    "# The Apple M chips.\n",
    "architecture = !(uname -m)\n",
    "IS_APPLE_M_CHIP = architecture[0] == \"arm64\"\n",
    "\n",
    "# We'll use these two variables to configure the steps that do not support\n",
    "# Local Mode.\n",
    "pipeline_session = PipelineSession(default_bucket=BUCKET) if not LOCAL_MODE else None\n",
    "\n",
    "if LOCAL_MODE:\n",
    "    config = {\n",
    "        \"session\": LocalPipelineSession(default_bucket=BUCKET),\n",
    "        \"instance_type\": \"local\",\n",
    "\n",
    "        # This is a dummy role that will be ignored when we run the\n",
    "        # pipeline in Local Mode.\n",
    "        \"role\": \"arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-11111111111111\",\n",
    "\n",
    "        # We need to use a custom Docker image when we run the pipeline\n",
    "        # in Local Model on an ARM64 machine.\n",
    "        \"image\": \"sagemaker-tensorflow-training-toolkit-local\" if IS_APPLE_M_CHIP else None,\n",
    "        \"framework_version\": None if IS_APPLE_M_CHIP else \"2.11\",\n",
    "        \"py_version\": None if IS_APPLE_M_CHIP else \"py39\",\n",
    "    }\n",
    "else:\n",
    "    config = {\n",
    "        \"session\": pipeline_session,\n",
    "        \"instance_type\": \"ml.m5.xlarge\",\n",
    "        \"role\": role,\n",
    "        \"image\": None,        \n",
    "        \"framework_version\": \"2.11\",\n",
    "        \"py_version\": \"py39\",\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11137928-6b4e-465c-8ad7-2297afbaa33c",
   "metadata": {},
   "source": [
    "# Exploratory Data Analysis\n",
    "\n",
    "Let's run Exploratory Data Analysis on the dataset. The goal of this section is to understand the data and the problem we are trying to solve."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb3fc999-1c6d-46ba-ab3a-0fdf819536ca",
   "metadata": {},
   "source": [
    "The first step is to create an S3 bucket where we will store the data and every resource we are going to create.\n",
    "\n",
    "<div class=\"alert\" style=\"background-color:#6e420c; color: #fff\"><strong>Note:</strong> \n",
    "    If you want to create a bucket in a region other than <strong>us-east-1</strong>, you need to use the \"--create-bucket-configuration\" argument when creating the bucket. You can see an example below.\n",
    "</div>\n",
    "\n",
    "Example of how to specify a region different from `us-east-1` when creating a bucket:\n",
    "\n",
    "```\n",
    "!aws s3api create-bucket --bucket $BUCKET --create-bucket-configuration LocationConstraint=\"eu-west-1\"\n",
    "```\n",
    "\n",
    "The `LocationConstraint` argument should specify the region where you want to create the bucket. The example above creates the bucket in the `eu-west-1` region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "806bcb29-2999-4126-90dd-0c929a68c746",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"Location\": \"/mlschool\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!aws s3api create-bucket --bucket $BUCKET"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "548c6d73-cadb-42b9-b126-e73c485ec58e",
   "metadata": {},
   "source": [
    "After we have a bucket, we can download the [Penguins dataset](https://www.kaggle.com/parulpandey/palmer-archipelago-antarctica-penguin-data) and store it in the bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "0b03948f-0eba-440f-b439-2b9aa2b87c80",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://mlschool/penguins/data/data.csv'"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import boto3\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import urllib.request\n",
    "from sagemaker.s3 import S3Uploader\n",
    "\n",
    "\n",
    "S3_LOCATION = f\"s3://{BUCKET}/penguins\"\n",
    "DATA_FILEPATH = CODE_FOLDER / \"data.csv\"\n",
    "\n",
    "urllib.request.urlretrieve(\n",
    "    \"https://storage.googleapis.com/download.tensorflow.org/data/palmer_penguins/penguins_size.csv\", \n",
    "    DATA_FILEPATH\n",
    ")\n",
    "\n",
    "S3Uploader.upload(local_path=str(DATA_FILEPATH), desired_s3_uri=f\"{S3_LOCATION}/data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a835695-557b-46d8-a901-a29bc57df5fe",
   "metadata": {},
   "source": [
    "Let's load the Penguins dataset. The data contains the following columns:\n",
    "\n",
    "1. `species`: The species of a penguin. This is the column we want to predict.\n",
    "2. `island`: The island where the penguin was found\n",
    "3. `culmen_length_mm`: The length of the penguin's culmen (bill) in millimeters\n",
    "4. `culmen_depth_mm`: The depth of the penguin's culmen in millimeters\n",
    "5. `flipper_length_mm`: The length of the penguin's flipper in millimeters\n",
    "6. `body_mass_g`: The body mass of the penguin in grams\n",
    "7. `sex`: The sex of the penguin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "f1cd2f0e-446d-48a9-a008-b4f1cc593bfc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>island</th>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <th>body_mass_g</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>39.1</td>\n",
       "      <td>18.7</td>\n",
       "      <td>181.0</td>\n",
       "      <td>3750.0</td>\n",
       "      <td>MALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>39.5</td>\n",
       "      <td>17.4</td>\n",
       "      <td>186.0</td>\n",
       "      <td>3800.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>40.3</td>\n",
       "      <td>18.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>3250.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Torgersen</td>\n",
       "      <td>36.7</td>\n",
       "      <td>19.3</td>\n",
       "      <td>193.0</td>\n",
       "      <td>3450.0</td>\n",
       "      <td>FEMALE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  species     island  culmen_length_mm  culmen_depth_mm  flipper_length_mm  \\\n",
       "0  Adelie  Torgersen              39.1             18.7              181.0   \n",
       "1  Adelie  Torgersen              39.5             17.4              186.0   \n",
       "2  Adelie  Torgersen              40.3             18.0              195.0   \n",
       "3  Adelie  Torgersen               NaN              NaN                NaN   \n",
       "4  Adelie  Torgersen              36.7             19.3              193.0   \n",
       "\n",
       "   body_mass_g     sex  \n",
       "0       3750.0    MALE  \n",
       "1       3800.0  FEMALE  \n",
       "2       3250.0  FEMALE  \n",
       "3          NaN     NaN  \n",
       "4       3450.0  FEMALE  "
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins = pd.read_csv(DATA_FILEPATH)\n",
    "penguins.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9eae10e-20c4-477e-b6b8-965c3a53566e",
   "metadata": {},
   "source": [
    "Now, let's get the summary statistics for the features in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "f2107c25-e730-4e22-a1b8-5bda53e61124",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>species</th>\n",
       "      <th>island</th>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <th>body_mass_g</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>344</td>\n",
       "      <td>344</td>\n",
       "      <td>342.000000</td>\n",
       "      <td>342.000000</td>\n",
       "      <td>342.000000</td>\n",
       "      <td>342.000000</td>\n",
       "      <td>334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>Adelie</td>\n",
       "      <td>Biscoe</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MALE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>152</td>\n",
       "      <td>168</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>43.921930</td>\n",
       "      <td>17.151170</td>\n",
       "      <td>200.915205</td>\n",
       "      <td>4201.754386</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.459584</td>\n",
       "      <td>1.974793</td>\n",
       "      <td>14.061714</td>\n",
       "      <td>801.954536</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32.100000</td>\n",
       "      <td>13.100000</td>\n",
       "      <td>172.000000</td>\n",
       "      <td>2700.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>39.225000</td>\n",
       "      <td>15.600000</td>\n",
       "      <td>190.000000</td>\n",
       "      <td>3550.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>44.450000</td>\n",
       "      <td>17.300000</td>\n",
       "      <td>197.000000</td>\n",
       "      <td>4050.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48.500000</td>\n",
       "      <td>18.700000</td>\n",
       "      <td>213.000000</td>\n",
       "      <td>4750.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>59.600000</td>\n",
       "      <td>21.500000</td>\n",
       "      <td>231.000000</td>\n",
       "      <td>6300.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       species  island  culmen_length_mm  culmen_depth_mm  flipper_length_mm  \\\n",
       "count      344     344        342.000000       342.000000         342.000000   \n",
       "unique       3       3               NaN              NaN                NaN   \n",
       "top     Adelie  Biscoe               NaN              NaN                NaN   \n",
       "freq       152     168               NaN              NaN                NaN   \n",
       "mean       NaN     NaN         43.921930        17.151170         200.915205   \n",
       "std        NaN     NaN          5.459584         1.974793          14.061714   \n",
       "min        NaN     NaN         32.100000        13.100000         172.000000   \n",
       "25%        NaN     NaN         39.225000        15.600000         190.000000   \n",
       "50%        NaN     NaN         44.450000        17.300000         197.000000   \n",
       "75%        NaN     NaN         48.500000        18.700000         213.000000   \n",
       "max        NaN     NaN         59.600000        21.500000         231.000000   \n",
       "\n",
       "        body_mass_g   sex  \n",
       "count    342.000000   334  \n",
       "unique          NaN     3  \n",
       "top             NaN  MALE  \n",
       "freq            NaN   168  \n",
       "mean    4201.754386   NaN  \n",
       "std      801.954536   NaN  \n",
       "min     2700.000000   NaN  \n",
       "25%     3550.000000   NaN  \n",
       "50%     4050.000000   NaN  \n",
       "75%     4750.000000   NaN  \n",
       "max     6300.000000   NaN  "
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins.describe(include='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2e19af7-9f0f-45fe-b7d3-f19721c02a2b",
   "metadata": {},
   "source": [
    "The distribution of the categories in our dataset are:\n",
    "\n",
    "- `species`: There are 3 species of penguins in the dataset: Adelie (152), Gentoo (124), and Chinstrap (68).\n",
    "- `island`: Penguins are from 3 islands: Biscoe (168), Dream (124), and Torgersen (52).\n",
    "- `sex`: We have 168 male penguins, 165 female penguins, and 1 penguin with an ambiguous gender ('.')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "1242122a-726e-4c37-a718-dd8e873d1612",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "species\n",
      "Adelie       152\n",
      "Gentoo       124\n",
      "Chinstrap     68\n",
      "Name: count, dtype: int64\n",
      "\n",
      "island\n",
      "Biscoe       168\n",
      "Dream        124\n",
      "Torgersen     52\n",
      "Name: count, dtype: int64\n",
      "\n",
      "sex\n",
      "MALE      168\n",
      "FEMALE    165\n",
      ".           1\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "species_distribution = penguins['species'].value_counts()\n",
    "island_distribution = penguins['island'].value_counts()\n",
    "sex_distribution = penguins['sex'].value_counts()\n",
    "\n",
    "print(species_distribution)\n",
    "print()\n",
    "print(island_distribution)\n",
    "print()\n",
    "print(sex_distribution)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9d98fdd-3b8c-40a2-b8dc-15162b4049e2",
   "metadata": {},
   "source": [
    "Let's replace the ambiguous value in the `sex` column with a null value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "cf1cf582-8831-4f83-bb17-2175afb193e8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sex\n",
       "MALE      168\n",
       "FEMALE    165\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins[\"sex\"] = penguins[\"sex\"].replace(\".\", np.nan)\n",
    "penguins[\"sex\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8425ce-ce4e-43e6-9ed8-0398b780cc66",
   "metadata": {},
   "source": [
    "Next, let's check for any missing values in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "cc42cb08-275c-4b05-9d2b-77052da2f336",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "species               0\n",
       "island                0\n",
       "culmen_length_mm      2\n",
       "culmen_depth_mm       2\n",
       "flipper_length_mm     2\n",
       "body_mass_g           2\n",
       "sex                  11\n",
       "dtype: int64"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b65207c-3e66-453a-87a1-751636c979ee",
   "metadata": {},
   "source": [
    "Let's get rid of the missing values. For now, we are going to replace the missing values with the most frequent value in the column. Later, we'll use a different strategy to replace missing numeric values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "3c57d55d-afd6-467a-a7a8-ff04132770ed",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "species              0\n",
       "island               0\n",
       "culmen_length_mm     0\n",
       "culmen_depth_mm      0\n",
       "flipper_length_mm    0\n",
       "body_mass_g          0\n",
       "sex                  0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "imputer = SimpleImputer(strategy=\"most_frequent\")\n",
    "penguins.iloc[:,:] = imputer.fit_transform(penguins)\n",
    "penguins.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5758214f-a4ab-4980-8892-91ec8d218ef3",
   "metadata": {},
   "source": [
    "Let's visualize the distribution of categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "60353941-e991-4d4b-b09a-63b42d107273",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "name": "species",
         "type": "bar",
         "x": [
          "Adelie",
          "Gentoo",
          "Chinstrap"
         ],
         "xaxis": "x",
         "y": [
          152,
          124,
          68
         ],
         "yaxis": "y"
        },
        {
         "name": "island",
         "type": "bar",
         "x": [
          "Biscoe",
          "Dream",
          "Torgersen"
         ],
         "xaxis": "x2",
         "y": [
          168,
          124,
          52
         ],
         "yaxis": "y2"
        },
        {
         "name": "sex",
         "type": "bar",
         "x": [
          "MALE",
          "FEMALE",
          "."
         ],
         "xaxis": "x3",
         "y": [
          168,
          165,
          1
         ],
         "yaxis": "y3"
        }
       ],
       "layout": {
        "height": 700,
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Distribution of Categorical Features"
        },
        "width": 960,
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ]
        },
        "xaxis2": {
         "anchor": "y2",
         "domain": [
          0,
          1
         ]
        },
        "xaxis3": {
         "anchor": "y3",
         "domain": [
          0,
          1
         ]
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0.7333333333333333,
          1
         ]
        },
        "yaxis2": {
         "anchor": "x2",
         "domain": [
          0.36666666666666664,
          0.6333333333333333
         ]
        },
        "yaxis3": {
         "anchor": "x3",
         "domain": [
          0,
          0.26666666666666666
         ]
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import plotly.express as px\n",
    "import plotly.graph_objs as go\n",
    "from plotly.subplots import make_subplots\n",
    "\n",
    "\n",
    "fig = make_subplots(rows=3, cols=1)\n",
    "\n",
    "fig.add_trace(go.Bar(x=species_distribution.index, y=species_distribution.values, name='species'), row=1, col=1)\n",
    "fig.add_trace(go.Bar(x=island_distribution.index, y=island_distribution.values, name='island'), row=2, col=1)\n",
    "fig.add_trace(go.Bar(x=sex_distribution.index, y=sex_distribution.values, name='sex'), row=3, col=1)\n",
    "\n",
    "fig.update_layout(height=700, width=960, title_text=\"Distribution of Categorical Features\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b04c8fae-35b4-4d8e-8fff-decee050af3a",
   "metadata": {},
   "source": [
    "Let's visualize the distribution of numerical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "ca68205b-e516-49d7-9b46-f77e25fd0698",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "name": "culmen_length_mm",
         "nbinsx": 20,
         "type": "histogram",
         "x": [
          39.1,
          39.5,
          40.3,
          41.1,
          36.7,
          39.3,
          38.9,
          39.2,
          34.1,
          42,
          37.8,
          37.8,
          41.1,
          38.6,
          34.6,
          36.6,
          38.7,
          42.5,
          34.4,
          46,
          37.8,
          37.7,
          35.9,
          38.2,
          38.8,
          35.3,
          40.6,
          40.5,
          37.9,
          40.5,
          39.5,
          37.2,
          39.5,
          40.9,
          36.4,
          39.2,
          38.8,
          42.2,
          37.6,
          39.8,
          36.5,
          40.8,
          36,
          44.1,
          37,
          39.6,
          41.1,
          37.5,
          36,
          42.3,
          39.6,
          40.1,
          35,
          42,
          34.5,
          41.4,
          39,
          40.6,
          36.5,
          37.6,
          35.7,
          41.3,
          37.6,
          41.1,
          36.4,
          41.6,
          35.5,
          41.1,
          35.9,
          41.8,
          33.5,
          39.7,
          39.6,
          45.8,
          35.5,
          42.8,
          40.9,
          37.2,
          36.2,
          42.1,
          34.6,
          42.9,
          36.7,
          35.1,
          37.3,
          41.3,
          36.3,
          36.9,
          38.3,
          38.9,
          35.7,
          41.1,
          34,
          39.6,
          36.2,
          40.8,
          38.1,
          40.3,
          33.1,
          43.2,
          35,
          41,
          37.7,
          37.8,
          37.9,
          39.7,
          38.6,
          38.2,
          38.1,
          43.2,
          38.1,
          45.6,
          39.7,
          42.2,
          39.6,
          42.7,
          38.6,
          37.3,
          35.7,
          41.1,
          36.2,
          37.7,
          40.2,
          41.4,
          35.2,
          40.6,
          38.8,
          41.5,
          39,
          44.1,
          38.5,
          43.1,
          36.8,
          37.5,
          38.1,
          41.1,
          35.6,
          40.2,
          37,
          39.7,
          40.2,
          40.6,
          32.1,
          40.7,
          37.3,
          39,
          39.2,
          36.6,
          36,
          37.8,
          36,
          41.5,
          46.5,
          50,
          51.3,
          45.4,
          52.7,
          45.2,
          46.1,
          51.3,
          46,
          51.3,
          46.6,
          51.7,
          47,
          52,
          45.9,
          50.5,
          50.3,
          58,
          46.4,
          49.2,
          42.4,
          48.5,
          43.2,
          50.6,
          46.7,
          52,
          50.5,
          49.5,
          46.4,
          52.8,
          40.9,
          54.2,
          42.5,
          51,
          49.7,
          47.5,
          47.6,
          52,
          46.9,
          53.5,
          49,
          46.2,
          50.9,
          45.5,
          50.9,
          50.8,
          50.1,
          49,
          51.5,
          49.8,
          48.1,
          51.4,
          45.7,
          50.7,
          42.5,
          52.2,
          45.2,
          49.3,
          50.2,
          45.6,
          51.9,
          46.8,
          45.7,
          55.8,
          43.5,
          49.6,
          50.8,
          50.2,
          46.1,
          50,
          48.7,
          50,
          47.6,
          46.5,
          45.4,
          46.7,
          43.3,
          46.8,
          40.9,
          49,
          45.5,
          48.4,
          45.8,
          49.3,
          42,
          49.2,
          46.2,
          48.7,
          50.2,
          45.1,
          46.5,
          46.3,
          42.9,
          46.1,
          44.5,
          47.8,
          48.2,
          50,
          47.3,
          42.8,
          45.1,
          59.6,
          49.1,
          48.4,
          42.6,
          44.4,
          44,
          48.7,
          42.7,
          49.6,
          45.3,
          49.6,
          50.5,
          43.6,
          45.5,
          50.5,
          44.9,
          45.2,
          46.6,
          48.5,
          45.1,
          50.1,
          46.5,
          45,
          43.8,
          45.5,
          43.2,
          50.4,
          45.3,
          46.2,
          45.7,
          54.3,
          45.8,
          49.8,
          46.2,
          49.5,
          43.5,
          50.7,
          47.7,
          46.4,
          48.2,
          46.5,
          46.4,
          48.6,
          47.5,
          51.1,
          45.2,
          45.2,
          49.1,
          52.5,
          47.4,
          50,
          44.9,
          50.8,
          43.4,
          51.3,
          47.5,
          52.1,
          47.5,
          52.2,
          45.5,
          49.5,
          44.5,
          50.8,
          49.4,
          46.9,
          48.4,
          51.1,
          48.5,
          55.9,
          47.2,
          49.1,
          47.3,
          46.8,
          41.7,
          53.4,
          43.3,
          48.1,
          50.5,
          49.8,
          43.5,
          51.5,
          46.2,
          55.1,
          44.5,
          48.8,
          47.2,
          41.1,
          46.8,
          50.4,
          45.2,
          49.9
         ],
         "xaxis": "x",
         "yaxis": "y"
        },
        {
         "name": "culmen_depth_mm",
         "nbinsx": 20,
         "type": "histogram",
         "x": [
          18.7,
          17.4,
          18,
          17,
          19.3,
          20.6,
          17.8,
          19.6,
          18.1,
          20.2,
          17.1,
          17.3,
          17.6,
          21.2,
          21.1,
          17.8,
          19,
          20.7,
          18.4,
          21.5,
          18.3,
          18.7,
          19.2,
          18.1,
          17.2,
          18.9,
          18.6,
          17.9,
          18.6,
          18.9,
          16.7,
          18.1,
          17.8,
          18.9,
          17,
          21.1,
          20,
          18.5,
          19.3,
          19.1,
          18,
          18.4,
          18.5,
          19.7,
          16.9,
          18.8,
          19,
          18.9,
          17.9,
          21.2,
          17.7,
          18.9,
          17.9,
          19.5,
          18.1,
          18.6,
          17.5,
          18.8,
          16.6,
          19.1,
          16.9,
          21.1,
          17,
          18.2,
          17.1,
          18,
          16.2,
          19.1,
          16.6,
          19.4,
          19,
          18.4,
          17.2,
          18.9,
          17.5,
          18.5,
          16.8,
          19.4,
          16.1,
          19.1,
          17.2,
          17.6,
          18.8,
          19.4,
          17.8,
          20.3,
          19.5,
          18.6,
          19.2,
          18.8,
          18,
          18.1,
          17.1,
          18.1,
          17.3,
          18.9,
          18.6,
          18.5,
          16.1,
          18.5,
          17.9,
          20,
          16,
          20,
          18.6,
          18.9,
          17.2,
          20,
          17,
          19,
          16.5,
          20.3,
          17.7,
          19.5,
          20.7,
          18.3,
          17,
          20.5,
          17,
          18.6,
          17.2,
          19.8,
          17,
          18.5,
          15.9,
          19,
          17.6,
          18.3,
          17.1,
          18,
          17.9,
          19.2,
          18.5,
          18.5,
          17.6,
          17.5,
          17.5,
          20.1,
          16.5,
          17.9,
          17.1,
          17.2,
          15.5,
          17,
          16.8,
          18.7,
          18.6,
          18.4,
          17.8,
          18.1,
          17.1,
          18.5,
          17.9,
          19.5,
          19.2,
          18.7,
          19.8,
          17.8,
          18.2,
          18.2,
          18.9,
          19.9,
          17.8,
          20.3,
          17.3,
          18.1,
          17.1,
          19.6,
          20,
          17.8,
          18.6,
          18.2,
          17.3,
          17.5,
          16.6,
          19.4,
          17.9,
          19,
          18.4,
          19,
          17.8,
          20,
          16.6,
          20.8,
          16.7,
          18.8,
          18.6,
          16.8,
          18.3,
          20.7,
          16.6,
          19.9,
          19.5,
          17.5,
          19.1,
          17,
          17.9,
          18.5,
          17.9,
          19.6,
          18.7,
          17.3,
          16.4,
          19,
          17.3,
          19.7,
          17.3,
          18.8,
          16.6,
          19.9,
          18.8,
          19.4,
          19.5,
          16.5,
          17,
          19.8,
          18.1,
          18.2,
          19,
          18.7,
          13.2,
          16.3,
          14.1,
          15.2,
          14.5,
          13.5,
          14.6,
          15.3,
          13.4,
          15.4,
          13.7,
          16.1,
          13.7,
          14.6,
          14.6,
          15.7,
          13.5,
          15.2,
          14.5,
          15.1,
          14.3,
          14.5,
          14.5,
          15.8,
          13.1,
          15.1,
          14.3,
          15,
          14.3,
          15.3,
          15.3,
          14.2,
          14.5,
          17,
          14.8,
          16.3,
          13.7,
          17.3,
          13.6,
          15.7,
          13.7,
          16,
          13.7,
          15,
          15.9,
          13.9,
          13.9,
          15.9,
          13.3,
          15.8,
          14.2,
          14.1,
          14.4,
          15,
          14.4,
          15.4,
          13.9,
          15,
          14.5,
          15.3,
          13.8,
          14.9,
          13.9,
          15.7,
          14.2,
          16.8,
          14.4,
          16.2,
          14.2,
          15,
          15,
          15.6,
          15.6,
          14.8,
          15,
          16,
          14.2,
          16.3,
          13.8,
          16.4,
          14.5,
          15.6,
          14.6,
          15.9,
          13.8,
          17.3,
          14.4,
          14.2,
          14,
          17,
          15,
          17.1,
          14.5,
          16.1,
          14.7,
          15.7,
          15.8,
          14.6,
          14.4,
          16.5,
          15,
          17,
          15.5,
          15,
          13.8,
          16.1,
          14.7,
          15.8,
          14,
          15.1,
          15.2,
          15.9,
          15.2,
          16.3,
          14.1,
          16,
          15.7,
          16.2,
          13.7,
          17,
          14.3,
          15.7,
          14.8,
          16.1
         ],
         "xaxis": "x2",
         "yaxis": "y2"
        },
        {
         "name": "flipper_length_mm",
         "nbinsx": 20,
         "type": "histogram",
         "x": [
          181,
          186,
          195,
          190,
          193,
          190,
          181,
          195,
          193,
          190,
          186,
          180,
          182,
          191,
          198,
          185,
          195,
          197,
          184,
          194,
          174,
          180,
          189,
          185,
          180,
          187,
          183,
          187,
          172,
          180,
          178,
          178,
          188,
          184,
          195,
          196,
          190,
          180,
          181,
          184,
          182,
          195,
          186,
          196,
          185,
          190,
          182,
          179,
          190,
          191,
          186,
          188,
          190,
          200,
          187,
          191,
          186,
          193,
          181,
          194,
          185,
          195,
          185,
          192,
          184,
          192,
          195,
          188,
          190,
          198,
          190,
          190,
          196,
          197,
          190,
          195,
          191,
          184,
          187,
          195,
          189,
          196,
          187,
          193,
          191,
          194,
          190,
          189,
          189,
          190,
          202,
          205,
          185,
          186,
          187,
          208,
          190,
          196,
          178,
          192,
          192,
          203,
          183,
          190,
          193,
          184,
          199,
          190,
          181,
          197,
          198,
          191,
          193,
          197,
          191,
          196,
          188,
          199,
          189,
          189,
          187,
          198,
          176,
          202,
          186,
          199,
          191,
          195,
          191,
          210,
          190,
          197,
          193,
          199,
          187,
          190,
          191,
          200,
          185,
          193,
          193,
          187,
          188,
          190,
          192,
          185,
          190,
          184,
          195,
          193,
          187,
          201,
          192,
          196,
          193,
          188,
          197,
          198,
          178,
          197,
          195,
          198,
          193,
          194,
          185,
          201,
          190,
          201,
          197,
          181,
          190,
          195,
          181,
          191,
          187,
          193,
          195,
          197,
          200,
          200,
          191,
          205,
          187,
          201,
          187,
          203,
          195,
          199,
          195,
          210,
          192,
          205,
          210,
          187,
          196,
          196,
          196,
          201,
          190,
          212,
          187,
          198,
          199,
          201,
          193,
          203,
          187,
          197,
          191,
          203,
          202,
          194,
          206,
          189,
          195,
          207,
          202,
          193,
          210,
          198,
          211,
          230,
          210,
          218,
          215,
          210,
          211,
          219,
          209,
          215,
          214,
          216,
          214,
          213,
          210,
          217,
          210,
          221,
          209,
          222,
          218,
          215,
          213,
          215,
          215,
          215,
          216,
          215,
          210,
          220,
          222,
          209,
          207,
          230,
          220,
          220,
          213,
          219,
          208,
          208,
          208,
          225,
          210,
          216,
          222,
          217,
          210,
          225,
          213,
          215,
          210,
          220,
          210,
          225,
          217,
          220,
          208,
          220,
          208,
          224,
          208,
          221,
          214,
          231,
          219,
          230,
          214,
          229,
          220,
          223,
          216,
          221,
          221,
          217,
          216,
          230,
          209,
          220,
          215,
          223,
          212,
          221,
          212,
          224,
          212,
          228,
          218,
          218,
          212,
          230,
          218,
          228,
          212,
          224,
          214,
          226,
          216,
          222,
          203,
          225,
          219,
          228,
          215,
          228,
          216,
          215,
          210,
          219,
          208,
          209,
          216,
          229,
          213,
          230,
          217,
          230,
          217,
          222,
          214,
          190,
          215,
          222,
          212,
          213
         ],
         "xaxis": "x3",
         "yaxis": "y3"
        },
        {
         "name": "body_mass_g",
         "nbinsx": 20,
         "type": "histogram",
         "x": [
          3750,
          3800,
          3250,
          3800,
          3450,
          3650,
          3625,
          4675,
          3475,
          4250,
          3300,
          3700,
          3200,
          3800,
          4400,
          3700,
          3450,
          4500,
          3325,
          4200,
          3400,
          3600,
          3800,
          3950,
          3800,
          3800,
          3550,
          3200,
          3150,
          3950,
          3250,
          3900,
          3300,
          3900,
          3325,
          4150,
          3950,
          3550,
          3300,
          4650,
          3150,
          3900,
          3100,
          4400,
          3000,
          4600,
          3425,
          2975,
          3450,
          4150,
          3500,
          4300,
          3450,
          4050,
          2900,
          3700,
          3550,
          3800,
          2850,
          3750,
          3150,
          4400,
          3600,
          4050,
          2850,
          3950,
          3350,
          4100,
          3050,
          4450,
          3600,
          3900,
          3550,
          4150,
          3700,
          4250,
          3700,
          3900,
          3550,
          4000,
          3200,
          4700,
          3800,
          4200,
          3350,
          3550,
          3800,
          3500,
          3950,
          3600,
          3550,
          4300,
          3400,
          4450,
          3300,
          4300,
          3700,
          4350,
          2900,
          4100,
          3725,
          4725,
          3075,
          4250,
          2925,
          3550,
          3750,
          3900,
          3175,
          4775,
          3825,
          4600,
          3200,
          4275,
          3900,
          4075,
          2900,
          3775,
          3350,
          3325,
          3150,
          3500,
          3450,
          3875,
          3050,
          4000,
          3275,
          4300,
          3050,
          4000,
          3325,
          3500,
          3500,
          4475,
          3425,
          3900,
          3175,
          3975,
          3400,
          4250,
          3400,
          3475,
          3050,
          3725,
          3000,
          3650,
          4250,
          3475,
          3450,
          3750,
          3700,
          4000,
          3500,
          3900,
          3650,
          3525,
          3725,
          3950,
          3250,
          3750,
          4150,
          3700,
          3800,
          3775,
          3700,
          4050,
          3575,
          4050,
          3300,
          3700,
          3450,
          4400,
          3600,
          3400,
          2900,
          3800,
          3300,
          4150,
          3400,
          3800,
          3700,
          4550,
          3200,
          4300,
          3350,
          4100,
          3600,
          3900,
          3850,
          4800,
          2700,
          4500,
          3950,
          3650,
          3550,
          3500,
          3675,
          4450,
          3400,
          4300,
          3250,
          3675,
          3325,
          3950,
          3600,
          4050,
          3350,
          3450,
          3250,
          4050,
          3800,
          3525,
          3950,
          3650,
          3650,
          4000,
          3400,
          3775,
          4100,
          3775,
          4500,
          5700,
          4450,
          5700,
          5400,
          4550,
          4800,
          5200,
          4400,
          5150,
          4650,
          5550,
          4650,
          5850,
          4200,
          5850,
          4150,
          6300,
          4800,
          5350,
          5700,
          5000,
          4400,
          5050,
          5000,
          5100,
          4100,
          5650,
          4600,
          5550,
          5250,
          4700,
          5050,
          6050,
          5150,
          5400,
          4950,
          5250,
          4350,
          5350,
          3950,
          5700,
          4300,
          4750,
          5550,
          4900,
          4200,
          5400,
          5100,
          5300,
          4850,
          5300,
          4400,
          5000,
          4900,
          5050,
          4300,
          5000,
          4450,
          5550,
          4200,
          5300,
          4400,
          5650,
          4700,
          5700,
          4650,
          5800,
          4700,
          5550,
          4750,
          5000,
          5100,
          5200,
          4700,
          5800,
          4600,
          6000,
          4750,
          5950,
          4625,
          5450,
          4725,
          5350,
          4750,
          5600,
          4600,
          5300,
          4875,
          5550,
          4950,
          5400,
          4750,
          5650,
          4850,
          5200,
          4925,
          4875,
          4625,
          5250,
          4850,
          5600,
          4975,
          5500,
          4725,
          5500,
          4700,
          5500,
          4575,
          5500,
          5000,
          5950,
          4650,
          5500,
          4375,
          5850,
          4875,
          6000,
          4925,
          3800,
          4850,
          5750,
          5200,
          5400
         ],
         "xaxis": "x4",
         "yaxis": "y4"
        }
       ],
       "layout": {
        "height": 700,
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Distribution of Numerical Features"
        },
        "width": 960,
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          0.45
         ]
        },
        "xaxis2": {
         "anchor": "y2",
         "domain": [
          0.55,
          1
         ]
        },
        "xaxis3": {
         "anchor": "y3",
         "domain": [
          0,
          0.45
         ]
        },
        "xaxis4": {
         "anchor": "y4",
         "domain": [
          0.55,
          1
         ]
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0.575,
          1
         ]
        },
        "yaxis2": {
         "anchor": "x2",
         "domain": [
          0.575,
          1
         ]
        },
        "yaxis3": {
         "anchor": "x3",
         "domain": [
          0,
          0.425
         ]
        },
        "yaxis4": {
         "anchor": "x4",
         "domain": [
          0,
          0.425
         ]
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = make_subplots(rows=2, cols=2)\n",
    "\n",
    "fig.add_trace(go.Histogram(x=penguins['culmen_length_mm'], name='culmen_length_mm', nbinsx=20), row=1, col=1)\n",
    "fig.add_trace(go.Histogram(x=penguins['culmen_depth_mm'], name='culmen_depth_mm', nbinsx=20), row=1, col=2)\n",
    "fig.add_trace(go.Histogram(x=penguins['flipper_length_mm'], name='flipper_length_mm', nbinsx=20), row=2, col=1)\n",
    "fig.add_trace(go.Histogram(x=penguins['body_mass_g'], name='body_mass_g', nbinsx=20), row=2, col=2)\n",
    "fig.update_layout(height=700, width=960, title_text=\"Distribution of Numerical Features\")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbf14768-888a-4478-88fa-c594fbc88ea5",
   "metadata": {},
   "source": [
    "Let's display a scatter matrix with every numeric feature from our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "62b728a4-95d5-40a4-8749-d061a892cfe4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "dimensions": [
          {
           "axis": {
            "matches": true
           },
           "label": "culmen_length_mm",
           "values": [
            39.1,
            39.5,
            40.3,
            41.1,
            36.7,
            39.3,
            38.9,
            39.2,
            34.1,
            42,
            37.8,
            37.8,
            41.1,
            38.6,
            34.6,
            36.6,
            38.7,
            42.5,
            34.4,
            46,
            37.8,
            37.7,
            35.9,
            38.2,
            38.8,
            35.3,
            40.6,
            40.5,
            37.9,
            40.5,
            39.5,
            37.2,
            39.5,
            40.9,
            36.4,
            39.2,
            38.8,
            42.2,
            37.6,
            39.8,
            36.5,
            40.8,
            36,
            44.1,
            37,
            39.6,
            41.1,
            37.5,
            36,
            42.3,
            39.6,
            40.1,
            35,
            42,
            34.5,
            41.4,
            39,
            40.6,
            36.5,
            37.6,
            35.7,
            41.3,
            37.6,
            41.1,
            36.4,
            41.6,
            35.5,
            41.1,
            35.9,
            41.8,
            33.5,
            39.7,
            39.6,
            45.8,
            35.5,
            42.8,
            40.9,
            37.2,
            36.2,
            42.1,
            34.6,
            42.9,
            36.7,
            35.1,
            37.3,
            41.3,
            36.3,
            36.9,
            38.3,
            38.9,
            35.7,
            41.1,
            34,
            39.6,
            36.2,
            40.8,
            38.1,
            40.3,
            33.1,
            43.2,
            35,
            41,
            37.7,
            37.8,
            37.9,
            39.7,
            38.6,
            38.2,
            38.1,
            43.2,
            38.1,
            45.6,
            39.7,
            42.2,
            39.6,
            42.7,
            38.6,
            37.3,
            35.7,
            41.1,
            36.2,
            37.7,
            40.2,
            41.4,
            35.2,
            40.6,
            38.8,
            41.5,
            39,
            44.1,
            38.5,
            43.1,
            36.8,
            37.5,
            38.1,
            41.1,
            35.6,
            40.2,
            37,
            39.7,
            40.2,
            40.6,
            32.1,
            40.7,
            37.3,
            39,
            39.2,
            36.6,
            36,
            37.8,
            36,
            41.5
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "culmen_depth_mm",
           "values": [
            18.7,
            17.4,
            18,
            17,
            19.3,
            20.6,
            17.8,
            19.6,
            18.1,
            20.2,
            17.1,
            17.3,
            17.6,
            21.2,
            21.1,
            17.8,
            19,
            20.7,
            18.4,
            21.5,
            18.3,
            18.7,
            19.2,
            18.1,
            17.2,
            18.9,
            18.6,
            17.9,
            18.6,
            18.9,
            16.7,
            18.1,
            17.8,
            18.9,
            17,
            21.1,
            20,
            18.5,
            19.3,
            19.1,
            18,
            18.4,
            18.5,
            19.7,
            16.9,
            18.8,
            19,
            18.9,
            17.9,
            21.2,
            17.7,
            18.9,
            17.9,
            19.5,
            18.1,
            18.6,
            17.5,
            18.8,
            16.6,
            19.1,
            16.9,
            21.1,
            17,
            18.2,
            17.1,
            18,
            16.2,
            19.1,
            16.6,
            19.4,
            19,
            18.4,
            17.2,
            18.9,
            17.5,
            18.5,
            16.8,
            19.4,
            16.1,
            19.1,
            17.2,
            17.6,
            18.8,
            19.4,
            17.8,
            20.3,
            19.5,
            18.6,
            19.2,
            18.8,
            18,
            18.1,
            17.1,
            18.1,
            17.3,
            18.9,
            18.6,
            18.5,
            16.1,
            18.5,
            17.9,
            20,
            16,
            20,
            18.6,
            18.9,
            17.2,
            20,
            17,
            19,
            16.5,
            20.3,
            17.7,
            19.5,
            20.7,
            18.3,
            17,
            20.5,
            17,
            18.6,
            17.2,
            19.8,
            17,
            18.5,
            15.9,
            19,
            17.6,
            18.3,
            17.1,
            18,
            17.9,
            19.2,
            18.5,
            18.5,
            17.6,
            17.5,
            17.5,
            20.1,
            16.5,
            17.9,
            17.1,
            17.2,
            15.5,
            17,
            16.8,
            18.7,
            18.6,
            18.4,
            17.8,
            18.1,
            17.1,
            18.5
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "flipper_length_mm",
           "values": [
            181,
            186,
            195,
            190,
            193,
            190,
            181,
            195,
            193,
            190,
            186,
            180,
            182,
            191,
            198,
            185,
            195,
            197,
            184,
            194,
            174,
            180,
            189,
            185,
            180,
            187,
            183,
            187,
            172,
            180,
            178,
            178,
            188,
            184,
            195,
            196,
            190,
            180,
            181,
            184,
            182,
            195,
            186,
            196,
            185,
            190,
            182,
            179,
            190,
            191,
            186,
            188,
            190,
            200,
            187,
            191,
            186,
            193,
            181,
            194,
            185,
            195,
            185,
            192,
            184,
            192,
            195,
            188,
            190,
            198,
            190,
            190,
            196,
            197,
            190,
            195,
            191,
            184,
            187,
            195,
            189,
            196,
            187,
            193,
            191,
            194,
            190,
            189,
            189,
            190,
            202,
            205,
            185,
            186,
            187,
            208,
            190,
            196,
            178,
            192,
            192,
            203,
            183,
            190,
            193,
            184,
            199,
            190,
            181,
            197,
            198,
            191,
            193,
            197,
            191,
            196,
            188,
            199,
            189,
            189,
            187,
            198,
            176,
            202,
            186,
            199,
            191,
            195,
            191,
            210,
            190,
            197,
            193,
            199,
            187,
            190,
            191,
            200,
            185,
            193,
            193,
            187,
            188,
            190,
            192,
            185,
            190,
            184,
            195,
            193,
            187,
            201
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "body_mass_g",
           "values": [
            3750,
            3800,
            3250,
            3800,
            3450,
            3650,
            3625,
            4675,
            3475,
            4250,
            3300,
            3700,
            3200,
            3800,
            4400,
            3700,
            3450,
            4500,
            3325,
            4200,
            3400,
            3600,
            3800,
            3950,
            3800,
            3800,
            3550,
            3200,
            3150,
            3950,
            3250,
            3900,
            3300,
            3900,
            3325,
            4150,
            3950,
            3550,
            3300,
            4650,
            3150,
            3900,
            3100,
            4400,
            3000,
            4600,
            3425,
            2975,
            3450,
            4150,
            3500,
            4300,
            3450,
            4050,
            2900,
            3700,
            3550,
            3800,
            2850,
            3750,
            3150,
            4400,
            3600,
            4050,
            2850,
            3950,
            3350,
            4100,
            3050,
            4450,
            3600,
            3900,
            3550,
            4150,
            3700,
            4250,
            3700,
            3900,
            3550,
            4000,
            3200,
            4700,
            3800,
            4200,
            3350,
            3550,
            3800,
            3500,
            3950,
            3600,
            3550,
            4300,
            3400,
            4450,
            3300,
            4300,
            3700,
            4350,
            2900,
            4100,
            3725,
            4725,
            3075,
            4250,
            2925,
            3550,
            3750,
            3900,
            3175,
            4775,
            3825,
            4600,
            3200,
            4275,
            3900,
            4075,
            2900,
            3775,
            3350,
            3325,
            3150,
            3500,
            3450,
            3875,
            3050,
            4000,
            3275,
            4300,
            3050,
            4000,
            3325,
            3500,
            3500,
            4475,
            3425,
            3900,
            3175,
            3975,
            3400,
            4250,
            3400,
            3475,
            3050,
            3725,
            3000,
            3650,
            4250,
            3475,
            3450,
            3750,
            3700,
            4000
           ]
          }
         ],
         "hovertemplate": "species=Adelie<br>%{xaxis.title.text}=%{x}<br>%{yaxis.title.text}=%{y}<extra></extra>",
         "legendgroup": "Adelie",
         "marker": {
          "color": "#636efa",
          "symbol": "circle"
         },
         "name": "Adelie",
         "showlegend": true,
         "type": "splom"
        },
        {
         "dimensions": [
          {
           "axis": {
            "matches": true
           },
           "label": "culmen_length_mm",
           "values": [
            46.5,
            50,
            51.3,
            45.4,
            52.7,
            45.2,
            46.1,
            51.3,
            46,
            51.3,
            46.6,
            51.7,
            47,
            52,
            45.9,
            50.5,
            50.3,
            58,
            46.4,
            49.2,
            42.4,
            48.5,
            43.2,
            50.6,
            46.7,
            52,
            50.5,
            49.5,
            46.4,
            52.8,
            40.9,
            54.2,
            42.5,
            51,
            49.7,
            47.5,
            47.6,
            52,
            46.9,
            53.5,
            49,
            46.2,
            50.9,
            45.5,
            50.9,
            50.8,
            50.1,
            49,
            51.5,
            49.8,
            48.1,
            51.4,
            45.7,
            50.7,
            42.5,
            52.2,
            45.2,
            49.3,
            50.2,
            45.6,
            51.9,
            46.8,
            45.7,
            55.8,
            43.5,
            49.6,
            50.8,
            50.2
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "culmen_depth_mm",
           "values": [
            17.9,
            19.5,
            19.2,
            18.7,
            19.8,
            17.8,
            18.2,
            18.2,
            18.9,
            19.9,
            17.8,
            20.3,
            17.3,
            18.1,
            17.1,
            19.6,
            20,
            17.8,
            18.6,
            18.2,
            17.3,
            17.5,
            16.6,
            19.4,
            17.9,
            19,
            18.4,
            19,
            17.8,
            20,
            16.6,
            20.8,
            16.7,
            18.8,
            18.6,
            16.8,
            18.3,
            20.7,
            16.6,
            19.9,
            19.5,
            17.5,
            19.1,
            17,
            17.9,
            18.5,
            17.9,
            19.6,
            18.7,
            17.3,
            16.4,
            19,
            17.3,
            19.7,
            17.3,
            18.8,
            16.6,
            19.9,
            18.8,
            19.4,
            19.5,
            16.5,
            17,
            19.8,
            18.1,
            18.2,
            19,
            18.7
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "flipper_length_mm",
           "values": [
            192,
            196,
            193,
            188,
            197,
            198,
            178,
            197,
            195,
            198,
            193,
            194,
            185,
            201,
            190,
            201,
            197,
            181,
            190,
            195,
            181,
            191,
            187,
            193,
            195,
            197,
            200,
            200,
            191,
            205,
            187,
            201,
            187,
            203,
            195,
            199,
            195,
            210,
            192,
            205,
            210,
            187,
            196,
            196,
            196,
            201,
            190,
            212,
            187,
            198,
            199,
            201,
            193,
            203,
            187,
            197,
            191,
            203,
            202,
            194,
            206,
            189,
            195,
            207,
            202,
            193,
            210,
            198
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "body_mass_g",
           "values": [
            3500,
            3900,
            3650,
            3525,
            3725,
            3950,
            3250,
            3750,
            4150,
            3700,
            3800,
            3775,
            3700,
            4050,
            3575,
            4050,
            3300,
            3700,
            3450,
            4400,
            3600,
            3400,
            2900,
            3800,
            3300,
            4150,
            3400,
            3800,
            3700,
            4550,
            3200,
            4300,
            3350,
            4100,
            3600,
            3900,
            3850,
            4800,
            2700,
            4500,
            3950,
            3650,
            3550,
            3500,
            3675,
            4450,
            3400,
            4300,
            3250,
            3675,
            3325,
            3950,
            3600,
            4050,
            3350,
            3450,
            3250,
            4050,
            3800,
            3525,
            3950,
            3650,
            3650,
            4000,
            3400,
            3775,
            4100,
            3775
           ]
          }
         ],
         "hovertemplate": "species=Chinstrap<br>%{xaxis.title.text}=%{x}<br>%{yaxis.title.text}=%{y}<extra></extra>",
         "legendgroup": "Chinstrap",
         "marker": {
          "color": "#EF553B",
          "symbol": "circle"
         },
         "name": "Chinstrap",
         "showlegend": true,
         "type": "splom"
        },
        {
         "dimensions": [
          {
           "axis": {
            "matches": true
           },
           "label": "culmen_length_mm",
           "values": [
            46.1,
            50,
            48.7,
            50,
            47.6,
            46.5,
            45.4,
            46.7,
            43.3,
            46.8,
            40.9,
            49,
            45.5,
            48.4,
            45.8,
            49.3,
            42,
            49.2,
            46.2,
            48.7,
            50.2,
            45.1,
            46.5,
            46.3,
            42.9,
            46.1,
            44.5,
            47.8,
            48.2,
            50,
            47.3,
            42.8,
            45.1,
            59.6,
            49.1,
            48.4,
            42.6,
            44.4,
            44,
            48.7,
            42.7,
            49.6,
            45.3,
            49.6,
            50.5,
            43.6,
            45.5,
            50.5,
            44.9,
            45.2,
            46.6,
            48.5,
            45.1,
            50.1,
            46.5,
            45,
            43.8,
            45.5,
            43.2,
            50.4,
            45.3,
            46.2,
            45.7,
            54.3,
            45.8,
            49.8,
            46.2,
            49.5,
            43.5,
            50.7,
            47.7,
            46.4,
            48.2,
            46.5,
            46.4,
            48.6,
            47.5,
            51.1,
            45.2,
            45.2,
            49.1,
            52.5,
            47.4,
            50,
            44.9,
            50.8,
            43.4,
            51.3,
            47.5,
            52.1,
            47.5,
            52.2,
            45.5,
            49.5,
            44.5,
            50.8,
            49.4,
            46.9,
            48.4,
            51.1,
            48.5,
            55.9,
            47.2,
            49.1,
            47.3,
            46.8,
            41.7,
            53.4,
            43.3,
            48.1,
            50.5,
            49.8,
            43.5,
            51.5,
            46.2,
            55.1,
            44.5,
            48.8,
            47.2,
            41.1,
            46.8,
            50.4,
            45.2,
            49.9
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "culmen_depth_mm",
           "values": [
            13.2,
            16.3,
            14.1,
            15.2,
            14.5,
            13.5,
            14.6,
            15.3,
            13.4,
            15.4,
            13.7,
            16.1,
            13.7,
            14.6,
            14.6,
            15.7,
            13.5,
            15.2,
            14.5,
            15.1,
            14.3,
            14.5,
            14.5,
            15.8,
            13.1,
            15.1,
            14.3,
            15,
            14.3,
            15.3,
            15.3,
            14.2,
            14.5,
            17,
            14.8,
            16.3,
            13.7,
            17.3,
            13.6,
            15.7,
            13.7,
            16,
            13.7,
            15,
            15.9,
            13.9,
            13.9,
            15.9,
            13.3,
            15.8,
            14.2,
            14.1,
            14.4,
            15,
            14.4,
            15.4,
            13.9,
            15,
            14.5,
            15.3,
            13.8,
            14.9,
            13.9,
            15.7,
            14.2,
            16.8,
            14.4,
            16.2,
            14.2,
            15,
            15,
            15.6,
            15.6,
            14.8,
            15,
            16,
            14.2,
            16.3,
            13.8,
            16.4,
            14.5,
            15.6,
            14.6,
            15.9,
            13.8,
            17.3,
            14.4,
            14.2,
            14,
            17,
            15,
            17.1,
            14.5,
            16.1,
            14.7,
            15.7,
            15.8,
            14.6,
            14.4,
            16.5,
            15,
            17,
            15.5,
            15,
            13.8,
            16.1,
            14.7,
            15.8,
            14,
            15.1,
            15.2,
            15.9,
            15.2,
            16.3,
            14.1,
            16,
            15.7,
            16.2,
            13.7,
            17,
            14.3,
            15.7,
            14.8,
            16.1
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "flipper_length_mm",
           "values": [
            211,
            230,
            210,
            218,
            215,
            210,
            211,
            219,
            209,
            215,
            214,
            216,
            214,
            213,
            210,
            217,
            210,
            221,
            209,
            222,
            218,
            215,
            213,
            215,
            215,
            215,
            216,
            215,
            210,
            220,
            222,
            209,
            207,
            230,
            220,
            220,
            213,
            219,
            208,
            208,
            208,
            225,
            210,
            216,
            222,
            217,
            210,
            225,
            213,
            215,
            210,
            220,
            210,
            225,
            217,
            220,
            208,
            220,
            208,
            224,
            208,
            221,
            214,
            231,
            219,
            230,
            214,
            229,
            220,
            223,
            216,
            221,
            221,
            217,
            216,
            230,
            209,
            220,
            215,
            223,
            212,
            221,
            212,
            224,
            212,
            228,
            218,
            218,
            212,
            230,
            218,
            228,
            212,
            224,
            214,
            226,
            216,
            222,
            203,
            225,
            219,
            228,
            215,
            228,
            216,
            215,
            210,
            219,
            208,
            209,
            216,
            229,
            213,
            230,
            217,
            230,
            217,
            222,
            214,
            190,
            215,
            222,
            212,
            213
           ]
          },
          {
           "axis": {
            "matches": true
           },
           "label": "body_mass_g",
           "values": [
            4500,
            5700,
            4450,
            5700,
            5400,
            4550,
            4800,
            5200,
            4400,
            5150,
            4650,
            5550,
            4650,
            5850,
            4200,
            5850,
            4150,
            6300,
            4800,
            5350,
            5700,
            5000,
            4400,
            5050,
            5000,
            5100,
            4100,
            5650,
            4600,
            5550,
            5250,
            4700,
            5050,
            6050,
            5150,
            5400,
            4950,
            5250,
            4350,
            5350,
            3950,
            5700,
            4300,
            4750,
            5550,
            4900,
            4200,
            5400,
            5100,
            5300,
            4850,
            5300,
            4400,
            5000,
            4900,
            5050,
            4300,
            5000,
            4450,
            5550,
            4200,
            5300,
            4400,
            5650,
            4700,
            5700,
            4650,
            5800,
            4700,
            5550,
            4750,
            5000,
            5100,
            5200,
            4700,
            5800,
            4600,
            6000,
            4750,
            5950,
            4625,
            5450,
            4725,
            5350,
            4750,
            5600,
            4600,
            5300,
            4875,
            5550,
            4950,
            5400,
            4750,
            5650,
            4850,
            5200,
            4925,
            4875,
            4625,
            5250,
            4850,
            5600,
            4975,
            5500,
            4725,
            5500,
            4700,
            5500,
            4575,
            5500,
            5000,
            5950,
            4650,
            5500,
            4375,
            5850,
            4875,
            6000,
            4925,
            3800,
            4850,
            5750,
            5200,
            5400
           ]
          }
         ],
         "hovertemplate": "species=Gentoo<br>%{xaxis.title.text}=%{x}<br>%{yaxis.title.text}=%{y}<extra></extra>",
         "legendgroup": "Gentoo",
         "marker": {
          "color": "#00cc96",
          "symbol": "circle"
         },
         "name": "Gentoo",
         "showlegend": true,
         "type": "splom"
        }
       ],
       "layout": {
        "dragmode": "select",
        "height": 800,
        "legend": {
         "title": {
          "text": "species"
         },
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Scatter Matrix of Numeric Features"
        },
        "width": 960
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = px.scatter_matrix(\n",
    "    penguins, \n",
    "    dimensions=[\"culmen_length_mm\", \"culmen_depth_mm\", \"flipper_length_mm\", \"body_mass_g\"], \n",
    "    color=\"species\")\n",
    "\n",
    "fig.update_layout(height=800, width=960, title_text=\"Scatter Matrix of Numeric Features\")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef241df0-3acd-4401-a2c6-b70723d7595b",
   "metadata": {},
   "source": [
    "Let's display the covariance matrix of the dataset. The \"covariance\" measures how changes in one variable are associated with changes in a second variable. In other words, the covariance measures the degree to which two variables are linearly associated.\n",
    "\n",
    "Here are three examples of what we get from interpreting the covariance matrix below:\n",
    "\n",
    "1. Penguins that weight more tend to have a larger culmen.\n",
    "2. The more a penguin weights, the shallower its culmen tends to be.\n",
    "3. There's a small variance between the culmen depth of penguins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "3daf3ba1-d218-4ad4-b862-af679b91273f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <th>body_mass_g</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <td>29.679415</td>\n",
       "      <td>-2.516984</td>\n",
       "      <td>50.260588</td>\n",
       "      <td>2596.971151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <td>-2.516984</td>\n",
       "      <td>3.877201</td>\n",
       "      <td>-16.108849</td>\n",
       "      <td>-742.660180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <td>50.260588</td>\n",
       "      <td>-16.108849</td>\n",
       "      <td>197.269501</td>\n",
       "      <td>9792.552037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>body_mass_g</th>\n",
       "      <td>2596.971151</td>\n",
       "      <td>-742.660180</td>\n",
       "      <td>9792.552037</td>\n",
       "      <td>640316.716388</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   culmen_length_mm  culmen_depth_mm  flipper_length_mm  \\\n",
       "culmen_length_mm          29.679415        -2.516984          50.260588   \n",
       "culmen_depth_mm           -2.516984         3.877201         -16.108849   \n",
       "flipper_length_mm         50.260588       -16.108849         197.269501   \n",
       "body_mass_g             2596.971151      -742.660180        9792.552037   \n",
       "\n",
       "                     body_mass_g  \n",
       "culmen_length_mm     2596.971151  \n",
       "culmen_depth_mm      -742.660180  \n",
       "flipper_length_mm    9792.552037  \n",
       "body_mass_g        640316.716388  "
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins.cov(numeric_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fbbe6bc-0104-4663-8c30-8f9566755739",
   "metadata": {},
   "source": [
    "Let's now display the correlation matrix. \"Correlation\" measures both the strength and direction of the linear relationship between two variables.\n",
    "\n",
    "Here are three examples of what we get from interpreting the correlation matrix below:\n",
    "\n",
    "1. Penguins that weight more tend to have larger flippers.\n",
    "2. Penguins with a shallower culmen tend to have larger flippers.\n",
    "3. The length and depth of the culmen have a slight negative correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "1d793e09-2cb9-47ff-a0e6-199a0f4fc1b3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <th>body_mass_g</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>culmen_length_mm</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.234635</td>\n",
       "      <td>0.656856</td>\n",
       "      <td>0.595720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>culmen_depth_mm</th>\n",
       "      <td>-0.234635</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.582472</td>\n",
       "      <td>-0.471339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flipper_length_mm</th>\n",
       "      <td>0.656856</td>\n",
       "      <td>-0.582472</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.871302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>body_mass_g</th>\n",
       "      <td>0.595720</td>\n",
       "      <td>-0.471339</td>\n",
       "      <td>0.871302</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   culmen_length_mm  culmen_depth_mm  flipper_length_mm  \\\n",
       "culmen_length_mm           1.000000        -0.234635           0.656856   \n",
       "culmen_depth_mm           -0.234635         1.000000          -0.582472   \n",
       "flipper_length_mm          0.656856        -0.582472           1.000000   \n",
       "body_mass_g                0.595720        -0.471339           0.871302   \n",
       "\n",
       "                   body_mass_g  \n",
       "culmen_length_mm      0.595720  \n",
       "culmen_depth_mm      -0.471339  \n",
       "flipper_length_mm     0.871302  \n",
       "body_mass_g           1.000000  "
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "penguins.corr(numeric_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8aec4c08-767c-4740-959c-2d76268c3513",
   "metadata": {},
   "source": [
    "Let's display the distribution of species by island."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "fd3711a6-5386-41da-b0f6-4a3b7baef4f1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "bingroup": "x",
         "hovertemplate": "species=Adelie<br>island=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "Adelie",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "Adelie",
         "nbinsx": 50,
         "offsetgroup": "Adelie",
         "orientation": "v",
         "showlegend": true,
         "type": "histogram",
         "x": [
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Torgersen",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream"
         ],
         "xaxis": "x",
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "bingroup": "x",
         "hovertemplate": "species=Chinstrap<br>island=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "Chinstrap",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "Chinstrap",
         "nbinsx": 50,
         "offsetgroup": "Chinstrap",
         "orientation": "v",
         "showlegend": true,
         "type": "histogram",
         "x": [
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream",
          "Dream"
         ],
         "xaxis": "x",
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "bingroup": "x",
         "hovertemplate": "species=Gentoo<br>island=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "Gentoo",
         "marker": {
          "color": "#00cc96",
          "pattern": {
           "shape": ""
          }
         },
         "name": "Gentoo",
         "nbinsx": 50,
         "offsetgroup": "Gentoo",
         "orientation": "v",
         "showlegend": true,
         "type": "histogram",
         "x": [
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe",
          "Biscoe"
         ],
         "xaxis": "x",
         "yaxis": "y"
        }
       ],
       "layout": {
        "bargap": 0.2,
        "bargroupgap": 0.1,
        "barmode": "relative",
        "height": 800,
        "legend": {
         "title": {
          "text": "species"
         },
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Distribution of Species by Island"
        },
        "width": 960,
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "island"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = px.histogram(penguins, x=\"island\", color=\"species\", nbins=50)\n",
    "\n",
    "fig.update_layout(\n",
    "    height=800, \n",
    "    width=960, \n",
    "    title_text=\"Distribution of Species by Island\",\n",
    "    xaxis_title_text='island',\n",
    "    yaxis_title_text='count',\n",
    "    bargap=0.2,\n",
    "    bargroupgap=0.1\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74ae740-3590-4dce-ac5a-6205975c83da",
   "metadata": {},
   "source": [
    "Let's display the distribution of species by sex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "45b0a87f-028d-477f-9b65-199728c0b7ee",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "bingroup": "x",
         "hovertemplate": "species=Adelie<br>sex=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "Adelie",
         "marker": {
          "color": "#636efa",
          "pattern": {
           "shape": ""
          }
         },
         "name": "Adelie",
         "nbinsx": 50,
         "offsetgroup": "Adelie",
         "orientation": "v",
         "showlegend": true,
         "type": "histogram",
         "x": [
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "MALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE"
         ],
         "xaxis": "x",
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "bingroup": "x",
         "hovertemplate": "species=Chinstrap<br>sex=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "Chinstrap",
         "marker": {
          "color": "#EF553B",
          "pattern": {
           "shape": ""
          }
         },
         "name": "Chinstrap",
         "nbinsx": 50,
         "offsetgroup": "Chinstrap",
         "orientation": "v",
         "showlegend": true,
         "type": "histogram",
         "x": [
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE"
         ],
         "xaxis": "x",
         "yaxis": "y"
        },
        {
         "alignmentgroup": "True",
         "bingroup": "x",
         "hovertemplate": "species=Gentoo<br>sex=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "Gentoo",
         "marker": {
          "color": "#00cc96",
          "pattern": {
           "shape": ""
          }
         },
         "name": "Gentoo",
         "nbinsx": 50,
         "offsetgroup": "Gentoo",
         "orientation": "v",
         "showlegend": true,
         "type": "histogram",
         "x": [
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "FEMALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "MALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE",
          "FEMALE",
          "MALE"
         ],
         "xaxis": "x",
         "yaxis": "y"
        }
       ],
       "layout": {
        "bargap": 0.2,
        "bargroupgap": 0.1,
        "barmode": "relative",
        "height": 800,
        "legend": {
         "title": {
          "text": "species"
         },
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Distribution of Species by Sex"
        },
        "width": 960,
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "sex"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = px.histogram(penguins, x=\"sex\", color=\"species\", nbins=50)\n",
    "\n",
    "fig.update_layout(\n",
    "    height=800, \n",
    "    width=960, \n",
    "    title_text=\"Distribution of Species by Sex\",\n",
    "    xaxis_title_text='sex',\n",
    "    yaxis_title_text='count',\n",
    "    bargap=0.2,\n",
    "    bargroupgap=0.1\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f16f6f11",
   "metadata": {},
   "source": [
    "# Training Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56b2c391",
   "metadata": {},
   "source": [
    "In this section, we'll create a [SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) to automate the process of building, evaluating, and registering a model. We'll use the [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/) to create the Pipeline. Check the [SageMaker Pipelines Overview](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-sdk.html) for an introduction to the fundamental components of a SageMaker Pipeline.\n",
    "\n",
    "We can define a caching policy to reuse the result of a previous successful run of a pipeline step. You can find more information about this topic in [Caching Pipeline Steps](https://docs.aws.amazon.com/sagemaker/latest/dg/pipelines-caching.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "544ccde5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import ipytest\n",
    "from sagemaker.workflow.steps import CacheConfig\n",
    "\n",
    "\n",
    "ipytest.autoconfig(raise_on_error=True)\n",
    "sagemaker_session = sagemaker.session.Session()\n",
    "sagemaker_client = boto3.client(\"sagemaker\")\n",
    "iam_client = boto3.client(\"iam\")\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "cache_config = CacheConfig(\n",
    "    enable_caching=True, \n",
    "    expire_after=\"15d\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "587d06e0-b711-4e3d-b424-6fa611a51f94",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Preprocessing\n",
    "\n",
    "We'll use a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) to split and transform the data.\n",
    "\n",
    "Let's create the preprocessing script. The Processing Step will spin up a Processing Job and run this script inside a container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "fb6ba7c0-1bd6-4fe5-8b7f-f6cbdfd3846c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting code/preprocessor.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {CODE_FOLDER}/preprocessor.py\n",
    "\n",
    "import os\n",
    "import sys\n",
    "import argparse\n",
    "import json\n",
    "import tarfile\n",
    "import tempfile\n",
    "import time\n",
    "import joblib\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from io import StringIO\n",
    "from pathlib import Path\n",
    "from sklearn.compose import ColumnTransformer, make_column_selector\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler, OrdinalEncoder\n",
    "\n",
    "\n",
    "def preprocess(base_directory):\n",
    "    \"\"\"\n",
    "    This function loads the supplied data, splits it and transforms it.\n",
    "    \"\"\"\n",
    "\n",
    "    df = _read_data_from_input_csv_files(base_directory)\n",
    "    \n",
    "    target_transformer = ColumnTransformer(\n",
    "        transformers=[(\"species\", OrdinalEncoder(), [0])]\n",
    "    )\n",
    "    \n",
    "    numeric_transformer = make_pipeline(\n",
    "        SimpleImputer(strategy=\"mean\"),\n",
    "        StandardScaler()\n",
    "    )\n",
    "\n",
    "    categorical_transformer = make_pipeline(\n",
    "        SimpleImputer(strategy=\"most_frequent\"),\n",
    "        OneHotEncoder()\n",
    "    )\n",
    "    \n",
    "    features_transformer = ColumnTransformer(\n",
    "        transformers=[\n",
    "            (\"numeric\", numeric_transformer, make_column_selector(dtype_exclude=\"object\")),\n",
    "            (\"categorical\", categorical_transformer, [\"island\"]),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    df_train, df_validation, df_test = _split_data(df)\n",
    "\n",
    "    _save_baseline(base_directory, df_test)\n",
    "\n",
    "    y_train = target_transformer.fit_transform(np.array(df_train.species.values).reshape(-1, 1))\n",
    "    y_validation = target_transformer.transform(np.array(df_validation.species.values).reshape(-1, 1))\n",
    "    y_test = target_transformer.transform(np.array(df_test.species.values).reshape(-1, 1))\n",
    "    \n",
    "    df_train = df_train.drop(\"species\", axis=1)\n",
    "    df_validation = df_validation.drop(\"species\", axis=1)\n",
    "    df_test = df_test.drop(\"species\", axis=1)\n",
    "\n",
    "    X_train = features_transformer.fit_transform(df_train)\n",
    "    X_validation = features_transformer.transform(df_validation)\n",
    "    X_test = features_transformer.transform(df_test)\n",
    "\n",
    "    _save_splits(base_directory, X_train, y_train, X_validation, y_validation, X_test, y_test)\n",
    "    _save_model(base_directory, target_transformer, features_transformer)\n",
    "    \n",
    "\n",
    "def _read_data_from_input_csv_files(base_directory):\n",
    "    \"\"\"\n",
    "    This function reads every CSV file available and concatenates\n",
    "    them into a single dataframe.\n",
    "    \"\"\"\n",
    "\n",
    "    input_directory = Path(base_directory) / \"input\"\n",
    "    files = [file for file in input_directory.glob(\"*.csv\")]\n",
    "    \n",
    "    if len(files) == 0:\n",
    "        raise ValueError(f\"The are no CSV files in {str(input_directory)}/\")\n",
    "        \n",
    "    raw_data = [pd.read_csv(file) for file in files]\n",
    "    df = pd.concat(raw_data)\n",
    "    \n",
    "    # Shuffle the data\n",
    "    return df.sample(frac=1, random_state=42)\n",
    "\n",
    "\n",
    "def _split_data(df):\n",
    "    \"\"\"\n",
    "    Splits the data into three sets: train, validation and test.\n",
    "    \"\"\"\n",
    "\n",
    "    df_train, temp = train_test_split(df, test_size=0.3)\n",
    "    df_validation, df_test = train_test_split(temp, test_size=0.5)\n",
    "\n",
    "    return df_train, df_validation, df_test\n",
    "\n",
    "\n",
    "def _save_baseline(base_directory, df_test):\n",
    "    \"\"\"\n",
    "    This function saves the untransformed test split to disk. This file will\n",
    "    be used later as a baseline to monitor the performance of the model.\n",
    "    \"\"\"\n",
    "\n",
    "    baseline_path = Path(base_directory) / f\"baseline\"\n",
    "    baseline_path.mkdir(parents=True, exist_ok=True)\n",
    "    df_test.to_csv(baseline_path / \"baseline.csv\", header=False, index=False)\n",
    "\n",
    "\n",
    "def _save_splits(base_directory, X_train, y_train, X_validation, y_validation, X_test, y_test):\n",
    "    \"\"\"\n",
    "    This function concatenates the transformed features and the target variable, and\n",
    "    saves each one of the split sets to disk.\n",
    "    \"\"\"\n",
    "\n",
    "    train = np.concatenate((X_train, y_train), axis=1)\n",
    "    validation = np.concatenate((X_validation, y_validation), axis=1)\n",
    "    test = np.concatenate((X_test, y_test), axis=1)\n",
    "\n",
    "    train_path = Path(base_directory) / \"train\"\n",
    "    validation_path = Path(base_directory) / \"validation\"\n",
    "    test_path = Path(base_directory) / \"test\"\n",
    "\n",
    "    train_path.mkdir(parents=True, exist_ok=True)\n",
    "    validation_path.mkdir(parents=True, exist_ok=True)\n",
    "    test_path.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "    pd.DataFrame(train).to_csv(train_path / \"train.csv\", header=False, index=False)\n",
    "    pd.DataFrame(validation).to_csv(validation_path / \"validation.csv\", header=False, index=False)\n",
    "    pd.DataFrame(test).to_csv(test_path / \"test.csv\", header=False, index=False)\n",
    "\n",
    "\n",
    "def _save_model(base_directory, target_transformer, features_transformer):\n",
    "    \"\"\"\n",
    "    This function creates a model.tar.gz file that contains the two transformation\n",
    "    pipelines we built to transform the data.\n",
    "    \"\"\"\n",
    "\n",
    "    with tempfile.TemporaryDirectory() as directory:\n",
    "        joblib.dump(target_transformer, os.path.join(directory, \"target.joblib\"))\n",
    "        joblib.dump(features_transformer, os.path.join(directory, \"features.joblib\"))\n",
    "    \n",
    "        model_path = Path(base_directory) / \"model\"\n",
    "        model_path.mkdir(parents=True, exist_ok=True)\n",
    "    \n",
    "        with tarfile.open(f\"{str(model_path / 'model.tar.gz')}\", \"w:gz\") as tar:\n",
    "            tar.add(os.path.join(directory, \"target.joblib\"), arcname=\"target.joblib\")\n",
    "            tar.add(os.path.join(directory, \"features.joblib\"), arcname=\"features.joblib\")\n",
    "\n",
    "    \n",
    "if __name__ == \"__main__\":\n",
    "    preprocess(base_directory=\"/opt/ml/processing\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f0a6dcc",
   "metadata": {},
   "source": [
    "Let's test the script to ensure everything is working as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "066b5ebb-90c8-49cd-9e34-7fde31412150",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\n",
      "\u001b[32m\u001b[32m\u001b[1m5 passed\u001b[0m\u001b[32m in 0.10s\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%%ipytest -s\n",
    "\n",
    "import os\n",
    "import shutil\n",
    "import tarfile\n",
    "import pytest\n",
    "import tempfile\n",
    "import joblib\n",
    "from preprocessor import preprocess\n",
    "\n",
    "\n",
    "@pytest.fixture(scope=\"function\", autouse=False)\n",
    "def directory():\n",
    "    directory = tempfile.mkdtemp()\n",
    "    input_directory = Path(directory) / \"input\"\n",
    "    input_directory.mkdir(parents=True, exist_ok=True)\n",
    "    shutil.copy2(DATA_FILEPATH, input_directory / \"data.csv\")\n",
    "    \n",
    "    directory = Path(directory)\n",
    "    preprocess(base_directory=directory)\n",
    "    \n",
    "    yield directory\n",
    "    \n",
    "    shutil.rmtree(directory)\n",
    "\n",
    "\n",
    "def test_preprocess_generates_data_splits(directory):\n",
    "    output_directories = os.listdir(directory)\n",
    "    \n",
    "    assert \"train\" in output_directories\n",
    "    assert \"validation\" in output_directories\n",
    "    assert \"test\" in output_directories\n",
    "\n",
    "\n",
    "def test_preprocess_generates_baseline(directory):\n",
    "    output_directories = os.listdir(directory)\n",
    "\n",
    "    assert \"baseline\" in output_directories\n",
    "\n",
    "\n",
    "def test_preprocess_creates_two_models(directory):\n",
    "    model_path = directory / \"model\"\n",
    "    tar = tarfile.open(model_path / \"model.tar.gz\", \"r:gz\")\n",
    "\n",
    "    assert \"features.joblib\" in tar.getnames()\n",
    "    assert \"target.joblib\" in tar.getnames()\n",
    "\n",
    "\n",
    "def test_splits_are_transformed(directory):\n",
    "    train = pd.read_csv(directory / \"train\" / \"train.csv\", header=None)\n",
    "    validation = pd.read_csv(directory / \"validation\" / \"validation.csv\", header=None)\n",
    "    test = pd.read_csv(directory / \"test\" / \"test.csv\", header=None)\n",
    "\n",
    "    # After transforming the data, the number of features should be 7:\n",
    "    # * 3 - island (one-hot encoded)\n",
    "    # * 1 - culmen_length_mm = 1\n",
    "    # * 1 - culmen_depth_mm\n",
    "    # * 1 - flipper_length_mm\n",
    "    # * 1 - body_mass_g\n",
    "    number_of_features = 7\n",
    "\n",
    "    # The transformed splits should have an additional column for the target\n",
    "    # variable.\n",
    "    assert train.shape[1] == number_of_features + 1\n",
    "    assert validation.shape[1] == number_of_features + 1\n",
    "    assert test.shape[1] == number_of_features + 1\n",
    "\n",
    "\n",
    "def test_baseline_is_not_transformed(directory):\n",
    "    baseline = pd.read_csv(directory / \"baseline\" / \"baseline.csv\", header=None)\n",
    "\n",
    "    island = baseline.iloc[:, 1].unique()\n",
    "\n",
    "    assert \"Biscoe\" in island\n",
    "    assert \"Torgersen\" in island\n",
    "    assert \"Dream\" in island"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9771e08-5560-492a-845e-f06988b6ae1b",
   "metadata": {},
   "source": [
    "The first step we need in the pipeline is a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) to run the preprocessing script. This Processing Step will create a SageMaker Processing Job in the background, run the script, and upload the output to S3. You can use Processing Jobs to perform data preprocessing, post-processing, feature engineering, data validation, and model evaluation. Check the [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "We can parameterize a SageMaker Pipeline to make it more flexible. To read more about these parameters, check [Pipeline Parameters](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html). The `dataset_location` represents the location of the data we can to process in our pipeline. We can execute the pipeline with different datasets by changing the value of this parameter.\n",
    "\n",
    "A processor gives the Processing Step information about the hardware and software that SageMaker should use to launch the Processing Job. To run the script, we need access to Scikit-Learn, so we can use the [SKLearnProcessor](https://sagemaker.readthedocs.io/en/stable/frameworks/sklearn/sagemaker.sklearn.html#scikit-learn-processor) processor that comes out-of-the-box with the SageMaker's Python SDK. The [Data Processing with Framework Processors](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job-frameworks.html) page discusses other built-in processors you can use. The [Docker Registry Paths and Example Code](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html) page contains information about the available framework versions for each region.\n",
    "\n",
    "The [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) requires a list of inputs that we need on the preprocessing script. In this case, the input is the dataset we stored in S3. We also have a few outputs that we want SageMaker to capture when the Processing Job finishes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "id": "2187b65e-e504-40a5-ad05-82f54885805c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to only available Python version: py3\n",
      "/Users/svpino/dev/ml.school/.venv/lib/python3.9/site-packages/sagemaker/workflow/pipeline_context.py:297: UserWarning:\n",
      "\n",
      "Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "from sagemaker.workflow.steps import ProcessingStep\n",
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from sagemaker.workflow.parameters import ParameterString\n",
    "\n",
    "\n",
    "dataset_location = ParameterString(\n",
    "    name=\"dataset_location\",\n",
    "    default_value=f\"{S3_LOCATION}/data\",\n",
    ")\n",
    "\n",
    "processor = SKLearnProcessor(\n",
    "    base_job_name=\"split-and-transform-data\",\n",
    "    framework_version=\"1.2-1\",\n",
    "\n",
    "    # By default, a new account doesn't have access to `ml.m5.xlarge` instances.\n",
    "    # If you haven't requested a quota increase yet, you can use an\n",
    "    # `ml.t3.medium` instance type instead. This will work out of the box, but\n",
    "    # the Processing Job will take significantly longer than it should have.\n",
    "    # To get access to `ml.m5.xlarge` instances, you can request a quota \n",
    "    # increase under the Service Quotas section in your AWS account.\n",
    "    instance_type=config[\"instance_type\"],\n",
    "    \n",
    "    instance_count=1,\n",
    "    role=config[\"role\"],\n",
    "    sagemaker_session=config[\"session\"],\n",
    ")\n",
    "\n",
    "split_and_transform_data_step = ProcessingStep(\n",
    "    name=\"split-and-transform-data\",\n",
    "    step_args=processor.run(\n",
    "        code=f\"{CODE_FOLDER}/preprocessor.py\",\n",
    "        inputs=[\n",
    "            ProcessingInput(source=dataset_location, destination=\"/opt/ml/processing/input\"),  \n",
    "        ],\n",
    "        outputs=[\n",
    "            ProcessingOutput(output_name=\"train\", source=\"/opt/ml/processing/train\"),\n",
    "            ProcessingOutput(output_name=\"validation\", source=\"/opt/ml/processing/validation\"),\n",
    "            ProcessingOutput(output_name=\"test\", source=\"/opt/ml/processing/test\"),\n",
    "            ProcessingOutput(output_name=\"model\", source=\"/opt/ml/processing/model\"),\n",
    "            \n",
    "            # The baseline output points to the test set before transforming the data. This set\n",
    "            # will be helpful to generate a quality baseline for the model performance.\n",
    "            ProcessingOutput(output_name=\"baseline\", source=\"/opt/ml/processing/baseline\"),\n",
    "        ]\n",
    "    ),\n",
    "    cache_config=cache_config\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8608092-7aab-4fd2-aa99-47c2db27bdb7",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "We'll use a [Training Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-training) to build a model.\n",
    "\n",
    "This following script is responsible for training a neural network using the train data, validating the model, and saving it so we can later use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "d92b121d-dcb9-43e8-9ee3-3ececb583e7e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting code/train.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {CODE_FOLDER}/train.py\n",
    "\n",
    "import os\n",
    "import argparse\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "from pathlib import Path\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "\n",
    "\n",
    "def train(model_directory, train_path, validation_path, epochs=50, batch_size=32):\n",
    "    train_files = [file for file in Path(train_path).glob(\"*.csv\")]\n",
    "    validation_files = [file for file in Path(validation_path).glob(\"*.csv\")]\n",
    "    \n",
    "    if len(train_files) == 0 or len(validation_files) == 0:\n",
    "        raise ValueError(\"The are no train or validation files\")\n",
    "        \n",
    "    train_data = [pd.read_csv(file, header=None) for file in train_files]\n",
    "    X_train = pd.concat(train_data)\n",
    "    y_train = X_train[X_train.columns[-1]]\n",
    "    X_train.drop(X_train.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    \n",
    "    validation_data = [pd.read_csv(file, header=None) for file in validation_files]\n",
    "    X_validation = pd.concat(validation_data)\n",
    "    y_validation = X_validation[X_validation.columns[-1]]\n",
    "    X_validation.drop(X_validation.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    model = Sequential([\n",
    "        Dense(10, input_shape=(X_train.shape[1],), activation=\"relu\"),\n",
    "        Dense(8, activation=\"relu\"),\n",
    "        Dense(3, activation=\"softmax\"),\n",
    "    ])\n",
    "    \n",
    "    model.compile(\n",
    "        optimizer=SGD(learning_rate=0.01),\n",
    "        loss=\"sparse_categorical_crossentropy\",\n",
    "        metrics=[\"accuracy\"]\n",
    "    )\n",
    "\n",
    "    model.fit(\n",
    "        X_train, \n",
    "        y_train, \n",
    "        validation_data=(X_validation, y_validation),\n",
    "        epochs=epochs, \n",
    "        batch_size=batch_size,\n",
    "        verbose=2,\n",
    "    )\n",
    "\n",
    "    predictions = np.argmax(model.predict(X_validation), axis=-1)\n",
    "    print(f\"Validation accuracy: {accuracy_score(y_validation, predictions)}\")\n",
    "    \n",
    "    model_filepath = Path(model_directory) / \"001\"\n",
    "    model.save(model_filepath)    \n",
    "    \n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    # Any hyperparameters provided by the training job are passed to \n",
    "    # the entry point as script arguments. \n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"--epochs\", type=int, default=50)\n",
    "    parser.add_argument(\"--batch_size\", type=int, default=32)\n",
    "    args, _ = parser.parse_known_args()\n",
    "    \n",
    "\n",
    "    train(\n",
    "        # This is the location where we need to save our model. SageMaker will\n",
    "        # create a model.tar.gz file with anything inside this directory when\n",
    "        # the training script finishes.\n",
    "        model_directory=os.environ[\"SM_MODEL_DIR\"],\n",
    "\n",
    "        # SageMaker creates one channel for each one of the inputs to the\n",
    "        # Training Step.\n",
    "        train_path=os.environ[\"SM_CHANNEL_TRAIN\"],\n",
    "        validation_path=os.environ[\"SM_CHANNEL_VALIDATION\"],\n",
    "\n",
    "        epochs=args.epochs,\n",
    "        batch_size=args.batch_size,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51c96e2d",
   "metadata": {},
   "source": [
    "Let's test the script to ensure everything is working as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "id": "14ea27ce-c453-4cb0-b309-dbecd732957e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8/8 - 0s - loss: 1.1870 - accuracy: 0.3208 - val_loss: 1.2022 - val_accuracy: 0.3077 - 228ms/epoch - 29ms/step\n",
      "2/2 [==============================] - 0s 2ms/step\n",
      "Validation accuracy: 0.3076923076923077\n",
      "INFO:tensorflow:Assets written to: /var/folders/4c/v1q3hy1x4mb5w0wpc72zl3_w0000gp/T/tmponqr06s6/model/001/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /var/folders/4c/v1q3hy1x4mb5w0wpc72zl3_w0000gp/T/tmponqr06s6/model/001/assets\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m.\u001b[0m\n",
      "\u001b[32m\u001b[32m\u001b[1m1 passed\u001b[0m\u001b[32m in 0.52s\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%%ipytest -s\n",
    "\n",
    "import os\n",
    "import shutil\n",
    "import tarfile\n",
    "import pytest\n",
    "import tempfile\n",
    "import joblib\n",
    "\n",
    "from preprocessor import preprocess\n",
    "from train import train\n",
    "\n",
    "\n",
    "@pytest.fixture(scope=\"function\", autouse=False)\n",
    "def directory():\n",
    "    directory = tempfile.mkdtemp()\n",
    "    input_directory = Path(directory) / \"input\"\n",
    "    input_directory.mkdir(parents=True, exist_ok=True)\n",
    "    shutil.copy2(DATA_FILEPATH, input_directory / \"data.csv\")\n",
    "    \n",
    "    directory = Path(directory)\n",
    "    \n",
    "    preprocess(base_directory=directory)\n",
    "    train(\n",
    "        model_directory=directory / \"model\",\n",
    "        train_path=directory / \"train\", \n",
    "        validation_path=directory / \"validation\",\n",
    "        epochs=1\n",
    "    )\n",
    "    \n",
    "    yield directory\n",
    "    \n",
    "    shutil.rmtree(directory)\n",
    "\n",
    "\n",
    "def test_train_saves_a_folder_with_model_assets(directory):\n",
    "    output = os.listdir(directory / \"model\")\n",
    "    assert \"001\" in output\n",
    "    \n",
    "    assets = os.listdir(directory / \"model\" / \"001\")\n",
    "    assert \"saved_model.pb\" in assets\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27cff4c1-6510-4d99-8ae1-cb14927b87c7",
   "metadata": {},
   "source": [
    "SageMaker uses the concept of an [Estimator](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html) to handle end-to-end training tasks. For this example, we will use the built-in [TensorFlow Estimator](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-estimator) to run the training script we wrote before. The [Docker Registry Paths and Example Code](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html) page contains information about the available framework versions for each region. Here, you can also check the available SageMaker [Deep Learning Container images](https://github.com/aws/deep-learning-containers/blob/master/available_images.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "id": "90fe82ae-6a2c-4461-bc83-bb52d8871e3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "\n",
    "estimator = TensorFlow(\n",
    "    base_job_name=\"training\",\n",
    "    entry_point=f\"{CODE_FOLDER}/train.py\",\n",
    "    \n",
    "    # SageMaker will pass these hyperparameters as arguments\n",
    "    # to the entry point of the training script.\n",
    "    hyperparameters={\n",
    "        \"epochs\": 50,\n",
    "        \"batch_size\": 32,\n",
    "    },\n",
    "\n",
    "    # SageMaker will track these metrics as part of the experiment\n",
    "    # associated to this pipeline. The metric definitions tells \n",
    "    # SageMaker how to parse the values from the Training Job logs.\n",
    "    metric_definitions=[\n",
    "        {\"Name\": \"loss\", \"Regex\": \"loss: ([0-9\\\\.]+)\"},\n",
    "        {\"Name\": \"accuracy\", \"Regex\": \"accuracy: ([0-9\\\\.]+)\"},\n",
    "        {\"Name\": \"val_loss\", \"Regex\": \"val_loss: ([0-9\\\\.]+)\"},\n",
    "        {\"Name\": \"val_accuracy\", \"Regex\": \"val_accuracy: ([0-9\\\\.]+)\"},\n",
    "    ],\n",
    "\n",
    "    image_uri=config[\"image\"],\n",
    "    framework_version=config[\"framework_version\"],\n",
    "    py_version=config[\"py_version\"],\n",
    "    instance_type=config[\"instance_type\"],\n",
    "    instance_count=1,\n",
    "    disable_profiler=True,\n",
    "    sagemaker_session=config[\"session\"],\n",
    "    role=config[\"role\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545d2b43-3bb5-4fe9-b3e4-cb8eb55c8a21",
   "metadata": {},
   "source": [
    "We can now create a [Training Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-training). This Training Step will create a SageMaker Training Job in the background, run the training script, and upload the output to S3. Check the [TrainingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TrainingStep) SageMaker's SDK documentation for more information. \n",
    "\n",
    "This step will receive the train and validation split from the previous step as inputs.\n",
    "\n",
    "Here, we are using two input channels, `train` and `validation`. SageMaker will automatically create an environment variable corresponding to each of these channels following the format `SM_CHANNEL_[channel_name]`:\n",
    "\n",
    "* `SM_CHANNEL_TRAIN`: This environment variable will contain the path to the data in the `train` channel\n",
    "* `SM_CHANNEL_VALIDATION`: This environment variable will contain the path to the data in the `validation` channel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "99e4850c-83d6-4f4e-a813-d5a3f4bb7486",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/svpino/dev/ml.school/.venv/lib/python3.9/site-packages/sagemaker/workflow/pipeline_context.py:297: UserWarning:\n",
      "\n",
      "Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.workflow.steps import TrainingStep\n",
    "from sagemaker.inputs import TrainingInput\n",
    "\n",
    "\n",
    "train_model_step = TrainingStep(\n",
    "    name=\"train-model\",\n",
    "    step_args=estimator.fit(\n",
    "        inputs={\n",
    "            \"train\": TrainingInput(\n",
    "                s3_data=split_and_transform_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                    \"train\"\n",
    "                ].S3Output.S3Uri,\n",
    "                content_type=\"text/csv\"\n",
    "            ),\n",
    "            \"validation\": TrainingInput(\n",
    "                s3_data=split_and_transform_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                    \"validation\"\n",
    "                ].S3Output.S3Uri,\n",
    "                content_type=\"text/csv\"\n",
    "            )\n",
    "        }\n",
    "    ),\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5814e258-c633-4e9a-85c5-6ed0f168b503",
   "metadata": {},
   "source": [
    "## Tuning\n",
    "\n",
    "We can also use a [Tuning Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-tuning) to build a model. This Tuning Step will create a SageMaker Hyperparameter Tuning Job in the background and use the training script to train different versions of the model to choose the best one.\n",
    "\n",
    "Since we could use the Training of the Tuning Step to create the model, we'll define a constant `USE_TUNING_STEP` to indicate which approach we want to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "40b33456",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "USE_TUNING_STEP = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e76a24ee",
   "metadata": {},
   "source": [
    "The [TuningStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep) requires a [HyperparameterTuner](https://sagemaker.readthedocs.io/en/stable/api/training/tuner.html) reference to configure the Hyperparameter Tuning Job.\n",
    "\n",
    "Here is the configuration that we'll use to find the best model:\n",
    "\n",
    "1. `objective_metric_name`: This is the name of the metric the tuner will use to determine the best model. We'll use `val_accuracy` to select the model with the highest validation accuracy.\n",
    "2. `objective_type`: This is the objective of the tuner. It can be `Minimize` or `Maximize`. Since we are using the validation accuracy of the model, we want the objective to be `Maximize.` If we were using the loss of the model, we would set the objective to `Minimize.`\n",
    "3. `metric_definitions`: This defines how SageMaker will determine the metric's value by looking at the output logs of the training process.\n",
    "\n",
    "The tuner expects the list of the hyperparameters you want to explore. You can use subclasses of the [Parameter](https://sagemaker.readthedocs.io/en/stable/api/training/parameter.html#sagemaker.parameter.ParameterRange) class to specify different types of hyperparameters. This example explores different values for the `epochs` hyperparameter.\n",
    "\n",
    "You can control the number of jobs and how many of them will run in parallel using the following two arguments:\n",
    "\n",
    "* `max_jobs`: Defines the maximum total number of training jobs to start for the hyperparameter tuning job.\n",
    "* `max_parallel_jobs`: Defines the maximum number of parallel training jobs to start.\n",
    "\n",
    "Finally, we'll create the Tuning Step using the tuner. Notice how the Tuning Step looks very similar to the Training Step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "038ff2e5-ed28-445b-bc03-4e996ec2286f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.tuner import HyperparameterTuner\n",
    "from sagemaker.parameter import IntegerParameter\n",
    "from sagemaker.workflow.steps import TuningStep\n",
    "\n",
    "\n",
    "tuner = HyperparameterTuner(\n",
    "    estimator,\n",
    "    objective_metric_name = \"val_accuracy\",\n",
    "    objective_type=\"Maximize\",\n",
    "    hyperparameter_ranges = {\n",
    "        \"epochs\": IntegerParameter(10, 50),\n",
    "    },\n",
    "    metric_definitions = [\n",
    "        {\"Name\": \"val_accuracy\", \"Regex\": \"val_accuracy: ([0-9\\\\.]+)\"}\n",
    "    ],\n",
    "    max_jobs=3,\n",
    "    max_parallel_jobs=3,\n",
    ")\n",
    "\n",
    "tune_model_step = TuningStep(\n",
    "    name = \"tune-model\",\n",
    "    step_args=tuner.fit(\n",
    "        inputs={\n",
    "            \"train\": TrainingInput(\n",
    "                s3_data=split_and_transform_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                    \"train\"\n",
    "                ].S3Output.S3Uri,\n",
    "                content_type=\"text/csv\"\n",
    "            ),\n",
    "            \"validation\": TrainingInput(\n",
    "                s3_data=split_and_transform_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                    \"validation\"\n",
    "                ].S3Output.S3Uri,\n",
    "                content_type=\"text/csv\"\n",
    "            )\n",
    "        },\n",
    "    ),\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eaa9691-f49f-48af-b272-3d4d17563b01",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Evaluation\n",
    "\n",
    "We'll use a [Processing Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-processing) to evaluate the model.\n",
    "\n",
    "Let's create the evaluation script. The Processing Step will spin up a Processing Job and run this script inside a container. This script is responsible for loading the model we created and evaluating it on the test set. Before finishing, this script will generate an evaluation report of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "3ee3ab26-afa5-4ceb-9f7a-005d5fdea646",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting code/evaluation.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {CODE_FOLDER}/evaluation.py\n",
    "\n",
    "import os\n",
    "import json\n",
    "import tarfile\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from pathlib import Path\n",
    "from tensorflow import keras\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "MODEL_PATH = \"/opt/ml/processing/model/\"\n",
    "TEST_PATH = \"/opt/ml/processing/test/\"\n",
    "OUTPUT_PATH = \"/opt/ml/processing/evaluation/\"\n",
    "\n",
    "\n",
    "def evaluate(model_path, test_path, output_path):\n",
    "    # The first step is to extract the model package so we can load \n",
    "    # it in memory.\n",
    "    with tarfile.open(Path(model_path) / \"model.tar.gz\") as tar:\n",
    "        tar.extractall(path=Path(model_path))\n",
    "        \n",
    "    model = keras.models.load_model(Path(model_path) / \"001\")\n",
    "    \n",
    "    X_test = pd.read_csv(Path(test_path) / \"test.csv\")\n",
    "    y_test = X_test[X_test.columns[-1]]\n",
    "    X_test.drop(X_test.columns[-1], axis=1, inplace=True)\n",
    "    \n",
    "    predictions = np.argmax(model.predict(X_test), axis=-1)\n",
    "    accuracy = accuracy_score(y_test, predictions)\n",
    "    print(f\"Test accuracy: {accuracy}\")\n",
    "\n",
    "    # Let's create an evaluation report using the model accuracy.\n",
    "    evaluation_report = {\n",
    "        \"metrics\": {\n",
    "            \"accuracy\": {\n",
    "                \"value\": accuracy\n",
    "            },\n",
    "        },\n",
    "    }\n",
    "    \n",
    "    Path(output_path).mkdir(parents=True, exist_ok=True)\n",
    "    with open(Path(output_path) / \"evaluation.json\", \"w\") as f:\n",
    "        f.write(json.dumps(evaluation_report))\n",
    "        \n",
    "        \n",
    "if __name__ == \"__main__\":\n",
    "    evaluate(\n",
    "        model_path=MODEL_PATH, \n",
    "        test_path=TEST_PATH,\n",
    "        output_path=OUTPUT_PATH\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73af95c7",
   "metadata": {},
   "source": [
    "Let's test the script to ensure everything is working as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "9a2540d8-278a-4953-bc54-0469d154427d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8/8 - 0s - loss: 1.1896 - accuracy: 0.0917 - val_loss: 1.1674 - val_accuracy: 0.1346 - 227ms/epoch - 28ms/step\n",
      "2/2 [==============================] - 0s 2ms/step\n",
      "Validation accuracy: 0.1346153846153846\n",
      "INFO:tensorflow:Assets written to: /var/folders/4c/v1q3hy1x4mb5w0wpc72zl3_w0000gp/T/tmp9xuc_muo/model/001/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /var/folders/4c/v1q3hy1x4mb5w0wpc72zl3_w0000gp/T/tmp9xuc_muo/model/001/assets\n",
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.RestoredOptimizer` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.RestoredOptimizer`.\n",
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2/2 [==============================] - 0s 2ms/step\n",
      "Test accuracy: 0.19607843137254902\n",
      "\u001b[32m.\u001b[0m"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8/8 - 0s - loss: 1.0395 - accuracy: 0.2375 - val_loss: 0.9944 - val_accuracy: 0.3269 - 230ms/epoch - 29ms/step\n",
      "2/2 [==============================] - 0s 2ms/step\n",
      "Validation accuracy: 0.3269230769230769\n",
      "INFO:tensorflow:Assets written to: /var/folders/4c/v1q3hy1x4mb5w0wpc72zl3_w0000gp/T/tmpr0ypzchk/model/001/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: /var/folders/4c/v1q3hy1x4mb5w0wpc72zl3_w0000gp/T/tmpr0ypzchk/model/001/assets\n",
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.RestoredOptimizer` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.RestoredOptimizer`.\n",
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2/2 [==============================] - 0s 2ms/step\n",
      "Test accuracy: 0.39215686274509803\n",
      "\u001b[32m.\u001b[0m\n",
      "\u001b[32m\u001b[32m\u001b[1m2 passed\u001b[0m\u001b[32m in 1.52s\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%%ipytest -s\n",
    "\n",
    "import os\n",
    "import shutil\n",
    "import tarfile\n",
    "import pytest\n",
    "import tempfile\n",
    "import joblib\n",
    "import json\n",
    "\n",
    "from preprocessor import preprocess\n",
    "from train import train\n",
    "from evaluation import evaluate\n",
    "\n",
    "\n",
    "@pytest.fixture(scope=\"function\", autouse=False)\n",
    "def directory():\n",
    "    directory = tempfile.mkdtemp()\n",
    "    input_directory = Path(directory) / \"input\"\n",
    "    input_directory.mkdir(parents=True, exist_ok=True)\n",
    "    shutil.copy2(DATA_FILEPATH, input_directory / \"data.csv\")\n",
    "    \n",
    "    directory = Path(directory)\n",
    "    \n",
    "    preprocess(base_directory=directory)\n",
    "    \n",
    "    train(\n",
    "        model_directory=directory / \"model\",\n",
    "        train_path=directory / \"train\", \n",
    "        validation_path=directory / \"validation\",\n",
    "        epochs=1\n",
    "    )\n",
    "    \n",
    "    # After training a model, we need to prepare a package just like\n",
    "    # SageMaker would. This package is what the evaluation script is\n",
    "    # expecting as an input.\n",
    "    with tarfile.open(directory / \"model.tar.gz\", \"w:gz\") as tar:\n",
    "        tar.add(directory / \"model\" / \"001\", arcname=\"001\")\n",
    "        \n",
    "    evaluate(\n",
    "        model_path=directory, \n",
    "        test_path=directory / \"test\",\n",
    "        output_path=directory / \"evaluation\",\n",
    "    )\n",
    "\n",
    "    yield directory / \"evaluation\"\n",
    "    \n",
    "    shutil.rmtree(directory)\n",
    "\n",
    "\n",
    "def test_evaluate_generates_evaluation_report(directory):\n",
    "    output = os.listdir(directory)\n",
    "    assert \"evaluation.json\" in output\n",
    "\n",
    "\n",
    "def test_evaluation_report_contains_accuracy(directory):\n",
    "    with open(directory / \"evaluation.json\", 'r') as file:\n",
    "        report = json.load(file)\n",
    "        \n",
    "    assert \"metrics\" in report\n",
    "    assert \"accuracy\" in report[\"metrics\"]\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec1109a-6c26-4464-8338-94960729d212",
   "metadata": {},
   "source": [
    "To run the evaluation script, we'll use a Processing Step with a [TensorFlowProcessor](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job-frameworks-tensorflow.html). \n",
    "\n",
    "We can use the `USE_TUNING_STEP` flag to determine whether we created the model using a Training Step or a Tuning Step. In case we are using the Tuning Step, we can use the [TuningStep.get_top_model_s3_uri()](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.TuningStep.get_top_model_s3_uri) function to get the model artifacts from the top performing training job of the Hyperparameter Tuning Job.\n",
    "\n",
    "The [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.steps.ProcessingStep) lets us specify a list of [PropertyFile](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.properties.PropertyFile) instances from the output of the job. We can use this to map the evaluation report generated in the evaluation script. Check [How to Build and Manage Property Files](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-propertyfile.html) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "48139a07-5c8e-4bc6-b666-bf9531f7f520",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "/Users/svpino/dev/ml.school/.venv/lib/python3.9/site-packages/sagemaker/workflow/pipeline_context.py:297: UserWarning:\n",
      "\n",
      "Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.tensorflow import TensorFlowProcessor\n",
    "from sagemaker.workflow.properties import PropertyFile\n",
    "\n",
    "\n",
    "tensorflow_processor = TensorFlowProcessor(\n",
    "    base_job_name=\"evaluation-processor\",\n",
    "    image_uri=config[\"image\"],\n",
    "    framework_version=config[\"framework_version\"],\n",
    "    py_version=config[\"py_version\"],\n",
    "    instance_type=config[\"instance_type\"],\n",
    "    instance_count=1,\n",
    "    role=config[\"role\"],\n",
    "    sagemaker_session=config[\"session\"],\n",
    ")\n",
    "\n",
    "# We want to map the evaluation report that we generate inside\n",
    "# the evaluation script so we can later reference it.\n",
    "evaluation_report = PropertyFile(\n",
    "    name=\"evaluation-report\",\n",
    "    output_name=\"evaluation\",\n",
    "    path=\"evaluation.json\"\n",
    ")\n",
    "\n",
    "# Notice how this step uses the model generated by the tuning or training\n",
    "# step, and the test set generated by the preprocessing step.\n",
    "evaluate_model_step = ProcessingStep(\n",
    "    name=\"evaluate-model\",\n",
    "    step_args=tensorflow_processor.run(\n",
    "        inputs=[\n",
    "            ProcessingInput(\n",
    "                source=split_and_transform_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "                    \"test\"\n",
    "                ].S3Output.S3Uri,\n",
    "                destination=\"/opt/ml/processing/test\"\n",
    "            ),\n",
    "            ProcessingInput(\n",
    "                source=(\n",
    "                    tune_model_step.get_top_model_s3_uri(top_k=0, s3_bucket=config[\"session\"].default_bucket()) \n",
    "                    if USE_TUNING_STEP \n",
    "                    else train_model_step.properties.ModelArtifacts.S3ModelArtifacts\n",
    "                ),\n",
    "                destination=\"/opt/ml/processing/model\",\n",
    "            )\n",
    "        ],\n",
    "        outputs=[\n",
    "            ProcessingOutput(output_name=\"evaluation\", source=\"/opt/ml/processing/evaluation\"),\n",
    "        ],\n",
    "        code=f\"{CODE_FOLDER}/evaluation.py\"\n",
    "    ),\n",
    "    property_files=[evaluation_report],\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2344a23c",
   "metadata": {},
   "source": [
    "## Inference Pipeline\n",
    "\n",
    "Deploying the model we trained directly to an endpoint doesn't lets us control the data that goes in and comes out of the endpoint. The TensorFlow model we trained requires the data to come in a specific format, which makes it useless to other applications. Fortunately, we can create an Inference Pipeline using SageMaker to control the data that goes in and comes out of the endpoint.\n",
    "\n",
    "Our inference pipeline will have three components:\n",
    "\n",
    "1. A preprocessing transformer that will transform the input data into the format the model expects. \n",
    "2. The TensorFlow model we trained.\n",
    "3. A postprocessing transformer that will transform the output of the model into a human-readable format.\n",
    "\n",
    "We want our endpoint to handle unprocessed data in CSV and JSON format and return the penguin's species. Here is an example of the payload input we want the endpoint to support:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"island\": \"Biscoe\",\n",
    "    \"culmen_length_mm\": 48.6,\n",
    "    \"culmen_depth_mm\": 16.0,\n",
    "    \"flipper_length_mm\": 230.0,\n",
    "    \"body_mass_g\": 5800.0,\n",
    "}\n",
    "```\n",
    "\n",
    "And here is an example of the output we'd like to get from the endpoint:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"prediction\": \"Adelie\", \n",
    "    \"confidence\": 0.802672\n",
    "}\n",
    "```\n",
    "\n",
    "Let's start by setting up a local folder where we will create the `inference.py` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "id": "f9550423",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "INFERENCE_CODE_FOLDER = CODE_FOLDER / \"inference\"\n",
    "Path(INFERENCE_CODE_FOLDER).mkdir(parents=True, exist_ok=True)\n",
    "sys.path.append(f\"./{INFERENCE_CODE_FOLDER}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df247a8d",
   "metadata": {},
   "source": [
    "### Preprocessing Script\n",
    "\n",
    "The first component of our inference pipeline is a transformer that will transform the input data into the format the model expects. We'll use the Scikit-Learn transformer we saved when we split and transformed the data. To deploy this transformer as part of an inference pipeline, we need to write a script that loads the transformer, uses it to modify the input data, and returns the output in the format the TensorFlow model expects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "id": "b7dbdbf1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting code/inference/preprocessing_component.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {INFERENCE_CODE_FOLDER}/preprocessing_component.py\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import json\n",
    "import joblib\n",
    "\n",
    "from io import StringIO\n",
    "\n",
    "try:\n",
    "    from sagemaker_containers.beta.framework import encoders, worker\n",
    "except ImportError:\n",
    "    # We don't have access to the `worker` instance when testing locally. \n",
    "    # We'll set it to None so we can change the way functions create a response.\n",
    "    worker = None\n",
    "\n",
    "\n",
    "TARGET_COLUMN = \"species\"\n",
    "FEATURE_COLUMNS = [\n",
    "    \"island\",\n",
    "    \"culmen_length_mm\",\n",
    "    \"culmen_depth_mm\", \n",
    "    \"flipper_length_mm\",\n",
    "    \"body_mass_g\",\n",
    "    \"sex\"\n",
    "]\n",
    "\n",
    "\n",
    "def input_fn(input_data, content_type):\n",
    "    \"\"\"\n",
    "    Parses the input payload and creates a Pandas DataFrame.\n",
    "    \n",
    "    This function will check whether the target column is present in the\n",
    "    input data, and will remove it.\n",
    "    \"\"\"\n",
    "    \n",
    "    if content_type == \"text/csv\":\n",
    "        df = pd.read_csv(StringIO(input_data), header=None, skipinitialspace=True)\n",
    "\n",
    "        if len(df.columns) == len(FEATURE_COLUMNS) + 1:\n",
    "            df = df.drop(df.columns[0], axis=1)\n",
    "        \n",
    "        df.columns = FEATURE_COLUMNS\n",
    "        return df\n",
    "    \n",
    "    if content_type == \"application/json\":\n",
    "        df = pd.DataFrame([json.loads(input_data)])\n",
    "        \n",
    "        if \"species\" in df.columns:\n",
    "            df = df.drop(\"species\", axis=1)\n",
    "        \n",
    "        return df\n",
    "    \n",
    "    else:\n",
    "        raise ValueError(f\"{content_type} is not supported.!\")\n",
    "\n",
    "\n",
    "def output_fn(prediction, accept):\n",
    "    \"\"\"\n",
    "    Formats the prediction output to generate a response.\n",
    "    \n",
    "    The default accept/content-type between containers for serial inference is JSON. \n",
    "    Since this model will preceed a TensorFlow model, we want to return a JSON object\n",
    "    following TensorFlow's input requirements.\n",
    "    \"\"\"\n",
    "    \n",
    "    if prediction is None:\n",
    "        raise Exception(f\"There was an error transforming the input data\")\n",
    "\n",
    "    if accept == \"text/csv\":\n",
    "        return worker.Response(encoders.encode(prediction, accept), mimetype=accept) if worker else prediction, accept \n",
    "    \n",
    "    if accept == \"application/json\":\n",
    "        instances = [p for p in prediction.tolist()]\n",
    "        response = {\"instances\": instances}\n",
    "        return worker.Response(json.dumps(response), mimetype=accept) if worker else (response, accept)\n",
    "\n",
    "    raise Exception(f\"{accept} accept type is not supported.\")\n",
    "\n",
    "\n",
    "def predict_fn(input_data, model):\n",
    "    \"\"\"\n",
    "    Preprocess the input using the transformer.\n",
    "    \"\"\"\n",
    "    \n",
    "    try:\n",
    "        response = model.transform(input_data)\n",
    "        return response\n",
    "    except ValueError as e:\n",
    "        print(\"Error transforming the input data\", e)\n",
    "        return None\n",
    "\n",
    "\n",
    "def model_fn(model_dir):\n",
    "    \"\"\"\n",
    "    Deserializes the model that will be used in this container.\n",
    "    \"\"\"\n",
    "    \n",
    "    return joblib.load(os.path.join(model_dir, \"features.joblib\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b71b5a8d",
   "metadata": {},
   "source": [
    "Let's test the script to ensure everything is working as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "id": "75abc675",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m                                                                                   [100%]\u001b[0m\n",
      "\u001b[32m\u001b[32m\u001b[1m10 passed\u001b[0m\u001b[32m in 0.07s\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%%ipytest\n",
    "\n",
    "import json\n",
    "\n",
    "from preprocessing_component import input_fn, predict_fn, output_fn, model_fn\n",
    "\n",
    "\n",
    "@pytest.fixture(scope=\"function\", autouse=False)\n",
    "def directory():\n",
    "    directory = tempfile.mkdtemp()\n",
    "    input_directory = Path(directory) / \"input\"\n",
    "    input_directory.mkdir(parents=True, exist_ok=True)\n",
    "    shutil.copy2(DATA_FILEPATH, input_directory / \"data.csv\")\n",
    "    \n",
    "    directory = Path(directory)\n",
    "    \n",
    "    preprocess(base_directory=directory)\n",
    "    \n",
    "    with tarfile.open(directory / \"model\" / \"model.tar.gz\") as tar:\n",
    "        tar.extractall(path=directory / \"model\")\n",
    "    \n",
    "    yield directory / \"model\"\n",
    "    \n",
    "    shutil.rmtree(directory)\n",
    "\n",
    "\n",
    "\n",
    "def test_input_csv_drops_target_column_if_present():\n",
    "    input_data = \"\"\"\n",
    "    Adelie, Torgersen, 39.1, 18.7, 181, 3750, MALE\n",
    "    \"\"\"\n",
    "    \n",
    "    df = input_fn(input_data, \"text/csv\")\n",
    "    assert len(df.columns) == 6 and \"species\" not in df.columns\n",
    "\n",
    "\n",
    "def test_input_json_drops_target_column_if_present():\n",
    "    input_data = json.dumps({\n",
    "        \"species\": \"Adelie\", \n",
    "        \"island\": \"Torgersen\",\n",
    "        \"culmen_length_mm\": 44.1,\n",
    "        \"culmen_depth_mm\": 18.0,\n",
    "        \"flipper_length_mm\": 210.0,\n",
    "        \"body_mass_g\": 4000.0,\n",
    "        \"sex\": \"MALE\"\n",
    "    })\n",
    "    \n",
    "    df = input_fn(input_data, \"application/json\")\n",
    "    assert len(df.columns) == 6 and \"species\" not in df.columns\n",
    "\n",
    "\n",
    "def test_input_csv_works_without_target_column():\n",
    "    input_data = \"\"\"\n",
    "    Torgersen, 39.1, 18.7, 181, 3750, MALE\n",
    "    \"\"\"\n",
    "    \n",
    "    df = input_fn(input_data, \"text/csv\")\n",
    "    assert len(df.columns) == 6\n",
    "\n",
    "\n",
    "def test_input_json_works_without_target_column():\n",
    "    input_data = json.dumps({\n",
    "        \"island\": \"Torgersen\",\n",
    "        \"culmen_length_mm\": 44.1,\n",
    "        \"culmen_depth_mm\": 18.0,\n",
    "        \"flipper_length_mm\": 210.0,\n",
    "        \"body_mass_g\": 4000.0,\n",
    "        \"sex\": \"MALE\"\n",
    "    })\n",
    "    \n",
    "    df = input_fn(input_data, \"application/json\")\n",
    "    assert len(df.columns) == 6\n",
    "\n",
    "\n",
    "def test_output_csv_raises_exception_if_prediction_is_none():\n",
    "    with pytest.raises(Exception):\n",
    "        output_fn(None, \"text/csv\")\n",
    "    \n",
    "    \n",
    "def test_output_json_raises_exception_if_prediction_is_none():\n",
    "    with pytest.raises(Exception):\n",
    "        output_fn(None, \"application/json\")\n",
    "    \n",
    "    \n",
    "def test_output_csv_returns_prediction():\n",
    "    prediction = np.array([\n",
    "        [-1.3944109908736013,1.15488062669371,-0.7954340636549508,-0.5536447804097907,0.0,1.0,0.0],\n",
    "        [1.0557485835338234,0.5040085971987002,-0.5824506029515057,-0.5851840035995248,0.0,1.0,0.0]\n",
    "    ])\n",
    "    \n",
    "    response = output_fn(prediction, \"text/csv\")\n",
    "    \n",
    "    assert response == (prediction, \"text/csv\")\n",
    "    \n",
    "    \n",
    "def test_output_json_returns_tensorflow_ready_input():\n",
    "    prediction = np.array([\n",
    "        [-1.3944109908736013,1.15488062669371,-0.7954340636549508,-0.5536447804097907,0.0,1.0,0.0],\n",
    "        [1.0557485835338234,0.5040085971987002,-0.5824506029515057,-0.5851840035995248,0.0,1.0,0.0]\n",
    "    ])\n",
    "    \n",
    "    response = output_fn(prediction, \"application/json\")\n",
    "    \n",
    "    assert response[0] == {\n",
    "        \"instances\": [\n",
    "            [-1.3944109908736013,1.15488062669371,-0.7954340636549508,-0.5536447804097907,0.0,1.0,0.0],\n",
    "            [1.0557485835338234,0.5040085971987002,-0.5824506029515057,-0.5851840035995248,0.0,1.0,0.0]\n",
    "        ]\n",
    "    }\n",
    "    \n",
    "    assert response[1] == \"application/json\"\n",
    "\n",
    "    \n",
    "def test_predict_transforms_data(directory):\n",
    "    input_data = \"\"\"\n",
    "    Torgersen, 39.1, 18.7, 181, 3750, MALE\n",
    "    \"\"\"\n",
    "    \n",
    "    model = model_fn(str(directory))\n",
    "    df = input_fn(input_data, \"text/csv\")\n",
    "    response = predict_fn(df, model)\n",
    "    assert type(response) is np.ndarray\n",
    "    \n",
    "\n",
    "def test_predict_returns_none_if_invalid_input(directory):\n",
    "    input_data = \"\"\"\n",
    "    Invalid, 39.1, 18.7, 181, 3750, MALE\n",
    "    \"\"\"\n",
    "    \n",
    "    model = model_fn(str(directory))\n",
    "    df = input_fn(input_data, \"text/csv\")\n",
    "    assert predict_fn(df, model) is None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fb90311",
   "metadata": {},
   "source": [
    "### Postprocessing Script\n",
    "\n",
    "The final component of our inference pipeline is a transformer that will transform the output from the model into a human-readable format. We'll use the Scikit-Learn target transformer we saved when we split and transformed the data. To deploy this transformer as part of an inference pipeline, we need to write a script that loads the transformer, uses it to modify the output from the model, and returns a human-readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "28cd8c25",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting code/inference/postprocessing_component.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {INFERENCE_CODE_FOLDER}/postprocessing_component.py\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import argparse\n",
    "import json\n",
    "import tarfile\n",
    "import joblib\n",
    "\n",
    "from pathlib import Path\n",
    "from io import StringIO\n",
    "\n",
    "from sklearn.compose import ColumnTransformer, make_column_selector\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder, StandardScaler, OrdinalEncoder\n",
    "from pickle import dump, load\n",
    "\n",
    "\n",
    "try:\n",
    "    from sagemaker_containers.beta.framework import encoders, worker\n",
    "except ImportError:\n",
    "    # We don't have access to the `worker` instance when testing locally. \n",
    "    # We'll set it to None so we can change the way functions create a response.\n",
    "    worker = None\n",
    "\n",
    "\n",
    "def input_fn(input_data, content_type):\n",
    "    if content_type == \"application/json\":\n",
    "        predictions = json.loads(input_data)[\"predictions\"]\n",
    "        return predictions\n",
    "    \n",
    "    else:\n",
    "        raise ValueError(f\"{content_type} is not supported.!\")\n",
    "\n",
    "\n",
    "def output_fn(prediction, accept):\n",
    "    if accept == \"text/csv\":\n",
    "        return worker.Response(encoders.encode(prediction, accept), mimetype=accept) if worker else (prediction, accept)\n",
    "    \n",
    "    if accept == \"application/json\":\n",
    "        response = []\n",
    "        for p, c in prediction:\n",
    "            response.append({\n",
    "                \"prediction\": p,\n",
    "                \"confidence\": c\n",
    "            })\n",
    "            \n",
    "        return worker.Response(json.dumps(response), mimetype=accept) if worker else (response, accept)\n",
    "    \n",
    "    raise RuntimeException(f\"{accept} accept type is not supported.\")\n",
    "\n",
    "\n",
    "def predict_fn(input_data, model):\n",
    "    \"\"\"\n",
    "    Transforms the prediction into its corresponding category.\n",
    "    \"\"\"\n",
    "    \n",
    "    predictions = np.argmax(input_data, axis=-1)\n",
    "    confidence = np.max(input_data, axis=-1)\n",
    "    return [(confidence, model[prediction]) for confidence, prediction in zip(confidence, predictions)]\n",
    "\n",
    "\n",
    "def model_fn(model_dir):\n",
    "    \"\"\"\n",
    "    Deserializes the target model and returns the list of fitted categories.\n",
    "    \"\"\"\n",
    "    \n",
    "    model = joblib.load(os.path.join(model_dir, \"target.joblib\"))\n",
    "    return model.named_transformers_[\"species\"].categories_[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58cccf41",
   "metadata": {},
   "source": [
    "Let's test the script to ensure everything is working as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "id": "74891b6f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m.\u001b[0m\u001b[32m                                                                                            [100%]\u001b[0m\n",
      "\u001b[32m\u001b[32m\u001b[1m1 passed\u001b[0m\u001b[32m in 0.01s\u001b[0m\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%%ipytest\n",
    "\n",
    "import json\n",
    "import numpy as np\n",
    "\n",
    "from postprocessing_component import predict_fn\n",
    "\n",
    "\n",
    "def test_predict_returns_prediction_as_last_column():\n",
    "    input_data = [\n",
    "        [0.6, 0.2, 0.2], \n",
    "        [0.1, 0.8, 0.1],\n",
    "        [0.2, 0.1, 0.7]\n",
    "    ]\n",
    "    \n",
    "    categories = [\"Adelie\", \"Gentoo\", \"Chinstrap\"]\n",
    "    \n",
    "    response = predict_fn(input_data, categories)\n",
    "    \n",
    "    assert response == [\n",
    "        (0.6, \"Adelie\"),\n",
    "        (0.8, \"Gentoo\"),\n",
    "        (0.7, \"Chinstrap\")\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a18f942",
   "metadata": {},
   "source": [
    "### Pipeline Model\n",
    "\n",
    "We can now create a [PipelineModel](https://sagemaker.readthedocs.io/en/stable/api/inference/pipeline.html#sagemaker.pipeline.PipelineModel) to define our inference pipeline. This Pipeline Model will create a SageMaker Model in the background and use the preprocessing and postprocessing scripts to transform the input and output of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "id": "27d3aef6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.functions import Join\n",
    "from sagemaker.sklearn.model import SKLearnModel\n",
    "from sagemaker.tensorflow.model import TensorFlowModel\n",
    "from sagemaker.pipeline import PipelineModel\n",
    "\n",
    "\n",
    "# We'll use the model we generated from the first step of the\n",
    "# pipeline as the input to the first and last components of the\n",
    "# inference pipeline. This model.tar.gz file contains the two\n",
    "# transformers we need to preprocess and postprocess the data.\n",
    "transformation_pipeline_model = Join(\n",
    "    on=\"/\",\n",
    "    values=[\n",
    "        split_and_transform_data_step.properties.ProcessingOutputConfig.Outputs[\n",
    "            \"model\"\n",
    "        ].S3Output.S3Uri,\n",
    "        \"model.tar.gz\",\n",
    "    ],\n",
    ")\n",
    "\n",
    "# This is the first component of the inference pipeline. It will\n",
    "# preprocess the data before sending it to the TensorFlow model.\n",
    "preprocessing_model = SKLearnModel(\n",
    "    model_data=transformation_pipeline_model,\n",
    "    entry_point=\"preprocessing_component.py\",\n",
    "    source_dir=str(INFERENCE_CODE_FOLDER),\n",
    "    framework_version=\"1.2-1\",\n",
    "    sagemaker_session=config[\"session\"],\n",
    "    role=config[\"role\"],\n",
    ")\n",
    "\n",
    "tensorflow_model = TensorFlowModel(\n",
    "    model_data=(\n",
    "        tune_model_step.get_top_model_s3_uri(\n",
    "            top_k=0, s3_bucket=config[\"session\"].default_bucket()\n",
    "        )\n",
    "        if USE_TUNING_STEP\n",
    "        else train_model_step.properties.ModelArtifacts.S3ModelArtifacts\n",
    "    ),\n",
    "    image_uri=config[\"image\"],\n",
    "    framework_version=config[\"framework_version\"],\n",
    "    sagemaker_session=config[\"session\"],\n",
    "    role=config[\"role\"],\n",
    ")\n",
    "\n",
    "# This is the last component of the inference pipeline. It will\n",
    "# postprocess the output from the TensorFlow model before sending \n",
    "# it back to the user.\n",
    "post_processing_model = SKLearnModel(\n",
    "    model_data=transformation_pipeline_model,\n",
    "    entry_point=\"postprocessing_component.py\",\n",
    "    source_dir=str(INFERENCE_CODE_FOLDER),\n",
    "    framework_version=\"1.2-1\",\n",
    "    sagemaker_session=config[\"session\"],\n",
    "    role=config[\"role\"],\n",
    ")\n",
    "\n",
    "# We can now create the inference pipeline using the three models.\n",
    "pipeline_model = PipelineModel(\n",
    "    name=\"inference-model\",\n",
    "    models=[preprocessing_model, tensorflow_model, post_processing_model],\n",
    "    sagemaker_session=config[\"session\"],\n",
    "    role=config[\"role\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e626107",
   "metadata": {},
   "source": [
    "## Data Quality Baseline\n",
    "\n",
    "We need to create a data quality baseline to compare against the real-time traffic our endpoint receives. This baseline will help us detect any data dristribution shifts.\n",
    "\n",
    "We'll use a [Quality Check Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-quality-check) to generate the baseline and configure the instance that will run the quality check using the [CheckJobConfig](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#checkjobconfig) class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "id": "8275ec63",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: .\n",
      "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.workflow.check_job_config import CheckJobConfig\n",
    "from sagemaker.workflow.quality_check_step import (\n",
    "    DataQualityCheckConfig,\n",
    "    QualityCheckStep,\n",
    ")\n",
    "from sagemaker.model_monitor.dataset_format import DatasetFormat\n",
    "\n",
    "\n",
    "DATA_QUALITY_LOCATION = f\"{S3_LOCATION}/monitoring/data-quality\"\n",
    "\n",
    "\n",
    "data_quality_baseline_step = QualityCheckStep(\n",
    "    name=\"generate-data-quality-baseline\",\n",
    "    check_job_config=CheckJobConfig(\n",
    "        instance_type=\"ml.c5.xlarge\",\n",
    "        instance_count=1,\n",
    "        volume_size_in_gb=20,\n",
    "        sagemaker_session=pipeline_session,\n",
    "        role=EXECUTION_ROLE,\n",
    "    ),\n",
    "    quality_check_config=DataQualityCheckConfig(\n",
    "        baseline_dataset=f\"{S3_LOCATION}/data\",\n",
    "        dataset_format=DatasetFormat.csv(header=True, output_columns_position=\"END\"),\n",
    "        output_s3_uri=DATA_QUALITY_LOCATION,\n",
    "    ),\n",
    "    skip_check=True,\n",
    "    register_new_baseline=True,\n",
    "    cache_config=cache_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0801286",
   "metadata": {},
   "source": [
    "## Model Quality Baseline\n",
    "\n",
    "To monitor the performance of the model, we need to generate a baseline performance. This baseline will help us detect any performance degradation.\n",
    "\n",
    "To create the baseline, we must generate predictions for the test set and compare them with the predictions from the model. We can do this by running a [Batch Transform Job](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) to generate predictions for every sample from the test set. We can use a [Transform Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-transform) as part of the pipeline to run this job. We'll configure the [Batch Transform Job](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) using a [Transform Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-transform)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "id": "1f74837d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/svpino/dev/ml.school/.venv/lib/python3.9/site-packages/sagemaker/workflow/pipeline_context.py:297: UserWarning:\n",
      "\n",
      "Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "\n",
      "INFO:sagemaker.tensorflow.model:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.workflow.steps import TransformStep\n",
    "from sagemaker.workflow.model_step import ModelStep\n",
    "from sagemaker.transformer import Transformer\n",
    "\n",
    "# The Transform Step requires a model to generate predictions, \n",
    "# so we need to create the inference pipeline model.\n",
    "create_model_step = ModelStep(\n",
    "    name=\"create\",\n",
    "    display_name=\"create-model\",\n",
    "    step_args=pipeline_model.create(\n",
    "        instance_type=\"ml.m5.xlarge\"\n",
    "    ),\n",
    ")\n",
    "\n",
    "transformer = Transformer(\n",
    "    model_name=create_model_step.properties.ModelName,\n",
    "    instance_type=config[\"instance_type\"],\n",
    "    instance_count=1,\n",
    "    strategy=\"MultiRecord\",\n",
    "    accept=\"text/csv\",\n",
    "    assemble_with=\"Line\",\n",
    "    output_path=f\"{S3_LOCATION}/transform\",\n",
    "    sagemaker_session=config[\"session\"]\n",
    ")\n",
    "\n",
    "generate_test_predictions_step = TransformStep(\n",
    "    name=\"generate-test-predictions\",\n",
    "    step_args=transformer.transform(\n",
    "        # We will use the baseline set we generated when we split the data.\n",
    "        # This set corresponds to the test split before the transformation step.\n",
    "        data=split_and_transform_data_step.properties.ProcessingOutputConfig.Outputs[\"baseline\"].S3Output.S3Uri,\n",
    "        \n",
    "        join_source=\"Input\",\n",
    "        split_type=\"Line\",\n",
    "        content_type=\"text/csv\",\n",
    "        input_filter=\"$\",\n",
    "        \n",
    "        # We want to output the first and the last field from the joint set.\n",
    "        # The first field corresponds to the groundtruth, and the last field\n",
    "        # corresponds to the prediction.\n",
    "        output_filter=\"$[0,-1]\",\n",
    "    ),\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10423fce",
   "metadata": {},
   "source": [
    "Let's now configure the [Quality Check Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-quality-check) and feed it the data we generated in the Transform Step.\n",
    "\n",
    "We'll use a [Model Quality Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-model-quality) to generate the baseline, and we'll configure the instance that will run the quality check using the [ModelQualityJobConfig](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#modelqualityjobconfig) class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "id": "b1226ccb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:Defaulting to the only supported framework/algorithm version: .\n",
      "INFO:sagemaker.image_uris:Ignoring unnecessary instance type: None.\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.workflow.quality_check_step import ModelQualityCheckConfig\n",
    "\n",
    "\n",
    "MODEL_QUALITY_LOCATION = f\"{S3_LOCATION}/monitoring/model-quality\"\n",
    "\n",
    "model_quality_baseline_step = QualityCheckStep(\n",
    "    name=\"generate-model-quality-baseline\",\n",
    "    \n",
    "    check_job_config = CheckJobConfig(\n",
    "        instance_type=\"ml.c5.xlarge\",\n",
    "        instance_count=1,\n",
    "        volume_size_in_gb=20,\n",
    "        sagemaker_session=pipeline_session,\n",
    "        role=EXECUTION_ROLE,\n",
    "    ),\n",
    "    \n",
    "    quality_check_config = ModelQualityCheckConfig(\n",
    "        # We are going to use the output of the Transform Step to generate\n",
    "        # the model quality baseline.\n",
    "        baseline_dataset=generate_test_predictions_step.properties.TransformOutput.S3OutputPath,\n",
    "        dataset_format=DatasetFormat.csv(header=False),\n",
    "\n",
    "        # We need to specify the problem type and the fields where the prediction\n",
    "        # and groundtruth are so the process knows how to interpret the results.\n",
    "        problem_type=\"MulticlassClassification\",\n",
    "        \n",
    "        # Since the data doesn't have headers, SageMaker will autocreate headers for it.\n",
    "        # _c0 corresponds to the first column, and _c1 corresponds to the second column.\n",
    "        ground_truth_attribute=\"_c0\",\n",
    "        inference_attribute=\"_c1\",\n",
    "\n",
    "        output_s3_uri=MODEL_QUALITY_LOCATION,\n",
    "    ),\n",
    "    \n",
    "    skip_check=True,\n",
    "    register_new_baseline=True,\n",
    "    cache_config=cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e712f3",
   "metadata": {},
   "source": [
    "## Registration\n",
    "\n",
    "We can now register the inference pipeline in the Model Registry.\n",
    "\n",
    "When we register a model, we can specify a set of [ModelMetrics](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_metrics.ModelMetrics) that will be saved in the Model Registry. We'll use the metrics that we calculated using the Quality Check Steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "id": "a66aca4f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.model_metrics import MetricsSource, ModelMetrics \n",
    "from sagemaker.drift_check_baselines import DriftCheckBaselines\n",
    "\n",
    "\n",
    "model_metrics = ModelMetrics(\n",
    "    model_data_statistics=MetricsSource(\n",
    "        s3_uri=data_quality_baseline_step.properties.CalculatedBaselineStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_data_constraints=MetricsSource(\n",
    "        s3_uri=data_quality_baseline_step.properties.CalculatedBaselineConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=model_quality_baseline_step.properties.CalculatedBaselineStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    \n",
    "    model_constraints=MetricsSource(\n",
    "        s3_uri=model_quality_baseline_step.properties.CalculatedBaselineConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    ")\n",
    "\n",
    "drift_check_baselines = DriftCheckBaselines(\n",
    "    model_data_statistics=MetricsSource(\n",
    "        s3_uri=data_quality_baseline_step.properties.BaselineUsedForDriftCheckStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_data_constraints=MetricsSource(\n",
    "        s3_uri=data_quality_baseline_step.properties.BaselineUsedForDriftCheckConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=model_quality_baseline_step.properties.BaselineUsedForDriftCheckStatistics,\n",
    "        content_type=\"application/json\",\n",
    "    ),\n",
    "    model_constraints=MetricsSource(\n",
    "        s3_uri=model_quality_baseline_step.properties.BaselineUsedForDriftCheckConstraints,\n",
    "        content_type=\"application/json\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8adbb84c",
   "metadata": {},
   "source": [
    "Let's use a [ModelStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.model_step.ModelStep) to register the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "id": "661ecb08",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.tensorflow.model:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    }
   ],
   "source": [
    "MODEL_PACKAGE_GROUP = \"penguins\"\n",
    "\n",
    "register_model_step = ModelStep(\n",
    "    name=\"register\",\n",
    "    display_name=\"register-model\",\n",
    "    step_args=pipeline_model.register(\n",
    "        model_package_group_name=MODEL_PACKAGE_GROUP,\n",
    "        model_metrics=model_metrics,\n",
    "        drift_check_baselines=drift_check_baselines,\n",
    "        approval_status=\"PendingManualApproval\",\n",
    "        \n",
    "        # Our inference pipeline model supports two content \n",
    "        # types: text/csv and application/json.\n",
    "        content_types=[\"text/csv\", \"application/json\"],\n",
    "        response_types=[\"text/csv\", \"application/json\"],       \n",
    "\n",
    "        # This is the suggested inference instance types when \n",
    "        # deploying the model or using it as part of a batch\n",
    "        # transform job.\n",
    "        inference_instances=[\"ml.m5.xlarge\"],\n",
    "        transform_instances=[\"ml.m5.xlarge\"],\n",
    "        \n",
    "        domain=\"MACHINE_LEARNING\",\n",
    "        task=\"CLASSIFICATION\",\n",
    "        framework=\"TENSORFLOW\",\n",
    "        framework_version=config[\"framework_version\"],\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c110f7-fe72-4db8-9d06-cfb9a0f2bfbd",
   "metadata": {},
   "source": [
    "## Condition Step\n",
    "\n",
    "We only want to register the model and generate the baseline predictions if the model's accuracy exceeds a predefined threshold. We can use a [Condition Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-condition) together with the evaluation report we generated to accomplish this. Check the [ConditionStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#conditionstep) SageMaker's SDK documentation for more information. In this example, we will use a [ConditionGreaterThanOrEqualTo](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.conditions.ConditionGreaterThanOrEqualTo) condition to compare the model's accuracy with the threshold. Look at the [Conditions](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_building_pipeline.html#conditions) section in the documentation for more information about the types of supported conditions.\n",
    "\n",
    "If the model's accuracy is not greater than or equal our threshold, we will send the pipeline to a [Fail Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-fail) with the appropriate error message. Check the [FailStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html#sagemaker.workflow.fail_step.FailStep) SageMaker's SDK documentation for more information.\n",
    "\n",
    "We are going to use a new [Pipeline Parameter](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-parameters.html) in our pipeline to specify the minimum accuracy that the model should reach for it to be registered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "id": "36e2a2b1-6711-4266-95d8-d2aebd52e199",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.fail_step import FailStep\n",
    "from sagemaker.workflow.functions import JsonGet\n",
    "from sagemaker.workflow.functions import Join\n",
    "from sagemaker.workflow.parameters import ParameterFloat\n",
    "from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo\n",
    "from sagemaker.workflow.condition_step import ConditionStep\n",
    "\n",
    "\n",
    "accuracy_threshold = ParameterFloat(name=\"accuracy_threshold\", default_value=0.70)\n",
    "\n",
    "condition_step = ConditionStep(\n",
    "    name=\"check-model-accuracy\",\n",
    "    conditions=[\n",
    "        # We want to check whether the accuracy of the model is greater than\n",
    "        # the threshold we defined.\n",
    "        ConditionGreaterThanOrEqualTo(\n",
    "            left=JsonGet(\n",
    "                step_name=evaluate_model_step.name,\n",
    "                property_file=evaluation_report,\n",
    "                json_path=\"metrics.accuracy.value\",\n",
    "            ),\n",
    "            right=accuracy_threshold,\n",
    "        )\n",
    "    ],\n",
    "    if_steps=[]\n",
    "    if LOCAL_MODE\n",
    "    else [\n",
    "        create_model_step,\n",
    "        generate_test_predictions_step,\n",
    "        model_quality_baseline_step,\n",
    "        register_model_step,\n",
    "    ],\n",
    "    # If the condition is not met, we want to fail the execution\n",
    "    # of the pipeline by sending it to a FailStep.\n",
    "    else_steps=[\n",
    "        FailStep(\n",
    "            name=\"fail\",\n",
    "            error_message=Join(\n",
    "                on=\" \",\n",
    "                values=[\n",
    "                    \"Execution failed because the model's accuracy was lower than\",\n",
    "                    accuracy_threshold,\n",
    "                ],\n",
    "            ),\n",
    "        )\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b9a3854",
   "metadata": {},
   "source": [
    "## Pipeline\n",
    "\n",
    "We can now create the SageMaker Pipeline and submit its definition to the SageMaker Pipelines service to create the pipeline if it doesn't exist or update it if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "id": "e143c13a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.processing:Uploaded None to s3://mlschool/cohort-pipeline/code/20c36254c1a14f23578c8c08d55a36e4/sourcedir.tar.gz\n",
      "INFO:sagemaker.processing:runproc.sh uploaded to s3://mlschool/cohort-pipeline/code/2c207c809cb0e0e9a1d77e5247f961f9/runproc.sh\n",
      "WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.\n",
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.processing:Uploaded None to s3://mlschool/cohort-pipeline/code/20c36254c1a14f23578c8c08d55a36e4/sourcedir.tar.gz\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.processing:runproc.sh uploaded to s3://mlschool/cohort-pipeline/code/2c207c809cb0e0e9a1d77e5247f961f9/runproc.sh\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'PipelineArn': 'arn:aws:sagemaker:us-east-1:325223348818:pipeline/cohort-pipeline',\n",
       " 'ResponseMetadata': {'RequestId': 'b51673e7-866f-4768-8796-40b8288d0656',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'b51673e7-866f-4768-8796-40b8288d0656',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '83',\n",
       "   'date': 'Thu, 19 Oct 2023 18:20:59 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "from sagemaker.workflow.pipeline_definition_config import PipelineDefinitionConfig\n",
    "\n",
    "\n",
    "pipeline = Pipeline(\n",
    "    name=\"cohort-pipeline\",\n",
    "    parameters=[\n",
    "        dataset_location,\n",
    "        accuracy_threshold,\n",
    "    ],\n",
    "    steps=[\n",
    "        split_and_transform_data_step,\n",
    "        train_model_step if not USE_TUNING_STEP else tune_model_step,\n",
    "        evaluate_model_step,\n",
    "        condition_step\n",
    "    ],\n",
    "    pipeline_definition_config=PipelineDefinitionConfig(use_custom_job_prefix=True),\n",
    "    sagemaker_session=config[\"session\"],\n",
    ")\n",
    "\n",
    "if not LOCAL_MODE:\n",
    "    # SageMaker doesn't support running any of these steps in Local Mode.\n",
    "    pipeline.steps.extend([data_quality_baseline_step])\n",
    "\n",
    "pipeline.upsert(role_arn=config[\"role\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e78dc888",
   "metadata": {},
   "source": [
    "<div class=\"alert\" style=\"background-color:#0066cc;\">To run the pipeline, comment out the <code style=\"background-color:#0066cc;\">%%script</code> cell magic line to execute the cell.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "id": "6f2c8a0c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_PipelineExecution(arn='arn:aws:sagemaker:us-east-1:325223348818:pipeline/cohort-pipeline/execution/tdzvhhg55yez', sagemaker_session=<sagemaker.workflow.pipeline_context.PipelineSession object at 0x3161532b0>)"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# %%script false --no-raise-error\n",
    "\n",
    "pipeline.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "520e87d5-634b-4833-a1da-ac106eae5ebe",
   "metadata": {},
   "source": [
    "## Quality Baselines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "384ad43d-20a8-4113-b75a-d86ce7d26ab6",
   "metadata": {},
   "source": [
    "Our pipeline generated data baseline statistics and constraints using our train set. We can take a look at what these values look like by downloading them from S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "id": "d882526e-e52d-4f1f-84d0-b3fb2edf2b9e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/json": {
       "dataset": {
        "item_count": 395
       },
       "features": [
        {
         "inferred_type": "String",
         "name": "species",
         "string_statistics": {
          "common": {
           "num_missing": 0,
           "num_present": 395
          },
          "distinct_count": 3,
          "distribution": {
           "categorical": {
            "buckets": [
             {
              "count": 171,
              "value": "Adelie"
             },
             {
              "count": 75,
              "value": "Chinstrap"
             },
             {
              "count": 149,
              "value": "Gentoo"
             }
            ]
           }
          }
         }
        },
        {
         "inferred_type": "String",
         "name": "island",
         "string_statistics": {
          "common": {
           "num_missing": 0,
           "num_present": 395
          },
          "distinct_count": 6,
          "distribution": {
           "categorical": {
            "buckets": [
             {
              "count": 52,
              "value": "Torgersen"
             },
             {
              "count": 168,
              "value": "Biscoe"
             },
             {
              "count": 124,
              "value": "Dream"
             },
             {
              "count": 19,
              "value": "Adelie"
             },
             {
              "count": 26,
              "value": "Gentoo"
             },
             {
              "count": 6,
              "value": "Chinstrap"
             }
            ]
           }
          }
         }
        },
        {
         "inferred_type": "String",
         "name": "culmen_length_mm",
         "string_statistics": {
          "common": {
           "num_missing": 51,
           "num_present": 344
          },
          "distinct_count": 162
         }
        },
        {
         "inferred_type": "String",
         "name": "culmen_depth_mm",
         "string_statistics": {
          "common": {
           "num_missing": 51,
           "num_present": 344
          },
          "distinct_count": 80,
          "distribution": {
           "categorical": {
            "buckets": [
             {
              "count": 2,
              "value": "21.2"
             },
             {
              "count": 6,
              "value": "16.6"
             },
             {
              "count": 9,
              "value": "17.8"
             },
             {
              "count": 6,
              "value": "19.5"
             },
             {
              "count": 2,
              "value": "16.4"
             },
             {
              "count": 5,
              "value": "19.1"
             },
             {
              "count": 9,
              "value": "19"
             },
             {
              "count": 4,
              "value": "15.3"
             },
             {
              "count": 5,
              "value": "14.4"
             },
             {
              "count": 9,
              "value": "17.3"
             },
             {
              "count": 5,
              "value": "15.9"
             },
             {
              "count": 3,
              "value": "19.8"
             },
             {
              "count": 6,
              "value": "16.1"
             },
             {
              "count": 1,
              "value": "13.6"
             },
             {
              "count": 2,
              "value": "15.5"
             },
             {
              "count": 4,
              "value": "18.3"
             },
             {
              "count": 10,
              "value": "15"
             },
             {
              "count": 1,
              "value": "20.2"
             },
             {
              "count": 51,
              "value": "NullValue"
             },
             {
              "count": 2,
              "value": "14.7"
             },
             {
              "count": 1,
              "value": "20.6"
             },
             {
              "count": 10,
              "value": "17.9"
             },
             {
              "count": 2,
              "value": "NA"
             },
             {
              "count": 6,
              "value": "17.2"
             },
             {
              "count": 2,
              "value": "17.7"
             },
             {
              "count": 4,
              "value": "13.9"
             },
             {
              "count": 2,
              "value": "16.7"
             },
             {
              "count": 2,
              "value": "13.5"
             },
             {
              "count": 3,
              "value": "14.8"
             },
             {
              "count": 1,
              "value": "20.5"
             },
             {
              "count": 8,
              "value": "14.5"
             },
             {
              "count": 6,
              "value": "14.2"
             },
             {
              "count": 4,
              "value": "16"
             },
             {
              "count": 5,
              "value": "19.4"
             },
             {
              "count": 3,
              "value": "15.6"
             },
             {
              "count": 10,
              "value": "18.6"
             },
             {
              "count": 3,
              "value": "19.9"
             },
             {
              "count": 1,
              "value": "13.2"
             },
             {
              "count": 1,
              "value": "20.1"
             },
             {
              "count": 1,
              "value": "14.9"
             },
             {
              "count": 10,
              "value": "18.5"
             },
             {
              "count": 5,
              "value": "18.2"
             },
             {
              "count": 8,
              "value": "17.1"
             },
             {
              "count": 5,
              "value": "14.6"
             },
             {
              "count": 12,
              "value": "17"
             },
             {
              "count": 3,
              "value": "14.1"
             },
             {
              "count": 1,
              "value": "13.1"
             },
             {
              "count": 2,
              "value": "14"
             },
             {
              "count": 4,
              "value": "16.8"
             },
             {
              "count": 4,
              "value": "17.6"
             },
             {
              "count": 2,
              "value": "19.3"
             },
             {
              "count": 1,
              "value": "21.5"
             },
             {
              "count": 3,
              "value": "16.2"
             },
             {
              "count": 1,
              "value": "13.3"
             },
             {
              "count": 6,
              "value": "17.5"
             },
             {
              "count": 6,
              "value": "15.7"
             },
             {
              "count": 4,
              "value": "14.3"
             },
             {
              "count": 3,
              "value": "21.1"
             },
             {
              "count": 6,
              "value": "20"
             },
             {
              "count": 7,
              "value": "18.8"
             },
             {
              "count": 6,
              "value": "18.7"
             },
             {
              "count": 4,
              "value": "13.8"
             },
             {
              "count": 4,
              "value": "15.2"
             },
             {
              "count": 1,
              "value": "20.8"
             },
             {
              "count": 4,
              "value": "19.2"
             },
             {
              "count": 3,
              "value": "20.3"
             },
             {
              "count": 4,
              "value": "15.8"
             },
             {
              "count": 2,
              "value": "16.9"
             },
             {
              "count": 5,
              "value": "18"
             },
             {
              "count": 9,
              "value": "18.1"
             },
             {
              "count": 5,
              "value": "18.4"
             },
             {
              "count": 1,
              "value": "17.4"
             },
             {
              "count": 4,
              "value": "16.3"
             },
             {
              "count": 2,
              "value": "19.7"
             },
             {
              "count": 1,
              "value": "13.4"
             },
             {
              "count": 3,
              "value": "19.6"
             },
             {
              "count": 3,
              "value": "15.1"
             },
             {
              "count": 3,
              "value": "20.7"
             },
             {
              "count": 9,
              "value": "18.9"
             },
             {
              "count": 2,
              "value": "15.4"
             },
             {
              "count": 6,
              "value": "13.7"
             },
             {
              "count": 4,
              "value": "16.5"
             }
            ]
           }
          }
         }
        },
        {
         "inferred_type": "String",
         "name": "flipper_length_mm",
         "string_statistics": {
          "common": {
           "num_missing": 51,
           "num_present": 344
          },
          "distinct_count": 56,
          "distribution": {
           "categorical": {
            "buckets": [
             {
              "count": 3,
              "value": "205"
             },
             {
              "count": 15,
              "value": "193"
             },
             {
              "count": 3,
              "value": "182"
             },
             {
              "count": 2,
              "value": "223"
             },
             {
              "count": 8,
              "value": "216"
             },
             {
              "count": 7,
              "value": "181"
             },
             {
              "count": 51,
              "value": "NullValue"
             },
             {
              "count": 6,
              "value": "201"
             },
             {
              "count": 10,
              "value": "197"
             },
             {
              "count": 2,
              "value": "NA"
             },
             {
              "count": 7,
              "value": "186"
             },
             {
              "count": 7,
              "value": "212"
             },
             {
              "count": 3,
              "value": "224"
             },
             {
              "count": 1,
              "value": "174"
             },
             {
              "count": 6,
              "value": "217"
             },
             {
              "count": 1,
              "value": "206"
             },
             {
              "count": 5,
              "value": "209"
             },
             {
              "count": 7,
              "value": "192"
             },
             {
              "count": 1,
              "value": "231"
             },
             {
              "count": 6,
              "value": "199"
             },
             {
              "count": 10,
              "value": "196"
             },
             {
              "count": 4,
              "value": "228"
             },
             {
              "count": 6,
              "value": "188"
             },
             {
              "count": 6,
              "value": "213"
             },
             {
              "count": 4,
              "value": "202"
             },
             {
              "count": 9,
              "value": "185"
             },
             {
              "count": 5,
              "value": "219"
             },
             {
              "count": 5,
              "value": "180"
             },
             {
              "count": 5,
              "value": "221"
             },
             {
              "count": 8,
              "value": "208"
             },
             {
              "count": 7,
              "value": "230"
             },
             {
              "count": 13,
              "value": "191"
             },
             {
              "count": 22,
              "value": "190"
             },
             {
              "count": 7,
              "value": "184"
             },
             {
              "count": 4,
              "value": "225"
             },
             {
              "count": 6,
              "value": "214"
             },
             {
              "count": 5,
              "value": "203"
             },
             {
              "count": 17,
              "value": "195"
             },
             {
              "count": 7,
              "value": "189"
             },
             {
              "count": 4,
              "value": "178"
             },
             {
              "count": 14,
              "value": "210"
             },
             {
              "count": 8,
              "value": "198"
             },
             {
              "count": 16,
              "value": "187"
             },
             {
              "count": 5,
              "value": "194"
             },
             {
              "count": 12,
              "value": "215"
             },
             {
              "count": 8,
              "value": "220"
             },
             {
              "count": 2,
              "value": "183"
             },
             {
              "count": 1,
              "value": "179"
             },
             {
              "count": 5,
              "value": "218"
             },
             {
              "count": 6,
              "value": "222"
             },
             {
              "count": 2,
              "value": "207"
             },
             {
              "count": 2,
              "value": "211"
             },
             {
              "count": 1,
              "value": "226"
             },
             {
              "count": 1,
              "value": "172"
             },
             {
              "count": 2,
              "value": "229"
             },
             {
              "count": 4,
              "value": "200"
             },
             {
              "count": 1,
              "value": "176"
             }
            ]
           }
          }
         }
        },
        {
         "inferred_type": "String",
         "name": "body_mass_g",
         "string_statistics": {
          "common": {
           "num_missing": 51,
           "num_present": 344
          },
          "distinct_count": 98,
          "distribution": {
           "categorical": {
            "buckets": [
             {
              "count": 6,
              "value": "4150"
             },
             {
              "count": 2,
              "value": "5950"
             },
             {
              "count": 2,
              "value": "3675"
             },
             {
              "count": 4,
              "value": "5200"
             },
             {
              "count": 2,
              "value": "4550"
             },
             {
              "count": 3,
              "value": "3475"
             },
             {
              "count": 8,
              "value": "4300"
             },
             {
              "count": 1,
              "value": "4475"
             },
             {
              "count": 8,
              "value": "3450"
             },
             {
              "count": 5,
              "value": "3200"
             },
             {
              "count": 3,
              "value": "5350"
             },
             {
              "count": 2,
              "value": "4950"
             },
             {
              "count": 2,
              "value": "6000"
             },
             {
              "count": 6,
              "value": "3300"
             },
             {
              "count": 2,
              "value": "5600"
             },
             {
              "count": 1,
              "value": "2925"
             },
             {
              "count": 3,
              "value": "5100"
             },
             {
              "count": 1,
              "value": "3575"
             },
             {
              "count": 1,
              "value": "4575"
             },
             {
              "count": 1,
              "value": "4075"
             },
             {
              "count": 2,
              "value": "4900"
             },
             {
              "count": 7,
              "value": "3600"
             },
             {
              "count": 51,
              "value": "NullValue"
             },
             {
              "count": 2,
              "value": "4925"
             },
             {
              "count": 5,
              "value": "4600"
             },
             {
              "count": 1,
              "value": "4375"
             },
             {
              "count": 4,
              "value": "3150"
             },
             {
              "count": 8,
              "value": "3400"
             },
             {
              "count": 10,
              "value": "3900"
             },
             {
              "count": 2,
              "value": "NA"
             },
             {
              "count": 3,
              "value": "5850"
             },
             {
              "count": 2,
              "value": "3175"
             },
             {
              "count": 5,
              "value": "4200"
             },
             {
              "count": 1,
              "value": "2975"
             },
             {
              "count": 3,
              "value": "4800"
             },
             {
              "count": 1,
              "value": "3825"
             },
             {
              "count": 1,
              "value": "3100"
             },
             {
              "count": 1,
              "value": "5750"
             },
             {
              "count": 7,
              "value": "3500"
             },
             {
              "count": 5,
              "value": "3325"
             },
             {
              "count": 1,
              "value": "5450"
             },
             {
              "count": 5,
              "value": "3250"
             },
             {
              "count": 3,
              "value": "5050"
             },
             {
              "count": 5,
              "value": "5500"
             },
             {
              "count": 1,
              "value": "6050"
             },
             {
              "count": 2,
              "value": "5150"
             },
             {
              "count": 1,
              "value": "3975"
             },
             {
              "count": 2,
              "value": "5800"
             },
             {
              "count": 1,
              "value": "2700"
             },
             {
              "count": 2,
              "value": "3425"
             },
             {
              "count": 3,
              "value": "4500"
             },
             {
              "count": 5,
              "value": "4450"
             },
             {
              "count": 4,
              "value": "5300"
             },
             {
              "count": 5,
              "value": "4750"
             },
             {
              "count": 3,
              "value": "3725"
             },
             {
              "count": 6,
              "value": "3650"
             },
             {
              "count": 12,
              "value": "3800"
             },
             {
              "count": 3,
              "value": "5250"
             },
             {
              "count": 5,
              "value": "4250"
             },
             {
              "count": 3,
              "value": "4725"
             },
             {
              "count": 4,
              "value": "2900"
             },
             {
              "count": 1,
              "value": "3275"
             },
             {
              "count": 8,
              "value": "4400"
             },
             {
              "count": 4,
              "value": "4850"
             },
             {
              "count": 5,
              "value": "3350"
             },
             {
              "count": 5,
              "value": "3750"
             },
             {
              "count": 4,
              "value": "3050"
             },
             {
              "count": 1,
              "value": "6300"
             },
             {
              "count": 5,
              "value": "5400"
             },
             {
              "count": 1,
              "value": "4675"
             },
             {
              "count": 1,
              "value": "4775"
             },
             {
              "count": 5,
              "value": "4000"
             },
             {
              "count": 1,
              "value": "4975"
             },
             {
              "count": 9,
              "value": "3550"
             },
             {
              "count": 6,
              "value": "4700"
             },
             {
              "count": 3,
              "value": "5650"
             },
             {
              "count": 2,
              "value": "3000"
             },
             {
              "count": 2,
              "value": "4625"
             },
             {
              "count": 5,
              "value": "4650"
             },
             {
              "count": 5,
              "value": "4100"
             },
             {
              "count": 5,
              "value": "5700"
             },
             {
              "count": 1,
              "value": "3875"
             },
             {
              "count": 10,
              "value": "3950"
             },
             {
              "count": 2,
              "value": "4350"
             },
             {
              "count": 11,
              "value": "3700"
             },
             {
              "count": 2,
              "value": "2850"
             },
             {
              "count": 1,
              "value": "3075"
             },
             {
              "count": 4,
              "value": "3775"
             },
             {
              "count": 1,
              "value": "3850"
             },
             {
              "count": 6,
              "value": "4050"
             },
             {
              "count": 6,
              "value": "5000"
             },
             {
              "count": 1,
              "value": "4275"
             },
             {
              "count": 6,
              "value": "5550"
             },
             {
              "count": 3,
              "value": "4875"
             },
             {
              "count": 2,
              "value": "3525"
             },
             {
              "count": 1,
              "value": "3625"
             }
            ]
           }
          }
         }
        },
        {
         "inferred_type": "String",
         "name": "sex",
         "string_statistics": {
          "common": {
           "num_missing": 51,
           "num_present": 344
          },
          "distinct_count": 4,
          "distribution": {
           "categorical": {
            "buckets": [
             {
              "count": 1,
              "value": "."
             },
             {
              "count": 51,
              "value": "NullValue"
             },
             {
              "count": 10,
              "value": "NA"
             },
             {
              "count": 168,
              "value": "MALE"
             },
             {
              "count": 165,
              "value": "FEMALE"
             }
            ]
           }
          }
         }
        }
       ],
       "version": 0
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 173,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "from IPython.display import JSON\n",
    "from sagemaker.s3 import S3Downloader\n",
    "\n",
    "\n",
    "statistics = f\"{DATA_QUALITY_LOCATION}/statistics.json\"\n",
    "\n",
    "response = None\n",
    "try:\n",
    "    response = json.loads(S3Downloader.read_file(statistics))\n",
    "except Exception as e:\n",
    "    pass\n",
    "\n",
    "JSON(response or {})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "id": "0a8a3276-46fe-484a-a8b8-d60ad05d63fc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/json": {
       "features": [
        {
         "completeness": 1,
         "inferred_type": "String",
         "name": "species",
         "string_constraints": {
          "domains": [
           "Adelie",
           "Chinstrap",
           "Gentoo"
          ]
         }
        },
        {
         "completeness": 1,
         "inferred_type": "String",
         "name": "island",
         "string_constraints": {
          "domains": [
           "Torgersen",
           "Biscoe",
           "Dream",
           "Adelie",
           "Gentoo",
           "Chinstrap"
          ]
         }
        },
        {
         "completeness": 0.8708860759493671,
         "inferred_type": "String",
         "name": "culmen_length_mm"
        },
        {
         "completeness": 0.8708860759493671,
         "inferred_type": "String",
         "name": "culmen_depth_mm",
         "string_constraints": {
          "domains": [
           "21.2",
           "16.6",
           "17.8",
           "19.5",
           "16.4",
           "19.1",
           "19",
           "15.3",
           "14.4",
           "17.3",
           "15.9",
           "19.8",
           "16.1",
           "13.6",
           "15.5",
           "18.3",
           "15",
           "20.2",
           "NullValue",
           "14.7",
           "20.6",
           "17.9",
           "NA",
           "17.2",
           "17.7",
           "13.9",
           "16.7",
           "13.5",
           "14.8",
           "20.5",
           "14.5",
           "14.2",
           "16",
           "19.4",
           "15.6",
           "18.6",
           "19.9",
           "13.2",
           "20.1",
           "14.9",
           "18.5",
           "18.2",
           "17.1",
           "14.6",
           "17",
           "14.1",
           "13.1",
           "14",
           "16.8",
           "17.6",
           "19.3",
           "21.5",
           "16.2",
           "13.3",
           "17.5",
           "15.7",
           "14.3",
           "21.1",
           "20",
           "18.8",
           "18.7",
           "13.8",
           "15.2",
           "20.8",
           "19.2",
           "20.3",
           "15.8",
           "16.9",
           "18",
           "18.1",
           "18.4",
           "17.4",
           "16.3",
           "19.7",
           "13.4",
           "19.6",
           "15.1",
           "20.7",
           "18.9",
           "15.4",
           "13.7",
           "16.5"
          ]
         }
        },
        {
         "completeness": 0.8708860759493671,
         "inferred_type": "String",
         "name": "flipper_length_mm",
         "string_constraints": {
          "domains": [
           "205",
           "193",
           "182",
           "223",
           "216",
           "181",
           "NullValue",
           "201",
           "197",
           "NA",
           "186",
           "212",
           "224",
           "174",
           "217",
           "206",
           "209",
           "192",
           "231",
           "199",
           "196",
           "228",
           "188",
           "213",
           "202",
           "185",
           "219",
           "180",
           "221",
           "208",
           "230",
           "191",
           "190",
           "184",
           "225",
           "214",
           "203",
           "195",
           "189",
           "178",
           "210",
           "198",
           "187",
           "194",
           "215",
           "220",
           "183",
           "179",
           "218",
           "222",
           "207",
           "211",
           "226",
           "172",
           "229",
           "200",
           "176"
          ]
         }
        },
        {
         "completeness": 0.8708860759493671,
         "inferred_type": "String",
         "name": "body_mass_g",
         "string_constraints": {
          "domains": [
           "4150",
           "5950",
           "3675",
           "5200",
           "4550",
           "3475",
           "4300",
           "4475",
           "3450",
           "3200",
           "5350",
           "4950",
           "6000",
           "3300",
           "5600",
           "2925",
           "5100",
           "3575",
           "4575",
           "4075",
           "4900",
           "3600",
           "NullValue",
           "4925",
           "4600",
           "4375",
           "3150",
           "3400",
           "3900",
           "NA",
           "5850",
           "3175",
           "4200",
           "2975",
           "4800",
           "3825",
           "3100",
           "5750",
           "3500",
           "3325",
           "5450",
           "3250",
           "5050",
           "5500",
           "6050",
           "5150",
           "3975",
           "5800",
           "2700",
           "3425",
           "4500",
           "4450",
           "5300",
           "4750",
           "3725",
           "3650",
           "3800",
           "5250",
           "4250",
           "4725",
           "2900",
           "3275",
           "4400",
           "4850",
           "3350",
           "3750",
           "3050",
           "6300",
           "5400",
           "4675",
           "4775",
           "4000",
           "4975",
           "3550",
           "4700",
           "5650",
           "3000",
           "4625",
           "4650",
           "4100",
           "5700",
           "3875",
           "3950",
           "4350",
           "3700",
           "2850",
           "3075",
           "3775",
           "3850",
           "4050",
           "5000",
           "4275",
           "5550",
           "4875",
           "3525",
           "3625"
          ]
         }
        },
        {
         "completeness": 0.8708860759493671,
         "inferred_type": "String",
         "name": "sex",
         "string_constraints": {
          "domains": [
           ".",
           "NullValue",
           "NA",
           "MALE",
           "FEMALE"
          ]
         }
        }
       ],
       "monitoring_config": {
        "datatype_check_threshold": 1,
        "distribution_constraints": {
         "categorical_comparison_threshold": 0.1,
         "categorical_drift_method": "LInfinity",
         "comparison_method": "Robust",
         "comparison_threshold": 0.1,
         "perform_comparison": "Enabled"
        },
        "domain_content_threshold": 1,
        "emit_metrics": "Enabled",
        "evaluate_constraints": "Enabled"
       },
       "version": 0
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 174,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "constraints = f\"{DATA_QUALITY_LOCATION}/constraints.json\"\n",
    "\n",
    "response = None\n",
    "try:\n",
    "    response = json.loads(S3Downloader.read_file(constraints))\n",
    "except Exception as e:\n",
    "    pass\n",
    "\n",
    "JSON(response or {})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da339a4-aaac-4df6-922f-b6967750b597",
   "metadata": {},
   "source": [
    "We also generated the baseline performance using the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "id": "d57bb7d6-bec2-4557-b836-a821d0db7446",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/json": {
       "multiclass_classification_constraints": {
        "accuracy": {
         "comparison_operator": "LessThanThreshold",
         "threshold": 0.7962962962962963
        },
        "weighted_f0_5": {
         "comparison_operator": "LessThanThreshold",
         "threshold": 0.7302051424858442
        },
        "weighted_f1": {
         "comparison_operator": "LessThanThreshold",
         "threshold": 0.7339666506333173
        },
        "weighted_f2": {
         "comparison_operator": "LessThanThreshold",
         "threshold": 0.7666609510745312
        },
        "weighted_precision": {
         "comparison_operator": "LessThanThreshold",
         "threshold": 0.843681917211329
        },
        "weighted_recall": {
         "comparison_operator": "LessThanThreshold",
         "threshold": 0.7962962962962962
        }
       },
       "version": 0
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 175,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "constraints = f\"{MODEL_QUALITY_LOCATION}/constraints.json\"\n",
    "\n",
    "response = None\n",
    "try:\n",
    "    response = json.loads(S3Downloader.read_file(constraints))\n",
    "except Exception as e:\n",
    "    pass\n",
    "\n",
    "JSON(response or {})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "436b60d5-6be3-428f-9979-4fe9eebfc5eb",
   "metadata": {},
   "source": [
    "# Model Deployment\n",
    "\n",
    "To deploy the model, we will use [Amazon EventBridge](https://aws.amazon.com/pm/eventbridge/) to trigger a Lambda function that will deploy the model whenever its status changes to \"Approved.\"\n",
    "\n",
    "\n",
    "## Lambda \n",
    "\n",
    "Let's start by writing the Lambda function to take the model information and create a new endpoint.\n",
    "\n",
    "We'll enable [Data Capture](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-data-capture.html) as part of the endpoint configuration. With Data Capture we can record the inputs and outputs of the endpoint to use them later for monitoring the model:\n",
    "* `InitialSamplingPercentage` represents the percentage of traffic that we want to capture. \n",
    "* `DestinationS3Uri` specifies the S3 location where we want to store the captured data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "id": "28c75afd-d0d0-47f4-b7b6-9d590c5b600e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting code/lambda.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {CODE_FOLDER}/lambda.py\n",
    "\n",
    "import os\n",
    "import json\n",
    "import boto3\n",
    "import time\n",
    "\n",
    "sagemaker = boto3.client(\"sagemaker\")\n",
    "\n",
    "def lambda_handler(event, context):\n",
    "    model_package_arn = event[\"detail\"][\"ModelPackageArn\"]\n",
    "    approval_status = event[\"detail\"][\"ModelApprovalStatus\"]\n",
    "    \n",
    "    print(f'Model: \"{model_package_arn}\". Approval Status: \"{approval_status}\"')\n",
    "    \n",
    "    # We only want to deploy the model if it's new approval\n",
    "    # status is \"Approved.\"\n",
    "    if approval_status != \"Approved\":\n",
    "        return {\n",
    "            \"statusCode\": 200,\n",
    "            \"body\": json.dumps(f'Skipping deployment. Approval status: \"{approval_status}\"')\n",
    "        }    \n",
    "    \n",
    "    \n",
    "    endpoint_name = os.environ[\"ENDPOINT\"]\n",
    "    data_capture_destination = os.environ[\"DATA_CAPTURE_DESTINATION\"]\n",
    "    role = os.environ[\"ROLE\"]\n",
    "    \n",
    "    timestamp = time.strftime(\"%m%d%H%M%S\", time.localtime())\n",
    "    model_name = f\"{endpoint_name}-model-{timestamp}\"\n",
    "    endpoint_config_name = f\"{endpoint_name}-config-{timestamp}\"\n",
    "\n",
    "    sagemaker.create_model(\n",
    "        ModelName=model_name, \n",
    "        ExecutionRoleArn=role, \n",
    "        Containers=[{\n",
    "            \"ModelPackageName\": model_package_arn\n",
    "        }] \n",
    "    )\n",
    "\n",
    "    sagemaker.create_endpoint_config(\n",
    "        EndpointConfigName=endpoint_config_name,\n",
    "        ProductionVariants=[{\n",
    "            \"ModelName\": model_name,\n",
    "            \"InstanceType\": \"ml.m5.xlarge\",\n",
    "            \"InitialVariantWeight\": 1,\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            \"VariantName\": \"AllTraffic\",\n",
    "        }],\n",
    "        \n",
    "        # We can enable Data Capture to record the inputs and outputs of the endpoint\n",
    "        # to use them later for monitoring the model. \n",
    "        DataCaptureConfig={\n",
    "            \"EnableCapture\": True,\n",
    "            \"InitialSamplingPercentage\": 100,\n",
    "            \"DestinationS3Uri\": data_capture_destination,\n",
    "            \"CaptureOptions\": [\n",
    "                {\n",
    "                    \"CaptureMode\": \"Input\"\n",
    "                },\n",
    "                {\n",
    "                    \"CaptureMode\": \"Output\"\n",
    "                },\n",
    "            ],\n",
    "            \"CaptureContentTypeHeader\": {\n",
    "                \"CsvContentTypes\": [\n",
    "                    \"text/csv\",\n",
    "                    \"application/octect-stream\"\n",
    "                ],\n",
    "                \"JsonContentTypes\": [\n",
    "                    \"application/json\",\n",
    "                    \"application/octect-stream\"\n",
    "                ]\n",
    "            }\n",
    "        },\n",
    "    )\n",
    "    \n",
    "    response = sagemaker.list_endpoints(NameContains=endpoint_name, MaxResults=1)\n",
    "\n",
    "    if len(response[\"Endpoints\"]) == 0:\n",
    "        # If the endpoint doesn't exist, let's create it.\n",
    "        sagemaker.create_endpoint(\n",
    "            EndpointName=endpoint_name, \n",
    "            EndpointConfigName=endpoint_config_name,\n",
    "        )\n",
    "    else:\n",
    "        # If the endpoint already exist, let's update it with the\n",
    "        # new configuration.\n",
    "        sagemaker.update_endpoint(\n",
    "            EndpointName=endpoint_name, \n",
    "            EndpointConfigName=endpoint_config_name,\n",
    "        )\n",
    "    \n",
    "    return {\n",
    "        \"statusCode\": 200,\n",
    "        \"body\": json.dumps(\"Endpoint deployed successfully\")\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c66a4446-6db4-4515-bb1d-88ee21013bb2",
   "metadata": {},
   "source": [
    "We need to ensure our Lambda function has permission to interact with SageMaker, so let's create a new role and then create the lambda function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "id": "9ab1cca3-ce3f-48e4-93c6-0ce6363d00bc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Role lambda-deployment-role already exists.\n"
     ]
    }
   ],
   "source": [
    "ENDPOINT = \"penguins-endpoint\"\n",
    "DATA_CAPTURE_DESTINATION = f\"{S3_LOCATION}/monitoring/data-capture\"\n",
    "\n",
    "\n",
    "lambda_role_name = \"lambda-deployment-role\"\n",
    "lambda_role_arn = None\n",
    "\n",
    "try:\n",
    "    response = iam_client.create_role(\n",
    "        RoleName = lambda_role_name,\n",
    "        AssumeRolePolicyDocument = json.dumps({\n",
    "            \"Version\": \"2012-10-17\",\n",
    "            \"Statement\": [\n",
    "                {\n",
    "                    \"Effect\": \"Allow\",\n",
    "                    \"Principal\": {\n",
    "                        \"Service\": [\n",
    "                            \"lambda.amazonaws.com\",\n",
    "                            \"events.amazonaws.com\"\n",
    "                        ]\n",
    "                    },\n",
    "                    \"Action\": \"sts:AssumeRole\",\n",
    "                }\n",
    "            ]\n",
    "        }),\n",
    "        Description=\"Lambda Endpoint Deployment\"\n",
    "    )\n",
    "\n",
    "    lambda_role_arn = response[\"Role\"][\"Arn\"]\n",
    "    \n",
    "    iam_client.attach_role_policy(\n",
    "        RoleName=\"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole\",\n",
    "        PolicyArn=lambda_role_arn\n",
    "    )\n",
    "    \n",
    "    iam_client.attach_role_policy(\n",
    "        RoleName=\"arn:aws:iam::aws:policy/AmazonSageMakerFullAccess\",\n",
    "        PolicyArn=lambda_role_arn\n",
    "    )\n",
    "    \n",
    "    print(f'Role \"{lambda_role_name}\" created with ARN \"{lambda_role_arn}\".')\n",
    "except iam_client.exceptions.EntityAlreadyExistsException:\n",
    "    print(f\"Role {lambda_role_name} already exists.\")\n",
    "    response = iam_client.get_role(RoleName=lambda_role_name)\n",
    "    lambda_role_arn = response[\"Role\"][\"Arn\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c878ea6-dc45-4c4f-8f9e-344f9d2c9690",
   "metadata": {},
   "source": [
    "Let's create the Lambda function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "id": "33461c72-9e67-4cd7-aa3d-f20ce3a65b2e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ResponseMetadata': {'RequestId': 'bf076fe2-6182-4b0d-9d0a-64928f1959e5',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'date': 'Thu, 19 Oct 2023 18:21:02 GMT',\n",
       "   'content-type': 'application/json',\n",
       "   'content-length': '1428',\n",
       "   'connection': 'keep-alive',\n",
       "   'x-amzn-requestid': 'bf076fe2-6182-4b0d-9d0a-64928f1959e5'},\n",
       "  'RetryAttempts': 0},\n",
       " 'FunctionName': 'deploy_fn',\n",
       " 'FunctionArn': 'arn:aws:lambda:us-east-1:325223348818:function:deploy_fn',\n",
       " 'Runtime': 'python3.11',\n",
       " 'Role': 'arn:aws:iam::325223348818:role/lambda-deployment-role',\n",
       " 'Handler': 'lambda.lambda_handler',\n",
       " 'CodeSize': 3142,\n",
       " 'Description': '',\n",
       " 'Timeout': 600,\n",
       " 'MemorySize': 128,\n",
       " 'LastModified': '2023-10-19T18:21:02.000+0000',\n",
       " 'CodeSha256': 'D7Eppm64Jw5OLgWC+q9XBJqxjTF6zDJuGvCphFBphhU=',\n",
       " 'Version': '$LATEST',\n",
       " 'Environment': {'Variables': {'ROLE': 'arn:aws:iam::325223348818:role/service-role/AmazonSageMaker-ExecutionRole-20230312T160501',\n",
       "   'DATA_CAPTURE_DESTINATION': 's3://mlschool/penguins/monitoring/data-capture',\n",
       "   'ENDPOINT': 'penguins-endpoint'}},\n",
       " 'TracingConfig': {'Mode': 'PassThrough'},\n",
       " 'RevisionId': '30c7912d-e566-4969-b8d5-06d710c85f13',\n",
       " 'Layers': [],\n",
       " 'State': 'Active',\n",
       " 'LastUpdateStatus': 'InProgress',\n",
       " 'LastUpdateStatusReason': 'The function is being created.',\n",
       " 'LastUpdateStatusReasonCode': 'Creating',\n",
       " 'PackageType': 'Zip',\n",
       " 'Architectures': ['x86_64'],\n",
       " 'EphemeralStorage': {'Size': 512},\n",
       " 'SnapStart': {'ApplyOn': 'None', 'OptimizationStatus': 'Off'},\n",
       " 'RuntimeVersionConfig': {'RuntimeVersionArn': 'arn:aws:lambda:us-east-1::runtime:6cf63f1a78b5c5e19617d6b4b111370fdbda415ea91bdfdc5aacef9fee76b64a'}}"
      ]
     },
     "execution_count": 178,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sagemaker.lambda_helper import Lambda\n",
    "\n",
    "\n",
    "deploy_lambda_fn = Lambda(\n",
    "    function_name=\"deploy_fn\",\n",
    "    execution_role_arn=lambda_role_arn,\n",
    "    script=str(CODE_FOLDER / \"lambda.py\"),\n",
    "    handler=\"lambda.lambda_handler\",\n",
    "    timeout=600,\n",
    "    session=sagemaker_session,\n",
    "    runtime=\"python3.11\",\n",
    "    environment={\n",
    "        \"Variables\": {\n",
    "            \"ENDPOINT\": ENDPOINT,\n",
    "            \"DATA_CAPTURE_DESTINATION\": DATA_CAPTURE_DESTINATION,\n",
    "            \"ROLE\": config[\"role\"]\n",
    "        }\n",
    "    }\n",
    ")\n",
    "\n",
    "lambda_response = None\n",
    "if not LOCAL_MODE:\n",
    "    lambda_response = deploy_lambda_fn.upsert()\n",
    "\n",
    "lambda_response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd385160-a72e-44fd-84c2-1a5f468cfbb3",
   "metadata": {
    "tags": []
   },
   "source": [
    "## EventBridge\n",
    "\n",
    "Let's create an EventBridge rule that triggers the deployment process whenever a model approval status becomes \"Approved\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "id": "28c651a7-7aca-4066-b698-27f1b2aa3815",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'RuleArn': 'arn:aws:events:us-east-1:325223348818:rule/model-approval-rule',\n",
       " 'ResponseMetadata': {'RequestId': '5ad9ab26-0e43-46ed-9ff2-77233589157a',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '5ad9ab26-0e43-46ed-9ff2-77233589157a',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '76',\n",
       "   'date': 'Thu, 19 Oct 2023 18:21:02 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "events_client = boto3.client(\"events\")\n",
    "\n",
    "event_pattern = f\"\"\"\n",
    "{{\n",
    "  \"source\": [\"aws.sagemaker\"],\n",
    "  \"detail-type\": [\"SageMaker Model Package State Change\"],\n",
    "  \"detail\": {{\n",
    "    \"ModelPackageGroupName\": [\"{MODEL_PACKAGE_GROUP}\"],\n",
    "    \"ModelApprovalStatus\": [\"Approved\"]\n",
    "  }}\n",
    "}}\n",
    "\"\"\"\n",
    "\n",
    "rule_response = None\n",
    "if not LOCAL_MODE:\n",
    "  rule_response = events_client.put_rule(\n",
    "      Name=\"model-approval-rule\",\n",
    "      EventPattern=event_pattern,\n",
    "      State=\"ENABLED\",\n",
    "      RoleArn=config[\"role\"],\n",
    "  )\n",
    "\n",
    "  response = events_client.put_targets(\n",
    "      Rule=\"model-approval-rule\",\n",
    "      Targets=[\n",
    "          {\n",
    "              \"Id\": \"1\",\n",
    "              \"Arn\": lambda_response[\"FunctionArn\"],\n",
    "          }\n",
    "      ]\n",
    "  )\n",
    "\n",
    "rule_response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1976e203-93c2-4a0c-a7ec-935ad4889a88",
   "metadata": {},
   "source": [
    "We need to give the event permissions to invoke the Lambda function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "id": "fedaa279-166d-4efb-bd86-7a451baf26ac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Function \"deploy_fn\" already has the correct permissions.\n"
     ]
    }
   ],
   "source": [
    "lambda_client = boto3.client(\"lambda\")\n",
    "\n",
    "if not LOCAL_MODE:\n",
    "    try:\n",
    "        response = lambda_client.add_permission(\n",
    "            Action=\"lambda:InvokeFunction\",\n",
    "            FunctionName=lambda_response[\"FunctionName\"],\n",
    "            Principal=\"events.amazonaws.com\",\n",
    "            SourceArn=rule_response[\"RuleArn\"],\n",
    "            StatementId=\"EventBridge\",\n",
    "        )\n",
    "    except lambda_client.exceptions.ResourceConflictException as e:\n",
    "        print(f'Function \"{lambda_response[\"FunctionName\"]}\" already has the correct permissions.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d56782b2-4b40-488b-8e4f-f8ecfa056b10",
   "metadata": {},
   "source": [
    "## Predictions\n",
    "\n",
    "Let's now test the endpoint we deployed automatically with the pipeline. We will use the function to create a predictor with a JSON encoder and decoder. \n",
    "\n",
    "<div class=\"alert\" style=\"background-color:#0066cc;\">Uncomment the <code style=\"background-color:#0066cc;\">%%script</code> cell magic line to execute this cell.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "id": "54d8465e-9f50-4e12-9b39-240f9a4b180c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.serializers import CSVSerializer\n",
    "\n",
    "\n",
    "waiter = sagemaker_client.get_waiter(\"endpoint_in_service\")\n",
    "waiter.wait(\n",
    "    EndpointName=ENDPOINT,\n",
    "    WaiterConfig={\n",
    "        \"Delay\": 10,\n",
    "        \"MaxAttempts\": 30\n",
    "    }\n",
    ")\n",
    "\n",
    "predictor = Predictor(\n",
    "    endpoint_name=ENDPOINT, \n",
    "    serializer=CSVSerializer(),\n",
    "    sagemaker_session=sagemaker_session\n",
    ")\n",
    "\n",
    "data = pd.read_csv(DATA_FILEPATH)\n",
    "data = data.drop(\"species\", axis=1)\n",
    "\n",
    "payload = data.iloc[:3].to_csv(header=False, index=False)\n",
    "response = predictor.predict(payload, initial_args={\"ContentType\": \"text/csv\"})\n",
    "print(response.decode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02a1e7af-933e-492d-948e-aa16cc67c3db",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Data Capture\n",
    "\n",
    "Let's check the S3 location where the endpoint stores the requests and responses that it receives.\n",
    "\n",
    "Notice that it make take a few minutes for the first few files to show up in S3. Keep running the following line until you get some."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "3f35e8db-24d7-4d4b-9264-78ee5070cf27",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['s3://mlschool/penguins/monitoring/data-capture/penguins-endpoint/AllTraffic/2023/09/25/13/15-40-735-9cc3750d-ba42-472c-903d-969695d2096d.jsonl',\n",
       " 's3://mlschool/penguins/monitoring/data-capture/penguins-endpoint/AllTraffic/2023/09/27/15/45-31-289-001fc69f-c352-4da2-b57a-a3a69fe3fecf.jsonl',\n",
       " 's3://mlschool/penguins/monitoring/data-capture/penguins-endpoint/AllTraffic/2023/10/05/16/50-04-992-e16242d1-925c-4b07-9289-dffa0e026679.jsonl']"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files = S3Downloader.list(DATA_CAPTURE_DESTINATION)[:3]\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3bc31d3-a277-446a-afd1-8bf7aab6173e",
   "metadata": {},
   "source": [
    "These files contain the data captured by the endpoint in a SageMaker-specific JSON-line format. Each inference request is captured in a single line in the `jsonl` file. The line contains both the input and output merged together.\n",
    "\n",
    "Let's read the first line from the first file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "id": "3dee0107-c9ca-4f75-873d-d47512c56797",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"captureData\": {\n",
      "    \"endpointInput\": {\n",
      "      \"observedContentType\": \"text/csv\",\n",
      "      \"mode\": \"INPUT\",\n",
      "      \"data\": \"Torgersen,39.1,18.7,181.0,3750.0,MALE\\nTorgersen,39.5,17.4,186.0,3800.0,FEMALE\\nTorgersen,40.3,18.0,195.0,3250.0,FEMALE\\n\",\n",
      "      \"encoding\": \"CSV\"\n",
      "    },\n",
      "    \"endpointOutput\": {\n",
      "      \"observedContentType\": \"application/json\",\n",
      "      \"mode\": \"OUTPUT\",\n",
      "      \"data\": \"[{\\\"prediction\\\": \\\"Adelie\\\", \\\"confidence\\\": 0.775418103}, {\\\"prediction\\\": \\\"Adelie\\\", \\\"confidence\\\": 0.775709867}, {\\\"prediction\\\": \\\"Adelie\\\", \\\"confidence\\\": 0.67967391}]\",\n",
      "      \"encoding\": \"JSON\"\n",
      "    }\n",
      "  },\n",
      "  \"eventMetadata\": {\n",
      "    \"eventId\": \"d33f9a23-5ae3-4403-9aa1-3759d7fa8015\",\n",
      "    \"inferenceTime\": \"2023-09-25T13:15:40Z\"\n",
      "  },\n",
      "  \"eventVersion\": \"0\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "if len(files):\n",
    "    lines = S3Downloader.read_file(files[0])\n",
    "    print(json.dumps(json.loads(lines.split(\"\\n\")[0]), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1729bdf6-59d1-47c6-9abb-43ed530b75e2",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "\n",
    "Let's now delete the endpoint.\n",
    "\n",
    "<div class=\"alert\" style=\"background-color:#0066cc;\">Uncomment the <code style=\"background-color:#0066cc;\">%%script</code> cell magic line to execute this cell.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "id": "8c3e851a-2416-4a0b-b8a1-c483cde3d776",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e544ae36-00b3-4bde-b133-c3a59bb7f1d8",
   "metadata": {},
   "source": [
    "# Model Monitoring\n",
    "\n",
    "In this session we'll set up a monitoring process to analyze the quality of the data our endpoint receives and the endpoint predictions. For this, we need to check the data received by the endpoint, generate ground truth labels, and compare them with a baseline performance.\n",
    "\n",
    "To enable this functionality, we need a couple of steps:\n",
    "\n",
    "1. Create baselines we can use to compare against real-time traffic.\n",
    "2. Set up a schedule to continuously evaluate and compare against the baselines.\n",
    "\n",
    "Notice that we use the baseline datasets we generated during the Processing Step. These baseline datasets are the same unprocessed data in JSON format. We do this because we need raw data to compare against the endpoint input.\n",
    "\n",
    "Check [Amazon SageMaker Model Monitor](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_monitoring.html) for a brief explanation of how to use SageMaker's Model Monitoring functionality. [Monitor models for data and model quality, bias, and explainability](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html) is a much more extensive guide to monitoring in Amazon SageMaker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "id": "2c7e2f9d-cc75-46bc-8700-f7123292fac5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "GROUND_TRUTH_LOCATION = f\"{S3_LOCATION}/monitoring/groundtruth\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b948aa92-8064-4f03-af08-0f6a8fc329cf",
   "metadata": {},
   "source": [
    "## Fake Traffic\n",
    "\n",
    "To test the monitoring functionality, we need to generate traffic to the endpoint.\n",
    "\n",
    "To generate traffic, we will repeatedly send every sample from the dataset to the endpoint to simulate real prediction requests."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a488020-e1d3-48a8-8cb7-a1d3c8d26a07",
   "metadata": {},
   "source": [
    "The following function will generate the traffic to the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "id": "87a3c4ce-aff7-4f48-9d1b-be98eb746e66",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from time import sleep\n",
    "from threading import Thread, Event\n",
    "\n",
    "\n",
    "def generate_traffic(predictor):\n",
    "    \n",
    "    def _predict(data, predictor, stop_traffic_thread):\n",
    "        for index, row in data.iterrows():\n",
    "            data = row.tolist()\n",
    "            data = ','.join(map(str, data))\n",
    "            predictor.predict(data, inference_id=str(index), initial_args={\"ContentType\": \"text/csv\"})\n",
    "            \n",
    "            sleep(1)\n",
    "\n",
    "            if stop_traffic_thread.is_set():\n",
    "                break\n",
    "\n",
    "    def _generate_prediction_data(data, predictor, stop_traffic_thread):\n",
    "        while True:\n",
    "            print(f\"Generating {data.shape[0]} predictions...\")\n",
    "            _predict(data, predictor, stop_traffic_thread)\n",
    "            \n",
    "            if stop_traffic_thread.is_set():\n",
    "                break\n",
    "\n",
    "                \n",
    "    stop_traffic_thread = Event()\n",
    "    \n",
    "    data = pd.read_csv(DATA_FILEPATH, header=0).dropna()\n",
    "    data.drop([\"species\"], axis=1, inplace=True)\n",
    "    \n",
    "    traffic_thread = Thread(\n",
    "        target=_generate_prediction_data,\n",
    "        args=(data, predictor, stop_traffic_thread,)\n",
    "    )\n",
    "    \n",
    "    traffic_thread.start()\n",
    "    \n",
    "    return stop_traffic_thread, traffic_thread"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "695ca746-8618-4c2e-b6f8-8d058bbfa4e9",
   "metadata": {},
   "source": [
    "Let's wait for the endpoint to be in service, and then we can start generating traffic to the endpoint.\n",
    "\n",
    "<div class=\"alert\" style=\"background-color:#0066cc;\">Uncomment the <code style=\"background-color:#0066cc;\">%%script</code> cell magic line to execute this cell.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "id": "9a65fc15-076a-41ab-b60c-144d97d5d75b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "waiter = sagemaker_client.get_waiter(\"endpoint_in_service\")\n",
    "waiter.wait(\n",
    "    EndpointName=ENDPOINT,\n",
    "    WaiterConfig={\n",
    "        \"Delay\": 10,\n",
    "        \"MaxAttempts\": 30\n",
    "    }\n",
    ")\n",
    "\n",
    "predictor = Predictor(\n",
    "    endpoint_name=ENDPOINT, \n",
    "    serializer=CSVSerializer(),\n",
    "    sagemaker_session=sagemaker_session\n",
    ")\n",
    "\n",
    "stop_traffic_thread, traffic_thread = generate_traffic(predictor)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72521fc9-2401-4489-85dc-a6a8dfa6547e",
   "metadata": {},
   "source": [
    "## Fake Labels\n",
    "\n",
    "To test the performance of the model, we need to label the samples captured by the endpoint. We can simulate the labeling process by generating a random label for every sample. Check [Ingest Ground Truth Labels and Merge Them With Predictions](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-merge.html) for more information about this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "id": "cb649a6e-fabe-4103-b1db-7c6a01fe959a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import random\n",
    "from datetime import datetime\n",
    "\n",
    "\n",
    "def generate_ground_truth_data(ground_truth_location):\n",
    "    \n",
    "    def _generate_ground_truth_record(inference_id):\n",
    "        random.seed(inference_id)\n",
    "\n",
    "        return {\n",
    "            \"groundTruthData\": {\n",
    "                \"data\": random.choice([\"Adelie\", \"Chinstrap\", \"Gentoo\"]),\n",
    "                \"encoding\": \"CSV\",\n",
    "            },\n",
    "            \"eventMetadata\": {\n",
    "                \"eventId\": str(inference_id),\n",
    "            },\n",
    "            \"eventVersion\": \"0\",\n",
    "        }\n",
    "\n",
    "\n",
    "    def _upload_ground_truth(records, upload_time):\n",
    "        records = [json.dumps(r) for r in records]\n",
    "        data = \"\\n\".join(records)\n",
    "        uri = f\"{ground_truth_location}/{upload_time:%Y/%m/%d/%H/%M%S}.jsonl\"\n",
    "\n",
    "        print(f\"Uploading ground truth data to {uri}...\")\n",
    "\n",
    "        S3Uploader.upload_string_as_file_body(data, uri)    \n",
    "\n",
    "                \n",
    "    def _generate_ground_truth_data(max_records, stop_ground_truth_thread):\n",
    "        while True:\n",
    "            records = [_generate_ground_truth_record(i) for i in range(max_records)]\n",
    "            _upload_ground_truth(records, datetime.utcnow())\n",
    "\n",
    "            if stop_ground_truth_thread.is_set():\n",
    "                break\n",
    "\n",
    "            sleep(30)\n",
    "\n",
    "\n",
    "    stop_ground_truth_thread = Event()\n",
    "    data = pd.read_csv(DATA_FILEPATH).dropna()\n",
    "    \n",
    "    groundtruth_thread = Thread(\n",
    "        target=_generate_ground_truth_data,\n",
    "        args=(len(data), stop_ground_truth_thread,)\n",
    "    )\n",
    "    \n",
    "    groundtruth_thread.start()\n",
    "    \n",
    "    return stop_ground_truth_thread, groundtruth_thread"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4dd7d22-aecb-4c2d-b6d2-a282ea8a17e8",
   "metadata": {},
   "source": [
    "We can now start generating fake labels.\n",
    "\n",
    "<div class=\"alert\" style=\"background-color:#0066cc;\">Uncomment the <code style=\"background-color:#0066cc;\">%%script</code> cell magic line to execute this cell.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "id": "0de56c8d-ebde-409a-a7c3-4103790117d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "stop_ground_truth_thread, groundtruth_thread = generate_ground_truth_data(\n",
    "    GROUND_TRUTH_LOCATION\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04fda9e8-08dc-4323-9e97-eb17194078a1",
   "metadata": {},
   "source": [
    "## Monitoring Jobs\n",
    "\n",
    "We can now schedule the Monitoring Jobs to continuously monitor the data going into the endpoint and the model performance. We will use the baseline we generated in the pipeline to determine when there's drift. Check [Schedule Data Quality Monitoring Jobs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-schedule-data-monitor.html) and [Schedule Model Quality Monitoring Jobs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-schedule.html) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f98d39f9-446d-4cfe-ad94-b50ab31caf68",
   "metadata": {},
   "source": [
    "The following functions will help us work with monitoring schedules later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "id": "da145ba1-4966-4dab-8a73-281db364cbc7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def describe_monitoring_schedules(endpoint_name):\n",
    "    schedules = []\n",
    "    response = sagemaker_client.list_monitoring_schedules(EndpointName=endpoint_name)[\"MonitoringScheduleSummaries\"]\n",
    "    for item in response:\n",
    "        name = item[\"MonitoringScheduleName\"]\n",
    "        schedule = {\n",
    "            \"MonitoringScheduleName\": name,\n",
    "            \"MonitoringType\": item[\"MonitoringType\"]\n",
    "        }\n",
    "        \n",
    "        description = sagemaker_client.describe_monitoring_schedule(\n",
    "            MonitoringScheduleName=name\n",
    "        )\n",
    "        \n",
    "        schedule[\"Status\"] = description[\"LastMonitoringExecutionSummary\"][\"MonitoringExecutionStatus\"]\n",
    "        \n",
    "        if schedule[\"Status\"] == \"Failed\":\n",
    "            schedule[\"FailureReason\"] = description[\"LastMonitoringExecutionSummary\"][\"FailureReason\"]\n",
    "        elif schedule[\"Status\"] == \"CompletedWithViolations\":\n",
    "            processing_job_arn = description[\"LastMonitoringExecutionSummary\"][\"ProcessingJobArn\"]\n",
    "            execution = MonitoringExecution.from_processing_arn(\n",
    "                sagemaker_session=sagemaker_session, \n",
    "                processing_job_arn=processing_job_arn\n",
    "            )\n",
    "            execution_destination = execution.output.destination\n",
    "\n",
    "            violations_filepath = os.path.join(execution_destination, \"constraint_violations.json\")\n",
    "            violations = json.loads(S3Downloader.read_file(violations_filepath))[\"violations\"]\n",
    "            \n",
    "            schedule[\"Violations\"] = violations\n",
    "\n",
    "        schedules.append(schedule)\n",
    "        \n",
    "    return schedules\n",
    "\n",
    "def describe_monitoring_schedule(endpoint_name, monitoring_type):\n",
    "    found = False\n",
    "    \n",
    "    schedules = describe_monitoring_schedules(endpoint_name)\n",
    "    for schedule in schedules:\n",
    "        if schedule[\"MonitoringType\"] == monitoring_type:\n",
    "            found = True\n",
    "            print(json.dumps(schedule, indent=2))\n",
    "\n",
    "    if not found:            \n",
    "        print(f\"There's no {monitoring_type} Monitoring Schedule.\")\n",
    "\n",
    "\n",
    "def describe_data_monitoring_schedule(endpoint_name):\n",
    "    describe_monitoring_schedule(endpoint_name, \"DataQuality\")\n",
    "\n",
    "    \n",
    "def describe_model_monitoring_schedule(endpoint_name):\n",
    "    describe_monitoring_schedule(endpoint_name, \"ModelQuality\")\n",
    "\n",
    "    \n",
    "def delete_monitoring_schedule(endpoint_name, monitoring_type):\n",
    "    attempts = 30\n",
    "    found = False\n",
    "    \n",
    "    response = sagemaker_client.list_monitoring_schedules(EndpointName=endpoint_name)[\"MonitoringScheduleSummaries\"]\n",
    "    for item in response:\n",
    "        if item[\"MonitoringType\"] == monitoring_type:\n",
    "            found = True\n",
    "            status = sagemaker_client.describe_monitoring_schedule(\n",
    "                MonitoringScheduleName=item[\"MonitoringScheduleName\"]\n",
    "            )[\"MonitoringScheduleStatus\"]\n",
    "            while status in (\"Pending\", \"InProgress\") and attempts > 0:\n",
    "                attempts -= 1\n",
    "                print(f\"Monitoring schedule status: {status}. Waiting for it to finish.\")\n",
    "                sleep(30)\n",
    "                \n",
    "                status = sagemaker_client.describe_monitoring_schedule(\n",
    "                    MonitoringScheduleName=item[\"MonitoringScheduleName\"]\n",
    "                )[\"MonitoringScheduleStatus\"]\n",
    "\n",
    "            if status not in (\"Pending\", \"InProgress\"):\n",
    "                sagemaker_client.delete_monitoring_schedule(\n",
    "                    MonitoringScheduleName=item[\"MonitoringScheduleName\"]\n",
    "                )\n",
    "                print(\"Monitoring schedule deleted.\")\n",
    "            else:\n",
    "                print(\"Waiting for monitoring schedule timed out\")\n",
    "                \n",
    "    if not found:            \n",
    "        print(f\"There's no {monitoring_type} Monitoring Schedule.\")\n",
    "\n",
    "        \n",
    "def delete_data_monitoring_schedule(endpoint_name):\n",
    "    delete_monitoring_schedule(endpoint_name, \"DataQuality\")\n",
    "\n",
    "    \n",
    "def delete_model_monitoring_schedule(endpoint_name):\n",
    "    delete_monitoring_schedule(endpoint_name, \"ModelQuality\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d936df76-e0b8-4dad-a04f-ef77ce2a2df1",
   "metadata": {},
   "source": [
    "### Data Monitoring\n",
    "\n",
    "SageMaker looks for violations in the data captured by the endpoint. By default, it combines the input data with the endpoint output and compare the result with the baseline we generated. If we let SageMaker do this, we will get a few violations, for example an \"extra column check\" violation because the field `confidence` doesn't exist in the baseline data.\n",
    "\n",
    "We can fix these violations by creating a preprocessing script configuring the data we want the monitoring job to use. Check [Preprocessing and Postprocessing](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-pre-and-post-processing.html) for more information about how to configure these scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "id": "cc119422-2e85-4e8c-86cd-6d59e353d09d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DATA_QUALITY_PREPROCESSOR = \"data_quality_preprocessor.py\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "id": "083b0bd0-4035-43fe-9b2c-946b12a5e266",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting code/data_quality_preprocessor.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {CODE_FOLDER}/{DATA_QUALITY_PREPROCESSOR}\n",
    "import json\n",
    "\n",
    "def preprocess_handler(inference_record):\n",
    "    input_data = inference_record.endpoint_input.data\n",
    "    output_data = json.loads(inference_record.endpoint_output.data)\n",
    "    \n",
    "    response = json.loads(input_data)\n",
    "    response[\"species\"] = output_data[\"prediction\"]\n",
    "\n",
    "    # The `response` variable contains the data that we want the\n",
    "    # monitoring job to use to compare with the baseline.\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "840d54c5-f09c-4559-a1d2-63587da0ad14",
   "metadata": {},
   "source": [
    "The monitoring schedule expects an S3 location pointing to the preprocessing script. Let's upload the script to the default bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "id": "96e5c0c1-7e40-47df-8f40-1d891db13875",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "if not LOCAL_MODE:\n",
    "    bucket = boto3.Session().resource(\"s3\").Bucket(pipeline_session.default_bucket())\n",
    "    prefix = \"penguins-monitoring\"\n",
    "    bucket.Object(os.path.join(prefix, DATA_QUALITY_PREPROCESSOR)).upload_file(str(CODE_FOLDER / DATA_QUALITY_PREPROCESSOR))\n",
    "    data_quality_preprocessor = f\"s3://{os.path.join(bucket.name, prefix, DATA_QUALITY_PREPROCESSOR)}\"\n",
    "    data_quality_preprocessor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56e107eb-546d-431c-b74d-1bfd412711b7",
   "metadata": {},
   "source": [
    "We can now set up the Data Quality Monitoring Job using the [DefaultModelMonitor](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_monitor.model_monitoring.DefaultModelMonitor) class. Notice how we specify the `record_preprocessor_script` using the S3 location where we uploaded our script.\n",
    "\n",
    "<div class=\"alert\" style=\"background-color:#0066cc;\">Uncomment the <code style=\"background-color:#0066cc;\">%%script</code> cell magic line to execute this cell.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "id": "15caf9e1-97fc-4379-893b-6062d4bd876e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "from sagemaker.model_monitor import CronExpressionGenerator, DefaultModelMonitor\n",
    "\n",
    "data_monitor = DefaultModelMonitor(\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    instance_count=1,\n",
    "    max_runtime_in_seconds=3600,\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "data_monitor.create_monitoring_schedule(\n",
    "    monitor_schedule_name=\"penguins-data-monitoring-schedule\",\n",
    "    endpoint_input=ENDPOINT,\n",
    "    record_preprocessor_script=data_quality_preprocessor,\n",
    "    statistics=f\"{DATA_QUALITY_LOCATION}/statistics.json\",\n",
    "    constraints=f\"{DATA_QUALITY_LOCATION}/constraints.json\",\n",
    "    schedule_cron_expression=CronExpressionGenerator.hourly(),\n",
    "    enable_cloudwatch_metrics=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "018800f7-315f-4f5e-b082-ba94bbde91ad",
   "metadata": {},
   "source": [
    "We can check the results of the monitoring job by looking at whether it generated any violations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "id": "2c04fdd4-cc03-496c-a0a1-405854505c46",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There's no DataQuality Monitoring Schedule.\n"
     ]
    }
   ],
   "source": [
    "describe_data_monitoring_schedule(ENDPOINT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a9d201d-f60f-49f2-b4e9-eb0a0159ecfd",
   "metadata": {},
   "source": [
    "### Model Monitoring\n",
    "\n",
    "To set up a Model Quality Monitoring Job, we can use the [ModelQualityMonitor](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_monitor.model_monitoring.ModelQualityMonitor) class. The [EndpointInput](https://sagemaker.readthedocs.io/en/v2.24.2/api/inference/model_monitor.html#sagemaker.model_monitor.model_monitoring.EndpointInput) instance configures the attribute the monitoring job should use to determine the prediction from the model.\n",
    "\n",
    "Check [Amazon SageMaker Model Quality Monitor](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_model_monitor/model_quality/model_quality_churn_sdk.html) for a complete tutorial on how to run a Model Monitoring Job in SageMaker.\n",
    "\n",
    "<div class=\"alert\" style=\"background-color:#0066cc;\">Uncomment the <code style=\"background-color:#0066cc;\">%%script</code> cell magic line to execute this cell.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "id": "070e0d73-5375-4fc3-b94c-da0574600c05",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "from sagemaker.model_monitor import ModelQualityMonitor, EndpointInput\n",
    "\n",
    "\n",
    "model_monitor = ModelQualityMonitor(\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    instance_count=1,\n",
    "    max_runtime_in_seconds=1800,\n",
    "    role=role\n",
    ")\n",
    "\n",
    "model_monitor.create_monitoring_schedule(\n",
    "    monitor_schedule_name=\"penguins-model-monitoring-schedule\",\n",
    "    \n",
    "    endpoint_input = EndpointInput(\n",
    "        endpoint_name=ENDPOINT,\n",
    "        inference_attribute=\"prediction\",\n",
    "        destination=\"/opt/ml/processing/input_data\",\n",
    "    ),\n",
    "    \n",
    "    problem_type=\"MulticlassClassification\",\n",
    "    ground_truth_input=GROUND_TRUTH_LOCATION,\n",
    "    \n",
    "    constraints=f\"{MODEL_QUALITY_LOCATION}/constraints.json\",\n",
    "    \n",
    "    schedule_cron_expression=CronExpressionGenerator.hourly(),\n",
    "    output_s3_uri=f\"{S3_LOCATION}/monitoring/model-quality\",\n",
    "    enable_cloudwatch_metrics=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d9e523e-49c5-4382-b28a-cdbece9bd0e0",
   "metadata": {},
   "source": [
    "We can check the results of the monitoring job by looking at whether it generated any violations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "id": "347de298-16f2-42e0-85c4-dfc916080020",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There's no ModelQuality Monitoring Schedule.\n"
     ]
    }
   ],
   "source": [
    "describe_model_monitoring_schedule(ENDPOINT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c267ea0-f9c0-4bf2-8281-8d21edebb2a0",
   "metadata": {},
   "source": [
    "### Clean up\n",
    "\n",
    "The following code will stop the generation of traffic and labels, delete the monitoring jobs, and delete the endpoint.\n",
    "\n",
    "<div class=\"alert\" style=\"background-color:#0066cc;\">Uncomment the <code style=\"background-color:#0066cc;\">%%script</code> cell magic line to execute this cell.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "id": "bb74dc04-54a1-4a3f-854f-4877f7f0b4a1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%script false --no-raise-error\n",
    "\n",
    "stop_traffic_thread.set()\n",
    "traffic_thread.join()\n",
    "\n",
    "stop_ground_truth_thread.set()\n",
    "groundtruth_thread.join()\n",
    "\n",
    "delete_data_monitoring_schedule(ENDPOINT)\n",
    "delete_model_monitoring_schedule(ENDPOINT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "791af080",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "lcc_arn": "arn:aws:sagemaker:us-east-1:325223348818:studio-lifecycle-config/packages",
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}