1-Clipper-Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "<div class=\"toc\" style=\"margin-top: 1em;\">\n",
    "   <ul class=\"toc-item\" id=\"toc-level0\">\n",
    "       <li><span><a href=\"#Jupyter-Notebooks\" data-toc-modified-id=\"Jupyter-Notebooks\"><span class=\"toc-item-num\">1.&nbsp;&nbsp;</span>Jupyter Notebooks</a></span></li>\n",
    "      <li>\n",
    "         <span><a href=\"#API-Overview\" data-toc-modified-id=\"API-Overview-1\"><span class=\"toc-item-num\">2.&nbsp;&nbsp;</span>API Overview</a></span>\n",
    "         <ul class=\"toc-item\">\n",
    "            <li><span><a href=\"#Context\" data-toc-modified-id=\"Context-1.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Context</a></span></li>\n",
    "            <li><span><a href=\"#Creating-a-ClipperConnection\" data-toc-modified-id=\"Create-a-ClipperConnection-1.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Creating a ClipperConnection</a></span></li>\n",
    "            <li><span><a href=\"#Starting-Clipper\" data-toc-modified-id=\"Starting-Clipper-1.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Starting Clipper</a></span></li>\n",
    "            <li>\n",
    "               <span><a href=\"#Deploying-a-model\" data-toc-modified-id=\"Deploying-a-model-1.4\"><span class=\"toc-item-num\">2.4&nbsp;&nbsp;</span>Deploying a model</a></span>\n",
    "               <ul class=\"toc-item\">\n",
    "                  <li>\n",
    "                     <span><a href=\"#Creating-a-model\" data-toc-modified-id=\"Create-the-model-1.4.1\"><span class=\"toc-item-num\">2.4.1&nbsp;&nbsp;</span>Creating a model</a></span>\n",
    "                  </li>\n",
    "                  <li><span><a href=\"#Deploying-to-Clipper\" data-toc-modified-id=\"Deploying-to-Clipper-1.4.2\"><span class=\"toc-item-num\">2.4.2&nbsp;&nbsp;</span>Deploying to Clipper</a></span></li>\n",
    "                  <li><span><a href=\"#A-Note-About-Types-[Optional]\" data-toc-modified-id=\"A-Note-About-Types-[Optional]-1.4.3\"><span class=\"toc-item-num\">2.4.3&nbsp;&nbsp;</span>A Note About Types [Optional]</a></span></li>\n",
    "               </ul>\n",
    "            </li>\n",
    "            <li><span><a href=\"#Registering-an-Application\" data-toc-modified-id=\"Registering-an-Application-1.5\"><span class=\"toc-item-num\">2.5&nbsp;&nbsp;</span>Registering an Application</a></span></li>\n",
    "            <li><span><a href=\"#Inspecting-Clipper\" data-toc-modified-id=\"Inspecting-Clipper-1.6\"><span class=\"toc-item-num\">2.6&nbsp;&nbsp;</span>Inspecting Clipper</a></span></li>\n",
    "            <li><span><a href=\"#Updating-the-Model\" data-toc-modified-id=\"Updating-the-Model-1.7\"><span class=\"toc-item-num\">2.7&nbsp;&nbsp;</span>Updating the Model</a></span></li>\n",
    "            <li><span><a href=\"#Adding-Model-Replicas\" data-toc-modified-id=\"Adding-Model-Replicas-1.8\"><span class=\"toc-item-num\">2.8&nbsp;&nbsp;</span>Adding Model Replicas</a></span></li>\n",
    "         </ul>\n",
    "      </li>\n",
    "<li>\n",
    "         <span><a href=\"#Example-Application---Image-Classification\" data-toc-modified-id=\"Example-Application---Image-Classification\"><span class=\"toc-item-num\">3.&nbsp;&nbsp;</span>Example Application - Image Classification</a></span>\n",
    "      <li><span><a href=\"#Example-Application---Custom-Docker-Containers\" data-toc-modified-id=\"Example-Application---Custom-Docker-Containers\"><span class=\"toc-item-num\">4.&nbsp;&nbsp;</span>Example Application - Custom Docker Containers</a></span></li>\n",
    "      <li><span><a href=\"#Restarting-Clipper\" data-toc-modified-id=\"Restarting-Clipper\"><span class=\"toc-item-num\">5.&nbsp;&nbsp;</span>Restarting Clipper</a></span></li>\n",
    "   </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jupyter Notebooks\n",
    "\n",
    "Welcome to the Clipper tutorial at Strata NYC 2018. You will be using [Jupyter Notebooks](http://jupyter.org/) for this tutorial. If you have never used a Jupyter notebook before, you may find it useful to spend a few minutes looking over the documentation. To get you started, two useful commands are:\n",
    "\n",
    "+ Shift-Enter will execute a cell. If it is a code cell, it will run the code. If it is a Markdown cell, it will render the Markdown.\n",
    "+ You can run shell commands directly in the notebook by prefixing them with an exclamation point. You will see several examples of this throughput the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='api_overview'></a>\n",
    "## API Overview\n",
    "\n",
    "In the first part of this exercise, you will explore how to create and interact with a Clipper cluster. The primary way of managing Clipper is with the Clipper Admin Python tool. This tutorial will walk you through all the things you can do with the Clipper Admin tool as well as explain what happens within Clipper when you issue each command. You can find the complete API documentation for the Clipper Admin tool on our website: <http://docs.clipper.ai>.\n",
    "\n",
    "**Goal:** Be familiar with how to create and manage a Clipper cluster, and understand what happens when you issue Clipper admin commands.\n",
    "\n",
    "> *NOTE: Throughout this exercise, you will building Docker images and deploying and managing a distributed system. Some commands may take a few minutes to run. If a cell does not complete immediately, that is perfectly normal. Thanks for your patience!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Context\n",
    "The Clipper Admin tool is distributed through Pip. You can install it with `pip install clipper_admin`, but it has already been installed in this notebook for you.\n",
    "\n",
    "Clipper uses Docker containers as its deployment mechanism. A running Clipper cluster consists of a collection of Docker containers communicating with each other over the network. As you issue commands against Clipper, you are communicating with these containers as well as creating new ones or destroying existing ones. As you explore the Clipper API throughout this exercise, we will illustrate how each command effects the cluster state.\n",
    "\n",
    "The main API for interacting with Clipper is exposed via a [`ClipperConnection`](http://docs.clipper.ai/en/develop/#clipper-connection) object. This is your handle to a Clipper cluster (this collection of Docker containers). It can be used to start, stop, inspect, and modify the cluster.\n",
    "\n",
    "In order to create a `ClipperConnection` object, you must provide it with a [`ContainerManager`](http://docs.clipper.ai/en/develop/#container-managers) object. While Docker is becoming an increasingly standard mechanism for deploying applications, there are many different tools for managing a Docker cluster. These tools broadly fall into the category of *Container Orchestration frameworks*. Some popular examples are [Kubernetes](https://kubernetes.io/), [Docker Swarm](https://docs.docker.com/engine/swarm/), and [DC/OS](https://dcos.io/). One of the reasons we run Clipper in Docker containers is to make the system as general as possible and support many different deployment scenarios. Within the Clipper Admin, we abstract away all of the Docker container-specific commands behind the `ContainerManager` interface. The `ClipperConnection` object makes Clipper-specific decisions about how to issue commands, and then makes any changes to the Docker configuration (for example, to launch a container for a newly deployed model) through the `ContainerManager`. To support different container orchestration frameworks that manage Docker containers in different ways, we create different implementations of the `ContainerManager` interface.\n",
    "\n",
    "Clipper currently provides two `ContainerManager` implementations: the `DockerContainerManager` and the `KubernetesContainerManager`. In this exercise, you will be using the `DockerContainerManager`, which runs Clipper directly on your local Docker instance. This `ContainerManager` is particularly useful for trying out Clipper without needing to set up an enterprise-grade container orchestration framework. The `DockerContainerManager` is not recommended for production use cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a ClipperConnection\n",
    "To start Clipper, you must first create a [`ClipperConnection`](http://docs.clipper.ai/en/develop/#clipper-connection) object with the type of `ContainerManager` you want to use. In this case, you will be using the `DockerContainerManager`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from clipper_admin import ClipperConnection, DockerContainerManager\n",
    "clipper_conn = ClipperConnection(DockerContainerManager())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Starting Clipper\n",
    "Now that you have a `ClipperConnection` object, you can start a Clipper cluster.\n",
    "\n",
    "The following command will start 3 Docker containers:\n",
    "1. The Query Frontend: The Query Frontend container listens for incoming prediction requests and schedules and routes them to the deployed models.\n",
    "2. The Management Frontend: The Management Frontend container manages and updates the cluster's internal configuration state, such as tracking which models are deployed and which application endpoints have been registered.\n",
    "3. A Redis instance: Redis is used to persistently store Clipper's internal configuration state. By default, Redis is started on port 6380 instead of the standard Redis default port 6379 to avoid collisions with any Redis instances that are already running.\n",
    "\n",
    "<img src=\"notebook-images/start_clipper.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "> ***NOTE:*** *Because Docker must download the Docker images from the internet (if they are not already cached) before it can start the containers, the first time you run this command it may take a few minutes to complete. Once the images have been downloaded once, they will be cached and future invocations of this command will complete much more quickly. Thanks for your patience.*\n",
    "\n",
    "If you try to start more than one Clipper cluster at once on the same host, the second execution of the command will fail because, by default, the second cluster will try to bind to the same ports as the first one. If you run into problems with the exercise and want to start over, you can completely stop the cluster with [`clipper_conn.stop_all()`](http://docs.clipper.ai/en/v0.3.0/clipper_connection.html#clipper_admin.ClipperConnection.stop_all_model_containers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.start_clipper()\n",
    "clipper_addr = clipper_conn.get_query_addr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, let us take a look at the containers Clipper has started. It is interesting to note that one simple call to the Python API was able to spin up so many containers!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!docker ps --filter label=ai.clipper.container.label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploying a model\n",
    "At its most basic, a trained model is just a function that takes some input and produces some output. As a result, one way to think about Clipper is as a function server. While these functions are often complex models, Clipper is not restricted to serving machine learning models.\n",
    "\n",
    "Many machine learning models are trained in commonly used machine learning frameworks such as [Scikit-Learn](http://scikit-learn.org/), [TensorFlow](https://www.tensorflow.org/), [PyTorch](https://pytorch.org/), etc. To support these common use cases, Clipper offers a wide variety of deployers for common machine learning frameworks to simplify deployment. Furthermore, any frameworks that produce models that can be pickled -- this includes Scikit-Learn and XGBoost -- do not even require a special deployer and can be deployed directly with the standard Clipper Python closure deployer.\n",
    "\n",
    "In this exercise, you'll start by deploying a Scikit-Learn model using the Python closure deployer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating a model\n",
    "You will start by creating and training a three-class logistic regression classifier.\n",
    "\n",
    "Complete the TODO's in the next cells to build your model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The code to train the model and produce the graph comes from this sklearn example:\n",
    "# http://scikit-learn.org/stable/auto_examples/linear_model/plot_iris_logistic.html\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import linear_model, datasets\n",
    "%matplotlib inline\n",
    "\n",
    "np.random.seed(5)\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X = iris.data[:, :2]  # We only take the first two features.\n",
    "Y = iris.target\n",
    "\n",
    "model = linear_model.LogisticRegression(C=1e5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you've initialized your model, it's time to train it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Plot the decision boundary. For that, we will assign a color to each\n",
    "# point in the mesh [x_min, x_max]x[y_min, y_max].\n",
    "h = .02  # Step size in the mesh\n",
    "x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n",
    "y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "Z = model.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(xx.shape)\n",
    "plt.figure(1, figsize=(4, 3))\n",
    "plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired)\n",
    "\n",
    "# Plot also the training points\n",
    "plt.scatter(X[:, 0], X[:, 1], c=Y, edgecolors='k', cmap=plt.cm.Paired)\n",
    "plt.xlabel('Sepal length')\n",
    "plt.ylabel('Sepal width')\n",
    "\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n",
    "plt.xticks(())\n",
    "plt.yticks(())\n",
    "plt.title('3 Way Classifier')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clipper's Model Deployers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the goals of Clipper is to make it simple to deploy and maintain machine-learning models in production. The prediction interface that models must implement is very simple, consisting of a single function. And the use of Docker makes it easy to include all of a model's dependencies in a self-contained environment. However, deploying a new type of model still entails writing and debugging a new model container and creating a Docker image.\n",
    "To make the model deployment process even simpler, Clipper provides a library of model deployers for common types of models. If your model can be deployed with one of these deployers, you no longer need to write a model container, create a Docker image, or even figure out how to save a model. Instead, you provide your trained model directly to the model deployer function within your Python process. The model deployer takes care of saving the model and building a Docker image that is compatible with your model type.\n",
    "\n",
    "Clipper provides model deployers for many common ML packages including PySpark, PyTorch, TensorFlow, etc. In addition, Clipper provides a Python model deployer that can deploy arbitrary Python functions as long as they can be pickled. Finally, if none of these fit your needs, you can always write a custom model container that can execute arbitrary code.\n",
    "\n",
    "To keep the base images small, Clipper model containers install only the required dependencies to ensure that a basic model will run. However, if your model requires custom Python packages installed, you can specify these additional packages to the model deployer and it will automatically use Pip to install them inside your model's Docker container. You will use this to install Scikit-Learn when you deploy the flower model.\n",
    "\n",
    "For more information about model deployers, check out the [documentation](http://docs.clipper.ai/en/v0.3.0/model_deployers.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define a prediction function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have a model that takes an array of length 2 as input -- a petal width and a sepal length -- and returns a flower label. Before you can deploy the model using the Python model deployer, you need to wrap that model in a function that conforms to Clipper's prediction interface.\n",
    "\n",
    "To improve performance during inference, many machine learning models exploit opportunities for data parallelism in the inference process. Because of this, Clipper tries to provide multiple inputs at once to a deployed model. Therefore, models deployed to Clipper must have a function interface that takes a list of inputs as an argument and returns a list of predictions as strings. Returning predictions as strings provides a lot of flexibility over what your models can return. Commonly, models in Clipper will return either a single number (such as a label or score) or JSON containing a richer representation of the model output (for example, by including confidence estimates of predicted labels).\n",
    "\n",
    "Define a `predict_flower` function below that takes a list of inputs and returns their predicted labels as strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_label_from_class(l):\n",
    "    if l == 0:\n",
    "        return 'Setosa'\n",
    "    elif l == 1:\n",
    "        return 'Virginica'\n",
    "    else:\n",
    "        return 'Versicolour'\n",
    "\n",
    "\n",
    "def predict_flowers(flowers):\n",
    "    labels = \"FIXME\" # TODO: Use the model to make predict labels\n",
    "    return [get_label_from_class(l) for l in labels]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def predict_flowers(flowers):\n",
    "    labels = model.predict(flowers)\n",
    "    return [get_label_from_class(l) for l in labels]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the following cell to test your function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert predict_flowers([X[0], X[101]]) == ['Setosa', 'Virginica']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can use the Python model deployer to deploy your model to Clipper. The following cell may take a few seconds to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from clipper_admin.deployers import python as python_deployer\n",
    "python_deployer.deploy_python_closure(\n",
    "    clipper_conn,\n",
    "    name=\"flowercat\",  # The name of the model in Clipper\n",
    "    version=1,  # A unique identifier to assign to this model.\n",
    "    input_type=\"floats\",  # The type of data the model function expects as input\n",
    "    func=predict_flowers, # The model function to deploy\n",
    "    pkgs_to_install=['scipy', 'scikit-learn'] # Packages to install in the new container. Must be a list\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clipper deploys each model in its own Docker container. After deploying the model, Clipper uses the DockerContainerManager to start a container for this model and create an RPC connection with the Clipper query frontend, as illustrated below (the changes to the cluster are highlighted in red).\n",
    "\n",
    "> *Once again, Clipper must download a Docker container from the internet the first time this command is run.*\n",
    "\n",
    "<img src=\"notebook-images/deploy_model.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "If you list the Clipper containers again, you can see the container running your word count model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!docker ps --filter label=ai.clipper.container.label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A Note About Types [Optional]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you deploy models and register applications, you must specify the input type that the model or application expects. The type that you specify has implications for how Clipper manages input serialization and deserialization. From the user's perspective, the input type affects the behavior of Clipper in two places. In the \"input\" field of the request JSON body, applications will reject requests where the value of that field is the wrong type. And the deployed model function will be called with a list of inputs of the specified type.\n",
    "\n",
    "The input type can be one of the following types:\n",
    "\n",
    "* \"ints\": The value of the \"input\" field in a request must be a JSON list of ints. The model function will be called with a list of numpy arrays of type numpy.int.\n",
    "* \"floats\": The value of the \"input\" field in a request must be a JSON list of doubles. The model function will be called with a list of numpy arrays of type numpy.float32.\n",
    "* \"doubles\": The value of the \"input\" field in a request must be a JSON list of doubles. The model function will be called with a list of numpy arrays of type numpy.float64.\n",
    "* \"bytes\": The value of the \"input\" field in a request must be a Base64 encoded string. The model function will be called with a list of numpy arrays of type numpy.int8.\n",
    "* \"strings\": The value of the \"input\" field in a request must be a string. The model function will be called with a list of strings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Registering an Application\n",
    "You've now deployed a model to Clipper, but you don't have any way to query it yet. Instead of automatically creating a REST endpoint when you deploy a model, Clipper introduces a layer of indirection: the application. Clients query a specific application in Clipper, and the application routes the query to the correct model. One advantage of this choice is it decouples endpoint versioning from model versioning, enabling data scientists to rapidly iterate and update models without stopping or modifying a live application.\n",
    "\n",
    "A single Clipper cluster can have many applications registered and many models deployed at once.\n",
    "\n",
    "When you register an application you configure certain elements of the application's behavior. These include:\n",
    "\n",
    "* The name to give the REST endpoint.\n",
    "* The input type that the application expects (Clipper will ensure applications only route requests to models with matching input types).\n",
    "* The latency service level objective (SLO) specified in microseconds. Clipper will manage how it schedules and routes queries for an application based on the specified latency SLO. For example, Clipper uses the specified SLO to configure its batching behavior to balance maximizing resource utilization by using larger batches with ensuring that SLOs can be met. In addition, Clipper will respond to requests by the end of the specified SLO with a default response if it has not received a prediction back from the model yet.\n",
    "* The default output: Clipper will respond with the default output to requests if a real prediction isn't available by the end of the service level objective.\n",
    "\n",
    "<img src=\"notebook-images/register_app.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "Register an application to query your classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.register_application(\n",
    "    name=\"flowercat-app\",\n",
    "    input_type=\"floats\",\n",
    "    default_output=\"Default\",\n",
    "    slo_micros=100000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you register an application with Clipper, it creates a REST endpoint for that application:\n",
    "\n",
    "```\n",
    "URL: /<app_name>/predict\n",
    "Method: POST\n",
    "Data Params: {\"input\": <input>}\n",
    "```\n",
    "\n",
    "If you want to send several inputs at once, you can batch them into a single request by providing a JSON object with the key `input_batch`. Note that this is separate from Clipper's internal batching mechanism, which will be used regardless of whether the inputs are provided in a single REST request or separate requests.\n",
    "\n",
    "```\n",
    "URL: /<app_name>/predict\n",
    "Method: POST\n",
    "Data Params: {\"input_batch\": <input_batch>}\n",
    "```\n",
    "\n",
    "Try querying the newly created application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests, json\n",
    "response = requests.post(\n",
    "     \"http://%s/%s/predict\" % (clipper_addr, 'flowercat-app'),\n",
    "     headers={\"Content-type\": \"application/json\"},\n",
    "     data=json.dumps({\n",
    "         'input': list(X[0]),\n",
    "     }))\n",
    "result = response.json()\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see that your application returned the default output of \"Default\". This is because even though you have deployed a model and registered an application, you have not told Clipper to route requests from the \"flowercat-app\" application to the \"flowercat\" model.\n",
    "\n",
    "You can do this by linking the model to the application.\n",
    "\n",
    "<img src=\"notebook-images/link_model.png\" width=\"50%\" height=\"50%\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.link_model_to_app(app_name=\"flowercat-app\", model_name=\"flowercat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you query the \"flowercat-app\" endpoint again, Clipper should return a predicted label. Try it with your own input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "     \"http://%s/%s/predict\" % (clipper_addr, 'flowercat-app'),\n",
    "     headers={\"Content-type\": \"application/json\"},\n",
    "     data=json.dumps({\n",
    "         'input': \"FIXME\", # TODO: Specify an input\n",
    "     }))\n",
    "result = response.json()\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "response = requests.post(\n",
    "     \"http://%s/%s/predict\" % (clipper_addr, 'flowercat-app'),\n",
    "     headers={\"Content-type\": \"application/json\"},\n",
    "     data=json.dumps({\n",
    "         'input': list(X[0]), # TODO: Specify an input\n",
    "     }))\n",
    "result = response.json()\n",
    "result\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting Clipper\n",
    "The ClipperConnection object has several methods to inspect various aspects of the Clipper cluster.\n",
    "\n",
    "You can list all of the applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.get_all_apps(verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or all of the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.get_all_models(verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also fetch the raw container logs from all of the Clipper docker containers. The command will print the paths to the log files for further examination. You can figure out which logs belong to which container based on the unique Docker container ID in the log filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.get_clipper_logs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Updating the Model\n",
    "Machine learning models are rarely static. Instead, data science tends to be an iterative process, with new and improved models being developed over time. Clipper supports this workflow by letting you deploy new versions of models. If you look back to where you linked your flowercat model to the application, you'll see that there is no mention of versioning in that method call. Instead, when a new version of a model is deployed, Clipper will automatically start routing requests to the new version.\n",
    "\n",
    "Create a new version of the \"flowercat\" model that returns the probabilities that an input is in each class instead.\n",
    "\n",
    "> *Hint: Check out the [Scikit-Learn Logistic Regression documentation](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression.predict_proba).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def predict_flower_probabilities(flowers):\n",
    "    # TODO: Return the predicted probabilities.\n",
    "    return \"FIXME\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def predict_flower_probabilities(flowers):\n",
    "    return model.predict_proba(flowers)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predict_flower_probabilities([X[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deploy this new version of the function as version \"2\". For this application, you are using a numeric versioning scheme. But Clipper simply treats versions as unique string identifiers, so you could use other versioning schemes (such as Git hashes or semantic versioning). Versions don't even have to be ordered, Clipper just tracks the currently active version.\n",
    "\n",
    "<img src=\"notebook-images/update_model.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "The following cell may take a few seconds to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "python_deployer.deploy_python_closure(\n",
    "    clipper_conn,\n",
    "    name=\"flowercat\",\n",
    "    version=2, # Note that you are specifying a new version\n",
    "    input_type=\"floats\",\n",
    "    func=predict_flower_probabilities, # The new predict function\n",
    "    pkgs_to_install=['scipy', 'scikit-learn']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "     \"http://%s/%s/predict\" % (clipper_addr, 'flowercat-app'),\n",
    "     headers={\"Content-type\": \"application/json\"},\n",
    "     data=json.dumps({\n",
    "         'input': list(X[0]),\n",
    "     }))\n",
    "result = response.json()\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes the \"new and improved\" model is not actually improved. If you deploy a model that isn't working well, you can roll back to any previous version. This just changes which version of the model the application routes requests to.\n",
    "\n",
    "<img src=\"notebook-images/rollback_version.png\" width=\"50%\" height=\"50%\" />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.set_model_version(name=\"flowercat\", version=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "response = requests.post(\n",
    "     \"http://%s/%s/predict\" % (clipper_addr, 'flowercat-app'),\n",
    "     headers={\"Content-type\": \"application/json\"},\n",
    "     data=json.dumps({\n",
    "         'input': list(X[0]),\n",
    "     }))\n",
    "result = response.json()\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Model Replicas\n",
    "Many machine learning models are computationally expensive and a single instance of the model may not meet the throughput demands of a serving workload. To increase prediction throughput, you can horizontally scale the application by creating additional model replicas. This creates additional Docker containers running the same model. Clipper will automatically load-balance requests across the set of available model replicas.\n",
    "\n",
    "Set the number of replicas for the currently active version (1) of the \"flowercat\" model to 4.\n",
    "\n",
    "<img src=\"notebook-images/add_replicas.png\" width=\"50%\" height=\"50%\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.set_num_replicas(\"flowercat\", num_replicas=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you list the Clipper Docker containers, you should now see four containers based on the image \"flowercat:1\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!docker ps --filter label=ai.clipper.container.label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to reduce the number of replicas of a model to free up hardware resource, you can use the same command.\n",
    "\n",
    "Set the number of replicas for \"flowercat\" back to 1.\n",
    "\n",
    "<img src=\"notebook-images/set_replicas.png\" width=\"50%\" height=\"50%\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.set_num_replicas(\"flowercat\", num_replicas=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!docker ps --filter label=ai.clipper.container.label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Application - Image Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the second part of this exercise, you will deploy a pre-trained [SqueezeNet](https://arxiv.org/abs/1602.07360) model that you will download from the PyTorch model zoo to classify images.\n",
    "\n",
    "You will create an application that labels images from the ImageNet dataset.\n",
    "\n",
    "Some sample images have already been downloaded for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an application\n",
    "\n",
    "For this tutorial, create an application named \"squeezenet-classifier\". Note that Clipper allows you to create the application before deploying any models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "app_name = \"squeezenet-classsifier\"\n",
    "default_output = \"default\"\n",
    "\n",
    "clipper_conn.register_application(\n",
    "    name=app_name,\n",
    "    input_type=\"bytes\",\n",
    "    default_output=default_output,\n",
    "    slo_micros=10000000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you list the applications registered with Clipper, you should see the newly registered \"squeezenet-classifier\" application show up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.get_all_apps()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start serving\n",
    "\n",
    "As soon as you register the application, the REST endpoint is live, even though no models have been linked yet. However, in this exercsie you will wait until you've deployed and linked a model before querying the application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the pretrained model from the PyTorch model zoo\n",
    "\n",
    "Several common machine learning libraries, including PyTorch and TensorFlow, maintain a \"model zoo\" of pre-trained models that can be downloaded and used for inference without the need to train them. Most of the models in these model zoos are state-of-the-art deep-learning models which require an enormous amount of compute power (often several thousand GPU-hours) and expertise to train well. PyTorch provides a simple API to download and import models available in their computer vision model zoo.\n",
    "\n",
    "Download the SqueezeNet model from the PyTorch model zoo and import it into your notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torchvision import models, transforms\n",
    "model = models.squeezenet1_1(pretrained=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploying the PyTorch Model\n",
    "\n",
    "Unlike the Scikit-Learn model in [Section 1](#API-Overview), PyTorch models cannot just be pickled and loaded. Instead, they must be saved using PyTorch's native serialization API. Because of this, you cannot use the generic Python model deployer to deploy the model to Clipper. Instead, you will use the Clipper PyTorch deployer to deploy it. The Docker container will load and reconstruct the model from the serialized model checkpoint when the container is started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocessing\n",
    "Before making the predict function, you will download some labels for the dataset to test the accuracy of the pre-trained model, and specify some preprocessing logic to transform the images into the format the model expects. The code for this cell comes from the this [tutorial](http://blog.outcome.io/pytorch-quick-start-classifying-an-image/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First we define the preproccessing on the images:\n",
    "normalize = transforms.Normalize(\n",
    "   mean=[0.485, 0.456, 0.406],\n",
    "   std=[0.229, 0.224, 0.225]\n",
    ")\n",
    "preprocess = transforms.Compose([\n",
    "   transforms.Scale(256),\n",
    "   transforms.CenterCrop(224),\n",
    "   transforms.ToTensor(),\n",
    "   normalize\n",
    "])\n",
    "\n",
    "# Then we download the labels:\n",
    "labels = {int(key):value for (key, value)\n",
    "          in requests.get('https://s3.amazonaws.com/outcome-blog/imagenet/labels.json').json().items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a predict function and add metrics\n",
    "Now you can define a predict function to deploy. This predict function will wrap the model you downloaded from the model zoo as well as the preprocessing logic you just specified.\n",
    "\n",
    "You will also use Clipper's metrics reporting functionality to export some custom metrics for this model. In particular, you will report the batch size for each batch sent to the model. These metrics will be aggregated along with several system metrics and reported to [Prometheus](https://prometheus.io/), a time-series database commonly used for metrics reporting (for example, Kubernetes uses Prometheus for its own system metrics). You can view a simple metrics dashboard automatically exported by Prometheus at port 9090 (`http:://[IP_ADDRESS]:9090`). In practice, a monitoring and alerting system such as [Grafana](https://grafana.com/) is typically layered on top of Promtheus. You will work through an example of connecting Grafana to visualize Clipper's metrics later in the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import clipper_admin.metrics as metrics\n",
    "\n",
    "def predict_torch_model(model, imgs):\n",
    "    import io\n",
    "    import PIL.Image\n",
    "    import torch\n",
    "    import clipper_admin.metrics as metrics\n",
    "    \n",
    "    metrics.add_metric(\"batch_size\", 'Gauge', 'Batch size passed to PyTorch predict function.')\n",
    "    metrics.report_metric('batch_size', \"FIXME\") # TODO: Fill in the batch size\n",
    "    \n",
    "    # We first prepare a batch from `imgs`\n",
    "    img_tensors = []\n",
    "    for img in imgs:\n",
    "        img_tensor = preprocess(PIL.Image.open(io.BytesIO(img)))\n",
    "        img_tensor.unsqueeze_(0)\n",
    "        img_tensors.append(img_tensor)\n",
    "    img_batch = torch.cat(img_tensors)\n",
    "    \n",
    "    # We perform a forward pass\n",
    "    with torch.no_grad():\n",
    "        model_output = model(img_batch)\n",
    "        \n",
    "    # Parse Result\n",
    "    img_labels = [labels[out.data.numpy().argmax()] for out in model_output]\n",
    "        \n",
    "    return img_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```py\n",
    "metrics.report_metric('batch_size', len(imgs))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *Once again, Clipper must download this Docker image from the internet, so this may take a minute. Thanks for your patience.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from clipper_admin.deployers import pytorch as pytorch_deployer\n",
    "pytorch_deployer.deploy_pytorch_model(\n",
    "    clipper_conn,\n",
    "    name=\"pytorch-model\",\n",
    "    version=1, \n",
    "    input_type=\"bytes\", \n",
    "    func=\"FIXME\", # TODO: Provide the predict function wrapper\n",
    "    pytorch_model=\"FIXME\", # TODO: Pass model to function\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Solution:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from clipper_admin.deployers import pytorch as pytorch_deployer\n",
    "pytorch_deployer.deploy_pytorch_model(\n",
    "    clipper_conn,\n",
    "    name= \"pytorch-model\",\n",
    "    version=1, \n",
    "    input_type=\"bytes\", \n",
    "    func=predict_torch_model, # TODO: Provide the predict function wrapper\n",
    "    pytorch_model= model, # TODO: Pass model to function\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.link_model_to_app(app_name=\"squeezenet-classsifier\", model_name=\"pytorch-model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize the metrics, you will connect Grafana to our Prometheus server. First, start Grafana in a Docker container. You will start it on port 3000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!docker run --label ai.clipper.container.label -d -p 3000:3000 grafana/grafana"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you open up the Grafana Dashboard, we need to explain how to add Prometheus as a data source. Please read to the end of this section to get instructions on how to set up Grafana and import a pre-made dashboard. There will be a code block to run at the end that will give you the URL to use to access Grafana.\n",
    "\n",
    "When you go to Grafana, you will be asked to log in. You can use the username `admin` and password `admin` to log in.\n",
    "\n",
    "Next, click the green `Add Datasource` button, which will take you to the page shown in the image below.\n",
    "\n",
    "<img src=\"notebook-images/grafana.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "Give the data source a name, such as `Clipper Metrics`, then change the settings so that `Type` is `Prometheus`, which you can select from the drop down menu. For the `URL` field under the `HTTP` tab, please run the following code block and copy paste the generated URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from IPython.display import display, Markdown\n",
    "\n",
    "this_ip = requests.get('http://ip.42.pl/raw').text\n",
    "\n",
    "display(Markdown(f\"\"\"\n",
    "Prometheus address: http://{this_ip}:9090\n",
    "\"\"\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also adjust the scrape interval so that Grafana queries Prometheus more often. When you are done, your setup page should look like the image below.\n",
    "\n",
    "<img src=\"notebook-images/grafana_w_info.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "When you click `Save and Test`, the page should show that the data source was successfully added, like so:\n",
    "\n",
    "<img src=\"notebook-images/grafana_test_succeeded.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "Next, click the plus button in the sidebar and select `Dashboard`:\n",
    "\n",
    "<img src=\"notebook-images/grafana_add_dashboard.png\" />\n",
    "\n",
    "This will take you to the following screen, where you should click `New Dashboard` in the upper left of the screen:\n",
    "\n",
    "<img src=\"notebook-images/grafana_new_dashboard.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "Which will take you to this screen, where you should click `Import Dashboard` in the lower right box\n",
    "\n",
    "<img src=\"notebook-images/grafana_import_dashboard.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "That will take you to this next screen, where you should copy and paste the ID `7904` into the `Grafana.com Dashboard` field like so:\n",
    "\n",
    "<img src=\"notebook-images/grafana_dashboard_id.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "This will lead you to the final page, with settings for the Dashboard. The last step is to select `Clipper Metrics` (or whatever else you named the data source you added at the start) from the drop down menu in the last field labelled `Clipper Metrics`. Once you do so, you can hit `Import` to view your dashboard. Be sure to send more queries to generate interesting metrics!\n",
    "\n",
    "<img src=\"notebook-images/grafana_set_data_source.png\" width=\"50%\" height=\"50%\" />\n",
    "\n",
    "To open Grafana, run the following code block, and navigate to the URL it outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(Markdown(f\"\"\"This is the location of your Grafana instance:\n",
    "    http://{this_ip}:3000\n",
    "\"\"\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "import json\n",
    "req_json = json.dumps({\n",
    "        \"input\":\n",
    "        base64.b64encode(open('images/cat.jpg', \"rb\").read()).decode() # bytes to unicode\n",
    "    })\n",
    "\n",
    "response = requests.post(\n",
    "     \"http://%s/%s/predict\" % (clipper_addr, 'squeezenet-classsifier'),\n",
    "     headers={\"Content-type\": \"application/json\"},\n",
    "     data=req_json)\n",
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *Note that squeezenet is a deep-learning model (specifically, a convolutional neural network). Even though it was developed to be relatively fast, it may take a few seconds to return a prediction since it is running on a CPU. Deploying the model to a GPU will accelerate it substantially.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's important to note that since we are sending requests in series, there aren't multiple requests in the system at any one time, and therefore the system will not have multiple queries to batch together. You will still be able to observe some batching, however, thanks to Clipper's batching exploration algorithm. Clipper will occasionally inject additional queries into the system (when it is underloaded) to explore the throughput-latency tradeoff for each model. This exploration algorithm estimates a function that predicts query latency based on the batch size, thereby enabling Clipper to choose the largest batch size (and therefore the highest throughput) that is still capable of returning within the deadline imposed by the latency SLO. Clipper estimates a different function for each model container, allowing the system to adapt to different model characteristics and varying resource loads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, Image\n",
    "# Blue is Rehan's cat's name. The last picture is an image of his cat.\n",
    "images = ['images/cat.jpg', 'images/dog.jpg', 'images/duck.jpg', 'images/blue.jpg']\n",
    "for img in images:\n",
    "    print('Output for', img)\n",
    "    display(Image(img, width=200, height=200))\n",
    "    req_json = json.dumps({\n",
    "            \"input\":\n",
    "            base64.b64encode(open(img, \"rb\").read()).decode() # bytes to unicode\n",
    "        })\n",
    "\n",
    "    response = requests.post(\n",
    "         \"http://%s/%s/predict\" % (clipper_addr, 'squeezenet-classsifier'),\n",
    "         headers={\"Content-type\": \"application/json\"},\n",
    "         data=req_json)\n",
    "    print(response.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Application - Custom Docker Containers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this final example, you will learn how to create a custom Docker model container. Custom Docker containers can be utilized when model containers rely on dependencies that cannot be installed with Pip. In this example, you will deploying a model developed in a pure C framework that must be compiled from source within the Docker container.\n",
    "\n",
    "You will deploy the [YoloV3 Real Time Object Detection System](https://pjreddie.com/darknet/yolo/) to predict both *labels and bounding boxes* for objects within an image. This is a harder machine learning task than the image classification task that you performed with SqueezeNet.\n",
    "\n",
    "YOLO (You Only Look Once) is a state-of-the-art object detection model, but unfortunately it does not expose a native Python API. Instead, it must be compiled from source and executed as a binary executable, not as a library. You will clone the repo (in this case, a fork we created with some minor modifications to extract the bounding box coordinates instead of directly drawing the on the image) and compile it, all within the Docker image.\n",
    "\n",
    "The first step in creating a Docker image is to write a [Dockerfile](https://docs.docker.com/engine/reference/builder/). This is like the source code for the image. After you have written the Dockerfile, you will build it. Unlike when you compile source code, building a Docker image actually runs the program specified in the Dockerfile. The end result is a Docker image -- a binary object that contains the results of executing the instructions in the Dockerfile sequentially from top to bottom. \n",
    "\n",
    "For your convenience, we have already written a Dockerfile for you called `PyDarknetDockerfile` and included it in the tutorial. However, before you build it you will read about each step in the Dockerfile, providing a blueprint for you to define your own custom Clipper model containers in the future.\n",
    "\n",
    "Here are the full contents of the Dockerfile:\n",
    "\n",
    "```dockerfile\n",
    "FROM clipper/python36-closure-container:0.3\n",
    "\n",
    "# Install Git\n",
    "RUN apt-get update \\\n",
    "    && apt-get install -y git\n",
    "\n",
    "# Install cURL\n",
    "RUN apt-get install -y curl\n",
    "\n",
    "# Clone Darknet Repo\n",
    "RUN git clone https://github.com/RehanSD/darknet.git /tmp/darknet\n",
    "RUN mv /tmp/darknet/* .\n",
    "\n",
    "# Make Darknet Project\n",
    "RUN make -j4\n",
    "\n",
    "#Download Weights\n",
    "RUN curl -o yolov3-tiny.weights https://pjreddie.com/media/files/yolov3-tiny.weights\n",
    "```\n",
    "\n",
    "In the following sections, we will break down each part of the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first line of the file ensures that you are starting off with a Docker image with all of the necessary Clipper dependencies installed. In the rest of the Dockerfile, you are extending the Clipper-provided container to simply install some additional dependencies.\n",
    "``` dockerfile\n",
    "FROM clipper/python36-closure-container:0.3\n",
    "```\n",
    "\n",
    "The next few lines install the basic dependencies we need in the container - git to clone the repo, annd curl to download the pretrained yolo model weights.\n",
    "\n",
    "```dockerfile\n",
    "# Install Git\n",
    "RUN apt-get update \\\n",
    "    && apt-get install -y git\n",
    "\n",
    "# Install cURL\n",
    "RUN apt-get install -y curl\n",
    "```\n",
    "\n",
    "The last few lines clone the darknet repo, compile it, and then download the pretrained model weights. We are using `yolov3-tiny` because it offers a good trade off between accuracy and inference speed on a CPU. (For a latency-sensitive production use case, you might run it on GPU.)\n",
    "\n",
    "```dockerfile\n",
    "# Clone Darknet Repo\n",
    "RUN git clone https://github.com/RehanSD/darknet.git /tmp/darknet\n",
    "RUN mv /tmp/darknet/* .\n",
    "\n",
    "# Make Darknet Project\n",
    "RUN make -j4\n",
    "\n",
    "#Download Weights\n",
    "RUN curl -o yolov3-tiny.weights https://pjreddie.com/media/files/yolov3-tiny.weights\n",
    "```\n",
    "\n",
    "\n",
    "Now that you have defined the image, you can build it with the following shell command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!docker build -t clipper/darknet-yolov3-container -f PyDarknetDockerfile ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict Function\n",
    "Now we build the predict function. We must first deserialize the image that we serialized in our request - similar to how we did in the PyTorch example, and then write it to a file to run darknet on it. Since darknet is a C executable, we must call it using the subprocess API. It is not strictly neccessary to print out the output of calling darknet, but it is useful to have the output for the sake of debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import subprocess\n",
    "import base64\n",
    "import io\n",
    "import os\n",
    "def yolo_pred(imgs):\n",
    "    import base64\n",
    "    import io\n",
    "    import os\n",
    "    import tempfile\n",
    "    import subprocess\n",
    "\n",
    "    num_imgs = len(imgs)\n",
    "    ret_coords = []\n",
    "    predict_procs = []\n",
    "    file_names = []\n",
    "    \n",
    "    # First, we save the images to file\n",
    "    for i in range(num_imgs):\n",
    "        # Create a temp file to write to\n",
    "        tmp = tempfile.NamedTemporaryFile('wb', delete=False, suffix='.jpg')\n",
    "        tmp.write(io.BytesIO(imgs[i]).getvalue())\n",
    "        tmp.close()\n",
    "        file_names.append(tmp.name)\n",
    "    \n",
    "    # Second, we call ./darknet executable to detect objects in images.\n",
    "    #   This is done in parallel.\n",
    "    for file_name in file_names:\n",
    "        process = subprocess.Popen(\n",
    "            ['./darknet',\n",
    "             'detector',\n",
    "             'test',\n",
    "             './cfg/coco.data',\n",
    "             './cfg/yolov3-tiny.cfg',\n",
    "             './yolov3-tiny.weights',\n",
    "             file_name,\n",
    "             '-json',\n",
    "             '-dont_show',\n",
    "             '-ext_output', '>',\n",
    "             '{}.txt'.format(file_name+'_result')], stdout=subprocess.PIPE)\n",
    "        predict_procs.append(process)\n",
    "        \n",
    "    # Lastly, we wait for all process to finished and return stdout of each process\n",
    "    for process in predict_procs:\n",
    "        process.wait()\n",
    "        ret_coords += [' '.join(map(lambda byte_str: byte_str.decode(), process.stdout))]\n",
    "\n",
    "    return ret_coords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Do not be concerned if this cell takes a couple of seconds to run.\n",
    "from clipper_admin.deployers import python as python_deployer\n",
    "python_deployer.deploy_python_closure(\n",
    "    clipper_conn,\n",
    "    name=\"yolov3\",  # The name of the model in Clipper\n",
    "    version=1,  # A unique identifier to assign to this model.\n",
    "    input_type=\"bytes\",  # The type of data the model function expects as input\n",
    "    func=yolo_pred, # The model function to deploy\n",
    "    base_image='clipper/darknet-yolov3-container'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.register_application(\n",
    "    name=\"darknet-app\",\n",
    "    input_type=\"bytes\",\n",
    "    default_output=\"Default\",\n",
    "    slo_micros=10000000 # 10 seconds\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.link_model_to_app(app_name=\"darknet-app\", model_name=\"yolov3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Please note the request may take a couple of seconds to return, as we are running a largeish model\n",
    "# on a CPU, which is slow.\n",
    "import requests, json\n",
    "url = \"http://%s/darknet-app/predict\" % clipper_addr\n",
    "req_json = json.dumps({\n",
    "    \"input\":\n",
    "    base64.b64encode(open('images/dog.jpg', \"rb\").read()).decode() # bytes to unicode\n",
    "})\n",
    "headers = {'Content-type': 'application/json'}\n",
    "r = requests.post(url, headers=headers, data=req_json)\n",
    "\n",
    "# Let's see what does YoloV3-tiny return\n",
    "print(r.json())\n",
    "print()\n",
    "print(r.json()['output'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's plot the result\n",
    "def _get_bounding_boxes(output_string):\n",
    "    \"\"\"Yield Rectangle object from output string\"\"\"\n",
    "    import re\n",
    "    import matplotlib.patches as patches\n",
    "    bbox_regex = re.compile(r\" ([a-z]+): [\\d]{2}%\\n Left: ([\\d]{1,3}), Bottom: ([\\d]{1,3}), Right: ([\\d]{1,3}), Top: ([\\d]{1,3})\")\n",
    "    matched = bbox_regex.findall(output_string)\n",
    "    \n",
    "    colors = ['b', 'g', 'r', 'c', 'm', 'y', 'k']\n",
    "    for c, m in zip(colors, matched):\n",
    "        cat, left, bottom, right, top = m\n",
    "        left, bottom, right, top = int(left), int(bottom), int(right), int(top)\n",
    "        yield patches.Rectangle((left,bottom), right-left, top-bottom, linewidth=1, facecolor='none', edgecolor=c,label=cat)\n",
    "\n",
    "def plot_bbox(im_name, output_string):\n",
    "    \"\"\"Plot the image with bbox overlay\"\"\"\n",
    "    import matplotlib.pyplot as plt\n",
    "    import matplotlib.patches as patches\n",
    "    from PIL import Image\n",
    "    import numpy as np\n",
    "    \n",
    "    fig,ax = plt.subplots(1, figsize=(10,8))\n",
    "    ax.imshow(np.array(Image.open(im_name)))\n",
    "    for p in _get_bounding_boxes(output_string):\n",
    "        ax.add_patch(p) \n",
    "    plt.title(\"Prediction result for image: {}\".format(im_name))\n",
    "    fig.legend()\n",
    "\n",
    "def predict_and_plot(im_name):\n",
    "    \"\"\"Make a prediction and plot image with bbox\"\"\"\n",
    "    url = \"http://%s/darknet-app/predict\" % clipper_addr\n",
    "    req_json = json.dumps({\n",
    "        \"input\":\n",
    "        base64.b64encode(open(im_name, \"rb\").read()).decode() # bytes to unicode\n",
    "    })\n",
    "    headers = {'Content-type': 'application/json'}\n",
    "    r = requests.post(url, headers=headers, data=req_json)\n",
    "    print(r.json())\n",
    "    plot_bbox(im_name, r.json()['output'])\n",
    "\n",
    "plot_bbox('images/dog.jpg', r.json()['output'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to experiment with other images under `images/detection/*.jpg`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls images/detection/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please note the request may take a couple of seconds to return, as we are running a largeish model\n",
    "# on a CPU, which is slow.\n",
    "# im_name = \"images/detection/*.jpg\"\n",
    "im_name = \"images/detection/cityscapes-3.jpg\"\n",
    "predict_and_plot(im_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stopping Clipper\n",
    "If you run into issues and want to completely stop Clipper, you can do this by calling [`ClipperConnection.stop_all()`](http://docs.clipper.ai/en/latest/#clipper_admin.ClipperConnection.stop_all)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clipper_conn.stop_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you list all the Docker containers a final time, you should see that all of the Clipper containers have been stopped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!docker ps --filter label=ai.clipper.container.label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now call `clipper_conn.start_clipper()` again without running into errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please continue with the rest of the tutorial in [this notebook](2-Pong-Game.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}