notebook_examples/aqua-deploy-llm-byoc.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "748139a6",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "@notebook{aqua-deploy-llm-byoc.ipynb,\n",
    "    title: Deploy LLM Models using BYOC,\n",
    "    summary: Deploy and perform inferencing using AI Quick Action models.,\n",
    "    developed_on: generalml_p311_cpu_x86_64_v1,\n",
    "    keywords: byoc, llm, quick action, deploy\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "443c3053",
   "metadata": {},
   "source": [
    "# Deploy LLM Models using BYOC\n",
    "\n",
    "This guide demonstrates how to deploy and perform inferencing using AI Quick Action registered models with Oracle Data Science Service Managed Containers (SMC) powered by vLLM. In this example, we will use a model downloaded from Hugging Face specifically, [Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) from Meta. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7b65a7a",
   "metadata": {},
   "source": [
    "## Required IAM Policies\n",
    "\n",
    "Add these [policies](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/llama2#required-iam-policies) to grant access to OCI services.\n",
    "\n",
    "## Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "06577ea6-2400-422c-b177-b9241137fbc0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Install required python packages\n",
    "\n",
    "!pip install oracle-ads\n",
    "!pip install oci\n",
    "!pip install huggingface_hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8993f0c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment this code and set the correct proxy links if have to setup proxy for internet\n",
    "# import os\n",
    "# os.environ['http_proxy']=\"http://myproxy\"\n",
    "# os.environ['https_proxy']=\"http://myproxy\"\n",
    "\n",
    "# Use os.environ['no_proxy'] to route traffic directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5b286246-71cd-4b5f-9864-0b8099a61974",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import ads\n",
    "import os\n",
    "\n",
    "ads.set_auth(\"resource_principal\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd6862f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract region information from the Notebook environment variables and signer.\n",
    "ads.common.utils.extract_region()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37a822e7",
   "metadata": {},
   "source": [
    "### Common variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fec27465",
   "metadata": {},
   "outputs": [],
   "source": [
    "# change as required for your environment\n",
    "compartment_id = os.environ[\"PROJECT_COMPARTMENT_OCID\"]\n",
    "project_id = os.environ[\"PROJECT_OCID\"]\n",
    "\n",
    "log_group_id = \"ocid1.loggroup.oc1.xxx.xxxxx\"\n",
    "log_id = \"cid1.log.oc1.xxx.xxxxx\"\n",
    "\n",
    "instance_shape = \"VM.GPU.A10.2\"\n",
    "\n",
    "region = \"<your-region>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6cf4280",
   "metadata": {},
   "source": [
    "## API Endpoint Usage\n",
    "\n",
    "The `/v1/completions` is for interacting with non-chat base models or the instruction trained chat model. This endpoint provides the completion for a single prompt and takes a single string as input, whereas the `/v1/chat/completions` endpoint provides the responses for a given dialog and requires the input in a specific format corresponding to the message history. This guide uses `/v1/chat/completions` endpoint.\n",
    "\n",
    "\n",
    "## Prepare The Model Artifacts\n",
    "\n",
    "To prepare Model artifacts for LLM model deployment:\n",
    "\n",
    "- Download the model files from huggingface to local directory using a valid huggingface token (only needed for gated models). If you don't have Huggingface Token, refer [this](https://huggingface.co/docs/hub/en/security-tokens) to generate one.\n",
    "- Upload the model folder to a [versioned bucket](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/usingversioning.htm) in Oracle Object Storage. If you don’t have an Object Storage bucket, create one using the OCI SDK or the Console. Create an Object Storage bucket. Make a note of the `namespace`, `compartment`, and `bucketname`. Configure the policies to allow the Data Science service to read and write the model artifact to the Object Storage bucket in your tenancy. An administrator must configure the policies in IAM in the Console.\n",
    "- Create model catalog entry for the model using the Object storage path\n",
    "\n",
    "### Model Download from HuggingFace Model Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed7f9920",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Login to huggingface using env variable\n",
    "HUGGINGFACE_TOKEN = \"<HUGGINGFACE_TOKEN>\"  # Your huggingface token\n",
    "!huggingface-cli login --token $HUGGINGFACE_TOKEN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d732d64d",
   "metadata": {},
   "source": [
    "[This](https://huggingface.co/docs/huggingface_hub/en/guides/cli#download-an-entire-repository) provides more information on using `huggingface-cli` to download an entire repository at a given revision. Models in the HuggingFace hub are stored in their own repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c9202bc-5134-49ed-b45c-1a3de23db060",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the the model that you want to deploy. Currently it is set to Meta-Llama3.1-8B-Instruct\n",
    "model_name = \"meta-llama/Meta-Llama-3.1-8B-Instruct\"\n",
    "local_dir = \"models/Meta-Llama-3.1-8B-Instruct\"\n",
    "\n",
    "!huggingface-cli download $model_namet --local-dir $local_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa12c79-b937-4633-b560-d423bb29592d",
   "metadata": {},
   "source": [
    "## Upload Model to OCI Object Storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ac60c1d-3f7b-438f-88c3-d865dab2b6a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_prefix = \"Meta-Llama-3.1-8B-Instruct/\"  # \"<bucket_prefix>\"\n",
    "bucket = \"<bucket_name>\"  # this should be a versioned bucket\n",
    "namespace = \"<bucket_namespace>\"\n",
    "\n",
    "!oci os object bulk-upload --src-dir $local_dir --prefix $model_prefix -bn $bucket -ns $namespace --auth \"resource_principal\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "253e2528-5d8a-4fb5-a066-fd7aad489409",
   "metadata": {},
   "source": [
    "## Create Model by Reference using ADS\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d538e7c6-fe9c-423f-a32d-bee59890f3f4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from ads.model.datascience_model import DataScienceModel\n",
    "\n",
    "artifact_path = f\"oci://{bucket}@{namespace}/{model_prefix}\"\n",
    "\n",
    "model = (\n",
    "    DataScienceModel()\n",
    "    .with_compartment_id(compartment_id)\n",
    "    .with_project_id(project_id)\n",
    "    .with_display_name(\"Meta-Llama-3.1-405B-Instruct-FP8\")\n",
    "    .with_artifact(artifact_path)\n",
    ")\n",
    "\n",
    "model.create(model_by_reference=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "321ed2d7",
   "metadata": {},
   "source": [
    "## Inference container\n",
    "\n",
    "vLLM is an easy-to-use library for LLM inference and server.  You can get the container image from [DockerHub](https://hub.docker.com/r/vllm/vllm-openai/tags).\n",
    "\n",
    "```python\n",
    "docker pull --platform linux/amd64 vllm/vllm-openai:latest\n",
    "```\n",
    "\n",
    "Currently, OCI Data Science Model Deployment only supports container images residing in the OCI Registry.  Before we can push the pulled vLLM container, make sure you have created a repository in your tenancy.  \n",
    "- Go to your tenancy Container Registry\n",
    "- Click on the Create repository button\n",
    "- Select Private under Access types\n",
    "- Set a name for Repository name.  We are using \"vllm-odsc \"in the example.\n",
    "- Click on Create button\n",
    "\n",
    "You may need to docker login to the Oracle Cloud Container Registry (OCIR) first, if you haven't done so before in order to push the image. To login, you have to use your API Auth Token that can be created under your Oracle Cloud Account->Auth Token. You need to login only once. Replace <region> with the OCI region you are using.\n",
    "\n",
    "```python\n",
    "docker login -u '<tenant-namespace>/<username>' <region>.ocir.io\n",
    "```\n",
    "\n",
    "If your tenancy is federated with Oracle Identity Cloud Service, use the format <tenancy-namespace>/oracleidentitycloudservice/<username>. You can then push the container image to the OCI Registry\n",
    "\n",
    "```python\n",
    "docker tag vllm/vllm-openai:v0.5.3.post1 -t <region>.ocir.io/<tenancy>/vllm-odsc/vllm-openai:latest\n",
    "docker push <region>.ocir.io/<tenancy>/vllm-odsc/vllm-openai:latest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ec1391a",
   "metadata": {},
   "outputs": [],
   "source": [
    "container_image = \"<region>.ocir.io/<tenancy>/vllm-odsc/vllm-openai:latest\"  # name given to vllm image pushed to oracle  container registry"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "475066cf-d67a-41b7-9732-9cb5e03baab8",
   "metadata": {},
   "source": [
    "### Import Model Deployment Modules\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d0e7220e-7262-4fd6-801e-a63da486ac9a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from ads.model.deployment import (\n",
    "    ModelDeployment,\n",
    "    ModelDeploymentContainerRuntime,\n",
    "    ModelDeploymentInfrastructure,\n",
    "    ModelDeploymentMode,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "313e873f-a74e-40f8-8af8-63a47c8d09f3",
   "metadata": {},
   "source": [
    "## Setup Model Deployment Infrastructure\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "021bf449-ac92-4bf6-a457-85027d50f77b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "infrastructure = (\n",
    "    ModelDeploymentInfrastructure()\n",
    "    .with_project_id(project_id)\n",
    "    .with_compartment_id(compartment_id)\n",
    "    .with_shape_name(instance_shape)\n",
    "    .with_bandwidth_mbps(10)\n",
    "    .with_replica(1)\n",
    "    .with_web_concurrency(1)\n",
    "    .with_access_log(\n",
    "        log_group_id=log_group_id,\n",
    "        log_id=log_id,\n",
    "    )\n",
    "    .with_predict_log(\n",
    "        log_group_id=log_group_id,\n",
    "        log_id=log_id,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a968590-6752-42f8-b6b6-f08c0893b281",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Configure Model Deployment Runtime\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "c8704bfc-4078-44ce-91a2-dc90871d7312",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "env_var = {\n",
    "    \"MODEL_DEPLOY_PREDICT_ENDPOINT\": \"/v1/completions\",\n",
    "}\n",
    "\n",
    "cmd_var = [\n",
    "    \"--model\",\n",
    "    f\"/opt/ds/model/deployed_model/{model_prefix}\",\n",
    "    \"--tensor-parallel-size\",\n",
    "    \"2\",\n",
    "    \"--port\",\n",
    "    \"8080\",\n",
    "    \"--served-model-name\",\n",
    "    \"odsc-llm\",\n",
    "    \"--host\",\n",
    "    \"0.0.0.0\",\n",
    "    \"--trust-remote-code\",\n",
    "]\n",
    "\n",
    "container_runtime = (\n",
    "    ModelDeploymentContainerRuntime()\n",
    "    .with_image(container_image)\n",
    "    .with_server_port(8080)\n",
    "    .with_health_check_port(8080)\n",
    "    .with_env(env_var)\n",
    "    .with_cmd(cmd_var)\n",
    "    .with_deployment_mode(ModelDeploymentMode.HTTPS)\n",
    "    .with_model_uri(model.id)\n",
    "    .with_region(region)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59b35e7b-76d4-4aab-8523-52dfe35d7ef3",
   "metadata": {},
   "source": [
    "## Deploy Model using Container Runtime\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "520c85aa-78ab-4df4-b6fb-eafd7ef5272e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "deployment = (\n",
    "    ModelDeployment()\n",
    "    .with_display_name(f\"{model_prefix} MD with BYOC\")\n",
    "    .with_description(f\"Deployment of {model_prefix} MD with vLLM BYOC container\")\n",
    "    .with_infrastructure(infrastructure)\n",
    "    .with_runtime(container_runtime)\n",
    ").deploy(wait_for_completion=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73c0f30c-b873-44b7-bba5-66ce7d73f723",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "deployment.watch()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e40643e-3b05-45b8-931f-586e04389620",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8b33722-41db-4e2a-a41a-ef665dae347e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from string import Template\n",
    "from datetime import datetime\n",
    "\n",
    "\n",
    "auth = ads.common.auth.default_signer()[\"signer\"]\n",
    "prompt = \"What amateur radio bands are best to use when there are solar flares?\"\n",
    "endpoint = f\"https://modeldeployment.us-ashburn-1.oci.customer-oci.com/{deployment.model_deployment_id}/predict\"\n",
    "\n",
    "current_date = datetime.now().strftime(\"%d %B %Y\")\n",
    "prompt_template = Template(\n",
    "    \"\"\"\"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n",
    "\n",
    "                    Cutting Knowledge Date: December 2023\n",
    "                    Today Date: {current_date}\n",
    "\n",
    "                    You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n",
    "\n",
    "                    $prompt<|eot_id|><|start_header_id|>assistant<|end_header_id|>\"\"\"\n",
    ")\n",
    "\n",
    "prompt = t.substitute(\n",
    "    prompt=\"What amateur radio bands are best to use when there are solar flares?\"\n",
    ")\n",
    "\n",
    "body = {\n",
    "    \"model\": \"odsc-llm\",  # this is a constant\n",
    "    \"prompt\": prompt,\n",
    "    \"max_tokens\": 500,\n",
    "    \"temperature\": 0,\n",
    "    \"top_p\": 0.9,\n",
    "}\n",
    "requests.post(endpoint, json=body, auth=auth, headers={}).json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf231669",
   "metadata": {},
   "source": [
    "#### Output:\n",
    "\n",
    "The raw output:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"data\": {\n",
    "    \"choices\": [\n",
    "      {\n",
    "        \"finish_reason\": \"stop\",\n",
    "        \"index\": 0,\n",
    "        \"logprobs\": null,\n",
    "        \"stop_reason\": null,\n",
    "        \"text\": \"\\n\\nDuring solar flares, radio communications can be disrupted due to increased ionization and geomagnetic storms. However, some amateur radio bands are more resilient to these conditions than others. Here are some bands that are often considered best to use during solar flares:\\n\\n1. **VHF (30 MHz - 300 MHz) and UHF (300 MHz - 3 GHz) bands**: These higher frequency bands are less affected by solar flares and geomagnetic storms. They are also less prone to ionospheric absorption, which can attenuate signals on lower frequency bands.\\n2. **6 meters (50 MHz)**: This band is often considered a good choice during solar flares, as it is less affected by ionospheric disturbances and can provide reliable local and regional communications.\\n3. **2 meters (144 MHz) and 70 cm (440 MHz)**: These bands are popular for local and regional communications and are often less affected by solar flares.\\n4. **Microwave bands (e.g., 1.2 GHz, 2.4 GHz, 5.8 GHz)**: These bands are even less affected by solar flares and can provide reliable communications over shorter distances.\\n\\nBands to avoid during solar flares:\\n\\n1. **HF (3 MHz - 30 MHz) bands**: These lower frequency bands are more susceptible to ionospheric absorption and geomagnetic storms, which can cause signal loss and disruption.\\n2. **160 meters (1.8 MHz) and 80 meters (3.5 MHz)**: These bands are often the most affected by solar flares and geomagnetic storms.\\n\\nKeep in mind that the impact of solar flares on amateur radio communications can vary depending on the intensity of the flare, the location of the communicating stations, and the time of day. It's always a good idea to monitor space weather forecasts and adjust your communication plans accordingly.\"\n",
    "      }\n",
    "    ],\n",
    "    \"created\": 1721939892,\n",
    "    \"id\": \"cmpl-4aac6ee35ffd477eaedadbb973efde18\",\n",
    "    \"model\": \"llama3.1\",\n",
    "    \"object\": \"text_completion\",\n",
    "    \"usage\": {\n",
    "      \"completion_tokens\": 384,\n",
    "      \"prompt_tokens\": 57,\n",
    "      \"total_tokens\": 441\n",
    "    }\n",
    "  },\n",
    "```\n",
    "\n",
    "During solar flares, radio communications can be disrupted due to increased ionization and geomagnetic storms. However, some amateur radio bands are more resilient to these conditions than others. Here are some bands that are often considered best to use during solar flares:\n",
    "\n",
    "1. **VHF (30 MHz - 300 MHz) and UHF (300 MHz - 3 GHz) bands**: These higher frequency bands are less affected by solar flares and geomagnetic storms. They are also less prone to ionospheric absorption, which can attenuate signals on lower frequency bands.\n",
    "2. **6 meters (50 MHz)**: This band is often considered a good choice during solar flares, as it is less affected by ionospheric disturbances and can provide reliable local and regional communications.\n",
    "3. **2 meters (144 MHz) and 70 cm (440 MHz)**: These bands are popular for local and regional communications and are often less affected by solar flares.\n",
    "4. **Microwave bands (e.g., 1.2 GHz, 2.4 GHz, 5.8 GHz)**: These bands are even less affected by solar flares and can provide reliable communications over shorter distances.\n",
    "\n",
    "Bands to avoid during solar flares:\n",
    "\n",
    "1. **HF (3 MHz - 30 MHz) bands**: These lower frequency bands are more susceptible to ionospheric absorption and geomagnetic storms, which can cause signal loss and disruption.\n",
    "2. **160 meters (1.8 MHz) and 80 meters (3.5 MHz)**: These bands are often the most affected by solar flares and geomagnetic storms.\n",
    "\n",
    "Keep in mind that the impact of solar flares on amateur radio communications can vary depending on the intensity of the flare, the location of the communicating stations, and the time of day. It's always a good idea to monitor space weather forecasts and adjust your communication plans accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c5a523",
   "metadata": {},
   "source": [
    "\n",
    "#### Using the model from [LangChain](https://python.langchain.com/v0.1/docs/integrations/llms/oci_model_deployment_endpoint/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8de1e18d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.llms import OCIModelDeploymentVLLM\n",
    "from string import Template\n",
    "from datetime import datetime\n",
    "\n",
    "ads.set_auth(\"resource_principal\")\n",
    "current_date = datetime.now().strftime(\"%d %B %Y\")\n",
    "\n",
    "llm = OCIModelDeploymentVLLM(\n",
    "    endpoint=f\"https://modeldeployment.us-ashburn-1.oci.customer-oci.com/{deployment.model_deployment_id}/predict\",\n",
    "    model=\"odsc-llm\",\n",
    ")\n",
    "\n",
    "llm.invoke(\n",
    "    input=Template(\n",
    "        f\"\"\"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n",
    "\n",
    "                    Cutting Knowledge Date: December 2023\n",
    "                    Today Date:{current_date}\n",
    "\n",
    "                    You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n",
    "\n",
    "                    $prompt<|eot_id|><|start_header_id|>assistant<|end_header_id|>\"\"\"\n",
    "    ).substitute(\n",
    "        prompt=\"What amateur radio bands are best to use when there are solar flares?\"\n",
    "    ),\n",
    "    max_tokens=500,\n",
    "    temperature=0.7,\n",
    "    p=0.8,\n",
    "    stop=[\"<|eot_id|>\"],\n",
    "    skip_special_tokens=False,\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch21_p39_gpu_v1]",
   "language": "python",
   "name": "conda-env-pytorch21_p39_gpu_v1-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}