introduction-to-bedrock/prompt-caching/getting_started_with_prompt_caching.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c3b87af2-6de5-4055-b410-237fff246f98",
   "metadata": {},
   "source": [
    "# Prompt Caching in Amazon Bedrock\n",
    "\n",
    "Prompt caching is a powerful feature in Amazon Bedrock that significantly reduces response latency for workloads with repetitive contexts. This notebook demonstrates how to implement prompt caching effectively for document-based chat applications.\n",
    "\n",
    "## What is Prompt Caching?\n",
    "\n",
    "Prompt caching allows you to store portions of your conversation context, enabling models to:\n",
    "- Reuse cached context instead of reprocessing inputs\n",
    "- Reduce response Time-To-First-Token (TTFT) for subsequent queries\n",
    "\n",
    "## When to Use Prompt Caching\n",
    "\n",
    "Prompt caching delivers maximum benefits for:\n",
    "- **Chat with Document**: By caching the document as input context on the first request, each user query becomes more efficient, perhaps enabling simpler architectures that avoid heavier solutions like vector databases.\n",
    "- **Coding assistants**: Reusing long code files in prompts enables near real-time inline suggestions, eliminating much of the time spent reprocessing code files.\n",
    "- **Agentic workflows**: Longer system prompts can be used to refine agent behavior without degrading the end-user experience. By caching the system prompts and complex tool definitions, the time to process each step in the agentic flow can be reduced.\n",
    "- **Few-Shot Learning**: Including numerous high-quality examples and complex instructions, such as for customer service or technical troubleshooting, can benefit from prompt caching.\n",
    "\n",
    "## Benefits of Prompt Caching\n",
    "\n",
    "- **Faster Response Times**: Avoid reprocessing the same context repeatedly\n",
    "- **Improved User Experience**: Reduced TTFT to create more natural conversations\n",
    "- **Cost Efficiency**: Potentially lower token usage by avoiding redundant processing\n",
    "\n",
    "## Implementation Example\n",
    "\n",
    "This notebook walks through a document-based chat implementation using prompt caching to demonstrate:\n",
    "1. How to properly structure cache points in your requests\n",
    "2. Performance comparisons with and without caching\n",
    "3. Best practices for cache management\n",
    "4. Measuring and optimizing cache effectiveness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8673631a-102f-49d6-a809-7ced95a81256",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install --upgrade boto3 pandas numpy matplotlib seaborn pytz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "057a87ff-672a-4307-a367-e49c26d2b0b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard libraries\n",
    "import json\n",
    "import time\n",
    "from enum import Enum\n",
    "\n",
    "# Data processing and visualization\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patheffects as path_effects\n",
    "import seaborn as sns\n",
    "\n",
    "# AWS and external services\n",
    "import boto3\n",
    "import requests\n",
    "\n",
    "bedrock_runtime = boto3.client('bedrock-runtime')\n",
    "boto3_bedrock = boto3.client('bedrock')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82524009-78ae-4ebf-b32f-5b8c65dff472",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Info:</b> You will need the latest boto3 which includes prompt caching in the Converse API.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5853d36-b884-4272-ad01-9de605fa2f0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"boto3 version: {boto3.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "478b986e-15f7-445e-ad79-7c368dc79091",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Info:</b> This notebook uses Anthropic Claude 3.5 Haiku as an example, please make sure you have enabled the model on Bedrock\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f574b52c-563f-4db2-ab21-f3fb7f5880dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "[models['modelId'] for models in boto3_bedrock.list_foundation_models()['modelSummaries']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5dcd14a-6c6f-4572-9793-bbe7091589f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id=\"us.anthropic.claude-3-5-haiku-20241022-v1:0\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7423052e-d2bf-4576-bc5e-5c264b1f7d74",
   "metadata": {},
   "source": [
    "### Use case: Chat with document\n",
    "\n",
    "To effectively use Prompt Caching, there is a [minimum number of tokens](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html#prompt-caching-models). Thus we need to request a long doc here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b77409a9-1e59-46df-aa94-d04a46c1ad60",
   "metadata": {},
   "outputs": [],
   "source": [
    "topics = [\n",
    "    'https://aws.amazon.com/blogs/aws/reduce-costs-and-latency-with-amazon-bedrock-intelligent-prompt-routing-and-prompt-caching-preview/',\n",
    "    'https://aws.amazon.com/blogs/machine-learning/enhance-conversational-ai-with-advanced-routing-techniques-with-amazon-bedrock/',\n",
    "    'https://aws.amazon.com/blogs/security/cost-considerations-and-common-options-for-aws-network-firewall-log-management/'\n",
    "]\n",
    "\n",
    "questions = [\n",
    "    'what is it about?',\n",
    "    'what are the use cases?',\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0162905-9bbe-45f4-b3f8-37cae0720a5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def chat_with_document(document, user_query, model_id):\n",
    "    instructions = (\n",
    "    \"I will provide you with a document, followed by a question about its content. \"\n",
    "    \"Your task is to analyze the document, extract relevant information, and provide \"\n",
    "    \"a comprehensive answer to the question. Please follow these detailed instructions:\"\n",
    "\n",
    "    \"\\n\\n1. Identifying Relevant Quotes:\"\n",
    "    \"\\n   - Carefully read through the entire document.\"\n",
    "    \"\\n   - Identify sections of the text that are directly relevant to answering the question.\"\n",
    "    \"\\n   - Select quotes that provide key information, context, or support for the answer.\"\n",
    "    \"\\n   - Quotes should be concise and to the point, typically no more than 2-3 sentences each.\"\n",
    "    \"\\n   - Choose a diverse range of quotes if multiple aspects of the question need to be addressed.\"\n",
    "    \"\\n   - Aim to select between 2 to 5 quotes, depending on the complexity of the question.\"\n",
    "\n",
    "    \"\\n\\n2. Presenting the Quotes:\"\n",
    "    \"\\n   - List the selected quotes under the heading 'Relevant quotes:'\"\n",
    "    \"\\n   - Number each quote sequentially, starting from [1].\"\n",
    "    \"\\n   - Present each quote exactly as it appears in the original text, enclosed in quotation marks.\"\n",
    "    \"\\n   - If no relevant quotes can be found, write 'No relevant quotes' instead.\"\n",
    "    \"\\n   - Example format:\"\n",
    "    \"\\n     Relevant quotes:\"\n",
    "    \"\\n     [1] \\\"This is the first relevant quote from the document.\\\"\"\n",
    "    \"\\n     [2] \\\"This is the second relevant quote from the document.\\\"\"\n",
    "\n",
    "    \"\\n\\n3. Formulating the Answer:\"\n",
    "    \"\\n   - Begin your answer with the heading 'Answer:' on a new line after the quotes.\"\n",
    "    \"\\n   - Provide a clear, concise, and accurate answer to the question based on the information in the document.\"\n",
    "    \"\\n   - Ensure your answer is comprehensive and addresses all aspects of the question.\"\n",
    "    \"\\n   - Use information from the quotes to support your answer, but do not repeat them verbatim.\"\n",
    "    \"\\n   - Maintain a logical flow and structure in your response.\"\n",
    "    \"\\n   - Use clear and simple language, avoiding jargon unless it's necessary and explained.\"\n",
    "\n",
    "    \"\\n\\n4. Referencing Quotes in the Answer:\"\n",
    "    \"\\n   - Do not explicitly mention or introduce quotes in your answer (e.g., avoid phrases like 'According to quote [1]').\"\n",
    "    \"\\n   - Instead, add the bracketed number of the relevant quote at the end of each sentence or point that uses information from that quote.\"\n",
    "    \"\\n   - If a sentence or point is supported by multiple quotes, include all relevant quote numbers.\"\n",
    "    \"\\n   - Example: 'The company's revenue grew by 15% last year. [1] This growth was primarily driven by increased sales in the Asian market. [2][3]'\"\n",
    "\n",
    "    \"\\n\\n5. Handling Uncertainty or Lack of Information:\"\n",
    "    \"\\n   - If the document does not contain enough information to fully answer the question, clearly state this in your answer.\"\n",
    "    \"\\n   - Provide any partial information that is available, and explain what additional information would be needed to give a complete answer.\"\n",
    "    \"\\n   - If there are multiple possible interpretations of the question or the document's content, explain this and provide answers for each interpretation if possible.\"\n",
    "\n",
    "    \"\\n\\n6. Maintaining Objectivity:\"\n",
    "    \"\\n   - Stick to the facts presented in the document. Do not include personal opinions or external information not found in the text.\"\n",
    "    \"\\n   - If the document presents biased or controversial information, note this objectively in your answer without endorsing or refuting the claims.\"\n",
    "\n",
    "    \"\\n\\n7. Formatting and Style:\"\n",
    "    \"\\n   - Use clear paragraph breaks to separate different points or aspects of your answer.\"\n",
    "    \"\\n   - Employ bullet points or numbered lists if it helps to organize information more clearly.\"\n",
    "    \"\\n   - Ensure proper grammar, punctuation, and spelling throughout your response.\"\n",
    "    \"\\n   - Maintain a professional and neutral tone throughout your answer.\"\n",
    "\n",
    "    \"\\n\\n8. Length and Depth:\"\n",
    "    \"\\n   - Provide an answer that is sufficiently detailed to address the question comprehensively.\"\n",
    "    \"\\n   - However, avoid unnecessary verbosity. Aim for clarity and conciseness.\"\n",
    "    \"\\n   - The length of your answer should be proportional to the complexity of the question and the amount of relevant information in the document.\"\n",
    "\n",
    "    \"\\n\\n9. Dealing with Complex or Multi-part Questions:\"\n",
    "    \"\\n   - For questions with multiple parts, address each part separately and clearly.\"\n",
    "    \"\\n   - Use subheadings or numbered points to break down your answer if necessary.\"\n",
    "    \"\\n   - Ensure that you've addressed all aspects of the question in your response.\"\n",
    "\n",
    "    \"\\n\\n10. Concluding the Answer:\"\n",
    "    \"\\n    - If appropriate, provide a brief conclusion that summarizes the key points of your answer.\"\n",
    "    \"\\n    - If the question asks for recommendations or future implications, include these based strictly on the information provided in the document.\"\n",
    "\n",
    "    \"\\n\\nRemember, your goal is to provide a clear, accurate, and well-supported answer based solely on the content of the given document. \"\n",
    "    \"Adhere to these instructions carefully to ensure a high-quality response that effectively addresses the user's query.\"\n",
    "    )\n",
    "\n",
    "    document_content =  f\"Here is the document:  <document> {document} </document>\"\n",
    "\n",
    "    messages_body = [\n",
    "        {\n",
    "            'role': 'user',\n",
    "            'content': [\n",
    "                {\n",
    "                'text': instructions\n",
    "                },\n",
    "                {\n",
    "                \"cachePoint\": {\n",
    "                    \"type\": \"default\"\n",
    "                }\n",
    "            },\n",
    "                {\n",
    "                'text': document_content\n",
    "                },\n",
    "                {\n",
    "                \"cachePoint\": {\n",
    "                    \"type\": \"default\"\n",
    "                }\n",
    "            },\n",
    "                {\n",
    "                'text': user_query\n",
    "                },\n",
    "            ]\n",
    "        },\n",
    "    ]\n",
    "\n",
    "    inference_config={\n",
    "        'maxTokens': 500,\n",
    "        'temperature': 0,\n",
    "        'topP': 1\n",
    "    }\n",
    "\n",
    "    response = bedrock_runtime.converse(\n",
    "                messages=messages_body,\n",
    "                modelId=model_id,\n",
    "                inferenceConfig=inference_config\n",
    "            )\n",
    "\n",
    "    output_message = response[\"output\"][\"message\"]\n",
    "    response_text = output_message[\"content\"][0][\"text\"]\n",
    "\n",
    "    print(\"Response text:\")\n",
    "    print(response_text)\n",
    "\n",
    "    print(\"Usage:\")\n",
    "    print(json.dumps(response[\"usage\"], indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92d725b9-0880-4d4b-9933-d41ad2a34a13",
   "metadata": {},
   "source": [
    "### Converse API with Prompt Caching\n",
    "\n",
    "**First invocation**\n",
    "\n",
    "When you first use the Converse API with prompt caching enabled, you'll initiate the cache creation process (indicated by **cacheWriteInputTokenCount** in the response). This is what happens during the first invocation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed1b6fc1-79d3-47ef-af52-cd4977cb4f30",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(topics[0])\n",
    "blog = response.text\n",
    "\n",
    "chat_with_document(\n",
    "    document=blog,\n",
    "    user_query=questions[0],\n",
    "    model_id=model_id\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe28e92-e26c-4838-9340-b7c792279689",
   "metadata": {},
   "source": [
    "**Subsequent invocations**\n",
    "\n",
    "When you submit a different question about the same document, the LLM retrieves context from the cache instead of reprocessing the entire document and you can observe the **cacheReadInputTokenCount** metric in the response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34aa4f53-df86-4499-8cc1-9a4d3f51bd73",
   "metadata": {},
   "outputs": [],
   "source": [
    "chat_with_document(\n",
    "    document=blog,\n",
    "    user_query=questions[1],\n",
    "    model_id=model_id\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f3b4daa-0d2e-43e4-a02d-9f7f8dde9b3b",
   "metadata": {},
   "source": [
    "**Multiple cache checkpoints**\n",
    "\n",
    "The **chat_with_document** function implements an advanced caching strategy with two distinct cache checkpoints:\n",
    "\n",
    "1. After the instruction prompt\n",
    "2. After the document content\n",
    "   \n",
    "This dual-checkpoint approach enables more granular cache utilization. When you switch to a different document while maintaining the same instructions, the system:\n",
    "\n",
    "- Retrieves the instruction context from cache (cacheReadInputTokenCount)\n",
    "- Creates a new cache entry for the document content (cacheWriteInputTokenCount)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b490ae0-e722-4b33-a027-f15f4dc82859",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(topics[1])\n",
    "blog = response.text\n",
    "\n",
    "chat_with_document(\n",
    "    document=blog,\n",
    "    user_query=questions[0],\n",
    "    model_id=model_id\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c8302c11-eaf5-48ca-8382-626d93cff77b",
   "metadata": {},
   "source": [
    "### Latency Optimization with Prompt Caching\n",
    "#### Measuring Performance Gains\n",
    "Prompt caching significantly improves time-to-first-token (TTFT) performance when working with repetitive contexts. The following benchmarking approach quantifies these improvements:\n",
    "\n",
    "**Performance Metrics**\n",
    "When evaluating prompt caching effectiveness, focus on these key metrics:\n",
    "\n",
    "- Time to First Token (TTFT): The interval between request submission and receipt of the first response token\n",
    "- Total Latency: Complete request-to-response completion time (indicated by Time-To-Last-Token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10f4be66-f123-4d85-a3c9-c777f4764f79",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(topics[2])\n",
    "blog = response.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d688b578-a01c-4798-a669-2cfedc4cf867",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CACHE(Enum):\n",
    "    ON = \"on\"\n",
    "    OFF = \"off\"\n",
    "    READ = \"read\"\n",
    "    WRITE = \"write\"\n",
    "    \n",
    "    def __lt__(self, other):\n",
    "        if self.__class__ is other.__class__:\n",
    "            return self.value < other.value\n",
    "        return NotImplemented"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fae355a-7f65-4b91-8f5b-442a7e6fe011",
   "metadata": {},
   "outputs": [],
   "source": [
    "TESTS = [\n",
    "    # {\n",
    "    #     'model_id':\"us.anthropic.claude-3-7-sonnet-20250219-v1:0\",\n",
    "    #     'model_name':\"Anthropic.Sonnet-3.7\",\n",
    "    #     'cache_mode': [CACHE.ON, CACHE.OFF],\n",
    "    # },\n",
    "    {\n",
    "        'model_id':\"us.anthropic.claude-3-5-haiku-20241022-v1:0\",\n",
    "        'model_name':\"Anthropic.Haiku-3.5\",\n",
    "        'cache_mode': [CACHE.ON, CACHE.OFF],\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d3219e6-c145-4238-a0e9-7043aa76b600",
   "metadata": {},
   "outputs": [],
   "source": [
    "converse_stream_api_cmd = {\n",
    "  \"modelId\": \"\",\n",
    "  \"messages\": [\n",
    "    {\n",
    "      \"role\": \"user\",\n",
    "      \"content\": []\n",
    "    }\n",
    "  ],\n",
    "  \"inferenceConfig\": {\n",
    "    \"maxTokens\": 100,\n",
    "    \"temperature\": 0.0\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "602cb954-23a8-4933-8b37-87e671d8ec6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def benchmark_ttft(test_configs, epochs=3):\n",
    "    datapoints = []\n",
    "    \n",
    "    for test_config in test_configs:        \n",
    "        print(\"[{}]\".format(test_config['model_name']))\n",
    "        \n",
    "        converse_stream_api_cmd[\"modelId\"] = test_config['model_id']\n",
    "            \n",
    "        for cache_mode in test_config['cache_mode']:\n",
    "            converse_cmd = converse_stream_api_cmd.copy()  # Create a copy for modification\n",
    "            \n",
    "            converse_cmd[\"messages\"][0][\"content\"] = [\n",
    "                {\n",
    "                    \"text\": blog\n",
    "                },\n",
    "                {\n",
    "                    \"text\": \"what is it about in 20 words.\"\n",
    "                }\n",
    "            ]\n",
    "            \n",
    "            if cache_mode == CACHE.ON:\n",
    "                converse_cmd[\"messages\"][0][\"content\"].append({\n",
    "                    \"cachePoint\": {\n",
    "                        \"type\": \"default\"\n",
    "                    }\n",
    "                })\n",
    "            \n",
    "            for epoch in range(epochs):\n",
    "                start_time = time.time()\n",
    "                \n",
    "                response = bedrock_runtime.converse_stream(**converse_cmd)\n",
    "                \n",
    "                time.time() - start_time\n",
    "                \n",
    "                ttft = None\n",
    "                \n",
    "                for i, chunk in enumerate(response['stream']):    \n",
    "                    if \"messageStart\" in chunk:\n",
    "                        pass\n",
    "                    elif \"contentBlockStop\" in chunk:\n",
    "                        pass\n",
    "                    elif \"messageStop\" in chunk:\n",
    "                        pass\n",
    "                    elif \"contentBlockDelta\" in chunk:\n",
    "                        text = chunk[\"contentBlockDelta\"].get('delta',{}).get(\"text\",None) \n",
    "                        if text is not None and not text:\n",
    "                            print('<empty>', end='')\n",
    "                        if text is not None:\n",
    "                            if not ttft:\n",
    "                                ttft = time.time() - start_time\n",
    "                    \n",
    "                    elif \"metadata\" in chunk:        \n",
    "                        if 'cacheReadInputTokenCount' in chunk[\"metadata\"]['usage']:\n",
    "                            if chunk[\"metadata\"]['usage']['cacheWriteInputTokens'] > 1:\n",
    "                                cache_result = CACHE.WRITE\n",
    "                            elif chunk[\"metadata\"]['usage']['cacheReadInputTokens'] > 1:\n",
    "                                cache_result = CACHE.READ\n",
    "                            else:\n",
    "                                print(json.dumps(chunk, sort_keys=False, indent=4))\n",
    "                                assert False, 'Unclear'\n",
    "                        else:\n",
    "                            cache_result = CACHE.OFF\n",
    "                                                \n",
    "                        latencyMs = chunk[\"metadata\"][\"metrics\"][\"latencyMs\"] / 1000\n",
    "                        requestId = response['ResponseMetadata']['RequestId']\n",
    "                        \n",
    "                        datapoints.append({ \n",
    "                            'model': test_config['model_name'],\n",
    "                            'cache': cache_result,\n",
    "                            'measure': 'first_token',\n",
    "                            'time': ttft,\n",
    "                            'requestId': requestId,\n",
    "                        })\n",
    "                        \n",
    "                        datapoints.append({ \n",
    "                            'model': test_config['model_name'],\n",
    "                            'cache': cache_result,\n",
    "                            'measure': 'last_token',\n",
    "                            'time': latencyMs,\n",
    "                            'requestId': requestId,\n",
    "                        })\n",
    "                        \n",
    "                        print(f\"{epoch:2} {cache_mode},{cache_result} | ttft={ttft:.1f}s | last={latencyMs:.1f}s | {requestId}\")\n",
    "                    \n",
    "                    else:\n",
    "                        end_time = time.time()\n",
    "                        print('\\n\\nchunk +{:.3f}s \\n{}'.format(\n",
    "                            time.time()-end_time,\n",
    "                            json.dumps(chunk, sort_keys=False, indent=4)\n",
    "                        ))\n",
    "                \n",
    "                time.sleep(30)\n",
    "    \n",
    "    return datapoints"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eedd7b89-c134-4c91-aed6-3732e6a920ba",
   "metadata": {},
   "source": [
    "**Run the benchmark**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d975e6b-0034-4818-a961-dfcf9921adc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "datapoints = benchmark_ttft(TESTS, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f13aefa3-1a01-47c9-b002-aad9efb9f934",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(datapoints)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5497af7e-8ac4-4c8b-bc9d-86cfd4612e3a",
   "metadata": {},
   "source": [
    "**Visualize the results**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c70af2a-2630-45af-8eb4-0eb120454c9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_median_labels(ax: plt.Axes, fmt: str = \".1f\") -> None:\n",
    "    \"\"\"Add text labels to the median lines of a seaborn boxplot.\n",
    "\n",
    "    Args:\n",
    "        ax: plt.Axes, e.g. the return value of sns.boxplot()\n",
    "        fmt: format string for the median value\n",
    "    \"\"\"\n",
    "    lines = ax.get_lines()\n",
    "    boxes = [c for c in ax.get_children() if \"Patch\" in str(c)]\n",
    "    start = 4\n",
    "    if not boxes:  # seaborn v0.13 => fill=False => no patches => +1 line\n",
    "        boxes = [c for c in ax.get_lines() if len(c.get_xdata()) == 5]\n",
    "        start += 1\n",
    "    lines_per_box = len(lines) // len(boxes)\n",
    "    for median in lines[start::lines_per_box]:\n",
    "        x, y = (data.mean() for data in median.get_data())\n",
    "        # choose value depending on horizontal or vertical plot orientation\n",
    "        value = x if len(set(median.get_xdata())) == 1 else y\n",
    "        text = ax.text(x, y, f'{value:{fmt}}', ha='center', va='center', color='white')\n",
    "        # create median-colored border around white text for contrast\n",
    "        text.set_path_effects([\n",
    "            path_effects.Stroke(linewidth=3, foreground=median.get_color()),\n",
    "            path_effects.Normal(),\n",
    "        ])\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "534d8f02-64da-4629-b850-c68985c87be6",
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set_style(\"whitegrid\")\n",
    "n_models = df['model'].nunique()\n",
    "\n",
    "f, axes = plt.subplots(n_models, 1, figsize=(6, n_models * 6.4))\n",
    "\n",
    "# Convert axes to array if there's only one model\n",
    "axes = np.array([axes]) if n_models == 1 else axes\n",
    "\n",
    "for i, model in enumerate(df['model'].unique()):\n",
    "    cond = df['model'] == model\n",
    "    df_i = df.loc[cond]\n",
    "    \n",
    "    ax = sns.boxplot(df_i,\n",
    "                     ax=axes[i],\n",
    "                     x='measure', \n",
    "                     y='time', \n",
    "                     hue=df_i[['cache']].apply(tuple, axis=1))        \n",
    "\n",
    "    ax.tick_params(axis='x', rotation=45)\n",
    "    ax.set_xlabel(None)\n",
    "    add_median_labels(ax)\n",
    "    ax.legend(loc='upper left')\n",
    "\n",
    "\n",
    "plt.title('Time to First Token (TTFT) by Cache Mode - us.haiku-3.5', fontsize=16)\n",
    "plt.xlabel('Cache Mode', fontsize=14)\n",
    "plt.ylabel('Time (seconds)', fontsize=14)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ae3bf18c-e87e-47a5-91a8-c781fd785e63",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This notebook expolored Amazon Bedrock's prompt caching feature, demonstrating how it works, when to use it, and how to use it effectively. It's important to carefully evaluate whether your use case will benefit from this feature. It depends on thoughtful prompt structuring, understanding the distinction between static and dynamic content, and selecting appropriate caching strategies for your specific needs.\n",
    "\n",
    "For more information about working with prompt caching on Amazon Bedrock, see the [Amazon Bedrock User Guide](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c9544ba-5df7-4666-b640-d24df87dc13a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}