MLOps/Pipelines/Vertex AI Pipelines - Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "403cb049",
   "metadata": {},
   "source": [
    "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Fvertex-ai-mlops%2FMLOps%2FPipelines&file=Vertex+AI+Pipelines+-+Introduction.ipynb)\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/MLOps/Pipelines/Vertex%20AI%20Pipelines%20-%20Introduction.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Fvertex-ai-mlops%2Fmain%2FMLOps%2FPipelines%2FVertex%2520AI%2520Pipelines%2520-%2520Introduction.ipynb\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/statmike/vertex-ai-mlops/blob/main/MLOps/Pipelines/Vertex%20AI%20Pipelines%20-%20Introduction.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/vertex-ai-mlops/main/MLOps/Pipelines/Vertex%20AI%20Pipelines%20-%20Introduction.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0e7a164-2a66-4b7f-8b15-1b5f246c8880",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "This is part of a [series of notebook based workflows](./readme.md) that teach all the ways to use pipelines within Vertex AI. The suggested order and description/reason is:\n",
    "\n",
    "||Notebook Workflow|Description|\n",
    "|---|---|---|\n",
    "||[Vertex AI Pipelines - Start Here](./Vertex%20AI%20Pipelines%20-%20Start%20Here.ipynb)|What are pipelines? Start here to go from code to pipeline and see it in action.|\n",
    "|_**This Notebook**_|[Vertex AI Pipelines - Introduction](./Vertex%20AI%20Pipelines%20-%20Introduction.ipynb)|Introduction to pipelines with the console and Vertex AI SDK|\n",
    "||[Vertex AI Pipelines - Components](./Vertex%20AI%20Pipelines%20-%20Components.ipynb)|An introduction to all the ways to create pipeline components from your code|\n",
    "||[Vertex AI Pipelines - IO](./Vertex%20AI%20Pipelines%20-%20IO.ipynb)|An overview of all the type of inputs and outputs for pipeline components|\n",
    "||[Vertex AI Pipelines - Control](./Vertex%20AI%20Pipelines%20-%20Control.ipynb)|An overview of controlling the flow of exectution for pipelines|\n",
    "||[Vertex AI Pipelines - Secret Manager](./Vertex%20AI%20Pipelines%20-%20Secret%20Manager.ipynb)|How to pass sensitive information to pipelines and components|\n",
    "||[Vertex AI Pipelines - GCS Read and Write](./Vertex%20AI%20Pipelines%20-%20GCS%20Read%20and%20Write.ipynb)|How to read/write to GCS from components, including container components.|\n",
    "||[Vertex AI Pipelines - Scheduling](./Vertex%20AI%20Pipelines%20-%20Scheduling.ipynb)|How to schedule pipeline execution|\n",
    "||[Vertex AI Pipelines - Notifications](./Vertex%20AI%20Pipelines%20-%20Notifications.ipynb)|How to send email notification of pipeline status.|\n",
    "||[Vertex AI Pipelines - Management](./Vertex%20AI%20Pipelines%20-%20Management.ipynb)|Managing, Reusing, and Storing pipelines and components|\n",
    "||[Vertex AI Pipelines - Testing](./Vertex%20AI%20Pipelines%20-%20Testing.ipynb)|Strategies for testing components and pipeliens locally and remotely to aide development.|\n",
    "||[Vertex AI Pipelines - Managing Pipeline Jobs](./Vertex%20AI%20Pipelines%20-%20Managing%20Pipeline%20Jobs.ipynb)|Manage runs of pipelines in an environment: list, check status, filtered list, cancel and delete jobs.|\n",
    "\n",
    "\n",
    "To discover these notebooks as part of an introduction to MLOps orchestration [start here](./readme.md).  To read more about MLOps also check out [the parent folder](../readme.md).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9608a8fd-eec7-4c9f-9d30-78e1275ad987",
   "metadata": {},
   "source": [
    "# Vertex AI Pipelines - Introduction\n",
    "\n",
    "When an ML workflow has more than one step it can benefit from a pipeline.  [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction) is a managed service that executes [Kubeflow Pipelines (KFP)](https://www.kubeflow.org/docs/components/pipelines/v2/introduction/) and [TensorFlow Extended (TFX)](https://www.tensorflow.org/tfx/guide/understanding_tfx_pipelines) pipelines.  For this introduction the focus will be on KFP due to its vast flexibility, it was even originally developed as a simplified way of running TFX on Kubernetes - [history of Kubeflow](https://www.kubeflow.org/docs/started/introduction/#history).\n",
    "This notebook based workflow will introduce KFP pipelines runninng on Vertex AI Pipelines and point to the additional detailed workflows within [this repository](./readme.md) that help with a deeper understanding.\n",
    "\n",
    "<p align=\"center\"><center>\n",
    "    <img src=\"../resources/images/screenshots/pipelines/intro/overview.png\" width=\"75%\">\n",
    "</center><p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "730ff978-b3d0-4b92-9780-7e861b34afa2",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Pipelines** - The Concept\n",
    "\n",
    "Pipelines are constructed from steps.  The steps are called **tasks** and utilize **components**.\n",
    "\n",
    "The tasks are connected by inputs and outputs.  This forms the execution graph of the pipeline.\n",
    "\n",
    "Inputs and outputs can be made up of **parameters** (`str`, `int`, `float`, `bool`, `dict`, `list`) as well as **artifacts**.  Artifacts are multiparameter objects that have defined schemas for machine learning objects (datasets, models, etc.) and automatically get stored as [Vertex AI ML Metadata](https://cloud.google.com/vertex-ai/docs/ml-metadata/introduction) and have lineage.\n",
    "\n",
    "**Kubeflow Pipelines** - Constructing Pipelines\n",
    "\n",
    "Pipelines are written in Python using the [`kfp` package](https://www.kubeflow.org/docs/components/pipelines/v2/installation/).  This SDK has decorators for multiple ways of writing [**components**](https://www.kubeflow.org/docs/components/pipelines/v2/components/) and constructing [**pipelines**](https://www.kubeflow.org/docs/components/pipelines/v2/pipelines/) from the components with features to control the execution and flow of the pipeline. Pipeline code is [**compiled**](https://www.kubeflow.org/docs/components/pipelines/v2/compile-a-pipeline/) into a YAML file using a single line command with the SDK.  \n",
    "\n",
    "**Vertex AI Pipelines** - Executing Pipelines\n",
    "\n",
    "The resulting YAML file is the input for [running the pipeline](https://cloud.google.com/vertex-ai/docs/pipelines/run-pipeline) with Vertex AI Pipelines.  The runs can even be scheduled with [Vertex AI Pipeline scheduler API](https://cloud.google.com/vertex-ai/docs/pipelines/schedule-pipeline-run).\n",
    "\n",
    "The runs of a pipeline can be directly monitored, and compared, in the Vertex AI console:\n",
    "\n",
    "**Pipeline Dashboard View In The Console: Overall Pipeline**\n",
    "<p align=\"center\"><center>\n",
    "    <img src=\"../resources/images/screenshots/pipelines/intro/overview.png\" width=\"75%\">\n",
    "</center><p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcb1be06-0703-4f05-a650-def0afb2f027",
   "metadata": {
    "id": "od_UkDpvRmgD",
    "tags": []
   },
   "source": [
    "---\n",
    "## Colab Setup\n",
    "\n",
    "To run this notebook in Colab run the cells in this section.  Otherwise, skip this section.\n",
    "\n",
    "This cell will authenticate to GCP (follow prompts in the popup)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9722edbb-c3ea-4376-bda5-173093b85d1d",
   "metadata": {
    "executionInfo": {
     "elapsed": 195,
     "status": "ok",
     "timestamp": 1683726184843,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "8UO9FnqyKBlF",
    "tags": []
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = 'statmike-mlops-349915' # replace with project ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "79d5cd52-fd6a-40ec-b729-b982eb98b351",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 68869,
     "status": "ok",
     "timestamp": 1683726253709,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "N98-KK7LRkjm",
    "outputId": "09ec5008-0def-4e1a-c349-c598ee752f78",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not a Colab Environment\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from google.colab import auth\n",
    "    auth.authenticate_user()\n",
    "    !gcloud config set project {PROJECT_ID}\n",
    "    print('Colab authorized to GCP')\n",
    "except Exception:\n",
    "    print('Not a Colab Environment')\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c2290c7-0b83-457b-b740-9640b4c29f1d",
   "metadata": {},
   "source": [
    "---\n",
    "## Installs\n",
    "\n",
    "The list `packages` contains tuples of package import names and install names.  If the import name is not found then the install name is used to install quitely for the current user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7c6739f0-727d-4919-a85c-2c452712bb05",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# tuples of (import name, install name, min_version)\n",
    "packages = [\n",
    "    ('google.cloud.aiplatform', 'google-cloud-aiplatform', '1.51.0'),\n",
    "    ('kfp', 'kfp'),\n",
    "]\n",
    "\n",
    "import importlib\n",
    "install = False\n",
    "for package in packages:\n",
    "    if not importlib.util.find_spec(package[0]):\n",
    "        print(f'installing package {package[1]}')\n",
    "        install = True\n",
    "        !pip install {package[1]} -U -q --user\n",
    "    elif len(package) == 3:\n",
    "        if importlib.metadata.version(package[0]) < package[2]:\n",
    "            print(f'updating package {package[1]}')\n",
    "            install = True\n",
    "            !pip install {package[1]} -U -q --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db68156d-2c9e-4ddd-9d3c-cf1149ce5626",
   "metadata": {},
   "source": [
    "### API Enablement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dac5fdfc-69b3-4954-9921-f943bf23296c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!gcloud services enable aiplatform.googleapis.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3558688e-c1ab-48df-a51c-cfe8ffe5b43e",
   "metadata": {},
   "source": [
    "### Restart Kernel (If Installs Occured)\n",
    "\n",
    "After a kernel restart the code submission can start with the next cell after this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4293181d-0a5e-4203-9847-7b233197d8a1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if install:\n",
    "    import IPython\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)\n",
    "    IPython.display.display(IPython.display.Markdown(\"\"\"<div class=\\\"alert alert-block alert-warning\\\">\n",
    "        <b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. The previous cells do not need to be run again⚠️</b>\n",
    "        </div>\"\"\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b7acf54-e4a0-414c-b65b-d3a19d22ac60",
   "metadata": {
    "id": "appt8-yVRtJ1"
   },
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2199197a-de7f-4c6a-b204-f9d0e0577e03",
   "metadata": {
    "id": "63mx2EozRxFP"
   },
   "source": [
    "Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "51f35ee8-fd15-45f2-913b-36c87be3a1bb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 2124,
     "status": "ok",
     "timestamp": 1683726390544,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "xzcoXjM5Rky5",
    "outputId": "b3bdcbc1-70d5-472e-aea2-42c74a42efde",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project = !gcloud config get-value project\n",
    "PROJECT_ID = project[0]\n",
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8f413d63-ecb4-42c5-a4c1-cb415e62e9bd",
   "metadata": {
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1683726390712,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "IxWrFtqYMfku",
    "tags": []
   },
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "SERIES = 'mlops'\n",
    "EXPERIMENT = 'pipeline-intro'\n",
    "\n",
    "# gcs bucket\n",
    "GCS_BUCKET = PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b184d93a-9af7-4601-82a1-cc592cf226b2",
   "metadata": {
    "id": "LuajVwCiO6Yg"
   },
   "source": [
    "Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "9ea8f342-a55a-47c0-882b-10e2fdf91192",
   "metadata": {
    "executionInfo": {
     "elapsed": 17761,
     "status": "ok",
     "timestamp": 1683726409304,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "LVC7zzSLRk2C",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os, time, importlib\n",
    "from typing import NamedTuple\n",
    "from IPython.display import Markdown as show_md\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "import kfp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d3aa007c-152b-42c0-8207-a8a3af0e8e36",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.11.0'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kfp.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "03845691-170e-4d0e-98ff-4be1189f3afc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.78.0'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aiplatform.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29e0fb56-67dd-4eb8-be65-7a75dd7113ff",
   "metadata": {
    "id": "EyAVFG9TO9H-"
   },
   "source": [
    "Clients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7e09bb7e-49fe-4086-b338-c5ec8e1f4509",
   "metadata": {
    "executionInfo": {
     "elapsed": 19,
     "status": "ok",
     "timestamp": 1683726409306,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "L0RPE13LOZce",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# vertex ai clients\n",
    "aiplatform.init(project = PROJECT_ID, location = REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c328f7bc-cc73-474c-bb6a-d2b32f1d8d1a",
   "metadata": {},
   "source": [
    "parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5fae4399-e55c-40d4-9e22-050f11ea8aac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DIR = f\"temp/{SERIES}-{EXPERIMENT}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d43fb4b5-efa6-496a-82b8-8ad24d4dd44f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1026793852137-compute@developer.gserviceaccount.com'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SERVICE_ACCOUNT = !gcloud config list --format='value(core.account)' \n",
    "SERVICE_ACCOUNT = SERVICE_ACCOUNT[0]\n",
    "SERVICE_ACCOUNT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de40a8b2-2050-48d6-ac78-b2c64b3d4604",
   "metadata": {},
   "source": [
    "environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8665838f-2345-44da-a9cd-e1be2341f445",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not os.path.exists(DIR):\n",
    "    os.makedirs(DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e12e93dd-7358-4c3c-9178-a71fd6d0c0b0",
   "metadata": {},
   "source": [
    "---\n",
    "## KFP Pipeline On Vertex AI Pipelines\n",
    "\n",
    "An example workflow: a KFP pipeline constructed from custom components and run on Vertex AI Pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bd0080c-71d5-438a-b425-7232f267367d",
   "metadata": {},
   "source": [
    "### Create Pipeline Components\n",
    "\n",
    "These are simple Python components, specifically lightweight Python components.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8da1aa88-a87a-49ec-ac43-ad11ee3e85dd",
   "metadata": {},
   "source": [
    "Simple component with multiple inputs of different data types and outputs a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "469614bd-e3ba-4ca9-aee2-fa59f34556e9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@kfp.dsl.component(\n",
    "    base_image = \"python:3.11\",\n",
    "    packages_to_install = [\"pandas\"]\n",
    ")\n",
    "def example_parameters(\n",
    "    in_str: str,\n",
    "    in_int: int,\n",
    "    in_float: float,\n",
    "    in_bool: bool,\n",
    "    in_list: list\n",
    ") -> dict:\n",
    "    results = dict(\n",
    "        ex_str = in_str,\n",
    "        ex_int = in_int,\n",
    "        ex_float = in_float,\n",
    "        ex_bool = in_bool,\n",
    "        ex_list = in_list\n",
    "    )\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e56aaa43-7109-44ab-82a9-6b1a7b889fdf",
   "metadata": {},
   "source": [
    "Simple component that take input parameters and outputs an artifact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2ea209fb-1bfe-4c8e-86f2-ac7201ad380f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@kfp.dsl.component(\n",
    "    base_image = \"python:3.10\",\n",
    "    packages_to_install = [\"pandas\"]\n",
    ")\n",
    "def example_artifact(\n",
    "    metadata: dict\n",
    ") -> kfp.dsl.Artifact:\n",
    "    ex_artifact = kfp.dsl.Artifact(\n",
    "        metadata = metadata,\n",
    "        uri = 'https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types'\n",
    "    )\n",
    "    return ex_artifact"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cb192dd-4e08-445b-addf-e37a2e44a59c",
   "metadata": {},
   "source": [
    "### Create Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "087bfbf0-4415-4b46-93f7-43f956095d3d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@kfp.dsl.pipeline(\n",
    "    name = f'{SERIES}-{EXPERIMENT}',\n",
    "    description = 'An introduction pipeline',\n",
    "    pipeline_root = f'gs://{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/pipeline_root'\n",
    ")\n",
    "def intro_pipeline(\n",
    "    in_str: str,\n",
    "    in_int: int,\n",
    "    in_float: float,\n",
    "    in_bool: bool,\n",
    "    in_list: list\n",
    ") -> dict:\n",
    "    \n",
    "    task1 = example_parameters(\n",
    "        in_str = in_str,\n",
    "        in_int = in_int,\n",
    "        in_float = in_float,\n",
    "        in_bool = in_bool,\n",
    "        in_list = in_list\n",
    "    )\n",
    "    \n",
    "    with kfp.dsl.If(in_bool == True, name = 'Proceed If True'):\n",
    "        task2 = example_artifact(metadata = task1.output)\n",
    "\n",
    "    return task1.output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce3ea375-7939-41b0-9f94-9e7e5b2ea8db",
   "metadata": {},
   "source": [
    "### Compile Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9ade9261-18bf-4049-8144-6835772f6d04",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "kfp.compiler.Compiler().compile(\n",
    "    pipeline_func = intro_pipeline,\n",
    "    package_path = f'{DIR}/{SERIES}-{EXPERIMENT}.yaml'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c54d6ca7-454f-4133-82a0-63d9666561e3",
   "metadata": {},
   "source": [
    "### Create Pipeline Job (With Vertex AI SDK)\n",
    "\n",
    "The compiled pipeline file can be submitted for running with the console or the SDK (shown here).  Check out the details in the documentation [here](https://cloud.google.com/vertex-ai/docs/pipelines/run-pipeline#create_a_pipeline_run) for an overview with the console.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "cc5ff736-d3be-4a39-91b8-e2710dd98e0b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline_job = aiplatform.PipelineJob(\n",
    "    display_name = f\"{SERIES}-{EXPERIMENT}\",\n",
    "    template_path = f\"{DIR}/{SERIES}-{EXPERIMENT}.yaml\",\n",
    "    parameter_values = dict(\n",
    "        in_str = 'An Example String',\n",
    "        in_int = 45,\n",
    "        in_float = 4.5,\n",
    "        in_bool = True,\n",
    "        in_list = [1, 27, 'Another String']\n",
    "    ),\n",
    "    pipeline_root = f'gs://{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/pipeline_root',\n",
    "    enable_caching = None # True (enabled), False (disable), None (defer to component level caching) \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3915c92-a627-4723-9b60-0977f54dd32d",
   "metadata": {},
   "source": [
    "### Submit Pipeline Job (On Vertex AI Pipelines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4d5ede69-a217-4a68-bade-37731c18becf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-pipeline-intro-20250309193845?project=1026793852137\n"
     ]
    }
   ],
   "source": [
    "response = pipeline_job.submit(\n",
    "    service_account = SERVICE_ACCOUNT\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "686eced5-6c92-4dd0-9535-1f0cbcda208f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "The Dashboard can be viewed here:\n",
       "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-pipeline-intro-20250309193845?project=1026793852137"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_md(f'The Dashboard can be viewed here:\\n{pipeline_job._dashboard_uri()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "37069dd0-997e-44c7-a915-6f75f4ca9f97",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob run completed. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845\n"
     ]
    }
   ],
   "source": [
    "pipeline_job.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50ffb1a7-6fbf-4236-8177-e640e5c52c84",
   "metadata": {},
   "source": [
    "---\n",
    "## Results: Vertex AI Console And SDK\n",
    "\n",
    "As soon as the job is submitted the response gives a link to the the dashboard view in the console for Vertex AI Pipelines.  Using this link, or navigating directly though Vertex AI, gives a dashboard view of the pipeline as it is running with progress indicatiors for each task.  This information call also be retrieved using the Vertex AI SDK. Both approaches will be covered in this section for:\n",
    "- Pipeline Run\n",
    "- Pipeline Run Tasks Level\n",
    "- Pipeline IO Artifacts\n",
    "- Artifact Lineage In Vertex AI ML Metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7ac76ce-c3ff-411a-9cd5-197da68f41f0",
   "metadata": {},
   "source": [
    "### Pipeline Run Overview\n",
    "**Pipelines Runs List In The Console: Review All Runs And Select Pipeline Run**\n",
    "<p><center>\n",
    "    <img src=\"../resources/images/screenshots/pipelines/intro/runs.png\" width=\"75%\">\n",
    "</center><p>\n",
    "\n",
    "\n",
    "**Pipeline Dashboard View In The Console: Overall Pipeline**\n",
    "<p><center>\n",
    "    <img src=\"../resources/images/screenshots/pipelines/intro/overview.png\" width=\"75%\">\n",
    "</center><p>\n",
    "\n",
    "Note that:\n",
    "- each node, or tasks, is displayed.\n",
    "- the connections between nodes are visual and reflect the input/output dependencies\n",
    "- the node purpose is reflected with an icon and the status is reflected with an icon and color coding\n",
    "- a summary of the pipeline run is provide on the right and includes all the pipelines inputs parameters.\n",
    "\n",
    "    \n",
    "The same information can be retrieved with the Vertex AI SDK as follows.\n",
    "- SDK Reference: [`aiplatform.PipelineJob`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.PipelineJob)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3197b368-a16c-4484-988a-91ac93ba15ef",
   "metadata": {},
   "source": [
    "#### Retrieve Latest Run: By Display Name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "1f02afef-4000-4d14-9ff9-f74757e3dfe9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.pipeline_jobs.PipelineJob object at 0x7f8d12413f40> \n",
       "resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aiplatform.PipelineJob.list(\n",
    "    project = PROJECT_ID,\n",
    "    location = REGION,\n",
    "    filter = f'display_name=\"{SERIES}-{EXPERIMENT}\"',\n",
    "    order_by='update_time desc'\n",
    ")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e27fb80e-3834-49d6-a4c5-dc1aa89392f0",
   "metadata": {},
   "source": [
    "#### Retrieve all runs to dataframe:\n",
    "- SDK Refrence: [`aiplatform.get_pipeline_df`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform#google_cloud_aiplatform_get_pipeline_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "4b9abeb4-b293-4d49-8895-f27df6142d88",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pipeline_name</th>\n",
       "      <th>run_name</th>\n",
       "      <th>param.input:in_list</th>\n",
       "      <th>param.input:in_str</th>\n",
       "      <th>param.input:in_bool</th>\n",
       "      <th>param.output:Output</th>\n",
       "      <th>param.input:in_int</th>\n",
       "      <th>param.input:in_float</th>\n",
       "      <th>param.vmlmd_lineage_integration</th>\n",
       "      <th>param.vertex-ai-pipelines-artifact-argument-binding</th>\n",
       "      <th>metric.ex_list</th>\n",
       "      <th>metric.ex_str</th>\n",
       "      <th>metric.ex_bool</th>\n",
       "      <th>metric.ex_int</th>\n",
       "      <th>metric.ex_float</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mlops-pipeline-intro</td>\n",
       "      <td>mlops-pipeline-intro-20250309193845</td>\n",
       "      <td>[1.0, 27.0, Another String]</td>\n",
       "      <td>An Example String</td>\n",
       "      <td>True</td>\n",
       "      <td>{'ex_float': 4.5, 'ex_str': 'An Example String...</td>\n",
       "      <td>45.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>{'pipeline_run_component': {'location_id': 'us...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mlops-pipeline-intro</td>\n",
       "      <td>mlops-pipeline-intro-20240513162405</td>\n",
       "      <td>[1.0, 27.0, Another String]</td>\n",
       "      <td>An Example String</td>\n",
       "      <td>True</td>\n",
       "      <td>{'ex_bool': True, 'ex_str': 'An Example String...</td>\n",
       "      <td>45.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>{'pipeline_run_component': {'task_name': 'mlop...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mlops-pipeline-intro</td>\n",
       "      <td>mlops-pipeline-intro-20240510115856</td>\n",
       "      <td>[1.0, 27.0, Another String]</td>\n",
       "      <td>An Example String</td>\n",
       "      <td>True</td>\n",
       "      <td>{'ex_int': 45.0, 'ex_bool': True, 'ex_list': [...</td>\n",
       "      <td>45.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>{'pipeline_run_component': {'location_id': 'us...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mlops-pipeline-intro</td>\n",
       "      <td>mlops-pipeline-intro-20240510114744</td>\n",
       "      <td>[1.0, 27.0, Another String]</td>\n",
       "      <td>An Example String</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>45.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>{'pipeline_run_component': {'task_name': 'mlop...</td>\n",
       "      <td>{'output:Output': ['projects/1026793852137/loc...</td>\n",
       "      <td>[1.0, 27.0, Another String]</td>\n",
       "      <td>An Example String</td>\n",
       "      <td>True</td>\n",
       "      <td>45.0</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          pipeline_name                             run_name  \\\n",
       "0  mlops-pipeline-intro  mlops-pipeline-intro-20250309193845   \n",
       "1  mlops-pipeline-intro  mlops-pipeline-intro-20240513162405   \n",
       "2  mlops-pipeline-intro  mlops-pipeline-intro-20240510115856   \n",
       "3  mlops-pipeline-intro  mlops-pipeline-intro-20240510114744   \n",
       "\n",
       "           param.input:in_list param.input:in_str  param.input:in_bool  \\\n",
       "0  [1.0, 27.0, Another String]  An Example String                 True   \n",
       "1  [1.0, 27.0, Another String]  An Example String                 True   \n",
       "2  [1.0, 27.0, Another String]  An Example String                 True   \n",
       "3  [1.0, 27.0, Another String]  An Example String                 True   \n",
       "\n",
       "                                 param.output:Output  param.input:in_int  \\\n",
       "0  {'ex_float': 4.5, 'ex_str': 'An Example String...                45.0   \n",
       "1  {'ex_bool': True, 'ex_str': 'An Example String...                45.0   \n",
       "2  {'ex_int': 45.0, 'ex_bool': True, 'ex_list': [...                45.0   \n",
       "3                                                NaN                45.0   \n",
       "\n",
       "   param.input:in_float                    param.vmlmd_lineage_integration  \\\n",
       "0                   4.5  {'pipeline_run_component': {'location_id': 'us...   \n",
       "1                   4.5  {'pipeline_run_component': {'task_name': 'mlop...   \n",
       "2                   4.5  {'pipeline_run_component': {'location_id': 'us...   \n",
       "3                   4.5  {'pipeline_run_component': {'task_name': 'mlop...   \n",
       "\n",
       "  param.vertex-ai-pipelines-artifact-argument-binding  \\\n",
       "0                                                NaN    \n",
       "1                                                NaN    \n",
       "2                                                NaN    \n",
       "3  {'output:Output': ['projects/1026793852137/loc...    \n",
       "\n",
       "                metric.ex_list      metric.ex_str metric.ex_bool  \\\n",
       "0                          NaN                NaN            NaN   \n",
       "1                          NaN                NaN            NaN   \n",
       "2                          NaN                NaN            NaN   \n",
       "3  [1.0, 27.0, Another String]  An Example String           True   \n",
       "\n",
       "   metric.ex_int  metric.ex_float  \n",
       "0            NaN              NaN  \n",
       "1            NaN              NaN  \n",
       "2            NaN              NaN  \n",
       "3           45.0              4.5  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aiplatform.get_pipeline_df(pipeline = f'{SERIES}-{EXPERIMENT}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "183e92db-0c1a-4b04-b288-ecb58b69e191",
   "metadata": {},
   "source": [
    "#### Review PipelineJob Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "72fddfbf-9721-41ec-bbff-1b2495393b5c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.pipeline_jobs.PipelineJob object at 0x7f8d1bcc4760> \n",
       "resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-intro-20250309193845"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e1a12b1c-e66c-4454-b0da-4d764b974cfc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "google.cloud.aiplatform.pipeline_jobs.PipelineJob"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(pipeline_job)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "76b63ff5-cdf7-4e3a-bfce-f3afae545861",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_SUCCEEDED: 4>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "26f4709f-c306-4107-a063-af5f4a4a1065",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mlops-pipeline-intro'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4a4951e1-a0e3-4fb5-a7ee-dd07277c4d8a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mlops-pipeline-intro-20250309193845'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "f4e387c5-6d31-4842-a24d-dfde03759a4b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['name', 'displayName', 'createTime', 'startTime', 'endTime', 'updateTime', 'pipelineSpec', 'state', 'jobDetail', 'labels', 'runtimeConfig', 'serviceAccount'])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.to_dict().keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "c17c96bf-a3cc-4d8a-9f39-330953f1c656",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['deploymentConfig', 'pipelineInfo', 'schemaVersion', 'components', 'sdkVersion', 'defaultPipelineRoot', 'root', 'deploymentSpec'])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.to_dict()['pipelineSpec'].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "961a5bd0-0a29-4fb5-83e9-f2e2d866187a",
   "metadata": {},
   "source": [
    "### Pipeline Run: Task Overview\n",
    "\n",
    "Each component used in the pipeline created a task.  These task can be retrieved, reviewed, and even examine for input/ouput values of parameters.\n",
    "\n",
    "**Pipeline Dashboard View In The Console: Task Level Details**\n",
    "<p><center>\n",
    "    <img src=\"../resources/images/screenshots/pipelines/intro/overview.png\" width=\"75%\">\n",
    "</center><p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "726304e5-4c65-4c0b-ba83-b0dc67fc439f",
   "metadata": {},
   "source": [
    "#### Review Tasks with `PipelineJob.task_details`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b41c6454-63d3-4292-b183-4a2d6643175a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('mlops-pipeline-intro-20250309193845', <State.SUCCEEDED: 3>),\n",
       " ('example-parameters', <State.SUCCEEDED: 3>),\n",
       " ('condition-1', <State.SUCCEEDED: 3>),\n",
       " ('example-artifact', <State.SUCCEEDED: 3>)]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(task.task_name, task.state) for task in pipeline_job.task_details]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "4f46c860-06b8-4ab0-854a-98c32d0f90d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tasks = {task.task_name: task for task in pipeline_job.task_details}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f6fa000c-90a8-44ee-b614-e970b0eb7c4c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#tasks['example-parameters'].execution.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83d33488-b0dc-4ea0-a7dc-4580408c9de3",
   "metadata": {},
   "source": [
    "#### Review Tasks With `PipelineJob.gca_resource` As Dictionary with `.to_dict()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "89da8a0c-81bb-40fb-adb3-6ea31877b1e4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input:in_int': 45.0,\n",
       " 'input:in_float': 4.5,\n",
       " 'input:in_str': 'An Example String',\n",
       " 'output:Output': {'ex_float': 4.5,\n",
       "  'ex_list': [1.0, 27.0, 'Another String'],\n",
       "  'ex_int': 45.0,\n",
       "  'ex_str': 'An Example String',\n",
       "  'ex_bool': True},\n",
       " 'input:in_bool': True,\n",
       " 'vmlmd_lineage_integration': {'pipeline_run_component': {'project_id': 'statmike-mlops-349915',\n",
       "   'pipeline_run_id': 'mlops-pipeline-intro-20250309193845',\n",
       "   'location_id': 'us-central1',\n",
       "   'task_name': 'example-parameters',\n",
       "   'parent_task_names': ['mlops-pipeline-intro-20250309193845']}},\n",
       " 'input:in_list': [1.0, 27.0, 'Another String']}"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.to_dict()['jobDetail']['taskDetails'][1]['execution']['metadata']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17e7703-4d69-4655-9c1b-b03ef38ec6e0",
   "metadata": {},
   "source": [
    "### Pipeline Run: `CustomJob` For Task:\n",
    "\n",
    "Each task runs as a Vertex AI Training Job.  The details of these jobs can also be reviewed directly from the console by click the 'VIEW JOB' link in the task overview panel.  This opens the exact training job under Vertex AI Training:\n",
    "\n",
    "**Vertex AI Training Job In The Console: Task Level Job Resources**\n",
    "<p><center>\n",
    "    <img src=\"../resources/images/screenshots/pipelines/intro/job.png\" width=\"75%\">\n",
    "</center><p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35a1770-d533-4190-b559-255269e2a275",
   "metadata": {},
   "source": [
    "### Retrieve `CustomJob` Details Using `PipelineJob` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "59040177-875b-470a-af64-ba33056ab7c6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/customJobs/9016432970570924032'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_job_id = pipeline_job.to_dict()['jobDetail']['taskDetails'][1]['executorDetail']['containerDetail']['mainJob']\n",
    "task_job_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "4a1a947a-53a7-4b6c-ab44-27984c10f185",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "task_job = aiplatform.CustomJob.get(resource_name = task_job_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "be628c39-046b-4016-b5ce-321c8a95a6bd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "google.cloud.aiplatform.jobs.CustomJob"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(task_job)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "b787e151-fcd4-4aec-8e0d-dda8df9f57db",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.jobs.CustomJob object at 0x7f8d11672e90> \n",
       "resource name: projects/1026793852137/locations/us-central1/customJobs/9016432970570924032"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "9547c5ae-f5f0-4548-98ad-210138173cc5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'caip_pipelines_5043319236559110144_-5043258316742983680'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_job.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "c8d7d818-609c-4990-987c-9040477d0976",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'projects/1026793852137/locations/us-central1/customJobs/9016432970570924032',\n",
       " 'displayName': 'caip_pipelines_5043319236559110144_-5043258316742983680',\n",
       " 'jobSpec': {'workerPoolSpecs': [{'machineSpec': {'machineType': 'e2-standard-4'},\n",
       "    'replicaCount': '1',\n",
       "    'diskSpec': {'bootDiskType': 'pd-ssd', 'bootDiskSizeGb': 100},\n",
       "    'containerSpec': {'imageUri': 'python:3.11',\n",
       "     'command': ['sh',\n",
       "      '-c',\n",
       "      '\\nif ! [ -x \"$(command -v pip)\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location \\'kfp==2.11.0\\' \\'--no-deps\\' \\'typing-extensions>=3.7.4,<5; python_version<\"3.9\"\\'  &&  python3 -m pip install --quiet --no-warn-script-location \\'pandas\\' && \"$0\" \"$@\"\\n',\n",
       "      'sh',\n",
       "      '-ec',\n",
       "      'program_path=$(mktemp -d)\\n\\nprintf \"%s\" \"$0\" > \"$program_path/ephemeral_component.py\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\\n',\n",
       "      '\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef example_parameters(\\n    in_str: str,\\n    in_int: int,\\n    in_float: float,\\n    in_bool: bool,\\n    in_list: list\\n) -> dict:\\n    results = dict(\\n        ex_str = in_str,\\n        ex_int = in_int,\\n        ex_float = in_float,\\n        ex_bool = in_bool,\\n        ex_list = in_list\\n    )\\n    return results\\n\\n'],\n",
       "     'args': ['--executor_input',\n",
       "      '{\"inputs\":{\"parameterValues\":{\"in_bool\":true,\"in_float\":4.5,\"in_int\":45,\"in_list\":[1,27,\"Another String\"],\"in_str\":\"An Example String\"},\"parameters\":{\"in_bool\":{\"stringValue\":\"true\"},\"in_float\":{\"doubleValue\":4.5},\"in_int\":{\"intValue\":\"45\"},\"in_list\":{\"stringValue\":\"[1,27,\\\\\"Another String\\\\\"]\"},\"in_str\":{\"stringValue\":\"An Example String\"}}},\"outputs\":{\"outputFile\":\"/gcs/statmike-mlops-349915/mlops/pipeline-intro/pipeline_root/1026793852137/mlops-pipeline-intro-20250309193845/example-parameters_-5043258316742983680/executor_output.json\",\"parameters\":{\"Output\":{\"outputFile\":\"/gcs/statmike-mlops-349915/mlops/pipeline-intro/pipeline_root/1026793852137/mlops-pipeline-intro-20250309193845/example-parameters_-5043258316742983680/Output\"}}}}',\n",
       "      '--function_to_execute',\n",
       "      'example_parameters'],\n",
       "     'env': [{'name': 'VERTEX_AI_PIPELINES_RUN_LABELS',\n",
       "       'value': '{\"vertex-ai-pipelines-run-billing-id\":\"5043319236559110144\"}'}]}}],\n",
       "  'scheduling': {'disableRetries': True},\n",
       "  'serviceAccount': '1026793852137-compute@developer.gserviceaccount.com'},\n",
       " 'state': 'JOB_STATE_SUCCEEDED',\n",
       " 'createTime': '2025-03-09T19:38:56.759902Z',\n",
       " 'startTime': '2025-03-09T19:40:13Z',\n",
       " 'endTime': '2025-03-09T19:40:33Z',\n",
       " 'updateTime': '2025-03-09T19:40:35.596483Z',\n",
       " 'labels': {'vertex_pipelines': '',\n",
       "  'vertex-ai-pipelines-run-billing-id': '5043319236559110144'}}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_job.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1507586c-ec21-4cca-b350-5bf4c42abf46",
   "metadata": {},
   "source": [
    "### Pipeline IO Artifacts\n",
    "\n",
    "The pipeline task created from the `example_artifact` component returned an artifact (`kfp.dsl.Artifact`) as output.  Pipelines automatically store artifacts in Vertex AI ML Metadata.\n",
    "\n",
    "**Pipeline Dashboard View In The Console: Artifact Info**\n",
    "<p><center>\n",
    "    <img src=\"../resources/images/screenshots/pipelines/intro/overview.png\" width=\"75%\">\n",
    "</center><p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a18dd0-c2be-4fe9-b356-33f2d3a43cfb",
   "metadata": {},
   "source": [
    "#### Retrieve Artifact Info From `PipelineJob`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "d5972905-e492-47ee-9382-e8eb09c38edd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input:metadata': {'ex_list': [1.0, 27.0, 'Another String'],\n",
       "  'ex_int': 45.0,\n",
       "  'ex_str': 'An Example String',\n",
       "  'ex_bool': True,\n",
       "  'ex_float': 4.5},\n",
       " 'vertex-ai-pipelines-artifact-argument-binding': {'output:Output': ['projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/17045140288337670150']},\n",
       " 'vmlmd_lineage_integration': {'pipeline_run_component': {'task_name': 'example-artifact',\n",
       "   'location_id': 'us-central1',\n",
       "   'project_id': 'statmike-mlops-349915',\n",
       "   'parent_task_names': ['mlops-pipeline-intro-20250309193845', 'condition-1'],\n",
       "   'pipeline_run_id': 'mlops-pipeline-intro-20250309193845'}}}"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.to_dict()['jobDetail']['taskDetails'][3]['execution']['metadata']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "753a7abc-402e-4246-a726-c733abe545cc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/17045140288337670150'"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifact_id = pipeline_job.to_dict()['jobDetail']['taskDetails'][3]['execution']['metadata']['vertex-ai-pipelines-artifact-argument-binding']['output:Output'][0]\n",
    "artifact_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "4069f6f6-29b8-4693-a716-dda20197195b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "artifact = aiplatform.Artifact.get(resource_id = artifact_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "bea5c667-9e64-4457-91b0-c4885b909ebf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types'"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifact.uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "58d99578-0057-47d4-89e2-ed17eb2d5cb7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ex_bool': True,\n",
       " 'ex_float': 4.5,\n",
       " 'ex_str': 'An Example String',\n",
       " 'ex_list': [1.0, 27.0, 'Another String'],\n",
       " 'ex_int': 45.0}"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifact.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3bf1e40-dd46-4256-b249-582cc49dbd24",
   "metadata": {},
   "source": [
    "#### Retrieve Pipeline Artifacts Vertex AI SDK\n",
    "\n",
    "- [`aiplatform.Artifact()`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Artifact)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "af920f02-b979-4b04-99c2-f2d72dfd8f96",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "artifacts = aiplatform.Artifact.list(\n",
    "    filter = \"metadata.ex_bool.bool_value = true\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "357f2324-adfa-4eaa-bcef-38698f041541",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<google.cloud.aiplatform.metadata.artifact.Artifact object at 0x7f8d12413ee0> \n",
       " resource name: projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/17045140288337670150\n",
       " uri: https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types\n",
       " schema_title:system.Artifact,\n",
       " <google.cloud.aiplatform.metadata.artifact.Artifact object at 0x7f8d37657040> \n",
       " resource name: projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/8967050471620078494\n",
       " uri: https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types\n",
       " schema_title:system.Artifact,\n",
       " <google.cloud.aiplatform.metadata.artifact.Artifact object at 0x7f8d37658970> \n",
       " resource name: projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/11017351580797271703\n",
       " uri: https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types\n",
       " schema_title:system.Artifact,\n",
       " <google.cloud.aiplatform.metadata.artifact.Artifact object at 0x7f8d37658c40> \n",
       " resource name: projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/2441918375101103701\n",
       " uri: https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types\n",
       " schema_title:system.Artifact,\n",
       " <google.cloud.aiplatform.metadata.artifact.Artifact object at 0x7f8d37658d00> \n",
       " resource name: projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/12015835593073198771\n",
       " uri: https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types\n",
       " schema_title:system.Artifact,\n",
       " <google.cloud.aiplatform.metadata.artifact.Artifact object at 0x7f8d37658f10> \n",
       " resource name: projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/2907691161866851018\n",
       " uri: https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types\n",
       " schema_title:system.Artifact,\n",
       " <google.cloud.aiplatform.metadata.artifact.Artifact object at 0x7f8d37659120> \n",
       " resource name: projects/1026793852137/locations/us-central1/metadataStores/default/artifacts/17948488352989447493\n",
       " uri: https://www.kubeflow.org/docs/components/pipelines/v2/data-types/artifacts/#artifact-types\n",
       " schema_title:system.Artifact]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "0d5bc287-681f-4776-889f-ca77d69ff766",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "google.cloud.aiplatform.metadata.artifact.Artifact"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(artifacts[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "c859f958-4d2e-4500-a8d4-e270e3ba5ed1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ex_bool': True,\n",
       " 'ex_int': 45.0,\n",
       " 'ex_str': 'An Example String',\n",
       " 'ex_list': [1.0, 27.0, 'Another String'],\n",
       " 'ex_float': 4.5}"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifacts[0].metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "4ab15c67-4034-4fe7-ad29-9e33c2195fc4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://console.cloud.google.com/vertex-ai/locations/us-central1/metadata-stores/default/artifacts/17045140288337670150?project=1026793852137'"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifacts[0].lineage_console_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "021295c2-4011-4170-854f-a195f7a542ec",
   "metadata": {},
   "source": [
    "### Pipeline IO Artifact Metadata Lineage\n",
    "\n",
    "The consolve view of artifact info include a link to the Vertex AI ML Metadata.  This link can also be directly retrieved for the artifact using the SDK with `lineage_console_uri` attribute.  This ones the lineage view of the artifact in the Vertex AI ML Metadata Console.\n",
    "\n",
    "\n",
    "**Vertex AI ML Metadata Console: Artifact Lineage**\n",
    "<p><center>\n",
    "    <img src=\"../resources/images/screenshots/pipelines/intro/metadata.png\" width=\"75%\">\n",
    "</center><p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "39bdfa91-7451-425d-975d-c48e0a8cc872",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://console.cloud.google.com/vertex-ai/locations/us-central1/metadata-stores/default/artifacts/17045140288337670150?project=statmike-mlops-349915'"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifact.lineage_console_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7b0c2b7-5867-4933-a550-5602f60eccec",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "conda-base-py",
   "name": "workbench-notebooks.m128",
   "type": "gcloud",
   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m128"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel) (Local)",
   "language": "python",
   "name": "conda-base-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}