MLOps/Pipelines/Vertex AI Pipelines - Managing Pipeline Jobs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "188c8627",
   "metadata": {},
   "source": [
    "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Fvertex-ai-mlops%2FMLOps%2FPipelines&file=Vertex+AI+Pipelines+-+Managing+Pipeline+Jobs.ipynb)\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/MLOps/Pipelines/Vertex%20AI%20Pipelines%20-%20Managing%20Pipeline%20Jobs.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Fvertex-ai-mlops%2Fmain%2FMLOps%2FPipelines%2FVertex%2520AI%2520Pipelines%2520-%2520Managing%2520Pipeline%2520Jobs.ipynb\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/statmike/vertex-ai-mlops/blob/main/MLOps/Pipelines/Vertex%20AI%20Pipelines%20-%20Managing%20Pipeline%20Jobs.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/vertex-ai-mlops/main/MLOps/Pipelines/Vertex%20AI%20Pipelines%20-%20Managing%20Pipeline%20Jobs.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee657411-57b5-4712-ac2d-d0e6dc6c693c",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "This is part of a [series of notebook based workflows](./readme.md) that teach all the ways to use pipelines within Vertex AI. The suggested order and description/reason is:\n",
    "\n",
    "|Link To Section|Notebook Workflow|Description|\n",
    "|---|---|---|\n",
    "||[Vertex AI Pipelines - Start Here](./Vertex%20AI%20Pipelines%20-%20Start%20Here.ipynb)|What are pipelines? Start here to go from code to pipeline and see it in action.|\n",
    "||[Vertex AI Pipelines - Introduction](./Vertex%20AI%20Pipelines%20-%20Introduction.ipynb)|Introduction to pipelines with the console and Vertex AI SDK|\n",
    "||[Vertex AI Pipelines - Components](./Vertex%20AI%20Pipelines%20-%20Components.ipynb)|An introduction to all the ways to create pipeline components from your code|\n",
    "||[Vertex AI Pipelines - IO](./Vertex%20AI%20Pipelines%20-%20IO.ipynb)|An overview of all the type of inputs and outputs for pipeline components|\n",
    "||[Vertex AI Pipelines - Control](./Vertex%20AI%20Pipelines%20-%20Control.ipynb)|An overview of controlling the flow of exectution for pipelines|\n",
    "||[Vertex AI Pipelines - Secret Manager](./Vertex%20AI%20Pipelines%20-%20Secret%20Manager.ipynb)|How to pass sensitive information to pipelines and components|\n",
    "||[Vertex AI Pipelines - GCS Read and Write](./Vertex%20AI%20Pipelines%20-%20GCS%20Read%20and%20Write.ipynb)|How to read/write to GCS from components, including container components.|\n",
    "||[Vertex AI Pipelines - Scheduling](./Vertex%20AI%20Pipelines%20-%20Scheduling.ipynb)|How to schedule pipeline execution|\n",
    "||[Vertex AI Pipelines - Notifications](./Vertex%20AI%20Pipelines%20-%20Notifications.ipynb)|How to send email notification of pipeline status.|\n",
    "||[Vertex AI Pipelines - Management](./Vertex%20AI%20Pipelines%20-%20Management.ipynb)|Managing, Reusing, and Storing pipelines and components|\n",
    "||[Vertex AI Pipelines - Testing](./Vertex%20AI%20Pipelines%20-%20Testing.ipynb)|Strategies for testing components and pipeliens locally and remotely to aide development.|\n",
    "|_**This Notebook**_|[Vertex AI Pipelines - Managing Pipeline Jobs](./Vertex%20AI%20Pipelines%20-%20Managing%20Pipeline%20Jobs.ipynb)|Manage runs of pipelines in an environment: list, check status, filtered list, cancel and delete jobs.|\n",
    "\n",
    "\n",
    "To discover these notebooks as part of an introduction to MLOps orchestration [start here](./readme.md).  To read more about MLOps also check out [the parent folder](../readme.md).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f442897-d93b-48d8-a7f5-8727c17896bd",
   "metadata": {},
   "source": [
    "# Vertex AI Pipelines - Managing Jobs\n",
    "\n",
    "Vertex AI Pipeline Jobs are runs of a pipeline.  These can be directly run by a user, started by API, or scheduled.  Withing Vetex AI a project can have many jobs running at any time and a history of all past jobs.  This workflow shows how to review and manage the jobs in an enviornment using the Python SDK.  For custom metrics in Cloud Logging check out [this helpful page](https://cloud.google.com/vertex-ai/docs/pipelines/metrics)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee0a49c1-7788-4597-a03a-535c2f6a5c1f",
   "metadata": {
    "id": "od_UkDpvRmgD",
    "tags": []
   },
   "source": [
    "---\n",
    "## Colab Setup\n",
    "\n",
    "To run this notebook in Colab run the cells in this section.  Otherwise, skip this section.\n",
    "\n",
    "This cell will authenticate to GCP (follow prompts in the popup)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d0e56518-6260-4eff-a997-ca4dfba8b759",
   "metadata": {
    "executionInfo": {
     "elapsed": 195,
     "status": "ok",
     "timestamp": 1683726184843,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "8UO9FnqyKBlF",
    "tags": []
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = 'statmike-mlops-349915' # replace with project ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ad6072b3-f85f-4dc4-8796-bbb9cbaf46c8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 68869,
     "status": "ok",
     "timestamp": 1683726253709,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "N98-KK7LRkjm",
    "outputId": "09ec5008-0def-4e1a-c349-c598ee752f78",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not a Colab Environment\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from google.colab import auth\n",
    "    auth.authenticate_user()\n",
    "    !gcloud config set project {PROJECT_ID}\n",
    "    print('Colab authorized to GCP')\n",
    "except Exception:\n",
    "    print('Not a Colab Environment')\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d45cd234-8782-4668-b77b-63a7fffd9598",
   "metadata": {},
   "source": [
    "---\n",
    "## Installs\n",
    "\n",
    "The list `packages` contains tuples of package import names and install names.  If the import name is not found then the install name is used to install quitely for the current user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "13c5feb0-1f17-4a32-b6ae-11fc473adf21",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# tuples of (import name, install name, min_version)\n",
    "packages = [\n",
    "    ('google.cloud.aiplatform', 'google-cloud-aiplatform', '1.51.0'),\n",
    "    ('kfp', 'kfp'),\n",
    "]\n",
    "\n",
    "import importlib\n",
    "install = False\n",
    "for package in packages:\n",
    "    if not importlib.util.find_spec(package[0]):\n",
    "        print(f'installing package {package[1]}')\n",
    "        install = True\n",
    "        !pip install {package[1]} -U -q --user\n",
    "    elif len(package) == 3:\n",
    "        if importlib.metadata.version(package[0]) < package[2]:\n",
    "            print(f'updating package {package[1]}')\n",
    "            install = True\n",
    "            !pip install {package[1]} -U -q --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c067cf10-9efe-4a73-8eb4-1a7f9ef416af",
   "metadata": {},
   "source": [
    "### API Enablement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3dcee0c8-105b-40d7-856d-c1b0c3bdb5df",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!gcloud services enable aiplatform.googleapis.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7fd018f-eff1-467b-b01a-7e5c120d7fc7",
   "metadata": {},
   "source": [
    "### Restart Kernel (If Installs Occured)\n",
    "\n",
    "After a kernel restart the code submission can start with the next cell after this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0747aea3-be35-4deb-b295-47199c991ac2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if install:\n",
    "    import IPython\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)\n",
    "    IPython.display.display(IPython.display.Markdown(\"\"\"<div class=\\\"alert alert-block alert-warning\\\">\n",
    "        <b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. The previous cells do not need to be run again⚠️</b>\n",
    "        </div>\"\"\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94d1c674-48a7-4da1-9283-594bcb9ec5ec",
   "metadata": {
    "id": "appt8-yVRtJ1"
   },
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "738585b0-7a5d-4f8b-8461-f20ae5c5cf85",
   "metadata": {
    "id": "63mx2EozRxFP"
   },
   "source": [
    "Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a31caec7-4659-4600-a2af-a78d32c61617",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 2124,
     "status": "ok",
     "timestamp": 1683726390544,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "xzcoXjM5Rky5",
    "outputId": "b3bdcbc1-70d5-472e-aea2-42c74a42efde",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project = !gcloud config get-value project\n",
    "PROJECT_ID = project[0]\n",
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "11c62634-2e32-4e0a-80e4-6588e7912603",
   "metadata": {
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1683726390712,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "IxWrFtqYMfku",
    "tags": []
   },
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "EXPERIMENT = 'pipeline-jobstate'\n",
    "SERIES = 'mlops'\n",
    "\n",
    "# gcs bucket\n",
    "GCS_BUCKET = PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7296eca2-7c8e-4593-bdab-0e63d12e95e3",
   "metadata": {
    "id": "LuajVwCiO6Yg"
   },
   "source": [
    "Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "1e1ddbda-3393-4ff8-967f-0849e010493f",
   "metadata": {
    "executionInfo": {
     "elapsed": 17761,
     "status": "ok",
     "timestamp": 1683726409304,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "LVC7zzSLRk2C",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os, datetime, collections\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "import kfp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "eebc9915-ba85-4b3a-9c17-2ccb0946a89e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.11.0'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kfp.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "af6fcd55-71d8-4534-9cd1-0caaee086beb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.78.0'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aiplatform.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3844cc5e-bae2-4354-bd32-fda64fcf3611",
   "metadata": {
    "id": "EyAVFG9TO9H-"
   },
   "source": [
    "Clients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d12273d5-cbc8-4c9b-a11c-11e7f6e572a9",
   "metadata": {
    "executionInfo": {
     "elapsed": 19,
     "status": "ok",
     "timestamp": 1683726409306,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "L0RPE13LOZce",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# vertex ai clients\n",
    "aiplatform.init(project = PROJECT_ID, location = REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a90ca01-1d37-47dc-a0b8-5dbce2e7ae58",
   "metadata": {},
   "source": [
    "parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5ecfc876-3e23-4d67-ade5-d6b4ac82a54d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DIR = f\"temp/{SERIES}-{EXPERIMENT}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "44345b63-6799-4716-8165-29097607809d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1026793852137-compute@developer.gserviceaccount.com'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SERVICE_ACCOUNT = !gcloud config list --format='value(core.account)' \n",
    "SERVICE_ACCOUNT = SERVICE_ACCOUNT[0]\n",
    "SERVICE_ACCOUNT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d95afe-6e13-4df0-aa3a-ed3974e87ecc",
   "metadata": {},
   "source": [
    "environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a3b00bf0-bba5-47af-b52c-af6f3c28be57",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not os.path.exists(DIR):\n",
    "    os.makedirs(DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17024378-a5b2-40aa-88ea-d3b40fe08e0e",
   "metadata": {},
   "source": [
    "---\n",
    "## Make A Simple Pipeline For Testing/Illustrating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0d32ff49-ec74-41db-a87a-f66ab0f960de",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@kfp.dsl.component(\n",
    "    base_image = \"python:3.11\"\n",
    ")\n",
    "def waiter(wait_time: int) -> dict:\n",
    "    import time\n",
    "    \n",
    "    time.sleep(wait_time)\n",
    "    \n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0a65d7ae-d50b-4618-a395-fcc76d2e5473",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "kfp.compiler.Compiler().compile(\n",
    "    pipeline_func = waiter,\n",
    "    package_path = f'{DIR}/{SERIES}-{EXPERIMENT}.yaml',\n",
    "    pipeline_name = f\"{SERIES}-{EXPERIMENT}-component\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8f0010c-4293-4185-8b5b-0e918f108dec",
   "metadata": {},
   "source": [
    "## Executing Pipeline Jobs And Monitoring Directly"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2251a78b-0ee4-4639-a773-605ab9cb5e15",
   "metadata": {},
   "source": [
    "### Execute A PipelineJob with `.run()`\n",
    "\n",
    "This method runs and locks the local context while it waits on the job to end.  It continually updates the state of the job in the output to the local session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "55db1100-bacb-4e90-a799-9064fca06cd7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline_job = aiplatform.PipelineJob(\n",
    "    display_name = f\"{SERIES}-{EXPERIMENT}-component\",\n",
    "    template_path = f\"{DIR}/{SERIES}-{EXPERIMENT}.yaml\",\n",
    "    parameter_values = dict(\n",
    "        wait_time = 10\n",
    "    ),\n",
    "    pipeline_root = f'gs://{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/pipeline_root',\n",
    "    enable_caching = False # True (enabled), False (disable), None (defer to component level caching) \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f28c4280-8c03-491b-8c2e-6ef776e89390",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214124\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214124')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-pipeline-jobstate-component-20250213214124?project=1026793852137\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214124 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214124 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214124 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214124 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob run completed. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214124\n"
     ]
    }
   ],
   "source": [
    "job_run = pipeline_job.run(\n",
    "    service_account = SERVICE_ACCOUNT\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974d5998-5d41-4dff-8a6a-f9dac4441d78",
   "metadata": {},
   "source": [
    "#### Understanding the Final State Of The PipelineJob\n",
    "\n",
    "Notice that the job continually reported state for the duration of running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4b285c72-a8f4-442c-8add-67f0df90cb98",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_SUCCEEDED: 4>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e5e1ad31-c9f1-40af-a6bc-a38201f1d0bb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.state.value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "cfd114d9-490b-449b-9544-7d712fe59879",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'PIPELINE_STATE_SUCCEEDED'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.state.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "46c8bc24-cea4-4a9c-bbe0-43618e498c22",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<enum 'PipelineState'>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(pipeline_job.state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7c7d6ee2-d868-4c92-a2c2-1da592e33726",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job.done()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32f6cc2-f508-426f-aafc-8e2fe1e0df05",
   "metadata": {},
   "source": [
    "https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.types.PipelineState"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ae0d5f16-1130-4845-a6d5-bdc2f0ba2b0b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mappingproxy({'PIPELINE_STATE_UNSPECIFIED': <PipelineState.PIPELINE_STATE_UNSPECIFIED: 0>,\n",
       "              'PIPELINE_STATE_QUEUED': <PipelineState.PIPELINE_STATE_QUEUED: 1>,\n",
       "              'PIPELINE_STATE_PENDING': <PipelineState.PIPELINE_STATE_PENDING: 2>,\n",
       "              'PIPELINE_STATE_RUNNING': <PipelineState.PIPELINE_STATE_RUNNING: 3>,\n",
       "              'PIPELINE_STATE_SUCCEEDED': <PipelineState.PIPELINE_STATE_SUCCEEDED: 4>,\n",
       "              'PIPELINE_STATE_FAILED': <PipelineState.PIPELINE_STATE_FAILED: 5>,\n",
       "              'PIPELINE_STATE_CANCELLING': <PipelineState.PIPELINE_STATE_CANCELLING: 6>,\n",
       "              'PIPELINE_STATE_CANCELLED': <PipelineState.PIPELINE_STATE_CANCELLED: 7>,\n",
       "              'PIPELINE_STATE_PAUSED': <PipelineState.PIPELINE_STATE_PAUSED: 8>})"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aiplatform.gapic.PipelineState.__members__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2e6ca6f-2181-4ac5-a395-43cbd22a7de8",
   "metadata": {},
   "source": [
    "### Execute A PipelineJob with `.submit()` and `.wait()`\n",
    "\n",
    "This method starts the job and returns the context to the local session without waiting on the job to complete.  The wait method can be used to force the session to wait on the job to end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "33dbe7e0-d058-48bc-94df-c4c369cee562",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline_job2 = aiplatform.PipelineJob(\n",
    "    display_name = f\"{SERIES}-{EXPERIMENT}-component\",\n",
    "    template_path = f\"{DIR}/{SERIES}-{EXPERIMENT}.yaml\",\n",
    "    parameter_values = dict(\n",
    "        wait_time = 10\n",
    "    ),\n",
    "    pipeline_root = f'gs://{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/pipeline_root',\n",
    "    enable_caching = False # True (enabled), False (disable), None (defer to component level caching) \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cbc1c0bc-69ea-4e11-8ee0-776bb2fc6531",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214746\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214746')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-pipeline-jobstate-component-20250213214746?project=1026793852137\n"
     ]
    }
   ],
   "source": [
    "pipeline_job2.submit(\n",
    "    service_account = SERVICE_ACCOUNT\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f2749b8-1a3d-483a-acde-2fa8f60ba4fb",
   "metadata": {},
   "source": [
    "Notice that the job returns to the local context after it is sumitted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "dede558e-58a7-4288-b8b5-65e8b75abc47",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "google.cloud.aiplatform.pipeline_jobs.PipelineJob"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(pipeline_job2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "602aab77-c24f-4e4d-a6b8-7335612c2c46",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_RUNNING: 3>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job2.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "021757bb-f7a4-4e10-b989-bdbef6a4e252",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job2.done()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b645ea70-8cc4-4604-b4e5-818eb4176586",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214746 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214746 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214746 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob run completed. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214746\n"
     ]
    }
   ],
   "source": [
    "pipeline_job2.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9b571e8c-663e-4a1f-8e72-27e9e0499f1e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_SUCCEEDED: 4>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job2.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "ac0dfeb6-d93f-4ccb-aee7-e1d660217d67",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job2.done()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "361b41e1-e795-49f6-99ec-48b744211d49",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214746'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job2.resource_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cb10007-675c-483b-a0d8-c3fccc5bae36",
   "metadata": {},
   "source": [
    "### Execute A PipelineJob And Monitor it Directly\n",
    "\n",
    "Similar to the previous example, execute a PipelineJob with `.submit()`.  Then `.get()` the job separately (as though in a different session at a different time) and directly monitor its status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "4cee8390-37da-44a7-9814-89d77404b282",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'02132025-215722'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datetime.datetime.now().strftime(\"%m%d%Y-%H%M%S\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5b0b4f42-2684-4790-bde9-788bcaa47bce",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "job_id = f'{SERIES}-{EXPERIMENT}-component-{datetime.datetime.now().strftime(\"%m%d%Y-%H%M%S\")}'\n",
    "\n",
    "pipeline_job3 = aiplatform.PipelineJob(\n",
    "    display_name = f\"{SERIES}-{EXPERIMENT}-component\",\n",
    "    job_id = job_id,\n",
    "    template_path = f\"{DIR}/{SERIES}-{EXPERIMENT}.yaml\",\n",
    "    parameter_values = dict(\n",
    "        wait_time = 100\n",
    "    ),\n",
    "    pipeline_root = f'gs://{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/pipeline_root',\n",
    "    enable_caching = False # True (enabled), False (disable), None (defer to component level caching) \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "91b8457e-bbc0-4532-9cae-b0d6c0e1779f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-pipeline-jobstate-component-02132025-215756?project=1026793852137\n"
     ]
    }
   ],
   "source": [
    "pipeline_job3.submit(\n",
    "    service_account = SERVICE_ACCOUNT\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "f9ac16e1-83dd-4a4d-99ba-d0ad5c19d298",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "retrieve_job3 = aiplatform.PipelineJob.get(resource_name = job_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "6f02c137-3ab4-457f-827e-48e0d41a0ff6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job3.resource_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "8f1e23e5-74ff-40a3-8de6-7df3c5b5000e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve_job3.resource_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "c29c08ce-a7f1-40de-8e4e-3569c1f7173b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve_job3.done()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e9bfcf34-446a-4b17-8c50-b73a195a2f04",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_RUNNING: 3>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve_job3.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "1b2e77f8-08a9-4dc4-be43-4b7c288dae20",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob run completed. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756\n"
     ]
    }
   ],
   "source": [
    "retrieve_job3.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "6dd9a033-ec96-4b59-b9de-0ffa8974acac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_SUCCEEDED: 4>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve_job3.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "c9e34ec5-d558-4326-b810-3714fb9a6f99",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_SUCCEEDED: 4>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job3.state"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "499fbc7b-68f9-482a-9afd-5ca7d5e6dcc9",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "## Listing Pipeline Jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68e4ec18-554e-40cb-8e3b-207b41578730",
   "metadata": {},
   "source": [
    "### All Runs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "6215ffd0-a18f-47e5-90c2-3a47933e1433",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "By enabling simple view, the PipelineJob resources returned from this method will not contain all fields.\n"
     ]
    }
   ],
   "source": [
    "jobs_list = aiplatform.PipelineJob.list(filter = None, enable_simple_view = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "58b5c6b1-943f-44f2-9cbc-74d996d64723",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(jobs_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "958540cf-f316-4593-8621-135bf08eea8d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<google.cloud.aiplatform.pipeline_jobs.PipelineJob object at 0x7f7f545117b0> \n",
       " resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-02132025-215756,\n",
       " <google.cloud.aiplatform.pipeline_jobs.PipelineJob object at 0x7f7f545130d0> \n",
       " resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214746,\n",
       " <google.cloud.aiplatform.pipeline_jobs.PipelineJob object at 0x7f7f54512f20> \n",
       " resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213214124]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "6958918e-09c7-467a-b21b-fff5cc676fda",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_SUCCEEDED: 4>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "8559f693-09ea-44eb-b642-797743ba5486",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "states = [job.state.name for job in jobs_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "207ed3ad-8be1-4dca-902e-4d39f7a32853",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'PIPELINE_STATE_SUCCEEDED': 196,\n",
       "         'PIPELINE_STATE_FAILED': 185,\n",
       "         'PIPELINE_STATE_CANCELLED': 3})"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collections.Counter(states)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30e21fa4-54ee-4bad-85cf-d51c3d35e6fb",
   "metadata": {},
   "source": [
    "### Named Runs: Filter By `display_name`\n",
    "\n",
    "This gets all the runs for a pipeline by using the pipelines display name as the filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "6cd806c4-6c74-4881-a862-3431122d60a6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "By enabling simple view, the PipelineJob resources returned from this method will not contain all fields.\n"
     ]
    }
   ],
   "source": [
    "jobs_list = aiplatform.PipelineJob.list(\n",
    "    filter = f'display_name = \"{SERIES}-{EXPERIMENT}-component\"', \n",
    "    enable_simple_view = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "65afa1cb-f891-4a07-a8ca-12d8075f1cfe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(jobs_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "c350dbb3-6930-43dc-9604-ab2b87252cfc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_SUCCEEDED: 4>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].state"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1d00063-28d0-400b-adc7-27d30ed1de7a",
   "metadata": {},
   "source": [
    "### Query Runs: Filter By `display_name` and `state`\n",
    "\n",
    "Get all successful runs of a pipeline, and order by creation time descending."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "25464dd6-b09e-4b7a-a954-0a5ee0aabb4e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "By enabling simple view, the PipelineJob resources returned from this method will not contain all fields.\n"
     ]
    }
   ],
   "source": [
    "jobs_list = aiplatform.PipelineJob.list(\n",
    "    filter = f'display_name = \"{SERIES}-{EXPERIMENT}-component\" AND state = \"PIPELINE_STATE_SUCCEEDED\"', \n",
    "    order_by = 'create_time desc',\n",
    "    enable_simple_view = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "5a86d845-be63-417a-bf02-8409822a43cc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(jobs_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "d7eda189-8efa-4542-a3c9-7567d0b83f3e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('mlops-pipeline-jobstate-component-02132025-215756', '2025-02-13 21:58:03'),\n",
       " ('mlops-pipeline-jobstate-component-20250213214746', '2025-02-13 21:47:47'),\n",
       " ('mlops-pipeline-jobstate-component-20250213214124', '2025-02-13 21:41:24'),\n",
       " ('mlops-pipeline-jobstate-component-20250212193454', '2025-02-12 19:35:07'),\n",
       " ('mlops-pipeline-jobstate-component-20250212192434', '2025-02-12 19:24:45'),\n",
       " ('mlops-pipeline-jobstate-component-20250212175926', '2025-02-12 17:59:46')]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(job.name, job.create_time.strftime(\"%Y-%m-%d %H:%M:%S\")) for job in jobs_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e6cb2ec-8b57-41c0-b9d4-d31292125400",
   "metadata": {},
   "source": [
    "### Active Runs: Filter by `state`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "d3d5d94a-aec5-40c8-b9b5-d3685b8cce1f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline_job4 = aiplatform.PipelineJob(\n",
    "    display_name = f\"{SERIES}-{EXPERIMENT}-component\",\n",
    "    template_path = f\"{DIR}/{SERIES}-{EXPERIMENT}.yaml\",\n",
    "    parameter_values = dict(\n",
    "        wait_time = 1000\n",
    "    ),\n",
    "    pipeline_root = f'gs://{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/pipeline_root',\n",
    "    enable_caching = False # True (enabled), False (disable), None (defer to component level caching) \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "377238a0-ecfe-4583-a3d4-639d93944335",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213223849\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213223849')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-pipeline-jobstate-component-20250213223849?project=1026793852137\n"
     ]
    }
   ],
   "source": [
    "pipeline_job4.submit(\n",
    "    service_account = SERVICE_ACCOUNT\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "9b4344b4-fc70-478a-a6f9-bbd447523286",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "By enabling simple view, the PipelineJob resources returned from this method will not contain all fields.\n"
     ]
    }
   ],
   "source": [
    "jobs_list = aiplatform.PipelineJob.list(\n",
    "    filter = f'display_name = \"{SERIES}-{EXPERIMENT}-component\" AND state = \"PIPELINE_STATE_RUNNING\"', \n",
    "    order_by = 'create_time desc',\n",
    "    enable_simple_view = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "a66a13db-ce60-4fc1-928c-31218068ee7d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('mlops-pipeline-jobstate-component-20250213223849', '2025-02-13 22:38:50')]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(job.name, job.create_time.strftime(\"%Y-%m-%d %H:%M:%S\")) for job in jobs_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "b57d0cf9-45e9-4eba-bdc2-cd816914d7a5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].done()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "e07c7515-dd4e-47ca-a1a5-6eed9826f34d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeWithNanoseconds(2025, 2, 13, 22, 38, 50, 258720, tzinfo=datetime.timezone.utc)"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].create_time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "521697a8-cec6-43c5-934b-ff6cba5928f0",
   "metadata": {},
   "source": [
    "### Jobs In Date Range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "a19d39f7-a726-467f-a193-3ee15644cc45",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "jobs_list = aiplatform.PipelineJob.list(\n",
    "    filter = 'create_time >= \"2024-06-01T00:00:00Z\" AND create_time < \"2024-07-01T00:00:00Z\"'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "b80dc714-434b-4ae0-acf4-124fcb19cbe7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(jobs_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "697aceb5-6993-4568-80a5-c59758c0d7b5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'PIPELINE_STATE_FAILED': 19, 'PIPELINE_STATE_SUCCEEDED': 14})"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "collections.Counter([job.state.name for job in jobs_list])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df89a620-725e-4381-87f9-560651d3402d",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "## Job Canceling, Deleting, And Cloning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c86888ef-a927-4a4b-a606-7ce50de79475",
   "metadata": {},
   "source": [
    "### Stop Jobs with `.cancel()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "de4657e0-adb5-4e19-aeb5-e90c8db2f435",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].done()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "bd890d26-b8d9-4594-b654-9f5829a18755",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_RUNNING: 3>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "34eb4ed4-339f-414f-ae01-d22531087a6a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "jobs_list[0].cancel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "a9b563ef-5149-474e-bc58-9f31416eabf8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_CANCELLING: 6>"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "cc60b607-6170-4a6c-a9b4-8a2a5899693e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].done()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "4bb7e894-3155-48ca-859e-6c688f2180bb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob run completed. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213223849\n"
     ]
    }
   ],
   "source": [
    "jobs_list[0].wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "2f7c5331-3fb4-41ad-9744-b2ebeadd0a7b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].done()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "e9b10593-9c83-4483-866b-5b51785ffb62",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_CANCELLED: 7>"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_list[0].state"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc2f16b4-ec56-4a44-a514-af271e7d12c6",
   "metadata": {},
   "source": [
    "### Delete Jobs with `.delete()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "dc0fb35a-880e-4536-a877-4149409d430f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting PipelineJob : projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213223849\n",
      "PipelineJob deleted. . Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213223849\n",
      "Deleting PipelineJob resource: projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213223849\n",
      "Delete PipelineJob backing LRO: projects/1026793852137/locations/us-central1/operations/3691838885597282304\n",
      "PipelineJob resource projects/1026793852137/locations/us-central1/pipelineJobs/mlops-pipeline-jobstate-component-20250213223849 deleted.\n"
     ]
    }
   ],
   "source": [
    "jobs_list[0].delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "eb0526d7-e3b0-49de-807d-f0e76fa883e2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Job not found - it was deleted!\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    jobs_list[0].state\n",
    "except:\n",
    "    print('Job not found - it was deleted!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e07420-62fa-4ace-955b-82d0f05acff5",
   "metadata": {},
   "source": [
    "### Clone A Job To Run It Again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "1c561ad4-524e-4912-96ce-8c9ce9d0c4c6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mlops-pipeline-jobstate-component-20250213214746'"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job2.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "aa334dc2-72b3-4477-aba2-41d44790b1ec",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PipelineState.PIPELINE_STATE_SUCCEEDED: 4>"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_job2.state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "ad06c801-cfad-49e8-9636-2ba822660ec2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "clone_job2 = pipeline_job2.clone()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "8a3ff10f-844a-4ac3-b864-f9096cb3bf80",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "google.cloud.aiplatform.pipeline_jobs.PipelineJob"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(clone_job2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "b3ea66c7-6752-49b3-94e9-f28155a1367f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/cloned-mlops-pipeline-jobstate-component-20250213224824\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/1026793852137/locations/us-central1/pipelineJobs/cloned-mlops-pipeline-jobstate-component-20250213224824')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/cloned-mlops-pipeline-jobstate-component-20250213224824?project=1026793852137\n"
     ]
    }
   ],
   "source": [
    "clone_job2.submit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "4dbbd36b-2995-4ce0-8c3a-cc7837c4eb34",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cloned-mlops-pipeline-jobstate-component-20250213224824'"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clone_job2.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "43732a79-7d22-40e7-8270-921dcf8c6a8a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob run completed. Resource name: projects/1026793852137/locations/us-central1/pipelineJobs/cloned-mlops-pipeline-jobstate-component-20250213224824\n"
     ]
    }
   ],
   "source": [
    "clone_job2.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "4c6b2290-830c-4103-b034-983aa76f9cab",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clone_job2.done()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d966acff-90da-477f-bd60-75cffa7f6446",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "763fd8c9-7f6c-4d6a-9620-16c1dcf1c153",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "conda-base-py",
   "name": "workbench-notebooks.m128",
   "type": "gcloud",
   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m128"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel) (Local)",
   "language": "python",
   "name": "conda-base-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}