notebooks/a1_init_dvc_and_track_data.ipynb

{"cells":[{"attachments":{},"cell_type":"markdown","metadata":{},"source":["# Initialize DVC and Start Tracking Data\n","\n","### 🎯 Introducing Data Versioning to our MLOps Workflow\n","\n","In the previous notebook (a0) we retrieved data, nicely packaged it in a single dataframe, serialized that dataframe and put it in a Cloud Object Storage Bucket.\n","\n","**Why do Data Versioning?**<br>\n","As time passes we want to ensure the quality of our model, so we regularly download the latest data and add it to our dataset.\n","If our model ever performs worse after adding a month of data to our training process, we want to be able to roll back and therefore require data versioning.\n","\n","---\n","\n","When this notebook is executed, we expect <br>**(1)** the existence of a private git repository for DVC tracking metadata, as well as <br>**(2)** the existence of historic ERA5/GloFAS data serialized (pickle) on Cloud Object Storage via the preceding notebook.\n","\n","Steps covered in this notebook:\n","1. Retrieve parameters and set-up COS connection\n","2. Set-up DVC situation\n","    - Clone empty *private* repository\n","    - ```dvc init```\n","    - Add COS instance as remote to DVC configuration file\n","3. Download dataset from COS\n","4. Track dataset (```git add```, ```dvc push```, ...)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Install required packages.\n","# TODO: Create IBM Cloud Software Configuration for those\n","!pip install ibm-cos-sdk ibm_watson_studio_pipelines 'dvc[s3]' # dvc[all] alternatively, however, COS is covered by S3"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["from botocore.client import Config\n","from sklearn.model_selection import train_test_split\n","from dataclasses import dataclass\n","import numpy as np\n","import pandas as pd\n","\n","from ibm_watson_studio_pipelines import WSPipelines\n","import ibm_boto3\n","\n","import logging\n","import os, types\n","import warnings\n","\n","warnings.filterwarnings(\"ignore\")"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["### 1. Retrieve parameters and set-up COS connection\n","\n","**Note**: If you are running this notebook outside of a Watson Studio Pipeline execution. Make sure to set the environment variables that the Pipeline environment would have passed to the notebook.\n","Refer to ```credentials.py```."]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Uncomment this cell and put your credentials in credentials.py to run locally.\n","from credentials2 import set_env_variables_for_credentials\n","set_env_variables_for_credentials()"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["## Retrieve cos credentials from global pipeline parameters\n","import json\n","# Get json from environment and convert to string\n","project_cos_credentials = json.loads(os.getenv('PROJECT_COS_CREDENTIALS'))\n","mlops_cos_credentials = json.loads(os.getenv('MLOPS_COS_CREDENTIALS'))\n","\n","## PROJECT COS \n","AUTH_ENDPOINT = project_cos_credentials['AUTH_ENDPOINT']\n","ENDPOINT_URL = project_cos_credentials['ENDPOINT_URL']\n","API_KEY_COS = project_cos_credentials['API_KEY']\n","BUCKET_PROJECT_COS = project_cos_credentials['BUCKET']\n","\n","## MLOPS COS\n","ENDPOINT_URL_MLOPS = mlops_cos_credentials['ENDPOINT_URL']\n","API_KEY_MLOPS = mlops_cos_credentials['API_KEY']\n","CRN_MLOPS = mlops_cos_credentials['CRN']\n","BUCKET_MLOPS  = mlops_cos_credentials['BUCKET']"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["CLOUD_API_KEY = os.getenv(\"CLOUD_API_KEY\")\n","DATA_FILENAME = os.getenv(\"serialized_data_filename\")"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# # @hidden_cell\n","# CLOUD_API_KEY = \"\"\n","# DATA_FILENAME = \"\""]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["## 2. Set-up DVC Situation"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["Clone a (preferably empty) git repository which will be used for data and model version tracking.<br> It will store ```.dvc``` files **and it will contain the remote locations as well as the corresponding access keys.**<br> Make sure to create a private repository or work with GitHub Enterprise.\n","\n","The following cells expect the repository to be empty, however they should be able to skip cells if they have already been completed. Their nature is non-overwriting."]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["#### 2.1. Clone Empty Repository for Versioning w/ DVC"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# NOTE: env set in credentials.py\n","!git clone $GIT_REPOSITORY"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && dvc init"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["#### 2.2. Add IBM COS Instance to dvc.config as remote\n","\n","To successfully complete this step, make sure that you create Cloud Object Storage \"Credentials\" for the COS Instance that you want to use.\n","<br>**Note:** Make sure to enable HMAC credentials when generating the \"Credentials\" in IBM Cloud."]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && dvc remote add -d -f ibm-cos s3://mlops-sustainability-data/"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && dvc remote modify ibm-cos endpointurl https://s3.eu-de.cloud-object-storage.appdomain.cloud"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && dvc remote modify ibm-cos access_key_id $HMAC_ADMIN_ACCESS_KEY"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && dvc remote modify ibm-cos secret_access_key $HMAC_ADMIN_SECRET_ACCESS_KEY"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && git commit .dvc/config -m \"Configure IBM COS (S3) as remote storage\""]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && dvc push"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["#### 2.3. Beginning tracking concatenated ERA5/GloFAS data\n","\n","The purpose of tracking the whole unsplitted dataset is solely for safety. We will track the train/test split data separately in a later notebook."]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && mkdir data"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!mv era5-glofas-merged.pkl dvc-testing/data"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && dvc add data/era5-glofas-merged.pkl"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# To track the cahnges with git, run:\n","!cd dvc-testing && git add data/.gitignore data/era5-glofas-merged.pkl.dvc"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && git commit -m \"Newest ERA5xGloFAS data\""]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && git push"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# To enable auto staging, run:\n","!dvc config core.autostage true"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && git config --global user.email os.getenv(\"GIT_USER_EMAIL\")\n","!cd dvc-testing && git config --global user.name os.getenv(\"GIT_USER_NAME\")"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# !cd dvc-testing && git add "]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && git commit -m \"Add concatenated ERA5 and GloFas data\""]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!cd dvc-testing && dvc push"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["### 3. Hand-off to Next Pipeline Node"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["validation_params = {}\n","validation_params[\"tracking_merged\"] = True"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["pipelines_client = WSPipelines.from_apikey(apikey=CLOUD_API_KEY)\n","pipelines_client.store_results(validation_params)"]}],"metadata":{"kernelspec":{"display_name":"Python 3.10","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.6"}},"nbformat":4,"nbformat_minor":1}