Tips/Python Custom Containers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2aef4cf8",
   "metadata": {},
   "source": [
    "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Fvertex-ai-mlops%2FTips&file=Python+Custom+Containers.ipynb)\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Custom%20Containers.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Fvertex-ai-mlops%2Fmain%2FTips%2FPython%2520Custom%2520Containers.ipynb\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Custom%20Containers.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/vertex-ai-mlops/main/Tips/Python%20Custom%20Containers.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11f5c7a1-37b6-4bae-8e4a-89b4d8bd7af7",
   "metadata": {},
   "source": [
    "# Python Custom Containers\n",
    "\n",
    "Containers are helpful.\n",
    "\n",
    "> TL;DR\n",
    "> Use containers to bring together **software** and training **code** so that you can easily launch jobs on different **compute** with different **parameters** to simplify the operations of ML training.\n",
    "\n",
    "At the point in our workflow where we train an ML model a lot of things come together to make it happen:\n",
    "- **compute** in running: CPUs, memory, networking, GPUs on one or more instances\n",
    "- **software** is running on the compute\n",
    "    - the required packages are installed with the software\n",
    "- training **code**/script is launched with the software\n",
    "- training **data** is read by the training code\n",
    "- **parameters** that the code uses to configure the training run\n",
    "\n",
    "It's tempting to develop in an IDE, like JupyterLab here, and then just make the VM behind it much larger.  The notebook here is running in JupyterLab hosted on **compute** running **software** and is being used to author **code** that reads **data** using **parameters** set as Python variables.  One of the issues with this is that typing these words just cost `$$$$` and this instance might not be able to run this notebook 10 times in parallel with different **parameters**.  \n",
    "\n",
    "A better way?  Keep using an enviorment like this to develop our **code** and make sure it works. Just use smaller **compute** and **data** during this development process.  Then, launch a sepearate, managed job, that runs the full training.  How? What if we could instruct a service to take the list of inputs above and run a job and only charge for the compute used during the duration of training? That is exactly what Vertex AI Training is used for.  With this in mind it also helps scale the usefulness of training as a next step:\n",
    "- specify distributed training, pools of compute instances\n",
    "- manage hyperparameter tuning with multiple parallel training jobs focusing in on the right values for hyperparameters\n",
    "- run many training jobs at the same time without managing compute but also controling the cost of scaling\n",
    "\n",
    "Vertex AI has a [list of provided pre-built training containers](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers) for the most popular frameworks.  They are made available in multiple release versions of common frameworks both with and without [CUDA](https://developer.nvidia.com/cuda-toolkit) already configured and setup for GPU based training.\n",
    "\n",
    "> Sources for Google provided pre-built containers can be found here:\n",
    "> - [Deep Learning Containers](https://cloud.google.com/deep-learning-containers/docs/choosing-container)\n",
    ">     - list these with `gcloud container images list --repository=\"gcr.io/deeplearning-platform-release\"`\n",
    "> - [Vertex AI Pre-Built Containers for Custom Training](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers)\n",
    ">     - list these with `gcloud container images list --repository=\"us-docker.pkg.dev/vertex-ai/training\"`\n",
    "> - [Vertex AI Pre-Built Containers for Prediction](https://cloud.google.com/vertex-ai/docs/predictions/pre-built-containers)\n",
    ">     - list these with `gcloud container images list --repository=\"us-docker.pkg.dev/vertex-ai/prediction\"`\n",
    "\n",
    "For Vertex AI Training Custom Jobs you:\n",
    "- specify the **compute** to use as input parameters or as worker pool specs\n",
    "- provide a URI for a container with the **software** to use on each worker\n",
    "- provide training **code** in one of three ways\n",
    "    - as a link to a Python script (file.py)\n",
    "    - as URI to GCS for a Python Source Distribution\n",
    "    - as a starting point to code already included on the container with the **software**\n",
    "- provide **data**\n",
    "    - as a **parameter** specifying the location the **code** can use to read/retrieve it\n",
    "    - or build the logic for connecting to the data source into the **code**\n",
    "\n",
    "If we learn the skill of building a derivative containers that packages our desired **software** and installs additional packages while also holding a copy of our **code**, and maybe even our **parameters**, then this ML training job become very simple to incorporate in our workflow!\n",
    "\n",
    "That is this notebooks goal.\n",
    "\n",
    "---\n",
    "## Notes:\n",
    "\n",
    "**Prerequisite:**\n",
    "\n",
    "This notebook depends on ML training code prepared in multiple forms by the [Python Packages](./Python%20Packages.ipynb) notebook.  Please run that notebook first before running this notebook. \n",
    "\n",
    "**We will use [Cloud Build](https://cloud.google.com/build) to construct containers.**\n",
    "\n",
    "- [API Overview](https://cloud.google.com/build/docs/api)\n",
    "    - REST API, gcloud CLI, and Client Libraries for Go, Java, Node.js, and Python\n",
    "- [Python Client for Cloud Build API](https://github.com/googleapis/python-cloudbuild)\n",
    "- [Python Client Library Documentation](https://cloud.google.com/python/docs/reference/cloudbuild/latest)\n",
    "\n",
    "**We will store built containers in [Artifact Registry](https://cloud.google.com/artifact-registry).**\n",
    "\n",
    "- [API Overview](https://cloud.google.com/artifact-registry/docs/apis)\n",
    "- [Python Client for Artifact Registry API](https://github.com/googleapis/python-artifact-registry)\n",
    "- [Python Client Library Documentation](https://cloud.google.com/python/docs/reference/artifactregistry/latest)\n",
    "\n",
    "**Notes on Python and Google Cloud:**\n",
    "\n",
    "Google Cloud APIs can be used with the [Google Cloud Python Client](https://github.com/googleapis/google-cloud-python).  The client has [libraries](https://github.com/googleapis/google-cloud-python#libraries) for Google Cloud services.  The documentation for each library is centralized in the [Python Cloud Client Libraries](https://cloud.google.com/python/docs/reference) reference documentation.\n",
    "- Also helpful: [Getting started with Python](https://cloud.google.com/python/docs/getting-started) in Google Cloud"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a30798bc-03ef-411f-8e14-7b589bb82efd",
   "metadata": {
    "id": "od_UkDpvRmgD",
    "tags": []
   },
   "source": [
    "---\n",
    "## Colab Setup\n",
    "\n",
    "To run this notebook in Colab click [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Custom%20Containers.ipynb) and run the cells in this section.  Otherwise, skip this section.\n",
    "\n",
    "This cell will authenticate to GCP (follow prompts in the popup)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e577228c-94ee-4c68-b81c-7c1b478f1967",
   "metadata": {
    "executionInfo": {
     "elapsed": 195,
     "status": "ok",
     "timestamp": 1683726184843,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "8UO9FnqyKBlF",
    "tags": []
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = 'statmike-mlops-349915' # replace with project ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "51a6dd90-c27c-49c8-bdff-edc4196028f0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 68869,
     "status": "ok",
     "timestamp": 1683726253709,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "N98-KK7LRkjm",
    "outputId": "09ec5008-0def-4e1a-c349-c598ee752f78",
    "tags": []
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "    from google.colab import auth\n",
    "    auth.authenticate_user()\n",
    "    !gcloud config set project {PROJECT_ID}\n",
    "except Exception:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "692c3c19-8444-4974-b720-c867a5185c41",
   "metadata": {},
   "source": [
    "---\n",
    "## Installs\n",
    "\n",
    "The list `packages` contains tuples of package import names and install names.  If the import name is not found then the install name is used to install quitely for the current user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5cf3f86d-bfa5-4ba4-8562-c27b297bcd44",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# tuples of (import name, install name)\n",
    "packages = [\n",
    "    ('google.cloud.artifactregistry_v1', 'google-cloud-artifact-registry'),\n",
    "    ('google.cloud.devtools', 'google-cloud-build')\n",
    "]\n",
    "\n",
    "import importlib\n",
    "install = False\n",
    "for package in packages:\n",
    "    if not importlib.util.find_spec(package[0]):\n",
    "        print(f'installing package {package[1]}')\n",
    "        install = True\n",
    "        !pip install {package[1]} -U -q --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be17632f-cc4e-4497-8feb-b75a42726dac",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Enable APIs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b23dd92b-45c2-4228-a27c-32c2cda2d239",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!gcloud services enable artifactregistry.googleapis.com\n",
    "!gcloud services enable cloudbuild.googleapis.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad209f0e-e59f-48e1-8be4-82a5aab15089",
   "metadata": {},
   "source": [
    "### Restart Kernel (If Installs Occured)\n",
    "\n",
    "After a kernel restart the code submission can start with the next cell after this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ad5388bb-2940-451b-89a8-1bbba49837f9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if install:\n",
    "    import IPython\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b1414d7-e68f-44b9-a331-4fb9a04e2538",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eed111b6-43f9-49e7-9770-63d2359779b2",
   "metadata": {},
   "source": [
    "inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "09680104-3052-4adb-b247-8ecaa4672a34",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project = !gcloud config get-value project\n",
    "PROJECT_ID = project[0]\n",
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b2a88a5e-0238-47ac-9e70-c761892c602d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "EXPERIMENT = 'containers'\n",
    "SERIES = 'tips'\n",
    "\n",
    "# specify a GCS Bucket\n",
    "GCS_BUCKET = PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7510728-c1d5-4015-b692-9867e894daa1",
   "metadata": {},
   "source": [
    "packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7629a781-519d-48b7-a135-812b31dea62d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os, shutil\n",
    "import importlib\n",
    "from google.cloud.devtools import cloudbuild_v1\n",
    "from google.cloud import artifactregistry_v1\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud import storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c94eba86-0904-47f4-b818-ac140346303f",
   "metadata": {},
   "source": [
    "clients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "31447726-a55a-4ef5-bd2d-2465215d22a9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ar_client = artifactregistry_v1.ArtifactRegistryClient()\n",
    "cb_client = cloudbuild_v1.CloudBuildClient()\n",
    "aiplatform.init(project = PROJECT_ID, location = REGION)\n",
    "gcs = storage.Client(project = PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53a5cfbe-2353-4418-901d-fa291c408233",
   "metadata": {},
   "source": [
    "parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f01a4cfe-8ade-4272-92a2-d78bc532cb89",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DIR = f'code/{EXPERIMENT}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fe1aaa9-983b-4ae8-a604-1f0993c7a159",
   "metadata": {},
   "source": [
    "environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3d3912e8-7365-4bda-8105-88f8fa298524",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['containers', 'packages']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if not os.path.exists(DIR):\n",
    "    os.makedirs(DIR)\n",
    "else:\n",
    "    shutil.rmtree(DIR, ignore_errors = True)\n",
    "    os.makedirs(DIR)\n",
    "\n",
    "# list contents of directory one level higher than DIR\n",
    "os.listdir(DIR + '/../')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d021aab2-f348-432a-be9a-648428132ed7",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup Artifact Registry\n",
    "\n",
    "Artifact registry organizes artifacts with repositories.  Each repository contains packages and is designated to hold a partifcular format of package: Docker images, Python Packages and [others](https://cloud.google.com/artifact-registry/docs/supported-formats#package)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea9e64b5-e813-4f51-add1-63292d5f227e",
   "metadata": {},
   "source": [
    "### List Repositories\n",
    "\n",
    "This may be empty if no repositories have been created for this project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "37d928c9-4968-4e8a-84f5-2c0e1cd67cb7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915\n",
      "projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-docker\n",
      "projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-python\n"
     ]
    }
   ],
   "source": [
    "for repo in ar_client.list_repositories(parent = f'projects/{PROJECT_ID}/locations/{REGION}'):\n",
    "    print(repo.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e69c2a28-2c2d-4499-aa88-387f4ac5c7c5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-python\"\n",
       "format_: PYTHON\n",
       "description: \"A repository for the statmike-mlops-349915 experiment that holds Python Packages.\"\n",
       "labels {\n",
       "  key: \"series\"\n",
       "  value: \"tips\"\n",
       "}\n",
       "labels {\n",
       "  key: \"experiment\"\n",
       "  value: \"packages\"\n",
       "}\n",
       "create_time {\n",
       "  seconds: 1663696036\n",
       "  nanos: 276742000\n",
       "}\n",
       "update_time {\n",
       "  seconds: 1703256229\n",
       "  nanos: 639447000\n",
       "}\n",
       "mode: STANDARD_REPOSITORY\n",
       "size_bytes: 7164\n",
       "cleanup_policy_dry_run: true"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e12ea93-29e5-4b97-a8ae-6758006db523",
   "metadata": {},
   "source": [
    "### Create Docker Image Repository\n",
    "\n",
    "Create an Artifact Registry Repository to hold Docker Images created by this notebook.  First, check to see if it is already created by a previous run and retrieve it if it has.  Otherwise, create!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "00dbbac5-ab76-4bfc-83ec-71a4d7e0ca80",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved existing repo: projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-docker\n"
     ]
    }
   ],
   "source": [
    "docker_repo = None\n",
    "for repo in ar_client.list_repositories(parent = f'projects/{PROJECT_ID}/locations/{REGION}'):\n",
    "    if f'{PROJECT_ID}-docker' in repo.name:\n",
    "        docker_repo = repo\n",
    "        print(f'Retrieved existing repo: {docker_repo.name}')\n",
    "\n",
    "if not docker_repo:\n",
    "    operation = ar_client.create_repository(\n",
    "        request = artifactregistry_v1.CreateRepositoryRequest(\n",
    "            parent = f'projects/{PROJECT_ID}/locations/{REGION}',\n",
    "            repository_id = f'{PROJECT_ID}-docker',\n",
    "            repository = artifactregistry_v1.Repository(\n",
    "                description = f'A repository for the {EXPERIMENT} experiment that holds docker images.',\n",
    "                name = f'{PROJECT_ID}-docker',\n",
    "                format_ = artifactregistry_v1.Repository.Format.DOCKER,\n",
    "                labels = {'series': SERIES, 'experiment': EXPERIMENT}\n",
    "            )\n",
    "        )\n",
    "    )\n",
    "    print('Creating Repository ...')\n",
    "    docker_repo = operation.result()\n",
    "    print(f'Completed creating repo: {docker_repo.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9d29aa6f-ff91-473a-97f3-b7c04ce0c9a9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-docker',\n",
       " 'DOCKER')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docker_repo.name, docker_repo.format_.name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344f3e6f-d5b5-4bdf-bd3d-85f93c0fecac",
   "metadata": {},
   "source": [
    "### Create Python Package Repository\n",
    "\n",
    "Create an Artifact Registry Repository to hold Python Packages created by this notebook.  First, check to see if it is already created by a previous run and retrieve it if it has.  Otherwise, create!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2739bb70-6bb3-4c86-bd4e-6930b9217560",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved existing repo: projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-python\n"
     ]
    }
   ],
   "source": [
    "python_repo = None\n",
    "for repo in ar_client.list_repositories(parent = f'projects/{PROJECT_ID}/locations/{REGION}'):\n",
    "    if f'{PROJECT_ID}-python' in repo.name:\n",
    "        python_repo = repo\n",
    "        print(f'Retrieved existing repo: {python_repo.name}')\n",
    "\n",
    "if not python_repo:\n",
    "    operation = ar_client.create_repository(\n",
    "        request = artifactregistry_v1.CreateRepositoryRequest(\n",
    "            parent = f'projects/{PROJECT_ID}/locations/{REGION}',\n",
    "            repository_id = f'{PROJECT_ID}-python',\n",
    "            repository = artifactregistry_v1.Repository(\n",
    "                description = f'A repository for the {PROJECT_ID} experiment that holds Python Packages.',\n",
    "                name = f'{PROJECT_ID}-python',\n",
    "                format_ = artifactregistry_v1.Repository.Format.PYTHON,\n",
    "                labels = {'series': SERIES, 'experiment': EXPERIMENT}\n",
    "            )\n",
    "        )\n",
    "    )\n",
    "    print('Creating Repository ...')\n",
    "    python_repo = operation.result()\n",
    "    print(f'Completed creating repo: {python_repo.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "36ab9f9d-7a2a-4f48-b3d2-c279f93247a6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-python',\n",
       " 'PYTHON')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "python_repo.name, python_repo.format_.name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a35f1519-0d36-48d0-822a-78084b5d1b5d",
   "metadata": {},
   "source": [
    "### List Repositories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "814c8ba4-556f-41be-ad24-4aebf360f983",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915\n",
      "projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-docker\n",
      "projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-python\n"
     ]
    }
   ],
   "source": [
    "for repo in ar_client.list_repositories(parent = f'projects/{PROJECT_ID}/locations/{REGION}'):\n",
    "    print(repo.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5382175c-d5a3-46d2-90d8-0c4b2e282676",
   "metadata": {},
   "source": [
    "---\n",
    "## Training Code\n",
    "\n",
    "ML Training code can take the form of a single script, a folder of scripts/modules, a Python Package distribution file. These could be located on the local disk, in GCS buckets, GitHub repository, or in a repository like Artifact Registry.  This notebook will explore many workflows for getting training code in all these forms and locations into a custom container.  \n",
    "\n",
    "The example training code being used here is from the [05 - TensorFlow](../05%20-%20TensorFlow/readme.md) series has a model training file named [train.py](../05%20-%20TensorFlow/code/train.py) which is created and annotated in the notebook [05 - Vertex AI Custom Model - TensorFlow - Notebook to Script.ipynb](../05%20-%20TensorFlow/05%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Notebook%20to%20Script.ipynb).\n",
    "\n",
    "The training code has already been prepared as local files, GCS hosted files and distribution, GitHub hosted files and distribution, and the distributions are stored in Artifact Registry as a Python Package.\n",
    "\n",
    "**Please pause here and review and run the tip in the [Python Packages](./Python%20Packages.ipynb) notebook first to prepare these versions for use in this notebook.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dded355-fcbd-4b88-9986-8608a07f0f17",
   "metadata": {},
   "source": [
    "### GCS Bucket: Files/Folders/Source Distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9e58e8bd-2b54-4053-828a-6c43d8bba473",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bucket = gcs.lookup_bucket(GCS_BUCKET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "185a89d3-6d27-4509-8523-fa64c2ad6662",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tips/code/packages/tips_trainer/dist/tips_trainer-0.1-py3-none-any.whl\n",
      "tips/code/packages/tips_trainer/dist/tips_trainer-0.1.tar.gz\n",
      "tips/code/packages/tips_trainer/pyproject.toml\n",
      "tips/code/packages/tips_trainer/src/tips_trainer.egg-info/PKG-INFO\n",
      "tips/code/packages/tips_trainer/src/tips_trainer.egg-info/SOURCES.txt\n",
      "tips/code/packages/tips_trainer/src/tips_trainer.egg-info/dependency_links.txt\n",
      "tips/code/packages/tips_trainer/src/tips_trainer.egg-info/requires.txt\n",
      "tips/code/packages/tips_trainer/src/tips_trainer.egg-info/top_level.txt\n",
      "tips/code/packages/tips_trainer/src/tips_trainer/__init__.py\n",
      "tips/code/packages/tips_trainer/src/tips_trainer/train.py\n"
     ]
    }
   ],
   "source": [
    "for blob in list(bucket.list_blobs(prefix = f'{SERIES}/code')):\n",
    "    print(blob.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8711345-fdff-4ffc-9425-6370a88a8d13",
   "metadata": {},
   "source": [
    "### Artifact Registry: Python Package Distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8ac4fec8-8525-49cd-9fac-56d8bcafe599",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ListPackagesPager<packages {\n",
       "  name: \"projects/statmike-mlops-349915/locations/us-central1/repositories/statmike-mlops-349915-python/packages/tips-trainer\"\n",
       "  create_time {\n",
       "    seconds: 1703256229\n",
       "    nanos: 454826000\n",
       "  }\n",
       "  update_time {\n",
       "    seconds: 1703256229\n",
       "    nanos: 639447000\n",
       "  }\n",
       "}\n",
       ">"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ar_client.list_packages(\n",
    "    parent = python_repo.name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4920042-fdaa-4741-8bff-1e9eee01b20b",
   "metadata": {},
   "source": [
    "---\n",
    "## Creating a Custom Container with Cloud Build\n",
    "\n",
    "Cloud Build creates and manages the build on GCP.  The API creates a build by providing:\n",
    "- location of the source\n",
    "- instructions\n",
    "- location to store the built artifacts\n",
    "\n",
    "The instruction part of Cloud Build has options:\n",
    "- Dockerfile\n",
    "- Build Config file (YAML or JSON)\n",
    "- Cloud Native Buildpacks\n",
    "\n",
    "This notebook uses the approach of using the Python Client for Cloud Build and not referencing any local files.  For that reason, the first step is creating a Dockerfile for the workflow and storing it in GCS. The next step is running Cloud Build and using the client to specify the Build config rather than a config file.  The steps of the build config start with getting the code (git clone, or copy from GCS) and copying the Dockerfile.  \n",
    "\n",
    "Other options:\n",
    "If you store the Dockerfile(s) in repository folder then you could go as far as having GitHub trigger a build on commit.  If docker is installed locally (and it is with user-managed notebook instances), then the dockerfile could be used locally with docker build to create the container.  \n",
    "\n",
    "There are multiple stand-alone subsections here that illustrate different workflows for building a custom container depending on where and how the training code is stored.  All of these method end up creating the same contents in a custom container and all have an entrypoint of `tip-trainer.train`.\n",
    "\n",
    "- [Workflow 1: copy script to container](#workflow1)\n",
    "- [Workflow 2: copy folder to container](#workflow2)\n",
    "- [Workflow 3: copy package to container](#workflow3)\n",
    "- [Workflow 4: pip install package from GCS to container](#workflow4)\n",
    "- [Workflow 5: pip install package from GitHub to container](#workflow5)\n",
    "- [Workflow 6: pip install package from Artifact Registry to container](#workflow6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9e7798e-08d7-4c61-a6de-7db08926fbc2",
   "metadata": {},
   "source": [
    "### Choose Computing Environment\n",
    "\n",
    "When using [custom training on Vertex AI](https://cloud.google.com/vertex-ai/docs/training/custom-training-methods) the compute environment is specified as parameters.  At a minimum this will include the [compute resources](https://cloud.google.com/vertex-ai/docs/training/configure-compute) and [container](https://cloud.google.com/vertex-ai/docs/training/configure-container-settings) URIs.\n",
    "\n",
    "This example uses minimal compute with a single node and no accelerators (GPU).\n",
    "\n",
    "For a container, a [pre-built custome training container](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers) for TensorFlow is being used and additional packages are being specified in the jobs parameters. This leads to two important considerations:\n",
    "- What is the Python version on the container? \n",
    "    - This will have an impact when additional packages are being installed and the packages may have version requirments.\n",
    "- What is the framework version on the container?\n",
    "    - Make sure the options you need from a framework are included in the version you pick."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5b668be0-7a5d-49e0-96cf-5547ab974248",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Resources\n",
    "#TRAIN_COMPUTE = 'n1-standard-4'\n",
    "#DEPLOY_COMPUTE = 'n1-standard-4'\n",
    "TRAIN_IMAGE = 'us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-12.py310:latest'\n",
    "#DEPLOY_IMAGE ='us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-12:latest'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11e6a417-1c37-424d-a020-4c2c0b6adebb",
   "metadata": {},
   "source": [
    "### Common Parameters for All Workflows\n",
    "\n",
    "Choose a Vertex AI Pre-Built container for ML Training from the list [here](https://cloud.google.com/vertex-ai/docs/training/pre-built-containers) and store in in the `TRAIN_IMAGE` parameter below.  Also, create a parameter the references the Artifact Registry custom repository for Docker images created above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "29afa08b-22a2-48d8-9444-effe1df00789",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "REPOSITORY = f\"{REGION}-docker.pkg.dev/{PROJECT_ID}/{docker_repo.name.split('/')[-1]}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d540b389-563b-48e4-a7aa-db8c040dcee1",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow1'></a>\n",
    "### Workflow 1: copy script to container\n",
    "\n",
    "This workflow builds a custom container using a Vertex AI Pre-Built container for traning as the base.  The code below stores the `requirements.txt`, `Dockerfile` and training script in the project GCS bucket and then launches a Cloud Build job that copies these and runs the Docker build process with Cloud Build.\n",
    "\n",
    "To see an an example Vertex AI Training Custom Job running with the custom container created here go to [Python Training - workflow 1](./Python%20Training.ipynb#workflow1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "60ef34f2-6329-4ba0-b52a-11fe3dd1c0d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "WORKFLOW = 'workflow_1' # name for this workflow\n",
    "CONTAINER = f'tips_trainer_{WORKFLOW}' # name for custom container\n",
    "SOURCEPATH = f'{SERIES}/{EXPERIMENT}/{WORKFLOW}' # in gcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cde31c46-41d3-492e-902c-d4ec2047a6fd",
   "metadata": {},
   "source": [
    "#### Source Code\n",
    "A copy of just the training code script that will be copied into the custom container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d4aee090-2f63-46bf-8fc6-3069d1161403",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Blob: statmike-mlops-349915, tips/containers/workflow_1/train.py, 1703256456536681>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket.copy_blob(\n",
    "    blob = bucket.get_blob(blob_name = f'{SERIES}/code/packages/tips_trainer/src/tips_trainer/train.py'),\n",
    "    destination_bucket = bucket,\n",
    "    new_name = f'{SOURCEPATH}/train.py'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c576dfd-b8bc-4fc2-8c44-f38107c543a3",
   "metadata": {},
   "source": [
    "#### Requirements.txt\n",
    "A list of requirments for the training code to run.  These packages will be added/updated on the pre-built container when the `Dockerfile` instructions runs `pip install ...`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "cc560708-68f1-4f18-96ac-895f716f8800",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "requirements = f\"\"\"\n",
    "tensorflow_io\n",
    "google-cloud-aiplatform>={aiplatform.__version__}\n",
    "db-dtypes\n",
    "protobuf>={importlib.metadata.version('protobuf')}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7f31f127-b618-4005-9459-26161c63045c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "blob = bucket.blob(f'{SOURCEPATH}/requirements.txt')\n",
    "blob.upload_from_string(requirements)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "929797c9-034c-40e9-b079-1c2a3b1c2378",
   "metadata": {},
   "source": [
    "#### Dockerfile\n",
    "The Docker build instructions.  This copies the `requirements.txt` and training script into the containers, `pip installs ...` the requirements, sets the entrypoint for the container to the training script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "57be688e-132e-4100-8b2b-9a4061ff68a6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dockerfile = f\"\"\"\n",
    "FROM {TRAIN_IMAGE}\n",
    "WORKDIR /training\n",
    "# copy requirements and install them\n",
    "COPY requirements.txt ./\n",
    "RUN pip install --no-cache-dir --upgrade pip \\\n",
    "  && pip install --no-cache-dir -r requirements.txt\n",
    "## Copies the trainer code to the docker image\n",
    "COPY train.py .\n",
    "## Sets up the entry point to invoke the trainer\n",
    "ENTRYPOINT [\"python\", \"train.py\"]\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b801ea8c-1b58-40d2-af00-a14be730edd9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "blob = bucket.blob(f'{SOURCEPATH}/Dockerfile')\n",
    "blob.upload_from_string(dockerfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f2ba39d-cfd4-4636-a6dd-0837fe7f3972",
   "metadata": {},
   "source": [
    "#### Build Custom Container\n",
    "Use the Cloud Build client to construct and run the build instructions.  Here the files collected in GCS are copied to the build instance, then the Docker build in run in the folder with the `Dockerfile`.  The resulting image is pushed to Artifact Registry (setup above).\n",
    "\n",
    "You may wish to speed up the build by increasing the vCPU available for the build.  Do this by uncommenting the line `build.options.machine_type = ` below.  Keep in mind that start-up time for the build may increase so this is really best for longer builds or when you run into timeout conditions.\n",
    "- [Increase vCPU for builds](https://cloud.google.com/build/docs/optimize-builds/increase-vcpu-for-builds#json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "05a0ff3a-dd11-4aee-8e74-2619484326f2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# setup the build config with empty list of steps - these will be added sequentially\n",
    "build = cloudbuild_v1.Build(\n",
    "    steps = []\n",
    ")\n",
    "# retrieve the source\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/gsutil',\n",
    "        'args': ['cp', '-r', f'gs://{GCS_BUCKET}/{SOURCEPATH}/*', '/workspace']\n",
    "    }\n",
    ")\n",
    "# docker build\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/docker',\n",
    "        'args': ['build', '-t', f'{REPOSITORY}/{CONTAINER}', '/workspace']\n",
    "    }    \n",
    ")\n",
    "# docker push\n",
    "#build.options.machine_type = 'N1_HIGHCPU_8'\n",
    "build.images = [f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d671c828-6c1c-4a79-ad58-0c312b6fba01",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/gsutil\"\n",
       "  args: \"cp\"\n",
       "  args: \"-r\"\n",
       "  args: \"gs://statmike-mlops-349915/tips/containers/workflow_1/*\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/docker\"\n",
       "  args: \"build\"\n",
       "  args: \"-t\"\n",
       "  args: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_1\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_1\""
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "f8aa3013-348c-42c9-8a1e-a9d0bffb7202",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "operation = cb_client.create_build(\n",
    "    project_id = PROJECT_ID,\n",
    "    build = build\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "c0186c43-3bd9-4ec2-a097-cc224f352d81",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Status.SUCCESS: 3>,\n",
       " images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_1\")"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = operation.result()\n",
    "response.status, response.artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "eec92e2b-cddd-4fec-b348-3af3446377e1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Custom Container with Artifact Registry in the Google Cloud Console:\n",
      "https://console.cloud.google.com/artifacts/docker/statmike-mlops-349915/us-central1/statmike-mlops-349915-docker?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the Custom Container with Artifact Registry in the Google Cloud Console:\\nhttps://console.cloud.google.com/artifacts/docker/{PROJECT_ID}/{REGION}/{PROJECT_ID}-docker?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "959f2640-bdd3-4317-9bc2-7dbd95beaf37",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow2'></a>\n",
    "### Workflow 2: copy folder to container\n",
    "\n",
    "This workflow builds a custom container using a Vertex AI Pre-Built container for traning as the base.  The code below stores the `requirements.txt`, `Dockerfile` and training script folder in the project GCS bucket and then launches a Cloud Build job that copies these and runs the Docker build process with Cloud Build.\n",
    "\n",
    "To see an an example Vertex AI Training Custom Job running with the custom container created here go to [Python Training - workflow 2](./Python%20Training.ipynb#workflow2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "80da5501-6fe8-4c9b-a9f7-03daddd7ee4c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "WORKFLOW = 'workflow_2' # name for this workflow\n",
    "CONTAINER = f'tips_trainer_{WORKFLOW}' # name for custom container\n",
    "SOURCEPATH = f'{SERIES}/{EXPERIMENT}/{WORKFLOW}' # in gcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f20b664-369c-400c-9ce1-567f395130f0",
   "metadata": {},
   "source": [
    "#### Source Code\n",
    "A copy of folder of training code that will be copied into the custom container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "71210fa2-3265-458c-a1c5-ffd3c32c7ff4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for blob in list(bucket.list_blobs(prefix = f'{SERIES}/code/packages/tips_trainer/src/tips_trainer/')):\n",
    "    foldername = '/'.join(blob.name.split('/')[-2:])\n",
    "    bucket.copy_blob(\n",
    "        blob = blob,\n",
    "        destination_bucket = bucket,\n",
    "        new_name = f\"{SOURCEPATH}/{foldername}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "05e7cd3e-6a95-4ed8-bb03-6734b5460529",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tips/containers/workflow_2/tips_trainer/__init__.py\n",
      "tips/containers/workflow_2/tips_trainer/train.py\n"
     ]
    }
   ],
   "source": [
    "for blob in list(bucket.list_blobs(prefix = SOURCEPATH)):\n",
    "    print(blob.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee895974-49cc-4b00-895b-0572ad3ec21e",
   "metadata": {},
   "source": [
    "#### Requirements.txt\n",
    "A list of requirments for the training code to run.  These packages will be added/updated on the pre-built container when the `Dockerfile` instructions runs `pip install ...`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "352de6fa-3f49-40de-a309-69acd6a207bb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "requirements = f\"\"\"\n",
    "tensorflow_io\n",
    "google-cloud-aiplatform>={aiplatform.__version__}\n",
    "db-dtypes\n",
    "protobuf>={importlib.metadata.version('protobuf')}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "e0289c92-d14a-426f-bf6b-aa8fc8e9f4b0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "blob = bucket.blob(f'{SOURCEPATH}/requirements.txt')\n",
    "blob.upload_from_string(requirements)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2da8d70-4351-4ae6-9594-1fd066304559",
   "metadata": {},
   "source": [
    "#### Dockerfile\n",
    "The Docker build instructions.  This copies the `requirements.txt` and training script into the containers, `pip installs ...` the requirements, sets the entrypoint for the container to the training script.\n",
    "\n",
    "Note: the `COPY` statement is moving multiple files so the destination must end in `/` here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "56e7869a-6802-4ef7-872f-23872a987ae1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dockerfile = f\"\"\"\n",
    "FROM {TRAIN_IMAGE}\n",
    "WORKDIR /training\n",
    "# copy requirements and install them\n",
    "COPY requirements.txt ./\n",
    "RUN pip install --no-cache-dir --upgrade pip \\\n",
    "  && pip install --no-cache-dir -r requirements.txt\n",
    "## Copies the trainer code to the docker image\n",
    "COPY tips_trainer/* ./tips_trainer/\n",
    "## Sets up the entry point to invoke the trainer\n",
    "ENTRYPOINT [\"python\", \"-m\", \"tips_trainer.train\"]\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "16af78fc-3396-4595-bd65-6891190bba2f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "blob = bucket.blob(f'{SOURCEPATH}/Dockerfile')\n",
    "blob.upload_from_string(dockerfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1714f2b6-0e64-485a-9d3f-9d6e8341129d",
   "metadata": {},
   "source": [
    "#### Build Custom Container\n",
    "Use the Cloud Build client to construct and run the build instructions.  Here the files collected in GCS are copied to the build instance, then the Docker build in run in the folder with the `Dockerfile`.  The resulting image is pushed to Artifact Registry (setup above).\n",
    "\n",
    "You may wish to speed up the build by increasing the vCPU available for the build.  Do this by uncommenting the line `build.options.machine_type = ` below.  Keep in mind that start-up time for the build may increase so this is really best for longer builds or when you run into timeout conditions.\n",
    "- [Increase vCPU for builds](https://cloud.google.com/build/docs/optimize-builds/increase-vcpu-for-builds#json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "c5e178d7-b094-4f64-81d3-f99bc5e3c7d1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# setup the build config with empty list of steps - these will be added sequentially\n",
    "build = cloudbuild_v1.Build(\n",
    "    steps = []\n",
    ")\n",
    "# retrieve the source\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/gsutil',\n",
    "        'args': ['cp', '-r', f'gs://{GCS_BUCKET}/{SOURCEPATH}/*', '/workspace']\n",
    "    }\n",
    ")\n",
    "# docker build\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/docker',\n",
    "        'args': ['build', '-t', f'{REPOSITORY}/{CONTAINER}', '/workspace']\n",
    "    }    \n",
    ")\n",
    "#build.options.machine_type = 'N1_HIGHCPU_8'\n",
    "# docker push\n",
    "build.images = [f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "05b1acb1-65fd-4db5-adbc-9a552c1155d8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/gsutil\"\n",
       "  args: \"cp\"\n",
       "  args: \"-r\"\n",
       "  args: \"gs://statmike-mlops-349915/tips/containers/workflow_2/*\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/docker\"\n",
       "  args: \"build\"\n",
       "  args: \"-t\"\n",
       "  args: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_2\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_2\""
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "5694ba96-33a7-4d2c-8e38-8063e51ae40b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "operation = cb_client.create_build(\n",
    "    project_id = PROJECT_ID,\n",
    "    build = build\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "1d2f4ca1-b31e-4ea1-870c-36b2e9e5e073",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Status.SUCCESS: 3>,\n",
       " images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_2\")"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = operation.result()\n",
    "response.status, response.artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "783902e1-b34b-4d45-b5bd-57b07ca1fab7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Custom Container with Artifact Registry in the Google Cloud Console:\n",
      "https://console.cloud.google.com/artifacts/docker/statmike-mlops-349915/us-central1/statmike-mlops-349915-docker?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the Custom Container with Artifact Registry in the Google Cloud Console:\\nhttps://console.cloud.google.com/artifacts/docker/{PROJECT_ID}/{REGION}/{PROJECT_ID}-docker?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6449b75e-8597-495f-a26d-966a004dec87",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow3'></a>\n",
    "### Workflow 3: copy package to container\n",
    "\n",
    "This workflow builds a custom container using a Vertex AI Pre-Built container for traning as the base.  The code below stores the Python package distribution and `Dockerfile` in the project GCS bucket and then launches a Cloud Build job that copies these and runs the Docker build process with Cloud Build.\n",
    "\n",
    "To see an an example Vertex AI Training Custom Job running with the custom container created here go to [Python Training - workflow 3](./Python%20Training.ipynb#workflow3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "f435cbd1-812c-454e-afa4-72a29deab262",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "WORKFLOW = 'workflow_3' # name for this workflow\n",
    "CONTAINER = f'tips_trainer_{WORKFLOW}' # name for custom container\n",
    "SOURCEPATH = f'{SERIES}/{EXPERIMENT}/{WORKFLOW}' # in gcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c3f1f98-c300-4419-81cc-68fc5be6f833",
   "metadata": {},
   "source": [
    "#### Built Python Distribution (.whl)\n",
    "A copy of the wheel (.whl) for the built Python Package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "6b353645-69e8-4898-91c2-aa9b07c140ae",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for blob in list(bucket.list_blobs(prefix = f'{SERIES}/code/packages/tips_trainer/dist/')):\n",
    "    filename = blob.name.split('/')[-1]\n",
    "    bucket.copy_blob(\n",
    "        blob = blob,\n",
    "        destination_bucket = bucket,\n",
    "        new_name = f\"{SOURCEPATH}/{filename}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "e76e85d1-01fd-4739-a81c-91a9f4c017cc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tips/containers/workflow_3/tips_trainer-0.1-py3-none-any.whl\n",
      "tips/containers/workflow_3/tips_trainer-0.1.tar.gz\n"
     ]
    }
   ],
   "source": [
    "for blob in list(bucket.list_blobs(prefix = SOURCEPATH)):\n",
    "    print(blob.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed8f57cc-7439-463e-90c4-2e0bb6fed792",
   "metadata": {},
   "source": [
    "#### Dockerfile\n",
    "The Docker build instructions.  This copies the `requirements.txt` and training script into the containers, `pip installs ...` the requirements, sets the entrypoint for the container to the training script.\n",
    "\n",
    "Note: the `COPY` statement is moving multiple files so the destination must end in `/` here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "60bb0638-e74c-41b4-b83f-43e319fbb9f7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dockerfile = f\"\"\"\n",
    "FROM {TRAIN_IMAGE}\n",
    "WORKDIR /training\n",
    "COPY * ./\n",
    "RUN pip install --no-cache-dir --upgrade pip \\\n",
    "  && pip install tips_trainer-0.1-py3-none-any.whl\n",
    "## Sets up the entry point to invoke the trainer\n",
    "ENTRYPOINT [\"python\", \"-m\", \"tips_trainer.train\"]\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "13dbd358-d4f9-4c1f-a5f2-b608045d761a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "blob = bucket.blob(f'{SOURCEPATH}/Dockerfile')\n",
    "blob.upload_from_string(dockerfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b94d694-5b9e-4f53-8257-c171f5e7801c",
   "metadata": {},
   "source": [
    "#### Build Custom Container\n",
    "Use the Cloud Build client to construct and run the build instructions.  Here the files collected in GCS are copied to the build instance, then the Docker build in run in the folder with the `Dockerfile`.  The resulting image is pushed to Artifact Registry (setup above).\n",
    "\n",
    "You may wish to speed up the build by increasing the vCPU available for the build.  Do this by uncommenting the line `build.options.machine_type = ` below.  Keep in mind that start-up time for the build may increase so this is really best for longer builds or when you run into timeout conditions.\n",
    "- [Increase vCPU for builds](https://cloud.google.com/build/docs/optimize-builds/increase-vcpu-for-builds#json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "aba99a29-5535-4039-819f-6fdc6c5c77ef",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# setup the build config with empty list of steps - these will be added sequentially\n",
    "build = cloudbuild_v1.Build(\n",
    "    steps = []\n",
    ")\n",
    "# retrieve the source\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/gsutil',\n",
    "        'args': ['cp', '-r', f'gs://{GCS_BUCKET}/{SOURCEPATH}/*', '/workspace']\n",
    "    }\n",
    ")\n",
    "# docker build\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/docker',\n",
    "        'args': ['build', '-t', f'{REPOSITORY}/{CONTAINER}', '/workspace']\n",
    "    }    \n",
    ")\n",
    "#build.options.machine_type = 'N1_HIGHCPU_8'\n",
    "# docker push\n",
    "build.images = [f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "9b31d245-70e8-4921-954c-fc03fffbebe9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/gsutil\"\n",
       "  args: \"cp\"\n",
       "  args: \"-r\"\n",
       "  args: \"gs://statmike-mlops-349915/tips/containers/workflow_3/*\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/docker\"\n",
       "  args: \"build\"\n",
       "  args: \"-t\"\n",
       "  args: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_3\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_3\""
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "a78939a7-b27c-4b73-ad40-3969accf9903",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "operation = cb_client.create_build(\n",
    "    project_id = PROJECT_ID,\n",
    "    build = build\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "0fccb572-8ea8-4a91-9edc-725646da7f0c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Status.SUCCESS: 3>,\n",
       " images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_3\")"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = operation.result()\n",
    "response.status, response.artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b3b26ae9-a17b-4eb9-8632-612426edad57",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Custom Container with Artifact Registry in the Google Cloud Console:\n",
      "https://console.cloud.google.com/artifacts/docker/statmike-mlops-349915/us-central1/statmike-mlops-349915-docker?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the Custom Container with Artifact Registry in the Google Cloud Console:\\nhttps://console.cloud.google.com/artifacts/docker/{PROJECT_ID}/{REGION}/{PROJECT_ID}-docker?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d8305a4-b746-4a8a-a494-fc72ed0b842a",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow4'></a>\n",
    "### Workflow 4: pip install package from GCS to container\n",
    "\n",
    "This workflow builds a custom container using a Vertex AI Pre-Built container for traning as the base.  The code below stores the `Dockerfile` in the project GCS bucket and then launches a Cloud Build job that copies this and runs the Docker build process with Cloud Build.\n",
    "\n",
    "To see an an example Vertex AI Training Custom Job running with the custom container created here go to [Python Training - workflow 4](./Python%20Training.ipynb#workflow4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "5f2392c2-a8d8-4af7-8455-68639746e533",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "WORKFLOW = 'workflow_4' # name for this workflow\n",
    "CONTAINER = f'tips_trainer_{WORKFLOW}' # name for custom container\n",
    "SOURCEPATH = f'{SERIES}/{EXPERIMENT}/{WORKFLOW}' # in gcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "047a913b-ce6f-4dde-ab01-babe10cf7886",
   "metadata": {},
   "source": [
    "#### Dockerfile\n",
    "The Docker build instructions.  This copies the `requirements.txt` and training script into the containers, `pip installs ...` the requirements, sets the entrypoint for the container to the training script.\n",
    "\n",
    "Note: the `COPY` statement is moving multiple files so the destination must end in `/` here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "5938a76c-cc57-4e18-af76-f811f0e8f416",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dockerfile = f\"\"\"\n",
    "FROM {TRAIN_IMAGE}\n",
    "WORKDIR /training\n",
    "COPY * ./\n",
    "RUN pip install --no-cache-dir --upgrade pip\n",
    "RUN pip install *.whl\n",
    "## Sets up the entry point to invoke the trainer\n",
    "ENTRYPOINT [\"python\", \"-m\", \"tips_trainer.train\"]\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "f3031661-507c-4e2e-a6b0-acf500a0c9c7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "blob = bucket.blob(f'{SOURCEPATH}/Dockerfile')\n",
    "blob.upload_from_string(dockerfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd16fe6-42f0-4011-b62e-359d6699c912",
   "metadata": {},
   "source": [
    "#### Build Custom Container\n",
    "Use the Cloud Build client to construct and run the build instructions.  Here the files collected in GCS are copied to the build instance, then the Docker build in run in the folder with the `Dockerfile`.  The resulting image is pushed to Artifact Registry (setup above).\n",
    "\n",
    "You may wish to speed up the build by increasing the vCPU available for the build.  Do this by uncommenting the line `build.options.machine_type = ` below.  Keep in mind that start-up time for the build may increase so this is really best for longer builds or when you run into timeout conditions.\n",
    "- [Increase vCPU for builds](https://cloud.google.com/build/docs/optimize-builds/increase-vcpu-for-builds#json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "106d580f-d0bb-4b6e-9d02-b0306279017f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# setup the build config with empty list of steps - these will be added sequentially\n",
    "build = cloudbuild_v1.Build(\n",
    "    steps = []\n",
    ")\n",
    "# retrieve the source\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/gsutil',\n",
    "        'args': ['cp', '-r', f'gs://{GCS_BUCKET}/{SOURCEPATH}/*', '/workspace']\n",
    "    }\n",
    ")\n",
    "# retrieve the python package from gcs\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/gsutil',\n",
    "        'args': ['cp', f'gs://{GCS_BUCKET}/{SERIES}/code/packages/tips_trainer/dist/*.whl', '/workspace']\n",
    "    }\n",
    ")\n",
    "# docker build\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/docker',\n",
    "        'args': ['build', '-t', f'{REPOSITORY}/{CONTAINER}', '/workspace']\n",
    "    }    \n",
    ")\n",
    "#build.options.machine_type = 'N1_HIGHCPU_8'\n",
    "# docker push\n",
    "build.images = [f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "98258090-5d1b-4c37-9d9f-a83e74cf4d82",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/gsutil\"\n",
       "  args: \"cp\"\n",
       "  args: \"-r\"\n",
       "  args: \"gs://statmike-mlops-349915/tips/containers/workflow_4/*\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/gsutil\"\n",
       "  args: \"cp\"\n",
       "  args: \"gs://statmike-mlops-349915/tips/code/packages/tips_trainer/dist/*.whl\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/docker\"\n",
       "  args: \"build\"\n",
       "  args: \"-t\"\n",
       "  args: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_4\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_4\""
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "f6ef2d96-5f8b-406f-8569-2551727fe0f1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "operation = cb_client.create_build(\n",
    "    project_id = PROJECT_ID,\n",
    "    build = build\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "d00bd264-695f-4ced-a0ac-63f4f30b2d76",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Status.SUCCESS: 3>,\n",
       " images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_4\")"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = operation.result()\n",
    "response.status, response.artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "f5153a87-e081-4ed8-a2c6-c09c20c7803a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Custom Container with Artifact Registry in the Google Cloud Console:\n",
      "https://console.cloud.google.com/artifacts/docker/statmike-mlops-349915/us-central1/statmike-mlops-349915-docker?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the Custom Container with Artifact Registry in the Google Cloud Console:\\nhttps://console.cloud.google.com/artifacts/docker/{PROJECT_ID}/{REGION}/{PROJECT_ID}-docker?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c16c5b27-7edc-448a-a7a5-023176e3300a",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow5'></a>\n",
    "### Workflow 5: pip install package from GitHub to container\n",
    "\n",
    "This workflow builds a custom container using a Vertex AI Pre-Built container for traning as the base.  The code below stores the `Dockerfile` in the project GCS bucket and then launches a Cloud Build job that copies this and runs the Docker build process with Cloud Build.\n",
    "\n",
    "To see an an example Vertex AI Training Custom Job running with the custom container created here go to [Python Training - workflow 5](./Python%20Training.ipynb#workflow5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "257ad906-1fd7-44e5-95fd-1b4544b682f9",
   "metadata": {},
   "source": [
    "```pip install https://github.com/statmike/vertex-ai-mlops/blob/main/Tips/code/tips_trainer/dist/tips_trainer-0.1-py3-none-any.whl?raw=true```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "37d655d7-a8f4-4a49-ab49-5c46b2b3a32b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "WORKFLOW = 'workflow_5' # name for this workflow\n",
    "CONTAINER = f'tips_trainer_{WORKFLOW}' # name for custom container\n",
    "SOURCEPATH = f'{SERIES}/{EXPERIMENT}/{WORKFLOW}' # in gcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22950778-ca95-453f-853e-116bc60bfe93",
   "metadata": {},
   "source": [
    "#### Dockerfile\n",
    "The Docker build instructions.  This copies the `requirements.txt` and training script into the containers, `pip installs ...` the requirements, sets the entrypoint for the container to the training script.\n",
    "\n",
    "Note: the `COPY` statement is moving multiple files so the destination must end in `/` here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "fd13f15f-dd99-49b7-bc32-b8568d853e53",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dockerfile = f\"\"\"\n",
    "FROM {TRAIN_IMAGE}\n",
    "WORKDIR /training\n",
    "COPY * ./\n",
    "RUN pip install --no-cache-dir --upgrade pip \\\n",
    "  && pip install https://github.com/statmike/vertex-ai-mlops/blob/main/Tips/code/packages/tips_trainer/dist/tips_trainer-0.1-py3-none-any.whl?raw=true\n",
    "## Sets up the entry point to invoke the trainer\n",
    "ENTRYPOINT [\"python\", \"-m\", \"tips_trainer.train\"]\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "5a8af4d7-5236-48ae-9351-63ad59de250b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "blob = bucket.blob(f'{SOURCEPATH}/Dockerfile')\n",
    "blob.upload_from_string(dockerfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27ff13b6-6ae7-498b-b8d6-733a235d3d6b",
   "metadata": {},
   "source": [
    "#### Build Custom Container\n",
    "Use the Cloud Build client to construct and run the build instructions.  Here the files collected in GCS are copied to the build instance, then the Docker build in run in the folder with the `Dockerfile`.  The resulting image is pushed to Artifact Registry (setup above).\n",
    "\n",
    "You may wish to speed up the build by increasing the vCPU available for the build.  Do this by uncommenting the line `build.options.machine_type = ` below.  Keep in mind that start-up time for the build may increase so this is really best for longer builds or when you run into timeout conditions.\n",
    "- [Increase vCPU for builds](https://cloud.google.com/build/docs/optimize-builds/increase-vcpu-for-builds#json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "2404efee-e7b3-42fd-b22d-b4a434af9cd0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# setup the build config with empty list of steps - these will be added sequentially\n",
    "build = cloudbuild_v1.Build(\n",
    "    steps = []\n",
    ")\n",
    "# retrieve the source\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/gsutil',\n",
    "        'args': ['cp', '-r', f'gs://{GCS_BUCKET}/{SOURCEPATH}/*', '/workspace']\n",
    "    }\n",
    ")\n",
    "# docker build\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/docker',\n",
    "        'args': ['build', '-t', f'{REPOSITORY}/{CONTAINER}', '/workspace']\n",
    "    }    \n",
    ")\n",
    "#build.options.machine_type = 'N1_HIGHCPU_8'\n",
    "# docker push\n",
    "build.images = [f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "db17e316-186c-4b4d-9153-5f5f82f2523e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/gsutil\"\n",
       "  args: \"cp\"\n",
       "  args: \"-r\"\n",
       "  args: \"gs://statmike-mlops-349915/tips/containers/workflow_5/*\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/docker\"\n",
       "  args: \"build\"\n",
       "  args: \"-t\"\n",
       "  args: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_5\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_5\""
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "2c654b2e-2cf6-4ed9-915c-a3821a36ad53",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "operation = cb_client.create_build(\n",
    "    project_id = PROJECT_ID,\n",
    "    build = build\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "7c454748-fc51-49e5-a63a-da805fcf6bb0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Status.SUCCESS: 3>,\n",
       " images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_5\")"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = operation.result()\n",
    "response.status, response.artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "68f01b75-36a2-4397-b311-d52c6158bb9c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Custom Container with Artifact Registry in the Google Cloud Console:\n",
      "https://console.cloud.google.com/artifacts/docker/statmike-mlops-349915/us-central1/statmike-mlops-349915-docker?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the Custom Container with Artifact Registry in the Google Cloud Console:\\nhttps://console.cloud.google.com/artifacts/docker/{PROJECT_ID}/{REGION}/{PROJECT_ID}-docker?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a491a4-432b-4f1a-b964-b3976de880f0",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow6'></a>\n",
    "### Workflow 6: pip install package from Artifact Registry to container\n",
    "\n",
    "This workflow builds a custom container using a Vertex AI Pre-Built container for traning as the base.  The code below stores the `Dockerfile` in the project GCS bucket and then launches a Cloud Build job that copies this and runs the Docker build process with Cloud Build.\n",
    "\n",
    "To see an an example Vertex AI Training Custom Job running with the custom container created here go to [Python Training - workflow 6](./Python%20Training.ipynb#workflow6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e41c011-66cd-4625-b874-df04f73ff735",
   "metadata": {},
   "source": [
    "```!pip install --index-url https://{REGION}-python.pkg.dev/{PROJECT_ID}/{PROJECT_ID}-python/simple tips-trainer```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "cf6abac0-6b51-4c28-b2d8-428615a4c13e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "WORKFLOW = 'workflow_6' # name for this workflow\n",
    "CONTAINER = f'tips_trainer_{WORKFLOW}' # name for custom container\n",
    "SOURCEPATH = f'{SERIES}/{EXPERIMENT}/{WORKFLOW}' # in gcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306c114c-ee33-4526-90ea-1624481d7966",
   "metadata": {},
   "source": [
    "#### Dockerfile\n",
    "The Docker build instructions.  This copies the `requirements.txt` and training script into the containers, `pip installs ...` the requirements, sets the entrypoint for the container to the training script.\n",
    "\n",
    "Note: the `COPY` statement is moving multiple files so the destination must end in `/` here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "4bdecc52-16a3-4640-8e61-151465f31f45",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dockerfile = f\"\"\"\n",
    "FROM {TRAIN_IMAGE}\n",
    "WORKDIR /training\n",
    "COPY * ./\n",
    "RUN pip install --no-cache-dir --upgrade pip\n",
    "RUN pip install *.whl\n",
    "## Sets up the entry point to invoke the trainer\n",
    "ENTRYPOINT [\"python\", \"-m\", \"tips_trainer.train\"]\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "5c624108-bce0-49fb-8578-e6541439594c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "blob = bucket.blob(f'{SOURCEPATH}/Dockerfile')\n",
    "blob.upload_from_string(dockerfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f857b8e-80b5-4a9d-ac0d-52bd0f6945d3",
   "metadata": {},
   "source": [
    "#### Build Custom Container\n",
    "Use the Cloud Build client to construct and run the build instructions.  Here the files collected in GCS are copied to the build instance, then the Docker build in run in the folder with the `Dockerfile`.  The resulting image is pushed to Artifact Registry (setup above).\n",
    "\n",
    "You may wish to speed up the build by increasing the vCPU available for the build.  Do this by uncommenting the line `build.options.machine_type = ` below.  Keep in mind that start-up time for the build may increase so this is really best for longer builds or when you run into timeout conditions.\n",
    "- [Increase vCPU for builds](https://cloud.google.com/build/docs/optimize-builds/increase-vcpu-for-builds#json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "2e5f7cae-7908-4aa5-999a-5de14ef1e0c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# setup the build config with empty list of steps - these will be added sequentially\n",
    "build = cloudbuild_v1.Build(\n",
    "    steps = []\n",
    ")\n",
    "# get permissions for service account\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'python',\n",
    "        'entrypoint': 'pip',\n",
    "        'args': ['download', '--no-deps', '-d', 'workspace', '--index-url', f'https://{REGION}-python.pkg.dev/{PROJECT_ID}/{PROJECT_ID}-python/simple', 'tips-trainer']\n",
    "    }\n",
    ")\n",
    "# retrieve the source\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/gsutil',\n",
    "        'args': ['cp', '-r', f'gs://{GCS_BUCKET}/{SOURCEPATH}/*', '/workspace']\n",
    "    }\n",
    ")\n",
    "# docker build\n",
    "build.steps.append(\n",
    "    {\n",
    "        'name': 'gcr.io/cloud-builders/docker',\n",
    "        'args': ['build', '-t', f'{REPOSITORY}/{CONTAINER}', '/workspace']\n",
    "    }    \n",
    ")\n",
    "#build.options.machine_type = 'N1_HIGHCPU_8'\n",
    "# docker push\n",
    "build.images = [f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "4bd1fee5-3541-45ae-8973-effed4d7ba5d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "steps {\n",
       "  name: \"python\"\n",
       "  args: \"download\"\n",
       "  args: \"--no-deps\"\n",
       "  args: \"-d\"\n",
       "  args: \"workspace\"\n",
       "  args: \"--index-url\"\n",
       "  args: \"https://us-central1-python.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-python/simple\"\n",
       "  args: \"tips-trainer\"\n",
       "  entrypoint: \"pip\"\n",
       "}\n",
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/gsutil\"\n",
       "  args: \"cp\"\n",
       "  args: \"-r\"\n",
       "  args: \"gs://statmike-mlops-349915/tips/containers/workflow_6/*\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "steps {\n",
       "  name: \"gcr.io/cloud-builders/docker\"\n",
       "  args: \"build\"\n",
       "  args: \"-t\"\n",
       "  args: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_6\"\n",
       "  args: \"/workspace\"\n",
       "}\n",
       "images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_6\""
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "82357c67-cf07-43cb-9a39-1618e72d2d1e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "operation = cb_client.create_build(\n",
    "    project_id = PROJECT_ID,\n",
    "    build = build\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "8bd2f2e0-aaf7-4d18-b6e6-0d276a14e2be",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Status.SUCCESS: 3>,\n",
       " images: \"us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_6\")"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = operation.result()\n",
    "response.status, response.artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "81a1ce6d-eb1e-4184-bfc5-8b03862af659",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Custom Container with Artifact Registry in the Google Cloud Console:\n",
      "https://console.cloud.google.com/artifacts/docker/statmike-mlops-349915/us-central1/statmike-mlops-349915-docker?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the Custom Container with Artifact Registry in the Google Cloud Console:\\nhttps://console.cloud.google.com/artifacts/docker/{PROJECT_ID}/{REGION}/{PROJECT_ID}-docker?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bba6fd5-ef56-4af2-bd0e-8cc3931db78f",
   "metadata": {},
   "source": [
    "---\n",
    "## Using Custom Containers in Vertex AI Training Jobs\n",
    "\n",
    "The prior notebook, [Python Packages](./Python%20Packages.ipynb), created versions of training code (script, folder, distribution) in multiple locations (local, GCS Bucket, Artifact Registry, GitHub).\n",
    "\n",
    "This notebook, [Python Custom Containers](./Python%20Custom%20Containers.ipynb), demonstrated many workflow for getting the training code into a custom container.\n",
    "\n",
    "The next notebook, [Python Training](./Python%20Training.ipynb), uses the training code and custom containers to run Vertex AI Training Custom Jobs demonstrated in [Python Training](./Python%20Training.ipynb).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecf3fc69-39a5-4a76-b813-c6df87f53ce4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "conda-root-py",
   "name": "workbench-notebooks.m113",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/workbench-notebooks:m113"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel) (Local)",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}