Tips/Python Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "57eeb6d5",
   "metadata": {},
   "source": [
    "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Fvertex-ai-mlops%2FTips&file=Python+Training.ipynb)\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Training.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Fvertex-ai-mlops%2Fmain%2FTips%2FPython%2520Training.ipynb\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Training.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/vertex-ai-mlops/main/Tips/Python%20Training.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c04da120-7bcb-4d5d-afa2-adae2435d48a",
   "metadata": {},
   "source": [
    "# Python Training - Vertex AI Training Custom Jobs\n",
    "\n",
    "ML Training with Python code as a Vertex AI Training Custom Job\n",
    "\n",
    "Why?  This notebook is an IDE that happens to also happen to have:\n",
    "- **compute**: CPU, Memory, GPU\n",
    "- **software**: container running with Python and loaded packages like TensorFlow, PyTorch, ...\n",
    "- **code**: user-written instruction for ML training\n",
    "\n",
    "But scaling this notebook instance to run our ML training code has limitations:\n",
    "- paying `$$$$` while typing and troubleshooting\n",
    "- running training code multiple times with different data sources\n",
    "- running training code with multiple configuration of hyperparameters for tuning\n",
    "- automating training code execution based on time or events\n",
    "\n",
    "Rather than scaling this notebook up to larger **compute** we want to launch a fit for purpose job that runs our training **code** using the **software** of choice on the needed **compute** to handle the size of our training data.  That is made simple with Vertex AI Training Custom Jobs.  \n",
    "\n",
    "Our training code can be in many locations and forms:\n",
    "- local files\n",
    "    - single script\n",
    "    - folders/modules\n",
    "    - Python Package Distribution\n",
    "- GCS Bucket\n",
    "    - single script\n",
    "    - folders/modules\n",
    "    - Python Package Distribution\n",
    "- GitHub\n",
    "    - single script\n",
    "    - folders/modules\n",
    "    - Python Package Distribution\n",
    "- Repository\n",
    "    - Python Package hosted on Artifact Registry\n",
    "    \n",
    "Vertex AI Training Custom Jobs can use training code from:\n",
    "- local files: single script\n",
    "- GCS Bucket: Python Source Distribution\n",
    "- Custom Container\n",
    "    - Built with code originating at any of the locations and forms above!\n",
    "\n",
    "<p align=\"center\" width=\"100%\">\n",
    "    <img src=\"../architectures/overview/training.png\" width=\"45%\">\n",
    "    &nbsp; &nbsp; &nbsp; &nbsp;\n",
    "    <img src=\"../architectures/overview/training2.png\" width=\"45%\">\n",
    "</p>\n",
    "\n",
    "---\n",
    "\n",
    "**Prerequisites:**\n",
    "\n",
    "The examples below use:\n",
    "- the code in various formats created in the [Python Packages](./Python%20Packages.ipynb) notebook\n",
    "- the custom containers created in multiple workflows by the [Python Custom Containers](./Python%20Custom%20Containers.ipynb) notebook\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5bda18b-a646-472e-9e7e-bc3fdfcc29fa",
   "metadata": {
    "id": "od_UkDpvRmgD",
    "tags": []
   },
   "source": [
    "---\n",
    "## Colab Setup\n",
    "\n",
    "To run this notebook in Colab click [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/Tips/Python%20Training.ipynb) and run the cells in this section.  Otherwise, skip this section.\n",
    "\n",
    "This cell will authenticate to GCP (follow prompts in the popup)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d80bad04-125c-48ba-a904-a629cff223f3",
   "metadata": {
    "executionInfo": {
     "elapsed": 195,
     "status": "ok",
     "timestamp": 1683726184843,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "8UO9FnqyKBlF",
    "tags": []
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = 'statmike-mlops-349915' # replace with project ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e4b7b194-49ac-434e-9ba3-a7967e1db890",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 68869,
     "status": "ok",
     "timestamp": 1683726253709,
     "user": {
      "displayName": "Mike Henderson",
      "userId": "07691629187611687318"
     },
     "user_tz": 240
    },
    "id": "N98-KK7LRkjm",
    "outputId": "09ec5008-0def-4e1a-c349-c598ee752f78",
    "tags": []
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "    from google.colab import auth\n",
    "    auth.authenticate_user()\n",
    "    !gcloud config set project {PROJECT_ID}\n",
    "except Exception:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "339f5021-f581-4878-bc24-79aacdb98ccc",
   "metadata": {},
   "source": [
    "---\n",
    "## Installs\n",
    "\n",
    "The list `packages` contains tuples of package import names and install names.  If the import name is not found then the install name is used to install quitely for the current user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0b8b2dcb-a710-4b98-b417-7e453d523efc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# tuples of (import name, install name)\n",
    "packages = [\n",
    "    ('google.cloud.aiplatform', 'google-cloud-aiplatform')\n",
    "]\n",
    "\n",
    "import importlib\n",
    "install = False\n",
    "for package in packages:\n",
    "    if not importlib.util.find_spec(package[0]):\n",
    "        print(f'installing package {package[1]}')\n",
    "        install = True\n",
    "        !pip install {package[1]} -U -q --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd7eee2c-9767-4abc-9356-a672216bf1f5",
   "metadata": {},
   "source": [
    "### Restart Kernel (If Installs Occured)\n",
    "\n",
    "After a kernel restart the code submission can start with the next cell after this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0a6ad63e-0ec3-4bc3-8d3e-cc8a0bbe1e0a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if install:\n",
    "    import IPython\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01ae9d29-f857-40d2-b241-40c31bfa5fcc",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "644496ff-7ffd-46cd-a85e-9540deeaaa57",
   "metadata": {},
   "source": [
    "inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "04784902-baa1-4456-9758-e96d21125817",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'statmike-mlops-349915'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project = !gcloud config get-value project\n",
    "PROJECT_ID = project[0]\n",
    "PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0ef8a8b7-7656-47d0-9836-e71a9245d26b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "REGION = 'us-central1'\n",
    "EXPERIMENT = 'training'\n",
    "SERIES = 'tips'\n",
    "\n",
    "# specify a GCS Bucket\n",
    "GCS_BUCKET = PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd77fd3f-d8a1-411d-aceb-10071431ec7d",
   "metadata": {},
   "source": [
    "packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "eb896eef-71a5-4a39-8789-eb8db72bb1f4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os, shutil\n",
    "import importlib\n",
    "from datetime import datetime\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud import storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32db0a98-b020-426d-9b31-f3a57fadf0ba",
   "metadata": {},
   "source": [
    "clients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "ae2e2c96-45ef-4e4c-9587-2753eee35ce0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "aiplatform.init(project = PROJECT_ID, location = REGION)\n",
    "gcs = storage.Client(project = PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2aa029a0-c5d4-4038-829c-84e226e98b52",
   "metadata": {},
   "source": [
    "parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8f4a2d4c-d55a-4c2a-ba23-1d0b574cf121",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DIR = f'code/{EXPERIMENT}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7cc91aa5-1da1-40e4-aff4-ed393af203c5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1026793852137-compute@developer.gserviceaccount.com'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SERVICE_ACCOUNT = !gcloud config list --format='value(core.account)' \n",
    "SERVICE_ACCOUNT = SERVICE_ACCOUNT[0]\n",
    "SERVICE_ACCOUNT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14e5d421-db29-4068-8d17-cd6fb6e8befa",
   "metadata": {},
   "source": [
    "List the service accounts current roles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8fe3feb3-8656-4ca1-ab11-b44acb9a9f09",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROLE\n",
      "roles/bigquery.admin\n",
      "roles/owner\n",
      "roles/run.admin\n",
      "roles/secretmanager.secretAccessor\n",
      "roles/storage.objectAdmin\n"
     ]
    }
   ],
   "source": [
    "!gcloud projects get-iam-policy $PROJECT_ID --filter=\"bindings.members:$SERVICE_ACCOUNT\" --format='table(bindings.role)' --flatten=\"bindings[].members\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81af207a-0995-477b-8421-b5c48b34a609",
   "metadata": {},
   "source": [
    ">Note: If the resulting list is missing [roles/storage.objectAdmin](https://cloud.google.com/storage/docs/access-control/iam-roles) then [revisit the setup notebook](../00%20-%20Setup/00%20-%20Environment%20Setup.ipynb#permissions) and add this permission to the service account with the provided instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d787e41-2ab6-43f4-a5ee-288d14546aec",
   "metadata": {},
   "source": [
    "environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a8f343ba-a908-4c95-94e5-48d15690785f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['training', 'containers', 'packages']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if not os.path.exists(DIR):\n",
    "    os.makedirs(DIR)\n",
    "else:\n",
    "    shutil.rmtree(DIR, ignore_errors = True)\n",
    "    os.makedirs(DIR)\n",
    "    \n",
    "# list contents of directory one level higher than DIR\n",
    "os.listdir(DIR + '/../')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aec69b6-1e5d-4700-857c-bdd17308e5c9",
   "metadata": {},
   "source": [
    "---\n",
    "## Local Files\n",
    "\n",
    "Some of the method covered below either launch Vertex AI Training from local code or actuall run the code locally.  If the prior notebooks in the series have been run in this enviornment then the code is already present: \n",
    "\n",
    "If this notebook is being run in isolation (like Colab) then the following cell will copy the results from a prior run of these notebooks to the local drive from GCS."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca1403ce-e053-4745-8f4e-8e0d75d4b096",
   "metadata": {},
   "source": [
    "Make sure training code is in local directory (if not already included in a clone of this repository):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "b80c6e83-531c-409c-ab9b-10f94339b253",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training code found\n"
     ]
    }
   ],
   "source": [
    "train_dir = f'code/packages'\n",
    "if not os.path.exists(train_dir):\n",
    "    print('not found locally, copying from GCS...')\n",
    "    bucket = gcs.lookup_bucket(GCS_BUCKET)\n",
    "    for blob in list(bucket.list_blobs(prefix = f'{SERIES}/{train_dir}')):\n",
    "        #print(blob.name)\n",
    "        file_path = f'./{train_dir}' + blob.name.split(f'{SERIES}/{train_dir}')[-1]\n",
    "        if not os.path.exists(os.path.dirname(file_path)):\n",
    "            os.makedirs(os.path.dirname(file_path))\n",
    "        blob.download_to_filename(f'./{train_dir}' + blob.name.split(f'{SERIES}/{train_dir}')[-1])\n",
    "    print('Training code download')\n",
    "else:\n",
    "    print('Training code found')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cc27f97-0fcc-47c8-8ed2-7cb8523a31cf",
   "metadata": {},
   "source": [
    "---\n",
    "## Vertex AI Training Custom Jobs Example Workflows\n",
    "\n",
    "Vertex AI Training Custom Jobs can use:\n",
    "- a local script\n",
    "- GCS housed Python source distribution\n",
    "- a custom container\n",
    "    - all the workflows from the [Python Custom Containers](./Python%20Custom%20Container.ipynb) notebook\n",
    "\n",
    "This section show examples of running Vertex AI Custom Jobs in many different workflows.  It also shows how to uses the workflow and test the training script locally, in the notebook instance.\n",
    "\n",
    "**Examples Workflows**\n",
    "- [Custom Container - Workflow 1 - Copy Script To Container](#workflow1)\n",
    "- [Custom Container - Workflow 2 - Copy Folder To Container](#workflow2)\n",
    "- [Custom Container - Workflow 3 - Copy Package To Container](#workflow3)\n",
    "- [Custom Container - Workflow 4 - pip install package from GCS to container](#workflow4)\n",
    "- [Custom Container - Workflow 5 - pip install package from GitHub to container](#workflow5)\n",
    "- [Custom Container - Workflow 6 - pip install package from Artifact Registry to container](#workflow6)\n",
    "- [Local Script](#script)\n",
    "- [Python Source Distribution](#source)\n",
    "- [Running in Notebook](#notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea5dc64-7bf2-4715-8d24-e1c36c8b7510",
   "metadata": {},
   "source": [
    "---\n",
    "### Common Prep for Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c77230d0-0d5a-4b31-9d50-67f553960a5f",
   "metadata": {},
   "source": [
    "#### Inputs & Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0ffaeaf9-c696-406e-a9f5-40c58b651a27",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Model Training\n",
    "VAR_TARGET = 'Class'\n",
    "VAR_OMIT = 'transaction_id,splits' # add more variables to the string with space delimiters\n",
    "EPOCHS = 10\n",
    "BATCH_SIZE = 100\n",
    "\n",
    "# source data\n",
    "BQ_PROJECT = PROJECT_ID\n",
    "BQ_DATASET = 'fraud'\n",
    "BQ_TABLE = 'fraud_prepped'\n",
    "\n",
    "# Experiment Tracking\n",
    "FRAMEWORK = 'tf'\n",
    "TASK = 'classification'\n",
    "MODEL_TYPE = 'dnn'\n",
    "EXPERIMENT_NAME = f'experiment-{SERIES}-{EXPERIMENT}-{FRAMEWORK}-{TASK}-{MODEL_TYPE}'\n",
    "\n",
    "# Resources\n",
    "TRAIN_COMPUTE = 'n1-standard-4'\n",
    "TRAIN_IMAGE = 'us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-12.py310:latest'\n",
    "REPOSITORY = f\"{REGION}-docker.pkg.dev/{PROJECT_ID}/{PROJECT_ID}-docker\"\n",
    "\n",
    "# parameters\n",
    "BUCKET = PROJECT_ID\n",
    "URI = f\"gs://{GCS_BUCKET}/{SERIES}/{EXPERIMENT}\"\n",
    "DIR = f\"temp/{EXPERIMENT}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e59315b7-cac6-40fc-bbf8-2cb67810cc0c",
   "metadata": {},
   "source": [
    "#### Tensorboard\n",
    "\n",
    "The example test jobs below are based on jobs in the `05 - TensorFlow` series and takes advantage of Vertex AI Experiments and mangaed TensorBoard.  This section creates a TensorBoard instance and gets other inputs for the jobs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "406cbd5a-a373-4123-a318-0be0842495ec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tb = aiplatform.Tensorboard.list(filter=f\"labels.series={SERIES}\")\n",
    "if tb:\n",
    "    tb = tb[0]\n",
    "else: \n",
    "    tb = aiplatform.Tensorboard.create(display_name = SERIES, labels = {'series' : f'{SERIES}'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "38f5b44d-4785-4be3-8562-f18f13b7462e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/tensorboards/8386058747931262976'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tb.resource_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dab266f-484c-4e88-9d06-d87642f742e0",
   "metadata": {},
   "source": [
    "#### Vertex AI Experiments\n",
    "\n",
    "The code in this section initializes the experiment that represents this notebook.  Throughout the notebook sections the model training and evaluation information will be logged to the experiment using as an experiment run using:\n",
    "\n",
    "- [.log_params](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform#google_cloud_aiplatform_log_params)\n",
    "- [.log_metrics](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform#google_cloud_aiplatform_log_metrics)\n",
    "- [.log_time_series_metrics](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform#google_cloud_aiplatform_log_time_series_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bd483fc7-a695-4da7-aefe-593919f31f00",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "aiplatform.init(experiment = EXPERIMENT_NAME, experiment_tensorboard = tb.resource_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae2d3684-d014-4838-8027-823f64c9de48",
   "metadata": {},
   "source": [
    "#### Vertex AI Training Custom Job Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "6eedea69-845f-4ef2-9e85-bf5b3d4672f4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "CMDARGS = [\n",
    "    \"--epochs=\" + str(EPOCHS),\n",
    "    \"--batch_size=\" + str(BATCH_SIZE),\n",
    "    \"--var_target=\" + VAR_TARGET,\n",
    "    \"--var_omit=\" + VAR_OMIT,\n",
    "    \"--project_id=\" + PROJECT_ID,\n",
    "    \"--bq_project=\" + BQ_PROJECT,\n",
    "    \"--bq_dataset=\" + BQ_DATASET,\n",
    "    \"--bq_table=\" + BQ_TABLE,\n",
    "    \"--region=\" + REGION,\n",
    "    \"--experiment=\" + EXPERIMENT,\n",
    "    \"--series=\" + SERIES,\n",
    "    \"--experiment_name=\" + EXPERIMENT_NAME,\n",
    "    \"--run_name=\" #updated by each workflow below\n",
    "]\n",
    "\n",
    "MACHINE_SPEC = {\n",
    "    \"machine_type\": TRAIN_COMPUTE,\n",
    "    \"accelerator_count\": 0\n",
    "}\n",
    "\n",
    "WORKER_POOL_SPEC = [\n",
    "    {\n",
    "        \"replica_count\": 1,\n",
    "        \"machine_spec\": MACHINE_SPEC,\n",
    "        \"container_spec\": {\n",
    "            \"image_uri\": '', # will be filled in below by the workflow\n",
    "            \"command\": [],\n",
    "            \"args\": [] # will be filled in below by the workflow\n",
    "        }\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4ade637-fbb9-4db4-91c2-458d0962bedf",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow1'></a>\n",
    "### Custom Container - Workflow 1 - Copy Script To Container\n",
    "\n",
    "The custom container used here was created by [Python Custom Containers - Workflow 1](./Python%20Custom%20Containers.ipynb#workflow1).\n",
    "\n",
    "> This is a modified version of notebook [05c - Vertex AI Custom Model - TensorFlow - Custom Job With Custom Container](../05%20-%20TensorFlow/05c%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Custom%20Job%20With%20Custom%20Container.ipynb).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1da0214-43a2-4e18-8acd-e419f8955b99",
   "metadata": {},
   "source": [
    "Job Parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5db4558e-6516-4964-bb1f-3839829b0b93",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'replica_count': 1,\n",
       "  'machine_spec': {'machine_type': 'n1-standard-4', 'accelerator_count': 0},\n",
       "  'container_spec': {'image_uri': 'us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_1',\n",
       "   'command': [],\n",
       "   'args': ['--epochs=10',\n",
       "    '--batch_size=100',\n",
       "    '--var_target=Class',\n",
       "    '--var_omit=transaction_id,splits',\n",
       "    '--project_id=statmike-mlops-349915',\n",
       "    '--bq_project=statmike-mlops-349915',\n",
       "    '--bq_dataset=fraud',\n",
       "    '--bq_table=fraud_prepped',\n",
       "    '--region=us-central1',\n",
       "    '--experiment=training',\n",
       "    '--series=tips',\n",
       "    '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       "    '--run_name=run-workflow-1-20231222150338']}}]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_1'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "WORKER_POOL_SPEC[0]['container_spec']['image_uri'] = f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"\n",
    "WORKER_POOL_SPEC[0]['container_spec']['args'] = CMDARGS\n",
    "WORKER_POOL_SPEC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c74b98a1-4949-4f9b-a47e-da25e4c1638f",
   "metadata": {},
   "source": [
    "Define the `aiplatform.CustomJob`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2a2eff52-6ccb-4947-a091-2f1dfffd21b1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "customJob = aiplatform.CustomJob(\n",
    "    display_name = f'{EXPERIMENT}_{SERIES}_{WORKFLOW}_{TIMESTAMP}',\n",
    "    worker_pool_specs = WORKER_POOL_SPEC,\n",
    "    base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    staging_bucket = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    labels = {'series' : f'{SERIES}', 'experiment' : f'{EXPERIMENT}', 'experiment_name' : f'{EXPERIMENT_NAME}', 'run_name' : f'{RUN_NAME}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad604e55-2ee2-4d6b-ac9a-727404b9b81d",
   "metadata": {},
   "source": [
    "Run the job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "79fb8945-0777-4974-865a-6ed7271c7415",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/1026793852137/locations/us-central1/customJobs/6701374593627586560\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/1026793852137/locations/us-central1/customJobs/6701374593627586560')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/6701374593627586560?project=1026793852137\n",
      "View Tensorboard:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+6701374593627586560\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6701374593627586560 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/1026793852137/locations/us-central1/customJobs/6701374593627586560\n"
     ]
    }
   ],
   "source": [
    "customJob.run(\n",
    "    service_account = SERVICE_ACCOUNT,\n",
    "    tensorboard = tb.resource_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd69468b-ec0c-4f81-9d38-ac4b895c95d7",
   "metadata": {},
   "source": [
    "Review the Job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "4975d849-b03a-4491-a953-d34b77379b5c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'training_tips_workflow_1_20231222150338'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e3cb4295-9c91-444b-a788-8124df345174",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/customJobs/6701374593627586560'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.resource_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "adbe81a0-2a0b-41f1-9fda-59fc186fbc80",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Job here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/training/6701374593627586560/cpu?cloudshell=false&project=statmike-mlops-349915\n",
      "Review the TensorBoard From the Job here:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+6701374593627586560\n"
     ]
    }
   ],
   "source": [
    "job_link = f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/training/{customJob.resource_name.split('/')[-1]}/cpu?cloudshell=false&project={PROJECT_ID}\"\n",
    "board_link = f\"https://{REGION}.tensorboard.googleusercontent.com/experiment/{tb.resource_name.replace('/', '+')}+experiments+{customJob.resource_name.split('/')[-1]}\"\n",
    "\n",
    "print(f'Review the Job here:\\n{job_link}')\n",
    "print(f'Review the TensorBoard From the Job here:\\n{board_link}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87d403a4-4967-44d6-9e1c-4fef6109bfbf",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow2'></a>\n",
    "### Custom Container - Workflow 2 - Copy Folder To Container\n",
    "\n",
    "The custom container used here was created by [Python Custom Containers - Workflow 2](./Python%20Custom%20Containers.ipynb#workflow2).\n",
    "\n",
    "> This is a modified version of notebook [05c - Vertex AI Custom Model - TensorFlow - Custom Job With Custom Container](../05%20-%20TensorFlow/05c%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Custom%20Job%20With%20Custom%20Container.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e0f28d7-f190-45d7-9ccd-17e38877752a",
   "metadata": {},
   "source": [
    "Job Parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "58849365-2f46-4836-88ba-77ee12707c79",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'replica_count': 1,\n",
       "  'machine_spec': {'machine_type': 'n1-standard-4', 'accelerator_count': 0},\n",
       "  'container_spec': {'image_uri': 'us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_2',\n",
       "   'command': [],\n",
       "   'args': ['--epochs=10',\n",
       "    '--batch_size=100',\n",
       "    '--var_target=Class',\n",
       "    '--var_omit=transaction_id,splits',\n",
       "    '--project_id=statmike-mlops-349915',\n",
       "    '--bq_project=statmike-mlops-349915',\n",
       "    '--bq_dataset=fraud',\n",
       "    '--bq_table=fraud_prepped',\n",
       "    '--region=us-central1',\n",
       "    '--experiment=training',\n",
       "    '--series=tips',\n",
       "    '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       "    '--run_name=run-workflow-2-20231222151326']}}]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_2'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "WORKER_POOL_SPEC[0]['container_spec']['image_uri'] = f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"\n",
    "WORKER_POOL_SPEC[0]['container_spec']['args'] = CMDARGS\n",
    "\n",
    "WORKER_POOL_SPEC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eef7be2e-ee07-48de-8dc6-74e4a314e60a",
   "metadata": {},
   "source": [
    "Define the `aiplatform.CustomJob`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "4944c6f4-9517-48a9-8517-d6a564271619",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "customJob = aiplatform.CustomJob(\n",
    "    display_name = f'{EXPERIMENT}_{SERIES}_{WORKFLOW}_{TIMESTAMP}',\n",
    "    worker_pool_specs = WORKER_POOL_SPEC,\n",
    "    base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    staging_bucket = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    labels = {'series' : f'{SERIES}', 'experiment' : f'{EXPERIMENT}', 'experiment_name' : f'{EXPERIMENT_NAME}', 'run_name' : f'{RUN_NAME}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48305361-fb4d-4b6f-a90c-b8e89fd006a8",
   "metadata": {},
   "source": [
    "Run the job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e10bbfd4-e144-4c4a-89d7-6aa4a2083636",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/1026793852137/locations/us-central1/customJobs/7323997242111557632\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/1026793852137/locations/us-central1/customJobs/7323997242111557632')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/7323997242111557632?project=1026793852137\n",
      "View Tensorboard:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+7323997242111557632\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/7323997242111557632 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/1026793852137/locations/us-central1/customJobs/7323997242111557632\n"
     ]
    }
   ],
   "source": [
    "customJob.run(\n",
    "    service_account = SERVICE_ACCOUNT,\n",
    "    tensorboard = tb.resource_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7e4339f-14b8-429f-915e-8694da5d94f6",
   "metadata": {},
   "source": [
    "Review the Job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "22bbdd1b-9f23-4f74-845e-8fd6508b5b07",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'training_tips_workflow_2_20231222151326'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "bed9e84f-87fe-4680-9e02-be590f4c2365",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/customJobs/7323997242111557632'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.resource_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "1a0f48d7-cff5-4189-9e74-117fb0dd8587",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Job here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/training/7323997242111557632/cpu?cloudshell=false&project=statmike-mlops-349915\n",
      "Review the TensorBoard From the Job here:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+7323997242111557632\n"
     ]
    }
   ],
   "source": [
    "job_link = f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/training/{customJob.resource_name.split('/')[-1]}/cpu?cloudshell=false&project={PROJECT_ID}\"\n",
    "board_link = f\"https://{REGION}.tensorboard.googleusercontent.com/experiment/{tb.resource_name.replace('/', '+')}+experiments+{customJob.resource_name.split('/')[-1]}\"\n",
    "\n",
    "print(f'Review the Job here:\\n{job_link}')\n",
    "print(f'Review the TensorBoard From the Job here:\\n{board_link}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "263cdffd-282b-44ea-b8fb-ba3542a75728",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow3'></a>\n",
    "### Custom Container - Workflow 3 - Copy Package To Container\n",
    "\n",
    "The custom container used here was created by [Python Custom Containers - Workflow 3](./Python%20Custom%20Containers.ipynb#workflow3).\n",
    "\n",
    "> This is a modified version of notebook [05c - Vertex AI Custom Model - TensorFlow - Custom Job With Custom Container](../05%20-%20TensorFlow/05c%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Custom%20Job%20With%20Custom%20Container.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d81ad1aa-f0eb-4d7b-a540-d553aac982ce",
   "metadata": {},
   "source": [
    "Job Parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "02193b17-2df6-427a-b219-9759a9e25c82",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'replica_count': 1,\n",
       "  'machine_spec': {'machine_type': 'n1-standard-4', 'accelerator_count': 0},\n",
       "  'container_spec': {'image_uri': 'us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_3',\n",
       "   'command': [],\n",
       "   'args': ['--epochs=10',\n",
       "    '--batch_size=100',\n",
       "    '--var_target=Class',\n",
       "    '--var_omit=transaction_id,splits',\n",
       "    '--project_id=statmike-mlops-349915',\n",
       "    '--bq_project=statmike-mlops-349915',\n",
       "    '--bq_dataset=fraud',\n",
       "    '--bq_table=fraud_prepped',\n",
       "    '--region=us-central1',\n",
       "    '--experiment=training',\n",
       "    '--series=tips',\n",
       "    '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       "    '--run_name=run-workflow-3-20231222152234']}}]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_3'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "WORKER_POOL_SPEC[0]['container_spec']['image_uri'] = f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"\n",
    "WORKER_POOL_SPEC[0]['container_spec']['args'] = CMDARGS\n",
    "\n",
    "WORKER_POOL_SPEC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17c833cc-ef4c-44e5-aa41-f845d0d2dd55",
   "metadata": {},
   "source": [
    "Define the `aiplatform.CustomJob`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "fdfe2083-1f1b-48bc-aadb-63a30afd9f1b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "customJob = aiplatform.CustomJob(\n",
    "    display_name = f'{EXPERIMENT}_{SERIES}_{WORKFLOW}_{TIMESTAMP}',\n",
    "    worker_pool_specs = WORKER_POOL_SPEC,\n",
    "    base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    staging_bucket = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    labels = {'series' : f'{SERIES}', 'experiment' : f'{EXPERIMENT}', 'experiment_name' : f'{EXPERIMENT_NAME}', 'run_name' : f'{RUN_NAME}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7347531b-7a66-4192-987c-6f9824e51a9f",
   "metadata": {},
   "source": [
    "Run the job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6c4e2a51-4203-4ea4-83ec-9dab789da1fb",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/1026793852137/locations/us-central1/customJobs/5643591631148941312\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/1026793852137/locations/us-central1/customJobs/5643591631148941312')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/5643591631148941312?project=1026793852137\n",
      "View Tensorboard:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+5643591631148941312\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5643591631148941312 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/1026793852137/locations/us-central1/customJobs/5643591631148941312\n"
     ]
    }
   ],
   "source": [
    "customJob.run(\n",
    "    service_account = SERVICE_ACCOUNT,\n",
    "    tensorboard = tb.resource_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0a070d7-0f7c-4cf7-b2e3-38d3ceaa826d",
   "metadata": {},
   "source": [
    "Review the Job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2792bdc3-38d8-418c-9c28-0a81646445b3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'training_tips_workflow_3_20231222152234'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "d83bfda6-dddd-4186-a408-38ad43f85623",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/customJobs/5643591631148941312'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.resource_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "5470fafb-565e-40ef-8eeb-26a9c138023e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Job here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/training/5643591631148941312/cpu?cloudshell=false&project=statmike-mlops-349915\n",
      "Review the TensorBoard From the Job here:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+5643591631148941312\n"
     ]
    }
   ],
   "source": [
    "job_link = f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/training/{customJob.resource_name.split('/')[-1]}/cpu?cloudshell=false&project={PROJECT_ID}\"\n",
    "board_link = f\"https://{REGION}.tensorboard.googleusercontent.com/experiment/{tb.resource_name.replace('/', '+')}+experiments+{customJob.resource_name.split('/')[-1]}\"\n",
    "\n",
    "print(f'Review the Job here:\\n{job_link}')\n",
    "print(f'Review the TensorBoard From the Job here:\\n{board_link}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49901764-2167-4264-8e8f-9c05e2fd4c1f",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow4'></a>\n",
    "### Custom Container - Workflow 4 - pip install package from GCS to container\n",
    "\n",
    "The custom container used here was created by [Python Custom Containers - Workflow 4](./Python%20Custom%20Containers.ipynb#workflow4).\n",
    "\n",
    "> This is a modified version of notebook [05c - Vertex AI Custom Model - TensorFlow - Custom Job With Custom Container](../05%20-%20TensorFlow/05c%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Custom%20Job%20With%20Custom%20Container.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "966d2c5a-3541-43cc-a8c6-1641745b2cbe",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'replica_count': 1,\n",
       "  'machine_spec': {'machine_type': 'n1-standard-4', 'accelerator_count': 0},\n",
       "  'container_spec': {'image_uri': 'us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_4',\n",
       "   'command': [],\n",
       "   'args': ['--epochs=10',\n",
       "    '--batch_size=100',\n",
       "    '--var_target=Class',\n",
       "    '--var_omit=transaction_id,splits',\n",
       "    '--project_id=statmike-mlops-349915',\n",
       "    '--bq_project=statmike-mlops-349915',\n",
       "    '--bq_dataset=fraud',\n",
       "    '--bq_table=fraud_prepped',\n",
       "    '--region=us-central1',\n",
       "    '--experiment=training',\n",
       "    '--series=tips',\n",
       "    '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       "    '--run_name=run-workflow-4-20231222153207']}}]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_4'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "WORKER_POOL_SPEC[0]['container_spec']['image_uri'] = f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"\n",
    "WORKER_POOL_SPEC[0]['container_spec']['args'] = CMDARGS\n",
    "\n",
    "WORKER_POOL_SPEC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46ecd647-3a8f-46a7-8a55-3887dd1bbf6e",
   "metadata": {},
   "source": [
    "Define the `aiplatform.CustomJob`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "26b41352-eea3-47df-a920-b303a0598015",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "customJob = aiplatform.CustomJob(\n",
    "    display_name = f'{EXPERIMENT}_{SERIES}_{WORKFLOW}_{TIMESTAMP}',\n",
    "    worker_pool_specs = WORKER_POOL_SPEC,\n",
    "    base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    staging_bucket = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    labels = {'series' : f'{SERIES}', 'experiment' : f'{EXPERIMENT}', 'experiment_name' : f'{EXPERIMENT_NAME}', 'run_name' : f'{RUN_NAME}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f13ae502-766d-4284-8f57-f220dab06193",
   "metadata": {},
   "source": [
    "Run the job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "67b484dd-7d86-40fb-9680-53d8cf9eed1b",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/1026793852137/locations/us-central1/customJobs/5738730173277143040\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/1026793852137/locations/us-central1/customJobs/5738730173277143040')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/5738730173277143040?project=1026793852137\n",
      "View Tensorboard:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+5738730173277143040\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/5738730173277143040 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/1026793852137/locations/us-central1/customJobs/5738730173277143040\n"
     ]
    }
   ],
   "source": [
    "customJob.run(\n",
    "    service_account = SERVICE_ACCOUNT,\n",
    "    tensorboard = tb.resource_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89e41075-cebe-49d4-918e-ded0337a0b2a",
   "metadata": {},
   "source": [
    "Review the Job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "6a5e7db9-f925-45b7-9543-c27419910cb0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'training_tips_workflow_4_20231222153207'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "085bc8fd-9ebc-45a3-ae9c-46e5b60a44ac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/customJobs/5738730173277143040'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.resource_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "e9320f8b-5a1f-46b5-8ac8-0179f3c7bfde",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Job here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/training/5738730173277143040/cpu?cloudshell=false&project=statmike-mlops-349915\n",
      "Review the TensorBoard From the Job here:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+5738730173277143040\n"
     ]
    }
   ],
   "source": [
    "job_link = f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/training/{customJob.resource_name.split('/')[-1]}/cpu?cloudshell=false&project={PROJECT_ID}\"\n",
    "board_link = f\"https://{REGION}.tensorboard.googleusercontent.com/experiment/{tb.resource_name.replace('/', '+')}+experiments+{customJob.resource_name.split('/')[-1]}\"\n",
    "\n",
    "print(f'Review the Job here:\\n{job_link}')\n",
    "print(f'Review the TensorBoard From the Job here:\\n{board_link}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f671a51e-db37-42e3-804d-e232602ed108",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow5'></a>\n",
    "### Custom Container - Workflow 5 - pip install package from GitHub to container\n",
    "\n",
    "The custom container used here was created by [Python Custom Containers - Workflow 5](./Python%20Custom%20Containers.ipynb#workflow5).\n",
    "\n",
    "> This is a modified version of notebook [05c - Vertex AI Custom Model - TensorFlow - Custom Job With Custom Container](../05%20-%20TensorFlow/05c%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Custom%20Job%20With%20Custom%20Container.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "bd8dd684-a022-4ace-a950-22ca14085526",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'replica_count': 1,\n",
       "  'machine_spec': {'machine_type': 'n1-standard-4', 'accelerator_count': 0},\n",
       "  'container_spec': {'image_uri': 'us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_5',\n",
       "   'command': [],\n",
       "   'args': ['--epochs=10',\n",
       "    '--batch_size=100',\n",
       "    '--var_target=Class',\n",
       "    '--var_omit=transaction_id,splits',\n",
       "    '--project_id=statmike-mlops-349915',\n",
       "    '--bq_project=statmike-mlops-349915',\n",
       "    '--bq_dataset=fraud',\n",
       "    '--bq_table=fraud_prepped',\n",
       "    '--region=us-central1',\n",
       "    '--experiment=training',\n",
       "    '--series=tips',\n",
       "    '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       "    '--run_name=run-workflow-5-20231222154055']}}]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_5'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "WORKER_POOL_SPEC[0]['container_spec']['image_uri'] = f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"\n",
    "WORKER_POOL_SPEC[0]['container_spec']['args'] = CMDARGS\n",
    "\n",
    "WORKER_POOL_SPEC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35946b4-7b87-4656-9461-ced767d7adec",
   "metadata": {},
   "source": [
    "Define the `aiplatform.CustomJob`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "7016321f-c938-450d-a42f-5c9ac487ddf7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "customJob = aiplatform.CustomJob(\n",
    "    display_name = f'{EXPERIMENT}_{SERIES}_{WORKFLOW}_{TIMESTAMP}',\n",
    "    worker_pool_specs = WORKER_POOL_SPEC,\n",
    "    base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    staging_bucket = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    labels = {'series' : f'{SERIES}', 'experiment' : f'{EXPERIMENT}', 'experiment_name' : f'{EXPERIMENT_NAME}', 'run_name' : f'{RUN_NAME}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78c8ab07-9e5b-4f00-9f97-d656727844ad",
   "metadata": {},
   "source": [
    "Run the job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "182495bc-1262-491d-b43b-138b570328ee",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/1026793852137/locations/us-central1/customJobs/6620872750288338944\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/1026793852137/locations/us-central1/customJobs/6620872750288338944')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/6620872750288338944?project=1026793852137\n",
      "View Tensorboard:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+6620872750288338944\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6620872750288338944 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/1026793852137/locations/us-central1/customJobs/6620872750288338944\n"
     ]
    }
   ],
   "source": [
    "customJob.run(\n",
    "    service_account = SERVICE_ACCOUNT,\n",
    "    tensorboard = tb.resource_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d65e5180-39be-4e1f-8aec-0fd346d9c106",
   "metadata": {},
   "source": [
    "Review the Job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "e20c6cfc-5960-4af5-8de0-3f7770b070d0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'training_tips_workflow_5_20231222154055'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "cac46ede-aa58-4501-b120-ab06a0ab2bb8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/customJobs/6620872750288338944'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.resource_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "e64cf7be-65c3-4c93-8fef-cdef56684add",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Job here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/training/6620872750288338944/cpu?cloudshell=false&project=statmike-mlops-349915\n",
      "Review the TensorBoard From the Job here:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+6620872750288338944\n"
     ]
    }
   ],
   "source": [
    "job_link = f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/training/{customJob.resource_name.split('/')[-1]}/cpu?cloudshell=false&project={PROJECT_ID}\"\n",
    "board_link = f\"https://{REGION}.tensorboard.googleusercontent.com/experiment/{tb.resource_name.replace('/', '+')}+experiments+{customJob.resource_name.split('/')[-1]}\"\n",
    "\n",
    "print(f'Review the Job here:\\n{job_link}')\n",
    "print(f'Review the TensorBoard From the Job here:\\n{board_link}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9299aaed-5db7-4246-b130-a858176aaab6",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'workflow6'></a>\n",
    "### Custom Container - Workflow 6 - pip install package from Artifact Registry to container\n",
    "\n",
    "The custom container used here was created by [Python Custom Containers - Workflow 6](./Python%20Custom%20Containers.ipynb#workflow6).\n",
    "\n",
    "> This is a modified version of notebook [05c - Vertex AI Custom Model - TensorFlow - Custom Job With Custom Container](../05%20-%20TensorFlow/05c%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Custom%20Job%20With%20Custom%20Container.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "f98735a4-d308-485c-992b-57764e1203f6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'replica_count': 1,\n",
       "  'machine_spec': {'machine_type': 'n1-standard-4', 'accelerator_count': 0},\n",
       "  'container_spec': {'image_uri': 'us-central1-docker.pkg.dev/statmike-mlops-349915/statmike-mlops-349915-docker/tips_trainer_workflow_6',\n",
       "   'command': [],\n",
       "   'args': ['--epochs=10',\n",
       "    '--batch_size=100',\n",
       "    '--var_target=Class',\n",
       "    '--var_omit=transaction_id,splits',\n",
       "    '--project_id=statmike-mlops-349915',\n",
       "    '--bq_project=statmike-mlops-349915',\n",
       "    '--bq_dataset=fraud',\n",
       "    '--bq_table=fraud_prepped',\n",
       "    '--region=us-central1',\n",
       "    '--experiment=training',\n",
       "    '--series=tips',\n",
       "    '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       "    '--run_name=run-workflow-6-20231222155043']}}]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_6'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "WORKER_POOL_SPEC[0]['container_spec']['image_uri'] = f\"{REPOSITORY}/tips_trainer_{WORKFLOW}\"\n",
    "WORKER_POOL_SPEC[0]['container_spec']['args'] = CMDARGS\n",
    "\n",
    "WORKER_POOL_SPEC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58a86481-0665-4cdf-ae60-88c61ab655d3",
   "metadata": {},
   "source": [
    "Define the `aiplatform.CustomJob`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "f89e4615-f3c6-49f5-a0bb-03c9ec41dc2c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "customJob = aiplatform.CustomJob(\n",
    "    display_name = f'{EXPERIMENT}_{SERIES}_{WORKFLOW}_{TIMESTAMP}',\n",
    "    worker_pool_specs = WORKER_POOL_SPEC,\n",
    "    base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    staging_bucket = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    labels = {'series' : f'{SERIES}', 'experiment' : f'{EXPERIMENT}', 'experiment_name' : f'{EXPERIMENT_NAME}', 'run_name' : f'{RUN_NAME}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e8378f0-b465-44e0-bde8-e52c043ce1a6",
   "metadata": {},
   "source": [
    "Run the job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "5858449f-b4a2-4a83-9986-c79a6a10eb11",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/1026793852137/locations/us-central1/customJobs/3664822544873029632\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/1026793852137/locations/us-central1/customJobs/3664822544873029632')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/3664822544873029632?project=1026793852137\n",
      "View Tensorboard:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+3664822544873029632\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/3664822544873029632 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/1026793852137/locations/us-central1/customJobs/3664822544873029632\n"
     ]
    }
   ],
   "source": [
    "customJob.run(\n",
    "    service_account = SERVICE_ACCOUNT,\n",
    "    tensorboard = tb.resource_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c3063de-7084-4ee4-8440-4f9271055031",
   "metadata": {},
   "source": [
    "Review the Job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "4e86e061-864b-4f62-9be4-41dde7685dfb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'training_tips_workflow_6_20231222155043'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.display_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "42817d67-430d-4d81-b54c-5fbf7b830490",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/1026793852137/locations/us-central1/customJobs/3664822544873029632'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "customJob.resource_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "34f37b74-090d-47de-8117-820fd5cf43ac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Job here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/training/3664822544873029632/cpu?cloudshell=false&project=statmike-mlops-349915\n",
      "Review the TensorBoard From the Job here:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+3664822544873029632\n"
     ]
    }
   ],
   "source": [
    "job_link = f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/training/{customJob.resource_name.split('/')[-1]}/cpu?cloudshell=false&project={PROJECT_ID}\"\n",
    "board_link = f\"https://{REGION}.tensorboard.googleusercontent.com/experiment/{tb.resource_name.replace('/', '+')}+experiments+{customJob.resource_name.split('/')[-1]}\"\n",
    "\n",
    "print(f'Review the Job here:\\n{job_link}')\n",
    "print(f'Review the TensorBoard From the Job here:\\n{board_link}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82c7fb2e-b4b3-43c2-ad1d-1e39ab791b7c",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'script'></a>\n",
    "### Local Script\n",
    "\n",
    "Run a single file training script with `aiplatform.CustomJob.from_local_script()`\n",
    "\n",
    "Notes:\n",
    "- This uses a single file `filename.py` from the local directory, not a GCS URI\n",
    "- When you run `aiplatform.CustomJob.from_local_script()` it responds with a message confirming the local script was copied to the GCS URI provide in the parameter `staging_bucket = `.\n",
    "\n",
    "This is a modified version of notebook [05a - Vertex AI Custom Model - TensorFlow - Custom Job With Python File](../05%20-%20TensorFlow/05a%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Custom%20Job%20With%20Python%20File.ipynb) that uses the local script for this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "6e8408f0-e716-4e51-903b-c6a82a6ca080",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['--epochs=10',\n",
       " '--batch_size=100',\n",
       " '--var_target=Class',\n",
       " '--var_omit=transaction_id,splits',\n",
       " '--project_id=statmike-mlops-349915',\n",
       " '--bq_project=statmike-mlops-349915',\n",
       " '--bq_dataset=fraud',\n",
       " '--bq_table=fraud_prepped',\n",
       " '--region=us-central1',\n",
       " '--experiment=training',\n",
       " '--series=tips',\n",
       " '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       " '--run_name=run-workflow-script-20231222164919']"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_script'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "CMDARGS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "d31af555-c7d1-404a-8870-e0bf1e2a34e7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training script copied to:\n",
      "gs://statmike-mlops-349915/tips/training/workflow_script/20231222164919/aiplatform-2023-12-22-16:51:05.462-aiplatform_custom_trainer_script-0.1.tar.gz.\n"
     ]
    }
   ],
   "source": [
    "customJob = aiplatform.CustomJob.from_local_script(\n",
    "    display_name = f'{EXPERIMENT}_{SERIES}_{WORKFLOW}_{TIMESTAMP}',\n",
    "    script_path = f\"./code/packages/tips_trainer/src/tips_trainer/train.py\",\n",
    "    container_uri = TRAIN_IMAGE,\n",
    "    args = CMDARGS,\n",
    "    requirements = ['tensorflow_io', f'google-cloud-aiplatform>={aiplatform.__version__}', 'db-dtypes', f\"protobuf>={importlib.metadata.version('protobuf')}\"],\n",
    "    replica_count = 1,\n",
    "    machine_type = TRAIN_COMPUTE,\n",
    "    accelerator_count = 0,\n",
    "    base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    staging_bucket = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    labels = {'series' : f'{SERIES}', 'experiment' : f'{EXPERIMENT}', 'experiment_name' : f'{EXPERIMENT_NAME}', 'run_name' : f'{RUN_NAME}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "e5daa8ad-c8c1-4fad-a593-dbf9167a9153",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/1026793852137/locations/us-central1/customJobs/6200067660105908224\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/1026793852137/locations/us-central1/customJobs/6200067660105908224')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/6200067660105908224?project=1026793852137\n",
      "View Tensorboard:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+6200067660105908224\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_QUEUED\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/6200067660105908224 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/1026793852137/locations/us-central1/customJobs/6200067660105908224\n"
     ]
    }
   ],
   "source": [
    "customJob.run(\n",
    "    service_account = SERVICE_ACCOUNT,\n",
    "    tensorboard = tb.resource_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "6e21365d-dc49-4a8f-baaa-12931dde6ffd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Job here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/training/6200067660105908224/cpu?cloudshell=false&project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "job_link = f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/training/{customJob.resource_name.split('/')[-1]}/cpu?cloudshell=false&project={PROJECT_ID}\"\n",
    "print(f'Review the Job here:\\n{job_link}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "6d71e99f-b767-4526-8a07-1a714f7140c6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the model output here:\n",
      "https://console.cloud.google.com/storage/browser/statmike-mlops-349915/tips/training/workflow_script/20231222164919?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f'Review the model output here:\\nhttps://console.cloud.google.com/storage/browser/{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/{WORKFLOW}/{TIMESTAMP}?project={PROJECT_ID}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d037b7a7-ea95-423f-8329-2559ba147d36",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'source'></a>\n",
    "### Python Source Distribution\n",
    "\n",
    "Use the Python Source Distribution to run with `aiplatform.CustomJob(..., worker_pool_specs = )` by specifying `python_package_spec = ` in the `worker_pool_specs`.\n",
    "\n",
    "Notes:\n",
    "- This uses a Python Source Distribution which is in the format `.tar.gz`, a compressed tarball\n",
    "- This project has a prepared Python Source Distribution in `./code/tips_trainer/dist/` which was created by [Python Packages](./Python%20Packages.ipynb)\n",
    "- The `python_package_spec` parameter of the `worker_pool_specs` has subparameter `package_uris` which allows a list of up to 100 source distributions.  These must be provided as GCS URIs like `gs://bucketname/path_to_file.tar.gz`\n",
    "\n",
    "This is a modified version of notebook [05b - Vertex AI Custom Model - TensorFlow - Custom Job With Python Source Distribution](../05%20-%20TensorFlow/05b%20-%20Vertex%20AI%20Custom%20Model%20-%20TensorFlow%20-%20Custom%20Job%20With%20Python%20Source%20Distribution.ipynb) that uses the source distribution stored in GCS for this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "c3393fdc-3327-4657-a187-775430608e71",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'replica_count': 1,\n",
       "  'machine_spec': {'machine_type': 'n1-standard-4', 'accelerator_count': 0},\n",
       "  'python_package_spec': {'executor_image_uri': 'us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-12.py310:latest',\n",
       "   'package_uris': ['gs://statmike-mlops-349915/tips/code/packages/tips_trainer/dist/tips_trainer-0.1.tar.gz'],\n",
       "   'python_module': 'tips_trainer.train',\n",
       "   'args': ['--epochs=10',\n",
       "    '--batch_size=100',\n",
       "    '--var_target=Class',\n",
       "    '--var_omit=transaction_id,splits',\n",
       "    '--project_id=statmike-mlops-349915',\n",
       "    '--bq_project=statmike-mlops-349915',\n",
       "    '--bq_dataset=fraud',\n",
       "    '--bq_table=fraud_prepped',\n",
       "    '--region=us-central1',\n",
       "    '--experiment=training',\n",
       "    '--series=tips',\n",
       "    '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       "    '--run_name=run-workflow-source-20231222170120']}}]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_source'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "\n",
    "# remove container_spec and replace with python_package_spec:\n",
    "SOURCE_WORKER_POOL_SPEC = WORKER_POOL_SPEC\n",
    "SOURCE_WORKER_POOL_SPEC[0].pop('container_spec', None)\n",
    "SOURCE_WORKER_POOL_SPEC[0]['python_package_spec'] = {\n",
    "            \"executor_image_uri\": TRAIN_IMAGE,\n",
    "            \"package_uris\": [f\"gs://{GCS_BUCKET}/{SERIES}/code/packages/tips_trainer/dist/tips_trainer-0.1.tar.gz\"],\n",
    "            \"python_module\": \"tips_trainer.train\",\n",
    "            \"args\": CMDARGS\n",
    "}\n",
    "\n",
    "SOURCE_WORKER_POOL_SPEC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "571c1e2a-23dd-4bba-b0bb-87074a39c179",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "customJob = aiplatform.CustomJob(\n",
    "    display_name = f'{EXPERIMENT}_{SERIES}_{WORKFLOW}_{TIMESTAMP}',\n",
    "    worker_pool_specs = SOURCE_WORKER_POOL_SPEC,\n",
    "    base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    staging_bucket = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\",\n",
    "    labels = {'series' : f'{SERIES}', 'experiment' : f'{EXPERIMENT}', 'experiment_name' : f'{EXPERIMENT_NAME}', 'run_name' : f'{RUN_NAME}'}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "9c9dd6ef-171b-45c3-9a9b-f0f137df330b",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating CustomJob\n",
      "CustomJob created. Resource name: projects/1026793852137/locations/us-central1/customJobs/1635669437765910528\n",
      "To use this CustomJob in another session:\n",
      "custom_job = aiplatform.CustomJob.get('projects/1026793852137/locations/us-central1/customJobs/1635669437765910528')\n",
      "View Custom Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/training/1635669437765910528?project=1026793852137\n",
      "View Tensorboard:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+1635669437765910528\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_PENDING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_RUNNING\n",
      "CustomJob projects/1026793852137/locations/us-central1/customJobs/1635669437765910528 current state:\n",
      "JobState.JOB_STATE_SUCCEEDED\n",
      "CustomJob run completed. Resource name: projects/1026793852137/locations/us-central1/customJobs/1635669437765910528\n"
     ]
    }
   ],
   "source": [
    "customJob.run(\n",
    "    service_account = SERVICE_ACCOUNT,\n",
    "    tensorboard = tb.resource_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "dfbbe9a9-bbd2-48b6-af33-304d242c49b1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Job here:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/training/1635669437765910528/cpu?cloudshell=false&project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "job_link = f\"https://console.cloud.google.com/vertex-ai/locations/{REGION}/training/{customJob.resource_name.split('/')[-1]}/cpu?cloudshell=false&project={PROJECT_ID}\"\n",
    "print(f'Review the Job here:\\n{job_link}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "0ca6c61f-c761-44f7-ad95-03f1fc377aa0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the model output here:\n",
      "https://console.cloud.google.com/storage/browser/statmike-mlops-349915/tips/training/workflow_source/20231222170120?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f'Review the model output here:\\nhttps://console.cloud.google.com/storage/browser/{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/{WORKFLOW}/{TIMESTAMP}?project={PROJECT_ID}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fba3947-8032-4973-87af-fade2070c7e7",
   "metadata": {},
   "source": [
    "---\n",
    "<a id = 'notebook'></a>\n",
    "## Notebook - local code\n",
    "\n",
    "\n",
    "Use the training script in the local notebook environment.  While the script is authored and packaged for running in a Vertex AI Training Custom Job it can also be used locally.  This is helpful for testing.  \n",
    "\n",
    "The job will be launched like any Python job with `python -m tips_trainer.train <list_of_args_here>`.  Choices for making the `tips_trainer.train` module/file/script available are:\n",
    "- pip install from Artifact Registry: `pip install --index-url https://{REGION}-python.pkg.dev/{PROJECT_ID}/{PROJECT_ID}-python/simple tips-trainer`\n",
    "- pip install from local directory: `pip install /code/packages/tips_trainer/dist/*.whl`\n",
    "- pip install from GitHub: `pip install https://github.com/statmike/vertex-ai-mlops/blob/main/Tips/code/packages/tips_trainer/dist/tips_trainer-0.1-py3-none-any.whl?raw=true`\n",
    "- run from local directory: `./code/packages/tips_trainer/src` or copy to directory of choice\n",
    "\n",
    "Notes:\n",
    "- Vetex AI Training Jobs set [environment variables for Cloud Storage locations](https://cloud.google.com/vertex-ai/docs/training/code-requirements#environment-variables).  Since this example is running in the training code in the local notebook instance rather than in a custom job, these will need to be set manually.\n",
    "    - `AIP_MODEL_DIR` - this extends `base_output_directory` with `/model`\n",
    "    - `AIP_TENSORBOARD_LOG_DIR` - this extends `base_output_directory` with `/log`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "c4addd07-75bc-49b7-8df2-cb3ec54b93df",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['--epochs=10',\n",
       " '--batch_size=100',\n",
       " '--var_target=Class',\n",
       " '--var_omit=transaction_id,splits',\n",
       " '--project_id=statmike-mlops-349915',\n",
       " '--bq_project=statmike-mlops-349915',\n",
       " '--bq_dataset=fraud',\n",
       " '--bq_table=fraud_prepped',\n",
       " '--region=us-central1',\n",
       " '--experiment=training',\n",
       " '--series=tips',\n",
       " '--experiment_name=experiment-tips-training-tf-classification-dnn',\n",
       " '--run_name=run-workflow-nb-local-20231222171048']"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WORKFLOW = 'workflow_nb_local'\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "RUN_NAME = f\"run-{WORKFLOW.replace('_', '-')}-{TIMESTAMP}\"\n",
    "\n",
    "CMDARGS[-1] = \"--run_name=\" + RUN_NAME\n",
    "CMDARGS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85011674-39bb-42f9-a427-2808dadceb8e",
   "metadata": {},
   "source": [
    "Set environment variable the code expects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "1ecd0031-3802-45c4-b366-222d76b173ca",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_output_dir = f\"{URI}/{WORKFLOW}/{TIMESTAMP}\"\n",
    "\n",
    "os.environ[\"AIP_MODEL_DIR\"] = base_output_dir + '/model'\n",
    "os.environ[\"AIP_TENSORBOARD_LOG_DIR\"] = base_output_dir + '/logs'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "d00db4b3-37b8-4010-8b1f-69b2b4b8e106",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://statmike-mlops-349915/tips/training/workflow_nb_local/20231222171048/model\n",
      "gs://statmike-mlops-349915/tips/training/workflow_nb_local/20231222171048/logs\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo $AIP_MODEL_DIR\n",
    "echo $AIP_TENSORBOARD_LOG_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02ae24dc-d7d6-4820-8a4c-279b391b28fd",
   "metadata": {},
   "source": [
    "Run the training code locally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "c69b69ff-c744-49c8-919b-c0f8b826ea16",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-12-22 17:10:56.202103: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2023-12-22 17:10:58.884932: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2023-12-22 17:10:58.885089: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2023-12-22 17:10:59.118144: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-12-22 17:10:59.924073: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2023-12-22 17:10:59.926912: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "Associating projects/1026793852137/locations/us-central1/metadataStores/default/contexts/experiment-tips-training-tf-classification-dnn-run-workflow-nb-local-20231222171048 to Experiment: experiment-tips-training-tf-classification-dnn\n",
      "2023-12-22 17:11:37.965516: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA\n",
      "2023-12-22 17:11:38.939459: W tensorflow_io/core/kernels/audio_video_mp3_kernels.cc:271] libmp3lame.so.0 or lame functions are not available\n",
      "Epoch 1/10\n",
      "2023-12-22 17:11:44.301115: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:11:44.301165: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "   2269/Unknown - 17s 6ms/step - loss: 0.0804 - accuracy: 0.9800 - auprc: 0.99332023-12-22 17:12:01.181151: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:12:01.181203: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 21s 8ms/step - loss: 0.0801 - accuracy: 0.9801 - auprc: 0.9933 - val_loss: 0.0239 - val_accuracy: 0.9977 - val_auprc: 0.9969\n",
      "Epoch 2/10\n",
      "2269/2281 [============================>.] - ETA: 0s - loss: 0.0137 - accuracy: 0.9983 - auprc: 0.99892023-12-22 17:12:19.194887: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:12:19.194931: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 17s 8ms/step - loss: 0.0137 - accuracy: 0.9983 - auprc: 0.9989 - val_loss: 0.0095 - val_accuracy: 0.9988 - val_auprc: 0.9992\n",
      "Epoch 3/10\n",
      "2275/2281 [============================>.] - ETA: 0s - loss: 0.0078 - accuracy: 0.9988 - auprc: 0.99962023-12-22 17:12:36.187204: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:12:36.187260: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 17s 7ms/step - loss: 0.0078 - accuracy: 0.9988 - auprc: 0.9996 - val_loss: 0.0076 - val_accuracy: 0.9990 - val_auprc: 0.9992\n",
      "Epoch 4/10\n",
      "2269/2281 [============================>.] - ETA: 0s - loss: 0.0064 - accuracy: 0.9989 - auprc: 0.99962023-12-22 17:12:53.067426: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:12:53.067492: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 17s 7ms/step - loss: 0.0064 - accuracy: 0.9989 - auprc: 0.9996 - val_loss: 0.0068 - val_accuracy: 0.9990 - val_auprc: 0.9994\n",
      "Epoch 5/10\n",
      "2274/2281 [============================>.] - ETA: 0s - loss: 0.0058 - accuracy: 0.9991 - auprc: 0.99952023-12-22 17:13:10.692216: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:13:10.692274: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 18s 8ms/step - loss: 0.0058 - accuracy: 0.9991 - auprc: 0.9995 - val_loss: 0.0063 - val_accuracy: 0.9991 - val_auprc: 0.9994\n",
      "Epoch 6/10\n",
      "2269/2281 [============================>.] - ETA: 0s - loss: 0.0054 - accuracy: 0.9991 - auprc: 0.99962023-12-22 17:13:27.779195: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:13:27.779255: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 17s 7ms/step - loss: 0.0053 - accuracy: 0.9991 - auprc: 0.9996 - val_loss: 0.0060 - val_accuracy: 0.9992 - val_auprc: 0.9994\n",
      "Epoch 7/10\n",
      "2270/2281 [============================>.] - ETA: 0s - loss: 0.0050 - accuracy: 0.9991 - auprc: 0.99962023-12-22 17:13:45.343421: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:13:45.343472: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 18s 8ms/step - loss: 0.0050 - accuracy: 0.9991 - auprc: 0.9996 - val_loss: 0.0059 - val_accuracy: 0.9992 - val_auprc: 0.9994\n",
      "Epoch 8/10\n",
      "2271/2281 [============================>.] - ETA: 0s - loss: 0.0048 - accuracy: 0.9992 - auprc: 0.99962023-12-22 17:14:04.860852: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:14:04.860910: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 19s 8ms/step - loss: 0.0048 - accuracy: 0.9992 - auprc: 0.9996 - val_loss: 0.0058 - val_accuracy: 0.9992 - val_auprc: 0.9995\n",
      "Epoch 9/10\n",
      "2269/2281 [============================>.] - ETA: 0s - loss: 0.0048 - accuracy: 0.9992 - auprc: 0.99962023-12-22 17:14:23.304328: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:14:23.304387: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 18s 8ms/step - loss: 0.0048 - accuracy: 0.9992 - auprc: 0.9996 - val_loss: 0.0056 - val_accuracy: 0.9992 - val_auprc: 0.9995\n",
      "Epoch 10/10\n",
      "2274/2281 [============================>.] - ETA: 0s - loss: 0.0045 - accuracy: 0.9992 - auprc: 0.99962023-12-22 17:14:42.638414: E tensorflow/core/framework/dataset.cc:769] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2023-12-22 17:14:42.638473: E tensorflow/core/framework/dataset.cc:773] UNIMPLEMENTED: Cannot compute input sources for dataset of type IO>BigQueryDataset, because the dataset does not implement `InputDatasets`. To fix this, your dataset should override the `InputDatasets` method. If it is a source dataset, it should return empty inputs.\n",
      "2281/2281 [==============================] - 19s 8ms/step - loss: 0.0045 - accuracy: 0.9992 - auprc: 0.9996 - val_loss: 0.0055 - val_accuracy: 0.9992 - val_auprc: 0.9995\n",
      "286/286 [==============================] - 2s 5ms/step - loss: 0.0053 - accuracy: 0.9993 - auprc: 0.9994\n",
      "283/283 [==============================] - 2s 5ms/step - loss: 0.0055 - accuracy: 0.9992 - auprc: 0.9995\n",
      "2281/2281 [==============================] - 20s 8ms/step - loss: 0.0063 - accuracy: 0.9992 - auprc: 0.9994\n"
     ]
    }
   ],
   "source": [
    "!cd ./code/packages/tips_trainer/src && python -m tips_trainer.train {(' ').join(CMDARGS)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "f9c751b8-953d-41bd-87bc-0a7767cb8231",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the Cloud Storage contents for this job here:\n",
      "https://console.cloud.google.com/storage/browser/statmike-mlops-349915/tips/training/workflow_nb_local/20231222171048?project=statmike-mlops-349915\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the Cloud Storage contents for this job here:\\nhttps://console.cloud.google.com/storage/browser/{GCS_BUCKET}/{SERIES}/{EXPERIMENT}/{WORKFLOW}/{TIMESTAMP}?project={PROJECT_ID}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "44442c8f-fac9-4051-9c34-020f4ae233f7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review the TensorBoard for this Experiment here:\n",
      "https://us-central1.tensorboard.googleusercontent.com/experiment/projects+1026793852137+locations+us-central1+tensorboards+8386058747931262976+experiments+experiment-tips-training-tf-classification-dnn\n"
     ]
    }
   ],
   "source": [
    "print(f\"Review the TensorBoard for this Experiment here:\\nhttps://{REGION}.tensorboard.googleusercontent.com/experiment/{tb.resource_name.replace('/', '+')}+experiments+{EXPERIMENT_NAME}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84877e94-f09a-4c20-9274-d3599ee0f974",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "conda-root-py",
   "name": "workbench-notebooks.m113",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/workbench-notebooks:m113"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel) (Local)",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}