Merge pull request #55 from WallarooLabs/wallaroo_llm_managed_endpoints

Author: johnhansarickWallaroo

.gitignore

@@ -37,3 +37,4 @@ resnet50_v1.onnx
 wallaroo-llms/llamav2/models/llama_byop_llamav2.zip
 wallaroo-llms/vector-database-embedding-with-ml-orchestrations/models/byop_bge_base2.zip
 wallaroo-llms/vector-database-embedding-with-ml-orchestrations/vector_db_orch.zip
+clip-vit-base-patch-32.zip

wallaroo-llms/llm-managed-inference-endpoint/llm-managed-inference-endpoint-llama-vertex/managed-inference-endpoint-vertex.ipynb

@@ -0,0 +1,351 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "22ad614e-2e4c-4635-a167-2b97c3f041af",
+   "metadata": {},
+   "source": [
+    "This tutorial and the assets can be downloaded as part of the [Wallaroo Tutorials repository](https://github.com/WallarooLabs/Wallaroo_Tutorials/blob/main/wallaroo-llms/).\n",
+    "\n",
+    "## Wallaroo Deployment of Managed Inference Endpoint Models with Google Vertex\n",
+    "\n",
+    "The following tutorial demonstrates uploading, deploying, inferencing and monitoring a [LLM with Managed Inference Endpoints](https://staging.docs.wallaroo.ai/wallaroo-llm/wallaroo-llm-package-deployment/wallaroo-llm-monitoring-external-endpoints/).\n",
+    "\n",
+    "These models leverage LLMs deployed in other services, with Wallaroo providing a single source for inference requests, logging results, monitoring for hate/abuse/racism and other factors, and tracking model drift through Wallaroo assays.\n",
+    "\n",
+    "## Provided Models\n",
+    "\n",
+    "The following models are provided:\n",
+    "\n",
+    "* `byop_llama2_vertex_v2_9.zip`: A [Wallaroo BYOP](https://docs.wallaroo.ai/wallaroo-developer-guides/wallaroo-sdk-guides/wallaroo-sdk-essentials-guide/wallaroo-sdk-model-uploads/wallaroo-sdk-model-arbitrary-python/) model that uses Google Vertex as a Managed Inference Endpoint.\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "This tutorial requires:\n",
+    "\n",
+    "* Wallaroo 2024.1 and above\n",
+    "* Credentials for authenticating to Google Vertex"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "314c3dd4",
+   "metadata": {},
+   "source": [
+    "## Tutorial Steps\n",
+    "\n",
+    "### Import Library\n",
+    "\n",
+    "The following libraries are used to upload and perform inferences on the LLM with Managed Inference Endpoints."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4e18447c-59ca-41fc-ae5f-0435849d30fd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "\n",
+    "import wallaroo\n",
+    "from wallaroo.pipeline   import Pipeline\n",
+    "from wallaroo.deployment_config import DeploymentConfigBuilder\n",
+    "from wallaroo.framework import Framework\n",
+    "from wallaroo.engine_config import Architecture\n",
+    "\n",
+    "import pyarrow as pa\n",
+    "import numpy as np\n",
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9654ee15",
+   "metadata": {},
+   "source": [
+    "### Connect to the Wallaroo Instance\n",
+    "\n",
+    "A connection to Wallaroo is opened through the Wallaroo SDK client.  The Python library is included in the Wallaroo install and available through the Jupyter Hub interface provided with your Wallaroo environment.\n",
+    "\n",
+    "This is accomplished using the `wallaroo.Client()` command, which provides a URL to grant the SDK permission to your specific Wallaroo environment.  When displayed, enter the URL into a browser and confirm permissions.  Store the connection into a variable that can be referenced later.\n",
+    "\n",
+    "If logging into the Wallaroo instance through the internal JupyterHub service, use `wl = wallaroo.Client()`.  For more information on Wallaroo Client settings, see the [Client Connection guide](https://docs.wallaroo.ai/wallaroo-developer-guides/wallaroo-sdk-guides/wallaroo-sdk-essentials-guide/wallaroo-sdk-essentials-client/).\n",
+    "\n",
+    "The `request_timeout` flag is used for Wallaroo BYOP models where the file size may require additional time to complete the upload process."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "00ea4e3f-c993-4a19-9d11-cb4c1ce300ab",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "wl = wallaroo.Client()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "25729b33-c6de-4fdc-9467-222c2bac820c",
+   "metadata": {},
+   "source": [
+    "### LLM with Managed Inference Endpoint Model Code\n",
+    "\n",
+    "The Wallaroo BYOP model `byop_llamav2_vertex_v2_9.zip` contains the following artifacts:\n",
+    "\n",
+    "* `main.py`: Python script that controls the behavior of the model.\n",
+    "* `requirements.txt`:  Python requirements file that sets the Python libraries used.\n",
+    "\n",
+    "The model performs the following.\n",
+    "\n",
+    "1. Accepts the inference request from the requester.\n",
+    "2. Load the credentials to the Google Vertex session from the provided environmental variables.  These are supplied during the [Set Deployment Configuration](#set-deployment-configuration) step.  The following code shows this process.\n",
+    "\n",
+    "    ```python\n",
+    "    credentials = Credentials.from_service_account_info(\n",
+    "          json.loads(os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"].replace(\"'\", '\"')),\n",
+    "          scopes=[\"https://www.googleapis.com/auth/cloud-platform\"],\n",
+    "      )\n",
+    "    ```\n",
+    "\n",
+    "3. Take the inference request, connect to Google and submit the request to the deployed LLM.  The inference result is returned to the BYOP model, which is then returned.\n",
+    "\n",
+    "    ```python\n",
+    "    def _predict(self, input_data: InferenceData):\n",
+    "        credentials.refresh(Request())\n",
+    "        token = credentials.token\n",
+    "\n",
+    "        headers = {\n",
+    "            \"Authorization\": f\"Bearer {token}\",\n",
+    "            \"Content-Type\": \"application/json\",\n",
+    "        }\n",
+    "        prompts = input_data[\"text\"].tolist()\n",
+    "        instances = [{\"prompt\": prompt, \"max_tokens\": 200} for prompt in prompts]\n",
+    "\n",
+    "        response = requests.post(\n",
+    "            f\"{self.model}\",\n",
+    "            json={\"instances\": instances},\n",
+    "            headers=headers,\n",
+    "        )\n",
+    "\n",
+    "        predictions = response.json()\n",
+    "\n",
+    "        if isinstance(predictions[\"predictions\"], str):\n",
+    "            generated_text = [\n",
+    "                prediction.split(\"Output:\\n\")[-1]\n",
+    "                for prediction in predictions[\"predictions\"]\n",
+    "            ]\n",
+    "        else:\n",
+    "            generated_text = [\n",
+    "                prediction[\"predictions\"][0].split(\"Output:\\n\")[-1]\n",
+    "                for prediction in predictions[\"predictions\"]\n",
+    "            ]\n",
+    "\n",
+    "        return {\"generated_text\": np.array(generated_text)}\n",
+    "    ```\n",
+    "\n",
+    "This model is contained in a Wallaroo pipeline which accepts the inference request, then returns the final result back to the requester."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0f2597e1",
+   "metadata": {},
+   "source": [
+    "### Upload LLM with Managed Inference Endpoint Model\n",
+    "\n",
+    "Uploading models uses the Wallaroo Client `upload_model` method, which takes the following parameters:\n",
+    "\n",
+    "| Parameter | Type | Description |\n",
+    "|---|---|---|\n",
+    "| `name` | `string` (*Required*) | The name of the model.  Model names are unique **per workspace**.  Models that are uploaded with the same name are assigned as a new **version** of the model. |\n",
+    "| `path` | `string` (*Required*) | The path to the model file being uploaded. |\n",
+    "| `framework` |`string` (*Required*) | The framework of the model from `wallaroo.framework`. |\n",
+    "| `input_schema` | `pyarrow.lib.Schema` (*Required*) | The input schema in Apache Arrow schema format. |\n",
+    "| `output_schema` | `pyarrow.lib.Schema` (*Required*) | The output schema in Apache Arrow schema format. |\n",
+    "\n",
+    "The following shows the upload parameters for the `byop_llama2_vertex_v2_9.zip` Wallaroo BYOP model with the following input and output schema:\n",
+    "\n",
+    "* Input:\n",
+    "  * `text` (*String*):  The input text.\n",
+    "* Output:\n",
+    "  * `generated_text` (*String*): The result returned from the GPT 3.5 model as a Managed Inference Endpoint.\n",
+    "\n",
+    "The uploaded model reference is saved to the variable `model`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d983d3f0-c19d-44e1-8c05-62200f3e0854",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_schema = pa.schema([\n",
+    "    pa.field(\"text\", pa.string()),\n",
+    "])\n",
+    "\n",
+    "output_schema = pa.schema([\n",
+    "    pa.field(\"generated_text\", pa.string())\n",
+    "])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2cba287f-ad6f-4c39-a10e-e4db47404c89",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = wl.upload_model('byop-llama-vertex-v1', \n",
+    "    './models/byop_llama2_vertex_v2_9.zip',\n",
+    "    framework=Framework.CUSTOM,\n",
+    "    input_schema=input_schema,\n",
+    "    output_schema=output_schema,\n",
+    ")\n",
+    "model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "91e052ce-5407-4995-aaf6-e1411a838cf8",
+   "metadata": {},
+   "source": [
+    "### Set Deployment Configuration\n",
+    "\n",
+    "The deployment configuration sets the resources assigned to the LLM with Managed Inference Endpoint.  For this example, following resources are applied.\n",
+    "\n",
+    "* `byop_llama2_vertex_v2_9.zip`: 2 cpus, 1 Gi RAM, plus the environmental variable `GOOGLE_APPLICATION_CREDENTIALS` loaded from the file `credentials.json`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2d8e3996-734c-4f7f-8922-5df049fd28c0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "deployment_config = DeploymentConfigBuilder() \\\n",
+    "    .cpus(1).memory('2Gi') \\\n",
+    "    .sidekick_cpus(model, 2) \\\n",
+    "    .sidekick_memory(model, '1Gi') \\\n",
+    "    .sidekick_env(model, {\"GOOGLE_APPLICATION_CREDENTIALS\": str(json.load(open(\"credentials.json\", 'r')))}) \\\n",
+    "    .build()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9151986",
+   "metadata": {},
+   "source": [
+    "### Deploy Model\n",
+    "\n",
+    "To deploy the model:\n",
+    "\n",
+    "1. We build a Wallaroo pipeline and assign the model as a pipeline step.  For this tutorial it is called `llama-vertex-pipe`.\n",
+    "2. The pipeline is deployed with the deployment configuration.\n",
+    "3. Once the resources allocation is complete, the model is ready for inferencing.\n",
+    "\n",
+    "See [Model Deploy](https://docs.wallaroo.ai/wallaroo-llm/wallaroo-llm-package-deployment/) for more details on deploying LLMs in Wallaroo."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "335576e5-c0ff-4c99-81f5-65e849235d1f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pipeline = wl.build_pipeline(\"llama-vertex-pipe\")\n",
+    "pipeline.add_model_step(model)\n",
+    "pipeline.deploy(deployment_config=deployment_config)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "80e00a9a-891e-4c20-9f6e-604d5b6b0276",
+   "metadata": {},
+   "source": [
+    "### Generate Inference Request\n",
+    "\n",
+    "The inference request will be submitted as a pandas DataFrame as a text entry."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fb2fa598-e32f-432e-a7b9-71d247f6de83",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_data = pd.DataFrame({'text': ['What happened to the Serge llama?', 'How are you doing?']})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ac742d79",
+   "metadata": {},
+   "source": [
+    "### Submit Inference Request\n",
+    "\n",
+    "The inference request is submitted to the pipeline with the `infer` method, which accepts either:\n",
+    "\n",
+    "* pandas DataFrame\n",
+    "* Apache Arrow Table\n",
+    "\n",
+    "The results are returned in the same format as submitted.  For this example, a pandas DataFrame is submitted, so a pandas DataFrame is returned.  The final generated text is displayed."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a418f8fc-9c2a-4b9f-864f-1b2680f971a4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pipeline.infer(input_data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "30b03c14-0519-4f47-9a91-e47ad5965d84",
+   "metadata": {},
+   "source": [
+    "## Undeploy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "481b0019-b217-4af7-862d-ac3cae0fcb37",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pipeline.undeploy()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}