text-video-search/src/python/create-feed-query-text-video-search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "excess-listing",
   "metadata": {},
   "source": [
    "# Text-Video search app example\n",
    "\n",
    "Create, deploy, feed and query the Vespa app using the Vespa python API.\n",
    "\n",
    "This example requires FFmpeg for video processing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bearing-adelaide",
   "metadata": {},
   "source": [
    "## Install required packages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "nonprofit-riverside",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recreational-characterization",
   "metadata": {},
   "source": [
    "## CLIP models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "operational-transcript",
   "metadata": {},
   "source": [
    "List the model variations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "moderate-earth",
   "metadata": {},
   "outputs": [],
   "source": [
    "import clip\n",
    "\n",
    "clip.available_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adolescent-freedom",
   "metadata": {},
   "source": [
    "Each model might have a different embedding size,\n",
    "and this information is needed when creating the schema of the text-video search application.\n",
    "Running the below outputs the dimensions for the supported models -\n",
    "as this is a large download, a copy of the output is listed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "third-magic",
   "metadata": {},
   "outputs": [],
   "source": [
    "#embedding_info = {\n",
    "#    name: clip.load(name)[0].visual.output_dim for name in clip.available_models()\n",
    "#}\n",
    "#embedding_info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c393f68-3261-43fb-85b1-7485fa8c41d5",
   "metadata": {},
   "source": [
    "````\n",
    "{'RN50': 1024,\n",
    " 'RN101': 512,\n",
    " 'RN50x4': 640,\n",
    " 'RN50x16': 768,\n",
    " 'RN50x64': 1024,\n",
    " 'ViT-B/32': 512,\n",
    " 'ViT-B/16': 512,\n",
    " 'ViT-L/14': 768,\n",
    " 'ViT-L/14@336px': 768}\n",
    "````"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "different-remainder",
   "metadata": {},
   "source": [
    "## Create and deploy a text-video search app\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thirty-territory",
   "metadata": {},
   "source": [
    "### Create the Vespa application package\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "international-question",
   "metadata": {},
   "source": [
    "The function `create_text_video_app` below uses [Pyvespa](https://pyvespa.readthedocs.io/en/latest/)\n",
    "to create an application package with fields to store image embeddings extracted from the videos\n",
    "that we want to search based on the selected CLIP models.\n",
    "It also declares the types of the text embeddings that we are going to send along with the query when searching for images,\n",
    "and creates one ranking profile for each (text, image) embedding model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38f995b3",
   "metadata": {},
   "source": [
    "For this demonstration we are going to use only one CLIP model but we could very well index all the available models for comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "present-encoding",
   "metadata": {},
   "outputs": [],
   "source": [
    "from embedding import create_text_video_app\n",
    "\n",
    "app_package = create_text_video_app({\"ViT-B/32\": 512})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "neutral-fence",
   "metadata": {},
   "source": [
    "Inspect how the `schema` of the resulting application package looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "southeast-liabilities",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "schema videosearch {\n",
      "    document videosearch {\n",
      "        field video_file_name type string {\n",
      "            indexing: summary | attribute\n",
      "        }\n",
      "        field vit_b_32_image type tensor<float>(x[512]) {\n",
      "            indexing: attribute | index\n",
      "            attribute {\n",
      "                distance-metric: euclidean\n",
      "            }\n",
      "            index {\n",
      "                hnsw {\n",
      "                    max-links-per-node: 16\n",
      "                    neighbors-to-explore-at-insert: 500\n",
      "                }\n",
      "            }\n",
      "        }\n",
      "    }\n",
      "    rank-profile vit_b_32_similarity inherits default {\n",
      "        first-phase {\n",
      "            expression {\n",
      "                closeness(vit_b_32_image)\n",
      "            }\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(app_package.schema.schema_to_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "meaning-report",
   "metadata": {},
   "source": [
    "### Deploy to Vespa Cloud\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "assured-possible",
   "metadata": {},
   "source": [
    "Refer to [Authenticating with Vespa Cloud](https://pyvespa.readthedocs.io/en/latest/authenticating-to-vespa-cloud.html)\n",
    "for any issues with the below (replace with your tenant name):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "printable-spray",
   "metadata": {},
   "outputs": [],
   "source": [
    "from vespa.deployment import VespaCloud\n",
    "\n",
    "vespa_cloud = VespaCloud(\n",
    "    tenant=\"mytenant\",\n",
    "    application=\"videosearch\",\n",
    "    application_package=app_package,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "983947a9-3231-46c0-a58d-ed68598b3b27",
   "metadata": {},
   "outputs": [],
   "source": [
    "app = vespa_cloud.deploy(\n",
    "    instance=\"clip-video-search\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "foreign-complaint",
   "metadata": {},
   "source": [
    "Alternatively, check [this guide](https://pyvespa.readthedocs.io/en/latest/getting-started-pyvespa.html) to deploy locally in a Docker container.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7230b645",
   "metadata": {},
   "source": [
    "## Download and convert the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cc37439",
   "metadata": {},
   "source": [
    "We are going to use the UCF101 dataset to allow users to follow along from\n",
    "their laptop. We downloaded a [zipped file](http://storage.googleapis.com/thumos14_files/UCF101_videos.zip)\n",
    "containing 13320 trimmed videos, each including one action,\n",
    "and a [text file](http://crcv.ucf.edu/THUMOS14/Class%20Index.txt) containing the list of action\n",
    "classes and their numerical index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d45fda4-ece7-4867-8713-8a46b291138e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "video_dir=os.getcwd()\n",
    "print(video_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d1a2759",
   "metadata": {},
   "source": [
    "There is better support for `.mp4` files, convert the `.avi` files to `.mp4` using [ffmpeg](https://www.ffmpeg.org/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58ebbbf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess\n",
    "\n",
    "def convert_from_avi_to_mp4(file_name):\n",
    "    outputfile = file_name.lower().replace(\".avi\", \".mp4\")\n",
    "    subprocess.call([\"ffmpeg\", \"-i\", file_name, outputfile])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73d68bcc",
   "metadata": {},
   "source": [
    "The code below takes quite a while and could be sped up by using multi-processing:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c0c099",
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "\n",
    "video_files = glob.glob(os.path.join(video_dir, \"*.avi\"))\n",
    "for file_name in video_files:\n",
    "    convert_from_avi_to_mp4(file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c1c4259",
   "metadata": {},
   "source": [
    "## Compute and feed embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suspended-supervision",
   "metadata": {},
   "source": [
    "The function below assumes you have downloaded the UCF101 dataset and converted it to .mp4.\n",
    "It extracts frames from the video, computes image embeddings according to a CLIP model and sends them to the Vespa app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bacterial-capital",
   "metadata": {},
   "outputs": [],
   "source": [
    "from embedding import compute_and_send_video_embeddings\n",
    "\n",
    "compute_and_send_video_embeddings(\n",
    "    app=app,\n",
    "    batch_size=32,\n",
    "    clip_model_names=[\"ViT-B/32\"],\n",
    "    number_frames_per_video=4,\n",
    "    video_dir=video_dir,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "organizational-delta",
   "metadata": {},
   "source": [
    "The function `compute_and_send_video_embeddings` is a more robust version of the following loop:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "canadian-accused",
   "metadata": {},
   "outputs": [],
   "source": [
    "for model_name in clip_model_names:\n",
    "    video_dataset = (\n",
    "        VideoFeedDataset(  ## PyTorch Dataset that outputs pyvespa-compatible data\n",
    "            video_dir=video_dir,  # Folder containing video files\n",
    "            model_name=model_name,  # CLIP model name used to convert image into vector\n",
    "            number_frames_per_video=4,  # Number of image frames to use per video\n",
    "        )\n",
    "    )\n",
    "    dataloader = DataLoader(  ## PyTorch Dataloader to loop through the dataset\n",
    "        video_dataset,\n",
    "        batch_size=batch_size,\n",
    "        collate_fn=lambda x: [\n",
    "            item for sublist in x for item in sublist\n",
    "        ],  # turn list of list into flat list\n",
    "    )\n",
    "    for idx, batch in enumerate(dataloader):\n",
    "        app.update_batch(batch=batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eea78554",
   "metadata": {},
   "source": [
    "## Query the application\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eff562f6",
   "metadata": {},
   "source": [
    "We created a custom class `VideoSearchApp` that implements a `query` method that is specific to text-video use case that we are demonstrating here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cadd128",
   "metadata": {},
   "outputs": [],
   "source": [
    "from embedding import VideoSearchApp\n",
    "\n",
    "video_app = VideoSearchApp(app=app, clip_model_name=\"ViT-B/32\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191275fb",
   "metadata": {},
   "source": [
    "It takes a `text` query, transform it into an embedding with the CLIP model, and for each video it takes the score of the frame of that video that is closest to the text in the joint embedding space to represent the score of the video. We can also select the number of videos that we want to retrieve.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51f14f55",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = video_app.query(text=\"playing soccer\", number_videos=4)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19a1f763",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Video, display\n",
    "\n",
    "for hit in result:\n",
    "    display(\n",
    "        Video(os.path.join(os.environ[\"VIDEO_DIR\"], hit[\"video_file_name\"]), embed=True)\n",
    "    )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}