Merge branch 'main' into evaluating-search-engines

Author: freddiev4

notebooks/en/_toctree.yml

@@ -74,7 +74,7 @@
           title: Phoenix Observability Dashboard on HF Spaces
         - local: search_and_learn
           title: Scaling Test-Time Compute for Longer Thinking in LLMs
-          
+
     - title: Computer Vision Recipes
       isExpanded: false
       sections:
@@ -110,12 +110,16 @@
           title: Smol Multimodal RAG, Building with ColSmolVLM and SmolVLM on Colab's Free-Tier GPU
         - local: fine_tuning_vlm_dpo_smolvlm_instruct
           title: Fine-tuning SmolVLM using direct preference optimization (DPO) with TRL on a consumer GPU
+        - local: structured_generation_vision_language_models
+          title: Structured Generation from Images or Documents Using Vision Language Models
 
     - title: Search Recipes
       isExpanded: false
       sections:
       - local: semantic_reranking_elasticsearch
         title: Semantic Reranking with Elasticsearch
+      - local: vector_search_with_hub_as_backend
+        title: Vector Search on Hugging Face with the Hub as Backend
 
     - title: Agents Recipes
       isExpanded: false

notebooks/en/advanced_rag.ipynb

@@ -28,7 +28,7 @@
     "\n",
     "In this notebook, we will take a look into many of these blue notes to see how to tune your RAG system and get the best performance.\n",
     "\n",
-    "__Let's dig into the model building!__ First, we install the required model dependancies."
+    "__Let's dig into the model building!__ First, we install the required model dependencies."
    ]
   },
   {
@@ -39,7 +39,7 @@
    },
    "outputs": [],
    "source": [
-    "!pip install -q torch transformers transformers accelerate bitsandbytes langchain sentence-transformers faiss-cpu openpyxl pacmap datasets langchain-community ragatouille"
+    "!pip install -q torch transformers accelerate bitsandbytes langchain sentence-transformers faiss-cpu openpyxl pacmap datasets langchain-community ragatouille"
    ]
   },
   {
@@ -272,7 +272,7 @@
     "👀 As you can see, __the chunk lengths are not aligned with our limit of 512 tokens__, and some documents are above the limit, thus some part of them will be lost in truncation!\n",
     " - So we should change the `RecursiveCharacterTextSplitter` class to count length in number of tokens instead of number of characters.\n",
     " - Then we can choose a specific chunk size, here we would choose a lower threshold than 512:\n",
-    "    - smaller documents could allow the split to focus more on specific ideas.\n",
+    "    - Smaller documents could allow the split to focus more on specific ideas.\n",
     "    - But too small chunks would split sentences in half, thus losing meaning again: the proper tuning is a matter of balance."
    ]
   },

notebooks/en/index.md

@@ -8,10 +8,11 @@ applications and solving various machine learning tasks using open-source tools
 Check out the recently added notebooks:
 
 - [Evaluating AI Search Engines with `judges` - the open-source library for LLM-as-a-judge evaluators](llm_judge_evaluating_ai_search_engines_with_judges_library)
+- [Structured Generation from Images or Documents Using Vision Language Models](structured_generation_vision_language_models)
+- [Vector Search on Hugging Face with the Hub as Backend](vector_search_with_hub_as_backend)
 - [Multi-Agent Order Management System with MongoDB](mongodb_smolagents_multi_micro_agents)
 - [Scaling Test-Time Compute for Longer Thinking in LLMs](search_and_learn)
 - [Signature-Aware Model Serving from MLflow with Ray Serve](mlflow_ray_serve)
-- [Fine-tuning SmolVLM using direct preference optimization (DPO) with TRL on a consumer GPU](fine_tuning_vlm_dpo_smolvlm_instruct)
 
 You can also check out the notebooks in the cookbook's [GitHub repo](https://github.com/huggingface/cookbook).
 

notebooks/en/mongodb_smolagents_multi_micro_agents.ipynb

@@ -1,12 +1,5 @@
 {
   "cells": [
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mongodb-developer/GenAI-Showcase/blob/main/notebooks/agents/smolagents_multi-agent_micro_agents.ipynb)"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -46,117 +39,15 @@
     },
     {
       "cell_type": "code",
-      "execution_count": 2,
+      "execution_count": null,
       "metadata": {
         "colab": {
           "base_uri": "https://localhost:8080/"
         },
         "id": "G8R5u8fuFg33",
         "outputId": "8703f072-a9ba-42ab-b9e2-92cdcb3e3de2"
       },
-      "outputs": [
-        {
-          "name": "stdout",
-          "output_type": "stream",
-          "text": [
-            "Requirement already satisfied: smolagents in /usr/local/lib/python3.10/dist-packages (1.0.0)\n",
-            "Requirement already satisfied: pymongo in /usr/local/lib/python3.10/dist-packages (4.10.1)\n",
-            "Requirement already satisfied: litellm in /usr/local/lib/python3.10/dist-packages (1.57.0)\n",
-            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from smolagents) (2.5.1+cu121)\n",
-            "Requirement already satisfied: torchaudio in /usr/local/lib/python3.10/dist-packages (from smolagents) (2.5.1+cu121)\n",
-            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from smolagents) (0.20.1+cu121)\n",
-            "Requirement already satisfied: transformers>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from smolagents) (4.47.1)\n",
-            "Requirement already satisfied: requests>=2.32.3 in /usr/local/lib/python3.10/dist-packages (from smolagents) (2.32.3)\n",
-            "Requirement already satisfied: rich>=13.9.4 in /usr/local/lib/python3.10/dist-packages (from smolagents) (13.9.4)\n",
-            "Requirement already satisfied: pandas>=2.2.3 in /usr/local/lib/python3.10/dist-packages (from smolagents) (2.2.3)\n",
-            "Requirement already satisfied: jinja2>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from smolagents) (3.1.4)\n",
-            "Requirement already satisfied: pillow>=11.0.0 in /usr/local/lib/python3.10/dist-packages (from smolagents) (11.0.0)\n",
-            "Requirement already satisfied: markdownify>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from smolagents) (0.14.1)\n",
-            "Requirement already satisfied: gradio>=5.8.0 in /usr/local/lib/python3.10/dist-packages (from smolagents) (5.9.1)\n",
-            "Requirement already satisfied: duckduckgo-search>=6.3.7 in /usr/local/lib/python3.10/dist-packages (from smolagents) (7.2.0)\n",
-            "Requirement already satisfied: python-dotenv>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from smolagents) (1.0.1)\n",
-            "Requirement already satisfied: e2b-code-interpreter>=1.0.3 in /usr/local/lib/python3.10/dist-packages (from smolagents) (1.0.3)\n",
-            "Requirement already satisfied: dnspython<3.0.0,>=1.16.0 in /usr/local/lib/python3.10/dist-packages (from pymongo) (2.7.0)\n",
-            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from litellm) (3.11.10)\n",
-            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from litellm) (8.1.7)\n",
-            "Requirement already satisfied: httpx<0.28.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from litellm) (0.27.2)\n",
-            "Requirement already satisfied: importlib-metadata>=6.8.0 in /usr/local/lib/python3.10/dist-packages (from litellm) (8.5.0)\n",
-            "Requirement already satisfied: jsonschema<5.0.0,>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from litellm) (4.23.0)\n",
-            "Requirement already satisfied: openai>=1.55.3 in /usr/local/lib/python3.10/dist-packages (from litellm) (1.57.4)\n",
-            "Requirement already satisfied: pydantic<3.0.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from litellm) (2.10.3)\n",
-            "Requirement already satisfied: tiktoken>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from litellm) (0.8.0)\n",
-            "Requirement already satisfied: tokenizers in /usr/local/lib/python3.10/dist-packages (from litellm) (0.21.0)\n",
-            "Requirement already satisfied: primp>=0.9.3 in /usr/local/lib/python3.10/dist-packages (from duckduckgo-search>=6.3.7->smolagents) (0.9.3)\n",
-            "Requirement already satisfied: lxml>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from duckduckgo-search>=6.3.7->smolagents) (5.3.0)\n",
-            "Requirement already satisfied: attrs>=21.3.0 in /usr/local/lib/python3.10/dist-packages (from e2b-code-interpreter>=1.0.3->smolagents) (24.3.0)\n",
-            "Requirement already satisfied: e2b<2.0.0,>=1.0.4 in /usr/local/lib/python3.10/dist-packages (from e2b-code-interpreter>=1.0.3->smolagents) (1.0.5)\n",
-            "Requirement already satisfied: aiofiles<24.0,>=22.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (23.2.1)\n",
-            "Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (3.7.1)\n",
-            "Requirement already satisfied: fastapi<1.0,>=0.115.2 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.115.6)\n",
-            "Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.5.0)\n",
-            "Requirement already satisfied: gradio-client==1.5.2 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (1.5.2)\n",
-            "Requirement already satisfied: huggingface-hub>=0.25.1 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.27.0)\n",
-            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (2.1.5)\n",
-            "Requirement already satisfied: numpy<3.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (1.26.4)\n",
-            "Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (3.10.12)\n",
-            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (24.2)\n",
-            "Requirement already satisfied: pydub in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.25.1)\n",
-            "Requirement already satisfied: python-multipart>=0.0.18 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.0.20)\n",
-            "Requirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (6.0.2)\n",
-            "Requirement already satisfied: ruff>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.8.6)\n",
-            "Requirement already satisfied: safehttpx<0.2.0,>=0.1.6 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.1.6)\n",
-            "Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (2.10.0)\n",
-            "Requirement already satisfied: starlette<1.0,>=0.40.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.41.3)\n",
-            "Requirement already satisfied: tomlkit<0.14.0,>=0.12.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.13.2)\n",
-            "Requirement already satisfied: typer<1.0,>=0.12 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.15.1)\n",
-            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (4.12.2)\n",
-            "Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio>=5.8.0->smolagents) (0.34.0)\n",
-            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==1.5.2->gradio>=5.8.0->smolagents) (2024.10.0)\n",
-            "Requirement already satisfied: websockets<15.0,>=10.0 in /usr/local/lib/python3.10/dist-packages (from gradio-client==1.5.2->gradio>=5.8.0->smolagents) (14.1)\n",
-            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<0.28.0,>=0.23.0->litellm) (2024.12.14)\n",
-            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<0.28.0,>=0.23.0->litellm) (1.0.7)\n",
-            "Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx<0.28.0,>=0.23.0->litellm) (3.10)\n",
-            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx<0.28.0,>=0.23.0->litellm) (1.3.1)\n",
-            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<0.28.0,>=0.23.0->litellm) (0.14.0)\n",
-            "Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=6.8.0->litellm) (3.21.0)\n",
-            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (2024.10.1)\n",
-            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (0.35.1)\n",
-            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (0.22.3)\n",
-            "Requirement already satisfied: beautifulsoup4<5,>=4.9 in /usr/local/lib/python3.10/dist-packages (from markdownify>=0.14.1->smolagents) (4.12.3)\n",
-            "Requirement already satisfied: six<2,>=1.15 in /usr/local/lib/python3.10/dist-packages (from markdownify>=0.14.1->smolagents) (1.17.0)\n",
-            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from openai>=1.55.3->litellm) (1.9.0)\n",
-            "Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from openai>=1.55.3->litellm) (0.8.2)\n",
-            "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai>=1.55.3->litellm) (4.67.1)\n",
-            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas>=2.2.3->smolagents) (2.8.2)\n",
-            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=2.2.3->smolagents) (2024.2)\n",
-            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas>=2.2.3->smolagents) (2024.2)\n",
-            "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.0.0->litellm) (0.7.0)\n",
-            "Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3.0.0,>=2.0.0->litellm) (2.27.1)\n",
-            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.3->smolagents) (3.4.0)\n",
-            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.3->smolagents) (2.2.3)\n",
-            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=13.9.4->smolagents) (3.0.0)\n",
-            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=13.9.4->smolagents) (2.18.0)\n",
-            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken>=0.7.0->litellm) (2024.11.6)\n",
-            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers>=4.0.0->smolagents) (3.16.1)\n",
-            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.0.0->smolagents) (0.4.5)\n",
-            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->litellm) (2.4.4)\n",
-            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->litellm) (1.3.2)\n",
-            "Requirement already satisfied: async-timeout<6.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->litellm) (4.0.3)\n",
-            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->litellm) (1.5.0)\n",
-            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->litellm) (6.1.0)\n",
-            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->litellm) (0.2.1)\n",
-            "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->litellm) (1.18.3)\n",
-            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->smolagents) (3.4.2)\n",
-            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch->smolagents) (1.13.1)\n",
-            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch->smolagents) (1.3.0)\n",
-            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5.0,>=3.0->gradio>=5.8.0->smolagents) (1.2.2)\n",
-            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4<5,>=4.9->markdownify>=0.14.1->smolagents) (2.6)\n",
-            "Requirement already satisfied: protobuf<6.0.0,>=3.20.0 in /usr/local/lib/python3.10/dist-packages (from e2b<2.0.0,>=1.0.4->e2b-code-interpreter>=1.0.3->smolagents) (4.25.5)\n",
-            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=13.9.4->smolagents) (0.1.2)\n",
-            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0,>=0.12->gradio>=5.8.0->smolagents) (1.5.4)\n"
-          ]
-        }
-      ],
+      "outputs": [],
       "source": [
         "!pip install smolagents pymongo litellm"
       ]