From 5df856060450fab0f4ce18f00230369656552ae9 Mon Sep 17 00:00:00 2001 From: shaniewill Date: Thu, 30 Jul 2026 01:40:45 +0200 Subject: [PATCH 1/3] New Version Lab --- lab-agent-vector-store-new-version.ipynb | 1498 ++++++++++++++++++++++ 1 file changed, 1498 insertions(+) create mode 100644 lab-agent-vector-store-new-version.ipynb diff --git a/lab-agent-vector-store-new-version.ipynb b/lab-agent-vector-store-new-version.ipynb new file mode 100644 index 0000000..57c77af --- /dev/null +++ b/lab-agent-vector-store-new-version.ipynb @@ -0,0 +1,1498 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "title", + "metadata": { + "id": "title" + }, + "source": [ + "# Lab | Agent & Vector Store — LangChain 1.x\n", + "\n", + "This updated version uses LangChain 1.x, `create_agent`, retrieval tools, the standalone Chroma integration, and `gpt-4o-mini`." + ] + }, + { + "cell_type": "markdown", + "id": "intro", + "metadata": { + "id": "intro" + }, + "source": [ + "## Intro\n", + "\n", + "In this lab, you will build an AI agent that knows when to consult different knowledge bases instead of relying on a single source of truth.\n", + "\n", + "The demo creates two vector stores: one for the State of the Union address and another for the official Ruff FAQ. Each retriever becomes a tool. The agent can call either tool—or both tools for a multi-source question—and synthesize a grounded answer." + ] + }, + { + "cell_type": "markdown", + "id": "install-heading", + "metadata": { + "id": "install-heading" + }, + "source": [ + "## Install dependencies\n", + "\n", + "Run the following cell once. Restart the notebook kernel after installation if these packages were upgraded in an active session.\n", + "\n", + "> LangChain 1.x requires Python 3.10 or newer." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "install", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "collapsed": true, + "id": "install", + "outputId": "ae0baf4c-88fc-4c3b-ed44-a22b0575e1b8" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Requirement already satisfied: langchain in /usr/local/lib/python3.12/dist-packages (1.3.14)\n", + "Requirement already satisfied: langchain-openai in /usr/local/lib/python3.12/dist-packages (1.4.1)\n", + "Requirement already satisfied: langchain-community in /usr/local/lib/python3.12/dist-packages (0.4.2)\n", + "Requirement already satisfied: langchain-text-splitters in /usr/local/lib/python3.12/dist-packages (1.1.2)\n", + "Requirement already satisfied: langchain-chroma in /usr/local/lib/python3.12/dist-packages (1.1.0)\n", + "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.12/dist-packages (4.15.0)\n", + "Requirement already satisfied: python-dotenv in /usr/local/lib/python3.12/dist-packages (1.2.2)\n", + "Requirement already satisfied: langchain-core<2.0.0,>=1.4.9 in /usr/local/lib/python3.12/dist-packages (from langchain) (1.5.2)\n", + "Requirement already satisfied: langgraph<1.3.0,>=1.2.5 in /usr/local/lib/python3.12/dist-packages (from langchain) (1.2.9)\n", + "Requirement already satisfied: pydantic<3.0.0,>=2.7.4 in /usr/local/lib/python3.12/dist-packages (from langchain) (2.13.4)\n", + "Requirement already satisfied: openai<3.0.0,>=2.45.0 in /usr/local/lib/python3.12/dist-packages (from langchain-openai) (2.45.0)\n", + "Requirement already satisfied: tiktoken<1.0.0,>=0.7.0 in /usr/local/lib/python3.12/dist-packages (from langchain-openai) (0.13.0)\n", + "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (3.14.1)\n", + "Requirement already satisfied: httpx-sse<1.0.0,>=0.4.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (0.4.3)\n", + "Requirement already satisfied: langchain-classic<2.0.0,>=1.0.7 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (1.0.8)\n", + "Requirement already satisfied: langsmith<1.0.0,>=0.1.125 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (0.10.2)\n", + "Requirement already satisfied: numpy>=1.26.2 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (2.0.2)\n", + "Requirement already satisfied: pydantic-settings<3.0.0,>=2.10.1 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (2.14.2)\n", + "Requirement already satisfied: pyyaml<7.0.0,>=5.3.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (6.0.3)\n", + "Requirement already satisfied: requests<3.0.0,>=2.32.5 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (2.34.2)\n", + "Requirement already satisfied: sqlalchemy<3.0.0,>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (2.0.51)\n", + "Requirement already satisfied: tenacity!=8.4.0,<10.0.0,>=8.1.0 in /usr/local/lib/python3.12/dist-packages (from langchain-community) (9.1.4)\n", + "Requirement already satisfied: chromadb<2.0.0,>=1.3.5 in /usr/local/lib/python3.12/dist-packages (from langchain-chroma) (1.5.9)\n", + "Requirement already satisfied: soupsieve>=1.6.1 in /usr/local/lib/python3.12/dist-packages (from beautifulsoup4) (2.8.4)\n", + "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.12/dist-packages (from beautifulsoup4) (4.16.0)\n", + "Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (2.7.1)\n", + "Requirement already satisfied: aiosignal>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.4.0)\n", + "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (26.1.0)\n", + "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.8.0)\n", + "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (6.7.1)\n", + "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (0.5.2)\n", + "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp<4.0.0,>=3.8.3->langchain-community) (1.24.2)\n", + "Requirement already satisfied: build>=1.0.3 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.5.0)\n", + "Requirement already satisfied: pybase64>=1.4.1 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.4.3)\n", + "Requirement already satisfied: uvicorn>=0.18.3 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.18.3->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.51.0)\n", + "Requirement already satisfied: onnxruntime>=1.14.1 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.28.0)\n", + "Requirement already satisfied: opentelemetry-api>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.44.0)\n", + "Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.44.0)\n", + "Requirement already satisfied: opentelemetry-sdk>=1.2.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.44.0)\n", + "Requirement already satisfied: tokenizers>=0.13.2 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.22.2)\n", + "Requirement already satisfied: pypika>=0.48.9 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.51.1)\n", + "Requirement already satisfied: tqdm>=4.65.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (4.67.3)\n", + "Requirement already satisfied: overrides>=7.3.1 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (7.7.0)\n", + "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (7.1.0)\n", + "Requirement already satisfied: grpcio>=1.58.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.82.1)\n", + "Requirement already satisfied: bcrypt>=4.0.1 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (5.0.0)\n", + "Requirement already satisfied: typer>=0.9.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.26.8)\n", + "Requirement already satisfied: kubernetes>=28.1.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (36.0.3)\n", + "Requirement already satisfied: mmh3>=4.0.1 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (5.2.1)\n", + "Requirement already satisfied: orjson>=3.9.12 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (3.11.9)\n", + "Requirement already satisfied: httpx>=0.27.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.28.1)\n", + "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (13.9.4)\n", + "Requirement already satisfied: jsonschema>=4.19.0 in /usr/local/lib/python3.12/dist-packages (from chromadb<2.0.0,>=1.3.5->langchain-chroma) (4.26.0)\n", + "Requirement already satisfied: jsonpatch<2.0.0,>=1.33.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core<2.0.0,>=1.4.9->langchain) (1.33)\n", + "Requirement already satisfied: langchain-protocol>=0.0.17 in /usr/local/lib/python3.12/dist-packages (from langchain-core<2.0.0,>=1.4.9->langchain) (0.0.18)\n", + "Requirement already satisfied: packaging>=23.2.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core<2.0.0,>=1.4.9->langchain) (26.2)\n", + "Requirement already satisfied: uuid-utils<1.0,>=0.12.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core<2.0.0,>=1.4.9->langchain) (0.17.0)\n", + "Requirement already satisfied: langgraph-checkpoint<5.0.0,>=4.1.0 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.3.0,>=1.2.5->langchain) (4.1.1)\n", + "Requirement already satisfied: langgraph-prebuilt<1.2.0,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.3.0,>=1.2.5->langchain) (1.1.0)\n", + "Requirement already satisfied: langgraph-sdk<0.5.0,>=0.4.2 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.3.0,>=1.2.5->langchain) (0.4.2)\n", + "Requirement already satisfied: xxhash>=3.5.0 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.3.0,>=1.2.5->langchain) (3.8.1)\n", + "Requirement already satisfied: anyio>=3.5.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (4.14.2)\n", + "Requirement already satisfied: distro>=1.7.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (1.9.0)\n", + "Requirement already satisfied: requests-toolbelt>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (1.0.0)\n", + "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (1.3.1)\n", + "Requirement already satisfied: websockets>=15.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (15.0.1)\n", + "Requirement already satisfied: zstandard>=0.23.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.1.125->langchain-community) (0.25.0)\n", + "Requirement already satisfied: jiter<1,>=0.10.0 in /usr/local/lib/python3.12/dist-packages (from openai<3.0.0,>=2.45.0->langchain-openai) (0.16.0)\n", + "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (0.7.0)\n", + "Requirement already satisfied: pydantic-core==2.46.4 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (2.46.4)\n", + "Requirement already satisfied: typing-inspection>=0.4.2 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (0.4.2)\n", + "Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.32.5->langchain-community) (3.4.9)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.32.5->langchain-community) (3.18)\n", + "Requirement already satisfied: urllib3<3,>=1.26 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.32.5->langchain-community) (2.5.0)\n", + "Requirement already satisfied: certifi>=2023.5.7 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.32.5->langchain-community) (2026.6.17)\n", + "Requirement already satisfied: greenlet>=1 in /usr/local/lib/python3.12/dist-packages (from sqlalchemy<3.0.0,>=1.4.0->langchain-community) (3.5.3)\n", + "Requirement already satisfied: regex in /usr/local/lib/python3.12/dist-packages (from tiktoken<1.0.0,>=0.7.0->langchain-openai) (2025.11.3)\n", + "Requirement already satisfied: pyproject_hooks in /usr/local/lib/python3.12/dist-packages (from build>=1.0.3->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.2.0)\n", + "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx>=0.27.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.0.9)\n", + "Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx>=0.27.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.16.0)\n", + "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.12/dist-packages (from jsonpatch<2.0.0,>=1.33.0->langchain-core<2.0.0,>=1.4.9->langchain) (3.1.1)\n", + "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.12/dist-packages (from jsonschema>=4.19.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (2025.9.1)\n", + "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.12/dist-packages (from jsonschema>=4.19.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.37.0)\n", + "Requirement already satisfied: rpds-py>=0.25.0 in /usr/local/lib/python3.12/dist-packages (from jsonschema>=4.19.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (2026.6.3)\n", + "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.12/dist-packages (from kubernetes>=28.1.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.17.0)\n", + "Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.12/dist-packages (from kubernetes>=28.1.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (2.9.0.post0)\n", + "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.12/dist-packages (from kubernetes>=28.1.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.9.0)\n", + "Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.12/dist-packages (from kubernetes>=28.1.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (2.0.0)\n", + "Requirement already satisfied: durationpy>=0.7 in /usr/local/lib/python3.12/dist-packages (from kubernetes>=28.1.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.10)\n", + "Requirement already satisfied: ormsgpack>=1.12.0 in /usr/local/lib/python3.12/dist-packages (from langgraph-checkpoint<5.0.0,>=4.1.0->langgraph<1.3.0,>=1.2.5->langchain) (1.12.2)\n", + "Requirement already satisfied: flatbuffers in /usr/local/lib/python3.12/dist-packages (from onnxruntime>=1.14.1->chromadb<2.0.0,>=1.3.5->langchain-chroma) (25.12.19)\n", + "Requirement already satisfied: protobuf>=4.25.8 in /usr/local/lib/python3.12/dist-packages (from onnxruntime>=1.14.1->chromadb<2.0.0,>=1.3.5->langchain-chroma) (5.29.6)\n", + "Requirement already satisfied: googleapis-common-protos~=1.57 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.75.0)\n", + "Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.44.0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.44.0)\n", + "Requirement already satisfied: opentelemetry-proto==1.44.0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.44.0)\n", + "Requirement already satisfied: opentelemetry-semantic-conventions==0.65b0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-sdk>=1.2.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.65b0)\n", + "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich>=10.11.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (4.2.0)\n", + "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich>=10.11.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (2.20.0)\n", + "Requirement already satisfied: huggingface-hub<2.0,>=0.16.4 in /usr/local/lib/python3.12/dist-packages (from tokenizers>=0.13.2->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.23.0)\n", + "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from typer>=0.9.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.5.4)\n", + "Requirement already satisfied: annotated-doc>=0.0.2 in /usr/local/lib/python3.12/dist-packages (from typer>=0.9.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.0.4)\n", + "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.12/dist-packages (from uvicorn>=0.18.3->uvicorn[standard]>=0.18.3->chromadb<2.0.0,>=1.3.5->langchain-chroma) (8.4.2)\n", + "Requirement already satisfied: httptools>=0.8.0 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.18.3->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.8.0)\n", + "Requirement already satisfied: uvloop>=0.15.1 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.18.3->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.22.1)\n", + "Requirement already satisfied: watchfiles>=0.20 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.18.3->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.2.0)\n", + "Requirement already satisfied: filelock>=3.10.0 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<2.0,>=0.16.4->tokenizers>=0.13.2->chromadb<2.0.0,>=1.3.5->langchain-chroma) (3.29.7)\n", + "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<2.0,>=0.16.4->tokenizers>=0.13.2->chromadb<2.0.0,>=1.3.5->langchain-chroma) (2025.3.0)\n", + "Requirement already satisfied: hf-xet<2.0.0,>=1.5.1 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<2.0,>=0.16.4->tokenizers>=0.13.2->chromadb<2.0.0,>=1.3.5->langchain-chroma) (1.5.1)\n", + "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (0.1.2)\n", + "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.12/dist-packages (from requests-oauthlib->kubernetes>=28.1.0->chromadb<2.0.0,>=1.3.5->langchain-chroma) (3.3.1)\n" + ] + } + ], + "source": [ + "%pip install -U langchain langchain-openai langchain-community langchain-text-splitters langchain-chroma beautifulsoup4 python-dotenv" + ] + }, + { + "cell_type": "markdown", + "id": "setup-heading", + "metadata": { + "id": "setup-heading" + }, + "source": [ + "## Initial setup\n", + "\n", + "Load the API key from `.env`, import the current LangChain APIs, and initialize a chat model that supports tool calling." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "imports", + "metadata": { + "id": "imports" + }, + "outputs": [], + "source": [ + "import os\n", + "from pathlib import Path\n", + "\n", + "from dotenv import find_dotenv, load_dotenv\n", + "from langchain.agents import create_agent\n", + "from langchain_chroma import Chroma\n", + "from langchain_community.document_loaders import TextLoader, WebBaseLoader\n", + "from langchain_core.tools import create_retriever_tool\n", + "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n", + "from langchain_text_splitters import RecursiveCharacterTextSplitter" + ] + }, + { + "cell_type": "code", + "source": [ + "from google.colab import userdata\n", + "os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')" + ], + "metadata": { + "id": "ESisSfWfw6de" + }, + "id": "ESisSfWfw6de", + "execution_count": 22, + "outputs": [] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "environment", + "metadata": { + "id": "environment" + }, + "outputs": [], + "source": [ + "load_dotenv(find_dotenv())\n", + "\n", + "if not os.getenv(\"OPENAI_API_KEY\"):\n", + " raise RuntimeError(\"OPENAI_API_KEY is missing. Add it to your .env file.\")" + ] + }, + { + "cell_type": "code", + "source": [ + "llm = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0)\n", + "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")" + ], + "metadata": { + "id": "p8_TMShgxAhr" + }, + "id": "p8_TMShgxAhr", + "execution_count": 24, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "first-source-heading", + "metadata": { + "id": "first-source-heading" + }, + "source": [ + "## Build the first knowledge base\n", + "\n", + "Load the State of the Union text, split it into overlapping chunks, embed the chunks, and store them in an in-memory Chroma collection." + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "path", + "metadata": { + "id": "path" + }, + "outputs": [], + "source": [ + "# This works whether the kernel starts in the lab folder or its parent folder.\n", + "doc_path = Path(\"/content/state_of_the_union.txt\")\n", + "if not doc_path.exists():\n", + " doc_path = Path(\"lab-agent-vector-store/datasets/state_of_the_union.txt\")\n", + "\n", + "if not doc_path.exists():\n", + " raise FileNotFoundError(\"Could not find datasets/state_of_the_union.txt\")" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "state-vector-store", + "metadata": { + "id": "state-vector-store" + }, + "outputs": [], + "source": [ + "documents = TextLoader(str(doc_path), encoding=\"utf-8\").load()\n", + "\n", + "text_splitter = RecursiveCharacterTextSplitter(\n", + " chunk_size=1000,\n", + " chunk_overlap=150,\n", + ")\n", + "state_texts = text_splitter.split_documents(documents)\n", + "\n", + "state_db = Chroma.from_documents(\n", + " documents=state_texts,\n", + " embedding=embeddings,\n", + " collection_name=\"state-of-union\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "second-source-heading", + "metadata": { + "id": "second-source-heading" + }, + "source": [ + "## Add a second knowledge source\n", + "\n", + "Load the current official Ruff FAQ and build a separate Chroma collection." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "ruff-vector-store", + "metadata": { + "id": "ruff-vector-store" + }, + "outputs": [], + "source": [ + "ruff_loader = WebBaseLoader(\"https://docs.astral.sh/ruff/faq/\")\n", + "ruff_documents = ruff_loader.load()\n", + "ruff_texts = text_splitter.split_documents(ruff_documents)\n", + "\n", + "ruff_db = Chroma.from_documents(\n", + " documents=ruff_texts,\n", + " embedding=embeddings,\n", + " collection_name=\"ruff-faq\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "tools-heading", + "metadata": { + "id": "tools-heading" + }, + "source": [ + "## Turn the retrievers into tools\n", + "\n", + "In the current agentic RAG pattern, retrieval tools return relevant evidence and the agent synthesizes the final answer. This replaces the old nested `RetrievalQA` chains." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "tools", + "metadata": { + "id": "tools" + }, + "outputs": [], + "source": [ + "state_of_union_tool = create_retriever_tool(\n", + " state_db.as_retriever(search_kwargs={\"k\": 4}),\n", + " name=\"search_state_of_union\",\n", + " description=(\n", + " \"Search the State of the Union address. Use this tool for questions \"\n", + " \"about statements, people, or topics mentioned in the address.\"\n", + " ),\n", + ")\n", + "\n", + "ruff_tool = create_retriever_tool(\n", + " ruff_db.as_retriever(search_kwargs={\"k\": 4}),\n", + " name=\"search_ruff_faq\",\n", + " description=(\n", + " \"Search the official Ruff FAQ. Use this tool for questions about Ruff, \"\n", + " \"Flake8, linting, formatting, or Jupyter Notebook support.\"\n", + " ),\n", + ")\n", + "\n", + "tools = [state_of_union_tool, ruff_tool]" + ] + }, + { + "cell_type": "markdown", + "id": "agent-heading", + "metadata": { + "id": "agent-heading" + }, + "source": [ + "## Create the agent\n", + "\n", + "`create_agent` is the LangChain 1.x replacement for the old `initialize_agent` and `AgentType.ZERO_SHOT_REACT_DESCRIPTION` APIs." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "agent", + "metadata": { + "id": "agent" + }, + "outputs": [], + "source": [ + "agent = create_agent(\n", + " model=llm,\n", + " tools=tools,\n", + " system_prompt=(\n", + " \"You are a helpful research assistant. Use the retrieval tools whenever \"\n", + " \"a question depends on the State of the Union address or Ruff documentation. \"\n", + " \"For questions involving both sources, call both tools before answering. \"\n", + " \"Base claims about these sources only on retrieved evidence.\"\n", + " ),\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "helper-heading", + "metadata": { + "id": "helper-heading" + }, + "source": [ + "### Helper for displaying the final response\n", + "\n", + "The new agent returns a state containing messages rather than the old AgentExecutor output." + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "ask-helper", + "metadata": { + "id": "ask-helper" + }, + "outputs": [], + "source": [ + "def ask_agent(question: str) -> str:\n", + " result = agent.invoke(\n", + " {\"messages\": [{\"role\": \"user\", \"content\": question}]}\n", + " )\n", + " answer = result[\"messages\"][-1].content\n", + " print(answer)\n", + " return answer" + ] + }, + { + "cell_type": "markdown", + "id": "tests-heading", + "metadata": { + "id": "tests-heading" + }, + "source": [ + "## Test tool selection\n", + "\n", + "First ask a question that should use only the State of the Union tool." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "state-test", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 128 + }, + "id": "state-test", + "outputId": "c8481341-8c3f-45e3-9625-9fa9fc8deacc" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "In the State of the Union address, President Biden mentioned Ketanji Brown Jackson when he discussed his responsibility as President to nominate someone to the United States Supreme Court. He highlighted her qualifications, stating that she is \"one of our nation’s top legal minds\" and emphasized her background as a former top litigator, a federal public defender, and a member of a family of public school educators and police officers. Biden noted that since her nomination, she has received broad support from various groups, including the Fraternal Order of Police and former judges from both political parties.\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'In the State of the Union address, President Biden mentioned Ketanji Brown Jackson when he discussed his responsibility as President to nominate someone to the United States Supreme Court. He highlighted her qualifications, stating that she is \"one of our nation’s top legal minds\" and emphasized her background as a former top litigator, a federal public defender, and a member of a family of public school educators and police officers. Biden noted that since her nomination, she has received broad support from various groups, including the Fraternal Order of Police and former judges from both political parties.'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 32 + } + ], + "source": [ + "ask_agent(\n", + " \"What did Biden say about Ketanji Brown Jackson in the State of the Union address?\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "ruff-test-heading", + "metadata": { + "id": "ruff-test-heading" + }, + "source": [ + "Now ask a question that should use only the Ruff tool." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "ruff-test", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 472 + }, + "id": "ruff-test", + "outputId": "c603ba6e-7494-4743-b1d6-64b293ae3a67" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Ruff offers several advantages over Flake8, making it an appealing choice for developers:\n", + "\n", + "1. **Performance**: Ruff is designed to be faster than Flake8, as it is implemented in Rust, which generally provides better performance compared to Python-based tools.\n", + "\n", + "2. **Comprehensive Rule Set**: Ruff implements over 900 linting rules, including all the rules from Flake8 and many additional ones. This extensive rule set allows for more thorough code analysis.\n", + "\n", + "3. **Native Plugin Support**: Ruff re-implements popular Flake8 plugins natively, which helps minimize conflicts and allows for easier management of rules. For example, you can toggle groups of rules on or off with a single command.\n", + "\n", + "4. **Drop-in Replacement**: Ruff can be used as a drop-in replacement for Flake8 under certain conditions (e.g., without or with a small number of plugins, alongside Black, and on Python 3 code). It implements all Flake8's F rules and a subset of E and W rules.\n", + "\n", + "5. **Formatter Integration**: Ruff includes a formatter that is designed to be a drop-in replacement for Black, ensuring that the output is nearly identical when formatting code that has already been formatted with Black.\n", + "\n", + "6. **Independence of Linter and Formatter**: You can use Ruff's linter and formatter independently, allowing for flexibility in how you integrate it into your workflow.\n", + "\n", + "7. **No Need for Rust Installation**: Ruff can be installed easily via PyPI without requiring a Rust installation, making it more accessible.\n", + "\n", + "Overall, if you're looking for a faster, more comprehensive, and flexible linting tool, Ruff may be the better choice over Flake8.\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "\"Ruff offers several advantages over Flake8, making it an appealing choice for developers:\\n\\n1. **Performance**: Ruff is designed to be faster than Flake8, as it is implemented in Rust, which generally provides better performance compared to Python-based tools.\\n\\n2. **Comprehensive Rule Set**: Ruff implements over 900 linting rules, including all the rules from Flake8 and many additional ones. This extensive rule set allows for more thorough code analysis.\\n\\n3. **Native Plugin Support**: Ruff re-implements popular Flake8 plugins natively, which helps minimize conflicts and allows for easier management of rules. For example, you can toggle groups of rules on or off with a single command.\\n\\n4. **Drop-in Replacement**: Ruff can be used as a drop-in replacement for Flake8 under certain conditions (e.g., without or with a small number of plugins, alongside Black, and on Python 3 code). It implements all Flake8's F rules and a subset of E and W rules.\\n\\n5. **Formatter Integration**: Ruff includes a formatter that is designed to be a drop-in replacement for Black, ensuring that the output is nearly identical when formatting code that has already been formatted with Black.\\n\\n6. **Independence of Linter and Formatter**: You can use Ruff's linter and formatter independently, allowing for flexibility in how you integrate it into your workflow.\\n\\n7. **No Need for Rust Installation**: Ruff can be installed easily via PyPI without requiring a Rust installation, making it more accessible.\\n\\nOverall, if you're looking for a faster, more comprehensive, and flexible linting tool, Ruff may be the better choice over Flake8.\"" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 33 + } + ], + "source": [ + "ask_agent(\"Why use Ruff over Flake8?\")" + ] + }, + { + "cell_type": "markdown", + "id": "direct-heading", + "metadata": { + "id": "direct-heading" + }, + "source": [ + "## Direct retrieval versus agent reasoning\n", + "\n", + "The old notebook used `return_direct=True`, which was tied to the old AgentExecutor loop. In the current architecture, compare the raw retriever output with the agent's synthesized answer instead.\n", + "\n", + "Direct retrieval is useful when another component will process the documents. The agent is useful when the answer must combine, explain, or compare evidence." + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "direct-retrieval", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "direct-retrieval", + "outputId": "0f3b173e-4038-4175-d69a-13ffd17dd688" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "--- Result 1 ---\n", + "Note that, in some cases, Ruff uses different rule codes and prefixes than would be found in the\n", + "originating Flake8 plugins. For example, Ruff uses TID252 to represent the I252 rule from\n", + "flake8-tidy-imports. This helps minimize conflicts across plugins and allows any individual plugin\n", + "to be toggled on or off with a single (e.g.) --select TID, as opposed to --select I2 (to avoid\n", + "conflicts with the isort rules, like I001).\n", + "Beyond the rule set, Ruff's primary limitation vis-à-vis Flake8 is that it does not support custom\n", + "lint rules. (Instead, popular Flake8 plugins are re-implemented in Rust as part of Ruff itself.)\n", + "One minor difference is that Ruff doesn't include all the 'opinionated' rules from flake8-bugbear.\n", + "How does Ruff's linter compare to Pylint?\n", + "At time of writing, Pylint implements \n", + "\n", + "--- Result 2 ---\n", + "How does Ruff's formatter compare to Black?\n", + "The Ruff formatter is designed to be a drop-in replacement for Black.\n", + "Specifically, the formatter is intended to emit near-identical output when run over Black-formatted\n", + "code. When run over extensive Black-formatted projects like Django and Zulip, > 99.9% of lines\n", + "are formatted identically. When migrating an existing project from Black to Ruff, you should expect\n", + "to see a few differences on the margins, but the vast majority of your code should be unchanged.\n", + "When run over non-Black-formatted code, the formatter makes some different decisions than Black,\n", + "and so more deviations should be expected, especially around the treatment of end-of-line comments.\n", + "See Style Guide for more.\n", + "How does Ruff's linter compare to Flake8?\n", + "Ruff can be used as a drop-in\n", + "\n", + "--- Result 3 ---\n", + "plugins, (2) alongside Black, and (3) on Python 3 code.\n", + "Under those conditions, Ruff implements every rule in Flake8. In practice, that means Ruff\n", + "implements all of the F rules (which originate from Pyflakes), along with a subset of the E and\n", + "W rules (which originate from pycodestyle).\n", + "Ruff also re-implements some of the most popular Flake8 plugins and related code quality tools\n", + "natively, including:\n", + "\n", + "--- Result 4 ---\n", + "Ruff can also replace Black, isort,\n", + "yesqa, eradicate, and\n", + "most of the rules implemented in pyupgrade.\n", + "If you're looking to use Ruff, but rely on an unsupported Flake8 plugin, feel free to file an\n", + "issue.\n", + "Do I have to use Ruff's linter and formatter together?\n", + "Nope! Ruff's linter and formatter can be used independently of one another -- you can use\n", + "Ruff as a formatter, but not a linter, or vice versa.\n", + "What versions of Python does Ruff support?\n", + "Ruff can lint code for any Python version from 3.7 onwards, including Python 3.13.\n", + "Ruff does not support Python 2. Ruff may run on pre-Python 3.7 code, although such versions\n", + "are not officially supported (e.g., Ruff does not respect type comments).\n", + "Ruff is installable under any Python version from 3.7 onwards.\n", + "Do I need to install Rust to use Ruff?\n", + "Nope\n", + "\n" + ] + } + ], + "source": [ + "query = \"Why use Ruff over Flake8?\"\n", + "retrieved_docs = ruff_db.as_retriever(search_kwargs={\"k\": 4}).invoke(query)\n", + "\n", + "for index, document in enumerate(retrieved_docs, start=1):\n", + " print(f\"--- Result {index} ---\")\n", + " print(document.page_content[:800])\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "multi-hop-heading", + "metadata": { + "id": "multi-hop-heading" + }, + "source": [ + "## Multi-source reasoning\n", + "\n", + "Ask a question that requires evidence from both vector stores. The system prompt tells the agent to call both tools before answering." + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "multi-hop-test", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 291 + }, + "id": "multi-hop-test", + "outputId": "c8bb86db-4cca-4530-8be2-58a55a7608d2" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Ruff has built-in support for linting and formatting Jupyter Notebooks. It can be used in conjunction with `nbQA`, a tool that allows you to run linters and code formatters over Jupyter Notebooks. After installing both Ruff and nbQA, you can run Ruff on a notebook using the command:\n", + "\n", + "```bash\n", + "$ nbqa ruff Untitled.ipynb\n", + "```\n", + "\n", + "This will lint the notebook and report any issues found.\n", + "\n", + "Regarding the State of the Union address, there was no mention of Ruff or any specific tools related to it. The address focused on various topics, including economic measures, law enforcement, and community safety, but did not reference Ruff or its capabilities.\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'Ruff has built-in support for linting and formatting Jupyter Notebooks. It can be used in conjunction with `nbQA`, a tool that allows you to run linters and code formatters over Jupyter Notebooks. After installing both Ruff and nbQA, you can run Ruff on a notebook using the command:\\n\\n```bash\\n$ nbqa ruff Untitled.ipynb\\n```\\n\\nThis will lint the notebook and report any issues found.\\n\\nRegarding the State of the Union address, there was no mention of Ruff or any specific tools related to it. The address focused on various topics, including economic measures, law enforcement, and community safety, but did not reference Ruff or its capabilities.'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 35 + } + ], + "source": [ + "ask_agent(\n", + " \"What tool can run Ruff over Jupyter Notebooks? \"\n", + " \"Did the president mention that tool in the State of the Union address?\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "exercise", + "metadata": { + "id": "exercise" + }, + "source": [ + "## Your turn\n", + "\n", + "Replace `state_of_the_union.txt` with a text dataset you would enjoy querying, then rebuild the same pipeline.\n", + "\n", + "1. Download a `.txt` dataset into this project or point `TextLoader` to it.\n", + "2. Load and split it with `RecursiveCharacterTextSplitter`.\n", + "3. Create a new Chroma collection with a unique `collection_name`.\n", + "4. Create a retriever tool with a clear name and description. The description should tell the model exactly when to use the tool.\n", + "5. Rebuild the agent with your new tool and the Ruff tool.\n", + "6. Ask one question for your dataset, one question for Ruff, and one multi-source question.\n", + "7. In a Markdown cell, record which tools the agent selected and compare its answer with direct retrieval.\n", + "\n", + "Tip: use `agent.stream(..., stream_mode=\"updates\")` if you want to inspect agent and tool events while learning." + ] + }, + { + "cell_type": "code", + "source": [ + "import os\n", + "import pandas as pd\n", + "from langchain_community.document_loaders import CSVLoader" + ], + "metadata": { + "id": "graSFIeTGl7Z" + }, + "id": "graSFIeTGl7Z", + "execution_count": 36, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "## Load Dataset" + ], + "metadata": { + "id": "YNxM45pPPNQf" + }, + "id": "YNxM45pPPNQf" + }, + { + "cell_type": "code", + "source": [ + "import kagglehub\n", + "\n", + "# Download latest version\n", + "path = kagglehub.dataset_download(\"goyaladi/climate-insights-dataset\")\n", + "\n", + "print(\"Path to dataset files:\", path)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "P931ONYWEd9o", + "outputId": "e1a5e3ff-db10-420e-fe70-a342b5001ca8" + }, + "id": "P931ONYWEd9o", + "execution_count": 37, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Using Colab cache for faster access to the 'climate-insights-dataset' dataset.\n", + "Path to dataset files: /kaggle/input/climate-insights-dataset\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "for file in os.listdir(path):\n", + " print(os.path.join(path, file))" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "QAaXSMbLGGaL", + "outputId": "79a75e31-c325-4a9d-f9eb-bda005a66014" + }, + "id": "QAaXSMbLGGaL", + "execution_count": 38, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "/kaggle/input/climate-insights-dataset/climate_change_data.csv\n", + "/kaggle/input/climate-insights-dataset/Readme.md\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "csv_path = os.path.join(path, \"climate_change_data.csv\")\n", + "\n", + "df = pd.read_csv(csv_path)\n", + "\n", + "print(\"Shape:\", df.shape)\n", + "df.head(10)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 381 + }, + "id": "QPeVzZ8cGV6-", + "outputId": "ad77bff6-3ebb-4e7e-f201-6313d7da9236" + }, + "id": "QPeVzZ8cGV6-", + "execution_count": 45, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Shape: (10000, 9)\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " Date Location Country \\\n", + "0 2000-01-01 00:00:00.000000000 New Williamtown Latvia \n", + "1 2000-01-01 20:09:43.258325832 North Rachel South Africa \n", + "2 2000-01-02 16:19:26.516651665 West Williamland French Guiana \n", + "3 2000-01-03 12:29:09.774977497 South David Vietnam \n", + "4 2000-01-04 08:38:53.033303330 New Scottburgh Moldova \n", + "5 2000-01-05 04:48:36.291629162 South Nathan Saint Helena \n", + "6 2000-01-06 00:58:19.549954995 Port Richardfurt Tuvalu \n", + "7 2000-01-06 21:08:02.808280828 Adambury Australia \n", + "8 2000-01-07 17:17:46.066606660 Williamsonberg Qatar \n", + "9 2000-01-08 13:27:29.324932493 North Thomas Chad \n", + "\n", + " Temperature CO2 Emissions Sea Level Rise Precipitation Humidity \\\n", + "0 10.688986 403.118903 0.717506 13.835237 23.631256 \n", + "1 13.814430 396.663499 1.205715 40.974084 43.982946 \n", + "2 27.323718 451.553155 -0.160783 42.697931 96.652600 \n", + "3 12.309581 422.404983 -0.475931 5.193341 47.467938 \n", + "4 13.210885 410.472999 1.135757 78.695280 61.789672 \n", + "5 6.229326 392.473317 1.122210 76.368331 48.973886 \n", + "6 21.646738 387.648437 0.058471 9.650389 11.402284 \n", + "7 19.730800 448.180275 0.001415 93.360755 21.526350 \n", + "8 19.858114 379.618820 0.584881 6.218846 30.861949 \n", + "9 14.121563 410.517072 -1.712224 15.351583 88.422794 \n", + "\n", + " Wind Speed \n", + "0 18.492026 \n", + "1 34.249300 \n", + "2 34.124261 \n", + "3 8.554563 \n", + "4 8.001164 \n", + "5 30.398908 \n", + "6 15.720944 \n", + "7 29.993495 \n", + "8 37.519472 \n", + "9 47.922521 " + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
DateLocationCountryTemperatureCO2 EmissionsSea Level RisePrecipitationHumidityWind Speed
02000-01-01 00:00:00.000000000New WilliamtownLatvia10.688986403.1189030.71750613.83523723.63125618.492026
12000-01-01 20:09:43.258325832North RachelSouth Africa13.814430396.6634991.20571540.97408443.98294634.249300
22000-01-02 16:19:26.516651665West WilliamlandFrench Guiana27.323718451.553155-0.16078342.69793196.65260034.124261
32000-01-03 12:29:09.774977497South DavidVietnam12.309581422.404983-0.4759315.19334147.4679388.554563
42000-01-04 08:38:53.033303330New ScottburghMoldova13.210885410.4729991.13575778.69528061.7896728.001164
52000-01-05 04:48:36.291629162South NathanSaint Helena6.229326392.4733171.12221076.36833148.97388630.398908
62000-01-06 00:58:19.549954995Port RichardfurtTuvalu21.646738387.6484370.0584719.65038911.40228415.720944
72000-01-06 21:08:02.808280828AdamburyAustralia19.730800448.1802750.00141593.36075521.52635029.993495
82000-01-07 17:17:46.066606660WilliamsonbergQatar19.858114379.6188200.5848816.21884630.86194937.519472
92000-01-08 13:27:29.324932493North ThomasChad14.121563410.517072-1.71222415.35158388.42279447.922521
\n", + "
\n", + "
\n", + "\n", + "
\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "
\n", + "\n", + "\n", + "
\n", + "
\n" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "dataframe", + "variable_name": "df", + "summary": "{\n \"name\": \"df\",\n \"rows\": 10000,\n \"fields\": [\n {\n \"column\": \"Date\",\n \"properties\": {\n \"dtype\": \"object\",\n \"num_unique_values\": 10000,\n \"samples\": [\n \"2014-05-19 04:55:31.053105280\",\n \"2010-10-09 22:53:01.998199808\",\n \"2003-12-25 04:27:00.162016192\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Location\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 7764,\n \"samples\": [\n \"Chavezmouth\",\n \"Shannonbury\",\n \"New Melissaborough\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Country\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 243,\n \"samples\": [\n \"Norfolk Island\",\n \"Tuvalu\",\n \"Cayman Islands\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Temperature\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 5.030615849200166,\n \"min\": -3.803588598564197,\n \"max\": 33.97695597775728,\n \"num_unique_values\": 10000,\n \"samples\": [\n 20.234153409062458,\n 20.371283774970337,\n 18.95790809475954\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"CO2 Emissions\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 49.696932706616785,\n \"min\": 182.1312203487651,\n \"max\": 582.8997013258188,\n \"num_unique_values\": 10000,\n \"samples\": [\n 439.74824466541327,\n 431.8294103755388,\n 460.4931160992976\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Sea Level Rise\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0.9913486683740719,\n \"min\": -4.092154970483335,\n \"max\": 4.116558983659821,\n \"num_unique_values\": 10000,\n \"samples\": [\n 0.968479648925933,\n -0.2556054913303217,\n -1.4337799591064333\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Precipitation\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 28.86241708886994,\n \"min\": 0.0101426705857776,\n \"max\": 99.99190025064706,\n \"num_unique_values\": 10000,\n \"samples\": [\n 8.121517170144099,\n 8.63171177897646,\n 53.52384692287132\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Humidity\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 28.929320027260594,\n \"min\": 0.0189976288834659,\n \"max\": 99.95966453034534,\n \"num_unique_values\": 10000,\n \"samples\": [\n 17.20567441925176,\n 6.913664063442915,\n 57.3292104870909\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Wind Speed\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 14.466648038545443,\n \"min\": 0.0017323274571046,\n \"max\": 49.99766399410564,\n \"num_unique_values\": 10000,\n \"samples\": [\n 10.982594790623562,\n 18.63121193175667,\n 46.72671448381816\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" + } + }, + "metadata": {}, + "execution_count": 45 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Convert CSV into Documents with CSVLoader" + ], + "metadata": { + "id": "vsiD29qOOGam" + }, + "id": "vsiD29qOOGam" + }, + { + "cell_type": "code", + "source": [ + "climate_loader = CSVLoader(\n", + " file_path=str(csv_path),\n", + " encoding=\"utf-8\",\n", + " )\n", + "\n", + "climate_documents = climate_loader.load()\n", + "\n", + "print(\"Number of documents:\", len(climate_documents))\n", + "print(climate_documents[0].page_content)\n", + "print(climate_documents[0].metadata)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "IvwSBOqzEnWU", + "outputId": "c6b0d3ba-41b1-4dc8-e3eb-c37b6f55357b" + }, + "id": "IvwSBOqzEnWU", + "execution_count": 40, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Number of documents: 10000\n", + "Date: 2000-01-01 00:00:00.000000000\n", + "Location: New Williamtown\n", + "Country: Latvia\n", + "Temperature: 10.688985961440224\n", + "CO2 Emissions: 403.118902532313\n", + "Sea Level Rise: 0.7175060280487084\n", + "Precipitation: 13.835236935170093\n", + "Humidity: 23.631256224753127\n", + "Wind Speed: 18.492026001060687\n", + "{'source': '/kaggle/input/climate-insights-dataset/climate_change_data.csv', 'row': 0}\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Chroma vector store" + ], + "metadata": { + "id": "JHPEC02KPXWw" + }, + "id": "JHPEC02KPXWw" + }, + { + "cell_type": "code", + "source": [ + "climate_db = Chroma.from_documents(\n", + " documents=climate_documents,\n", + " embedding=embeddings,\n", + " collection_name=\"climate-insights\",\n", + ")" + ], + "metadata": { + "id": "guYRvqdCMKBW" + }, + "id": "guYRvqdCMKBW", + "execution_count": 41, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "## Climate_tool" + ], + "metadata": { + "id": "aBqw1nhaPZuR" + }, + "id": "aBqw1nhaPZuR" + }, + { + "cell_type": "code", + "source": [ + "climate_tool = create_retriever_tool(\n", + " climate_db.as_retriever(search_kwargs={\"k\": 6}),\n", + " name=\"search_climate_insights\",\n", + " description=(\n", + " \"Search climate observations from the Climate Insights Dataset. \"\n", + " \"The dataset contains records with date, location, country, \"\n", + " \"temperature, CO2 concentration, sea-level rise, precipitation, \"\n", + " \"humidity, and wind speed. Use this tool to retrieve climate \"\n", + " \"records for particular dates, countries, locations, or indicators. \"\n", + " \"This tool retrieves records but does not perform reliable\"\n", + " \"whole-dataset statistical calculations.\"\n", + " ),\n", + ")" + ], + "metadata": { + "id": "nnQbPaJDFC89" + }, + "id": "nnQbPaJDFC89", + "execution_count": 42, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + " tools = [\n", + " climate_tool,\n", + " ruff_tool,\n", + " ]" + ], + "metadata": { + "id": "M5k9w_4PNDM4" + }, + "id": "M5k9w_4PNDM4", + "execution_count": 43, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "## Create_agent" + ], + "metadata": { + "id": "9gcsne0iPiHm" + }, + "id": "9gcsne0iPiHm" + }, + { + "cell_type": "code", + "source": [ + "agent = create_agent(\n", + " model=llm,\n", + " tools=tools,\n", + " system_prompt=(\n", + " \"You are a helpful research assistant. \"\n", + " \"Use search_climate_insights for questions about records in \"\n", + " \"the Climate Insights Dataset. Use search_ruff_faq for \"\n", + " \"questions about Ruff, Flake8, linting, formatting, or \"\n", + " \"Jupyter Notebook support. For questions involving both \"\n", + " \"sources, call both tools before answering. \"\n", + " \"Do not claim that a retrieved record represents a global\"\n", + " \"trend unless the retrieved evidence is sufficient. \"\n", + " \"Clearly distinguish retrieved values from calculations \"\n", + " \"or general climate knowledge.\"\n", + " ),\n", + " )" + ], + "metadata": { + "id": "XaFg1FGSNIhY" + }, + "id": "XaFg1FGSNIhY", + "execution_count": 44, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "## ask_agent" + ], + "metadata": { + "id": "ybFt0P6BPlPv" + }, + "id": "ybFt0P6BPlPv" + }, + { + "cell_type": "code", + "source": [ + "ask_agent(\n", + " \"Retrieve climate observations from Qutar and summarize \"\n", + " \"the indicators found in the matching records.\"\n", + " )" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 488 + }, + "id": "lhUlIo6DOrmH", + "outputId": "d390bd4d-f590-4eb6-9a45-e71a700dfd21" + }, + "id": "lhUlIo6DOrmH", + "execution_count": 46, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "I found several climate observations from Qatar. Here’s a summary of the indicators from the matching records:\n", + "\n", + "1. **Temperature**:\n", + " - Ranges from approximately 13.73°C to 26.73°C.\n", + "\n", + "2. **CO2 Emissions**:\n", + " - Values range from about 385.25 ppm to 404.01 ppm.\n", + "\n", + "3. **Sea Level Rise**:\n", + " - Observations show a range from -0.88 meters to 1.23 meters.\n", + "\n", + "4. **Precipitation**:\n", + " - Recorded precipitation ranges from approximately 49.25 mm to 93.72 mm.\n", + "\n", + "5. **Humidity**:\n", + " - Humidity levels vary from about 16.46% to 62.53%.\n", + "\n", + "6. **Wind Speed**:\n", + " - Wind speeds range from approximately 16.07 km/h to 38.74 km/h.\n", + "\n", + "These records provide a snapshot of various climate indicators in Qatar at different times and locations.\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'I found several climate observations from Qatar. Here’s a summary of the indicators from the matching records:\\n\\n1. **Temperature**:\\n - Ranges from approximately 13.73°C to 26.73°C.\\n\\n2. **CO2 Emissions**:\\n - Values range from about 385.25 ppm to 404.01 ppm.\\n\\n3. **Sea Level Rise**:\\n - Observations show a range from -0.88 meters to 1.23 meters.\\n\\n4. **Precipitation**:\\n - Recorded precipitation ranges from approximately 49.25 mm to 93.72 mm.\\n\\n5. **Humidity**:\\n - Humidity levels vary from about 16.46% to 62.53%.\\n\\n6. **Wind Speed**:\\n - Wind speeds range from approximately 16.07 km/h to 38.74 km/h.\\n\\nThese records provide a snapshot of various climate indicators in Qatar at different times and locations.'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 46 + } + ] + }, + { + "cell_type": "code", + "source": [ + "ask_agent(\"Does Ruff support linting Jupyter Notebooks?\")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 255 + }, + "id": "W6zxk6W8O5B0", + "outputId": "5e6b57f9-92da-4bce-9d76-36a3bcae0d6e" + }, + "id": "W6zxk6W8O5B0", + "execution_count": 47, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Yes, Ruff does support linting and formatting Jupyter Notebooks. It has built-in support for this functionality, and you can also integrate it with nbQA, a tool that allows you to run linters and code formatters over Jupyter Notebooks. \n", + "\n", + "To lint a Jupyter Notebook using Ruff, you can run the following command after installing both Ruff and nbQA:\n", + "\n", + "```bash\n", + "$ nbqa ruff Untitled.ipynb\n", + "```\n", + "\n", + "This will check the notebook for linting issues and report any errors found.\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'Yes, Ruff does support linting and formatting Jupyter Notebooks. It has built-in support for this functionality, and you can also integrate it with nbQA, a tool that allows you to run linters and code formatters over Jupyter Notebooks. \\n\\nTo lint a Jupyter Notebook using Ruff, you can run the following command after installing both Ruff and nbQA:\\n\\n```bash\\n$ nbqa ruff Untitled.ipynb\\n```\\n\\nThis will check the notebook for linting issues and report any errors found.'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 47 + } + ] + }, + { + "cell_type": "code", + "source": [ + "sample = df.iloc[0]\n", + "\n", + "question = (\n", + " f\"Summarize the climate record for {sample['Location']} \"\n", + " f\"in {sample['Country']} on {sample['Date']}. \"\n", + " \"Also explain whether Ruff can lint the Jupyter Notebook \"\n", + " \"used to analyze this CSV dataset.\"\n", + ")\n", + "\n", + "ask_agent(question)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 490 + }, + "id": "KamAeG5oPAwx", + "outputId": "72d07714-982e-4b24-f827-581eea743ddd" + }, + "id": "KamAeG5oPAwx", + "execution_count": 48, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "### Climate Record for New Williamtown, Latvia on January 1, 2000\n", + "\n", + "- **Date**: 2000-01-01\n", + "- **Location**: New Williamtown, Latvia\n", + "- **Temperature**: 10.69 °C\n", + "- **CO2 Emissions**: 403.12 ppm\n", + "- **Sea Level Rise**: 0.72 mm\n", + "- **Precipitation**: 13.84 mm\n", + "- **Humidity**: 23.63%\n", + "- **Wind Speed**: 18.49 km/h\n", + "\n", + "### Ruff and Jupyter Notebook Support\n", + "\n", + "Ruff does support linting and formatting for Jupyter Notebooks. You can use Ruff in conjunction with `nbQA`, a tool that allows you to run linters and code formatters over Jupyter Notebooks. After installing Ruff and nbQA, you can lint a notebook by running a command like:\n", + "\n", + "```bash\n", + "$ nbqa ruff your_notebook.ipynb\n", + "```\n", + "\n", + "This will check the notebook for linting issues, similar to how it would for regular Python scripts.\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'### Climate Record for New Williamtown, Latvia on January 1, 2000\\n\\n- **Date**: 2000-01-01\\n- **Location**: New Williamtown, Latvia\\n- **Temperature**: 10.69 °C\\n- **CO2 Emissions**: 403.12 ppm\\n- **Sea Level Rise**: 0.72 mm\\n- **Precipitation**: 13.84 mm\\n- **Humidity**: 23.63%\\n- **Wind Speed**: 18.49 km/h\\n\\n### Ruff and Jupyter Notebook Support\\n\\nRuff does support linting and formatting for Jupyter Notebooks. You can use Ruff in conjunction with `nbQA`, a tool that allows you to run linters and code formatters over Jupyter Notebooks. After installing Ruff and nbQA, you can lint a notebook by running a command like:\\n\\n```bash\\n$ nbqa ruff your_notebook.ipynb\\n```\\n\\nThis will check the notebook for linting issues, similar to how it would for regular Python scripts.'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 48 + } + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.10+" + }, + "colab": { + "provenance": [], + "gpuType": "T4" + }, + "accelerator": "GPU" + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file From 7ae564c2a2844ca28e6e57d5a43a20534ef1039f Mon Sep 17 00:00:00 2001 From: Shan Date: Thu, 30 Jul 2026 01:44:08 +0200 Subject: [PATCH 2/3] Delete old version lab-agent-vector-store.ipynb --- lab-agent-vector-store.ipynb | 556 ----------------------------------- 1 file changed, 556 deletions(-) delete mode 100644 lab-agent-vector-store.ipynb diff --git a/lab-agent-vector-store.ipynb b/lab-agent-vector-store.ipynb deleted file mode 100644 index ed58a74..0000000 --- a/lab-agent-vector-store.ipynb +++ /dev/null @@ -1,556 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "efad6479-7fb1-40c4-8610-8963f8de307f", - "metadata": {}, - "source": [ - "# Lab | Agent & Vector store\n" - ] - }, - { - "cell_type": "markdown", - "id": "25fb0c6d", - "metadata": {}, - "source": [ - "
\n", - "\n", - "## Intro\n", - "\n", - "In this lab you'll build an AI agent that knows when to consult *different* knowledge bases to answer a question — instead of relying on a single source of truth.\n", - "\n", - "Here's what to expect:\n", - "\n", - "1. **Follow a full worked demo** — We'll walk through every step together: ingesting the *state of the union* speech and the *Ruff* docs into two vector stores, wrapping each in a `RetrievalQA` tool, and building an agent that picks the right tool (or both!) depending on the question.\n", - "\n", - "2. **Replicate it yourself with a new dataset** — Then, you'll swap in a dataset of your choice and rebuild the same pipeline, adapting the prompts and tools along the way.\n", - "\n", - "By the end of this lab, you'll understand how to build multi-source AI agents and be able to apply the pattern to your own datasets." - ] - }, - { - "cell_type": "markdown", - "id": "68b24990", - "metadata": {}, - "source": [ - "
\n", - "\n", - "## Combine agents and vector stores\n", - "\n", - "Let's get into the demo. We'll wrap each vector store in a `RetrievalQA` chain and hand it to an agent as a `Tool`. The agent then decides, at each step, which tool to call based purely on its description — this is what lets it route between multiple knowledge sources.\n", - "\n", - "There are two flavors of this pattern, both of which we'll try below:\n", - "\n", - "- **Agent as reasoner** — the agent calls a tool and can keep reasoning afterward (e.g. to combine results from multiple sources).\n", - "- **Agent as router** (`return_direct=True`) — the agent just picks the right tool and returns its answer immediately, no extra reasoning." - ] - }, - { - "cell_type": "markdown", - "id": "9b22020a", - "metadata": {}, - "source": [ - "
\n", - "\n", - "## Install dependencies\n", - "\n", - "Uncomment and run the cells below to install the dependencies required for this notebook.\n", - "\n", - "Tip: Use a virtual environment to keep this project's dependencies isolated from your system Python and other projects." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "12df91e0-7322-43c2-96cd-0159d017a1e1", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# !pip install \"langchain<0.3\" \"langchain-core<0.3\" \"langchain-community<0.3\" \"langchain-openai<0.2\"" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "936c402a", - "metadata": {}, - "outputs": [], - "source": [ - "# !pip install python-dotenv==1.2.2 chromadb==1.5.9 beautifulsoup4==4.15.0" - ] - }, - { - "cell_type": "markdown", - "id": "565a6825", - "metadata": {}, - "source": [ - "
\n", - "\n", - "## Initial Setup\n", - "\n", - "Before building anything, we need to load our API credentials, instantiate the LLM we'll use throughout the notebook, and locate the sample document we'll be querying." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2868cff1-a3ef-426f-b8d7-7fe89047a5b7", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "from langchain.chains import RetrievalQA\n", - "from langchain_community.vectorstores import Chroma\n", - "from langchain_openai import OpenAI, OpenAIEmbeddings\n", - "from langchain_text_splitters import CharacterTextSplitter\n", - "from langchain_community.document_loaders import TextLoader" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "aeff3a2d-b0cc-429f-a34c-4381d71f1a5f", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "import os\n", - "from dotenv import load_dotenv, find_dotenv\n", - "_ = load_dotenv(find_dotenv())\n", - "\n", - "OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "14331eec-fd46-42e0-b6e7-adaf21824ef7", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "llm = OpenAI(temperature=0, api_key=OPENAI_API_KEY)" - ] - }, - { - "cell_type": "markdown", - "id": "b3f2857f", - "metadata": {}, - "source": [ - "
\n", - "\n", - "Now let's ingest the state of the union speech: load the raw text, split it into manageable chunks, embed those chunks, and store them in a Chroma vector store." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "bbb327d2", - "metadata": {}, - "outputs": [], - "source": [ - "doc_path = \"./datasets/state_of_the_union.txt\"" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f2675861", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "loader = TextLoader(doc_path)\n", - "documents = loader.load()\n", - "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n", - "texts = text_splitter.split_documents(documents)\n", - "\n", - "embeddings = OpenAIEmbeddings(api_key=OPENAI_API_KEY)\n", - "\n", - "docsearch = Chroma.from_documents(texts, embeddings, collection_name=\"state-of-union\")" - ] - }, - { - "cell_type": "markdown", - "id": "fe780cdc", - "metadata": {}, - "source": [ - "
\n", - "\n", - "## Adding a second knowledge source\n", - "\n", - "To show how an agent can route between multiple tools, let's add a second vector store — this time built from the Ruff FAQ web page instead of a local file." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "bc5403d4", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "state_of_union = RetrievalQA.from_chain_type(\n", - " llm=llm, chain_type=\"stuff\", retriever=docsearch.as_retriever()\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1431cded", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "from langchain_community.document_loaders import WebBaseLoader" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "915d3ff3", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "loader = WebBaseLoader(\"https://beta.ruff.rs/docs/faq/\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "96a2edf8", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "docs = loader.load()\n", - "ruff_texts = text_splitter.split_documents(docs)\n", - "ruff_db = Chroma.from_documents(ruff_texts, embeddings, collection_name=\"ruff\")\n", - "ruff = RetrievalQA.from_chain_type(\n", - " llm=llm, chain_type=\"stuff\", retriever=ruff_db.as_retriever()\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "c0a6c031", - "metadata": {}, - "source": [ - "
\n", - "\n", - "## Create the Agent\n", - "\n", - "With both `RetrievalQA` chains ready, we wrap each one in a `Tool` (giving it a name and a description the agent will use to decide when to call it), then hand both tools to an agent." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "eb142786", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Import things that are needed generically\n", - "from langchain.agents import AgentType, Tool, initialize_agent\n", - "from langchain_openai import OpenAI" - ] - }, - { - "cell_type": "markdown", - "id": "6fa28c55", - "metadata": {}, - "source": [ - "
\n", - "\n", - "Let's try it out — first with a question only the state of the union tool can answer, then one only Ruff can answer. Watch the verbose output to see which tool the agent picks each time." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "850bc4e9", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "tools = [\n", - " Tool(\n", - " name=\"State of Union QA System\",\n", - " func=state_of_union.run,\n", - " description=\"useful for when you need to answer questions about the most recent state of the union address. Input should be a fully formed question.\",\n", - " ),\n", - " Tool(\n", - " name=\"Ruff QA System\",\n", - " func=ruff.run,\n", - " description=\"useful for when you need to answer questions about ruff (a python linter). Input should be a fully formed question.\",\n", - " ),\n", - "]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fc47f230", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Construct the agent. We will use the default agent type here.\n", - "# See documentation for a full list of options.\n", - "agent = initialize_agent(\n", - " tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "10ca2db8", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "agent.invoke(\n", - " \"What did biden say about ketanji brown jackson in the state of the union address?\"\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "4e91b811", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "agent.invoke(\"Why use ruff over flake8?\")" - ] - }, - { - "cell_type": "markdown", - "id": "787a9b5e", - "metadata": {}, - "source": [ - "## Use the Agent solely as a router" - ] - }, - { - "cell_type": "markdown", - "id": "9161ba91", - "metadata": {}, - "source": [ - "
\n", - "\n", - "You can also set `return_direct=True` if you intend to use the agent as a router and just want to directly return the result of the RetrievalQAChain.\n", - "\n", - "Notice that in the above examples the agent did some extra work after querying the RetrievalQAChain. You can avoid that and just return the result directly." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f59b377e", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "tools = [\n", - " Tool(\n", - " name=\"State of Union QA System\",\n", - " func=state_of_union.run,\n", - " description=\"useful for when you need to answer questions about the most recent state of the union address. Input should be a fully formed question.\",\n", - " return_direct=True,\n", - " ),\n", - " Tool(\n", - " name=\"Ruff QA System\",\n", - " func=ruff.run,\n", - " description=\"useful for when you need to answer questions about ruff (a python linter). Input should be a fully formed question.\",\n", - " return_direct=True,\n", - " ),\n", - "]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8615707a", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "agent = initialize_agent(\n", - " tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "36e718a9", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "agent.invoke(\n", - " \"What did biden say about ketanji brown jackson in the state of the union address?\"\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "edfd0a1a", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "agent.invoke(\"Why use ruff over flake8?\")" - ] - }, - { - "cell_type": "markdown", - "id": "49a0cbbe", - "metadata": {}, - "source": [ - "
\n", - "\n", - "## Multi-Hop vector store reasoning\n", - "\n", - "Because vector stores are easily usable as tools in agents, it is easy to use answer multi-hop questions that depend on vector stores using the existing agent framework." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "d397a233", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "tools = [\n", - " Tool(\n", - " name=\"State of Union QA System\",\n", - " func=state_of_union.run,\n", - " description=\"useful for when you need to answer questions about the most recent state of the union address. Input should be a fully formed question, not referencing any obscure pronouns from the conversation before.\",\n", - " ),\n", - " Tool(\n", - " name=\"Ruff QA System\",\n", - " func=ruff.run,\n", - " description=\"useful for when you need to answer questions about ruff (a python linter). Input should be a fully formed question, not referencing any obscure pronouns from the conversation before.\",\n", - " ),\n", - "]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "06157240", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Construct the agent. We will use the default agent type here.\n", - "# See documentation for a full list of options.\n", - "agent = initialize_agent(\n", - " tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "b492b520", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "agent.invoke(\n", - " \"What tool does ruff use to run over Jupyter Notebooks? Did the president mention that tool in the state of the union?\"\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "b26b4963", - "metadata": {}, - "source": [ - "

\n", - "\n", - "---\n", - "\n", - "
\n", - "\n", - "## 🚀 Your turn\n", - "\n", - "Time to make this lab your own! \n", - "\n", - "Replace the `state_of_the_union.txt` dataset with something you'd actually enjoy chatting with. A great place to start is the [sonnets.txt dataset](https://github.com/martin-gorner/tensorflow-rnn-shakespeare/blob/master/shakespeare/sonnets.txt) —or any other .txt file from that repository. Of course, you're not limited to those options. Pick any text that interests you and see how your chatbot responds.\n", - "\n", - "Here's what to do:\n", - "\n", - "1. **Get your data** — Download your chosen `.txt` file into this project folder (or point `TextLoader` at it directly).\n", - "2. **Rebuild the vector store** — Load, split, and embed your new document, then create a fresh `RetrievalQA` chain for it (give it a descriptive `collection_name`!).\n", - "3. **Rewrite the tool description** — Update the `Tool`'s `name` and `description` so the agent knows *when* it should reach for this new tool instead of the Ruff or state-of-the-union ones.\n", - "4. **Rebuild the agent** — Combine your new tool with the existing Ruff tool (or drop it if you'd rather keep just your new dataset + one other source).\n", - "5. **Put it to the test** — Ask your agent:\n", - " - A direct question that only your new dataset can answer.\n", - " - A question that only the Ruff tool can answer.\n", - " - A multi-hop question that requires combining *both* tools' knowledge, like the Jupyter/Ruff example above.\n", - "6. **Reflect** — In a markdown cell, briefly note whether the agent picked the right tool(s) each time, and what happened when you set `return_direct=True` vs. not.\n", - "\n", - "⭐️ **Bonus points:**\n", - "- Instead of modifying this same file, create a new file `solution.ipynb` and replicate the process from scratch.\n", - "\n", - "💡 **Tip:** \n", - "- Watch the `verbose=True` agent logs closely — they show you the agent's reasoning step by step, which is the best way to understand *why* it picked a particular tool.\n", - "\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python (LangChain 0.2.x)", - "language": "python", - "name": "langchain-v0.2.x" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.13" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} From 64c7fe2297bcd501cfb051ad2609df9b40150d08 Mon Sep 17 00:00:00 2001 From: Shan Date: Thu, 30 Jul 2026 09:16:16 +0200 Subject: [PATCH 3/3] Rename lab-agent-vector-store notebook file --- ...ent-vector-store-new-version.ipynb => lab-agent-vector-store | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename lab-agent-vector-store-new-version.ipynb => lab-agent-vector-store (99%) diff --git a/lab-agent-vector-store-new-version.ipynb b/lab-agent-vector-store similarity index 99% rename from lab-agent-vector-store-new-version.ipynb rename to lab-agent-vector-store index 57c77af..f44ec90 100644 --- a/lab-agent-vector-store-new-version.ipynb +++ b/lab-agent-vector-store @@ -1495,4 +1495,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} \ No newline at end of file +}