diff --git a/.DS_Store b/.DS_Store
new file mode 100644
index 0000000..2c86323
Binary files /dev/null and b/.DS_Store differ
diff --git a/lab-extractive-question-answering.ipynb b/lab-extractive-question-answering.ipynb
deleted file mode 100644
index 7a04dde..0000000
--- a/lab-extractive-question-answering.ipynb
+++ /dev/null
@@ -1,738 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "split-aluminum",
- "metadata": {
- "id": "split-aluminum",
- "papermill": {
- "duration": 0.048394,
- "end_time": "2021-04-15T21:06:39.560571",
- "exception": false,
- "start_time": "2021-04-15T21:06:39.512177",
- "status": "completed"
- },
- "tags": []
- },
- "source": [
- "# LAB | Extractive Question Answering"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e3f44179",
- "metadata": {},
- "source": [
- "\n",
- "\n",
- "
\n",
- "\n",
- "**Run this notebook in [Google Colab](https://colab.research.google.com/) for GPU acceleration.**\n",
- "\n",
- "
\n",
- "\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "prospective-turner",
- "metadata": {
- "id": "prospective-turner",
- "papermill": {
- "duration": 0.045573,
- "end_time": "2021-04-15T21:06:39.651272",
- "exception": false,
- "start_time": "2021-04-15T21:06:39.605699",
- "status": "completed"
- },
- "tags": []
- },
- "source": [
- "This notebook demonstrates how Pinecone helps you build an extractive question-answering application. To build an extractive question-answering system, we need three main components:\n",
- "\n",
- "- A vector index to store and run semantic search\n",
- "- A retriever model for embedding context passages\n",
- "- A reader model to extract answers\n",
- "\n",
- "We will use the SQuAD dataset, which consists of **questions** and **context** paragraphs containing question **answers**. We generate embeddings for the context passages using the retriever, index them in the vector database, and query with semantic search to retrieve the top k most relevant contexts containing potential answers to our question. We then use the reader model to extract the answers from the returned contexts."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "oC3GG-dWkZJ6",
- "metadata": {
- "id": "oC3GG-dWkZJ6"
- },
- "source": [
- "Let's get started by installing the packages needed for notebook to run:"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "terminal-export",
- "metadata": {
- "id": "terminal-export",
- "papermill": {
- "duration": 0.044413,
- "end_time": "2021-04-15T21:06:39.741951",
- "exception": false,
- "start_time": "2021-04-15T21:06:39.697538",
- "status": "completed"
- },
- "tags": []
- },
- "source": [
- "# Install Dependencies"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "expressed-executive",
- "metadata": {
- "execution": {
- "iopub.execute_input": "2021-04-15T21:06:39.845309Z",
- "iopub.status.busy": "2021-04-15T21:06:39.842494Z",
- "iopub.status.idle": "2021-04-15T21:08:22.163939Z",
- "shell.execute_reply": "2021-04-15T21:08:22.164616Z"
- },
- "id": "expressed-executive",
- "papermill": {
- "duration": 102.376674,
- "end_time": "2021-04-15T21:08:22.165052",
- "exception": false,
- "start_time": "2021-04-15T21:06:39.788378",
- "status": "completed"
- },
- "tags": []
- },
- "outputs": [],
- "source": [
- "!pip install -q \\\n",
- " transformers==4.46.3 \\\n",
- " datasets==4.5.0 \\\n",
- " pinecone==5.4.2 \\\n",
- " sentence-transformers==3.3.1 \\\n",
- " huggingface_hub==0.26.5\n",
- " \n",
- "!rm -rf ~/.cache/huggingface"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "2DCPtl6IhgSz",
- "metadata": {
- "id": "2DCPtl6IhgSz"
- },
- "outputs": [],
- "source": [
- "# Force-kill the kernel process to restart it, ensuring the newly\n",
- "# installed/uninstalled package versions are loaded in a fresh process\n",
- "\n",
- "import os\n",
- "import sys\n",
- "\n",
- "print(\n",
- " \"⚠️ WARNING:\\n\"\n",
- " \"This code cell will restart the kernel to load the newly installed packages...\\n\"\n",
- " \"If you used 'Run all', it will stop here — that's expected.\\n\"\n",
- " \"See the note below on how to continue running the rest of the notebook.\",\n",
- " flush=True,\n",
- ")\n",
- "sys.stdout.flush()\n",
- "\n",
- "os.kill(os.getpid(), 9)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d339487b",
- "metadata": {},
- "source": [
- "\n",
- "\n",
- "⚠️ **Do not use \"Run all\" for this whole notebook in one go.** The cell above force-restarts the kernel (`os.kill`) so the newly installed packages load cleanly. Colab's \"Run all\" cannot continue past a kernel restart — it will always stop right after that cell, no matter how many times you retry it.\n",
- "\n",
- "Instead, run it in two steps:\n",
- "1. Run only the two cells above (install + restart). Wait for the kernel to finish restarting (a few seconds).\n",
- "2. Click on the next cell below, then use **Runtime → Run after** (or manually run the remaining cells one by one / select them and press Shift+Enter). Do not click \"Run all\" again, since it would just re-trigger the restart at the same spot.\n",
- "\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "29ad3840",
- "metadata": {
- "id": "29ad3840"
- },
- "source": [
- "# Load Dataset"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "hgIieQukgagu",
- "metadata": {
- "id": "hgIieQukgagu"
- },
- "source": [
- "Now let's load the SQUAD dataset from the HuggingFace Model Hub. We load the dataset into a pandas dataframe and filter the title, question, and context columns, and we drop any duplicate context passages."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "J250IJeh7NIb",
- "metadata": {
- "id": "J250IJeh7NIb"
- },
- "outputs": [],
- "source": [
- "from datasets import load_dataset\n",
- "\n",
- "# load the squad dataset into a pandas dataframe\n",
- "df = load_dataset(\"rajpurkar/squad\", split=\"train\").to_pandas()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "FcmeNO97dHDO",
- "metadata": {
- "id": "FcmeNO97dHDO"
- },
- "outputs": [],
- "source": [
- "# select only title and context column\n",
- "df = None\n",
- "# drop rows containing duplicate context passages\n",
- "df = None\n",
- "df"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "57bbcb57",
- "metadata": {
- "id": "57bbcb57"
- },
- "source": [
- "# Pinecone Setup"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e24d904c",
- "metadata": {
- "id": "e24d904c"
- },
- "source": [
- "Pinecone is a managed vector database that lets us store context embeddings and run fast semantic search over them. In this section, we set up the connection to Pinecone using an API key and create the index that will hold our context embeddings."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "472402a1",
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import getpass\n",
- "from pinecone import Pinecone\n",
- "\n",
- "#\n",
- "# For this notebook, you'll need a Pinecone API key, \n",
- "# to get one, sign up for free at https://app.pinecone.io/,\n",
- "# then go to \"API Keys\" and copy the default key (or create a new one).\n",
- "#\n",
- "# Once you have your Pinecone API key,\n",
- "# load it from Colab's \"Secrets\" manager \n",
- "# (the key icon in the left sidebar). \n",
- "# Add a secret named PINECONE_API_KEY there and enable notebook access.\n",
- "#\n",
- "# Note: Colab Secrets rely on a UI popup to grant access, which only works when\n",
- "# running in the Colab UI itself. If you're running this notebook through\n",
- "# another editor (e.g. VS Code's Colab extension), that popup isn't available,\n",
- "# so we fall back to an environment variable or a secure manual prompt instead.\n",
- "api_key = os.environ.get('PINECONE_API_KEY')\n",
- "\n",
- "\n",
- "if not api_key:\n",
- " try:\n",
- " from google.colab import userdata\n",
- " api_key = userdata.get('PINECONE_API_KEY')\n",
- " except Exception:\n",
- " api_key = None\n",
- "\n",
- "if not api_key:\n",
- " api_key = getpass.getpass('Enter your Pinecone API key: ')\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7ca5ef09",
- "metadata": {},
- "source": [
- "
\n",
- "\n",
- "Once we have the API key, we can initialize the connection:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "092d1e71",
- "metadata": {
- "id": "092d1e71"
- },
- "outputs": [],
- "source": [
- "from pinecone import Pinecone, ServerlessSpec\n",
- "\n",
- "spec = ServerlessSpec(\n",
- " cloud=\"aws\", region=\"us-east-1\"\n",
- ")\n",
- "\n",
- "# connect to pinecone environment\n",
- "pc = Pinecone(\n",
- " api_key = api_key\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "58028e12",
- "metadata": {
- "id": "58028e12"
- },
- "source": [
- "Now we create a new index called \"question-answering\" — we can name the index anything we want. We specify the metric type as \"cosine\" and dimension as 384 because the retriever we use to generate context embeddings is optimized for cosine similarity and outputs 384-dimension vectors."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b3206184",
- "metadata": {
- "id": "b3206184"
- },
- "outputs": [],
- "source": [
- "index_name = None\n",
- "\n",
- "# check if the extractive-question-answering index exists\n",
- "if index_name not in pinecone.list_indexes().names():\n",
- " # create the index if it does not exist\n",
- " None\n",
- "# connect to extractive-question-answering index we created\n",
- "index = pinecone.Index(index_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6e84a3e5",
- "metadata": {
- "id": "6e84a3e5"
- },
- "source": [
- "# Initialize Retriever"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "oZzhGS1Lpj0g",
- "metadata": {
- "id": "oZzhGS1Lpj0g"
- },
- "source": [
- "Next, we need to initialize our retriever. The retriever will mainly do two things:\n",
- "\n",
- "- Generate embeddings for all context passages (context vectors/embeddings)\n",
- "- Generate embeddings for our questions (query vector/embedding)\n",
- "\n",
- "The retriever will generate embeddings in a way that the questions and context passages containing answers to our questions are nearby in the vector space. We can use cosine similarity to calculate the similarity between the query and context embeddings to find the context passages that contain potential answers to our question.\n",
- "\n",
- "We will use a SentenceTransformer model named ``multi-qa-MiniLM-L6-cos-v1`` designed for semantic search and trained on 215M (question, answer) pairs from diverse sources as our retriever."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "31a85bb3",
- "metadata": {
- "id": "31a85bb3"
- },
- "outputs": [],
- "source": [
- "import torch\n",
- "from sentence_transformers import SentenceTransformer\n",
- "\n",
- "# set device to GPU if available\n",
- "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
- "# load the retriever model from huggingface model hub\n",
- "retriever = None #use the 'multi-qa-MiniLM-L6-cos-v1' model from HuggingFace to build the retriever\n",
- "\n",
- "retriever"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8aaad0a2",
- "metadata": {
- "id": "8aaad0a2"
- },
- "source": [
- "# Generate Embeddings and Upsert"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "Hgy7AagJtO_p",
- "metadata": {
- "id": "Hgy7AagJtO_p"
- },
- "source": [
- "Next, we need to generate embeddings for the context passages. We will do this in batches to help us more quickly generate embeddings and upload them to the Pinecone index. When passing the documents to Pinecone, we need an id (a unique value), context embedding, and metadata for each document representing context passages in the dataset. The metadata is a dictionary containing data relevant to our embeddings, such as the article title, context passage, etc."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a17824ef",
- "metadata": {
- "id": "a17824ef",
- "tags": []
- },
- "outputs": [],
- "source": [
- "from tqdm.auto import tqdm\n",
- "\n",
- "# we will use batches of 64\n",
- "batch_size = 64\n",
- "\n",
- "for i in tqdm(range(0, len(df), batch_size)):\n",
- " # find end of batch\n",
- " None\n",
- " # extract batch\n",
- " None\n",
- " # generate embeddings for batch\n",
- " emb = None\n",
- " # get metadata\n",
- " meta = None\n",
- " # create unique IDs\n",
- " ids = None\n",
- " # add all to upsert list\n",
- " to_upsert = None\n",
- " # upsert/insert these records to pinecone\n",
- " _ = index.upsert(vectors=to_upsert)\n",
- "\n",
- "# check that we have all vectors in index\n",
- "index.describe_index_stats()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "YFyBYafuJ0y0",
- "metadata": {
- "id": "YFyBYafuJ0y0"
- },
- "source": [
- "# Initialize Reader"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "HgdiLCz5ynOk",
- "metadata": {
- "id": "HgdiLCz5ynOk"
- },
- "source": [
- "We use the `deepset/electra-base-squad2` model from the HuggingFace model hub as our reader model. We load this model into a \"question-answering\" pipeline from HuggingFace transformers and feed it our questions and context passages individually. The model gives a prediction for each context we pass through the pipeline."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "hg9XTDkIJzH_",
- "metadata": {
- "id": "hg9XTDkIJzH_"
- },
- "outputs": [],
- "source": [
- "from transformers import pipeline\n",
- "\n",
- "model_name = 'deepset/electra-base-squad2'\n",
- "# load the reader model into a question-answering pipeline\n",
- "\n",
- "device_id = 0 if torch.cuda.is_available() else -1\n",
- "reader = pipeline(tokenizer=model_name, model=model_name, task='question-answering', device=device_id)\n",
- "\n",
- "reader"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e14d89d6",
- "metadata": {
- "id": "e14d89d6"
- },
- "source": [
- "Now all the components we need are ready. Let's write some helper functions to execute our queries. The `get_context` function retrieves the context embeddings containing answers to our question from the Pinecone index, and the `extract_answer` function extracts the answers from these context passages."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "lyYaY3QEQiHZ",
- "metadata": {
- "id": "lyYaY3QEQiHZ"
- },
- "outputs": [],
- "source": [
- "# gets context passages from the pinecone index\n",
- "def get_context(question, top_k):\n",
- " # generate embeddings for the question\n",
- " xq = None\n",
- " # search pinecone index for context passage with the answer\n",
- " xc = None\n",
- " # extract the context passage from pinecone search result\n",
- " c = None\n",
- " return c\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "Dc9VYOiUQA7B",
- "metadata": {
- "id": "Dc9VYOiUQA7B"
- },
- "outputs": [],
- "source": [
- "from pprint import pprint\n",
- "\n",
- "# extracts answer from the context passage\n",
- "def extract_answer(question, context):\n",
- " results = []\n",
- " for c in context:\n",
- " # feed the reader the question and contexts to extract answers\n",
- " answer = reader(question=question, context=c)\n",
- " # add the context to answer dict for printing both together\n",
- " answer[\"context\"] = c\n",
- " results.append(answer)\n",
- " # sort the result based on the score from reader model\n",
- " sorted_result = pprint(sorted(results, key=lambda x: x['score'], reverse=True))\n",
- " return sorted_result"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5E3a3dkJ5ZQD",
- "metadata": {
- "id": "5E3a3dkJ5ZQD"
- },
- "outputs": [],
- "source": [
- "question = \"How much oil is Egypt producing in a day?\"\n",
- "context = get_context(question, top_k = 1)\n",
- "context"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "heKNVbWQ_LtC",
- "metadata": {
- "id": "heKNVbWQ_LtC"
- },
- "source": [
- "As we can see, the retiever is working fine and gets us the context passage that contains the answer to our question. Now let's use the reader to extract the exact answer from the context passage."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "DQ4GWdbMSjPl",
- "metadata": {
- "id": "DQ4GWdbMSjPl"
- },
- "outputs": [],
- "source": [
- "extract_answer(question, context)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fMD_ABuDAyhN",
- "metadata": {
- "id": "fMD_ABuDAyhN"
- },
- "source": [
- "The reader model predicted with 99% accuracy the correct answer *691,000 bbl/d* as seen from the context passage. Let's run few more queries."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "_4NRgV4mGWoj",
- "metadata": {
- "id": "_4NRgV4mGWoj"
- },
- "outputs": [],
- "source": [
- "question = \"What are the first names of the men that invented youtube?\"\n",
- "context = get_context(question, top_k=1)\n",
- "extract_answer(question, context)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "juXlctWgJgMF",
- "metadata": {
- "id": "juXlctWgJgMF"
- },
- "outputs": [],
- "source": [
- "question = \"What is Albert Eistein famous for?\"\n",
- "context = get_context(question, top_k=1)\n",
- "extract_answer(question, context)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "OhCgeny_BVno",
- "metadata": {
- "id": "OhCgeny_BVno"
- },
- "source": [
- "Let's run another question. This time for top 3 context passages from the retriever."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "iXACn71xmett",
- "metadata": {
- "id": "iXACn71xmett"
- },
- "outputs": [],
- "source": [
- "question = \"Who was the first person to step foot on the moon?\"\n",
- "context = get_context(question, top_k=3)\n",
- "extract_answer(question, context)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "27fecadc-6976-43b2-90e1-788a8633ecc7",
- "metadata": {
- "id": "27fecadc-6976-43b2-90e1-788a8633ecc7"
- },
- "source": [
- "
\n",
- "\n",
- "### 🚀 Challenge\n",
- "\n",
- "Try adding a few more questions of your own in new cells below (reusing `get_context` and `extract_answer`). Experiment with different `top_k` values and observe:\n",
- "\n",
- "- Does the retriever always return a context that actually contains the answer?\n",
- "- How does the reader's confidence score change with harder or more ambiguous questions?\n",
- "- What happens with questions that have no good answer in the dataset?"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "898daa2f",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ...\n",
- "# ...\n",
- "# ..."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d4ed6384",
- "metadata": {},
- "source": [
- "
\n",
- "\n",
- "---\n",
- "\n",
- "### Clean Up\n",
- "\n",
- "Now that we're done experimenting, let's delete the Pinecone index we created. This frees up the resources it was using (and, when using a paid plan, avoids being billed for an index we no longer need)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "c53493d1",
- "metadata": {},
- "outputs": [],
- "source": [
- "# To delete the Pinecone index, \n",
- "# uncomment the line below + run this code cell\n",
- "\n",
- "# pc.delete_index(index_name)"
- ]
- }
- ],
- "metadata": {
- "accelerator": "GPU",
- "colab": {
- "gpuType": "T4",
- "machine_shape": "hm",
- "provenance": []
- },
- "environment": {
- "name": "tf2-gpu.2-3.m65",
- "type": "gcloud",
- "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m65"
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.8"
- },
- "papermill": {
- "default_parameters": {},
- "duration": 333.240754,
- "end_time": "2021-04-15T21:12:11.363566",
- "environment_variables": {},
- "exception": null,
- "input_path": "/notebooks/question_answering/question_answering.ipynb",
- "output_path": "/notebooks/tmp/question_answering/question_answering.ipynb",
- "parameters": {},
- "start_time": "2021-04-15T21:06:38.122812",
- "version": "2.3.3"
- },
- "vscode": {
- "interpreter": {
- "hash": "5fe10bf018ef3e697f9035d60bf60847932a12bface18908407fd371fe880db9"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/lab_extractive_question_answering(6).ipynb b/lab_extractive_question_answering(6).ipynb
new file mode 100644
index 0000000..2add059
--- /dev/null
+++ b/lab_extractive_question_answering(6).ipynb
@@ -0,0 +1,1698 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "split-aluminum",
+ "metadata": {
+ "id": "split-aluminum"
+ },
+ "source": [
+ "# LAB | Extractive Question Answering"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "prospective-turner",
+ "metadata": {
+ "id": "prospective-turner"
+ },
+ "source": [
+ "This notebook demonstrates how Pinecone helps you build an extractive question-answering application. To build an extractive question-answering system, we need three main components:\n",
+ "\n",
+ "- A vector index to store and run semantic search\n",
+ "- A retriever model for embedding context passages\n",
+ "- A reader model to extract answers\n",
+ "\n",
+ "We will use the SQuAD dataset, which consists of **questions** and **context** paragraphs containing question **answers**. We generate embeddings for the context passages using the retriever, index them in the vector database, and query with semantic search to retrieve the top k most relevant contexts containing potential answers to our question. We then use the reader model to extract the answers from the returned contexts."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "oC3GG-dWkZJ6",
+ "metadata": {
+ "id": "oC3GG-dWkZJ6"
+ },
+ "source": [
+ "Let's get started by installing the packages needed for notebook to run:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5924add8",
+ "metadata": {
+ "id": "5924add8"
+ },
+ "source": [
+ "Install Dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "id": "XFkQCVF8uNL3",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1000
+ },
+ "id": "XFkQCVF8uNL3",
+ "outputId": "098e276a-f92d-4660-b66e-c2df768eaec4"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Requirement already satisfied: langchain in /usr/local/lib/python3.12/dist-packages (1.3.14)\n",
+ "Requirement already satisfied: langchain-core in /usr/local/lib/python3.12/dist-packages (1.5.3)\n",
+ "Requirement already satisfied: langchain-classic in /usr/local/lib/python3.12/dist-packages (1.0.8)\n",
+ "Requirement already satisfied: langchain-pinecone in /usr/local/lib/python3.12/dist-packages (0.2.13)\n",
+ "Requirement already satisfied: langchain-huggingface in /usr/local/lib/python3.12/dist-packages (1.2.2)\n",
+ "Requirement already satisfied: datasets in /usr/local/lib/python3.12/dist-packages (5.0.1)\n",
+ "Requirement already satisfied: pinecone-client in /usr/local/lib/python3.12/dist-packages (6.0.0)\n",
+ "Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.12/dist-packages (5.6.1)\n",
+ "Requirement already satisfied: torch in /usr/local/lib/python3.12/dist-packages (2.13.0)\n",
+ "Requirement already satisfied: langgraph<1.3.0,>=1.2.5 in /usr/local/lib/python3.12/dist-packages (from langchain) (1.2.9)\n",
+ "Requirement already satisfied: pydantic<3.0.0,>=2.7.4 in /usr/local/lib/python3.12/dist-packages (from langchain) (2.13.4)\n",
+ "Requirement already satisfied: jsonpatch<2.0.0,>=1.33.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core) (1.33)\n",
+ "Requirement already satisfied: langchain-protocol>=0.0.17 in /usr/local/lib/python3.12/dist-packages (from langchain-core) (0.0.18)\n",
+ "Requirement already satisfied: langsmith<1.0.0,>=0.3.45 in /usr/local/lib/python3.12/dist-packages (from langchain-core) (0.10.2)\n",
+ "Requirement already satisfied: packaging>=23.2.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core) (26.2)\n",
+ "Requirement already satisfied: pyyaml<7.0.0,>=5.3.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core) (6.0.3)\n",
+ "Requirement already satisfied: tenacity!=8.4.0,<10.0.0,>=8.1.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core) (9.1.4)\n",
+ "Requirement already satisfied: typing-extensions<5.0.0,>=4.7.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core) (4.16.0)\n",
+ "Requirement already satisfied: uuid-utils<1.0,>=0.12.0 in /usr/local/lib/python3.12/dist-packages (from langchain-core) (0.17.0)\n",
+ "Requirement already satisfied: langchain-text-splitters<2.0.0,>=1.1.2 in /usr/local/lib/python3.12/dist-packages (from langchain-classic) (1.1.2)\n",
+ "Requirement already satisfied: requests<3.0.0,>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from langchain-classic) (2.32.4)\n",
+ "Requirement already satisfied: sqlalchemy<3.0.0,>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from langchain-classic) (2.0.51)\n",
+ "Requirement already satisfied: pinecone<8.0.0,>=6.0.0 in /usr/local/lib/python3.12/dist-packages (from pinecone[asyncio]<8.0.0,>=6.0.0->langchain-pinecone) (7.3.0)\n",
+ "Requirement already satisfied: numpy!=2.0.2,>=1.26.4 in /usr/local/lib/python3.12/dist-packages (from langchain-pinecone) (2.5.1)\n",
+ "Requirement already satisfied: langchain-openai>=0.3.11 in /usr/local/lib/python3.12/dist-packages (from langchain-pinecone) (1.4.1)\n",
+ "Requirement already satisfied: httpx>=0.28.0 in /usr/local/lib/python3.12/dist-packages (from langchain-pinecone) (0.28.1)\n",
+ "Requirement already satisfied: simsimd>=5.9.11 in /usr/local/lib/python3.12/dist-packages (from langchain-pinecone) (6.5.16)\n",
+ "Requirement already satisfied: huggingface-hub<2.0.0,>=0.33.4 in /usr/local/lib/python3.12/dist-packages (from langchain-huggingface) (1.26.0)\n",
+ "Requirement already satisfied: tokenizers<1.0.0,>=0.19.1 in /usr/local/lib/python3.12/dist-packages (from langchain-huggingface) (0.22.2)\n",
+ "Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from datasets) (3.32.2)\n",
+ "Requirement already satisfied: pyarrow>=21.0.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (25.0.0)\n",
+ "Requirement already satisfied: dill<0.4.2,>=0.3.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.3.8)\n",
+ "Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (from datasets) (2.2.2)\n",
+ "Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.12/dist-packages (from datasets) (4.70.0)\n",
+ "Requirement already satisfied: xxhash in /usr/local/lib/python3.12/dist-packages (from datasets) (3.8.1)\n",
+ "Requirement already satisfied: multiprocess<0.70.20 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.70.16)\n",
+ "Collecting fsspec<=2026.6.0,>=2023.1.0 (from fsspec[http]<=2026.6.0,>=2023.1.0->datasets)\n",
+ " Downloading fsspec-2026.6.0-py3-none-any.whl.metadata (10 kB)\n",
+ "Requirement already satisfied: certifi>=2019.11.17 in /usr/local/lib/python3.12/dist-packages (from pinecone-client) (2026.7.22)\n",
+ "Requirement already satisfied: pinecone-plugin-interface<0.0.8,>=0.0.7 in /usr/local/lib/python3.12/dist-packages (from pinecone-client) (0.0.7)\n",
+ "Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.12/dist-packages (from pinecone-client) (2.9.0.post0)\n",
+ "Requirement already satisfied: urllib3>=1.26.5 in /usr/local/lib/python3.12/dist-packages (from pinecone-client) (2.5.0)\n",
+ "Requirement already satisfied: transformers<6.0.0,>=4.41.0 in /usr/local/lib/python3.12/dist-packages (from sentence-transformers) (5.14.1)\n",
+ "Requirement already satisfied: scikit-learn>=0.22.0 in /usr/local/lib/python3.12/dist-packages (from sentence-transformers) (1.9.0)\n",
+ "Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from sentence-transformers) (1.18.0)\n",
+ "Requirement already satisfied: setuptools>=77.0.3 in /usr/local/lib/python3.12/dist-packages (from torch) (83.0.0)\n",
+ "Requirement already satisfied: sympy>=1.13.3 in /usr/local/lib/python3.12/dist-packages (from torch) (1.14.0)\n",
+ "Requirement already satisfied: networkx>=2.5.1 in /usr/local/lib/python3.12/dist-packages (from torch) (3.6.1)\n",
+ "Requirement already satisfied: jinja2 in /usr/local/lib/python3.12/dist-packages (from torch) (3.1.6)\n",
+ "Requirement already satisfied: cuda-toolkit==13.0.3 in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (13.0.3.0)\n",
+ "Requirement already satisfied: cuda-bindings<14,>=13.0.3 in /usr/local/lib/python3.12/dist-packages (from torch) (13.3.1)\n",
+ "Requirement already satisfied: nvidia-cudnn-cu13==9.20.0.48 in /usr/local/lib/python3.12/dist-packages (from torch) (9.20.0.48)\n",
+ "Requirement already satisfied: nvidia-cusparselt-cu13==0.8.1 in /usr/local/lib/python3.12/dist-packages (from torch) (0.8.1)\n",
+ "Requirement already satisfied: nvidia-nccl-cu13==2.29.7 in /usr/local/lib/python3.12/dist-packages (from torch) (2.29.7)\n",
+ "Requirement already satisfied: nvidia-nvshmem-cu13==3.4.5 in /usr/local/lib/python3.12/dist-packages (from torch) (3.4.5)\n",
+ "Requirement already satisfied: triton==3.7.1 in /usr/local/lib/python3.12/dist-packages (from torch) (3.7.1)\n",
+ "Requirement already satisfied: nvidia-cublas==13.1.1.3.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (13.1.1.3)\n",
+ "Requirement already satisfied: nvidia-cuda-nvrtc==13.0.88.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (13.0.88)\n",
+ "Requirement already satisfied: nvidia-cuda-runtime==13.0.96.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (13.0.96)\n",
+ "Requirement already satisfied: nvidia-cufft==12.0.0.61.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (12.0.0.61)\n",
+ "Requirement already satisfied: nvidia-nvjitlink<14,>=13.0.88 in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (13.3.33)\n",
+ "Requirement already satisfied: nvidia-cufile==1.15.1.6.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (1.15.1.6)\n",
+ "Requirement already satisfied: nvidia-cuda-cupti==13.0.85.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (13.0.85)\n",
+ "Requirement already satisfied: nvidia-curand==10.4.0.35.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (10.4.0.35)\n",
+ "Requirement already satisfied: nvidia-cusolver==12.0.4.66.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (12.0.4.66)\n",
+ "Requirement already satisfied: nvidia-cusparse==12.6.3.3.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (12.6.3.3)\n",
+ "Requirement already satisfied: nvidia-nvtx==13.0.85.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==13.0.3; platform_system == \"Linux\"->torch) (13.0.85)\n",
+ "Requirement already satisfied: cuda-pathfinder>=1.4.2 in /usr/local/lib/python3.12/dist-packages (from cuda-bindings<14,>=13.0.3->torch) (1.6.0)\n",
+ "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.12/dist-packages (from fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (3.14.1)\n",
+ "Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx>=0.28.0->langchain-pinecone) (4.14.2)\n",
+ "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx>=0.28.0->langchain-pinecone) (1.0.9)\n",
+ "Requirement already satisfied: idna in /usr/local/lib/python3.12/dist-packages (from httpx>=0.28.0->langchain-pinecone) (3.18)\n",
+ "Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx>=0.28.0->langchain-pinecone) (0.16.0)\n",
+ "Requirement already satisfied: click<9.0.0,>=8.4.2 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<2.0.0,>=0.33.4->langchain-huggingface) (8.4.2)\n",
+ "Requirement already satisfied: hf-xet<2.0.0,>=1.5.1 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub<2.0.0,>=0.33.4->langchain-huggingface) (1.5.2)\n",
+ "Requirement already satisfied: jsonpointer>=1.9 in /usr/local/lib/python3.12/dist-packages (from jsonpatch<2.0.0,>=1.33.0->langchain-core) (3.1.1)\n",
+ "Requirement already satisfied: openai<3.0.0,>=2.45.0 in /usr/local/lib/python3.12/dist-packages (from langchain-openai>=0.3.11->langchain-pinecone) (2.45.0)\n",
+ "Requirement already satisfied: tiktoken<1.0.0,>=0.7.0 in /usr/local/lib/python3.12/dist-packages (from langchain-openai>=0.3.11->langchain-pinecone) (0.13.0)\n",
+ "Requirement already satisfied: langgraph-checkpoint<5.0.0,>=4.1.0 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.3.0,>=1.2.5->langchain) (4.1.1)\n",
+ "Requirement already satisfied: langgraph-prebuilt<1.2.0,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.3.0,>=1.2.5->langchain) (1.1.0)\n",
+ "Requirement already satisfied: langgraph-sdk<0.5.0,>=0.4.2 in /usr/local/lib/python3.12/dist-packages (from langgraph<1.3.0,>=1.2.5->langchain) (0.4.2)\n",
+ "Requirement already satisfied: distro>=1.7.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.3.45->langchain-core) (1.9.0)\n",
+ "Requirement already satisfied: orjson>=3.9.14 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.3.45->langchain-core) (3.11.9)\n",
+ "Requirement already satisfied: requests-toolbelt>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.3.45->langchain-core) (1.0.0)\n",
+ "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.3.45->langchain-core) (1.3.1)\n",
+ "Requirement already satisfied: websockets>=15.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.3.45->langchain-core) (15.0.1)\n",
+ "Requirement already satisfied: zstandard>=0.23.0 in /usr/local/lib/python3.12/dist-packages (from langsmith<1.0.0,>=0.3.45->langchain-core) (0.25.0)\n",
+ "Requirement already satisfied: pinecone-plugin-assistant<2.0.0,>=1.6.0 in /usr/local/lib/python3.12/dist-packages (from pinecone<8.0.0,>=6.0.0->pinecone[asyncio]<8.0.0,>=6.0.0->langchain-pinecone) (1.8.0)\n",
+ "Requirement already satisfied: aiohttp-retry<3.0.0,>=2.9.1 in /usr/local/lib/python3.12/dist-packages (from pinecone[asyncio]<8.0.0,>=6.0.0->langchain-pinecone) (2.9.1)\n",
+ "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (0.7.0)\n",
+ "Requirement already satisfied: pydantic-core==2.46.4 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (2.46.4)\n",
+ "Requirement already satisfied: typing-inspection>=0.4.2 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.7.4->langchain) (0.4.2)\n",
+ "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.5.3->pinecone-client) (1.17.0)\n",
+ "Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.0.0->langchain-classic) (3.4.9)\n",
+ "Requirement already satisfied: joblib>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from scikit-learn>=0.22.0->sentence-transformers) (1.5.3)\n",
+ "Requirement already satisfied: narwhals>=2.0.1 in /usr/local/lib/python3.12/dist-packages (from scikit-learn>=0.22.0->sentence-transformers) (2.24.0)\n",
+ "Requirement already satisfied: threadpoolctl>=3.5.0 in /usr/local/lib/python3.12/dist-packages (from scikit-learn>=0.22.0->sentence-transformers) (3.6.0)\n",
+ "Requirement already satisfied: greenlet>=1 in /usr/local/lib/python3.12/dist-packages (from sqlalchemy<3.0.0,>=1.4.0->langchain-classic) (3.5.3)\n",
+ "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy>=1.13.3->torch) (1.3.0)\n",
+ "Requirement already satisfied: regex>=2025.10.22 in /usr/local/lib/python3.12/dist-packages (from transformers<6.0.0,>=4.41.0->sentence-transformers) (2026.7.19)\n",
+ "Requirement already satisfied: typer in /usr/local/lib/python3.12/dist-packages (from transformers<6.0.0,>=4.41.0->sentence-transformers) (0.27.1)\n",
+ "Requirement already satisfied: safetensors>=0.8.0 in /usr/local/lib/python3.12/dist-packages (from transformers<6.0.0,>=4.41.0->sentence-transformers) (0.8.0)\n",
+ "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.12/dist-packages (from jinja2->torch) (3.0.3)\n",
+ "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2025.2)\n",
+ "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2026.3)\n",
+ "Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (2.7.1)\n",
+ "Requirement already satisfied: aiosignal>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.4.0)\n",
+ "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (26.1.0)\n",
+ "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.8.0)\n",
+ "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (6.7.1)\n",
+ "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (0.5.2)\n",
+ "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.24.2)\n",
+ "Requirement already satisfied: ormsgpack>=1.12.0 in /usr/local/lib/python3.12/dist-packages (from langgraph-checkpoint<5.0.0,>=4.1.0->langgraph<1.3.0,>=1.2.5->langchain) (1.12.2)\n",
+ "Requirement already satisfied: jiter<1,>=0.10.0 in /usr/local/lib/python3.12/dist-packages (from openai<3.0.0,>=2.45.0->langchain-openai>=0.3.11->langchain-pinecone) (0.16.0)\n",
+ "Collecting packaging>=23.2.0 (from langchain-core)\n",
+ " Using cached packaging-24.2-py3-none-any.whl.metadata (3.2 kB)\n",
+ "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from typer->transformers<6.0.0,>=4.41.0->sentence-transformers) (1.5.4)\n",
+ "Requirement already satisfied: rich>=13.8.0 in /usr/local/lib/python3.12/dist-packages (from typer->transformers<6.0.0,>=4.41.0->sentence-transformers) (15.0.0)\n",
+ "Requirement already satisfied: annotated-doc>=0.0.2 in /usr/local/lib/python3.12/dist-packages (from typer->transformers<6.0.0,>=4.41.0->sentence-transformers) (0.0.5)\n",
+ "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.8.0->typer->transformers<6.0.0,>=4.41.0->sentence-transformers) (4.2.0)\n",
+ "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.8.0->typer->transformers<6.0.0,>=4.41.0->sentence-transformers) (2.20.0)\n",
+ "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich>=13.8.0->typer->transformers<6.0.0,>=4.41.0->sentence-transformers) (0.1.2)\n",
+ "Downloading fsspec-2026.6.0-py3-none-any.whl (203 kB)\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m203.9/203.9 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25hUsing cached packaging-24.2-py3-none-any.whl (65 kB)\n",
+ "Installing collected packages: packaging, fsspec\n",
+ " Attempting uninstall: packaging\n",
+ " Found existing installation: packaging 26.2\n",
+ " Uninstalling packaging-26.2:\n",
+ " Successfully uninstalled packaging-26.2\n",
+ " Attempting uninstall: fsspec\n",
+ " Found existing installation: fsspec 2026.7.0\n",
+ " Uninstalling fsspec-2026.7.0:\n",
+ " Successfully uninstalled fsspec-2026.7.0\n",
+ "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
+ "cuml-cu12 26.2.0 requires cuda-toolkit[cublas,cufft,curand,cusolver,cusparse]==12.*, but you have cuda-toolkit 13.0.3.0 which is incompatible.\n",
+ "pyiceberg 0.11.1 requires rich<15.0.0,>=10.11.0, but you have rich 15.0.0 which is incompatible.\n",
+ "cudf-cu12 26.2.1 requires cuda-toolkit[nvcc,nvrtc]==12.*, but you have cuda-toolkit 13.0.3.0 which is incompatible.\n",
+ "torchvision 0.26.0+cu128 requires torch==2.11.0, but you have torch 2.13.0 which is incompatible.\n",
+ "bigframes 2.42.0 requires rich<14,>=12.4.4, but you have rich 15.0.0 which is incompatible.\n",
+ "gcsfs 2025.3.0 requires fsspec==2025.3.0, but you have fsspec 2026.6.0 which is incompatible.\u001b[0m\u001b[31m\n",
+ "\u001b[0mSuccessfully installed fsspec-2026.6.0 packaging-24.2\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "application/vnd.colab-display-data+json": {
+ "pip_warning": {
+ "packages": [
+ "fsspec",
+ "packaging"
+ ]
+ },
+ "id": "d304d170d60e4ed6a077eee1cc425742"
+ }
+ },
+ "metadata": {}
+ }
+ ],
+ "source": [
+ "!pip install -U langchain langchain-core langchain-classic langchain-pinecone langchain-huggingface datasets pinecone-client sentence-transformers torch"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "ZheZeF_WsSge",
+ "metadata": {
+ "id": "ZheZeF_WsSge"
+ },
+ "outputs": [],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "from pinecone import Pinecone, ServerlessSpec\n",
+ "\n",
+ "# Clear PINECONE_API_KEY from environment to ensure re-prompting if needed\n",
+ "if \"PINECONE_API_KEY\" in os.environ:\n",
+ " del os.environ[\"PINECONE_API_KEY\"]\n",
+ "\n",
+ "# Prompt for API key if not set"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "VfNIPuXrfkPZ",
+ "metadata": {
+ "id": "VfNIPuXrfkPZ",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "42cd2ce8-1169-4eea-954c-e03174bf6e82"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Environment ready ✅\n"
+ ]
+ }
+ ],
+ "source": [
+ "from google.colab import userdata\n",
+ "\n",
+ "\n",
+ "def load_colab_secret(secret_name: str) -> None:\n",
+ " \"\"\"Load a Colab secret into an environment variable.\"\"\"\n",
+ " try:\n",
+ " secret_value = userdata.get(secret_name)\n",
+ "\n",
+ " if not secret_value:\n",
+ " raise ValueError(\"Secret is empty.\")\n",
+ "\n",
+ " os.environ[secret_name] = secret_value\n",
+ "\n",
+ " except Exception as exc:\n",
+ " raise RuntimeError(\n",
+ " f\"Could not access {secret_name}. \"\n",
+ " \"Make sure it exists in Colab's Secrets tab and \"\n",
+ " \"that notebook access is enabled.\"\n",
+ " ) from exc\n",
+ "\n",
+ "\n",
+ "load_colab_secret(\"OPENAI_API_KEY\")\n",
+ "load_colab_secret(\"PINECONE_API_KEY\")\n",
+ "load_colab_secret(\"HF_TOKEN\")\n",
+ "\n",
+ "print(\"Environment ready ✅\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "load_colab_secret(\"OPENAI_API_KEY\")\n",
+ "load_colab_secret(\"PINECONE_API_KEY\")\n",
+ "load_colab_secret(\"HF_TOKEN\")\n",
+ "\n",
+ "pinecone_api_key = os.environ.get(\"PINECONE_API_KEY\") # ← ajoute ça\n",
+ "\n",
+ "print(\"Environment ready ✅\")"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "hkrWvxk59Rjt",
+ "outputId": "9f1c63b9-9d2a-48b3-d555-60fd60864210"
+ },
+ "id": "hkrWvxk59Rjt",
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Environment ready ✅\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "29ad3840",
+ "metadata": {
+ "id": "29ad3840"
+ },
+ "source": [
+ "# Load Dataset"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "hgIieQukgagu",
+ "metadata": {
+ "id": "hgIieQukgagu"
+ },
+ "source": [
+ "Now let's load the SQUAD dataset from the HuggingFace Model Hub. We load the dataset into a pandas dataframe and filter the title, question, and context columns, and we drop any duplicate context passages."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "J250IJeh7NIb",
+ "metadata": {
+ "id": "J250IJeh7NIb"
+ },
+ "outputs": [],
+ "source": [
+ "from datasets import load_dataset\n",
+ "\n",
+ "# load the squad dataset into a pandas dataframe\n",
+ "\n",
+ "df = load_dataset(\"rajpurkar/squad\", split=\"train\").to_pandas()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "FcmeNO97dHDO",
+ "metadata": {
+ "id": "FcmeNO97dHDO",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 597
+ },
+ "outputId": "222c2cd3-3f13-45c9-f7bd-724f49186353"
+ },
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ " title \\\n",
+ "0 University_of_Notre_Dame \n",
+ "5 University_of_Notre_Dame \n",
+ "10 University_of_Notre_Dame \n",
+ "15 University_of_Notre_Dame \n",
+ "20 University_of_Notre_Dame \n",
+ "... ... \n",
+ "87574 Kathmandu \n",
+ "87579 Kathmandu \n",
+ "87584 Kathmandu \n",
+ "87589 Kathmandu \n",
+ "87594 Kathmandu \n",
+ "\n",
+ " context \n",
+ "0 Architecturally, the school has a Catholic cha... \n",
+ "5 As at most other universities, Notre Dame's st... \n",
+ "10 The university is the major seat of the Congre... \n",
+ "15 The College of Engineering was established in ... \n",
+ "20 All of Notre Dame's undergraduate students are... \n",
+ "... ... \n",
+ "87574 Institute of Medicine, the central college of ... \n",
+ "87579 Football and Cricket are the most popular spor... \n",
+ "87584 The total length of roads in Nepal is recorded... \n",
+ "87589 The main international airport serving Kathman... \n",
+ "87594 Kathmandu Metropolitan City (KMC), in order to... \n",
+ "\n",
+ "[18891 rows x 2 columns]"
+ ],
+ "text/html": [
+ "\n",
+ " \n",
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " title | \n",
+ " context | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " University_of_Notre_Dame | \n",
+ " Architecturally, the school has a Catholic cha... | \n",
+ "
\n",
+ " \n",
+ " | 5 | \n",
+ " University_of_Notre_Dame | \n",
+ " As at most other universities, Notre Dame's st... | \n",
+ "
\n",
+ " \n",
+ " | 10 | \n",
+ " University_of_Notre_Dame | \n",
+ " The university is the major seat of the Congre... | \n",
+ "
\n",
+ " \n",
+ " | 15 | \n",
+ " University_of_Notre_Dame | \n",
+ " The College of Engineering was established in ... | \n",
+ "
\n",
+ " \n",
+ " | 20 | \n",
+ " University_of_Notre_Dame | \n",
+ " All of Notre Dame's undergraduate students are... | \n",
+ "
\n",
+ " \n",
+ " | ... | \n",
+ " ... | \n",
+ " ... | \n",
+ "
\n",
+ " \n",
+ " | 87574 | \n",
+ " Kathmandu | \n",
+ " Institute of Medicine, the central college of ... | \n",
+ "
\n",
+ " \n",
+ " | 87579 | \n",
+ " Kathmandu | \n",
+ " Football and Cricket are the most popular spor... | \n",
+ "
\n",
+ " \n",
+ " | 87584 | \n",
+ " Kathmandu | \n",
+ " The total length of roads in Nepal is recorded... | \n",
+ "
\n",
+ " \n",
+ " | 87589 | \n",
+ " Kathmandu | \n",
+ " The main international airport serving Kathman... | \n",
+ "
\n",
+ " \n",
+ " | 87594 | \n",
+ " Kathmandu | \n",
+ " Kathmandu Metropolitan City (KMC), in order to... | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
18891 rows × 2 columns
\n",
+ "
\n",
+ "
\n",
+ "
\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0msentence_transformers\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSentenceTransformer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0;31m# Set device to GPU if available\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mdevice\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'cuda'\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_available\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m'cpu'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.12/dist-packages/sentence_transformers/__init__.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0mexport_static_quantized_openvino_model\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m )\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0msentence_transformers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbase\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msampler\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mDefaultBatchSampler\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mMultiDatasetDefaultBatchSampler\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m from sentence_transformers.cross_encoder import (\n\u001b[1;32m 17\u001b[0m \u001b[0mCrossEncoder\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.12/dist-packages/sentence_transformers/base/__init__.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mdata_collator\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mBaseDataCollator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mBaseModel\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mpeft_mixin\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mPeftAdapterMixin\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m from .sampler import (\n",
+ "\u001b[0;32m/usr/local/lib/python3.12/dist-packages/sentence_transformers/base/model.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mpackaging\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mversion\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 23\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mtorch\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mtransformers\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mPreTrainedModel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_datasets_available\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_torch_npu_available\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 25\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mtransformers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdynamic_module_utils\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mget_relative_import_files\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 26\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mtransformers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutils\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mlogging\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mtransformers_logging\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m 2428\u001b[0m ) from e\n\u001b[1;32m 2429\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2430\u001b[0;31m raise ModuleNotFoundError(\n\u001b[0m\u001b[1;32m 2431\u001b[0m \u001b[0;34mf\"Could not import module '{name}'. Are this object's requirements defined correctly?\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2432\u001b[0m ) from e\n",
+ "\u001b[0;31mModuleNotFoundError\u001b[0m: Could not import module 'PreTrainedModel'. Are this object's requirements defined correctly?",
+ "",
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
+ ],
+ "errorDetails": {
+ "actions": [
+ {
+ "action": "open_url",
+ "actionText": "Open Examples",
+ "url": "/notebooks/snippets/importing_libraries.ipynb"
+ }
+ ]
+ }
+ }
+ ],
+ "source": [
+ "import torch\n",
+ "from sentence_transformers import SentenceTransformer\n",
+ "\n",
+ "# Set device to GPU if available\n",
+ "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
+ "print(f\"Using device: {device}\")\n",
+ "\n",
+ "# Load retriever model (multi-qa-MiniLM-L6-cos-v1 - 384 dimensions)\n",
+ "retriever = SentenceTransformer('multi-qa-MiniLM-L6-cos-v1', device=device) #use the 'multi-qa-MiniLM-L6-cos-v1' model from HuggingFace to build the retriever\n",
+ "\n",
+ "# ✅ Now retriever is loaded and ready\n",
+ "print(\"Retriever model loaded:\", retriever)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8aaad0a2",
+ "metadata": {
+ "id": "8aaad0a2"
+ },
+ "source": [
+ "# Generate Embeddings and Upsert"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "Hgy7AagJtO_p",
+ "metadata": {
+ "id": "Hgy7AagJtO_p"
+ },
+ "source": [
+ "Next, we need to generate embeddings for the context passages. We will do this in batches to help us more quickly generate embeddings and upload them to the Pinecone index. When passing the documents to Pinecone, we need an id (a unique value), context embedding, and metadata for each document representing context passages in the dataset. The metadata is a dictionary containing data relevant to our embeddings, such as the article title, context passage, etc."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "id": "a17824ef",
+ "metadata": {
+ "id": "a17824ef",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 301,
+ "referenced_widgets": [
+ "2608c03bf0204fd59c849121f510bdcf",
+ "a5d9beda38c14a46be36ce8914039b4c",
+ "4dc401302a104b129a79f5548d9b846f",
+ "c1c3a28a52b84aaa9c1fa8e6bb34a150",
+ "b326b4a9d21f44f284663bbd82a170f5",
+ "16831727859b43068bb32a6d823400b4",
+ "006ae383a6564077a84d89fbe895d97b",
+ "19189531c6ac49548289add512ce6400",
+ "c6ed5a0bd13a4be1a0753fa554947a1f",
+ "55b350e0ab0941bbb9219b7540cd7a04",
+ "4649a2c96c8b48018312df14f451c12d"
+ ]
+ },
+ "outputId": "b85f13e7-1cc5-45af-bb21-6f3307b2e2b0"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ " 0%| | 0/296 [00:00\u001b[0;34m()\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;31m# generate embeddings for batch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0memb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mretriever\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"context\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtolist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtolist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0;31m# get metadata\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mNameError\u001b[0m: name 'retriever' is not defined"
+ ]
+ }
+ ],
+ "source": [
+ "from tqdm.auto import tqdm\n",
+ "\n",
+ "# we will use batches of 64\n",
+ "batch_size = 64\n",
+ "\n",
+ "for i in tqdm(range(0, len(df), batch_size)):\n",
+ " # find end of batch\n",
+ " i_end = min(i + batch_size, len(df))\n",
+ " # extract batch\n",
+ " batch = df.iloc[i:i_end]\n",
+ "\n",
+ " # generate embeddings for batch\n",
+ " emb = retriever.encode(batch[\"context\"].tolist()).tolist()\n",
+ "\n",
+ " # get metadata\n",
+ " meta = [{\"title\": row[\"title\"], \"context\": row[\"context\"]} for _, row in batch.iterrows()]\n",
+ " # create unique IDs\n",
+ " ids = [str(x) for x in range(i, i_end)]\n",
+ " # add all to upsert list\n",
+ " to_upsert = list(zip(ids, emb, meta))\n",
+ " # upsert/insert these records to pinecone\n",
+ " _ = index.upsert(vectors=to_upsert)\n",
+ "\n",
+ "# check that we have all vectors in index\n",
+ "index.describe_index_stats()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "YFyBYafuJ0y0",
+ "metadata": {
+ "id": "YFyBYafuJ0y0"
+ },
+ "source": [
+ "# Initialize Reader"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "HgdiLCz5ynOk",
+ "metadata": {
+ "id": "HgdiLCz5ynOk"
+ },
+ "source": [
+ "We use the `deepset/electra-base-squad2` model from the HuggingFace model hub as our reader model. We load this model into a \"question-answering\" pipeline from HuggingFace transformers and feed it our questions and context passages individually. The model gives a prediction for each context we pass through the pipeline."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "hg9XTDkIJzH_",
+ "metadata": {
+ "id": "hg9XTDkIJzH_"
+ },
+ "outputs": [],
+ "source": [
+ "from transformers import pipeline\n",
+ "\n",
+ "model_name = 'deepset/electra-base-squad2'\n",
+ "# load the reader model into a question-answering pipeline\n",
+ "reader = pipeline(tokenizer=model_name, model=model_name, task='question-answering', device=device)\n",
+ "reader"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e14d89d6",
+ "metadata": {
+ "id": "e14d89d6"
+ },
+ "source": [
+ "Now all the components we need are ready. Let's write some helper functions to execute our queries. The `get_context` function retrieves the context embeddings containing answers to our question from the Pinecone index, and the `extract_answer` function extracts the answers from these context passages."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c190cd87",
+ "metadata": {
+ "id": "c190cd87"
+ },
+ "outputs": [],
+ "source": [
+ "# gets context passages from the pinecone index\n",
+ "def get_context(question, top_k):\n",
+ " # generate embeddings for the question\n",
+ " xq = retriever.encode(question).tolist()\n",
+ " # search pinecone index for context passage with the answer\n",
+ " xc = index.query(vector=xq, top_k=top_k, include_metadata=True)\n",
+ " # extract the context passage from pinecone search result\n",
+ " c = [x[\"metadata\"][\"context\"] for x in xc[\"matches\"]]\n",
+ " return c\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9b7780b7",
+ "metadata": {
+ "id": "9b7780b7"
+ },
+ "outputs": [],
+ "source": [
+ "from pprint import pprint\n",
+ "\n",
+ "# extracts answer from the context passage\n",
+ "def extract_answer(question, context):\n",
+ " results = []\n",
+ " for c in context:\n",
+ " # feed the reader the question and contexts to extract answers\n",
+ " answer = reader(question=question, context=c)\n",
+ " # add the context to answer dict for printing both together\n",
+ " answer[\"context\"] = c\n",
+ " results.append(answer)\n",
+ " # sort the result based on the score from reader model\n",
+ " sorted_result = pprint(sorted(results, key=lambda x: x['score'], reverse=True))\n",
+ " return sorted_result\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5E3a3dkJ5ZQD",
+ "metadata": {
+ "id": "5E3a3dkJ5ZQD"
+ },
+ "outputs": [],
+ "source": [
+ "question = \"How much oil is Egypt producing in a day?\"\n",
+ "context = get_context(question, top_k = 3)\n",
+ "context"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "heKNVbWQ_LtC",
+ "metadata": {
+ "id": "heKNVbWQ_LtC"
+ },
+ "source": [
+ "As we can see, the retiever is working fine and gets us the context passage that contains the answer to our question. Now let's use the reader to extract the exact answer from the context passage."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "DQ4GWdbMSjPl",
+ "metadata": {
+ "id": "DQ4GWdbMSjPl"
+ },
+ "outputs": [],
+ "source": [
+ "extract_answer(question, context)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fMD_ABuDAyhN",
+ "metadata": {
+ "id": "fMD_ABuDAyhN"
+ },
+ "source": [
+ "The reader model predicted with 99% accuracy the correct answer *691,000 bbl/d* as seen from the context passage. Let's run few more queries."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "_4NRgV4mGWoj",
+ "metadata": {
+ "id": "_4NRgV4mGWoj"
+ },
+ "outputs": [],
+ "source": [
+ "question = \"What are the first names of the men that invented youtube?\"\n",
+ "context = get_context(question, top_k=1)\n",
+ "extract_answer(question, context)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "juXlctWgJgMF",
+ "metadata": {
+ "id": "juXlctWgJgMF"
+ },
+ "outputs": [],
+ "source": [
+ "question = \"What is Albert Eistein famous for?\"\n",
+ "context = get_context(question, top_k=1)\n",
+ "extract_answer(question, context)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "OhCgeny_BVno",
+ "metadata": {
+ "id": "OhCgeny_BVno"
+ },
+ "source": [
+ "Let's run another question. This time for top 3 context passages from the retriever."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "iXACn71xmett",
+ "metadata": {
+ "id": "iXACn71xmett"
+ },
+ "outputs": [],
+ "source": [
+ "question = \"Who was the first person to step foot on the moon?\"\n",
+ "context = get_context(question, top_k=3)\n",
+ "extract_answer(question, context)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c9oCWHy0tPpV",
+ "metadata": {
+ "id": "c9oCWHy0tPpV"
+ },
+ "source": [
+ "The result looks pretty good."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d83f9f55-b099-4280-a12e-1d0192f4f5aa",
+ "metadata": {
+ "id": "d83f9f55-b099-4280-a12e-1d0192f4f5aa"
+ },
+ "outputs": [],
+ "source": [
+ "pc.delete_index(index_name)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "27fecadc-6976-43b2-90e1-788a8633ecc7",
+ "metadata": {
+ "id": "27fecadc-6976-43b2-90e1-788a8633ecc7"
+ },
+ "source": [
+ "### Add a few more questions. What did you observe?"
+ ]
+ }
+ ],
+ "metadata": {
+ "accelerator": "GPU",
+ "colab": {
+ "provenance": []
+ },
+ "environment": {
+ "name": "tf2-gpu.2-3.m65",
+ "type": "gcloud",
+ "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m65"
+ },
+ "gpuClass": "standard",
+ "kernelspec": {
+ "display_name": "langchain_env",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.19"
+ },
+ "papermill": {
+ "default_parameters": {},
+ "duration": 333.240754,
+ "end_time": "2021-04-15T21:12:11.363566",
+ "environment_variables": {},
+ "exception": null,
+ "input_path": "/notebooks/question_answering/question_answering.ipynb",
+ "output_path": "/notebooks/tmp/question_answering/question_answering.ipynb",
+ "parameters": {},
+ "start_time": "2021-04-15T21:06:38.122812",
+ "version": "2.3.3"
+ },
+ "widgets": {
+ "application/vnd.jupyter.widget-state+json": {
+ "version_major": 2,
+ "version_minor": 0,
+ "state": {
+ "2608c03bf0204fd59c849121f510bdcf": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "HBoxModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "HBoxModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "HBoxView",
+ "box_style": "",
+ "children": [
+ "IPY_MODEL_a5d9beda38c14a46be36ce8914039b4c",
+ "IPY_MODEL_4dc401302a104b129a79f5548d9b846f",
+ "IPY_MODEL_c1c3a28a52b84aaa9c1fa8e6bb34a150"
+ ],
+ "layout": "IPY_MODEL_b326b4a9d21f44f284663bbd82a170f5"
+ }
+ },
+ "a5d9beda38c14a46be36ce8914039b4c": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "HTMLModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "HTMLModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "HTMLView",
+ "description": "",
+ "description_tooltip": null,
+ "layout": "IPY_MODEL_16831727859b43068bb32a6d823400b4",
+ "placeholder": "",
+ "style": "IPY_MODEL_006ae383a6564077a84d89fbe895d97b",
+ "value": " 0%"
+ }
+ },
+ "4dc401302a104b129a79f5548d9b846f": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "FloatProgressModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "FloatProgressModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "ProgressView",
+ "bar_style": "danger",
+ "description": "",
+ "description_tooltip": null,
+ "layout": "IPY_MODEL_19189531c6ac49548289add512ce6400",
+ "max": 296,
+ "min": 0,
+ "orientation": "horizontal",
+ "style": "IPY_MODEL_c6ed5a0bd13a4be1a0753fa554947a1f",
+ "value": 0
+ }
+ },
+ "c1c3a28a52b84aaa9c1fa8e6bb34a150": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "HTMLModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "HTMLModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "HTMLView",
+ "description": "",
+ "description_tooltip": null,
+ "layout": "IPY_MODEL_55b350e0ab0941bbb9219b7540cd7a04",
+ "placeholder": "",
+ "style": "IPY_MODEL_4649a2c96c8b48018312df14f451c12d",
+ "value": " 0/296 [00:00<?, ?it/s]"
+ }
+ },
+ "b326b4a9d21f44f284663bbd82a170f5": {
+ "model_module": "@jupyter-widgets/base",
+ "model_name": "LayoutModel",
+ "model_module_version": "1.2.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/base",
+ "_model_module_version": "1.2.0",
+ "_model_name": "LayoutModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "LayoutView",
+ "align_content": null,
+ "align_items": null,
+ "align_self": null,
+ "border": null,
+ "bottom": null,
+ "display": null,
+ "flex": null,
+ "flex_flow": null,
+ "grid_area": null,
+ "grid_auto_columns": null,
+ "grid_auto_flow": null,
+ "grid_auto_rows": null,
+ "grid_column": null,
+ "grid_gap": null,
+ "grid_row": null,
+ "grid_template_areas": null,
+ "grid_template_columns": null,
+ "grid_template_rows": null,
+ "height": null,
+ "justify_content": null,
+ "justify_items": null,
+ "left": null,
+ "margin": null,
+ "max_height": null,
+ "max_width": null,
+ "min_height": null,
+ "min_width": null,
+ "object_fit": null,
+ "object_position": null,
+ "order": null,
+ "overflow": null,
+ "overflow_x": null,
+ "overflow_y": null,
+ "padding": null,
+ "right": null,
+ "top": null,
+ "visibility": null,
+ "width": null
+ }
+ },
+ "16831727859b43068bb32a6d823400b4": {
+ "model_module": "@jupyter-widgets/base",
+ "model_name": "LayoutModel",
+ "model_module_version": "1.2.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/base",
+ "_model_module_version": "1.2.0",
+ "_model_name": "LayoutModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "LayoutView",
+ "align_content": null,
+ "align_items": null,
+ "align_self": null,
+ "border": null,
+ "bottom": null,
+ "display": null,
+ "flex": null,
+ "flex_flow": null,
+ "grid_area": null,
+ "grid_auto_columns": null,
+ "grid_auto_flow": null,
+ "grid_auto_rows": null,
+ "grid_column": null,
+ "grid_gap": null,
+ "grid_row": null,
+ "grid_template_areas": null,
+ "grid_template_columns": null,
+ "grid_template_rows": null,
+ "height": null,
+ "justify_content": null,
+ "justify_items": null,
+ "left": null,
+ "margin": null,
+ "max_height": null,
+ "max_width": null,
+ "min_height": null,
+ "min_width": null,
+ "object_fit": null,
+ "object_position": null,
+ "order": null,
+ "overflow": null,
+ "overflow_x": null,
+ "overflow_y": null,
+ "padding": null,
+ "right": null,
+ "top": null,
+ "visibility": null,
+ "width": null
+ }
+ },
+ "006ae383a6564077a84d89fbe895d97b": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "DescriptionStyleModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "DescriptionStyleModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "StyleView",
+ "description_width": ""
+ }
+ },
+ "19189531c6ac49548289add512ce6400": {
+ "model_module": "@jupyter-widgets/base",
+ "model_name": "LayoutModel",
+ "model_module_version": "1.2.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/base",
+ "_model_module_version": "1.2.0",
+ "_model_name": "LayoutModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "LayoutView",
+ "align_content": null,
+ "align_items": null,
+ "align_self": null,
+ "border": null,
+ "bottom": null,
+ "display": null,
+ "flex": null,
+ "flex_flow": null,
+ "grid_area": null,
+ "grid_auto_columns": null,
+ "grid_auto_flow": null,
+ "grid_auto_rows": null,
+ "grid_column": null,
+ "grid_gap": null,
+ "grid_row": null,
+ "grid_template_areas": null,
+ "grid_template_columns": null,
+ "grid_template_rows": null,
+ "height": null,
+ "justify_content": null,
+ "justify_items": null,
+ "left": null,
+ "margin": null,
+ "max_height": null,
+ "max_width": null,
+ "min_height": null,
+ "min_width": null,
+ "object_fit": null,
+ "object_position": null,
+ "order": null,
+ "overflow": null,
+ "overflow_x": null,
+ "overflow_y": null,
+ "padding": null,
+ "right": null,
+ "top": null,
+ "visibility": null,
+ "width": null
+ }
+ },
+ "c6ed5a0bd13a4be1a0753fa554947a1f": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "ProgressStyleModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "ProgressStyleModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "StyleView",
+ "bar_color": null,
+ "description_width": ""
+ }
+ },
+ "55b350e0ab0941bbb9219b7540cd7a04": {
+ "model_module": "@jupyter-widgets/base",
+ "model_name": "LayoutModel",
+ "model_module_version": "1.2.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/base",
+ "_model_module_version": "1.2.0",
+ "_model_name": "LayoutModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "LayoutView",
+ "align_content": null,
+ "align_items": null,
+ "align_self": null,
+ "border": null,
+ "bottom": null,
+ "display": null,
+ "flex": null,
+ "flex_flow": null,
+ "grid_area": null,
+ "grid_auto_columns": null,
+ "grid_auto_flow": null,
+ "grid_auto_rows": null,
+ "grid_column": null,
+ "grid_gap": null,
+ "grid_row": null,
+ "grid_template_areas": null,
+ "grid_template_columns": null,
+ "grid_template_rows": null,
+ "height": null,
+ "justify_content": null,
+ "justify_items": null,
+ "left": null,
+ "margin": null,
+ "max_height": null,
+ "max_width": null,
+ "min_height": null,
+ "min_width": null,
+ "object_fit": null,
+ "object_position": null,
+ "order": null,
+ "overflow": null,
+ "overflow_x": null,
+ "overflow_y": null,
+ "padding": null,
+ "right": null,
+ "top": null,
+ "visibility": null,
+ "width": null
+ }
+ },
+ "4649a2c96c8b48018312df14f451c12d": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "DescriptionStyleModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "DescriptionStyleModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/base",
+ "_view_module_version": "1.2.0",
+ "_view_name": "StyleView",
+ "description_width": ""
+ }
+ }
+ }
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
|