Added support for llms.txt and llms-full.txt

comet-ml · Dec 30, 2024 · 4e723eb · 4e723eb
1 parent b61bfd4
commit 4e723eb
Show file tree

Hide file tree

Showing 76 changed files with 235 additions and 215 deletions.
diff --git a/apps/opik-documentation/documentation/docs/changelog.md b/apps/opik-documentation/documentation/docs/changelog.md
@@ -1,10 +1,33 @@
 ---
-sidebar_position: 101
 sidebar_label: Changelog
+description: Weelkly changelog for Opik
 ---
 
 # Weekly Changelog
 
+## Week of 2024-12-30
+
+**Opik Dashboard**:
+
+- Added duration chart to the project dashboard
+- Prompt metadata can now be set and viewed in the UI, this can be used to store any additional information about the prompt
+- Playground prompts and settings are now cached when you navigate away from the page
+
+**SDK**:
+
+- Introduced a new `OPIK_TRACK_DISABLE` environment variable to disable the tracking of traces and spans
+- We now log usage information for traces logged using the LlamaIndex integration
+
+## Week of 2024-12-23
+
+**SDK**:
+
+- Improved error messages when getting a rate limit when using the `evaluate` method
+- Added support for a new metadata field in the `Prompt` object, this field is used to store any additional information about the prompt.
+- Updated the library used to create uuidv7 IDs
+- New Guardrails integration
+- New DSPY integration
+
 ## Week of 2024-12-16
 
 **Opik Dashboard**:

diff --git a/apps/opik-documentation/documentation/docs/cookbook/aisuite.ipynb b/apps/opik-documentation/documentation/docs/cookbook/aisuite.ipynb
@@ -1,5 +1,14 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "description: Cookbook that showcases Opik's integration with the aisuite Python SDK\n",
+    "---"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

diff --git a/apps/opik-documentation/documentation/docs/cookbook/anthropic.ipynb b/apps/opik-documentation/documentation/docs/cookbook/anthropic.ipynb
@@ -1,5 +1,14 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "description: Cookbook that showcases Opik's integration with the Anthropic Python SDK\n",
+    "---"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

diff --git a/apps/opik-documentation/documentation/docs/cookbook/bedrock.ipynb b/apps/opik-documentation/documentation/docs/cookbook/bedrock.ipynb
@@ -4,6 +4,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "---\n",
+    "description: Cookbook that showcases Opik's integration with AWS Bedrock\n",
+    "---\n",
+    "\n",
     "# Using Opik with AWS Bedrock\n",
     "\n",
     "Opik integrates with AWS Bedrock to provide a simple way to log traces for all Bedrock LLM calls. This works for all supported models, including if you are using the streaming API.\n"

diff --git a/apps/opik-documentation/documentation/docs/cookbook/evaluate_hallucination_metric.ipynb b/apps/opik-documentation/documentation/docs/cookbook/evaluate_hallucination_metric.ipynb
@@ -1,5 +1,15 @@
 {
  "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "sidebar_label: Evaluating Opik's Hallucination Metric\n",
+    "description: Cookbook that evalutes Opik's Hallucination Metric, showcasing both how to use the `evaluation` functionality in the platform as well as the quality of the Hallucination metric included in the SDK. It is a complex example that doesn't always align with how the `evaluate` function works.\n",
+    "---"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -51,7 +61,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -73,17 +83,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "OPIK: Created a \"HaluEval\" dataset at https://www.comet.com/opik/jacques-comet/redirect/datasets?name=HaluEval.\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "# Create dataset\n",
     "import opik\n",
@@ -128,70 +130,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Evaluation: 100%|██████████| 50/50 [00:07<00:00,  6.37it/s]\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">╭─ HaluEval (50 samples) ───────────────────╮\n",
-       "│                                           │\n",
-       "│ <span style=\"font-weight: bold\">Total time:       </span> 00:00:08               │\n",
-       "│ <span style=\"font-weight: bold\">Number of samples:</span> 50                     │\n",
-       "│                                           │\n",
-       "│ <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Correct hallucination score: 0.8000 (avg)</span> │\n",
-       "│                                           │\n",
-       "╰───────────────────────────────────────────╯\n",
-       "</pre>\n"
-      ],
-      "text/plain": [
-       "╭─ HaluEval (50 samples) ───────────────────╮\n",
-       "│                                           │\n",
-       "│ \u001b[1mTotal time:       \u001b[0m 00:00:08               │\n",
-       "│ \u001b[1mNumber of samples:\u001b[0m 50                     │\n",
-       "│                                           │\n",
-       "│ \u001b[1;32mCorrect hallucination score: 0.8000 (avg)\u001b[0m │\n",
-       "│                                           │\n",
-       "╰───────────────────────────────────────────╯\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Uploading results to Opik <span style=\"color: #808000; text-decoration-color: #808000\">...</span> \n",
-       "</pre>\n"
-      ],
-      "text/plain": [
-       "Uploading results to Opik \u001b[33m...\u001b[0m \n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">View the results <a href=\"https://www.comet.com/opik/jacques-comet/experiments/0193aaa8-7d37-7512-8450-0ed502c074e4/compare?experiments=%5B%2206756b2b-4988-7701-8000-8c2b063aa4dc%22%5D\" target=\"_blank\">in your Opik dashboard</a>.\n",
-       "</pre>\n"
-      ],
-      "text/plain": [
-       "View the results \u001b]8;id=436842;https://www.comet.com/opik/jacques-comet/experiments/0193aaa8-7d37-7512-8450-0ed502c074e4/compare?experiments=%5B%2206756b2b-4988-7701-8000-8c2b063aa4dc%22%5D\u001b\\in your Opik dashboard\u001b]8;;\u001b\\.\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
+   "outputs": [],
    "source": [
     "from opik.evaluation.metrics import Hallucination, Equals\n",
     "from opik.evaluation import evaluate\n",

diff --git a/apps/opik-documentation/documentation/docs/cookbook/evaluate_moderation_metric.ipynb b/apps/opik-documentation/documentation/docs/cookbook/evaluate_moderation_metric.ipynb
@@ -4,6 +4,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "---\n",
+    "sidebar_label: Evaluate Opik's Moderation Metric\n",
+    "description: Cookbook that evalutes Opik's Moderation Metric, showcasing both how to use the `evaluation` functionality in the platform as well as the quality of the Modelration metric included in the SDK. It is a complex example that doesn't always align with how the `evaluate` function works.\n",
+    "---\n",
+    "\n",
     "# Evaluating Opik's Moderation Metric\n",
     "\n",
     "For this guide we will be evaluating the Moderation metric included in the LLM Evaluation SDK which will showcase both how to use the `evaluation` functionality in the platform as well as the quality of the Moderation metric included in the SDK."
@@ -22,35 +27,18 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Note: you may need to restart the kernel to use updated packages.\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "%pip install --upgrade --quiet opik pandas"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "OPIK: Your Opik API key is available in your account settings, can be found at https://www.comet.com/api/my/settings/ for Opik cloud\n",
-      "OPIK: Configuration saved to file: /Users/jacquesverre/.opik.config\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "import opik\n",
     "\n",
@@ -68,7 +56,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -90,7 +78,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -149,73 +137,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Evaluation:   0%|          | 0/50 [00:00<?, ?it/s]OPIK: Started logging traces to the \"Default Project\" project at https://www.comet.com/opik/jacques-comet/redirect/projects?name=Default%20Project.\n",
-      "OPIK: Started logging traces to the \"Default Project\" project at https://www.comet.com/opik/jacques-comet/redirect/projects?name=Default%20Project.\n",
-      "OPIK: Started logging traces to the \"Default Project\" project at https://www.comet.com/opik/jacques-comet/redirect/projects?name=Default%20Project.\n",
-      "Evaluation: 100%|██████████| 50/50 [00:05<00:00,  8.86it/s]\n"
-     ]
-    },
-    {
-     "data": {
-      "text/html": [
-       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">╭─ OpenAIModerationDataset (50 samples) ─╮\n",
-       "│                                        │\n",
-       "│ <span style=\"font-weight: bold\">Total time:       </span> 00:00:06            │\n",
-       "│ <span style=\"font-weight: bold\">Number of samples:</span> 50                  │\n",
-       "│                                        │\n",
-       "│ <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Correct moderation score: 0.8600 (avg)</span> │\n",
-       "│                                        │\n",
-       "╰────────────────────────────────────────╯\n",
-       "</pre>\n"
-      ],
-      "text/plain": [
-       "╭─ OpenAIModerationDataset (50 samples) ─╮\n",
-       "│                                        │\n",
-       "│ \u001b[1mTotal time:       \u001b[0m 00:00:06            │\n",
-       "│ \u001b[1mNumber of samples:\u001b[0m 50                  │\n",
-       "│                                        │\n",
-       "│ \u001b[1;32mCorrect moderation score: 0.8600 (avg)\u001b[0m │\n",
-       "│                                        │\n",
-       "╰────────────────────────────────────────╯\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Uploading results to Opik <span style=\"color: #808000; text-decoration-color: #808000\">...</span> \n",
-       "</pre>\n"
-      ],
-      "text/plain": [
-       "Uploading results to Opik \u001b[33m...\u001b[0m \n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">View the results <a href=\"https://www.comet.com/opik/jacques-comet/experiments/01939721-3a29-7f9a-ac8b-737923e30e31/compare?experiments=%5B%2206766a9d-2e22-7b5a-8000-afa08cbccbb4%22%5D\" target=\"_blank\">in your Opik dashboard</a>.\n",
-       "</pre>\n"
-      ],
-      "text/plain": [
-       "View the results \u001b]8;id=373788;https://www.comet.com/opik/jacques-comet/experiments/01939721-3a29-7f9a-ac8b-737923e30e31/compare?experiments=%5B%2206766a9d-2e22-7b5a-8000-afa08cbccbb4%22%5D\u001b\\in your Opik dashboard\u001b]8;;\u001b\\.\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
+   "outputs": [],
    "source": [
     "from opik.evaluation.metrics import Moderation, Equals\n",
     "from opik.evaluation import evaluate\n",

diff --git a/apps/opik-documentation/documentation/docs/cookbook/gemini.ipynb b/apps/opik-documentation/documentation/docs/cookbook/gemini.ipynb
@@ -4,6 +4,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "---\n",
+    "description: Cookbook that showcases Opik's integration with the Gemini Python SDK\n",
+    "---\n",
+    "\n",
     "# Using Opik with Gemini\n",
     "\n",
     "Opik integrates with Gemini to provide a simple way to log traces for all Gemini LLM calls. This works for all Gemini models."

diff --git a/apps/opik-documentation/documentation/docs/cookbook/groq.ipynb b/apps/opik-documentation/documentation/docs/cookbook/groq.ipynb
@@ -4,6 +4,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "---\n",
+    "description: Cookbook that showcases Opik's integration with Groq\n",
+    "---\n",
+    "\n",
     "# Using Opik with Groq\n",
     "\n",
     "Opik integrates with Groq to provide a simple way to log traces for all Groq LLM calls. This works for all Groq models."