weightsandbiasees_weave_intro_notebook.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "298501c0",
      "metadata": {
        "id": "298501c0"
      },
      "source": [
        "<img src=\"http://wandb.me/logo-im-png\" width=\"400\" alt=\"Weights & Biases\" />\n",
        "<!--- @wandbcode{intro-colab} -->"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "974b2c96",
      "metadata": {
        "id": "974b2c96"
      },
      "source": [
        "# 🏃‍♀️ Quickstart\n",
        "\n",
        "Get started using Weave to:\n",
        "- Log and debug language model inputs, outputs, and traces\n",
        "- Build rigorous, apples-to-apples evaluations for language model use cases\n",
        "- Organize all the information generated across the LLM workflow, from experimentation to evaluations to production\n",
        "\n",
        "See the full Weave documentation [here](https://wandb.me/weave).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "56dc5c9d",
      "metadata": {
        "id": "56dc5c9d"
      },
      "source": [
        "## 🪄 Install `weave` library and login\n",
        "\n",
        "\n",
        "Start by installing the library and logging in to your account.\n",
        "\n",
        "In this example, we're using openai so you should [add an openai API key](https://platform.openai.com/docs/quickstart/step-2-setup-your-api-key).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "61a6d209",
      "metadata": {
        "id": "61a6d209"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install weave openai set-env-colab-kaggle-dotenv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0921bbef-d089-4960-b624-505172f90d76",
      "metadata": {
        "id": "0921bbef-d089-4960-b624-505172f90d76"
      },
      "outputs": [],
      "source": [
        "# Set your OpenAI API key\n",
        "from set_env import set_env\n",
        "\n",
        "# Put your OPENAI_API_KEY in the secrets panel to the left 🗝️\n",
        "_ = set_env(\"OPENAI_API_KEY\")\n",
        "# os.environ[\"OPENAI_API_KEY\"] = \"sk-...\" # alternatively, put your key here\n",
        "\n",
        "PROJECT = \"weave-intro-notebook\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "09467a33",
      "metadata": {
        "id": "09467a33"
      },
      "source": [
        "# Track inputs & outputs of functions\n",
        "\n",
        "Weave allows users to track function calls: the code, inputs, outputs, and even LLM tokens & costs! In the following sections we will cover:\n",
        "\n",
        "* Custom Functions\n",
        "* Vendor Integrations\n",
        "* Nested Function Calling\n",
        "* Error Tracking\n",
        "\n",
        "Note: in all cases, we will:\n",
        "\n",
        "```python\n",
        "import weave                    # import the weave library\n",
        "weave.init('project-name')      # initialize tracking for a specific W&B project\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "99453d48",
      "metadata": {
        "id": "99453d48"
      },
      "source": [
        "## Track custom functions\n",
        "\n",
        "Add the @weave.op decorator to the functions you want to track"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cdcd1bb7",
      "metadata": {
        "id": "cdcd1bb7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177
        },
        "outputId": "59412082-7ac8-44bd-a37c-e7b544ed22bf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Please login to Weights & Biases (https://wandb.ai/) to continue:\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "        window._wandbApiKey = new Promise((resolve, reject) => {\n",
              "            function loadScript(url) {\n",
              "            return new Promise(function(resolve, reject) {\n",
              "                let newScript = document.createElement(\"script\");\n",
              "                newScript.onerror = reject;\n",
              "                newScript.onload = resolve;\n",
              "                document.body.appendChild(newScript);\n",
              "                newScript.src = url;\n",
              "            });\n",
              "            }\n",
              "            loadScript(\"https://cdn.jsdelivr.net/npm/postmate/build/postmate.min.js\").then(() => {\n",
              "            const iframe = document.createElement('iframe')\n",
              "            iframe.style.cssText = \"width:0;height:0;border:none\"\n",
              "            document.body.appendChild(iframe)\n",
              "            const handshake = new Postmate({\n",
              "                container: iframe,\n",
              "                url: 'https://wandb.ai/authorize'\n",
              "            });\n",
              "            const timeout = setTimeout(() => reject(\"Couldn't auto authenticate\"), 5000)\n",
              "            handshake.then(function(child) {\n",
              "                child.on('authorize', data => {\n",
              "                    clearTimeout(timeout)\n",
              "                    resolve(data)\n",
              "                });\n",
              "            });\n",
              "            })\n",
              "        });\n",
              "    "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Logging into wandb.ai. (Learn how to deploy a W&B server locally: https://wandb.me/wandb-server)\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: You can find your API key in your browser here: https://wandb.ai/authorize\n",
            "wandb: Paste an API key from your profile and hit enter, or press ctrl+c to quit:"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " ··········\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Logged in as Weights & Biases user: rosm.\n",
            "View Weave data at https://wandb.ai/rosmur/weave-intro-notebook/weave\n",
            "That was so easy, it was a piece of cake!\n"
          ]
        }
      ],
      "source": [
        "from openai import OpenAI\n",
        "\n",
        "import weave\n",
        "\n",
        "weave.init(PROJECT)\n",
        "\n",
        "client = OpenAI()\n",
        "response = client.chat.completions.create(\n",
        "    model=\"gpt-3.5-turbo-1106\",\n",
        "    messages=[\n",
        "        {\n",
        "            \"role\": \"system\",\n",
        "            \"content\": \"You are a grammar checker, correct the following user input.\",\n",
        "        },\n",
        "        {\"role\": \"user\", \"content\": \"That was so easy, it was a piece of pie!\"},\n",
        "    ],\n",
        "    temperature=0,\n",
        ")\n",
        "generation = response.choices[0].message.content\n",
        "print(generation)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "174cabef",
      "metadata": {
        "id": "174cabef"
      },
      "source": [
        "You can find your interactive dashboard by clicking any of the  👆 wandb links above."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5ef6f27b",
      "metadata": {
        "id": "5ef6f27b"
      },
      "source": [
        "## Vendor Integrations (OpenAI, Anthropic, Mistral, etc...)\n",
        "\n",
        "Here, we're automatically tracking all calls to `openai`. We automatically track a lot of LLM libraries, but it's really easy to add support for whatever LLM you're using, as you'll see below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ac8a2b59",
      "metadata": {
        "id": "ac8a2b59",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d05e9884-f833-45a5-9a79-0c7df4ae5ea9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "🍩 https://wandb.ai/rosmur/weave-intro-notebook/r/call/dd9003af-a7f4-4081-bc0f-81cabab5905d\n",
            "hello\n"
          ]
        }
      ],
      "source": [
        "import weave\n",
        "\n",
        "weave.init(PROJECT)\n",
        "\n",
        "\n",
        "@weave.op()\n",
        "def strip_user_input(user_input):\n",
        "    return user_input.strip()\n",
        "\n",
        "\n",
        "result = strip_user_input(\"    hello    \")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1ed90e37",
      "metadata": {
        "id": "1ed90e37"
      },
      "source": [
        "After adding `weave.op` and calling the function, visit the link and see it tracked within your project.\n",
        "\n",
        "💡 We automatically track your code, have a look at the code tab!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "35fe43dc",
      "metadata": {
        "id": "35fe43dc"
      },
      "source": [
        "## Track nested functions\n",
        "\n",
        "Now that you've seen the basics, let's combine all of the above and track some deeply nested functions alongside LLM calls.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "422050d0",
      "metadata": {
        "id": "422050d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 329
        },
        "outputId": "013136c1-000e-411c-92ba-d2fd2ee56bd7"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "WeaveInitError",
          "evalue": "You must call `weave.init(<project_name>)` first",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mWeaveInitError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-12-08ff62d03da0>\u001b[0m in \u001b[0;36m<cell line: 32>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 32\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcorrect_grammar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"   That was so easy, it was a piece of pie!    \"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     33\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/trace/op.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     62\u001b[0m         \u001b[0mmaybe_client\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgraph_client_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     63\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mmaybe_client\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 64\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresolve_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     65\u001b[0m         \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtyping\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"WeaveClient\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmaybe_client\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-12-08ff62d03da0>\u001b[0m in \u001b[0;36mcorrect_grammar\u001b[0;34m(user_input)\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m     \u001b[0mstripped\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstrip_user_input\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muser_input\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m     response = client.chat.completions.create(\n\u001b[0m\u001b[1;32m     19\u001b[0m         \u001b[0mmodel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"gpt-3.5-turbo-1106\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m         messages=[\n",
            "\u001b[0;32m/usr/lib/python3.10/functools.py\u001b[0m in \u001b[0;36m_method\u001b[0;34m(cls_or_self, *args, **keywords)\u001b[0m\n\u001b[1;32m    386\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0m_method\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls_or_self\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m/\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    387\u001b[0m             \u001b[0mkeywords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 388\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls_or_self\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    389\u001b[0m         \u001b[0m_method\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__isabstractmethod__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__isabstractmethod__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    390\u001b[0m         \u001b[0m_method\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_partialmethod\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/monitoring/openai/openai.py\u001b[0m in \u001b[0;36mcreate\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    183\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_streaming_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    184\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 185\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    186\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mChatCompletion\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/monitoring/openai/openai.py\u001b[0m in \u001b[0;36m_create\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mChatCompletion\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    188\u001b[0m         \u001b[0mnamed_args\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbind_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mold_create\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 189\u001b[0;31m         \u001b[0;32mwith\u001b[0m \u001b[0mlog_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcreate_op\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnamed_args\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mfinish_run\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    190\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_base_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    191\u001b[0m             \u001b[0mfinish_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_dump\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexclude_unset\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.10/contextlib.py\u001b[0m in \u001b[0;36m__enter__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    133\u001b[0m         \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    134\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 135\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    136\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    137\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"generator didn't yield\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/monitoring/openai/openai.py\u001b[0m in \u001b[0;36mlog_run\u001b[0;34m(call_name, inputs)\u001b[0m\n\u001b[1;32m    257\u001b[0m     \u001b[0mcall_name\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mtyping\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mOp\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    258\u001b[0m ) -> Iterator[Callable]:\n\u001b[0;32m--> 259\u001b[0;31m     \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgraph_client_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequire_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    260\u001b[0m     \u001b[0mparent_run\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrun_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_current_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    261\u001b[0m     \u001b[0;31m# TODO: client should not need refs passed in.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/graph_client_context.py\u001b[0m in \u001b[0;36mrequire_graph_client\u001b[0;34m()\u001b[0m\n\u001b[1;32m     27\u001b[0m     \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mWeaveInitError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"You must call `weave.init(<project_name>)` first\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mWeaveInitError\u001b[0m: You must call `weave.init(<project_name>)` first"
          ]
        }
      ],
      "source": [
        "from openai import OpenAI\n",
        "\n",
        "import weave\n",
        "\n",
        "weave.init(PROJECT)\n",
        "\n",
        "\n",
        "@weave.op()\n",
        "def strip_user_input(user_input):\n",
        "    return user_input.strip()\n",
        "\n",
        "\n",
        "@weave.op()\n",
        "def correct_grammar(user_input):\n",
        "    client = OpenAI()\n",
        "\n",
        "    stripped = strip_user_input(user_input)\n",
        "    response = client.chat.completions.create(\n",
        "        model=\"gpt-3.5-turbo-1106\",\n",
        "        messages=[\n",
        "            {\n",
        "                \"role\": \"system\",\n",
        "                \"content\": \"You are a grammar checker, correct the following user input.\",\n",
        "            },\n",
        "            {\"role\": \"user\", \"content\": stripped},\n",
        "        ],\n",
        "        temperature=0,\n",
        "    )\n",
        "    return response.choices[0].message.content\n",
        "\n",
        "\n",
        "result = correct_grammar(\"   That was so easy, it was a piece of pie!    \")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6f483413",
      "metadata": {
        "id": "6f483413"
      },
      "source": [
        "## Track Errors\n",
        "\n",
        "Whenever your code crashes, weave will highlight what caused the issue. This is especially useful for finding things like JSON parsing issues that can occasionally happen when parsing data from LLM responses."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "14e7e103",
      "metadata": {
        "id": "14e7e103",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 365
        },
        "outputId": "58b126d2-34a1-400b-ab07-03e813256cf9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "🍩 https://wandb.ai/rosmur/weave-intro-notebook/r/call/c8eaa2ce-effa-4451-832c-9ef3ffce343e\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "BadRequestError",
          "evalue": "Error code: 400 - {'error': {'message': \"'messages' must contain the word 'json' in some form, to use 'response_format' of type 'json_object'.\", 'type': 'invalid_request_error', 'param': 'messages', 'code': None}}",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mBadRequestError\u001b[0m                           Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-6-917370b0d565>\u001b[0m in \u001b[0;36m<cell line: 35>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     33\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 35\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcorrect_grammar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"   That was so easy, it was a piece of pie!    \"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     36\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/trace/op.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    106\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    107\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 108\u001b[0;31m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresolve_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    109\u001b[0m             \u001b[0;31m# TODO: can we get rid of this?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    110\u001b[0m             \u001b[0mres\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbox\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbox\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-6-917370b0d565>\u001b[0m in \u001b[0;36mcorrect_grammar\u001b[0;34m(user_input)\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0mstripped\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstrip_user_input\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muser_input\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m     response = client.chat.completions.create(\n\u001b[0m\u001b[1;32m     21\u001b[0m         \u001b[0mmodel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"gpt-3.5-turbo-1106\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m         messages=[\n",
            "\u001b[0;32m/usr/lib/python3.10/functools.py\u001b[0m in \u001b[0;36m_method\u001b[0;34m(cls_or_self, *args, **keywords)\u001b[0m\n\u001b[1;32m    386\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0m_method\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls_or_self\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m/\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    387\u001b[0m             \u001b[0mkeywords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 388\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls_or_self\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    389\u001b[0m         \u001b[0m_method\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__isabstractmethod__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__isabstractmethod__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    390\u001b[0m         \u001b[0m_method\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_partialmethod\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/monitoring/openai/openai.py\u001b[0m in \u001b[0;36mcreate\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    183\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_streaming_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    184\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 185\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    186\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mChatCompletion\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/monitoring/openai/openai.py\u001b[0m in \u001b[0;36m_create\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    188\u001b[0m         \u001b[0mnamed_args\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbind_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mold_create\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mlog_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcreate_op\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnamed_args\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mfinish_run\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 190\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_base_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    191\u001b[0m             \u001b[0mfinish_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_dump\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexclude_unset\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    192\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/openai/_utils/_utils.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    275\u001b[0m                         \u001b[0mmsg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mf\"Missing required argument: {quote(missing[0])}\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    276\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 277\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    278\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    279\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m  \u001b[0;31m# type: ignore\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/openai/resources/chat/completions.py\u001b[0m in \u001b[0;36mcreate\u001b[0;34m(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, parallel_tool_calls, presence_penalty, response_format, seed, service_tier, stop, stream, stream_options, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)\u001b[0m\n\u001b[1;32m    638\u001b[0m         \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mfloat\u001b[0m \u001b[0;34m|\u001b[0m \u001b[0mhttpx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTimeout\u001b[0m \u001b[0;34m|\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;34m|\u001b[0m \u001b[0mNotGiven\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNOT_GIVEN\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    639\u001b[0m     ) -> ChatCompletion | Stream[ChatCompletionChunk]:\n\u001b[0;32m--> 640\u001b[0;31m         return self._post(\n\u001b[0m\u001b[1;32m    641\u001b[0m             \u001b[0;34m\"/chat/completions\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    642\u001b[0m             body=maybe_transform(\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/openai/_base_client.py\u001b[0m in \u001b[0;36mpost\u001b[0;34m(self, path, cast_to, body, options, files, stream, stream_cls)\u001b[0m\n\u001b[1;32m   1248\u001b[0m             \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"post\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjson_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfiles\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mto_httpx_files\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfiles\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1249\u001b[0m         )\n\u001b[0;32m-> 1250\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mResponseT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcast_to\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mopts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstream\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstream\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstream_cls\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstream_cls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1251\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1252\u001b[0m     def patch(\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/openai/_base_client.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, cast_to, options, remaining_retries, stream, stream_cls)\u001b[0m\n\u001b[1;32m    929\u001b[0m         \u001b[0mstream_cls\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0m_StreamT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m|\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    930\u001b[0m     ) -> ResponseT | _StreamT:\n\u001b[0;32m--> 931\u001b[0;31m         return self._request(\n\u001b[0m\u001b[1;32m    932\u001b[0m             \u001b[0mcast_to\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcast_to\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    933\u001b[0m             \u001b[0moptions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/openai/_base_client.py\u001b[0m in \u001b[0;36m_request\u001b[0;34m(self, cast_to, options, remaining_retries, stream, stream_cls)\u001b[0m\n\u001b[1;32m   1028\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1029\u001b[0m             \u001b[0mlog\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Re-raising status error\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1030\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_status_error_from_response\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1031\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1032\u001b[0m         return self._process_response(\n",
            "\u001b[0;31mBadRequestError\u001b[0m: Error code: 400 - {'error': {'message': \"'messages' must contain the word 'json' in some form, to use 'response_format' of type 'json_object'.\", 'type': 'invalid_request_error', 'param': 'messages', 'code': None}}"
          ]
        }
      ],
      "source": [
        "import json\n",
        "\n",
        "from openai import OpenAI\n",
        "\n",
        "import weave\n",
        "\n",
        "weave.init(PROJECT)\n",
        "\n",
        "\n",
        "@weave.op()\n",
        "def strip_user_input(user_input):\n",
        "    return user_input.strip()\n",
        "\n",
        "\n",
        "@weave.op()\n",
        "def correct_grammar(user_input):\n",
        "    client = OpenAI()\n",
        "\n",
        "    stripped = strip_user_input(user_input)\n",
        "    response = client.chat.completions.create(\n",
        "        model=\"gpt-3.5-turbo-1106\",\n",
        "        messages=[\n",
        "            {\n",
        "                \"role\": \"system\",\n",
        "                \"content\": \"You are a grammar checker, correct the following user input.\",\n",
        "            },\n",
        "            {\"role\": \"user\", \"content\": stripped},\n",
        "        ],\n",
        "        temperature=0,\n",
        "        response_format={\"type\": \"json_object\"},\n",
        "    )\n",
        "    return json.loads(response.choices[0].message.content)\n",
        "\n",
        "\n",
        "result = correct_grammar(\"   That was so easy, it was a piece of pie!    \")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f957c9c8",
      "metadata": {
        "id": "f957c9c8"
      },
      "source": [
        "# Tracking Objects\n",
        "\n",
        "Organizing experimentation is difficult when there are many moving pieces. You can capture and organize the experimental details of your app like your system prompt or the model you're using within `weave.Objects`. This helps organize and compare different iterations of your app. In this section, we will cover:\n",
        "\n",
        "* General Object Tracking\n",
        "* Tracking Models\n",
        "* Tracking Datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8f6129cc",
      "metadata": {
        "id": "8f6129cc"
      },
      "source": [
        "## General Object Tracking\n",
        "\n",
        "Many times, it is useful to track & version data, just like you track and version code. For example, here we define a `SystemPrompt(weave.Object)` object that can be shared between teammates"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "08e4e852",
      "metadata": {
        "id": "08e4e852",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 311
        },
        "outputId": "cdd99184-d846-4c5d-cb58-bc76d4ee3d25"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "WeaveInitError",
          "evalue": "You must call `weave.init(<project_name>)` first",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mWeaveInitError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-13-5395c089a246>\u001b[0m in \u001b[0;36m<cell line: 13>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0mprompt\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"You are a grammar checker, correct the following user input.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m )\n\u001b[0;32m---> 13\u001b[0;31m \u001b[0mweave\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpublish\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msystem_prompt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/api.py\u001b[0m in \u001b[0;36mpublish\u001b[0;34m(obj, name)\u001b[0m\n\u001b[1;32m    201\u001b[0m         \u001b[0mA\u001b[0m \u001b[0mweave\u001b[0m \u001b[0mRef\u001b[0m \u001b[0mto\u001b[0m \u001b[0mthe\u001b[0m \u001b[0msaved\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    202\u001b[0m     \"\"\"\n\u001b[0;32m--> 203\u001b[0;31m     \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_graph_client_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequire_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    204\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    205\u001b[0m     \u001b[0msave_name\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/graph_client_context.py\u001b[0m in \u001b[0;36mrequire_graph_client\u001b[0;34m()\u001b[0m\n\u001b[1;32m     27\u001b[0m     \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mWeaveInitError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"You must call `weave.init(<project_name>)` first\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mWeaveInitError\u001b[0m: You must call `weave.init(<project_name>)` first"
          ]
        }
      ],
      "source": [
        "import weave\n",
        "\n",
        "weave.init(PROJECT)\n",
        "\n",
        "\n",
        "class SystemPrompt(weave.Object):\n",
        "    prompt: str\n",
        "\n",
        "\n",
        "system_prompt = SystemPrompt(\n",
        "    prompt=\"You are a grammar checker, correct the following user input.\"\n",
        ")\n",
        "weave.publish(system_prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0c873438",
      "metadata": {
        "id": "0c873438"
      },
      "source": [
        "## Model Tracking\n",
        "\n",
        "Models are so common of an object type, that we have a special class to represent them: `weave.Model`. The only requirement is that we define a `predict` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e3e60d81",
      "metadata": {
        "id": "e3e60d81",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 329
        },
        "outputId": "c88d0459-37b6-4244-cd63-e2d79425f41c"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "WeaveInitError",
          "evalue": "You must call `weave.init(<project_name>)` first",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mWeaveInitError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-9-6bfd774648a1>\u001b[0m in \u001b[0;36m<cell line: 32>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     30\u001b[0m )\n\u001b[1;32m     31\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 32\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcorrector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"     That was so easy, it was a piece of pie!       \"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     33\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/trace/op.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    175\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    176\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 177\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mOp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marg0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    178\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    179\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/trace/op.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     62\u001b[0m         \u001b[0mmaybe_client\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgraph_client_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     63\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mmaybe_client\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 64\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresolve_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     65\u001b[0m         \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtyping\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"WeaveClient\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmaybe_client\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-9-6bfd774648a1>\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, user_input)\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0muser_input\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mOpenAI\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m         response = client.chat.completions.create(\n\u001b[0m\u001b[1;32m     17\u001b[0m             \u001b[0mmodel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopenai_model_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m             messages=[\n",
            "\u001b[0;32m/usr/lib/python3.10/functools.py\u001b[0m in \u001b[0;36m_method\u001b[0;34m(cls_or_self, *args, **keywords)\u001b[0m\n\u001b[1;32m    386\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0m_method\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls_or_self\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m/\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    387\u001b[0m             \u001b[0mkeywords\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 388\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls_or_self\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkeywords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    389\u001b[0m         \u001b[0m_method\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__isabstractmethod__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__isabstractmethod__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    390\u001b[0m         \u001b[0m_method\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_partialmethod\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/monitoring/openai/openai.py\u001b[0m in \u001b[0;36mcreate\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    183\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_streaming_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    184\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 185\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    186\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mChatCompletion\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/monitoring/openai/openai.py\u001b[0m in \u001b[0;36m_create\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mChatCompletion\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    188\u001b[0m         \u001b[0mnamed_args\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbind_params\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mold_create\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 189\u001b[0;31m         \u001b[0;32mwith\u001b[0m \u001b[0mlog_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcreate_op\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnamed_args\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mfinish_run\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    190\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_base_create\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    191\u001b[0m             \u001b[0mfinish_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_dump\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexclude_unset\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.10/contextlib.py\u001b[0m in \u001b[0;36m__enter__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    133\u001b[0m         \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    134\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 135\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    136\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    137\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"generator didn't yield\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/monitoring/openai/openai.py\u001b[0m in \u001b[0;36mlog_run\u001b[0;34m(call_name, inputs)\u001b[0m\n\u001b[1;32m    257\u001b[0m     \u001b[0mcall_name\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mtyping\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mOp\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    258\u001b[0m ) -> Iterator[Callable]:\n\u001b[0;32m--> 259\u001b[0;31m     \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgraph_client_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequire_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    260\u001b[0m     \u001b[0mparent_run\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrun_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_current_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    261\u001b[0m     \u001b[0;31m# TODO: client should not need refs passed in.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/graph_client_context.py\u001b[0m in \u001b[0;36mrequire_graph_client\u001b[0;34m()\u001b[0m\n\u001b[1;32m     27\u001b[0m     \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mWeaveInitError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"You must call `weave.init(<project_name>)` first\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mWeaveInitError\u001b[0m: You must call `weave.init(<project_name>)` first"
          ]
        }
      ],
      "source": [
        "from openai import OpenAI\n",
        "\n",
        "import weave\n",
        "\n",
        "weave.init(PROJECT)\n",
        "\n",
        "\n",
        "class OpenAIGrammarCorrector(weave.Model):\n",
        "    # Properties are entirely user-defined\n",
        "    openai_model_name: str\n",
        "    system_message: str\n",
        "\n",
        "    @weave.op()\n",
        "    def predict(self, user_input):\n",
        "        client = OpenAI()\n",
        "        response = client.chat.completions.create(\n",
        "            model=self.openai_model_name,\n",
        "            messages=[\n",
        "                {\"role\": \"system\", \"content\": self.system_message},\n",
        "                {\"role\": \"user\", \"content\": user_input},\n",
        "            ],\n",
        "            temperature=0,\n",
        "        )\n",
        "        return response.choices[0].message.content\n",
        "\n",
        "\n",
        "corrector = OpenAIGrammarCorrector(\n",
        "    openai_model_name=\"gpt-3.5-turbo-1106\",\n",
        "    system_message=\"You are a grammar checker, correct the following user input.\",\n",
        ")\n",
        "\n",
        "result = corrector.predict(\"     That was so easy, it was a piece of pie!       \")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "624b04c1",
      "metadata": {
        "id": "624b04c1"
      },
      "source": [
        "## Dataset Tracking\n",
        "\n",
        "Similar to models, a `weave.Dataset` object exists to help track, organize, and operate on datasets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ac2c5145",
      "metadata": {
        "id": "ac2c5145",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 311
        },
        "outputId": "cd8d7afd-8d1c-47df-e48f-3877d3293fcf"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "WeaveInitError",
          "evalue": "You must call `weave.init(<project_name>)` first",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mWeaveInitError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-8-18f33a8d67d7>\u001b[0m in \u001b[0;36m<cell line: 16>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     14\u001b[0m )\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m \u001b[0mweave\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpublish\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/api.py\u001b[0m in \u001b[0;36mpublish\u001b[0;34m(obj, name)\u001b[0m\n\u001b[1;32m    201\u001b[0m         \u001b[0mA\u001b[0m \u001b[0mweave\u001b[0m \u001b[0mRef\u001b[0m \u001b[0mto\u001b[0m \u001b[0mthe\u001b[0m \u001b[0msaved\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    202\u001b[0m     \"\"\"\n\u001b[0;32m--> 203\u001b[0;31m     \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_graph_client_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequire_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    204\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    205\u001b[0m     \u001b[0msave_name\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/weave/graph_client_context.py\u001b[0m in \u001b[0;36mrequire_graph_client\u001b[0;34m()\u001b[0m\n\u001b[1;32m     27\u001b[0m     \u001b[0mclient\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_graph_client\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mWeaveInitError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"You must call `weave.init(<project_name>)` first\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mclient\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mWeaveInitError\u001b[0m: You must call `weave.init(<project_name>)` first"
          ]
        }
      ],
      "source": [
        "dataset = weave.Dataset(\n",
        "    name=\"grammar-correction\",\n",
        "    rows=[\n",
        "        {\n",
        "            \"user_input\": \"   That was so easy, it was a piece of pie!   \",\n",
        "            \"expected\": \"That was so easy, it was a piece of cake!\",\n",
        "        },\n",
        "        {\"user_input\": \"  I write good   \", \"expected\": \"I write well\"},\n",
        "        {\n",
        "            \"user_input\": \"  GPT-3 is smartest AI model.   \",\n",
        "            \"expected\": \"GPT-3 is the smartest AI model.\",\n",
        "        },\n",
        "    ],\n",
        ")\n",
        "\n",
        "weave.publish(dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f7d43bc3",
      "metadata": {
        "id": "f7d43bc3"
      },
      "source": [
        "Notice that we saved a versioned `GrammarCorrector` object that captures the configurations you're experimenting with."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "71c3aaea",
      "metadata": {
        "id": "71c3aaea"
      },
      "source": [
        "## Retrieve Published Objects & Ops\n",
        "\n",
        "You can publish objects and then retrieve them in your code. You can even call functions from your retrieved objects!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "caa5c5e0",
      "metadata": {
        "id": "caa5c5e0"
      },
      "outputs": [],
      "source": [
        "import weave\n",
        "\n",
        "weave.init(PROJECT)\n",
        "\n",
        "corrector = OpenAIGrammarCorrector(\n",
        "    openai_model_name=\"gpt-3.5-turbo-1106\",\n",
        "    system_message=\"You are a grammar checker, correct the following user input.\",\n",
        ")\n",
        "\n",
        "ref = weave.publish(corrector)\n",
        "print(ref.uri())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9ff9cfe7",
      "metadata": {
        "id": "9ff9cfe7"
      },
      "outputs": [],
      "source": [
        "import weave\n",
        "\n",
        "weave.init(PROJECT)\n",
        "\n",
        "# Note: this url is available from the UI after publishing the object!\n",
        "ref_url = f\"weave:///{ref.entity}/{PROJECT}/object/{ref.name}:{ref.digest}\"\n",
        "fetched_collector = weave.ref(ref_url).get()\n",
        "\n",
        "# Notice: this object was loaded from remote location!\n",
        "result = fetched_collector.predict(\"That was so easy, it was a piece of pie!\")\n",
        "\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "77b94dd5",
      "metadata": {
        "id": "77b94dd5"
      },
      "source": [
        "# Evaluation\n",
        "\n",
        "Evaluation-driven development helps you reliably iterate on an application. The `Evaluation` class is designed to assess the performance of a `Model` on a given `Dataset` or set of examples using scoring functions.\n",
        "\n",
        "See a preview of the API below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9c597f9a",
      "metadata": {
        "id": "9c597f9a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 216
        },
        "outputId": "5f1ad221-09f9-40c4-f783-2600c3d3b75e"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'dataset' is not defined",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-7-3f1ef2de8219>\u001b[0m in \u001b[0;36m<cell line: 13>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;31m# Score your examples using scoring functions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m evaluation = Evaluation(\n\u001b[0;32m---> 14\u001b[0;31m     \u001b[0mdataset\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m,\u001b[0m  \u001b[0;31m# can be a list of dictionaries or a weave.Dataset object\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m     \u001b[0mscorers\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mexact_match\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m  \u001b[0;31m# can be a list of scoring functions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m )\n",
            "\u001b[0;31mNameError\u001b[0m: name 'dataset' is not defined"
          ]
        }
      ],
      "source": [
        "import weave\n",
        "from weave import Evaluation\n",
        "\n",
        "\n",
        "# Define any custom scoring function\n",
        "@weave.op()\n",
        "def exact_match(expected: str, model_output: dict) -> dict:\n",
        "    # Here is where you'd define the logic to score the model output\n",
        "    return {\"match\": expected == model_output}\n",
        "\n",
        "\n",
        "# Score your examples using scoring functions\n",
        "evaluation = Evaluation(\n",
        "    dataset=dataset,  # can be a list of dictionaries or a weave.Dataset object\n",
        "    scorers=[exact_match],  # can be a list of scoring functions\n",
        ")\n",
        "\n",
        "# Start tracking the evaluation\n",
        "weave.init(PROJECT)\n",
        "# Run the evaluation\n",
        "summary = await evaluation.evaluate(corrector)  # can be a model or simple function"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d997d9e6",
      "metadata": {
        "id": "d997d9e6"
      },
      "source": [
        "## What's next?\n",
        "\n",
        "Follow the [Build an Evaluation pipeline](http://wandb.me/weave_eval_tut) tutorial to learn more about Evaluation and begin iteratively improving your applications."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    },
    "colab": {
      "provenance": [],
      "toc_visible": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}