src/langchain/notebooks/langsmith_walkthrough.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "1a4596ea-a631-416d-a2a4-3577c140493d",
      "metadata": {
        "id": "1a4596ea-a631-416d-a2a4-3577c140493d",
        "tags": []
      },
      "source": [
        "# [LangSmith Walkthrough](https://python.langchain.com/docs/langsmith/walkthrough \"Original notebook\")\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/langsmith/walkthrough.ipynb)\n",
        "\n",
        "LangChain makes it easy to prototype LLM applications and Agents. However, delivering LLM applications to production can be deceptively difficult. You will likely have to heavily customize and iterate on your prompts, chains, and other components to create a high-quality product.\n",
        "\n",
        "To aid in this process, we've launched LangSmith, a unified platform for debugging, testing, and monitoring your LLM applications.\n",
        "\n",
        "When might this come in handy? You may find it useful when you want to:\n",
        "\n",
        "- Quickly debug a new chain, agent, or set of tools\n",
        "- Visualize how components (chains, llms, retrievers, etc.) relate and are used\n",
        "- Evaluate different prompts and LLMs for a single component\n",
        "- Run a given chain several times over a dataset to ensure it consistently meets a quality bar\n",
        "- Capture usage traces and using LLMs or analytics pipelines to generate insights\n",
        "\n",
        "#### You may wish to skip to the section on [Evaluation](#evaluate-agent)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "138fbb8f-960d-4d26-9dd5-6d6acab3ee55",
      "metadata": {
        "id": "138fbb8f-960d-4d26-9dd5-6d6acab3ee55"
      },
      "source": [
        "## Prerequisites\n",
        "\n",
        "**[Create a LangSmith account](https://smith.langchain.com/) and create an API key (see bottom left corner). Familiarize yourself with the platform by looking through the [docs](https://docs.smith.langchain.com/)**\n",
        "\n",
        "Note LangSmith is in closed beta; we're in the process of rolling it out to more users. However, you can fill out the form on the website for expedited access.\n",
        "\n",
        "Now, let's get started!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2d77d064-41b4-41fb-82e6-2d16461269ec",
      "metadata": {
        "id": "2d77d064-41b4-41fb-82e6-2d16461269ec",
        "tags": []
      },
      "source": [
        "## Log runs to LangSmith\n",
        "\n",
        "First, configure your environment variables to tell LangChain to log traces. This is done by setting the `LANGCHAIN_TRACING_V2` environment variable to true.\n",
        "You can tell LangChain which project to log to by setting the `LANGCHAIN_PROJECT` environment variable (if this isn't set, runs will be logged to the `default` project). This will automatically create the project for you if it doesn't exist. You must also set the `LANGCHAIN_ENDPOINT` and `LANGCHAIN_API_KEY` environment variables.\n",
        "\n",
        "For more information on other ways to set up tracing, please reference the [LangSmith documentation](https://docs.smith.langchain.com/docs/).\n",
        "\n",
        "**NOTE:** You must also set your `OPENAI_API_KEY` environment variables in order to run the following tutorial.\n",
        "\n",
        "**NOTE:** You can only access an API key when you first create it. Keep it somewhere safe.\n",
        "\n",
        "**NOTE:** You can also use a context manager in python to log traces using\n",
        "```python\n",
        "from langchain.callbacks.manager import tracing_v2_enabled\n",
        "\n",
        "with tracing_v2_enabled(project_name=\"My Project\"):\n",
        "    agent.run(\"How many people live in canada as of 2023?\")\n",
        "```\n",
        "\n",
        "However, in this example, we will use environment variables."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "e4780363-f05a-4649-8b1a-9b449f960ce4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e4780363-f05a-4649-8b1a-9b449f960ce4",
        "outputId": "6629ef72-3db8-498f-9544-517b6632f66c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Note: you may need to restart the kernel to use updated packages.\n"
          ]
        }
      ],
      "source": [
        "%pip install -Uq langchain langchain-openai langsmith langchainhub \\\n",
        "    pandas duckduckgo-search"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "904db9a5-f387-4a57-914c-c8af8d39e249",
      "metadata": {
        "id": "904db9a5-f387-4a57-914c-c8af8d39e249",
        "tags": []
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from uuid import uuid4\n",
        "\n",
        "unique_id = uuid4().hex[0:8]\n",
        "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
        "os.environ[\"LANGCHAIN_PROJECT\"] = f\"Tracing Walkthrough - {unique_id}\"\n",
        "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n",
        "os.environ[\"LANGCHAIN_API_KEY\"] = \"\"  # Update to your API key\n",
        "\n",
        "# Used by the agent in this tutorial\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8ee7f34b-b65c-4e09-ad52-e3ace78d0221",
      "metadata": {
        "id": "8ee7f34b-b65c-4e09-ad52-e3ace78d0221",
        "tags": []
      },
      "source": [
        "Create the langsmith client to interact with the API"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "510b5ca0",
      "metadata": {
        "id": "510b5ca0",
        "tags": []
      },
      "outputs": [],
      "source": [
        "from langsmith import Client\n",
        "\n",
        "client = Client()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ca27fa11-ddce-4af0-971e-c5c37d5b92ef",
      "metadata": {
        "id": "ca27fa11-ddce-4af0-971e-c5c37d5b92ef"
      },
      "source": [
        "Create a LangChain component and log runs to the platform. In this example, we will create a ReAct-style agent with access to a general search tool (DuckDuckGo). The agent's prompt can be viewed in the [Hub here](https://smith.langchain.com/hub/wfh/langsmith-agent-prompt)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "a0fbfbba-3c82-4298-a312-9cec016d9d2e",
      "metadata": {
        "id": "a0fbfbba-3c82-4298-a312-9cec016d9d2e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\Users\\dae\\.vscode\\Software\\Build-AI\\.venv\\lib\\site-packages\\langchain_core\\_api\\deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.chat_models.openai.ChatOpenAI` was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import ChatOpenAI`.\n",
            "  warn_deprecated(\n"
          ]
        }
      ],
      "source": [
        "from langchain import hub\n",
        "from langchain.agents import AgentExecutor\n",
        "from langchain.agents.format_scratchpad import format_to_openai_function_messages\n",
        "from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser\n",
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain.tools import DuckDuckGoSearchResults\n",
        "from langchain.tools.render import format_tool_to_openai_function\n",
        "\n",
        "# Fetches the latest version of this prompt\n",
        "prompt = hub.pull(\"wfh/langsmith-agent-prompt:latest\")\n",
        "\n",
        "llm = ChatOpenAI(\n",
        "    model=\"gpt-3.5-turbo-1106\",\n",
        "    temperature=0.25,\n",
        ")\n",
        "\n",
        "tools = [\n",
        "    DuckDuckGoSearchResults(\n",
        "        name=\"duck_duck_go\"\n",
        "    ),  # General internet search using DuckDuckGo\n",
        "]\n",
        "\n",
        "llm_with_tools = llm.bind(functions=[format_tool_to_openai_function(t) for t in tools])\n",
        "\n",
        "runnable_agent = (\n",
        "    {\n",
        "        \"input\": lambda x: x[\"input\"],\n",
        "        \"agent_scratchpad\": lambda x: format_to_openai_function_messages(\n",
        "            x[\"intermediate_steps\"]\n",
        "        ),\n",
        "    }\n",
        "    | prompt\n",
        "    | llm_with_tools\n",
        "    | OpenAIFunctionsAgentOutputParser()\n",
        ")\n",
        "\n",
        "agent_executor = AgentExecutor(\n",
        "    agent=runnable_agent, tools=tools, handle_parsing_errors=True\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cab51e1e-8270-452c-ba22-22b5b5951899",
      "metadata": {
        "id": "cab51e1e-8270-452c-ba22-22b5b5951899"
      },
      "source": [
        "We are running the agent concurrently on multiple inputs to reduce latency. Runs get logged to LangSmith in the background so execution latency is unaffected."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "19537902-b95c-4390-80a4-f6c9a937081e",
      "metadata": {
        "id": "19537902-b95c-4390-80a4-f6c9a937081e",
        "tags": []
      },
      "outputs": [],
      "source": [
        "inputs = [\n",
        "    \"What is LangChain?\",\n",
        "    \"What's LangSmith?\",\n",
        "    \"When was Llama-v2 released?\",\n",
        "    \"What is the langsmith cookbook?\",\n",
        "    \"When did langchain first announce the hub?\",\n",
        "]\n",
        "\n",
        "results = agent_executor.batch([{\"input\": x} for x in inputs], return_exceptions=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "9a6a764c-5d7a-4de7-a916-3ecc987d5bb6",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9a6a764c-5d7a-4de7-a916-3ecc987d5bb6",
        "outputId": "f928b4ff-0fff-4840-caab-bbcaddbb73cb"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'input': 'What is LangChain?',\n",
              "  'output': 'LangChain is a framework for creating applications that leverage the capabilities of language models. It simplifies the creation of applications using large language models (LLMs) such as GPT-4. LangChain provides a standard interface for chains, integrations with other tools, and end-to-end chains for common applications. It is a Python library that helps create applications that combine large language models with other sources of information or computations. You can learn more about LangChain and how to use it for various applications through the provided links.'},\n",
              " {'input': \"What's LangSmith?\",\n",
              "  'output': 'LangSmith is a unified platform for debugging, testing, and deploying intelligent agents and chains based on any LLM framework. It is developed by LangChain and aims to help developers close the gap between prototype and production. LangSmith provides full visibility into model inputs and outputs, and it is designed to elevate LLM applications to production-grade quality. You can find more information about LangSmith on their official website: [LangSmith Official Website](https://blog.langchain.dev/announcing-langsmith/)'},\n",
              " {'input': 'When was Llama-v2 released?',\n",
              "  'output': 'Llama-v2 was released on July 18, 2023.'},\n",
              " {'input': 'What is the langsmith cookbook?',\n",
              "  'output': 'The LangSmith Cookbook is a repository that stores code tutorials showing different ways to get more out of LangSmith. LangSmith is a platform that helps you debug, test, evaluate, and monitor your Language Learning Model (LLM) applications. The cookbook recipes are meant to complement the LangSmith Documentation by showing common use cases and tactics within an end-to-end context.\\n\\nYou can find the LangSmith Cookbook on GitHub at the following link: [LangSmith Cookbook on GitHub](https://github.com/whatif-dev/langchain-langsmith-cookbook)'},\n",
              " {'input': 'When did langchain first announce the hub?',\n",
              "  'output': 'LangChain first announced the LangChain Hub on September 5, 2023.'}]"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "results[:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9decb964-be07-4b6c-9802-9825c8be7b64",
      "metadata": {
        "id": "9decb964-be07-4b6c-9802-9825c8be7b64"
      },
      "source": [
        "Assuming you've successfully set up your environment, your agent traces should show up in the `Projects` section in the [app](https://smith.langchain.com/). Congrats!\n",
        "\n",
        "![Initial Runs](https://github.com/langchain-ai/langchain/blob/master/docs/docs/langsmith/img/log_traces.png?raw=1)\n",
        "\n",
        "It looks like the agent isn't effectively using the tools though. Let's evaluate this so we have a baseline."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6c43c311-4e09-4d57-9ef3-13afb96ff430",
      "metadata": {
        "id": "6c43c311-4e09-4d57-9ef3-13afb96ff430"
      },
      "source": [
        "## Evaluate Agent\n",
        "\n",
        "In addition to logging runs, LangSmith also allows you to test and evaluate your LLM applications.\n",
        "\n",
        "In this section, you will leverage LangSmith to create a benchmark dataset and run AI-assisted evaluators on an agent. You will do so in a few steps:\n",
        "\n",
        "1. Create a dataset\n",
        "2. Initialize a new agent to benchmark\n",
        "3. Configure evaluators to grade an agent's output\n",
        "4. Run the agent over the dataset and evaluate the results"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "beab1a29-b79d-4a99-b5b1-0870c2d772b1",
      "metadata": {
        "id": "beab1a29-b79d-4a99-b5b1-0870c2d772b1"
      },
      "source": [
        "### 1. Create a LangSmith dataset\n",
        "\n",
        "Below, we use the LangSmith client to create a dataset from the input questions from above and a list labels. You will use these later to measure performance for a new agent. A dataset is a collection of examples, which are nothing more than input-output pairs you can use as test cases to your application.\n",
        "\n",
        "For more information on datasets, including how to create them from CSVs or other files or how to create them in the platform, please refer to the [LangSmith documentation](https://docs.smith.langchain.com/)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "43fd40b2-3f02-4e51-9343-705aafe90a36",
      "metadata": {
        "id": "43fd40b2-3f02-4e51-9343-705aafe90a36"
      },
      "outputs": [],
      "source": [
        "outputs = [\n",
        "    \"LangChain is an open-source framework for building applications using large language models. It is also the name of the company building LangSmith.\",\n",
        "    \"LangSmith is a unified platform for debugging, testing, and monitoring language model applications and agents powered by LangChain\",\n",
        "    \"July 18, 2023\",\n",
        "    \"The langsmith cookbook is a github repository containing detailed examples of how to use LangSmith to debug, evaluate, and monitor large language model-powered applications.\",\n",
        "    \"September 5, 2023\",\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "17580c4b-bd04-4dde-9d21-9d4edd25b00d",
      "metadata": {
        "id": "17580c4b-bd04-4dde-9d21-9d4edd25b00d",
        "tags": []
      },
      "outputs": [],
      "source": [
        "dataset_name = f\"agent-qa-{unique_id}\"\n",
        "\n",
        "dataset = client.create_dataset(\n",
        "    dataset_name,\n",
        "    description=\"An example dataset of questions over the LangSmith documentation.\",\n",
        ")\n",
        "\n",
        "for query, answer in zip(inputs, outputs):\n",
        "    client.create_example(\n",
        "        inputs={\"input\": query}, outputs={\"output\": answer}, dataset_id=dataset.id\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8adfd29c-b258-49e5-94b4-74597a12ba16",
      "metadata": {
        "id": "8adfd29c-b258-49e5-94b4-74597a12ba16",
        "tags": []
      },
      "source": [
        "### 2. Initialize a new agent to benchmark\n",
        "\n",
        "LangSmith lets you evaluate any LLM, chain, agent, or even a custom function. Conversational agents are stateful (they have memory); to ensure that this state isn't shared between dataset runs, we will pass in a `chain_factory` (aka a `constructor`) function to initialize for each call.\n",
        "\n",
        "In this case, we will test an agent that uses OpenAI's function calling endpoints."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "f42d8ecc-d46a-448b-a89c-04b0f6907f75",
      "metadata": {
        "id": "f42d8ecc-d46a-448b-a89c-04b0f6907f75",
        "tags": []
      },
      "outputs": [],
      "source": [
        "from langchain import hub\n",
        "from langchain.agents import AgentExecutor, AgentType, initialize_agent, load_tools\n",
        "from langchain.agents.format_scratchpad import format_to_openai_function_messages\n",
        "from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser\n",
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain.tools.render import format_tool_to_openai_function\n",
        "\n",
        "\n",
        "# Since chains can be stateful (e.g. they can have memory), we provide\n",
        "# a way to initialize a new chain for each row in the dataset. This is done\n",
        "# by passing in a factory function that returns a new chain for each row.\n",
        "def agent_factory(prompt):\n",
        "    llm_with_tools = llm.bind(\n",
        "        functions=[format_tool_to_openai_function(t) for t in tools]\n",
        "    )\n",
        "    runnable_agent = (\n",
        "        {\n",
        "            \"input\": lambda x: x[\"input\"],\n",
        "            \"agent_scratchpad\": lambda x: format_to_openai_function_messages(\n",
        "                x[\"intermediate_steps\"]\n",
        "            ),\n",
        "        }\n",
        "        | prompt\n",
        "        | llm_with_tools\n",
        "        | OpenAIFunctionsAgentOutputParser()\n",
        "    )\n",
        "    return AgentExecutor(agent=runnable_agent, tools=tools, handle_parsing_errors=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9cb9ef53",
      "metadata": {
        "id": "9cb9ef53"
      },
      "source": [
        "### 3. Configure evaluation\n",
        "\n",
        "Manually comparing the results of chains in the UI is effective, but it can be time consuming.\n",
        "It can be helpful to use automated metrics and AI-assisted feedback to evaluate your component's performance.\n",
        "\n",
        "Below, we will create some pre-implemented run evaluators that do the following:\n",
        "- Compare results against ground truth labels.\n",
        "- Measure semantic (dis)similarity using embedding distance\n",
        "- Evaluate 'aspects' of the agent's response in a reference-free manner using custom criteria\n",
        "\n",
        "For a longer discussion of how to select an appropriate evaluator for your use case and how to create your own\n",
        "custom evaluators, please refer to the [LangSmith documentation](https://docs.smith.langchain.com/).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "a25dc281",
      "metadata": {
        "id": "a25dc281",
        "tags": []
      },
      "outputs": [],
      "source": [
        "from langchain.evaluation import EvaluatorType\n",
        "from langchain.smith import RunEvalConfig\n",
        "\n",
        "evaluation_config = RunEvalConfig(\n",
        "    # Evaluators can either be an evaluator type (e.g., \"qa\", \"criteria\", \"embedding_distance\", etc.) or a configuration for that evaluator\n",
        "    evaluators=[\n",
        "        # Measures whether a QA response is \"Correct\", based on a reference answer\n",
        "        # You can also select via the raw string \"qa\"\n",
        "        EvaluatorType.QA,\n",
        "        # Measure the embedding distance between the output and the reference answer\n",
        "        # Equivalent to: EvalConfig.EmbeddingDistance(embeddings=OpenAIEmbeddings())\n",
        "        EvaluatorType.EMBEDDING_DISTANCE,\n",
        "        # Grade whether the output satisfies the stated criteria.\n",
        "        # You can select a default one such as \"helpfulness\" or provide your own.\n",
        "        RunEvalConfig.LabeledCriteria(\"helpfulness\"),\n",
        "        # The LabeledScoreString evaluator outputs a score on a scale from 1-10.\n",
        "        # You can use default criteria or write our own rubric\n",
        "        RunEvalConfig.LabeledScoreString(\n",
        "            {\n",
        "                \"accuracy\": \"\"\"\n",
        "Score 1: The answer is completely unrelated to the reference.\n",
        "Score 3: The answer has minor relevance but does not align with the reference.\n",
        "Score 5: The answer has moderate relevance but contains inaccuracies.\n",
        "Score 7: The answer aligns with the reference but has minor errors or omissions.\n",
        "Score 10: The answer is completely accurate and aligns perfectly with the reference.\"\"\"\n",
        "            },\n",
        "            normalize_by=10,\n",
        "        ),\n",
        "    ],\n",
        "    # You can add custom StringEvaluator or RunEvaluator objects here as well, which will automatically be\n",
        "    # applied to each prediction. Check out the docs for examples.\n",
        "    custom_evaluators=[],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "07885b10",
      "metadata": {
        "id": "07885b10",
        "tags": []
      },
      "source": [
        "### 4. Run the agent and evaluators\n",
        "\n",
        "Use the [run_on_dataset](https://api.python.langchain.com/en/latest/smith/langchain.smith.evaluation.runner_utils.run_on_dataset.html#langchain.smith.evaluation.runner_utils.run_on_dataset) (or asynchronous [arun_on_dataset](https://api.python.langchain.com/en/latest/smith/langchain.smith.evaluation.runner_utils.arun_on_dataset.html#langchain.smith.evaluation.runner_utils.arun_on_dataset)) function to evaluate your model. This will:\n",
        "1. Fetch example rows from the specified dataset.\n",
        "2. Run your agent (or any custom function) on each example.\n",
        "3. Apply evaluators to the resulting run traces and corresponding reference examples to generate automated feedback.\n",
        "\n",
        "The results will be visible in the LangSmith app."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "af8c8469-d70d-46d9-8fcd-517a1ccc7c4b",
      "metadata": {
        "id": "af8c8469-d70d-46d9-8fcd-517a1ccc7c4b"
      },
      "outputs": [],
      "source": [
        "from langchain import hub\n",
        "\n",
        "# We will test this version of the prompt\n",
        "prompt = hub.pull(\"wfh/langsmith-agent-prompt:798e7324\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "3733269b-8085-4644-9d5d-baedcff13a2f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3733269b-8085-4644-9d5d-baedcff13a2f",
        "outputId": "bec26a4c-1dd6-4289-f7be-9591ec64d25e",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "View the evaluation results for project 'runnable-agent-test-5d466cbc-4626d0a1' at:\n",
            "https://smith.langchain.com/o/0f7461cf-206f-5c85-aa8d-48c6c48bafc5/datasets/911e7abe-f52f-41e5-a0e8-a9bdd9f039e4/compare?selectedSessions=fb899c5f-1e8f-4dc0-b99d-e307e80e9915\n",
            "\n",
            "View all tests for Dataset agent-qa-4626d0a1 at:\n",
            "https://smith.langchain.com/o/0f7461cf-206f-5c85-aa8d-48c6c48bafc5/datasets/911e7abe-f52f-41e5-a0e8-a9bdd9f039e4\n",
            "[>                                                 ] 0/5"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\Users\\dae\\.vscode\\Software\\Build-AI\\.venv\\lib\\site-packages\\langchain_core\\_api\\deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.embeddings.openai.OpenAIEmbeddings` was deprecated in langchain-community 0.1.0 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import OpenAIEmbeddings`.\n",
            "  warn_deprecated(\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[------------------------------------------------->] 5/5"
          ]
        },
        {
          "data": {
            "text/html": [
              "<h3>Experiment Results:</h3>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>feedback.correctness</th>\n",
              "      <th>feedback.embedding_cosine_distance</th>\n",
              "      <th>feedback.helpfulness</th>\n",
              "      <th>feedback.score_string:accuracy</th>\n",
              "      <th>error</th>\n",
              "      <th>execution_time</th>\n",
              "      <th>run_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>5.000000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unique</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>top</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9a0c79e4-b2f2-4fe9-b247-e46d93ec07f6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>freq</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.600000</td>\n",
              "      <td>0.117986</td>\n",
              "      <td>0.800000</td>\n",
              "      <td>0.780000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8.584452</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.547723</td>\n",
              "      <td>0.100740</td>\n",
              "      <td>0.447214</td>\n",
              "      <td>0.216795</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.855749</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.039155</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.807878</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.043848</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.700000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.105066</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.071396</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.700000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.930291</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.159452</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10.116441</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.276078</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10.962582</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        feedback.correctness  feedback.embedding_cosine_distance  \\\n",
              "count               5.000000                            5.000000   \n",
              "unique                   NaN                                 NaN   \n",
              "top                      NaN                                 NaN   \n",
              "freq                     NaN                                 NaN   \n",
              "mean                0.600000                            0.117986   \n",
              "std                 0.547723                            0.100740   \n",
              "min                 0.000000                            0.039155   \n",
              "25%                 0.000000                            0.043848   \n",
              "50%                 1.000000                            0.071396   \n",
              "75%                 1.000000                            0.159452   \n",
              "max                 1.000000                            0.276078   \n",
              "\n",
              "        feedback.helpfulness  feedback.score_string:accuracy error  \\\n",
              "count               5.000000                        5.000000     0   \n",
              "unique                   NaN                             NaN     0   \n",
              "top                      NaN                             NaN   NaN   \n",
              "freq                     NaN                             NaN   NaN   \n",
              "mean                0.800000                        0.780000   NaN   \n",
              "std                 0.447214                        0.216795   NaN   \n",
              "min                 0.000000                        0.500000   NaN   \n",
              "25%                 1.000000                        0.700000   NaN   \n",
              "50%                 1.000000                        0.700000   NaN   \n",
              "75%                 1.000000                        1.000000   NaN   \n",
              "max                 1.000000                        1.000000   NaN   \n",
              "\n",
              "        execution_time                                run_id  \n",
              "count         5.000000                                     5  \n",
              "unique             NaN                                     5  \n",
              "top                NaN  9a0c79e4-b2f2-4fe9-b247-e46d93ec07f6  \n",
              "freq               NaN                                     1  \n",
              "mean          8.584452                                   NaN  \n",
              "std           1.855749                                   NaN  \n",
              "min           6.807878                                   NaN  \n",
              "25%           7.105066                                   NaN  \n",
              "50%           7.930291                                   NaN  \n",
              "75%          10.116441                                   NaN  \n",
              "max          10.962582                                   NaN  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import functools\n",
        "\n",
        "from langchain.smith import (\n",
        "    arun_on_dataset,\n",
        "    run_on_dataset,\n",
        ")\n",
        "\n",
        "chain_results = run_on_dataset(\n",
        "    dataset_name=dataset_name,\n",
        "    llm_or_chain_factory=functools.partial(agent_factory, prompt=prompt),\n",
        "    evaluation=evaluation_config,\n",
        "    verbose=True,\n",
        "    client=client,\n",
        "    project_name=f\"runnable-agent-test-5d466cbc-{unique_id}\",\n",
        "    tags=[\n",
        "        \"testing-notebook\",\n",
        "        \"prompt:5d466cbc\",\n",
        "    ],  # Optional, adds a tag to the resulting chain runs\n",
        ")\n",
        "\n",
        "# Sometimes, the agent will error due to parsing issues, incompatible tool inputs, etc.\n",
        "# These are logged as warnings here and captured as errors in the tracing UI."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cdacd159-eb4d-49e9-bb2a-c55322c40ed4",
      "metadata": {
        "id": "cdacd159-eb4d-49e9-bb2a-c55322c40ed4",
        "tags": []
      },
      "source": [
        "### Review the test results\n",
        "\n",
        "You can review the test results tracing UI below by clicking the URL in the output above or navigating to the \"Testing & Datasets\" page in LangSmith  **\"agent-qa-{unique_id}\"** dataset.\n",
        "\n",
        "![test results](https://github.com/langchain-ai/langchain/blob/master/docs/docs/langsmith/img/test_results.png?raw=1)\n",
        "\n",
        "This will show the new runs and the feedback logged from the selected evaluators. You can also explore a summary of the results in tabular format below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "9da60638-5be8-4b5f-a721-2c6627aeaf0c",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 487
        },
        "id": "9da60638-5be8-4b5f-a721-2c6627aeaf0c",
        "outputId": "0a239cc9-f352-4292-bc7b-9d424717c0fc"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>inputs.input</th>\n",
              "      <th>outputs.input</th>\n",
              "      <th>outputs.output</th>\n",
              "      <th>reference.output</th>\n",
              "      <th>feedback.correctness</th>\n",
              "      <th>feedback.embedding_cosine_distance</th>\n",
              "      <th>feedback.helpfulness</th>\n",
              "      <th>feedback.score_string:accuracy</th>\n",
              "      <th>error</th>\n",
              "      <th>execution_time</th>\n",
              "      <th>run_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>63e0a356-2427-4c1d-a6f7-5d6bdff43ff2</th>\n",
              "      <td>When did langchain first announce the hub?</td>\n",
              "      <td>When did langchain first announce the hub?</td>\n",
              "      <td>The LangChain Hub was announced in a blog post...</td>\n",
              "      <td>September 5, 2023</td>\n",
              "      <td>0</td>\n",
              "      <td>0.276078</td>\n",
              "      <td>0</td>\n",
              "      <td>0.5</td>\n",
              "      <td>None</td>\n",
              "      <td>7.105066</td>\n",
              "      <td>9a0c79e4-b2f2-4fe9-b247-e46d93ec07f6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>b6c85061-53b3-44fb-9e22-9fa082d9592f</th>\n",
              "      <td>What is the langsmith cookbook?</td>\n",
              "      <td>What is the langsmith cookbook?</td>\n",
              "      <td>The LangSmith Cookbook is a repository that st...</td>\n",
              "      <td>The langsmith cookbook is a github repository ...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.043848</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>None</td>\n",
              "      <td>10.116441</td>\n",
              "      <td>4e037b33-9cef-4d6e-a751-423cdeaa4568</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>0ad3af26-a8f0-467c-b09f-baec7b5d5fae</th>\n",
              "      <td>When was Llama-v2 released?</td>\n",
              "      <td>When was Llama-v2 released?</td>\n",
              "      <td>LangChain Llama-v2 was released on July 18, 2023.</td>\n",
              "      <td>July 18, 2023</td>\n",
              "      <td>1</td>\n",
              "      <td>0.159452</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>None</td>\n",
              "      <td>6.807878</td>\n",
              "      <td>9bd4f235-115e-44a2-8c7f-ae46e9833c6a</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ec05057b-9780-44e0-9b02-e2cc82d6f1b1</th>\n",
              "      <td>What's LangSmith?</td>\n",
              "      <td>What's LangSmith?</td>\n",
              "      <td>LangSmith is a unified platform for debugging,...</td>\n",
              "      <td>LangSmith is a unified platform for debugging,...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.039155</td>\n",
              "      <td>1</td>\n",
              "      <td>0.7</td>\n",
              "      <td>None</td>\n",
              "      <td>7.930291</td>\n",
              "      <td>883cc23e-8da5-4fb2-b35b-9a427a4410de</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14f1f5a1-68c1-410e-ad09-f12fcb66c6ee</th>\n",
              "      <td>What is LangChain?</td>\n",
              "      <td>What is LangChain?</td>\n",
              "      <td>LangChain is a framework for creating applicat...</td>\n",
              "      <td>LangChain is an open-source framework for buil...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.071396</td>\n",
              "      <td>1</td>\n",
              "      <td>0.7</td>\n",
              "      <td>None</td>\n",
              "      <td>10.962582</td>\n",
              "      <td>9cf36921-1095-46d5-86f6-c5924739956d</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                                    inputs.input  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2  When did langchain first announce the hub?   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f             What is the langsmith cookbook?   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae                 When was Llama-v2 released?   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1                           What's LangSmith?   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee                          What is LangChain?   \n",
              "\n",
              "                                                                   outputs.input  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2  When did langchain first announce the hub?   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f             What is the langsmith cookbook?   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae                 When was Llama-v2 released?   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1                           What's LangSmith?   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee                          What is LangChain?   \n",
              "\n",
              "                                                                         outputs.output  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2  The LangChain Hub was announced in a blog post...   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f  The LangSmith Cookbook is a repository that st...   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae  LangChain Llama-v2 was released on July 18, 2023.   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1  LangSmith is a unified platform for debugging,...   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee  LangChain is a framework for creating applicat...   \n",
              "\n",
              "                                                                       reference.output  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2                                  September 5, 2023   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f  The langsmith cookbook is a github repository ...   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae                                      July 18, 2023   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1  LangSmith is a unified platform for debugging,...   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee  LangChain is an open-source framework for buil...   \n",
              "\n",
              "                                      feedback.correctness  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2                     0   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f                     1   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae                     1   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1                     1   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee                     0   \n",
              "\n",
              "                                      feedback.embedding_cosine_distance  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2                            0.276078   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f                            0.043848   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae                            0.159452   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1                            0.039155   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee                            0.071396   \n",
              "\n",
              "                                      feedback.helpfulness  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2                     0   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f                     1   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae                     1   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1                     1   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee                     1   \n",
              "\n",
              "                                      feedback.score_string:accuracy error  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2                             0.5  None   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f                             1.0  None   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae                             1.0  None   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1                             0.7  None   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee                             0.7  None   \n",
              "\n",
              "                                      execution_time  \\\n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2        7.105066   \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f       10.116441   \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae        6.807878   \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1        7.930291   \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee       10.962582   \n",
              "\n",
              "                                                                    run_id  \n",
              "63e0a356-2427-4c1d-a6f7-5d6bdff43ff2  9a0c79e4-b2f2-4fe9-b247-e46d93ec07f6  \n",
              "b6c85061-53b3-44fb-9e22-9fa082d9592f  4e037b33-9cef-4d6e-a751-423cdeaa4568  \n",
              "0ad3af26-a8f0-467c-b09f-baec7b5d5fae  9bd4f235-115e-44a2-8c7f-ae46e9833c6a  \n",
              "ec05057b-9780-44e0-9b02-e2cc82d6f1b1  883cc23e-8da5-4fb2-b35b-9a427a4410de  \n",
              "14f1f5a1-68c1-410e-ad09-f12fcb66c6ee  9cf36921-1095-46d5-86f6-c5924739956d  "
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "chain_results.to_dataframe()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "13aad317-73ff-46a7-a5a0-60b5b5295f02",
      "metadata": {
        "id": "13aad317-73ff-46a7-a5a0-60b5b5295f02"
      },
      "source": [
        "### (Optional) Compare to another prompt\n",
        "\n",
        "Now that we have our test run results, we can make changes to our agent and benchmark them. Let's try this again with a different prompt and see the results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "5eeb023f-ded2-4d0f-b910-2a57d9675853",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5eeb023f-ded2-4d0f-b910-2a57d9675853",
        "outputId": "ff5870b3-d974-42f2-de37-277f1053dc9b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "View the evaluation results for project 'runnable-agent-test-39f3bbd0-4626d0a1' at:\n",
            "https://smith.langchain.com/o/0f7461cf-206f-5c85-aa8d-48c6c48bafc5/datasets/911e7abe-f52f-41e5-a0e8-a9bdd9f039e4/compare?selectedSessions=3daeaf66-8bb2-44e2-9c2c-582485f2be9e\n",
            "\n",
            "View all tests for Dataset agent-qa-4626d0a1 at:\n",
            "https://smith.langchain.com/o/0f7461cf-206f-5c85-aa8d-48c6c48bafc5/datasets/911e7abe-f52f-41e5-a0e8-a9bdd9f039e4\n",
            "[------------------------------------------------->] 5/5"
          ]
        },
        {
          "data": {
            "text/html": [
              "<h3>Experiment Results:</h3>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>feedback.correctness</th>\n",
              "      <th>feedback.embedding_cosine_distance</th>\n",
              "      <th>feedback.helpfulness</th>\n",
              "      <th>feedback.score_string:accuracy</th>\n",
              "      <th>error</th>\n",
              "      <th>execution_time</th>\n",
              "      <th>run_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>5.0</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>5.0</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unique</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>top</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4f0c1c68-a2e2-41be-8113-3fb200ef959a</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>freq</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.085867</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.880000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.464101</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.055507</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.164317</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.426526</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.037770</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.700000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5.444651</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.043254</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.700000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.702335</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.056364</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7.934884</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.138944</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8.075832</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.153003</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>NaN</td>\n",
              "      <td>9.162802</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        feedback.correctness  feedback.embedding_cosine_distance  \\\n",
              "count                    5.0                            5.000000   \n",
              "unique                   NaN                                 NaN   \n",
              "top                      NaN                                 NaN   \n",
              "freq                     NaN                                 NaN   \n",
              "mean                     1.0                            0.085867   \n",
              "std                      0.0                            0.055507   \n",
              "min                      1.0                            0.037770   \n",
              "25%                      1.0                            0.043254   \n",
              "50%                      1.0                            0.056364   \n",
              "75%                      1.0                            0.138944   \n",
              "max                      1.0                            0.153003   \n",
              "\n",
              "        feedback.helpfulness  feedback.score_string:accuracy error  \\\n",
              "count                    5.0                        5.000000     0   \n",
              "unique                   NaN                             NaN     0   \n",
              "top                      NaN                             NaN   NaN   \n",
              "freq                     NaN                             NaN   NaN   \n",
              "mean                     1.0                        0.880000   NaN   \n",
              "std                      0.0                        0.164317   NaN   \n",
              "min                      1.0                        0.700000   NaN   \n",
              "25%                      1.0                        0.700000   NaN   \n",
              "50%                      1.0                        1.000000   NaN   \n",
              "75%                      1.0                        1.000000   NaN   \n",
              "max                      1.0                        1.000000   NaN   \n",
              "\n",
              "        execution_time                                run_id  \n",
              "count         5.000000                                     5  \n",
              "unique             NaN                                     5  \n",
              "top                NaN  4f0c1c68-a2e2-41be-8113-3fb200ef959a  \n",
              "freq               NaN                                     1  \n",
              "mean          7.464101                                   NaN  \n",
              "std           1.426526                                   NaN  \n",
              "min           5.444651                                   NaN  \n",
              "25%           6.702335                                   NaN  \n",
              "50%           7.934884                                   NaN  \n",
              "75%           8.075832                                   NaN  \n",
              "max           9.162802                                   NaN  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "candidate_prompt = hub.pull(\"wfh/langsmith-agent-prompt:39f3bbd0\")\n",
        "\n",
        "chain_results = run_on_dataset(\n",
        "    dataset_name=dataset_name,\n",
        "    llm_or_chain_factory=functools.partial(agent_factory, prompt=candidate_prompt),\n",
        "    evaluation=evaluation_config,\n",
        "    verbose=True,\n",
        "    client=client,\n",
        "    project_name=f\"runnable-agent-test-39f3bbd0-{unique_id}\",\n",
        "    tags=[\n",
        "        \"testing-notebook\",\n",
        "        \"prompt:39f3bbd0\",\n",
        "    ],  # Optional, adds a tag to the resulting chain runs\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "591c819e-9932-45cf-adab-63727dd49559",
      "metadata": {
        "id": "591c819e-9932-45cf-adab-63727dd49559"
      },
      "source": [
        "## Exporting datasets and runs\n",
        "\n",
        "LangSmith lets you export data to common formats such as CSV or JSONL directly in the web app. You can also use the client to fetch runs for further analysis, to store in your own database, or to share with others. Let's fetch the run traces from the evaluation run.\n",
        "\n",
        "**Note: It may be a few moments before all the runs are accessible.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "33bfefde-d1bb-4f50-9f7a-fd572ee76820",
      "metadata": {
        "id": "33bfefde-d1bb-4f50-9f7a-fd572ee76820",
        "tags": []
      },
      "outputs": [],
      "source": [
        "runs = client.list_runs(project_name=chain_results[\"project_name\"], execution_order=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "6595c888-1f5c-4ae3-9390-0a559f5575d1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6595c888-1f5c-4ae3-9390-0a559f5575d1",
        "outputId": "a44c055f-792b-4e27-9de5-354b464deb86",
        "tags": []
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'correctness': {'n': 5, 'avg': 1.0},\n",
              " 'embedding_cosine_distance': {'n': 5, 'avg': 0.08582},\n",
              " 'helpfulness': {'n': 5, 'avg': 1.0},\n",
              " 'score_string:accuracy': {'n': 5, 'avg': 0.8800000000000001}}"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# After some time, these will be populated.\n",
        "client.read_project(project_name=chain_results[\"project_name\"]).feedback_stats"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2646f0fb-81d4-43ce-8a9b-54b8e19841e2",
      "metadata": {
        "id": "2646f0fb-81d4-43ce-8a9b-54b8e19841e2",
        "tags": []
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "Congratulations! You have successfully traced and evaluated an agent using LangSmith!\n",
        "\n",
        "This was a quick guide to get started, but there are many more ways to use LangSmith to speed up your developer flow and produce better results.\n",
        "\n",
        "For more information on how you can get the most out of LangSmith, check out [LangSmith documentation](https://docs.smith.langchain.com/), and please reach out with questions, feature requests, or feedback at [support@langchain.dev](mailto:support@langchain.dev)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}