diff --git a/examples/tutorials/rag-agent-testing.ipynb b/examples/tutorials/rag-agent-testing.ipynb index 6d2317d01..7bb3fccf4 100644 --- a/examples/tutorials/rag-agent-testing.ipynb +++ b/examples/tutorials/rag-agent-testing.ipynb @@ -12,7 +12,7 @@ "source": [ "# Building and Testing Corrective RAG\n", "\n", - "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://drive.google.com/file/d/1KUCIBFtDytL2gFgyFC46t_Yjp3jDjdUF/view?usp=sharing)\n", "\n", "[Corrective-RAG (CRAG)](https://arxiv.org/abs/2401.15884) is a strategy for RAG that incorporates self-reflection / self-grading on retrieved documents. \n", "\n", @@ -57,7 +57,7 @@ "outputs": [], "source": [ "%%capture --no-stderr\n", - "%pip install -U langchain tavily-python langgraph matplotlib langchain_community tiktoken langchain-openai scikit-learn" + "%pip install -U langchain tavily-python langgraph matplotlib langchain_community tiktoken langchain-openai scikit-learn langchain_fireworks langchainhub" ] }, { @@ -79,9 +79,7 @@ "_set_env(\"OPENAI_API_KEY\")\n", "_set_env(\"LANGCHAIN_API_KEY\")\n", "_set_env(\"TAVILY_API_KEY\")\n", - "_set_env(\"FIREWORKS_API_KEY\")\n", - "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n", - "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"" + "_set_env(\"FIREWORKS_API_KEY\")" ] }, { @@ -89,7 +87,7 @@ "id": "7b59fdc0-ee15-4fb0-b004-86089c26acd8", "metadata": {}, "source": [ - "You can also set API keys as shown here:" + "You can also set API keys and env variables here:" ] }, { @@ -101,6 +99,13 @@ "source": [ "import os\n", "\n", + "os.environ[\"OPENAI_API_KEY\"] = \"xxx\"\n", + "os.environ[\"LANGCHAIN_API_KEY\"] = \"xxx\"\n", + "os.environ[\"TAVILY_API_KEY\"] = \"xxx\"\n", + "os.environ[\"FIREWORKS_API_KEY\"] = \"xxx\"\n", + "\n", + "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n", + "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n", "os.environ[\"LANGCHAIN_PROJECT\"] = \"corrective-rag-agent-testing\"" ] }, @@ -184,7 +189,7 @@ "\n", "### Tools\n", "\n", - "First, we'll index 3 blog posts and store them in a vectorstore, [Chroma](https://python.langchain.com/v0.2/docs/integrations/vectorstores/chroma/) and define [this as a tool](https://python.langchain.com/v0.2/docs/concepts/#tools).\n" + "First, we'll index 3 blog posts and store them in a vectorstore and define [this as a tool](https://python.langchain.com/v0.2/docs/concepts/#tools).\n" ] }, { @@ -1073,8 +1078,7 @@ "from langchain_openai import ChatOpenAI\n", "\n", "# Grade prompt\n", - "grade_prompt_answer_accuracy = prompt = hub.pull(\"langchain-ai/rag-answer-vs-reference\")\n", - "\n", + "grade_prompt_answer_accuracy = hub.pull(\"langchain-ai/rag-answer-vs-reference\")\n", "\n", "def answer_evaluator(run, example) -> dict:\n", " \"\"\"\n", @@ -1136,7 +1140,6 @@ " \"generate_answer\",\n", "]\n", "\n", - "\n", "def check_trajectory_react(root_run: Run, example: Example) -> dict:\n", " \"\"\"\n", " Check if all expected tools are called in exact order and without any additional tool calls.\n", @@ -1176,6 +1179,36 @@ "The results from each experiment will be logged to our `dataset_name` in LangSmith.\n" ] }, + { + "cell_type": "code", + "execution_count": null, + "id": "6faa646a-b13a-4b74-aea5-a4afe5cbd2d6", + "metadata": {}, + "outputs": [], + "source": [ + "from langsmith.evaluation import evaluate\n", + "\n", + "experiment_prefix = f\"react-agent-{model_tested}\"\n", + "experiment_results = evaluate(\n", + " predict_react_agent_answer,\n", + " data=dataset_name,\n", + " evaluators=[answer_evaluator, check_trajectory_react],\n", + " experiment_prefix=experiment_prefix + \"-answer-and-tool-use\",\n", + " num_repetitions=3,\n", + " metadata={\"version\": metadata},\n", + ")\n", + "\n", + "experiment_prefix = f\"custom-agent-{model_tested}\"\n", + "experiment_results = evaluate(\n", + " predict_custom_agent_answer,\n", + " data=dataset_name,\n", + " evaluators=[answer_evaluator, check_trajectory_custom],\n", + " experiment_prefix=experiment_prefix + \"-answer-and-tool-use\",\n", + " num_repetitions=3,\n", + " metadata={\"version\": metadata},\n", + ")" + ] + }, { "attachments": { "953411a8-f352-4c8f-a923-d3ff171c6080.png": { @@ -1201,11 +1234,12 @@ "\n", "What we can see: \n", "\n", - "* ReAct agent has more freedom to select different trajectories of tool use.\n", - "* But, this means that it can deviate from our expected control flow, reducing reliability.\n", - "* With `GPT-4o`, the difference is fairly minor between our two agents, with `ReAct` scoring 73% vs `Custom` scoring 100% on tool use trajectory.\n", - "* However, as we move to `Firefunction-v2` with `Llama 3` we can see that the performance in our `ReAct` agent drops to `53%` on tool use trajectory.\n", - "* One of the powerful things about our `Custom` agent is that it retains reliable performance even as we dial down model capacity.\n", + "* ReAct agent has freedom to select many tool use trajectories.\n", + "* But, LLMs are non-deterministic and tool-calling is challenging.\n", + "* The tool calling trajectory score is worse for ReAct than Custom.\n", + "* The effect is further worse when we moved to `Firefunction-v2`.\n", + "* But the Custom agent shows strong performance in reasoning trajectory.\n", + "* This is true even as we dial down model capacity.\n", "* It can even be run locally [see here](https://github.com/langchain-ai/langgraph/blob/main/examples/rag/langgraph_crag_local.ipynb)! \n", "\n", "![Screenshot 2024-06-23 at 1.32.31 PM.png](attachment:953411a8-f352-4c8f-a923-d3ff171c6080.png)"