Add CoLab link (#791)

examples/tutorials/rag-agent-testing.ipynb

@@ -12,7 +12,7 @@
    "source": [
     "# Building and Testing Corrective RAG\n",
     "\n",
-    "\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://drive.google.com/file/d/1KUCIBFtDytL2gFgyFC46t_Yjp3jDjdUF/view?usp=sharing)\n",
     "\n",
     "[Corrective-RAG (CRAG)](https://arxiv.org/abs/2401.15884) is a strategy for RAG that incorporates self-reflection / self-grading on retrieved documents. \n",
     "\n",
@@ -57,7 +57,7 @@
    "outputs": [],
    "source": [
     "%%capture --no-stderr\n",
-    "%pip install -U langchain tavily-python langgraph matplotlib langchain_community tiktoken langchain-openai scikit-learn"
+    "%pip install -U langchain tavily-python langgraph matplotlib langchain_community tiktoken langchain-openai scikit-learn langchain_fireworks langchainhub"
    ]
   },
   {
@@ -79,9 +79,7 @@
     "_set_env(\"OPENAI_API_KEY\")\n",
     "_set_env(\"LANGCHAIN_API_KEY\")\n",
     "_set_env(\"TAVILY_API_KEY\")\n",
-    "_set_env(\"FIREWORKS_API_KEY\")\n",
-    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
-    "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\""
+    "_set_env(\"FIREWORKS_API_KEY\")"
    ]
   },
   {
@@ -89,7 +87,7 @@
    "id": "7b59fdc0-ee15-4fb0-b004-86089c26acd8",
    "metadata": {},
    "source": [
-    "You can also set API keys as shown here:"
+    "You can also set API keys and env variables here:"
    ]
   },
   {
@@ -101,6 +99,13 @@
    "source": [
     "import os\n",
     "\n",
+    "os.environ[\"OPENAI_API_KEY\"] = \"xxx\"\n",
+    "os.environ[\"LANGCHAIN_API_KEY\"] = \"xxx\"\n",
+    "os.environ[\"TAVILY_API_KEY\"] = \"xxx\"\n",
+    "os.environ[\"FIREWORKS_API_KEY\"] = \"xxx\"\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
+    "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n",
     "os.environ[\"LANGCHAIN_PROJECT\"] = \"corrective-rag-agent-testing\""
    ]
   },
@@ -184,7 +189,7 @@
     "\n",
     "### Tools\n",
     "\n",
-    "First, we'll index 3 blog posts and store them in a vectorstore, [Chroma](https://python.langchain.com/v0.2/docs/integrations/vectorstores/chroma/) and define [this as a tool](https://python.langchain.com/v0.2/docs/concepts/#tools).\n"
+    "First, we'll index 3 blog posts and store them in a vectorstore and define [this as a tool](https://python.langchain.com/v0.2/docs/concepts/#tools).\n"
    ]
   },
   {
@@ -1073,8 +1078,7 @@
     "from langchain_openai import ChatOpenAI\n",
     "\n",
     "# Grade prompt\n",
-    "grade_prompt_answer_accuracy = prompt = hub.pull(\"langchain-ai/rag-answer-vs-reference\")\n",
-    "\n",
+    "grade_prompt_answer_accuracy = hub.pull(\"langchain-ai/rag-answer-vs-reference\")\n",
     "\n",
     "def answer_evaluator(run, example) -> dict:\n",
     "    \"\"\"\n",
@@ -1136,7 +1140,6 @@
     "    \"generate_answer\",\n",
     "]\n",
     "\n",
-    "\n",
     "def check_trajectory_react(root_run: Run, example: Example) -> dict:\n",
     "    \"\"\"\n",
     "    Check if all expected tools are called in exact order and without any additional tool calls.\n",
@@ -1176,6 +1179,36 @@
     "The results from each experiment will be logged to our `dataset_name` in LangSmith.\n"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6faa646a-b13a-4b74-aea5-a4afe5cbd2d6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langsmith.evaluation import evaluate\n",
+    "\n",
+    "experiment_prefix = f\"react-agent-{model_tested}\"\n",
+    "experiment_results = evaluate(\n",
+    "    predict_react_agent_answer,\n",
+    "    data=dataset_name,\n",
+    "    evaluators=[answer_evaluator, check_trajectory_react],\n",
+    "    experiment_prefix=experiment_prefix + \"-answer-and-tool-use\",\n",
+    "    num_repetitions=3,\n",
+    "    metadata={\"version\": metadata},\n",
+    ")\n",
+    "\n",
+    "experiment_prefix = f\"custom-agent-{model_tested}\"\n",
+    "experiment_results = evaluate(\n",
+    "    predict_custom_agent_answer,\n",
+    "    data=dataset_name,\n",
+    "    evaluators=[answer_evaluator, check_trajectory_custom],\n",
+    "    experiment_prefix=experiment_prefix + \"-answer-and-tool-use\",\n",
+    "    num_repetitions=3,\n",
+    "    metadata={\"version\": metadata},\n",
+    ")"
+   ]
+  },
   {
    "attachments": {
     "953411a8-f352-4c8f-a923-d3ff171c6080.png": {
@@ -1201,11 +1234,12 @@
     "\n",
     "What we can see: \n",
     "\n",
-    "* ReAct agent has more freedom to select different trajectories of tool use.\n",
-    "* But, this means that it can deviate from our expected control flow, reducing reliability.\n",
-    "* With `GPT-4o`, the difference is fairly minor between our two agents, with `ReAct` scoring 73% vs `Custom` scoring 100% on tool use trajectory.\n",
-    "* However, as we move to `Firefunction-v2` with `Llama 3` we can see that the performance in our `ReAct` agent drops to `53%` on tool use trajectory.\n",
-    "* One of the powerful things about our `Custom` agent is that it retains reliable performance even as we dial down model capacity.\n",
+    "* ReAct agent has freedom to select many tool use trajectories.\n",
+    "* But, LLMs are non-deterministic and tool-calling is challenging.\n",
+    "* The tool calling trajectory score is worse for ReAct than Custom.\n",
+    "* The effect is further worse when we moved to `Firefunction-v2`.\n",
+    "* But the Custom agent shows strong performance in reasoning trajectory.\n",
+    "* This is true even as we dial down model capacity.\n",
     "* It can even be run locally [see here](https://github.com/langchain-ai/langgraph/blob/main/examples/rag/langgraph_crag_local.ipynb)! \n",
     "\n",
     "![Screenshot 2024-06-23 at 1.32.31 PM.png](attachment:953411a8-f352-4c8f-a923-d3ff171c6080.png)"