{
 "cells": [
  {
   "attachments": {
    "2701e468-83b5-40f8-b1d3-05d6e6aedcdb.png": {
     "image/png": "[encoded data, 129548 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "ae9e7280-1997-4ab6-9810-31dcd7c68d92",
   "metadata": {},
   "source": [
    "# Building and Testing Corrective RAG\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://drive.google.com/file/d/1KUCIBFtDytL2gFgyFC46t_Yjp3jDjdUF/view?usp=sharing)\n",
    "\n",
    "[Corrective-RAG (CRAG)](https://arxiv.org/abs/2401.15884) is a strategy for RAG that incorporates self-reflection / self-grading on retrieved documents. \n",
    "\n",
    "Here's we're going to show two different way to implement Corrective-RAG: \n",
    "\n",
    "(1) We will implement this as a [ReAct agent](https://react-lm.github.io/). \n",
    "\n",
    "(2) We will implement this agent as a custom / user-defined control flow in LangGraph. \n",
    "\n",
    "Then, we will show how to test these agents. \n",
    "\n",
    "(1) We'll build an [evaluation set of question-answer pairs for RAG in LangSmith](https://docs.smith.langchain.com/tutorials/Developers/agents#eval). \n",
    "\n",
    "(2) We'll evaluate end-to-end performance of our agents along with the specific reasoning trace of each one.\n",
    "\n",
    "This notebook will reference ideas from [these slides](https://docs.google.com/presentation/d/1QWkXi4DYjfw94eHcy9RMLqpQdJtS2C_kx_u7wAUvlZE/edit?usp=sharing).\n",
    "\n",
    "![Screenshot 2024-06-24 at 10.56.06 AM.png](attachment:2701e468-83b5-40f8-b1d3-05d6e6aedcdb.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07af6cb7-39e7-4011-a19e-9953df1f85c8",
   "metadata": {},
   "source": [
    "## Environment\n",
    "\n",
    "We'll start with [OpenAI](https://python.langchain.com/v0.2/docs/integrations/chat/openai/) as our LLM, but this can be easily swapped. Optionally, we also show [Fireworks](https://fireworks.ai/). \n",
    "\n",
    "We'll use [Tavily](https://python.langchain.com/v0.2/docs/integrations/tools/tavily_search/) for web search.\n",
    "\n",
    "We'll use a vectorstore with [OpenAI embeddings](https://python.langchain.com/v0.2/docs/integrations/text_embedding/openai/#embed-documents).\n",
    "\n",
    "We'll use [LangSmith](https://docs.smith.langchain.com/) for tracing and evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "045a7bcd-fdf7-4c30-a5b7-a6a4bf40d2c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install -U langchain tavily-python langgraph matplotlib langchain_community tiktoken langchain-openai scikit-learn langchain_fireworks langchainhub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8cd373f6-75b6-4122-a00c-1f72b0c0ad1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import getpass\n",
    "\n",
    "\n",
    "def _set_env(var: str):\n",
    "    if not os.environ.get(var):\n",
    "        os.environ[var] = getpass.getpass(f\"{var}: \")\n",
    "\n",
    "\n",
    "_set_env(\"OPENAI_API_KEY\")\n",
    "_set_env(\"LANGCHAIN_API_KEY\")\n",
    "_set_env(\"TAVILY_API_KEY\")\n",
    "_set_env(\"FIREWORKS_API_KEY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b59fdc0-ee15-4fb0-b004-86089c26acd8",
   "metadata": {},
   "source": [
    "You can also set API keys and env variables here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "54b183a4-d895-4d73-aecf-7890dac7b5c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"xxx\"\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = \"xxx\"\n",
    "os.environ[\"TAVILY_API_KEY\"] = \"xxx\"\n",
    "os.environ[\"FIREWORKS_API_KEY\"] = \"xxx\"\n",
    "\n",
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n",
    "os.environ[\"LANGCHAIN_PROJECT\"] = \"corrective-rag-agent-testing\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47904568-d17f-4fe0-8d02-2c29dde1401f",
   "metadata": {},
   "source": [
    "## LLM\n",
    "\n",
    "Select the LLM to use for both of our agents.\n",
    "\n",
    "This can be easily swapped with [other models that support tool use](https://python.langchain.com/v0.2/docs/integrations/chat/)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8d0866a9-70d4-4c43-b538-80ddb50e0e32",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "model_tested = \"gpt-4o\"\n",
    "metadata = \"CRAG, gpt-4o\"\n",
    "llm = ChatOpenAI(model_name=model_tested, temperature=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fb6216b-6ba8-4bb6-aa8a-07d185fbcb35",
   "metadata": {},
   "source": [
    "For example, we can also test [Firefunction-v2](https://fireworks.ai/blog/firefunction-v2-launch-post), a Llama 3-70b fine-tuned for tool calling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b7dc9931-f037-40b2-a9e9-da22aee2fe62",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_fireworks import ChatFireworks\n",
    "\n",
    "model_tested = \"firefunction-v2\"\n",
    "metadata = \"CRAG, firefunction-v2\"\n",
    "llm = ChatFireworks(model=\"accounts/fireworks/models/firefunction-v2\", temperature=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3e646bb-0f9b-40dc-b382-8e4338df2cfe",
   "metadata": {},
   "source": [
    "We can test [Mistal-Lage-v2](https://mistral.ai/news/mistral-large-2407/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "9a454a8f-80a7-417a-819d-37a3e52e0412",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_mistralai.chat_models import ChatMistralAI\n",
    "\n",
    "model_tested = \"mistral-large-2407\"\n",
    "metadata = \"CRAG, mistral-large-2407\"\n",
    "llm = ChatMistralAI(model=model_tested, temperature=0)"
   ]
  },
  {
   "attachments": {
    "599c43c4-9da4-4875-b644-f3ed2b34a179.png": {
     "image/png": "[encoded data, 387000 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "e29e8360-267a-4509-bfa4-68ae4656eb85",
   "metadata": {},
   "source": [
    "## ReAct Agent\n",
    "\n",
    "First, we'll define our [ReAct agent](https://python.langchain.com/v0.2/docs/tutorials/agents/) for corrective RAG!\n",
    "\n",
    "LLM powered autonomous agents [use three things to accomplish tasks](https://lilianweng.github.io/posts/2023-06-23-agent/):\n",
    "\n",
    "- `Tools`: Actions to gather external information that is missing from the model weights\n",
    "- `Planning`: Often this is self-criticism and self-reflection over past actions\n",
    "- `Memory`: Ability to recall information (e.g., from the user or from past actions)\n",
    "\n",
    "[ReAct](https://arxiv.org/abs/2210.03629) is a popular that incorporates explicit steps for LLM to:\n",
    "\n",
    "- `Action`: Choose an action (e.g., often a tool)\n",
    "- `Observation`: Observe the outputs of the action\n",
    "- `Think`: Reason about the next step\n",
    "\n",
    "These steps follow a loop until the agent stops calling tools and decides to return a final answer.\n",
    "\n",
    "Let's set up our ReAct agent by first defining a few tools.\n",
    "\n",
    "![Screenshot 2024-06-23 at 9.31.55 AM.png](attachment:599c43c4-9da4-4875-b644-f3ed2b34a179.png)\n",
    "\n",
    "### Tools\n",
    "\n",
    "First, we'll index 3 blog posts and store them in a vectorstore and define [this as a tool](https://python.langchain.com/v0.2/docs/concepts/#tools).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d3560759-9e2d-4362-a06f-3a257d0a3088",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "USER_AGENT environment variable not set, consider setting it to identify your requests.\n"
     ]
    }
   ],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_community.vectorstores import SKLearnVectorStore\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_core.tools import tool\n",
    "\n",
    "# List of URLs to load documents from\n",
    "urls = [\n",
    "    \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n",
    "    \"https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/\",\n",
    "    \"https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/\",\n",
    "]\n",
    "\n",
    "# Load documents from the URLs\n",
    "docs = [WebBaseLoader(url).load() for url in urls]\n",
    "docs_list = [item for sublist in docs for item in sublist]\n",
    "\n",
    "# Initialize a text splitter with specified chunk size and overlap\n",
    "text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n",
    "    chunk_size=250, chunk_overlap=0\n",
    ")\n",
    "\n",
    "# Split the documents into chunks\n",
    "doc_splits = text_splitter.split_documents(docs_list)\n",
    "\n",
    "# Add the document chunks to the \"vector store\" using OpenAIEmbeddings\n",
    "vectorstore = SKLearnVectorStore.from_documents(\n",
    "    documents=doc_splits,\n",
    "    embedding=OpenAIEmbeddings(),\n",
    ")\n",
    "retriever = vectorstore.as_retriever(k=4)"
   ]
  },
  {
   "attachments": {
    "4fee1fb2-a317-49da-8792-d9bb4edd1444.png": {
     "image/png": "[encoded data, 89892 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "35c682d5-f753-403d-ae13-8623827ad22f",
   "metadata": {},
   "source": [
    "When we define this as a `tool` and bind it too the LLM, then the LLM can:\n",
    "\n",
    "* Choose to call this tool\n",
    "* Formulate the payload needed to run the tool\n",
    "\n",
    "![Screenshot 2024-06-24 at 12.17.54 PM.png](attachment:4fee1fb2-a317-49da-8792-d9bb4edd1444.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "a8cd1080-f8da-41b8-8e52-9a75d9ce1258",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a tool, which we will connect to our agent\n",
    "@tool\n",
    "def retrieve_documents(query: str) -> list:\n",
    "    \"\"\"Retrieve documents from the vector store based on the query.\"\"\"\n",
    "    return retriever.invoke(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02566999-a411-4154-a021-e1e6edf3264b",
   "metadata": {},
   "source": [
    "We'll create a tool that instructs our agent to reflect on the retrieved documents.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "3a4f8620-d072-49d8-bbeb-76fbf9ee4097",
   "metadata": {},
   "outputs": [],
   "source": [
    "@tool\n",
    "def grade_document_retrieval(step_by_step_reasoning: str, score: int) -> str:\n",
    "    \"\"\"You are a teacher grading a quiz. You will be given:\n",
    "    1/ a QUESTION\n",
    "    2/ a set of comma separated FACTS provided by the student\n",
    "\n",
    "    You are grading RELEVANCE RECALL:\n",
    "    A score of 1 means that ANY of the FACTS are relevant to the QUESTION.\n",
    "    A score of 0 means that NONE of the FACTS are relevant to the QUESTION.\n",
    "\n",
    "    If your score is 1: then call a tool to generate the answer, generate_answer\n",
    "    If your score is 0: then call a tool to perform web search, web_search.\"\"\"\n",
    "    if score == 1:\n",
    "        return \"Docs are relevant. Generate the answer to the question.\"\n",
    "    return \"Docs are not relevant. Use web search to find more documents.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfb9090a-f339-41d4-bebd-6e89ebdb3d6e",
   "metadata": {},
   "source": [
    "We'll create a tool for web search using Tavily.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "f3411e9a-53ef-45d3-9c95-0f64b7218c25",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.schema import Document\n",
    "from langchain_community.tools.tavily_search import TavilySearchResults\n",
    "\n",
    "web_search_tool = TavilySearchResults()\n",
    "\n",
    "\n",
    "@tool\n",
    "def web_search(query: str) -> str:\n",
    "    \"\"\"Run web search on the question.\"\"\"\n",
    "    web_results = web_search_tool.invoke({\"query\": query})\n",
    "    return [\n",
    "        Document(page_content=d[\"content\"], metadata={\"url\": d[\"url\"]})\n",
    "        for d in web_results\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad568da3-8f42-4742-8744-bc651fb2269b",
   "metadata": {},
   "source": [
    "Finally, we'll create a tool that instructs our agent to produce the final distillated answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e9b474e9-cebd-4452-a694-4e4cc622fd52",
   "metadata": {},
   "outputs": [],
   "source": [
    "@tool\n",
    "def generate_answer(answer: str) -> str:\n",
    "    \"\"\"You are an assistant for question-answering tasks.\n",
    "    Use the retrieved documents to answer the user question.\n",
    "    If you don't know the answer, just say that you don't know.\n",
    "    Use three sentences maximum and keep the answer concise\"\"\"\n",
    "    return f\"Here is the answer to the user question: {answer}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5428f9b8-38b0-427f-a340-b3d769ea5889",
   "metadata": {},
   "source": [
    "Now, we store these in a tools list, which we will bind to our LLM.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "c3856ef6-e5db-4fa6-97e2-c9495f159d77",
   "metadata": {},
   "outputs": [],
   "source": [
    "tools = [retrieve_documents, grade_document_retrieval, web_search, generate_answer]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49c40cb2-5f29-488b-b524-bb388c56fc84",
   "metadata": {},
   "source": [
    "### Planning\n",
    "\n",
    "Now, we'll define an LLM assistant for our agent, which can use the tools we defined above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "13f4be82-ef3b-4d3a-901d-82eb28abb594",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Annotated, List\n",
    "from typing_extensions import TypedDict\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.runnables import Runnable, RunnableConfig\n",
    "from langgraph.graph.message import AnyMessage, add_messages\n",
    "\n",
    "\n",
    "class State(TypedDict):\n",
    "    messages: Annotated[list[AnyMessage], add_messages]\n",
    "\n",
    "\n",
    "class Assistant:\n",
    "    def __init__(self, runnable: Runnable):\n",
    "        \"\"\"\n",
    "        Initialize the Assistant with a runnable object.\n",
    "\n",
    "        Args:\n",
    "            runnable (Runnable): The runnable instance to invoke.\n",
    "        \"\"\"\n",
    "        self.runnable = runnable\n",
    "\n",
    "    def __call__(self, state: State, config: RunnableConfig):\n",
    "        \"\"\"\n",
    "        Call method to invoke the LLM and handle its responses.\n",
    "        Re-prompt the assistant if the response is not a tool call or meaningful text.\n",
    "\n",
    "        Args:\n",
    "            state (State): The current state containing messages.\n",
    "            config (RunnableConfig): The configuration for the runnable.\n",
    "\n",
    "        Returns:\n",
    "            dict: The final state containing the updated messages.\n",
    "        \"\"\"\n",
    "        while True:\n",
    "            result = self.runnable.invoke(state)  # Invoke the LLM\n",
    "            if not result.tool_calls and (\n",
    "                not result.content\n",
    "                or isinstance(result.content, list)\n",
    "                and not result.content[0].get(\"text\")\n",
    "            ):\n",
    "                messages = state[\"messages\"] + [(\"user\", \"Respond with a real output.\")]\n",
    "                state = {**state, \"messages\": messages}\n",
    "            else:\n",
    "                break\n",
    "        return {\"messages\": result}\n",
    "\n",
    "\n",
    "# Create the primary assistant prompt template\n",
    "primary_assistant_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \" You are a helpful assistant tasked with answering user questions using the provided vector store. \"\n",
    "            \" Use the provided vector store to retrieve documents. Then grade them to ensure they are relevant before answering the question. \",\n",
    "        ),\n",
    "        (\"placeholder\", \"{messages}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "# Prompt our LLM and bind tools\n",
    "assistant_runnable = primary_assistant_prompt | llm.bind_tools(tools)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcd86ae2-26ac-499a-9ccd-f69adc613801",
   "metadata": {},
   "source": [
    "### Memory\n",
    "\n",
    "Finally, we'll using [LangGraph](https://langchain-ai.github.io/langgraph/) to orchestrate our agent.\n",
    "\n",
    "LangGraph is a library for building agents, which allows us to define 2 nodes:\n",
    "\n",
    "- `Assistant`, which will contain the `assistant_runnable` defined above\n",
    "- `Tool`, which will call the tool when the assistant instructs and return the tool outputs to our assistant\n",
    " \n",
    "LangGraph allows us to define state, which will serve as short-term memory over the lifetime of our agent.\n",
    "\n",
    "First, there are some utilities that we'll use to help define the graph. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "a0c66d91-8804-4fd7-a29c-2852faa31d19",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.runnables import RunnableLambda\n",
    "from langchain_core.messages import ToolMessage\n",
    "from langgraph.prebuilt import ToolNode\n",
    "\n",
    "\n",
    "def create_tool_node_with_fallback(tools: list) -> dict:\n",
    "    return ToolNode(tools).with_fallbacks(\n",
    "        [RunnableLambda(handle_tool_error)], exception_key=\"error\"\n",
    "    )\n",
    "\n",
    "\n",
    "def handle_tool_error(state: State) -> dict:\n",
    "    error = state.get(\"error\")\n",
    "    tool_calls = state[\"messages\"][-1].tool_calls\n",
    "    return {\n",
    "        \"messages\": [\n",
    "            ToolMessage(\n",
    "                content=f\"Error: {repr(error)}\\n please fix your mistakes.\",\n",
    "                tool_call_id=tc[\"id\"],\n",
    "            )\n",
    "            for tc in tool_calls\n",
    "        ]\n",
    "    }"
   ]
  },
  {
   "attachments": {
    "3bc82b64-f1e5-44b0-aca6-29e6928a1554.png": {
     "image/png": "[encoded data, 96788 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "95463344-47ad-4639-bede-f0a61746620b",
   "metadata": {},
   "source": [
    "Now, we'll define our graph.\n",
    "\n",
    "We'll simply cycle between 2 nodes: \n",
    "\n",
    "* Our `assistant` node reasons and plans which tool to use\n",
    "* Our `tool` node that executes the tool\n",
    "\n",
    "It will continue as long as our `assistant` is making a tool call. \n",
    "\n",
    "This shows the general flow:\n",
    "\n",
    "![Screenshot 2024-06-24 at 12.35.48 PM.png](attachment:3bc82b64-f1e5-44b0-aca6-29e6928a1554.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "3e13495c-dfe2-4eea-8786-7d3b8f8138e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "[encoded data, 10812 characters]",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from langgraph.checkpoint.memory import MemorySaver\n",
    "from langgraph.graph import START, END, StateGraph\n",
    "from langgraph.prebuilt import tools_condition\n",
    "from IPython.display import Image, display\n",
    "\n",
    "# Graph\n",
    "builder = StateGraph(State)\n",
    "\n",
    "# Define nodes: these do the work\n",
    "builder.add_node(\"assistant\", Assistant(assistant_runnable))\n",
    "builder.add_node(\"tools\", create_tool_node_with_fallback(tools))\n",
    "\n",
    "# Define edges: these determine how the control flow moves\n",
    "builder.add_edge(START, \"assistant\")\n",
    "builder.add_conditional_edges(\n",
    "    \"assistant\",\n",
    "    # If the latest message (result) from assistant is a tool call -> tools_condition routes to tools\n",
    "    # If the latest message (result) from assistant is a not a tool call -> tools_condition routes to END\n",
    "    tools_condition,\n",
    ")\n",
    "builder.add_edge(\"tools\", \"assistant\")\n",
    "\n",
    "# The checkpointer lets the graph persist its state\n",
    "memory = MemorySaver()\n",
    "react_graph = builder.compile(checkpointer=memory)\n",
    "\n",
    "# Show\n",
    "display(Image(react_graph.get_graph(xray=True).draw_mermaid_png()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40614db2-fca0-43c7-8f76-ca38388c62ab",
   "metadata": {},
   "source": [
    "Finally, we can our agent!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "71301681-ddc5-46a9-8092-96693dcdef1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "\n",
    "\n",
    "def predict_react_agent_answer(example: dict):\n",
    "    \"\"\"Use this for answer evaluation\"\"\"\n",
    "\n",
    "    config = {\"configurable\": {\"thread_id\": str(uuid.uuid4())}}\n",
    "    messages = react_graph.invoke({\"messages\": (\"user\", example[\"input\"])}, config)\n",
    "    return {\"response\": messages[\"messages\"][-1].content, \"messages\": messages}\n",
    "\n",
    "\n",
    "example = {\"input\": \"What are the types of agent memory?\"}\n",
    "response = predict_react_agent_answer(example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "87c2aea6-49d9-4d32-9da6-7851fda021ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['retrieve_documents', 'grade_document_retrieval', 'generate_answer']"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def find_tool_calls_react(messages):\n",
    "    \"\"\"\n",
    "    Find all tool calls in the messages returned from the ReAct agent\n",
    "    \"\"\"\n",
    "    tool_calls = [\n",
    "        tc[\"name\"] for m in messages[\"messages\"] for tc in getattr(m, \"tool_calls\", [])\n",
    "    ]\n",
    "    return tool_calls\n",
    "\n",
    "\n",
    "find_tool_calls_react(response[\"messages\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6efe335f-f362-460e-9874-0a8e93eca516",
   "metadata": {},
   "source": [
    "With LangSmith set, we can see the trace:\n",
    "\n",
    "https://smith.langchain.com/public/4510c99f-abdf-4fdd-aebf-270f4e49e02f/r\n",
    "\n",
    "We see follow the tool calls to clearly see the reasoning trace:\n",
    "\n",
    "* Our agent first calls `retrieve_documents` tool to get documents.\n",
    " \n",
    "* Then, it calls `grade_document_retrieval` tool and deems the documents to be relevant.\n",
    " \n",
    "* Finally, it calls `generate_answer` to produce the final answer."
   ]
  },
  {
   "attachments": {
    "d3d350f1-b69a-4843-9860-24206515b97e.png": {
     "image/png": "[encoded data, 109540 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "ec64535c-75ec-41eb-91d6-ca75db35c3bf",
   "metadata": {},
   "source": [
    "## Custom LangGraph Agent\n",
    "\n",
    "Now, we'll do the same as shown above. But rather that following the ReAct agent control flow, we'll use a custom control flow.\n",
    "\n",
    "Here we'll explicitly define the majority of the control flow, only using an LLM to define a single branch point following grading.\n",
    "\n",
    "![Screenshot 2024-06-23 at 9.56.52 AM.png](attachment:d3d350f1-b69a-4843-9860-24206515b97e.png)\n",
    "\n",
    "First, we'll define a RAG chain to produce our final answer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "9520cb91-5ba2-4915-8aa3-ce0e86d4381b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts import PromptTemplate\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "    template=\"\"\"You are an assistant for question-answering tasks. \n",
    "    \n",
    "    Use the following documents to answer the question. \n",
    "    \n",
    "    If you don't know the answer, just say that you don't know. \n",
    "    \n",
    "    Use three sentences maximum and keep the answer concise:\n",
    "    Question: {question} \n",
    "    Documents: {documents} \n",
    "    Answer: \n",
    "    \"\"\",\n",
    "    input_variables=[\"question\", \"documents\"],\n",
    ")\n",
    "\n",
    "rag_chain = prompt | llm | StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9658fc32-0e13-4044-95dd-e16d48442e6e",
   "metadata": {},
   "source": [
    "Now, we'll define a document grader to review our documents.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "c949a92e-9f76-41a0-8008-728b9e51dc1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.pydantic_v1 import BaseModel, Field\n",
    "\n",
    "\n",
    "# Data model for the output\n",
    "class GradeDocuments(BaseModel):\n",
    "    \"\"\"Binary score for relevance check on retrieved documents.\"\"\"\n",
    "\n",
    "    binary_score: str = Field(\n",
    "        description=\"Documents are relevant to the question, 'yes' or 'no'\"\n",
    "    )\n",
    "\n",
    "\n",
    "# LLM with tool call\n",
    "structured_llm_grader = llm.with_structured_output(GradeDocuments)\n",
    "\n",
    "# Prompt\n",
    "system = \"\"\"You are a teacher grading a quiz. You will be given: \n",
    "1/ a QUESTION \n",
    "2/ a set of comma separated FACTS provided by the student\n",
    "\n",
    "You are grading RELEVANCE RECALL:\n",
    "A score of 1 means that ANY of the FACTS are relevant to the QUESTION. \n",
    "A score of 0 means that NONE of the FACTS are relevant to the QUESTION. \n",
    "1 is the highest (best) score. 0 is the lowest score you can give. \n",
    "\n",
    "Explain your reasoning in a step-by-step manner. Ensure your reasoning and conclusion are correct. \n",
    "\n",
    "Avoid simply stating the correct answer at the outset.\"\"\"\n",
    "\n",
    "grade_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system),\n",
    "        (\"human\", \"FACTS: \\n\\n {documents} \\n\\n QUESTION: {question}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "retrieval_grader = grade_prompt | structured_llm_grader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5c9763d-40ac-4466-8b9e-ef606b8df699",
   "metadata": {},
   "source": [
    "Finally, we'll define our graph.\n",
    "\n",
    "Before, our state was a set of messages.\n",
    "\n",
    "Here, our state is a dict of specific entities that we want to persist.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "769ade7f-e19f-467c-b359-ca7e5ded45c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "[encoded data, 19676 characters]",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Image, display\n",
    "\n",
    "\n",
    "class GraphState(TypedDict):\n",
    "    \"\"\"\n",
    "    Represents the state of our graph.\n",
    "\n",
    "    Attributes:\n",
    "        question: question\n",
    "        generation: LLM generation\n",
    "        search: whether to add search\n",
    "        documents: list of documents\n",
    "    \"\"\"\n",
    "\n",
    "    question: str\n",
    "    generation: str\n",
    "    search: str\n",
    "    documents: List[str]\n",
    "    steps: List[str]\n",
    "\n",
    "\n",
    "def retrieve(state):\n",
    "    \"\"\"\n",
    "    Retrieve documents\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        state (dict): New key added to state, documents, that contains retrieved documents\n",
    "    \"\"\"\n",
    "    question = state[\"question\"]\n",
    "    documents = retriever.invoke(question)\n",
    "    steps = state[\"steps\"]\n",
    "    steps.append(\"retrieve_documents\")\n",
    "    return {\"documents\": documents, \"question\": question, \"steps\": steps}\n",
    "\n",
    "\n",
    "def generate(state):\n",
    "    \"\"\"\n",
    "    Generate answer\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        state (dict): New key added to state, generation, that contains LLM generation\n",
    "    \"\"\"\n",
    "\n",
    "    question = state[\"question\"]\n",
    "    documents = state[\"documents\"]\n",
    "    generation = rag_chain.invoke({\"documents\": documents, \"question\": question})\n",
    "    steps = state[\"steps\"]\n",
    "    steps.append(\"generate_answer\")\n",
    "    return {\n",
    "        \"documents\": documents,\n",
    "        \"question\": question,\n",
    "        \"generation\": generation,\n",
    "        \"steps\": steps,\n",
    "    }\n",
    "\n",
    "\n",
    "def grade_documents(state):\n",
    "    \"\"\"\n",
    "    Determines whether the retrieved documents are relevant to the question.\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        state (dict): Updates documents key with only filtered relevant documents\n",
    "    \"\"\"\n",
    "\n",
    "    question = state[\"question\"]\n",
    "    documents = state[\"documents\"]\n",
    "    steps = state[\"steps\"]\n",
    "    steps.append(\"grade_document_retrieval\")\n",
    "    filtered_docs = []\n",
    "    search = \"No\"\n",
    "    for d in documents:\n",
    "        score = retrieval_grader.invoke(\n",
    "            {\"question\": question, \"documents\": d.page_content}\n",
    "        )\n",
    "        grade = score.binary_score\n",
    "        if grade == \"yes\":\n",
    "            filtered_docs.append(d)\n",
    "        else:\n",
    "            search = \"Yes\"\n",
    "            continue\n",
    "    return {\n",
    "        \"documents\": filtered_docs,\n",
    "        \"question\": question,\n",
    "        \"search\": search,\n",
    "        \"steps\": steps,\n",
    "    }\n",
    "\n",
    "\n",
    "def web_search(state):\n",
    "    \"\"\"\n",
    "    Web search based on the re-phrased question.\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        state (dict): Updates documents key with appended web results\n",
    "    \"\"\"\n",
    "\n",
    "    question = state[\"question\"]\n",
    "    documents = state.get(\"documents\", [])\n",
    "    steps = state[\"steps\"]\n",
    "    steps.append(\"web_search\")\n",
    "    web_results = web_search_tool.invoke({\"query\": question})\n",
    "    documents.extend(\n",
    "        [\n",
    "            Document(page_content=d[\"content\"], metadata={\"url\": d[\"url\"]})\n",
    "            for d in web_results\n",
    "        ]\n",
    "    )\n",
    "    return {\"documents\": documents, \"question\": question, \"steps\": steps}\n",
    "\n",
    "\n",
    "def decide_to_generate(state):\n",
    "    \"\"\"\n",
    "    Determines whether to generate an answer, or re-generate a question.\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        str: Binary decision for next node to call\n",
    "    \"\"\"\n",
    "    search = state[\"search\"]\n",
    "    if search == \"Yes\":\n",
    "        return \"search\"\n",
    "    else:\n",
    "        return \"generate\"\n",
    "\n",
    "\n",
    "# Graph\n",
    "workflow = StateGraph(GraphState)\n",
    "\n",
    "# Define the nodes\n",
    "workflow.add_node(\"retrieve\", retrieve)  # retrieve\n",
    "workflow.add_node(\"grade_documents\", grade_documents)  # grade documents\n",
    "workflow.add_node(\"generate\", generate)  # generatae\n",
    "workflow.add_node(\"web_search\", web_search)  # web search\n",
    "\n",
    "# Build graph\n",
    "workflow.set_entry_point(\"retrieve\")\n",
    "workflow.add_edge(\"retrieve\", \"grade_documents\")\n",
    "workflow.add_conditional_edges(\n",
    "    \"grade_documents\",\n",
    "    decide_to_generate,\n",
    "    {\n",
    "        \"search\": \"web_search\",\n",
    "        \"generate\": \"generate\",\n",
    "    },\n",
    ")\n",
    "workflow.add_edge(\"web_search\", \"generate\")\n",
    "workflow.add_edge(\"generate\", END)\n",
    "\n",
    "custom_graph = workflow.compile()\n",
    "\n",
    "display(Image(custom_graph.get_graph(xray=True).draw_mermaid_png()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3dbbfde9-37cc-427f-abee-64cc3366c3b7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def predict_custom_agent_answer(example: dict):\n",
    "    config = {\"configurable\": {\"thread_id\": str(uuid.uuid4())}}\n",
    "    state_dict = custom_graph.invoke(\n",
    "        {\"question\": example[\"input\"], \"steps\": []}, config\n",
    "    )\n",
    "    return {\"response\": state_dict[\"generation\"], \"steps\": state_dict[\"steps\"]}\n",
    "\n",
    "\n",
    "example = {\"input\": \"What are the types of agent memory?\"}\n",
    "response = predict_custom_agent_answer(example)\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf4e5b46-4905-4e31-9c9d-420d7806b987",
   "metadata": {},
   "source": [
    "We can see from our trace that our agent follows the specified control flow:\n",
    "\n",
    "https://smith.langchain.com/public/7f14c989-4505-4bc1-9c87-9126899f8714/r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca8c5f4-4053-4380-98a6-3b6eef27883e",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Now we've defined two different agent architectures that do roughly the same thing!\n",
    "\n",
    "We can evaluate them. See our [conceptual guide](https://docs.smith.langchain.com/concepts/evaluation#agents) for context on agent evaluation.\n",
    "\n",
    "### Response\n",
    "\n",
    "First, we can assess how well [our agent performs on a set of question-answer pairs](https://docs.smith.langchain.com/tutorials/Developers/agents#response-evaluation).\n",
    "\n",
    "We'll create a dataset and save it in LangSmith.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2ef03dad-d161-4002-abb4-857dc034d2fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import Client\n",
    "\n",
    "client = Client()\n",
    "\n",
    "# Create a dataset\n",
    "examples = [\n",
    "    (\n",
    "        \"How does the ReAct agent use self-reflection? \",\n",
    "        \"ReAct integrates reasoning and acting, performing actions - such tools like Wikipedia search API - and then observing / reasoning about the tool outputs.\",\n",
    "    ),\n",
    "    (\n",
    "        \"What are the types of biases that can arise with few-shot prompting?\",\n",
    "        \"The biases that can arise with few-shot prompting include (1) Majority label bias, (2) Recency bias, and (3) Common token bias.\",\n",
    "    ),\n",
    "    (\n",
    "        \"What are five types of adversarial attacks?\",\n",
    "        \"Five types of adversarial attacks are (1) Token manipulation, (2) Gradient based attack, (3) Jailbreak prompting, (4) Human red-teaming, (5) Model red-teaming.\",\n",
    "    ),\n",
    "    (\n",
    "        \"Who did the Chicago Bears draft first in the 2024 NFL draft”?\",\n",
    "        \"The Chicago Bears drafted Caleb Williams first in the 2024 NFL draft.\",\n",
    "    ),\n",
    "    (\"Who won the 2024 NBA finals?\", \"The Boston Celtics on the 2024 NBA finals\"),\n",
    "]\n",
    "\n",
    "# Save it\n",
    "dataset_name = \"Corrective RAG Agent Testing\"\n",
    "if not client.has_dataset(dataset_name=dataset_name):\n",
    "    dataset = client.create_dataset(dataset_name=dataset_name)\n",
    "    inputs, outputs = zip(\n",
    "        *[({\"input\": text}, {\"output\": label}) for text, label in examples]\n",
    "    )\n",
    "    client.create_examples(inputs=inputs, outputs=outputs, dataset_id=dataset.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94a7586a-a28a-4405-9ba0-a366c33fac1c",
   "metadata": {},
   "source": [
    "Now, we'll use an `LLM as a grader` to compare both agent responses to our ground truth reference answer.\n",
    "\n",
    "[Here](https://smith.langchain.com/hub/rlm/rag-answer-vs-reference) is the default prompt that we can use.\n",
    "\n",
    "We'll use `gpt-4o` as our LLM grader.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "88a78e4e-c1d4-456a-9f4c-a207e3255086",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import hub\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "# Grade prompt\n",
    "grade_prompt_answer_accuracy = hub.pull(\"langchain-ai/rag-answer-vs-reference\")\n",
    "\n",
    "\n",
    "def answer_evaluator(run, example) -> dict:\n",
    "    \"\"\"\n",
    "    A simple evaluator for RAG answer accuracy\n",
    "    \"\"\"\n",
    "\n",
    "    # Get the question, the ground truth reference answer, RAG chain answer prediction\n",
    "    input_question = example.inputs[\"input\"]\n",
    "    reference = example.outputs[\"output\"]\n",
    "    prediction = run.outputs[\"response\"]\n",
    "\n",
    "    # Define an LLM grader\n",
    "    llm = ChatOpenAI(model=\"gpt-4o\", temperature=0)\n",
    "    answer_grader = grade_prompt_answer_accuracy | llm\n",
    "\n",
    "    # Run evaluator\n",
    "    score = answer_grader.invoke(\n",
    "        {\n",
    "            \"question\": input_question,\n",
    "            \"correct_answer\": reference,\n",
    "            \"student_answer\": prediction,\n",
    "        }\n",
    "    )\n",
    "    score = score[\"Score\"]\n",
    "    return {\"key\": \"answer_v_reference_score\", \"score\": score}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb515ab4-6ac1-4049-836a-f980dec1d8c1",
   "metadata": {},
   "source": [
    "### Trajectory\n",
    "\n",
    "Second, [we can assess the list of tool calls](https://docs.smith.langchain.com/tutorials/Developers/agents#trajectory) that each agent makes relative to expected trajectories.\n",
    "\n",
    "This evaluates the specific reasoning traces taken by our agents!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a6d07b7c-a6ab-4b71-9e31-fdb61348b7cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith.schemas import Example, Run\n",
    "\n",
    "# Reasoning traces that we expect the agents to take\n",
    "expected_trajectory_1 = [\n",
    "    \"retrieve_documents\",\n",
    "    \"grade_document_retrieval\",\n",
    "    \"web_search\",\n",
    "    \"generate_answer\",\n",
    "]\n",
    "expected_trajectory_2 = [\n",
    "    \"retrieve_documents\",\n",
    "    \"grade_document_retrieval\",\n",
    "    \"generate_answer\",\n",
    "]\n",
    "\n",
    "\n",
    "def check_trajectory_react(root_run: Run, example: Example) -> dict:\n",
    "    \"\"\"\n",
    "    Check if all expected tools are called in exact order and without any additional tool calls.\n",
    "    \"\"\"\n",
    "    messages = root_run.outputs[\"messages\"]\n",
    "    tool_calls = find_tool_calls_react(messages)\n",
    "    print(f\"Tool calls ReAct agent: {tool_calls}\")\n",
    "    if tool_calls == expected_trajectory_1 or tool_calls == expected_trajectory_2:\n",
    "        score = 1\n",
    "    else:\n",
    "        score = 0\n",
    "\n",
    "    return {\"score\": int(score), \"key\": \"tool_calls_in_exact_order\"}\n",
    "\n",
    "\n",
    "def check_trajectory_custom(root_run: Run, example: Example) -> dict:\n",
    "    \"\"\"\n",
    "    Check if all expected tools are called in exact order and without any additional tool calls.\n",
    "    \"\"\"\n",
    "    tool_calls = root_run.outputs[\"steps\"]\n",
    "    print(f\"Tool calls custom agent: {tool_calls}\")\n",
    "    if tool_calls == expected_trajectory_1 or tool_calls == expected_trajectory_2:\n",
    "        score = 1\n",
    "    else:\n",
    "        score = 0\n",
    "\n",
    "    return {\"score\": int(score), \"key\": \"tool_calls_in_exact_order\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0e20c97-d4b4-46bc-9148-1e46671d8754",
   "metadata": {},
   "source": [
    "Finally, we can kick off our evaluations.\n",
    "\n",
    "The results from each experiment will be logged to our `dataset_name` in LangSmith.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6faa646a-b13a-4b74-aea5-a4afe5cbd2d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith.evaluation import evaluate\n",
    "\n",
    "experiment_prefix = f\"react-agent-{model_tested}\"\n",
    "experiment_results = evaluate(\n",
    "    predict_react_agent_answer,\n",
    "    data=dataset_name,\n",
    "    evaluators=[answer_evaluator, check_trajectory_react],\n",
    "    experiment_prefix=experiment_prefix + \"-answer-and-tool-use\",\n",
    "    num_repetitions=5,\n",
    "    metadata={\"version\": metadata},\n",
    ")\n",
    "\n",
    "experiment_prefix = f\"custom-agent-{model_tested}\"\n",
    "experiment_results = evaluate(\n",
    "    predict_custom_agent_answer,\n",
    "    data=dataset_name,\n",
    "    evaluators=[answer_evaluator, check_trajectory_custom],\n",
    "    experiment_prefix=experiment_prefix + \"-answer-and-tool-use\",\n",
    "    num_repetitions=5,\n",
    "    metadata={\"version\": metadata},\n",
    ")"
   ]
  },
  {
   "attachments": {
    "953411a8-f352-4c8f-a923-d3ff171c6080.png": {
     "image/png": "[encoded data, 180396 characters]"
    },
    "c108caa7-47bc-4015-9fa0-7d73388f8460.png": {
     "image/png": "[encoded data, 79284 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "74c1ca7b-8357-481a-bc76-c4980e4ca6d5",
   "metadata": {},
   "source": [
    "We ran 3 repetitions on `Firefunction-v2` and `GPT-4o` with ReAct and Custom agents. \n",
    "\n",
    "we can see the results logged here:\n",
    "\n",
    "![Screenshot 2024-06-23 at 8.13.09 PM.png](attachment:c108caa7-47bc-4015-9fa0-7d73388f8460.png)\n",
    " \n",
    "Now we can look at a summary of the results here:\n",
    "\n",
    "https://smith.langchain.com/public/6c6c4543-1e56-4126-ab18-a84a31ff4946/d\n",
    "\n",
    "What we can see: \n",
    "\n",
    "* ReAct agent has freedom to select many tool use trajectories.\n",
    "* But, LLMs are non-deterministic and tool-calling is challenging.\n",
    "* The tool calling trajectory score is worse for ReAct than Custom.\n",
    "* The effect is further worse when we moved to `Firefunction-v2`.\n",
    "* But the Custom agent shows strong performance in reasoning trajectory.\n",
    "* This is true even as we dial down model capacity.\n",
    "* It can even be run locally [see here](https://github.com/langchain-ai/langgraph/blob/main/examples/rag/langgraph_crag_local.ipynb)! \n",
    "\n",
    "![Screenshot 2024-06-23 at 1.32.31 PM.png](attachment:953411a8-f352-4c8f-a923-d3ff171c6080.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfa533b4-007f-4f86-b146-02c56a4e667a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
