diff --git a/docs/_scripts/copy_notebooks.py b/docs/_scripts/copy_notebooks.py index 9cc2e8a76..1a3af4487 100644 --- a/docs/_scripts/copy_notebooks.py +++ b/docs/_scripts/copy_notebooks.py @@ -63,6 +63,7 @@ _MANUAL = { "human_in_the_loop/wait-user-input.ipynb", "human_in_the_loop/review-tool-calls.ipynb", "node-retries.ipynb", + "react_diagrams.png", "react-agent-structured-output.ipynb", ], "tutorials": [ diff --git a/examples/option1.png b/examples/option1.png new file mode 100644 index 000000000..485a48b65 Binary files /dev/null and b/examples/option1.png differ diff --git a/examples/option2.png b/examples/option2.png new file mode 100644 index 000000000..b24107505 Binary files /dev/null and b/examples/option2.png differ diff --git a/examples/react-agent-structured-output.ipynb b/examples/react-agent-structured-output.ipynb index 91e2b0dc6..6ffd9775f 100644 --- a/examples/react-agent-structured-output.ipynb +++ b/examples/react-agent-structured-output.ipynb @@ -8,74 +8,65 @@ "\n", "You might want your agent to return its output in a structured format. For example, if the output of the agent is used by some other downstream software, you may want the output to be in the same structured format every time the agent is invoked to ensure consistency.\n", "\n", - "This guide shows how you can do this. We will be using a basic [ReAct agent](https://langchain-ai.github.io/langgraph/how-tos/create-react-agent/) (a model node and a tool-calling node) together with a third node at the end that will format response for the user.\n", + "This notebook will walk through two different options for forcing a function calling agent to structure its output. We will be using a basic [ReAct agent](https://langchain-ai.github.io/langgraph/how-tos/create-react-agent/) (a model node and a tool-calling node) together with a third node at the end that will format response for the user. Both of the options will use the same graph structure as shown in the diagram below, but will have different mechanisms under the hood.\n", "\n", + "![React Diagrams](./react_diagrams.png)\n", + "\n", + "**Option 1**\n", + "\n", + "![option1](./option1.png)\n", + "\n", + "The first way you can force your tool calling agent to have structured output is to bind the output you would like as an additional tool for the `agent` node to use. In contrast to the basic ReAct agent, the `agent` node in this case is not selecting between `tools` and `END` but rather selecting between the specific tools it calls. The expected flow in this case is that the LLM in the `agent` node will first select the action tool, and after receiving the action tool output it will call the response tool, which will then route to the `respond` node which simply structures the arguments from the `agent` node tool call.\n", + "\n", + "**Pros and Cons**\n", + "\n", + "The benefit to this format is that you only need one LLM, and can save money and latency because of this. The downside to this option is that it isn't guaranteed that the single LLM will call the correct tool when you want it to. We can help the LLM by setting `tool_choice` to `any` when we use `bind_tools` which forces the LLM to select at least one tool at every turn, but this is far from a fool proof strategy. In addition, another downside is that the agent might call *multiple* tools, so we need to check for this explicitly in our routing function (or if we are using OpenAI we an set `parallell_tool_calling=False` to ensure only one tool is called at a time).\n", + "\n", + "**Option 2**\n", + "\n", + "![option1](./option2.png)\n", + "\n", + "The second way you can force your tool calling agent to have structured output is to use a second LLM (in this case `model_with_structured_output`) to respond to the user. \n", + "\n", + "In this case, you will define a basic ReAct agent normally, but instead of having the `agent` node choose between the `tools` node and ending the conversation, the `agent` node will choose between the `tools` node and the `respond` node. The `respond` node will contain a second LLM that uses structured output, and once called will return directly to the user. You can think of this method as basic ReAct with one extra step before responding to the user. \n", + "\n", + "**Pros and Cons**\n", + "\n", + "The benefit of this method is that it guarantees structured output (as long as `.with_structured_output` works as expected with the LLM). The downside to using this approach is that it requires making an additional LLM call before responding to the user, which can increase costs as well as latency. In addition, by not providing the `agent` node LLM with information about the desired output schema there is a risk that the `agent` LLM will fail to call the correct tools required to answer in the correct output schema.\n", + "\n", + "Note that both of these options will follow the exact same graph structure (see the diagram above), in that they are both exact replicas of the basic ReAct architecture but with a `respond` node before the end.\n", "\n", "## Setup\n", "\n", - "### Structured Output\n", + "For our setup we need to define how we want to structure our output, define our graph state, and also our tools and the models we are going to use.\n", "\n", - "First we need to define how we want to structure our output. To do this, we will use the `with_structured_output` method from LangChain, which you can read more about [here](https://python.langchain.com/v0.2/docs/how_to/structured_output/)." + "To use structured output, we will use the `with_structured_output` method from LangChain, which you can read more about [here](https://python.langchain.com/v0.2/docs/how_to/structured_output/).\n", + "\n", + "We are going to use a single tool in this example for finding the weather, and will return a structured weather response to the user." ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "from pydantic import BaseModel, Field\n", - "\n", - "class WeatherResponse(BaseModel):\n", - " temperature: float = Field(description=\"The temperature in fahrenheit\")\n", - " wind_directon: str = Field(description=\"The direction of the wind in abbreviated form\")\n", - " wind_speed: float = Field(description=\"The speed of the wind in km/h\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Graph State\n", - "\n", - "We can now define our graph state:" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "from typing import Annotated, Any\n", - "from typing_extensions import TypedDict\n", - "from langgraph.graph.message import add_messages\n", - "\n", - "class AgentState(TypedDict):\n", - " # list of chat messages from user, LLM, and tools\n", - " messages: Annotated[list, add_messages]\n", - " # Final structured response from the agent\n", - " final_response: WeatherResponse" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Tools + Models\n", - "\n", - "We can now instantiate the tools and models we are going to use in our graph. We are going to use a single tool in this example for finding the weather, and we are going to have two models in our graph, one that does the function calling and one that does the responding." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "from typing import Literal\n", + "from typing import Literal\n", "from langchain_core.tools import tool\n", "from langchain_anthropic import ChatAnthropic\n", + "from langgraph.graph import MessagesState\n", + "\n", + "class WeatherResponse(BaseModel):\n", + " \"\"\"Respond to the user with this\"\"\"\n", + " temperature: float = Field(description=\"The temperature in fahrenheit\")\n", + " wind_directon: str = Field(description=\"The direction of the wind in abbreviated form\")\n", + " wind_speed: float = Field(description=\"The speed of the wind in km/h\")\n", + "\n", + "# Inherit 'messages' key from MessagesState, which is a list of chat messages \n", + "class AgentState(MessagesState):\n", + " # Final structured response from the agent\n", + " final_response: WeatherResponse\n", "\n", "@tool\n", "def get_weather(city: Literal[\"nyc\", \"sf\"]):\n", @@ -99,47 +90,52 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "## Option 1: Bind output as tool\n", + "\n", + "Let's now examine how we would use the single LLM option.\n", + "\n", "### Define Graph\n", "\n", - "Now that we have defined our tools and models, we can define our graph." + "The graph definition is very similar to the one above, the only difference is we no longer call an LLM in the `response` node, and instead bind the `WeatherResponse` tool to our LLM that already contains the `get_weather` tool." ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from langgraph.graph import StateGraph, END\n", "from langgraph.prebuilt import ToolNode\n", - "from langchain_core.messages import HumanMessage\n", "\n", - "# Define the function that determines whether to continue or not\n", - "def should_continue(state: AgentState):\n", - " messages = state[\"messages\"]\n", - " last_message = messages[-1]\n", - " # If there is no function call, then we respond to the user\n", - " if not last_message.tool_calls:\n", - " return \"respond\"\n", - " # Otherwise if there is, we continue\n", - " else:\n", - " return \"continue\"\n", + "tools = [get_weather, WeatherResponse]\n", + "\n", + "# Force the model to use tools by passing tool_choice=\"any\" \n", + "model_with_response_tool = model.bind_tools(tools,tool_choice=\"any\")\n", "\n", "# Define the function that calls the model\n", "def call_model(state: AgentState):\n", - " response = model_with_tools.invoke(state['messages'])\n", + " response = model_with_response_tool.invoke(state['messages'])\n", " # We return a list, because this will get added to the existing list\n", " return {\"messages\": [response]}\n", "\n", "# Define the function that responds to the user\n", "def respond(state: AgentState):\n", - " # We call the model with structured output in order to return the same format to the user every time\n", - " # state['messages'][-2] is the last ToolMessage in the convo, which we convert to a HumanMessage for the model to use\n", - " # We could also pass the entire chat history, but this saves tokens since all we care to structure is the output of the tool\n", - " response = model_with_structured_output.invoke([HumanMessage(content=state['messages'][-2].content)])\n", + " # Construct the final answer from the arguments of the last tool call\n", + " response = WeatherResponse(**state['messages'][-1].tool_calls[0]['args'])\n", " # We return the final answer\n", " return {\"final_response\": response}\n", "\n", + "# Define the function that determines whether to continue or not\n", + "def should_continue(state: AgentState):\n", + " messages = state[\"messages\"]\n", + " last_message = messages[-1]\n", + " # If there is only one tool call and it is the response tool call we respond to the user\n", + " if len(last_message.tool_calls) == 1 and last_message.tool_calls[0]['name'] == \"WeatherResponse\":\n", + " return \"respond\"\n", + " # Otherwise we will use the tool node again\n", + " else:\n", + " return \"continue\"\n", "\n", "# Define a new graph\n", "workflow = StateGraph(AgentState)\n", @@ -172,14 +168,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Usage\n", + "### Usage\n", "\n", - "We can now invoke our graph to verify that the output is being structured as desired:" + "Now we can run our graph to check that it worked as intended:" ] }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ @@ -188,7 +184,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 12, "metadata": {}, "outputs": [ { @@ -197,7 +193,129 @@ "WeatherResponse(temperature=75.0, wind_directon='SE', wind_speed=3.0)" ] }, - "execution_count": 6, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "answer" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Again, the agent returned a `WeatherResponse` object as we expected." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Option 2: 2 LLMs\n", + "\n", + "Let's now dive into how we would use a second LLM to force structured output.\n", + "\n", + "### Define Graph\n", + "\n", + "We can now define our graph:" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "from langgraph.graph import StateGraph, END\n", + "from langgraph.prebuilt import ToolNode\n", + "from langchain_core.messages import HumanMessage\n", + "\n", + "# Define the function that calls the model\n", + "def call_model(state: AgentState):\n", + " response = model_with_tools.invoke(state['messages'])\n", + " # We return a list, because this will get added to the existing list\n", + " return {\"messages\": [response]}\n", + "\n", + "# Define the function that responds to the user\n", + "def respond(state: AgentState):\n", + " # We call the model with structured output in order to return the same format to the user every time\n", + " # state['messages'][-2] is the last ToolMessage in the convo, which we convert to a HumanMessage for the model to use\n", + " # We could also pass the entire chat history, but this saves tokens since all we care to structure is the output of the tool\n", + " response = model_with_structured_output.invoke([HumanMessage(content=state['messages'][-2].content)])\n", + " # We return the final answer\n", + " return {\"final_response\": response}\n", + "\n", + "# Define the function that determines whether to continue or not\n", + "def should_continue(state: AgentState):\n", + " messages = state[\"messages\"]\n", + " last_message = messages[-1]\n", + " # If there is no function call, then we respond to the user\n", + " if not last_message.tool_calls:\n", + " return \"respond\"\n", + " # Otherwise if there is, we continue\n", + " else:\n", + " return \"continue\"\n", + "\n", + "# Define a new graph\n", + "workflow = StateGraph(AgentState)\n", + "\n", + "# Define the two nodes we will cycle between\n", + "workflow.add_node(\"agent\", call_model)\n", + "workflow.add_node(\"respond\", respond)\n", + "workflow.add_node(\"tools\", ToolNode(tools))\n", + "\n", + "# Set the entrypoint as `agent`\n", + "# This means that this node is the first one called\n", + "workflow.set_entry_point(\"agent\")\n", + "\n", + "# We now add a conditional edge\n", + "workflow.add_conditional_edges(\n", + " \"agent\",\n", + " should_continue,\n", + " {\n", + " \"continue\": \"tools\",\n", + " \"respond\": \"respond\",\n", + " },\n", + ")\n", + "\n", + "workflow.add_edge(\"tools\", \"agent\")\n", + "workflow.add_edge(\"respond\", END)\n", + "graph = workflow.compile()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Usage\n", + "\n", + "We can now invoke our graph to verify that the output is being structured as desired:" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "answer = graph.invoke(input={\"messages\": [(\"human\", \"what's the weather in SF?\")]})['final_response']" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "WeatherResponse(temperature=75.0, wind_directon='SE', wind_speed=4.83)" + ] + }, + "execution_count": 15, "metadata": {}, "output_type": "execute_result" } diff --git a/examples/react_diagrams.png b/examples/react_diagrams.png new file mode 100644 index 000000000..9d755884c Binary files /dev/null and b/examples/react_diagrams.png differ