mirror of
https://github.com/langchain-ai/langgraph.git
synced 2026-08-20 22:52:29 +02:00
William FHandGitHub
feca5e1970
Plus: 1. Improve docstrings of add_node 2. Update crosslinking of sqlite and aiosqlite docstrings 3. Fix a bunch of links so we can turn on strict validation
466 lines
27 KiB
Plaintext
466 lines
27 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "51466c8d-8ce4-4b3d-be4e-18fdbeda5f53",
|
||
"metadata": {},
|
||
"source": [
|
||
"# How to stream LLM tokens from your graph\n",
|
||
"\n",
|
||
"In this example we will stream tokens from the language model powering an agent. We will use a ReAct agent as an example. The main thing to bear in mind here is that using [async nodes](./async.ipynb) typically offers the best behavior for this, since we will be using the `astream_events` method.\n",
|
||
"\n",
|
||
"This how-to guide closely follows the others in this directory, so we will call out differences with the **STREAMING** tag below (if you just want to search for those).\n",
|
||
"\n",
|
||
"<div class=\"admonition tip\">\n",
|
||
" <p class=\"admonition-title\">Note</p>\n",
|
||
" <p>\n",
|
||
" In this how-to, we will create our agent from scratch to be transparent (but verbose). You can accomplish similar functionality using the <code>create_react_agent(model, tools=tool)</code> (<a href=\"https://langchain-ai.github.io/langgraph/reference/prebuilt/#create_react_agent\">API doc</a>) constructor. This may be more appropriate if you are used to LangChain’s <a href=\"https://python.langchain.com/v0.2/docs/how_to/agent_executor/#concepts\">AgentExecutor</a> class.\n",
|
||
" </p>\n",
|
||
"</div> \n",
|
||
"\n",
|
||
"<div class=\"admonition warning\">\n",
|
||
" <p class=\"admonition-title\">Note on Python < 3.11</p>\n",
|
||
" <p>\n",
|
||
" When using python 3.8, 3.9, or 3.10, please ensure you manually pass the RunnableConfig through to the llm when invoking it like so: <code>llm.ainvoke(..., config)</code>.\n",
|
||
" The <a href=\"https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.base.Runnable.html#langchain_core.runnables.base.Runnable.astream_events\">astream_events</a> method collects all events from your nested code using a streaming tracer passed as a callback. In 3.11 and above, this is automatically handled via <a href=\"https://docs.python.org/3/library/contextvars.html\">contextvar</a>'s; prior to 3.11, <a href=\"https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task\">asyncio's tasks</a> lacked proper contextvar support, meaning that the callbacks will only propagate if you manually pass the config through. We do this in the <code>call_model</code> method below.\n",
|
||
" </p>\n",
|
||
"</div> "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7cbd446a-808f-4394-be92-d45ab818953c",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Setup\n",
|
||
"\n",
|
||
"First we need to install the packages required"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 26,
|
||
"id": "af4ce0ba-7596-4e5f-8bf8-0b0bd6e62833",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%%capture --no-stderr\n",
|
||
"%pip install --quiet -U langgraph langchain_openai langsmith"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d67b5425",
|
||
"metadata": {},
|
||
"source": [
|
||
"Next, we need to set API keys for OpenAI (the LLM we will use)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "a372be6f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import getpass\n",
|
||
"import os\n",
|
||
"\n",
|
||
"\n",
|
||
"def _set_env(var: str):\n",
|
||
" if not os.environ.get(var):\n",
|
||
" os.environ[var] = getpass.getpass(f\"{var}: \")\n",
|
||
"\n",
|
||
"\n",
|
||
"_set_env(\"OPENAI_API_KEY\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cc088bbd",
|
||
"metadata": {},
|
||
"source": [
|
||
"Optionally, we can set API key for [LangSmith tracing](https://smith.langchain.com/), which will give us best-in-class observability."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "907bf5e8",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
|
||
"_set_env(\"LANGCHAIN_API_KEY\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cd420984",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Set up the state\n",
|
||
"\n",
|
||
"The main type of graph in `langgraph` is the [StateGraph](https://langchain-ai.github.io/langgraph/reference/graphs/#langgraph.graph.StateGraph).\n",
|
||
"This graph is parameterized by a `State` object that it passes around to each node.\n",
|
||
"Each node then returns operations the graph uses to `update` that state.\n",
|
||
"These operations can either SET specific attributes on the state (e.g. overwrite the existing values) or ADD to the existing attribute.\n",
|
||
"Whether to set or add is denoted by annotating the `State` object you use to construct the graph.\n",
|
||
"\n",
|
||
"For this example, the state we will track will just be a list of messages.\n",
|
||
"We want each node to just add messages to that list.\n",
|
||
"Therefore, we will use a `TypedDict` with one key (`messages`) and annotate it so that the `messages` attribute is \"append-only\"."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "17ef4967",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from typing import Annotated\n",
|
||
"\n",
|
||
"from typing_extensions import TypedDict\n",
|
||
"\n",
|
||
"from langgraph.graph.message import add_messages\n",
|
||
"\n",
|
||
"# Add messages essentially does this with more\n",
|
||
"# robust handling\n",
|
||
"# def add_messages(left: list, right: list):\n",
|
||
"# return left + right\n",
|
||
"\n",
|
||
"\n",
|
||
"class State(TypedDict):\n",
|
||
" messages: Annotated[list, add_messages]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "81ed4e9c",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Set up the tools\n",
|
||
"\n",
|
||
"We will first define the tools we want to use.\n",
|
||
"For this simple example, we will use create a placeholder search engine.\n",
|
||
"It is really easy to create your own tools - see documentation [here](https://python.langchain.com/v0.2/docs/how_to/custom_tools) on how to do that.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "9a8bc61e",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_core.tools import tool\n",
|
||
"\n",
|
||
"\n",
|
||
"@tool\n",
|
||
"def search(query: str):\n",
|
||
" \"\"\"Call to surf the web.\"\"\"\n",
|
||
" # This is a placeholder, but don't tell the LLM that...\n",
|
||
" return [\"Cloudy with a chance of hail.\"]\n",
|
||
"\n",
|
||
"\n",
|
||
"tools = [search]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b0aa12b9",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can now wrap these tools in a simple [ToolNode](https://langchain-ai.github.io/langgraph/reference/prebuilt/#toolnode).\n",
|
||
"This is a simple class that takes in a list of messages containing an [AIMessages with tool_calls](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessage.html#langchain_core.messages.ai.AIMessage.tool_calls), runs the tools, and returns the output as [ToolMessage](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.ToolMessage.html#langchain_core.messages.tool.ToolMessage)s.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "4d6ac180",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langgraph.prebuilt import ToolNode\n",
|
||
"\n",
|
||
"tool_node = ToolNode(tools)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4f13e0a5",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Set up the model\n",
|
||
"\n",
|
||
"Now we need to load the chat model we want to use.\n",
|
||
"This should satisfy two criteria:\n",
|
||
"\n",
|
||
"1. It should work with messages, since our state is primarily a list of messages (chat history).\n",
|
||
"2. It should work with tool calling, since we are using a prebuilt [ToolNode](https://langchain-ai.github.io/langgraph/reference/prebuilt/#toolnode)\n",
|
||
"\n",
|
||
"**Note:** these model requirements are not requirements for using LangGraph - they are just requirements for this particular example.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "42c0af37",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from langchain_openai import ChatOpenAI\n",
|
||
"\n",
|
||
"model = ChatOpenAI(model=\"gpt-3.5-turbo\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "8a592001",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"After we've done this, we should make sure the model knows that it has these tools available to call.\n",
|
||
"We can do this by converting the LangChain tools into the format for function calling, and then bind them to the model class.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "2bbdd3bc",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"model = model.bind_tools(tools)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e03c5094-9297-4d19-a04e-3eedc75cefb4",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Define the nodes\n",
|
||
"\n",
|
||
"We now need to define a few different nodes in our graph.\n",
|
||
"In `langgraph`, a node can be either a function or a [runnable](https://python.langchain.com/v0.2/docs/concepts/#langchain-expression-language-lcel).\n",
|
||
"There are two main nodes we need for this:\n",
|
||
"\n",
|
||
"1. The agent: responsible for deciding what (if any) actions to take.\n",
|
||
"2. A function to invoke tools: if the agent decides to take an action, this node will then execute that action.\n",
|
||
"\n",
|
||
"We will also need to define some edges.\n",
|
||
"Some of these edges may be conditional.\n",
|
||
"The reason they are conditional is that based on the output of a node, one of several paths may be taken.\n",
|
||
"The path that is taken is not known until that node is run (the LLM decides).\n",
|
||
"\n",
|
||
"1. Conditional Edge: after the agent is called, we should either:\n",
|
||
" a. If the agent said to take an action, then the function to invoke tools should be called\n",
|
||
" b. If the agent said that it was finished, then it should finish\n",
|
||
"2. Normal Edge: after the tools are invoked, it should always go back to the agent to decide what to do next\n",
|
||
"\n",
|
||
"Let's define the nodes, as well as a function to decide how what conditional edge to take.\n",
|
||
"\n",
|
||
"**STREAMING**\n",
|
||
"\n",
|
||
"We define each node as an async function.\n",
|
||
"\n",
|
||
"<div class=\"admonition note\">\n",
|
||
" <p class=\"admonition-title\">Manual Callback Propagation</p>\n",
|
||
" <p>\n",
|
||
" Note that in <code>call_model(state: State, config: RunnableConfig):</code> below, we a) accept the <a href=\"https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.config.RunnableConfig.html#langchain_core.runnables.config.RunnableConfig\">RunnableConfig</a> in the node and b) pass this in as the second arg for <code>llm.ainvoke(..., config)</code>. This is optional for python 3.11 and later. If you ever have a problem where the LLM tokens are not streamed when using `astream_events` and you are using an older version of python, it's worth checking to ensure that the callbacks are manually propagated.</p>\n",
|
||
"</div> "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "3b541bb9-900c-40d0-964d-7b5dfee30667",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from typing import Literal\n",
|
||
"\n",
|
||
"from langchain_core.runnables import RunnableConfig\n",
|
||
"\n",
|
||
"from langgraph.graph import END, START, StateGraph\n",
|
||
"\n",
|
||
"\n",
|
||
"# Define the function that determines whether to continue or not\n",
|
||
"def should_continue(state: State) -> Literal[\"__end__\", \"tools\"]:\n",
|
||
" messages = state[\"messages\"]\n",
|
||
" last_message = messages[-1]\n",
|
||
" # If there is no function call, then we finish\n",
|
||
" if not last_message.tool_calls:\n",
|
||
" return END\n",
|
||
" # Otherwise if there is, we continue\n",
|
||
" else:\n",
|
||
" return \"tools\"\n",
|
||
"\n",
|
||
"\n",
|
||
"# Define the function that calls the model\n",
|
||
"async def call_model(state: State, config: RunnableConfig):\n",
|
||
" messages = state[\"messages\"]\n",
|
||
" # Note: Passing the config through explicitly is required for python < 3.11\n",
|
||
" # Since context var support wasn't added before then: https://docs.python.org/3/library/asyncio-task.html#creating-tasks\n",
|
||
" response = await model.ainvoke(messages, config)\n",
|
||
" # We return a list, because this will get added to the existing list\n",
|
||
" return {\"messages\": response}"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ffd6e892-946c-4899-8cc0-7c9291c1f73b",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Define the graph\n",
|
||
"\n",
|
||
"We can now put it all together and define the graph!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "813ae66c-3b58-4283-a02a-36da72a2ab90",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Define a new graph\n",
|
||
"workflow = StateGraph(State)\n",
|
||
"\n",
|
||
"# Define the two nodes we will cycle between\n",
|
||
"workflow.add_node(\"agent\", call_model)\n",
|
||
"workflow.add_node(\"tools\", tool_node)\n",
|
||
"\n",
|
||
"# Set the entrypoint as `agent`\n",
|
||
"# This means that this node is the first one called\n",
|
||
"workflow.add_edge(START, \"agent\")\n",
|
||
"\n",
|
||
"# We now add a conditional edge\n",
|
||
"workflow.add_conditional_edges(\n",
|
||
" # First, we define the start node. We use `agent`.\n",
|
||
" # This means these are the edges taken after the `agent` node is called.\n",
|
||
" \"agent\",\n",
|
||
" # Next, we pass in the function that will determine which node is called next.\n",
|
||
" should_continue,\n",
|
||
")\n",
|
||
"\n",
|
||
"workflow.add_edge(\"tools\", \"agent\")\n",
|
||
"\n",
|
||
"# Finally, we compile it!\n",
|
||
"# This compiles it into a LangChain Runnable,\n",
|
||
"# meaning you can use it as you would any other runnable\n",
|
||
"app = workflow.compile()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "72785b66",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/4gHYSUNDX1BST0ZJTEUAAQEAAAHIAAAAAAQwAABtbnRyUkdCIFhZWiAH4AABAAEAAAAAAABhY3NwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAA9tYAAQAAAADTLQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlkZXNjAAAA8AAAACRyWFlaAAABFAAAABRnWFlaAAABKAAAABRiWFlaAAABPAAAABR3dHB0AAABUAAAABRyVFJDAAABZAAAAChnVFJDAAABZAAAAChiVFJDAAABZAAAAChjcHJ0AAABjAAAADxtbHVjAAAAAAAAAAEAAAAMZW5VUwAAAAgAAAAcAHMAUgBHAEJYWVogAAAAAAAAb6IAADj1AAADkFhZWiAAAAAAAABimQAAt4UAABjaWFlaIAAAAAAAACSgAAAPhAAAts9YWVogAAAAAAAA9tYAAQAAAADTLXBhcmEAAAAAAAQAAAACZmYAAPKnAAANWQAAE9AAAApbAAAAAAAAAABtbHVjAAAAAAAAAAEAAAAMZW5VUwAAACAAAAAcAEcAbwBvAGcAbABlACAASQBuAGMALgAgADIAMAAxADb/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCADaAMcDASIAAhEBAxEB/8QAHQABAAIDAAMBAAAAAAAAAAAAAAYHBAUIAQMJAv/EAFAQAAEEAQICBAkGCAsGBwAAAAEAAgMEBQYRBxITITFVCBQWIkFRYZTRFRcyNpPhI1Jxc3SBsrMJGCQzQlZidpWh0iU1U3KRsSZDRVSCkqL/xAAbAQEAAgMBAQAAAAAAAAAAAAAAAgMBBAUGB//EADoRAAIBAgIGBQkHBQAAAAAAAAABAgMRBBMSITFRUpEVQWGhsQUUMmJxgcHR8DM0Y3Ky4fEiQlOCwv/aAAwDAQACEQMRAD8A+qaIiAIiIAiIgCIiALV+VOFH/q9D3lnxW0VFaEweNm0VgZJMfVfI6jCXOdC0knkHWTsqq1enhqeZNN60tXbf5G5h8Pntq9rFw+VWF74oe8s+KeVWF74oe8s+KrvyexfdtP7BnwTyexfdtP7BnwXO6Vw/BLmjd6O9buLE8qsL3xQ95Z8U8qsL3xQ95Z8VXfk9i+7af2DPgnk9i+7af2DPgnSuH4Jc0OjvW7ixPKrC98UPeWfFPKrC98UPeWfFV35PYvu2n9gz4J5PYvu2n9gz4J0rh+CXNDo71u4sTyqwvfFD3lnxTyqwvfFD3lnxVd+T2L7tp/YM+CeT2L7tp/YM+CdK4fglzQ6O9buLE8qsL3xQ95Z8Vl0slUyTHPqWobTWnZzoJA8A+3YqsPJ7F920/sGfBbXhdVhp5vVUcEMcEYlrnkjaGj+a9QW5hsXSxblGEWmlfXbel8TXr4PJhp6Vyw0RFtHOCIiAIiIAiIgCIiAIiIAiIgCpjh/9RtP/AKBB+wFc6pjh/wDUbT/6BB+wFy/Kf3X/AGXhI6/k70pG/REXkzuENj4v6Sm1nJpSLKmfORSGF8ENWZ8bZBH0hjMoYYw8MBdyc3Nt6FHOGHhB4PiJhs/kJYLeJjw89wzOsUrLIxWgkc0SmR8TW8xa3mMY3c3cgjcFRR3yrp/jkxui8PqelBlMuXakrX6B+Rp4uhIddhnPU2XdsY2a7z9utvVucLT1/WOkNB8S9MYTT+Wr6whv5fJ4y7JQLqdhk1gyROimP4N8hbJuGE78zSCFvZULatrt18zUzJX19vVyLSwHHHROp8fmrmPzRfFhqpu3o56c8E0MAa5xk6KRjXubs12xaCDtsFF9ZeE/pbB6NOoML43nq/jlKq18WPtthcLEnLztk6EtfytDzs3fzmhnU5zQao8nMjc1Hqm9jcPry9TyHD/JYtt7U0Fh8893drxGI37uj3BPKA1rHO5gwFWTr/SuVm8GXT1DHYizZyGLgwtp+Lgi2nLa0teSWNrDsecNjd5vbuNu1ZyqUZK/W11mMypKL7C4sFm6uo8RWyVLp/FbDeePxmtJXk23286ORrXt7OxwCz1qtMahj1Tha+TipZDHxzc21fKVH1bDdnEedG8Bzd9txv6CFtVotWdjbWtBZPDb6waq/O1v3Sxlk8NvrBqr87W/dLu+R/tKn5f+omhjvsfeT9EReiPOBERAEREAREQBERAEREAREQBUxw/+o2n/ANAg/YCudQSpwfxdCrDWr5TMw14WBkcbbnU1oGwA6lr4nDrFUcvSs7p9z+Zv4SvGg25dZWbuAHDNxJOgNNknrJOLh/0rzJwC4aSvc9+gtOPe4kuc7GQkk+s+arQ+aqj3xm/ffuT5qqPfGb99+5c3oyp/m8Td88ocPcjS0KFbFUa1KnBHVp1o2wwwQtDWRsaAGtaB1AAAAD2LIWy+aqj3xm/ffuT5qqPfGb99+5V9Efirkyfn9LczWoq08Gyrd4n4HWVvOZvKSTYvVeRxFYwWOjArwuaIwerrOxO59Kt35qqPfGb99+5Oh/xVyZnpCluZA9S8KdGayyPyhntK4fM3uQR+M3qUc0nKN9m8zgTsNz1e1an+L9wy3+oGm/8AC4f9KtL5qqPfGb99+5Pmqo98Zv337lYvJc1qVbxIPG0Hrce5EV0vo7BaJoSUtP4ejhKckpmfBQrthY55ABcQ0AE7NaN/YFIeG31g1V+drfulk/NVR74zfvv3Lc6X0fT0objqs1qxJbe18sluXpHEtGw69vUt3CYPzWU5ynpNq2x70/ga+IxVOrT0Io3qIi3TlBERAEREAREQBERAEREAREQBERAEREAREQHO/gU/VTiT/f8AzP7xi6IXO/gU/VTiT/f/ADP7xi6IQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQHO/gU/VTiT/AH/zP7xi6IXO/gU/VTiT/f8AzP7xi6IQBERAEREAREQBERAEREAREQBERAEREAREQBERAEXgnYbnsUIv8TBPIY8DjzlmAgG7LL0NU+1j9nGQe1rS0+h3qnGEp7CcISqO0VcnC5N/hHuCcvE/gozUePY6TL6QdLeEY/p1HhosgDs3AYyTf1RuA7VdB1pq1x3FTCs/smSZ2369h/2Xrsar1TbgkgnpYKaGVpY+OTpXNc0jYgg9oIVmUuJczY80rbj5UeBRwLPHTjhi6d2sZtOYjbJZUubux0bCOSI+g9I/laR28vOR2L7Srl3wfuEE3g5Y7P1NOQYyY5i863LNadIXsjG4igBAG7WAu2J6yXOPp2FseWerv/bYT/7TJlLiXMeaVtxZSKu4OIGoasgNzB07sG/WaFwtlA9jJGhp/W8KX6e1NQ1NWfLSkdzRnlmgmYY5YXep7HdY9h7COsEjrUZU5RWltXY7/wAe8qnRnT9JG1REVRSEREAREQBERAEREAREQBERAEREAREQFe67yrs1lnafjd/III2y5DY/zxd9CA/2SAXPHpBa07tc4HDAAAAGwHoWDTe6bO6mlk/nXZSRrvXs1jGt/wDy1qwNd52vpjRmbytrKR4SGpTllORlh6ZtYhp2f0f9PY7eb/S7PSrK+qSgti8ev63WPR4eCp0k/eb1Fy9h+NOvNLz6xrX48vnTV0nNqLGHP4qvRsOfG/kI6Ou7rjPO07PDXjlI9q9MfGjUejr+Xyb9aR8RMRR0XLnnsq1q8UMNx0kbYo3uibuGOHOWgnmAD9+bqI17Es+J1Oi590Rqbiw3PYuXJ1M1ewtyvM/IzZWjja0NM9C58b65r2Hvc3nDW8sgcdnb824WFoLX+uYcJwf1NmtUDNVtYTRUL2MOPggjidJVllZLG5jQ8PBh87clp5jytYNgMElVW5/X8nRrZGvLg1wcWnlcAd9j6j/1WLbNrHzsyuNBOSrNJbGHcrbDO0xP9YPoJ+idiPTvS/gx4HKUJdd2bepr2UrN1RlaxpT167I3SiwN7BcyNrud2x3aDyDc7NHUr0U4TcJaSMq1WH9S2lh4rJ181jKmQqP6StaibNE71tcAR/kVlqHcJnudoqJp+hFdvRR/8jbczWj9QAH6lMVfVioVJRXU2eZktGTQREVREIiIAiIgCIiAIiIAiIgCIiAIiICs9S0HYHWE8rgRSzPLJG8nzW2WMDXM9hcxjXD18sh9HXpdYaTx2utL5PT+XidNjchA6CZrHFrtj6WkdhB2IPrAUq1rxA0nX1fhOHuY8Zs5nUMb5a1SvVleGRx7uMzpWDaLlc1uz9wWuLSNttxgX9MahwT+WCEahpAgNkjeyK00f22uLWPPtaW7/i+u6Uc6zT1+J18NiYaGXUKjn4FRYyTJZyDUGps9qV+FtYlk13KMifNFIAWxh7Yg2Ite0ObIxoIJJdzdihPB/hXqzH5Kzhsth7uN0Dcx09bJ4nOWsdY8ZkeGtZ0PiULC0BvOHF53II6gRuugDeyLep2ms013pArNdt+sOIT5Qv8A9XM17p96j5vV3eBt3o3TUu8h2iuDlfRL+SHVOpspRjqOpVsfk77Za9aI7dTWhgLi0NAaXlxA6gesr90eDOEoab0LhI7V81NH2YrVB7pGc8ro4ZImiU8mxHLK4nlDesD8h2ml+IVPWte9Pg8dlMnDRuS4+y+CruIrEZAkjPX2t3G63Xyhf/q5mvdPvTzeruJ6dFdaI3pjhdR0hq3M5vG5XKxQZaeS3Yw7p2Oo+MScvPM1pZzhzuXc+ftuT1KU5O6+jULoYjYtPPR164OxmlP0WD8p9PoG57AvMMefvyCOnpq4wk7dNfljrxN/L5zn/wDRhUPg4/8ACjh3xHyGnNYa2pQa1oFkb2WKssVWsJI2v5YpC0s35XtDnF/MTuNmjzRlUtB6VTlfW+Wz3lNTE06UbQd2XlpXBjTencfjek6Z9eINkl/4kh63v/W4k/rW1UV0pxW0XrtzG6c1bhM5I/fljx+Qimf1Akjla4kHYE7behSpQlJybk9rOC3fWERFEwEREAREQBERAEREAREQBEVcT8VWap1brPQelI7UOrMJjhKclkMdL8mQ2ZGAwxvf1cx2ex5a3taSQTsQAJnm9U4fTUmPZlspTxsmQssp02WpmxusTuOzY4wT5zj6h1qAWTqji5BxA0plsNl+H+FjkbRxeo8fkmC5dbuTJNEGgmJuwYBvvuHO7CCBstN8LW38RpO3xEGL1trXAiSSLOPxzIhHK9wJdEzrDCA1g3G2/IHbNPULBQGo0tpmppLT+KxFWSxZhxtVlOGxdlM07mNAA55D1uJ5RufTstuiIAiIgK64J5b5XxOpX+QPzfdDn7sHivQdD8o8rm/y7boo+bpe3m2dvt9JysVQvhdi9bYrH5tmucxSzNyXMWZsdJSYGthoOI6CJ20ce72jfc7O7fpFTRAF88/4UXgKJIsbxWxNY8zOTHZvkHo7IJnfuyfbEF9DFqdV6VxWuNN5LAZ2lHkcRkYHVrVWQkCRjhsRuCC0+kOBBBAIIIBQHyT8CfA6b0TxO0zr/ibWyWG0w+WSPAZW3jwcVYvt3aHSzPBDejO5Y4DYSM5udvREH6/QTx2YY5oZGyxSND2SMcHNc0jcEEdoK02V0Np3N6Rdpa9hKFjTZrtqDFOrt8WbC0AMY1gGzQ0Acu23LsNttgucptCcQvBHmfe0Ay5xC4VB5ks6OsSGTI4lhO5dSkPXIwf8M9f5SXPAHVSKF8KeMGlONWmGZ3SeUjyFXcMnhPmT1ZPTHLGetjh7eo9oJHWpogCIiAIiIAiIgCIiALW5nPU8N0EU1qrHetl0dKpYsMidalDd+jZzHrP5N9u1bJVTxftaJr6+4Ws1RTu2c1LmJG4CSqSI4bXRHmdLs4bt5fWD1+hAYkeh8zx70HgJeI+NyGh71XKDJHC4PNO8+NjiYY7MkYG/a1xDT1OY0gt62i4AAN9htv2ryiAIiIAiIgCIiAqLwdodAY2prfG6Ez9nNOj1Lds5iK75stS9I4dLGGmNhEYLSGnYg7HZztjtbqgvEvSGpMjpyz83uYoaS1JLehuy3J6DZorvJyh0c4A5iHNa1pcPO2aADss3T3FHTWo9a57R1PKxTanwLYnZCiY3Rua17WuD2h30m+cAS0nYkAnrG4EtREQBERAUPxV8GU5TU79fcM8v5A8R2AmS3AzellR2mO5CBs8E/wBMAuHaQ4hu2fwS4/X9balv6C1tpuxpHiTiqnjlqi0GSnbr8wZ4zWlG4MZc4DYncE7bu2dtdK504Df+NvCS4461d+ErUbdXSdF/4ni0fNZbv7ZXNKA6LREQBERAEX5e9sbS5xDWtG5JOwAWt8qsL3xQ95Z8VJRlLYgbRFq/KrC98UPeWfFPKrC98UPeWfFSy58LM2ZmZGWzDj7MlKBlq4yJzoYJZeiZI8A8rS/Z3KCdhvsdt99j2L536m/hUcrRz0NSXhPVpT460+O7XvZcyzNc3drmMcIG9E8OB3JDvVsvoJ5VYXvih7yz4r5s+HB4NY1H4Q+nsro+WpJT1tYbBckge10VO23YSTSbHZrHM/CE+kslKZc+FizO3vBg472/CL4aP1fZ0u/SsL70tWtA+540LEbGs3ma/o2dXOXs227Yz1+q3FDNA0tJcN9F4XS+GydCHGYqqyrCDaj5nBo63O6+tzju4n0kkrf+VWF74oe8s+KZc+FizNoi1flVhe+KHvLPinlVhe+KHvLPimXPhYszaIsalkqmSY51S1Baa07OdBIHgH27FZKg007MwERFgBRbXmlchnNOZ1umcjDpnVV6mK1fPNqMmki5SSwODh5zQXP2B7OdxHWpSiApDWfhO6V4CYuDEcS87zasqYOPI2PEsfIyPJSb9G5tXfzXPMg+jzAAHmcWtDi3Y+DD4ReN8Jfh0/UlPHOwt2tbkp3cY+x05geNnNIfyt5muY5p35R18w6+Xc8o+HpwK478aNa1L9PTtDLaQw4mjxVTD2mPnja8t55ZhIGPdJJyM3awFrA0NG55nvg38Hnm9UcFvCFm0JqbA5bCs1VVfH4pepyQubPAx8schDgDy8gmG4G3ng77BAfUVFh3MvQx7uW1dr1neqaVrD/mVj+VWF74oe8s+KmoSetIzY86nz9bSmmstm7h5aeNqTXJjvtsyNhe7/JpVM+BDgLOL8HjB5XID/a2pbFnUFx+3032ZXOa79cfRrE8M/W0TfB51HicFer3MxnnQYWvFBM15/DytZJzbE7N6PpNyrd0rY07pXTWIwVHK0fFMbUhpQNFhn0I2Bjerf1NCzlz4WLMk6L8RSsmjD43tkY7sc07g/rX7VZgIiIDV6q+rGY/Q5v2CqswGAxj8FjnOx1RzjWjJJgbufNHsVp6q+rGY/Q5v2Cq709/uDG/o0X7AWtjJyjQjou2v4HnfLUnGFOz638B5PYvu2n9gz4J5PYvu2n9gz4LYIuLm1OJ8zymnLea/wAnsX3bT+wZ8E8nsX3bT+wZ8FFM/wActD6Yz8uGyeejrXYHsjsO6CV8FZztuVs0zWGOIncHZ7gdiD6V69RceNDaUy+SxmTzZgu4x0bb0bKdiUVQ+NsjHyuZGWsYWvaedxDe0b7ggT062995ao1nsT7yX+T2L7tp/YM+CeT2L7tp/YM+Cj2seL2kdA2KNfNZhkFi9GZq8FeCWzI+IdsnLE1xDB+OQG+1Y/BHXV3iXwq07qfIxVobuSgMsjKjXNiBD3NHKHOcdtgO0lY06ttLSdvaLVVDTd7fXyJT5PYvu2n9gz4J5PYvu2n9gz4LYIo5tTifMq05bz28LqsNPN6pjrwxwRiWueSNoaP5r1BWGoDw2+sGqvzlb90p8vTSbai3wx/Sj6HhNeHp+xeAREUDaCIiA/E00deF8sr2xRRtLnvedmtA6yST2BVnldQXtYOLop7GMwm/4KKImOe038eR30mNPaGN2dtsXEEljdzxStGShjMQCOjydvo5wd/OhYx0j29X4xa1pHpDj+RadXXyoqS2vuXzvyOrg6EZLMkauDSuGrD8HiqYPpcYGlx9PWSNz+te3yfxfdtP7BvwUEwHHDG5vi3qLQrqd2GxjDBHDZFKy6Od7o3vk5n9FyRNbyANLnbP380nsWwwnHDQ+otSswOOz0djIyySQw7QSthsSR787IpiwRyObsdwxxPUfUqXVqPW5PmdRSh1Mlfk/i+7af2Dfgh09iiCDjaZB9HQM+CiruOGh2ar8nDno/lTxoUCBBKYBZP/AJBn5eiEno5Ofm36tt1qtP8AGWtDjNd5TVVipisZp7UU2IimijeS+MMhLN2guL5HOlI2aOvq2CxmT4mNOG8ndfAQYyfxjEOfhLW4PSUdmNdt1bPj25Hj/mB9m2wKnmkNWPzRloX42QZeuwPkbECIp2E7CWPck7b9RaSSw9RJBa51faU1ditbYdmUw88lim57o+aWvJA8OadnAska1zSD6CAvflbRw9rGZiM8slK1Hzn1wyPEcrfb5rubY9W7W9m24vpzlWapzd77Pb1e41cRQjUg5R2lvoiKk4Bq9VfVjMfoc37BVd6e/wBwY39Gi/YCsTVX1YzH6HN+wVXODiZPpzHxyND431I2ua4bggsG4K1Mb9hH2/A835b9Cn7X8DZIoAPB/wCGYII0BpsEekYuH/SvH8X7hl/UDTf+Fw/6VxLR3/XM8xanvfL9ykKPD+DFZrWWmtY6c4gZb5ZzluzDNgb135LvVLMnMDKI5mxRlocWva8Dqb1c26mM+jshWPhC1ocRddWyGOgr40GB7vHA3Eti5YiR+EIcOXq3PN1dqvqCCOtBHDCxsUUbQxjGDYNaBsAB6l7FY6rZsPFSbv8AW1P4HN+iX5fhXreDMZjSufzFXN6Vw9OvZxlB9mSjNXjeJq0rB50XM57Xbu2buDudx1WD4M2Jv4PgTpCjk6NnGX4arhLUuROiliPSPOzmu6wdiFZ6iupeFOjNZZL5Qz2lcPmb3II/Gb1KOaTlG+zeZwJ2G56vasOekrMjKsqitJW2d2pEqRV//F94Zf1A03/hcP8ApUm0tovAaIpy1NPYajhKssnSyQ0K7YWPfsBzENA3OwA39ig7dRQ1C2pvl+5JOG31g1V+crfulPlAeG31g1V+crfulPl6h+jD8sf0o+g4T7vT9i8AiIom2EREBBOJ9cstabv7Exw3HwPIG/KJInBpPs5g0f8AyC1isDOYatqHE2sdca51ewzlcWHZzT2hzT6HAgEH0EAqtJJLOEvNxmY5YrhPLBYA5Yro9Do+vqdt9KPtad+1vK510k6kFbbHw2373c7GCqq2W9pT/Q5HAcbNeV58TlzU1dRoRY/L0Kb568L44pYn9NI3qiLS5rvO23Cg2nsXnsto7hRw9Zo/MYnL6Vy9Czk8hZpmOhFHUJMkkVj6Mpm7AGbn8IebbYrqRFqG+6V+v62nKMun9Qs4Rz8ImaUzLtRyZxzhm/Ez8nmI5HxoXTZ+juI9vN35+YbcqkU2n462P4rYTVGltTXKlnUzM3TtYKq58jmSCDopqz2ncyRPiLnNG5AHYd9l0YiXMZK3ldcCcnqnKaNtP1Uy6ZIshPDj7OUqircs0m7dFLPEAOSQ+cD1N3DQSBupfqaub+PioMBMl61BVaAN/pSN5j+QN5nH2ArYXL1fHwGazMyCIEDnkdsNz2D8p9S3WjNOT3chFncjXfWbE0jH1JmlsjOYbOmkafoucOpre1rS7m63FrNmgnGSqvYvHd9dRXWqKjTs3rJ0iIqzzp6LtSO/TnqygmKeN0bwDsdiNj/3UNh4SY6vCyKPLZpkbGhrWi71ADqA7FOUU1OUVZEJQjP0kn7SE/NVR74zfvv3J81VHvjN++/cpsizmPs5Ihk0uBckQn5qqPfGb99+5Pmqo98Zv337lNkTMfZyQyaXAuSIT81VHvjN++/cnzVUe+M3779ymyJmPs5IZNLgXJEJ+aqj3xm/ffuT5qqPfGb99+5TZEzH2ckMmlwLkjRaX0fT0objqs1qxJbc18sluXpHEtGw69vUt6iKMpOTuy1JJWQREUTIREQBY2RxlTL1H1b1aK3Wf9KKZgc0+rqKyUWU2ndAhkvCjDb/AMms5SiwdkcGQlLB+QOLgP1L1/NRQ73zXvv3Kbors+pvLVVqL+5kI+aih3vmvffuXkcKMfv15bNOHq8dI/7BTZEz6m8znVOJkdw2gMHhLbbcNR1i6z6Nq5M+xI3q280vJ5er8Xb0+tSJEVcpym7ydyptyd2ERFAwf//Z",
|
||
"text/plain": [
|
||
"<IPython.core.display.Image object>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"from IPython.display import Image, display\n",
|
||
"\n",
|
||
"display(Image(app.get_graph().draw_mermaid_png()))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "2a1b56c5-bd61-4192-8bdb-458a1e9f0159",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Streaming LLM Tokens\n",
|
||
"\n",
|
||
"You can access the LLM tokens as they are produced by each node. \n",
|
||
"In this case only the \"agent\" node produces LLM tokens.\n",
|
||
"In order for this to work properly, you must be using an LLM that supports streaming as well as have set it when constructing the LLM (e.g. `ChatOpenAI(model=\"gpt-3.5-turbo-1106\", streaming=True)`)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "cfd140f0-a5a6-4697-8115-322242f197b5",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"/Users/wfh/code/lc/langgraph/.venv/lib/python3.12/site-packages/langchain_core/_api/beta_decorator.py:87: LangChainBetaWarning: This API is in beta and may change in the future.\n",
|
||
" warn_beta(\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"--\n",
|
||
"Starting tool: search with inputs: {'query': 'weather in San Francisco'}\n",
|
||
"Done tool: search\n",
|
||
"Tool output was: ['Cloudy with a chance of hail.']\n",
|
||
"--\n",
|
||
"The| weather| in| San| Francisco| is| currently| cloudy| with| a| chance| of| hail|.|"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from langchain_core.messages import HumanMessage\n",
|
||
"\n",
|
||
"inputs = [HumanMessage(content=\"what is the weather in sf\")]\n",
|
||
"async for event in app.astream_events({\"messages\": inputs}, version=\"v1\"):\n",
|
||
" kind = event[\"event\"]\n",
|
||
" if kind == \"on_chat_model_stream\":\n",
|
||
" content = event[\"data\"][\"chunk\"].content\n",
|
||
" if content:\n",
|
||
" # Empty content in the context of OpenAI or Anthropic usually means\n",
|
||
" # that the model is asking for a tool to be invoked.\n",
|
||
" # So we only print non-empty content\n",
|
||
" print(content, end=\"|\")\n",
|
||
" elif kind == \"on_tool_start\":\n",
|
||
" print(\"--\")\n",
|
||
" print(\n",
|
||
" f\"Starting tool: {event['name']} with inputs: {event['data'].get('input')}\"\n",
|
||
" )\n",
|
||
" elif kind == \"on_tool_end\":\n",
|
||
" print(f\"Done tool: {event['name']}\")\n",
|
||
" print(f\"Tool output was: {event['data'].get('output')}\")\n",
|
||
" print(\"--\")"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.12.2"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|