How to stream messages from your graph

LangGraph Cloud supports multiple streaming modes. The main ones are:

values: This streaming mode streams back values of the graph. This is the full state of the graph after each node is called.
updates: This streaming mode streams back updates to the graph. This is the update to the state of the graph after each node is called.
messages: This streaming mode streams back messages - both complete messages (at the end of a node) as well as tokens for any messages generated inside a node. This mode is primarily meant for powering chat applications.

This guide covers stream_mode="messages".

In order to use this mode, the state of the graph you are interacting with MUST have a messages key that is a list of messages. Eg, the state should look something like:

from typing import TypedDict, Annotated
from langgraph.graph import add_messages
from langchain_core.messages import AnyMessage

class State(TypedDict):
    messages: Annotated[list[AnyMessage], add_messages]

OR it should be an instance or subclass of from langgraph.graph import MessageState (MessageState is just a helper type hint equivalent to the above).

With stream_mode="messages" two things will be streamed back:

It outputs messages produced by any chat model called inside (unless tagged in a special way)
It outputs messages returned from nodes (to allow for nodes to return ToolMessages and the like

In [1]:

from langgraph_sdk import get_client

In [2]:

client = get_client()

In [3]:

assistant_id = "agent"

In [4]:

thread = await client.threads.create()
thread

Out [4]:

{'thread_id': 'e1431c95-e241-4d1d-a252-27eceb1e5c86',
 'created_at': '2024-06-21T15:48:59.808924+00:00',
 'updated_at': '2024-06-21T15:48:59.808924+00:00',
 'metadata': {}}

In [5]:

runs = await client.runs.list(thread["thread_id"])
runs

Out [5]:

[]

In [6]:

# Helper function for formatting messages


def format_tool_calls(tool_calls):
    if tool_calls:
        formatted_calls = []
        for call in tool_calls:
            formatted_calls.append(
                f"Tool Call ID: {call['id']}, Function: {call['name']}, Arguments: {call['args']}"
            )
        return "\n".join(formatted_calls)
    return "No tool calls"

In [7]:

input = {"messages": [{"role": "user", "content": "what's the weather in sf"}]}
config = {"configurable": {"model_name": "openai"}}

async for event in client.runs.stream(
    thread["thread_id"],
    assistant_id,
    input=input,
    config=config,
    stream_mode="messages",
):
    if event.event == "metadata":
        print(f"Metadata: Run ID - {event.data['run_id']}")
        print("-" * 50)
    elif event.event == "messages/partial":
        for data_item in event.data:
            if "role" in data_item and data_item["role"] == "user":
                print(f"Human: {data_item['content']}")
            else:
                tool_calls = data_item.get("tool_calls", [])
                invalid_tool_calls = data_item.get("invalid_tool_calls", [])
                content = data_item.get("content", "")
                response_metadata = data_item.get("response_metadata", {})

                if content:
                    print(f"AI: {content}")

                if tool_calls:
                    print("Tool Calls:")
                    print(format_tool_calls(tool_calls))

                if invalid_tool_calls:
                    print("Invalid Tool Calls:")
                    print(format_tool_calls(invalid_tool_calls))

                if response_metadata:
                    finish_reason = response_metadata.get("finish_reason", "N/A")
                    print(f"Response Metadata: Finish Reason - {finish_reason}")
        print("-" * 50)

Metadata: Run ID - 1ef2fe5c-6a1d-6575-bc09-d7832711c17e
--------------------------------------------------
Invalid Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: 
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': ''}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current'}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather'}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in'}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in San'}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in San Francisco'}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in San Francisco'}
--------------------------------------------------
Tool Calls:
Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in San Francisco'}
Response Metadata: Finish Reason - tool_calls
--------------------------------------------------
--------------------------------------------------
AI: The
--------------------------------------------------
AI: The current
--------------------------------------------------
AI: The current weather
--------------------------------------------------
AI: The current weather in
--------------------------------------------------
AI: The current weather in San
--------------------------------------------------
AI: The current weather in San Francisco
--------------------------------------------------
AI: The current weather in San Francisco is
--------------------------------------------------
AI: The current weather in San Francisco is over
--------------------------------------------------
AI: The current weather in San Francisco is overcast
--------------------------------------------------
AI: The current weather in San Francisco is overcast with
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F).
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-s
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-south
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 k
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph).
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%,
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles).
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is 
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is 3
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is 3.
--------------------------------------------------
AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is 3.
Response Metadata: Finish Reason - stop
--------------------------------------------------

24 KiB Raw Blame History

How to stream messages from your graph

24 KiB

Raw Blame History