{ "cells": [ { "cell_type": "markdown", "id": "51466c8d-8ce4-4b3d-be4e-18fdbeda5f53", "metadata": {}, "source": [ "# How to stream messages from your graph\n", "\n", "LangGraph Cloud supports multiple streaming modes. The main ones are:\n", "\n", "- `values`: This streaming mode streams back values of the graph. This is the **full state of the graph** after each node is called.\n", "- `updates`: This streaming mode streams back updates to the graph. This is the **update to the state of the graph** after each node is called.\n", "- `messages`: This streaming mode streams back messages - both complete messages (at the end of a node) as well as **tokens** for any messages generated inside a node. This mode is primarily meant for powering chat applications.\n", "\n", "\n", "This guide covers `stream_mode=\"messages\"`.\n", "\n", "In order to use this mode, the state of the graph you are interacting with MUST have a messages key that is a list of messages.\n", "Eg, the state should look something like:\n", "\n", "```python\n", "from typing import TypedDict, Annotated\n", "from langgraph.graph import add_messages\n", "from langchain_core.messages import AnyMessage\n", "\n", "class State(TypedDict):\n", " messages: Annotated[list[AnyMessage], add_messages]\n", "```\n", "\n", "OR it should be an instance or subclass of `from langgraph.graph import MessageState` (`MessageState` is just a helper type hint equivalent to the above).\n", "\n", "With `stream_mode=\"messages\"` two things will be streamed back:\n", "\n", "- It outputs messages produced by any chat model called inside (unless tagged in a special way)\n", "- It outputs messages returned from nodes (to allow for nodes to return `ToolMessages` and the like" ] }, { "cell_type": "code", "execution_count": 1, "id": "521d975b-e94b-4c37-bfa1-82d969e2a4dc", "metadata": {}, "outputs": [], "source": [ "from langgraph_sdk import get_client" ] }, { "cell_type": "code", "execution_count": 2, "id": "27a1392b-86c3-464e-99a8-90ffc965f3ec", "metadata": {}, "outputs": [], "source": [ "client = get_client()" ] }, { "cell_type": "code", "execution_count": 3, "id": "714e9f92-86b4-4cd8-9d68-cfc45d56ed2c", "metadata": {}, "outputs": [], "source": [ "assistant_id = \"agent\"" ] }, { "cell_type": "code", "execution_count": 4, "id": "56aa5159-5583-4134-9210-709b969bda6f", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'thread_id': 'e1431c95-e241-4d1d-a252-27eceb1e5c86',\n", " 'created_at': '2024-06-21T15:48:59.808924+00:00',\n", " 'updated_at': '2024-06-21T15:48:59.808924+00:00',\n", " 'metadata': {}}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "thread = await client.threads.create()\n", "thread" ] }, { "cell_type": "code", "execution_count": 5, "id": "147c3f98-f889-4f05-a090-6b31f2a0b291", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "runs = await client.runs.list(thread[\"thread_id\"])\n", "runs" ] }, { "cell_type": "code", "execution_count": 6, "id": "040795c6-5d9f-4729-9132-f3b0f94d9e94", "metadata": {}, "outputs": [], "source": [ "# Helper function for formatting messages\n", "\n", "\n", "def format_tool_calls(tool_calls):\n", " if tool_calls:\n", " formatted_calls = []\n", " for call in tool_calls:\n", " formatted_calls.append(\n", " f\"Tool Call ID: {call['id']}, Function: {call['name']}, Arguments: {call['args']}\"\n", " )\n", " return \"\\n\".join(formatted_calls)\n", " return \"No tool calls\"" ] }, { "cell_type": "code", "execution_count": 7, "id": "7da70e20-1a4e-4df2-b996-1927f474c835", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Metadata: Run ID - 1ef2fe5c-6a1d-6575-bc09-d7832711c17e\n", "--------------------------------------------------\n", "Invalid Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: \n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': ''}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current'}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather'}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in'}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in San'}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in San Francisco'}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in San Francisco'}\n", "--------------------------------------------------\n", "Tool Calls:\n", "Tool Call ID: call_cg14F20jMBqWYrNgEkdWHwB3, Function: tavily_search_results_json, Arguments: {'query': 'current weather in San Francisco'}\n", "Response Metadata: Finish Reason - tool_calls\n", "--------------------------------------------------\n", "--------------------------------------------------\n", "AI: The\n", "--------------------------------------------------\n", "AI: The current\n", "--------------------------------------------------\n", "AI: The current weather\n", "--------------------------------------------------\n", "AI: The current weather in\n", "--------------------------------------------------\n", "AI: The current weather in San\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is over\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of \n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F).\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-s\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-south\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at \n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 k\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph).\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at \n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%,\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is \n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles).\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is \n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is 3\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is 3.\n", "--------------------------------------------------\n", "AI: The current weather in San Francisco is overcast with a temperature of 13.9°C (57.0°F). The wind is blowing from the south-southwest at 6.9 mph (11.2 kph). The humidity is at 81%, and the visibility is 16 km (9 miles). The UV index is 3.\n", "Response Metadata: Finish Reason - stop\n", "--------------------------------------------------\n" ] } ], "source": [ "input = {\"messages\": [{\"role\": \"user\", \"content\": \"what's the weather in sf\"}]}\n", "config = {\"configurable\": {\"model_name\": \"openai\"}}\n", "\n", "async for event in client.runs.stream(\n", " thread[\"thread_id\"],\n", " assistant_id,\n", " input=input,\n", " config=config,\n", " stream_mode=\"messages\",\n", "):\n", " if event.event == \"metadata\":\n", " print(f\"Metadata: Run ID - {event.data['run_id']}\")\n", " print(\"-\" * 50)\n", " elif event.event == \"messages/partial\":\n", " for data_item in event.data:\n", " if \"role\" in data_item and data_item[\"role\"] == \"user\":\n", " print(f\"Human: {data_item['content']}\")\n", " else:\n", " tool_calls = data_item.get(\"tool_calls\", [])\n", " invalid_tool_calls = data_item.get(\"invalid_tool_calls\", [])\n", " content = data_item.get(\"content\", \"\")\n", " response_metadata = data_item.get(\"response_metadata\", {})\n", "\n", " if content:\n", " print(f\"AI: {content}\")\n", "\n", " if tool_calls:\n", " print(\"Tool Calls:\")\n", " print(format_tool_calls(tool_calls))\n", "\n", " if invalid_tool_calls:\n", " print(\"Invalid Tool Calls:\")\n", " print(format_tool_calls(invalid_tool_calls))\n", "\n", " if response_metadata:\n", " finish_reason = response_metadata.get(\"finish_reason\", \"N/A\")\n", " print(f\"Response Metadata: Finish Reason - {finish_reason}\")\n", " print(\"-\" * 50)" ] } ], "metadata": { "kernelspec": { "display_name": "langgraph-example-dev", "language": "python", "name": "langgraph-example-dev" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 5 }