WEBDIENSTE/langgraph
1
0
Fork 0
mirror of https://github.com/langchain-ai/langgraph.git synced 2026-10-01 14:05:10 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: concepts for cloud and doc-reorg (#2196)

Browse Source 
Update langgraph documentation

---------

Co-authored-by: Vadym Barda <vadym@langchain.dev>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
Co-authored-by: Nuno Campos <nuno@langchain.dev>
Co-authored-by: Chester Curme <chester.curme@gmail.com>
Co-authored-by: William FH <13333726+hinthornw@users.noreply.github.com>
This commit is contained in:
Harrison Chase
2024-10-30 23:34:02 -04:00
committed by GitHub
co-authored by Vadym Barda Eugene Yurtsev Nuno Campos Chester Curme William FH
parent d6d6ab9b73
commit d0e59f406b
45 changed files with 3536 additions and 1248 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/concepts/streaming.md
+119 -1
View File
@@ -9,8 +9,23 @@ There are several different modes you can specify when calling these methods (e.
- [`"values"`](../how-tos/stream-values.ipynb): This streams the full value of the state after each step of the graph.
- [`"updates"`](../how-tos/stream-updates.ipynb): This streams the updates to the state after each step of the graph. If multiple updates are made in the same step (e.g. multiple nodes are run) then those updates are streamed separately.
- [`"custom"`](../how-tos/streaming-content.ipynb): This streams custom data from inside your graph nodes.
- [`"messages"`](../how-tos/streaming-tokens.ipynb): This streams LLM tokens and metadata for the graph node where LLM is invoked.
- `"debug"`: This streams as much information as possible throughout the execution of the graph.
You can also specify multiple streaming modes at the same time by passing them as a list. When you do this, the streamed outputs will be tuples `(stream_mode, data)`. For example:
```python
graph.stream(..., stream_mode=["updates", "messages"])
```
```
...
('messages', (AIMessageChunk(content='Hi'), {'langgraph_step': 3, 'langgraph_node': 'agent', ...}))
...
('updates', {'agent': {'messages': [AIMessage(content="Hi, how can I help you?")]}})
```
The below visualization shows the difference between the `values` and `updates` modes:
![values vs updates](../static/values_vs_updates.png)
@@ -130,4 +145,107 @@ guide for that [here](../how-tos/streaming-tokens.ipynb).
!!! warning "ASYNC IN PYTHON<=3.10"
You may fail to see events being emitted from inside a node when using `.astream_events` in Python <= 3.10. If you're using a Langchain RunnableLambda, a RunnableGenerator, or Tool asynchronously inside your node, you will have to propagate callbacks to these objects manually. This is because LangChain cannot automatically propagate callbacks to child objects in this case. Please see examples [here](../how-tos/streaming-content.ipynb) and [here](../how-tos/streaming-events-from-within-tools.ipynb).
You may fail to see events being emitted from inside a node when using `.astream_events` in Python <= 3.10. If you're using a Langchain RunnableLambda, a RunnableGenerator, or Tool asynchronously inside your node, you will have to propagate callbacks to these objects manually. This is because LangChain cannot automatically propagate callbacks to child objects in this case. Please see examples [here](../how-tos/streaming-content.ipynb) and [here](../how-tos/streaming-events-from-within-tools.ipynb).
## LangGraph Platform
Streaming is critical for making LLM applications feel responsive to end users. When creating a streaming run, the streaming mode determines what data is streamed back to the API client. LangGraph Platform supports five streaming modes:
- `values`: Stream the full state of the graph after each [super-step](https://langchain-ai.github.io/langgraph/concepts/low_level/#graphs) is executed. See the [how-to guide](../cloud/how-tos/stream_values.md) for streaming values.
- `messages`: Stream complete messages (at the end of node execution) as well as tokens for any messages generated inside a node. This mode is primarily meant for powering chat applications. This is only an option if your graph contains a `messages` key. See the [how-to guide](../cloud/how-tos/stream_messages.md) for streaming messages.
- `updates`: Streams updates to the state of the graph after each node is executed. See the [how-to guide](../cloud/how-tos/stream_updates.md) for streaming updates.
- `events`: Stream all events (including the state of the graph) that occur during graph execution. See the [how-to guide](../cloud/how-tos/stream_events.md) for streaming events. This can be used to do token-by-token streaming for LLMs.
- `debug`: Stream debug events throughout graph execution. See the [how-to guide](../cloud/how-tos/stream_debug.md) for streaming debug events.
You can also specify multiple streaming modes at the same time. See the [how-to guide](../cloud/how-tos/stream_multiple.md) for configuring multiple streaming modes at the same time.
See the [API reference](../cloud/reference/api/api_ref.html#tag/threads-runs/POST/threads/{thread_id}/runs/stream) for how to create streaming runs.
Streaming modes `values`, `updates`, and `debug` are very similar to modes available in the LangGraph library - for a deeper conceptual explanation of those, you can see the [previous section](#streaming-graph-outputs-stream-and-astream).
Streaming mode `events` is the same as using `.astream_events` in the LangGraph library - for a deeper conceptual explanation of this, you can see the [previous section](#streaming-graph-outputs-stream-and-astream).
### `stream_mode="messages"`
Streaming mode `messages` is for streaming back messages from the LLM. Assuming we have a simple [ReAct](./agentic_concepts.md#react-implementation)-style agent deployed, what does this stream look like?
All events emitted have two attributes:
- `event`: This is the name of the event
- `data`: This is data associated with the event
!!! note
Streaming mode `messages` is different from the one in the LangGraph library:
- LangGraph Server streams event objects with messages in the `data` field, while LangGraph library streams tuples (`AIMessageChunk`, metadata).
- In LangGraph Server, metadata is streamed only once per message (`messages/metadata`), before the individual tokens are streamed (`messages/partial`), while in LangGraph library it's streamed with every `AIMessageChunk` (for each LLM token).
- LangGraph Server also streams additional events (`metadata`, `messages/complete`, see below for more details).
Let's run it on a question that should trigger a tool call:
```python
thread = await client.threads.create()
input = {"messages": [{"role": "user", "content": "what's the weather in sf?"}]}
events = []
async for event in client.runs.stream(
thread["thread_id"],
assistant_id="agent", # This may need to change depending on the graph you deployed
input=input,
stream_mode="messages",
):
print(event.event)
```
```shell
metadata
messages/metadata
messages/partial
...
messages/partial
messages/metadata
messages/complete
messages/metadata
messages/partial
...
messages/partial
end
```
We first get some `metadata` - this is metadata about the run.
```python
StreamPart(event='metadata', data={'run_id': '1ef657cf-ae55-6f65-97d4-f4ed1dbdabc6'})
```
We then get a `messages/metadata` - this is letting us know that a new message is starting and provides additional information about the LLM as well as the node where the LLM is invoked.
```python
StreamPart(event='messages/metadata', data={'run-985c0f14-9f43-40d4-a505-4637fc58e333': {'metadata': {'created_by': 'system', 'run_id': '1ef657de-7594-66df-8eb2-31518e4a1ee2', 'graph_id': 'agent', 'thread_id': 'c178eab5-e293-423c-8e7d-1d113ffe7cd9', 'model_name': 'openai', 'assistant_id': 'fe096781-5601-53d2-b2f6-0d3403f7e9ca', 'langgraph_step': 1, 'langgraph_node': 'agent', 'langgraph_triggers': ['start:agent'], 'langgraph_task_idx': 0, 'ls_provider': 'openai', 'ls_model_name': 'gpt-4o', 'ls_model_type': 'chat', 'ls_temperature': 0.0}}})
```
We then get a BUNCH of `messages/partial` events - these are the individual tokens from the LLM! In the case below, we can see the START of a tool call.
```python
StreamPart(event='messages/partial', data=[{'content': '', 'additional_kwargs': {'tool_calls': [{'index': 0, 'id': 'call_w8Hr8dHGuZCPgRfd5FqRBArs', 'function': {'arguments': '', 'name': 'tavily_search_results_json'}, 'type': 'function'}]}, 'response_metadata': {}, 'type': 'ai', 'name': None, 'id': 'run-985c0f14-9f43-40d4-a505-4637fc58e333', 'example': False, 'tool_calls': [], 'invalid_tool_calls': [{'name': 'tavily_search_results_json', 'args': '', 'id': 'call_w8Hr8dHGuZCPgRfd5FqRBArs', 'error': None}], 'usage_metadata': None}])
```
The last `messages/partial` event for a given message will contain all of the tokens streamed for that message. In our case, it is now a complete tool call:
```python
StreamPart(event='messages/partial', data=[{'content': '', 'additional_kwargs': {'tool_calls': [{'index': 0, 'id': 'call_w8Hr8dHGuZCPgRfd5FqRBArs', 'function': {'arguments': '{"query":"current weather in San Francisco"}', 'name': 'tavily_search_results_json'}, 'type': 'function'}]}, 'response_metadata': {'finish_reason': 'tool_calls', 'model_name': 'gpt-4o-2024-05-13', 'system_fingerprint': 'fp_157b3831f5'}, 'type': 'ai', 'name': None, 'id': 'run-985c0f14-9f43-40d4-a505-4637fc58e333', 'example': False, 'tool_calls': [{'name': 'tavily_search_results_json', 'args': {'query': 'current weather in San Francisco'}, 'id': 'call_w8Hr8dHGuZCPgRfd5FqRBArs'}], 'invalid_tool_calls': [], 'usage_metadata': None}])
```
After that, we get another `messages/metadata`, now followed by a `messages/complete` event. This event is emitted for a tool message - our agent has called a tool, gotten a response, and now inserting it into the state in the form of a tool message.
```python
StreamPart(event='messages/complete', data=[{'content': '[{"url": "https://www.weatherapi.com/", "content": "{\'location\': {\'name\': \'San Francisco\', \'region\': \'California\', \'country\': \'United States of America\', \'lat\': 37.78, \'lon\': -122.42, \'tz_id\': \'America/Los_Angeles\', \'localtime_epoch\': 1724877689, \'localtime\': \'2024-08-28 13:41\'}, \'current\': {\'last_updated_epoch\': 1724877000, \'last_updated\': \'2024-08-28 13:30\', \'temp_c\': 23.3, \'temp_f\': 73.9, \'is_day\': 1, \'condition\': {\'text\': \'Partly cloudy\', \'icon\': \'//cdn.weatherapi.com/weather/64x64/day/116.png\', \'code\': 1003}, \'wind_mph\': 15.0, \'wind_kph\': 24.1, \'wind_degree\': 310, \'wind_dir\': \'NW\', \'pressure_mb\': 1014.0, \'pressure_in\': 29.93, \'precip_mm\': 0.0, \'precip_in\': 0.0, \'humidity\': 57, \'cloud\': 25, \'feelslike_c\': 25.0, \'feelslike_f\': 77.1, \'windchill_c\': 20.9, \'windchill_f\': 69.6, \'heatindex_c\': 23.3, \'heatindex_f\': 74.0, \'dewpoint_c\': 12.9, \'dewpoint_f\': 55.2, \'vis_km\': 16.0, \'vis_miles\': 9.0, \'uv\': 6.0, \'gust_mph\': 19.5, \'gust_kph\': 31.3}}"}]', 'additional_kwargs': {}, 'response_metadata': {}, 'type': 'tool', 'name': 'tavily_search_results_json', 'id': '0112eba5-7660-4375-9f24-c7a1d6777b97', 'tool_call_id': 'call_w8Hr8dHGuZCPgRfd5FqRBArs'}])
```
After that, we see the agent doing another LLM call and streaming back a response. We then get an `end` event:
```python
StreamPart(event='end', data=None)
```
And that's it! This is more focused streaming mode specifically focused on streaming back messages. See this [how-to guide](../cloud/how-tos/stream_messages.md) for more information.