mirror of
https://github.com/langchain-ai/langgraph.git
synced 2026-09-11 20:27:54 +02:00
[docs]: conceptual guides for streaming from specific nodes/LLMs (#1537)
* x * changes * typo
This commit is contained in:
@@ -493,7 +493,7 @@ workflow.add_edge("call_model", END)
|
||||
app = workflow.compile()
|
||||
|
||||
inputs = [{"role": "user", "content": "hi!"}]
|
||||
async for event in app.astream_events({"messages": inputs}, version="v1"):
|
||||
async for event in app.astream_events({"messages": inputs}, version="v2"):
|
||||
kind = event["event"]
|
||||
print(f"{kind}: {event['name']}")
|
||||
```
|
||||
@@ -567,4 +567,72 @@ guide for that [here](../how-tos/streaming-tokens.ipynb).
|
||||
|
||||
|
||||
!!! warning "ASYNC IN PYTHON<=3.10"
|
||||
You may fail to see events being emitted from inside a node when using `.astream_events` in Python <= 3.10. If you're using a Langchain RunnableLambda, a RunnableGenerator, or Tool asynchronously inside your node, you will have to propagate callbacks to these objects manually. This is because LangChain cannot automatically propagate callbacks to child objects in this case. Please see examples [here](../how-tos/streaming-content.ipynb) and [here](../how-tos/streaming-events-from-within-tools.ipynb).
|
||||
You may fail to see events being emitted from inside a node when using `.astream_events` in Python <= 3.10. If you're using a Langchain RunnableLambda, a RunnableGenerator, or Tool asynchronously inside your node, you will have to propagate callbacks to these objects manually. This is because LangChain cannot automatically propagate callbacks to child objects in this case. Please see examples [here](../how-tos/streaming-content.ipynb) and [here](../how-tos/streaming-events-from-within-tools.ipynb).
|
||||
|
||||
#### Only stream tokens from specific nodes/LLMs
|
||||
|
||||
|
||||
There are certain cases where you have multiple nodes in your graph that make LLM calls, and you do not wish to stream the tokens from every single LLM call. For example, you may use one LLM as a planner for the next steps to take, and another LLM somewhere else in the graph that actually responds to the user. In that case, you most likely WON'T want to stream tokens from the planner LLM but WILL want to stream them from the respond to user LLM. Below we show two different ways of doing this, one by streaming from specific nodes only and the second by streaming from specific LLMs only.
|
||||
|
||||
First, let's define our graph:
|
||||
|
||||
```python
|
||||
from langchain_openai import ChatOpenAI
|
||||
from langgraph.graph import StateGraph, MessagesState, START, END
|
||||
|
||||
model_1 = ChatOpenAI(model="gpt-3.5-turbo", name="model_1")
|
||||
model_2 = ChatOpenAI(model="gpt-3.5-turbo", name="model_2")
|
||||
|
||||
def call_first_model(state: MessagesState):
|
||||
response = model_1.invoke(state['messages'])
|
||||
return {"messages": response}
|
||||
|
||||
def call_second_model(state: MessagesState):
|
||||
response = model_2.invoke(state['messages'])
|
||||
return {"messages": response}
|
||||
|
||||
workflow = StateGraph(MessagesState)
|
||||
workflow.add_node(call_first_model)
|
||||
workflow.add_node(call_second_model)
|
||||
workflow.add_edge(START, "call_first_model")
|
||||
workflow.add_edge("call_first_model", "call_second_model")
|
||||
workflow.add_edge("call_second_model", END)
|
||||
app = workflow.compile()
|
||||
```
|
||||
|
||||
**Streaming from specific node**
|
||||
|
||||
In the case that we only want the output from a single node, we can use the event metadata to filter node names:
|
||||
|
||||
```python
|
||||
inputs = [{"role": "user", "content": "hi!"}]
|
||||
|
||||
async for event in app.astream_events({"messages": inputs}, version="v2"):
|
||||
# Get chat model tokens from a particular node
|
||||
if event["event"] == "on_chat_model_stream" and event['metadata'].get('langgraph_node','') == "call_second_model":
|
||||
print(event["data"]["chunk"].content, end="|", flush=True)
|
||||
```
|
||||
|
||||
```shell
|
||||
|Hello|!| How| can| I| help| you| today|?||
|
||||
```
|
||||
|
||||
As we can see only the response from the second LLM was streamed (you can tell because we only received a single response, if we had streamed both we would have received two "Hello! How can I help you today?" messages).
|
||||
|
||||
**Streaming from specific LLM**
|
||||
|
||||
Sometimes you might want to stream from specific LLMs instead of specific nodes. This could be the case if you have multiple LLM calls inside a single node, and only want to stream the output of a specific one or if you use the same LLM in different nodes and want to stream it's output anytime it is called. We can do this by using the `name` parameter for LLMs and events:
|
||||
|
||||
```python
|
||||
inputs = [{"role": "user", "content": "hi!"}]
|
||||
async for event in app.astream_events({"messages": inputs}, version="v2"):
|
||||
# Get chat model tokens from a particular LLM inside a particular node
|
||||
if event["event"] == "on_chat_model_stream" and event['name'] == "model_2":
|
||||
print(event["data"]["chunk"].content, end="|", flush=True)
|
||||
```
|
||||
|
||||
```shell
|
||||
|Hello|!| How| can| I| assist| you| today|?||
|
||||
```
|
||||
|
||||
As expected, we only see a single LLM response since the response from `model_1` was not streamed.
|
||||
Reference in New Issue
Block a user