diff --git a/docs/docs/how-tos/streaming-content.ipynb b/docs/docs/how-tos/streaming-content.ipynb index f4fa01623..a5c7261c3 100644 --- a/docs/docs/how-tos/streaming-content.ipynb +++ b/docs/docs/how-tos/streaming-content.ipynb @@ -7,13 +7,44 @@ "source": [ "# How to stream custom data\n", "\n", - "The most common use case for streaming from inside a node is to stream LLM tokens, but you may also want to stream custom data. For example, you might have some long-running streaming functions you may wish to render for the user. \n", + "
\n", + "

Prerequisites

\n", + "

\n", + " This guide assumes familiarity with the following:\n", + "

\n", + "

\n", + "
\n", + "\n", + "The most common use case for streaming from inside a node is to stream LLM tokens, but you may also want to stream custom data.\n", + "\n", + "For example, if you have a long-running tool call, you can dispatch custom events between the steps and use these custom events to monitor progress. You could also surface these custom events to an end user of your application to show them how the current task is progressing.\n", "\n", "You can do so in two ways:\n", "* using graph's `.stream` / `.astream` methods with `stream_mode=\"custom\"`\n", "* emitting custom events using [adispatch_custom_events](https://python.langchain.com/docs/how_to/callbacks_custom_events/).\n", "\n", - "Below is a simple toy example that shows both.\n", + "Below we'll see how to use both APIs.\n", "\n", "## Setup\n", "\n", @@ -114,7 +145,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 5, "id": "00a91b15-82c7-443c-acb6-a7406df15cee", "metadata": {}, "outputs": [ @@ -122,7 +153,15 @@ "name": "stdout", "output_type": "stream", "text": [ - "Four|score|and|seven|years|ago|our|fathers|...|" + "Four\n", + "score\n", + "and\n", + "seven\n", + "years\n", + "ago\n", + "our\n", + "fathers\n", + "...\n" ] } ], @@ -131,30 +170,59 @@ "\n", "inputs = [HumanMessage(content=\"What are you thinking about?\")]\n", "async for chunk in app.astream({\"messages\": inputs}, stream_mode=\"custom\"):\n", - " print(chunk, end=\"|\", flush=True)" + " print(chunk, flush=True)" ] }, { "cell_type": "markdown", - "id": "29035302-3111-45bf-ac69-50ab940f8cb4", + "id": "c7b9f1f0-c170-40dc-9c22-289483dfbc99", "metadata": {}, "source": [ - "## Stream custom data using `.astream_events`" + "You will likely need to use [multiple streaming modes](https://langchain-ai.github.io/langgraph/how-tos/stream-multiple/) as you will\n", + "want access to both the custom data and the state updates." ] }, { - "cell_type": "markdown", - "id": "822e91c3-03be-4778-9fa5-a6ec57be3e52", + "cell_type": "code", + "execution_count": 6, + "id": "f8ed22d4-6ce6-4b04-a68b-2ea516e3ab15", "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "('custom', 'Four')\n", + "('custom', 'score')\n", + "('custom', 'and')\n", + "('custom', 'seven')\n", + "('custom', 'years')\n", + "('custom', 'ago')\n", + "('custom', 'our')\n", + "('custom', 'fathers')\n", + "('custom', '...')\n", + "('updates', {'model': {'messages': [AIMessage(content='Four score and seven years ago our fathers ...', additional_kwargs={}, response_metadata={})]}})\n" + ] + } + ], "source": [ - "If you are already using graph's `.astream_events` method in your workflow, you can also stream custom data by emitting custom events using `adispatch_custom_event`" + "from langchain_core.messages import HumanMessage\n", + "\n", + "inputs = [HumanMessage(content=\"What are you thinking about?\")]\n", + "async for chunk in app.astream({\"messages\": inputs}, stream_mode=[\"custom\", \"updates\"]):\n", + " print(chunk, flush=True)" ] }, { + "attachments": {}, "cell_type": "markdown", - "id": "0fb6c3e5-7377-4f93-a8c6-44582ee3bc1a", + "id": "ca976d6a-7c64-4603-8bb4-dee95428c33d", "metadata": {}, "source": [ + "## Stream custom data using `.astream_events`\n", + "\n", + "If you are already using graph's `.astream_events` method in your workflow, you can also stream custom data by emitting custom events using `adispatch_custom_event`\n", + "\n", "
\n", "

ASYNC IN PYTHON<=3.10

\n", "

\n", @@ -178,7 +246,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 19, "id": "486a01a0", "metadata": {}, "outputs": [], @@ -229,7 +297,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 20, "id": "ce773a40", "metadata": {}, "outputs": [ @@ -270,7 +338,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.9" + "version": "3.11.4" } }, "nbformat": 4, diff --git a/docs/docs/how-tos/streaming-events-from-within-tools-without-langchain.ipynb b/docs/docs/how-tos/streaming-events-from-within-tools-without-langchain.ipynb index 2d702c1f2..d0404adf0 100644 --- a/docs/docs/how-tos/streaming-events-from-within-tools-without-langchain.ipynb +++ b/docs/docs/how-tos/streaming-events-from-within-tools-without-langchain.ipynb @@ -1,26 +1,49 @@ { "cells": [ { + "attachments": {}, "cell_type": "markdown", - "id": "b23ced4e-dc29-43be-9f94-0c36bb181b8a", - "metadata": {}, - "source": [ - "# How to stream events from within a tool (without LangChain LLMs / tools)" - ] - }, - { - "cell_type": "markdown", - "id": "7044eeb8-4074-4f9c-8a62-962488744557", - "metadata": {}, - "source": [ - "In this example we will stream tokens from within tools that an agent is using. We'll also be using OpenAI client library directly, without using LangChain chat models. We will use a ReAct agent as an example." - ] - }, - { - "cell_type": "markdown", - "id": "a37f60af-43ea-4aa6-847a-df8cc47065f5", + "id": "18e6e213-b398-4a7e-b342-ba225e97b424", "metadata": {}, "source": [ + "# How to stream events from within a tool (without LangChain LLMs / tools)\n", + "\n", + "\n", + "

\n", + "

Prerequisites

\n", + "

\n", + " This guide assumes familiarity with the following:\n", + "

\n", + "

\n", + "
\n", + "\n", + "In this guide, we will demonstrate how to stream tokens from tools used by a custom ReAct agent, without relying on LangChain’s chat models or tool-calling functionalities. \n", + "\n", + "We will use the OpenAI client library directly for the chat model interaction. The tool execution will be implemented from scratch.\n", + "\n", + "This showcases how LangGraph can be utilized independently of built-in LangChain components like chat models or tools.\n", + "\n", "## Setup\n", "\n", "First, let's install the required packages and set our API keys" @@ -28,7 +51,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 7, "id": "47f79af8-58d8-4a48-8d9a-88823d88701f", "metadata": {}, "outputs": [], @@ -39,7 +62,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "id": "0cf6b41d-7fcb-40b6-9a72-229cdd00a094", "metadata": {}, "outputs": [], @@ -70,24 +93,19 @@ ] }, { + "attachments": {}, "cell_type": "markdown", - "id": "e3d02ebb-c2e1-4ef7-b187-810d55139317", - "metadata": {}, - "source": [ - "## Define the graph" - ] - }, - { - "cell_type": "markdown", - "id": "3ba684f1-d46b-42e4-95cf-9685209a5992", + "id": "7d766c7d-34ea-455b-8bcb-f2f12d100e1d", "metadata": {}, "source": [ + "## Define the graph\n", + "\n", "### Define a node that will call OpenAI API" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 9, "id": "d59234f9-173e-469d-a725-c13e0979663e", "metadata": {}, "outputs": [], @@ -190,7 +208,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 10, "id": "b90941d8-afe4-42ec-9262-9c3b87c3b1ec", "metadata": {}, "outputs": [], @@ -261,7 +279,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 14, "id": "228260be-1f9a-4195-80e0-9604f8a5dba6", "metadata": {}, "outputs": [], @@ -299,12 +317,14 @@ "id": "d046e2ef-f208-4831-ab31-203b2e75a49a", "metadata": {}, "source": [ - "## Stream tokens from within the tool" + "## Stream tokens from within the tool\n", + "\n", + "Here, we'll use the `astream_events` API to stream back individual events. Please see [astream_events](https://python.langchain.com/docs/concepts/#astream_events) for more details." ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 15, "id": "45c96a79-4147-42e3-89fd-d942b2b49f6c", "metadata": {}, "outputs": [ @@ -344,7 +364,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.9" + "version": "3.11.4" } }, "nbformat": 4, diff --git a/docs/docs/how-tos/streaming-events-from-within-tools.ipynb b/docs/docs/how-tos/streaming-events-from-within-tools.ipynb index 8de3b5f7a..5ec94c90d 100644 --- a/docs/docs/how-tos/streaming-events-from-within-tools.ipynb +++ b/docs/docs/how-tos/streaming-events-from-within-tools.ipynb @@ -1,26 +1,53 @@ { "cells": [ { + "attachments": {}, "cell_type": "markdown", - "id": "b23ced4e-dc29-43be-9f94-0c36bb181b8a", - "metadata": {}, - "source": [ - "# How to stream events from within a tool" - ] - }, - { - "cell_type": "markdown", - "id": "7044eeb8-4074-4f9c-8a62-962488744557", - "metadata": {}, - "source": [ - "If your LangGraph graph needs to use tools that call LLMs (or any other LangChain `Runnable` objects -- other graphs, LCEL chains, retrievers, etc.), you might want to stream events from the underlying `Runnable`. This guide shows how you can do that." - ] - }, - { - "cell_type": "markdown", - "id": "a37f60af-43ea-4aa6-847a-df8cc47065f5", + "id": "04b012ac-e0b5-483e-a645-d13d0e215aad", "metadata": {}, "source": [ + "# How to stream data from within a tool\n", + "\n", + "
\n", + "

Prerequisites

\n", + "

\n", + " This guide assumes familiarity with the following:\n", + "

\n", + "

\n", + "
\n", + "\n", + "If your graph involves tools that invoke LLMs (or any other LangChain `Runnable` objects like other graphs, `LCEL` chains, or retrievers), you might want to surface partial results during the execution of the tool, especially if the tool takes a longer time to run.\n", + "\n", + "A common scenario is streaming LLM tokens generated by a tool calling an LLM, though this applies to any use of Runnable objects. \n", + "\n", + "This guide shows how to stream data from within a tool using the `astream` API with `stream_mode=\"messages\"` and also the more granular `astream_events` API. The `astream` API should be sufficient for most use cases.\n", + "\n", "## Setup\n", "\n", "First, let's install the required packages and set our API keys" @@ -28,7 +55,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 3, "id": "47f79af8-58d8-4a48-8d9a-88823d88701f", "metadata": {}, "outputs": [], @@ -39,7 +66,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "id": "0cf6b41d-7fcb-40b6-9a72-229cdd00a094", "metadata": {}, "outputs": [], @@ -74,30 +101,9 @@ "id": "e3d02ebb-c2e1-4ef7-b187-810d55139317", "metadata": {}, "source": [ - "## Define the graph" - ] - }, - { - "cell_type": "markdown", - "id": "d74a1760-a063-4d05-8c6f-9d16bc31fa82", - "metadata": {}, - "source": [ - "We'll use a prebuilt ReAct agent for this guide" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "083757a9-26d7-481e-8f3d-3e34bcba154b", - "metadata": {}, - "outputs": [], - "source": [ - "from langchain_core.callbacks import Callbacks\n", - "from langchain_core.prompts import ChatPromptTemplate\n", - "from langchain_core.tools import tool\n", + "## Define the graph\n", "\n", - "from langgraph.prebuilt import create_react_agent\n", - "from langchain_openai import ChatOpenAI" + "We'll use a prebuilt ReAct agent for this guide" ] }, { @@ -108,7 +114,7 @@ "
\n", "

ASYNC IN PYTHON<=3.10

\n", "

\n", - "Any Langchain RunnableLambda, a RunnableGenerator, or Tool that invokes other runnables and is running async in python<=3.10, will have to propagate callbacks to child objects manually. This is because LangChain cannot automatically propagate callbacks to child objects in this case.\n", + "Any Langchain `RunnableLambda`, a `RunnableGenerator`, or `Tool` that invokes other runnables and is running async in python<=3.10, will have to propagate callbacks to child objects **manually**. This is because LangChain cannot automatically propagate callbacks to child objects in this case.\n", " \n", "This is a common reason why you may fail to see events being emitted from custom runnables or tools.\n", "

\n", @@ -117,84 +123,61 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "2cb38dd9-74d8-456d-9e39-4655f2bf3f37", + "execution_count": 5, + "id": "f1975577-a485-42bd-b0f1-d3e987faf52b", "metadata": {}, "outputs": [], "source": [ + "from langchain_core.callbacks import Callbacks\n", + "from langchain_core.messages import HumanMessage\n", + "from langchain_core.tools import tool\n", + "\n", + "from langgraph.prebuilt import create_react_agent\n", + "from langchain_openai import ChatOpenAI\n", + "\n", + "\n", "@tool\n", "async def get_items(\n", - " place: str, callbacks: Callbacks\n", - ") -> str: # <--- Accept callbacks (Python <= 3.10)\n", + " place: str,\n", + " callbacks: Callbacks, # <--- Manually accept callbacks (needed for Python <= 3.10)\n", + ") -> str:\n", " \"\"\"Use this tool to look up which items are in the given place.\"\"\"\n", - " template = ChatPromptTemplate.from_messages(\n", + " # Attention when using async, you should be invoking the LLM using ainvoke!\n", + " # If you fail to do so, streaming will not WORK.\n", + " return await llm.ainvoke(\n", " [\n", - " (\n", - " \"human\",\n", - " \"Can you tell me what kind of items i might find in the following place: '{place}'. \"\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": f\"Can you tell me what kind of items i might find in the following place: '{place}'. \"\n", " \"List at least 3 such items separating them by a comma. And include a brief description of each item..\",\n", - " )\n", - " ]\n", + " }\n", + " ],\n", + " {\"callbacks\": callbacks},\n", " )\n", - " chain = template | llm.with_config(\n", - " {\n", - " \"run_name\": \"Get Items LLM\",\n", - " \"tags\": [\"tool_llm\"],\n", - " \"callbacks\": callbacks, # <-- Propagate callbacks (Python <= 3.10)\n", - " }\n", - " )\n", - " chunks = [chunk async for chunk in chain.astream({\"place\": place})]\n", - " return \"\".join(chunk.content for chunk in chunks)" - ] - }, - { - "cell_type": "markdown", - "id": "17279b8a-049d-483d-af63-8a875098e71f", - "metadata": {}, - "source": [ - "We're adding a custom tag (`tool_llm`) to our LLM runnable within the tool. This will allow us to filter events that we'll stream from the compiled graph (`agent`) Runnable below" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "7254310e-7016-45f7-9795-6d52a1160086", - "metadata": {}, - "outputs": [], - "source": [ - "llm = ChatOpenAI(model_name=\"gpt-3.5-turbo\")\n", + "\n", + "\n", + "llm = ChatOpenAI(model_name=\"gpt-4o\")\n", "tools = [get_items]\n", "agent = create_react_agent(llm, tools=tools)" ] }, { "cell_type": "markdown", - "id": "b7d88960-a66b-4699-adee-c12d40b4318a", + "id": "15cb55cc-b59d-4743-b6a3-13db75414d2c", "metadata": {}, "source": [ - "## Stream events from the graph" + "## Using stream_mode=\"messages\"\n", + "\n", + "Using `stream_mode=\"messages\"` is a good option if you don't have any complex LCEL logic inside of nodes (or you don't need super granular progress from within the LCEL chain)." ] }, { "cell_type": "code", - "execution_count": 25, - "id": "ec461f66", + "execution_count": 6, + "id": "4c9cdad3-3e9a-444f-9d9d-eae20b8d3486", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "1|.| Books| -| A| collection| of| written| or| printed| works| bound| together| and| typically| held| upright| on| a| shelf| for| easy| access| and| storage|.\n", - "|2|.| Picture| frames| -| Decor|ative| frames| used| to| display| photographs| or| artwork| on| a| shelf|,| adding| a| personal| touch| to| the| space|.\n", - "|3|.| Decor|ative| figur|ines| -| Small| sculptures| or| statues| that| are| placed| on| a| shelf| for| decorative| purposes|,| adding| visual| interest| and| personality| to| the| room|.|" - ] - } - ], + "outputs": [], "source": [ - "from langchain_core.messages import HumanMessage\n", - "\n", - "inputs = [HumanMessage(content=\"what is the weather in sf\")]\n", "final_message = \"\"\n", "async for msg, metadata in agent.astream(\n", " {\"messages\": [(\"human\", \"what items are on the shelf?\")]}, stream_mode=\"messages\"\n", @@ -213,32 +196,58 @@ ] }, { - "cell_type": "code", - "execution_count": 26, - "id": "1b35d72f", + "attachments": {}, + "cell_type": "markdown", + "id": "81656193-1cbf-4721-a8df-0e316fd510e5", "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'The items on the shelf are:\\n1. Books\\n2. Picture frames\\n3. Decorative figurines'" - ] - }, - "execution_count": 26, - "metadata": {}, - "output_type": "execute_result" - } - ], "source": [ - "final_message" + "## Using stream events API\n", + "\n", + "For simplicity, the `get_items` tool doesn't use any complex LCEL logic inside it -- it only invokes an LLM.\n", + "\n", + "However, if the tool were more complex (e.g., using a RAG chain inside it), and you wanted to see more granular events from within the chain, then you can use the astream events API.\n", + "\n", + "The example below only illustrates how to invoke the API.\n", + "\n", + "
\n", + "

Use async for the astream events API

\n", + "

\n", + " You should generally be using `async` code (e.g., using `ainvoke` to invoke the llm) to be able to leverage the astream events API properly.\n", + "

\n", + "
" ] }, { - "cell_type": "markdown", - "id": "d7f9457c-5665-4cd5-9a99-d54c84270616", + "cell_type": "code", + "execution_count": 7, + "id": "c3acdec9-0a24-4348-921e-435c8ea6f9fe", "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "|In| a| bedroom|,| you| might| find| the| following| items|:\n", + "\n", + "|1|.| **|Bed|**|:| The| central| piece| of| furniture| in| a| bedroom|,| typically| consisting| of| a| mattress| on| a| frame|,| where| people| sleep|.| It| often| includes| bedding| such| as| sheets|,| blankets|,| and| pillows| for| comfort|.\n", + "\n", + "|2|.| **|Ward|robe|**|:| A| large|,| tall| cupboard| or| fre|estanding| piece| of| furniture| used| for| storing| clothes|.| It| may| have| hanging| space|,| shelves|,| and| sometimes| drawers| for| organizing| garments| and| accessories|.\n", + "\n", + "|3|.| **|Night|stand|**|:| A| small| table| or| cabinet| placed| beside| the| bed|,| used| for| holding| items| like| a| lamp|,| alarm| clock|,| books|,| or| personal| belongings| that| might| be| needed| during| the| night| or| early| morning|.||" + ] + } + ], "source": [ - "You can see that the content of the final message is the same as the output we streamed above" + "from langchain_core.messages import HumanMessage\n", + "\n", + "async for event in agent.astream_events(\n", + " {\"messages\": [{\"role\": \"user\", \"content\": \"what's in the bedroom.\"}]}, version=\"v2\"\n", + "):\n", + " if (\n", + " event[\"event\"] == \"on_chat_model_stream\"\n", + " and event[\"metadata\"].get(\"langgraph_node\") == \"tools\"\n", + " ):\n", + " print(event[\"data\"][\"chunk\"].content, end=\"|\", flush=True)" ] } ], @@ -258,7 +267,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.9" + "version": "3.11.4" } }, "nbformat": 4, diff --git a/docs/docs/how-tos/streaming-tokens-without-langchain.ipynb b/docs/docs/how-tos/streaming-tokens-without-langchain.ipynb index 14f0ef2a9..bd7eba37c 100644 --- a/docs/docs/how-tos/streaming-tokens-without-langchain.ipynb +++ b/docs/docs/how-tos/streaming-tokens-without-langchain.ipynb @@ -347,7 +347,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.9" + "version": "3.11.4" } }, "nbformat": 4,