From b5cbaad301bff00df4747a5933de868dfb71148f Mon Sep 17 00:00:00 2001 From: Harrison Chase Date: Mon, 15 Jan 2024 17:51:33 -0800 Subject: [PATCH] cr --- README.md | 9 + examples/async.ipynb | 4 +- .../async.ipynb => streaming-tokens.ipynb} | 195 ++++-------------- 3 files changed, 53 insertions(+), 155 deletions(-) rename examples/{chat_agent_executor_with_function_calling/async.ipynb => streaming-tokens.ipynb} (67%) diff --git a/README.md b/README.md index c19cfab94..87aab4f13 100644 --- a/README.md +++ b/README.md @@ -452,6 +452,15 @@ We also have a lot of examples highlighting how to slightly modify the base chat - [Force calling a tool first](examples/agent_executor/force-calling-a-tool-first.ipynb): How to always call a specific tool first - [Managing agent steps](examples/agent_executor/managing-agent-steps.ipynb): How to more explicitly manage intermediate steps that an agent takes +### Async + +If you are running LangGraph in async workflows, you may want to create the nodes to be async by default. +In order for a walkthrough on how to do that, see [this documentation](examples/async.ipynb) + +### Streaming Tokens + +Sometimes language models take a while to respond and you may want to stream tokens to end users. +For a guide on how to do this, see [this documentation](examples/streaming-tokens.ipynb) ## Documentation diff --git a/examples/async.ipynb b/examples/async.ipynb index 2fae3e3f9..7d4a8081c 100644 --- a/examples/async.ipynb +++ b/examples/async.ipynb @@ -7,9 +7,7 @@ "source": [ "# Async\n", "\n", - "In this example we will build a chat executor with native async implementations of the core logic. This enables taking advantage of Chat Models which have async clients, removing the need for calling the model in a separate thread.\n", - "\n", - "For more information" + "In this example we will build a chat executor with native async implementations of the core logic. This enables taking advantage of Chat Models which have async clients, removing the need for calling the model in a separate thread." ] }, { diff --git a/examples/chat_agent_executor_with_function_calling/async.ipynb b/examples/streaming-tokens.ipynb similarity index 67% rename from examples/chat_agent_executor_with_function_calling/async.ipynb rename to examples/streaming-tokens.ipynb index e9cb177c8..d1dfed602 100644 --- a/examples/chat_agent_executor_with_function_calling/async.ipynb +++ b/examples/streaming-tokens.ipynb @@ -5,9 +5,14 @@ "id": "51466c8d-8ce4-4b3d-be4e-18fdbeda5f53", "metadata": {}, "source": [ - "# Chat Executor: Async\n", + "# Streaming Tokens\n", "\n", - "In this example we will build a chat executor with native async implementations of the core logic. This enables taking advantage of Chat Models which have async clients, removing the need for calling the model in a separate thread." + "In this example we will focus on explaining how to stream tokens from a language model that is powering an agent. We will use a chat agent executor as an example. There a few specific things we need to do in order to properly stream tokens. They are: \n", + "\n", + "1. Set `streaming=True` when creating the LLM\n", + "2. Create nodes with [async methods](./async.ipynb) - this is best practice because in order to stream tokens we will use the `async_log` method.\n", + "\n", + "we will call them out with the **STREAMING** tag below (if you just want to search for those)." ] }, { @@ -149,7 +154,11 @@ "1. It should work with messages. We will represent all agent state in the form of messages, so it needs to be able to work well with them.\n", "2. It should work with OpenAI function calling. This means it should either be an OpenAI model or a model that exposes a similar interface.\n", "\n", - "Note: these model requirements are not requirements for using LangGraph - they are just requirements for this one example.\n" + "Note: these model requirements are not requirements for using LangGraph - they are just requirements for this one example.\n", + "\n", + "**STREAMING**\n", + "\n", + "Here, we set `streaming=True` when creating the model." ] }, { @@ -249,7 +258,7 @@ "\n", "Let's define the nodes, as well as a function to decide how what conditional edge to take.\n", "\n", - "**MODIFICATION**\n", + "**STREAMING**\n", "\n", "We define each node as an async function." ] @@ -362,117 +371,14 @@ "app = workflow.compile()" ] }, - { - "cell_type": "markdown", - "id": "547c3931-3dae-4281-ad4e-4b51305594d4", - "metadata": {}, - "source": [ - "## Use it!\n", - "\n", - "We can now use it!\n", - "This now exposes the [same interface](https://python.langchain.com/docs/expression_language/) as all other LangChain runnables." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "8edb04b9-40b6-46f1-a7a8-4b2d8aba7752", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'messages': [HumanMessage(content='what is the weather in sf'),\n", - " AIMessage(content='', additional_kwargs={'function_call': {'arguments': '{\\n \"query\": \"weather in San Francisco\"\\n}', 'name': 'tavily_search_results_json'}}),\n", - " FunctionMessage(content=\"[{'url': 'https://www.whereandwhen.net/when/north-america/california/san-francisco-ca/january/', 'content': 'Best time to go to San Francisco? Weather in San Francisco in january 2024 How was the weather last january? Here is the day by day recorded weather in San Francisco in january 2023: Seasonal average climate and temperature of San Francisco in january 8% 46% 29% 12% 8% Evolution of daily average temperature and precipitation in San Francisco in januaryWeather in San Francisco in january 2024. The weather in San Francisco in january comes from statistical datas on the past years. You can view the weather statistics the entire month, but also by using the tabs for the beginning, the middle and the end of the month. ... 16-01-2023 45°F to 52°F. 17-01-2023 45°F to 54°F. 18-01-2023 47°F to ...'}]\", name='tavily_search_results_json'),\n", - " AIMessage(content=\"I'm sorry, but I couldn't find the current weather in San Francisco. However, you can check the weather forecast for San Francisco on websites like Weather.com or AccuWeather.\")]}" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "from langchain_core.messages import HumanMessage\n", - "\n", - "inputs = {\"messages\": [HumanMessage(content=\"what is the weather in sf\")]}\n", - "await app.ainvoke(inputs)" - ] - }, - { - "cell_type": "markdown", - "id": "5a9e8155-70c5-4973-912c-dc55104b2acf", - "metadata": {}, - "source": [ - "This may take a little bit - it's making a few calls behind the scenes.\n", - "In order to start seeing some intermediate results as they happen, we can use streaming - see below for more information on that.\n", - "\n", - "## Streaming\n", - "\n", - "LangGraph has support for several different types of streaming.\n", - "\n", - "### Streaming Node Output\n", - "\n", - "One of the benefits of using LangGraph is that it is easy to stream output as it's produced by each node.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "f544977e-31f7-41f0-88c4-ec9c27b8cecb", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Output from node 'agent':\n", - "---\n", - "{'messages': [AIMessage(content='', additional_kwargs={'function_call': {'arguments': '{\\n \"query\": \"weather in San Francisco\"\\n}', 'name': 'tavily_search_results_json'}})]}\n", - "\n", - "---\n", - "\n", - "Output from node 'action':\n", - "---\n", - "{'messages': [FunctionMessage(content=\"[{'url': 'https://www.weather2travel.com/california/san-francisco/january/', 'content': 'San Francisco weather in January 2024 Expect 13°C daytime maximum temperatures long-term weather averages for San Francisco in January before you book your next holiday to California in 2024/2025. San Francisco January sunrise & sunset times How sunny is it in San Francisco in January?San Francisco weather in January 2024 Expect 13°C daytime maximum temperatures in the shade with on average 6 hours of sunshine per day in San Francisco in January. Check more long-term weather averages for San Francisco in January before you book your next holiday to California in 2024/2025. 13 13°C max day temperature 6'}]\", name='tavily_search_results_json')]}\n", - "\n", - "---\n", - "\n", - "Output from node 'agent':\n", - "---\n", - "{'messages': [AIMessage(content='The weather in San Francisco is currently not available. However, in January, the average daytime maximum temperature is around 13°C with an average of 6 hours of sunshine per day. Please note that this information is based on long-term weather averages and may vary.')]}\n", - "\n", - "---\n", - "\n", - "Output from node '__end__':\n", - "---\n", - "{'messages': [HumanMessage(content='what is the weather in sf'), AIMessage(content='', additional_kwargs={'function_call': {'arguments': '{\\n \"query\": \"weather in San Francisco\"\\n}', 'name': 'tavily_search_results_json'}}), FunctionMessage(content=\"[{'url': 'https://www.weather2travel.com/california/san-francisco/january/', 'content': 'San Francisco weather in January 2024 Expect 13°C daytime maximum temperatures long-term weather averages for San Francisco in January before you book your next holiday to California in 2024/2025. San Francisco January sunrise & sunset times How sunny is it in San Francisco in January?San Francisco weather in January 2024 Expect 13°C daytime maximum temperatures in the shade with on average 6 hours of sunshine per day in San Francisco in January. Check more long-term weather averages for San Francisco in January before you book your next holiday to California in 2024/2025. 13 13°C max day temperature 6'}]\", name='tavily_search_results_json'), AIMessage(content='The weather in San Francisco is currently not available. However, in January, the average daytime maximum temperature is around 13°C with an average of 6 hours of sunshine per day. Please note that this information is based on long-term weather averages and may vary.')]}\n", - "\n", - "---\n", - "\n" - ] - } - ], - "source": [ - "inputs = {\"messages\": [HumanMessage(content=\"what is the weather in sf\")]}\n", - "async for output in app.astream(inputs):\n", - " # stream() yields dictionaries with output keyed by node name\n", - " for key, value in output.items():\n", - " print(f\"Output from node '{key}':\")\n", - " print(\"---\")\n", - " print(value)\n", - " print(\"\\n---\\n\")" - ] - }, { "cell_type": "markdown", "id": "2a1b56c5-bd61-4192-8bdb-458a1e9f0159", "metadata": {}, "source": [ - "### Streaming LLM Tokens\n", + "## Streaming LLM Tokens\n", "\n", - "You can also access the LLM tokens as they are produced by each node. \n", + "You can access the LLM tokens as they are produced by each node. \n", "In this case only the \"agent\" node produces LLM tokens.\n", "In order for this to work properly, you must be using an LLM that supports streaming as well as have set it when constructing the LLM (e.g. `ChatOpenAI(model=\"gpt-3.5-turbo-1106\", streaming=True)`)\n" ] @@ -502,64 +408,49 @@ "content='' additional_kwargs={'function_call': {'arguments': '}', 'name': ''}}\n", "content=''\n", "content=''\n", - "content='The'\n", + "content='I'\n", + "content=\"'m\"\n", + "content=' sorry'\n", + "content=','\n", + "content=' but'\n", + "content=' I'\n", + "content=' couldn'\n", + "content=\"'t\"\n", + "content=' find'\n", + "content=' the'\n", + "content=' current'\n", "content=' weather'\n", "content=' in'\n", "content=' San'\n", "content=' Francisco'\n", - "content=' is'\n", - "content=' currently'\n", - "content=' not'\n", - "content=' available'\n", "content='.'\n", "content=' However'\n", "content=','\n", - "content=' in'\n", - "content=' January'\n", - "content=','\n", + "content=' you'\n", + "content=' can'\n", + "content=' check'\n", "content=' the'\n", - "content=' average'\n", - "content=' daytime'\n", - "content=' maximum'\n", - "content=' temperature'\n", - "content=' is'\n", - "content=' around'\n", - "content=' '\n", - "content='13'\n", - "content='°C'\n", - "content=' with'\n", - "content=' an'\n", - "content=' average'\n", - "content=' of'\n", - "content=' '\n", - "content='6'\n", - "content=' hours'\n", - "content=' of'\n", - "content=' sunshine'\n", - "content=' per'\n", - "content=' day'\n", - "content='.'\n", - "content=' Please'\n", - "content=' note'\n", - "content=' that'\n", - "content=' this'\n", - "content=' information'\n", - "content=' is'\n", - "content=' based'\n", - "content=' on'\n", - "content=' long'\n", - "content='-term'\n", "content=' weather'\n", - "content=' averages'\n", - "content=' and'\n", - "content=' may'\n", - "content=' vary'\n", + "content=' forecast'\n", + "content=' for'\n", + "content=' San'\n", + "content=' Francisco'\n", + "content=' on'\n", + "content=' websites'\n", + "content=' like'\n", + "content=' Weather'\n", + "content='.com'\n", + "content=' or'\n", + "content=' Acc'\n", + "content='u'\n", + "content='Weather'\n", "content='.'\n", "content=''\n" ] } ], "source": [ + "from langchain_core.messages import HumanMessage\n", "inputs = {\"messages\": [HumanMessage(content=\"what is the weather in sf\")]}\n", "async for output in app.astream_log(inputs, include_types=[\"llm\"]):\n", " # astream_log() yields the requested logs (here LLMs) in JSONPatch format\n",