mirror of
https://github.com/langchain-ai/langgraph.git
synced 2026-08-26 09:32:25 +02:00
Shared state conceptual docs (#1958)
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> Co-authored-by: William Fu-Hinthorn <13333726+hinthornw@users.noreply.github.com>
This commit is contained in:
Lance Martin
co-authored by
Harrison Chase
William Fu-Hinthorn
Harrison Chase
William Fu-Hinthorn
parent
61798d09d2
commit
d5da547850
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 177 KiB |
+253
-117
@@ -2,41 +2,146 @@
|
||||
|
||||
## What is Memory?
|
||||
|
||||
Memory in the context of LLMs and AI applications refers to the ability to process, retain, and utilize information from past interactions or data sources. Examples include:
|
||||
Memory in the context of LLMs and AI applications refers to the ability to process, store, and effectively recall information from past interactions. With memory, your agents can learn from feedback and provide more relevant outputs to users.
|
||||
This guide is divided into two sections based on the scope of memory recall: short-term memory and long-term memory.
|
||||
|
||||
- Managing what messages (e.g., from a long message history) are sent to a chat model to limit token usage
|
||||
- Summarizing past conversations to give a chat model context from prior interactions
|
||||
- Selecting few shot examples (e.g., from a dataset) to guide model responses
|
||||
- Maintaining persistent data (e.g., user preferences) across multiple chat sessions
|
||||
- Allowing an LLM to update its own prompt using past information (e.g., meta-prompting)
|
||||
- Retrieving information relevant to a conversation or question from a long-term storage system
|
||||
**Short-term memory**, or thread-scoped memory, can be recalled _at any time_ **from within** a single conversational thread with a user. LangGraph manages short-term memory as a part of your agent's [state](low_level.md#state). State is persisted to a database using a [checkpointer](persistence.md#checkpoints), so the thread can be resumed at any time. Updates to short-term memory are triggered any time you invoke the graph or any time a step completes.
|
||||
|
||||
Below, we'll discuss each of these examples in some detail.
|
||||
**Long-term memory** is shared **across** conversational threads. It can be recalled _at any time_ and in any node. Memories are scoped to any custom namespace, not just within a single thread. LangGraph provides [stores](persistence.md#memory-store) to let you save and recall long-term memories.
|
||||
|
||||
## Managing Messages
|
||||
Both are important to understand and implement for your application.
|
||||
|
||||
|
||||
|
||||
## Short-term memory
|
||||
|
||||
Short-term memory refers to the ability to accurately remember previous turns within a single thread. A thread organizes multiple turns from a single conversation or session, similar to the way an email or slack thread groups messages in a single conversation. Reading from and writing updates to the memory is scoped to within a single thread.
|
||||
|
||||
LangGraph agents access short-term memory using checkpointed state. Using chatbots as a common example, the state would contain conversation history as a list of chat messages. Every user request and every assistant response are appended as messages to the state LangGraph saves the state updates in checkpoints scoped by the conversation's distinct [thread](persistence.md#threads) ID. For every subsequent user request, LangGraph can load the state from the appropriate checkpoint so your chatbot can see the entire conversation history.
|
||||
|
||||
If a user uploads files, if the bot generates code artifacts, or if the agent performs other side-effects, all of these objects could be stored and checkpointed as a part of your graph's state. That way, your bot can access the entire shared context for each conversation while keeping the context for each conversation separate from others.
|
||||
|
||||
Since conversation history is the most common form of representing short-term memory, in the next section, we will cover techniques for managing conversation history when interactions become **long**. If you want to stick to the high-level concepts, continue on to the [long-term memory](#long-term-memory) section.
|
||||
|
||||
Long conversations pose a challenge to today's LLMs. The full history may (a) not even fit inside an LLM's context window, resulting in an irrecoverable error. Even _if_ your LLM technically supports the full context length, most LLMs (b) still perform poorly over long contexts. They get "distracted" by stale or off-topic content, all while suffering from slower response times and higher costs.
|
||||
|
||||
Managing short-term memory is an exercise of balancing [precision & recall](https://en.wikipedia.org/wiki/Precision_and_recall#:~:text=Precision%20can%20be%20seen%20as,irrelevant%20ones%20are%20also%20returned) with your application's other performance requirements (latency & cost). As always, it's important to think critically about how you represent information for your LLM and to look at your data. We cover a few common techniques for managing message lists below and hope to provide sufficient context for you to pick the best tradeoffs for your application:
|
||||
|
||||
- [Editing message lists](#editing-message-lists): How to think about trimming and filtering a list of messages before passing to language model.
|
||||
- [Summarizing past conversations](#summarizing-past-conversations): A common technique to use when you don't just want to filter the list of messages.
|
||||
|
||||
### Editing message lists
|
||||
|
||||
Chat models accept instructions through [messages](https://python.langchain.com/docs/concepts/#messages), which can serve as general instructions (e.g., a system message) or user-provided instructions (e.g., human messages). In chat applications, messages often alternate between human inputs and model responses, accumulating in a list over time. Because context windows are limited and token-rich message lists can be costly, many applications can benefit from approaches to actively manage messages.
|
||||
Chat models accept context using [messages](https://python.langchain.com/docs/concepts/#messages). Messages transmit both developer or application-provided instructions (e.g., a system message) and user-provided instructions (e.g., human messages) all while recording the trajectory of the conversation or interaction. In chat applications, messages often alternate between human inputs and model responses, accumulating as a list over time. Because context windows are limited and token-rich message lists can be costly, many applications can benefit from using techniques to manually remove or forget stale information.
|
||||
|
||||
The most directed approach is to remove specific messages from a list. This can be done using [RemoveMessage](https://langchain-ai.github.io/langgraph/how-tos/memory/delete-messages/#manually-deleting-messages) based upon the message `id`, a unique identifier for each message. In the below example, we keep only the last two messages in the list using `RemoveMessage` to remove older messages based upon their `id`.
|
||||
|
||||
|
||||
The most direct approach is to remove old messages from a list (similar to a [least-recently used cache](https://en.wikipedia.org/wiki/Page_replacement_algorithm#Least_recently_used)).
|
||||
|
||||
The typical technique for deleting content from a list in LangGraph is to return an update from a node telling the system to delete some portion of the list. You get to define what this update looks like, but a common approach would be to let you return an object or dictionary specifying which values to retain.
|
||||
|
||||
```python
|
||||
from langchain_core.messages import RemoveMessage
|
||||
def manage_list(existing: list, updates: Union[list, dict]):
|
||||
if isinstance(updates, list):
|
||||
# Normal case, add to the history
|
||||
return existing + updates
|
||||
elif isinstance(updates, dict) and updates["type"] == "keep":
|
||||
# You get to decide what this looks like.
|
||||
# For example, you could simplify and just accept a string "DELETE"
|
||||
# and clear the entire list.
|
||||
return existing[updates["from"]:updates["to"]]
|
||||
# etc. We define how to interpret updates
|
||||
|
||||
# Message list
|
||||
messages = [AIMessage("Hi.", name="Bot", id="1")]
|
||||
messages.append(HumanMessage("Hi.", name="Lance", id="2"))
|
||||
messages.append(AIMessage("So you said you were researching ocean mammals?", name="Bot", id="3"))
|
||||
messages.append(HumanMessage("Yes, I know about whales. But what others should I learn about?", name="Lance", id="4"))
|
||||
class State(TypedDict):
|
||||
my_list: Annotated[list, manage_list]
|
||||
|
||||
# Isolate messages to delete
|
||||
delete_messages = [RemoveMessage(id=m.id) for m in messages[:-2]]
|
||||
print(delete_messages)
|
||||
[RemoveMessage(content='', id='1'), RemoveMessage(content='', id='2')]
|
||||
def my_node(state: State):
|
||||
return {
|
||||
# We return an update for the field "my_list" saying to
|
||||
# keep only values from index -5 to the end (deleting the rest)
|
||||
"my_list": {"type": "keep", "from": -5, "to": None}
|
||||
}
|
||||
```
|
||||
|
||||
Because the context window for chat model is denominated in tokens, it can be useful to trim message lists based upon some number of tokens that we want to retain. To do this, we can use [`trim_messages`](https://python.langchain.com/docs/how_to/trim_messages/#trimming-based-on-token-count) and specify number of token to keep from the list, as well as the `strategy` (e.g., keep the last `max_tokens`).
|
||||
LangGraph will call the `manage_list` "[reducer](low_level.md#reducers)" function any time an update is returned under the key "my_list". Within that function, we define what types of updates to accept. Typically, messages will be added to the existing list (the conversation will grow); however, we've also added support to accept a dictionary that lets you "keep" certain parts of the state. This lets you programmatically drop old message context.
|
||||
|
||||
Another common approach is to let you return a list of "remove" objects that specify the IDs of all messages to delete. If you're using the LangChain messages and the [`add_messages`](https://langchain-ai.github.io/langgraph/reference/graphs/#langgraph.graph.message.add_messages) reducer (or `MessagesState`, which uses the same underlying functionality) in LangGraph, you can do this using a `RemoveMessage`.
|
||||
|
||||
```python
|
||||
from langchain_core.messages import RemoveMessage, AIMessage
|
||||
from langgraph.graph import add_messages
|
||||
# ... other imports
|
||||
|
||||
class State(TypedDict):
|
||||
# add_messages will default to upserting messages by ID to the existing list
|
||||
# if a RemoveMessage is returned, it will delete the message in the list by ID
|
||||
messages: Annotated[list, add_messages]
|
||||
|
||||
def my_node_1(state: State):
|
||||
# Add an AI message to the `messages` list in the state
|
||||
return {"messages": [AIMessage(content="Hi")]}
|
||||
|
||||
def my_node_2(state: State):
|
||||
# Delete all but the last 2 messages from the `messages` list in the state
|
||||
delete_messages = [RemoveMessage(id=m.id) for m in state['messages'][:-2]]
|
||||
return {"messages": delete_messages}
|
||||
|
||||
```
|
||||
|
||||
In the example above, the `add_messages` reducer allows us to [append](https://langchain-ai.github.io/langgraph/concepts/low_level/#serialization) new messages to the `messages` state key as shown in `my_node_1`. When it sees a `RemoveMessage`, it will delete the message with that ID from the list (and the RemoveMessage will then be discarded). For more information on LangChain-specific message handling, check out [this how-to on using `RemoveMessage` ](https://langchain-ai.github.io/langgraph/how-tos/memory/delete-messages/).
|
||||
|
||||
See this how-to [guide](https://langchain-ai.github.io/langgraph/how-tos/memory/manage-conversation-history/) and module 2 from our [LangChain Academy](https://github.com/langchain-ai/langchain-academy/tree/main/module-2) course for example usage.
|
||||
|
||||
### Summarizing Past Conversations
|
||||
|
||||
The problem with trimming or removing messages, as shown above, is that we may lose information from culling of the message queue. Because of this, some applications benefit from a more sophisticated approach of summarizing the message history using a chat model.
|
||||
|
||||
|
||||
|
||||
Simple prompting and orchestration logic can be used to achieve this. As an example, in LangGraph we can extend the [MessagesState](https://langchain-ai.github.io/langgraph/concepts/low_level/#working-with-messages-in-graph-state) to include a `summary` key.
|
||||
|
||||
```python
|
||||
from langgraph.graph import MessagesState
|
||||
class State(MessagesState):
|
||||
summary: str
|
||||
```
|
||||
|
||||
Then, we can generate a summary of the chat history, using any existing summary as context for the next summary. This `summarize_conversation` node can be called after some number of messages have accumulated in the `messages` state key.
|
||||
|
||||
```python
|
||||
def summarize_conversation(state: State):
|
||||
|
||||
# First, we get any existing summary
|
||||
summary = state.get("summary", "")
|
||||
|
||||
# Create our summarization prompt
|
||||
if summary:
|
||||
|
||||
# A summary already exists
|
||||
summary_message = (
|
||||
f"This is a summary of the conversation to date: {summary}\n\n"
|
||||
"Extend the summary by taking into account the new messages above:"
|
||||
)
|
||||
|
||||
else:
|
||||
summary_message = "Create a summary of the conversation above:"
|
||||
|
||||
# Add prompt to our history
|
||||
messages = state["messages"] + [HumanMessage(content=summary_message)]
|
||||
response = model.invoke(messages)
|
||||
|
||||
# Delete all but the 2 most recent messages
|
||||
delete_messages = [RemoveMessage(id=m.id) for m in state["messages"][:-2]]
|
||||
return {"summary": response.content, "messages": delete_messages}
|
||||
```
|
||||
|
||||
See this how-to [here](https://langchain-ai.github.io/langgraph/how-tos/memory/add-summary-conversation-history/) and module 2 from our [LangChain Academy](https://github.com/langchain-ai/langchain-academy/tree/main/module-2) course for example usage.
|
||||
|
||||
### Knowing **when** to remove messages
|
||||
|
||||
Most LLMs have a maximum supported context window (denominated in tokens). A simple way to decide when to truncate messages is to count the tokens in the message history and truncate whenever it approaches that limit. Naive truncation is straightforward to implement on your own, though there are a few "gotchas". Some model APIs further restrict the sequence of message types (must start with human message, cannot have consecutive messages of the same type, etc.). If you're using LangChain, you can use the [`trim_messages`](https://python.langchain.com/docs/how_to/trim_messages/#trimming-based-on-token-count) utility and specify the number of tokens to keep from the list, as well as the `strategy` (e.g., keep the last `max_tokens`) to use for handling the boundary.
|
||||
|
||||
Below is an example.
|
||||
|
||||
```python
|
||||
from langchain_core.messages import trim_messages
|
||||
@@ -46,7 +151,7 @@ trim_messages(
|
||||
strategy="last",
|
||||
# Remember to adjust based on your model
|
||||
# or else pass a custom token_encoder
|
||||
token_counter=ChatOpenAI(model="gpt-4o"),
|
||||
token_counter=ChatOpenAI(model="gpt-4"),
|
||||
# Most chat models expect that chat history starts with either:
|
||||
# (1) a HumanMessage or
|
||||
# (2) a SystemMessage followed by a HumanMessage
|
||||
@@ -67,109 +172,140 @@ trim_messages(
|
||||
include_system=True,
|
||||
)
|
||||
```
|
||||
### Usage with LangGraph
|
||||
|
||||
When building agents in LangGraph, we commonly want to manage a list of messages in the graph state. Because this is such a common use case, [MessagesState](https://langchain-ai.github.io/langgraph/concepts/low_level/#working-with-messages-in-graph-state) is a built-in LangGraph state schema that includes a `messages` key, which is a list of messages. `MessagesState` also includes an `add_messages` reducer for updating the messages list with new messages as the application runs. The `add_messages` reducer allows us to [append](https://langchain-ai.github.io/langgraph/concepts/low_level/#serialization) new messages to the `messages` state key as shown below. When we perform a state update with `{"messages": new_message}` returned from `my_node`, the `add_messages` reducer appends `new_message` to the existing list of messages.
|
||||
## Long-term memory
|
||||
|
||||
Long-term memory refers to the ability of a system to remember information across different conversations (or sessions). While short-term memory is always scoped to a "thread", long-term memory is saved within custom scopes, or "namespaces."
|
||||
|
||||
Long-term memories are saved in a [store](persistence.md#memory-store). Each memory is a JSON document stored in a custom `namespace` under a distinct `key` in that namespace. You can think of namespaces like "folders" or "directories" on your computer. They're one way of organizing information into arbitrary collections. Common things to include in a namespace would be a user or organiation ID, a schema type, or other contextual information that makes it easier to manage. To take the analogy further, the `key` would be akin to the memory's "filename", and the value would contain the contents. This permits arbitrary hierarchical organization of memories while still letting you organize and search memories using content folders to support cross-cutting searches.
|
||||
|
||||
```python
|
||||
def my_node(state: State):
|
||||
# Add a new message to the state
|
||||
new_message = HumanMessage(content="message")
|
||||
return {"messages": new_message}
|
||||
from langgraph.store.memory import InMemoryStore
|
||||
|
||||
store = InMemoryStore()
|
||||
user_id = "my-user"
|
||||
application_context = "chitchat"
|
||||
namespace = (user_id, application_context)
|
||||
store.put(namespace, key="a-memory", {"rules": ["User likes short, direct language", "User only speaks English & python"], "my-key": "my-value"})
|
||||
# get the "memory" by ID
|
||||
item = store.get(namespace)
|
||||
# list "memories" within this namespace, filtering on content equivalence
|
||||
items = store.search(namespace, filter={"my-key": "my-value"})
|
||||
```
|
||||
|
||||
The `add_messages` reducer built into `MessagesState` [also works with the `RemoveMessage` utility that we discussed above](https://langchain-ai.github.io/langgraph/how-tos/memory/delete-messages/). In this case, we can perform a state update with a list of `delete_messages` to remove specific messages from the `messages` list.
|
||||
When adding long-term memory to your agent, it's important to think about how & how often to **write memories**, how to **stores and manage memory updates**, and how to **recall & represent memories** for the LLM in your application. These questions are all interdependent: how you want to recall & format memories for the LLM dictates what you should store and how to manage it. Furthermore, each technique has tradeoffs. The right approach for you largely depends on your application's needs.
|
||||
LangGraph aims to give you the low-level primitives to directly control the long-term memory of your application, based on memory [Store](persistence.md#memory-store)'s.
|
||||
|
||||
Long-term memory is far from a solved problem. While it is hard to provide generic advice, we have provided a few reliable patterns below for your consideration as you implement long-term memory.
|
||||
|
||||
**Do you want to write memories "in the hot path" or "in the background"**
|
||||
|
||||
Memory can be updated either as part of your primary application logic (e.g. "in the hot path" of the application) or as a background task (as a separate function that generates memories based on the primary application's state). We document some tradeoffs for each approach in [the writing memories section below](#writing-memories)
|
||||
|
||||
**Do you want to manage memories as a single profile or as a list of events?**
|
||||
|
||||
Managing memories as a single, continuously updated "profile" or "schema" is useful when there is well-scoped, specific information you want to remember about a user, organization, or other entity (including the agent itself). You can define the schema of the profile ahead of time, and then use an LLM to update this based on interactions. Querying the "memory" is easy since it's a simple GET operation on a JSON document.We explain this in more detail in [remember a profile](#manage-individual-profiles). This technique can provide higher precision (on known information use cases) at the expense of lower recall (since you have to anticipate and model your domain, and updates to the doc tend to delete or rewrite away old information at a greater frequency).
|
||||
|
||||
Managing long-term memory as a collection of documents, on the other hand, lets you store an unbounded amount of information. This technique is useful when you want to repeatedly extract & remember items over a long time horizon but can be more complicated to query and manage over time.
|
||||
Similar to the "profile" memory, you still define schema(s) for each memory. Rather than overwriting a single document, you instead will insert new ones (and potentially update or re-contextualize existing ones in the process). We explain this approach in more detail in ["managing a collection of memories"](#manage-a-collection-of-memories)
|
||||
|
||||
**Do you want to present memories to your agent as updated instructions or as few-shot examples?**
|
||||
|
||||
Memories are typically provided to the LLM as a part of the system prompt. Some common ways to "frame" memories for the LLM include providing raw information as "memories from previous interactions with user A", as system instructions or rules, or as few-shot examples.
|
||||
|
||||
Framing memories as "learning rules or instructions" typically means dedicating a portion of the system prompt to instructions the LLM can manage itself. After each conversation, you can prompt the LLM to evaluate its performance and update the instructions to better handle this type of task in the future. We explain this approach in more detail in [this section](#update-own-instructions).
|
||||
|
||||
Storing memories as few-shot examples lets you store and manage instructions as cause and effect. Each memory stores an input or context and expected response. Including a reasoning trajectory (a chain-of-thought) can also help provide sufficient context so that the memory is less likely to be mis-used in the future. We elaborate on this concept more in [this section](#few-shot-examples)
|
||||
|
||||
We will expand on techniques for writing, managing, and recalling & formatting memories in the following section.
|
||||
|
||||
### Writing memories
|
||||
|
||||
Humans form long-term memories when we sleep, but when and how should our agents create new memories? The two most common ways we see agents write memories are "in the hot path" and "in the background".
|
||||
|
||||
|
||||
|
||||
#### Writing memories in the hot path
|
||||
|
||||
This involves creating memories while the application is running. To provide a popular production example, ChatGPT manages memories using a "save_memories" tool to upsert memories as content strings. It decides whether (and how) to use this tool every time it receives a user message and multi-tasks memory management with the rest of the user instructions.
|
||||
|
||||
This has a few benefits. First of all, it happens "in real time". If the user starts a new thread right away that memory will be present. The user also transparently sees when memories are stored, since the bot has to explicitly decide to store information and can relate that to the user.
|
||||
|
||||
This also has several downsides. It complicates the decisions the agent must make (what to commit to memory). This complication can degrade its tool-calling performance and reduce task completion rates. It will slow down the final response since it needs to decide what to commit to memory. It also typically leads to fewer things being saved to memory (since the assistant is multi-tasking), which will cause **lower recall** in later conversations.
|
||||
|
||||
#### Writing memories in the background
|
||||
|
||||
This involves updating memory as a conceptually separate task, typically as a completely separate graph or function. Since it happens in the background, it incurs no latency. It also splits up the application logic from the memory logic, making it more modular and easy to manage. It also lets you separate the timing of memory creation, letting you avoid redundant work. Your agent can focus on accomplishing its immediate task without having to consciously think about what it needs to remember.
|
||||
|
||||
This approach is not without its downsides, however. You have to think about how often to write memories. If it doesn't run in realtime, the user's interactions on other threads won't benefit from the new context. You also have to think about when to trigger this job. We typically recommend scheduling memories after some point of time, cancelling and re-scheduling for the future if new events occur on a given thread. Other popular choices are to form memories on some cron schedule or to let the user or application logic manually trigger memory formation.
|
||||
|
||||
### Managing memories
|
||||
|
||||
Once you've sorted out memory scheduling, it's important to think about **how to update memory with new information**.
|
||||
|
||||
There are two ends of the spectrum: on one hand, you could continuously update a single document (memory profile). On the other, you could only ever insert new documents every time you receive new information.
|
||||
|
||||
We will outline some tradeoffs between these two approaches below, understanding that most people will find it most appropriate to combine approaches and to settle somewhere in the middle.
|
||||
|
||||
#### Manage individual profiles
|
||||
|
||||
A profile is generally just a JSON document with various key-value pairs you've selected to represent your domain. When remembering a profile, you will want to make sure that you are **updating** the profile each time. As a result, you will want to pass in the previous profile and ask the LLM to generate a new profile (or some JSON patch to apply to the old profile).
|
||||
|
||||
The larger the document, the more error-prone this can become. If your document becomes **too** large, you may want to consider splitting up the profiles into separate sections. You will likely need to use generation with retries and/or **strict** decoding when generating documents to ensure the memory schemas remains valid.
|
||||
|
||||
|
||||
|
||||
#### Manage a collection of memories
|
||||
|
||||
Saving memories as a collection of documents simplifies some things Each individual memory can be more narrowly scoped and easier to generate. It also means you're less likely to **lose** information over time, since it's easier for an LLM to generate _new_ objects for new information than it is for it to reconcile that new information with information in a dense profile. This tends to lead to higher recall downstream.
|
||||
|
||||
This approach shifts some complexity to how you prompt the LLM to apply memory updates. You now have to enable the LLM to _delete_ or _update_ existing items in the list. This can be tricky to prompt the LLM to do. Some LLMs may default to over-inserting; others may default to over-updating. Tuning the behavior here is best done through evals, something you can do with a tool like [LangSmith](https://docs.smith.langchain.com/tutorials/Developers/evaluation).
|
||||
|
||||
This also shifts complexity to memory **search** (recall). You have to think about what relevant items to use. Right now we support filtering by metadata. We will be adding semantic search shortly.
|
||||
|
||||
Finally, this shifts some complexity to how you represent the memories for the LLM (and by extension, the schemas you use to save each memories). It's very easy to write memories that can easily be mistaken out-of-context. It's important to prompt the LLM to include all necessary contextual information in the given memory so that when you use it in later conversations it doesn't mistakenly mis-apply that information.
|
||||
|
||||
|
||||
|
||||
### Representing memories
|
||||
|
||||
Once you have saved memories, the way you then retrieve and present the memory content for the LLM can play a large role in how well your LLM incorporates that information in its responses.
|
||||
The following sections present a couple of common approaches. Note that these sections also will largely inform how you write and manage memories. Everything in memory is connected!
|
||||
|
||||
#### Update own instructions
|
||||
|
||||
While instructions are often static text written by the developer, many AI applications benefit from letting the users personalize the rules and instructions the agent should follow whenever it interacts with that user. This ideally can be inferred by its interactions with the user (so the user doesn't have to explicitly change settings in yoru app). In this sense, instructions are a form of long-form memory!
|
||||
|
||||
One way to apply this is using "reflection" or "Meta-prompting" steps. Prompt the LLM with the current instruction set (from the system prompt) and a conversation with the user, and instruct the LLM to refine its instructions. This approach allows the system to dynamically update and improve its own behavior, potentially leading to better performance on various tasks. This is particularly useful for tasks where the instructions are challenging to specify a priori.
|
||||
|
||||
Meta-prompting uses past information to refine prompts. For instance, a [Tweet generator](https://www.youtube.com/watch?v=Vn8A3BxfplE) employs meta-prompting to enhance its paper summarization prompt for Twitter. You could implement this using LangGraph's memory store to save updated instructions in a shared namespace. In this case, we will namespace the memories as "agent_instructions" and key the memory based on the agent.
|
||||
|
||||
```python
|
||||
def my_node(state: State):
|
||||
# Delete messages from state
|
||||
delete_messages = [RemoveMessage(id=m.id) for m in state['messages'][:-2]]
|
||||
return {"messages": delete_messages}
|
||||
# Node that *uses* the instructions
|
||||
def call_model(state: State, store: BaseStore):
|
||||
namespace = ("agent_instructions", )
|
||||
instructions = store.get(namespace, key="agent_a")[0]
|
||||
# Application logic
|
||||
prompt = prompt_template.format(instructions=instructions.value["instructions"])
|
||||
...
|
||||
|
||||
|
||||
# Node that updates instructions
|
||||
def update_instructions(state: State, store: BaseStore):
|
||||
namespace = ("instructions",)
|
||||
current_instructions = store.search(namespace)[0]
|
||||
# Memory logic
|
||||
prompt = prompt_template.format(instructions=instructions.value["instructions"], conversation=state["messages"])
|
||||
output = llm.invoke(prompt)
|
||||
new_instructions = output['new_instructions']
|
||||
store.put(("agent_instructions",), "agent_a", {"instructions": new_instructions})
|
||||
...
|
||||
```
|
||||
|
||||
See this how-to [guide](https://langchain-ai.github.io/langgraph/how-tos/memory/manage-conversation-history/) and module 2 from our [LangChain Academy](https://github.com/langchain-ai/langchain-academy/tree/main/module-2) course for example usage.
|
||||
|
||||
|
||||
## Summarizing Past Conversations
|
||||
#### Few Shot Examples
|
||||
|
||||
The problem with trimming or removing messages, as shown above, is that we may loose information from culling of the message queue. Because of this, some applications benefit from a more sophisticated approach of summarizing the message history using a chat model.
|
||||
Sometimes it's easier to "show" than "tell." LLMs learn well from examples. Few-shot learning lets you ["program"](https://x.com/karpathy/status/1627366413840322562) your LLM by updating the prompt with input-output examples to illustrate the intended behavior. While various [best-practices](https://python.langchain.com/docs/concepts/#1-generating-examples) can be used to generate few-shot examples, often the challenge lies in selecting the most relevant examples based on user input.
|
||||
|
||||
Simple prompting and orchestration logic can be used to achieve this. As an example, in LangGraph we can extend the [MessagesState](https://langchain-ai.github.io/langgraph/concepts/low_level/#working-with-messages-in-graph-state) to include a `summary` key.
|
||||
|
||||
```python
|
||||
from langgraph.graph import MessagesState
|
||||
class State(MessagesState):
|
||||
summary: str
|
||||
```
|
||||
|
||||
Then, we can generate a summary of the chat history, using any existing summary as context for the next summary. This `summarize_conversation` node can be called after some number of messages have accumulated in the `messages` state key.
|
||||
|
||||
```python
|
||||
def summarize_conversation(state: State):
|
||||
|
||||
# First, we get any existing summary
|
||||
summary = state.get("summary", "")
|
||||
|
||||
# Create our summarization prompt
|
||||
if summary:
|
||||
|
||||
# A summary already exists
|
||||
summary_message = (
|
||||
f"This is summary of the conversation to date: {summary}\n\n"
|
||||
"Extend the summary by taking into account the new messages above:"
|
||||
)
|
||||
|
||||
else:
|
||||
summary_message = "Create a summary of the conversation above:"
|
||||
|
||||
# Add prompt to our history
|
||||
messages = state["messages"] + [HumanMessage(content=summary_message)]
|
||||
response = model.invoke(messages)
|
||||
|
||||
# Delete all but the 2 most recent messages
|
||||
delete_messages = [RemoveMessage(id=m.id) for m in state["messages"][:-2]]
|
||||
return {"summary": response.content, "messages": delete_messages}
|
||||
```
|
||||
|
||||
See this how-to [here](https://langchain-ai.github.io/langgraph/how-tos/memory/add-summary-conversation-history/) and module 2 from our [LangChain Academy](https://github.com/langchain-ai/langchain-academy/tree/main/module-2) course for example usage.
|
||||
|
||||
## Few Shot Examples
|
||||
|
||||
Few-shot learning is a powerful technique where LLMs can be ["programmed"](https://x.com/karpathy/status/1627366413840322562) inside the prompt with input-output examples to perform diverse tasks. While various [best-practices](https://python.langchain.com/docs/concepts/#1-generating-examples) can be used to generate few-shot examples, often the challenge lies in selecting the most relevant examples based on user input.
|
||||
|
||||
LangChain [`ExampleSelectors`](https://python.langchain.com/docs/how_to/#example-selectors) can be used to customize few-shot example selection from a collection of examples using criteria such as length, semantic similarity, semantic ngram overlap, or maximal marginal relevance.
|
||||
|
||||
If few-shot examples are stored in a [LangSmith Dataset](https://docs.smith.langchain.com/how_to_guides/datasets), then dynamic few-shot example selectors can be used out-of-the box to achieve this same goal. LangSmith will index the dataset for you and enable retrieval of few shot examples that are most relevant to the user input based upon keyword similarity ([using a BM25-like algorithm](https://docs.smith.langchain.com/how_to_guides/datasets/index_datasets_for_dynamic_few_shot_example_selection) for keyword based similarity).
|
||||
|
||||
See this how-to [video](https://www.youtube.com/watch?v=37VaU7e7t5o) for example usage of dynamic few-shot example selection in LangSmith. Also, see this [blog post](https://blog.langchain.dev/few-shot-prompting-to-improve-tool-calling-performance/) showcasing few-shot prompting to improve tool calling performance and this [blog post](https://blog.langchain.dev/aligning-llm-as-a-judge-with-human-preferences/) using few-shot example to align an LLMs to human preferences.
|
||||
|
||||
## Maintaining Data Across Chat Sessions
|
||||
|
||||
LangGraph's [persistence layer](https://langchain-ai.github.io/langgraph/concepts/persistence/#persistence) has checkpointers that utilize various storage systems, including an in-memory key-value store or different databases. These checkpoints capture the graph state at each execution step and accumulate in a thread, which can be accessed at a later time using a thread ID to resume a previous graph execution. We add persistence to our graph by passing a checkpointer to the `compile` method, as shown here.
|
||||
|
||||
```python
|
||||
# Compile the graph with a checkpointer
|
||||
checkpointer = MemorySaver()
|
||||
graph = workflow.compile(checkpointer=checkpointer)
|
||||
|
||||
# Invoke the graph with a thread ID
|
||||
config = {"configurable": {"thread_id": "1"}}
|
||||
graph.invoke(input_state, config)
|
||||
|
||||
# get the latest state snapshot at a later time
|
||||
config = {"configurable": {"thread_id": "1"}}
|
||||
graph.get_state(config)
|
||||
```
|
||||
|
||||
Persistence is critical sustaining a long-running chat sessions. For example, a chat between a user and an AI assistant may have interruptions. Persistence ensures that a user can continue that particular chat session at any later point in time. However, what happens if a user initiates a new chat session with an assistant? This spawns a new thread, and the information from the previous session (thread) is not retained. This motivates the need for a memory service that can maintain data across chat sessions (threads).
|
||||
|
||||
## Meta-prompting
|
||||
|
||||
Meta-prompting uses an LLM to generate or refine its own prompts or instructions. This approach allows the system to dynamically update and improve its own behavior, potentially leading to better performance on various tasks. This is particularly useful for tasks where the instructions are challenging to specify a priori.
|
||||
|
||||
Meta-prompting can use past information to update the prompt. As an example, this [Tweet generator](https://www.youtube.com/watch?v=Vn8A3BxfplE) uses meta-prompting to iteratively improve the summarization prompt used to generate high quality paper summaries for Twitter. In this case, we used a LangSmith dataset to house several papers that we wanted to summarize, generated summaries using a naive summarization prompt, manually reviewed the summaries, captured feedback from human review using the LangSmith Annotation Queue, and passed this feedback to a chat model to re-generate the summarization prompt. The process was repeated in a loop until the summaries met our criteria in human review.
|
||||
|
||||
## Retrieving relevant information from long-term storage
|
||||
|
||||
A central challenge that spans many different memory use-case can be summarized simply: how can we retrieve *relevant information* from a long-term storage system and pass it to a chat model? As an example, assume we have a system that stores a large number of specific details about a user, but the user asks a specific question related to restaurant recommendations. It would be costly to trivially extract *all* personal user information and pass it to a chat model. Instead, we want to extract only the information that is most relevant to the user's current chat interaction (e,g,. food preferences, location, etc.) and pass it to the chat model.
|
||||
|
||||
There is a large body of work on retrieval that aims to address this challenge. See our tutorials focused on [RAG, or Retrieval Augmented Generation](https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_adaptive_rag/), our conceptual docs on [retrieval](https://python.langchain.com/docs/concepts/#retrieval), and our [open source repository](https://github.com/langchain-ai/rag-from-scratch) along with [videos](https://www.youtube.com/playlist?list=PLfaIDFEXuae2LXbO1_PKyVJiQ23ZztA0x) on this topic.
|
||||
Note that the memory store is just one way to store data as few-shot examples. If you want to have more developer involvement, or tie few-shots more closely to your evaluation harness, you can also use a [LangSmith Dataset](https://docs.smith.langchain.com/how_to_guides/datasets) to store your data. Then dynamic few-shot example selectors can be used out-of-the box to achieve this same goal. LangSmith will index the dataset for you and enable retrieval of few shot examples that are most relevant to the user input based upon keyword similarity ([using a BM25-like algorithm](https://docs.smith.langchain.com/how_to_guides/datasets/index_datasets_for_dynamic_few_shot_example_selection) for keyword based similarity). See this how-to [video](https://www.youtube.com/watch?v=37VaU7e7t5o) for example usage of dynamic few-shot example selection in LangSmith. Also, see this [blog post](https://blog.langchain.dev/few-shot-prompting-to-improve-tool-calling-performance/) showcasing few-shot prompting to improve tool calling performance and this [blog post](https://blog.langchain.dev/aligning-llm-as-a-judge-with-human-preferences/) using few-shot example to align an LLMs to human preferences.
|
||||
@@ -216,6 +216,148 @@ The final thing you can optionally specify when calling `update_state` is `as_no
|
||||
|
||||
|
||||
|
||||
## Memory Store
|
||||
|
||||
|
||||
|
||||
A [state schema](low_level.md#schema) specifies a set of keys that are populated as a graph is executed. As discussed above, state can be written by a checkpointer to a thread at each graph step, enabling state persistence.
|
||||
|
||||
But, what if we want to retrain some information *across threads*? Consider the case of a chatbot where we want to retain specific information about the user across *all* chat conversations (e.g., threads) with that user!
|
||||
|
||||
With checkpointers alone, we cannot share information across threads. This motivates the need for the `Store` interface. As an illustration, we can define an `InMemoryStore` to store information about a user across threads. We simply compile our graph with a checkpointer, as before, and will our new `in_memory_store`.
|
||||
First, let's showcase this in isolation without using LangGraph.
|
||||
|
||||
```python
|
||||
from langgraph.store.memory import InMemoryStore
|
||||
in_memory_store = InMemoryStore()
|
||||
```
|
||||
|
||||
Memories are namespaced by a `tuple`, which in this specific example will be `(<user_id>, "memories")`. The namespace can be any length and represent anything, does not have be user specific.
|
||||
|
||||
```python
|
||||
user_id = "1"
|
||||
namespace_for_memory = (user_id, "memories")
|
||||
```
|
||||
|
||||
We use the `store.put` to save memories to our namespace in the store. When we do this, we specify the namespace, as defined above, and a key-value pair for the memory: the key is simply a unique identifier for the memory (`memory_id`) and the value (a dictionary) is the memory itself.
|
||||
|
||||
```python
|
||||
memory_id = str(uuid.uuid4())
|
||||
memory = {"food_preference" : "I like pizza"}
|
||||
in_memory_store.put(namespace_for_memory, memory_id, memory)
|
||||
```
|
||||
|
||||
We can read out memories in our namespace using `store.search`, which will return all memories for a given user as a list. The most recent memory is the last in the list.
|
||||
|
||||
```python
|
||||
memories = in_memory_store.search(namespace_for_memory)
|
||||
memories[-1].dict()
|
||||
{'value': {'food_preference': 'I like pizza'},
|
||||
'key': '07e0caf4-1631-47b7-b15f-65515d4c1843',
|
||||
'namespace': ['1', 'memories'],
|
||||
'created_at': '2024-10-02T17:22:31.590602+00:00',
|
||||
'updated_at': '2024-10-02T17:22:31.590605+00:00'}
|
||||
```
|
||||
|
||||
Each memory type is a Python class with certain attributes. We can access it as a dictionary by converting via `.dict` as above.
|
||||
The attributes it has are:
|
||||
|
||||
- `value`: The value (itself a dictionary) of this memory
|
||||
- `key`: The UUID for this memory in this namespace
|
||||
- `namespace`: A list of strings, the namespace of this memory type
|
||||
- `created_at`: Timestamp for when this memory was created
|
||||
- `updated_at`: Timestamp for when this memory was updated
|
||||
|
||||
With this all in place, we use the `in_memory_store` in LangGraph. The `in_memory_store` works hand-in-hand with the checkpointer: the checkpointer saves state to threads, as discussed above, and the the `in_memory_store` allows us to store arbitrary information for access *across* threads. We compile the graph with both the checkpointer and the `in_memory_store` as follows.
|
||||
|
||||
```python
|
||||
from langgraph.checkpoint.memory import MemorySaver
|
||||
|
||||
# We need this because we want to enable threads (conversations)
|
||||
checkpointer = MemorySaver()
|
||||
|
||||
# ... Define the graph ...
|
||||
|
||||
# Compile the graph with the checkpointer and store
|
||||
graph = graph.compile(checkpointer=checkpointer, store=in_memory_store)
|
||||
```
|
||||
|
||||
We invoke the graph with a `thread_id`, as before, and also with a `user_id`, which we'll use to namespace our memories to this particular user as we showed above.
|
||||
|
||||
```python
|
||||
# Invoke the graph
|
||||
user_id = "1"
|
||||
config = {"configurable": {"thread_id": "1", "user_id": user_id}}
|
||||
|
||||
# First let's just say hi to the AI
|
||||
for update in graph.stream(
|
||||
{"messages": [{"role": "user", "content": "hi"}]}, config, stream_mode="updates"
|
||||
):
|
||||
print(update)
|
||||
```
|
||||
|
||||
We can access the `in_memory_store` and the `user_id` in *any node* by passing `store: BaseStore` and `config: RunnableConfig` as node arguments. Just as we saw above, simply use the `put` method to save memories to the store.
|
||||
|
||||
```python
|
||||
def update_memory(state: MessagesState, config: RunnableConfig, *, store: BaseStore):
|
||||
|
||||
# Get the user id from the config
|
||||
user_id = config["configurable"]["user_id"]
|
||||
|
||||
# Namespace the memory
|
||||
namespace = (user_id, "memories")
|
||||
|
||||
# ... Analyze conversation and create a new memory
|
||||
|
||||
# Create a new memory ID
|
||||
memory_id = str(uuid.uuid4())
|
||||
|
||||
# We create a new memory
|
||||
store.put(namespace, memory_id, {"memory": memory})
|
||||
|
||||
```
|
||||
|
||||
As we showed above, we can also access the store in any node and use `search` to get memories. Recall the the memories are returned as a list of objects that can be converted to a dictionary.
|
||||
|
||||
```python
|
||||
memories[-1].dict()
|
||||
{'value': {'food_preference': 'I like pizza'},
|
||||
'key': '07e0caf4-1631-47b7-b15f-65515d4c1843',
|
||||
'namespace': ['1', 'memories'],
|
||||
'created_at': '2024-10-02T17:22:31.590602+00:00',
|
||||
'updated_at': '2024-10-02T17:22:31.590605+00:00'}
|
||||
```
|
||||
|
||||
We can access the memories and use them in our model call.
|
||||
|
||||
```python
|
||||
def call_model(state: MessagesState, config: RunnableConfig, *, store: BaseStore):
|
||||
|
||||
# Get the user id from the config
|
||||
user_id = config["configurable"]["user_id"]
|
||||
|
||||
# Get the memories for the user from the store
|
||||
memories = store.search(("memories", user_id))
|
||||
info = "\n".join([d.value["memory"] for d in memories])
|
||||
|
||||
# ... Use memories in the model call
|
||||
```
|
||||
|
||||
If we create a new thread, we can still access the same memories so long as the `user_id` is the same.
|
||||
|
||||
```python
|
||||
# Invoke the graph
|
||||
config = {"configurable": {"thread_id": "2", "user_id": "1"}}
|
||||
|
||||
# Let's say hi again
|
||||
for update in graph.stream(
|
||||
{"messages": [{"role": "user", "content": "hi, tell me about my memories"}]}, config, stream_mode="updates"
|
||||
):
|
||||
print(update)
|
||||
```
|
||||
|
||||
When we use the LangGraph API, either locally (e.g., in LangGraph Studio) or with LangGraph Cloud, the memory store is available to use by default and does not need to be specified during graph compilation.
|
||||
|
||||
## Checkpointer libraries
|
||||
|
||||
Under the hood, checkpointing is powered by checkpointer objects that conform to [BaseCheckpointSaver][langgraph.checkpoint.base.BaseCheckpointSaver] interface. LangGraph provides several checkpointer implementations, all implemented via standalone, installable libraries:
|
||||
|
||||
Reference in New Issue
Block a user