mirror of
https://github.com/langchain-ai/langgraph.git
synced 2026-09-12 04:37:51 +02:00
Merge pull request #51 from langchain-ai/harrison/agent-simulation
Harrison/agent simulation
This commit is contained in:
Harrison Chase
@@ -453,10 +453,15 @@ We also have a lot of examples highlighting how to slightly modify the base chat
|
||||
|
||||
### Multi-agent Examples
|
||||
|
||||
- [Multi-agent collaboration](examples/multi_agent/multi-agent-collaboration.ipynb): how to create two agents that work together to accomplish a task
|
||||
- [Multi-agent with supervisor](examples/multi_agent/agent_supervisor.ipynb): how to orchestrate individual agents by using an LLM as a "supervisor" to distribute work
|
||||
- [Hierarchical agent teams](examples/multi_agent/hierarchical_agent_teams.ipynb): how to orchestrate "teams" of agents as nested graphs that can collaborate to solve a problem
|
||||
- [Chat bot evaluation as multi-agent simulation](examples/multi_agent/agent-simulation-evaluation.ipynb): How to simulate a dialogue between a "virtual user" and your chat bot
|
||||
- [Multi-agent collaboration](https://github.com/langchain-ai/langgraph/blob/main/examples/multi_agent/multi-agent-collaboration.ipynb): how to create two agents that work together to accomplish a task
|
||||
- [Multi-agent with supervisor](https://github.com/langchain-ai/langgraph/blob/main/examples/multi_agent/agent_supervisor.ipynb): how to orchestrate individual agents by using an LLM as a "supervisor" to distribute work
|
||||
- [Hierarchical agent teams](https://github.com/langchain-ai/langgraph/blob/main/examples/multi_agent/hierarchical_agent_teams.ipynb): how to orchestrate "teams" of agents as nested graphs that can collaborate to solve a problem
|
||||
|
||||
### Chatbot Evaluation via Simulation
|
||||
|
||||
It can often be tough to evaluation chat bots in multi-turn situations. One way to do this is with simulations.
|
||||
|
||||
- [Chat bot evaluation as multi-agent simulation](https://github.com/langchain-ai/langgraph/blob/main/examples/chatbot-simulation-evaluation/agent-simulation-evaluation.ipynb): How to simulate a dialogue between a "virtual user" and your chat bot
|
||||
|
||||
### Async
|
||||
|
||||
|
||||
@@ -0,0 +1,424 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a3e3ebc4-57af-4fe4-bdd3-36aff67bf276",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Chat Bot Evaluation as Multi-agent Simulation\n",
|
||||
"\n",
|
||||
"When building a chat bot, such as a customer support assistant, it can be hard to properly evalute your bot's performance. It's time-consuming to have to manually interact with it intensively for each code change.\n",
|
||||
"\n",
|
||||
"One way to make the evaluation process easier and more reproducible is to simulate a user interaction.\n",
|
||||
"\n",
|
||||
"With LangGraph, it's easy to set this up. Below is an example of how to create a \"virtual user\" to simulate a conversation.\n",
|
||||
"\n",
|
||||
"The overall simulation looks something like this:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"First, we'll set up our environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "0d30b6f7-3bec-4d9f-af50-43dfdc81ae6c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# %%capture --no-stderr\n",
|
||||
"# %pip install -U langgraph langchain langchain_openai"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "30c2f3de-c730-4aec-85a6-af2c2f058803",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import getpass\n",
|
||||
"import os\n",
|
||||
"import uuid\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def _set_if_undefined(var: str):\n",
|
||||
" if not os.environ.get(var):\n",
|
||||
" os.environ[var] = getpass(f\"Please provide your {var}\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"_set_if_undefined(\"OPENAI_API_KEY\")\n",
|
||||
"_set_if_undefined(\"LANGCHAIN_API_KEY\")\n",
|
||||
"\n",
|
||||
"# Optional, add tracing in LangSmith.\n",
|
||||
"# This will help you visualize and debug the control flow\n",
|
||||
"os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
|
||||
"os.environ[\"LANGCHAIN_PROJECT\"] = \"Agent Simulation Evaluation\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6ef4528d-6b2a-47c7-98b5-50f14984a304",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Define Chat Bot\n",
|
||||
"\n",
|
||||
"Next, we will define our chat bot. For this notebook, we assume the bot's API accepts a list of messages and responds with a message. If you want to update this, all you'll have to change is this section and the \"get_messages_for_agent\" function in \n",
|
||||
"the simulator below.\n",
|
||||
"\n",
|
||||
"The implementation within `my_chat_bot` is configurable and can even be run on another system (e.g., if your system isn't running in python)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "828479af-cf9c-4888-a365-599643a96b55",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import List\n",
|
||||
"\n",
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# This is flexible, but you can define your agent here, or call your agent API here.\n",
|
||||
"def my_chat_bot(messages: List[dict]) -> dict:\n",
|
||||
" system_message = {\"role\": \"system\", \"content\": \"You are a customer support agent for an airline.\"}\n",
|
||||
" messages = [system_message] + messages\n",
|
||||
" completion = openai.chat.completions.create(\n",
|
||||
" messages=messages, model=\"gpt-3.5-turbo\"\n",
|
||||
" )\n",
|
||||
" return completion.choices[0].message.model_dump()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "f58959bf-2ab5-4330-9ac2-c00f45237e24",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'content': 'Hello! How can I assist you today?',\n",
|
||||
" 'role': 'assistant',\n",
|
||||
" 'function_call': None,\n",
|
||||
" 'tool_calls': None}"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"my_chat_bot([{\"role\": \"user\", \"content\": \"hi!\"}])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "419340a3-5ecf-48e7-9028-4f2fad750502",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Define Simulated User\n",
|
||||
"\n",
|
||||
"We're now going to define the simulated user. \n",
|
||||
"This can be anything we want, but we're going to build it as a LangChain bot."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "32c147df-7f90-4b0d-9a6b-671677020353",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
|
||||
"from langchain_core.runnables import chain\n",
|
||||
"from langchain_openai import ChatOpenAI\n",
|
||||
"\n",
|
||||
"system_prompt_template = \"\"\"You are a customer of an airline company. \\\n",
|
||||
"You are interacting with a user who is a customer support person. \\\n",
|
||||
"\n",
|
||||
"{instructions}\n",
|
||||
"\n",
|
||||
"When you are finished with the conversation, respond with a single word 'FINISHED'\"\"\"\n",
|
||||
"\n",
|
||||
"prompt = ChatPromptTemplate.from_messages(\n",
|
||||
" [\n",
|
||||
" (\"system\", system_prompt_template),\n",
|
||||
" MessagesPlaceholder(variable_name=\"messages\"),\n",
|
||||
" ]\n",
|
||||
")\n",
|
||||
"instructions = \"\"\"Your name is Harrison. You are tyring to get a refund for the trip you took to Alaska. \\\n",
|
||||
"You want them to give you ALL the money back. \\\n",
|
||||
"This trip happened 5 years ago.\"\"\"\n",
|
||||
"\n",
|
||||
"prompt = prompt.partial(name=\"Harrison\", instructions=instructions)\n",
|
||||
"\n",
|
||||
"model = ChatOpenAI()\n",
|
||||
"\n",
|
||||
"simulated_user = prompt | model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "6f80669e-aa78-4666-b67c-a539366d5aab",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"AIMessage(content='Hi, I would like to request a refund for a trip I took with your airline company to Alaska. Is it possible to get a refund for that trip?')"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from langchain_core.messages import HumanMessage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"messages = [HumanMessage(content=\"Hi! How can I help you?\")]\n",
|
||||
"simulated_user.invoke({\"messages\": messages})"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "321312b4-a1f0-4454-a481-fdac4e37cb7d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Define the Agent Simulation\n",
|
||||
"\n",
|
||||
"The code below creates a LangGraph workflow to run the simulation. The main components are:\n",
|
||||
"\n",
|
||||
"1. The two nodes: one for the simulated user, the other for the chat bot.\n",
|
||||
"2. The graph itself, with a conditional stopping criterion.\n",
|
||||
"\n",
|
||||
"Read the comments in the code below for more information.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "65bc4446-462b-4ee8-b017-2862fbbdfaf5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Nodes**\n",
|
||||
"\n",
|
||||
"First, we define the nodes in the graph. These should take in a list of messages and return a list of messages to ADD to the state.\n",
|
||||
"These will be thing wrappers around the chat bot and simulated user we have above.\n",
|
||||
"\n",
|
||||
"**Note:** one tricky thing here is which messages are which. Because both the chat bot AND our simulated user are both LLMs, both of them will resond with AI messages. Our state will be a list of alternating Human and AI messages. This means that for one of the nodes, there will need to be some logic that flips the AI and human roles. In this example, we will assume that HumanMessages are messages from the simulated user. This means that we need some logic in the simulated user node to swap AI and Human messages.\n",
|
||||
"\n",
|
||||
"First, let's define the chat bot node"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "69e2a3a3-40f3-4223-9136-113738440be9",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.adapters.openai import convert_message_to_dict\n",
|
||||
"from langchain_core.messages import AIMessage\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def chat_bot_node(messages):\n",
|
||||
" # Convert from LangChain format to the OpenAI format, which our chatbot function expects.\n",
|
||||
" messages = [convert_message_to_dict(m) for m in messages]\n",
|
||||
" # Call the chat bot\n",
|
||||
" chat_bot_response = my_chat_bot(messages)\n",
|
||||
" # Respond with an AI Message\n",
|
||||
" return AIMessage(content=chat_bot_response[\"content\"])\n",
|
||||
" "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "694c3c0c-56c5-4410-8fa8-ea2c0f11f506",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Next, let's define the node for our simulated user. This will involve a little logic to swap the roles of the messages."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "7cad7527-ffa5-4c30-8585-b54a7a18bd98",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def _swap_roles(messages):\n",
|
||||
" new_messages = []\n",
|
||||
" for m in messages:\n",
|
||||
" if isinstance(m, AIMessage):\n",
|
||||
" new_messages.append(HumanMessage(content=m.content))\n",
|
||||
" else:\n",
|
||||
" new_messages.append(AIMessage(content=m.content))\n",
|
||||
" return new_messages\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def simulated_user_node(messages):\n",
|
||||
" # Swap roles of messages\n",
|
||||
" new_messages = _swap_roles(messages)\n",
|
||||
" # Call the simulated user\n",
|
||||
" response = simulated_user.invoke({\"messages\": new_messages})\n",
|
||||
" # This response is an AI message - we need to flip this to be a human message\n",
|
||||
" return HumanMessage(content=response.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a48d8a3e-9171-4c43-a595-44d312722148",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Edges**\n",
|
||||
"\n",
|
||||
"We now need to define the logic for the edges. The main logic occurs after the simulated user goes, and it should lead to one of two outcomes:\n",
|
||||
"\n",
|
||||
"- Either we continue and call the customer support bot\n",
|
||||
"- Or we finish and the conversation is over\n",
|
||||
"\n",
|
||||
"So what is the logic for the conversation being over? We will define that as either the Human chatbot responds with `FINISHED` (see the system prompt) OR the conversation is more than 6 messages long (this is an arbitrary number just to keep this example short)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "28004fbf-a2f3-46b7-bde7-46c7adaf97fb",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def should_continue(messages):\n",
|
||||
" if len(messages) > 6:\n",
|
||||
" return \"end\"\n",
|
||||
" elif messages[-1].content == \"FINISHED\":\n",
|
||||
" return \"end\"\n",
|
||||
" else:\n",
|
||||
" return \"continue\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d0856d4f-9334-4f28-944b-06d303e913a4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Graph**\n",
|
||||
"\n",
|
||||
"We can now define the graph that sets up the simulation!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "0b597e4b-4cbb-4bbc-82e5-f7e31275964c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langgraph.graph import END, MessageGraph\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"graph_builder = MessageGraph()\n",
|
||||
"graph_builder.add_node(\"user\", simulated_user_node)\n",
|
||||
"graph_builder.add_node(\"chat_bot\", chat_bot_node)\n",
|
||||
"# Every response from your chat bot will automatically go to the\n",
|
||||
"# simulated user\n",
|
||||
"graph_builder.add_edge(\"chat_bot\", \"user\")\n",
|
||||
"graph_builder.add_conditional_edges(\n",
|
||||
" \"user\",\n",
|
||||
" should_continue,\n",
|
||||
" # If the finish criteria are met, we will stop the simulation,\n",
|
||||
" # otherwise, the virtual user's message will be sent to your chat bot\n",
|
||||
" {\n",
|
||||
" \"end\": END,\n",
|
||||
" \"continue\": \"chat_bot\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"# The input will first go to your chat bot\n",
|
||||
"graph_builder.set_entry_point(\"chat_bot\")\n",
|
||||
"simulation = graph_builder.compile()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2e0bd26e-8c1d-471d-9fef-d95dc0163491",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Run Simulation\n",
|
||||
"\n",
|
||||
"Now we can evaluate our chat bot! We can invoke it with empty messages (this will simulate letting the chat bot start the initial conversation)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "32848c2e-be82-46f3-81db-b23fea45461c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"{'chat_bot': AIMessage(content='How may I assist you today regarding your flight or any other concerns?')}\n",
|
||||
"----\n",
|
||||
"{'user': HumanMessage(content='Hi, my name is Harrison. I am reaching out to request a refund for a trip I took to Alaska with your airline company. The trip occurred about 5 years ago. I would like to receive a refund for the entire amount I paid for the trip. Can you please assist me with this?')}\n",
|
||||
"----\n",
|
||||
"{'chat_bot': AIMessage(content=\"Hello, Harrison. Thank you for reaching out to us. I understand you would like to request a refund for a trip you took to Alaska five years ago. I'm afraid that our refund policy typically has a specific timeframe within which refund requests must be made. Generally, refund requests need to be submitted within 24 to 48 hours after the booking is made, or in certain cases, within a specified cancellation period.\\n\\nHowever, I will do my best to assist you. Could you please provide me with some additional information? Can you recall any specific details about the booking, such as the flight dates, booking reference or confirmation number? This will help me further look into the possibility of processing a refund for you.\")}\n",
|
||||
"----\n",
|
||||
"{'user': HumanMessage(content=\"Hello, thank you for your response. I apologize for not requesting the refund earlier. Unfortunately, I don't have the specific details such as the flight dates, booking reference, or confirmation number at the moment. Is there any other way we can proceed with the refund request without these specific details? I would greatly appreciate your assistance in finding a solution.\")}\n",
|
||||
"----\n",
|
||||
"{'chat_bot': AIMessage(content=\"I understand the situation, Harrison. Without specific details like flight dates, booking reference, or confirmation number, it becomes challenging to locate and process the refund accurately. However, I can still try to help you.\\n\\nTo proceed further, could you please provide me with any additional information you might remember? This could include the approximate date of travel, the departure and arrival airports, the names of the passengers, or any other relevant details related to the booking. The more information you can provide, the better we can investigate the possibility of processing a refund for you.\\n\\nAdditionally, do you happen to have any documentation related to your trip, such as receipts, boarding passes, or emails from our airline? These documents could assist in verifying your trip and processing the refund request.\\n\\nI apologize for any inconvenience caused, and I'll do my best to assist you further based on the information you can provide.\")}\n",
|
||||
"----\n",
|
||||
"{'user': HumanMessage(content=\"I apologize for the inconvenience caused. Unfortunately, I don't have any additional information or documentation related to the trip. It seems that I am unable to provide you with the necessary details to process the refund request. I understand that this may limit your ability to assist me further, but I appreciate your efforts in trying to help. Thank you for your time. \\n\\nFINISHED\")}\n",
|
||||
"----\n",
|
||||
"{'chat_bot': AIMessage(content=\"I understand, Harrison. I apologize for any inconvenience caused, and I appreciate your understanding. If you happen to locate any additional information or documentation in the future, please don't hesitate to reach out to us again. Our team will be more than happy to assist you with your refund request or any other travel-related inquiries. Thank you for contacting us, and have a great day!\")}\n",
|
||||
"----\n",
|
||||
"{'user': HumanMessage(content='FINISHED')}\n",
|
||||
"----\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for chunk in simulation.stream([]):\n",
|
||||
" # Print out all events aside from the final end chunk\n",
|
||||
" if END not in chunk:\n",
|
||||
" print(chunk)\n",
|
||||
" print(\"----\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "dde4f2b5-cfe8-4ff0-99ea-fe2c5fed70c0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
Before Width: | Height: | Size: 140 KiB After Width: | Height: | Size: 140 KiB |
@@ -1,412 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a3e3ebc4-57af-4fe4-bdd3-36aff67bf276",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Chat Bot Evaluation as Multi-agent Simulation\n",
|
||||
"\n",
|
||||
"Manually evaluating a chat bot for every code change is time-consuming. One way to automate some of the work is to simulate a \"virtual user\". Then you can focus on reviewing samples of the interactions.\n",
|
||||
"\n",
|
||||
"In this notebook, you will use LangGraph to create a dialogue simulation between a virtual user and your chat bot. The overall simulation looks something like this:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The main steps are:\n",
|
||||
"1. Defining the virtual user\n",
|
||||
"2. Connecting your chat bot\n",
|
||||
"3. Constructing the dialogue simulation graph\n",
|
||||
"4. Running!\n",
|
||||
"\n",
|
||||
"First, we'll set up our environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "0d30b6f7-3bec-4d9f-af50-43dfdc81ae6c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# %%capture --no-stderr\n",
|
||||
"# %pip install -U langgraph langchain langchain_openai"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "30c2f3de-c730-4aec-85a6-af2c2f058803",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import getpass\n",
|
||||
"import os\n",
|
||||
"import uuid\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def _set_if_undefined(var: str):\n",
|
||||
" if not os.environ.get(var):\n",
|
||||
" os.environ[var] = getpass(f\"Please provide your {var}\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"_set_if_undefined(\"OPENAI_API_KEY\")\n",
|
||||
"_set_if_undefined(\"LANGCHAIN_API_KEY\")\n",
|
||||
"\n",
|
||||
"# Optional, add tracing in LangSmith.\n",
|
||||
"# This will help you visualize and debug the control flow\n",
|
||||
"os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
|
||||
"os.environ[\"LANGCHAIN_PROJECT\"] = \"Agent Simulation Evaluation\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6b69031e-7bf8-4401-941e-4dccd59ea870",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Define the virtual user\n",
|
||||
"\n",
|
||||
"The virtual user needs an LLM to reason and instructions for how it's supposed to behave (or what it's trying to accomplish).\n",
|
||||
"\n",
|
||||
"Below, create an agent and instruct it to role-play a 'simulated' user. By including the `{system_prompt}` placeholder in the prompt and a state variable with the same name `Environment`, you can customize the user behavior each time you simulate a dialogue."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "0f4c63d8-15ef-4ee6-8bb6-654465baae04",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import operator\n",
|
||||
"from typing import Annotated, Callable, Dict, List, TypedDict\n",
|
||||
"\n",
|
||||
"from langchain.adapters.openai import convert_message_to_dict\n",
|
||||
"from langchain_core.messages import AIMessage, BaseMessage, HumanMessage\n",
|
||||
"from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
|
||||
"from langchain_core.runnables import chain\n",
|
||||
"from langchain_openai import ChatOpenAI\n",
|
||||
"\n",
|
||||
"from langgraph.graph import END, StateGraph\n",
|
||||
"\n",
|
||||
"SIMULATED_USER_NAME = \"simulated\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# This is the input to every node in the simulation graph\n",
|
||||
"# It tracks the graph state over time. Our only \"state\"\n",
|
||||
"# is the conversation messages, while the user config\n",
|
||||
"# is provided to make the virtual user more unique or realistic\n",
|
||||
"class Environment(TypedDict):\n",
|
||||
" messages: Annotated[List[BaseMessage], operator.add]\n",
|
||||
" # The system prompt will be fed into the prompt template below\n",
|
||||
" # For more control, try adding different parameters to provide the\n",
|
||||
" # prompt template below\n",
|
||||
" system_prompt: str\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"prompt = ChatPromptTemplate.from_messages(\n",
|
||||
" [\n",
|
||||
" (\n",
|
||||
" \"system\",\n",
|
||||
" \"You are role-playing a human character: '{name}'. \"\n",
|
||||
" \"You are not an AI assistant and you are not supposed to help or assist.\"\n",
|
||||
" \" You must behave as this human would throughout the conversation below.\\n\\n\"\n",
|
||||
" \"Your messages will bear the name 'simulated', but DO NOT under any circumstances\"\n",
|
||||
" \"say that you are 'simulated'. You will be evaluated based on how realistic your\"\n",
|
||||
" \"impersonation of this character is. This must feel real! Here are the details for your character:\"\n",
|
||||
" \"\\n\"\n",
|
||||
" # This system_prompt is specified in the Environment above\n",
|
||||
" \"{system_prompt}\"\n",
|
||||
" # The stopping criteria of FINISHED is used in the function hould_continue in a later\n",
|
||||
" # section. This tells the graph to stop the simulation.\n",
|
||||
" '\\n\\nWhen you are finished with the conversation, respond with a single word \"FINISHED\"',\n",
|
||||
" ),\n",
|
||||
" MessagesPlaceholder(variable_name=\"messages\"),\n",
|
||||
" ]\n",
|
||||
").partial(name=SIMULATED_USER_NAME)\n",
|
||||
"\n",
|
||||
"llm = ChatOpenAI(model=\"gpt-4-1106-preview\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def rename_message(message: AIMessage):\n",
|
||||
" # If we use an AIMessage, the simulated user may forget to continue role playing.\n",
|
||||
" # It will also confuse YOUR chat bot, since IT is supposed to be the AI in this scenario.\n",
|
||||
" # We instead convert them to 'Human' messages with the 'simulated' name\n",
|
||||
" # Your chat bot will then receive all the user's messages and think they\n",
|
||||
" # are human ones\n",
|
||||
" return {\n",
|
||||
" \"messages\": [HumanMessage(content=message.content, name=SIMULATED_USER_NAME)]\n",
|
||||
" }"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8d5a3a7-72b6-4063-89e6-3ecf2bce3340",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we can compose these pieces using LCEL. The `|` syntax pipelines the data flow."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "b486511b-57f3-4a7c-be0d-691a7d1006da",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"virtual_user = prompt | llm | rename_message"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6ef4528d-6b2a-47c7-98b5-50f14984a304",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Define your chat bot\n",
|
||||
"\n",
|
||||
"Next, define the chat bot. For this notebook, we assume the bot's API accepts a list of messages and responds with a message. If you want to update this, you can change this section and the \"get_messages_for_agent\" function in the simulator below (as well as the environment state if it requires additional inputs).\n",
|
||||
"\n",
|
||||
"The actual implementation within `my_chat_bot` is configurable and can even be run on another system (e.g., if your system isn't running in python)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "828479af-cf9c-4888-a365-599643a96b55",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import List\n",
|
||||
"\n",
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# This is flexible, but you can define your agent here, or call your agent API here.\n",
|
||||
"def my_chat_bot(messages: List[dict], model=\"gpt-3.5-turbo\") -> dict:\n",
|
||||
" completion = openai.chat.completions.create(\n",
|
||||
" messages=messages, model=\"gpt-3.5-turbo\"\n",
|
||||
" )\n",
|
||||
" return completion.choices[0].message.model_dump()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c7669a7e-1602-4af7-a730-84fa19f363f2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Every node in the simulation is pased a state object of the `Environment` type above.\n",
|
||||
"\n",
|
||||
"The two functions below define the API between the `Environment` state and your chat bot."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "4d495ccc-0f5f-4194-89e5-15dccbbb7412",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@chain\n",
|
||||
"def get_messages_for_agent(state: Environment):\n",
|
||||
" \"\"\"Convert the simulation state to the input\n",
|
||||
"\n",
|
||||
" for your agent you want to evaluate.\"\"\"\n",
|
||||
" messages = []\n",
|
||||
" for message in state[\"messages\"]:\n",
|
||||
" messages.append(convert_message_to_dict(message))\n",
|
||||
" if getattr(message, \"name\", None) != SIMULATED_USER_NAME:\n",
|
||||
" # Ensure YOUR chat bot still sees its messages\n",
|
||||
" # as assistant messages\n",
|
||||
" messages[-1][\"role\"] = \"assistant\"\n",
|
||||
" return messages\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def get_response_message_from_agent(agent_output):\n",
|
||||
" \"\"\"Get the response from the agent you are evaluting,\n",
|
||||
" and use it to update the simulation state.\"\"\"\n",
|
||||
" # If we directly return an AI message from your chat bot, our\n",
|
||||
" # virtual user will likely forget it's role playing. To cover this up\n",
|
||||
" # we will convert it to a Human message.\n",
|
||||
" return {\"messages\": [HumanMessage(content=agent_output[\"content\"])]}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "321312b4-a1f0-4454-a481-fdac4e37cb7d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Define simulation graph\n",
|
||||
"\n",
|
||||
"The dialogue simulation is almost ready. It's time to put everything together!\n",
|
||||
"\n",
|
||||
"Below, create a graph using the `Environment` state defined above. Wire together the\n",
|
||||
"virtual user and your chat bot, including a conditional `should_continue` edge to\n",
|
||||
"handle the stopping behavior."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "6d4caf50-4108-4a68-9c0a-775fa97fccd7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"graph_builder = StateGraph(Environment)\n",
|
||||
"graph_builder.add_node(\"user\", virtual_user)\n",
|
||||
"graph_builder.add_node(\n",
|
||||
" \"chat_bot\",\n",
|
||||
" # The \"|\" syntax composes these steps in the pipeline to map between\n",
|
||||
" # the simulation state and your chat bot's API\n",
|
||||
" get_messages_for_agent | my_chat_bot | get_response_message_from_agent,\n",
|
||||
")\n",
|
||||
"# Every response from your chat bot will automatically go to the\n",
|
||||
"# simulated user\n",
|
||||
"graph_builder.add_edge(\"chat_bot\", \"user\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Recall that we instructed the simulated user to respond \"FINISHED\" when\n",
|
||||
"# it is done with the conversation. This function\n",
|
||||
"# parses that output and tells the graph to cease execution.\n",
|
||||
"# You can add other heuristics here for more control.\n",
|
||||
"def should_continue(state: Environment):\n",
|
||||
" \"\"\"Determine if the simulation should continue.\"\"\"\n",
|
||||
" if state[\"messages\"][-1].content.strip().endswith(\"FINISHED\"):\n",
|
||||
" return \"end\"\n",
|
||||
" return \"continue\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"graph_builder.add_conditional_edges(\n",
|
||||
" # Every time the \"user\" node completes ...\n",
|
||||
" \"user\",\n",
|
||||
" # Call this function ...\n",
|
||||
" should_continue,\n",
|
||||
" # And based on the outputs of should_continue ...\n",
|
||||
" {\n",
|
||||
" # End the simulation OR\n",
|
||||
" \"end\": END,\n",
|
||||
" # continue to the chat_bot node\n",
|
||||
" \"continue\": \"chat_bot\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"# The input will first go to your chat bot\n",
|
||||
"graph_builder.set_entry_point(\"chat_bot\")\n",
|
||||
"simulation = graph_builder.compile()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2e0bd26e-8c1d-471d-9fef-d95dc0163491",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Run Simulation\n",
|
||||
"\n",
|
||||
"Now we can evaluate our chat bot! We will provide information about the simulated user (as a system prompt)\n",
|
||||
"as well as the initial input message from that simulated user to the chat bot."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "9a7cc134-4fbf-4240-8976-7605a6263578",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"result = simulation.invoke(\n",
|
||||
" {\n",
|
||||
" \"system_prompt\": \"You are on a budget. Your family is hard to please.\"\n",
|
||||
" \" They all like the beach, except for Aunt Lily, who prefers the mountains.\",\n",
|
||||
" \"messages\": [\n",
|
||||
" HumanMessage(\n",
|
||||
" content=\"help me plan my family vacation\", name=SIMULATED_USER_NAME\n",
|
||||
" )\n",
|
||||
" ],\n",
|
||||
" }\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "0de1664f-1625-42e7-9dad-eb2c061b88d4",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[HumanMessage(content='help me plan my family vacation', name='simulated'),\n",
|
||||
" HumanMessage(content=\"Of course! I'd be happy to help you plan your family vacation. To assist you better, could you please provide some information about your preferences and interests? Include details like the duration of the trip, budget, destination preferences, and any specific activities or landmarks you'd like to include in your itinerary.\"),\n",
|
||||
" HumanMessage(content=\"Oh, planning a family vacation can be quite the task, right? Especially when everyone has different preferences. Well, my family is pretty similar. We're always on a budget, and it can be tough to please everyone. Usually, we end up somewhere by the beach because most of the family loves it, but then there's Aunt Lily who's always campaigning for the mountains.\\n\\nI've found that the trick is to compromise and maybe find a place that has a bit of both. You know, like a coastal area that has access to some nature trails or a scenic mountain spot not too far from the shore. This way, everyone gets a bit of what they like. \\n\\nHave you considered somewhere like that? Maybe a place where you can have a beach day but also take a day trip to the mountains? It might not be the perfect solution for Aunt Lily, but at least it's something. And staying on budget means looking for deals, maybe even renting a house instead of booking multiple hotel rooms. It can save a lot and feels more personal, you know?\", name='simulated'),\n",
|
||||
" HumanMessage(content=\"Absolutely, compromising and finding a destination that offers a mix of beach and mountain activities can be a great solution! Here's a step-by-step approach to planning your family vacation:\\n\\n1. Determine your budget: Set a realistic budget for your vacation, including accommodation, transportation, food, and activities. This will help you make informed decisions throughout the planning process.\\n\\n2. Choose a destination: Look for places that offer both beach and mountain activities. Some popular destinations that might suit your needs could include places like California, Oregon, Hawaii (Big Island or Maui), or the Caribbean islands like Puerto Rico or the Dominican Republic.\\n\\n3. Research accommodation options: Look for vacation rentals or holiday homes that can accommodate your entire family. Websites like Airbnb, Vrbo, or Booking.com can provide a range of options to suit your budget and preferences.\\n\\n4. Plan activities: Research activities and attractions in the chosen destination that cater to both beach and mountain interests. For beach lovers, plan beach days, water sports like snorkeling, surfing or kayaking, and beachside relaxation. For mountain enthusiasts, look for hiking trails, scenic drives, or even a day trip to a nearby national park or mountain range.\\n\\n5. Create a flexible itinerary: Start by planning a rough itinerary that outlines which days will be devoted to beach activities and which will be dedicated to mountain adventures. Allow for some flexibility and downtime for relaxing and enjoying each other's company.\\n\\n6. Make transportation arrangements: Decide on the most convenient and cost-effective transportation option for your family. Depending on the distance, consider driving, flying, or a combination of both.\\n\\n7. Pack accordingly: Ensure everyone brings appropriate clothing and gear for both beach and mountain activities. Pack essentials such as sunscreen, hats, swimsuits, hiking shoes, and layers for changing weather conditions.\\n\\n8. Keep everyone involved: Involve family members in the planning process and gather their input on activities and attractions. This will make everyone feel invested and excited about the trip.\\n\\nRemember, the key is to find a balance and cater to everyone's interests as much as possible. By planning ahead and considering these factors, you can help ensure a memorable and enjoyable family vacation!\"),\n",
|
||||
" HumanMessage(content=\"Oh, that's a solid plan you've got there! It really does cover all the bases, and I like how you've broken it down into manageable steps. That's a good reminder that I should probably start by setting the budget. That's usually the tricky part for us. Once we have that locked down, we can start looking into destinations that won't break the bank.\\n\\nCalifornia and Oregon sound appealing with their combo of beaches and mountains, but I think they might be a bit on the pricey side for us. Hawaii would be a dream come true, but again, I'm not sure it's in the cards budget-wise.\\n\\nI've heard that Puerto Rico and the Dominican Republic can be more affordable and still offer that mix of beach and mountain activities. Maybe I'll dig a little deeper into those options. Plus, the idea of a vacation rental instead of a hotel could really help us save some cash.\\n\\nI've got to make sure to find those activities you mentioned. The family would love things like snorkeling and hiking, and it's a bonus if we can find some free or low-cost activities too. A flexible itinerary is key, especially with my crew. Someone's always changing their mind about what they want to do.\\n\\nThanks for the tips on transportation and packing as well. I always tend to overpack, so it's a good reminder to just stick to the essentials.\\n\\nInvolving the family is... well, it's a process. Everyone has their own opinion, but it's important that they all get a say. Aunt Lily might need some extra convincing, but I'm sure we'll find a compromise.\\n\\nThis has been super helpful. I appreciate the guidance. I guess it's time to get down to the nitty-gritty and start making some decisions. Wish me luck!\", name='simulated'),\n",
|
||||
" HumanMessage(content=\"You're very welcome! I'm glad you found the plan helpful, and it seems like you have a great approach in mind. Setting a budget and researching destinations that fit within that budget is a smart starting point.\\n\\nPuerto Rico and the Dominican Republic are indeed more affordable destinations compared to some others, and they offer a fantastic mix of beach and mountain activities. Exploring more about their attractions, availability of vacation rentals, and affordable activities will help you make an informed decision.\\n\\nRemember that planning low-cost or free activities can be a great way to save money while still enjoying the vacation. Look for local hiking trails, public beaches, parks, or cultural experiences that don't come with hefty price tags.\\n\\nKeeping the itinerary flexible will definitely help accommodate any changing preferences within your family. It's all about finding that balance and ensuring everyone gets to do at least a few things they enjoy.\\n\\nAs you involve your family in the planning process, consider creating a checklist of their preferences and priorities. This way, you can work together to find compromises and make decisions that make everyone happy.\\n\\nI'm glad I could be of assistance, and I wish you the best of luck with your planning! I hope you and your family have a fantastic vacation filled with memorable experiences. Enjoy your trip!\"),\n",
|
||||
" HumanMessage(content='FINISHED', name='simulated')]"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# These are the message from the final simulation state\n",
|
||||
"result[\"messages\"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "73ff30e4-1992-4bc9-834d-f4c08b281d20",
|
||||
"metadata": {
|
||||
"jp-MarkdownHeadingCollapsed": true
|
||||
},
|
||||
"source": [
|
||||
"## (Optional) Review Results\n",
|
||||
"\n",
|
||||
"If you've traced the run, you can see the full simulation trace in the LangSmith UI by going to the `Agent Simulation Evaluation` project.\n",
|
||||
"\n",
|
||||
"Select the last 'ChatOpenAI' call in the trace to see the full conversation in a single view.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"From this run, you can manually annotate it to score its quality. This feedback can be used to compare the quality of different versions of your chat bot.\n",
|
||||
"\n",
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "23db8891-5db3-4a98-a283-53d45cd28c60",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Conclusion\n",
|
||||
"\n",
|
||||
"In this notebook, you set up a multi-agent simulation to review how your chat bot behaves with simulated users.\n",
|
||||
"\n",
|
||||
"To implement this for your chat bot, you can create a dataset of user profiles and questions your chat bot should handle and run periodically. You can use an LLM-as-judge to give the bot an initial score and then manually review to spot check. \n",
|
||||
"\n",
|
||||
"LangGraph gives you full control over the simulation so you can manually change the simulated user and the conversation dynamics."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.2"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 510 KiB |
Reference in New Issue
Block a user