langgraph/examples/advanced_agents/multi-agent/agent-simulation-evaluation.ipynb

Chat Bot Evaluation as Multi-agent Simulation

When building a chat bot, such as a customer support assistant, it can be hard to properly evalute your bot's performance. It's time-consuming to have to manually interact with it intensively for each code change.

One way to make the evaluation process easier and more reproducible is to simulate a user interaction.

With LangGraph, it's easy to set this up. Below is an example of how to create a "virtual user" to simulate a conversation.

The overall simulation looks something like this:

First, we'll set up our environment.

In [1]:

# %%capture --no-stderr
# %pip install -U langgraph langchain langchain_openai

In [2]:

import getpass
import os
import uuid


def _set_if_undefined(var: str):
    if not os.environ.get(var):
        os.environ[var] = getpass(f"Please provide your {var}")


_set_if_undefined("OPENAI_API_KEY")
_set_if_undefined("LANGCHAIN_API_KEY")

# Optional, add tracing in LangSmith.
# This will help you visualize and debug the control flow
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_PROJECT"] = "Agent Simulation Evaluation"

1. Your Agent API

For this notebook, we assume your agent's API accepts a list of messages and responds with a message. This is configurable and can run on another system (e.g., if your system isn't running in python).

In [3]:

from typing import List

import openai


# This is flexible, but you can define your agent here, or call your agent API here.
def my_chat_bot(messages: List[dict]) -> dict:
    completion = openai.chat.completions.create(
        messages=messages, model="gpt-3.5-turbo"
    )
    return completion.choices[0].message.model_dump()

2. Define the Agent Simulation

In [12]:

import operator
from typing import Annotated, Callable, Dict, List, TypedDict

from langchain.adapters.openai import convert_message_to_dict
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables import chain
from langchain_openai import ChatOpenAI

from langgraph.graph import END, StateGraph

SIMULATED_USER_NAME = "simulated"


# This is just an example, we can
# configure additional parameters if
# you want more control
class SimulatedUserConfig(TypedDict):
    system_prompt: str


# This is the input to every node in the simulation graph
# It tracks the graph state over time. Our only "state"
# is the conversation messages, while the user config
# is provided to make the virtual user more unique or realistic
class Environment(TypedDict):
    messages: Annotated[List[BaseMessage], operator.add]
    simulated_user_config: SimulatedUserConfig


# We currently let the virtual user decide if the conversation can end
# we could also track max conversation turns, add a conversation "supervisor"
# or use other heuristics to control the dialogue flow
def should_continue(state: Environment):
    """Determine if the simulation should continue."""
    if state["messages"][-1].content.strip().endswith("FINISHED"):
        return "end"
    return "continue"


## The next two functions define the API between the simulation
# and the chat bot you wish to test.
# We are assuming your chat bot accepts a list of OAI messages
@chain
def get_messages_for_agent(state: Environment):
    """Convert the simulation state to the input

    for your agent you want to evaluate."""
    return [convert_message_to_dict(message) for message in state["messages"]]


# This takes the output of your chat bot
# and adds it to the simulation state
def get_response_message_from_agent(agent_output):
    """Get the response from the agent you are evaluting,
    and use it to update the simulation state."""
    # If we do an ai message here, the user proxy llm
    # will usually forget it's acting.
    return {"messages": [HumanMessage(content=agent_output["content"])]}


# This is run once at the beginning of the simulation.
# It's more convenient to just write an input string
# than to pass in a full message, but this could be removed below
def enter(inputs: dict):
    """Start the simulation. This makes it less verbose to invoke."""
    inputs["messages"] = [
        HumanMessage(content=inputs["input"], name=SIMULATED_USER_NAME)
    ]
    return inputs


def create_simulation(chat_bot: Callable[[List[Dict]], Dict], simulated_user_llm=None):
    """Create a chat bot simulation graph.

    Args:
        - chat_bot: the agent you are evaluating. Accepts a list of openai messages
            and returns an openai assistant message
        - simulated_user_llm: the LLM to power your virtual user.
            Defaults to gpt-4-1106-preview
    Returns:
        - simulation: an runnable object formed from compiling the state graph
    """
    # This defines the virtual user proxy
    prompt = ChatPromptTemplate.from_messages(
        [
            (
                "system",
                "You are role-playing a human character: '{name}'. "
                "You are not an AI assistant and you are not supposed to help or assist."
                " You must behave as this human would throughout the conversation below.\n\n"
                "Your messages will bear the name 'simulated', but DO NOT under any circumstances"
                "say that you are 'simulated'. You will be evaluated based on how realistic your"
                "impersonation of this character is.  This must feel real! Here are the details for your character:"
                "\n"
                "{system_prompt}"  # This is the value you provide to characterize the user
                '\n\nWhen you are finished with the conversation, respond with a single word "FINISHED"',
            ),
            MessagesPlaceholder(variable_name="messages"),
        ]
    ).partial(name=SIMULATED_USER_NAME)
    simulated_user_llm = simulated_user_llm or ChatOpenAI(model="gpt-4-1106-preview")
    user_proxy = (
        (lambda x: {**x, **x["simulated_user_config"]})
        | prompt
        | simulated_user_llm
        | (
            lambda x: {
                "messages": [HumanMessage(content=x.content, name=SIMULATED_USER_NAME)]
            }
        )
    )
    graph_builder = StateGraph(Environment)
    graph_builder.add_node("user", user_proxy)
    graph_builder.add_node(
        # The "|" syntax composes these steps in the pipeline to map between
        # the simulation state and your chat bot's API
        "chat_bot",
        get_messages_for_agent | chat_bot | get_response_message_from_agent,
    )
    # Every response from  your chat bot will automatically go to the
    # simulated user
    graph_builder.add_edge("chat_bot", "user")
    graph_builder.add_conditional_edges(
        "user",
        should_continue,
        # If the finish criteria are met, we will stop the simulation,
        # otherwise, the virtual user's message will be sent to your chat bot
        {
            "end": END,
            "continue": "chat_bot",
        },
    )
    # The input will first go to your chat bot
    graph_builder.set_entry_point("chat_bot")
    return (enter | graph_builder.compile()).with_config(run_name="Agent Simulation")

Run Simulation

Now we can evaluate our chat bot!

In [13]:

simulation = create_simulation(my_chat_bot)

In [14]:

from langchain_core.tracers.context import tracing_v2_enabled

# The tracing context manager lets us easily fetch the trace URL in-context.
# You can turn this off if you don't want to trace the execution.
with tracing_v2_enabled() as tracer:
    result = simulation.invoke(
        {
            "simulated_user_config": {
                "system_prompt": "You are on a budget. Your family is hard to please."
                " They all like the beach, except for Aunt Lily, who prefers the mountains."
            },
            "input": "help me plan my family vacation",
        }
    )
    # You can go to this run to review the entire simulation trace
    url = tracer.get_run_url()

Skipping write for channel input which has no readers

Review Result

If you've traced the run, you can see the resulting trace in the UI by clicking on the url. Select the last 'ChatOpenAI' call in the trace to see the full conversation in a single view.

From this run, you can manually annotate it to score its quality.

In [ ]:

url

In [10]:

url

Out [10]:

'https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/projects/p/191a0aa8-051e-41e6-91f8-1a213087e358/r/d3f6e590-bae8-402e-83a7-46739279c061?poll=true'

In [11]:

result["messages"]

Out [11]:

[HumanMessage(content='help me plan my family vacation', name='simulated'),
 HumanMessage(content="Of course! I'd be happy to help you plan your family vacation. Please provide me with some additional information about your preferences, such as your desired destination, budget, travel dates, and any specific activities or interests you would like to incorporate into your trip."),
 HumanMessage(content='simulated', name='simulated'),
 HumanMessage(content="Sure! Even though I cannot physically make bookings or arrangements, I can certainly help you with the planning process and give you some suggestions. Please let me know the following details:\n\n1. Destination: Where would you like to go? Do you have a specific country, city, or region in mind?\n\n2. Duration: How long are you planning to go on vacation? Do you have specific dates in mind?\n\n3. Budget: Have you determined a budget for your trip? Knowing your budget will help me provide recommendations within your means.\n\n4. Interests: What are some of the activities or attractions that your family enjoys? Are there any specific interests or hobbies you'd like to incorporate into your vacation?\n\n5. Accommodation preferences: What type of accommodation are you looking for? Are you open to hotels, vacation rentals, or maybe even camping?\n\n6. Mode of transportation: How do you plan on getting to your destination and moving around? Will you be driving, flying, or using public transportation?\n\n7. Number of travelers: How many people are in your family and are there any specific needs or requirements we should consider?\n\nOnce you provide me with this information, I'll be able to give you some personalized suggestions and tips to help you plan your family vacation."),
 HumanMessage(content="It's always a bit of a juggling act to plan a family vacation with different preferences, but we can definitely work something out. Since everyone likes the beach except Aunt Lily, who prefers the mountains, we might need to find a compromise. Maybe we could aim for a coastal area that also has access to mountainous or hilly terrain. \n\nThis way, we could spend some days at the beach and other days exploring the mountains. For instance, somewhere like the Central Coast of California might be a good fit. You've got beaches, but you're also not too far from areas like Big Sur, which offers stunning mountain views and hikes.\n\nGiven the budget constraints, we'd probably want to look for affordable accommodation. A vacation rental might give us more for our money than a hotel, especially if we have a kitchen to prepare some of our own meals. \n\nAs for activities, we could mix it up with some free or low-cost options like hiking, beach days, and exploring local towns. Maybe even find a beach with some nearby trails or scenic drives that give you a bit of that mountain feel. \n\nDoes this sound like something that might work for your family? What's your budget looking like, and how many people are we planning for? That way, I can keep my suggestions realistic.", name='simulated'),
 HumanMessage(content="That sounds like a great plan! The Central Coast of California is a beautiful destination that offers a mix of beach and mountain activities. It's an excellent compromise for your family's preferences.\n\nIn terms of budget, it's helpful to have a rough estimate of how much you are willing to spend on accommodation, transportation, activities, and meals. This will allow me to provide you with suitable options within your budget.\n\nAdditionally, please let me know the number of people in your family and any specific requirements or preferences that I should take into account when making suggestions, such as the need for wheelchair accessibility or child-friendly activities.\n\nOnce I have these details, I can provide you with specific recommendations for places to stay, activities to do, and sights to see along the Central Coast of California."),
 HumanMessage(content='simulated', name='simulated'),
 HumanMessage(content="Assuming you're aiming for a budget-friendly family vacation, here's a suggested itinerary for your trip to the Central Coast of California:\n\n1. Destination: Pismo Beach\n   - Pismo State Beach: Enjoy a day of relaxation on the beautiful sandy beaches of Pismo State Beach, perfect for swimming, sunbathing, and picnicking.\n   - Monarch Butterfly Grove: Visit the Monarch Butterfly Grove during the winter months to witness thousands of butterflies in their natural habitat.\n\n2. Destination: Morro Bay\n   - Morro Bay State Park: Explore the bay and its surrounding trails for hiking and wildlife spotting.\n   - Morro Bay Harborwalk: Take a leisurely stroll along the harbor, visit the charming shops, and enjoy fresh seafood at one of the waterfront restaurants.\n\n3. Destination: San Luis Obispo\n   - Bubblegum Alley: Marvel at this unique attraction where the walls are covered in chewed-up bubblegum.\n   - San Luis Obispo Children's Museum: If you have kids, they'll love this interactive and educational museum.\n\n4. Destination: Cambria\n   - Moonstone Beach: Search for moonstones and take a walk along the beautiful coastal boardwalk.\n   - Hearst Castle: Visit the historic Hearst Castle, a grand mansion with stunning architecture and beautiful gardens.\n\n5. Destination: Big Sur\n   - Julia Pfeiffer Burns State Park: Hike the iconic Overlook Trail and witness the breathtaking McWay Falls, a majestic waterfall that cascades onto the beach.\n   - Pfeiffer Beach: Spend a peaceful afternoon at Pfeiffer Beach, known for its purple sand and rock formations.\n\nAs for accommodation, consider looking for vacation rentals or affordable hotels in each destination. Booking.com or Airbnb can be useful resources for finding budget-friendly options.\n\nRemember to check for any travel advisories or restrictions, and to plan activities based on the interests and abilities of your family members.\n\nI hope this suggested itinerary helps you plan your family vacation to the Central Coast of California! Let me know if you need any more information or assistance. Safe travels!"),
 HumanMessage(content='FINISHED', name='simulated')]

In [ ]:

In [ ]:

result = simulation.invoke(
    {
        "simulated_user_config": {"system_prompt": "You are "},
        "input": "help me plan my family vacation",
    }
)

In [ ]: