Update flow

examples/multi_agent/agent-simulation-evaluation.ipynb

@@ -7,16 +7,18 @@
    "source": [
     "# Chat Bot Evaluation as Multi-agent Simulation\n",
     "\n",
-    "When building a chat bot, such as a customer support assistant, it can be hard to properly evalute your bot's performance. It's time-consuming to have to manually interact with it intensively for each code change.\n",
+    "Manually evaluating a chat bot on every code change is time-consuming. One way to automate some of the work is to simulate a \"virtual user\". Then you can focus on reviewing samples of the interactions.\n",
     "\n",
-    "One way to make the evaluation process easier and more reproducible is to simulate a user interaction.\n",
-    "\n",
-    "With LangGraph, it's easy to set this up. Below is an example of how to create a \"virtual user\" to simulate a conversation.\n",
-    "\n",
-    "The overall simulation looks something like this:\n",
+    "In thos notebook, you will use LangGraph to create a dialogue simulation between a virtual user and your chat bot. The overall simulation looks something like this:\n",
     "\n",
     "![diagram](./img/virtual_user_diagram.png)\n",
     "\n",
+    "The main steps are:\n",
+    "1. Defining the virtual user\n",
+    "2. Connecting your chat bot\n",
+    "3. Constructing the dialogue simulation graph\n",
+    "4. Running!\n",
+    "\n",
     "First, we'll set up our environment."
    ]
   },
@@ -296,7 +298,7 @@
    "id": "2e0bd26e-8c1d-471d-9fef-d95dc0163491",
    "metadata": {},
    "source": [
-    "## 3. Run Simulation\n",
+    "## 4. Run Simulation\n",
     "\n",
     "Now we can evaluate our chat bot! We will provide information about the simulated user (as a system prompt)\n",
     "as well as the initial input message from that simulated user to the chat bot."
@@ -309,8 +311,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# The tracing context manager lets us easily fetch the trace URL in-context.\n",
-    "# You can turn this off if you don't want to trace the execution.\n",
     "result = simulation.invoke(\n",
     "    {\n",
     "        \"system_prompt\": \"You are on a budget. Your family is hard to please.\"\n",
@@ -324,33 +324,6 @@
     ")"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "fa91e733-3493-43c2-ba21-171cf78deef8",
-   "metadata": {},
-   "outputs": [
-    {
-     "ename": "SyntaxError",
-     "evalue": "invalid syntax (1300180531.py, line 5)",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[0;36m  Cell \u001b[0;32mIn[9], line 5\u001b[0;36m\u001b[0m\n\u001b[0;31m    \"messages\": [\u001b[0m\n\u001b[0m              ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
-     ]
-    }
-   ],
-   "source": [
-    "result = simulation.invoke(\n",
-    "    {\n",
-    "        \"system_prompt\": \"You are on a budget. Your family is hard to please.\"\n",
-    "        \" They all like the beach, except for Aunt Lily, who prefers the mountains.\"\n",
-    "        \"messages\": [\n",
-    "        HumanMessage(content=\"help me plan my family vacation\", name=SIMULATED_USER_NAME)\n",
-    "        ],\n",
-    "    }\n",
-    ")"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 10,
@@ -361,11 +334,11 @@
      "data": {
       "text/plain": [
        "[HumanMessage(content='help me plan my family vacation', name='simulated'),\n",
-       " HumanMessage(content=\"Of course! I would be happy to help you plan your family vacation. Please provide me with some more details:\\n\\n1. Destination: Where are you thinking of going? Do you have any specific preferences or requirements?\\n2. Duration: How long would you like your vacation to be?\\n3. Budget: What is your approximate budget for the trip?\\n4. Number of people: How many people will be traveling with you?\\n5. Interests: What are the interests or activities your family members enjoy?\\n6. Age of children: If you have children, please provide their ages.\\n\\nOnce I have this information, I will be able to provide you with a customized vacation plan that suits your family's needs and preferences.\"),\n",
-       " HumanMessage(content=\"Well, planning a family vacation can be a bit of a balancing act, especially when everyone has different preferences. First off, since we're on a budget, we've got to consider somewhere that's not going to break the bank. And since everyone likes the beach—except Aunt Lily, who's more into the mountains—we might want to look for a place that offers a bit of both.\\n\\nMaybe we could find a coastal area that's near some mountains or hills. That way, most of the family can enjoy the beach while Aunt Lily has the option to explore the mountains or go on a hike. The duration of the trip will probably depend on how much time we can take off work and what our budget can handle.\\n\\nSpeaking of budget, we've got to keep costs in mind for accommodations, food, travel, and any activities we plan to do. Since there are a few of us traveling, maybe renting a house or apartment through a site like Airbnb could be more cost-effective than booking multiple hotel rooms.\\n\\nAs for the number of people, I'd have to get a headcount, but let's assume it's the usual gang. That way, we can start looking into group discounts or family rates for activities and travel.\\n\\nWe also need to think about what kind of activities we want access to. Besides the beach and hiking, do we want to be near a town or city for dining out and entertainment? Or would we prefer a more secluded spot where we can cook our meals and have some quiet time?\\n\\nLastly, we don't have to worry about children's ages for this trip, which simplifies things a bit.\\n\\nSo, what do you think? Is there a place you know of that could fit this mix of interests and constraints?\", name='simulated'),\n",
-       " HumanMessage(content=\"Based on your preferences and requirements, I can suggest a few possible destinations that offer a mix of beach and mountain activities:\\n\\n1. California, USA: Consider areas like Santa Cruz or Half Moon Bay, which offer beautiful coastal landscapes and are near the Santa Cruz Mountains for hiking and exploring.\\n\\n2. Costa Rica: This country has stunning beaches along the Pacific and Caribbean coasts, as well as rainforests and mountains for hiking and wildlife encounters.\\n\\n3. Barcelona, Spain: The city offers a vibrant beachfront, while being in close proximity to the mountains of Montserrat for hiking and scenic views.\\n\\n4. Bali, Indonesia: With its gorgeous beaches and lush, mountainous landscapes, Bali provides a diverse range of activities for everyone.\\n\\n5. Cape Town, South Africa: Known for its stunning coastal scenery, Cape Town also offers Table Mountain and nearby hiking trails for exploring the mountains.\\n\\n6. Thailand: Places like Phuket or Krabi provide beautiful beaches for relaxation and activities, while being close to mountainous regions like Khao Sok National Park or Elephant Hills.\\n\\nOnce you've chosen a destination, I can help you with specific recommendations for accommodations, activities, and budgeting. Let me know which option interests you the most, or if you have any other preferences!\"),\n",
-       " HumanMessage(content=\"Oh, those are some fantastic suggestions! But you know, going international might be a stretch for our budget. California sounds like a good middle ground, though. I've heard Santa Cruz has some nice beaches, and being close to the mountains could be perfect for Aunt Lily.\\n\\nI'm thinking we could make it a road trip if it's within a reasonable distance. That way, we could save on flights and have the flexibility to explore. We'd just have to work out the logistics of car rentals and gas prices, but that could be part of the adventure.\\n\\nRenting a house or a larger apartment might work out well for us, especially if we can find a place with a kitchen. It'll save us a lot on eating out, and I know a couple of us enjoy cooking, so it could be fun to prepare meals together.\\n\\nFor activities, it's a mix. Some will want to lounge on the beach, others might want to try surfing or stand-up paddleboarding, and I'm sure Aunt Lily will want to hit the trails. It'll be a challenge to schedule everything, but maybe we can have some group activities and also allow for some time when everyone can do their own thing.\\n\\nSo, I think I'll start looking into Santa Cruz and see what kind of deals I can find. I'll have to get everyone's input, of course, but it's a solid starting point. Thanks for the brainstorming help!\", name='simulated'),\n",
-       " HumanMessage(content=\"You're very welcome! Santa Cruz sounds like a great choice for a family road trip, combining the beach and mountain activities you're looking for. Renting a house or larger apartment will give you the flexibility and cost-saving advantages you mentioned.\\n\\nWhen it comes to activities, Santa Cruz offers a range of options to suit everyone's interests. The beach is perfect for relaxing, swimming, and trying out water sports like surfing or stand-up paddleboarding. For Aunt Lily, there are beautiful hiking trails in nearby Santa Cruz Mountains, such as Henry Cowell Redwoods State Park or Big Basin Redwoods State Park.\\n\\nDo check for any specific guidelines or restrictions in the area regarding COVID-19 before finalizing your plans.\\n\\nTake your time researching accommodations, car rentals, and budget-friendly deals in Santa Cruz. It's always great to involve everyone in the decision-making process to ensure a memorable and enjoyable vacation for the whole family.\\n\\nIf you need any further assistance or information during your planning process, feel free to reach out. Have a fantastic family vacation in Santa Cruz!\"),\n",
+       " HumanMessage(content=\"Of course! I'd be happy to help you plan your family vacation. To assist you better, could you please provide some information about your preferences and interests? Include details like the duration of the trip, budget, destination preferences, and any specific activities or landmarks you'd like to include in your itinerary.\"),\n",
+       " HumanMessage(content=\"Oh, planning a family vacation can be quite the task, right? Especially when everyone has different preferences. Well, my family is pretty similar. We're always on a budget, and it can be tough to please everyone. Usually, we end up somewhere by the beach because most of the family loves it, but then there's Aunt Lily who's always campaigning for the mountains.\\n\\nI've found that the trick is to compromise and maybe find a place that has a bit of both. You know, like a coastal area that has access to some nature trails or a scenic mountain spot not too far from the shore. This way, everyone gets a bit of what they like. \\n\\nHave you considered somewhere like that? Maybe a place where you can have a beach day but also take a day trip to the mountains? It might not be the perfect solution for Aunt Lily, but at least it's something. And staying on budget means looking for deals, maybe even renting a house instead of booking multiple hotel rooms. It can save a lot and feels more personal, you know?\", name='simulated'),\n",
+       " HumanMessage(content=\"Absolutely, compromising and finding a destination that offers a mix of beach and mountain activities can be a great solution! Here's a step-by-step approach to planning your family vacation:\\n\\n1. Determine your budget: Set a realistic budget for your vacation, including accommodation, transportation, food, and activities. This will help you make informed decisions throughout the planning process.\\n\\n2. Choose a destination: Look for places that offer both beach and mountain activities. Some popular destinations that might suit your needs could include places like California, Oregon, Hawaii (Big Island or Maui), or the Caribbean islands like Puerto Rico or the Dominican Republic.\\n\\n3. Research accommodation options: Look for vacation rentals or holiday homes that can accommodate your entire family. Websites like Airbnb, Vrbo, or Booking.com can provide a range of options to suit your budget and preferences.\\n\\n4. Plan activities: Research activities and attractions in the chosen destination that cater to both beach and mountain interests. For beach lovers, plan beach days, water sports like snorkeling, surfing or kayaking, and beachside relaxation. For mountain enthusiasts, look for hiking trails, scenic drives, or even a day trip to a nearby national park or mountain range.\\n\\n5. Create a flexible itinerary: Start by planning a rough itinerary that outlines which days will be devoted to beach activities and which will be dedicated to mountain adventures. Allow for some flexibility and downtime for relaxing and enjoying each other's company.\\n\\n6. Make transportation arrangements: Decide on the most convenient and cost-effective transportation option for your family. Depending on the distance, consider driving, flying, or a combination of both.\\n\\n7. Pack accordingly: Ensure everyone brings appropriate clothing and gear for both beach and mountain activities. Pack essentials such as sunscreen, hats, swimsuits, hiking shoes, and layers for changing weather conditions.\\n\\n8. Keep everyone involved: Involve family members in the planning process and gather their input on activities and attractions. This will make everyone feel invested and excited about the trip.\\n\\nRemember, the key is to find a balance and cater to everyone's interests as much as possible. By planning ahead and considering these factors, you can help ensure a memorable and enjoyable family vacation!\"),\n",
+       " HumanMessage(content=\"Oh, that's a solid plan you've got there! It really does cover all the bases, and I like how you've broken it down into manageable steps. That's a good reminder that I should probably start by setting the budget. That's usually the tricky part for us. Once we have that locked down, we can start looking into destinations that won't break the bank.\\n\\nCalifornia and Oregon sound appealing with their combo of beaches and mountains, but I think they might be a bit on the pricey side for us. Hawaii would be a dream come true, but again, I'm not sure it's in the cards budget-wise.\\n\\nI've heard that Puerto Rico and the Dominican Republic can be more affordable and still offer that mix of beach and mountain activities. Maybe I'll dig a little deeper into those options. Plus, the idea of a vacation rental instead of a hotel could really help us save some cash.\\n\\nI've got to make sure to find those activities you mentioned. The family would love things like snorkeling and hiking, and it's a bonus if we can find some free or low-cost activities too. A flexible itinerary is key, especially with my crew. Someone's always changing their mind about what they want to do.\\n\\nThanks for the tips on transportation and packing as well. I always tend to overpack, so it's a good reminder to just stick to the essentials.\\n\\nInvolving the family is... well, it's a process. Everyone has their own opinion, but it's important that they all get a say. Aunt Lily might need some extra convincing, but I'm sure we'll find a compromise.\\n\\nThis has been super helpful. I appreciate the guidance. I guess it's time to get down to the nitty-gritty and start making some decisions. Wish me luck!\", name='simulated'),\n",
+       " HumanMessage(content=\"You're very welcome! I'm glad you found the plan helpful, and it seems like you have a great approach in mind. Setting a budget and researching destinations that fit within that budget is a smart starting point.\\n\\nPuerto Rico and the Dominican Republic are indeed more affordable destinations compared to some others, and they offer a fantastic mix of beach and mountain activities. Exploring more about their attractions, availability of vacation rentals, and affordable activities will help you make an informed decision.\\n\\nRemember that planning low-cost or free activities can be a great way to save money while still enjoying the vacation. Look for local hiking trails, public beaches, parks, or cultural experiences that don't come with hefty price tags.\\n\\nKeeping the itinerary flexible will definitely help accommodate any changing preferences within your family. It's all about finding that balance and ensuring everyone gets to do at least a few things they enjoy.\\n\\nAs you involve your family in the planning process, consider creating a checklist of their preferences and priorities. This way, you can work together to find compromises and make decisions that make everyone happy.\\n\\nI'm glad I could be of assistance, and I wish you the best of luck with your planning! I hope you and your family have a fantastic vacation filled with memorable experiences. Enjoy your trip!\"),\n",
        " HumanMessage(content='FINISHED', name='simulated')]"
       ]
      },