diff --git a/examples/advanced_agents/multi-agent/agent-simulation-evaluation.ipynb b/examples/advanced_agents/multi-agent/agent-simulation-evaluation.ipynb index ff65ad927..948dc5e78 100644 --- a/examples/advanced_agents/multi-agent/agent-simulation-evaluation.ipynb +++ b/examples/advanced_agents/multi-agent/agent-simulation-evaluation.ipynb @@ -324,28 +324,7 @@ }, { "cell_type": "code", - "execution_count": 10, - "id": "c1037dd1-7e2e-4158-b248-bc7a8839e62e", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/projects/p/191a0aa8-051e-41e6-91f8-1a213087e358/r/d3f6e590-bae8-402e-83a7-46739279c061?poll=true'" - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "url" - ] - }, - { - "cell_type": "code", - "execution_count": 11, + "execution_count": 15, "id": "0de1664f-1625-42e7-9dad-eb2c061b88d4", "metadata": {}, "outputs": [ @@ -353,17 +332,15 @@ "data": { "text/plain": [ "[HumanMessage(content='help me plan my family vacation', name='simulated'),\n", - " HumanMessage(content=\"Of course! I'd be happy to help you plan your family vacation. Please provide me with some additional information about your preferences, such as your desired destination, budget, travel dates, and any specific activities or interests you would like to incorporate into your trip.\"),\n", - " HumanMessage(content='simulated', name='simulated'),\n", - " HumanMessage(content=\"Sure! Even though I cannot physically make bookings or arrangements, I can certainly help you with the planning process and give you some suggestions. Please let me know the following details:\\n\\n1. Destination: Where would you like to go? Do you have a specific country, city, or region in mind?\\n\\n2. Duration: How long are you planning to go on vacation? Do you have specific dates in mind?\\n\\n3. Budget: Have you determined a budget for your trip? Knowing your budget will help me provide recommendations within your means.\\n\\n4. Interests: What are some of the activities or attractions that your family enjoys? Are there any specific interests or hobbies you'd like to incorporate into your vacation?\\n\\n5. Accommodation preferences: What type of accommodation are you looking for? Are you open to hotels, vacation rentals, or maybe even camping?\\n\\n6. Mode of transportation: How do you plan on getting to your destination and moving around? Will you be driving, flying, or using public transportation?\\n\\n7. Number of travelers: How many people are in your family and are there any specific needs or requirements we should consider?\\n\\nOnce you provide me with this information, I'll be able to give you some personalized suggestions and tips to help you plan your family vacation.\"),\n", - " HumanMessage(content=\"It's always a bit of a juggling act to plan a family vacation with different preferences, but we can definitely work something out. Since everyone likes the beach except Aunt Lily, who prefers the mountains, we might need to find a compromise. Maybe we could aim for a coastal area that also has access to mountainous or hilly terrain. \\n\\nThis way, we could spend some days at the beach and other days exploring the mountains. For instance, somewhere like the Central Coast of California might be a good fit. You've got beaches, but you're also not too far from areas like Big Sur, which offers stunning mountain views and hikes.\\n\\nGiven the budget constraints, we'd probably want to look for affordable accommodation. A vacation rental might give us more for our money than a hotel, especially if we have a kitchen to prepare some of our own meals. \\n\\nAs for activities, we could mix it up with some free or low-cost options like hiking, beach days, and exploring local towns. Maybe even find a beach with some nearby trails or scenic drives that give you a bit of that mountain feel. \\n\\nDoes this sound like something that might work for your family? What's your budget looking like, and how many people are we planning for? That way, I can keep my suggestions realistic.\", name='simulated'),\n", - " HumanMessage(content=\"That sounds like a great plan! The Central Coast of California is a beautiful destination that offers a mix of beach and mountain activities. It's an excellent compromise for your family's preferences.\\n\\nIn terms of budget, it's helpful to have a rough estimate of how much you are willing to spend on accommodation, transportation, activities, and meals. This will allow me to provide you with suitable options within your budget.\\n\\nAdditionally, please let me know the number of people in your family and any specific requirements or preferences that I should take into account when making suggestions, such as the need for wheelchair accessibility or child-friendly activities.\\n\\nOnce I have these details, I can provide you with specific recommendations for places to stay, activities to do, and sights to see along the Central Coast of California.\"),\n", - " HumanMessage(content='simulated', name='simulated'),\n", - " HumanMessage(content=\"Assuming you're aiming for a budget-friendly family vacation, here's a suggested itinerary for your trip to the Central Coast of California:\\n\\n1. Destination: Pismo Beach\\n - Pismo State Beach: Enjoy a day of relaxation on the beautiful sandy beaches of Pismo State Beach, perfect for swimming, sunbathing, and picnicking.\\n - Monarch Butterfly Grove: Visit the Monarch Butterfly Grove during the winter months to witness thousands of butterflies in their natural habitat.\\n\\n2. Destination: Morro Bay\\n - Morro Bay State Park: Explore the bay and its surrounding trails for hiking and wildlife spotting.\\n - Morro Bay Harborwalk: Take a leisurely stroll along the harbor, visit the charming shops, and enjoy fresh seafood at one of the waterfront restaurants.\\n\\n3. Destination: San Luis Obispo\\n - Bubblegum Alley: Marvel at this unique attraction where the walls are covered in chewed-up bubblegum.\\n - San Luis Obispo Children's Museum: If you have kids, they'll love this interactive and educational museum.\\n\\n4. Destination: Cambria\\n - Moonstone Beach: Search for moonstones and take a walk along the beautiful coastal boardwalk.\\n - Hearst Castle: Visit the historic Hearst Castle, a grand mansion with stunning architecture and beautiful gardens.\\n\\n5. Destination: Big Sur\\n - Julia Pfeiffer Burns State Park: Hike the iconic Overlook Trail and witness the breathtaking McWay Falls, a majestic waterfall that cascades onto the beach.\\n - Pfeiffer Beach: Spend a peaceful afternoon at Pfeiffer Beach, known for its purple sand and rock formations.\\n\\nAs for accommodation, consider looking for vacation rentals or affordable hotels in each destination. Booking.com or Airbnb can be useful resources for finding budget-friendly options.\\n\\nRemember to check for any travel advisories or restrictions, and to plan activities based on the interests and abilities of your family members.\\n\\nI hope this suggested itinerary helps you plan your family vacation to the Central Coast of California! Let me know if you need any more information or assistance. Safe travels!\"),\n", + " HumanMessage(content=\"Sure! I'd be happy to help you plan your family vacation. Can you provide more details about your preferences, such as the destination, budget, duration of the trip, and any specific activities or attractions you have in mind?\"),\n", + " HumanMessage(content=\"Oh, planning family vacations is always a bit of a juggling act, isn't it? We've got a variety of tastes in my family too, so I totally get where you're coming from. We're on a budget, so we usually look for places that won't break the bank. Everyone loves the beach—it's just Aunt Lily who's the odd one out, preferring the mountains.\\n\\nHere's a thought, maybe you can find a coastal area that's near some mountains? That way, the majority of the family gets to enjoy the sand and surf while Aunt Lily isn't too far from a mountain getaway. Depending on where you live, there might be some places not too far away that offer both. \\n\\nFor instance, places like the Central Coast of California have beaches and they're not too far from mountains. Or you could look into a spot like the South of France, if you're up for international travel and can find some deals. I've also heard that places like Costa Rica have both, but I've never been there myself.\\n\\nAs for the budget, I'm always on the lookout for off-season deals or vacation rentals that can accommodate the whole family. It can be way more cost-effective than booking multiple hotel rooms, and you can save a bit by cooking meals at the rental rather than eating out all the time.\\n\\nHave you thought about any specific destinations yet?\", name='simulated'),\n", + " HumanMessage(content=\"Those are great suggestions! Finding a destination that offers both beach and mountain options can be a great compromise for your family. Here are a few more specific destination ideas that might fit your preferences:\\n\\n1. The Oregon Coast, USA: Known for its stunning coastline and nearby mountain ranges like the Cascade Range, the Oregon Coast offers a mix of beautiful beaches, charming coastal towns, and opportunities for hiking in the mountains.\\n\\n2. Bali, Indonesia: This tropical island destination offers gorgeous beaches as well as volcanic mountains like Mount Batur. You can relax on the beach, explore temples, try water sports, and even trek through rice terraces and lush forests.\\n\\n3. Split, Croatia: Located on the stunning Dalmatian Coast, Split offers a mix of beach relaxation and nearby mountain hiking opportunities in places like the Biokovo nature park. Plus, you can explore the historic Old Town and nearby islands like Hvar.\\n\\n4. Cape Town, South Africa: With its iconic Table Mountain and beautiful Atlantic beaches like Camps Bay, Cape Town provides the best of both worlds. You can take a cable car up Table Mountain, visit the penguins at Boulders Beach, and even go on a wine tour in the nearby Cape Winelands.\\n\\nWhen it comes to budget-friendly options, consider booking vacation rentals, researching affordable or all-inclusive resorts, and keeping an eye out for discounts on flights and attractions. It's also advisable to be flexible with your travel dates, as traveling during the offseason can often result in more affordable prices.\\n\\nLet me know if you need any more information or help with planning specific activities or accommodations in any of these destinations!\"),\n", + " HumanMessage(content=\"Oh, those are some fantastic ideas, really! Each of those spots has something unique to offer. I'll definitely have to look into the Oregon Coast. It has that rugged charm, and I've heard it's not as pricey as California. Bali sounds like a dream, honestly, but I have to admit, international travel might be a bit much for the budget this time around. \\n\\nCroatia is one of those places I've always wanted to visit, with all that beautiful coastline and history, but again, might be a stretch budget-wise. Cape Town would be an adventure for sure, but South Africa is a big trip. It's probably out of our range for now.\\n\\nI really appreciate the suggestions about being flexible with travel dates and looking at vacation rentals. That's the kind of approach we usually take. We try to avoid the peak seasons to save some money and find those hidden deals.\\n\\nIt sounds like you've done a fair bit of traveling yourself, or you're just really good at sniffing out the cool spots to visit. Do you travel a lot?\", name='simulated'),\n", + " HumanMessage(content=\"I'm glad you found the suggestions helpful! The Oregon Coast is definitely a more budget-friendly option compared to some other coastal destinations. It offers stunning landscapes, charming towns, and the opportunity to explore both the beach and the mountains.\\n\\nBali is indeed a dream destination, but it's understandable that international travel might not fit within the budget this time. It's always good to keep it in mind for future trips though, as it offers a unique cultural experience along with beautiful beaches and mountains.\\n\\nCroatia is known for its stunning coastline and historic cities like Split and Dubrovnik, but it can sometimes be on the more expensive side. It's always worth checking for deals and considering different accommodation options to make it more affordable.\\n\\nAnd yes, South Africa and Cape Town are definitely big trips. If it's not feasible for the current vacation, you can always keep it on your bucket list for future adventures when the budget allows.\\n\\nAs for your question, I do love to travel and explore different places whenever I get the chance. I'm also always researching and learning about new destinations to be able to offer suggestions and help others plan their trips. It's a passion of mine! If you ever need more help or have any specific questions about the destinations or planning, feel free to ask.\"),\n", " HumanMessage(content='FINISHED', name='simulated')]" ] }, - "execution_count": 11, + "execution_count": 15, "metadata": {}, "output_type": "execute_result" } @@ -373,26 +350,17 @@ ] }, { - "cell_type": "code", - "execution_count": null, - "id": "e871cb1f-b3d4-44d7-83a8-c61f65898568", + "cell_type": "markdown", + "id": "23db8891-5db3-4a98-a283-53d45cd28c60", "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "a046369d-bce9-4215-9d1a-e5522d586916", - "metadata": {}, - "outputs": [], "source": [ - "result = simulation.invoke(\n", - " {\n", - " \"simulated_user_config\": {\"system_prompt\": \"You are \"},\n", - " \"input\": \"help me plan my family vacation\",\n", - " }\n", - ")" + "## Conclusion\n", + "\n", + "In this notebook, you set up a multi-agent simulation to review how your chat bot behaves with simulated users.\n", + "\n", + "To implement this for your chat bot, you can create a dataset of user profiles and questions your chat bot should handle and run periodically. You can use an LLM-as-judge to give the bot an initial score and then manually review to spot check. \n", + "\n", + "LangGraph gives you full control over the simulation so you can manually change the simulated user and the conversation dynamics." ] }, {