{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1a7d688-561c-4175-acfc-a6537f6dd042",
   "metadata": {},
   "source": [
    "## Web Voyager\n",
    "\n",
    "[WebVoyager](https://arxiv.org/abs/2401.13919) by He, et. al., is a vision-enabled web-browsing agent capable of controlling the mouse and keyboard.\n",
    "\n",
    "It works by viewing annotated browser screenshots for each turn, then choosing the next step to take. The agent architecture is a basic reasoning and action (ReAct) loop. \n",
    "The unique aspects of this agent are:\n",
    "- It's usage of [Set-of-Marks](https://som-gpt4v.github.io/)-like image annotations to serve as UI affordances for the agent\n",
    "- It's application in the browser by using tools to control both the mouse and keyboard\n",
    "\n",
    "The overall design looks like the following:\n",
    "\n",
    "<!-- ![Voyager Image](./img/web-voyager.excalidraw.png) -->\n",
    "<img src=\"./img/web-voyager.excalidraw.png\" width=\"50%\">\n",
    "\n",
    "\n",
    "## Configure environment\n",
    "\n",
    "We will first set up LangSmith tracing. Though optional, this lets us inspect and debug agent's trajectory for a given input.\n",
    "\n",
    "You can sign up at [smith.langchain.com](https://smith.langchain.com/) to get an API key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af83b042",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install -U --quiet langgraph langsmith langchain_openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "02f6e962-c6a5-43fa-b069-616c78c86c26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional: add tracing to visualize the agent trajectories\n",
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "\n",
    "def _getpass(env_var: str):\n",
    "    if not os.environ.get(env_var):\n",
    "        os.environ[env_var] = getpass(f\"{env_var}=\")\n",
    "\n",
    "\n",
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_PROJECT\"] = \"Web-Voyager\"\n",
    "_getpass(\"LANGCHAIN_API_KEY\")\n",
    "_getpass(\"OPENAI_API_KEY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15d2e932-e1ce-4f2e-93e9-c8caf44b2afc",
   "metadata": {},
   "source": [
    "#### Install Agent requirements\n",
    "\n",
    "The only additional requirement we have is the [playwright](https://playwright.dev/) browser. Uncomment and install below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "50b70dbe-ea14-440c-99ab-9cd171d78742",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install --upgrade --quiet  playwright > /dev/null\n",
    "# !playwright install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e2a14fa9-8ca7-4a7a-9827-8fbd465b6959",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "# This is just required for running async playwright in a Jupyter notebook\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0ee0f97-eb4e-4a13-b4f4-fc6439eec6a6",
   "metadata": {},
   "source": [
    "## Define Graph State\n",
    "\n",
    "The state provides the inputs to each node in the graph.\n",
    "\n",
    "In our case, the agent will track the webpage object (within the browser), annotated images + bounding boxes, the user's initial request, and the messages containing the agent scratchpad, system prompt, and other information.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "51467b74-1be9-46fc-a3c0-0051eec62ed4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List, Optional, TypedDict\n",
    "\n",
    "from langchain_core.messages import BaseMessage, SystemMessage\n",
    "from playwright.async_api import Page\n",
    "\n",
    "\n",
    "class BBox(TypedDict):\n",
    "    x: float\n",
    "    y: float\n",
    "    text: str\n",
    "    type: str\n",
    "    ariaLabel: str\n",
    "\n",
    "\n",
    "class Prediction(TypedDict):\n",
    "    action: str\n",
    "    args: Optional[List[str]]\n",
    "\n",
    "\n",
    "# This represents the state of the agent\n",
    "# as it proceeds through execution\n",
    "class AgentState(TypedDict):\n",
    "    page: Page  # The Playwright web page lets us interact with the web environment\n",
    "    input: str  # User request\n",
    "    img: str  # b64 encoded screenshot\n",
    "    bboxes: List[BBox]  # The bounding boxes from the browser annotation function\n",
    "    prediction: Prediction  # The Agent's output\n",
    "    # A system message (or messages) containing the intermediate steps\n",
    "    scratchpad: List[BaseMessage]\n",
    "    observation: str  # The most recent response from a tool"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8016a06a-3a90-46a4-85d3-510b83dfcef4",
   "metadata": {},
   "source": [
    "## Define tools\n",
    "\n",
    "The agent has 6 simple tools:\n",
    "\n",
    "1. Click (at labeled box)\n",
    "2. Type\n",
    "3. Scroll\n",
    "4. Wait\n",
    "5. Go back\n",
    "6. Go to search engine (Google)\n",
    "\n",
    "\n",
    "We define them below here as functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b77978f0-c20f-495f-8026-8e1ee0b56e0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "import platform\n",
    "\n",
    "\n",
    "async def click(state: AgentState):\n",
    "    # - Click [Numerical_Label]\n",
    "    page = state[\"page\"]\n",
    "    click_args = state[\"prediction\"][\"args\"]\n",
    "    if click_args is None or len(click_args) != 1:\n",
    "        return f\"Failed to click bounding box labeled as number {click_args}\"\n",
    "    bbox_id = click_args[0]\n",
    "    bbox_id = int(bbox_id)\n",
    "    try:\n",
    "        bbox = state[\"bboxes\"][bbox_id]\n",
    "    except Exception:\n",
    "        return f\"Error: no bbox for : {bbox_id}\"\n",
    "    x, y = bbox[\"x\"], bbox[\"y\"]\n",
    "    await page.mouse.click(x, y)\n",
    "    # TODO: In the paper, they automatically parse any downloaded PDFs\n",
    "    # We could add something similar here as well and generally\n",
    "    # improve response format.\n",
    "    return f\"Clicked {bbox_id}\"\n",
    "\n",
    "\n",
    "async def type_text(state: AgentState):\n",
    "    page = state[\"page\"]\n",
    "    type_args = state[\"prediction\"][\"args\"]\n",
    "    if type_args is None or len(type_args) != 2:\n",
    "        return (\n",
    "            f\"Failed to type in element from bounding box labeled as number {type_args}\"\n",
    "        )\n",
    "    bbox_id = type_args[0]\n",
    "    bbox_id = int(bbox_id)\n",
    "    bbox = state[\"bboxes\"][bbox_id]\n",
    "    x, y = bbox[\"x\"], bbox[\"y\"]\n",
    "    text_content = type_args[1]\n",
    "    await page.mouse.click(x, y)\n",
    "    # Check if MacOS\n",
    "    select_all = \"Meta+A\" if platform.system() == \"Darwin\" else \"Control+A\"\n",
    "    await page.keyboard.press(select_all)\n",
    "    await page.keyboard.press(\"Backspace\")\n",
    "    await page.keyboard.type(text_content)\n",
    "    await page.keyboard.press(\"Enter\")\n",
    "    return f\"Typed {text_content} and submitted\"\n",
    "\n",
    "\n",
    "async def scroll(state: AgentState):\n",
    "    page = state[\"page\"]\n",
    "    scroll_args = state[\"prediction\"][\"args\"]\n",
    "    if scroll_args is None or len(scroll_args) != 2:\n",
    "        return \"Failed to scroll due to incorrect arguments.\"\n",
    "\n",
    "    target, direction = scroll_args\n",
    "\n",
    "    if target.upper() == \"WINDOW\":\n",
    "        # Not sure the best value for this:\n",
    "        scroll_amount = 500\n",
    "        scroll_direction = (\n",
    "            -scroll_amount if direction.lower() == \"up\" else scroll_amount\n",
    "        )\n",
    "        await page.evaluate(f\"window.scrollBy(0, {scroll_direction})\")\n",
    "    else:\n",
    "        # Scrolling within a specific element\n",
    "        scroll_amount = 200\n",
    "        target_id = int(target)\n",
    "        bbox = state[\"bboxes\"][target_id]\n",
    "        x, y = bbox[\"x\"], bbox[\"y\"]\n",
    "        scroll_direction = (\n",
    "            -scroll_amount if direction.lower() == \"up\" else scroll_amount\n",
    "        )\n",
    "        await page.mouse.move(x, y)\n",
    "        await page.mouse.wheel(0, scroll_direction)\n",
    "\n",
    "    return f\"Scrolled {direction} in {'window' if target.upper() == 'WINDOW' else 'element'}\"\n",
    "\n",
    "\n",
    "async def wait(state: AgentState):\n",
    "    sleep_time = 5\n",
    "    await asyncio.sleep(sleep_time)\n",
    "    return f\"Waited for {sleep_time}s.\"\n",
    "\n",
    "\n",
    "async def go_back(state: AgentState):\n",
    "    page = state[\"page\"]\n",
    "    await page.go_back()\n",
    "    return f\"Navigated back a page to {page.url}.\"\n",
    "\n",
    "\n",
    "async def to_google(state: AgentState):\n",
    "    page = state[\"page\"]\n",
    "    await page.goto(\"https://www.google.com/\")\n",
    "    return \"Navigated to google.com.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed4d4d9f-9971-477c-b391-1a73dee34573",
   "metadata": {},
   "source": [
    "## Define Agent\n",
    "\n",
    "The agent is driven by a multi-modal model and decides the action to take for each step. It is composed of a few runnable objects:\n",
    "\n",
    "1. A `mark_page` function to annotate the current page with bounding boxes\n",
    "2. A prompt to hold the user question, annotated image, and agent scratchpad\n",
    "3. GPT-4V to decide the next steps\n",
    "4. Parsing logic to extract the action\n",
    "\n",
    "\n",
    "Let's first define the annotation step:\n",
    "#### Browser Annotations\n",
    "\n",
    "This function annotates all buttons, inputs, text areas, etc. with numbered bounding boxes. GPT-4V then just has to refer to a bounding box\n",
    "when taking actions, reducing the complexity of the overall task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "03604500-a16c-4ea0-b6a9-ab81d2ecc00a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "\n",
    "from langchain_core.runnables import chain as chain_decorator\n",
    "\n",
    "# Some javascript we will run on each step\n",
    "# to take a screenshot of the page, select the\n",
    "# elements to annotate, and add bounding boxes\n",
    "with open(\"mark_page.js\") as f:\n",
    "    mark_page_script = f.read()\n",
    "\n",
    "\n",
    "@chain_decorator\n",
    "async def mark_page(page):\n",
    "    await page.evaluate(mark_page_script)\n",
    "    for _ in range(10):\n",
    "        try:\n",
    "            bboxes = await page.evaluate(\"markPage()\")\n",
    "            break\n",
    "        except Exception:\n",
    "            # May be loading...\n",
    "            asyncio.sleep(3)\n",
    "    screenshot = await page.screenshot()\n",
    "    # Ensure the bboxes don't follow us around\n",
    "    await page.evaluate(\"unmarkPage()\")\n",
    "    return {\n",
    "        \"img\": base64.b64encode(screenshot).decode(),\n",
    "        \"bboxes\": bboxes,\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e407ca7-1a8a-49ca-bec0-402529f60fe5",
   "metadata": {},
   "source": [
    "#### Agent definition\n",
    "\n",
    "Now we'll compose this function with the prompt, llm and output parser to complete our agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d5e47466-93fc-4b84-9334-c39ebe2682d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import hub\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "\n",
    "async def annotate(state):\n",
    "    marked_page = await mark_page.with_retry().ainvoke(state[\"page\"])\n",
    "    return {**state, **marked_page}\n",
    "\n",
    "\n",
    "def format_descriptions(state):\n",
    "    labels = []\n",
    "    for i, bbox in enumerate(state[\"bboxes\"]):\n",
    "        text = bbox.get(\"ariaLabel\") or \"\"\n",
    "        if not text.strip():\n",
    "            text = bbox[\"text\"]\n",
    "        el_type = bbox.get(\"type\")\n",
    "        labels.append(f'{i} (<{el_type}/>): \"{text}\"')\n",
    "    bbox_descriptions = \"\\nValid Bounding Boxes:\\n\" + \"\\n\".join(labels)\n",
    "    return {**state, \"bbox_descriptions\": bbox_descriptions}\n",
    "\n",
    "\n",
    "def parse(text: str) -> dict:\n",
    "    action_prefix = \"Action: \"\n",
    "    if not text.strip().split(\"\\n\")[-1].startswith(action_prefix):\n",
    "        return {\"action\": \"retry\", \"args\": f\"Could not parse LLM Output: {text}\"}\n",
    "    action_block = text.strip().split(\"\\n\")[-1]\n",
    "\n",
    "    action_str = action_block[len(action_prefix) :]\n",
    "    split_output = action_str.split(\" \", 1)\n",
    "    if len(split_output) == 1:\n",
    "        action, action_input = split_output[0], None\n",
    "    else:\n",
    "        action, action_input = split_output\n",
    "    action = action.strip()\n",
    "    if action_input is not None:\n",
    "        action_input = [\n",
    "            inp.strip().strip(\"[]\") for inp in action_input.strip().split(\";\")\n",
    "        ]\n",
    "    return {\"action\": action, \"args\": action_input}\n",
    "\n",
    "\n",
    "# Will need a later version of langchain to pull\n",
    "# this image prompt template\n",
    "prompt = hub.pull(\"wfh/web-voyager\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "833f1708-bdaf-4cea-8c26-ed6f09eb8c90",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = ChatOpenAI(model=\"gpt-4-vision-preview\", max_tokens=4096)\n",
    "agent = annotate | RunnablePassthrough.assign(\n",
    "    prediction=format_descriptions | prompt | llm | StrOutputParser() | parse\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7802b9fe-e75b-4779-b45d-003c218dba48",
   "metadata": {},
   "source": [
    "## Define graph\n",
    "\n",
    "We've created most of the important logic. We have one more function to define that will help us update the graph state after a tool is called."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e2b68881-1c40-48f4-b046-07bf226ceb60",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "\n",
    "def update_scratchpad(state: AgentState):\n",
    "    \"\"\"After a tool is invoked, we want to update\n",
    "    the scratchpad so the agent is aware of its previous steps\"\"\"\n",
    "    old = state.get(\"scratchpad\")\n",
    "    if old:\n",
    "        txt = old[0].content\n",
    "        last_line = txt.rsplit(\"\\n\", 1)[-1]\n",
    "        step = int(re.match(r\"\\d+\", last_line).group()) + 1\n",
    "    else:\n",
    "        txt = \"Previous action observations:\\n\"\n",
    "        step = 1\n",
    "    txt += f\"\\n{step}. {state['observation']}\"\n",
    "\n",
    "    return {**state, \"scratchpad\": [SystemMessage(content=txt)]}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eed8f97-a535-4416-ac60-c06b357aecf5",
   "metadata": {},
   "source": [
    "Now we can compose everything into a graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c8aa7106-cc79-49d9-8f9d-c9c13019ef5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.runnables import RunnableLambda\n",
    "\n",
    "from langgraph.graph import END, StateGraph\n",
    "\n",
    "graph_builder = StateGraph(AgentState)\n",
    "\n",
    "\n",
    "graph_builder.add_node(\"agent\", agent)\n",
    "graph_builder.set_entry_point(\"agent\")\n",
    "\n",
    "graph_builder.add_node(\"update_scratchpad\", update_scratchpad)\n",
    "graph_builder.add_edge(\"update_scratchpad\", \"agent\")\n",
    "\n",
    "tools = {\n",
    "    \"Click\": click,\n",
    "    \"Type\": type_text,\n",
    "    \"Scroll\": scroll,\n",
    "    \"Wait\": wait,\n",
    "    \"GoBack\": go_back,\n",
    "    \"Google\": to_google,\n",
    "}\n",
    "\n",
    "\n",
    "for node_name, tool in tools.items():\n",
    "    graph_builder.add_node(\n",
    "        node_name,\n",
    "        # The lambda ensures the function's string output is mapped to the \"observation\"\n",
    "        # key in the AgentState\n",
    "        RunnableLambda(tool) | (lambda observation: {\"observation\": observation}),\n",
    "    )\n",
    "    # Always return to the agent (by means of the update-scratchpad node)\n",
    "    graph_builder.add_edge(node_name, \"update_scratchpad\")\n",
    "\n",
    "\n",
    "def select_tool(state: AgentState):\n",
    "    # Any time the agent completes, this function\n",
    "    # is called to route the output to a tool or\n",
    "    # to the end user.\n",
    "    action = state[\"prediction\"][\"action\"]\n",
    "    if action == \"ANSWER\":\n",
    "        return END\n",
    "    if action == \"retry\":\n",
    "        return \"agent\"\n",
    "    return action\n",
    "\n",
    "\n",
    "graph_builder.add_conditional_edges(\"agent\", select_tool)\n",
    "\n",
    "graph = graph_builder.compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d11071f-f7ad-434d-99b7-14ebbbc92506",
   "metadata": {},
   "source": [
    "## Run agent\n",
    "\n",
    "Now that we've created the whole agent executor, we can run it on a few questions! We'll start our browser at \"google.com\" and then let it control the rest.\n",
    "\n",
    "Below is a helper function to help print out the steps to the notebook (and display the intermediate screenshots)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5d0fb86f-6923-4eb7-a61a-567498c0eba7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import display\n",
    "from playwright.async_api import async_playwright\n",
    "\n",
    "browser = await async_playwright().start()\n",
    "# We will set headless=False so we can watch the agent navigate the web.\n",
    "browser = await browser.chromium.launch(headless=False, args=None)\n",
    "page = await browser.new_page()\n",
    "_ = await page.goto(\"https://www.google.com\")\n",
    "\n",
    "\n",
    "async def call_agent(question: str, page, max_steps: int = 150):\n",
    "    event_stream = graph.astream(\n",
    "        {\n",
    "            \"page\": page,\n",
    "            \"input\": question,\n",
    "            \"scratchpad\": [],\n",
    "        },\n",
    "        {\n",
    "            \"recursion_limit\": max_steps,\n",
    "        },\n",
    "    )\n",
    "    final_answer = None\n",
    "    steps = []\n",
    "    async for event in event_stream:\n",
    "        # We'll display an event stream here\n",
    "        if \"agent\" not in event:\n",
    "            continue\n",
    "        pred = event[\"agent\"].get(\"prediction\") or {}\n",
    "        action = pred.get(\"action\")\n",
    "        action_input = pred.get(\"args\")\n",
    "        display.clear_output(wait=False)\n",
    "        steps.append(f\"{len(steps) + 1}. {action}: {action_input}\")\n",
    "        print(\"\\n\".join(steps))\n",
    "        display.display(display.Image(base64.b64decode(event[\"agent\"][\"img\"])))\n",
    "        if \"ANSWER\" in action:\n",
    "            final_answer = action_input[0]\n",
    "            break\n",
    "    return final_answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c5e4da43-4f99-4c3a-9873-4dc5e76a9098",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Type: ['7', 'WebVoyager paper arXiv']\n",
      "2. Click: ['32']\n",
      "3. Click: ['3']\n",
      "4. ANSWER;: ['The \"WebVoyager\" paper discusses the development of an end-to-end web agent that leverages large multimodal models. The abstract highlights the importance of such agents in automating complex tasks on the web, which remains a challenging domain due to the heterogeneity in structure and the semantic gap between humans and machines. The paper proposes a solution that combines neural symbolic models and multimodal web environments, aiming to advance the capabilities of these agents to perform web browsing tasks effectively. Further details would require a more in-depth analysis of the paper\\'s content beyond the abstract.']\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 149504 characters]",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final response: The \"WebVoyager\" paper discusses the development of an end-to-end web agent that leverages large multimodal models. The abstract highlights the importance of such agents in automating complex tasks on the web, which remains a challenging domain due to the heterogeneity in structure and the semantic gap between humans and machines. The paper proposes a solution that combines neural symbolic models and multimodal web environments, aiming to advance the capabilities of these agents to perform web browsing tasks effectively. Further details would require a more in-depth analysis of the paper's content beyond the abstract.\n"
     ]
    }
   ],
   "source": [
    "res = await call_agent(\"Could you explain the WebVoyager paper (on arxiv)?\", page)\n",
    "print(f\"Final response: {res}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d24da6b5-044e-474f-a664-875f760916ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. retry: Could not parse LLM Output: I'm sorry, but the image provided does not contain an XKCD comic. The image shows a page from a scientific paper titled \"WebVoyager 2: Building an End-to-End Web Agent with Large Multimodal Models.\" If you provide the XKCD comic you're referring to, I'd be happy to explain the humor in it.\n",
      "2. retry: Could not parse LLM Output: I'm sorry, but I cannot assist with that request.\n",
      "3. Google: None\n",
      "4. Type: ['6', 'xkcd.com']\n",
      "5. Click: ['25']\n",
      "6. ANSWER;: ['The XKCD comic titled \"Relationship Advice\" pokes fun at the sometimes exaggerated way people talk about the challenges of relationships. It starts with one character stating that relationships require constant work and are like a job, which is a common sentiment. However, the other character takes this comparison to an extreme, calling it a \"grueling ordeal\" and a \"crushing burden,\" which humorously exaggerates the difficulties of maintaining a relationship. The punchline comes when, after this escalation, the second character insists they\\'re fine and that it\\'s all normal, which satirizes how people might downplay their struggles to appear in control or deny the extent of their challenges. The humor lies in the hyperbole and the relatable nature of discussing relationship difficulties, as well as the contrast between the characters\\' statements and the insistence that everything is okay.']\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 348460 characters]",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final response: The XKCD comic titled \"Relationship Advice\" pokes fun at the sometimes exaggerated way people talk about the challenges of relationships. It starts with one character stating that relationships require constant work and are like a job, which is a common sentiment. However, the other character takes this comparison to an extreme, calling it a \"grueling ordeal\" and a \"crushing burden,\" which humorously exaggerates the difficulties of maintaining a relationship. The punchline comes when, after this escalation, the second character insists they're fine and that it's all normal, which satirizes how people might downplay their struggles to appear in control or deny the extent of their challenges. The humor lies in the hyperbole and the relatable nature of discussing relationship difficulties, as well as the contrast between the characters' statements and the insistence that everything is okay.\n"
     ]
    }
   ],
   "source": [
    "res = await call_agent(\n",
    "    \"Please explain the today's XKCD comic for me. Why is it funny?\", page\n",
    ")\n",
    "print(f\"Final response: {res}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a3386b5c-802e-4e95-8c20-1222388476be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Google: None\n",
      "2. Type: ['6', 'latest blog posts from langchain']\n",
      "3. Click: ['27']\n",
      "4. Click: ['14']\n",
      "5. Click: ['0']\n",
      "6. retry: Could not parse LLM Output: Thought: The latest blog posts from Langchain are displayed on the right side of the screen with titles and reading time. I will provide the titles of the featured blog posts as seen on the screen.\n",
      "\n",
      "Action: ANSWER; The latest blog posts from Langchain are:\n",
      "1. OpenGPTs - 7 min read\n",
      "2. LangGraph: Multi-Agent Workflows - 6 min read\n",
      "3. LangGraph - 7 min read\n",
      "4. LangChain v0.1.0 - 10 min read\n",
      "7. ANSWER;: ['The latest blog posts from Langchain are \"OpenGPTs,\" \"LangGraph: Multi-Agent Workflows,\" and \"LangGraph.\"']\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 529916 characters]",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final response: The latest blog posts from Langchain are \"OpenGPTs,\" \"LangGraph: Multi-Agent Workflows,\" and \"LangGraph.\"\n"
     ]
    }
   ],
   "source": [
    "res = await call_agent(\"What are the latest blog posts from langchain?\", page)\n",
    "print(f\"Final response: {res}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "319e95d3-4362-45f4-9c83-540913b23f41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Google: None\n",
      "2. Type: ['6', 'Google Maps']\n",
      "3. Click: ['0']\n",
      "4. Click: ['0']\n",
      "5. Wait: None\n",
      "6. Click: ['22']\n",
      "7. Click: ['0']\n",
      "8. Click: ['2']\n",
      "9. Type: ['0', 'San Francisco downtown to SFO']\n",
      "10. Click: ['1']\n",
      "11. Click: ['2']\n",
      "12. Type: ['8', 'San Francisco International Airport SFO']\n",
      "13. Click: ['14']\n",
      "14. Click: ['28']\n",
      "15. Scroll: ['WINDOW', 'up']\n",
      "16. Scroll: ['WINDOW', 'up']\n",
      "17. Click: ['10']\n",
      "18. Click: ['28']\n",
      "19. ANSWER;: ['To arrive at San Francisco International Airport (SFO) by 7:00 AM starting from downtown San Francisco, you should leave by 6:46 AM according to the current Google Maps information, which estimates a 44-minute travel time.']\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 711108 characters]",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final response: To arrive at San Francisco International Airport (SFO) by 7:00 AM starting from downtown San Francisco, you should leave by 6:46 AM according to the current Google Maps information, which estimates a 44-minute travel time.\n"
     ]
    }
   ],
   "source": [
    "res = await call_agent(\n",
    "    \"Could you check google maps to see when i should leave to get to SFO by 7 o'clock? starting from SF downtown.\",\n",
    "    page,\n",
    ")\n",
    "print(f\"Final response: {res}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcb656d9-fff5-42b0-9b16-60716a51ca0b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
