openswarm/backend/OLDapps/agents/browser/README.md

browser/ — Browser Automation Sub-Agents

This package implements autonomous browser agents that can control browser tabs in the frontend. The main agent can delegate browser tasks (navigate, click, type, screenshot, etc.) and these sub-agents execute them independently.

Architecture

There are two layers of tools here, which is important to understand:

┌──────────────────────────────────────────────────────────────────┐
│                        Main Agent (Claude SDK)                   │
│                                                                  │
│  Uses DELEGATION tools (MCP):                                    │
│    CreateBrowserAgent  — spin up a new browser + assign a task   │
│    BrowserAgent        — assign a task to an existing browser    │
│    BrowserAgents       — parallel tasks on multiple browsers     │
│                                                                  │
│  These are defined in browser_agent_mcp_schemas.py               │
│  and served by browser_agent_mcp_server.py (stdio subprocess)    │
└───────────────────────────────────┬──────────────────────────────┘
                                    │ HTTP POST to /browser-agent/run
                                    ▼
┌──────────────────────────────────────────────────────────────────┐
│                     Browser Agent (Anthropic API)                 │
│                                                                  │
│  Uses EXECUTION tools (direct):                                  │
│    BrowserScreenshot   — capture current page                    │
│    BrowserGetText      — get visible text content                │
│    BrowserNavigate     — go to a URL                             │
│    BrowserClick        — click an element by CSS selector        │
│    BrowserType         — type text into an element               │
│    BrowserEvaluate     — run JavaScript on the page              │
│    BrowserGetElements  — query elements by selector              │
│    BrowserScroll       — scroll the page                         │
│    BrowserWait         — wait for a specified duration            │
│                                                                  │
│  These are defined in schemas.py                                 │
│  and executed by executor.py (via WebSocket to frontend)         │
└───────────────────────────────────┬──────────────────────────────┘
                                    │ ws_manager.send_browser_command
                                    ▼
┌──────────────────────────────────────────────────────────────────┐
│                    Frontend Browser Iframe                        │
│                                                                  │
│  Receives WebSocket commands, executes in the actual browser,    │
│  and returns results (screenshots, text, element lists)          │
└──────────────────────────────────────────────────────────────────┘

Files

schemas.py — Browser Tool Definitions + System Prompt

Pure data file with no imports. Defines everything the browser agent needs to operate.

BROWSER_TOOLS_SCHEMA — List of 9 Anthropic-compatible tool definitions:

Tool	Parameters	Description
BrowserScreenshot	(none)	Capture a screenshot of the current page
BrowserGetText	(none)	Get all visible text content from the page
BrowserNavigate	url	Navigate to a URL
BrowserClick	selector	Click an element by CSS selector
BrowserType	selector, text	Type text into an input element
BrowserEvaluate	expression	Execute JavaScript and return the result
BrowserGetElements	selector	Query DOM elements by CSS selector
BrowserScroll	direction (up/down), amount (pixels)	Scroll the page
BrowserWait	duration (ms)	Wait for a specified duration

ACTION_MAP — Maps tool names to short action strings for the WebSocket protocol:

BrowserScreenshot → screenshot    BrowserClick     → click
BrowserGetText    → get_text      BrowserType      → type
BrowserNavigate   → navigate      BrowserEvaluate  → evaluate
BrowserGetElements→ get_elements  BrowserScroll    → scroll
BrowserWait       → wait

SYSTEM_PROMPT — Multi-paragraph instructions for the browser agent, including:

• Always screenshot first to see the current state
• Wait 2-3 seconds after navigation before screenshots
• Use BrowserGetElements before clicking to find correct selectors
• Don't get stuck in loops — try alternative approaches
• Provide clear summaries of what was accomplished

MAX_TURNS — 25 (maximum LLM turns per browser agent run)

---

executor.py — Tool Execution Bridge

Bridges between the browser agent's tool calls and the actual browser in the frontend.

execute_browser_tool(tool_name, tool_input, browser_id, tab_id="") (async)

1. Looks up the action string from ACTION_MAP
2. Sends the command to the frontend via ws_manager.send_browser_command()
3. Waits up to 30 seconds for the frontend to return a result
4. Returns the raw result dict

_format_tool_result(result, tool_name)

• Converts raw browser results into Anthropic content blocks
• Special case for BrowserScreenshot: returns an image content block with base64 PNG
• Other tools: returns text content blocks

_request_browser_approval(session, tool_name, tool_input) (async)

• Wraps the generic request_approval() from execution/approval.py
• Uses browser-specific defaults: 300s timeout, analytics tracking disabled

---

runner.py — Core Browser Agent Loop

The main engine that runs browser agents. Uses the Anthropic API directly (not the Claude Agent SDK).

run_browser_agent(task, browser_id, model, dashboard_id?, tab_id?, pre_selected?, initial_url?, parent_session_id?) (async)

Full lifecycle of a single browser agent:

1. Setup — Creates an AgentSession in "browser-agent" mode with the parent session ID
2. Initial navigation — If initial_url is provided, navigates and takes an initial screenshot
3. Agent loop (up to MAX_TURNS): a. Calls the Anthropic API with the conversation history + browser tools b. For each tool call in the response:
   • Checks builtin permissions for approval requirements
   • Requests approval if needed (via _request_browser_approval)
   • Executes the tool via execute_browser_tool
   • Formats the result and appends to conversation
   • Logs the action for the action log c. If no tool calls → agent is done (the response is the summary) d. If cancelled → stop early
4. Completion — Takes a final screenshot, sets status, emits via WebSocket
5. Returns {session_id, browser_id, summary, action_log, final_screenshot}

Error handling:

• API errors → logged, session status set to error
• Cancellation → session status set to stopped
• Always emits final status via WebSocket

_create_browser_card(dashboard_id, url, parent_session_id?) (async)

• Creates a new browser card on the dashboard
• Adds a BrowserTab with the given URL
• Positions the card in the layout
• Persists the dashboard and broadcasts dashboard:browser_card_added
• Returns the new browser_id

run_browser_agents(tasks, model, dashboard_id?, pre_selected_browser_ids?, parent_session_id?) (async)

• Runs multiple browser agents in parallel via asyncio.gather
• For tasks without a browser_id: calls _create_browser_card first
• For tasks with pre_selected_browser_ids: assigns available pre-selected IDs
• Records browser_agent.batch_completed analytics
• Returns list of result dicts

---

browser_agent_mcp_schemas.py — MCP Delegation Tool Schemas

Pure data file. Defines the 3 high-level tools the main agent uses to delegate browser work.

Tool	Parameters	Description
CreateBrowserAgent	task, initial_url	Create a new browser card and run a task on it
BrowserAgent	browser_id, task, tab_id?	Run a task on an existing browser card
BrowserAgents	tasks[] (each with browser_id?, task, tab_id?, initial_url?)	Run multiple browser tasks in parallel

These are the tools listed by the MCP server when the SDK calls tools/list.

---

browser_agent_mcp_server.py — Stdio MCP Server

A standalone script launched as a subprocess by the Claude Agent SDK. It implements the JSON-RPC MCP protocol and proxies browser agent requests to the OpenSwarm backend via HTTP.

Lifecycle:

1. The Claude Agent SDK starts this process with stdin/stdout pipes
2. It receives initialize → responds with server info and capabilities
3. It receives tools/list → returns the 3 delegation tools from browser_agent_mcp_schemas.py
4. It receives tools/call → dispatches to handle_tool_call
5. handle_tool_call POSTs to http://127.0.0.1:{port}/api/agents/browser-agent/run
6. The backend runs the browser agents and returns results
7. Results are formatted into MCP content blocks and returned to the SDK

Environment variables:

Variable	Default	Purpose
OPENSWARM_PORT	8325	Backend server port
OPENSWARM_AGENT_MODEL	sonnet	Model for browser agents
OPENSWARM_DASHBOARD_ID	—	Dashboard to create browser cards on
OPENSWARM_PRE_SELECTED_BROWSER_IDS	—	Comma-separated pre-selected browser IDs
OPENSWARM_PARENT_SESSION_ID	—	Parent session for child tracking

Screenshot compression:

• If a screenshot's base64 exceeds 400KB, it's compressed via PIL (if available)
• Resized to max 1024px wide, converted to JPEG at quality 60
• Falls back gracefully if PIL isn't installed

No direct Python imports from the backend — communicates purely via HTTP. This isolation is necessary because it runs as a separate subprocess.

---

__init__.py — Package Exports

Re-exports run_browser_agent and run_browser_agents from runner.py for convenient importing:

from backend.apps.agents.browser import run_browser_agent, run_browser_agents

Key Concepts

Browser Cards

Browser cards are UI elements in the dashboard that contain an embedded browser. Each card has:

• A unique browser_id
• One or more BrowserTab instances (each with a URL)
• A position in the dashboard layout

Browser agents are always associated with a specific browser card.

Parent-Child Sessions

When the main agent spawns browser agents, the browser agent sessions are linked to the parent via parent_session_id. This enables:

• Querying all browser agents for a given session
• Stopping all children when the parent stops
• Tracking sub-agent costs and analytics

Two API Paths

There are two ways browser agents get triggered:

1. Via MCP (agent-initiated): The main agent calls CreateBrowserAgent → MCP server → HTTP → run_browser_agents → runner.py
2. Via REST (user-initiated): Direct POST to /browser-agent/run → run_browser_agents → runner.py

Both paths end up in the same run_browser_agents function.

Tool Approval in Browser Context

Browser agents share the same HITL approval system as the main agent, but with:

• Shorter timeout (300s vs 600s)
• Analytics tracking disabled (to avoid double-counting)
• Permission checks against the same builtin permission policies