This package implements autonomous browser agents that can control browser tabs in the frontend. The main agent can delegate browser tasks (navigate, click, type, screenshot, etc.) and these sub-agents execute them independently.

Architecture

There are two layers of tools here, which is important to understand:

┌──────────────────────────────────────────────────────────────────┐
│                        Main Agent (Claude SDK)                   │
│                                                                  │
│  Uses DELEGATION tools (MCP):                                    │
│    CreateBrowserAgent  — spin up a new browser + assign a task   │
│    BrowserAgent        — assign a task to an existing browser    │
│    BrowserAgents       — parallel tasks on multiple browsers     │
│                                                                  │
│  These are defined in browser_agent_mcp_schemas.py               │
│  and served by browser_agent_mcp_server.py (stdio subprocess)    │
└───────────────────────────────────┬──────────────────────────────┘
                                    │ HTTP POST to /browser-agent/run
                                    ▼
┌──────────────────────────────────────────────────────────────────┐
│                     Browser Agent (Anthropic API)                 │
│                                                                  │
│  Uses EXECUTION tools (direct):                                  │
│    BrowserScreenshot   — capture current page                    │
│    BrowserGetText      — get visible text content                │
│    BrowserNavigate     — go to a URL                             │
│    BrowserClick        — click an element by CSS selector        │
│    BrowserType         — type text into an element               │
│    BrowserEvaluate     — run JavaScript on the page              │
│    BrowserGetElements  — query elements by selector              │
│    BrowserScroll       — scroll the page                         │
│    BrowserWait         — wait for a specified duration            │
│                                                                  │
│  These are defined in schemas.py                                 │
│  and executed by executor.py (via WebSocket to frontend)         │
└───────────────────────────────────┬──────────────────────────────┘
                                    │ ws_manager.send_browser_command
                                    ▼
┌──────────────────────────────────────────────────────────────────┐
│                    Frontend Browser Iframe                        │
│                                                                  │
│  Receives WebSocket commands, executes in the actual browser,    │
│  and returns results (screenshots, text, element lists)          │
└──────────────────────────────────────────────────────────────────┘

Files

`schemas.py` — Browser Tool Definitions + System Prompt

Pure data file with no imports. Defines everything the browser agent needs to operate.

BROWSER_TOOLS_SCHEMA — List of 9 Anthropic-compatible tool definitions:

Tool	Parameters	Description
`BrowserScreenshot`	(none)	Capture a screenshot of the current page
`BrowserGetText`	(none)	Get all visible text content from the page
`BrowserNavigate`	`url`	Navigate to a URL
`BrowserClick`	`selector`	Click an element by CSS selector
`BrowserType`	`selector`, `text`	Type text into an input element
`BrowserEvaluate`	`expression`	Execute JavaScript and return the result
`BrowserGetElements`	`selector`	Query DOM elements by CSS selector
`BrowserScroll`	`direction` (up/down), `amount` (pixels)	Scroll the page
`BrowserWait`	`duration` (ms)	Wait for a specified duration

ACTION_MAP — Maps tool names to short action strings for the WebSocket protocol:

BrowserScreenshot → screenshot    BrowserClick     → click
BrowserGetText    → get_text      BrowserType      → type
BrowserNavigate   → navigate      BrowserEvaluate  → evaluate
BrowserGetElements→ get_elements  BrowserScroll    → scroll
BrowserWait       → wait

SYSTEM_PROMPT — Multi-paragraph instructions for the browser agent, including:

Always screenshot first to see the current state
Wait 2-3 seconds after navigation before screenshots
Use BrowserGetElements before clicking to find correct selectors
Don't get stuck in loops — try alternative approaches
Provide clear summaries of what was accomplished

MAX_TURNS — 25 (maximum LLM turns per browser agent run)

`executor.py` — Tool Execution Bridge

Bridges between the browser agent's tool calls and the actual browser in the frontend.

execute_browser_tool(tool_name, tool_input, browser_id, tab_id="") (async)

Looks up the action string from ACTION_MAP
Sends the command to the frontend via ws_manager.send_browser_command()
Waits up to 30 seconds for the frontend to return a result
Returns the raw result dict

_format_tool_result(result, tool_name)

Converts raw browser results into Anthropic content blocks
Special case for BrowserScreenshot: returns an image content block with base64 PNG
Other tools: returns text content blocks

_request_browser_approval(session, tool_name, tool_input) (async)

Wraps the generic request_approval() from execution/approval.py
Uses browser-specific defaults: 300s timeout, analytics tracking disabled

`runner.py` — Core Browser Agent Loop

The main engine that runs browser agents. Uses the Anthropic API directly (not the Claude Agent SDK).

run_browser_agent(task, browser_id, model, dashboard_id?, tab_id?, pre_selected?, initial_url?, parent_session_id?) (async)

Full lifecycle of a single browser agent:

Setup — Creates an AgentSession in "browser-agent" mode with the parent session ID
Initial navigation — If initial_url is provided, navigates and takes an initial screenshot
Agent loop (up to MAX_TURNS): a. Calls the Anthropic API with the conversation history + browser tools b. For each tool call in the response:
- Checks builtin permissions for approval requirements
- Requests approval if needed (via _request_browser_approval)
- Executes the tool via execute_browser_tool
- Formats the result and appends to conversation
- Logs the action for the action log c. If no tool calls → agent is done (the response is the summary) d. If cancelled → stop early
Completion — Takes a final screenshot, sets status, emits via WebSocket
Returns {session_id, browser_id, summary, action_log, final_screenshot}

Error handling:

API errors → logged, session status set to error
Cancellation → session status set to stopped
Always emits final status via WebSocket

_create_browser_card(dashboard_id, url, parent_session_id?) (async)

Creates a new browser card on the dashboard
Adds a BrowserTab with the given URL
Positions the card in the layout
Persists the dashboard and broadcasts dashboard:browser_card_added
Returns the new browser_id

run_browser_agents(tasks, model, dashboard_id?, pre_selected_browser_ids?, parent_session_id?) (async)

Runs multiple browser agents in parallel via asyncio.gather
For tasks without a browser_id: calls _create_browser_card first
For tasks with pre_selected_browser_ids: assigns available pre-selected IDs
Records browser_agent.batch_completed analytics
Returns list of result dicts

`browser_agent_mcp_schemas.py` — MCP Delegation Tool Schemas

Pure data file. Defines the 3 high-level tools the main agent uses to delegate browser work.

Tool	Parameters	Description
`CreateBrowserAgent`	`task`, `initial_url`	Create a new browser card and run a task on it
`BrowserAgent`	`browser_id`, `task`, `tab_id?`	Run a task on an existing browser card
`BrowserAgents`	`tasks[]` (each with `browser_id?`, `task`, `tab_id?`, `initial_url?`)	Run multiple browser tasks in parallel

These are the tools listed by the MCP server when the SDK calls tools/list.

`browser_agent_mcp_server.py` — Stdio MCP Server

A standalone script launched as a subprocess by the Claude Agent SDK. It implements the JSON-RPC MCP protocol and proxies browser agent requests to the OpenSwarm backend via HTTP.

Lifecycle:

The Claude Agent SDK starts this process with stdin/stdout pipes
It receives initialize → responds with server info and capabilities
It receives tools/list → returns the 3 delegation tools from browser_agent_mcp_schemas.py
It receives tools/call → dispatches to handle_tool_call
handle_tool_call POSTs to http://127.0.0.1:{port}/api/agents/browser-agent/run
The backend runs the browser agents and returns results
Results are formatted into MCP content blocks and returned to the SDK

Environment variables:

Variable	Default	Purpose
`OPENSWARM_PORT`	`8325`	Backend server port
`OPENSWARM_AGENT_MODEL`	`sonnet`	Model for browser agents
`OPENSWARM_DASHBOARD_ID`	—	Dashboard to create browser cards on
`OPENSWARM_PRE_SELECTED_BROWSER_IDS`	—	Comma-separated pre-selected browser IDs
`OPENSWARM_PARENT_SESSION_ID`	—	Parent session for child tracking

Screenshot compression:

If a screenshot's base64 exceeds 400KB, it's compressed via PIL (if available)
Resized to max 1024px wide, converted to JPEG at quality 60
Falls back gracefully if PIL isn't installed

No direct Python imports from the backend — communicates purely via HTTP. This isolation is necessary because it runs as a separate subprocess.

`init.py` — Package Exports

Re-exports run_browser_agent and run_browser_agents from runner.py for convenient importing:

from backend.apps.agents.browser import run_browser_agent, run_browser_agents

Key Concepts

Browser Cards

Browser cards are UI elements in the dashboard that contain an embedded browser. Each card has:

A unique browser_id
One or more BrowserTab instances (each with a URL)
A position in the dashboard layout

Browser agents are always associated with a specific browser card.

Parent-Child Sessions

When the main agent spawns browser agents, the browser agent sessions are linked to the parent via parent_session_id. This enables:

Querying all browser agents for a given session
Stopping all children when the parent stops
Tracking sub-agent costs and analytics

Two API Paths

There are two ways browser agents get triggered:

Via MCP (agent-initiated): The main agent calls CreateBrowserAgent → MCP server → HTTP → run_browser_agents → runner.py
Via REST (user-initiated): Direct POST to /browser-agent/run → run_browser_agents → runner.py

Both paths end up in the same run_browser_agents function.

Tool Approval in Browser Context

Browser agents share the same HITL approval system as the main agent, but with:

Shorter timeout (300s vs 600s)
Analytics tracking disabled (to avoid double-counting)
Permission checks against the same builtin permission policies

README.md

browser/ — Browser Automation Sub-Agents

Architecture

Files

schemas.py — Browser Tool Definitions + System Prompt

executor.py — Tool Execution Bridge

runner.py — Core Browser Agent Loop

browser_agent_mcp_schemas.py — MCP Delegation Tool Schemas

browser_agent_mcp_server.py — Stdio MCP Server

__init__.py — Package Exports