WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-09-30 05:24:50 +02:00
Code Issues Packages Projects Releases Wiki Activity

[eric] docs: record every browser tool installed and run, including the ones that would not

Browse Source 
Five of nine installed and driven; three produced comparable numbers. playwright-mcp
snapshots in ~130ms median and finds editables on all five sites including both ACE
editors, which puts it alongside agent-browser (~50ms) and against our
BrowserFindComposer's 13,991ms median.

The three that did not run are recorded with the reason, because 'could not test' is a
result: stagehand parses modelName as provider/model on the first slash and every id our
9Router exposes contains one (0 of 27 are slash-free), so it resolves the wrong provider
-- a config collision with our lane, not a defect, and its browser half worked fine.
OpenCLI needs a hand-installed Chrome extension. Skyvern is a Postgres deployment.
browsergym cannot build greenlet on py3.13.

Also noted: playwright-mcp's fill column reflects my own JS, not their element finder, so
only its snapshot times and editable counts are quoted as signal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ciregenz
2026-08-08 10:30:58 -07:00
co-authored by Claude Opus 5
parent 72c5effbff
commit 6dc6c0ecf2
1 changed files with 60 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
e2e/browser-v3/COMPETITIVE_2026-08-08.md
+60
View File
@@ -58,6 +58,66 @@ Category matters and is easy to get wrong: **agent-browser is a primitive layer
browser-use is an **agent**, stagehand is an **SDK**. Benchmarking the first against the second is a
category error; the comparison in s.1 is against our own deterministic finder, which is its true peer.
---
## 2b. Every tool actually installed and run, with what happened
Nine were claimed; I installed and drove five, and three of those produced comparable numbers. What
blocked the others is recorded because "we could not test it" is a result, not a gap to paper over.
| tool | installed | benchmarked | outcome |
| --- | --- | --- | --- |
| **browser-use** 0.9.x | yes (uv, py3.13) | **yes, 43 trials** | full agent numbers, s.3 |
| **agent-browser** 0.33.2 | yes (npm -g) | **yes, 5 sites** | fastest perception measured, s.1 |
| **@playwright/mcp** 0.0.79 | yes (npm -g) | **yes, 5 sites** | 5/5 editables found, below |
| **@browserbasehq/stagehand** 3.7.1 | yes (npm) | no | model-naming conflict, below |
| **@jackwener/opencli** 1.8.6 | yes (npm -g) | no | needs a hand-installed Chrome extension |
| **skyvern** 1.0.48 | no | no | 145 deps incl. Postgres/alembic/fastapi: a deployment |
| **browsergym** 0.14.3 | no | no | greenlet 3.0.3 will not build on py3.13 (pins playwright 1.44) |
| UI-TARS-desktop | no | no | desktop GUI app, no headless benchmark surface |
| BrowserOS | no | no | full Chromium fork; a browser swap, not a library test |
### playwright-mcp (n=5, headless, isolated profile)
Driven over raw JSON-RPC on stdio, 24 tools exposed.
| site | navigate | **snapshot** | editables | verified |
| --- | --- | --- | --- | --- |
| gtranslate | 1.18s | **0.13s** | 1 | yes |
| deepl | 1.10s | **0.05s** | 2 | yes |
| w3schools | 1.10s | **1.71s** | 1 | yes |
| regex101 | 2.42s | **0.02s** | 3 | yes |
| onlinegdb | 2.21s | **0.01s** | 4 | yes |
**Snapshot median ~130ms**, and it finds editables on all five including both ACE sites and deepl.
Honest limit: I filled via `browser_evaluate` (raw JS), so `verified` reflects my JS, not their
element finder. The **snapshot times and editable counts are the real signal**; the fill column only
shows the page was fillable. It also ran headless on a fresh profile, so deepl never challenged it --
the same advantage browser-use had, not a property of the tool.
### stagehand: a naming collision, not a defect
It parses `modelName` as `provider/model` on the first `/`. Every model id our 9Router exposes
contains a slash (`cc/claude-opus-4-8`, and **0 of 27 ids are slash-free**), so `anthropic/cc/...`
resolves to the wrong provider and falls through to a real OpenAI client: "OpenAI API key is missing".
Pointing it at 9Router's OpenAI-compatible endpoint instead got "No credentials for provider: openai"
from the router. Its browser half worked fine (pages loaded, `observe` returned in 98-4,973ms), so
this is a config incompatibility with OUR lane. A direct provider key would benchmark it in minutes.
### OpenCLI: correct instincts, manual setup
Could not run: `opencli browser` requires the OpenCLI Chrome extension loaded by hand, and every
command timed out at ~45s with "Make sure Chrome/Chromium is open and the OpenCLI extension is
enabled."
Two design points visible without running it, and both are things we should copy:
- `fill` returns `{filled, verified, text, actual}` -- **read-back verification built into the
primitive**, the same discipline as our send-script receipts, at the layer below the agent.
- Shipped adapters state postconditions outright ("Fails if the row is already read **or the
postcondition cannot be verified**").
---
## 3. browser-use, measured as an agent (n=43 trials)