Twelve versions, every technique measured on all 125 tasks, negative results recorded
alongside the wins (chain-split -4.7, nudge inside noise, 36-step runway bought
nothing). Perception was most of the gap: subtree-text names alone were +12 points.
Category ledger vs their whole stack: lead 5, tie 3, behind 1 (forms -- long widget
flows, needs a scripted autocomplete primitive, not more steps). Product-port list
staged in priority order with each item's measured arena win attached.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The subtree-text name fix cracked the compound cluster (18/20 vs their 17) and email
(9/10 vs their 6). Category ledger vs their whole stack: LEAD 5, tie 3, behind 1
(forms 14v18). v11 closes the forms mechanics: click(x,y) auto-rewrites to
mouse_click, a submit click never rides in a chain behind unverified actions (the
product's verify-then-send principle, now measured), and 500 max_tokens stops long
select_option payloads truncating mid-quote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
24-step cap + progressive vision recovered the step-capped flows: +6.4 points, zero
false claims, still 9x faster than the real browser-use (69.6% at 44.5s). We now LEAD
their whole stack on drag (10v6), spatial (6v4) and text_entry (13v12), tie reasoning.
v10 root-causes the biggest remaining cluster (click_compound 10v17): styled links
render as '(alink)' -- their visible text is a child StaticText the name fallback never
read, so every link in a tab panel was indistinguishable and a guess was terminal.
Nameless rows now resolve by their own subtree text before the DOM class hint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Scripted exact-match fastpath + exact-name guard moved the wins where near-miss clicks
were terminal: drag 8/13 vs their 6/13, spatial 7/13 vs their 4/13, at 4.8s and zero
false claims. The remaining systematic gap is runway, not ability: 21 of 54 losses die
exactly at the 12-step cap, all in the email/forms/long-flow cluster their 44s median
happily outlasts. v9 sweeps with a 24-step cap (wins end at median 3 steps, so the win
path pays nothing) plus progressive vision -- any episode that burns 6 actions gets a
screenshot every turn thereafter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Ablation on the full 125: multi-action alone 50.4% at 4.3s (faster than the v5 base it
improves on -- fewer LLM calls is both the speed and the accuracy win); adaptive vision
adds +6.4 points for 301 screenshot calls that only fire when stuck or the goal reads
spatial, so the median win pays +0.3s. circle-center fell in one step once the model
could see the circle. v7v (vision-only) completes the ablation table.
v8 queued: scripted exact-match fast path (the product's own scripted-then-model shape)
-- when the goal's quoted target is visible as exactly one row, click it with no model
call. The tab/section losses are all terminal near-miss clicks on a goal-named link,
and the fastest correct click is the one no model gets to second-guess.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Controlled (view-only difference): our v5 beats the browser-use-shaped arm 48.0% to
36.0% at 2.6x the speed with fewer tokens; ahead in 6 categories, tied in 2, one email
task behind inside single-seed noise. Whole-stack: the real browser-use agent scores
69.6% -- but at 44.5s median (10x ours) and 16 false success claims (12.8% of tasks)
against our zero across 875 episodes. Their rate edge is loop machinery (vision steps,
multi-action, raw JS-evaluate), not perception; the same evaluate hatch that buys the
rate produces the confident wrongness. Next frontier ranked by measured value:
screenshot-conditioned steps, multi-action turns, a VERIFIED evaluate primitive.
v5's eval-memory prompt is the champion arm; v6's loop-breaker nudge measured inside
noise (46.4%) -- recorded, kept, not adopted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
v3 (44.8%, beats the browser-use-shaped arm's 36.0% by 8.8 points at 3x the speed):
page-text panel, select options on the row, input twins exempt from dedupe. v4 queued:
DOM-attribute names for nameless icons and deep sibling context, which turn the email
suite's unlabeled trash/star icons into addressable rows.
Product fix landed with tests: interactiveRanking.ts collapsed adjacent same-named
inputs and adjacent nameless icons as twins -- measured on MiniWoB, that single
collapse cost enter-password and the whole email suite. Inputs and nameless rows are
now exempt; named icon+label pairs still collapse. 17/17 tests, tsc clean.
ARENA.md carries the full scoreboard, per-category verdicts, what was ingested from
whom, and the honest gaps (drag and vision-less spatial) neither stack solves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
125 tasks, one external grader (WOB_REWARD_GLOBAL through BrowserGym), one recorder
book, agents never grade themselves. Arms: our shipped perception ported line-for-line
(deterministic + LLM), a browser-use-shaped flat-dump arm on the same model, the real
browser-use agent attached over CDP to the same seeded page, and Stagehand wiring.
Techniques already ingested from the competitor diffs, each landing measured points:
clickable-but-unroled elements (canvas/svg -- their flat dump solved spatial tasks
purely because the canvas was in it), coordinate actions (our click_point equivalent),
input twins exempt from dedupe (two same-named password boxes are two fields), select
options rendered on the row, and the page-text panel (our BrowserGetText equivalent).
v1 25.6% -> v2 32.0% on identical tasks/model, wall clock down 3.8s -> 3.2s.
Supervision is out-of-process: SIGALRM exceptions got swallowed inside stack retry
loops twice, so a watcher kills the process tree when the recorder stops moving and
resumes exactly the missing tasks. 502-dead LLM lanes book as infra, never as skill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Every other measurement in this session, ours and the competitors', was scored by a
harness I wrote and could therefore flatter. MiniWoB's reward comes from the task
definition, so it is ground truth nobody in this repo controls.
Working setup recorded because the obvious one fails: pypi browsergym cannot build
greenlet 3.0.3 on py3.13, so it needs a 3.12 venv, plus miniwob-plusplus's HTML served
locally. 341 envs register (125 MiniWoB, 215 AssistantBench, 0 WebArena -- that one needs
self-hosted sites).
The result reframes the competitive work: a ~20-line deterministic axtree heuristic with
NO model solves 5/15 = 33% at 1.44s median and zero cost. Published agent scores sit at
70-90%, so the real question for any model-driven agent, ours included, is what it buys
over the free baseline at 10-40x the latency.
miniwob_baseline.py takes task names as argv so a real policy can be dropped in for a
like-for-like number on the same 15 or all 125.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five of nine installed and driven; three produced comparable numbers. playwright-mcp
snapshots in ~130ms median and finds editables on all five sites including both ACE
editors, which puts it alongside agent-browser (~50ms) and against our
BrowserFindComposer's 13,991ms median.
The three that did not run are recorded with the reason, because 'could not test' is a
result: stagehand parses modelName as provider/model on the first slash and every id our
9Router exposes contains one (0 of 27 are slash-free), so it resolves the wrong provider
-- a config collision with our lane, not a defect, and its browser half worked fine.
OpenCLI needs a hand-installed Chrome extension. Skyvern is a Postgres deployment.
browsergym cannot build greenlet on py3.13.
Also noted: playwright-mcp's fill column reflects my own JS, not their element finder, so
only its snapshot times and editable counts are quoted as signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Numbers from this box, with the unverifiable ones marked as such.
The result that changes priorities: a deterministic accessibility-tree snapshot
(agent-browser, Rust CLI, no LLM) perceives in ~50ms median against our
BrowserFindComposer's 13,991ms median / 30,003ms p95 over 119 calls, and fills+verifies
onlinegdb in 326ms -- a site our agent cannot reach at all. Also fills w3schools (810ms)
and regex101 (31ms). That is a 40-450x perception gap with no model in the loop.
Also recorded: all nine claimed repos verified to exist with matching stars and
AGPL-compatible licences; browser-use benchmarked as an agent over 43 trials (we are
faster on every site we reach, 9.8s vs 35.7s median, but our arm ran with its fallback
disabled by dry-run so reach is not comparable); direct CDP evidence that deepl serves
our browser a Cloudflare challenge while browser-use's fresh-per-run profile is never
flagged; OpenCLI's API-first-with-browser-fallback design as the route to a learned path
that actually replays; and why Polar was not benchmarked (their ToS forbids using the
product to build a competing one, and their 98.0 concatenates two benchmarks and appears
on neither leaderboard).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A trial whose card was reaped before the readback could not be verified either way,
and excluding those rows was too charitable: it flipped openswarm 60.9% -> 100% and
McNemar p=0.02 -> 1.0 purely on my read timing. The backend's own dryrun-report
(composer/textboxes/filled) is the same bar the card readback applies, so it stands
in when the card is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Accumulating and resumable: a 95% CI narrow enough to separate two systems needs tens
of trials per site per arm, which is hours, so rows append to results/vs_raw.jsonl and
the report is computed from everything on disk. `vs_bench.py report` re-reads without
running anything.
Paired by construction: both arms run the same site back-to-back in one iteration, so
a site that changes mid-day (deepl started redirecting to a locale path during this
work) moves BOTH arms rather than one. Discordant pairs are tested with McNemar exact,
which is the right test for paired binary outcomes at small discordant counts, and
proportions use Wilson rather than the normal approximation -- at 20/20 the naive
interval is [100%, 100%], a certainty no 20-trial sample carries.
Three confounds found while building it, each of which had produced a wrong number:
1. The stop directive cannot be shared. "Do NOT submit" trips OpenSwarm's is_readonly()
and the send-script declines outright ("read-only directive in user request"), so
our arm measured 0/2 while browser-use scored 2/2 on the identical prompt. That is
the prompt disabling one side, not a capability gap. OpenSwarm now gets the bare task
and is held to reach-only by the backend's own OSW_SENDSCRIPT_DRYRUN=1; browser-use
has no dry-run mode, so its constraint stays in the prompt. Verified at the
classifier: is_readonly(BU)=True, is_readonly(OSW)=False.
2. Verification must be scoped to the cards a trial CREATED. A dashboard accumulates one
card per run and they persist in the saved layout; reading "all cards" checked seven
stale pages, and a regex101 run graded 0/1 because its own card had been reaped while
leftovers answered instead.
3. browser-use's readback goes through ITS session (get_tabs + get_or_create_cdp_session)
rather than a second websocket. Every earlier verifier bug -- hardcoded port, ws:// vs
http://, page-vs-webview target filtering -- lived in hand-rolled target discovery.
The browser is deliberately NOT held constant. browser-use cannot screenshot an Electron
webview and falls back to the app's page target on 100% of steps (49/49 measured), so a
shared browser blinds it; each side runs in the browser it was built for and that is
recorded as a caveat rather than hidden.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Added to compare browser-use against us with the BROWSER held constant, and the
result is that it cannot be held constant, which is worth writing down.
What works: with the port exposed, browser-use attaches, sees the browser cards
(they are CDP targets of type 'webview'), navigates one, and completes a task that
verifies independently. Cards persist across restarts because they live in the saved
dashboard layout, so there is something to attach to.
What does not: browser-use's screenshot path cannot capture a webview and falls back
to the app's own 'page' target -- 49 fallbacks in 49 agent steps, i.e. it drove the
web page while LOOKING at the OpenSwarm UI for every single step. A comparison run
that way handicaps the rival for an infrastructural reason and measures neither
agent, so the shared-browser numbers are not publishable as a fair head-to-head.
Two hazards found the hard way, both worth knowing before anyone repeats this:
- browser-use filters CDP targets to page/tab, so 'webview' is invisible to it, and
Electron refuses Target.createTarget ("Not supported") so it cannot open its own.
Pointed at Electron unrestricted it therefore grabs the app's page target and
navigates the OpenSwarm UI away, killing the renderer: the next sweep failed
infra_no_renderer 1/1 because of exactly that.
- An external agent must never be allowed to kill the shared session. Doing so after
trial 1 made trials 2-15 fail ConnectError, which a naive summary published as
"browser-use REACH 4/15" -- ten rows of my own teardown scored against the agent.
Off by default: an open debugging port is a local attack surface, and this only
exists for measurement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Read browser-use's own DOM code (dom/service.py,
is_element_visible_according_to_all_parents) and it decides visibility on CSS
alone: display, visibility, opacity, plus bounds existence and a viewport
intersection with 1000px of slack. There is NO minimum size gate anywhere.
Ours had `r.width >= 80 && r.height >= 16`, and that floor was the entire reason
ACE / CodeMirror-5 / Monaco composers were unreachable: their real input is a ~1x1
offscreen textarea that paints into a sibling div, so it fails a size test by
construction while being the correct fill target. Measured: w3schools and onlinegdb
scored 0/3 while every contenteditable editor (CodeMirror 6, Quill, TinyMCE,
CKEditor) passed -- the input MECHANISM split the suite, not the site.
This replaces the proxyHost patch from the previous commit, which reached the same
sites by walking up to a composer-sized ancestor. Their rule is simpler, more
general, and needs no ancestor heuristic, so proxyHost is deleted rather than kept
alongside. The floor was also doing less work than it looked: junk is excluded by
requiring a rich role or a nearby submit AND score >= 2, and a honeypot is
display:none / opacity:0, which still fails.
Measured on the merged tree, same lane (OSW_MODEL=opus-4-8-cc):
- anon w3schools 3/3, i.e. ACE still reached with proxyHost gone
- holdout 18/20 = 90%, byte-identical to before: no regression
- other_ms unchanged at 1211ms
deepl regressed to not_measurable in the same sweep and it is NOT this change:
A/B'd by restoring the old floor and re-running, it fails identically. deepl now
redirects to /en/translator/l/en/pt-BR and serves one textbox; the redirect predates
the change in the logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six of the eleven defects here were in the MEASUREMENT, not the product, and they
were wrong in both directions.
Harness, all of which silently produced wrong numbers:
- coverage.py preflight refused every sweep on a box holding exactly one backend:
stack.sh's supervisor is a `bash -c` quoting the whole uvicorn line, so it carries
both "-m uvicorn backend.main" AND the venv python path. Discriminate on POSITION.
- stack.sh status reported 2 backends over 1 and 0 webpack over a live dev server
(webpack retitles its process). A status check whose job is preventing a second
stack, failing in the direction that lets one land.
- c7_run.sh/c8_run.sh slice r6_be.log while stack.sh names logs by TAG: a stack under
any other tag hands every trial an empty slice and the sweep reports a confident
0/108. Now refuses loudly; it caught this exact mistake on first use.
- "Browser command timed out" was bucketed infra. It is ONE command blowing its own
budget, not a dead webview: all 4 such rows were BrowserFindComposer at exactly its
30s cap, every run completed after, zero card-gone markers in the whole log. Filed
as infra it read as 11.8% flake AND lifted holdout reach 70% -> 84%.
- api_retry / rate_limit_error now grade as infra. A provider 429 storm turned clean
15-21s exclusions into 188s product_no_composer rows.
- bench.py prints reach BOTH ways when a row is UNVERIFIED. An exclusion resting on
the agent's own word quietly flatters the score, and coverage.py's own instruction
to confirm it by hand goes unread (I quoted a 100% that excluded onlinegdb).
Timing was measuring 0.2% of the run: prestage completes BEFORE metrics_started_at,
so other_ms was 25ms of a 12700ms median while prestage (4146ms, ~61%) sat in no
bucket at all. prestage_ms/task_ms are now recorded; total_ms is deliberately NOT
redefined, which would invalidate every before/after already taken against it.
Product:
- find_composer rejected ACE/CodeMirror-5/Monaco composers. Their input is a ~1x1
offscreen textarea that paints into a sibling div, so it can never pass a size
gate. Accept it when a VISIBLE ancestor is composer-sized; honeypots stay out
because the input itself must not be display:none/visibility:hidden/opacity:0.
anon reach 80% -> 100%, holdout 89% -> 90%, p95 38.6s -> 9.7s.
- the composer poll slept a blind 0+1.2+1.4 = 2.6s whenever prestage staged nothing,
which is nearly every run, and it was the whole of other_ms's suspicious constancy
(2610-2613ms regardless of tools_ms). Stop when two reads are identical, the rule
the opener poll 40 lines below already applies. other_ms -53.7%, tools_ms flat.
- prestage no longer navigates to the page it is already on, nor sleeps 0.35s before
its first settle probe.
- is_replay_boundary reasoned from the NAME alone, so x.com's composer textbox named
"Post text" was ruled an irreversible send and truncated its replay to a bare
navigate. Excluded by ROLE; first_unsafe_step now passes role through at all.
Measured on this box: reach 100% (83% if onlinegdb's unverified exclusion is bogus),
0 false successes in ~155 runs, prestage tier-0/1 2702ms, other_ms -53.7%, infra
flake 0/158, holdout 18/20. Criteria 2/4/9 need live writes and are untouched.
Full evidence, including what did NOT work, in e2e/browser-v3/RESULTS_2026-08-06.md.
Also drops the tracked electron/node_modules symlink pointing at another machine's
Downloads folder; it is dangling on every other checkout and re-breaks the install on
any stash or checkout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- modes test: replace expect.fail() (not a Playwright API, threw TypeError)
with annotate+skip when a clean profile has no edit-or-create entry point
- theme x toggle matrix: clicking Dark/Light updates the settings draft, so
Save before asserting localStorage flips; conditional Save handles the
already-that-mode case
- both were hidden until the dashboard-precondition fix unblocked the serial
chain past them; full suite passes locally now
