Numbers from this box, with the unverifiable ones marked as such.
The result that changes priorities: a deterministic accessibility-tree snapshot
(agent-browser, Rust CLI, no LLM) perceives in ~50ms median against our
BrowserFindComposer's 13,991ms median / 30,003ms p95 over 119 calls, and fills+verifies
onlinegdb in 326ms -- a site our agent cannot reach at all. Also fills w3schools (810ms)
and regex101 (31ms). That is a 40-450x perception gap with no model in the loop.
Also recorded: all nine claimed repos verified to exist with matching stars and
AGPL-compatible licences; browser-use benchmarked as an agent over 43 trials (we are
faster on every site we reach, 9.8s vs 35.7s median, but our arm ran with its fallback
disabled by dry-run so reach is not comparable); direct CDP evidence that deepl serves
our browser a Cloudflare challenge while browser-use's fresh-per-run profile is never
flagged; OpenCLI's API-first-with-browser-fallback design as the route to a learned path
that actually replays; and why Polar was not benchmarked (their ToS forbids using the
product to build a competing one, and their 98.0 concatenates two benchmarks and appears
on neither leaderboard).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A trial whose card was reaped before the readback could not be verified either way,
and excluding those rows was too charitable: it flipped openswarm 60.9% -> 100% and
McNemar p=0.02 -> 1.0 purely on my read timing. The backend's own dryrun-report
(composer/textboxes/filled) is the same bar the card readback applies, so it stands
in when the card is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Accumulating and resumable: a 95% CI narrow enough to separate two systems needs tens
of trials per site per arm, which is hours, so rows append to results/vs_raw.jsonl and
the report is computed from everything on disk. `vs_bench.py report` re-reads without
running anything.
Paired by construction: both arms run the same site back-to-back in one iteration, so
a site that changes mid-day (deepl started redirecting to a locale path during this
work) moves BOTH arms rather than one. Discordant pairs are tested with McNemar exact,
which is the right test for paired binary outcomes at small discordant counts, and
proportions use Wilson rather than the normal approximation -- at 20/20 the naive
interval is [100%, 100%], a certainty no 20-trial sample carries.
Three confounds found while building it, each of which had produced a wrong number:
1. The stop directive cannot be shared. "Do NOT submit" trips OpenSwarm's is_readonly()
and the send-script declines outright ("read-only directive in user request"), so
our arm measured 0/2 while browser-use scored 2/2 on the identical prompt. That is
the prompt disabling one side, not a capability gap. OpenSwarm now gets the bare task
and is held to reach-only by the backend's own OSW_SENDSCRIPT_DRYRUN=1; browser-use
has no dry-run mode, so its constraint stays in the prompt. Verified at the
classifier: is_readonly(BU)=True, is_readonly(OSW)=False.
2. Verification must be scoped to the cards a trial CREATED. A dashboard accumulates one
card per run and they persist in the saved layout; reading "all cards" checked seven
stale pages, and a regex101 run graded 0/1 because its own card had been reaped while
leftovers answered instead.
3. browser-use's readback goes through ITS session (get_tabs + get_or_create_cdp_session)
rather than a second websocket. Every earlier verifier bug -- hardcoded port, ws:// vs
http://, page-vs-webview target filtering -- lived in hand-rolled target discovery.
The browser is deliberately NOT held constant. browser-use cannot screenshot an Electron
webview and falls back to the app's page target on 100% of steps (49/49 measured), so a
shared browser blinds it; each side runs in the browser it was built for and that is
recorded as a caveat rather than hidden.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Added to compare browser-use against us with the BROWSER held constant, and the
result is that it cannot be held constant, which is worth writing down.
What works: with the port exposed, browser-use attaches, sees the browser cards
(they are CDP targets of type 'webview'), navigates one, and completes a task that
verifies independently. Cards persist across restarts because they live in the saved
dashboard layout, so there is something to attach to.
What does not: browser-use's screenshot path cannot capture a webview and falls back
to the app's own 'page' target -- 49 fallbacks in 49 agent steps, i.e. it drove the
web page while LOOKING at the OpenSwarm UI for every single step. A comparison run
that way handicaps the rival for an infrastructural reason and measures neither
agent, so the shared-browser numbers are not publishable as a fair head-to-head.
Two hazards found the hard way, both worth knowing before anyone repeats this:
- browser-use filters CDP targets to page/tab, so 'webview' is invisible to it, and
Electron refuses Target.createTarget ("Not supported") so it cannot open its own.
Pointed at Electron unrestricted it therefore grabs the app's page target and
navigates the OpenSwarm UI away, killing the renderer: the next sweep failed
infra_no_renderer 1/1 because of exactly that.
- An external agent must never be allowed to kill the shared session. Doing so after
trial 1 made trials 2-15 fail ConnectError, which a naive summary published as
"browser-use REACH 4/15" -- ten rows of my own teardown scored against the agent.
Off by default: an open debugging port is a local attack surface, and this only
exists for measurement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Read browser-use's own DOM code (dom/service.py,
is_element_visible_according_to_all_parents) and it decides visibility on CSS
alone: display, visibility, opacity, plus bounds existence and a viewport
intersection with 1000px of slack. There is NO minimum size gate anywhere.
Ours had `r.width >= 80 && r.height >= 16`, and that floor was the entire reason
ACE / CodeMirror-5 / Monaco composers were unreachable: their real input is a ~1x1
offscreen textarea that paints into a sibling div, so it fails a size test by
construction while being the correct fill target. Measured: w3schools and onlinegdb
scored 0/3 while every contenteditable editor (CodeMirror 6, Quill, TinyMCE,
CKEditor) passed -- the input MECHANISM split the suite, not the site.
This replaces the proxyHost patch from the previous commit, which reached the same
sites by walking up to a composer-sized ancestor. Their rule is simpler, more
general, and needs no ancestor heuristic, so proxyHost is deleted rather than kept
alongside. The floor was also doing less work than it looked: junk is excluded by
requiring a rich role or a nearby submit AND score >= 2, and a honeypot is
display:none / opacity:0, which still fails.
Measured on the merged tree, same lane (OSW_MODEL=opus-4-8-cc):
- anon w3schools 3/3, i.e. ACE still reached with proxyHost gone
- holdout 18/20 = 90%, byte-identical to before: no regression
- other_ms unchanged at 1211ms
deepl regressed to not_measurable in the same sweep and it is NOT this change:
A/B'd by restoring the old floor and re-running, it fails identically. deepl now
redirects to /en/translator/l/en/pt-BR and serves one textbox; the redirect predates
the change in the logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six of the eleven defects here were in the MEASUREMENT, not the product, and they
were wrong in both directions.
Harness, all of which silently produced wrong numbers:
- coverage.py preflight refused every sweep on a box holding exactly one backend:
stack.sh's supervisor is a `bash -c` quoting the whole uvicorn line, so it carries
both "-m uvicorn backend.main" AND the venv python path. Discriminate on POSITION.
- stack.sh status reported 2 backends over 1 and 0 webpack over a live dev server
(webpack retitles its process). A status check whose job is preventing a second
stack, failing in the direction that lets one land.
- c7_run.sh/c8_run.sh slice r6_be.log while stack.sh names logs by TAG: a stack under
any other tag hands every trial an empty slice and the sweep reports a confident
0/108. Now refuses loudly; it caught this exact mistake on first use.
- "Browser command timed out" was bucketed infra. It is ONE command blowing its own
budget, not a dead webview: all 4 such rows were BrowserFindComposer at exactly its
30s cap, every run completed after, zero card-gone markers in the whole log. Filed
as infra it read as 11.8% flake AND lifted holdout reach 70% -> 84%.
- api_retry / rate_limit_error now grade as infra. A provider 429 storm turned clean
15-21s exclusions into 188s product_no_composer rows.
- bench.py prints reach BOTH ways when a row is UNVERIFIED. An exclusion resting on
the agent's own word quietly flatters the score, and coverage.py's own instruction
to confirm it by hand goes unread (I quoted a 100% that excluded onlinegdb).
Timing was measuring 0.2% of the run: prestage completes BEFORE metrics_started_at,
so other_ms was 25ms of a 12700ms median while prestage (4146ms, ~61%) sat in no
bucket at all. prestage_ms/task_ms are now recorded; total_ms is deliberately NOT
redefined, which would invalidate every before/after already taken against it.
Product:
- find_composer rejected ACE/CodeMirror-5/Monaco composers. Their input is a ~1x1
offscreen textarea that paints into a sibling div, so it can never pass a size
gate. Accept it when a VISIBLE ancestor is composer-sized; honeypots stay out
because the input itself must not be display:none/visibility:hidden/opacity:0.
anon reach 80% -> 100%, holdout 89% -> 90%, p95 38.6s -> 9.7s.
- the composer poll slept a blind 0+1.2+1.4 = 2.6s whenever prestage staged nothing,
which is nearly every run, and it was the whole of other_ms's suspicious constancy
(2610-2613ms regardless of tools_ms). Stop when two reads are identical, the rule
the opener poll 40 lines below already applies. other_ms -53.7%, tools_ms flat.
- prestage no longer navigates to the page it is already on, nor sleeps 0.35s before
its first settle probe.
- is_replay_boundary reasoned from the NAME alone, so x.com's composer textbox named
"Post text" was ruled an irreversible send and truncated its replay to a bare
navigate. Excluded by ROLE; first_unsafe_step now passes role through at all.
Measured on this box: reach 100% (83% if onlinegdb's unverified exclusion is bogus),
0 false successes in ~155 runs, prestage tier-0/1 2702ms, other_ms -53.7%, infra
flake 0/158, holdout 18/20. Criteria 2/4/9 need live writes and are untouched.
Full evidence, including what did NOT work, in e2e/browser-v3/RESULTS_2026-08-06.md.
Also drops the tracked electron/node_modules symlink pointing at another machine's
Downloads folder; it is dangling on every other checkout and re-breaks the install on
any stash or checkout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- modes test: replace expect.fail() (not a Playwright API, threw TypeError)
with annotate+skip when a clean profile has no edit-or-create entry point
- theme x toggle matrix: clicking Dark/Light updates the settings draft, so
Save before asserting localStorage flips; conditional Save handles the
already-that-mode case
- both were hidden until the dashboard-precondition fix unblocked the serial
chain past them; full suite passes locally now
