The untainted re-run replaces the asterisked 48%: their best-model cell is 16 points
behind ours at 4.5x the wall. Claude-5 lanes reject the temperature param outright --
v17's 400-loop root-caused, field omitted; decode variance now attacked at the action
layer (fill-verify, verify-terminal) instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
temperature=0 kills decode randomness (measured cause of same-seed run-to-run flips);
mechanical fill-verify re-issues a fill exactly when the fresh look shows the value did
not stick (feature-triggered, fires on observed mismatch only); the sweep runs
--isolate so the copy-paste playwright wedge cannot poison neighbors. Generalization
protocol: zero task-name logic (audited -- 3 matches are all evidence citations in
comments), two seeds (42+43: seeds change task CONTENT, so cross-seed consistency is
the in-family generalization test), and full-125 regression not just target tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The four pattern-fixes landed where aimed (email perfect 10/10, text_entry 16/17,
click_basic 12/13) while the untouched hard cluster held. Across v14|v15|v16 on opus
the architecture demonstrably solves 112/125 = 89.6%; the binding constraint is now
which-90% selection variance, not capability. bu-real-opus5 clean re-run launched on
the idle box (load 1.7) to replace its tainted cell.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The confounders disclosure (claim asymmetry, co-run load, seed noise) vanished in the
full-file rewrite -- exactly the content that must never be lost silently. Restored,
plus the 2026-08-11 host-load window: bu-real-opus5's cell asterisked pending clean
re-run, tainted v16 attempts discarded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The module-scope import of browsergym.assistantbench pulls in HuggingFace datasets'
multiprocessing at interpreter start; 248 of 252 v16 MiniWoB episodes died to the
resulting 'no running event loop' cascade before the import moved inside env setup,
loaded only when a dotted task name actually needs it. v16 relaunched clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
AssistantBench pages make BrowserGym's obs extraction slow and flaky (context-destroyed
retries, unmarked frames); one step-hang kill corrupted process-wide playwright state
and every later episode instant-failed as infra. supervisor --isolate spawns one task
per process, containing the blast radius. Step deadline raised to 150s for live pages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
data/ now also carries logs/ (raw sweep + supervisor stdout, including browser-use's
own internal plan/eval lines) next to the episode book and screenshots; MANIFEST.md
documents the full schema. Goal capture widened to 600 chars for the research suites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The verify-terminal prompt line now leaves history after its one call (it was
poisoning every later turn with a stale instruction). ARENA.md gains a stated-
confounders section: the false-claim metric is asymmetric by construction (their API
claims, ours never does), co-run CPU load, and the single-seed noise floor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Verify-before-terminal (the variance killer: same task flips run to run on premature
submits), look-act-look-again after any mouse geometry action, rapid-fire multi-cap 6
for guess-and-check games, and sub-step value confirmation for booking flows. Queued
on opus-5 behind the last fair-grid cell, aiming to stabilize the 85.6% union.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
v14-opus5 82.4% at 6.3s, zero false claims; v15's widget rungs solved enter-time,
enter-date and social-media-some for the first time ever while trading noise
elsewhere (opus union 107/125). browser-use on sonnet-5 is running 63% at 42s --
worse than its sonnet-4-6 74.4%: their loop does not scale with the model, ours does.
Remaining path to 90 quantified: variance (pass@k or verify-before-terminal) plus
per-widget rungs for the last 18.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
sonnet-4-6 pair: ours 77.6% @ 6.5s / 0 false vs their 74.4% @ 37.4s / 8 false. Their
false claims persist across models (16 haiku, 8 sonnet) -- structural, not model error.
sonnet-5 plateaus (76.0%): past sonnet the constraint is widget primitives, not brains.
Opus-5 and bu-real-sonnet-5 sweeping; AssistantBench pair chained behind them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
70.4% vs v10's 75.2. Task-level diff: SoM cost 5-6 wins (mark labels literally cover
the words on highlight-text; marked screenshots pushed the model to freehand mouse
work on tasks element clicks already won). Scripted drag net +1, autocomplete resolver
clean gains with zero losses. v14 keeps the two proven primitives, drops the overlay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Twelve versions, every technique measured on all 125 tasks, negative results recorded
alongside the wins (chain-split -4.7, nudge inside noise, 36-step runway bought
nothing). Perception was most of the gap: subtree-text names alone were +12 points.
Category ledger vs their whole stack: lead 5, tie 3, behind 1 (forms -- long widget
flows, needs a scripted autocomplete primitive, not more steps). Product-port list
staged in priority order with each item's measured arena win attached.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The subtree-text name fix cracked the compound cluster (18/20 vs their 17) and email
(9/10 vs their 6). Category ledger vs their whole stack: LEAD 5, tie 3, behind 1
(forms 14v18). v11 closes the forms mechanics: click(x,y) auto-rewrites to
mouse_click, a submit click never rides in a chain behind unverified actions (the
product's verify-then-send principle, now measured), and 500 max_tokens stops long
select_option payloads truncating mid-quote.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
24-step cap + progressive vision recovered the step-capped flows: +6.4 points, zero
false claims, still 9x faster than the real browser-use (69.6% at 44.5s). We now LEAD
their whole stack on drag (10v6), spatial (6v4) and text_entry (13v12), tie reasoning.
v10 root-causes the biggest remaining cluster (click_compound 10v17): styled links
render as '(alink)' -- their visible text is a child StaticText the name fallback never
read, so every link in a tab panel was indistinguishable and a guess was terminal.
Nameless rows now resolve by their own subtree text before the DOM class hint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Scripted exact-match fastpath + exact-name guard moved the wins where near-miss clicks
were terminal: drag 8/13 vs their 6/13, spatial 7/13 vs their 4/13, at 4.8s and zero
false claims. The remaining systematic gap is runway, not ability: 21 of 54 losses die
exactly at the 12-step cap, all in the email/forms/long-flow cluster their 44s median
happily outlasts. v9 sweeps with a 24-step cap (wins end at median 3 steps, so the win
path pays nothing) plus progressive vision -- any episode that burns 6 actions gets a
screenshot every turn thereafter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Ablation on the full 125: multi-action alone 50.4% at 4.3s (faster than the v5 base it
improves on -- fewer LLM calls is both the speed and the accuracy win); adaptive vision
adds +6.4 points for 301 screenshot calls that only fire when stuck or the goal reads
spatial, so the median win pays +0.3s. circle-center fell in one step once the model
could see the circle. v7v (vision-only) completes the ablation table.
v8 queued: scripted exact-match fast path (the product's own scripted-then-model shape)
-- when the goal's quoted target is visible as exactly one row, click it with no model
call. The tab/section losses are all terminal near-miss clicks on a goal-named link,
and the fastest correct click is the one no model gets to second-guess.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Controlled (view-only difference): our v5 beats the browser-use-shaped arm 48.0% to
36.0% at 2.6x the speed with fewer tokens; ahead in 6 categories, tied in 2, one email
task behind inside single-seed noise. Whole-stack: the real browser-use agent scores
69.6% -- but at 44.5s median (10x ours) and 16 false success claims (12.8% of tasks)
against our zero across 875 episodes. Their rate edge is loop machinery (vision steps,
multi-action, raw JS-evaluate), not perception; the same evaluate hatch that buys the
rate produces the confident wrongness. Next frontier ranked by measured value:
screenshot-conditioned steps, multi-action turns, a VERIFIED evaluate primitive.
v5's eval-memory prompt is the champion arm; v6's loop-breaker nudge measured inside
noise (46.4%) -- recorded, kept, not adopted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
v3 (44.8%, beats the browser-use-shaped arm's 36.0% by 8.8 points at 3x the speed):
page-text panel, select options on the row, input twins exempt from dedupe. v4 queued:
DOM-attribute names for nameless icons and deep sibling context, which turn the email
suite's unlabeled trash/star icons into addressable rows.
Product fix landed with tests: interactiveRanking.ts collapsed adjacent same-named
inputs and adjacent nameless icons as twins -- measured on MiniWoB, that single
collapse cost enter-password and the whole email suite. Inputs and nameless rows are
now exempt; named icon+label pairs still collapse. 17/17 tests, tsc clean.
ARENA.md carries the full scoreboard, per-category verdicts, what was ingested from
whom, and the honest gaps (drag and vision-less spatial) neither stack solves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
125 tasks, one external grader (WOB_REWARD_GLOBAL through BrowserGym), one recorder
book, agents never grade themselves. Arms: our shipped perception ported line-for-line
(deterministic + LLM), a browser-use-shaped flat-dump arm on the same model, the real
browser-use agent attached over CDP to the same seeded page, and Stagehand wiring.
Techniques already ingested from the competitor diffs, each landing measured points:
clickable-but-unroled elements (canvas/svg -- their flat dump solved spatial tasks
purely because the canvas was in it), coordinate actions (our click_point equivalent),
input twins exempt from dedupe (two same-named password boxes are two fields), select
options rendered on the row, and the page-text panel (our BrowserGetText equivalent).
v1 25.6% -> v2 32.0% on identical tasks/model, wall clock down 3.8s -> 3.2s.
Supervision is out-of-process: SIGALRM exceptions got swallowed inside stack retry
loops twice, so a watcher kills the process tree when the recorder stops moving and
resumes exactly the missing tasks. 502-dead LLM lanes book as infra, never as skill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Every other measurement in this session, ours and the competitors', was scored by a
harness I wrote and could therefore flatter. MiniWoB's reward comes from the task
definition, so it is ground truth nobody in this repo controls.
Working setup recorded because the obvious one fails: pypi browsergym cannot build
greenlet 3.0.3 on py3.13, so it needs a 3.12 venv, plus miniwob-plusplus's HTML served
locally. 341 envs register (125 MiniWoB, 215 AssistantBench, 0 WebArena -- that one needs
self-hosted sites).
The result reframes the competitive work: a ~20-line deterministic axtree heuristic with
NO model solves 5/15 = 33% at 1.44s median and zero cost. Published agent scores sit at
70-90%, so the real question for any model-driven agent, ours included, is what it buys
over the free baseline at 10-40x the latency.
miniwob_baseline.py takes task names as argv so a real policy can be dropped in for a
like-for-like number on the same 15 or all 125.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five of nine installed and driven; three produced comparable numbers. playwright-mcp
snapshots in ~130ms median and finds editables on all five sites including both ACE
editors, which puts it alongside agent-browser (~50ms) and against our
BrowserFindComposer's 13,991ms median.
The three that did not run are recorded with the reason, because 'could not test' is a
result: stagehand parses modelName as provider/model on the first slash and every id our
9Router exposes contains one (0 of 27 are slash-free), so it resolves the wrong provider
-- a config collision with our lane, not a defect, and its browser half worked fine.
OpenCLI needs a hand-installed Chrome extension. Skyvern is a Postgres deployment.
browsergym cannot build greenlet on py3.13.
Also noted: playwright-mcp's fill column reflects my own JS, not their element finder, so
only its snapshot times and editable counts are quoted as signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Numbers from this box, with the unverifiable ones marked as such.
The result that changes priorities: a deterministic accessibility-tree snapshot
(agent-browser, Rust CLI, no LLM) perceives in ~50ms median against our
BrowserFindComposer's 13,991ms median / 30,003ms p95 over 119 calls, and fills+verifies
onlinegdb in 326ms -- a site our agent cannot reach at all. Also fills w3schools (810ms)
and regex101 (31ms). That is a 40-450x perception gap with no model in the loop.
Also recorded: all nine claimed repos verified to exist with matching stars and
AGPL-compatible licences; browser-use benchmarked as an agent over 43 trials (we are
faster on every site we reach, 9.8s vs 35.7s median, but our arm ran with its fallback
disabled by dry-run so reach is not comparable); direct CDP evidence that deepl serves
our browser a Cloudflare challenge while browser-use's fresh-per-run profile is never
flagged; OpenCLI's API-first-with-browser-fallback design as the route to a learned path
that actually replays; and why Polar was not benchmarked (their ToS forbids using the
product to build a competing one, and their 98.0 concatenates two benchmarks and appears
on neither leaderboard).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wait was a bare `timeout=300.0` at the call site: invisible, unchangeable, and
longer than WALL_BUDGET_S (180s) itself, so a single prompt could outlive the whole
run's budget.
Measured 2026-08-08. deepl bot-detected the browser profile, the agent correctly
refused to solve the challenge ("handing to the user, not solving it") and asked for
help via RequestHumanIntervention. Headless, nobody answered, so it burned the full
306s and then denied -- the identical verdict it can reach instantly. That single
wait consumed the entire 420s task budget and was the whole of what looked like a
"249s spawn stall" while profiling. Cron runs, scheduled agents, CI and benchmarks
all sit in exactly this position.
Two changes, neither of which weakens the gate:
- ws_manager.has_listener(session_id) reports whether ANY socket would receive the
session's events, reading the same two lists send_to_session broadcasts to so it
cannot drift from where messages actually go.
- p_request_browser_approval checks it BEFORE building a request, and declines with
an honest reason when no UI is attached. The decision is unchanged (deny); only
the five minutes of waiting for it are gone.
The timeout is now P_APPROVAL_TIMEOUT_S, overridable via OSW_APPROVAL_TIMEOUT_S for
automation contexts that want a different budget.
A human at the keyboard sees no change: with a socket attached the request is sent
and awaited exactly as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A trial whose card was reaped before the readback could not be verified either way,
and excluding those rows was too charitable: it flipped openswarm 60.9% -> 100% and
McNemar p=0.02 -> 1.0 purely on my read timing. The backend's own dryrun-report
(composer/textboxes/filled) is the same bar the card readback applies, so it stands
in when the card is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Accumulating and resumable: a 95% CI narrow enough to separate two systems needs tens
of trials per site per arm, which is hours, so rows append to results/vs_raw.jsonl and
the report is computed from everything on disk. `vs_bench.py report` re-reads without
running anything.
Paired by construction: both arms run the same site back-to-back in one iteration, so
a site that changes mid-day (deepl started redirecting to a locale path during this
work) moves BOTH arms rather than one. Discordant pairs are tested with McNemar exact,
which is the right test for paired binary outcomes at small discordant counts, and
proportions use Wilson rather than the normal approximation -- at 20/20 the naive
interval is [100%, 100%], a certainty no 20-trial sample carries.
Three confounds found while building it, each of which had produced a wrong number:
1. The stop directive cannot be shared. "Do NOT submit" trips OpenSwarm's is_readonly()
and the send-script declines outright ("read-only directive in user request"), so
our arm measured 0/2 while browser-use scored 2/2 on the identical prompt. That is
the prompt disabling one side, not a capability gap. OpenSwarm now gets the bare task
and is held to reach-only by the backend's own OSW_SENDSCRIPT_DRYRUN=1; browser-use
has no dry-run mode, so its constraint stays in the prompt. Verified at the
classifier: is_readonly(BU)=True, is_readonly(OSW)=False.
2. Verification must be scoped to the cards a trial CREATED. A dashboard accumulates one
card per run and they persist in the saved layout; reading "all cards" checked seven
stale pages, and a regex101 run graded 0/1 because its own card had been reaped while
leftovers answered instead.
3. browser-use's readback goes through ITS session (get_tabs + get_or_create_cdp_session)
rather than a second websocket. Every earlier verifier bug -- hardcoded port, ws:// vs
http://, page-vs-webview target filtering -- lived in hand-rolled target discovery.
The browser is deliberately NOT held constant. browser-use cannot screenshot an Electron
webview and falls back to the app's page target on 100% of steps (49/49 measured), so a
shared browser blinds it; each side runs in the browser it was built for and that is
recorded as a caveat rather than hidden.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Added to compare browser-use against us with the BROWSER held constant, and the
result is that it cannot be held constant, which is worth writing down.
What works: with the port exposed, browser-use attaches, sees the browser cards
(they are CDP targets of type 'webview'), navigates one, and completes a task that
verifies independently. Cards persist across restarts because they live in the saved
dashboard layout, so there is something to attach to.
What does not: browser-use's screenshot path cannot capture a webview and falls back
to the app's own 'page' target -- 49 fallbacks in 49 agent steps, i.e. it drove the
web page while LOOKING at the OpenSwarm UI for every single step. A comparison run
that way handicaps the rival for an infrastructural reason and measures neither
agent, so the shared-browser numbers are not publishable as a fair head-to-head.
Two hazards found the hard way, both worth knowing before anyone repeats this:
- browser-use filters CDP targets to page/tab, so 'webview' is invisible to it, and
Electron refuses Target.createTarget ("Not supported") so it cannot open its own.
Pointed at Electron unrestricted it therefore grabs the app's page target and
navigates the OpenSwarm UI away, killing the renderer: the next sweep failed
infra_no_renderer 1/1 because of exactly that.
- An external agent must never be allowed to kill the shared session. Doing so after
trial 1 made trials 2-15 fail ConnectError, which a naive summary published as
"browser-use REACH 4/15" -- ten rows of my own teardown scored against the agent.
Off by default: an open debugging port is a local attack surface, and this only
exists for measurement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
