mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-09-13 21:27:41 +02:00
ciregenzandClaude Fable 5
9456402524
bu_real.score() reads WOB_REWARD_GLOBAL, which does not exist on AssistantBench pages, so browser-use's 0.000 is a harness bug not a performance result; ours (env-reward path) is likely fine but both must be re-scored through the AssistantBench reward before any claim. Caught by user skepticism -- exactly the tunnel-vision guard the method depends on. MiniWoB numbers are unaffected (that path is correct there). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
MiniWoB arena
Every browser stack we care about, scored on the same 125 MiniWoB tasks by MiniWoB itself.
Nothing in this directory decides success; the reward comes from the task's own JS
(WOB_REWARD_GLOBAL), read through BrowserGym. That is the property none of our other suites
have, and it is why cross-stack claims ("ours is better/worse than X") should cite THIS data.
Arms
| arm | what it is | entrypoint |
|---|---|---|
flat |
no-LLM floor: flat axtree, first label match | run.py --arm flat |
openswarm |
no-LLM port of our shipped perception + action ladder | run.py --arm openswarm |
bu |
LLM given a browser-use-shaped flat axtree dump | run.py --arm bu |
osw-llm |
same LLM given our ranked/deduped/capped element menu | run.py --arm osw-llm |
bu-real |
the actual browser-use agent, attached over CDP | bu_real.py |
sh-real |
the actual Stagehand agent, attached over CDP | sh_real.py |
bu vs osw-llm is the controlled experiment (same model, same action layer, only the page view
differs). bu-real/sh-real are the shipping competitors, whole-stack.
Ground rules
- One recorder (
recorder.py): every episode appends todata/all.jsonl; screenshots underdata/shots/<run>/<arm>/<task>-s<seed>/. Reruns supersede bystarted_at; nothing is rewritten. - Agents never grade themselves.
claimed_successvssuccessis recorded precisely to count false-success claims per arm. - Infra failures (
error_classstartinginfra) are excluded from rates but always reported. ranking.pymust stay line-for-line withfrontend/src/shared/interactiveRanking.ts; if either changes, change both.
Running
# serve MiniWoB HTML once: cd miniwob-plusplus/miniwob/html && python3 -m http.server 8099
MINIWOB_URL=http://localhost:8099/miniwob/ python run.py --arm osw-llm --tasks all --seeds 1
python report.py # scoreboard; --md ARENA.md for the markdown version
python diffs.py --ours osw-llm --theirs bu-real # evidence trail for every loss
Needs the browsergym venv (Python 3.12 — 3.13 cannot build greenlet 3.0.3).