mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-09-23 18:14:53 +02:00

Files

T

ciregenzandClaude Fable 5 228d65c9e9 arena: CompWoB wired -- the generalization referee, with zero new scoring code

101 composed tasks registered through BrowserGym's own MiniWoB task class pointed at
the composed pages, so the reward path is the already-canary-validated page-owned
machinery; task ids discovered from the served directory so registry drift is
impossible. Canary passed (real composed goal, reward global live). This is the
benchmark built to expose memorization (specialists fall 95->61 on it) -- our v22 and
browser-use both sweep all 101 under identical isolated protocol.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ

2026-08-12 19:08:11 -07:00

data

arena: central data store manifest -- one directory holds every episode, screenshot and raw log

2026-08-10 21:16:05 -07:00

ARENA.md

arena: GOAL MET -- v22 at 92.0% single run, 90.8% two-seed mean, zero false claims

2026-08-12 17:59:28 -07:00

bu_real.py

arena: v21 -- the retry that could not truncate

2026-08-12 15:47:05 -07:00

compwob.py

arena: CompWoB wired -- the generalization referee, with zero new scoring code

2026-08-12 19:08:11 -07:00

diffs.py

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

llm_policy.py

arena: v22 -- truncation made structurally impossible (action-first reply order)

2026-08-12 15:50:17 -07:00

perception.py

arena: the "forms" deficit was never forms -- it was viewport-blind perception

2026-08-12 10:59:41 -07:00

policies.py

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

ranking.py

arena: the "forms" deficit was never forms -- it was viewport-blind perception

2026-08-12 10:59:41 -07:00

README.md

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

recorder.py

arena: v7 lands multi-action + adaptive vision -- 56.8% at 4.8s, still zero false claims

2026-08-10 01:34:03 -07:00

report.py

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

run.py

arena: CompWoB wired -- the generalization referee, with zero new scoring code

2026-08-12 19:08:11 -07:00

sh_driver.mjs

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

sh_real.py

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

supervisor.py

arena: live-web episodes get one process each -- a single hang was poisoning whole rounds

2026-08-10 21:41:26 -07:00

tasks.py

arena: CompWoB wired -- the generalization referee, with zero new scoring code

2026-08-12 19:08:11 -07:00

README.md

MiniWoB arena

Every browser stack we care about, scored on the same 125 MiniWoB tasks by MiniWoB itself. Nothing in this directory decides success; the reward comes from the task's own JS (WOB_REWARD_GLOBAL), read through BrowserGym. That is the property none of our other suites have, and it is why cross-stack claims ("ours is better/worse than X") should cite THIS data.

Arms

arm	what it is	entrypoint
`flat`	no-LLM floor: flat axtree, first label match	`run.py --arm flat`
`openswarm`	no-LLM port of our shipped perception + action ladder	`run.py --arm openswarm`
`bu`	LLM given a browser-use-shaped flat axtree dump	`run.py --arm bu`
`osw-llm`	same LLM given our ranked/deduped/capped element menu	`run.py --arm osw-llm`
`bu-real`	the actual browser-use agent, attached over CDP	`bu_real.py`
`sh-real`	the actual Stagehand agent, attached over CDP	`sh_real.py`

bu vs osw-llm is the controlled experiment (same model, same action layer, only the page view differs). bu-real/sh-real are the shipping competitors, whole-stack.

Ground rules

One recorder (recorder.py): every episode appends to data/all.jsonl; screenshots under data/shots/<run>/<arm>/<task>-s<seed>/. Reruns supersede by started_at; nothing is rewritten.
Agents never grade themselves. claimed_success vs success is recorded precisely to count false-success claims per arm.
Infra failures (error_class starting infra) are excluded from rates but always reported.
ranking.py must stay line-for-line with frontend/src/shared/interactiveRanking.ts; if either changes, change both.

Running

# serve MiniWoB HTML once:  cd miniwob-plusplus/miniwob/html && python3 -m http.server 8099
MINIWOB_URL=http://localhost:8099/miniwob/ python run.py --arm osw-llm --tasks all --seeds 1
python report.py            # scoreboard; --md ARENA.md for the markdown version
python diffs.py --ours osw-llm --theirs bu-real   # evidence trail for every loss

Needs the browsergym venv (Python 3.12 — 3.13 cannot build greenlet 3.0.3).