mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-09-08 02:37:45 +02:00

Files

T

ciregenzandClaude Fable 5 9db052cf56 arena: clean bu-real-opus5 cell lands -- 66.4% at 28.3s; temperature lever dead on Claude 5

The untainted re-run replaces the asterisked 48%: their best-model cell is 16 points
behind ours at 4.5x the wall. Claude-5 lanes reject the temperature param outright --
v17's 400-loop root-caused, field omitted; decode variance now attacked at the action
layer (fill-verify, verify-terminal) instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ

2026-08-12 04:25:07 -07:00

data

arena: central data store manifest -- one directory holds every episode, screenshot and raw log

2026-08-10 21:16:05 -07:00

ARENA.md

arena: clean bu-real-opus5 cell lands -- 66.4% at 28.3s; temperature lever dead on Claude 5

2026-08-12 04:25:07 -07:00

bu_real.py

arena: assistantbench import quarantined -- HF datasets machinery was corrupting playwright process-wide

2026-08-11 02:12:03 -07:00

diffs.py

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

llm_policy.py

arena: v17 -- variance attacked at the source, with overfitting guards built in

2026-08-12 02:18:43 -07:00

perception.py

arena: v9 63.2% at 4.9s -- runway was the constraint; v10 fixes the last blindness

2026-08-10 04:13:17 -07:00

policies.py

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

ranking.py

arena: v3/v4 ingest rounds + the dedupe bug those rounds exposed in the shipped ranker

2026-08-09 16:17:11 -07:00

README.md

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

recorder.py

arena: v7 lands multi-action + adaptive vision -- 56.8% at 4.8s, still zero false claims

2026-08-10 01:34:03 -07:00

report.py

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

run.py

arena: assistantbench import quarantined -- HF datasets machinery was corrupting playwright process-wide

2026-08-11 02:12:03 -07:00

sh_driver.mjs

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

sh_real.py

arena: every browser stack scored by MiniWoB itself, plus the first two ingest wins

2026-08-09 14:41:02 -07:00

supervisor.py

arena: live-web episodes get one process each -- a single hang was poisoning whole rounds

2026-08-10 21:41:26 -07:00

tasks.py

arena: multi-model ledger -- ours leads the same-model pair on every axis at every tier tested

2026-08-10 16:51:25 -07:00

README.md

MiniWoB arena

Every browser stack we care about, scored on the same 125 MiniWoB tasks by MiniWoB itself. Nothing in this directory decides success; the reward comes from the task's own JS (WOB_REWARD_GLOBAL), read through BrowserGym. That is the property none of our other suites have, and it is why cross-stack claims ("ours is better/worse than X") should cite THIS data.

Arms

arm	what it is	entrypoint
`flat`	no-LLM floor: flat axtree, first label match	`run.py --arm flat`
`openswarm`	no-LLM port of our shipped perception + action ladder	`run.py --arm openswarm`
`bu`	LLM given a browser-use-shaped flat axtree dump	`run.py --arm bu`
`osw-llm`	same LLM given our ranked/deduped/capped element menu	`run.py --arm osw-llm`
`bu-real`	the actual browser-use agent, attached over CDP	`bu_real.py`
`sh-real`	the actual Stagehand agent, attached over CDP	`sh_real.py`

bu vs osw-llm is the controlled experiment (same model, same action layer, only the page view differs). bu-real/sh-real are the shipping competitors, whole-stack.

Ground rules

One recorder (recorder.py): every episode appends to data/all.jsonl; screenshots under data/shots/<run>/<arm>/<task>-s<seed>/. Reruns supersede by started_at; nothing is rewritten.
Agents never grade themselves. claimed_success vs success is recorded precisely to count false-success claims per arm.
Infra failures (error_class starting infra) are excluded from rates but always reported.
ranking.py must stay line-for-line with frontend/src/shared/interactiveRanking.ts; if either changes, change both.

Running

# serve MiniWoB HTML once:  cd miniwob-plusplus/miniwob/html && python3 -m http.server 8099
MINIWOB_URL=http://localhost:8099/miniwob/ python run.py --arm osw-llm --tasks all --seeds 1
python report.py            # scoreboard; --md ARENA.md for the markdown version
python diffs.py --ours osw-llm --theirs bu-real   # evidence trail for every loss

Needs the browsergym venv (Python 3.12 — 3.13 cannot build greenlet 3.0.3).