WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-08-30 11:49:50 +02:00
Code Issues Packages Projects Releases Wiki Activity

arena: v36 PASSES (3/8 incl click-collapsible pair; first ingestion win) -> v40 champion; v37 deferred to chore-scale

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
ciregenz
2026-08-16 01:30:09 -07:00
co-authored by Claude Fable 5
parent 25892f30b1
commit 2e07047ff7
2 changed files with 18 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
e2e/browser-v3/arena/ARENA.md
+9
View File
@@ -586,6 +586,15 @@ replacement is strictly safer than the click it replaces).
Order: v37 then v36, sequential, immediately after the Claude chore column finishes (auto-chained).
VERDICTS (2026-08-16):
- **v36 escape token: PASS — 3/8 targets flipped (click-collapsible-2 AND -nodelay, both
never-won hard-cluster residents, + click-dialog-2_login-user-popup), controls 12/12.** First
competitor ingestion to clear its gate. PROMOTED: v40 champion = v35 stack + escape_token.
- v37 note scratchpad: fail on short-horizon proxies (0/8; controls 12/12 — mechanism safe,
never harmful). Honest reading: the gate words rarely trigger on these tasks and short tasks
don't need durable memory; the REAL test is chore-scale aggregation, which needs an uncapped
chore rerun (expensive, deferred). Flag stays off in champion until that evidence exists.
## Open-source / SOTA reference points (2026 leaderboards, for honest comparison)
- WebArena: SOTA WebTactix/DeepSeek-v3.2 74.3%; frontier models 64-68%; human 78. (leaderboard.steel.dev)
e2e/browser-v3/arena/llm_policy.py
+9
View File
@@ -999,6 +999,15 @@ def build(name: str, model: str = "", endpoint: str = "", **_: Any) -> Any:
scripted_drag=True, auto_complete=True, som=False,
native_pickers=True, verify_terminal=True, post_mouse_vision=True,
multi_cap=6, fill_verify=True, **v17)
if name == "osw-llm-v40": # CHAMPION: v35 stack + v36 escape token (first passed ingestion)
v40 = dict(v7, system=OSW_SYSTEM_V8 + OSW_SYSTEM_V9_WIDGETS + OSW_SYSTEM_V16 + OSW_SYSTEM_V30
+ OSW_SYSTEM_V36, max_tokens=800)
return OpenSwarmLlmPolicy(name=name, multi=True, vision="progressive", fastpath=True,
scripted_drag=True, auto_complete=True, som=False,
native_pickers=True, verify_terminal=True, post_mouse_vision=True,
multi_cap=6, fill_verify=True, dispatch=True, offscreen=True,
local_ctx=True, blocker_probe=True, suppress_wrappers=True,
force_unblock=True, native_js_fallback=True, escape_token=True, **v40)
if name == "osw-llm-v39": # v35 + mutation-diff action feedback (Agent-E observer, text-side)
v39 = dict(v7, system=OSW_SYSTEM_V8 + OSW_SYSTEM_V9_WIDGETS + OSW_SYSTEM_V16 + OSW_SYSTEM_V30,
max_tokens=800)