WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-08-30 19:59:38 +02:00
Code Issues Packages Projects Releases Wiki Activity

arena: v40 union FAILS confirm (interaction regression) -- pairwise isolation running (v40c first)

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
ciregenz
2026-08-16 02:33:55 -07:00
co-authored by Claude Fable 5
parent 7581706f5f
commit 76eada6987
2 changed files with 18 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
e2e/browser-v3/arena/ARENA.md
+8
View File
@@ -619,6 +619,14 @@ champion re-registered as the UNION (escape_token + table_md + mutation_diff). P
a combined CONFIRMATION pilot (union of the three target sets + 12 controls) runs before the
champion 3-seed — passed-alone does not guarantee passed-together (interaction risk).
CONFIRM VERDICT (2026-08-16): **interaction regression — gate FAILS.** Targets 11/20 but
controls 10/12 (login-popup pair lost) and book-flight regressed vs BOTH individual pilots
(won under v38-alone and v39-alone, lost combined). The union does not compose. Isolation
running pairwise, cheapest discriminator first: v40c = table_md+mutation_diff (no escape) on
the same 32-task set — if book-flight loses there, the conflict is the two prompt-additive
mechanisms crowding each other; if it wins, escape_token is implicated. No promotion until a
composing set passes with controls 12/12.
## Open-source / SOTA reference points (2026 leaderboards, for honest comparison)
- WebArena: SOTA WebTactix/DeepSeek-v3.2 74.3%; frontier models 64-68%; human 78. (leaderboard.steel.dev)
e2e/browser-v3/arena/llm_policy.py
+10
View File
@@ -999,6 +999,16 @@ def build(name: str, model: str = "", endpoint: str = "", **_: Any) -> Any:
scripted_drag=True, auto_complete=True, som=False,
native_pickers=True, verify_terminal=True, post_mouse_vision=True,
multi_cap=6, fill_verify=True, **v17)
if name == "osw-llm-v40c": # pair isolate: table_md + mutation_diff (no escape token)
c = dict(v7, system=OSW_SYSTEM_V8 + OSW_SYSTEM_V9_WIDGETS + OSW_SYSTEM_V16 + OSW_SYSTEM_V30,
max_tokens=800)
return OpenSwarmLlmPolicy(name=name, multi=True, vision="progressive", fastpath=True,
scripted_drag=True, auto_complete=True, som=False,
native_pickers=True, verify_terminal=True, post_mouse_vision=True,
multi_cap=6, fill_verify=True, dispatch=True, offscreen=True,
local_ctx=True, blocker_probe=True, suppress_wrappers=True,
force_unblock=True, native_js_fallback=True,
table_md=True, mutation_diff=True, **c)
if name == "osw-llm-v40": # CHAMPION: v35 stack + ALL passed ingestions (escape+table_md+mutation_diff)
v40 = dict(v7, system=OSW_SYSTEM_V8 + OSW_SYSTEM_V9_WIDGETS + OSW_SYSTEM_V16 + OSW_SYSTEM_V30
+ OSW_SYSTEM_V36, max_tokens=800)