WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-09-30 05:24:50 +02:00
Code Issues Packages Projects Releases Wiki Activity

arena: WebArena FINAL = 13 strict/15% + 29.2 partial/34% (was falsely 0/3.83) -- ~7-8x true-credit correction from instrument fixes

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
ciregenz
2026-08-16 18:47:42 -07:00
co-authored by Claude Fable 5
parent 728ccab6b6
commit c44308ed94
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
e2e/browser-v3/arena/ARENA.md
+4 -1
View File
@@ -212,7 +212,10 @@ all claimed_success=False. This is the biggest correction of the effort and a te
standing method's 'presume instrument bug behind a clean zero' rule -- we spent cycles booking a
false 0 as capability. Context: GPT-4 generic ~14% on EASIER all-sites WebArena; ~18% generic on
the HARD reddit+gitlab partition is competitive. Clause 8 (>=65) still OPEN, but the record is now
honest. (Number finalizes at 100/100; interim at 57/100 = 18% strict / 36% partial.)
honest. FINAL (85/100 clean, 15 infra-excluded): **13 strict = 15% | partial-sum 29.2 = 34.3%
avg credit | 0 false | 49s median** -- vs the wrongly-booked 0 strict / 3.83 (a ~7-8x correction
in true credit, purely from removing our own instrument suppression). 15% generic strict on the
HARD reddit+gitlab partition (GPT-4 ~14% on easier all-sites) is competitive; clause 8 >=65 OPEN.
## WebArena Verified — first realistic-workflow measurement (100-task seeded partition)