mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-08-26 22:42:23 +02:00
arena: RETRACT the AssistantBench numbers -- bu_real scored it with a MiniWoB-only global
bu_real.score() reads WOB_REWARD_GLOBAL, which does not exist on AssistantBench pages, so browser-use's 0.000 is a harness bug not a performance result; ours (env-reward path) is likely fine but both must be re-scored through the AssistantBench reward before any claim. Caught by user skepticism -- exactly the tunnel-vision guard the method depends on. MiniWoB numbers are unaffected (that path is correct there). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
co-authored by
Claude Fable 5
parent
5542039c56
commit
9456402524
@@ -61,10 +61,10 @@ change goals/values, not just RNG). The ~6-point pass@1-vs-pass@2 gap is decode
|
||||
Claude-5 lanes give no temperature control over; the remaining stable losses are the four
|
||||
engineering clusters (long forms, pixel precision, console emulation, stateful games).
|
||||
|
||||
**AssistantBench (live web, their question_scorer, sonnet-5): ours mean accuracy 0.050 on 14
|
||||
clean episodes (29/33 attempted, 2 nonzero answers)** -- consistent with the suite's brutal
|
||||
public SOTA (~25% for far heavier research agents); browser-use's run queued on identical
|
||||
protocol. Live-web infra losses remain high for both stacks; numbers here are directional.
|
||||
**AssistantBench (live web, sonnet-5): INVALID pending re-run.** ours read 0.050; browser-use read
|
||||
0.000 -- but the bu_real scorer was reading MiniWoB's WOB_REWARD_GLOBAL, which does not exist on
|
||||
AssistantBench pages, so its 0.000 is a scoring bug, not a result. Both arms need re-scoring
|
||||
through the AssistantBench env reward before any AssistantBench claim is made. Do not cite these.
|
||||
|
||||
v16 (verify-terminal, look-act-look, rapid-fire, sub-step confirm) FINAL: 82.1% on clean episodes
|
||||
-- statistically tied with v14, but the fixes hit their targets: email 10/10 (their best 6/10),
|
||||
|
||||
Reference in New Issue
Block a user