arena: RETRACT the AssistantBench numbers -- bu_real scored it with a MiniWoB-only global

bu_real.score() reads WOB_REWARD_GLOBAL, which does not exist on AssistantBench pages,
so browser-use's 0.000 is a harness bug not a performance result; ours (env-reward
path) is likely fine but both must be re-scored through the AssistantBench reward
before any claim. Caught by user skepticism -- exactly the tunnel-vision guard the
method depends on. MiniWoB numbers are unaffected (that path is correct there).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
ciregenz
2026-08-12 10:30:54 -07:00
co-authored by Claude Fable 5
parent 5542039c56
commit 9456402524
+4 -4
View File
@@ -61,10 +61,10 @@ change goals/values, not just RNG). The ~6-point pass@1-vs-pass@2 gap is decode
Claude-5 lanes give no temperature control over; the remaining stable losses are the four
engineering clusters (long forms, pixel precision, console emulation, stateful games).
**AssistantBench (live web, their question_scorer, sonnet-5): ours mean accuracy 0.050 on 14
clean episodes (29/33 attempted, 2 nonzero answers)** -- consistent with the suite's brutal
public SOTA (~25% for far heavier research agents); browser-use's run queued on identical
protocol. Live-web infra losses remain high for both stacks; numbers here are directional.
**AssistantBench (live web, sonnet-5): INVALID pending re-run.** ours read 0.050; browser-use read
0.000 -- but the bu_real scorer was reading MiniWoB's WOB_REWARD_GLOBAL, which does not exist on
AssistantBench pages, so its 0.000 is a scoring bug, not a result. Both arms need re-scoring
through the AssistantBench env reward before any AssistantBench claim is made. Do not cite these.
v16 (verify-terminal, look-act-look, rapid-fire, sub-step confirm) FINAL: 82.1% on clean episodes
-- statistically tied with v14, but the fixes hit their targets: email 10/10 (their best 6/10),