From 945640252409bd0e78c3ec00e16bbe8d76a4890b Mon Sep 17 00:00:00 2001 From: ciregenz Date: Wed, 12 Aug 2026 10:30:54 -0700 Subject: [PATCH] arena: RETRACT the AssistantBench numbers -- bu_real scored it with a MiniWoB-only global bu_real.score() reads WOB_REWARD_GLOBAL, which does not exist on AssistantBench pages, so browser-use's 0.000 is a harness bug not a performance result; ours (env-reward path) is likely fine but both must be re-scored through the AssistantBench reward before any claim. Caught by user skepticism -- exactly the tunnel-vision guard the method depends on. MiniWoB numbers are unaffected (that path is correct there). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ --- e2e/browser-v3/arena/ARENA.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/e2e/browser-v3/arena/ARENA.md b/e2e/browser-v3/arena/ARENA.md index 8738578f..915053ed 100644 --- a/e2e/browser-v3/arena/ARENA.md +++ b/e2e/browser-v3/arena/ARENA.md @@ -61,10 +61,10 @@ change goals/values, not just RNG). The ~6-point pass@1-vs-pass@2 gap is decode Claude-5 lanes give no temperature control over; the remaining stable losses are the four engineering clusters (long forms, pixel precision, console emulation, stateful games). -**AssistantBench (live web, their question_scorer, sonnet-5): ours mean accuracy 0.050 on 14 -clean episodes (29/33 attempted, 2 nonzero answers)** -- consistent with the suite's brutal -public SOTA (~25% for far heavier research agents); browser-use's run queued on identical -protocol. Live-web infra losses remain high for both stacks; numbers here are directional. +**AssistantBench (live web, sonnet-5): INVALID pending re-run.** ours read 0.050; browser-use read +0.000 -- but the bu_real scorer was reading MiniWoB's WOB_REWARD_GLOBAL, which does not exist on +AssistantBench pages, so its 0.000 is a scoring bug, not a result. Both arms need re-scoring +through the AssistantBench env reward before any AssistantBench claim is made. Do not cite these. v16 (verify-terminal, look-act-look, rapid-fire, sub-step confirm) FINAL: 82.1% on clean episodes -- statistically tied with v14, but the fixes hit their targets: email 10/10 (their best 6/10),