From c69e71e7be08cb422767f8fa8925bf14e12b0546 Mon Sep 17 00:00:00 2001 From: ciregenz Date: Tue, 11 Aug 2026 04:20:51 -0700 Subject: [PATCH] arena: restore the caveats section the definitive rewrite silently dropped, add load-taint window The confounders disclosure (claim asymmetry, co-run load, seed noise) vanished in the full-file rewrite -- exactly the content that must never be lost silently. Restored, plus the 2026-08-11 host-load window: bu-real-opus5's cell asterisked pending clean re-run, tainted v16 attempts discarded. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ --- e2e/browser-v3/arena/ARENA.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/e2e/browser-v3/arena/ARENA.md b/e2e/browser-v3/arena/ARENA.md index 3cf74dd0..34415ba8 100644 --- a/e2e/browser-v3/arena/ARENA.md +++ b/e2e/browser-v3/arena/ARENA.md @@ -117,6 +117,19 @@ Staged, evidence attached, in priority order (each mapped to its arena win): 5. Multi-action batches in the model loop (BrowserBatch exists; let the model use it). 6. Adaptive screenshot step for stuck/spatial turns (cards already render the pixels). +## Measurement caveats (confounders, stated) + +- **False-claim asymmetry**: browser-use's API surfaces an explicit success claim and ours never + claims at all, so our 0 is partly structural -- read "ours never lies" as "ours never claims". +- **Host-load taint window (2026-08-11 ~00:30-04:00)**: the machine ran at load ~100 (user apps: + ~88 Chrome processes, VS Code, a dev server). bu-real-opus5's 48.0% cell and two v16 launch + attempts fall inside it; the v16 attempts were discarded for a clean re-run and the + bu-real-opus5 cell carries an asterisk pending re-run -- fairness cuts both ways. Measurement + now auto-pauses when load exceeds the guard. +- **Co-run CPU load**: concurrent sweeps never share an LLM lane but do share the machine; + champion numbers were re-verified solo with an external stopwatch. +- **Single seed per cell**: adjacent-variant differences under ±4 tasks are noise and labeled so. + ## Remaining gaps (named, not hidden) - forms 14/22 vs their 18/22 — long widget flows (autocomplete, multi-item select). Needs a