From 65006c6cd5c26a42bd4b7c8f103e1bd54eece697 Mon Sep 17 00:00:00 2001 From: ciregenz Date: Mon, 17 Aug 2026 17:53:51 -0700 Subject: [PATCH] arena: v48 plan-state NOT-A-LIFT at scale (completion wall confirmed non-scaffolding-movable, all 4 mechanisms) + book harness-vs-product distinction (arena numbers = benchmark harness, product write capability is higher) --- e2e/browser-v3/arena/ARENA.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/e2e/browser-v3/arena/ARENA.md b/e2e/browser-v3/arena/ARENA.md index 7cdbd816..017fd1c0 100644 --- a/e2e/browser-v3/arena/ARENA.md +++ b/e2e/browser-v3/arena/ARENA.md @@ -1155,3 +1155,22 @@ solved class. NEXT: re-measure verified compose-writes on the self-hosted sites comments/posts) using the send-script fill/verify/send/verify pattern -- reproducible, deterministic. Live-LinkedIn demonstration stays the user's to run via the product (real account + ToS + their authenticated session -- not something the benchmark harness should autonomously drive). + +## Two honest findings (2026-08-17) +1. v48 plan-state at SCALE (65 informative tasks) = NOT a real lift: 9 strict / 25.1 partial at + 47/65 clean, vs v42 base 13/38.2 (proportional ~9-10 strict) -- within noise, slightly below. + The small-pilot +1 did not hold. CONCLUSION across all 4 capability mechanisms (v45 read-your- + writes FLAT, v46 done-gate NEGATIVE, v47 scratchpad FLAT, v48 plan-state NOT-A-LIFT): WebArena + long-horizon COMPLETION is NOT scaffolding-movable on the generic benchmark harness at the + opus-5 ceiling. It needs model-planning-capability (fine-tuning, excluded) or non-generic + multi-agent. Memory/planning scaffolding settled as a non-lever, per the study's own criterion. +2. HARNESS vs PRODUCT (the load-bearing distinction): EVERY arena number measures the clean + BENCHMARK HARNESS (osw-llm-*), deliberately stripped for fair reproducible scoring. It is NOT + the browser-v3 PRODUCT, which carries production machinery the arena omits: verified-action + predicates, staged send-script (fill/verify/send/verify-cleared), delivery-check, prestaging, + playbooks. On reads/navigation the two are comparable; on WRITES the product is materially more + capable (its code proves compose-writes on X/Reddit/LinkedIn/Gmail) -- so the arena's 7-14% + verified-write number UNDERSTATES the product. Honest framing: the generic-agent RESEARCH + numbers (MiniWoB/CompWoB/WebArena/writes) are the harness; the PRODUCT's shipped write capability + is higher and measured elsewhere (live, via the app). Reproducible product-write measurement = + run the product agent against the self-hosted sites (future work; different stack).