The 6 'strict wins' previously reported were fractional checkpoint credits (5x0.667, 1x0.5)
mislabeled by the MiniWoB success predicate (reward>0). ARENA.md books the correction;
EpisodeRecord now carries an explicit strict bit so the conflation is unrepresentable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Single-step turns + full compressed history did not move one long-composition target
(controls unharmed). The 20-minute pilot killed the hypothesis before a 2-hour sweep
could -- exactly what the post-v24 rules were for. Next hypothesis, pre-registered:
inter-section settle pacing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ac/code
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Full-coverage fair pair after two harness retractions: their patient per-step
eval/memory loop owns long compositions (5-7-part 4/6 vs our 0/8) at 48s/8-false-claims
cost against our 17s/0. v25 ingests the mechanism pre-registered and gated (single-step
turns + compressed full history on >=4-clause goals only); pilot with controls running
per the new discipline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Ground truth: one CompWoB page (enter-text-second family) is broken upstream (clears
an input before creating it); isolate mode took remaining[:1] every round, so the
supervisor retried that one impossible page 201 straight times and 72 healthy tasks
starved unseen behind it. browser-use never received them. Supervisor now rotates
through remaining tasks and blacklists any that fail setup three times; their fair
completion run is live. Score one more for verify-the-verifier: both prior narratives
about their CompWoB failures were wrong, and the record now says so.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Symmetric audit: the composed pages' own genProblem loses an asset race and crashes
reset -- 1/101 times on our launch path, 73/101 on the CDP-port path, same pages,
same seed. Blaming their instrumentation was wrong and is withdrawn; both harness
paths now retry the reset (same courtesy both ways) and their CompWoB completion
sweep is rerunning for a genuine full-coverage number.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Docker Desktop proved uninitializable without GUI on this box (three engine rebuilds,
factory reset included); colima on macOS's Virtualization.framework started first try
and swallowed the 131GB of site images Docker Desktop could not load. Bring-up script
now targets the colima socket; shopping/admin/forum booting while gitlab downloads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
60.4% (-5): short tasks paid for the scaffolding, >=5-part stayed 0/8 through a third
mechanism. The cluster's invariance across runway/ordering/checklist localizes the
real need -- mid-episode replanning against current page state -- product-primitive
territory, same frontier as MiniWoB's hard-10. Champion for CompWoB remains v22/v23
at 65.3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Long-chain traces showed clean execution collapsing into scroll-flailing once the
agent lost its place. The goal is decomposed once into numbered clauses, rendered
every turn with the model's own last-reported position echoed back; content-agnostic
scaffolding, no task knowledge. Sweeping CompWoB against the 65.3 baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
browser-use cannot load 73 of 101 composed pages (instrumentation crash: 28-task
coverage ceiling, 2 more false claims on what loads). Our ordering fix won its exact
predicted tasks; long-chain traces isolate the true frontier as lost mid-chain
bookkeeping, to be attacked with a rendered clause checklist (v24).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
macOS silently deletes /tmp files by age: pyvenv.cfg vanished mid-sweep and 84 spawns
died import-less while site-packages survived -- the exact silent-error class the goal
names. Venv, serving trees and compwob discovery now live in ~/.cache/arena (durable);
servers rebound; registration re-verified (101). Episode data was never at risk: the
book lives in the repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
CompWoB's simple-pair autopsy: with "Ok" ambiguous, the fastpath clicked the SECOND
quoted target first, and order-enforcing pages fail terminally on that. It now acts
only on the first unsatisfied quoted target and otherwise defers to the model --
sequence is part of the task, structurally respected. Lands in v23's queued sweep.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Idempotent start+configure for shopping/shopping_admin/forum/gitlab per the official
environment_docker README, with port health checks; fires automatically the moment the
190GB image pipeline finishes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The stable game losses share one blindness: the page answers every action in its text
(Colder!, Too high, score counters) and the agent's memory recorded only its own
actions. History lines now carry the page-text delta each action caused -- a general
feedback mechanism, nothing task-shaped. Queued behind the CompWoB pair.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Clean-harness CompWoB opened 6/6 on 2-3-clause compositions and 0/3 on 7-8-clause
monsters whose traces show correct work hitting the step cap. Episode budget now grows
with clause count (feature-triggered from the goal text, capped at 3x) -- a general
long-instruction mechanism, not a benchmark tweak. Resume keeps the six clean wins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Two instrument bugs found before any number shipped: composed pages need the LEGACY
MiniWoB core (modern core terminated on the first click), and validate() string-
compares page.url so the '../' base-url trick killed every episode after step one --
boring URLs are robust URLs. Canary now clicks ONE then TWO with the episode alive.
The 0/17 zero from the broken setup is void; the pair sweep restarts clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
101 composed tasks registered through BrowserGym's own MiniWoB task class pointed at
the composed pages, so the reward path is the already-canary-validated page-owned
machinery; task ids discovered from the served directory so registry drift is
impossible. Canary passed (real composed goal, reward global live). This is the
benchmark built to expose memorization (specialists fall 95->61 on it) -- our v22 and
browser-use both sweep all 101 under identical isolated protocol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
25.6 -> 92.0 across 22 versions, all third-party-scored, zero benchmark-specific
logic, zero demos. Sixteen-plus points above the published generic-agent field and
inside the demonstration-trained band. The decisive gains were perception completeness
and structural impossibility fixes -- the design philosophy the user demanded,
measured all the way up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Action-first ordering (+3.2 over v20) proved the design-out-the-class principle:
truncation deaths ended, book-flight fell for the first time in twenty sweeps, forms
19/22, drag and email perfect, zero false claims at 8.5s median. Seed-43 running as
the cross-seed decider; the residue is the named product-primitive cluster plus an
empty-reply subclass whose structural close is reply prefill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Per the design-out-failure-classes principle: the action line now LEADS the reply and
the PLAN trails as optional commentary, so a truncated reply can only lose commentary,
never the action -- the entire empty-action loss class becomes unrepresentable. The
bare-prompt retry stays as defense-in-depth behind it. v21 (patch-only variant) was
killed mid-sweep: the prompt edit contaminated its remaining tasks, and a mixed run is
not a measurement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
v20's strict retry reused the full verbose system prompt, so 7 episodes truncated on
the retry exactly as on the original. The retry now swaps to a bare action-only
prompt for that one call and restores state after. Also stated plainly in ARENA.md
terms: our 0 false claims is structural (the arm never claims; the referee speaks),
distinct from their volunteered wrong claims.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
User-supplied 2026 survey gives the right comparison class: generic-harness MiniWoB
tops out at 71.5 (GPT-5) / 74.9 (best harness). v20's 86.4 zero-tuning single-run sits
far above it, and our +60-of-harness vs +7-of-model finding reproduces the survey's
Orby insight at scale. MiniWoB demoted to regression-suite status per the survey
rubric; Fable 5 sweep launched to complete the first known Claude-5 triplet and answer
model-limited-vs-harness-limited directly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Across v17/v18/v19 the flaky-task losses end on an EMPTY action: a verbose PLAN eats
the token cap before the action line, the reply parses to nothing, and the runner ends
the episode. The stable path to 90 is not the hard-10, it is converting the 18 flaky
tasks (always-won 97 + flaky 18 = 115/125 = 92 ceiling): v20 adds one terse strict
retry on any unparseable reply plus 800-token headroom.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Symmetric honesty: not just bu_real's scorer bug -- OUR arm only delivered an answer to
the validator on 5 of 14 clean episodes, and live-web flakiness eats a third as infra.
Every abench number (ours 0.050, theirs 0.000) reflects answer-delivery plumbing, not
research ability. Withdrawn and marked do-not-cite; a real abench run needs the
BrowserGym-native loop, not the CDP-attach shim. MiniWoB remains the one trustworthy
measurement here -- and the one where every claim holds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Off-screen rows + group ordinals solved the social-media class outright (first time in
any single run) and nudged forms; the headline held at 84.8 because decode variance
returned equivalent tasks elsewhere. v17/v18/v19 = 85.2/85.5/84.8: the
prompt-and-perception ceiling is ~85 pass@1, 91.2 labeled pass@2. What remains is
engineering, not tuning -- the four mechanical primitives, which belong in the product.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Loss traces killed the label: social-media-some fails because the @ashlea row the goal
names sits below the fold and our menu DROPPED every below-fold element, so the agent
scrolled blind; search-engine fails because "the 2nd result" is uncountable when result
links interleave with pagination links. Both are perception bugs with generic fixes:
rendered-but-off-screen rows now appear marked `off` (the action layer scrolls them into
view, so hiding them only blinded us), and rows inside repeated structures carry `#k/n`
group ordinals so "the 2nd result/post/story" is directly addressable. Verified: the
@ashlea retweet button is now row 23 (#6/11 off) where before it was absent entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Routing by page/goal features did not move the headline past v17's 85.2 pass@1, but it
reshaped the categories: drag 13/13 (from 8), email 10/10, click_compound 19/20,
text_entry 15/17 -- LEAD 7 / tie 1 / BEHIND 1 against browser-use's best clean cell.
The lesson is precise: strategy routing is not a substitute for the missing form-flow
primitive, which is the one cluster that never yields to prompting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The MiniWoB page global does not exist on AssistantBench pages, so every competitor
episode there scored a silent 0.0 -- a perfect-zero that looked like a result. Dotted
task ids now score through env.task.validate() exactly as BrowserGym's own step does,
and the retracted numbers can be re-measured honestly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
bu_real.score() reads WOB_REWARD_GLOBAL, which does not exist on AssistantBench pages,
so browser-use's 0.000 is a harness bug not a performance result; ours (env-reward
path) is likely fine but both must be re-scored through the AssistantBench reward
before any claim. Caught by user skepticism -- exactly the tunnel-vision guard the
method depends on. MiniWoB numbers are unaffected (that path is correct there).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
At first sight of a page the episode picks its mode -- form (per-field verify, no
gambled chains), geometry (eyes every turn, measure-act-remeasure), console (keyboard
primitives), game (many tiny turns, scoreboard) -- from page/goal FEATURES only, task
names audited absent. Targets the four stable loss clusters that pass@1 85.2 left.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Cross-seed spread 2.4 points: the mechanisms generalize across task content, not just
RNG. The 90 goal is met only under the honestly-labeled pass@2 protocol; single-run
85.2 is the true champion number, and the residual gap is decode variance Claude-5
lanes expose no temperature control over, plus the four named engineering clusters.
AssistantBench ours lands at 0.050 mean accuracy (suite SOTA ~25%); bu-real queued on
the identical protocol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Per-task process isolation recovered the entire playwright-wedge class and mechanical
fill-verify held the targeted text/email categories; the 20 remaining losses collapse
onto exactly the four engineering clusters named in the plan (long forms, pixel
precision, console emulation, stateful games). Seed-43 sweep running as the
cross-seed generalization check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The untainted re-run replaces the asterisked 48%: their best-model cell is 16 points
behind ours at 4.5x the wall. Claude-5 lanes reject the temperature param outright --
v17's 400-loop root-caused, field omitted; decode variance now attacked at the action
layer (fill-verify, verify-terminal) instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
temperature=0 kills decode randomness (measured cause of same-seed run-to-run flips);
mechanical fill-verify re-issues a fill exactly when the fresh look shows the value did
not stick (feature-triggered, fires on observed mismatch only); the sweep runs
--isolate so the copy-paste playwright wedge cannot poison neighbors. Generalization
protocol: zero task-name logic (audited -- 3 matches are all evidence citations in
comments), two seeds (42+43: seeds change task CONTENT, so cross-seed consistency is
the in-family generalization test), and full-125 regression not just target tasks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
