The 6 'strict wins' previously reported were fractional checkpoint credits (5x0.667, 1x0.5)
mislabeled by the MiniWoB success predicate (reward>0). ARENA.md books the correction;
EpisodeRecord now carries an explicit strict bit so the conflation is unrepresentable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Single-step turns + full compressed history did not move one long-composition target
(controls unharmed). The 20-minute pilot killed the hypothesis before a 2-hour sweep
could -- exactly what the post-v24 rules were for. Next hypothesis, pre-registered:
inter-section settle pacing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ac/code
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Full-coverage fair pair after two harness retractions: their patient per-step
eval/memory loop owns long compositions (5-7-part 4/6 vs our 0/8) at 48s/8-false-claims
cost against our 17s/0. v25 ingests the mechanism pre-registered and gated (single-step
turns + compressed full history on >=4-clause goals only); pilot with controls running
per the new discipline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Ground truth: one CompWoB page (enter-text-second family) is broken upstream (clears
an input before creating it); isolate mode took remaining[:1] every round, so the
supervisor retried that one impossible page 201 straight times and 72 healthy tasks
starved unseen behind it. browser-use never received them. Supervisor now rotates
through remaining tasks and blacklists any that fail setup three times; their fair
completion run is live. Score one more for verify-the-verifier: both prior narratives
about their CompWoB failures were wrong, and the record now says so.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Symmetric audit: the composed pages' own genProblem loses an asset race and crashes
reset -- 1/101 times on our launch path, 73/101 on the CDP-port path, same pages,
same seed. Blaming their instrumentation was wrong and is withdrawn; both harness
paths now retry the reset (same courtesy both ways) and their CompWoB completion
sweep is rerunning for a genuine full-coverage number.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Docker Desktop proved uninitializable without GUI on this box (three engine rebuilds,
factory reset included); colima on macOS's Virtualization.framework started first try
and swallowed the 131GB of site images Docker Desktop could not load. Bring-up script
now targets the colima socket; shopping/admin/forum booting while gitlab downloads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
60.4% (-5): short tasks paid for the scaffolding, >=5-part stayed 0/8 through a third
mechanism. The cluster's invariance across runway/ordering/checklist localizes the
real need -- mid-episode replanning against current page state -- product-primitive
territory, same frontier as MiniWoB's hard-10. Champion for CompWoB remains v22/v23
at 65.3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Long-chain traces showed clean execution collapsing into scroll-flailing once the
agent lost its place. The goal is decomposed once into numbered clauses, rendered
every turn with the model's own last-reported position echoed back; content-agnostic
scaffolding, no task knowledge. Sweeping CompWoB against the 65.3 baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
browser-use cannot load 73 of 101 composed pages (instrumentation crash: 28-task
coverage ceiling, 2 more false claims on what loads). Our ordering fix won its exact
predicted tasks; long-chain traces isolate the true frontier as lost mid-chain
bookkeeping, to be attacked with a rendered clause checklist (v24).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
macOS silently deletes /tmp files by age: pyvenv.cfg vanished mid-sweep and 84 spawns
died import-less while site-packages survived -- the exact silent-error class the goal
names. Venv, serving trees and compwob discovery now live in ~/.cache/arena (durable);
servers rebound; registration re-verified (101). Episode data was never at risk: the
book lives in the repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
CompWoB's simple-pair autopsy: with "Ok" ambiguous, the fastpath clicked the SECOND
quoted target first, and order-enforcing pages fail terminally on that. It now acts
only on the first unsatisfied quoted target and otherwise defers to the model --
sequence is part of the task, structurally respected. Lands in v23's queued sweep.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Idempotent start+configure for shopping/shopping_admin/forum/gitlab per the official
environment_docker README, with port health checks; fires automatically the moment the
190GB image pipeline finishes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
The stable game losses share one blindness: the page answers every action in its text
(Colder!, Too high, score counters) and the agent's memory recorded only its own
actions. History lines now carry the page-text delta each action caused -- a general
feedback mechanism, nothing task-shaped. Queued behind the CompWoB pair.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Clean-harness CompWoB opened 6/6 on 2-3-clause compositions and 0/3 on 7-8-clause
monsters whose traces show correct work hitting the step cap. Episode budget now grows
with clause count (feature-triggered from the goal text, capped at 3x) -- a general
long-instruction mechanism, not a benchmark tweak. Resume keeps the six clean wins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
Two instrument bugs found before any number shipped: composed pages need the LEGACY
MiniWoB core (modern core terminated on the first click), and validate() string-
compares page.url so the '../' base-url trick killed every episode after step one --
boring URLs are robust URLs. Canary now clicks ONE then TWO with the episode alive.
The 0/17 zero from the broken setup is void; the pair sweep restarts clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
101 composed tasks registered through BrowserGym's own MiniWoB task class pointed at
the composed pages, so the reward path is the already-canary-validated page-owned
machinery; task ids discovered from the served directory so registry drift is
impossible. Canary passed (real composed goal, reward global live). This is the
benchmark built to expose memorization (specialists fall 95->61 on it) -- our v22 and
browser-use both sweep all 101 under identical isolated protocol.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
