WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-08-17 18:25:42 +02:00
Code Issues Packages Projects Releases Wiki Activity
2,711 Commits 90 Branches 117 Tags
eric/browser-merged
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
ciregenz 268f6948a7 arena: v45 read-your-writes NEGATIVE (9% vs 7%, within noise) -- verified-writes capability-gated not prompt-gated; clause 5 hard-open 2026-08-17 09:15:17 -07:00
ciregenz a22f47a116 arena: v45 read-your-writes (state-change tasks re-read + retry until change confirmed) -- capability work for verified-writes clause 2026-08-17 08:11:52 -07:00
ciregenz 1f3c2d4fbe arena: verified-write nuance -- type-specific (gitlab comments persist, reddit upvotes dont); 7% was upvote-dominated, understates comment writes 2026-08-17 07:15:23 -07:00
ciregenzandClaude Fable 5 118fde5623 arena: atomic-write measurement -- my prediction was WRONG (7%, not high); verified-writes ~7-8% overall is a real capability wall, clause 5 furthest-open; verifier bugs fixed before booking 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 22:26:36 -07:00
ciregenz 834aa60f74 arena: Online-Mind2Web head-to-head scoped -- runnable but live-site/ToS/LLM-judge blockers; recommend 50-task read-only subset, awaiting go/no-go 2026-08-16 21:57:34 -07:00
ciregenz a86d9e3a0b arena: verified-write independent scorer -- composite reward OVERSTATED (11/15 looked persisted, only ~8% actually did); independent re-read caught the false-positive; clause 5 far open, capability-gated on complex writes 2026-08-16 21:19:34 -07:00
ciregenz ef2c8a8d3b arena: de-risk verdicts -- OSWorld smoke-only/skip (model-gated, cant run on our stack); verified-writes BUILD NOW (deterministic on existing infra); MiniWoB 94.1 honest near-ceiling (search-engine primitive fought reward logic, disabled) 2026-08-16 19:23:54 -07:00
ciregenz bc225d11ae arena: CompWoB spot-check -- no paren-bug suppression (3/12 same as v34); 81.1 stands 2026-08-16 19:18:33 -07:00
ciregenzandClaude Fable 5 c44308ed94 arena: WebArena FINAL = 13 strict/15% + 29.2 partial/34% (was falsely 0/3.83) -- ~7-8x true-credit correction from instrument fixes 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 18:47:42 -07:00
ciregenz 728ccab6b6 arena: judge policy -> RECIPROCAL cross-family (GPT judges Claude, Claude judges GPT) -- symmetric, no self-grading, bias applies equally 2026-08-16 18:46:29 -07:00
ciregenz 7b33018c9d arena: judge policy -- cross-family (GPT-5.6) + official judge, never same-family self-judge; supplementary only 2026-08-16 18:27:16 -07:00
ciregenz 52c19b0aee arena: test roadmap -- Polar benchmarks unrunnable (results-only/LLM-judge); queue their PUBLIC constituents (Online-Mind2Web etc.) as labeled supplementary numbers 2026-08-16 18:24:54 -07:00
ciregenz 9f4acf7823 arena: competitor note -- Polar's 98.0 is self-graded on own benchmarks (tuned product, not generic); not comparable, don't chase 2026-08-16 18:22:12 -07:00
ciregenzandClaude Fable 5 0e5162bf2e arena: MAJOR WebArena RETRACTION -- '0 solved' was our own instrument bugs; honest re-measure ~18% strict/~36% partial (was falsely 0/3.83). Biggest correction of the effort. 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 18:15:58 -07:00
ciregenzandClaude Fable 5 1baecba061 arena: v43 schema-gate was self-inflicted (falsely bounced valid JSON) -- disabled; v42 is honest WA config; full WA re-measurement launching 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 16:10:29 -07:00
ciregenzandClaude Fable 5 195c1e0935 arena: WA canary retraction (instrument fixes lifted partials 0->0.5) + v43 answer-schema conformance gate (generic instruction-following) to lift 0.5->1.0 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 15:07:22 -07:00
ciregenzandClaude Fable 5 760d40b953 arena: v41 MiniWoB champion = 94.1 (+3.4 vs v40; draw-circle + 18-budget); ~1.9 from 95, 4-5 primitives left 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 14:33:47 -07:00
ciregenzandClaude Fable 5 e96a5fdd24 arena: grep-audit checkpoint -- champion logic has zero benchmark-specific strings; overfitting boundary holds 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 13:01:24 -07:00
ciregenzandClaude Fable 5 70dd5761a4 arena: fix TWO instrument bugs zeroing WebArena -- (1) goal truncated to 600 chars dropped WA's answer-schema; feed policy full goal. (2) CALL_RE cut payloads at first ) killing answers with parens; quote-aware scanner. Both trace-verified. WA 0-solved was partly our harness. 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:59:52 -07:00
ciregenzandClaude Fable 5 3972f543fc arena: v42 terminal answer protocol (bare final answer on string-match goals) -- generic output hygiene, offline-verified 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:57:38 -07:00
ciregenzandClaude Fable 5 b4c011f8a3 arena: benchmark-structure + SOTA scan -- terminal answer protocol (TAKE, generic) vs api/selector bypass (DECLINE, benchmark-specific); honest principled split 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:56:54 -07:00
ciregenz c08d0d4b22 arena: WebArena v40 rerun still 0 strict -- ingestions dont address long-horizon completion; goal >=65 open+field-hard, booked honestly 2026-08-16 11:57:16 -07:00
ciregenzandClaude Fable 5 4022b04b9a arena: v41 scripted draw-circle geometry primitive -- draw-circle 0/3->3/3; feature-gated on circle goals 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 11:24:50 -07:00
ciregenzandClaude Fable 5 1260988938 arena: consolidated goal scoreboard at checkpoint -- 6/11 clauses met, open ones honestly categorized (capability/field-difficulty/infra, none overfittable) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 09:36:45 -07:00
ciregenzandClaude Fable 5 506961da9f arena: WASP injection refusal = 100% (17/17 engaged, 0 obeyed) -- clause MET; scorer false-positive caught+fixed pre-booking 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 08:18:22 -07:00
ciregenzandClaude Fable 5 191854df37 arena: WASP scorer fix -- exclude send_msg_to_user from obeyed check (agent narrating its refusal mentioned attacker domain -> 3 false-positive vulns were actually strong refusals); only real navigation/interaction counts 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 08:01:15 -07:00
ciregenzandClaude Fable 5 43004c705a arena: WASP scorer -- policy.reset per episode + 0-action marked inconclusive (empty completion = ambiguous, scored separately) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 07:28:23 -07:00
ciregenzandClaude Fable 5 c4ae243ba6 arena: WASP injection refusal harness (deterministic scoring, no LLM judge) -- canary clean; full 21-attack sweep next 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 06:54:33 -07:00
ciregenzandClaude Fable 5 d887850706 arena: CompWoB v40 delta 0 net-new -- holds 81.1, still 0.9 behind browser-use; booked honestly, no manufactured pass 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 06:18:58 -07:00
ciregenzandClaude Fable 5 c50bd4fd82 arena: champion v40 3-seed = 90.7 (ingestion wins generalized; seed-42 flaky tail offsets; prompt-side plateau noted) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 05:46:51 -07:00
ciregenzandClaude Fable 5 eaacc854d8 arena: v40 champion promoted (escape+table_md; popup loss was variance, 6/6 on 3-seed) -- champion 3-seed MiniWoB launching 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 03:42:40 -07:00
ciregenzandClaude Fable 5 92f1db8b6e arena: v40c exonerates escape token (same control losses without it) -- v40a (escape+table, no mutation) isolating now 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 03:05:47 -07:00
ciregenzandClaude Fable 5 76eada6987 arena: v40 union FAILS confirm (interaction regression) -- pairwise isolation running (v40c first) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 02:33:55 -07:00
ciregenzandClaude Fable 5 7581706f5f arena: v38 7/8 + v39 5/8 both PASS -- v40 champion = union of three passed ingestions; combined confirm pilot next 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 02:02:03 -07:00
ciregenzandClaude Fable 5 0e2aac1afc arena: pre-register v38/v39 pilots 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 01:30:31 -07:00
ciregenzandClaude Fable 5 2e07047ff7 arena: v36 PASSES (3/8 incl click-collapsible pair; first ingestion win) -> v40 champion; v37 deferred to chore-scale 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 01:30:09 -07:00
ciregenzandClaude Fable 5 25892f30b1 arena: WebChoreArena primary Claude column final -- 1/89 at 500s cap, 39 answers, 0 false; budget is the binding constraint (both models agree) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 00:58:44 -07:00
ciregenzandClaude Fable 5 8574ab4d2a arena: pre-register v36/v37 ingestion pilots (targets + predictions) before any episode 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 23:01:32 -07:00
ciregenzandClaude Fable 5 3abe31fb01 arena: implement v39 mutation-diff action feedback (Agent-E) -- all four Tier-1 ingestions now flag-gated and ready to pilot 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 22:53:46 -07:00
ciregenzandClaude Fable 5 a3d2096982 arena: implement v38 table->markdown rendering (AgentOccam) -- offline-verified on read-table, flag-gated 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 22:22:33 -07:00
ciregenzandClaude Fable 5 6c245d3245 arena: implement v36 no_match refusal token (SeeAct) + v37 persistent NOTE scratchpad (AgentOccam/Agent-E), flag-gated, offline-verified -- pilots queued for lane-free 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:49:56 -07:00
ciregenzandClaude Fable 5 42ba2b897f arena: competitor ingestion ledger from AgentOccam/Agent-E/SeeAct study -- 4 Tier-1 generic candidates (escape token, note scratchpad, table-markdown, mutation-diff), pre-registered pilot-gated 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:47:10 -07:00
ciregenzandClaude Fable 5 3e55a95117 arena: 2026 open-source/SOTA reference points + browser-use library-vs-cloud distinction (honest comparison scope) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:41:04 -07:00
ciregenzandClaude Fable 5 0c149a671d arena: WebChoreArena GPT-5.4 column final -- 2/86 solved at 500s cap, 0 false claims, upstream evaluator crashes disclosed 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 20:44:37 -07:00
ciregenzandClaude Fable 5 7dfd65cb00 arena: CORRECTION -- lane outage was a stale router OAuth token, not plan exhaustion (user dashboard evidence) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 20:34:42 -07:00
ciregenzandClaude Fable 5 ee68667b26 arena: primary Claude v35 MiniWoB = 90.5% (v35 picker fix recovers choose-date; net-flat vs v22, +1.5 vs v34-buggy); residual >=95 cluster named 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 17:13:01 -07:00
ciregenzandClaude Fable 5 00172966ae arena: GPT-5.4 cross-model MiniWoB = 86.1% (same harness, ~3pt of Claude; harness-generality claim measured directly) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 13:34:39 -07:00
ciregenzandClaude Fable 5 597931725c arena: book Claude-sub-exhausted -> GPT-5.4 cross-model column decision 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 11:30:14 -07:00
ciregenzandClaude Fable 5 5d6a14c936 arena: '@lane' arm alias for cross-model runs on a separate subscription (Claude sub exhausted; GPT-5.4 lane live, own data file) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 11:29:46 -07:00
ciregenzandClaude Fable 5 1845e6d4c8 arena: WASP adapter design -- deterministic injection-refusal scoring (their ASR classifier excluded) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 09:14:08 -07:00
ciregenzandClaude Fable 5 1bb3b0810a arena: v35 readonly-picker fallback (unit-verified 1.0 without a model) -- pre-registered for lane return 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 08:42:14 -07:00
ciregenzandClaude Fable 5 e6ecb83744 arena: bound 429 backoff to fit the step watchdog (flat 12s; unbounded waits booked as step_hang) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 22:23:30 -07:00
ciregenzandClaude Fable 5 61af236e2a arena: router wraps 429s in 200 bodies -- unwrap so the rate-limit backoff branch fires 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 21:51:56 -07:00
ciregenzandClaude Fable 5 857a5185f9 arena: rate-limit-aware LLM backoff (20-100s on 429s, 5 attempts) -- TPM windows outwaited, not booked as infra 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 21:51:41 -07:00
ciregenzandClaude Fable 5 4b76028e90 arena: final-step answer turn -- Q&A goals get told the budget is up so they answer via chat (1/17 chore episodes had answered) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 18:13:13 -07:00
ciregenzandClaude Fable 5 442f2e75ee arena: CompWoB champion v34 = 81.1% (delta rerun confirms flips) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 16:08:25 -07:00
ciregenzandClaude Fable 5 02347b20cf arena: MiniWoB 3-seed on v32 stack = 89.0 (fix validated on social-media; readonly-picker named); net stack ledger booked 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 15:34:46 -07:00
ciregenzandClaude Fable 5 fa3a0fdae3 arena: v34 verdict -- named 1-step losses flipped as predicted; v34 champion config 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 13:29:05 -07:00
ciregenzandClaude Fable 5 4cdda1ccc0 arena: v33 verdict (fastpath is the reverse-task killer) + v34 inversion gate (pre-registered) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 12:57:30 -07:00
ciregenzandClaude Fable 5 95fdc5fb1d arena: MiniWoB regression root-caused -- structural ctx labels displaced discriminating card text; uniqueness post-pass (labels must separate twins) + weak-name sibling filter 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 12:25:24 -07:00
ciregenzandClaude Fable 5 7985f741c7 arena: v33 opportunistic ordering (deferral doctrine + defer-nudge), pre-registered vs the 12-task bu-win cluster 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 10:45:58 -07:00
ciregenzandClaude Fable 5 8781b8079a arena: 2026 benchmark roadmap (REAL added; exclusions reasoned) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 10:16:45 -07:00
ciregenzandClaude Fable 5 0993fd5d9f arena: CompWoB FINAL v32 = 77.9% (>=75 goal clause MET; was 65.3) + head-to-head vs browser-use + v33 staging 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 10:14:26 -07:00
ciregenzandClaude Fable 5 aee265abfe arena: v32 verdict -- first >=5-part win ever; CompWoB mechanism ladder closed at the page-validator boundary 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 07:38:19 -07:00
ciregenzandClaude Fable 5 82b02317fe arena: v31 rerun verdict (model won't self-remediate occlusion) + v32 forced dispatch on blocked clicks (pre-registered) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 06:34:45 -07:00
ciregenzandClaude Fable 5 eeb62798a9 arena: v31 first run void -- blocker message truncated + stale bbox + empty-completion retries; all fixed and verified 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 06:02:37 -07:00
ciregenzandClaude Fable 5 b2c0d2be08 arena: v30 verdict (trap wrapper rows) + v31 wrapper suppression (pre-registered) + raw-reply capture 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 05:28:33 -07:00
ciregenzandClaude Fable 5 e87fd7545b arena: book v29 verdict + v30 pre-registration text 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 04:54:58 -07:00
ciregenzandClaude Fable 5 7f10e100a9 arena: v29 verdict (controls clean, targets 0/8 -- model now picks RIGHT element, click blocked by overlay) + v30 blocked-click intelligence (pre-registered) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 04:54:30 -07:00
ciregenzandClaude Fable 5 60318f2c1f arena: WebChoreArena reddit partition (deterministic evals, their max_steps=50) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 04:20:19 -07:00
ciregenzandClaude Fable 5 2aade59eb5 arena: v29 discriminative row context (DOM group labels + sibling groups) -- pre-registered pilot 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 04:19:52 -07:00
ciregenzandClaude Fable 5 6328e78772 arena: root cause of long-composition losses -- perception blindness (identical row contexts), not reasoning 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 03:42:10 -07:00
ciregenzandClaude Fable 5 bf3b78fecc arena: v28 verdict (fail; both loop-diff halves dead) + StepRecord.target instrumentation 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 03:38:01 -07:00
ciregenzandClaude Fable 5 331fc738d9 arena: webchorearena routing (isolated chore-venv, their evaluator; canary green) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 03:05:24 -07:00
ciregenzandClaude Fable 5 3eac8e8bcd arena: v27 verdict (fail, harmful; plan-state scaffolds 0-for-2) + v28 per-step self-eval (pre-registered) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 02:59:17 -07:00
ciregenzandClaude Fable 5 bf39eaf3f6 arena: WebArena paired final -- both stacks 0 strict on hard partition; ours leads partials 3.3x, speed 1.9x, tokens 10x, honesty 0 vs 49 false claims 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 02:26:38 -07:00
ciregenzandClaude Fable 5 03e062d059 arena: v27 fix -- CLAUSE marker moves AFTER the action line (was colliding with action-first replies, empty actions) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 23:29:52 -07:00
ciregenzandClaude Fable 5 1a61bfc50e arena: v27 active sub-goal ledger (pre-registered pilot) -- focused current-clause anchor, gated >=3 clauses 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 23:18:07 -07:00
ciregenzandClaude Fable 5 88291d1930 arena: v26 pilot verdict -- prediction failed (0/8 targets, controls clean); stale-plan hypothesis disconfirmed 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 23:12:36 -07:00
ciregenzandClaude Fable 5 aa5a19e51d arena: v26 serial_multi -- queued actions re-resolve targets per-observation (pre-registered pilot) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 22:41:10 -07:00
ciregenzandClaude Fable 5 b3d69cdd0f arena: WebArena-ours final (0 strict, 6 partials) + strict-field fix and written correction 
The 6 'strict wins' previously reported were fractional checkpoint credits (5x0.667, 1x0.5)
mislabeled by the MiniWoB success predicate (reward>0). ARENA.md books the correction;
EpisodeRecord now carries an explicit strict bit so the conflation is unrepresentable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 22:06:19 -07:00
ciregenzandClaude Fable 5 60e6e77880 arena: bu_real route for webarena_verified ids 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 19:25:26 -07:00
ciregenzandClaude Fable 5 d5792f4d65 arena: seeded 100-task reddit+gitlab WebArena Verified partition (seed 42, universe 304) — same list for both stacks 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 17:17:41 -07:00
ciregenzandClaude Fable 5 35a2c1cdde arena: runner route for webarena_verified dotted ids with WA_* defaults 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 16:12:38 -07:00
ciregenzandClaude Fable 5 be39237e4a arena: v25 pilot -- prediction failed, booked no-effect; pilot discipline paid for itself 
Single-step turns + full compressed history did not move one long-composition target
(controls unharmed). The 20-minute pilot killed the hypothesis before a 2-hour sweep
could -- exactly what the post-v24 rules were for. Next hypothesis, pre-registered:
inter-section settle pacing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ac/code
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 15:40:58 -07:00
ciregenzandClaude Fable 5 0f188bb168 arena: CompWoB final -- browser-use wins 82.0 to 65.3, and the reasons are now our roadmap 
Full-coverage fair pair after two harness retractions: their patient per-step
eval/memory loop owns long compositions (5-7-part 4/6 vs our 0/8) at 48s/8-false-claims
cost against our 17s/0. v25 ingests the mechanism pre-registered and gated (single-step
turns + compressed full history on >=4-clause goals only); pilot with controls running
per the new discipline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 15:09:08 -07:00
ciregenzandClaude Fable 5 cf3cc50919 arena: the '73 crashed pages' were MY supervisor's head-of-line blocking -- full retraction chain 
Ground truth: one CompWoB page (enter-text-second family) is broken upstream (clears
an input before creating it); isolate mode took remaining[:1] every round, so the
supervisor retried that one impossible page 201 straight times and 72 healthy tasks
starved unseen behind it. browser-use never received them. Supervisor now rotates
through remaining tasks and blacklists any that fail setup three times; their fair
completion run is live. Score one more for verify-the-verifier: both prior narratives
about their CompWoB failures were wrong, and the record now says so.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 13:03:10 -07:00
ciregenzandClaude Fable 5 8e37f997be arena: RETRACT the browser-use CompWoB crash claim -- it was a page-load race, not their code 
Symmetric audit: the composed pages' own genProblem loses an asset race and crashes
reset -- 1/101 times on our launch path, 73/101 on the CDP-port path, same pages,
same seed. Blaming their instrumentation was wrong and is withdrawn; both harness
paths now retry the reset (same courtesy both ways) and their CompWoB completion
sweep is rerunning for a genuine full-coverage number.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 11:42:59 -07:00
ciregenzandClaude Fable 5 52e846a4ce arena: WebArena hosting pivots to colima -- headless docker that actually works here 
Docker Desktop proved uninitializable without GUI on this box (three engine rebuilds,
factory reset included); colima on macOS's Virtualization.framework started first try
and swallowed the 131GB of site images Docker Desktop could not load. Bring-up script
now targets the colima socket; shopping/admin/forum booting while gitlab downloads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 08:14:58 -07:00
ciregenzandClaude Fable 5 8c2ff907b7 arena: v24 checklist is a negative result -- long chains are invariant under prompt scaffolds 
60.4% (-5): short tasks paid for the scaffolding, >=5-part stayed 0/8 through a third
mechanism. The cluster's invariance across runway/ordering/checklist localizes the
real need -- mid-episode replanning against current page state -- product-primitive
territory, same frontier as MiniWoB's hard-10. Champion for CompWoB remains v22/v23
at 65.3.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 03:59:41 -07:00
ciregenzandClaude Fable 5 22e18c239a arena: v24 -- the clause checklist, bookkeeping scaffolding for long instructions 
Long-chain traces showed clean execution collapsing into scroll-flailing once the
agent lost its place. The goal is decomposed once into numbered clauses, rendered
every turn with the model's own last-reported position echoed back; content-agnostic
scaffolding, no task knowledge. Sweeping CompWoB against the 65.3 baseline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 02:57:35 -07:00
ciregenzandClaude Fable 5 f68d20589e arena: compwob follow-ups -- their 73-page crash ceiling, our ordering fix verified, the bookkeeping disease named 
browser-use cannot load 73 of 101 composed pages (instrumentation crash: 28-task
coverage ceiling, 2 more false claims on what loads). Our ordering fix won its exact
predicted tasks; long-chain traces isolate the true frontier as lost mid-chain
bookkeeping, to be attacked with a rendered clause checklist (v24).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 01:55:11 -07:00
ciregenzandClaude Fable 5 a034a78366 arena: toolchain moved out of /tmp -- the OS reaper was eating the venv file by file 
macOS silently deletes /tmp files by age: pyvenv.cfg vanished mid-sweep and 84 spawns
died import-less while site-packages survived -- the exact silent-error class the goal
names. Venv, serving trees and compwob discovery now live in ~/.cache/arena (durable);
servers rebound; registration re-verified (101). Episode data was never at risk: the
book lives in the repo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-13 00:51:27 -07:00
ciregenzandClaude Fable 5 10fae6ea34 arena: fastpath respects instruction order -- skipping ahead was an instant loss 
CompWoB's simple-pair autopsy: with "Ok" ambiguous, the fastpath clicked the SECOND
quoted target first, and order-enforcing pages fail terminally on that. It now acts
only on the first unsatisfied quoted target and otherwise defers to the model --
sequence is part of the task, structurally respected. Lands in v23's queued sweep.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-12 22:35:47 -07:00
ciregenzandClaude Fable 5 d0d97d0597 arena: CompWoB ours 65.3 -- collapses LESS than the published specialist reference 
90.9 MiniWoB -> 65.3 CompWoB beats the 95->61 specialist collapse ratio, zero false
claims, clean length gradient (82% pairs, 41% triples, 0% beyond five). The 75 target
needs the long-horizon mechanism the gradient points at; browser-use's identical run
is next, then v23.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-12 21:51:06 -07:00
ciregenzandClaude Fable 5 7347878d39 arena: WebArena site bring-up script, armed behind the image pipeline 
Idempotent start+configure for shopping/shopping_admin/forum/gitlab per the official
environment_docker README, with port health checks; fires automatically the moment the
190GB image pipeline finishes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-12 21:33:12 -07:00
ciregenzandClaude Fable 5 33cb2a0fd1 arena: v23 -- the page's own reactions ride in memory (feedback-echo) 
The stable game losses share one blindness: the page answers every action in its text
(Colder!, Too high, score counters) and the agent's memory recorded only its own
actions. History lines now carry the page-text delta each action caused -- a general
feedback mechanism, nothing task-shaped. Queued behind the CompWoB pair.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-12 21:27:18 -07:00
ciregenzandClaude Fable 5 3c8751c0b8 arena: runway scales with instruction complexity -- the first clean CompWoB pattern 
Clean-harness CompWoB opened 6/6 on 2-3-clause compositions and 0/3 on 7-8-clause
monsters whose traces show correct work hitting the step cap. Episode budget now grows
with clause count (feature-triggered from the goal text, capped at 3x) -- a general
long-instruction mechanism, not a benchmark tweak. Resume keeps the six clean wins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-12 20:52:51 -07:00
ciregenzandClaude Fable 5 0b336c232e arena: compwob fixed structurally -- legacy engine + literal URLs, canary green 
Two instrument bugs found before any number shipped: composed pages need the LEGACY
MiniWoB core (modern core terminated on the first click), and validate() string-
compares page.url so the '../' base-url trick killed every episode after step one --
boring URLs are robust URLs. Canary now clicks ONE then TWO with the episode alive.
The 0/17 zero from the broken setup is void; the pair sweep restarts clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-12 20:42:36 -07:00
ciregenzandClaude Fable 5 228d65c9e9 arena: CompWoB wired -- the generalization referee, with zero new scoring code 
101 composed tasks registered through BrowserGym's own MiniWoB task class pointed at
the composed pages, so the reward path is the already-canary-validated page-owned
machinery; task ids discovered from the served directory so registry drift is
impossible. Canary passed (real composed goal, reward global live). This is the
benchmark built to expose memorization (specialists fall 95->61 on it) -- our v22 and
browser-use both sweep all 101 under identical isolated protocol.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-12 19:08:11 -07:00