mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-08-17 18:25:42 +02:00
arena: pre-register v36/v37 ingestion pilots (targets + predictions) before any episode
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
co-authored by
Claude Fable 5
parent
3abe31fb01
commit
8574ab4d2a
@@ -558,6 +558,22 @@ TIER 3 (gated/expensive — pilot only if Tier 1-2 leave a gap):
|
||||
NOT ingested (out of scope / already have): per-site tips files (task-specific), combobox
|
||||
flattening (we have native pickers), fine-tuned transfer weights (specialist).
|
||||
|
||||
## PRE-REGISTERED (2026-08-15, before any v36/v37 episode): first two ingestion pilots
|
||||
|
||||
v37 NOTE SCRATCHPAD pilot — targets (8): form-sequence-3, hot-cold, search-engine, text-editor
|
||||
(MiniWoB losses needing running state) + 4 unsolved long CompWoB compositions (7-8-part +
|
||||
checkbox-transfer chains, cross-section bookkeeping). Prediction: >=2/8 flip, controls 12/12
|
||||
(gate: collection/aggregate goal words — controls' goals lack them, so the block never renders).
|
||||
The REAL payoff target is chore/WebArena aggregation (measured later via delta reruns); this
|
||||
pilot verifies mechanism safety + short-horizon lift.
|
||||
|
||||
v36 NO_MATCH REFUSAL pilot — targets (8): click-menu, search-engine, click-collapsible-2 pair
|
||||
(near-miss-prone MiniWoB losses) + 4 dialog-first/widget CompWoB losses where a wrong click is
|
||||
terminal. Prediction: >=2/8 flip, controls 12/12 (no_match only fires when chosen; wrong-click
|
||||
replacement is strictly safer than the click it replaces).
|
||||
|
||||
Order: v37 then v36, sequential, immediately after the Claude chore column finishes (auto-chained).
|
||||
|
||||
## Open-source / SOTA reference points (2026 leaderboards, for honest comparison)
|
||||
|
||||
- WebArena: SOTA WebTactix/DeepSeek-v3.2 74.3%; frontier models 64-68%; human 78. (leaderboard.steel.dev)
|
||||
|
||||
Reference in New Issue
Block a user