arena: v34 verdict -- named 1-step losses flipped as predicted; v34 champion config

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
ciregenz
2026-08-14 13:29:05 -07:00
co-authored by Claude Fable 5
parent 4cdda1ccc0
commit fa3a0fdae3
+9
View File
@@ -421,6 +421,15 @@ feature-trigger, no task names, no new scaffolding; normal goals keep the script
(unit-checked both ways). Prediction: >=3 of the 7 reverse-order targets flip (the two 1-step
losses at minimum), controls 12/12 (no control goal contains the conjunctions).
VERDICT (2026-08-14): **partially confirmed — the two named 1-step losses BOTH flipped (2-step
wins), plus one non-reverse target; 3/12 targets, reverse 2/8, controls 12/12.** The strict
>=3-reverse bar missed, booked as such. The gate keeps: measured wins, zero cost, correctness
over speed on inversion grammar. v34 is the champion config going forward. The remaining 9
bu-edge tasks fail deep (dialog-first and checkbox+widget combos, 8-24 steps in) — each its own
dig; deprioritized behind WebChoreArena and the unmeasured goal clauses. CompWoB estimate under
v34: ~77/95 ≈ 81% (74 + the 3 pilot flips at sweep seed); 12-task delta rerun queued to book it
properly after the 3-seed rerun frees the lane.
## Benchmark roadmap (2026 landscape survey, method-filtered)
Rules: third-party scoring, reproducible from a committed artifact, no LLM-judge (or deterministic