mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-08-17 18:25:42 +02:00
arena: v43 schema-gate was self-inflicted (falsely bounced valid JSON) -- disabled; v42 is honest WA config; full WA re-measurement launching
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
co-authored by
Claude Fable 5
parent
195c1e0935
commit
1baecba061
@@ -688,6 +688,19 @@ suppressing credit -- the true partial score is higher. Next: v43 answer-schema
|
||||
(validate send_msg against the schema the TASK ITSELF provides; bounce errors back -- generic
|
||||
instruction-following, knows no answers) to lift 0.5 -> 1.0.
|
||||
|
||||
## v43 schema-gate: SELF-INFLICTED bug, disabled (2026-08-16)
|
||||
|
||||
The v43 answer-schema gate was HARMFUL and is disabled. Trace (13.627): the agent emitted valid
|
||||
JSON ({"task_type":"MUTATE","status":"SUCCESS",...}) but my gate's extraction regex only matched
|
||||
quoted-string payloads, not raw ({...}) objects, so it FALSELY bounced every valid answer and the
|
||||
env never received a final message. Caught by reading the trace (the discipline that also caught
|
||||
the WASP false-positive). Honest conclusion: v42 (instrument fixes + answer_protocol, NO gate) is
|
||||
the WebArena config; instrument fixes recovered partials 0->0.5 (post-creation credit that was
|
||||
being zeroed), but 0.5->1.0 needs the exact answer CONTENT the evaluator expects (retrieved_data
|
||||
values / task_type enum) -- a genuine capability/knowledge wall, not a format bug. Full WA
|
||||
re-measurement on v42 launched to book the honest partial-sum (was suppressed to 3.83 under the
|
||||
truncation bug).
|
||||
|
||||
## Champion v41 3-seed MiniWoB (2026-08-16): 94.1 (92.0 / 95.2 / 95.2) — +3.4 vs v40
|
||||
|
||||
Two capability fixes landed: scripted draw-circle geometry (0/3->3/3) + universal 18-step budget
|
||||
|
||||
@@ -1093,7 +1093,7 @@ def build(name: str, model: str = "", endpoint: str = "", **_: Any) -> Any:
|
||||
local_ctx=True, blocker_probe=True, suppress_wrappers=True,
|
||||
force_unblock=True, native_js_fallback=True, escape_token=True,
|
||||
table_md=True, draw_circle=True, answer_protocol=True,
|
||||
schema_gate=True, **v43)
|
||||
schema_gate=False, **v43) # DISABLED: gate falsely bounced valid JSON
|
||||
if name == "osw-llm-v42": # v41 + terminal answer protocol (string-match benchmarks)
|
||||
v42 = dict(v7, system=OSW_SYSTEM_V8 + OSW_SYSTEM_V9_WIDGETS + OSW_SYSTEM_V16 + OSW_SYSTEM_V30
|
||||
+ OSW_SYSTEM_V36, max_tokens=800)
|
||||
|
||||
Reference in New Issue
Block a user