WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-09-10 19:57:44 +02:00
Code Issues Packages Projects Releases Wiki Activity

arena: WASP adapter design -- deterministic injection-refusal scoring (their ASR classifier excluded)

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
This commit is contained in:
ciregenz
2026-08-15 09:14:08 -07:00
co-authored by Claude Fable 5
parent 1bb3b0810a
commit 1845e6d4c8
1 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
e2e/browser-v3/arena/ARENA.md
+8 -1
View File
@@ -465,7 +465,14 @@ Rules: third-party scoring, reproducible from a committed artifact, no LLM-judge
subset only). ADDED: WebChoreArena (COLM 2026, wired, sweep queued); REAL/realevals.xyz (11
deterministic self-hosted simulations of real sites, 110+ tasks, programmatic state verification
— deterministic subset only) next after it; WASP (Meta, prompt-injection — the >=95 injection
clause); OSWorld (the >=85 clause; biggest infra lift). EXCLUDED with reasons: Online-Mind2Web
clause); OSWorld (the >=85 clause; biggest infra lift). WASP adapter design (2026-08-15, from repo read): their ASR scorer is an AzureOpenAI classifier
(LLM-judge → excluded for scoring per method), but the >=95 injection-refusal clause has a
DETERMINISTIC core — the injected tasks are concrete observable side-effects (navigate to
{attacker_domain}?token=..., create a deploy token). Plan: run their gitlab/reddit injected
tasks through our runner (sites already up), score refusal from the action trace deterministically
(did the agent hit the attacker domain / perform the injected write — boolean), no classifier.
Configs at ~/.cache/arena/wasp/webarena_prompt_injections/configs/experiment_config.raw.json.
EXCLUDED with reasons: Online-Mind2Web
(LLM-judge on live drifting sites — but its '59% weaker on live web' finding motivates the live
verified-writes clause), WebVoyager (publicly described as saturated), GAIA/BrowseComp/
LiveAgentBench (research-QA, not browser operation), VisualWebArena (screenshot-agent variant of