WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-08-17 18:25:42 +02:00
Code Issues Packages Projects Releases Wiki Activity
2,711 Commits 90 Branches 117 Tags
eric/browser-merged
Commit Graph
2711 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
ciregenz 268f6948a7 arena: v45 read-your-writes NEGATIVE (9% vs 7%, within noise) -- verified-writes capability-gated not prompt-gated; clause 5 hard-open 2026-08-17 09:15:17 -07:00
ciregenz a22f47a116 arena: v45 read-your-writes (state-change tasks re-read + retry until change confirmed) -- capability work for verified-writes clause 2026-08-17 08:11:52 -07:00
ciregenz 1f3c2d4fbe arena: verified-write nuance -- type-specific (gitlab comments persist, reddit upvotes dont); 7% was upvote-dominated, understates comment writes 2026-08-17 07:15:23 -07:00
ciregenzandClaude Fable 5 118fde5623 arena: atomic-write measurement -- my prediction was WRONG (7%, not high); verified-writes ~7-8% overall is a real capability wall, clause 5 furthest-open; verifier bugs fixed before booking 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 22:26:36 -07:00
ciregenz 834aa60f74 arena: Online-Mind2Web head-to-head scoped -- runnable but live-site/ToS/LLM-judge blockers; recommend 50-task read-only subset, awaiting go/no-go 2026-08-16 21:57:34 -07:00
ciregenz a86d9e3a0b arena: verified-write independent scorer -- composite reward OVERSTATED (11/15 looked persisted, only ~8% actually did); independent re-read caught the false-positive; clause 5 far open, capability-gated on complex writes 2026-08-16 21:19:34 -07:00
ciregenz ef2c8a8d3b arena: de-risk verdicts -- OSWorld smoke-only/skip (model-gated, cant run on our stack); verified-writes BUILD NOW (deterministic on existing infra); MiniWoB 94.1 honest near-ceiling (search-engine primitive fought reward logic, disabled) 2026-08-16 19:23:54 -07:00
ciregenz bc225d11ae arena: CompWoB spot-check -- no paren-bug suppression (3/12 same as v34); 81.1 stands 2026-08-16 19:18:33 -07:00
ciregenzandClaude Fable 5 c44308ed94 arena: WebArena FINAL = 13 strict/15% + 29.2 partial/34% (was falsely 0/3.83) -- ~7-8x true-credit correction from instrument fixes 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 18:47:42 -07:00
ciregenz 728ccab6b6 arena: judge policy -> RECIPROCAL cross-family (GPT judges Claude, Claude judges GPT) -- symmetric, no self-grading, bias applies equally 2026-08-16 18:46:29 -07:00
ciregenz 7b33018c9d arena: judge policy -- cross-family (GPT-5.6) + official judge, never same-family self-judge; supplementary only 2026-08-16 18:27:16 -07:00
ciregenz 52c19b0aee arena: test roadmap -- Polar benchmarks unrunnable (results-only/LLM-judge); queue their PUBLIC constituents (Online-Mind2Web etc.) as labeled supplementary numbers 2026-08-16 18:24:54 -07:00
ciregenz 9f4acf7823 arena: competitor note -- Polar's 98.0 is self-graded on own benchmarks (tuned product, not generic); not comparable, don't chase 2026-08-16 18:22:12 -07:00
ciregenzandClaude Fable 5 0e5162bf2e arena: MAJOR WebArena RETRACTION -- '0 solved' was our own instrument bugs; honest re-measure ~18% strict/~36% partial (was falsely 0/3.83). Biggest correction of the effort. 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 18:15:58 -07:00
ciregenzandClaude Fable 5 1baecba061 arena: v43 schema-gate was self-inflicted (falsely bounced valid JSON) -- disabled; v42 is honest WA config; full WA re-measurement launching 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 16:10:29 -07:00
ciregenzandClaude Fable 5 195c1e0935 arena: WA canary retraction (instrument fixes lifted partials 0->0.5) + v43 answer-schema conformance gate (generic instruction-following) to lift 0.5->1.0 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 15:07:22 -07:00
ciregenzandClaude Fable 5 760d40b953 arena: v41 MiniWoB champion = 94.1 (+3.4 vs v40; draw-circle + 18-budget); ~1.9 from 95, 4-5 primitives left 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 14:33:47 -07:00
ciregenzandClaude Fable 5 e96a5fdd24 arena: grep-audit checkpoint -- champion logic has zero benchmark-specific strings; overfitting boundary holds 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 13:01:24 -07:00
ciregenzandClaude Fable 5 70dd5761a4 arena: fix TWO instrument bugs zeroing WebArena -- (1) goal truncated to 600 chars dropped WA's answer-schema; feed policy full goal. (2) CALL_RE cut payloads at first ) killing answers with parens; quote-aware scanner. Both trace-verified. WA 0-solved was partly our harness. 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:59:52 -07:00
ciregenzandClaude Fable 5 3972f543fc arena: v42 terminal answer protocol (bare final answer on string-match goals) -- generic output hygiene, offline-verified 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:57:38 -07:00
ciregenzandClaude Fable 5 b4c011f8a3 arena: benchmark-structure + SOTA scan -- terminal answer protocol (TAKE, generic) vs api/selector bypass (DECLINE, benchmark-specific); honest principled split 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:56:54 -07:00
ciregenz c08d0d4b22 arena: WebArena v40 rerun still 0 strict -- ingestions dont address long-horizon completion; goal >=65 open+field-hard, booked honestly 2026-08-16 11:57:16 -07:00
ciregenzandClaude Fable 5 4022b04b9a arena: v41 scripted draw-circle geometry primitive -- draw-circle 0/3->3/3; feature-gated on circle goals 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 11:24:50 -07:00
ciregenzandClaude Fable 5 1260988938 arena: consolidated goal scoreboard at checkpoint -- 6/11 clauses met, open ones honestly categorized (capability/field-difficulty/infra, none overfittable) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 09:36:45 -07:00
ciregenzandClaude Fable 5 506961da9f arena: WASP injection refusal = 100% (17/17 engaged, 0 obeyed) -- clause MET; scorer false-positive caught+fixed pre-booking 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 08:18:22 -07:00
ciregenzandClaude Fable 5 191854df37 arena: WASP scorer fix -- exclude send_msg_to_user from obeyed check (agent narrating its refusal mentioned attacker domain -> 3 false-positive vulns were actually strong refusals); only real navigation/interaction counts 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 08:01:15 -07:00
ciregenzandClaude Fable 5 43004c705a arena: WASP scorer -- policy.reset per episode + 0-action marked inconclusive (empty completion = ambiguous, scored separately) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 07:28:23 -07:00
ciregenzandClaude Fable 5 c4ae243ba6 arena: WASP injection refusal harness (deterministic scoring, no LLM judge) -- canary clean; full 21-attack sweep next 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 06:54:33 -07:00
ciregenzandClaude Fable 5 d887850706 arena: CompWoB v40 delta 0 net-new -- holds 81.1, still 0.9 behind browser-use; booked honestly, no manufactured pass 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 06:18:58 -07:00
ciregenzandClaude Fable 5 c50bd4fd82 arena: champion v40 3-seed = 90.7 (ingestion wins generalized; seed-42 flaky tail offsets; prompt-side plateau noted) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 05:46:51 -07:00
ciregenzandClaude Fable 5 eaacc854d8 arena: v40 champion promoted (escape+table_md; popup loss was variance, 6/6 on 3-seed) -- champion 3-seed MiniWoB launching 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 03:42:40 -07:00
ciregenzandClaude Fable 5 92f1db8b6e arena: v40c exonerates escape token (same control losses without it) -- v40a (escape+table, no mutation) isolating now 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 03:05:47 -07:00
ciregenzandClaude Fable 5 76eada6987 arena: v40 union FAILS confirm (interaction regression) -- pairwise isolation running (v40c first) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 02:33:55 -07:00
ciregenzandClaude Fable 5 7581706f5f arena: v38 7/8 + v39 5/8 both PASS -- v40 champion = union of three passed ingestions; combined confirm pilot next 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 02:02:03 -07:00
ciregenzandClaude Fable 5 0e2aac1afc arena: pre-register v38/v39 pilots 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 01:30:31 -07:00
ciregenzandClaude Fable 5 2e07047ff7 arena: v36 PASSES (3/8 incl click-collapsible pair; first ingestion win) -> v40 champion; v37 deferred to chore-scale 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 01:30:09 -07:00
ciregenzandClaude Fable 5 25892f30b1 arena: WebChoreArena primary Claude column final -- 1/89 at 500s cap, 39 answers, 0 false; budget is the binding constraint (both models agree) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 00:58:44 -07:00
ciregenzandClaude Fable 5 8574ab4d2a arena: pre-register v36/v37 ingestion pilots (targets + predictions) before any episode 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 23:01:32 -07:00
ciregenzandClaude Fable 5 3abe31fb01 arena: implement v39 mutation-diff action feedback (Agent-E) -- all four Tier-1 ingestions now flag-gated and ready to pilot 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 22:53:46 -07:00
ciregenzandClaude Fable 5 a3d2096982 arena: implement v38 table->markdown rendering (AgentOccam) -- offline-verified on read-table, flag-gated 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 22:22:33 -07:00
ciregenzandClaude Fable 5 6c245d3245 arena: implement v36 no_match refusal token (SeeAct) + v37 persistent NOTE scratchpad (AgentOccam/Agent-E), flag-gated, offline-verified -- pilots queued for lane-free 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:49:56 -07:00
ciregenzandClaude Fable 5 42ba2b897f arena: competitor ingestion ledger from AgentOccam/Agent-E/SeeAct study -- 4 Tier-1 generic candidates (escape token, note scratchpad, table-markdown, mutation-diff), pre-registered pilot-gated 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:47:10 -07:00
ciregenzandClaude Fable 5 3e55a95117 arena: 2026 open-source/SOTA reference points + browser-use library-vs-cloud distinction (honest comparison scope) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:41:04 -07:00
ciregenzandClaude Fable 5 0c149a671d arena: WebChoreArena GPT-5.4 column final -- 2/86 solved at 500s cap, 0 false claims, upstream evaluator crashes disclosed 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 20:44:37 -07:00
ciregenzandClaude Fable 5 7dfd65cb00 arena: CORRECTION -- lane outage was a stale router OAuth token, not plan exhaustion (user dashboard evidence) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 20:34:42 -07:00
ciregenzandClaude Fable 5 ee68667b26 arena: primary Claude v35 MiniWoB = 90.5% (v35 picker fix recovers choose-date; net-flat vs v22, +1.5 vs v34-buggy); residual >=95 cluster named 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 17:13:01 -07:00
ciregenzandClaude Fable 5 00172966ae arena: GPT-5.4 cross-model MiniWoB = 86.1% (same harness, ~3pt of Claude; harness-generality claim measured directly) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 13:34:39 -07:00
ciregenzandClaude Fable 5 597931725c arena: book Claude-sub-exhausted -> GPT-5.4 cross-model column decision 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 11:30:14 -07:00
ciregenzandClaude Fable 5 5d6a14c936 arena: '@lane' arm alias for cross-model runs on a separate subscription (Claude sub exhausted; GPT-5.4 lane live, own data file) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 11:29:46 -07:00
ciregenzandClaude Fable 5 1845e6d4c8 arena: WASP adapter design -- deterministic injection-refusal scoring (their ASR classifier excluded) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 09:14:08 -07:00