WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-08-31 20:29:56 +02:00
Code Issues Packages Projects Releases Wiki Activity
2,700 Commits 92 Branches 124 Tags
52c19b0aee0b89385a1f7c8c720c9caee1a785d7
Commit Graph
2700 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
ciregenz 52c19b0aee arena: test roadmap -- Polar benchmarks unrunnable (results-only/LLM-judge); queue their PUBLIC constituents (Online-Mind2Web etc.) as labeled supplementary numbers 2026-08-16 18:24:54 -07:00
ciregenz 9f4acf7823 arena: competitor note -- Polar's 98.0 is self-graded on own benchmarks (tuned product, not generic); not comparable, don't chase 2026-08-16 18:22:12 -07:00
ciregenzandClaude Fable 5 0e5162bf2e arena: MAJOR WebArena RETRACTION -- '0 solved' was our own instrument bugs; honest re-measure ~18% strict/~36% partial (was falsely 0/3.83). Biggest correction of the effort. 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 18:15:58 -07:00
ciregenzandClaude Fable 5 1baecba061 arena: v43 schema-gate was self-inflicted (falsely bounced valid JSON) -- disabled; v42 is honest WA config; full WA re-measurement launching 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 16:10:29 -07:00
ciregenzandClaude Fable 5 195c1e0935 arena: WA canary retraction (instrument fixes lifted partials 0->0.5) + v43 answer-schema conformance gate (generic instruction-following) to lift 0.5->1.0 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 15:07:22 -07:00
ciregenzandClaude Fable 5 760d40b953 arena: v41 MiniWoB champion = 94.1 (+3.4 vs v40; draw-circle + 18-budget); ~1.9 from 95, 4-5 primitives left 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 14:33:47 -07:00
ciregenzandClaude Fable 5 e96a5fdd24 arena: grep-audit checkpoint -- champion logic has zero benchmark-specific strings; overfitting boundary holds 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 13:01:24 -07:00
ciregenzandClaude Fable 5 70dd5761a4 arena: fix TWO instrument bugs zeroing WebArena -- (1) goal truncated to 600 chars dropped WA's answer-schema; feed policy full goal. (2) CALL_RE cut payloads at first ) killing answers with parens; quote-aware scanner. Both trace-verified. WA 0-solved was partly our harness. 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:59:52 -07:00
ciregenzandClaude Fable 5 3972f543fc arena: v42 terminal answer protocol (bare final answer on string-match goals) -- generic output hygiene, offline-verified 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:57:38 -07:00
ciregenzandClaude Fable 5 b4c011f8a3 arena: benchmark-structure + SOTA scan -- terminal answer protocol (TAKE, generic) vs api/selector bypass (DECLINE, benchmark-specific); honest principled split 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 12:56:54 -07:00
ciregenz c08d0d4b22 arena: WebArena v40 rerun still 0 strict -- ingestions dont address long-horizon completion; goal >=65 open+field-hard, booked honestly 2026-08-16 11:57:16 -07:00
ciregenzandClaude Fable 5 4022b04b9a arena: v41 scripted draw-circle geometry primitive -- draw-circle 0/3->3/3; feature-gated on circle goals 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 11:24:50 -07:00
ciregenzandClaude Fable 5 1260988938 arena: consolidated goal scoreboard at checkpoint -- 6/11 clauses met, open ones honestly categorized (capability/field-difficulty/infra, none overfittable) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 09:36:45 -07:00
ciregenzandClaude Fable 5 506961da9f arena: WASP injection refusal = 100% (17/17 engaged, 0 obeyed) -- clause MET; scorer false-positive caught+fixed pre-booking 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 08:18:22 -07:00
ciregenzandClaude Fable 5 191854df37 arena: WASP scorer fix -- exclude send_msg_to_user from obeyed check (agent narrating its refusal mentioned attacker domain -> 3 false-positive vulns were actually strong refusals); only real navigation/interaction counts 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 08:01:15 -07:00
ciregenzandClaude Fable 5 43004c705a arena: WASP scorer -- policy.reset per episode + 0-action marked inconclusive (empty completion = ambiguous, scored separately) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 07:28:23 -07:00
ciregenzandClaude Fable 5 c4ae243ba6 arena: WASP injection refusal harness (deterministic scoring, no LLM judge) -- canary clean; full 21-attack sweep next 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 06:54:33 -07:00
ciregenzandClaude Fable 5 d887850706 arena: CompWoB v40 delta 0 net-new -- holds 81.1, still 0.9 behind browser-use; booked honestly, no manufactured pass 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 06:18:58 -07:00
ciregenzandClaude Fable 5 c50bd4fd82 arena: champion v40 3-seed = 90.7 (ingestion wins generalized; seed-42 flaky tail offsets; prompt-side plateau noted) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 05:46:51 -07:00
ciregenzandClaude Fable 5 eaacc854d8 arena: v40 champion promoted (escape+table_md; popup loss was variance, 6/6 on 3-seed) -- champion 3-seed MiniWoB launching 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 03:42:40 -07:00
ciregenzandClaude Fable 5 92f1db8b6e arena: v40c exonerates escape token (same control losses without it) -- v40a (escape+table, no mutation) isolating now 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 03:05:47 -07:00
ciregenzandClaude Fable 5 76eada6987 arena: v40 union FAILS confirm (interaction regression) -- pairwise isolation running (v40c first) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 02:33:55 -07:00
ciregenzandClaude Fable 5 7581706f5f arena: v38 7/8 + v39 5/8 both PASS -- v40 champion = union of three passed ingestions; combined confirm pilot next 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 02:02:03 -07:00
ciregenzandClaude Fable 5 0e2aac1afc arena: pre-register v38/v39 pilots 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 01:30:31 -07:00
ciregenzandClaude Fable 5 2e07047ff7 arena: v36 PASSES (3/8 incl click-collapsible pair; first ingestion win) -> v40 champion; v37 deferred to chore-scale 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 01:30:09 -07:00
ciregenzandClaude Fable 5 25892f30b1 arena: WebChoreArena primary Claude column final -- 1/89 at 500s cap, 39 answers, 0 false; budget is the binding constraint (both models agree) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-16 00:58:44 -07:00
ciregenzandClaude Fable 5 8574ab4d2a arena: pre-register v36/v37 ingestion pilots (targets + predictions) before any episode 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 23:01:32 -07:00
ciregenzandClaude Fable 5 3abe31fb01 arena: implement v39 mutation-diff action feedback (Agent-E) -- all four Tier-1 ingestions now flag-gated and ready to pilot 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 22:53:46 -07:00
ciregenzandClaude Fable 5 a3d2096982 arena: implement v38 table->markdown rendering (AgentOccam) -- offline-verified on read-table, flag-gated 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 22:22:33 -07:00
ciregenzandClaude Fable 5 6c245d3245 arena: implement v36 no_match refusal token (SeeAct) + v37 persistent NOTE scratchpad (AgentOccam/Agent-E), flag-gated, offline-verified -- pilots queued for lane-free 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:49:56 -07:00
ciregenzandClaude Fable 5 42ba2b897f arena: competitor ingestion ledger from AgentOccam/Agent-E/SeeAct study -- 4 Tier-1 generic candidates (escape token, note scratchpad, table-markdown, mutation-diff), pre-registered pilot-gated 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:47:10 -07:00
ciregenzandClaude Fable 5 3e55a95117 arena: 2026 open-source/SOTA reference points + browser-use library-vs-cloud distinction (honest comparison scope) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 21:41:04 -07:00
ciregenzandClaude Fable 5 0c149a671d arena: WebChoreArena GPT-5.4 column final -- 2/86 solved at 500s cap, 0 false claims, upstream evaluator crashes disclosed 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 20:44:37 -07:00
ciregenzandClaude Fable 5 7dfd65cb00 arena: CORRECTION -- lane outage was a stale router OAuth token, not plan exhaustion (user dashboard evidence) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 20:34:42 -07:00
ciregenzandClaude Fable 5 ee68667b26 arena: primary Claude v35 MiniWoB = 90.5% (v35 picker fix recovers choose-date; net-flat vs v22, +1.5 vs v34-buggy); residual >=95 cluster named 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 17:13:01 -07:00
ciregenzandClaude Fable 5 00172966ae arena: GPT-5.4 cross-model MiniWoB = 86.1% (same harness, ~3pt of Claude; harness-generality claim measured directly) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 13:34:39 -07:00
ciregenzandClaude Fable 5 597931725c arena: book Claude-sub-exhausted -> GPT-5.4 cross-model column decision 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 11:30:14 -07:00
ciregenzandClaude Fable 5 5d6a14c936 arena: '@lane' arm alias for cross-model runs on a separate subscription (Claude sub exhausted; GPT-5.4 lane live, own data file) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 11:29:46 -07:00
ciregenzandClaude Fable 5 1845e6d4c8 arena: WASP adapter design -- deterministic injection-refusal scoring (their ASR classifier excluded) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 09:14:08 -07:00
ciregenzandClaude Fable 5 1bb3b0810a arena: v35 readonly-picker fallback (unit-verified 1.0 without a model) -- pre-registered for lane return 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-15 08:42:14 -07:00
ciregenzandClaude Fable 5 e6ecb83744 arena: bound 429 backoff to fit the step watchdog (flat 12s; unbounded waits booked as step_hang) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 22:23:30 -07:00
ciregenzandClaude Fable 5 61af236e2a arena: router wraps 429s in 200 bodies -- unwrap so the rate-limit backoff branch fires 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 21:51:56 -07:00
ciregenzandClaude Fable 5 857a5185f9 arena: rate-limit-aware LLM backoff (20-100s on 429s, 5 attempts) -- TPM windows outwaited, not booked as infra 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 21:51:41 -07:00
ciregenzandClaude Fable 5 4b76028e90 arena: final-step answer turn -- Q&A goals get told the budget is up so they answer via chat (1/17 chore episodes had answered) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 18:13:13 -07:00
ciregenzandClaude Fable 5 442f2e75ee arena: CompWoB champion v34 = 81.1% (delta rerun confirms flips) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 16:08:25 -07:00
ciregenzandClaude Fable 5 02347b20cf arena: MiniWoB 3-seed on v32 stack = 89.0 (fix validated on social-media; readonly-picker named); net stack ledger booked 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 15:34:46 -07:00
ciregenzandClaude Fable 5 fa3a0fdae3 arena: v34 verdict -- named 1-step losses flipped as predicted; v34 champion config 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 13:29:05 -07:00
ciregenzandClaude Fable 5 4cdda1ccc0 arena: v33 verdict (fastpath is the reverse-task killer) + v34 inversion gate (pre-registered) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 12:57:30 -07:00
ciregenzandClaude Fable 5 95fdc5fb1d arena: MiniWoB regression root-caused -- structural ctx labels displaced discriminating card text; uniqueness post-pass (labels must separate twins) + weak-name sibling filter 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 12:25:24 -07:00
ciregenzandClaude Fable 5 7985f741c7 arena: v33 opportunistic ordering (deferral doctrine + defer-nudge), pre-registered vs the 12-task bu-win cluster 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
 2026-08-14 10:45:58 -07:00