WEBDIENSTE/openswarm
1
0
Fork 0
mirror of https://github.com/openswarm-ai/openswarm.git synced 2026-09-22 09:34:53 +02:00
Code Issues Packages Projects Releases Wiki Activity

arena: product-write attempt -- found the real harness (vs_bench.py, cooperative editors, verified readback) but stack won't boot headless (backend deps fail, needs Electron); handoff = user runs app, then vs_bench.py gives the honest product-write number

Browse Source
This commit is contained in:
ciregenz
2026-08-17 19:33:42 -07:00
parent d488c8fc18
commit aa78bff506
1 changed files with 16 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
e2e/browser-v3/arena/ARENA.md
+16
View File
@@ -1190,3 +1190,19 @@ full app (Electron frontend + backend + ws_manager) pointed at the self-hosted s
integration project, not a headless run. Honest final: research/generic-harness numbers
(MiniWoB/CompWoB/WebArena/writes) = benchmark harness; product write capability = live-proven,
not cheaply benchmarkable headless. The arena was never the product; it was the fair-scoring proxy.
## Product-write measurement attempt (2026-08-17): harness found, stack won't boot headless
GOOD NEWS: the exact reproducible product-write harness ALREADY EXISTS -- e2e/browser-v3/vs_bench.py
runs "one OpenSwarm trial through its REAL dispatch path, verified by reading the card back", head-
to-head vs browser-use, on COOPERATIVE demo editors from bench.py (quilljs playground, tinymce,
ckeditor, codemirror -- "write X in the editor"): safe, no-account, low-risk live writes, the exact
class the research recommended. This is the honest product-write number, NOT the stripped arena's
7-14%. BLOCKER: it needs the app stack at :8326, which won't boot in this autonomous session --
run.sh backend dep-install FAILED ("Failed to install Python dependencies"), the backend .venv
can't import backend.main, and browser cards need Electron (display + logged-in session) anyway.
CONFIRMS the architecture finding: the product's write capability is welded to the running app; it
cannot be measured from a stripped headless harness -- which is precisely why it is live-proven and
why the arena number understates it. HANDOFF: with the user's app running (their working dev env
handles deps/Electron/login), `python e2e/browser-v3/vs_bench.py` yields the real product-write
number + a browser-use head-to-head, verified by readback, on the cooperative editors. That is the
one measurement that fairly scores the shipped product's writes.