ciregenzandClaude Opus 5
830a995c6a
Every other measurement in this session, ours and the competitors', was scored by a harness I wrote and could therefore flatter. MiniWoB's reward comes from the task definition, so it is ground truth nobody in this repo controls. Working setup recorded because the obvious one fails: pypi browsergym cannot build greenlet 3.0.3 on py3.13, so it needs a 3.12 venv, plus miniwob-plusplus's HTML served locally. 341 envs register (125 MiniWoB, 215 AssistantBench, 0 WebArena -- that one needs self-hosted sites). The result reframes the competitive work: a ~20-line deterministic axtree heuristic with NO model solves 5/15 = 33% at 1.44s median and zero cost. Published agent scores sit at 70-90%, so the real question for any model-driven agent, ours included, is what it buys over the free baseline at 10-40x the latency. miniwob_baseline.py takes task names as argv so a real policy can be dropped in for a like-for-like number on the same 15 or all 125. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
End-to-end tests (packaged app, macOS + Windows)
Playwright tests that launch the packaged OpenSwarm desktop app (the real built binary, asar + bundled python-env + real paths) and drive it the way a user would. The same specs run unchanged on macOS and Windows; CI builds the artifact per-OS, then runs these. No provider API key is needed (no agent turn), so the suite is hermetic and deterministic on a clean machine.
What it checks (per OS)
- Main window paints the React shell (first meaningful paint).
- The preload bridge (
window.openswarm) is exposed. - The real backend the app spawned reaches HTTP-ready (
/api/health/check-> 200). - Provenance: the running app's
getBuildInfo()sha matcheselectron/build-info.json. - App version is reported.
Run locally
- Build the app first (produces
electron/dist/...):- Windows:
pwsh scripts/build-app-win.ps1 - macOS:
bash scripts/build-app.sh
- Windows:
- Then:
cd e2e PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm ci # Electron ships its own Chromium npm test
Override the binary location with E2E_APP_PATH=/path/to/app if your build
output lives elsewhere. Auto-detection covers win-unpacked/OpenSwarm.exe and the
mac OpenSwarm.app variants.
CI
.github/workflows/e2e.yml runs this on a windows-latest + macos-latest
matrix: it builds the unsigned app, then runs the suite. Tag-driven signed
releases are covered separately by release-windows.yml / release-macos.yml.