ciregenzandClaude Opus 5
72c5effbff
Numbers from this box, with the unverifiable ones marked as such. The result that changes priorities: a deterministic accessibility-tree snapshot (agent-browser, Rust CLI, no LLM) perceives in ~50ms median against our BrowserFindComposer's 13,991ms median / 30,003ms p95 over 119 calls, and fills+verifies onlinegdb in 326ms -- a site our agent cannot reach at all. Also fills w3schools (810ms) and regex101 (31ms). That is a 40-450x perception gap with no model in the loop. Also recorded: all nine claimed repos verified to exist with matching stars and AGPL-compatible licences; browser-use benchmarked as an agent over 43 trials (we are faster on every site we reach, 9.8s vs 35.7s median, but our arm ran with its fallback disabled by dry-run so reach is not comparable); direct CDP evidence that deepl serves our browser a Cloudflare challenge while browser-use's fresh-per-run profile is never flagged; OpenCLI's API-first-with-browser-fallback design as the route to a learned path that actually replays; and why Polar was not benchmarked (their ToS forbids using the product to build a competing one, and their 98.0 concatenates two benchmarks and appears on neither leaderboard). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
End-to-end tests (packaged app, macOS + Windows)
Playwright tests that launch the packaged OpenSwarm desktop app (the real built binary, asar + bundled python-env + real paths) and drive it the way a user would. The same specs run unchanged on macOS and Windows; CI builds the artifact per-OS, then runs these. No provider API key is needed (no agent turn), so the suite is hermetic and deterministic on a clean machine.
What it checks (per OS)
- Main window paints the React shell (first meaningful paint).
- The preload bridge (
window.openswarm) is exposed. - The real backend the app spawned reaches HTTP-ready (
/api/health/check-> 200). - Provenance: the running app's
getBuildInfo()sha matcheselectron/build-info.json. - App version is reported.
Run locally
- Build the app first (produces
electron/dist/...):- Windows:
pwsh scripts/build-app-win.ps1 - macOS:
bash scripts/build-app.sh
- Windows:
- Then:
cd e2e PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm ci # Electron ships its own Chromium npm test
Override the binary location with E2E_APP_PATH=/path/to/app if your build
output lives elsewhere. Auto-detection covers win-unpacked/OpenSwarm.exe and the
mac OpenSwarm.app variants.
CI
.github/workflows/e2e.yml runs this on a windows-latest + macos-latest
matrix: it builds the unsigned app, then runs the suite. Tag-driven signed
releases are covered separately by release-windows.yml / release-macos.yml.