ciregenzandClaude Fable 5
9456402524
bu_real.score() reads WOB_REWARD_GLOBAL, which does not exist on AssistantBench pages, so browser-use's 0.000 is a harness bug not a performance result; ours (env-reward path) is likely fine but both must be re-scored through the AssistantBench reward before any claim. Caught by user skepticism -- exactly the tunnel-vision guard the method depends on. MiniWoB numbers are unaffected (that path is correct there). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
End-to-end tests (packaged app, macOS + Windows)
Playwright tests that launch the packaged OpenSwarm desktop app (the real built binary, asar + bundled python-env + real paths) and drive it the way a user would. The same specs run unchanged on macOS and Windows; CI builds the artifact per-OS, then runs these. No provider API key is needed (no agent turn), so the suite is hermetic and deterministic on a clean machine.
What it checks (per OS)
- Main window paints the React shell (first meaningful paint).
- The preload bridge (
window.openswarm) is exposed. - The real backend the app spawned reaches HTTP-ready (
/api/health/check-> 200). - Provenance: the running app's
getBuildInfo()sha matcheselectron/build-info.json. - App version is reported.
Run locally
- Build the app first (produces
electron/dist/...):- Windows:
pwsh scripts/build-app-win.ps1 - macOS:
bash scripts/build-app.sh
- Windows:
- Then:
cd e2e PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm ci # Electron ships its own Chromium npm test
Override the binary location with E2E_APP_PATH=/path/to/app if your build
output lives elsewhere. Auto-detection covers win-unpacked/OpenSwarm.exe and the
mac OpenSwarm.app variants.
CI
.github/workflows/e2e.yml runs this on a windows-latest + macos-latest
matrix: it builds the unsigned app, then runs the suite. Tag-driven signed
releases are covered separately by release-windows.yml / release-macos.yml.