ciregenzandClaude Fable 5
dfe819b095
125 tasks, one external grader (WOB_REWARD_GLOBAL through BrowserGym), one recorder book, agents never grade themselves. Arms: our shipped perception ported line-for-line (deterministic + LLM), a browser-use-shaped flat-dump arm on the same model, the real browser-use agent attached over CDP to the same seeded page, and Stagehand wiring. Techniques already ingested from the competitor diffs, each landing measured points: clickable-but-unroled elements (canvas/svg -- their flat dump solved spatial tasks purely because the canvas was in it), coordinate actions (our click_point equivalent), input twins exempt from dedupe (two same-named password boxes are two fields), select options rendered on the row, and the page-text panel (our BrowserGetText equivalent). v1 25.6% -> v2 32.0% on identical tasks/model, wall clock down 3.8s -> 3.2s. Supervision is out-of-process: SIGALRM exceptions got swallowed inside stack retry loops twice, so a watcher kills the process tree when the recorder stops moving and resumes exactly the missing tasks. 502-dead LLM lanes book as infra, never as skill. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ
End-to-end tests (packaged app, macOS + Windows)
Playwright tests that launch the packaged OpenSwarm desktop app (the real built binary, asar + bundled python-env + real paths) and drive it the way a user would. The same specs run unchanged on macOS and Windows; CI builds the artifact per-OS, then runs these. No provider API key is needed (no agent turn), so the suite is hermetic and deterministic on a clean machine.
What it checks (per OS)
- Main window paints the React shell (first meaningful paint).
- The preload bridge (
window.openswarm) is exposed. - The real backend the app spawned reaches HTTP-ready (
/api/health/check-> 200). - Provenance: the running app's
getBuildInfo()sha matcheselectron/build-info.json. - App version is reported.
Run locally
- Build the app first (produces
electron/dist/...):- Windows:
pwsh scripts/build-app-win.ps1 - macOS:
bash scripts/build-app.sh
- Windows:
- Then:
cd e2e PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm ci # Electron ships its own Chromium npm test
Override the binary location with E2E_APP_PATH=/path/to/app if your build
output lives elsewhere. Auto-detection covers win-unpacked/OpenSwarm.exe and the
mac OpenSwarm.app variants.
CI
.github/workflows/e2e.yml runs this on a windows-latest + macos-latest
matrix: it builds the unsigned app, then runs the suite. Tag-driven signed
releases are covered separately by release-windows.yml / release-macos.yml.