openswarm/e2e/browser-v3/HANDOFF.md

Handoff: browser v3, state as of 2026-08-06

Branch eric/browser-merged, 45 commits past origin/eric/dev, pushed. Browser suite 707 passing, tsc 0, linter no new violations, tree clean. The goal and the harness are described in README.md.

Six of nine criteria pass. Three are open, and none of them is open because the fix is unknown.

Scorecard

#	criterion	target	before	now
1	composer reach	>=90%	57%	93% (28/30)	PASS
2	verified writes	>=95%	no honest data	LinkedIn proven end to end; N too small	OPEN
3	false success	0	unknown	0 in every live round	PASS
4	median write	<=12s	~21s	1.99s (p95 12.2s)	PASS
5	cold start tier-0/1	<=3s	~16s	0.13s	PASS
6	other_ms	-50%	none	-70% (417 -> 126ms)	PASS
7	infra flake	<=1%	60%	34 runs, 0 failures; sample too small	OPEN
8	holdout	>=80%, <=10pt	untested	87%, 6pt gap	PASS
9	learned path	remove or >=50%	0/55	recording 4/4 = 100%; replays 0 of 2	OPEN

Per-site reach at N=5 (criterion 1): x 5/5, linkedin 5/5, reddit 5/5, instagram 5/5 (was 1/10), youtube 4/5, twitch 4/5 (was 0/3). gmail, tiktok and substack are excluded, see Exclusions.

Timing at n=28 successful runs (criterion 4/6): wall median 1988ms / p95 12180ms; browser tools median 1621ms; other_ms median 126ms, down from 417ms. Wall fell too, so nothing moved from one bucket into another.

The finding that matters most

Several "product failures" were the measuring instrument. This is the single most useful thing to carry forward, because it recurred six times in one session and each instance cost hours chasing a bug that did not exist.

The worst case: criterion 2 failed on LinkedIn for days. LinkedIn was never broken. The canary proved a write landed by grepping the backend log for the marker string, and the backend does not log page text. A grep for every canary marker ever generated, across every backend log on the machine, returns zero lines. The audit could only ever answer "could not look", and that was being read as a product failure. probe_evidence.py settled it: the session API carries the model's answer and never the tool results, so auditing it is asking the same model whose claim is under audit.

Every dead grep found, and what each did:

where	needle	occurrences in source
canary delivered	DELIVERY CONFIRMED	0
canary saw_page	two [browser-action] X strings	0
canary receipt	only the fast-lane string; 2 of 3 producers missed	1 of 3
skillstats replay_full	replay(ed|ing) N steps	0
bench infra_browser	card is unavailable	0
stack.sh status	pgrep -fc (no such flag on macOS)	printed 0 over a live stack

verify_markers.py now checks all 32 harness literals against the source that prints them: 32/32 present. Run it before trusting any number.

What is open, why, and exactly what to do

Criterion 2, verified writes

The instrument was rebuilt (950755af) and the first result was linkedin PASS: posted, receipt-verified, deleted, verified gone. That is the first clean end-to-end LinkedIn round this project has recorded. One site is not >=95% across sites, so the criterion is not met.

Blocked on nothing technical. Two things to do first:

1. Verify reddit's Title field name. af68dcd0 prefers the textbox whose accessible name contains the field word the task used, and requires exactly one match. The unit tests use an assumed listing ([21]<textbox "Title" />) that has never been confirmed against the live page; the backend only logs textboxes=3. If reddit's title input is named something else, the fix silently does nothing. reddit is 1 of the 3 sites in this denominator.
2. Then ROUNDS=7 c2_rounds.sh and c2_tally.py.

Criterion 7, infrastructure flake

Last clean sample: 34 of 108 runs, 0 infra failures, 0 backend restarts. Encouraging but partial. An earlier 5.4% reading was contaminated: a second OpenSwarm checkout was up on :8324 and the shared 9router on :20128 was being evicted.

Needs a box with nothing on :8324, then N=12 c7_run.sh for the full 108.

Criterion 9, learned fast path

The starting baseline, over 427 gate decisions on dry sweeps: 95 eligible, 0 recorded, 0 replays, with one refusal reason (host empty or no robust steps). Read on its own that number is misleading, see "measure this on a LIVE run" below; but the refusal reason was a real bug.

The gate was never the problem. Every eligible run died inside record_skill: distill_steps reads clicked_name, and browser_send_script.py wrote the element name into result_summary prose only. An unnameable click truncates the distillation, truncation drops the typing steps, and the navigation-only remainder is correctly refused. Fixed in 438a96eb, proven directly:

OLD shape (no clicked_name) -> []
NEW shape -> ['BrowserNavigate', 'BrowserClickByName', 'BrowserClickByName']

Safety held rather than added: a BrowserClickIndex distills to a name-only BrowserClickByName, so the payload never enters the skill, and the send click's tool name matches no distill branch, so a replay cannot re-fire a send.

Recording is PROVEN live, at 100%. On the live canary rounds (r4): 14 runs reached the gate, 4 were eligible, 4 recorded, 0 refused. Two skills persist on disk with exactly the shape the unit test predicts, both timestamped after the fix:

x.com            BrowserNavigate(https://x.com/compose/post)
                 BrowserClickByName(textbox "Post text")
www.linkedin.com BrowserNavigate(https://www.linkedin.com/feed/?shareActive=true)
                 BrowserClickByName(textbox "Text editor for creating content")

Measure this on a LIVE run, never a dry one. Recording is gated on delivery_verified, which a dry run can never produce, so a dry sweep correctly records nothing. Reading a dry log as a verdict is how this was first misreported as "0/95, the fix did not fire". skillstats.py now prints a note when it sees that shape. (It also had two counting bugs of its own, fixed: it counted the slots unfillable FAILURE line under a label that read "skill matched", and it never surfaced quarantines.)

What still fails is REPLAY: 0 of 2.

1. x.com truncates to a bare navigate. is_replay_boundary flags the composer click as irreversible because the accessible name Post text contains "post". It is a textbox, and focusing a textbox is reversible, but the guard matches on the name only. The prefix therefore replays a single navigate, which prestage already does in 0.13s.
2. linkedin quarantines. replay step failed (BrowserClickByName: No element matching role="textbox" name="Text editor for creating content"). LinkedIn's composer is lazily mounted and does not exist at navigate time; the recorded skill never learned the opener click that reveals it. Quarantine is correct behaviour here, not a bug.

So criterion 9 clears the >=50% recording bar and fails the "successful replays and measured benefit" half. Before investing in either replay fix, answer the question that decides it: prestage already reaches a tier-0/1 composer in 0.13s, so what is a replay actually worth? If the answer is "nothing", removal is the honest path and the criterion explicitly allows it. The two browser-memory endpoints have no frontend caller (grep of frontend/src and electron/ returns nothing), so removal is not user-visible.

Product bugs found by the fixed instrument

• reddit: the task named a field and nobody read it (af68dcd0). "create a text post whose title is exactly X" filled the body, Title stayed empty, and reddit's submit stayed DISABLED on every attempt. A field word from the user's own sentence now beats the compose-shape guess. 6 tests, including that no-hint behaviour is byte-identical.
• disqus: a stalled top document denied its own child frames (7ba99ba4). The still-loading retry rethrew on a second failure, killing find_composer before the child-frame search ran. An ad-heavy page keeps the top document loading while the composer, in an embedded iframe with its own load state, is perfectly readable.

Known false negatives, filed not fixed

Both under-claim rather than over-claim, so neither violates criterion 3, but both read as drift on every live round.

• A torn-down browser card wedges, and navigate lies about it. Reproduced 4x: the reply is {"text":"Navigated to https://x.com/home","url":"https://x.com/home"} in ~80ms while the webview never leaves reddit. Anything trusting that reply reads the wrong page and answers confidently about it. The audit now re-reads location.href and requires a host match.
• Delete reports failure on deletions that provably worked. X, 2 for 2: "the deletion was never confirmed", and an independent read shows the post gone both times. LinkedIn showed the same shape.

Exclusions (predefined, never quietly dropped)

gmail, substack and tiktok are signed out; tiktok is additionally captcha-walled. That is 14 of 45 rows on the known suite. Each exclusion is judged on the page's own evidence, never on the product's claim. Solving a bot-detection challenge is off-limits, so a captcha-walled page is one this system is choosing not to reach, and scoring it against reach would charge us for a rule we intend to keep.

Account hygiene

Every marker written to a real account during this work was cleaned up and independently verified gone: X profile scan shows 0 markers with the newest genuine post predating the tests, and the LinkedIn post permalink returns nav chrome with no content.

Personal handles are not committed. Both live sites read their account from the environment (OSW_CANARY_X_HANDLE, OSW_CANARY_REDDIT_HANDLE), and an unset handle makes the round refuse loudly rather than audit a malformed URL.

Environment notes

• The stack is backend :8326, webpack :3026, Electron on its own --user-data-dir. It never touches :8324 or :3000, which belong to whatever else is on the machine.
• The backend was SIGTERMed three times mid-measurement, always cleanly, always with nothing in its log. Cause unknown. stack.sh now supervises it and writes [stack] BACKEND RESTARTED into the log the harness slices, so bench.py's infra_backend bucket can exclude any trial spanning a restart instead of scoring it as a product failure.
• stack.sh down once matched pkill -f "uvicorn backend.main", which is exactly what the other checkout runs, and it executed while that checkout was live. It is now scoped to this stack's own ports and profile and cannot match a process it did not start.

Next executable steps, in order

1. verify_markers.py (32/32 expected).
2. Confirm reddit's Title field name against the live page.
3. Decide criterion 9: is a replay worth anything against a 0.13s prestage? If not, remove.
4. N=12 c7_run.sh on a box with nothing on :8324. Gives criteria 1, 4, 5, 6, 7.
5. skillstats.py on a LIVE round's log (not the dry sweep) for criterion 9.
6. ROUNDS=7 c2_rounds.sh then c2_tally.py. Gives criterion 2.
7. N=2 c8_run.sh to re-check criterion 8 after the disqus fix.