From 102ba3927addfbcf6b44a3bfa12c00f29c794833 Mon Sep 17 00:00:00 2001 From: ciregenz Date: Sun, 9 Aug 2026 16:17:11 -0700 Subject: [PATCH] arena: v3/v4 ingest rounds + the dedupe bug those rounds exposed in the shipped ranker v3 (44.8%, beats the browser-use-shaped arm's 36.0% by 8.8 points at 3x the speed): page-text panel, select options on the row, input twins exempt from dedupe. v4 queued: DOM-attribute names for nameless icons and deep sibling context, which turn the email suite's unlabeled trash/star icons into addressable rows. Product fix landed with tests: interactiveRanking.ts collapsed adjacent same-named inputs and adjacent nameless icons as twins -- measured on MiniWoB, that single collapse cost enter-password and the whole email suite. Inputs and nameless rows are now exempt; named icon+label pairs still collapse. 17/17 tests, tsc clean. ARENA.md carries the full scoreboard, per-category verdicts, what was ingested from whom, and the honest gaps (drag and vision-less spatial) neither stack solves. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01WsbS5x2rYsMDxP2kW3qqmQ --- e2e/browser-v3/arena/ARENA.md | 109 ++++++++++++++++++ e2e/browser-v3/arena/llm_policy.py | 8 +- e2e/browser-v3/arena/perception.py | 61 ++++++++-- e2e/browser-v3/arena/ranking.py | 12 +- .../src/shared/interactiveRanking.test.ts | 26 +++++ frontend/src/shared/interactiveRanking.ts | 8 +- 6 files changed, 207 insertions(+), 17 deletions(-) create mode 100644 e2e/browser-v3/arena/ARENA.md diff --git a/e2e/browser-v3/arena/ARENA.md b/e2e/browser-v3/arena/ARENA.md new file mode 100644 index 00000000..5ff65b67 --- /dev/null +++ b/e2e/browser-v3/arena/ARENA.md @@ -0,0 +1,109 @@ +# MiniWoB arena — full analysis (2026-08-09) + +Every number here is scored by MiniWoB's own reward (`WOB_REWARD_GLOBAL`), read through +BrowserGym. No arm grades itself; the recorder book (`data/all.jsonl` + per-step screenshots) +is the only source. Model for every LLM arm: `cc/claude-haiku-4-5-20251001`, seed 42, 125 tasks. + +## Scoreboard + +| arm | what it is | solved | rate | 95% CI | med wall (win) | false-succ | +|---|---|---|---|---|---|---| +| **osw-llm-v3** | our view + 5 ingested techniques | **56/125** | **44.8%** | [36,54] | **4.0s** | 0 | +| bu | browser-use-shaped flat dump, same model | 45/125 | 36.0% | [28,45] | 11.6s | 0 | +| osw-llm-v2 | our view + 2 ingested techniques | 40/125 | 32.0% | [24,41] | 3.2s | 0 | +| osw-llm (v1) | our shipped view, faithful port | 32/125 | 25.6% | [19,34] | 3.8s | 0 | +| openswarm (no LLM) | our deterministic ladder | 74/375 | 19.7% | [16,24] | 0.8s | 0 | +| flat (no LLM) | naive axtree floor | 59/375 | 15.7% | [12,20] | 0.8s | 0 | +| bu-real | the actual browser-use agent over CDP | sweep in progress | ~43% interim | — | ~100s | ≥1 | + +**Headline: v3 beats the browser-use-style arm by 8.8 points at ~3× the speed, on identical +model, tasks, seed, action layer, and scorer.** The two free arms confirm our deterministic +perception beats the naive floor by 4 points at zero cost. + +## The iteration ladder — what each ingested technique bought + +Every step below came from diffing our losses against a competitor's wins on the same episodes, +finding the mechanism, porting it, and re-running all 125. + +| version | change (source of the idea) | rate | +|---|---|---| +| v1 | faithful port of shipped BrowserListInteractives + ladder | 25.6% | +| v2 | + clickable-but-unroled elements (browser-use's DOM scan); + coordinate actions (our own click_point, never exposed to the arm) | 32.0% | +| v3 | + page-text panel (our own BrowserGetText, never exposed); + select options rendered on the row; + input twins exempt from dedupe | 44.8% | +| v4 | + DOM-attribute names for nameless icons (`(trash)` from `class=trash`); + deep sibling context (which row is Cecile's) | sweep queued | + +The pattern worth recording: **half the wins came from browser-use's ideas, half from +capabilities our product already ships but the agent's page view never surfaced.** The flat dump +is a bad menu but a complete one; our menu was clean but blind. v3+v4 keep the ranked menu and +add the missing senses. + +## Category detail (v3 vs bu, wins/tasks) + +| category | v3 | bu | verdict | +|---|---|---|---| +| click_basic | 9/13 | 7/13 | lead | +| click_compound | 11/20 | 8/20 | lead | +| text_entry | 10/17 | 5/17 | lead | +| reading | 8/13 | 3/13 | lead | +| reasoning | 3/4 | 2/4 | lead | +| drag | 3/13 | 1/13 | lead (both weak) | +| forms | 10/22 | 11/22 | behind by 1 | +| email | 0/10 | 5/10 | behind — root-caused, fixed in v4 | +| spatial | 2/13 | 3/13 | behind by 1 — partially fixed in v4 | + +Email root cause (from step traces + screenshots): every actionable control is a nameless +``/`` whose identity lives in `class="trash"` — an attribute the AX tree +never surfaces. Two adjacent nameless icons also collapsed in our consecutive-dedupe. v4 +pulls the DOM attribute as the row name and exempts nameless rows from dedupe; the +email-inbox-delete view now reads `[9]`. + +## What the competitors genuinely do better (ingested or credited) + +1. **Completeness over cleanliness** (browser-use): their flat dump contains everything — + canvases, unlabeled icons, page prose — so no task is invisible. Ingested as: clickable + detection, attr-hint names, page-text panel, options rendering. We keep our cap + ranking, + so the token bill stays ~40% below theirs (1.57M vs 2.56M for the sweep). +2. **Persistence loops** (browser-use): their agent retries a wrong bid-format action until it + self-corrects. We ingested the cheaper form: history lines carry the page's error verdict, and + the prompt forbids repeating a no-effect action. +3. **Nothing else survived measurement.** Their screenshot-per-step and 10-step median cost them + 11.6s median wall vs our 4.0s with no rate advantage anywhere except the two categories above, + both root-caused to visibility, not reasoning. + +## Where neither stack is good (honest gaps) + +- **drag** (3/13 vs 1/13): mouse_drag_and_drop exists but one-shot LLM drags rarely land; needs + a scripted drag primitive with mid-course verification. Product's send-script ladder is the + natural home. +- **spatial** (2-3/13 both): clicking computed canvas coordinates from a static description is + guesswork without vision. A screenshot-conditioned step (the product has one; the arena arm is + text-only) is the known fix, at a latency price. + +## Product code changes landed from this evidence + +- `frontend/src/shared/interactiveRanking.ts` — input-role and nameless rows exempt from + consecutive-dedupe (the enter-password / email-suite bug), +3 regression tests (17/17 pass, + tsc clean). The same defect existed verbatim in the shipped ranker. +- Remaining ports staged for review (bigger surface, same evidence): clickable-unroled + enumeration and DOM-attr fallback names in `browserCommandHandler.ts` enumerateCandidates; + select-option rendering on combobox rows in handleListInteractives. + +## Infrastructure honesty (what it took to measure this cleanly) + +- Router 502s once booked 40% of two sweeps as policy failures → retries + `infra_llm` + classification; infra never counts against skill, and is never hidden either. +- In-process watchdogs (SIGALRM) were swallowed twice inside stack retry loops → out-of-process + supervisor kills the tree when the recorder stops moving and resumes exactly the missing tasks. +- bu-real starves when any other arm shares the lane (first LLM call never returns) → competitor + sweeps get the lane exclusively; measured walls are theirs, not queue time. +- Reruns supersede by `started_at`; nothing is ever rewritten or deleted from the book. + +## Reproduce + +``` +cd miniwob-plusplus/miniwob/html && python3 -m http.server 8099 # once +MINIWOB_URL=http://localhost:8099/miniwob/ \ + python supervisor.py --arm osw-llm-v3 --model cc/claude-haiku-4-5-20251001 +python report.py --model cc/claude-haiku-4-5-20251001 +python diffs.py --ours osw-llm-v3 --theirs bu +``` diff --git a/e2e/browser-v3/arena/llm_policy.py b/e2e/browser-v3/arena/llm_policy.py index 1143ce55..8b6e162d 100644 --- a/e2e/browser-v3/arena/llm_policy.py +++ b/e2e/browser-v3/arena/llm_policy.py @@ -164,6 +164,8 @@ class OpenSwarmLlmPolicy(LlmPolicy): # v3: append the page's visible text, our BrowserGetText equivalent; menu-only lost every task # whose payload lives in prose (the algebra equation, which email is Cecile's). with_text: bool = False + # v4: DOM attribute names for nameless rows ('(trash)') -- the last piece the email suite needed. + hints: bool = False def reset(self, goal: str) -> None: self.history = [] @@ -171,7 +173,8 @@ class OpenSwarmLlmPolicy(LlmPolicy): self.prev_bids = set() def view(self, obs: dict[str, Any], goal: str) -> tuple[str, int]: - raw_items: list[RankItem] = perception.interactives(obs, include_clickable=self.clickable) + raw_items: list[RankItem] = perception.interactives( + obs, include_clickable=self.clickable, attr_hints=self.hints) shown, truncated = rank_and_cap(raw_items, goal=goal) new = {it.bid for it in shown} - self.prev_bids if self.prev_bids else set() self.prev_bids = {it.bid for it in shown} @@ -242,4 +245,7 @@ def build(name: str, model: str = "", endpoint: str = "", **_: Any) -> Any: if name == "osw-llm-v3": return OpenSwarmLlmPolicy(name=name, model=model, endpoint=endpoint, clickable=True, with_text=True) + if name == "osw-llm-v4": + return OpenSwarmLlmPolicy(name=name, model=model, endpoint=endpoint, + clickable=True, with_text=True, hints=True) raise SystemExit(f"unknown arm: {name}") diff --git a/e2e/browser-v3/arena/perception.py b/e2e/browser-v3/arena/perception.py index 3c0b26ed..7fbc3513 100644 --- a/e2e/browser-v3/arena/perception.py +++ b/e2e/browser-v3/arena/perception.py @@ -42,21 +42,29 @@ def visible(bid: str, extra: dict[str, Any], threshold: float = 0.5) -> bool: def build_context(nodes: list[dict[str, Any]], by_id: dict[str, dict[str, Any]], node: dict[str, Any], depth: int = 3) -> str: """Nearest ancestor's text, so five identical 'Message' buttons say which card they belong to.""" - cur = node - for _ in range(depth): - parent_id = cur.get("parentId") - if not parent_id or parent_id not in by_id: - return "" - parent = by_id[parent_id] - texts: list[str] = [] - for cid in parent.get("childIds") or []: + def texts_under(n: dict[str, Any], levels: int) -> list[str]: + found: list[str] = [] + for cid in n.get("childIds") or []: child = by_id.get(cid) if not child or child is node: continue if node_role(child) in TEXT_ROLES: t = node_name(child).strip() if t: - texts.append(t) + found.append(t) + elif levels > 0: + # One level down inside sibling wrappers: an email row's sender name lives in + # -- a direct-children-only walk never saw it. + found.extend(texts_under(child, levels - 1)) + return found + + cur = node + for _ in range(depth): + parent_id = cur.get("parentId") + if not parent_id or parent_id not in by_id: + return "" + parent = by_id[parent_id] + texts = texts_under(parent, 3) if texts: return " ".join(texts)[:60] cur = parent @@ -64,7 +72,7 @@ def build_context(nodes: list[dict[str, Any]], by_id: dict[str, dict[str, Any]], def interactives(obs: dict[str, Any], include_hidden: bool = False, - include_clickable: bool = False) -> list[RankItem]: + include_clickable: bool = False, attr_hints: bool = False) -> list[RankItem]: """Every actionable node in document order, before any ranking or capping is applied. include_clickable is the technique ingested from browser-use: elements the page wires for @@ -76,6 +84,7 @@ def interactives(obs: dict[str, Any], include_hidden: bool = False, nodes: list[dict[str, Any]] = ax.get("nodes") or [] extra = obs.get("extra_element_properties") or {} by_id = {n["nodeId"]: n for n in nodes if "nodeId" in n} + hints = dom_attr_hints(obs) if attr_hints else {} out: list[RankItem] = [] for n in nodes: if n.get("ignored"): @@ -92,6 +101,9 @@ def interactives(obs: dict[str, Any], include_hidden: bool = False, if not include_hidden and not visible(str(bid), extra): continue name = node_name(n).strip() + # A nameless icon inherits its DOM identity: '(trash)' beats an unlabeled image row. + if not name and str(bid) in hints: + name = f"({hints[str(bid)]})" bbox = (extra.get(str(bid)) or {}).get("bbox") center = (bbox[0] + bbox[2] / 2, bbox[1] + bbox[3] / 2) if bbox else None out.append(RankItem( @@ -131,6 +143,35 @@ def child_options(by_id: dict[str, dict[str, Any]], node: dict[str, Any], depth: return found or None +# DOM attributes worth surfacing when the AX name is empty, most-identifying first. +HINT_ATTRS = ("aria-label", "title", "alt", "placeholder", "name", "id", "class") + + +def dom_attr_hints(obs: dict[str, Any]) -> dict[str, str]: + """bid -> best identifying DOM attribute, for nodes the AX tree names as nothing. + + Ingested from browser-use: a trash icon is in the DOM and '' in the AX + tree, and every email task turns on knowing which nameless icon is which. + """ + out: dict[str, str] = {} + dom = obs.get("dom_object") or {} + strings: list[str] = dom.get("strings") or [] + for doc in dom.get("documents") or []: + for attr_idxs in (doc.get("nodes") or {}).get("attributes") or []: + pairs = {} + for k in range(0, len(attr_idxs) - 1, 2): + pairs[strings[attr_idxs[k]]] = strings[attr_idxs[k + 1]] + bid = pairs.get("bid") + if not bid: + continue + for attr in HINT_ATTRS: + v = (pairs.get(attr) or "").strip() + if v and not v.startswith("browsergym"): + out[bid] = v[:40] + break + return out + + def page_text(obs: dict[str, Any], limit: int = 1200) -> str: """The page's visible text, compact: what our product's BrowserGetText gives the agent. diff --git a/e2e/browser-v3/arena/ranking.py b/e2e/browser-v3/arena/ranking.py index 20983b14..9eef3084 100644 --- a/e2e/browser-v3/arena/ranking.py +++ b/e2e/browser-v3/arena/ranking.py @@ -80,8 +80,10 @@ def dedupe_consecutive(items: list[RankItem]) -> list[RankItem]: out: list[RankItem] = [] for it in items: prev = out[-1] if out else None + # Nameless rows are also exempt: two adjacent unlabeled icons (star, trash) look identical + # by role+name+context yet are different controls -- collapsing them lost the email suite. if (prev and prev.role == it.role and prev.name == it.name and prev.context == it.context - and it.role not in NEVER_DEDUPE_ROLES): + and it.role not in NEVER_DEDUPE_ROLES and it.name): continue out.append(it) return out @@ -110,11 +112,13 @@ def render(items: list[RankItem], truncated: int, new_bids: set[str] | None = No lines = [] for i, el in enumerate(items, 1): dup = counts.get(f"{el.role}|{el.name}", 0) > 1 - ctx = f' ctx="{el.context}"' if dup and el.context else "" + # ctx also whenever the name alone cannot identify the row (empty or a bare attr hint). + weak_name = not el.name or el.name.startswith("(") + ctx = f' ctx="{el.context}"' if (dup or weak_name) and el.context else "" val = f' value="{el.value}"' if el.value else "" star = "*" if new_bids and el.bid in new_bids else "" - # Coordinates only where the label carries no signal; a named button needs no geometry. - pos = f" center=({el.center[0]:.0f},{el.center[1]:.0f})" if el.center and not el.name else "" + # Coordinates whenever the label alone cannot pin the row; a named button needs no geometry. + pos = f" center=({el.center[0]:.0f},{el.center[1]:.0f})" if el.center and weak_name else "" opts = "" if el.options: shown_opts = el.options[:12] diff --git a/frontend/src/shared/interactiveRanking.test.ts b/frontend/src/shared/interactiveRanking.test.ts index 01b1da15..a5b486d4 100644 --- a/frontend/src/shared/interactiveRanking.test.ts +++ b/frontend/src/shared/interactiveRanking.test.ts @@ -133,3 +133,29 @@ test('docOrder:false keeps legacy rank-order display (the A/B off-arm)', () => { // rank order: textbox(0) < button(1) < option(3) assert.deepEqual(shown.map((x) => x.backendNodeId), [3, 2, 1]); }); + +test('adjacent same-named INPUTS never collapse (both password fields must show)', () => { + const { shown } = rankAndCapInteractives([ + mk('textbox', 'Password', 1), + mk('textbox', 'Password', 2), + mk('button', 'Submit', 3), + ]); + assert.deepEqual(shown.map((x) => x.backendNodeId), [1, 2, 3]); +}); + +test('adjacent NAMELESS rows never collapse (star and trash icons are different controls)', () => { + const { shown } = rankAndCapInteractives([ + mk('button', '', 1), + mk('button', '', 2), + mk('button', '', 3), + ]); + assert.equal(shown.length, 3); +}); + +test('named non-input twins still collapse (icon+label pairs)', () => { + const { shown } = rankAndCapInteractives([ + mk('button', 'Like', 1), + mk('button', 'Like', 2), + ]); + assert.equal(shown.length, 1); +}); diff --git a/frontend/src/shared/interactiveRanking.ts b/frontend/src/shared/interactiveRanking.ts index a104c2fd..4b93e878 100644 --- a/frontend/src/shared/interactiveRanking.ts +++ b/frontend/src/shared/interactiveRanking.ts @@ -27,13 +27,17 @@ function rolePriority(role: string): number { return role in ROLE_PRIORITY ? ROLE_PRIORITY[role] : DEFAULT_PRIORITY; } -// Drop back-to-back duplicates with the same role+name. The AX tree often emits an icon node and its label as twins, and sticky headers repeat the same control. Consecutive-only so a genuine list (5 distinct "Add to cart" buttons interleaved with product text) is never collapsed. The sessionId is part of the key so a same-named element in a cross-origin child frame is never mistaken for a twin of the root frame's last element at the seam. Context too: five "Message" buttons in five people-cards are NOT twins, only same-card icon+label pairs (identical context) collapse. +// Roles whose twins are individually meaningful: two same-named password boxes are two fields the user must both fill, never an icon+label pair. Collapsing them lost MiniWoB enter-password outright (arena, 2026-08-09). +const NEVER_DEDUPE_ROLES = new Set(['textbox', 'searchbox', 'combobox', 'spinbutton', 'listbox']); + +// Drop back-to-back duplicates with the same role+name. The AX tree often emits an icon node and its label as twins, and sticky headers repeat the same control. Consecutive-only so a genuine list (5 distinct "Add to cart" buttons interleaved with product text) is never collapsed. The sessionId is part of the key so a same-named element in a cross-origin child frame is never mistaken for a twin of the root frame's last element at the seam. Context too: five "Message" buttons in five people-cards are NOT twins, only same-card icon+label pairs (identical context) collapse. Inputs and NAMELESS rows never collapse: adjacent unlabeled icons (star, trash) match on every key yet are different controls -- that collapse lost the whole MiniWoB email suite. function dedupeConsecutive(items: RankItem[]): RankItem[] { const out: RankItem[] = []; for (const it of items) { const prev = out[out.length - 1]; if (prev && prev.role === it.role && prev.name === it.name - && prev.sessionId === it.sessionId && (prev.context || '') === (it.context || '')) continue; + && prev.sessionId === it.sessionId && (prev.context || '') === (it.context || '') + && !NEVER_DEDUPE_ROLES.has(it.role) && it.name !== '') continue; out.push(it); } return out;