arena: de-risk verdicts -- OSWorld smoke-only/skip (model-gated, cant run on our stack); verified-writes BUILD NOW (deterministic on existing infra); MiniWoB 94.1 honest near-ceiling (search-engine primitive fought reward logic, disabled)

2026-08-17 18:25:42 +02:00 · 2026-08-16 19:23:54 -07:00
parent bc225d11ae
commit ef2c8a8d3b
3 changed files with 98 additions and 1 deletions
@@ -1013,3 +1013,35 @@ to v34. The paren-parser bug did NOT suppress CompWoB (its goals are MiniWoB-sty
 answers, no long JSON answer channel where parens appeared). 81.1 stands honest, no correction.
 The instrument bug was WebArena-specific (that benchmark appends a JSON answer schema + long
 answers with parens).
+
+## De-risk research verdicts (2026-08-16): OSWorld + verified-writes -- PRIORITIES RE-ORDERED
+
+Studied the frontier's actual code/infra before committing (per user directive). Both decisive:
+
+OSWORLD (clause 4, >=85): **SMOKE-TEST-ONLY / effectively skip.** (a) Our macOS+colima stack
+CANNOT run it -- needs KVM/x86; only VMware-Fusion-ARM works and only for a serial smoke run;
+full sweeps need AWS(~$90) or HUD. (b) SOTA is now 85-86% but via OFF-THE-SHELF models on MINIMAL
+harnesses (screenshot->computer-use-tool->pixel coords) -- it's MODEL-gated, not fine-tuning or
+harness-gated. Anthropic's published Opus/Sonnet = 81-83.5% on the reference loop; with our
+Opus/Sonnet ceiling (no fable-5) ~83% is the realistic max, and the reference harness already
+captures it -- our marginal add is +2-4pt (bBoN/hybrid-actions) at real desktop-infra cost. (c)
+MiniWoB skill doesn't transfer (OSWorld = LibreOffice/GIMP/OS, not web DOM). VERDICT: 2-3 day
+capped smoke test at most (control=reference loop vs treatment=+our layer, invest only if +>=3pt);
+otherwise SKIP -- paying desktop-infra cost to reproduce a free baseline. Booked as low-ROI.
+
+VERIFIED-WRITES (clause 5, >=95): **BUILD NOW -- feasible + deterministic on EXISTING infra,
+higher ROI than assumed.** No frontier benchmark verifies writes on real EXTERNAL sites (Web Bench
+uses $3k human review) -- so 'live' need not mean external; postmill/gitlab are real production
+software, the reproducible analogue. We ALREADY HAVE the tasks: config_files/test_reddit.raw.json
+(15 program_html write tasks) + test_gitlab.raw.json (14) -- create post/comment/license, verified
+by server-state re-read, NO LLM judge. Clean pattern (independent-channel read-your-writes):
+pre-assert absence -> agent writes -> fresh-session re-read + type-aware match -> GitLab REST API
+cross-check. Build: extract program_html write subset, webarena-verified evaluator, pre-assert +
+container-reset (design out false positives), abort-on-fail scoring, Wilson CI (>=95 needs
+~103/108). This is a whole unmeasured clause we can honestly measure -- PIVOT here.
+
+MINIWOB last-mile: 94.1 is the honest near-ceiling for the scripted-primitive approach. draw-circle
+(geometry) + book-flight (budget) worked; search-engine/hot-cold/text-editor primitives FIGHT
+MiniWoB's own reward machinery (search-engine pick_result tried 2 ways, 0/3 -- disabled). Remaining
+0.9pt to 95 needs framework-native primitives with diminishing returns; deprioritized BELOW the
+higher-ROI verified-writes clause. Two seeds already hit 95.2; 94.1 mean is variance-dragged.
@@ -192,6 +192,8 @@ class LlmPolicy:
    table_md: bool = False
    # v41: scripted freehand-circle geometry (run.py-side ring path). Feature-gated on circle goals.
    draw_circle: bool = False
+    # v44: scripted 'click the Nth search result' (run.py-side search+paginate+click). Gated.
+    pick_result: bool = False
    # v42: terminal answer protocol (run.py-side). Force a bare final answer on string-match goals.
    answer_protocol: bool = False
    # v43: answer-schema conformance gate (run.py-side). Validate send_msg against the task's own
@@ -704,6 +706,11 @@ class OpenSwarmLlmPolicy(LlmPolicy):
            d = LlmDecision(action="draw_circle()", n_interactive=n, note="scripted-circle")
            self.note("draw_circle()  (scripted circular path)", obs)
            return d
+        m_nth = re.search(r"(\d+)(?:st|nd|rd|th)\s+(?:search\s+)?result", goal, re.I)
+        if self.pick_result and not self.history and m_nth:
+            d = LlmDecision(action=f"pick_result({m_nth.group(1)})", n_interactive=n, note="scripted-result")
+            self.note(f"pick_result({m_nth.group(1)})  (scripted search + Nth result)", obs)
+            return d
        fv = self.try_fill_verify()
        if fv:
            # A fill that didn't stick invalidates everything planned on top of it.
@@ -895,7 +902,7 @@ says to interact with the covering element LATER, moving it aside now keeps that
 _VERB_RE = re.compile(
    r"\b(click|dblclick|fill|clear|select_option|hover|focus|press|scroll|drag_and_drop|noop"
    r"|mouse_click|mouse_dblclick|mouse_move|mouse_drag_and_drop|keyboard_type|keyboard_press"
-    r"|goto|go_back|go_forward|send_msg_to_user|report_infeasible|no_match|draw_circle)\s*\(")
+    r"|goto|go_back|go_forward|send_msg_to_user|report_infeasible|no_match|draw_circle|pick_result|noop)\s*\(")
 # Kept for callers that only need the verb set; payload capture is now the quote-aware scanner.
 CALL_RE = _VERB_RE

@@ -1104,6 +1111,16 @@ def build(name: str, model: str = "", endpoint: str = "", **_: Any) -> Any:
                                  local_ctx=True, blocker_probe=True, suppress_wrappers=True,
                                  force_unblock=True, native_js_fallback=True, escape_token=True,
                                  table_md=True, draw_circle=True, answer_protocol=True, **v42)
+    if name == "osw-llm-v44":  # v41 champion + scripted Nth-search-result primitive
+        v44 = dict(v7, system=OSW_SYSTEM_V8 + OSW_SYSTEM_V9_WIDGETS + OSW_SYSTEM_V16 + OSW_SYSTEM_V30
+                   + OSW_SYSTEM_V36, max_tokens=800)
+        return OpenSwarmLlmPolicy(name=name, multi=True, vision="progressive", fastpath=True,
+                                  scripted_drag=True, auto_complete=True, som=False,
+                                  native_pickers=True, verify_terminal=True, post_mouse_vision=True,
+                                  multi_cap=6, fill_verify=True, dispatch=True, offscreen=True,
+                                  local_ctx=True, blocker_probe=True, suppress_wrappers=True,
+                                  force_unblock=True, native_js_fallback=True, escape_token=True,
+                                  table_md=True, draw_circle=True, pick_result=False, **v44)  # pick_result disabled: fought MiniWoB reward logic
    if name == "osw-llm-v41":  # v40 champion + scripted draw-circle geometry primitive
        v41 = dict(v7, system=OSW_SYSTEM_V8 + OSW_SYSTEM_V9_WIDGETS + OSW_SYSTEM_V16 + OSW_SYSTEM_V30
                   + OSW_SYSTEM_V36, max_tokens=800)
@@ -274,6 +274,54 @@ def run_episode(arm: str, task: str, seed: int, rec: Recorder, args: argparse.Na
                    rec_step.action_error = "schema-gate: non-conforming answer bounced"
                    ep.add(rec_step); ep.steps = step
                    continue
+            # v44 (gated, scripted): "click the Nth search result" is an ordinal-resolution problem
+            # the model miscounts (measured: clicks the wrong row, confuses pagination with results).
+            # Script it: run the search, walk result pages counting a.search-title, click the Nth.
+            if (getattr(policy, "pick_result", False)
+                    and re.match(r"pick_result\(", decision.action or "")):
+                try:
+                    m_n = re.search(r"pick_result\((\d+)", decision.action)
+                    n = int(m_n.group(1)) if m_n else 1
+                    m_q = re.search(r'"([^"]+)"', full_goal)
+                    term = m_q.group(1) if m_q else ""
+                    pg = env.unwrapped.page
+                    def do_search():
+                        pg.evaluate(
+                            """(term) => {
+                                 const inp = document.querySelector('input[type=text], input.searchbar, input');
+                                 if (inp) { inp.value = term; inp.dispatchEvent(new Event('input',{bubbles:true})); }
+                                 const btn = [...document.querySelectorAll('button,input[type=submit]')]
+                                   .find(b => /search/i.test(b.textContent||b.value||''));
+                                 if (btn) btn.click();
+                               }""", term)
+                        # Walk result pages counting a.search-title until the Nth, clicking it.
+                        for _ in range(12):
+                            pg.wait_for_timeout(300)
+                            cnt = pg.evaluate("() => document.querySelectorAll('a.search-title').length")
+                            if cnt >= n:
+                                box = pg.evaluate(
+                                    "(n) => { const e=document.querySelectorAll('a.search-title')[n-1];"
+                                    " const r=e.getBoundingClientRect(); return [r.x+r.width/2, r.y+r.height/2]; }", n)
+                                pg.mouse.click(box[0], box[1])
+                                return True
+                            nxt = pg.query_selector("a.page-link:has-text('>')")
+                            if not nxt:
+                                if cnt:
+                                    pg.evaluate("(n) => { const l=document.querySelectorAll('a.search-title'); l[Math.min(n,l.length)-1].click(); }", n)
+                                return cnt > 0
+                            nxt.click()
+                        return False
+                    ok = with_deadline(do_search, 12)
+                    obs, reward, terminated, truncated, _ = with_deadline(
+                        lambda: env.step("noop()"), args.step_timeout)
+                    rec_step.action = f"pick_result({n}) [scripted search+Nth]"
+                    rec_step.reward = float(reward or 0)
+                    ep.add(rec_step); ep.reward = max(ep.reward, float(reward or 0)); ep.steps = step
+                    if terminated or truncated:
+                        ep.terminated, ep.truncated = bool(terminated), bool(truncated); break
+                    continue
+                except Exception:
+                    pass
            # v41 (gated, scripted geometry): a freehand-circle goal is a geometry problem the
            # model cannot trace by hand (measured: random short mouse jitters, never a circle).
            # Detect the SVG center marker and drive a true circular path, then submit. One-shot.