import { getWebview, type BrowserWebview } from './browserRegistry'; import { dashboardWs } from './ws/WebSocketManager'; import { resolveInput } from './resolveUrl'; import { rankAndCapInteractives, type RankItem } from './interactiveRanking'; import { shouldStopWaiting, SETTLE_FLOOR_MS, SETTLE_POLL_MS, SETTLE_PROBE_JS } from './browserSettle'; let initialized = false; export type BrowserAction = 'screenshot' | 'get_text' | 'navigate' | 'click' | 'type' | 'evaluate' | 'get_elements' | 'scroll' | 'wait' | 'press_key' | 'list_interactives' | 'click_index' | 'batch' | 'detect_webmcp' | 'list_routes' | 'replay_route' | 'click_by_name'; export interface BrowserActivity { action: BrowserAction; detail?: string; coords?: { xPercent: number; yPercent: number }; } type ActivityListener = (browserId: string, activity: BrowserActivity | null) => void; const activityMap = new Map(); const listeners = new Set(); // A webview keeps churning for a beat after an action lands; capturing it into the // dashboard snapshot during that churn is what crashes the renderer (SharedImage // 'non-existent mailbox' -> V8 ToLocalChecked), so we hold "busy" this long past // the last command before letting the thumbnail capture run again. const BUSY_COOLDOWN_MS = 1500; let lastActivityAt = 0; function setActivity(browserId: string, activity: BrowserActivity | null) { lastActivityAt = Date.now(); if (activity) { activityMap.set(browserId, activity); } else { activityMap.delete(browserId); } listeners.forEach((fn) => fn(browserId, activity)); } export function getActivity(browserId: string): BrowserActivity | null { return activityMap.get(browserId) ?? null; } // True while an agent is actively driving any browser webview (a command is in // flight, or one finished within the cooldown). The dashboard thumbnail capture // checks this and skips rather than screenshot a live, churning webview. export function isAnyBrowserBusy(): boolean { if (activityMap.size > 0) return true; return Date.now() - lastActivityAt < BUSY_COOLDOWN_MS; } export function subscribeActivity(fn: ActivityListener): () => void { listeners.add(fn); return () => { listeners.delete(fn); }; } const ACTION_LABELS: Record = { screenshot: 'Capturing...', get_text: 'Reading...', navigate: 'Navigating...', click: 'Clicking...', type: 'Typing...', evaluate: 'Evaluating...', get_elements: 'Inspecting...', scroll: 'Scrolling...', wait: 'Waiting...', press_key: 'Pressing key...', list_interactives: 'Reading page structure...', click_index: 'Clicking element...', click_by_name: 'Clicking element...', batch: 'Running batch...', }; export function getActionLabel(action: string): string { return ACTION_LABELS[action] ?? 'Working...'; } async function handleScreenshot(wv: BrowserWebview): Promise> { // capturePage throws UnknownVizError if the webview hasn't composited a frame // yet (the Viz compositor races the first paint, reliably bit turn-0 captures). // Retry a few times with a short backoff so a cold first screenshot succeeds // instead of burning a whole agent turn on a transient error. let lastErr: any; for (let attempt = 0; attempt < 4; attempt++) { try { const nativeImage = await wv.capturePage(); if (!nativeImage.isEmpty()) { // Stable PNG capture. The resize()+toJPEG() variant was reverted: it's the // prime suspect for the renderer "V8 Empty MaybeLocal" crash, NativeImage's // JPEG codec returns an empty image on some retina captures, which is the // shape of that native fault. A stable app beats a faster screenshot. const dataUrl = nativeImage.toDataURL(); const base64 = dataUrl.replace(/^data:image\/\w+;base64,/, ''); return { image: base64, url: wv.getURL(), title: wv.getTitle() }; } lastErr = new Error('capturePage returned an empty image (frame not painted yet)'); } catch (err: any) { lastErr = err; } await new Promise((r) => setTimeout(r, 250 * (attempt + 1))); } return { error: `Screenshot failed after retries: ${lastErr?.message || String(lastErr)}` }; } // Count the safe (GET) API endpoints captured for this site so the backend can // nudge the agent toward the fast network path. Best-effort, never throws. async function countSafeRoutes(wv: BrowserWebview): Promise { try { const bridge = (window as any).openswarm?.cdpRoutesGet as | ((id: number, origin?: string) => Promise) | undefined; if (!bridge) return 0; let origin = ''; try { origin = new URL(wv.getURL()).origin; } catch {} const routes = (await bridge(wv.getWebContentsId(), origin)) || []; return routes.filter((r) => r && r.safe).length; } catch { return 0; } } async function handleGetText(wv: BrowserWebview): Promise> { const text: string = await wv.executeJavaScript( 'document.body.innerText.substring(0, 15000)' ); // Sampled HERE (on a read), not on navigate: by the time the agent reads the // page, the SPA's XHR/fetch have fired, so routes are actually captured. const routes_available = await countSafeRoutes(wv); return { text, url: wv.getURL(), title: wv.getTitle(), routes_available }; } async function handleNavigate(wv: BrowserWebview, params: Record): Promise> { const raw = params.url as string; if (!raw) return { error: 'url parameter is required' }; const url = resolveInput(raw); try { await wv.loadURL(url); } catch (err: any) { if (!err?.message?.includes('ERR_ABORTED')) throw err; } // Route-count is sampled on the next READ (handleGetText), not here: at // navigate-return the SPA's XHRs haven't fired yet, so this would always be ~0. return { text: `Navigated to ${url}`, url }; } async function handleClick(wv: BrowserWebview, params: Record): Promise> { const selector = params.selector as string; if (!selector) return { error: 'selector parameter is required' }; const safeSelector = JSON.stringify(selector); const code = `(()=>{ const el = document.querySelector(${safeSelector}); if (!el) return { error: 'Element not found: ' + ${safeSelector} }; el.scrollIntoView({ block: 'center', behavior: 'instant' }); const rect = el.getBoundingClientRect(); const x = rect.left + rect.width / 2; const y = rect.top + rect.height / 2; const opts = { bubbles: true, cancelable: true, clientX: x, clientY: y, button: 0 }; el.dispatchEvent(new PointerEvent('pointerdown', { ...opts, pointerId: 1 })); el.dispatchEvent(new MouseEvent('mousedown', opts)); el.dispatchEvent(new PointerEvent('pointerup', { ...opts, pointerId: 1 })); el.dispatchEvent(new MouseEvent('mouseup', opts)); el.dispatchEvent(new MouseEvent('click', opts)); return { text: 'Clicked element: ' + el.tagName.toLowerCase() + (el.id ? '#' + el.id : ''), url: location.href, clickX: window.innerWidth > 0 ? x / window.innerWidth : 0.5, clickY: window.innerHeight > 0 ? y / window.innerHeight : 0.5, }; })()`; const result = await wv.executeJavaScript(code); return result; } async function handleType(wv: BrowserWebview, params: Record): Promise> { const selector = params.selector as string; const text = params.text as string; if (!selector) return { error: 'selector parameter is required' }; if (text == null) return { error: 'text parameter is required' }; const safeSelector = JSON.stringify(selector); const safeText = JSON.stringify(text); const code = `(async ()=>{ const el = document.querySelector(${safeSelector}); if (!el) return { error: 'Element not found: ' + ${safeSelector} }; el.scrollIntoView({ block: 'center', behavior: 'instant' }); el.focus(); if (el.select) el.select(); document.execCommand('selectAll', false); document.execCommand('delete', false); document.execCommand('insertText', false, ${safeText}); el.dispatchEvent(new InputEvent('input', { bubbles: true, cancelable: true, inputType: 'insertText', data: ${safeText}, })); el.dispatchEvent(new Event('change', { bubbles: true })); return { text: 'Typed into: ' + el.tagName.toLowerCase() + (el.id ? '#' + el.id : ''), }; })()`; const result = await wv.executeJavaScript(code); return result; } // Electron sendInputEvent expects names like 'Up', 'Enter', 'Space', not 'ArrowUp'/' '/'Esc'. const KEY_NAME_MAP: Record = { ArrowUp: 'Up', ArrowDown: 'Down', ArrowLeft: 'Left', ArrowRight: 'Right', ' ': 'Space', Spacebar: 'Space', Esc: 'Escape', Del: 'Delete', }; async function handlePressKey(wv: BrowserWebview, params: Record): Promise> { const rawKey = (params.key as string) || ''; if (!rawKey) return { error: 'key parameter is required' }; const keyCode = KEY_NAME_MAP[rawKey] || rawKey; await wv.executeJavaScript('document.body && document.body.focus && document.body.focus(); true'); // Native OS-level key events have isTrusted=true, so hostile sites' keyboard handlers respect them. wv.sendInputEvent({ type: 'keyDown', keyCode }); wv.sendInputEvent({ type: 'char', keyCode }); wv.sendInputEvent({ type: 'keyUp', keyCode }); return { text: `Pressed ${rawKey}` }; } // CDP Accessibility.getFullAXTree sees computed roles/names even on hostile sites with unlabeled DOMs. const INTERACTIVE_ROLES = new Set([ 'button', 'link', 'textbox', 'combobox', 'checkbox', 'menuitem', 'tab', 'switch', 'searchbox', 'slider', 'listbox', 'option', 'radio', 'menuitemcheckbox', 'menuitemradio', 'spinbutton', 'treeitem', ]); interface InteractiveElement { index: number; role: string; name: string; backendNodeId: number; sessionId?: string; } function extractAxValue(prop: any): string { if (!prop) return ''; if (typeof prop === 'string') return prop; if (prop.value !== undefined) { if (typeof prop.value === 'string') return prop.value; if (typeof prop.value === 'object' && prop.value && 'value' in prop.value) { return String(prop.value.value || ''); } } return ''; } interface CdpResult { ok: boolean; result?: any; error?: string } // sessionId undefined => root frame; a child-frame sessionId => that OOPIF. async function sendCdp(wv: BrowserWebview, method: string, params?: Record, sessionId?: string): Promise { const wcId = wv.getWebContentsId(); const bridge = (window as any).openswarm?.sendCdpCommand as | ((id: number, m: string, p?: any, s?: string) => Promise) | undefined; if (!bridge) throw new Error('CDP bridge not available, restart the app'); const resp = await bridge(wcId, method, params, sessionId); if (!resp || !resp.ok) { throw new Error(resp?.error || `CDP ${method} failed`); } return resp.result; } interface ChildSession { sessionId: string; frameId: string; parentSessionId: string | null; url: string } async function getChildSessions(wv: BrowserWebview): Promise { const bridge = (window as any).openswarm?.cdpChildSessionsGet as | ((id: number) => Promise) | undefined; if (!bridge) return []; try { return (await bridge(wv.getWebContentsId())) || []; } catch { return []; } } function axNodesToCandidates(nodes: any[], sessionId?: string): RankItem[] { const out: RankItem[] = []; for (const node of nodes) { if (node.ignored) continue; const role = extractAxValue(node.role); if (!INTERACTIVE_ROLES.has(role)) continue; const name = extractAxValue(node.name); if (!name && role !== 'textbox' && role !== 'searchbox' && role !== 'combobox') continue; const backendNodeId = node.backendDOMNodeId; if (backendNodeId == null) continue; out.push({ role, name: name.slice(0, 80), backendNodeId, sessionId }); } return out; } // Cumulative top-left offset of a frame within the root viewport: climb the // session chain adding each owning