import { getWebview, type BrowserWebview } from './browserRegistry'; import { store } from './state/store'; import { resumeBrowserCard } from './state/dashboardLayoutSlice'; import { dashboardWs } from './ws/WebSocketManager'; import { resolveInput } from './resolveUrl'; import { rankAndCapInteractives, type RankItem } from './interactiveRanking'; import { shouldStopWaiting, SETTLE_POLL_MS, settleProbeJs } from './browserSettle'; let initialized = false; export type BrowserAction = 'screenshot' | 'get_text' | 'get_console' | 'navigate' | 'click' | 'type' | 'evaluate' | 'get_elements' | 'scroll' | 'wait' | 'press_key' | 'list_interactives' | 'click_index' | 'batch' | 'detect_webmcp' | 'list_routes' | 'replay_route' | 'click_by_name'; export interface BrowserActivity { action: BrowserAction; detail?: string; coords?: { xPercent: number; yPercent: number }; } type ActivityListener = (browserId: string, activity: BrowserActivity | null) => void; const activityMap = new Map(); const listeners = new Set(); // A webview keeps churning for a beat after an action lands; capturing it into the // dashboard snapshot during that churn is what crashes the renderer (SharedImage // 'non-existent mailbox' -> V8 ToLocalChecked), so we hold "busy" this long past // the last command before letting the thumbnail capture run again. const BUSY_COOLDOWN_MS = 1500; let lastActivityAt = 0; function setActivity(browserId: string, activity: BrowserActivity | null) { lastActivityAt = Date.now(); if (activity) { activityMap.set(browserId, activity); } else { activityMap.delete(browserId); } listeners.forEach((fn) => fn(browserId, activity)); } export function getActivity(browserId: string): BrowserActivity | null { return activityMap.get(browserId) ?? null; } // True while an agent is actively driving any browser webview (a command is in // flight, or one finished within the cooldown). The dashboard thumbnail capture // checks this and skips rather than screenshot a live, churning webview. export function isAnyBrowserBusy(): boolean { if (activityMap.size > 0) return true; return Date.now() - lastActivityAt < BUSY_COOLDOWN_MS; } export function subscribeActivity(fn: ActivityListener): () => void { listeners.add(fn); return () => { listeners.delete(fn); }; } const ACTION_LABELS: Record = { screenshot: 'Capturing...', get_text: 'Reading...', navigate: 'Navigating...', click: 'Clicking...', type: 'Typing...', evaluate: 'Evaluating...', get_elements: 'Inspecting...', scroll: 'Scrolling...', wait: 'Waiting...', press_key: 'Pressing key...', list_interactives: 'Reading page structure...', click_index: 'Clicking element...', click_by_name: 'Clicking element...', batch: 'Running batch...', }; export function getActionLabel(action: string): string { return ACTION_LABELS[action] ?? 'Working...'; } // Draw each cached element's index as a colored chip on the live page right // before capture (browser-use's trick): the screenshot then speaks the same // numbers as BrowserListInteractives, so the vision side can act by index. const _ANNOTATION_COLORS = ['#e5484d', '#0091ff', '#30a46c', '#f76b15', '#8e4ec6', '#00a2c7']; const _ANNOTATE_BUDGET_MS = 1500; // One unsettled CDP bridge promise must never hang the screenshot; race each call. function _cdpTimeout(p: Promise, ms: number): Promise { return Promise.race([p, new Promise((_, rej) => setTimeout(() => rej(new Error('cdp call timed out')), ms))]); } async function annotateElements(wv: BrowserWebview): Promise { const cacheBridge = (window as any).openswarm?.cdpCacheGet; const cached = cacheBridge ? await _cdpTimeout(cacheBridge(wv.getWebContentsId()), 500) : null; if (!cached || typeof cached !== 'object') return 0; const deadline = Date.now() + _ANNOTATE_BUDGET_MS; const drawOne = async (idxStr: string, entry: any): Promise => { const backendNodeId = typeof entry === 'number' ? entry : entry?.backendNodeId; // v1 is root-frame only: OOPIF rects are frame-local and would land wrong if (!backendNodeId || entry?.sessionId) return false; try { const t = await _cdpTimeout(sendCdp(wv, 'DOM.resolveNode', { backendNodeId }), 300); const r = await _cdpTimeout(sendCdp(wv, 'Runtime.callFunctionOn', { objectId: t.object.objectId, functionDeclaration: 'function(idx, color) {' + ' const r = this.getBoundingClientRect();' + ' if (r.width <= 0 || r.height <= 0) return false;' + ' if (r.bottom < 0 || r.top > innerHeight || r.right < 0 || r.left > innerWidth) return false;' + ' let c = document.getElementById("__osw_annotations__");' + ' if (!c) { c = document.createElement("div"); c.id = "__osw_annotations__";' + ' c.style.cssText = "position:fixed;inset:0;z-index:2147483647;pointer-events:none;";' + ' document.documentElement.appendChild(c); }' + ' const box = document.createElement("div");' + ' box.style.cssText = "position:fixed;left:" + r.left + "px;top:" + r.top + "px;width:" + r.width + "px;height:" + r.height + "px;border:2px solid " + color + ";box-sizing:border-box;";' + ' const tag = document.createElement("span");' + ' tag.textContent = String(idx);' + ' tag.style.cssText = "position:absolute;left:-2px;top:-16px;background:" + color + ";color:#fff;font:bold 11px/14px monospace;padding:0 4px;border-radius:2px;";' + ' if (r.top < 18) { tag.style.top = "-2px"; }' + ' box.appendChild(tag); c.appendChild(box); return true;' + ' }', arguments: [{ value: Number(idxStr) }, { value: _ANNOTATION_COLORS[Number(idxStr) % _ANNOTATION_COLORS.length] }], returnByValue: true, }), 300); return r?.result?.value === true; } catch { return false; } // node gone or call timed out; skip }; let drawn = 0; const entries = Object.entries(cached); const CHUNK = 10; for (let i = 0; i < entries.length && Date.now() < deadline; i += CHUNK) { const results = await Promise.allSettled( entries.slice(i, i + CHUNK).map(([idxStr, e]) => drawOne(idxStr, e)), ); drawn += results.filter((r) => r.status === 'fulfilled' && r.value === true).length; } return drawn; } async function removeAnnotations(wv: BrowserWebview): Promise { try { await _cdpTimeout(sendCdp(wv, 'Runtime.evaluate', { expression: 'document.getElementById("__osw_annotations__")?.remove()', }), 800); } catch { /* page navigated mid-capture; the overlay died with it */ } } async function handleScreenshot(wv: BrowserWebview, params?: Record): Promise> { if (params?.annotate !== false) { let drawn = 0; try { drawn = await annotateElements(wv); if (drawn > 0) { const shot = await captureRetry(wv); if (shot.image) shot.text = `Screenshot with ${drawn} numbered boxes matching your element list (pass annotate:false for a clean shot).`; return shot; } } catch { /* annotation is decoration; a plain shot always beats an error */ } finally { if (drawn > 0) await removeAnnotations(wv); } } return captureRetry(wv); } async function captureRetry(wv: BrowserWebview): Promise> { // capturePage throws UnknownVizError if the webview hasn't composited a frame // yet (the Viz compositor races the first paint, reliably bit turn-0 captures). // Retry a few times with a short backoff so a cold first screenshot succeeds // instead of burning a whole agent turn on a transient error. let lastErr: any; for (let attempt = 0; attempt < 4; attempt++) { try { const nativeImage = await wv.capturePage(); if (!nativeImage.isEmpty()) { // Stable PNG capture. The resize()+toJPEG() variant was reverted: it's the // prime suspect for the renderer "V8 Empty MaybeLocal" crash, NativeImage's // JPEG codec returns an empty image on some retina captures, which is the // shape of that native fault. A stable app beats a faster screenshot. const dataUrl = nativeImage.toDataURL(); const base64 = dataUrl.replace(/^data:image\/\w+;base64,/, ''); return { image: base64, url: wv.getURL(), title: wv.getTitle() }; } lastErr = new Error('capturePage returned an empty image (frame not painted yet)'); } catch (err: any) { lastErr = err; } await new Promise((r) => setTimeout(r, 250 * (attempt + 1))); } return { error: `Screenshot failed after retries: ${lastErr?.message || String(lastErr)}` }; } // Count the safe (GET) API endpoints captured for this site so the backend can // nudge the agent toward the fast network path. Best-effort, never throws. async function countSafeRoutes(wv: BrowserWebview): Promise { try { const bridge = (window as any).openswarm?.cdpRoutesGet as | ((id: number, origin?: string) => Promise) | undefined; if (!bridge) return 0; let origin = ''; try { origin = new URL(wv.getURL()).origin; } catch {} const routes = (await bridge(wv.getWebContentsId(), origin)) || []; return routes.filter((r) => r && r.safe).length; } catch { return 0; } } async function handleGetText(wv: BrowserWebview): Promise> { const text: string = await wv.executeJavaScript( 'document.body.innerText.substring(0, 15000)' ); // Sampled HERE (on a read), not on navigate: by the time the agent reads the // page, the SPA's XHR/fetch have fired, so routes are actually captured. const routes_available = await countSafeRoutes(wv); return { text, url: wv.getURL(), title: wv.getTitle(), routes_available }; } // Recent warn+error console output for this webview (captured in main.js). Lets a // stuck agent see the page's OWN errors (JS exceptions, failed loads) instead of // guessing. Read-only, fail-safe: any miss returns an empty, honest result. async function handleGetConsole(wv: BrowserWebview): Promise> { try { const bridge = (window as any).openswarm?.getWebviewConsole as | ((id: number) => Promise>) | undefined; if (!bridge) return { text: 'Console capture is unavailable here.', errors: [] }; const errors = (await bridge(wv.getWebContentsId())) || []; if (errors.length === 0) { return { text: 'No console warnings or errors recorded on this page.', errors: [], url: wv.getURL() }; } const lines = errors.map( (e) => `[${e.level}] ${e.message}${e.source ? ` (${e.source}:${e.line ?? '?'})` : ''}`, ); return { text: `Page console, ${errors.length} recent warning(s)/error(s), newest last:\n${lines.join('\n')}`, errors, url: wv.getURL(), }; } catch (err: any) { return { text: `Could not read console: ${err?.message || String(err)}`, errors: [] }; } } async function handleNavigate(wv: BrowserWebview, params: Record): Promise> { const raw = params.url as string; if (!raw) return { error: 'url parameter is required' }; const url = resolveInput(raw); // loadURL resolves only on the full 'load' event, which heavy SPAs (LinkedIn, // Gmail) hold open with persistent connections long past our timeout even though // the page is usable in a second. Return the moment the DOM is ready and let the // agent's next wait settle the rest, the way a person clicks before every // background request has finished. let removeReady = () => {}; const domReady = new Promise((resolve) => { const onReady = () => resolve(); wv.addEventListener('dom-ready', onReady, { once: true }); removeReady = () => wv.removeEventListener('dom-ready', onReady); }); const fullyLoaded = wv.loadURL(url).catch((err: any) => { // A superseded navigation aborts the old load; that's normal, not a failure. if (err?.message?.includes('ERR_ABORTED')) return; throw err; }); fullyLoaded.catch(() => {}); // a late load failure shouldn't throw once dom-ready returned try { await Promise.race([fullyLoaded, domReady]); } finally { removeReady(); } // Route-count is sampled on the next READ (handleGetText), not here: at // navigate-return the SPA's XHRs haven't fired yet, so this would always be ~0. return { text: `Navigated to ${url}`, url }; } async function handleClick(wv: BrowserWebview, params: Record): Promise> { const selector = params.selector as string; if (!selector) return { error: 'selector parameter is required' }; const safeSelector = JSON.stringify(selector); const code = `(()=>{ const el = document.querySelector(${safeSelector}); if (!el) return { error: 'Element not found: ' + ${safeSelector} }; el.scrollIntoView({ block: 'center', behavior: 'instant' }); const rect = el.getBoundingClientRect(); const x = rect.left + rect.width / 2; const y = rect.top + rect.height / 2; const opts = { bubbles: true, cancelable: true, clientX: x, clientY: y, button: 0 }; el.dispatchEvent(new PointerEvent('pointerdown', { ...opts, pointerId: 1 })); el.dispatchEvent(new MouseEvent('mousedown', opts)); el.dispatchEvent(new PointerEvent('pointerup', { ...opts, pointerId: 1 })); el.dispatchEvent(new MouseEvent('mouseup', opts)); el.dispatchEvent(new MouseEvent('click', opts)); return { text: 'Clicked element: ' + el.tagName.toLowerCase() + (el.id ? '#' + el.id : ''), url: location.href, clickX: window.innerWidth > 0 ? x / window.innerWidth : 0.5, clickY: window.innerHeight > 0 ? y / window.innerHeight : 0.5, }; })()`; const result = await wv.executeJavaScript(code); return result; } async function handleType(wv: BrowserWebview, params: Record): Promise> { const selector = params.selector as string; const text = params.text as string; if (!selector) return { error: 'selector parameter is required' }; if (text == null) return { error: 'text parameter is required' }; const safeSelector = JSON.stringify(selector); const safeText = JSON.stringify(text); const code = `(async ()=>{ const el = document.querySelector(${safeSelector}); if (!el) return { error: 'Element not found: ' + ${safeSelector} }; el.scrollIntoView({ block: 'center', behavior: 'instant' }); el.focus(); if (el.select) el.select(); document.execCommand('selectAll', false); document.execCommand('delete', false); document.execCommand('insertText', false, ${safeText}); el.dispatchEvent(new InputEvent('input', { bubbles: true, cancelable: true, inputType: 'insertText', data: ${safeText}, })); el.dispatchEvent(new Event('change', { bubbles: true })); return { text: 'Typed into: ' + el.tagName.toLowerCase() + (el.id ? '#' + el.id : ''), }; })()`; const result = await wv.executeJavaScript(code); return result; } // Electron sendInputEvent expects names like 'Up', 'Enter', 'Space', not 'ArrowUp'/' '/'Esc'. const KEY_NAME_MAP: Record = { ArrowUp: 'Up', ArrowDown: 'Down', ArrowLeft: 'Left', ArrowRight: 'Right', ' ': 'Space', Spacebar: 'Space', Esc: 'Escape', Del: 'Delete', }; async function handlePressKey(wv: BrowserWebview, params: Record): Promise> { const rawKey = (params.key as string) || ''; if (!rawKey) return { error: 'key parameter is required' }; const keyCode = KEY_NAME_MAP[rawKey] || rawKey; await wv.executeJavaScript('document.body && document.body.focus && document.body.focus(); true'); // Native OS-level key events have isTrusted=true, so hostile sites' keyboard handlers respect them. wv.sendInputEvent({ type: 'keyDown', keyCode }); wv.sendInputEvent({ type: 'char', keyCode }); wv.sendInputEvent({ type: 'keyUp', keyCode }); return { text: `Pressed ${rawKey}` }; } // CDP Accessibility.getFullAXTree sees computed roles/names even on hostile sites with unlabeled DOMs. const INTERACTIVE_ROLES = new Set([ 'button', 'link', 'textbox', 'combobox', 'checkbox', 'menuitem', 'tab', 'switch', 'searchbox', 'slider', 'listbox', 'option', 'radio', 'menuitemcheckbox', 'menuitemradio', 'spinbutton', 'treeitem', ]); interface InteractiveElement { index: number; role: string; name: string; backendNodeId: number; sessionId?: string; value?: string; } function extractAxValue(prop: any): string { if (!prop) return ''; if (typeof prop === 'string') return prop; if (prop.value !== undefined) { if (typeof prop.value === 'string') return prop.value; if (typeof prop.value === 'object' && prop.value && 'value' in prop.value) { return String(prop.value.value || ''); } } return ''; } interface CdpResult { ok: boolean; result?: any; error?: string } // sessionId undefined => root frame; a child-frame sessionId => that OOPIF. async function sendCdp(wv: BrowserWebview, method: string, params?: Record, sessionId?: string): Promise { const wcId = wv.getWebContentsId(); const bridge = (window as any).openswarm?.sendCdpCommand as | ((id: number, m: string, p?: any, s?: string) => Promise) | undefined; if (!bridge) throw new Error('CDP bridge not available, restart the app'); const resp = await bridge(wcId, method, params, sessionId); if (!resp || !resp.ok) { throw new Error(resp?.error || `CDP ${method} failed`); } return resp.result; } interface ChildSession { sessionId: string; frameId: string; parentSessionId: string | null; url: string } async function getChildSessions(wv: BrowserWebview): Promise { const bridge = (window as any).openswarm?.cdpChildSessionsGet as | ((id: number) => Promise) | undefined; if (!bridge) return []; try { return (await bridge(wv.getWebContentsId())) || []; } catch { return []; } } // Roles whose name is useful as disambiguating context for a nearby control // (the person's name above a "Message" button, the section heading of a form). const _CONTEXT_ROLES = new Set(['heading', 'statictext', 'link']); const _CONTEXT_MAX_CHARS = 60; const _CONTEXT_LOOKBACK = 30; function axNodesToCandidates(nodes: any[], sessionId?: string): RankItem[] { const byId = new Map(); const parentOf = new Map(); const orderOf = new Map(); for (let i = 0; i < nodes.length; i++) { const node = nodes[i]; if (node.nodeId != null) { byId.set(String(node.nodeId), node); orderOf.set(String(node.nodeId), i); } } for (const node of nodes) { for (const c of node.childIds || []) parentOf.set(String(c), String(node.nodeId)); } const isCandidate = (n: any): boolean => { if (!n || n.ignored || n.backendDOMNodeId == null) return false; return INTERACTIVE_ROLES.has(extractAxValue(n.role)); }; // Which card/section does this control sit in? Nearest named non-interactive // ancestor wins (a listitem's name aggregates its card text); else the nearest // preceding heading/text/link in document order (browser-use's trick). This is // what tells "Message" for Tyler apart from "Message" for everyone else. const contextOf = (node: any, ownName: string): string => { let p = parentOf.get(String(node.nodeId)); for (let hops = 0; p && hops < 12; hops++) { const anc = byId.get(p); if (anc && !anc.ignored && !isCandidate(anc)) { const ancName = extractAxValue(anc.name).trim(); if (ancName && ancName !== ownName) return ancName.slice(0, _CONTEXT_MAX_CHARS); } p = parentOf.get(p); } const pos = orderOf.get(String(node.nodeId)); if (pos == null) return ''; for (let i = pos - 1; i >= 0 && i >= pos - _CONTEXT_LOOKBACK; i--) { const prev = nodes[i]; if (!prev || prev.ignored) continue; if (!_CONTEXT_ROLES.has(extractAxValue(prev.role).toLowerCase())) continue; const prevName = extractAxValue(prev.name).trim(); if (prevName && prevName !== ownName && prevName.length >= 3) { return prevName.slice(0, _CONTEXT_MAX_CHARS); } } return ''; }; // A same-named interactive ancestor owns this hit target (a link inside a // button, an icon twin inside its labeled wrapper); listing both just gives // the model two indexes for one click. Names must match so a menu never // swallows its menuitems. const twinOfAncestor = (node: any, name: string): boolean => { if (!name) return false; let p = parentOf.get(String(node.nodeId)); while (p) { const anc = byId.get(p); if (isCandidate(anc)) return extractAxValue(anc.name).slice(0, 80) === name; p = parentOf.get(p); } return false; }; const out: RankItem[] = []; for (const node of nodes) { if (node.ignored) continue; const role = extractAxValue(node.role); if (!INTERACTIVE_ROLES.has(role)) continue; const name = extractAxValue(node.name); if (!name && role !== 'textbox' && role !== 'searchbox' && role !== 'combobox') continue; const backendNodeId = node.backendDOMNodeId; if (backendNodeId == null) continue; const shortName = name.slice(0, 80); if (twinOfAncestor(node, shortName)) continue; let value = ''; if (role === 'textbox' || role === 'searchbox' || role === 'combobox') { const isProtected = (node.properties || []).some( (p: any) => p?.name === 'protected' && p?.value?.value === true, ); value = isProtected ? '' : extractAxValue(node.value).slice(0, 60); } out.push({ role, name: shortName, backendNodeId, sessionId, context: contextOf(node, name), value }); } return out; } // Cumulative top-left offset of a frame within the root viewport: climb the // session chain adding each owning