import { getWebview, type BrowserWebview } from './browserRegistry'; import { store } from './state/store'; import { resumeBrowserCard } from './state/dashboardLayoutSlice'; import { dashboardWs } from './ws/WebSocketManager'; import { resolveInput } from './resolveUrl'; import { rankAndCapInteractives, type RankItem } from './interactiveRanking'; import { shouldStopWaiting, SETTLE_POLL_MS, settleProbeJs } from './browserSettle'; let initialized = false; export type BrowserAction = 'screenshot' | 'get_text' | 'get_console' | 'navigate' | 'click' | 'type' | 'evaluate' | 'get_elements' | 'scroll' | 'wait' | 'press_key' | 'list_interactives' | 'click_index' | 'batch' | 'detect_webmcp' | 'list_routes' | 'replay_route' | 'click_by_name'; export interface BrowserActivity { action: BrowserAction; detail?: string; coords?: { xPercent: number; yPercent: number }; } type ActivityListener = (browserId: string, activity: BrowserActivity | null) => void; const activityMap = new Map(); const listeners = new Set(); // A webview keeps churning for a beat after an action lands; capturing it into the dashboard snapshot during that churn is what crashes the renderer (SharedImage 'non-existent mailbox' -> V8 ToLocalChecked), so we hold "busy" this long past the last command before letting the thumbnail capture run again. const BUSY_COOLDOWN_MS = 1500; let lastActivityAt = 0; function setActivity(browserId: string, activity: BrowserActivity | null) { lastActivityAt = Date.now(); if (activity) { activityMap.set(browserId, activity); } else { activityMap.delete(browserId); } listeners.forEach((fn) => fn(browserId, activity)); } export function getActivity(browserId: string): BrowserActivity | null { return activityMap.get(browserId) ?? null; } // True while an agent is actively driving any browser webview (a command is in flight, or one finished within the cooldown). The dashboard thumbnail capture checks this and skips rather than screenshot a live, churning webview. export function isAnyBrowserBusy(): boolean { if (activityMap.size > 0) return true; return Date.now() - lastActivityAt < BUSY_COOLDOWN_MS; } export function subscribeActivity(fn: ActivityListener): () => void { listeners.add(fn); return () => { listeners.delete(fn); }; } const ACTION_LABELS: Record = { screenshot: 'Capturing...', get_text: 'Reading...', navigate: 'Navigating...', click: 'Clicking...', type: 'Typing...', evaluate: 'Evaluating...', get_elements: 'Inspecting...', scroll: 'Scrolling...', wait: 'Waiting...', press_key: 'Pressing key...', list_interactives: 'Reading page structure...', click_index: 'Clicking element...', click_by_name: 'Clicking element...', batch: 'Running batch...', }; export function getActionLabel(action: string): string { return ACTION_LABELS[action] ?? 'Working...'; } // Draw each cached element's index as a colored chip on the live page right before capture (browser-use's trick): the screenshot then speaks the same numbers as BrowserListInteractives, so the vision side can act by index. const _ANNOTATION_COLORS = ['#e5484d', '#0091ff', '#30a46c', '#f76b15', '#8e4ec6', '#00a2c7']; const _ANNOTATE_BUDGET_MS = 1500; // One unsettled CDP bridge promise must never hang the screenshot; race each call. function _cdpTimeout(p: Promise, ms: number): Promise { return Promise.race([p, new Promise((_, rej) => setTimeout(() => rej(new Error('cdp call timed out')), ms))]); } async function annotateElements(wv: BrowserWebview): Promise { const cacheBridge = (window as any).openswarm?.cdpCacheGet; const cached = cacheBridge ? await _cdpTimeout(cacheBridge(wv.getWebContentsId()), 500) : null; if (!cached || typeof cached !== 'object') return 0; const deadline = Date.now() + _ANNOTATE_BUDGET_MS; const drawOne = async (idxStr: string, entry: any): Promise => { const backendNodeId = typeof entry === 'number' ? entry : entry?.backendNodeId; // v1 is root-frame only: OOPIF rects are frame-local and would land wrong if (!backendNodeId || entry?.sessionId) return false; try { const t = await _cdpTimeout(sendCdp(wv, 'DOM.resolveNode', { backendNodeId }), 300); const r = await _cdpTimeout(sendCdp(wv, 'Runtime.callFunctionOn', { objectId: t.object.objectId, functionDeclaration: 'function(idx, color) {' + ' const r = this.getBoundingClientRect();' + ' if (r.width <= 0 || r.height <= 0) return false;' + ' if (r.bottom < 0 || r.top > innerHeight || r.right < 0 || r.left > innerWidth) return false;' + ' let c = document.getElementById("__osw_annotations__");' + ' if (!c) { c = document.createElement("div"); c.id = "__osw_annotations__";' + ' c.style.cssText = "position:fixed;inset:0;z-index:2147483647;pointer-events:none;";' + ' document.documentElement.appendChild(c); }' + ' const box = document.createElement("div");' + ' box.style.cssText = "position:fixed;left:" + r.left + "px;top:" + r.top + "px;width:" + r.width + "px;height:" + r.height + "px;border:2px solid " + color + ";box-sizing:border-box;";' + ' const tag = document.createElement("span");' + ' tag.textContent = String(idx);' + ' tag.style.cssText = "position:absolute;left:-2px;top:-16px;background:" + color + ";color:#fff;font:bold 11px/14px monospace;padding:0 4px;border-radius:2px;";' + ' if (r.top < 18) { tag.style.top = "-2px"; }' + ' box.appendChild(tag); c.appendChild(box); return true;' + ' }', arguments: [{ value: Number(idxStr) }, { value: _ANNOTATION_COLORS[Number(idxStr) % _ANNOTATION_COLORS.length] }], returnByValue: true, }), 300); return r?.result?.value === true; } catch { return false; } // node gone or call timed out; skip }; let drawn = 0; const entries = Object.entries(cached); const CHUNK = 10; for (let i = 0; i < entries.length && Date.now() < deadline; i += CHUNK) { const results = await Promise.allSettled( entries.slice(i, i + CHUNK).map(([idxStr, e]) => drawOne(idxStr, e)), ); drawn += results.filter((r) => r.status === 'fulfilled' && r.value === true).length; } return drawn; } async function removeAnnotations(wv: BrowserWebview): Promise { try { await _cdpTimeout(sendCdp(wv, 'Runtime.evaluate', { expression: 'document.getElementById("__osw_annotations__")?.remove()', }), 800); } catch { /* page navigated mid-capture; the overlay died with it */ } } async function handleScreenshot(wv: BrowserWebview, params?: Record): Promise> { if (params?.annotate !== false) { let drawn = 0; try { drawn = await annotateElements(wv); if (drawn > 0) { const shot = await captureRetry(wv); if (shot.image) shot.text = `Screenshot with ${drawn} numbered boxes matching your element list (pass annotate:false for a clean shot).`; return shot; } } catch { /* annotation is decoration; a plain shot always beats an error */ } finally { if (drawn > 0) await removeAnnotations(wv); } } return captureRetry(wv); } async function captureRetry(wv: BrowserWebview): Promise> { // capturePage throws UnknownVizError if the webview hasn't composited a frame yet (the Viz compositor races the first paint, reliably bit turn-0 captures). Retry a few times with a short backoff so a cold first screenshot succeeds instead of burning a whole agent turn on a transient error. let lastErr: any; for (let attempt = 0; attempt < 4; attempt++) { try { const nativeImage = await wv.capturePage(); if (!nativeImage.isEmpty()) { // Stable PNG capture. The resize()+toJPEG() variant was reverted: it's the prime suspect for the renderer "V8 Empty MaybeLocal" crash, NativeImage's JPEG codec returns an empty image on some retina captures, which is the shape of that native fault. A stable app beats a faster screenshot. const dataUrl = nativeImage.toDataURL(); const base64 = dataUrl.replace(/^data:image\/\w+;base64,/, ''); return { image: base64, url: wv.getURL(), title: wv.getTitle() }; } lastErr = new Error('capturePage returned an empty image (frame not painted yet)'); } catch (err: any) { lastErr = err; } await new Promise((r) => setTimeout(r, 250 * (attempt + 1))); } return { error: `Screenshot failed after retries: ${lastErr?.message || String(lastErr)}` }; } // Count the safe (GET) API endpoints captured for this site so the backend can nudge the agent toward the fast network path. Best-effort, never throws. async function countSafeRoutes(wv: BrowserWebview): Promise { try { const bridge = (window as any).openswarm?.cdpRoutesGet as | ((id: number, origin?: string) => Promise) | undefined; if (!bridge) return 0; let origin = ''; try { origin = new URL(wv.getURL()).origin; } catch {} const routes = (await bridge(wv.getWebContentsId(), origin)) || []; return routes.filter((r) => r && r.safe).length; } catch { return 0; } } async function handleGetText(wv: BrowserWebview): Promise> { const text: string = await wv.executeJavaScript( 'document.body.innerText.substring(0, 15000)' ); // Sampled HERE (on a read), not on navigate: by the time the agent reads the page, the SPA's XHR/fetch have fired, so routes are actually captured. const routes_available = await countSafeRoutes(wv); return { text, url: wv.getURL(), title: wv.getTitle(), routes_available }; } // Recent warn+error console output for this webview (captured in main.js). Lets a stuck agent see the page's OWN errors (JS exceptions, failed loads) instead of guessing. Read-only, fail-safe: any miss returns an empty, honest result. async function handleGetConsole(wv: BrowserWebview): Promise> { try { const bridge = (window as any).openswarm?.getWebviewConsole as | ((id: number) => Promise>) | undefined; if (!bridge) return { text: 'Console capture is unavailable here.', errors: [] }; const errors = (await bridge(wv.getWebContentsId())) || []; if (errors.length === 0) { return { text: 'No console warnings or errors recorded on this page.', errors: [], url: wv.getURL() }; } const lines = errors.map( (e) => `[${e.level}] ${e.message}${e.source ? ` (${e.source}:${e.line ?? '?'})` : ''}`, ); return { text: `Page console, ${errors.length} recent warning(s)/error(s), newest last:\n${lines.join('\n')}`, errors, url: wv.getURL(), }; } catch (err: any) { return { text: `Could not read console: ${err?.message || String(err)}`, errors: [] }; } } async function handleNavigate(wv: BrowserWebview, params: Record): Promise> { const raw = params.url as string; if (!raw) return { error: 'url parameter is required' }; const url = resolveInput(raw); // loadURL resolves only on the full 'load' event, which heavy SPAs (LinkedIn, Gmail) hold open with persistent connections long past our timeout even though the page is usable in a second. Return the moment the DOM is ready and let the agent's next wait settle the rest, the way a person clicks before every background request has finished. let removeReady = () => {}; const domReady = new Promise((resolve) => { const onReady = () => resolve(); wv.addEventListener('dom-ready', onReady, { once: true }); removeReady = () => wv.removeEventListener('dom-ready', onReady); }); const fullyLoaded = wv.loadURL(url).catch((err: any) => { // A superseded navigation aborts the old load; that's normal, not a failure. if (err?.message?.includes('ERR_ABORTED')) return; throw err; }); fullyLoaded.catch(() => {}); // a late load failure shouldn't throw once dom-ready returned try { await Promise.race([fullyLoaded, domReady]); } finally { removeReady(); } // Route-count is sampled on the next READ (handleGetText), not here: at navigate-return the SPA's XHRs haven't fired yet, so this would always be ~0. return { text: `Navigated to ${url}`, url }; } async function handleClick(wv: BrowserWebview, params: Record): Promise> { const selector = params.selector as string; if (!selector) return { error: 'selector parameter is required' }; const safeSelector = JSON.stringify(selector); const code = `(()=>{ const el = document.querySelector(${safeSelector}); if (!el) return { error: 'Element not found: ' + ${safeSelector} }; el.scrollIntoView({ block: 'center', behavior: 'instant' }); const rect = el.getBoundingClientRect(); const x = rect.left + rect.width / 2; const y = rect.top + rect.height / 2; const opts = { bubbles: true, cancelable: true, clientX: x, clientY: y, button: 0 }; el.dispatchEvent(new PointerEvent('pointerdown', { ...opts, pointerId: 1 })); el.dispatchEvent(new MouseEvent('mousedown', opts)); el.dispatchEvent(new PointerEvent('pointerup', { ...opts, pointerId: 1 })); el.dispatchEvent(new MouseEvent('mouseup', opts)); el.dispatchEvent(new MouseEvent('click', opts)); return { text: 'Clicked element: ' + el.tagName.toLowerCase() + (el.id ? '#' + el.id : ''), url: location.href, clickX: window.innerWidth > 0 ? x / window.innerWidth : 0.5, clickY: window.innerHeight > 0 ? y / window.innerHeight : 0.5, }; })()`; const result = await wv.executeJavaScript(code); return result; } async function handleType(wv: BrowserWebview, params: Record): Promise> { const selector = params.selector as string; const text = params.text as string; if (!selector) return { error: 'selector parameter is required' }; if (text == null) return { error: 'text parameter is required' }; const safeSelector = JSON.stringify(selector); const safeText = JSON.stringify(text); const code = `(async ()=>{ const el = document.querySelector(${safeSelector}); if (!el) return { error: 'Element not found: ' + ${safeSelector} }; el.scrollIntoView({ block: 'center', behavior: 'instant' }); el.focus(); if (el.select) el.select(); document.execCommand('selectAll', false); document.execCommand('delete', false); document.execCommand('insertText', false, ${safeText}); el.dispatchEvent(new InputEvent('input', { bubbles: true, cancelable: true, inputType: 'insertText', data: ${safeText}, })); el.dispatchEvent(new Event('change', { bubbles: true })); return { text: 'Typed into: ' + el.tagName.toLowerCase() + (el.id ? '#' + el.id : ''), }; })()`; const result = await wv.executeJavaScript(code); return result; } // Electron sendInputEvent expects names like 'Up', 'Enter', 'Space', not 'ArrowUp'/' '/'Esc'. const KEY_NAME_MAP: Record = { ArrowUp: 'Up', ArrowDown: 'Down', ArrowLeft: 'Left', ArrowRight: 'Right', ' ': 'Space', Spacebar: 'Space', Esc: 'Escape', Del: 'Delete', }; async function handlePressKey(wv: BrowserWebview, params: Record): Promise> { const rawKey = (params.key as string) || ''; if (!rawKey) return { error: 'key parameter is required' }; const keyCode = KEY_NAME_MAP[rawKey] || rawKey; await wv.executeJavaScript('document.body && document.body.focus && document.body.focus(); true'); // Native OS-level key events have isTrusted=true, so hostile sites' keyboard handlers respect them. wv.sendInputEvent({ type: 'keyDown', keyCode }); wv.sendInputEvent({ type: 'char', keyCode }); wv.sendInputEvent({ type: 'keyUp', keyCode }); return { text: `Pressed ${rawKey}` }; } // CDP Accessibility.getFullAXTree sees computed roles/names even on hostile sites with unlabeled DOMs. const INTERACTIVE_ROLES = new Set([ 'button', 'link', 'textbox', 'combobox', 'checkbox', 'menuitem', 'tab', 'switch', 'searchbox', 'slider', 'listbox', 'option', 'radio', 'menuitemcheckbox', 'menuitemradio', 'spinbutton', 'treeitem', ]); interface InteractiveElement { index: number; role: string; name: string; backendNodeId: number; sessionId?: string; value?: string; } function extractAxValue(prop: any): string { if (!prop) return ''; if (typeof prop === 'string') return prop; if (prop.value !== undefined) { if (typeof prop.value === 'string') return prop.value; if (typeof prop.value === 'object' && prop.value && 'value' in prop.value) { return String(prop.value.value || ''); } } return ''; } interface CdpResult { ok: boolean; result?: any; error?: string } // sessionId undefined => root frame; a child-frame sessionId => that OOPIF. async function sendCdp(wv: BrowserWebview, method: string, params?: Record, sessionId?: string): Promise { const wcId = wv.getWebContentsId(); const bridge = (window as any).openswarm?.sendCdpCommand as | ((id: number, m: string, p?: any, s?: string) => Promise) | undefined; if (!bridge) throw new Error('CDP bridge not available, restart the app'); const resp = await bridge(wcId, method, params, sessionId); if (!resp || !resp.ok) { throw new Error(resp?.error || `CDP ${method} failed`); } return resp.result; } interface ChildSession { sessionId: string; frameId: string; parentSessionId: string | null; url: string } async function getChildSessions(wv: BrowserWebview): Promise { const bridge = (window as any).openswarm?.cdpChildSessionsGet as | ((id: number) => Promise) | undefined; if (!bridge) return []; try { return (await bridge(wv.getWebContentsId())) || []; } catch { return []; } } // Roles whose name is useful as disambiguating context for a nearby control (the person's name above a "Message" button, the section heading of a form). const _CONTEXT_ROLES = new Set(['heading', 'statictext', 'link']); const _CONTEXT_MAX_CHARS = 60; const _CONTEXT_LOOKBACK = 30; function axNodesToCandidates(nodes: any[], sessionId?: string): RankItem[] { const byId = new Map(); const parentOf = new Map(); const orderOf = new Map(); for (let i = 0; i < nodes.length; i++) { const node = nodes[i]; if (node.nodeId != null) { byId.set(String(node.nodeId), node); orderOf.set(String(node.nodeId), i); } } for (const node of nodes) { for (const c of node.childIds || []) parentOf.set(String(c), String(node.nodeId)); } const isCandidate = (n: any): boolean => { if (!n || n.ignored || n.backendDOMNodeId == null) return false; return INTERACTIVE_ROLES.has(extractAxValue(n.role)); }; // Which card/section does this control sit in? Nearest named non-interactive ancestor wins (a listitem's name aggregates its card text); else the nearest preceding heading/text/link in document order (browser-use's trick). This is what tells "Message" for Tyler apart from "Message" for everyone else. const contextOf = (node: any, ownName: string): string => { let p = parentOf.get(String(node.nodeId)); for (let hops = 0; p && hops < 12; hops++) { const anc = byId.get(p); if (anc && !anc.ignored && !isCandidate(anc)) { const ancName = extractAxValue(anc.name).trim(); if (ancName && ancName !== ownName) return ancName.slice(0, _CONTEXT_MAX_CHARS); } p = parentOf.get(p); } const pos = orderOf.get(String(node.nodeId)); if (pos == null) return ''; for (let i = pos - 1; i >= 0 && i >= pos - _CONTEXT_LOOKBACK; i--) { const prev = nodes[i]; if (!prev || prev.ignored) continue; if (!_CONTEXT_ROLES.has(extractAxValue(prev.role).toLowerCase())) continue; const prevName = extractAxValue(prev.name).trim(); if (prevName && prevName !== ownName && prevName.length >= 3) { return prevName.slice(0, _CONTEXT_MAX_CHARS); } } return ''; }; // A same-named interactive ancestor owns this hit target (a link inside a button, an icon twin inside its labeled wrapper); listing both just gives the model two indexes for one click. Names must match so a menu never swallows its menuitems. const twinOfAncestor = (node: any, name: string): boolean => { if (!name) return false; let p = parentOf.get(String(node.nodeId)); while (p) { const anc = byId.get(p); if (isCandidate(anc)) return extractAxValue(anc.name).slice(0, 80) === name; p = parentOf.get(p); } return false; }; const out: RankItem[] = []; for (const node of nodes) { if (node.ignored) continue; const role = extractAxValue(node.role); if (!INTERACTIVE_ROLES.has(role)) continue; const name = extractAxValue(node.name); if (!name && role !== 'textbox' && role !== 'searchbox' && role !== 'combobox') continue; const backendNodeId = node.backendDOMNodeId; if (backendNodeId == null) continue; const shortName = name.slice(0, 80); if (twinOfAncestor(node, shortName)) continue; let value = ''; if (role === 'textbox' || role === 'searchbox' || role === 'combobox') { const isProtected = (node.properties || []).some( (p: any) => p?.name === 'protected' && p?.value?.value === true, ); value = isProtected ? '' : extractAxValue(node.value).slice(0, 60); } out.push({ role, name: shortName, backendNodeId, sessionId, context: contextOf(node, name), value }); } return out; } // Cumulative top-left offset of a frame within the root viewport: climb the session chain adding each owning