Commit Graph
59 Commits
Author SHA1 Message Date
ciregenz f7472b3dbf [eric] browser: recover a card that opens straight onto a JSON URL (initial-load path + homepage fallback) 2026-07-16 10:37:32 -07:00
ciregenz de3261d2d9 [eric] browser: a navigate landing on a raw JSON/API URL hands data to the agent and recovers the card off the wall 2026-07-15 23:06:02 -07:00
ciregenz 1f74cc13de [eric] browser: general capture-replay write tier (site's own captured route, same-origin + gated, default off) 2026-07-14 23:47:17 -07:00
ciregenz 42a3249976 [eric] browser: broaden compose-trigger vocabulary (create/new-thread/start-a-thread variants)
The reveal opener regex only matched 'create a post', missing bare 'Create' and 'New thread' / 'Start a thread' buttons that real compose surfaces use. Adds those; safe by construction (a broadened trigger can never produce a false success, a fill against a real composer is still required, so a trigger that opens a signup/join upsell instead of a composer yields found:false). Validated CDP-direct: LinkedIn-post and YouTube-comment still flip (no regression), and Threads (which gates posting behind a 'Join Threads' upsell for this session) correctly does NOT false-fill
2026-07-13 14:59:21 -07:00
ciregenz 937bb35dca [eric] browser: evalInPage runs via CDP Runtime.evaluate, not webContents.executeJavaScript
The executeJavaScript path is SUSPENDED by Electron until a page stops loading, so on a page whose trackers never let it settle (Reddit, LinkedIn) every in-page command queues into the 15s backend timeout = the mount-poll wedge that has blocked every agent-loop run. CDP Runtime.evaluate runs in the browser process, out of the renderer's load-blocked event loop, so it does not suspend. This reuses the already-in-production CDP bridge (sendCdp -> send-cdp-command -> wc.debugger.sendCommand, keyed by getWebContentsId, with attach + per-wc serialization + timeout already built) that the AX/click commands use; only the eval path was left on executeJavaScript. No-regression by construction: on any CDP INFRA failure (debugger can't attach because DevTools or a remote-debugging port holds the webContents, or the bridge errors) it falls back to the original executeJavaScript path, and a real page-side exception still surfaces as a throw; window.__OSW_CDP_EVAL__ = false forces the old path. Unwrap logic extracted to cdpEval.ts + 6 unit tests. NOTE not live-demonstrable on a --remote-debugging-port bench: that port globally blocks wc.debugger.attach, so the whole app CDP path (this AND the existing list_interactives) times out there; production and normal dev do not use the port, so attach works and the existing CDP commands prove it
2026-07-12 21:55:59 -07:00
ciregenz 82fe16767c [eric] browser: handleType reads back its fill and falls back to real keystrokes = the main type action now detects when execCommand insertText did not commit (editors that reject synthetic input) and retypes via wv.sendInputEvent, same lever as the composer finder's keystrokeFill 2026-07-12 21:26:49 -07:00
ciregenz 19a3e8ad1a [eric] browser: keystroke-fill fallback + cross-nav reveal retry (both flag-gated, default off)
Item 1: when the finder's execCommand insertText read-back fails (editors that reject synthetic input), retype the same text as real OS-level key events via wv.sendInputEvent, then verify from the marked element OR document.activeElement (editors like Reddit swap the node on activation, staling the selector). Item 3: when an open-first reveal navigates to a new document (Reddit thread, TikTok video, GitHub issue), the send-script re-perceives after a load beat and calls the finder once more on the destination, bounded to 2 tries. 4 new tests (cross-nav retry fires / no wasted retry) + structural tests split into test_browser_reveal.py to stay under the 300-line cap. Live real-path validation of the sendInputEvent leg is owed on a healthy rig: the eric/browser agent loop wedges at fresh-card webview mount (15.25s cap, renderer starvation, predates the suspend fix) at low load too
2026-07-12 20:52:15 -07:00
ciregenz a3099f035c [eric] browser: reveal-and-find gains activation-click + shadow-safe collection = a placeholder composer that fills empty (YouTube 'Add a comment') gets clicked, rescanned, and the real editor filled; deepMatch collects only selector matches (few hundred) not every node (the old all-node 4000 cap truncated before a heavy SPA's late-in-DOM composer, which hid X's OWN box); openFirstItem broadened to chat/message lists (data-testid=conversation, message/chat links) and pierces shadow DOM; CDP-direct n=2 reproduced: reveal converts LinkedIn post (modal) + YouTube comment (activation), controls stay filled 2026-07-12 20:03:05 -07:00
ciregenz 64675115f8 [eric] browser: reveal-and-find reach primitive (OSW_COMPOSER_REVEAL, default off) = when no composer is painted the finder takes ONE reversible reveal action (click a scored compose-trigger, open the first list item, or scroll) and rescans, bounded; SPA route changes keep the JS context alive so X/LinkedIn/Reddit/YouTube stay drivable in one call; never clicks an irreversible control (hard-block send/submit/pay/delete); send-script forwards reveal only under the flag; 2 tests (reveal forwarded when on, false by default) 2026-07-12 19:29:07 -07:00
ciregenz 2d49687ae0 [eric] browser: composer finder pierces shadow DOM (Reddit shreddit, Telegram, WhatsApp, Slack build their composer inside web-component shadow roots a light-DOM querySelectorAll can't see); bounded recursive walk (depth 8, 400-node cap). Live finder-probe on real DOMs: correctly picks X's contenteditable 'Post text' + excludes the search combobox; the remaining wall is that lazy/dialog/chat-gated composers aren't rendered at scan time (detection is downstream of render) 2026-07-10 00:56:05 -07:00
ciregenz 4c85dd6314 [eric] browser: structural composer finder (OSW_COMPOSER_STRUCT, default off) = an in-page primitive that ranks editable regions (contenteditable/textarea/text-input near a Send/Post/Reply control, excluding search boxes) instead of matching AX names, marks the winner + fills+reads-back in-page (the only reliable commit-check for a React contenteditable whose text never reaches the AX tree); send-script uses it as a fallback when the name detector misses, unblocking unnamed/non-standard composers (Reddit Lexical, X DM); 3 tests, all safety gates + dry-run stop unchanged 2026-07-10 00:13:44 -07:00
ciregenz e0162779a7 [eric] browser: gate document-order display behind OSW_DOC_ORDER (default on) so the A/B can measure it against legacy rank-order; 14 ranking tests 2026-07-08 14:57:24 -07:00
ciregenz b60f0b94aa [eric] browser: wrong-target detectability probe = the click hit-test now reports WHAT element was actually under the click point, so a click that lands on the wrong element (occluded/stale/moved, and still changes the page) is finally measurable, the failure a page-change metric can't see 2026-07-07 16:47:47 -07:00
ciregenz 2b63953650 [eric] browser: click-effect probe wraps ALL clickBackendNode exits (was only on the coordinate branch; real-site clicks exit via the covered-element JS path, so the metric captured ~none) 2026-07-07 15:48:37 -07:00
ciregenz 8f2565e027 [eric] browser: thread self-heal + click-effect metric into BATCH sub-clicks too (most clicks are batched; gating only top-level click_index measured ~none of them) + scan batch sub-results for the telemetry 2026-07-07 15:41:07 -07:00
ciregenz f54f474de7 [eric] browser: click-effect metric (measure before fixing) = fingerprint the page before/after a click to catch the INVISIBLE failure the tool-error counter misses (a click that succeeds but hits the wrong/dead element = nothing changes); flag-gated OSW_CLICK_EFFECT_PROBE, zero cost off, 3 unit tests 2026-07-07 15:33:33 -07:00
ciregenz 5ca7308fa5 [eric] browser: self-healing click (Phase 1) = a failed cached-index click auto-escalates to fresh by-name resolution in-code before the model ever sees the error (stale index = ~half of all runs' tool-errors, each costs a ~3s re-strategize turn); error-triggered so no double-act, OSW_SELFHEAL_CLICK=0 kill switch, 5 unit tests 2026-07-07 15:05:41 -07:00
ciregenz 46c300e4ee [eric] browser: unwedge stuck-loading webviews (executeJavaScript queues until load stops; recaptcha/tracker stragglers held isLoading for minutes = every command timed out); guarded eval stops stragglers only after dom-ready 2026-07-07 02:18:13 -07:00
ciregenz c04a663996 [eric] browser-bridge: /api/browser-session/action + renderer perform_action + shim client 2026-06-30 17:19:58 -07:00
ciregenz 63c32554ae [eric] mcp: reddit session-borrow MCP (free, full human action set; WIP) 2026-06-29 23:05:15 -07:00
ciregenz 8eb48dfc42 [eric] app-use: strip pierre's transient [app-agent] trace scaffolding + write-only controls cache (no speculative scaffolding) 2026-06-29 23:03:45 -07:00
SirKentut 75df12a1b7 [pierre] feat: merge app-use (OPENSWARM_APP bridge + AppAgent) onto eric/dev, convention-clean
Brings the app-use feature (OPENSWARM_APP bridge, AppAgent delegation tool,
BrowserClickPoint, autopilot supervision) onto the linted eric/dev base.

- Resolved 9 conflicts favoring eric's refactored structure; renamed all of
  pierre's _-prefixed names to p_/P_, kept eric's prior_messages resume guard.
- Made the 4 test-exercised bridge helpers public (p-private rule).
- Wired AppAgent into browser_delegation_tools + BuiltinTool registry and
  threaded selected_app_output_ids -> OPENSWARM_SELECTED_APP_IDS (eric had
  forward-ported the prompt but not the tool, so it was unreachable).
- Linter clean (underscore/p-private/ruff/cycles 0); 301 backend tests pass.
2026-06-26 02:44:10 -07:00
ciregenz 52464e66ee [eric] frontend: one-line every // comment + collapse 3+ blank runs across frontend/src (161 files, -1652 lines, TS-AST-verified code-identical) 2026-06-25 04:20:41 -07:00
SirKentut edd018cb4c [pierre] feat: handle click_point via native CDP mouse event in browser command handler 2026-06-18 02:35:25 -07:00
SirKentut b498ed3097 implemented app-describe for agent use: exposing the command tree of an app 2026-06-13 01:01:11 -07:00
AidanandGitHub 3f4d91d8b4 [aidan] bug: cross-origin iframe perception + spawned card end UX (#78)
Browser sub-agents can now drive Google Drive’s share-doc flow end to end: same-process cross-origin iframes (the share dialog) are visible through a frame-tree-aware AX walk, and combobox autocomplete fields fill in one click_index call via DOM.focus + Input.insertText. When a spawned browser card’s agent finishes, the card now fades through a “Task done” pill with a Keep button for ~3s instead of snapping out, so the user sees what happened.
2026-06-13 00:58:44 -07:00
ciregenz b8a0c0814c [eric] dashboard: suspend offscreen browser webviews to a snapshot, cuts RAM/CPU at scale 2026-06-09 14:06:18 -07:00
ciregenz 1480fddf51 [eric] browser: replay clicks scroll-and-retry when off-screen, long names resolve by stable prefix 2026-06-05 17:21:50 -07:00
ciregenz 0992e25bf2 [eric] browser: numbered index overlays on screenshots, vision and clicks share one index language 2026-06-05 11:03:20 -07:00
ciregenz 059185dd4b [eric] browser: kill the double-fill, textbox rows show live value, typing echoes verified read-back, expect demands literal page text 2026-06-05 10:29:12 -07:00
ciregenz c0ff256afd [eric] browser: ctx-tagged twin rows + stable indices across looks in list_interactives 2026-06-05 00:46:24 -07:00
ciregenz 05fa871b24 [eric] browser: occlusion filter + containment dedup + new-element markers, click flash, request-id dedupe 2026-06-04 23:03:20 -07:00
ciregenz b02aecbb58 [eric] browser: hit-test clicks before dispatch and verify typed text, honest fallbacks for both 2026-06-04 19:04:38 -07:00
ciregenz 53ffe92c14 [eric] browser: return navigate on dom-ready so heavy SPAs stop hanging the 20s cap 2026-06-04 18:40:06 -07:00
ciregenz f7ac55660a [eric] browser: bound AX perception and cap the tracker-iframe walk so a busy page fails clearly, not silently 2026-06-04 17:15:29 -07:00
ciregenz f03dde7e0c [eric] browser: focus text boxes by node and insert text directly, reaches compose iframes clicks miss 2026-06-04 16:38:56 -07:00
ciregenz f0144690af [eric] browser: end the wait on the target or DOM-settle, not network-idle, so it stops waiting blind 2026-06-04 14:57:38 -07:00
ciregenz 2640930380 [eric] browser: add BrowserGetConsole so a stuck agent can read the page's console errors 2026-06-04 13:20:39 -07:00
ciregenz 5dcdc88005 [eric] browser: abort a batch on the first failed sub-action, not just on navigation 2026-06-04 12:10:09 -07:00
ciregenz 27686e270b [eric] browser: settle-aware in-batch wait and a terminal read sub-action 2026-06-04 04:04:44 -07:00
ciregenz a562a5c3c3 [eric] browser: prune stale screenshots + capture downscaled JPEG to cut agent context cost 2026-06-03 22:17:05 -07:00
ciregenz f121fa442c [eric] browser: surface a redacted concrete route example so the agent can compose replay_route with {{value}} 2026-06-03 13:17:57 -07:00
ciregenz 50230c88c5 [eric] browser: also sample route count on evaluate (the agent's real read path), not just get_text 2026-06-03 02:11:11 -07:00
ciregenz 7175f4b27b [eric] browser: sample captured-route count on read, not navigate, so the network nudge actually fires 2026-06-03 02:05:11 -07:00
ciregenz 21b225ded4 [eric] browser: nudge the agent toward the fast captured-API tier once per host 2026-06-03 00:50:23 -07:00
ciregenz e911898387 [eric] browser: add robust click_by_name and share enumerate/click helpers 2026-06-02 12:43:21 -07:00
ciregenz 54249bc015 [eric] browser: retry cold screenshot capture to dodge the turn-0 viz race 2026-06-02 12:27:28 -07:00
ciregenz 15781d2a3b [eric] browser: list captured routes and replay safe GETs to skip the UI (tier 2) 2026-06-02 04:11:50 -07:00
ciregenz 6e006dd472 [eric] browser: detect a page's declared WebMCP tools (tier 1) 2026-06-02 04:03:10 -07:00
ciregenz 21360cd2ea [eric] browser: rank/cap/goal-boost the element list and click into cross-origin iframes 2026-06-02 03:55:58 -07:00