ciregenz
19a3e8ad1a
[eric] browser: keystroke-fill fallback + cross-nav reveal retry (both flag-gated, default off)
...
Item 1: when the finder's execCommand insertText read-back fails (editors that reject synthetic input), retype the same text as real OS-level key events via wv.sendInputEvent, then verify from the marked element OR document.activeElement (editors like Reddit swap the node on activation, staling the selector). Item 3: when an open-first reveal navigates to a new document (Reddit thread, TikTok video, GitHub issue), the send-script re-perceives after a load beat and calls the finder once more on the destination, bounded to 2 tries. 4 new tests (cross-nav retry fires / no wasted retry) + structural tests split into test_browser_reveal.py to stay under the 300-line cap. Live real-path validation of the sendInputEvent leg is owed on a healthy rig: the eric/browser agent loop wedges at fresh-card webview mount (15.25s cap, renderer starvation, predates the suspend fix) at low load too
2026-07-12 20:52:15 -07:00
ciregenz
a3099f035c
[eric] browser: reveal-and-find gains activation-click + shadow-safe collection = a placeholder composer that fills empty (YouTube 'Add a comment') gets clicked, rescanned, and the real editor filled; deepMatch collects only selector matches (few hundred) not every node (the old all-node 4000 cap truncated before a heavy SPA's late-in-DOM composer, which hid X's OWN box); openFirstItem broadened to chat/message lists (data-testid=conversation, message/chat links) and pierces shadow DOM; CDP-direct n=2 reproduced: reveal converts LinkedIn post (modal) + YouTube comment (activation), controls stay filled
2026-07-12 20:03:05 -07:00
ciregenz
64675115f8
[eric] browser: reveal-and-find reach primitive (OSW_COMPOSER_REVEAL, default off) = when no composer is painted the finder takes ONE reversible reveal action (click a scored compose-trigger, open the first list item, or scroll) and rescans, bounded; SPA route changes keep the JS context alive so X/LinkedIn/Reddit/YouTube stay drivable in one call; never clicks an irreversible control (hard-block send/submit/pay/delete); send-script forwards reveal only under the flag; 2 tests (reveal forwarded when on, false by default)
2026-07-12 19:29:07 -07:00
ciregenz
2d49687ae0
[eric] browser: composer finder pierces shadow DOM (Reddit shreddit, Telegram, WhatsApp, Slack build their composer inside web-component shadow roots a light-DOM querySelectorAll can't see); bounded recursive walk (depth 8, 400-node cap). Live finder-probe on real DOMs: correctly picks X's contenteditable 'Post text' + excludes the search combobox; the remaining wall is that lazy/dialog/chat-gated composers aren't rendered at scan time (detection is downstream of render)
2026-07-10 00:56:05 -07:00
ciregenz
4c85dd6314
[eric] browser: structural composer finder (OSW_COMPOSER_STRUCT, default off) = an in-page primitive that ranks editable regions (contenteditable/textarea/text-input near a Send/Post/Reply control, excluding search boxes) instead of matching AX names, marks the winner + fills+reads-back in-page (the only reliable commit-check for a React contenteditable whose text never reaches the AX tree); send-script uses it as a fallback when the name detector misses, unblocking unnamed/non-standard composers (Reddit Lexical, X DM); 3 tests, all safety gates + dry-run stop unchanged
2026-07-10 00:13:44 -07:00
ciregenz
e0162779a7
[eric] browser: gate document-order display behind OSW_DOC_ORDER (default on) so the A/B can measure it against legacy rank-order; 14 ranking tests
2026-07-08 14:57:24 -07:00
ciregenz
b60f0b94aa
[eric] browser: wrong-target detectability probe = the click hit-test now reports WHAT element was actually under the click point, so a click that lands on the wrong element (occluded/stale/moved, and still changes the page) is finally measurable, the failure a page-change metric can't see
2026-07-07 16:47:47 -07:00
ciregenz
2b63953650
[eric] browser: click-effect probe wraps ALL clickBackendNode exits (was only on the coordinate branch; real-site clicks exit via the covered-element JS path, so the metric captured ~none)
2026-07-07 15:48:37 -07:00
ciregenz
8f2565e027
[eric] browser: thread self-heal + click-effect metric into BATCH sub-clicks too (most clicks are batched; gating only top-level click_index measured ~none of them) + scan batch sub-results for the telemetry
2026-07-07 15:41:07 -07:00
ciregenz
f54f474de7
[eric] browser: click-effect metric (measure before fixing) = fingerprint the page before/after a click to catch the INVISIBLE failure the tool-error counter misses (a click that succeeds but hits the wrong/dead element = nothing changes); flag-gated OSW_CLICK_EFFECT_PROBE, zero cost off, 3 unit tests
2026-07-07 15:33:33 -07:00
ciregenz
5ca7308fa5
[eric] browser: self-healing click (Phase 1) = a failed cached-index click auto-escalates to fresh by-name resolution in-code before the model ever sees the error (stale index = ~half of all runs' tool-errors, each costs a ~3s re-strategize turn); error-triggered so no double-act, OSW_SELFHEAL_CLICK=0 kill switch, 5 unit tests
2026-07-07 15:05:41 -07:00
ciregenz
46c300e4ee
[eric] browser: unwedge stuck-loading webviews (executeJavaScript queues until load stops; recaptcha/tracker stragglers held isLoading for minutes = every command timed out); guarded eval stops stragglers only after dom-ready
2026-07-07 02:18:13 -07:00
ciregenz
c04a663996
[eric] browser-bridge: /api/browser-session/action + renderer perform_action + shim client
2026-06-30 17:19:58 -07:00
ciregenz
63c32554ae
[eric] mcp: reddit session-borrow MCP (free, full human action set; WIP)
2026-06-29 23:05:15 -07:00
ciregenz
8eb48dfc42
[eric] app-use: strip pierre's transient [app-agent] trace scaffolding + write-only controls cache (no speculative scaffolding)
2026-06-29 23:03:45 -07:00
SirKentut
75df12a1b7
[pierre] feat: merge app-use (OPENSWARM_APP bridge + AppAgent) onto eric/dev, convention-clean
...
Brings the app-use feature (OPENSWARM_APP bridge, AppAgent delegation tool,
BrowserClickPoint, autopilot supervision) onto the linted eric/dev base.
- Resolved 9 conflicts favoring eric's refactored structure; renamed all of
pierre's _-prefixed names to p_/P_, kept eric's prior_messages resume guard.
- Made the 4 test-exercised bridge helpers public (p-private rule).
- Wired AppAgent into browser_delegation_tools + BuiltinTool registry and
threaded selected_app_output_ids -> OPENSWARM_SELECTED_APP_IDS (eric had
forward-ported the prompt but not the tool, so it was unreachable).
- Linter clean (underscore/p-private/ruff/cycles 0); 301 backend tests pass.
2026-06-26 02:44:10 -07:00
ciregenz
52464e66ee
[eric] frontend: one-line every // comment + collapse 3+ blank runs across frontend/src (161 files, -1652 lines, TS-AST-verified code-identical)
2026-06-25 04:20:41 -07:00
SirKentut
edd018cb4c
[pierre] feat: handle click_point via native CDP mouse event in browser command handler
2026-06-18 02:35:25 -07:00
SirKentut
b498ed3097
implemented app-describe for agent use: exposing the command tree of an app
2026-06-13 01:01:11 -07:00
Aidan and GitHub
3f4d91d8b4
[aidan] bug: cross-origin iframe perception + spawned card end UX ( #78 )
...
Browser sub-agents can now drive Google Drive’s share-doc flow end to end: same-process cross-origin iframes (the share dialog) are visible through a frame-tree-aware AX walk, and combobox autocomplete fields fill in one click_index call via DOM.focus + Input.insertText. When a spawned browser card’s agent finishes, the card now fades through a “Task done” pill with a Keep button for ~3s instead of snapping out, so the user sees what happened.
2026-06-13 00:58:44 -07:00
ciregenz
b8a0c0814c
[eric] dashboard: suspend offscreen browser webviews to a snapshot, cuts RAM/CPU at scale
2026-06-09 14:06:18 -07:00
ciregenz
1480fddf51
[eric] browser: replay clicks scroll-and-retry when off-screen, long names resolve by stable prefix
2026-06-05 17:21:50 -07:00
ciregenz
0992e25bf2
[eric] browser: numbered index overlays on screenshots, vision and clicks share one index language
2026-06-05 11:03:20 -07:00
ciregenz
059185dd4b
[eric] browser: kill the double-fill, textbox rows show live value, typing echoes verified read-back, expect demands literal page text
2026-06-05 10:29:12 -07:00
ciregenz
c0ff256afd
[eric] browser: ctx-tagged twin rows + stable indices across looks in list_interactives
2026-06-05 00:46:24 -07:00
ciregenz
05fa871b24
[eric] browser: occlusion filter + containment dedup + new-element markers, click flash, request-id dedupe
2026-06-04 23:03:20 -07:00
ciregenz
b02aecbb58
[eric] browser: hit-test clicks before dispatch and verify typed text, honest fallbacks for both
2026-06-04 19:04:38 -07:00
ciregenz
53ffe92c14
[eric] browser: return navigate on dom-ready so heavy SPAs stop hanging the 20s cap
2026-06-04 18:40:06 -07:00
ciregenz
f7ac55660a
[eric] browser: bound AX perception and cap the tracker-iframe walk so a busy page fails clearly, not silently
2026-06-04 17:15:29 -07:00
ciregenz
f03dde7e0c
[eric] browser: focus text boxes by node and insert text directly, reaches compose iframes clicks miss
2026-06-04 16:38:56 -07:00
ciregenz
f0144690af
[eric] browser: end the wait on the target or DOM-settle, not network-idle, so it stops waiting blind
2026-06-04 14:57:38 -07:00
ciregenz
2640930380
[eric] browser: add BrowserGetConsole so a stuck agent can read the page's console errors
2026-06-04 13:20:39 -07:00
ciregenz
5dcdc88005
[eric] browser: abort a batch on the first failed sub-action, not just on navigation
2026-06-04 12:10:09 -07:00
ciregenz
27686e270b
[eric] browser: settle-aware in-batch wait and a terminal read sub-action
2026-06-04 04:04:44 -07:00
ciregenz
a562a5c3c3
[eric] browser: prune stale screenshots + capture downscaled JPEG to cut agent context cost
2026-06-03 22:17:05 -07:00
ciregenz
f121fa442c
[eric] browser: surface a redacted concrete route example so the agent can compose replay_route with {{value}}
2026-06-03 13:17:57 -07:00
ciregenz
50230c88c5
[eric] browser: also sample route count on evaluate (the agent's real read path), not just get_text
2026-06-03 02:11:11 -07:00
ciregenz
7175f4b27b
[eric] browser: sample captured-route count on read, not navigate, so the network nudge actually fires
2026-06-03 02:05:11 -07:00
ciregenz
21b225ded4
[eric] browser: nudge the agent toward the fast captured-API tier once per host
2026-06-03 00:50:23 -07:00
ciregenz
e911898387
[eric] browser: add robust click_by_name and share enumerate/click helpers
2026-06-02 12:43:21 -07:00
ciregenz
54249bc015
[eric] browser: retry cold screenshot capture to dodge the turn-0 viz race
2026-06-02 12:27:28 -07:00
ciregenz
15781d2a3b
[eric] browser: list captured routes and replay safe GETs to skip the UI (tier 2)
2026-06-02 04:11:50 -07:00
ciregenz
6e006dd472
[eric] browser: detect a page's declared WebMCP tools (tier 1)
2026-06-02 04:03:10 -07:00
ciregenz
21360cd2ea
[eric] browser: rank/cap/goal-boost the element list and click into cross-origin iframes
2026-06-02 03:55:58 -07:00
eric
6a2b55f23b
[eric] browser: wait briefly for webview re-register before card-not-found to absorb the remount race
2026-06-01 11:55:17 -07:00
ciregenz
f59bf0db9b
defluff (frontend + backend): strip em-dashes + shorten docstrings + drop dead UI files (cosmetic only, no schedule code)
2026-05-20 05:36:17 -07:00
ciregenz
1dad8a57fc
[eric] browser sub-agent: loop detection, ReportProgress, CDP ax tree, batching, compaction, hide-don't-unmount Dashboard so browser cards survive route navigation
2026-04-07 20:00:32 -07:00
haikdc
9943ccafd5
[Haik]: (Agent browser render size matches that of user browser spawn) (queing messages) (chat loading bubble fix) (tethers to sub agents and branches are now permament and inputs are swapped to a continue chat button) (auto focus for everything) (history card position fix when redeployed) (dashboard persistance fixed) (non browser link clicks open in the in app browser now) (chat message tooltips for copy, branching, and editing / swapping between edited messages)
2026-03-18 23:45:54 -07:00
haikdc
6e1fc1dee2
[Haik]: ckpt (Round 1 of fixing mcp server issues in prod - still has bugs) (Fixed some browser related issues like popup windows rendering out of app and weird js framework related errors when u open sites like spotify) (Added option in settings to toggle item selection to be on by default when selecting a new chat) (Browser agent actions now show up in chat output) (Browser agent actions each have their own custom icon)
2026-03-17 23:52:20 -07:00
haikdc
1717d456cf
[Haik]: (Now u can rename the dashboards via the sidebar) (Default system prompt - and a button to reset to the default if u ever modify it) (Completely revamped dashboard icon, when clicked it expands to show all active items in the dashboard, when u click an item, it jumps to it auto focussing) (this auto focus jump also has a custom animation and highlighting)
2026-03-16 03:56:18 -07:00
ciregenz