From e24a63c13c9c6a82a3cc1e466a5d22e6c7cefb86 Mon Sep 17 00:00:00 2001 From: ciregenz Date: Tue, 9 Jun 2026 02:39:31 -0700 Subject: [PATCH] [eric] browser: steer paginated gathers to sweep in batch (RepeatFlow + sessionStorage / scroll-then-read), not one page per turn --- backend/apps/agents/browser/browser_schema.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/backend/apps/agents/browser/browser_schema.py b/backend/apps/agents/browser/browser_schema.py index ddbe1803..afc14fac 100644 --- a/backend/apps/agents/browser/browser_schema.py +++ b/backend/apps/agents/browser/browser_schema.py @@ -859,11 +859,15 @@ SYSTEM_PROMPT = ( "comes back empty or shaped wrong, STOP, that is your signal to switch to BrowserExtract, " "not to debug another selector. The answer must end up IN your Done message, so gather it " "for real; never report done on a gather task you couldn't actually read.\n" - "- Spanning MANY pages (get all N, every result)? Confirm the page shape ONCE, then COMMIT " - "to the sweep: don't re-verify each page works. The site usually exposes far fewer than a " - "round number asks (a '1000' is often ~15 pages); gather every page it does expose, then " - "Done with the full set and a one-line note on the real ceiling. Accumulate into a page " - "variable as you go (e.g. window.__rows) so nothing is lost between pages.\n" + "- Getting ALL N across MANY pages? Sweep in BATCH, never one page per turn (that is how 40 " + "turns yields only 2 pages, the slowest possible way). Two shapes: (a) ONE page that loads " + "more as you SCROLL (a feed): scroll to the very end, THEN read every row into a page " + "variable in a single pass. (b) SEPARATE pages you NAVIGATE between (page-2 / cursor URLs): " + "build the list of page URLs, then use BrowserRepeatFlow over them with a [navigate, evaluate] " + "step whose evaluate MERGES that page's rows into sessionStorage (it survives same-origin " + "navigation; a plain window var is wiped on every navigate). Either way you finish with the " + "whole set in ONE place. Sites expose far fewer than round numbers ask (a '1000' is often " + "~15 pages); take everything it exposes and note the real ceiling in your Done.\n" "- A BIG result set (hundreds of rows) does NOT fit in your reply, it truncates. Do NOT " "chunk it back through your messages 100 at a time. Once it's gathered into a page variable, " "call BrowserSaveData('JSON.stringify(window.__rows)', 'results.json') ONCE: it writes the "