mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-09-13 13:17:40 +02:00
ciregenzandClaude Opus 5
3c50359146
Six of the eleven defects here were in the MEASUREMENT, not the product, and they were wrong in both directions. Harness, all of which silently produced wrong numbers: - coverage.py preflight refused every sweep on a box holding exactly one backend: stack.sh's supervisor is a `bash -c` quoting the whole uvicorn line, so it carries both "-m uvicorn backend.main" AND the venv python path. Discriminate on POSITION. - stack.sh status reported 2 backends over 1 and 0 webpack over a live dev server (webpack retitles its process). A status check whose job is preventing a second stack, failing in the direction that lets one land. - c7_run.sh/c8_run.sh slice r6_be.log while stack.sh names logs by TAG: a stack under any other tag hands every trial an empty slice and the sweep reports a confident 0/108. Now refuses loudly; it caught this exact mistake on first use. - "Browser command timed out" was bucketed infra. It is ONE command blowing its own budget, not a dead webview: all 4 such rows were BrowserFindComposer at exactly its 30s cap, every run completed after, zero card-gone markers in the whole log. Filed as infra it read as 11.8% flake AND lifted holdout reach 70% -> 84%. - api_retry / rate_limit_error now grade as infra. A provider 429 storm turned clean 15-21s exclusions into 188s product_no_composer rows. - bench.py prints reach BOTH ways when a row is UNVERIFIED. An exclusion resting on the agent's own word quietly flatters the score, and coverage.py's own instruction to confirm it by hand goes unread (I quoted a 100% that excluded onlinegdb). Timing was measuring 0.2% of the run: prestage completes BEFORE metrics_started_at, so other_ms was 25ms of a 12700ms median while prestage (4146ms, ~61%) sat in no bucket at all. prestage_ms/task_ms are now recorded; total_ms is deliberately NOT redefined, which would invalidate every before/after already taken against it. Product: - find_composer rejected ACE/CodeMirror-5/Monaco composers. Their input is a ~1x1 offscreen textarea that paints into a sibling div, so it can never pass a size gate. Accept it when a VISIBLE ancestor is composer-sized; honeypots stay out because the input itself must not be display:none/visibility:hidden/opacity:0. anon reach 80% -> 100%, holdout 89% -> 90%, p95 38.6s -> 9.7s. - the composer poll slept a blind 0+1.2+1.4 = 2.6s whenever prestage staged nothing, which is nearly every run, and it was the whole of other_ms's suspicious constancy (2610-2613ms regardless of tools_ms). Stop when two reads are identical, the rule the opener poll 40 lines below already applies. other_ms -53.7%, tools_ms flat. - prestage no longer navigates to the page it is already on, nor sleeps 0.35s before its first settle probe. - is_replay_boundary reasoned from the NAME alone, so x.com's composer textbox named "Post text" was ruled an irreversible send and truncated its replay to a bare navigate. Excluded by ROLE; first_unsafe_step now passes role through at all. Measured on this box: reach 100% (83% if onlinegdb's unverified exclusion is bogus), 0 false successes in ~155 runs, prestage tier-0/1 2702ms, other_ms -53.7%, infra flake 0/158, holdout 18/20. Criteria 2/4/9 need live writes and are untouched. Full evidence, including what did NOT work, in e2e/browser-v3/RESULTS_2026-08-06.md. Also drops the tracked electron/node_modules symlink pointing at another machine's Downloads folder; it is dangling on every other checkout and re-breaks the install on any stash or checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
63 lines
3.6 KiB
Python
63 lines
3.6 KiB
Python
"""Popular sites whose composer is published to ANONYMOUS users.
|
|
|
|
The known suite is nine login-gated sites, so on any box without live sessions its reach denominator
|
|
is zero and criterion 1 cannot be scored at all -- which is exactly what happened on 2026-08-06,
|
|
where 8 of 9 known sites were signed out and the suite measured nothing. Session state is an account
|
|
fact, not a product fact, and a benchmark that can only run on one person's logged-in profile is not
|
|
reproducible by anyone else.
|
|
|
|
This suite fixes that: every host below serves a real editable composer with no account, so a miss is
|
|
always OUR miss and never a session state. It is a COMPLEMENT to the known suite, never a replacement
|
|
-- the login-gated sites test flows (opener clicks, lazily-mounted overlays, DM-vs-post disambiguation)
|
|
that these do not, so the original TASKS stay exactly as they were and both are reported separately.
|
|
|
|
Selection rules, so this cannot quietly become a suite we tuned against:
|
|
- No host that appears in HOLDOUT. Checked by `assert_disjoint_from_holdout` at import.
|
|
- Shape spread on purpose: plain textarea, rich contenteditable, CodeMirror, ACE, wikitext.
|
|
A suite of only textareas would flatter us; the known suite's wins are nearly all contenteditable.
|
|
- Reach only, in dry run. Nothing here is ever submitted, and none of these pages is a real
|
|
account surface, so a dry sweep leaves no trace at all.
|
|
"""
|
|
|
|
# Popular, anonymous, and shape-diverse. The task wording carries the payload in quotes because the
|
|
# send-script requires a quoted payload to arm at all; dry mode stops before the irreversible click.
|
|
ANON = {
|
|
"gtranslate": 'Go to translate.google.com and type "coverage probe alpha" into the source text box',
|
|
"deepl": 'Go to deepl.com/translator and type "coverage probe alpha" into the source text box',
|
|
"w3schools": 'Go to w3schools.com/html/tryit.asp?filename=tryhtml_default and write '
|
|
'"coverage probe alpha" in the code editor',
|
|
"wikisandbox": 'Go to en.wikipedia.org/wiki/Wikipedia:Sandbox?action=edit and write '
|
|
'"coverage probe alpha" in the edit box',
|
|
"regex101": 'Go to regex101.com and write "coverage probe alpha" in the test string box',
|
|
"onlinegdb": 'Go to onlinegdb.com and write "coverage probe alpha" in the code editor',
|
|
}
|
|
|
|
# The page each task must END on, same contract as coverage.ON_PAGE: reaching *a* box is not reach,
|
|
# reaching THE box on THE surface is.
|
|
ANON_PAGE = {
|
|
"gtranslate": r"translate\.google\.com",
|
|
"deepl": r"deepl\.com",
|
|
"w3schools": r"w3schools\.com",
|
|
"wikisandbox": r"wikipedia\.org/w/index\.php|wikipedia\.org/wiki/Wikipedia:Sandbox",
|
|
"regex101": r"regex101\.com",
|
|
"onlinegdb": r"onlinegdb\.com",
|
|
}
|
|
|
|
# What each host is here to exercise, so a per-shape reading is possible rather than one aggregate.
|
|
ANON_SHAPE = {
|
|
"gtranslate": "plain textarea", "deepl": "rich contenteditable",
|
|
"w3schools": "textarea in a framed editor", "wikisandbox": "wikitext textarea",
|
|
"regex101": "CodeMirror", "onlinegdb": "ACE editor",
|
|
}
|
|
|
|
|
|
def assert_disjoint_from_holdout(holdout_hosts) -> None:
|
|
"""A known-suite host that is also a holdout host burns the holdout, because everything we fix
|
|
against the first is by definition tuned against the second. Checked in code rather than trusted
|
|
to review: this file and HOLDOUT are edited months apart by people who cannot see each other."""
|
|
overlap = set(ANON) & set(holdout_hosts)
|
|
if overlap:
|
|
raise AssertionError(
|
|
f"anon suite overlaps the frozen holdout on {sorted(overlap)}; "
|
|
"that burns the holdout, pick different hosts")
|