mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-08-23 13:02:23 +02:00
[eric] browser-v3: move the measurement harness into the repo, out of a temp dir
This commit is contained in:
ciregenz
@@ -0,0 +1,4 @@
|
||||
# Run output: logs, per-trial artifacts, and a multi-gigabyte Electron profile with live logins.
|
||||
# The raw jsonl rows under results/ ARE committed; they are the evidence. Nothing else here is.
|
||||
runs/
|
||||
results/*.log
|
||||
@@ -0,0 +1,170 @@
|
||||
# Handoff: browser v3, state as of 2026-08-06
|
||||
|
||||
Branch `eric/browser-merged`, 41 commits past `origin/eric/dev`. Browser suite 707 passing, tsc 0,
|
||||
linter no new violations, tree clean. The goal and the harness are described in `README.md`.
|
||||
|
||||
**Six of nine criteria pass. Three are open, and none of them is open because the fix is unknown.**
|
||||
|
||||
## Scorecard
|
||||
|
||||
| # | criterion | target | before | now | |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 1 | composer reach | >=90% | 57% | **93%** (28/30) | PASS |
|
||||
| 2 | verified writes | >=95% | no honest data | LinkedIn proven end to end; N too small | OPEN |
|
||||
| 3 | false success | 0 | unknown | **0** in every live round | PASS |
|
||||
| 4 | median write | <=12s | ~21s | **1.99s** (p95 12.2s) | PASS |
|
||||
| 5 | cold start tier-0/1 | <=3s | ~16s | **0.13s** | PASS |
|
||||
| 6 | other_ms | -50% | none | **-70%** (417 -> 126ms) | PASS |
|
||||
| 7 | infra flake | <=1% | 60% | 34 runs, 0 failures; sample too small | OPEN |
|
||||
| 8 | holdout | >=80%, <=10pt | untested | **87%**, 6pt gap | PASS |
|
||||
| 9 | learned path | remove or >=50% | 0/55 | recording bug fixed; live rate unmeasured | OPEN |
|
||||
|
||||
Per-site reach at N=5 (criterion 1): x 5/5, linkedin 5/5, reddit 5/5, **instagram 5/5** (was 1/10),
|
||||
youtube 4/5, **twitch 4/5** (was 0/3). gmail, tiktok and substack are excluded, see Exclusions.
|
||||
|
||||
Timing at n=28 successful runs (criterion 4/6): wall median 1988ms / p95 12180ms; browser tools
|
||||
median 1621ms; **`other_ms` median 126ms, down from 417ms**. Wall fell too, so nothing moved from one
|
||||
bucket into another.
|
||||
|
||||
## The finding that matters most
|
||||
|
||||
**Several "product failures" were the measuring instrument.** This is the single most useful thing to
|
||||
carry forward, because it recurred six times in one session and each instance cost hours chasing a
|
||||
bug that did not exist.
|
||||
|
||||
The worst case: criterion 2 failed on LinkedIn for days. **LinkedIn was never broken.** The canary
|
||||
proved a write landed by grepping the backend log for the marker string, and the backend does not log
|
||||
page text. A grep for every canary marker ever generated, across every backend log on the machine,
|
||||
returns zero lines. The audit could only ever answer "could not look", and that was being read as a
|
||||
product failure. `probe_evidence.py` settled it: the session API carries the model's *answer* and
|
||||
never the tool results, so auditing it is asking the same model whose claim is under audit.
|
||||
|
||||
Every dead grep found, and what each did:
|
||||
|
||||
| where | needle | occurrences in source |
|
||||
| --- | --- | --- |
|
||||
| canary `delivered` | `DELIVERY CONFIRMED` | 0 |
|
||||
| canary `saw_page` | two `[browser-action] X` strings | 0 |
|
||||
| canary receipt | only the fast-lane string; 2 of 3 producers missed | 1 of 3 |
|
||||
| skillstats `replay_full` | `replay(ed\|ing) N steps` | 0 |
|
||||
| bench `infra_browser` | `card is unavailable` | 0 |
|
||||
| `stack.sh status` | `pgrep -fc` (no such flag on macOS) | printed 0 over a live stack |
|
||||
|
||||
`verify_markers.py` now checks all 32 harness literals against the source that prints them: 32/32
|
||||
present. **Run it before trusting any number.**
|
||||
|
||||
## What is open, why, and exactly what to do
|
||||
|
||||
### Criterion 2, verified writes
|
||||
|
||||
The instrument was rebuilt (`950755af`) and the first result was linkedin **PASS**: posted,
|
||||
receipt-verified, deleted, verified gone. That is the first clean end-to-end LinkedIn round this
|
||||
project has recorded. One site is not >=95% across sites, so the criterion is not met.
|
||||
|
||||
Blocked on nothing technical. Two things to do first:
|
||||
|
||||
1. **Verify reddit's Title field name.** `af68dcd0` prefers the textbox whose accessible name
|
||||
contains the field word the task used, and requires *exactly one* match. The unit tests use an
|
||||
assumed listing (`[21]<textbox "Title" />`) that has never been confirmed against the live page;
|
||||
the backend only logs `textboxes=3`. If reddit's title input is named something else, the fix
|
||||
silently does nothing. reddit is 1 of the 3 sites in this denominator.
|
||||
2. Then `ROUNDS=7 c2_rounds.sh` and `c2_tally.py`.
|
||||
|
||||
### Criterion 7, infrastructure flake
|
||||
|
||||
Last clean sample: **34 of 108 runs, 0 infra failures, 0 backend restarts.** Encouraging but partial.
|
||||
An earlier 5.4% reading was contaminated: a second OpenSwarm checkout was up on :8324 and the shared
|
||||
9router on :20128 was being evicted.
|
||||
|
||||
Needs a box with nothing on :8324, then `N=12 c7_run.sh` for the full 108.
|
||||
|
||||
### Criterion 9, learned fast path
|
||||
|
||||
Baseline is now measured properly: **427 gate decisions, 95 eligible, 0 recorded (0%), 0 replays**,
|
||||
with one refusal reason (`host empty or no robust steps`).
|
||||
|
||||
The gate was never the problem. Every eligible run died inside `record_skill`: `distill_steps` reads
|
||||
`clicked_name`, and `browser_send_script.py` wrote the element name into `result_summary` prose only.
|
||||
An unnameable click truncates the distillation, truncation drops the typing steps, and the
|
||||
navigation-only remainder is correctly refused. Fixed in `438a96eb`, proven directly:
|
||||
|
||||
```
|
||||
OLD shape (no clicked_name) -> []
|
||||
NEW shape -> ['BrowserNavigate', 'BrowserClickByName', 'BrowserClickByName']
|
||||
```
|
||||
|
||||
Safety held rather than added: a `BrowserClickIndex` distills to a name-only `BrowserClickByName`, so
|
||||
the payload never enters the skill, and the send click's tool name matches no distill branch, so a
|
||||
replay cannot re-fire a send.
|
||||
|
||||
**Not yet proven live.** A later dry sweep still showed 0/11, but the eligible hosts were
|
||||
`accounts.google.com`, `substack.com` and `twitch.tv` (signed-out and no-composer runs), which the
|
||||
fix does not touch. Separately, `skillstats.py` reported 8 prefix replays alongside 0 "matched",
|
||||
which cannot both be true, so it has a counting bug of its own. **The honest reading is that
|
||||
criterion 9 has no verdict yet, not that the fix failed.**
|
||||
|
||||
Fix the `skillstats` counter, then read the recording rate off a fresh 108-run sweep. If replay shows
|
||||
no benefit on off-table hosts, removal is the honest answer and the criterion explicitly allows it;
|
||||
the two `browser-memory` endpoints have no frontend caller, so removal is not user-visible.
|
||||
|
||||
## Product bugs found by the fixed instrument
|
||||
|
||||
- **reddit: the task named a field and nobody read it** (`af68dcd0`). "create a text post whose title
|
||||
is exactly X" filled the body, Title stayed empty, and reddit's submit stayed DISABLED on every
|
||||
attempt. A field word from the user's own sentence now beats the compose-shape guess. 6 tests,
|
||||
including that no-hint behaviour is byte-identical.
|
||||
- **disqus: a stalled top document denied its own child frames** (`7ba99ba4`). The still-loading
|
||||
retry rethrew on a second failure, killing `find_composer` before the child-frame search ran. An
|
||||
ad-heavy page keeps the top document loading while the composer, in an embedded iframe with its own
|
||||
load state, is perfectly readable.
|
||||
|
||||
## Known false negatives, filed not fixed
|
||||
|
||||
Both under-claim rather than over-claim, so neither violates criterion 3, but both read as drift on
|
||||
every live round.
|
||||
|
||||
- **A torn-down browser card wedges, and `navigate` lies about it.** Reproduced 4x: the reply is
|
||||
`{"text":"Navigated to https://x.com/home","url":"https://x.com/home"}` in ~80ms while the webview
|
||||
never leaves reddit. Anything trusting that reply reads the wrong page and answers confidently
|
||||
about it. The audit now re-reads `location.href` and requires a host match.
|
||||
- **Delete reports failure on deletions that provably worked.** X, 2 for 2: "the deletion was never
|
||||
confirmed", and an independent read shows the post gone both times. LinkedIn showed the same shape.
|
||||
|
||||
## Exclusions (predefined, never quietly dropped)
|
||||
|
||||
gmail, substack and tiktok are signed out; tiktok is additionally captcha-walled. That is 14 of 45
|
||||
rows on the known suite. Each exclusion is judged on the page's own evidence, never on the product's
|
||||
claim. Solving a bot-detection challenge is off-limits, so a captcha-walled page is one this system
|
||||
is *choosing* not to reach, and scoring it against reach would charge us for a rule we intend to keep.
|
||||
|
||||
## Account hygiene
|
||||
|
||||
Every marker written to a real account during this work was cleaned up and independently verified
|
||||
gone: X profile scan shows 0 markers with the newest genuine post predating the tests, and the
|
||||
LinkedIn post permalink returns nav chrome with no content.
|
||||
|
||||
Personal handles are **not** committed. Both live sites read their account from the environment
|
||||
(`OSW_CANARY_X_HANDLE`, `OSW_CANARY_REDDIT_HANDLE`), and an unset handle makes the round refuse
|
||||
loudly rather than audit a malformed URL.
|
||||
|
||||
## Environment notes
|
||||
|
||||
- The stack is backend :8326, webpack :3026, Electron on its own `--user-data-dir`. It never touches
|
||||
:8324 or :3000, which belong to whatever else is on the machine.
|
||||
- The backend was SIGTERMed three times mid-measurement, always cleanly, always with nothing in its
|
||||
log. Cause unknown. `stack.sh` now supervises it and writes `[stack] BACKEND RESTARTED` into the log
|
||||
the harness slices, so `bench.py`'s `infra_backend` bucket can exclude any trial spanning a restart
|
||||
instead of scoring it as a product failure.
|
||||
- `stack.sh down` once matched `pkill -f "uvicorn backend.main"`, which is exactly what the other
|
||||
checkout runs, and it executed while that checkout was live. It is now scoped to this stack's own
|
||||
ports and profile and cannot match a process it did not start.
|
||||
|
||||
## Next executable steps, in order
|
||||
|
||||
1. `verify_markers.py` (32/32 expected).
|
||||
2. Confirm reddit's Title field name against the live page.
|
||||
3. Fix the `skillstats.py` replay counter.
|
||||
4. `N=12 c7_run.sh` on a box with nothing on :8324. Gives criteria 1, 4, 5, 6, 7.
|
||||
5. `skillstats.py` on that sweep's log. Gives criterion 9's live recording rate.
|
||||
6. `ROUNDS=7 c2_rounds.sh` then `c2_tally.py`. Gives criterion 2.
|
||||
7. `N=2 c8_run.sh` to re-check criterion 8 after the disqus fix.
|
||||
@@ -0,0 +1,100 @@
|
||||
# v3 holdout set, FROZEN
|
||||
|
||||
Frozen 2026-08-02 before any site-specific v3 work. Repo HEAD at freeze: `1508e03c`.
|
||||
|
||||
Integrity claim: no commit touching twitch, tiktok, or instagram behaviour exists at or before
|
||||
`1508e03c`. Verify with `git log 1508e03c --oneline`. If a later commit tunes any host below, this
|
||||
holdout is burned and a new one must be frozen.
|
||||
|
||||
Selection rule: the host string appears NOWHERE under `backend/apps/agents/browser/`. Checked
|
||||
against the 106 hosts the module references (`seed_playbooks.py` carries most of them). Candidates
|
||||
rejected for contamination: news.ycombinator.com, soundcloud.com, quora.com, pinterest.com,
|
||||
wikipedia.org, github.com, stackoverflow.com, medium.com.
|
||||
|
||||
## The six
|
||||
|
||||
| host | composer architecture | task |
|
||||
| --- | --- | --- |
|
||||
| bsky.app | React SPA, modal composer | create a post |
|
||||
| mastodon.social | inline composer on the home timeline | create a post |
|
||||
| dev.to | URL-routed form (`/new`), markdown textarea | create a post |
|
||||
| lobste.rs | server-rendered form (`/stories/new`) | create a story |
|
||||
| pastebin.com | plain textarea, **no login required** | create a paste |
|
||||
| meta.discourse.org | Discourse SPA, "New Topic" composer | create a topic |
|
||||
|
||||
Spread is deliberate: 2 plain forms, 2 SPA modals, 1 inline, 1 URL-routed. If reach holds on the
|
||||
plain forms and collapses on the SPA modals, that localises the generalisation failure instead of
|
||||
producing one useless aggregate.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Reach only, in dry run.** No holdout submit, ever. These are not the user's accounts.
|
||||
2. **No tuning against these hosts** until the first evaluation is recorded in `holdout_run1.json`.
|
||||
3. Sign-in state is measured, never assumed. A signed-out host is reported NOT MEASURABLE with the
|
||||
page's own evidence and leaves the denominator, exactly as gmail does on the known suite.
|
||||
pastebin.com is the one host guaranteed measurable regardless of session state, which is why it
|
||||
is in the set.
|
||||
4. Every attempt is published, including failures, retries, and exclusions.
|
||||
|
||||
## Addendum, frozen 2026-08-02 at HEAD `ab898964`, before evaluating any of it
|
||||
|
||||
The first holdout was unmeasurable: 5 of 6 sites are signed out and I am not permitted to sign in,
|
||||
so it tested nothing. It did earn its keep once (pastebin exposed the header-login-widget bug), but
|
||||
a holdout that cannot be scored is not a generalisation test.
|
||||
|
||||
Fix: six more hosts, chosen because they publish a composer to anonymous users, so session state can
|
||||
never be the reason a run fails. Same contamination rule, all six verified absent from everything
|
||||
under `backend/apps/agents/browser/`. Frozen before the first run, per criterion 8.
|
||||
|
||||
| host | composer architecture | task |
|
||||
| --- | --- | --- |
|
||||
| rentry.co | plain textarea, markdown | create a paste |
|
||||
| dpaste.org | plain textarea in a server form | create a paste |
|
||||
| controlc.com | plain textarea | create a paste |
|
||||
| txti.es | plain textarea, minimal markup | create a page |
|
||||
| justpaste.it | rich contenteditable editor | create a note |
|
||||
| telegra.ph | rich contenteditable (Telegram's editor) | create a page |
|
||||
|
||||
Four plain textareas and two contenteditable rich editors on purpose. The known suite's wins are
|
||||
almost all contenteditable (x, linkedin, youtube, twitch), so a holdout of only rich editors would
|
||||
flatter us, and one of only textareas would not exercise the path that actually carries production.
|
||||
|
||||
Same rules as above: reach only, dry run, never submitted, every attempt published.
|
||||
|
||||
## Editor-shape addendum, frozen 2026-08-04 at HEAD `e445ca3e`, before evaluating any of it
|
||||
|
||||
Eric's observation, and it is the sharpest critique of this benchmark so far: the suite was picked by
|
||||
site POPULARITY, and popularity is not what determines whether we generalize. The **editor library**
|
||||
is. The web's writing surfaces cluster into roughly eight shapes and most sites just adopt one, so
|
||||
coverage should be counted per shape, not per famous site.
|
||||
|
||||
What the existing suites actually cover, audited:
|
||||
|
||||
| shape | covered by | samples |
|
||||
| --- | --- | --- |
|
||||
| plain `<textarea>` | rentry, dpaste, controlc, txti, pastebin | 5 |
|
||||
| raw contenteditable | telegra.ph, justpaste.it | 2 |
|
||||
| Draft.js family | x.com | 1 |
|
||||
| Slate / Lexical | twitch | 1 |
|
||||
| web components / shadow DOM | youtube | 1 |
|
||||
| multi-field form | reddit | 1 |
|
||||
| **iframe-embedded composer** | gmail (SIGNED OUT) | **0 measurable** |
|
||||
| **Quill / TinyMCE / CKEditor** | nothing | **0** |
|
||||
| canvas-based (Docs, Figma) | nothing | 0, likely out of scope |
|
||||
|
||||
The last two rows are the real hole, and they are not niche: every WordPress admin, every helpdesk
|
||||
reply box, and every Disqus thread lives there. Disqus is the highest-value single target because it
|
||||
is BOTH an iframe AND a rich editor, and it is embedded on millions of sites.
|
||||
|
||||
Frozen set, all verified absent from everything under `backend/apps/agents/browser/`, and all public
|
||||
demo pages so no login is required and grading stays honest:
|
||||
|
||||
| host | shape | task |
|
||||
| --- | --- | --- |
|
||||
| disqus.com | iframe + rich editor | leave a comment on the demo thread |
|
||||
| quilljs.com | Quill | write in the playground editor |
|
||||
| tiny.cloud | TinyMCE | write in the demo editor |
|
||||
| ckeditor.com | CKEditor 5 | write in the demo editor |
|
||||
| codepen.io | CodeMirror in an iframe | write in the HTML pane |
|
||||
|
||||
Reach only, dry run, never submitted. Every attempt published, including failures and exclusions.
|
||||
@@ -0,0 +1,106 @@
|
||||
# Browser v3: the goal, where we are, and how to pick this up
|
||||
|
||||
This directory is the measurement harness for the browser write path, plus the evidence it has
|
||||
produced so far. It exists because the browser agent's numbers were repeatedly wrong in ways that
|
||||
looked like product bugs, and the only defence is an instrument you can re-run and audit.
|
||||
|
||||
**Read `HANDOFF.md` first if you are continuing this work.** It has the current scorecard, the three
|
||||
open criteria, and the exact next commands.
|
||||
|
||||
## The goal
|
||||
|
||||
Nine exit criteria, all of which must hold at once:
|
||||
|
||||
| # | criterion | target | baseline |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | composer reach across the known suite | >=90% | 57% |
|
||||
| 2 | verified writes across sites and repetitions | >=95% | no honest data |
|
||||
| 3 | false success claims | **exactly 0** | unknown |
|
||||
| 4 | median successful-write wall time | <=12s | ~21s |
|
||||
| 5 | cold-start / prestage on a tier-0/1 hit | <=3s | ~16s |
|
||||
| 6 | `other_ms` (the part our code owns) | -50% | unmeasured |
|
||||
| 7 | infrastructure flake over >=100 site-runs | <=1% | 60% |
|
||||
| 8 | frozen-holdout reach, and its gap to the known set | >=80%, <=10pt | untested |
|
||||
| 9 | learned fast path: remove it, or >=50% recording with real replays | either | 0 of 55 |
|
||||
|
||||
Definitions that decide arguments later:
|
||||
|
||||
- **Reach** = navigated to the intended surface AND page-verified the correct editable composer.
|
||||
Filling *a* box is not reach; filling *the* box is.
|
||||
- **Verified write** = the exact content was independently confirmed at the correct destination.
|
||||
Typed-only, unsubmitted, wrong-composer and unverified all count as failures.
|
||||
- **False success** = reporting success without postcondition evidence. This is the hard gate.
|
||||
- Unknowns, timeouts and silent failures are failures, never omissions.
|
||||
|
||||
## Layout
|
||||
|
||||
| file | what it does |
|
||||
| --- | --- |
|
||||
| `coverage.py` | **The one grader.** Site tasks, surface rules, exclusion logic. Never reimplement grading elsewhere; two graders drift and the harness starts lying in a new way. |
|
||||
| `bench.py` | N trials per site, infra-vs-product bucketing, timing split, one artifact per trial. |
|
||||
| `stack.sh` | Boots the isolated stack (backend :8326, webpack :3026, Electron on its own profile). `up dry`, `up live`, `down`, `status`. |
|
||||
| `keep_renderer.sh` | Keeps an Electron renderer alive for the length of a sweep. |
|
||||
| `c7_run.sh` | Known suite at N=12 = 108 site-runs. Criteria 1, 4, 5, 6, 7 in one pass. |
|
||||
| `c8_run.sh` | The frozen holdout (criterion 8). |
|
||||
| `c2_rounds.sh` | Live write rounds on the authorised accounts (criterion 2). |
|
||||
| `c2_tally.py` | Aggregates the live rounds: verified writes, false successes, stranded markers. |
|
||||
| `skillstats.py` | Criterion 9: recording rate, replay rate, and every refusal reason. |
|
||||
| `verify_markers.py` | **Run this before trusting any number.** Checks all 32 literals the harness greps for against the source that prints them. |
|
||||
| `rawbrowser.py` | Drives a browser card with no model in the loop. The audit channel, and the tool for cleaning up a stranded test post by hand. |
|
||||
| `probe_evidence.py` | Dumps where a run's page text actually surfaces. Written to settle the question empirically instead of by assumption. |
|
||||
| `HOLDOUT_FROZEN.md` | The holdout sites and the commit each was frozen at, before its first run. |
|
||||
| `results/*.jsonl` | Raw per-trial rows. Every attempt, including failures and exclusions. |
|
||||
|
||||
## Running it
|
||||
|
||||
Everything needs the isolated stack and a quiet box.
|
||||
|
||||
```bash
|
||||
e2e/browser-v3/stack.sh up dry # dry: the backend refuses the irreversible click
|
||||
N=12 e2e/browser-v3/c7_run.sh # criteria 1, 4, 5, 6, 7
|
||||
N=2 e2e/browser-v3/c8_run.sh # criterion 8
|
||||
e2e/browser-v3/skillstats.py runs/*.log # criterion 9
|
||||
e2e/browser-v3/stack.sh down
|
||||
```
|
||||
|
||||
Live write tests post to real accounts, so they need explicit handles and a live backend:
|
||||
|
||||
```bash
|
||||
e2e/browser-v3/stack.sh up live
|
||||
OSW_CANARY_X_HANDLE=<handle> OSW_CANARY_REDDIT_HANDLE=<handle> \
|
||||
ROUNDS=7 e2e/browser-v3/c2_rounds.sh
|
||||
e2e/browser-v3/c2_tally.py runs/c2_r7.txt
|
||||
```
|
||||
|
||||
`OSW_BENCH_DIR` relocates run output (logs plus a multi-gigabyte browser profile) off the repo disk.
|
||||
|
||||
## Rules this harness enforces on itself
|
||||
|
||||
These are not style preferences. Each one is a bug that already shipped a wrong number.
|
||||
|
||||
1. **Never grep for a string the code does not print.** Five separate dead greps were found in one
|
||||
session, each silently turning a metric into a constant. `verify_markers.py` exists to make that
|
||||
class impossible. Run it first.
|
||||
2. **Absence is not evidence unless the read succeeded.** "The marker is not on the page" and "we
|
||||
never got a good look at the page" are different answers. Collapsing them into `False` falsely
|
||||
accused a working LinkedIn send of lying.
|
||||
3. **Never grade the guard with the guard.** The audit reads the destination itself and consults no
|
||||
model. A receipt reporting on its own correctness proves nothing.
|
||||
4. **Publish every attempt, exclusion and retry.** An excluded row is a claim that the product was
|
||||
not on trial, and that claim has to survive being read out loud.
|
||||
5. **A site being signed out is an account state, not a coverage failure.** Exclusions are judged on
|
||||
the page's own evidence (a sign-in URL, a visible password field, a bot-detection challenge),
|
||||
never on the product's say-so.
|
||||
6. **One backend per box.** A second OpenSwarm evicts the shared 9router and every timing becomes
|
||||
noise: the same sweep once scored 1/9 and 4/9 with zero code change. `stack.sh status` warns when
|
||||
:8324 is occupied, and `stack.sh down` is scoped so it can never kill another checkout's stack.
|
||||
|
||||
## Safety
|
||||
|
||||
- Live rounds write only to accounts the operator explicitly names by environment variable, and every
|
||||
post is deleted in the same run and re-verified gone.
|
||||
- Markers are random and carry no removal words, which would otherwise trip the removal classifier.
|
||||
- Dry mode is enforced by the backend (`OSW_SENDSCRIPT_DRYRUN=1`), never by a flag this harness sets
|
||||
and does not own. An earlier draft set it in the wrong process, which would have posted for real.
|
||||
- The canary prints its marker on every row, not just failures. It is the only record of what was put
|
||||
on a real account, so a stranded post stays cleanable by hand.
|
||||
@@ -0,0 +1,252 @@
|
||||
"""v3 benchmark: N trials per site, structured traces, honest denominators.
|
||||
|
||||
Two things this does that the old sweep did not.
|
||||
|
||||
1. N trials. One run per site is noise: on 2026-08-02 x went 2-of-3 and instagram 1-of-3, and a
|
||||
single-run number would have called either one whatever the last run happened to be.
|
||||
2. It separates INFRASTRUCTURE failure from PRODUCT failure and publishes both. A router death is
|
||||
not a coverage miss, and hiding it inflates the score; calling a real bug "infrastructure" is the
|
||||
mistake that hid the surrogate crash for a whole session. Every attempt lands in the raw file.
|
||||
|
||||
Timings come from browser_metrics/tasks.jsonl (the wall/llm/tools/other split), not from log
|
||||
scraping, because the wall clock cannot tell "our code got faster" from "the model took fewer turns".
|
||||
|
||||
Grading is imported from coverage.py, never reimplemented. Two graders would drift and the harness
|
||||
would start lying again in a new way.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
# coverage.py sits BESIDE this file. Pointing one directory higher was a live trap rather than a
|
||||
# typo: PyPI ships a package called `coverage` (the code-coverage tool), so on any machine that has
|
||||
# it installed `import coverage` would quietly bind to that instead of the grader, and every verdict
|
||||
# would come from a module that has never heard of a composer. Insert this directory FIRST.
|
||||
sys.path.insert(0, HERE)
|
||||
import coverage as C # noqa: E402 the ONE grader
|
||||
|
||||
assert hasattr(C, "TASKS"), f"imported the wrong 'coverage' module: {C.__file__}"
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(HERE))
|
||||
METRICS = os.path.join(ROOT, "backend/data/browser_metrics/tasks.jsonl")
|
||||
|
||||
# Frozen 2026-08-02 at HEAD 1508e03c, before any site-specific v3 work. See HOLDOUT_FROZEN.md.
|
||||
# Reach only, dry run, never submitted: these are not the user's accounts.
|
||||
HOLDOUT = {
|
||||
"bsky": 'Go to bsky.app and create a post with exactly this text: "coverage probe alpha"',
|
||||
"mastodon": 'Go to mastodon.social and create a post with exactly this text: "coverage probe alpha"',
|
||||
"devto": 'Go to dev.to and start a new post with the body "coverage probe alpha"',
|
||||
"lobsters": 'Go to lobste.rs and start a new story with the text "coverage probe alpha"',
|
||||
"pastebin": 'Go to pastebin.com and create a new paste with the text "coverage probe alpha"',
|
||||
"discourse": 'Go to meta.discourse.org and start a new topic with the body "coverage probe alpha"',
|
||||
}
|
||||
HOLDOUT_PAGE = {
|
||||
"bsky": r"bsky\.app",
|
||||
"mastodon": r"mastodon\.social",
|
||||
"devto": r"dev\.to/new",
|
||||
"lobsters": r"lobste\.rs/stories/new",
|
||||
"pastebin": r"pastebin\.com",
|
||||
"discourse": r"discourse\.org",
|
||||
}
|
||||
# Addendum, frozen at HEAD ab898964 before its first run. The six above are mostly signed out and
|
||||
# therefore unscoreable; these publish a composer to anonymous users, so a failure is always OUR
|
||||
# failure and never a session state. 4 plain textareas + 2 contenteditable rich editors, because a
|
||||
# holdout of only rich editors would flatter a suite whose wins are nearly all contenteditable.
|
||||
HOLDOUT.update({
|
||||
"rentry": 'Go to rentry.co and create a paste with the text "coverage probe alpha"',
|
||||
"dpaste": 'Go to dpaste.org and create a paste with the text "coverage probe alpha"',
|
||||
"controlc": 'Go to controlc.com and create a paste with the text "coverage probe alpha"',
|
||||
"txti": 'Go to txti.es and create a page with the text "coverage probe alpha"',
|
||||
"justpaste": 'Go to justpaste.it and create a note with the text "coverage probe alpha"',
|
||||
"telegraph": 'Go to telegra.ph and create a page with the text "coverage probe alpha"',
|
||||
})
|
||||
HOLDOUT_PAGE.update({
|
||||
"rentry": r"rentry\.co",
|
||||
"dpaste": r"dpaste\.org",
|
||||
"controlc": r"controlc\.com",
|
||||
"txti": r"txti\.es",
|
||||
"justpaste": r"justpaste\.it",
|
||||
"telegraph": r"telegra\.ph",
|
||||
})
|
||||
|
||||
# Infrastructure, not coverage. Kept separate and PUBLISHED, never folded into either column.
|
||||
INFRA = (
|
||||
# The backend itself. It has died mid-sweep on a clean SIGTERM with nothing in its log
|
||||
# (2026-08-05), and an unnoticed death turns every following trial into a fake product failure.
|
||||
# `BACKEND RESTARTED` is written by the supervisor in stack.sh; BACKEND_DOWN by the harness when
|
||||
# it cannot reach :8326 at all.
|
||||
("infra_backend", (C.BACKEND_DOWN, "[stack] BACKEND RESTARTED")),
|
||||
("infra_router", ("9Router watchdog", "9Router process died", "No AI provider connected")),
|
||||
# The webview is gone or wedged. "card is unavailable" was in this list and appears NOWHERE in
|
||||
# the codebase, so a third of this bucket could never match; these are P_CARD_GONE_MARKERS
|
||||
# (browser_loop.py) plus the renderer's own read-timeout wording, all grep-verified.
|
||||
("infra_browser", ("Browser command timed out", "too busy to read", "not an electron webview",
|
||||
"no dashboard is connected", "page unresponsive")),
|
||||
# No renderer attached. Its own bucket because it is neither the router nor the page: Electron
|
||||
# launched before webpack was serving, hit a dead URL and quit, and 44 straight runs then failed
|
||||
# with "dispatch refused: no dashboard". Filed as generic harness trouble that read as a mystery;
|
||||
# named, it says exactly which process to restart.
|
||||
("infra_no_renderer", ("dispatch refused: no dashboard", "no OpenSwarm window is connected")),
|
||||
)
|
||||
|
||||
|
||||
# Editor-shape addendum, frozen at HEAD e445ca3e before its first run. Coverage should be counted per
|
||||
# EDITOR LIBRARY, not per famous site: the web's writing surfaces cluster into ~8 shapes and most
|
||||
# sites adopt one. These five fill the two holes the popularity-picked suites left entirely empty,
|
||||
# iframe-embedded composers and the classic CMS editors. All public demos, so no login gates them.
|
||||
HOLDOUT.update({
|
||||
"disqus": 'Go to https://blog.disqus.com/ and open the first blog post, then write a '
|
||||
'comment "coverage probe alpha" in the Disqus comment box at the bottom',
|
||||
"quill": 'Go to quilljs.com/playground and write "coverage probe alpha" in the editor',
|
||||
"tinymce": 'Go to tiny.cloud and write "coverage probe alpha" in the demo editor',
|
||||
"ckeditor": 'Go to ckeditor.com/ckeditor-5/demo and write "coverage probe alpha" in the editor',
|
||||
"codemirror": 'Go to codepen.io/pen and write "coverage probe alpha" in the HTML pane',
|
||||
})
|
||||
HOLDOUT_PAGE.update({
|
||||
"disqus": r"disqus\.com", "quill": r"quilljs\.com", "tinymce": r"tiny\.cloud",
|
||||
"ckeditor": r"ckeditor\.com", "codemirror": r"codepen\.io",
|
||||
})
|
||||
|
||||
|
||||
# Buckets that leave the reach denominator. Defined ONCE: the two call sites used to list them by
|
||||
# hand and a new bucket added to only one of them silently changes the score.
|
||||
EXCLUDED = ("not_measurable", "infra_backend", "infra_router", "infra_browser",
|
||||
"infra_harness", "infra_no_renderer")
|
||||
|
||||
|
||||
def classify(v, slice_):
|
||||
"""One bucket per trial. Unknowns and timeouts are failures, never silently dropped."""
|
||||
for name, needles in INFRA:
|
||||
if any(n in slice_ for n in needles):
|
||||
return name
|
||||
if v.get("invalid"):
|
||||
return "infra_harness"
|
||||
if v.get("unmeasurable"):
|
||||
return "not_measurable"
|
||||
if v["composer"]:
|
||||
return "ok"
|
||||
d = v["detail"]
|
||||
if "FILL DIED" in d:
|
||||
return "product_fill_died"
|
||||
if "WRONG SURFACE" in d:
|
||||
return "product_wrong_surface"
|
||||
return "product_no_composer"
|
||||
|
||||
|
||||
def metrics_between(t0, t1):
|
||||
"""The browser sub-agent's own timing rows for this trial's window."""
|
||||
out = []
|
||||
try:
|
||||
with open(METRICS, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if t0 <= float(r.get("ts", 0)) <= t1 and r.get("llm_ms") is not None:
|
||||
out.append(r)
|
||||
except OSError:
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def trial(site, task, n_idx):
|
||||
t0 = time.time()
|
||||
slice_, wall = C.run(site, task)
|
||||
t1 = time.time()
|
||||
v = C.verdict(site, slice_)
|
||||
v["site"], v["trial"] = site, n_idx
|
||||
bucket = classify(v, slice_)
|
||||
rows = metrics_between(t0 - 2, t1 + 2)
|
||||
best = max(rows, key=lambda r: r.get("total_ms", 0)) if rows else {}
|
||||
rec = {
|
||||
"site": site, "trial": n_idx, "bucket": bucket, "wall_s": wall,
|
||||
"detail": v["detail"], "composer": bool(v["composer"]), "submit": bool(v["submit"]),
|
||||
"total_ms": best.get("total_ms"), "llm_ms": best.get("llm_ms"),
|
||||
"tools_ms": best.get("tools_ms"), "other_ms": best.get("other_ms"),
|
||||
"turns": best.get("turns"),
|
||||
}
|
||||
art = os.path.join(HERE, "results", f"{site}_{n_idx}.log")
|
||||
os.makedirs(os.path.dirname(art), exist_ok=True)
|
||||
with open(art, "w", encoding="utf-8") as f:
|
||||
f.write(slice_)
|
||||
rec["artifact"] = os.path.relpath(art, HERE)
|
||||
return rec
|
||||
|
||||
|
||||
def report(recs, label):
|
||||
print(f"\n=== {label}: {len(recs)} attempts ===")
|
||||
by_site = {}
|
||||
for r in recs:
|
||||
by_site.setdefault(r["site"], []).append(r)
|
||||
print(f" {'site':<11}{'reach':>8} buckets")
|
||||
for site, rs in by_site.items():
|
||||
meas = [r for r in rs if r["bucket"] not in EXCLUDED]
|
||||
ok = sum(r["bucket"] == "ok" for r in rs)
|
||||
rate = f"{ok}/{len(meas)}" if meas else "0/0"
|
||||
seen = {}
|
||||
for r in rs:
|
||||
seen[r["bucket"]] = seen.get(r["bucket"], 0) + 1
|
||||
print(f" {site:<11}{rate:>8} {seen}")
|
||||
meas = [r for r in recs if r["bucket"] not in EXCLUDED]
|
||||
ok = [r for r in recs if r["bucket"] == "ok"]
|
||||
infra = [r for r in recs if r["bucket"].startswith("infra")]
|
||||
print(f"\n REACH (product-measurable denominator): {len(ok)}/{len(meas)} "
|
||||
f"= {round(100 * len(ok) / len(meas)) if meas else 0}%")
|
||||
print(f" not measurable (signed out): {sum(r['bucket'] == 'not_measurable' for r in recs)}")
|
||||
print(f" INFRASTRUCTURE failures: {len(infra)}/{len(recs)} "
|
||||
f"= {round(100 * len(infra) / len(recs), 1) if recs else 0}% {[r['bucket'] for r in infra]}")
|
||||
lat = [r for r in ok if r.get("total_ms")]
|
||||
if lat:
|
||||
for key in ("total_ms", "llm_ms", "tools_ms", "other_ms"):
|
||||
vals = sorted(r[key] for r in lat if r.get(key) is not None)
|
||||
if vals:
|
||||
p95 = vals[min(len(vals) - 1, int(0.95 * len(vals)))]
|
||||
print(f" {key:<10} median={round(statistics.median(vals))}ms p95={p95}ms n={len(vals)}")
|
||||
else:
|
||||
print(" no successful runs carried a timing row")
|
||||
|
||||
|
||||
USAGE = """usage: bench.py [known|holdout] [N] [site ...]
|
||||
|
||||
known the known suite (default)
|
||||
holdout the frozen holdout, see HOLDOUT_FROZEN.md
|
||||
N trials per site (default 5)
|
||||
site ... restrict to these sites
|
||||
|
||||
Needs the isolated stack up: e2e/browser-v3/stack.sh up dry
|
||||
"""
|
||||
|
||||
if __name__ == "__main__":
|
||||
# An unrecognised first argument used to be treated as a SUITE NAME, so `bench.py --help`
|
||||
# quietly started firing real trials at a backend that was not there. The first thing anyone
|
||||
# new types is --help, so refuse anything that is not a real suite.
|
||||
if len(sys.argv) > 1 and sys.argv[1] not in ("known", "holdout"):
|
||||
print(USAGE)
|
||||
sys.exit(0 if sys.argv[1] in ("-h", "--help", "help") else 2)
|
||||
suite = sys.argv[1] if len(sys.argv) > 1 else "known"
|
||||
n = int(sys.argv[2]) if len(sys.argv) > 2 else 5
|
||||
only = sys.argv[3:]
|
||||
if suite == "holdout":
|
||||
tasks, C.ON_PAGE = HOLDOUT, {**C.ON_PAGE, **HOLDOUT_PAGE}
|
||||
C.PUBLIC = C.PUBLIC + tuple(HOLDOUT)
|
||||
else:
|
||||
tasks = C.TASKS
|
||||
if only:
|
||||
tasks = {k: v for k, v in tasks.items() if k in only}
|
||||
recs = []
|
||||
out = os.path.join(HERE, "results", f"{suite}_raw.jsonl")
|
||||
os.makedirs(os.path.dirname(out), exist_ok=True)
|
||||
with open(out, "a", encoding="utf-8") as fh:
|
||||
for i in range(1, n + 1):
|
||||
for site, task in tasks.items():
|
||||
r = trial(site, task, i)
|
||||
recs.append(r)
|
||||
fh.write(json.dumps(r) + "\n")
|
||||
fh.flush()
|
||||
print(f" [{i}/{n}] {site:<11}{r['bucket']:<22}{r['wall_s']:>6.1f}s {r['detail'][:62]}",
|
||||
flush=True)
|
||||
report(recs, f"{suite} suite, N={n}")
|
||||
Executable
+35
@@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
# Criterion 2: verified writes, on the three accounts authorised for real sends.
|
||||
#
|
||||
# Needs a LIVE backend (`stack.sh up live`). Every round posts a unique marker, audits the
|
||||
# destination by reading it, deletes, and audits again. Markers print on every row so anything
|
||||
# stranded can be cleaned up by hand.
|
||||
# Where logs, profiles and run output go. Defaults to runs/ beside this harness; override with
|
||||
# OSW_BENCH_DIR to keep multi-gigabyte browser profiles off the repo disk.
|
||||
SP="${OSW_BENCH_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/runs}"
|
||||
mkdir -p "$SP"
|
||||
# Repo root from this script's own location, so the harness works in any checkout.
|
||||
TREE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
cd "$TREE" || exit 1
|
||||
OUT="$SP/c2_r7.txt"
|
||||
|
||||
# The operator names the accounts, this file never does. These rounds POST to real profiles, so the
|
||||
# handles have to be a deliberate act by whoever runs it, not a default baked into an open-source
|
||||
# repo. Refuse rather than run: a missing handle would post and then audit a malformed URL, and the
|
||||
# round would report "unprovable" as though the product were at fault.
|
||||
for v in OSW_CANARY_X_HANDLE OSW_CANARY_REDDIT_HANDLE; do
|
||||
if [ -z "${!v}" ]; then
|
||||
echo "refusing: \$$v is not set. These rounds write to real accounts; name them explicitly."
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
|
||||
: > "$OUT"
|
||||
for i in $(seq 1 "${ROUNDS:-7}"); do
|
||||
echo "########## ROUND $i $(date +%H:%M:%S)" >> "$OUT"
|
||||
OSW_CANARY_BASE=http://127.0.0.1:8326 OSW_CANARY_LOG="$SP/r7_be.log" \
|
||||
timeout 2400 ./backend/.venv/bin/python scripts/browser_canary.py --live --sites x,linkedin,reddit \
|
||||
>> "$OUT" 2>&1
|
||||
echo " (backend restarts so far: $(grep -c 'BACKEND RESTARTED' "$SP/r7_be.log" 2>/dev/null))" >> "$OUT"
|
||||
done
|
||||
echo "########## DONE $(date +%H:%M:%S)" >> "$OUT"
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Criterion 2 across rounds: verified writes, false successes, and every marker that was left live.
|
||||
|
||||
The canary prints a per-round summary; the criterion asks about the whole sample. Parses the round
|
||||
output rather than re-running anything, so the numbers can be re-derived from the artifact.
|
||||
|
||||
python3 c2_tally.py c2_r7.txt
|
||||
"""
|
||||
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
ROW = re.compile(r"^\s+\[(PASS|DRIFT)\]\s+(\S+)\s+(canary[0-9a-f]+)?\s*stage=(\S+)\s+(.*)$")
|
||||
PROVEN = re.compile(r"verified writes: (\d+)/(\d+) proven")
|
||||
LIARS = re.compile(r"FALSE SUCCESS CLAIMS: (\d+)")
|
||||
# re.M or `$` only matches the very end of the file and this finds nothing at all,
|
||||
# which reads as "nothing was stranded" -- the exact wrong direction for this line.
|
||||
STRANDED = re.compile(r"MANUAL CLEANUP NEEDED: (.+)$", re.M)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
text = open(sys.argv[1], errors="ignore").read()
|
||||
rounds = text.count("########## ROUND")
|
||||
proven = denom = liars = 0
|
||||
for m in PROVEN.finditer(text):
|
||||
proven += int(m.group(1))
|
||||
denom += int(m.group(2))
|
||||
for m in LIARS.finditer(text):
|
||||
liars += int(m.group(1))
|
||||
|
||||
per_site = {}
|
||||
stages = Counter()
|
||||
for line in text.splitlines():
|
||||
m = ROW.match(line)
|
||||
if not m:
|
||||
continue
|
||||
flag, site, marker, stage, detail = m.groups()
|
||||
d = per_site.setdefault(site, Counter())
|
||||
d[stage] += 1
|
||||
d["rows"] += 1
|
||||
if flag == "PASS":
|
||||
d["pass"] += 1
|
||||
stages[stage] += 1
|
||||
|
||||
print(f"rounds completed: {rounds}")
|
||||
print(f"\nVERIFIED WRITES: {proven}/{denom}"
|
||||
f" = {round(100*proven/denom) if denom else 0}% (criterion 2 wants >=95%)")
|
||||
print(f"FALSE SUCCESS CLAIMS: {liars} (criterion 3 hard gate, must be 0)")
|
||||
print("\nper site (a row is one full post+audit+delete+audit round trip):")
|
||||
print(f" {'site':<10}{'rows':>5}{'pass':>6} stages")
|
||||
for site, d in sorted(per_site.items()):
|
||||
st = {k: v for k, v in d.items() if k not in ("rows", "pass")}
|
||||
print(f" {site:<10}{d['rows']:>5}{d['pass']:>6} {st}")
|
||||
|
||||
stranded = []
|
||||
for m in STRANDED.finditer(text):
|
||||
stranded.extend(x.strip() for x in m.group(1).split(","))
|
||||
print(f"\nmarkers the canary could not clean up: {len(stranded)}")
|
||||
for s in stranded:
|
||||
print(f" {s}")
|
||||
if stranded:
|
||||
print(" ^ verify by hand with bench/rawbrowser.py before calling this run finished")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+24
@@ -0,0 +1,24 @@
|
||||
#!/bin/bash
|
||||
# Criteria 1, 4, 5, 6, 7 and 9 in one pass: the known suite at N=12 = 108 site-runs, which is the
|
||||
# >=100 sample criterion 7 asks for, while the same trials carry the reach, timing and skill data.
|
||||
#
|
||||
# Dry run only. The backend must be up with OSW_SENDSCRIPT_DRYRUN=1 (`stack.sh up dry`), so the
|
||||
# irreversible click is refused in the backend, not by a flag this script sets and does not own.
|
||||
# Where logs, profiles and run output go. Defaults to runs/ beside this harness; override with
|
||||
# OSW_BENCH_DIR to keep multi-gigabyte browser profiles off the repo disk.
|
||||
SP="${OSW_BENCH_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/runs}"
|
||||
mkdir -p "$SP"
|
||||
# The harness itself lives beside this script; SP is only for run OUTPUT.
|
||||
HARNESS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Repo root from this script's own location, so the harness works in any checkout.
|
||||
TREE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
cd "$TREE" || exit 1
|
||||
|
||||
N="${N:-12}"
|
||||
OUT="$SP/c7_r6.txt"
|
||||
: > "$OUT"
|
||||
echo "started $(date +%H:%M:%S), N=$N, log=$SP/r6_be.log" >> "$OUT"
|
||||
OSW_BASE=http://127.0.0.1:8326 OSW_LOG="$SP/r6_be.log" \
|
||||
./backend/.venv/bin/python "$HARNESS/bench.py" known "$N" >> "$OUT" 2>&1
|
||||
echo "########## DONE $(date +%H:%M:%S)" >> "$OUT"
|
||||
echo "backend restarts during the sweep: $(grep -c 'BACKEND RESTARTED' "$SP/r6_be.log")" >> "$OUT"
|
||||
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/bin/bash
|
||||
# Criterion 8: the frozen holdout, re-measured after the round's changes. Dry run, same stack.
|
||||
# Where logs, profiles and run output go. Defaults to runs/ beside this harness; override with
|
||||
# OSW_BENCH_DIR to keep multi-gigabyte browser profiles off the repo disk.
|
||||
SP="${OSW_BENCH_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/runs}"
|
||||
mkdir -p "$SP"
|
||||
# The harness itself lives beside this script; SP is only for run OUTPUT.
|
||||
HARNESS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# Repo root from this script's own location, so the harness works in any checkout.
|
||||
TREE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
cd "$TREE" || exit 1
|
||||
N="${N:-2}"
|
||||
OUT="$SP/c8_r6.txt"
|
||||
: > "$OUT"
|
||||
echo "started $(date +%H:%M:%S), N=$N" >> "$OUT"
|
||||
OSW_BASE=http://127.0.0.1:8326 OSW_LOG="$SP/r6_be.log" \
|
||||
./backend/.venv/bin/python "$HARNESS/bench.py" holdout "$N" >> "$OUT" 2>&1
|
||||
echo "########## DONE $(date +%H:%M:%S)" >> "$OUT"
|
||||
@@ -0,0 +1,289 @@
|
||||
"""Write-path coverage across sites with a GENUINE session, measured in dry run.
|
||||
|
||||
Reports the two halves separately, because they fail for different reasons and only one of them was
|
||||
ever instrumented: did we REACH a composer and commit a fill, and can we RESOLVE that composer's
|
||||
submit. The second half is new. Without it every sweep said "ready to send" about sends that would
|
||||
have blind-tapped a coordinate (reddit, live).
|
||||
|
||||
Never posts: the backend runs with OSW_SENDSCRIPT_DRYRUN=1, which stops at the irreversible click.
|
||||
"""
|
||||
import json, os, re, sys, time, urllib.request
|
||||
|
||||
# Repo root, derived from this file (e2e/browser-v3/x.py -> two levels up). Hardcoding an
|
||||
# absolute path made this harness silently useless on any other checkout.
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
BASE = os.environ.get("OSW_BASE", "http://127.0.0.1:8326") + "/api/agents"
|
||||
LOG = os.environ.get("OSW_LOG", "")
|
||||
MODEL = "opus-4-8"
|
||||
|
||||
# This list was captioned "only sites this profile is genuinely signed into" and that was not true:
|
||||
# gmail and substack hit a login wall on every single sweep, and their rows were scored as composer
|
||||
# misses, which quietly charged our code for an account state. Sign-in is re-derived per run now,
|
||||
# from the page's own evidence, and an unmeasurable site leaves the denominator instead of losing.
|
||||
TASKS = {
|
||||
"x": 'Go to x.com and post this tweet, exactly: "coverage probe alpha"',
|
||||
"linkedin": 'Go to linkedin.com and create a post with exactly this text: "coverage probe alpha"',
|
||||
"reddit": 'Go to reddit.com/r/test/submit and create a text post with body "coverage probe alpha"',
|
||||
"gmail": 'Go to gmail and start a new email with the body "coverage probe alpha"',
|
||||
"youtube": 'Go to youtube.com, open the first video, and write the comment "coverage probe alpha"',
|
||||
"instagram": 'Go to instagram.com and write a comment "coverage probe alpha" on the first post',
|
||||
"tiktok": 'Go to tiktok.com and write a comment "coverage probe alpha" on the first video',
|
||||
"substack": 'Go to substack.com and start a new note with the text "coverage probe alpha"',
|
||||
"twitch": 'Go to twitch.tv, open the first live channel, and write "coverage probe alpha" in chat',
|
||||
}
|
||||
|
||||
def req(method, url, body=None):
|
||||
tok = open(os.path.join(ROOT, "backend/data/auth.token")).read().strip()
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
r = urllib.request.Request(url, data=data, method=method,
|
||||
headers={"Content-Type": "application/json", "Authorization": "Bearer " + tok})
|
||||
with urllib.request.urlopen(r, timeout=300) as resp:
|
||||
return json.loads(resp.read().decode() or "{}")
|
||||
|
||||
def loglines():
|
||||
try: return sum(1 for _ in open(LOG, errors="ignore"))
|
||||
except OSError: return 0
|
||||
|
||||
BACKEND_DOWN = "[harness] BACKEND UNREACHABLE"
|
||||
|
||||
|
||||
def run(site, prompt, budget=200):
|
||||
mark = loglines()
|
||||
# A dead backend used to raise URLError straight out of here and kill the whole sweep mid-run,
|
||||
# losing every trial after it. It is also not a product failure and must never be graded as one,
|
||||
# so it gets a marker the classifier can bucket as infrastructure.
|
||||
try:
|
||||
dash = req("GET", BASE.replace("/agents","") + "/dashboards/list")
|
||||
ds = dash if isinstance(dash, list) else dash.get("dashboards", [])
|
||||
sid = req("POST", f"{BASE}/launch", {"mode":"agent","model":MODEL,"provider":"anthropic",
|
||||
"dashboard_id": ds[0]["id"], "name": f"cov-{site}"})["session"]["id"]
|
||||
except Exception as e:
|
||||
time.sleep(20) # give the supervisor a chance to bring it back before the next trial
|
||||
return f"{BACKEND_DOWN} ({type(e).__name__}: {str(e)[:80]})", 0.0
|
||||
t0 = time.time()
|
||||
try: req("POST", f"{BASE}/sessions/{sid}/message", {"prompt":prompt,"mode":"agent","model":MODEL})
|
||||
except Exception: pass
|
||||
while time.time() - t0 < budget:
|
||||
try:
|
||||
s = req("GET", f"{BASE}/sessions/{sid}")
|
||||
if str(s.get("status") or "") in ("completed","error","stopped"): break
|
||||
except Exception: pass
|
||||
time.sleep(2)
|
||||
time.sleep(1.5)
|
||||
try: sl = "".join(open(LOG, errors="ignore").readlines()[mark:])
|
||||
except OSError: sl = ""
|
||||
# Tear the trial down before the next one starts. Each run leaves a chat session and a live
|
||||
# browser card with its own webview; across 45 runs that is 45 webviews the renderer keeps
|
||||
# compositing, and a starved renderer stops answering, which reads as a product failure. It
|
||||
# already cost one whole sweep (24 of 45 rows came back "no dashboard" and scored a meaningless
|
||||
# 29%). delete_session stops the browser-agent children first, so this reaps the cards too.
|
||||
# Read the log slice BEFORE deleting, or the teardown's own lines land in the graded window.
|
||||
try: req("DELETE", f"{BASE}/sessions/{sid}")
|
||||
except Exception: pass
|
||||
return sl, round(time.time()-t0, 1)
|
||||
|
||||
DRY = re.compile(r"DRYRUN: WOULD send \(fill committed, send_button_listed=(\w+), "
|
||||
r"submit_resolved=(\w+), submit_rank=(\d+), submit=(.+?)\); not clicking")
|
||||
DECL = re.compile(r"\[browser-sendscript\] decline: (.+)")
|
||||
DISABLED = re.compile(r"submit (.+?) is present but DISABLED")
|
||||
FILL = re.compile(r"\[browser-sendscript\] fill target (.+?) \[(-?\d+)\] on (\S*)")
|
||||
# The composer was found AND focused and the text still would not go in. The old grader printed
|
||||
# "no send-script activity" for this, which reads as "nothing happened", and it hid the twitch and
|
||||
# instagram failures for as long as they existed. A dead fill is a code gap, and it gets a name.
|
||||
FILLERR = re.compile(r"fill errored \((.+?)\); handing")
|
||||
# Signed out, judged on the page's OWN evidence rather than on the product's label: a sign-in URL
|
||||
# or a visible password field. Grading this with the product's verdict would be grading the guard
|
||||
# with the guard, which is the mistake that made every earlier number worthless.
|
||||
WALL = re.compile(r"decline: login/auth wall \('([^']*)'\)(?: triggered by (url|password field))?")
|
||||
SIGNIN_URL = re.compile(r"(accounts\.google\.com|/login|/signin|/sign[-_]?in|/i/flow/login|/auth/)", re.I)
|
||||
# Matched on the page's own words, not on the product's verdict, same rule as everything else here.
|
||||
CAPTCHA = re.compile(
|
||||
r"enter (?:the )?(?:code|characters) you hear|play the audio|press (?:and|&) hold|"
|
||||
r"drag (?:the )?(?:slider|puzzle)|i'?m not a robot|verify you are (?:a )?human|"
|
||||
r"select all (?:images|squares) (?:with|containing)|recaptcha|hcaptcha|turnstile|"
|
||||
r"bot-detection challenge",
|
||||
re.I)
|
||||
SAYS_OUT = "decline: signed OUT"
|
||||
|
||||
# The page each task must end up on, written out by hand from what these sites actually use. This
|
||||
# is the half the harness was missing: it scored "a composer got filled" and never asked whose.
|
||||
# The instagram DM that started all this was filled on instagram.com/<someone>/, a PROFILE, while
|
||||
# the task said "the first post". The URL alone rejects it, and unlike a name check it still works
|
||||
# when the structural finder reports its box as a bare 'contenteditable'.
|
||||
ON_PAGE = {
|
||||
"x": r"x\.com/(home|compose)",
|
||||
"linkedin": r"linkedin\.com/(feed|posts)",
|
||||
"reddit": r"reddit\.com/r/[^/]+/submit",
|
||||
"gmail": r"mail\.google\.com/",
|
||||
"youtube": r"youtube\.com/watch",
|
||||
"instagram": r"instagram\.com/(p|reel)/",
|
||||
"tiktok": r"tiktok\.com/@[^/]+/video/",
|
||||
"substack": r"substack\.com/",
|
||||
"twitch": r"twitch\.tv/[^/?]+(/|\?|$)",
|
||||
}
|
||||
# Graded independently of the product's own surface_mismatch on purpose: grading the guard with the
|
||||
# guard proves nothing. These are the names a DM box carries on the sites above.
|
||||
DM_NAME = re.compile(r"\b(message|messages|dm)\b", re.I)
|
||||
PUBLIC = ("x", "linkedin", "reddit", "youtube", "instagram", "tiktok", "substack")
|
||||
|
||||
# A dry run can never actually send, so the completion gate correctly calls EVERY run a ghost
|
||||
# ("declared done but the send was not confirmed") and the fast path fires one recovery dispatch.
|
||||
# That recovery reuses the same card, wherever it has drifted to, and fills whatever it finds. Those
|
||||
# fills are an artifact of measuring in dry run and say nothing about coverage. Scoring them made
|
||||
# reddit and youtube look systematically broken (3/3 and 2/2 "wrong page") when both had already hit
|
||||
# the right composer seconds earlier: reddit on /r/test/submit, youtube on /watch.
|
||||
RECOVERY = "one recovery dispatch"
|
||||
|
||||
def gradeable(slice_):
|
||||
"""The part of the run that happened before dry-run recovery muddied it."""
|
||||
cut = slice_.find(RECOVERY)
|
||||
return slice_[:cut] if cut > 0 else slice_
|
||||
|
||||
def last(rx, slice_):
|
||||
"""A run can legitimately fill more than once (reddit tries /r/test/submit, then bare /submit),
|
||||
so the LAST attempt inside the gradeable window owns the verdict; taking the first scored a page
|
||||
the run had already left."""
|
||||
ms = list(rx.finditer(gradeable(slice_)))
|
||||
return ms[-1] if ms else None
|
||||
|
||||
def surface(site, slice_):
|
||||
"""(ok, description) for the box that actually got filled. ok=False means we reached SOMETHING,
|
||||
which the old harness counted as a win, and it was the wrong thing."""
|
||||
f = last(FILL, slice_)
|
||||
if not f:
|
||||
return None, ""
|
||||
name, url = f.group(1), f.group(3)
|
||||
if site in PUBLIC and DM_NAME.search(name):
|
||||
return False, f"DM box {name} on {url[:44]}"
|
||||
want = ON_PAGE.get(site)
|
||||
if want and not re.search(want, url):
|
||||
return False, f"wrong page {url[:52]}"
|
||||
return True, f"{name} on {url[:40]}"
|
||||
|
||||
# A run whose LLM lane was flapping underneath it measured the machine, not the write path. Every
|
||||
# one of these was live in the 2026-08-01 sweep, which scored 1/9 while nothing about the code had
|
||||
# changed: eight other OpenSwarm backends were on the box, each one's dev logic killing the shared
|
||||
# 9router and starting its own on :20128. Wall times went 30s -> 202s and prestage got skipped.
|
||||
# "prestage] skipped" used to be in this list bare, and that was too wide: prestage skips for CODE
|
||||
# reasons too, and a live run on twitch died with "'utf-8' codec can't encode '\ud83e'" (half an
|
||||
# emoji) which this harness then filed as an environment problem and told me to re-run. A real bug
|
||||
# wearing an INVALID badge is worse than no harness. Only provider-shaped reasons count as sick.
|
||||
UNHEALTHY = ("9Router watchdog", "9Router process died", "No AI provider connected",
|
||||
"prestage] skipped (No AI provider", "prestage] skipped (classifier",
|
||||
"dispatch refused: no dashboard")
|
||||
|
||||
def row(**kw):
|
||||
base = {"composer": False, "reached": False, "submit": False, "rank": 0,
|
||||
"invalid": False, "unmeasurable": "", "detail": ""}
|
||||
base.update(kw)
|
||||
return base
|
||||
|
||||
def unmeasurable(slice_):
|
||||
"""Why this run could not measure our code at all, or '' if it could.
|
||||
|
||||
Being signed out is an account state, not a coverage failure, and charging our code for it is
|
||||
how gmail and substack sat in the miss column across five sweeps. Read the page's own evidence
|
||||
(a sign-in URL, a visible password field), and when the only thing available is the agent's own
|
||||
say-so, return it labelled UNVERIFIED so it can never be quietly banked as a clean excuse.
|
||||
"""
|
||||
# A bot-detection challenge is not a coverage failure. Solving one is off-limits, so a page
|
||||
# behind a captcha is a page our code is not allowed to reach, and scoring it against reach
|
||||
# charges us for a rule we are choosing to keep. Eric caught this on tiktok: an audio captcha
|
||||
# over a logged-out feed, while the row read "no composer" as though the finder had missed.
|
||||
cap = CAPTCHA.search(slice_)
|
||||
if cap:
|
||||
return f"bot-detection challenge, PROVEN ({cap.group(0)[:44]})"
|
||||
w = WALL.search(slice_)
|
||||
if w:
|
||||
url, trigger = w.group(1), w.group(2) or ""
|
||||
if trigger == "password field":
|
||||
return f"signed out, PROVEN (password field on {url[:38]})"
|
||||
if SIGNIN_URL.search(url):
|
||||
return f"signed out, PROVEN (sign-in URL {url[:42]})"
|
||||
return f"login wall claimed, UNVERIFIED (no evidence for {url[:34]})"
|
||||
if SAYS_OUT in slice_:
|
||||
return "signed out per the agent, UNVERIFIED (no wall evidence)"
|
||||
return ""
|
||||
|
||||
def verdict(site, slice_):
|
||||
sick = [s for s in UNHEALTHY if s in slice_]
|
||||
if sick:
|
||||
return row(invalid=True, detail=f"INVALID: stack was unhealthy ({sick[0]})")
|
||||
ok, where = surface(site, slice_)
|
||||
reached = ok is not None
|
||||
# The harness prompts a PARENT agent, which rewrites the task for the browser sub-agent. When
|
||||
# that rewrite happens to quote two different strings the send script honestly declines, and
|
||||
# that is the harness's phrasing failing, not this site's coverage. Scoring it as a miss put
|
||||
# x at "n" on a run where x was fine 4 minutes earlier. Call it INVALID and re-run the site.
|
||||
if not reached and "no unambiguous quoted payload" in slice_:
|
||||
return row(invalid=True, detail="INVALID: parent rephrased the task, payload came through ambiguous")
|
||||
why = unmeasurable(slice_)
|
||||
if why and not reached:
|
||||
return row(unmeasurable=why, detail=f"NOT MEASURABLE: {why}")
|
||||
if ok is False:
|
||||
return row(reached=True, detail=f"WRONG SURFACE: {where}")
|
||||
m = last(DRY, slice_)
|
||||
d = DISABLED.search(slice_)
|
||||
if m or d:
|
||||
submit = bool(m) and m.group(2) == "True"
|
||||
rank = int(m.group(3)) if m else 0
|
||||
detail = (m.group(4)[:34] if m else f"DISABLED {d.group(1)[:24]}")
|
||||
return row(composer=True, reached=reached, submit=submit, rank=rank,
|
||||
detail=f"{detail} <- {where}")
|
||||
# Reached the right box, focused it, and the text would not go in. Its own column: this is the
|
||||
# one failure mode a better composer finder cannot fix, and it was invisible until now.
|
||||
fe = last(FILLERR, slice_)
|
||||
if fe:
|
||||
return row(reached=True, detail=f"FILL DIED on {where}: {fe.group(1)[:46]}")
|
||||
c = DECL.findall(slice_)
|
||||
return row(reached=reached,
|
||||
detail=("decline: " + c[-1][:46]) if c else "no send-script activity")
|
||||
|
||||
def preflight():
|
||||
"""Refuse to measure on a box that cannot hold still. One 9router serves :20128 and every dev
|
||||
backend's startup kills the one it does not own, so a second stack anywhere on the machine turns
|
||||
this sweep into a coin flip. Better to print why than to publish a number that means nothing."""
|
||||
import subprocess
|
||||
# Count the real interpreters only. `pgrep -f "uvicorn backend.main"` also matches the
|
||||
# supervising shell (its command line quotes the whole uvicorn line), so with a supervisor in
|
||||
# place this refused to run every single time, on a box holding exactly one backend.
|
||||
ps = subprocess.run(["ps", "-Ao", "command"], capture_output=True, text=True).stdout
|
||||
n = sum(1 for ln in ps.splitlines()
|
||||
if "-m uvicorn backend.main" in ln and "/bin/python" in ln)
|
||||
router = urllib.request.urlopen("http://127.0.0.1:20128/v1/models", timeout=5).status
|
||||
if n > 1 or router != 200:
|
||||
print(f"REFUSING: {n} backends on this box (want 1), 9router HTTP {router} (want 200).")
|
||||
print("Another session's stack is fighting yours for the router. Wait it out.")
|
||||
sys.exit(2)
|
||||
|
||||
if __name__ == "__main__":
|
||||
if os.environ.get("OSW_SKIP_PREFLIGHT") != "1":
|
||||
preflight()
|
||||
only = sys.argv[1:] or list(TASKS)
|
||||
rows = []
|
||||
for site in only:
|
||||
if site not in TASKS: continue
|
||||
sl, wall = run(site, TASKS[site])
|
||||
v = verdict(site, sl); v["site"] = site; v["wall"] = wall
|
||||
rows.append(v)
|
||||
print(f" {site:11s} composer={'Y' if v['composer'] else 'n'} "
|
||||
f"submit={'Y' if v['submit'] else 'n'} rank={v['rank']} {wall:6.1f}s {v['detail']}", flush=True)
|
||||
invalid = [r for r in rows if r["invalid"]]
|
||||
unmeas = [r for r in rows if r["unmeasurable"] and not r["invalid"]]
|
||||
graded = [r for r in rows if not r["invalid"] and not r["unmeasurable"]]
|
||||
n = len(graded)
|
||||
wrong = sum("WRONG SURFACE" in r["detail"] for r in graded)
|
||||
died = sum("FILL DIED" in r["detail"] for r in graded)
|
||||
print(f"\n=== RIGHT composer {sum(r['composer'] for r in graded)}/{n} measurable | "
|
||||
f"submit resolved {sum(r['submit'] for r in graded)}/{n} | "
|
||||
f"WRONG surface {wrong} | fill DIED {died} ===")
|
||||
# Printed, never folded in. An excluded row is a claim that our code was not on trial, and that
|
||||
# claim has to survive being read out loud, especially the ones resting on the agent's own word.
|
||||
for label, group in (("NOT MEASURABLE", unmeas), ("INVALID, re-run", invalid)):
|
||||
for r in group:
|
||||
print(f" {label}: {r['site']} ({r['unmeasurable'] or r['detail']})")
|
||||
shaky = [r["site"] for r in unmeas if "UNVERIFIED" in r["unmeasurable"]]
|
||||
if shaky:
|
||||
print(f" ^ {', '.join(shaky)} rest on the agent's own word. Confirm by hand before quoting "
|
||||
f"any number that leaves them out.")
|
||||
json.dump(rows, open(os.environ.get("OSW_OUT","/tmp/cov.json"), "w"), indent=1)
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/bin/bash
|
||||
# Keep an Electron renderer alive for the length of a sweep.
|
||||
#
|
||||
# A 45-run sweep takes over an hour and the app does not reliably survive it: one run quit partway
|
||||
# through and 24 of 45 rows came back "dispatch refused: no dashboard", which scored as a 29% reach
|
||||
# that measured nothing but my own dead window. Relaunching by hand between runs is not a fix, it
|
||||
# just moves the gap to whenever I am not looking.
|
||||
#
|
||||
# Restarts on exit, and waits for webpack first: Electron launched against a dead :3026 loads a
|
||||
# blank page and quits immediately, which is how the window went missing the first time.
|
||||
# Where logs, profiles and run output go. Defaults to runs/ beside this harness; override with
|
||||
# OSW_BENCH_DIR to keep multi-gigabyte browser profiles off the repo disk.
|
||||
SP="${OSW_BENCH_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/runs}"
|
||||
mkdir -p "$SP"
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/electron" || exit 1
|
||||
|
||||
while true; do
|
||||
until curl -s -o /dev/null --max-time 4 "http://localhost:${OPENSWARM_DEV_PORT:-3026}"; do sleep 5; done
|
||||
ELECTRON_DEV=1 OPENSWARM_DEV_PORT="${OPENSWARM_DEV_PORT:-3026}" OPENSWARM_PORT="${OPENSWARM_PORT:-8326}" \
|
||||
./node_modules/.bin/electron . --user-data-dir="$SP/udd" >> "$SP/renderer.log" 2>&1
|
||||
echo "[keep_renderer] electron exited at $(date +%H:%M:%S), restarting" >> "$SP/renderer.log"
|
||||
sleep 5
|
||||
done
|
||||
@@ -0,0 +1,110 @@
|
||||
"""Where does a browser run's page text actually SURFACE? Answer it before trusting any audit.
|
||||
|
||||
The canary proves a write landed by looking for the marker string in evidence the model did not
|
||||
author. It has been looking in the backend LOG, and the backend never logs page text: a grep for
|
||||
every canary marker ever generated across every backend log on this box returns nothing. So the
|
||||
audit could only ever return "could not look", which is exactly what LinkedIn and reddit scored.
|
||||
|
||||
This probe runs one read-only task and dumps every place the marker could live (parent session
|
||||
messages, child sessions, tool results), so the replacement evidence channel is chosen on evidence
|
||||
rather than on where I assume the text goes.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
# Repo root, derived from this file (e2e/browser-v3/x.py -> two levels up). Hardcoding an
|
||||
# absolute path made this harness silently useless on any other checkout.
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
BASE = os.environ.get("OSW_BASE", "http://127.0.0.1:8326") + "/api/agents"
|
||||
MODEL = "opus-4-8"
|
||||
|
||||
|
||||
def req(method: str, url: str, body=None):
|
||||
tok = open(os.path.join(ROOT, "backend/data/auth.token")).read().strip()
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
r = urllib.request.Request(url, data=data, method=method,
|
||||
headers={"Content-Type": "application/json",
|
||||
"Authorization": "Bearer " + tok})
|
||||
with urllib.request.urlopen(r, timeout=260) as resp:
|
||||
return json.loads(resp.read().decode() or "{}")
|
||||
|
||||
|
||||
def walk(node, path="", out=None, needle=""):
|
||||
"""Every JSON path whose leaf string contains `needle`."""
|
||||
out = [] if out is None else out
|
||||
if isinstance(node, dict):
|
||||
for k, v in node.items():
|
||||
walk(v, f"{path}.{k}", out, needle)
|
||||
elif isinstance(node, list):
|
||||
for i, v in enumerate(node):
|
||||
walk(v, f"{path}[{i}]", out, needle)
|
||||
elif isinstance(node, str) and needle and needle in node:
|
||||
out.append((path, node[:160]))
|
||||
return out
|
||||
|
||||
|
||||
def main() -> int:
|
||||
task = sys.argv[1]
|
||||
needle = sys.argv[2]
|
||||
dash = req("GET", BASE.replace("/agents", "") + "/dashboards/list")
|
||||
ds = dash if isinstance(dash, list) else dash.get("dashboards", [])
|
||||
sid = req("POST", f"{BASE}/launch", {"mode": "agent", "model": MODEL, "provider": "anthropic",
|
||||
"dashboard_id": ds[0]["id"], "name": "probe-evidence"})["session"]["id"]
|
||||
print(f"session {sid}\ntask {task}\nneedle {needle}\n", flush=True)
|
||||
t0 = time.time()
|
||||
try:
|
||||
req("POST", f"{BASE}/sessions/{sid}/message", {"prompt": task, "mode": "agent", "model": MODEL})
|
||||
except Exception:
|
||||
pass
|
||||
status = ""
|
||||
while time.time() - t0 < 240:
|
||||
try:
|
||||
s = req("GET", f"{BASE}/sessions/{sid}")
|
||||
status = str(s.get("status") or "")
|
||||
if status in ("completed", "error", "stopped"):
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(2)
|
||||
time.sleep(2)
|
||||
|
||||
s = req("GET", f"{BASE}/sessions/{sid}")
|
||||
print(f"status={status} wall={round(time.time()-t0,1)}s")
|
||||
print(f"parent top-level keys: {sorted(s.keys())}")
|
||||
print(f"parent messages: {len(s.get('messages') or [])}")
|
||||
hits = walk(s, "session", needle=needle)
|
||||
print(f"\n--- marker in PARENT session json: {len(hits)} hits ---")
|
||||
for p, v in hits[:14]:
|
||||
print(f" {p}\n {v!r}")
|
||||
|
||||
# Child sessions: the sub-agent path puts the browser work in its own session.
|
||||
kids = []
|
||||
try:
|
||||
allsess = req("GET", f"{BASE}/sessions")
|
||||
rows = allsess if isinstance(allsess, list) else allsess.get("sessions", [])
|
||||
kids = [r for r in rows if str(r.get("parent_session_id") or "") == sid]
|
||||
except Exception as e:
|
||||
print(f"(child enumeration failed: {e})")
|
||||
print(f"\n--- child sessions: {len(kids)} ---")
|
||||
for k in kids:
|
||||
full = req("GET", f"{BASE}/sessions/{k['id']}")
|
||||
kh = walk(full, "child", needle=needle)
|
||||
print(f" {k['id']} name={k.get('name')!r} msgs={len(full.get('messages') or [])} marker_hits={len(kh)}")
|
||||
for p, v in kh[:8]:
|
||||
print(f" {p}\n {v!r}")
|
||||
|
||||
json.dump(s, open(os.environ.get("OSW_DUMP", "/tmp/probe_session.json"), "w"), indent=1)
|
||||
print(f"\nfull parent dump -> {os.environ.get('OSW_DUMP', '/tmp/probe_session.json')}")
|
||||
try:
|
||||
req("DELETE", f"{BASE}/sessions/{sid}")
|
||||
except Exception:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,98 @@
|
||||
"""Drive a browser card directly, with no model anywhere in the loop.
|
||||
|
||||
The audit channel the canary now uses, exposed as a CLI so I can check a real account by hand
|
||||
without hand-escaping JS through three layers of shell quoting (which silently sent a mangled
|
||||
expression once and made a working navigate look like a domain-gate refusal).
|
||||
|
||||
rawbrowser.py cards
|
||||
rawbrowser.py read <url> [substring]
|
||||
rawbrowser.py eval <url> <js>
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# Repo root, derived from this file (e2e/browser-v3/x.py -> two levels up). Hardcoding an
|
||||
# absolute path made this harness silently useless on any other checkout.
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
API = os.environ.get("OSW_BASE", "http://127.0.0.1:8326") + "/api"
|
||||
|
||||
|
||||
def req(method, url, body=None):
|
||||
tok = open(os.path.join(ROOT, "backend/data/auth.token")).read().strip()
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
r = urllib.request.Request(url, data=data, method=method,
|
||||
headers={"Content-Type": "application/json",
|
||||
"Authorization": "Bearer " + tok})
|
||||
with urllib.request.urlopen(r, timeout=180) as resp:
|
||||
return json.loads(resp.read().decode() or "{}")
|
||||
|
||||
|
||||
def cards():
|
||||
ds = req("GET", f"{API}/dashboards/list")
|
||||
ds = ds if isinstance(ds, list) else ds.get("dashboards", [])
|
||||
if not ds:
|
||||
return []
|
||||
full = req("GET", f"{API}/dashboards/{ds[0]['id']}")
|
||||
return list(((full.get("layout") or {}).get("browser_cards") or {}))
|
||||
|
||||
|
||||
def cmd(bid, action, params):
|
||||
return req("POST", f"{API}/browser/command",
|
||||
{"action": action, "browser_id": bid, "params": params})
|
||||
|
||||
|
||||
BODY = ('(()=>{try{return {body:(document.body&&document.body.innerText)||"",'
|
||||
'u:location.href};}catch(e){return {body:"",u:""};}})()')
|
||||
|
||||
|
||||
def main():
|
||||
what = sys.argv[1] if len(sys.argv) > 1 else "cards"
|
||||
cs = cards()
|
||||
if what == "cards":
|
||||
print("\n".join(cs) or "(none)")
|
||||
return 0
|
||||
if not cs:
|
||||
print("no browser card on the dashboard; run a browser task first")
|
||||
return 2
|
||||
url = sys.argv[2]
|
||||
want = (urlparse(url).hostname or "").lower().replace("www.", "")
|
||||
# A card whose session was torn down mid-run WEDGES: it answers navigate with "Navigated to
|
||||
# <url>" in ~80ms and stays where it was. So never trust the reply, check where we landed, and
|
||||
# move to another card if this one did not go.
|
||||
bid = ""
|
||||
for cand in ([os.environ["OSW_CARD"]] if os.environ.get("OSW_CARD") else list(reversed(cs))):
|
||||
cmd(cand, "navigate", {"url": url})
|
||||
time.sleep(float(os.environ.get("OSW_SETTLE", "5")))
|
||||
got = cmd(cand, "evaluate", {"expression": BODY})
|
||||
try:
|
||||
landed = json.loads(got.get("text") or "{}").get("u") or ""
|
||||
except ValueError:
|
||||
landed = ""
|
||||
if (urlparse(landed).hostname or "").lower().replace("www.", "") == want:
|
||||
bid = cand
|
||||
break
|
||||
print(f"[card {cand} wedged, landed on {landed[:60]!r}]", file=sys.stderr)
|
||||
if not bid:
|
||||
print(f"no card could reach {url}", file=sys.stderr)
|
||||
return 3
|
||||
if what == "eval":
|
||||
print(json.dumps(cmd(bid, "evaluate", {"expression": sys.argv[3]}), indent=1))
|
||||
return 0
|
||||
r = cmd(bid, "evaluate", {"expression": BODY})
|
||||
v = json.loads(r.get("text") or "{}")
|
||||
body = str(v.get("body") or "")
|
||||
print(f"[url] {v.get('u')}\n[chars] {len(body)}")
|
||||
if len(sys.argv) > 3:
|
||||
needle = sys.argv[3]
|
||||
print(f"[{needle!r} present] {needle in body}")
|
||||
print(body[:3000])
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,20 @@
|
||||
{"site": "rentry", "trial": 1, "bucket": "ok", "wall_s": 38.7, "detail": "'go' <- 'contenteditable' on https://rentry.co/", "composer": true, "submit": true, "total_ms": 5166, "llm_ms": 0, "tools_ms": 2072, "other_ms": 3094, "turns": 1, "artifact": "results/rentry_1.log"}
|
||||
{"site": "dpaste", "trial": 1, "bucket": "infra_browser", "wall_s": 71.2, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 35756, "llm_ms": 0, "tools_ms": 31997, "other_ms": 3759, "turns": 1, "artifact": "results/dpaste_1.log"}
|
||||
{"site": "controlc", "trial": 1, "bucket": "ok", "wall_s": 34.7, "detail": "'create paste' <- 'textarea' on https://controlc.com/", "composer": true, "submit": true, "total_ms": 6413, "llm_ms": 0, "tools_ms": 3344, "other_ms": 3069, "turns": 1, "artifact": "results/controlc_1.log"}
|
||||
{"site": "txti", "trial": 1, "bucket": "product_no_composer", "wall_s": 31.7, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 5828, "llm_ms": 0, "tools_ms": 2924, "other_ms": 2904, "turns": 1, "artifact": "results/txti_1.log"}
|
||||
{"site": "justpaste", "trial": 1, "bucket": "ok", "wall_s": 31.9, "detail": "'publish' <- 'input' on https://justpaste.it/", "composer": true, "submit": true, "total_ms": 6524, "llm_ms": 0, "tools_ms": 3456, "other_ms": 3068, "turns": 1, "artifact": "results/justpaste_1.log"}
|
||||
{"site": "telegraph", "trial": 1, "bucket": "ok", "wall_s": 29.8, "detail": "'publish' <- 'contenteditable' on https://telegra.ph/", "composer": true, "submit": true, "total_ms": 3812, "llm_ms": 0, "tools_ms": 1022, "other_ms": 2790, "turns": 1, "artifact": "results/telegraph_1.log"}
|
||||
{"site": "quill", "trial": 1, "bucket": "ok", "wall_s": 33.3, "detail": "'share your edits' <- 'contenteditable' on https://quilljs.com/playground/snow", "composer": true, "submit": true, "total_ms": 5320, "llm_ms": 0, "tools_ms": 2286, "other_ms": 3034, "turns": 1, "artifact": "results/quill_1.log"}
|
||||
{"site": "tinymce", "trial": 1, "bucket": "ok", "wall_s": 34.0, "detail": "'send' <- 'textarea' on https://www.tiny.cloud/", "composer": true, "submit": true, "total_ms": 8948, "llm_ms": 0, "tools_ms": 5838, "other_ms": 3110, "turns": 1, "artifact": "results/tinymce_1.log"}
|
||||
{"site": "ckeditor", "trial": 1, "bucket": "ok", "wall_s": 32.8, "detail": "'no submit control in the composer <- 'Rich Text Editor. Editing area: main. Press \u23250 for help.' on https://ckeditor.com/ckeditor-5/demo/fea", "composer": true, "submit": false, "total_ms": 2478, "llm_ms": 0, "tools_ms": 2024, "other_ms": 454, "turns": 1, "artifact": "results/ckeditor_1.log"}
|
||||
{"site": "codemirror", "trial": 1, "bucket": "not_measurable", "wall_s": 55.7, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 16007, "llm_ms": 0, "tools_ms": 10349, "other_ms": 5658, "turns": 1, "artifact": "results/codemirror_1.log"}
|
||||
{"site": "rentry", "trial": 2, "bucket": "ok", "wall_s": 28.4, "detail": "'go' <- 'contenteditable' on https://rentry.co/", "composer": true, "submit": true, "total_ms": 3808, "llm_ms": 0, "tools_ms": 964, "other_ms": 2844, "turns": 1, "artifact": "results/rentry_2.log"}
|
||||
{"site": "dpaste", "trial": 2, "bucket": "infra_browser", "wall_s": 68.0, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 37815, "llm_ms": 0, "tools_ms": 34733, "other_ms": 3082, "turns": 1, "artifact": "results/dpaste_2.log"}
|
||||
{"site": "controlc", "trial": 2, "bucket": "infra_router", "wall_s": 28.7, "detail": "INVALID: stack was unhealthy (No AI provider connected)", "composer": false, "submit": false, "total_ms": 7074, "llm_ms": 0, "tools_ms": 3392, "other_ms": 3682, "turns": 1, "artifact": "results/controlc_2.log"}
|
||||
{"site": "txti", "trial": 2, "bucket": "product_no_composer", "wall_s": 32.3, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 5158, "llm_ms": 0, "tools_ms": 2298, "other_ms": 2860, "turns": 1, "artifact": "results/txti_2.log"}
|
||||
{"site": "justpaste", "trial": 2, "bucket": "ok", "wall_s": 25.5, "detail": "'publish' <- 'input' on https://justpaste.it/", "composer": true, "submit": true, "total_ms": 5041, "llm_ms": 0, "tools_ms": 2148, "other_ms": 2893, "turns": 1, "artifact": "results/justpaste_2.log"}
|
||||
{"site": "telegraph", "trial": 2, "bucket": "ok", "wall_s": 30.3, "detail": "'publish' <- 'contenteditable' on https://telegra.ph/", "composer": true, "submit": true, "total_ms": 4126, "llm_ms": 0, "tools_ms": 1313, "other_ms": 2813, "turns": 1, "artifact": "results/telegraph_2.log"}
|
||||
{"site": "quill", "trial": 2, "bucket": "infra_router", "wall_s": 24.9, "detail": "INVALID: stack was unhealthy (No AI provider connected)", "composer": false, "submit": false, "total_ms": 4778, "llm_ms": 0, "tools_ms": 1923, "other_ms": 2855, "turns": 1, "artifact": "results/quill_2.log"}
|
||||
{"site": "tinymce", "trial": 2, "bucket": "ok", "wall_s": 64.7, "detail": "'send' <- 'textarea' on https://www.tiny.cloud/", "composer": true, "submit": true, "total_ms": 7947, "llm_ms": 0, "tools_ms": 4528, "other_ms": 3419, "turns": 1, "artifact": "results/tinymce_2.log"}
|
||||
{"site": "ckeditor", "trial": 2, "bucket": "ok", "wall_s": 20.4, "detail": "'no submit control in the composer <- 'Rich Text Editor. Editing area: main. Press \u23250 for help.' on https://ckeditor.com/ckeditor-5/demo/fea", "composer": true, "submit": false, "total_ms": 1746, "llm_ms": 0, "tools_ms": 1457, "other_ms": 289, "turns": 1, "artifact": "results/ckeditor_2.log"}
|
||||
{"site": "codemirror", "trial": 2, "bucket": "ok", "wall_s": 38.0, "detail": "'no submit control in the composer <- 'textarea' on https://codepen.io/pen", "composer": true, "submit": false, "total_ms": 4657, "llm_ms": 0, "tools_ms": 1817, "other_ms": 2840, "turns": 1, "artifact": "results/codemirror_2.log"}
|
||||
@@ -0,0 +1,158 @@
|
||||
{"site": "x", "trial": 1, "bucket": "ok", "wall_s": 11.7, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 726, "llm_ms": 0, "tools_ms": 680, "other_ms": 46, "turns": 1, "artifact": "results/x_1.log"}
|
||||
{"site": "linkedin", "trial": 1, "bucket": "ok", "wall_s": 20.5, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 726, "llm_ms": 0, "tools_ms": 680, "other_ms": 46, "turns": 1, "artifact": "results/linkedin_1.log"}
|
||||
{"site": "reddit", "trial": 1, "bucket": "ok", "wall_s": 7.7, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 333, "llm_ms": 0, "tools_ms": 313, "other_ms": 20, "turns": 1, "artifact": "results/reddit_1.log"}
|
||||
{"site": "gmail", "trial": 1, "bucket": "not_measurable", "wall_s": 11.4, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 125, "llm_ms": 0, "tools_ms": 107, "other_ms": 18, "turns": 1, "artifact": "results/gmail_1.log"}
|
||||
{"site": "youtube", "trial": 1, "bucket": "ok", "wall_s": 31.5, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=zdqCyJcT", "composer": true, "submit": true, "total_ms": 10519, "llm_ms": 0, "tools_ms": 7903, "other_ms": 2616, "turns": 1, "artifact": "results/youtube_1.log"}
|
||||
{"site": "instagram", "trial": 1, "bucket": "ok", "wall_s": 14.4, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbphiQPlmjFa", "composer": true, "submit": true, "total_ms": 3005, "llm_ms": 0, "tools_ms": 391, "other_ms": 2614, "turns": 1, "artifact": "results/instagram_1.log"}
|
||||
{"site": "tiktok", "trial": 1, "bucket": "not_measurable", "wall_s": 34.8, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/)", "composer": false, "submit": false, "total_ms": 10336, "llm_ms": 0, "tools_ms": 3515, "other_ms": 6821, "turns": 1, "artifact": "results/tiktok_1.log"}
|
||||
{"site": "substack", "trial": 1, "bucket": "not_measurable", "wall_s": 13.5, "detail": "NOT MEASURABLE: signed out, PROVEN (password field on https://substack.com/)", "composer": false, "submit": false, "total_ms": 202, "llm_ms": 0, "tools_ms": 193, "other_ms": 9, "turns": 1, "artifact": "results/substack_1.log"}
|
||||
{"site": "twitch", "trial": 1, "bucket": "ok", "wall_s": 12.2, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/cinna", "composer": true, "submit": true, "total_ms": 560, "llm_ms": 0, "tools_ms": 533, "other_ms": 27, "turns": 1, "artifact": "results/twitch_1.log"}
|
||||
{"site": "x", "trial": 2, "bucket": "ok", "wall_s": 10.6, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 490, "llm_ms": 0, "tools_ms": 470, "other_ms": 20, "turns": 1, "artifact": "results/x_2.log"}
|
||||
{"site": "linkedin", "trial": 2, "bucket": "ok", "wall_s": 20.6, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 387, "llm_ms": 0, "tools_ms": 352, "other_ms": 35, "turns": 1, "artifact": "results/linkedin_2.log"}
|
||||
{"site": "reddit", "trial": 2, "bucket": "ok", "wall_s": 8.0, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 327, "llm_ms": 0, "tools_ms": 319, "other_ms": 8, "turns": 1, "artifact": "results/reddit_2.log"}
|
||||
{"site": "gmail", "trial": 2, "bucket": "not_measurable", "wall_s": 11.6, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 384, "llm_ms": 0, "tools_ms": 376, "other_ms": 8, "turns": 1, "artifact": "results/gmail_2.log"}
|
||||
{"site": "youtube", "trial": 2, "bucket": "ok", "wall_s": 31.5, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=nlc41gME", "composer": true, "submit": true, "total_ms": 10192, "llm_ms": 0, "tools_ms": 7579, "other_ms": 2613, "turns": 1, "artifact": "results/youtube_2.log"}
|
||||
{"site": "instagram", "trial": 2, "bucket": "ok", "wall_s": 10.3, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbphiQPlmjFa", "composer": true, "submit": true, "total_ms": 2975, "llm_ms": 0, "tools_ms": 360, "other_ms": 2615, "turns": 1, "artifact": "results/instagram_2.log"}
|
||||
{"site": "tiktok", "trial": 2, "bucket": "product_no_composer", "wall_s": 47.6, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 12896, "llm_ms": 0, "tools_ms": 7273, "other_ms": 5623, "turns": 1, "artifact": "results/tiktok_2.log"}
|
||||
{"site": "substack", "trial": 2, "bucket": "not_measurable", "wall_s": 15.6, "detail": "NOT MEASURABLE: signed out, PROVEN (password field on https://substack.com/)", "composer": false, "submit": false, "total_ms": 210, "llm_ms": 0, "tools_ms": 201, "other_ms": 9, "turns": 1, "artifact": "results/substack_2.log"}
|
||||
{"site": "twitch", "trial": 2, "bucket": "ok", "wall_s": 11.6, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/cinna", "composer": true, "submit": true, "total_ms": 809, "llm_ms": 0, "tools_ms": 795, "other_ms": 14, "turns": 1, "artifact": "results/twitch_2.log"}
|
||||
{"site": "x", "trial": 3, "bucket": "ok", "wall_s": 10.3, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 562, "llm_ms": 0, "tools_ms": 553, "other_ms": 9, "turns": 1, "artifact": "results/x_3.log"}
|
||||
{"site": "linkedin", "trial": 3, "bucket": "ok", "wall_s": 18.5, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 361, "llm_ms": 0, "tools_ms": 351, "other_ms": 10, "turns": 1, "artifact": "results/linkedin_3.log"}
|
||||
{"site": "reddit", "trial": 3, "bucket": "ok", "wall_s": 7.6, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 361, "llm_ms": 0, "tools_ms": 351, "other_ms": 10, "turns": 1, "artifact": "results/reddit_3.log"}
|
||||
{"site": "gmail", "trial": 3, "bucket": "not_measurable", "wall_s": 11.6, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 120, "llm_ms": 0, "tools_ms": 113, "other_ms": 7, "turns": 1, "artifact": "results/gmail_3.log"}
|
||||
{"site": "youtube", "trial": 3, "bucket": "infra_router", "wall_s": 28.9, "detail": "INVALID: stack was unhealthy (No AI provider connected)", "composer": false, "submit": false, "total_ms": 11732, "llm_ms": 0, "tools_ms": 9073, "other_ms": 2659, "turns": 1, "artifact": "results/youtube_3.log"}
|
||||
{"site": "instagram", "trial": 3, "bucket": "infra_harness", "wall_s": 80.2, "detail": "INVALID: parent rephrased the task, payload came through ambiguous", "composer": false, "submit": false, "total_ms": 3309, "llm_ms": 0, "tools_ms": 920, "other_ms": 2389, "turns": 1, "artifact": "results/instagram_3.log"}
|
||||
{"site": "tiktok", "trial": 3, "bucket": "not_measurable", "wall_s": 54.3, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 13522, "llm_ms": 0, "tools_ms": 7841, "other_ms": 5681, "turns": 1, "artifact": "results/tiktok_3.log"}
|
||||
{"site": "substack", "trial": 3, "bucket": "not_measurable", "wall_s": 16.0, "detail": "NOT MEASURABLE: signed out, PROVEN (password field on https://substack.com/)", "composer": false, "submit": false, "total_ms": 305, "llm_ms": 0, "tools_ms": 286, "other_ms": 19, "turns": 1, "artifact": "results/substack_3.log"}
|
||||
{"site": "twitch", "trial": 3, "bucket": "ok", "wall_s": 9.9, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/cinna", "composer": true, "submit": true, "total_ms": 399, "llm_ms": 0, "tools_ms": 384, "other_ms": 15, "turns": 1, "artifact": "results/twitch_3.log"}
|
||||
{"site": "x", "trial": 4, "bucket": "ok", "wall_s": 9.5, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 331, "llm_ms": 0, "tools_ms": 310, "other_ms": 21, "turns": 1, "artifact": "results/x_4.log"}
|
||||
{"site": "linkedin", "trial": 4, "bucket": "ok", "wall_s": 19.8, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 399, "llm_ms": 0, "tools_ms": 384, "other_ms": 15, "turns": 1, "artifact": "results/linkedin_4.log"}
|
||||
{"site": "reddit", "trial": 4, "bucket": "ok", "wall_s": 10.8, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 683, "llm_ms": 0, "tools_ms": 664, "other_ms": 19, "turns": 1, "artifact": "results/reddit_4.log"}
|
||||
{"site": "gmail", "trial": 4, "bucket": "not_measurable", "wall_s": 14.1, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 152, "llm_ms": 0, "tools_ms": 144, "other_ms": 8, "turns": 1, "artifact": "results/gmail_4.log"}
|
||||
{"site": "youtube", "trial": 4, "bucket": "ok", "wall_s": 37.5, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=nXWaTUBH", "composer": true, "submit": true, "total_ms": 10470, "llm_ms": 0, "tools_ms": 7842, "other_ms": 2628, "turns": 1, "artifact": "results/youtube_4.log"}
|
||||
{"site": "instagram", "trial": 4, "bucket": "ok", "wall_s": 12.7, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbphiQPlmjFa", "composer": true, "submit": true, "total_ms": 1401, "llm_ms": 0, "tools_ms": 1388, "other_ms": 13, "turns": 1, "artifact": "results/instagram_4.log"}
|
||||
{"site": "tiktok", "trial": 4, "bucket": "product_no_composer", "wall_s": 35.6, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 13145, "llm_ms": 0, "tools_ms": 7519, "other_ms": 5626, "turns": 1, "artifact": "results/tiktok_4.log"}
|
||||
{"site": "substack", "trial": 4, "bucket": "not_measurable", "wall_s": 13.7, "detail": "NOT MEASURABLE: signed out, PROVEN (password field on https://substack.com/)", "composer": false, "submit": false, "total_ms": 225, "llm_ms": 0, "tools_ms": 214, "other_ms": 11, "turns": 1, "artifact": "results/substack_4.log"}
|
||||
{"site": "twitch", "trial": 4, "bucket": "ok", "wall_s": 10.2, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/cinna", "composer": true, "submit": true, "total_ms": 392, "llm_ms": 0, "tools_ms": 376, "other_ms": 16, "turns": 1, "artifact": "results/twitch_4.log"}
|
||||
{"site": "x", "trial": 5, "bucket": "ok", "wall_s": 15.8, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 744, "llm_ms": 0, "tools_ms": 728, "other_ms": 16, "turns": 1, "artifact": "results/x_5.log"}
|
||||
{"site": "linkedin", "trial": 5, "bucket": "ok", "wall_s": 19.8, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 430, "llm_ms": 0, "tools_ms": 417, "other_ms": 13, "turns": 1, "artifact": "results/linkedin_5.log"}
|
||||
{"site": "reddit", "trial": 5, "bucket": "ok", "wall_s": 7.5, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 368, "llm_ms": 0, "tools_ms": 343, "other_ms": 25, "turns": 1, "artifact": "results/reddit_5.log"}
|
||||
{"site": "gmail", "trial": 5, "bucket": "not_measurable", "wall_s": 12.0, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 192, "llm_ms": 0, "tools_ms": 182, "other_ms": 10, "turns": 1, "artifact": "results/gmail_5.log"}
|
||||
{"site": "youtube", "trial": 5, "bucket": "ok", "wall_s": 38.2, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=3-EkbYNP", "composer": true, "submit": true, "total_ms": 11844, "llm_ms": 0, "tools_ms": 9154, "other_ms": 2690, "turns": 1, "artifact": "results/youtube_5.log"}
|
||||
{"site": "instagram", "trial": 5, "bucket": "ok", "wall_s": 12.7, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbphiQPlmjFa", "composer": true, "submit": true, "total_ms": 1734, "llm_ms": 0, "tools_ms": 1668, "other_ms": 66, "turns": 1, "artifact": "results/instagram_5.log"}
|
||||
{"site": "tiktok", "trial": 5, "bucket": "not_measurable", "wall_s": 43.8, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 15671, "llm_ms": 0, "tools_ms": 8749, "other_ms": 6922, "turns": 1, "artifact": "results/tiktok_5.log"}
|
||||
{"site": "substack", "trial": 5, "bucket": "not_measurable", "wall_s": 13.6, "detail": "NOT MEASURABLE: signed out, PROVEN (password field on https://substack.com/)", "composer": false, "submit": false, "total_ms": 229, "llm_ms": 0, "tools_ms": 220, "other_ms": 9, "turns": 1, "artifact": "results/substack_5.log"}
|
||||
{"site": "twitch", "trial": 5, "bucket": "ok", "wall_s": 11.7, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/cinna", "composer": true, "submit": true, "total_ms": 748, "llm_ms": 0, "tools_ms": 706, "other_ms": 42, "turns": 1, "artifact": "results/twitch_5.log"}
|
||||
{"site": "x", "trial": 6, "bucket": "ok", "wall_s": 10.9, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 591, "llm_ms": 0, "tools_ms": 567, "other_ms": 24, "turns": 1, "artifact": "results/x_6.log"}
|
||||
{"site": "linkedin", "trial": 6, "bucket": "ok", "wall_s": 19.6, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 385, "llm_ms": 0, "tools_ms": 366, "other_ms": 19, "turns": 1, "artifact": "results/linkedin_6.log"}
|
||||
{"site": "reddit", "trial": 6, "bucket": "ok", "wall_s": 7.4, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 385, "llm_ms": 0, "tools_ms": 366, "other_ms": 19, "turns": 1, "artifact": "results/reddit_6.log"}
|
||||
{"site": "gmail", "trial": 6, "bucket": "not_measurable", "wall_s": 11.7, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 529, "llm_ms": 0, "tools_ms": 520, "other_ms": 9, "turns": 1, "artifact": "results/gmail_6.log"}
|
||||
{"site": "youtube", "trial": 6, "bucket": "ok", "wall_s": 29.5, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=skI5KmOI", "composer": true, "submit": true, "total_ms": 10506, "llm_ms": 0, "tools_ms": 7888, "other_ms": 2618, "turns": 1, "artifact": "results/youtube_6.log"}
|
||||
{"site": "instagram", "trial": 6, "bucket": "ok", "wall_s": 13.2, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbphiQPlmjFa", "composer": true, "submit": true, "total_ms": 3083, "llm_ms": 0, "tools_ms": 465, "other_ms": 2618, "turns": 1, "artifact": "results/instagram_6.log"}
|
||||
{"site": "tiktok", "trial": 6, "bucket": "not_measurable", "wall_s": 19.5, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@karlodecke)", "composer": false, "submit": false, "total_ms": 485, "llm_ms": 0, "tools_ms": 448, "other_ms": 37, "turns": 1, "artifact": "results/tiktok_6.log"}
|
||||
{"site": "substack", "trial": 6, "bucket": "not_measurable", "wall_s": 16.4, "detail": "NOT MEASURABLE: signed out, PROVEN (password field on https://substack.com/)", "composer": false, "submit": false, "total_ms": 363, "llm_ms": 0, "tools_ms": 327, "other_ms": 36, "turns": 1, "artifact": "results/substack_6.log"}
|
||||
{"site": "twitch", "trial": 6, "bucket": "ok", "wall_s": 12.5, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/cinna", "composer": true, "submit": true, "total_ms": 850, "llm_ms": 0, "tools_ms": 832, "other_ms": 18, "turns": 1, "artifact": "results/twitch_6.log"}
|
||||
{"site": "x", "trial": 7, "bucket": "ok", "wall_s": 11.9, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 993, "llm_ms": 0, "tools_ms": 532, "other_ms": 461, "turns": 1, "artifact": "results/x_7.log"}
|
||||
{"site": "linkedin", "trial": 7, "bucket": "infra_harness", "wall_s": 18.5, "detail": "INVALID: stack was unhealthy (prestage] skipped (No AI provider)", "composer": false, "submit": false, "total_ms": 432, "llm_ms": 0, "tools_ms": 420, "other_ms": 12, "turns": 1, "artifact": "results/linkedin_7.log"}
|
||||
{"site": "reddit", "trial": 7, "bucket": "infra_router", "wall_s": 209.0, "detail": "INVALID: stack was unhealthy (9Router process died)", "composer": false, "submit": false, "total_ms": 411, "llm_ms": 0, "tools_ms": 398, "other_ms": 13, "turns": 1, "artifact": "results/reddit_7.log"}
|
||||
{"site": "gmail", "trial": 7, "bucket": "not_measurable", "wall_s": 12.0, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 161, "llm_ms": 0, "tools_ms": 150, "other_ms": 11, "turns": 1, "artifact": "results/gmail_7.log"}
|
||||
{"site": "youtube", "trial": 7, "bucket": "product_no_composer", "wall_s": 36.0, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 10626, "llm_ms": 0, "tools_ms": 7995, "other_ms": 2631, "turns": 1, "artifact": "results/youtube_7.log"}
|
||||
{"site": "instagram", "trial": 7, "bucket": "infra_router", "wall_s": 201.7, "detail": "INVALID: stack was unhealthy (9Router watchdog)", "composer": false, "submit": false, "total_ms": null, "llm_ms": null, "tools_ms": null, "other_ms": null, "turns": null, "artifact": "results/instagram_7.log"}
|
||||
{"site": "x", "trial": 1, "bucket": "ok", "wall_s": 20.6, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 4087, "llm_ms": 0, "tools_ms": 3442, "other_ms": 645, "turns": 1, "artifact": "results/x_1.log"}
|
||||
{"site": "linkedin", "trial": 1, "bucket": "ok", "wall_s": 20.3, "detail": "'post' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 1493, "llm_ms": 0, "tools_ms": 1388, "other_ms": 105, "turns": 1, "artifact": "results/linkedin_1.log"}
|
||||
{"site": "reddit", "trial": 1, "bucket": "ok", "wall_s": 8.1, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 705, "llm_ms": 0, "tools_ms": 695, "other_ms": 10, "turns": 1, "artifact": "results/reddit_1.log"}
|
||||
{"site": "gmail", "trial": 1, "bucket": "not_measurable", "wall_s": 13.7, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 220, "llm_ms": 0, "tools_ms": 206, "other_ms": 14, "turns": 1, "artifact": "results/gmail_1.log"}
|
||||
{"site": "youtube", "trial": 1, "bucket": "ok", "wall_s": 40.8, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=b2dhv1Pl", "composer": true, "submit": true, "total_ms": 12529, "llm_ms": 0, "tools_ms": 9817, "other_ms": 2712, "turns": 1, "artifact": "results/youtube_1.log"}
|
||||
{"site": "instagram", "trial": 1, "bucket": "ok", "wall_s": 19.0, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbphiQPlmjFa", "composer": true, "submit": true, "total_ms": 2087, "llm_ms": 0, "tools_ms": 2048, "other_ms": 39, "turns": 1, "artifact": "results/instagram_1.log"}
|
||||
{"site": "tiktok", "trial": 1, "bucket": "not_measurable", "wall_s": 37.6, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@sams.extre)", "composer": false, "submit": false, "total_ms": 424, "llm_ms": 0, "tools_ms": 410, "other_ms": 14, "turns": 1, "artifact": "results/tiktok_1.log"}
|
||||
{"site": "substack", "trial": 1, "bucket": "not_measurable", "wall_s": 13.6, "detail": "NOT MEASURABLE: signed out, PROVEN (password field on https://substack.com/)", "composer": false, "submit": false, "total_ms": 317, "llm_ms": 0, "tools_ms": 307, "other_ms": 10, "turns": 1, "artifact": "results/substack_1.log"}
|
||||
{"site": "twitch", "trial": 1, "bucket": "product_wrong_surface", "wall_s": 18.1, "detail": "WRONG SURFACE: wrong page https://www.twitch.tv/", "composer": false, "submit": false, "total_ms": 6946, "llm_ms": 0, "tools_ms": 4325, "other_ms": 2621, "turns": 1, "artifact": "results/twitch_1.log"}
|
||||
{"site": "x", "trial": 2, "bucket": "ok", "wall_s": 11.6, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 3174, "llm_ms": 0, "tools_ms": 1642, "other_ms": 1532, "turns": 1, "artifact": "results/x_2.log"}
|
||||
{"site": "linkedin", "trial": 2, "bucket": "ok", "wall_s": 13.5, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 1211, "llm_ms": 0, "tools_ms": 1198, "other_ms": 13, "turns": 1, "artifact": "results/linkedin_2.log"}
|
||||
{"site": "reddit", "trial": 2, "bucket": "ok", "wall_s": 7.6, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 815, "llm_ms": 0, "tools_ms": 801, "other_ms": 14, "turns": 1, "artifact": "results/reddit_2.log"}
|
||||
{"site": "gmail", "trial": 2, "bucket": "not_measurable", "wall_s": 11.6, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 148, "llm_ms": 0, "tools_ms": 138, "other_ms": 10, "turns": 1, "artifact": "results/gmail_2.log"}
|
||||
{"site": "youtube", "trial": 2, "bucket": "ok", "wall_s": 33.6, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=A4YuhvxN", "composer": true, "submit": true, "total_ms": 10560, "llm_ms": 0, "tools_ms": 7943, "other_ms": 2617, "turns": 1, "artifact": "results/youtube_2.log"}
|
||||
{"site": "instagram", "trial": 2, "bucket": "product_no_composer", "wall_s": 29.8, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 7222, "llm_ms": 0, "tools_ms": 4603, "other_ms": 2619, "turns": 1, "artifact": "results/instagram_2.log"}
|
||||
{"site": "tiktok", "trial": 2, "bucket": "not_measurable", "wall_s": 21.5, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@poongcrew/)", "composer": false, "submit": false, "total_ms": 275, "llm_ms": 0, "tools_ms": 256, "other_ms": 19, "turns": 1, "artifact": "results/tiktok_2.log"}
|
||||
{"site": "substack", "trial": 2, "bucket": "not_measurable", "wall_s": 17.6, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 6017, "llm_ms": 0, "tools_ms": 3400, "other_ms": 2617, "turns": 1, "artifact": "results/substack_2.log"}
|
||||
{"site": "twitch", "trial": 2, "bucket": "ok", "wall_s": 9.9, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 777, "llm_ms": 0, "tools_ms": 764, "other_ms": 13, "turns": 1, "artifact": "results/twitch_2.log"}
|
||||
{"site": "x", "trial": 3, "bucket": "ok", "wall_s": 14.0, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 2361, "llm_ms": 0, "tools_ms": 1831, "other_ms": 530, "turns": 1, "artifact": "results/x_3.log"}
|
||||
{"site": "linkedin", "trial": 3, "bucket": "ok", "wall_s": 13.5, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 1458, "llm_ms": 0, "tools_ms": 1435, "other_ms": 23, "turns": 1, "artifact": "results/linkedin_3.log"}
|
||||
{"site": "reddit", "trial": 3, "bucket": "ok", "wall_s": 7.5, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 953, "llm_ms": 0, "tools_ms": 928, "other_ms": 25, "turns": 1, "artifact": "results/reddit_3.log"}
|
||||
{"site": "gmail", "trial": 3, "bucket": "not_measurable", "wall_s": 11.6, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 166, "llm_ms": 0, "tools_ms": 155, "other_ms": 11, "turns": 1, "artifact": "results/gmail_3.log"}
|
||||
{"site": "youtube", "trial": 3, "bucket": "ok", "wall_s": 35.7, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=c0jZ7sPy", "composer": true, "submit": true, "total_ms": 10746, "llm_ms": 0, "tools_ms": 8127, "other_ms": 2619, "turns": 1, "artifact": "results/youtube_3.log"}
|
||||
{"site": "instagram", "trial": 3, "bucket": "ok", "wall_s": 12.3, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbO3F_Mgf7kg", "composer": true, "submit": true, "total_ms": 1344, "llm_ms": 0, "tools_ms": 1308, "other_ms": 36, "turns": 1, "artifact": "results/instagram_3.log"}
|
||||
{"site": "tiktok", "trial": 3, "bucket": "not_measurable", "wall_s": 17.4, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@callmetash)", "composer": false, "submit": false, "total_ms": 367, "llm_ms": 0, "tools_ms": 352, "other_ms": 15, "turns": 1, "artifact": "results/tiktok_3.log"}
|
||||
{"site": "substack", "trial": 3, "bucket": "not_measurable", "wall_s": 17.9, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 5991, "llm_ms": 0, "tools_ms": 3376, "other_ms": 2615, "turns": 1, "artifact": "results/substack_3.log"}
|
||||
{"site": "twitch", "trial": 3, "bucket": "ok", "wall_s": 11.4, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 784, "llm_ms": 0, "tools_ms": 772, "other_ms": 12, "turns": 1, "artifact": "results/twitch_3.log"}
|
||||
{"site": "x", "trial": 4, "bucket": "ok", "wall_s": 9.7, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 1562, "llm_ms": 0, "tools_ms": 1046, "other_ms": 516, "turns": 1, "artifact": "results/x_4.log"}
|
||||
{"site": "linkedin", "trial": 4, "bucket": "ok", "wall_s": 13.6, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 1415, "llm_ms": 0, "tools_ms": 1391, "other_ms": 24, "turns": 1, "artifact": "results/linkedin_4.log"}
|
||||
{"site": "reddit", "trial": 4, "bucket": "ok", "wall_s": 7.4, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 973, "llm_ms": 0, "tools_ms": 956, "other_ms": 17, "turns": 1, "artifact": "results/reddit_4.log"}
|
||||
{"site": "gmail", "trial": 4, "bucket": "not_measurable", "wall_s": 12.1, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 172, "llm_ms": 0, "tools_ms": 162, "other_ms": 10, "turns": 1, "artifact": "results/gmail_4.log"}
|
||||
{"site": "youtube", "trial": 4, "bucket": "ok", "wall_s": 37.0, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=C0zbypHp", "composer": true, "submit": true, "total_ms": 10787, "llm_ms": 0, "tools_ms": 8167, "other_ms": 2620, "turns": 1, "artifact": "results/youtube_4.log"}
|
||||
{"site": "instagram", "trial": 4, "bucket": "ok", "wall_s": 13.7, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbO3F_Mgf7kg", "composer": true, "submit": true, "total_ms": 1135, "llm_ms": 0, "tools_ms": 1122, "other_ms": 13, "turns": 1, "artifact": "results/instagram_4.log"}
|
||||
{"site": "tiktok", "trial": 4, "bucket": "not_measurable", "wall_s": 19.5, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@joshh_x0nx)", "composer": false, "submit": false, "total_ms": 367, "llm_ms": 0, "tools_ms": 354, "other_ms": 13, "turns": 1, "artifact": "results/tiktok_4.log"}
|
||||
{"site": "substack", "trial": 4, "bucket": "infra_harness", "wall_s": 78.1, "detail": "INVALID: parent rephrased the task, payload came through ambiguous", "composer": false, "submit": false, "total_ms": 3916, "llm_ms": 0, "tools_ms": 736, "other_ms": 3180, "turns": 1, "artifact": "results/substack_4.log"}
|
||||
{"site": "twitch", "trial": 4, "bucket": "ok", "wall_s": 9.3, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 747, "llm_ms": 0, "tools_ms": 731, "other_ms": 16, "turns": 1, "artifact": "results/twitch_4.log"}
|
||||
{"site": "x", "trial": 5, "bucket": "ok", "wall_s": 12.1, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 3034, "llm_ms": 0, "tools_ms": 2434, "other_ms": 600, "turns": 1, "artifact": "results/x_5.log"}
|
||||
{"site": "linkedin", "trial": 5, "bucket": "ok", "wall_s": 17.1, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 2063, "llm_ms": 0, "tools_ms": 2006, "other_ms": 57, "turns": 1, "artifact": "results/linkedin_5.log"}
|
||||
{"site": "reddit", "trial": 5, "bucket": "ok", "wall_s": 7.5, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 934, "llm_ms": 0, "tools_ms": 922, "other_ms": 12, "turns": 1, "artifact": "results/reddit_5.log"}
|
||||
{"site": "gmail", "trial": 5, "bucket": "not_measurable", "wall_s": 11.4, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 171, "llm_ms": 0, "tools_ms": 162, "other_ms": 9, "turns": 1, "artifact": "results/gmail_5.log"}
|
||||
{"site": "youtube", "trial": 5, "bucket": "ok", "wall_s": 33.6, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=hopQXbkX", "composer": true, "submit": true, "total_ms": 10570, "llm_ms": 0, "tools_ms": 7954, "other_ms": 2616, "turns": 1, "artifact": "results/youtube_5.log"}
|
||||
{"site": "instagram", "trial": 5, "bucket": "ok", "wall_s": 13.8, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbO3F_Mgf7kg", "composer": true, "submit": true, "total_ms": 1106, "llm_ms": 0, "tools_ms": 1094, "other_ms": 12, "turns": 1, "artifact": "results/instagram_5.log"}
|
||||
{"site": "tiktok", "trial": 5, "bucket": "not_measurable", "wall_s": 49.9, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 14469, "llm_ms": 0, "tools_ms": 8843, "other_ms": 5626, "turns": 1, "artifact": "results/tiktok_5.log"}
|
||||
{"site": "substack", "trial": 5, "bucket": "not_measurable", "wall_s": 25.7, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 4412, "llm_ms": 0, "tools_ms": 1797, "other_ms": 2615, "turns": 1, "artifact": "results/substack_5.log"}
|
||||
{"site": "twitch", "trial": 5, "bucket": "ok", "wall_s": 11.5, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 971, "llm_ms": 0, "tools_ms": 954, "other_ms": 17, "turns": 1, "artifact": "results/twitch_5.log"}
|
||||
{"site": "x", "trial": 6, "bucket": "ok", "wall_s": 10.5, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 1816, "llm_ms": 0, "tools_ms": 1302, "other_ms": 514, "turns": 1, "artifact": "results/x_6.log"}
|
||||
{"site": "linkedin", "trial": 6, "bucket": "ok", "wall_s": 14.3, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 1419, "llm_ms": 0, "tools_ms": 1401, "other_ms": 18, "turns": 1, "artifact": "results/linkedin_6.log"}
|
||||
{"site": "reddit", "trial": 6, "bucket": "ok", "wall_s": 10.7, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 900, "llm_ms": 0, "tools_ms": 874, "other_ms": 26, "turns": 1, "artifact": "results/reddit_6.log"}
|
||||
{"site": "gmail", "trial": 6, "bucket": "not_measurable", "wall_s": 15.6, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 186, "llm_ms": 0, "tools_ms": 176, "other_ms": 10, "turns": 1, "artifact": "results/gmail_6.log"}
|
||||
{"site": "youtube", "trial": 6, "bucket": "ok", "wall_s": 39.4, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=zmS7kzSH", "composer": true, "submit": true, "total_ms": 10487, "llm_ms": 0, "tools_ms": 7869, "other_ms": 2618, "turns": 1, "artifact": "results/youtube_6.log"}
|
||||
{"site": "instagram", "trial": 6, "bucket": "product_no_composer", "wall_s": 23.4, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 7748, "llm_ms": 0, "tools_ms": 5128, "other_ms": 2620, "turns": 1, "artifact": "results/instagram_6.log"}
|
||||
{"site": "tiktok", "trial": 6, "bucket": "not_measurable", "wall_s": 19.5, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@futurewag7)", "composer": false, "submit": false, "total_ms": 302, "llm_ms": 0, "tools_ms": 292, "other_ms": 10, "turns": 1, "artifact": "results/tiktok_6.log"}
|
||||
{"site": "substack", "trial": 6, "bucket": "not_measurable", "wall_s": 17.6, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 6000, "llm_ms": 0, "tools_ms": 3385, "other_ms": 2615, "turns": 1, "artifact": "results/substack_6.log"}
|
||||
{"site": "twitch", "trial": 6, "bucket": "ok", "wall_s": 11.5, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 743, "llm_ms": 0, "tools_ms": 733, "other_ms": 10, "turns": 1, "artifact": "results/twitch_6.log"}
|
||||
{"site": "x", "trial": 7, "bucket": "ok", "wall_s": 12.4, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 1629, "llm_ms": 0, "tools_ms": 1110, "other_ms": 519, "turns": 1, "artifact": "results/x_7.log"}
|
||||
{"site": "linkedin", "trial": 7, "bucket": "ok", "wall_s": 13.7, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 2291, "llm_ms": 0, "tools_ms": 2276, "other_ms": 15, "turns": 1, "artifact": "results/linkedin_7.log"}
|
||||
{"site": "reddit", "trial": 7, "bucket": "ok", "wall_s": 7.8, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 987, "llm_ms": 0, "tools_ms": 965, "other_ms": 22, "turns": 1, "artifact": "results/reddit_7.log"}
|
||||
{"site": "gmail", "trial": 7, "bucket": "not_measurable", "wall_s": 14.1, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 201, "llm_ms": 0, "tools_ms": 192, "other_ms": 9, "turns": 1, "artifact": "results/gmail_7.log"}
|
||||
{"site": "youtube", "trial": 7, "bucket": "ok", "wall_s": 32.0, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=9xtAKakz", "composer": true, "submit": true, "total_ms": 10755, "llm_ms": 0, "tools_ms": 8141, "other_ms": 2614, "turns": 1, "artifact": "results/youtube_7.log"}
|
||||
{"site": "instagram", "trial": 7, "bucket": "product_wrong_surface", "wall_s": 17.8, "detail": "WRONG SURFACE: wrong page https://www.instagram.com/reels/DbktzhHuBnj/", "composer": false, "submit": false, "total_ms": 571, "llm_ms": 0, "tools_ms": 560, "other_ms": 11, "turns": 1, "artifact": "results/instagram_7.log"}
|
||||
{"site": "tiktok", "trial": 7, "bucket": "not_measurable", "wall_s": 18.2, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@cadelandmi)", "composer": false, "submit": false, "total_ms": 352, "llm_ms": 0, "tools_ms": 341, "other_ms": 11, "turns": 1, "artifact": "results/tiktok_7.log"}
|
||||
{"site": "substack", "trial": 7, "bucket": "not_measurable", "wall_s": 17.9, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 6121, "llm_ms": 0, "tools_ms": 3506, "other_ms": 2615, "turns": 1, "artifact": "results/substack_7.log"}
|
||||
{"site": "twitch", "trial": 7, "bucket": "ok", "wall_s": 11.2, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 701, "llm_ms": 0, "tools_ms": 688, "other_ms": 13, "turns": 1, "artifact": "results/twitch_7.log"}
|
||||
{"site": "x", "trial": 8, "bucket": "ok", "wall_s": 14.0, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 3956, "llm_ms": 0, "tools_ms": 3324, "other_ms": 632, "turns": 1, "artifact": "results/x_8.log"}
|
||||
{"site": "linkedin", "trial": 8, "bucket": "ok", "wall_s": 18.4, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 4182, "llm_ms": 0, "tools_ms": 3480, "other_ms": 702, "turns": 1, "artifact": "results/linkedin_8.log"}
|
||||
{"site": "reddit", "trial": 8, "bucket": "ok", "wall_s": 7.4, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 887, "llm_ms": 0, "tools_ms": 873, "other_ms": 14, "turns": 1, "artifact": "results/reddit_8.log"}
|
||||
{"site": "gmail", "trial": 8, "bucket": "not_measurable", "wall_s": 13.8, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 225, "llm_ms": 0, "tools_ms": 213, "other_ms": 12, "turns": 1, "artifact": "results/gmail_8.log"}
|
||||
{"site": "youtube", "trial": 8, "bucket": "ok", "wall_s": 31.6, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=hN2InEt1", "composer": true, "submit": true, "total_ms": 10583, "llm_ms": 0, "tools_ms": 7956, "other_ms": 2627, "turns": 1, "artifact": "results/youtube_8.log"}
|
||||
{"site": "instagram", "trial": 8, "bucket": "ok", "wall_s": 9.9, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbkML-VgTm9/", "composer": true, "submit": true, "total_ms": 900, "llm_ms": 0, "tools_ms": 889, "other_ms": 11, "turns": 1, "artifact": "results/instagram_8.log"}
|
||||
{"site": "tiktok", "trial": 8, "bucket": "not_measurable", "wall_s": 25.6, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@wkpnkjhzi0)", "composer": false, "submit": false, "total_ms": 857, "llm_ms": 0, "tools_ms": 847, "other_ms": 10, "turns": 1, "artifact": "results/tiktok_8.log"}
|
||||
{"site": "substack", "trial": 8, "bucket": "not_measurable", "wall_s": 25.7, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 4410, "llm_ms": 0, "tools_ms": 1794, "other_ms": 2616, "turns": 1, "artifact": "results/substack_8.log"}
|
||||
{"site": "twitch", "trial": 8, "bucket": "ok", "wall_s": 14.3, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 716, "llm_ms": 0, "tools_ms": 703, "other_ms": 13, "turns": 1, "artifact": "results/twitch_8.log"}
|
||||
{"site": "x", "trial": 9, "bucket": "ok", "wall_s": 13.5, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 1726, "llm_ms": 0, "tools_ms": 1210, "other_ms": 516, "turns": 1, "artifact": "results/x_9.log"}
|
||||
{"site": "linkedin", "trial": 9, "bucket": "ok", "wall_s": 13.6, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 2096, "llm_ms": 0, "tools_ms": 2076, "other_ms": 20, "turns": 1, "artifact": "results/linkedin_9.log"}
|
||||
{"site": "reddit", "trial": 9, "bucket": "ok", "wall_s": 8.2, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 1061, "llm_ms": 0, "tools_ms": 1046, "other_ms": 15, "turns": 1, "artifact": "results/reddit_9.log"}
|
||||
{"site": "gmail", "trial": 9, "bucket": "not_measurable", "wall_s": 14.3, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 207, "llm_ms": 0, "tools_ms": 198, "other_ms": 9, "turns": 1, "artifact": "results/gmail_9.log"}
|
||||
{"site": "youtube", "trial": 9, "bucket": "ok", "wall_s": 38.1, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=l2qHL3e7", "composer": true, "submit": true, "total_ms": 10445, "llm_ms": 0, "tools_ms": 7829, "other_ms": 2616, "turns": 1, "artifact": "results/youtube_9.log"}
|
||||
{"site": "instagram", "trial": 9, "bucket": "product_wrong_surface", "wall_s": 22.6, "detail": "WRONG SURFACE: wrong page https://www.instagram.com/reels/DbkML-VgTm9/", "composer": false, "submit": false, "total_ms": 1954, "llm_ms": 0, "tools_ms": 1941, "other_ms": 13, "turns": 1, "artifact": "results/instagram_9.log"}
|
||||
{"site": "tiktok", "trial": 9, "bucket": "not_measurable", "wall_s": 19.5, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@smartround)", "composer": false, "submit": false, "total_ms": 357, "llm_ms": 0, "tools_ms": 346, "other_ms": 11, "turns": 1, "artifact": "results/tiktok_9.log"}
|
||||
{"site": "substack", "trial": 9, "bucket": "not_measurable", "wall_s": 23.6, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 4450, "llm_ms": 0, "tools_ms": 1835, "other_ms": 2615, "turns": 1, "artifact": "results/substack_9.log"}
|
||||
{"site": "twitch", "trial": 9, "bucket": "ok", "wall_s": 12.4, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 721, "llm_ms": 0, "tools_ms": 710, "other_ms": 11, "turns": 1, "artifact": "results/twitch_9.log"}
|
||||
{"site": "x", "trial": 10, "bucket": "ok", "wall_s": 12.5, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 1753, "llm_ms": 0, "tools_ms": 1220, "other_ms": 533, "turns": 1, "artifact": "results/x_10.log"}
|
||||
{"site": "linkedin", "trial": 10, "bucket": "ok", "wall_s": 19.9, "detail": "'comment' <- 'Text editor for creating content' on https://www.linkedin.com/feed/?shareActi", "composer": true, "submit": true, "total_ms": 4378, "llm_ms": 0, "tools_ms": 4355, "other_ms": 23, "turns": 1, "artifact": "results/linkedin_10.log"}
|
||||
{"site": "reddit", "trial": 10, "bucket": "ok", "wall_s": 7.5, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 1167, "llm_ms": 0, "tools_ms": 1156, "other_ms": 11, "turns": 1, "artifact": "results/reddit_10.log"}
|
||||
{"site": "gmail", "trial": 10, "bucket": "not_measurable", "wall_s": 14.4, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 298, "llm_ms": 0, "tools_ms": 274, "other_ms": 24, "turns": 1, "artifact": "results/gmail_10.log"}
|
||||
{"site": "youtube", "trial": 10, "bucket": "ok", "wall_s": 35.6, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=QbYdd0io", "composer": true, "submit": true, "total_ms": 10818, "llm_ms": 0, "tools_ms": 8202, "other_ms": 2616, "turns": 1, "artifact": "results/youtube_10.log"}
|
||||
{"site": "instagram", "trial": 10, "bucket": "ok", "wall_s": 12.0, "detail": "'post' <- 'Add a comment\u2026' on https://www.instagram.com/p/DbphiQPlmjFa", "composer": true, "submit": true, "total_ms": 3174, "llm_ms": 0, "tools_ms": 555, "other_ms": 2619, "turns": 1, "artifact": "results/instagram_10.log"}
|
||||
{"site": "tiktok", "trial": 10, "bucket": "not_measurable", "wall_s": 53.8, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 13248, "llm_ms": 0, "tools_ms": 7625, "other_ms": 5623, "turns": 1, "artifact": "results/tiktok_10.log"}
|
||||
{"site": "substack", "trial": 10, "bucket": "not_measurable", "wall_s": 23.2, "detail": "NOT MEASURABLE: signed out per the agent, UNVERIFIED (no wall evidence)", "composer": false, "submit": false, "total_ms": 6162, "llm_ms": 0, "tools_ms": 3548, "other_ms": 2614, "turns": 1, "artifact": "results/substack_10.log"}
|
||||
{"site": "twitch", "trial": 10, "bucket": "ok", "wall_s": 11.8, "detail": "'send chat' <- 'Send a message' on https://www.twitch.tv/caedrel", "composer": true, "submit": true, "total_ms": 6162, "llm_ms": 0, "tools_ms": 3548, "other_ms": 2614, "turns": 1, "artifact": "results/twitch_10.log"}
|
||||
{"site": "x", "trial": 11, "bucket": "ok", "wall_s": 60.7, "detail": "'post' <- 'Post text' on https://x.com/compose/post", "composer": true, "submit": true, "total_ms": 22712, "llm_ms": 0, "tools_ms": 20722, "other_ms": 1990, "turns": 1, "artifact": "results/x_11.log"}
|
||||
{"site": "linkedin", "trial": 11, "bucket": "infra_router", "wall_s": 61.4, "detail": "INVALID: stack was unhealthy (No AI provider connected)", "composer": false, "submit": false, "total_ms": 4298, "llm_ms": 0, "tools_ms": 4141, "other_ms": 157, "turns": 1, "artifact": "results/linkedin_11.log"}
|
||||
{"site": "reddit", "trial": 11, "bucket": "ok", "wall_s": 31.1, "detail": "'the submit control is present but <- 'Post body text field' on https://www.reddit.com/r/test/submit/?ty", "composer": true, "submit": false, "total_ms": 2283, "llm_ms": 0, "tools_ms": 2233, "other_ms": 50, "turns": 1, "artifact": "results/reddit_11.log"}
|
||||
{"site": "gmail", "trial": 11, "bucket": "not_measurable", "wall_s": 39.5, "detail": "NOT MEASURABLE: signed out, PROVEN (sign-in URL https://accounts.google.com/v3/signin/acco)", "composer": false, "submit": false, "total_ms": 1468, "llm_ms": 0, "tools_ms": 1445, "other_ms": 23, "turns": 1, "artifact": "results/gmail_11.log"}
|
||||
{"site": "youtube", "trial": 11, "bucket": "ok", "wall_s": 60.3, "detail": "'post' <- 'contenteditable' on https://www.youtube.com/watch?v=6dROuF_T", "composer": true, "submit": true, "total_ms": 16285, "llm_ms": 0, "tools_ms": 13547, "other_ms": 2738, "turns": 1, "artifact": "results/youtube_11.log"}
|
||||
{"site": "instagram", "trial": 11, "bucket": "product_no_composer", "wall_s": 82.3, "detail": "decline: no composer, opener, or structural editable", "composer": false, "submit": false, "total_ms": 25483, "llm_ms": 0, "tools_ms": 19770, "other_ms": 5713, "turns": 1, "artifact": "results/instagram_11.log"}
|
||||
{"site": "tiktok", "trial": 11, "bucket": "not_measurable", "wall_s": 42.5, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://www.tiktok.com/@elliegetaj)", "composer": false, "submit": false, "total_ms": 929, "llm_ms": 0, "tools_ms": 900, "other_ms": 29, "turns": 1, "artifact": "results/tiktok_11.log"}
|
||||
{"site": "substack", "trial": 11, "bucket": "not_measurable", "wall_s": 24.9, "detail": "NOT MEASURABLE: login wall claimed, UNVERIFIED (no evidence for https://substack.com/)", "composer": false, "submit": false, "total_ms": 921, "llm_ms": 0, "tools_ms": 800, "other_ms": 121, "turns": 1, "artifact": "results/substack_11.log"}
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Criterion 9: does the learned fast path record, and does replaying it help?
|
||||
|
||||
Reads a backend log and reports the two rates the criterion names, plus the reason every refusal
|
||||
gave. Every string below is grep-verified against backend/apps/agents/browser/, because the last
|
||||
instrument that measured this layer looked for a line the code never prints.
|
||||
|
||||
python3 skillstats.py <backend.log> [...]
|
||||
"""
|
||||
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
# (label, regex). Anchored on the exact logger.info text in browser_skills.py / browser_agent.py.
|
||||
PATTERNS = [
|
||||
("gate_passed", re.compile(r"record gate: honest=True informational=False removal=False unconfirmed_send=False")),
|
||||
("gate_refused", re.compile(r"record gate: (?!honest=True informational=False removal=False unconfirmed_send=False)")),
|
||||
("recorded", re.compile(r"\[browser-skills\] (learned|EDITED) \d+-step skill for (\S+)")),
|
||||
("re_derived", re.compile(r"re-derived identical \d+-step skill")),
|
||||
("not_recorded", re.compile(r"NOT recorded \(([^)]*)\)")),
|
||||
("matched", re.compile(r"\[browser-skills\] skill matched on (\S+)")),
|
||||
("no_skill", re.compile(r"no skill for host=")),
|
||||
("replay_prefix", re.compile(r"PREFIX replay: (\d+)/(\d+) steps on (\S+)")),
|
||||
("replay_attempt", re.compile(r"REPLAY attempt: (\d+) steps on (\S+)")),
|
||||
("replay_ok", re.compile(r"REPLAY SUCCEEDED in (\d+)ms")),
|
||||
("replay_failed", re.compile(r"replay step failed \(")),
|
||||
("not_replayed", re.compile(r"skill on (\S+) not replayed: (.+?);")),
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
text = ""
|
||||
for p in sys.argv[1:]:
|
||||
try:
|
||||
text += open(p, errors="ignore").read()
|
||||
except OSError as e:
|
||||
print(f"skip {p}: {e}")
|
||||
if not text:
|
||||
print("no log content")
|
||||
return 2
|
||||
|
||||
counts = Counter()
|
||||
hosts_recorded = Counter()
|
||||
refusals = Counter()
|
||||
not_replayed = Counter()
|
||||
for line in text.splitlines():
|
||||
for label, rx in PATTERNS:
|
||||
m = rx.search(line)
|
||||
if not m:
|
||||
continue
|
||||
counts[label] += 1
|
||||
if label == "recorded":
|
||||
hosts_recorded[m.group(2)] += 1
|
||||
if label == "not_recorded":
|
||||
refusals[m.group(1)] += 1
|
||||
if label == "not_replayed":
|
||||
not_replayed[m.group(2)[:60]] += 1
|
||||
|
||||
gate_passed = counts["gate_passed"]
|
||||
recorded = counts["recorded"] + counts["re_derived"]
|
||||
print("=== RECORDING ===")
|
||||
print(f" runs reaching the record gate : {gate_passed + counts['gate_refused']}")
|
||||
print(f" gate PASSED (eligible) : {gate_passed}")
|
||||
print(f" skills recorded : {recorded}"
|
||||
+ (f" ({100*recorded//gate_passed}% of eligible)" if gate_passed else ""))
|
||||
for why, n in refusals.most_common():
|
||||
print(f" refused: {why} x{n}")
|
||||
for h, n in hosts_recorded.most_common(10):
|
||||
print(f" recorded on {h} x{n}")
|
||||
|
||||
print("\n=== REPLAY ===")
|
||||
print(f" skill matched for the host : {counts['matched']}")
|
||||
print(f" no skill for the host : {counts['no_skill']}")
|
||||
print(f" full replay attempts : {counts['replay_attempt']}")
|
||||
print(f" full replays SUCCEEDED : {counts['replay_ok']}")
|
||||
print(f" prefix replays performed : {counts['replay_prefix']}")
|
||||
print(f" replays refused (unsafe step) : {counts['not_replayed']}")
|
||||
for why, n in not_replayed.most_common(6):
|
||||
print(f" {why} x{n}")
|
||||
print(f" replay step failures : {counts['replay_failed']}")
|
||||
replayed = counts["replay_attempt"] + counts["replay_prefix"]
|
||||
print(f"\n RECORDING RATE {recorded}/{gate_passed}"
|
||||
f" = {100*recorded//gate_passed if gate_passed else 0}% (criterion 9 wants >=50%)")
|
||||
print(f" REPLAYS PERFORMED {replayed}, of which succeeded {counts['replay_ok']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+129
@@ -0,0 +1,129 @@
|
||||
#!/bin/bash
|
||||
# The isolated v3 measurement stack: backend :8326, webpack :3026, Electron on its own profile.
|
||||
#
|
||||
# One script because rebuilding it by hand three times cost three different half-booted stacks, and a
|
||||
# half-booted stack does not fail loudly, it just measures nothing and blames the product.
|
||||
#
|
||||
# ./stack.sh up dry backend refuses the irreversible click (coverage sweeps)
|
||||
# ./stack.sh up live backend really clicks send (canary write tests)
|
||||
# ./stack.sh down everything, SIGTERM then SIGKILL, ports verified free
|
||||
# ./stack.sh status what is up right now
|
||||
#
|
||||
# Never touches :8324 / :3000. Those belong to whatever else is on this box.
|
||||
|
||||
# Where logs, profiles and run output go. Defaults to runs/ beside this harness; override with
|
||||
# OSW_BENCH_DIR to keep multi-gigabyte browser profiles off the repo disk.
|
||||
SP="${OSW_BENCH_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/runs}"
|
||||
mkdir -p "$SP"
|
||||
# The harness itself lives beside this script; SP is only for run OUTPUT.
|
||||
HARNESS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
TREE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
TAG="${TAG:-run}"
|
||||
|
||||
status() {
|
||||
# Two bugs lived in the first version of these four lines, and both printed a reassuring 0 over a
|
||||
# fully live stack. macOS pgrep has no -c flag at all, so `pgrep -fc` exits on a usage error and
|
||||
# the count renders empty; and the patterns were the words I type rather than the words ps prints
|
||||
# (uvicorn runs as `python -m uvicorn`). A status check that reads clean while the box is busy is
|
||||
# the worst instrument in this directory, because its whole job is to stop a second stack landing
|
||||
# on the first. Count with grep on real ps output.
|
||||
echo "=== procs ==="
|
||||
local ps_out
|
||||
ps_out="$(ps -Ao pid,command)"
|
||||
for p in "\-m uvicorn backend.main" "bin/webpack" "MacOS/Electron" \
|
||||
"keep_renderer" "bench\.py" "browser_canary"; do
|
||||
printf " %-34s %s\n" "$p" "$(echo "$ps_out" | grep -cE "$p")"
|
||||
done
|
||||
echo "=== ports ==="
|
||||
for pt in 8324 8326 3000 3026 20128; do
|
||||
printf " %-6s %s\n" "$pt" "$(lsof -ti tcp:$pt 2>/dev/null | tr '\n' ' ')"
|
||||
done
|
||||
# Anything on :8324 is the OTHER checkout. Say so out loud: while it is up, the single 9router on
|
||||
# :20128 is contended and every timing this stack produces is noise (a sweep once scored 1/9 vs
|
||||
# 4/9 with zero code change, purely on that contention).
|
||||
if [ -n "$(lsof -ti tcp:8324 2>/dev/null)" ]; then
|
||||
echo " !! another OpenSwarm is on :8324. Do NOT measure, and do NOT kill it."
|
||||
fi
|
||||
}
|
||||
|
||||
down() {
|
||||
# SCOPED TO THIS STACK ONLY. The first version matched on `uvicorn backend.main`, which is exactly
|
||||
# what the OTHER OpenSwarm checkout on this box runs too: one careless `stack.sh down` would have
|
||||
# killed a colleague's backend on :8324 mid-session. Nothing here may match a process this script
|
||||
# did not start, so identify them by MY ports and MY profile directory, never by a generic name.
|
||||
pkill -f "keep_renderer" 2>/dev/null
|
||||
pkill -f "BACKEND RESTARTED" 2>/dev/null
|
||||
pkill -f "user-data-dir=$SP/udd" 2>/dev/null
|
||||
pkill -f "browser_canary" 2>/dev/null
|
||||
pkill -f "$HARNESS/bench.py" 2>/dev/null
|
||||
sleep 3
|
||||
for pt in 8326 3026; do
|
||||
pids=$(lsof -ti tcp:$pt 2>/dev/null)
|
||||
[ -n "$pids" ] && kill -9 $pids 2>/dev/null
|
||||
done
|
||||
pkill -9 -f "user-data-dir=$SP/udd" 2>/dev/null
|
||||
sleep 1
|
||||
status
|
||||
}
|
||||
|
||||
up() {
|
||||
local mode="${1:-dry}"
|
||||
local dry=0
|
||||
[ "$mode" = "dry" ] && dry=1
|
||||
|
||||
# A stale stack under a fresh one is the single most expensive failure here: two backends fight
|
||||
# over the one 9router and every number becomes a coin flip. Always start from nothing.
|
||||
down >/dev/null 2>&1
|
||||
|
||||
# Supervised, because the backend has died mid-measurement on a clean SIGTERM with no error in
|
||||
# its log (2026-08-05 07:44, 52 minutes in, no other stack on the box). An unsupervised death does
|
||||
# not announce itself: the harness just starts recording connection errors as product failures.
|
||||
# The restart marker goes into the same log the harness slices, so any trial that spans a restart
|
||||
# can be excluded instead of counted.
|
||||
cd "$TREE" || exit 1
|
||||
nohup bash -c '
|
||||
while true; do
|
||||
OPENSWARM_PORT=8326 OSW_SENDSCRIPT_DRYRUN='"$dry"' OPENSWARM_DEV=1 \
|
||||
./backend/.venv/bin/python -m uvicorn backend.main:app --port 8326 --host 127.0.0.1 \
|
||||
>> "'"$SP/${TAG}_be.log"'" 2>&1
|
||||
echo "[stack] BACKEND RESTARTED at $(date +%H:%M:%S)" >> "'"$SP/${TAG}_be.log"'"
|
||||
sleep 4
|
||||
done' > /dev/null 2>&1 &
|
||||
disown
|
||||
|
||||
cd "$TREE/frontend" || exit 1
|
||||
OPENSWARM_DEV_PORT=3026 OPENSWARM_PORT=8326 \
|
||||
./node_modules/.bin/webpack serve --mode development \
|
||||
> "$SP/${TAG}_wp.log" 2>&1 &
|
||||
|
||||
# A real authenticated 200, not just "something accepted a TCP connection". /api/health does not
|
||||
# exist (it 404s), and curl calls a 404 a success, so the old check passed the instant the socket
|
||||
# opened and handed the next step a backend that had not finished booting.
|
||||
echo "waiting for backend :8326 ..."
|
||||
for i in $(seq 1 90); do
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 3 \
|
||||
-H "Authorization: Bearer $(cat "$TREE/backend/data/auth.token" 2>/dev/null)" \
|
||||
"http://127.0.0.1:8326/api/dashboards/list")
|
||||
[ "$code" = "200" ] && break
|
||||
sleep 2
|
||||
done
|
||||
echo "waiting for webpack :3026 ..."
|
||||
for i in $(seq 1 120); do
|
||||
curl -s -o /dev/null --max-time 3 "http://localhost:3026" && break
|
||||
sleep 2
|
||||
done
|
||||
|
||||
OPENSWARM_DEV_PORT=3026 OPENSWARM_PORT=8326 nohup "$HARNESS/keep_renderer.sh" \
|
||||
>> "$SP/renderer.log" 2>&1 &
|
||||
disown
|
||||
sleep 25
|
||||
echo "mode=$mode (OSW_SENDSCRIPT_DRYRUN=$dry) tag=$TAG"
|
||||
status
|
||||
}
|
||||
|
||||
case "$1" in
|
||||
up) up "$2" ;;
|
||||
down) down ;;
|
||||
status) status ;;
|
||||
*) echo "usage: stack.sh {up dry|up live|down|status}"; exit 1 ;;
|
||||
esac
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Every literal this harness greps for must exist in the code that is supposed to print it.
|
||||
|
||||
Written after the fourth instance of the same bug. `DELIVERY CONFIRMED` appeared in the canary and
|
||||
nowhere in the backend. `saw_page` listed two `[browser-action] X` strings that are never emitted.
|
||||
`replay_full` matched a phrase no logger uses. Each one silently turned a measurement into a
|
||||
constant, and each cost hours of chasing a product bug that did not exist.
|
||||
|
||||
A grep whose needle is absent from the source cannot fail loudly, so make it fail here instead.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
TREE = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
SP = os.path.dirname(HERE)
|
||||
|
||||
# (label, literal, where it must appear). A literal is a fixed substring of a real log line, with
|
||||
# the f-string holes cut out, so it can be checked with a plain fixed-string grep.
|
||||
MARKERS = [
|
||||
# canary receipt detection
|
||||
("receipt/sendscript", "done sent_receipt=", "backend/apps"),
|
||||
("receipt/agent-loop", "two-sided receipt passed", "backend/apps"),
|
||||
("receipt/autosend", "code-send delivered (receipt verified)", "backend/apps"),
|
||||
# coverage.py grading
|
||||
("dryrun report", "DRYRUN: WOULD send (fill committed", "backend/apps"),
|
||||
("decline", "[browser-sendscript] decline: ", "backend/apps"),
|
||||
("disabled submit", "is present but DISABLED", "backend/apps"),
|
||||
("fill target", "[browser-sendscript] fill target ", "backend/apps"),
|
||||
("fill errored", "fill errored (", "backend/apps"),
|
||||
("login wall", "decline: login/auth wall", "backend/apps"),
|
||||
("signed out", "decline: signed OUT", "backend/apps"),
|
||||
("recovery", "one recovery dispatch", "backend/apps"),
|
||||
# coverage.py UNHEALTHY (infrastructure)
|
||||
("router watchdog", "9Router watchdog", "backend/apps"),
|
||||
("router died", "9Router process died", "backend/apps"),
|
||||
("no provider", "No AI provider connected", "backend/apps"),
|
||||
("no dashboard", "dispatch refused: no dashboard", "backend/apps"),
|
||||
# bench.py infra buckets
|
||||
("browser timeout", "Browser command timed out", "backend/apps"),
|
||||
("card gone/webview", "not an electron webview", "backend/apps"),
|
||||
("card gone/dashboard", "no dashboard is connected", "backend/apps"),
|
||||
("card gone/unresponsive", "page unresponsive", "backend/apps"),
|
||||
("busy to read", "too busy to read", "frontend/src"),
|
||||
# skillstats.py
|
||||
("skill record gate", "record gate: honest=", "backend/apps"),
|
||||
("skill recorded", "-step skill for ", "backend/apps"),
|
||||
("skill re-derived", "re-derived identical ", "backend/apps"),
|
||||
("skill not recorded", "NOT recorded (", "backend/apps"),
|
||||
("skill matched", "skill matched on ", "backend/apps"),
|
||||
("no skill", "no skill for host=", "backend/apps"),
|
||||
("prefix replay", "PREFIX replay: ", "backend/apps"),
|
||||
("replay attempt", "REPLAY attempt: ", "backend/apps"),
|
||||
("replay succeeded", "REPLAY SUCCEEDED in ", "backend/apps"),
|
||||
("replay step failed", "replay step failed (", "backend/apps"),
|
||||
("not replayed", " not replayed: ", "backend/apps"),
|
||||
# the honesty marker the packaged smokes also grep for
|
||||
("send-not-verified", "[send clicked, NOT verified]", "backend/apps"),
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
bad = []
|
||||
for label, needle, where in MARKERS:
|
||||
r = subprocess.run(["grep", "-rF", "--", needle, os.path.join(TREE, where)],
|
||||
capture_output=True, text=True)
|
||||
n = len([ln for ln in r.stdout.splitlines() if ln.strip()])
|
||||
status = "ok " if n else "DEAD"
|
||||
if not n:
|
||||
bad.append((label, needle))
|
||||
print(f" {status} {label:<22} x{n:<3} {needle!r}")
|
||||
print()
|
||||
if bad:
|
||||
print(f"{len(bad)} DEAD marker(s): a grep for these can never match, so whatever they")
|
||||
print("measure is a constant. Fix the needle or delete the check.")
|
||||
for label, needle in bad:
|
||||
print(f" - {label}: {needle!r}")
|
||||
return 1
|
||||
print(f"all {len(MARKERS)} markers exist in the source that prints them")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user