mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-09-11 20:27:44 +02:00
ciregenzandClaude Opus 5
3c50359146
Six of the eleven defects here were in the MEASUREMENT, not the product, and they were wrong in both directions. Harness, all of which silently produced wrong numbers: - coverage.py preflight refused every sweep on a box holding exactly one backend: stack.sh's supervisor is a `bash -c` quoting the whole uvicorn line, so it carries both "-m uvicorn backend.main" AND the venv python path. Discriminate on POSITION. - stack.sh status reported 2 backends over 1 and 0 webpack over a live dev server (webpack retitles its process). A status check whose job is preventing a second stack, failing in the direction that lets one land. - c7_run.sh/c8_run.sh slice r6_be.log while stack.sh names logs by TAG: a stack under any other tag hands every trial an empty slice and the sweep reports a confident 0/108. Now refuses loudly; it caught this exact mistake on first use. - "Browser command timed out" was bucketed infra. It is ONE command blowing its own budget, not a dead webview: all 4 such rows were BrowserFindComposer at exactly its 30s cap, every run completed after, zero card-gone markers in the whole log. Filed as infra it read as 11.8% flake AND lifted holdout reach 70% -> 84%. - api_retry / rate_limit_error now grade as infra. A provider 429 storm turned clean 15-21s exclusions into 188s product_no_composer rows. - bench.py prints reach BOTH ways when a row is UNVERIFIED. An exclusion resting on the agent's own word quietly flatters the score, and coverage.py's own instruction to confirm it by hand goes unread (I quoted a 100% that excluded onlinegdb). Timing was measuring 0.2% of the run: prestage completes BEFORE metrics_started_at, so other_ms was 25ms of a 12700ms median while prestage (4146ms, ~61%) sat in no bucket at all. prestage_ms/task_ms are now recorded; total_ms is deliberately NOT redefined, which would invalidate every before/after already taken against it. Product: - find_composer rejected ACE/CodeMirror-5/Monaco composers. Their input is a ~1x1 offscreen textarea that paints into a sibling div, so it can never pass a size gate. Accept it when a VISIBLE ancestor is composer-sized; honeypots stay out because the input itself must not be display:none/visibility:hidden/opacity:0. anon reach 80% -> 100%, holdout 89% -> 90%, p95 38.6s -> 9.7s. - the composer poll slept a blind 0+1.2+1.4 = 2.6s whenever prestage staged nothing, which is nearly every run, and it was the whole of other_ms's suspicious constancy (2610-2613ms regardless of tools_ms). Stop when two reads are identical, the rule the opener poll 40 lines below already applies. other_ms -53.7%, tools_ms flat. - prestage no longer navigates to the page it is already on, nor sleeps 0.35s before its first settle probe. - is_replay_boundary reasoned from the NAME alone, so x.com's composer textbox named "Post text" was ruled an irreversible send and truncated its replay to a bare navigate. Excluded by ROLE; first_unsafe_step now passes role through at all. Measured on this box: reach 100% (83% if onlinegdb's unverified exclusion is bogus), 0 false successes in ~155 runs, prestage tier-0/1 2702ms, other_ms -53.7%, infra flake 0/158, holdout 18/20. Criteria 2/4/9 need live writes and are untouched. Full evidence, including what did NOT work, in e2e/browser-v3/RESULTS_2026-08-06.md. Also drops the tracked electron/node_modules symlink pointing at another machine's Downloads folder; it is dangling on every other checkout and re-breaks the install on any stash or checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
28 lines
1.5 KiB
Bash
Executable File
28 lines
1.5 KiB
Bash
Executable File
#!/bin/bash
|
|
# Criterion 8: the frozen holdout, re-measured after the round's changes. Dry run, same stack.
|
|
# Where logs, profiles and run output go. Defaults to runs/ beside this harness; override with
|
|
# OSW_BENCH_DIR to keep multi-gigabyte browser profiles off the repo disk.
|
|
SP="${OSW_BENCH_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/runs}"
|
|
mkdir -p "$SP"
|
|
# The harness itself lives beside this script; SP is only for run OUTPUT.
|
|
HARNESS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# Repo root from this script's own location, so the harness works in any checkout.
|
|
TREE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
|
cd "$TREE" || exit 1
|
|
N="${N:-2}"
|
|
# Same TAG trap as c7_run.sh: this slices r6_be.log while stack.sh names its log after TAG, so a
|
|
# stack booted under any other tag hands every trial an empty slice and the sweep reports a
|
|
# confident score it never measured. Stale logs count as absent for the same reason.
|
|
LOGF="$SP/r6_be.log"
|
|
if [ ! -s "$LOGF" ] || [ -z "$(find "$LOGF" -mmin -5 2>/dev/null)" ]; then
|
|
echo "REFUSING: $LOGF missing, empty, or stale (>5min). The live backend is logging elsewhere." >&2
|
|
echo "Boot with TAG=r6: 'TAG=r6 $HARNESS/stack.sh up dry'" >&2
|
|
exit 2
|
|
fi
|
|
OUT="$SP/c8_r6.txt"
|
|
: > "$OUT"
|
|
echo "started $(date +%H:%M:%S), N=$N" >> "$OUT"
|
|
OSW_BASE=http://127.0.0.1:8326 OSW_LOG="$SP/r6_be.log" \
|
|
./backend/.venv/bin/python "$HARNESS/bench.py" holdout "$N" >> "$OUT" 2>&1
|
|
echo "########## DONE $(date +%H:%M:%S)" >> "$OUT"
|