mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-09-10 11:47:43 +02:00
ciregenzandClaude Opus 5
3c50359146
Six of the eleven defects here were in the MEASUREMENT, not the product, and they were wrong in both directions. Harness, all of which silently produced wrong numbers: - coverage.py preflight refused every sweep on a box holding exactly one backend: stack.sh's supervisor is a `bash -c` quoting the whole uvicorn line, so it carries both "-m uvicorn backend.main" AND the venv python path. Discriminate on POSITION. - stack.sh status reported 2 backends over 1 and 0 webpack over a live dev server (webpack retitles its process). A status check whose job is preventing a second stack, failing in the direction that lets one land. - c7_run.sh/c8_run.sh slice r6_be.log while stack.sh names logs by TAG: a stack under any other tag hands every trial an empty slice and the sweep reports a confident 0/108. Now refuses loudly; it caught this exact mistake on first use. - "Browser command timed out" was bucketed infra. It is ONE command blowing its own budget, not a dead webview: all 4 such rows were BrowserFindComposer at exactly its 30s cap, every run completed after, zero card-gone markers in the whole log. Filed as infra it read as 11.8% flake AND lifted holdout reach 70% -> 84%. - api_retry / rate_limit_error now grade as infra. A provider 429 storm turned clean 15-21s exclusions into 188s product_no_composer rows. - bench.py prints reach BOTH ways when a row is UNVERIFIED. An exclusion resting on the agent's own word quietly flatters the score, and coverage.py's own instruction to confirm it by hand goes unread (I quoted a 100% that excluded onlinegdb). Timing was measuring 0.2% of the run: prestage completes BEFORE metrics_started_at, so other_ms was 25ms of a 12700ms median while prestage (4146ms, ~61%) sat in no bucket at all. prestage_ms/task_ms are now recorded; total_ms is deliberately NOT redefined, which would invalidate every before/after already taken against it. Product: - find_composer rejected ACE/CodeMirror-5/Monaco composers. Their input is a ~1x1 offscreen textarea that paints into a sibling div, so it can never pass a size gate. Accept it when a VISIBLE ancestor is composer-sized; honeypots stay out because the input itself must not be display:none/visibility:hidden/opacity:0. anon reach 80% -> 100%, holdout 89% -> 90%, p95 38.6s -> 9.7s. - the composer poll slept a blind 0+1.2+1.4 = 2.6s whenever prestage staged nothing, which is nearly every run, and it was the whole of other_ms's suspicious constancy (2610-2613ms regardless of tools_ms). Stop when two reads are identical, the rule the opener poll 40 lines below already applies. other_ms -53.7%, tools_ms flat. - prestage no longer navigates to the page it is already on, nor sleeps 0.35s before its first settle probe. - is_replay_boundary reasoned from the NAME alone, so x.com's composer textbox named "Post text" was ruled an irreversible send and truncated its replay to a bare navigate. Excluded by ROLE; first_unsafe_step now passes role through at all. Measured on this box: reach 100% (83% if onlinegdb's unverified exclusion is bogus), 0 false successes in ~155 runs, prestage tier-0/1 2702ms, other_ms -53.7%, infra flake 0/158, holdout 18/20. Criteria 2/4/9 need live writes and are untouched. Full evidence, including what did NOT work, in e2e/browser-v3/RESULTS_2026-08-06.md. Also drops the tracked electron/node_modules symlink pointing at another machine's Downloads folder; it is dangling on every other checkout and re-breaks the install on any stash or checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
41 lines
2.3 KiB
Bash
Executable File
41 lines
2.3 KiB
Bash
Executable File
#!/bin/bash
|
|
# Criteria 1, 4, 5, 6, 7 and 9 in one pass: the known suite at N=12 = 108 site-runs, which is the
|
|
# >=100 sample criterion 7 asks for, while the same trials carry the reach, timing and skill data.
|
|
#
|
|
# Dry run only. The backend must be up with OSW_SENDSCRIPT_DRYRUN=1 (`stack.sh up dry`), so the
|
|
# irreversible click is refused in the backend, not by a flag this script sets and does not own.
|
|
# Where logs, profiles and run output go. Defaults to runs/ beside this harness; override with
|
|
# OSW_BENCH_DIR to keep multi-gigabyte browser profiles off the repo disk.
|
|
SP="${OSW_BENCH_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/runs}"
|
|
mkdir -p "$SP"
|
|
# The harness itself lives beside this script; SP is only for run OUTPUT.
|
|
HARNESS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
# Repo root from this script's own location, so the harness works in any checkout.
|
|
TREE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
|
cd "$TREE" || exit 1
|
|
|
|
N="${N:-12}"
|
|
# This sweep slices r6_be.log, but stack.sh names its log after TAG ($SP/${TAG}_be.log). Boot with
|
|
# any other TAG and bench.py reads a file that does not exist: every slice comes back empty, every
|
|
# trial grades as a silent miss, and the run reports a confident 0/108 that measured nothing. That
|
|
# is the half-booted-stack failure this directory keeps re-learning, so check it instead of trusting
|
|
# the operator to remember. Stale counts too: a leftover log from an earlier sweep passes an
|
|
# existence test while the live backend writes somewhere else entirely.
|
|
LOGF="$SP/r6_be.log"
|
|
if [ ! -s "$LOGF" ]; then
|
|
echo "REFUSING: $LOGF missing or empty. Boot with TAG=r6: 'TAG=r6 $HARNESS/stack.sh up dry'" >&2
|
|
exit 2
|
|
fi
|
|
if [ -z "$(find "$LOGF" -mmin -5 2>/dev/null)" ]; then
|
|
echo "REFUSING: $LOGF has not been written in 5min, so the live backend is logging elsewhere." >&2
|
|
echo "Boot with TAG=r6: 'TAG=r6 $HARNESS/stack.sh up dry'" >&2
|
|
exit 2
|
|
fi
|
|
OUT="$SP/c7_r6.txt"
|
|
: > "$OUT"
|
|
echo "started $(date +%H:%M:%S), N=$N, log=$SP/r6_be.log" >> "$OUT"
|
|
OSW_BASE=http://127.0.0.1:8326 OSW_LOG="$SP/r6_be.log" \
|
|
./backend/.venv/bin/python "$HARNESS/bench.py" known "$N" >> "$OUT" 2>&1
|
|
echo "########## DONE $(date +%H:%M:%S)" >> "$OUT"
|
|
echo "backend restarts during the sweep: $(grep -c 'BACKEND RESTARTED' "$SP/r6_be.log")" >> "$OUT"
|