Three #3481 review findings still hold on main 8ce348fb:
- csv-export-delimiter's grader checks only the dialect (CRLF, semicolons,
quoting), so a constant '"x";"y"' export or one emitting the column
names scores 1. Add decoy-constant and decoy-keys overlays and pin that
each scores below 1 and that graderIsSound reports them; a scenario whose
decoy passes is not sound.
- The pilot runner (run.js) marks errored keys done, so they are never
retried, and armSummary counts provider-error and timed-out rows in
trials, passes and means. Pin the summary on valid rows only, and drive
run.js end to end: a provider error is retried on resume, keeps its own
row and transcript, and the summary reports it apart.
- Disabling run-intent.js's abort on an unobserved hook left every test
passing. Drive both runners end to end with a gated trial whose hook
never runs: each must stop and record nothing for it.
fake-claude.js gains FAKE_CLAUDE_SKIP_HOOK and FAKE_CLAUDE_PROVIDER_ERROR.
The intent end-to-end setup moves into syntheticIntentRunner, shared with
the existing resume test.
Against main: 2 intent failures (the decoy cases) and 2 pilot failures
(summary, retry). The two abort tests pass on main, since both aborts
exist; each fails when its runner's abort is disabled
(if (false && !row.hookObserved)).
silence-hole and coverage-hole are claims about the gate, and an ungated arm
never fires it, so classify labelled every ungated miss a coverage-hole by
construction. In the 48-trial run that padded the hole counts with five rows
that say nothing about the gate: coverage-hole read 7 when only 2 were the
gate's. Label an ungated miss unaided-miss, which is the baseline rate and is
excluded from the holes list.
The hidden-intent report printed bare counts. run.js reported Wilson intervals,
a bootstrap paired difference and an exact McNemar test, and that machinery was
lost in the new runner, so four trials rendered as a table that reads like a
result. On this run it is not one: gate 1/2 against off 0/2 gives intervals of
9%-91% and 0%-66%, a paired difference of -50% with a 95% interval spanning
-100% to 0%, and McNemar p = 1.000.
Reuse lib.js's pairedComparison and ai-eval-lib's wilson rather than adding a
second statistics path, and state the shortfall in words whenever a comparison
has fewer than twenty pairs: about 20 trials per arm are needed for 80% power on
a 40-point pass-rate difference at a two-sided 5% level, about 36 for 30 points,
and about 160 for the 15-point threshold the pilot pre-registered.
Mean turns and cost sit in the same table, because the only efficiency figure
this harness has produced points the other way: the gate roughly doubles turns
and cost per trial.
materializeTree passed the archive as an absolute path. GNU tar, which Git for
Windows bundles and puts first on PATH in Git Bash, reads an -f argument
containing a colon as a host:path remote spec and fails with "Cannot connect to
C: resolve failed". bsdtar, the tar in system32 that PowerShell resolves, has no
remote-spec parsing, so the fault only appeared under a bash-based shell - and
there it failed a committed test, not just a dry run.
Name the archive relatively and run tar with dest as its cwd, which extracts
identically under both tars and makes -C unnecessary. git archive keeps the
absolute -o path, which git handles on Windows.
Two tests, because the environment-dependent kind of fault needs both: one
injects a spawn and asserts no tar argument is absolute or drive-lettered, which
holds the invariant on any platform; the other prepends Git's bundled GNU tar to
PATH and extracts into a drive-letter destination, so the original failure is
reproducible from PowerShell too, where it was previously invisible.
The suite was the only file in tests/docker using node:test. The repository's
dominant convention, in 229 of 327 test files including the sibling
gateguard-effectiveness suite, is a local test() helper that counts passes and
sets process.exitCode, which is what tests/run-all.js reads.
Adopt that helper verbatim, along with the sibling's tempDir wrapper so the
scenario fixtures resolve symlinked temp paths. Assertions are unchanged.
Four defects found by running the repository's own checks and one more trial
against it.
ESLint rejected an unused `armSettings` import in intent-eval.js; the caller
supplies `armSettingsByName` already.
A timeout was lost whenever a later turn came back clean: `a || b ? c : false`
evaluates the condition first, so `timedOut` was recomputed from the new turn
instead of being kept.
Isolating each trial's config directory, to stop persisted sessions piling up
under the projects folder, puts the credentials out of reach. Every turn then
returns "Not logged in", and the harness graded that as a coverage-hole - an
auth failure recorded as a finding. Keep the real config directory and delete
only the session folders this run's work root created, and adopt lib.js's own
guards: a trial that is unauthenticated, or whose gated arm edited files with no
metrics line and no denial, stops the run instead of being recorded.
Record denials per turn as well as in total. The hook marks a file checked once
per session and --resume keeps one session id for the conversation, so a zero
after a non-zero is the latch rather than the gate staying quiet, and
silence-hole has to be read with that column in view.
Verified: a real trial authenticates, produces a graded silence-hole, and leaves
the projects directory at the same count it started with.
The existing harness cannot measure what it set out to measure. Every trap
hides its deciding fact in the repository, which agents reliably discover, and
trials run one-shot under "do not ask clarifying questions", so a fact-forcing
gate is evaluated with no user to answer it. Its scenarios also declare the
question the gate ought to raise, chosen by reading the gate's own taxonomy, so
the suite passes by construction and can never find a missing question.
Add a second runner that measures effectiveness instead of coverage:
- scenarios-intent/ keeps the deciding fact in intent.json, as a choice between
plausible conventions the repository cannot settle, so an agent that does not
ask can only guess and the ungated arm cannot sit at the ceiling.
- user-sim.js plays that user with a separate cheap model: it answers the
question in front of it, never volunteers, and reports which fact ids it
disclosed, so asking something and asking the deciding thing are told apart.
An unparseable judge is a recorded failure, not a silent stonewall.
- session.js runs multi-turn sessions joined by --session-id and --resume, and
tells every arm the user is reachable, so the comparison is symmetric and the
gate's contribution is whether it makes the agent ask the deciding question.
- coverage.js classifies each trial from behaviour alone into working, lucky, or
a follow-through, targeting, silence or coverage hole, and reports the
questions the gate actually raised rather than asserting declared ones.
- run-intent.js refuses minus-target, which needs targets these scenarios
deliberately lack, and checks every grader separates the start, the trap and
the reference before a session is billed.
A scenario counts as carrying no information only when nothing varies across
arms. Trap strength is reported separately: the ungated arm failing is what a
working trap looks like, so it must not make a scenario unusable.
haelyra