- csv-export-delimiter's grader adds a fourth check: the data rows are
the exported values, with an optional header (the hidden intent says a
header is fine either way). Reference scores 1; both decoys 0.75.
- graderIsSound grades every decoy-* overlay and is sound only when each
scores below 1; --check-graders and the pre-billing check print the
decoy scores through formatGraderScores.
- isValidTrial and completedTrialKeys move to lib.js; coverage.js and
run-intent.js use them from there.
- run.js resumes with completedTrialKeys, so a provider-error or timed-out
key is retried, and writes one transcript per attempt
(<key>__<attemptId>.jsonl, recorded on the row) so a retry never
overwrites the failed attempt's record.
- armSummary computes trials, passes and means over valid rows and counts
the failed attempts apart as providerErrors.
Docs: question-effectiveness.md states the decoy rule and that a
csv-export-delimiter pass in the reported runs reflects the dialect only;
its Measures section and the harness README describe retries, per-attempt
transcripts and the hookless-trial stop.
Three #3481 review findings still hold on main 8ce348fb:
- csv-export-delimiter's grader checks only the dialect (CRLF, semicolons,
quoting), so a constant '"x";"y"' export or one emitting the column
names scores 1. Add decoy-constant and decoy-keys overlays and pin that
each scores below 1 and that graderIsSound reports them; a scenario whose
decoy passes is not sound.
- The pilot runner (run.js) marks errored keys done, so they are never
retried, and armSummary counts provider-error and timed-out rows in
trials, passes and means. Pin the summary on valid rows only, and drive
run.js end to end: a provider error is retried on resume, keeps its own
row and transcript, and the summary reports it apart.
- Disabling run-intent.js's abort on an unobserved hook left every test
passing. Drive both runners end to end with a gated trial whose hook
never runs: each must stop and record nothing for it.
fake-claude.js gains FAKE_CLAUDE_SKIP_HOOK and FAKE_CLAUDE_PROVIDER_ERROR.
The intent end-to-end setup moves into syntheticIntentRunner, shared with
the existing resume test.
Against main: 2 intent failures (the decoy cases) and 2 pilot failures
(summary, retry). The two abort tests pass on main, since both aborts
exist; each fails when its runner's abort is disabled
(if (false && !row.hookObserved)).
- questionIdsFor and condensedHintFor take an optional
{ externalContract: true }; without it a code change that handles data
asks data-schema only, and an unknown profile gets the original four
questions and condensed hint.
- The hook reads GATEGUARD_EXTERNAL_CONTRACT_QUESTION with the same
spellings as GATEGUARD_METRICS and passes it to the full denial, the
condensed denial and the metrics record, so all three agree.
- The effectiveness harness's arm patch forwards the option, so an arm
never drops it silently.
Docs: change-profile.md gains an External-contract question section and
qualifies "byte-identical to the fixed four questions"; design-notes.md,
SKILL.md (switch table, data rule, word-for-word set) and CHANGELOG say
the fifth question is opt-in; question-effectiveness.md and the harness
README note that the runner clears GATEGUARD_* variables, so a gate arm
from a current tree measures the gate without the question.
skill-triggers@1.json: registryDigest refreshed for the SKILL.md body
change; the description and triggers are unchanged.
soft-limit-overage asked for { allowed, overage }. Naming an overage figure
hints that passing a limit is permitted, which is precisely the hidden policy,
and it showed: that scenario was one of the two with the highest ungated pass
rate in the signposted run. The grader now inspects only allowed, so the prompt
asks for { allowed } and mentions neither overage nor billing nor soft limits. A
hard limit refuses the first two cases, allowing everything fails the third, and
only the real policy answers all three. --check-graders still separates start,
trap and reference on all six.
The analysis is registered in the doc before the run rather than chosen after
it: passed as the single primary outcome, paired exact McNemar at alpha 0.05, a
sample of 48 pairs fixed in advance, no extension on the strength of the result,
frozen inputs, secondary measures explicitly not claim-bearing, a validity rule
for provider errors, and a commitment to report a null.
The previous prompts signposted what they withheld - "the dialect our consumer
expects", "the calendar our reports are read against", "retrying only the
failures that are safe to repeat" - which invited a question in every arm and
pushed ungated pass rates to 3/4 and 4/4. That bounded the result to a regime
where the agent was already told something was missing.
Each prompt now reads as an ordinary ticket. Shape a grader checks stays, since
removing it would score field-name guessing instead of the hidden decision, but
it is stated mechanically: checkQuota still documents { allowed, overage } and
defines overage as how far usage exceeds the limit, which does not say whether
exceeding it is allowed, so the soft-limit policy stays hidden.
Graders, workspaces and intents are unchanged, so this isolates one variable:
whether the gate changes behaviour when nothing cues the agent that a convention
it cannot see decides the task.
Three of six scenarios sat at floor in the 48-trial run, and the run data says
the cause was the scenarios, not the agents.
csv-export-delimiter was asked about in 5 of 8 trials yet disclosedDecisive
never fired: the simulated user disclosed delimiter five times and lineending
twice, and quoting never. disclosedDecisive requires every decisive fact, and a
user who answers only what is asked and never volunteers cannot deliver three of
them to one question. username-collation failed the same way, with only case
ever disclosed.
soft-limit-overage failed differently and worse: the facts reached the agent in
4 of 8 trials and every trial still scored 0, because the grader demanded an
{ allowed, overage } shape that nothing disclosed. It was scoring whether the
agent guessed field names, not whether it learned the policy.
So each scenario now carries exactly one decisive fact, which is one hidden
decision a single well-aimed question can obtain, and anything a grader checks
about shape is stated in the prompt. The three working scenarios are changed the
same way, so disclosedDecisive means the same thing across the corpus: their
second and third decisive facts move into the prompt or become non-decisive
colour.
silence-hole and coverage-hole are claims about the gate, and an ungated arm
never fires it, so classify labelled every ungated miss a coverage-hole by
construction. In the 48-trial run that padded the hole counts with five rows
that say nothing about the gate: coverage-hole read 7 when only 2 were the
gate's. Label an ungated miss unaided-miss, which is the baseline rate and is
excluded from the holes list.
One scenario was the whole blind corpus, which cannot show whether the gate
generalises. These five were written from ambiguities a developer meets, not
from the gate's question list, and each keeps its deciding fact out of the
repository while leaving a plausible wrong answer in plain sight:
- csv-export-delimiter: a comma-parsing importer sits next to an export whose
consumer needs semicolons, CRLF and every field quoted.
- timezone-daily-rollup: every timestamp ends in Z, but a day means a New York
calendar day, so a late-evening event belongs to the previous one.
- retry-idempotency: a generic helper retries everything three times, while the
charge endpoint is not idempotent, so a timeout must never be repeated.
- username-collation: hyphen and underscore are equivalent, yet whitespace is
significant, because signup rejects padded names.
- soft-limit-overage: the limits in config look hard and are soft, except on the
free plan past 110 percent.
Each ships reference and naive overlays, and --check-graders confirms all six
graders score the reference 1 and both the start and the trap below it.
The hidden-intent report printed bare counts. run.js reported Wilson intervals,
a bootstrap paired difference and an exact McNemar test, and that machinery was
lost in the new runner, so four trials rendered as a table that reads like a
result. On this run it is not one: gate 1/2 against off 0/2 gives intervals of
9%-91% and 0%-66%, a paired difference of -50% with a 95% interval spanning
-100% to 0%, and McNemar p = 1.000.
Reuse lib.js's pairedComparison and ai-eval-lib's wilson rather than adding a
second statistics path, and state the shortfall in words whenever a comparison
has fewer than twenty pairs: about 20 trials per arm are needed for 80% power on
a 40-point pass-rate difference at a two-sided 5% level, about 36 for 30 points,
and about 160 for the 15-point threshold the pilot pre-registered.
Mean turns and cost sit in the same table, because the only efficiency figure
this harness has produced points the other way: the gate roughly doubles turns
and cost per trial.
materializeTree passed the archive as an absolute path. GNU tar, which Git for
Windows bundles and puts first on PATH in Git Bash, reads an -f argument
containing a colon as a host:path remote spec and fails with "Cannot connect to
C: resolve failed". bsdtar, the tar in system32 that PowerShell resolves, has no
remote-spec parsing, so the fault only appeared under a bash-based shell - and
there it failed a committed test, not just a dry run.
Name the archive relatively and run tar with dest as its cwd, which extracts
identically under both tars and makes -C unnecessary. git archive keeps the
absolute -o path, which git handles on Windows.
Two tests, because the environment-dependent kind of fault needs both: one
injects a spawn and asserts no tar argument is absolute or drive-lettered, which
holds the invariant on any platform; the other prepends Git's bundled GNU tar to
PATH and extracts into a drive-letter destination, so the original failure is
reproducible from PowerShell too, where it was previously invisible.
Four defects found by running the repository's own checks and one more trial
against it.
ESLint rejected an unused `armSettings` import in intent-eval.js; the caller
supplies `armSettingsByName` already.
A timeout was lost whenever a later turn came back clean: `a || b ? c : false`
evaluates the condition first, so `timedOut` was recomputed from the new turn
instead of being kept.
Isolating each trial's config directory, to stop persisted sessions piling up
under the projects folder, puts the credentials out of reach. Every turn then
returns "Not logged in", and the harness graded that as a coverage-hole - an
auth failure recorded as a finding. Keep the real config directory and delete
only the session folders this run's work root created, and adopt lib.js's own
guards: a trial that is unauthenticated, or whose gated arm edited files with no
metrics line and no denial, stops the run instead of being recorded.
Record denials per turn as well as in total. The hook marks a file checked once
per session and --resume keeps one session id for the conversation, so a zero
after a non-zero is the latch rather than the gate staying quiet, and
silence-hole has to be read with that column in view.
Verified: a real trial authenticates, produces a graded silence-hole, and leaves
the projects directory at the same count it started with.
The existing harness cannot measure what it set out to measure. Every trap
hides its deciding fact in the repository, which agents reliably discover, and
trials run one-shot under "do not ask clarifying questions", so a fact-forcing
gate is evaluated with no user to answer it. Its scenarios also declare the
question the gate ought to raise, chosen by reading the gate's own taxonomy, so
the suite passes by construction and can never find a missing question.
Add a second runner that measures effectiveness instead of coverage:
- scenarios-intent/ keeps the deciding fact in intent.json, as a choice between
plausible conventions the repository cannot settle, so an agent that does not
ask can only guess and the ungated arm cannot sit at the ceiling.
- user-sim.js plays that user with a separate cheap model: it answers the
question in front of it, never volunteers, and reports which fact ids it
disclosed, so asking something and asking the deciding thing are told apart.
An unparseable judge is a recorded failure, not a silent stonewall.
- session.js runs multi-turn sessions joined by --session-id and --resume, and
tells every arm the user is reachable, so the comparison is symmetric and the
gate's contribution is whether it makes the agent ask the deciding question.
- coverage.js classifies each trial from behaviour alone into working, lucky, or
a follow-through, targeting, silence or coverage hole, and reports the
questions the gate actually raised rather than asserting declared ones.
- run-intent.js refuses minus-target, which needs targets these scenarios
deliberately lack, and checks every grader separates the start, the trap and
the reference before a session is billed.
A scenario counts as carrying no information only when nothing varies across
arms. Trap strength is reported separately: the ungated arm failing is what a
working trap looks like, so it must not make a scenario unusable.
codex login binds its callback server to container loopback, so the host
browser redirect to localhost:1455 is refused. Publish 127.0.0.1:1455 on
the opt-in networked service only and bridge it to loopback with a small
node relay. Document the flow, fresh-state retry, and token lifetime in
the hardened-harness skill.
Follow-up to #2625.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
haelyra