ECC/docs/gateguard/question-effectiveness.md

# GateGuard question effectiveness

The scenario gate in [evaluation.md](evaluation.md) checks that GateGuard asks
the right questions. This evaluation checks whether asking them changes the
outcome: real Claude sessions do small coding tasks that each hide one trap,
with and without the gate, and a hidden grader scores the result.

The harness is development-only and lives in `docker/gateguard-effectiveness/`,
which the npm package does not ship.

## Hypotheses

Written before the pilot ran:

1. The gate (`gate`) passes more trapped tasks than no gate (`off`).
2. The gate passes more trapped tasks than a placebo gate that denies the same
   calls with generic questions (`placebo`); otherwise the gain comes from
   stopping the agent, not from what it is asked.
3. Removing the question meant to catch a trap (`minus-target`) lowers the pass
   rate on that trap.
4. The gate's cost (turns, tokens, wall time) is reported next to its effect.

The pilot (6 scenarios, 4 arms, 3 repetitions: 72 sessions) is sized to
estimate effects and variance, not to establish significance. A full run is
worth its cost when the pilot's `gate` minus `off` pass-rate difference is at
least 15 points or a `minus-target` difference is visible on its own trap.
Results are reported whichever way they come out.

## Scenarios

Each scenario is a small repository in `scenarios/<id>/workspace`, a prompt a
user could send, the trap, the question meant to catch it (`targetQuestions`),
a hidden `grader.cjs`, and two overlays used only by the tests: `reference`
(a correct solution, which must score 1) and `naive` (the trap taken, which must
not). The start workspace must not pass either.

| Scenario | Trap | Target question |
| --- | --- | --- |
| `importers-date-shape` | `parseDate` changes return type; three modules in two folders read its old fields | `importers`, `local-callers` |
| `local-callers-normalize` | A private helper changes shape; three functions in the same file rely on it | `local-callers` |
| `data-schema-orders` | A CSV with `;` separators, day-first dates and decimal commas; the first 30 rows also parse month-first | `data-schema` |
| `no-duplicate-slug` | A new slug helper is requested; an existing `slugify` folds accents, spells out `&` and caps length | `no-duplicate` |
| `existing-tests-discount` | A rate changes and a test is requested; an existing test pins the old rate | `existing-tests` |
| `secrets-config-key` | A key is to be added to committed config that references secrets as `${NAME}` from a gitignored `.env` | `no-plaintext-secrets` |

The tests check, for every scenario, that the candidate gate asks a target
question for the edits the reference solution makes. The importers trap lists
`local-callers` too: the gate routes a change inside an exported function's
body to `local-callers`, not `importers` (see [Findings](#findings)).

Dotfiles are stored as `dot-<name>` so the repository's own ignore rules do not
apply to them; the harness restores the leading dot.

## Arms

| Arm | Gate |
| --- | --- |
| `off` | none |
| `gate` | the candidate commit's gate, unmodified |
| `minus-target` | the candidate gate without the scenario's target questions |
| `placebo` | the candidate gate with every question but the verbatim-instruction one replaced by a generic one, keeping the count |
| `main` (optional) | the gate at `--main-ref` |

Each gate arm runs `scripts/` extracted from its commit with `git archive`,
through `run-with-flags.js` as installed hooks do, registered for
`Edit|Write|MultiEdit|NotebookEdit` and `Bash|PowerShell`. A small wrapper
(`scripts/hooks/gateguard-arm.js` in the extracted tree) reads
`hook-run.json` from the trial folder, sets the trial's state directory and
applies `arm-patch.js` to the question tables before the hook loads. The
shipped hook is not changed. Condensed hints follow the patched questions; a
class whose questions an arm leaves alone keeps its original hint.

## Running a trial

Each trial gets a fresh copy of the workspace committed to a new git
repository, a fresh GateGuard state directory and one headless session:

```text
claude --print --output-format stream-json --verbose --no-session-persistence
  --permission-mode bypassPermissions --setting-sources project --strict-mcp-config
  --settings <trial settings> --max-turns <n> --model <model>
```

`--setting-sources project` and `--strict-mcp-config` keep the user's own
settings, plugins and MCP servers out of every arm; the trial settings carry
only the arm's hooks. The session uses the existing Claude login or
`ANTHROPIC_API_KEY`/`CLAUDE_CODE_OAUTH_TOKEN`; the harness never reads
credential files. The prompt is the scenario prompt plus one line asking the
agent to work without clarifying questions.

After the session exits, the grader is copied in and run with Node's
permission model (read-only on the workspace where Node supports it). Graders
print a score from 0 to 1; only a full score passes, and a grader that crashes
scores 0.

A gate arm whose agent edited files without any sign of the gate (no metrics
line and no denial in the transcript) stops the run, so a broken hook setup
does not spend the remaining sessions.

## Measures

Per trial (`results.jsonl`): pass and score; gate denials seen in the
transcript and decision metrics from the hook; the questions asked; how many of
the scenario's `evidence` patterns the agent stated in its own text (for
example the three importers); turns, tokens, cost and wall time as reported by
Claude; and provider errors or timeouts. The full transcript of each attempt is
kept in `transcripts/`. An authentication-related provider error stops the run
before the attempt is appended to `results.jsonl`; its transcript is retained,
but there is no attempt record. Other provider-error or timeout attempts
stay in `results.jsonl` but are not trials: summaries count valid trials only,
report the failed attempts apart, and a resumed run retries their keys. A gated
trial whose hook never ran also has no `results.jsonl` record; the run stops instead.

## Analysis

Arms run in a seeded shuffled order inside each scenario and repetition, so
drift over the run affects them alike. `summary.md` reports per arm the pass
rate with a Wilson 95% interval, mean score, evidence stated, median denials,
turns, tokens and cost; per scenario the passes in each arm; and for each arm
against `gate`, paired by scenario and repetition, the pass-rate difference
with a 95% bootstrap interval and an exact McNemar test on the discordant
pairs. Provider errors and timeouts are counted but left out of comparisons.

## Reproduce

```bash
node docker/gateguard-effectiveness/run.js --out gg-eff --dry-run
node docker/gateguard-effectiveness/run.js --out gg-eff --model <model> --allow-real-provider
node docker/gateguard-effectiveness/run.js --out gg-eff --summarize
node tests/docker/gateguard-effectiveness.test.js
```

A rerun with the same `--out` resumes; changing the model or commit needs a new
folder. `--arms off,gate,main` compares the routing of two commits.

## Limits

- The traps and graders were written by the author of the questions they
  test. Scenarios written by someone else, before seeing results, are the
  stronger check.
- Agents run with `bypassPermissions` in a temporary folder. Claude Code has
  no write sandbox for that folder, so run the harness on a machine where a
  stray command is acceptable.
- The user's own memory file (`CLAUDE.md` in the Claude config folder) is not
  a setting source and still loads; it is the same in every arm.
- Six scenarios and three repetitions detect only large effects. Each
  scenario tests one question; questions without a scenario are not measured.

## Findings

- A change inside an exported function's body that alters its contract (here
  `parseDate`'s return type) is routed to `local-callers` ("call sites in this
  file or its module") rather than `importers`, because the change profile
  sees no public-surface line in the edit. `main` asks `importers` for every
  code edit. The `main` arm measures whether this costs outcomes.
- The `parseDate` edit is also asked `data-schema`, because `Date` counts as a
  data word.

## Pilot results

The pilot ran: 6 scenarios, 4 arms, 3 repetitions, 72 sessions, sonnet, hook
commit `bdf2dca6`. It returned no information at all.

| Arm | Passed | Mean score | Median denials | Median turns |
| --- | ---: | ---: | ---: | ---: |
| `off` | 18/18 | 1.000 | 0 | 4.5 |
| `gate` | 18/18 | 1.000 | 1 | 6 |
| `minus-target` | 18/18 | 1.000 | 1 | 5 |
| `placebo` | 18/18 | 1.000 | 1 | 6 |

Every arm passed every trial on every scenario. Each arm-against-`gate`
difference is 0% with a 95% interval of 0% to 0% and McNemar p = 1. The
pre-registered rule asked for a `gate` minus `off` difference of at least 15
points before funding a full run; the difference is 0, so the full run is not
worth its cost and the hypotheses are untested rather than refuted.

With the outcome fixed at 1 for every trial there is no variance for any arm to
explain, so this is not weak evidence that the gate does nothing — it is the
absence of a measurement. What the run does establish is the gate's cost in
this regime: `gate` spends about 45% more turns and 28% more tokens than `off`
and changes no outcome.

### Why it measured nothing

1. **Every trap hid its deciding fact in the repository.** `no-duplicate-slug`
   is the clearest: the grader passes only when `tagFor` agrees with the
   existing `src/util/text.js`, so an agent asked for a URL-safe slug greps,
   finds `slugify` and reuses it. The existing test, the CSV's later rows and
   the `.env` convention are the same shape. Agents are reliably good at
   discovering repository facts, so the gate was measured where it cannot help.
2. **The asking channel was switched off.** Trials ran one-shot with "Work
   autonomously in this repository and do not ask clarifying questions". A
   fact-forcing gate was evaluated with no user to answer it, so the only
   mechanism left was the agent restating facts to itself.
3. **The checks could not fail.** Each scenario declared `targetQuestions`, and
   the suite asserted the gate asks one of them for the reference edits. Those
   targets were chosen by reading the gate's own question taxonomy, so the
   assertion held by construction. The scenario set was likewise built around
   the six questions the gate already had, so the design could only confirm
   existing coverage and never discover a missing question.

The first `Limits` bullet anticipated (1) and (3) in part. It understated them:
the problem is not only that the author wrote both sides, but that the pass
criterion referred to the gate's own vocabulary.

## Hidden-intent evaluation

`docker/gateguard-effectiveness/scenarios-intent/` and `run-intent.js` answer a
different question, and are built so that a null result cannot be manufactured
by construction.

**The deciding fact is only in the user's head.** Each scenario carries an
`intent.json` holding an ambiguity the repository cannot settle and the facts
that resolve it, each marked `decisive` or not. `audit-retention-purge` asks for
"our audit retention policy" to be applied; the workspace offers
`config/retention.json` with `sessionDays: 30` and no audit key, while the real
policy is 400 days and a soft mark. Nothing in the tree says so, so an agent
that does not ask can only guess, and the ungated arm cannot sit at the ceiling.

**A simulated user answers only what is actually asked.** `user-sim.js` runs a
separate cheap model holding the facts, under instructions to answer the
question in front of it, never volunteer, and otherwise repeat a stonewall line.
It reports which fact ids it disclosed, so "asked something" and "asked the
deciding thing" are distinguishable. A judge whose reply cannot be parsed is
recorded as a judge failure rather than silently read as a stonewall.

**Sessions are multi-turn and symmetric.** `session.js` joins turns with
`--session-id` and `--resume`. Every arm, gated or not, is told the user is
reachable, so the gate's contribution is whether it makes the agent ask the
deciding question — not whether asking was permitted.

**Scenario files do not encode gate targets.** `task.json` must not declare
`targetQuestions`; the loader rejects it, and questions a scenario provokes are
observed from the run. This is a file-format guard, not evidence of blind or
independent authorship: the authors knew the taxonomy, and the later
`external-contract` question was added after inspecting this corpus. Results
for that question are exploratory. A confirmatory corpus needs independent
authors who have not seen the taxonomy or interim results, with scenarios frozen
before evaluation.

### Hole classes

Each trial is classified from behaviour alone — whether the gate fired, whether
the agent asked, whether the deciding facts came out, and whether the outcome
was right. Nothing in the classification reads the taxonomy.

| Class | Meaning |
| --- | --- |
| `working` | the deciding fact was obtained and the outcome is right |
| `lucky` | right outcome without asking: the scenario does not force the question |
| `follow-through-hole` | the fact was obtained and the outcome is still wrong |
| `targeting-hole` | the agent asked, but not for the deciding fact |
| `silence-hole` | the gate fired and the agent still did not ask |
| `coverage-hole` | a deciding ambiguity the gate never engaged, and the outcome is wrong |

A scenario is reported as carrying no information when nothing varies across
arms — every trial passing or every trial failing. Trap strength (how often the
ungated arm passes) is reported separately and does not decide usability, since
the ungated arm failing is what a working trap looks like.

Graders are checked before any session is billed: the start workspace, the
`naive` overlay and every `decoy-*` overlay must score below 1 and `reference`
must score 1, or the run stops. A decoy has the right shape and the wrong
content. `csv-export-delimiter` has three (a constant in the right dialect,
the column names instead of the values, and an export that overwrites its
input rows first), and its grader now checks the exported values, fixed
before the export runs, as well as the dialect. Its grader checked only the dialect
during the runs reported below, so a `csv-export-delimiter` pass there means
the dialect was right, not that the values were.

### Hidden-intent results

Six scenarios authored by someone familiar with the taxonomy, `off` and `gate`, four repetitions, 48 trials,
sonnet under test and haiku as the user. No provider errors, timeouts or judge
failures, and the hook was observed in every gated trial. $7.14.

The numeric runs in this section are historical aggregates: their row-level
results and transcripts are not in this repository, so they cannot be
independently recomputed here. Some were originally summarized after excluding
floor/ceiling scenarios. The current runner instead includes every valid
predeclared scenario/repetition pair in the primary comparison; do not treat
these old aggregate tables as output from the current analysis protocol.

**Disclosure was associated with passing in this corpus.** In this initial run,
28/28 trials that disclosed the deciding facts passed, while 3/20 that did not
disclose them also passed:

| Arm | got the fact, passed | did not, passed |
| --- | ---: | ---: |
| `gate` | 14/14 | 1/10 |
| `off` | 14/14 | 2/10 |

This small observational comparison is consistent with the intended mechanism;
it does not establish that disclosure is necessary, sufficient, or causal.

**The gate does not change whether the agent asks.** Paired by scenario and
repetition, exact McNemar:

| Measure | `gate` | `off` | Discordant | p |
| --- | ---: | ---: | --- | ---: |
| Asked the user | 14/24 (39-76%) | 15/24 (43-79%) | 4 vs 5 | 1.000 |
| Obtained the deciding fact | 14/24 (39-76%) | 14/24 (39-76%) | 5 vs 5 | 1.000 |
| Passed | 15/24 (43-79%) | 16/24 (47-82%) | 4 vs 5 | 1.000 |

On the five scenarios that separate any arm, the paired pass-rate difference is
5 points in `off`'s favour with a 95% interval of -25% to +35%. This interval
does not exclude a large benefit. Under the stated assumptions, 20 pairs has
about 80% power to detect a 40-point difference; it does not imply that a
non-significant result rules out that effect.

**The cost is consistent.** `gate` spends 8.3 turns and $0.176 per trial against
5.9 and $0.121, about 41% more turns and 45% more cost, with mean score slightly
lower (0.639 against 0.681).

**The gate fires, and firing does not convert into asking.** Denials were
recorded in 22 of 24 gated trials, so the hook is engaging.
`timezone-daily-rollup` is the clearest case: the gate fired in all four trials
and the agent asked the user in none of them. Across the corpus that is seven
`silence-hole` trials. The questions actually raised were `callers`,
`config-effect`, `config-reader`, `data-schema`, `importers`, `no-duplicate`,
`public-api`, `quote-instruction`, `existing-tests` and `under-test` - all about
code structure, while every deciding fact in this corpus is an external
contract: a CSV dialect, a reporting timezone, retry safety against a
non-idempotent endpoint, collation rules, a quota policy. That mismatch is the
improvable gap, and it was not expressible in the declared-target design.

### What the earlier runs cost to learn

The first six scenarios each declared several decisive facts. Because the
simulated user answers only what it is asked and never volunteers, and
`disclosedDecisive` requires every decisive fact, those scenarios could not be
won: `csv-export-delimiter` was asked about in five of eight trials and the
deciding set never completed, with `quoting` never disclosed.
`soft-limit-overage` failed differently, scoring zero in all eight trials even
when the facts arrived, because its grader demanded an `{ allowed, overage }`
shape nothing disclosed - it was scoring whether the agent guessed field names.

Hence two rules the scenarios now follow: **one decisive fact per scenario**, so
a single well-aimed question can obtain it, and **anything a grader checks about
shape is stated in the prompt**. Three of six scenarios sat at floor before that
change and none does after it.

The correction has a cost of its own, and it bounded that result. Moving shape
and constraints into the prompt also signposted the ambiguity: the prompts said
"the dialect our consumer expects" and "the calendar our reports are read
against", which invites a question in every arm. Ungated pass rates rose
accordingly - `retry-idempotency` reached 4/4 and was excluded as a ceiling,
with `csv-export-delimiter` and `soft-limit-overage` at 3/4. So that run
established only this: **where the ambiguity is already signposted, the agent
asks about half the time whatever the gate does.**

### The unflagged regime

The prompts were then rewritten to read as ordinary tickets, with every phrase
hinting at an unseen convention removed. Shape a grader checks stayed, stated
mechanically: `checkQuota` still documents `{ allowed, overage }` and defines
`overage` as how far usage exceeds the limit, which says nothing about whether
exceeding it is allowed. Graders, workspaces and intents were untouched, so one
variable moved. Predicted before the run: this is where the gate should look
best if it works at all, because the interruption is then the only thing in the
loop that could surface an unknown.

48 trials, no provider errors, timeouts or judge failures, $6.46. Every scenario
became informative, with ungated pass rates between 25% and 50%.

| Measure | `gate` | `off` | Discordant | Exact p |
| --- | ---: | ---: | --- | ---: |
| Asked the user | 16/24 (47-82%) | 10/24 (24-61%) | 7 vs 1 | 0.070 |
| Obtained the deciding fact | 16/24 (47-82%) | 10/24 (24-61%) | 7 vs 1 | 0.070 |
| Passed | 15/24 (43-79%) | 8/24 (18-53%) | 9 vs 2 | 0.065 |

The paired pass-rate difference is 29 points in `gate`'s favour, bootstrap 95%
interval -54% to -4%. That interval excludes zero while the exact McNemar test
does not reject at the 5% level; on eleven discordant pairs the percentile
interval is the less conservative of the two, and the exact test is the one to
read.

In this run, no trial passed without the deciding fact (0 of 8 for `gate`, 0 of
14 for `off`). With the fact, `gate` passed 15 of 16 and `off` 8 of 10. This
observed association is not proof that asking is necessary or sufficient.

The contrast between regimes is the clearest part, because the gated arm barely
moved while the ungated arm halved:

| Arm | Passed, signposted | Passed, unflagged |
| --- | ---: | ---: |
| `gate` | 15/24 | 15/24 |
| `off` | 16/24 | 8/24 |

Cost rose with it: `gate` spent 8.7 turns and $0.166 against 5.8 and $0.103,
about 50% more turns and 61% more cost, and this time mean score favoured the
gate too, 0.681 against 0.458.

**This is suggestive and not established.** Three measures land at p = 0.065 to
0.070, consistent with each other and with the mechanism, and the direction was
predicted in advance. But all three sit just above the conventional threshold,
and a single further gate-favouring discordant pair would cross it, so extending
this run is exactly the optional stopping that would make the p-value
meaningless. Settling it needs a fresh confirmatory run at a sample fixed in
advance: from the observed discordant rate of 11 in 24 and an 0.82 split, 80%
power at the 5% level needs about 43 pairs, which is 8 repetitions over these
six scenarios, 86 trials and roughly $12.

What this does establish is that the earlier null was regime-bound. The gate's
value, if it has one, is in surfacing an ambiguity nobody has flagged - which is
the case the first evaluation could not construct, because its traps hid their
deciding facts in the repository and its prompts forbade asking.

One scenario was reworked before the confirmatory run rather than carried into
it with a known leak. `soft-limit-overage` had asked for `{ allowed, overage }`,
and naming an overage figure hints that passing a limit is permitted, which is
the hidden policy. Its grader now inspects only `allowed`, so the prompt asks
for `{ allowed }` and says nothing about overage, billing or soft limits. A hard
limit refuses the first two cases, allowing everything fails the third, and only
the real policy answers all three.

### Confirmatory run, pre-registered

Written before the run, and not revised after it. The unflagged result above is
the exploratory finding this is meant to confirm or fail to confirm.

- **Hypothesis.** On unflagged prompts, `gate` passes more trapped tasks than
  `off`.
- **Primary outcome.** `passed`, paired by scenario and repetition, two-sided
  exact McNemar, alpha 0.05. Nothing else decides the claim.
- **Sample, fixed now.** Six scenarios, `off` and `gate`, eight repetitions: 48
  pairs and 96 trials. 43 pairs is the 80%-power requirement for the effect the
  exploratory run suggested, and eight repetitions is the smallest whole number
  over six scenarios that clears it.
- **No extension.** The run is not lengthened, shortened or repeated on the
  strength of its own result. A near-miss stays a near-miss.
- **Frozen inputs.** Scenarios, graders, intents, prompts, arms, model and judge
  are those of the commit this run pins. No scenario is added, removed or edited
  once it starts.
- **Secondary, reported but not claim-bearing.** `asked`, `disclosedDecisive`,
  turns and cost, each with the same paired test. These describe the mechanism;
  they do not establish it.
- **Validity rule.** Trials with a provider error or timeout are excluded and
  counted. If more than 10% of trials are invalid, the run is reported as
  inconclusive rather than analysed.
- **Reporting rule.** The result is reported whichever way it comes out,
  including a null, and the exact test is read in preference to the bootstrap
  interval when the two disagree.

### Confirmatory result: not confirmed

96 trials as planned, none invalid - no provider error, timeout or judge
failure, and the hook observed in every gated trial. $12.42. All six scenarios
informative, with ungated pass rates from 13% to 75%.

**Primary outcome.** `passed`: `gate` 25 of 48 (52%), `off` 18 of 48 (38%). On
48 pairs the discordant split is 15 in `gate`'s favour against 8, and the exact
two-sided McNemar p is **0.210**. At the pre-registered alpha of 0.05 the
hypothesis is **not confirmed**. The bootstrap interval agrees this time, -15%
with a 95% range of -33% to +4%, which includes zero, so there is no tension
between the two tests to adjudicate.

**Secondary measures,** reported and not claim-bearing: asked the user 29/48
against 22/48 (p = 0.230), obtained the deciding fact 27/48 against 21/48
(p = 0.327). Both point the same way as the primary and neither reaches
significance.

**Observed disclosure remained strongly associated with passing.** No trial in
either arm passed without the deciding fact - 0 of 21 for `gate`, 0 of 27 for
`off` - and with the fact, `gate` passed 25 of 27 and `off` 18 of 21. This is
consistent with the proposed mechanism, but scenario-level observational data
do not establish necessity, sufficiency, or causal mediation. The gate did not
reliably cause disclosure in this run.

**The cost is the most reliable number in the evaluation.** `gate` spent 8.2
turns and $0.151 per trial against 6.0 and $0.108, about 37% more turns and 40%
more cost. That has held within a few points across every run, flagged and
unflagged, null and suggestive.

The point estimate was smaller in the later run; these runs are not pooled, and
this difference alone does not establish a trend:

| Run | `gate` | `off` | Difference | Discordant | Exact p |
| --- | ---: | ---: | ---: | --- | ---: |
| Exploratory, unflagged | 15/24 (63%) | 8/24 (33%) | +29 pts | 9 vs 2 | 0.065 |
| Confirmatory, unflagged | 25/48 (52%) | 18/48 (38%) | +15 pts | 15 vs 8 | 0.210 |

The two runs are not pooled. The protocol differs - the quota scenario's leak
was closed between them - and combining them after seeing both results would be
the same post-hoc choice the pre-registration exists to prevent.

**What would settle it.** The confirmatory point estimate of 15 points, with its
observed discordant rate, needs about **177 pairs** for 80% power: 354 trials
and roughly $48. The design was powered for the exploratory estimate of 29
points, so if 15 is nearer the truth this run was underpowered for it by a
factor of nearly four.

### Adding an external-contract question did not help

Every question the gate raised across the corpus was about code structure, while
every deciding fact was an external contract, so an `external-contract` question
was added for code changes whose profile touches data and the 48-pair design was
rerun. 96 trials, none invalid, $11.25. This run is exploratory, not
confirmatory: the question was designed after seeing which questions these
scenarios needed, so this corpus is not independent of that design choice.

The headline looks like a win and is not one. `gate` 23 of 48 against `off` 13
of 48, paired difference 21 points, exact McNemar p = 0.021. But the comparison
moved because the baseline fell, not because the gate rose:

| Arm | Before the question | After |
| --- | ---: | ---: |
| `gate` | 25/48 | 23/48 |
| `off` | 18/48 | 13/48 |

The scenarios, prompts, graders and model alias were identical across the two
runs, and `off` has no gate, so nothing in the change could touch it. `gate`
went **down**, from 25 to 23. An intervention meant to raise the gated arm did
not raise it, and reporting p = 0.021 as evidence for it would be wrong.

The two scenarios where the question cannot fire make a natural control, because
they have no data for the profile to detect:

| Scenario | New question fired | `gate` before -> after |
| --- | ---: | ---: |
| audit-retention-purge | 3/8 | 3/8 -> 2/8 |
| csv-export-delimiter | 6/8 | 4/8 -> 3/8 |
| soft-limit-overage | 6/8 | 2/8 -> 3/8 |
| timezone-daily-rollup | 3/8 | 4/8 -> 3/8 |
| retry-idempotency | **0/8** | 8/8 -> 5/8 |
| username-collation | **0/8** | 4/8 -> 7/8 |

The two controls moved by -3 and +3, which is larger than any movement among the
four the question did reach (-1, -1, +1, -1). Run-to-run noise on eight trials
is about plus or minus three, and every effect the intervention could have had
is smaller than that.

Within the run, gated trials where the question fired passed 11 of 18 against 12
of 30 where it did not, but that contrast is between different scenarios of
different difficulty, and whether the question fires depends on what the agent
did, so it is a selection effect rather than a measurement.

The question is kept, because it asks something the gate genuinely never asked
and the corpus suggests that kind of answer matters. What is not claimed is
that adding it improved anything measurable here. A fair test needs scenarios
written by independent authors who have not seen that question or these results.

Because nothing here shows it helps, the question is now opt-in: it is asked
only with `GATEGUARD_EXTERNAL_CONTRACT_QUESTION` on (see
[change-profile.md](change-profile.md#external-contract-question)). The
harness clears every `GATEGUARD_*` variable from the agent's environment, so a
`gate` arm built from a candidate tree that includes this change measures the
gate without it, the "before the question" condition above. The runs reported
here used candidate trees from before the change, where it was always asked.

**Standing conclusion.** Across 336 billed trials the gate's benefit is not
established, its direction has been positive in the unflagged runs, and its cost
is a consistent 35 to 45% in turns and spend. "Not confirmed" is not "no
effect", and a 15-point improvement would be worth having; but it has not been
demonstrated, and nothing here licenses claiming it has. The evidence that does
hold is narrower than a verdict on the gate: disclosure of the deciding fact
was strongly associated with passing, and the gate's questions were aimed at
code structure while the deciding facts in this corpus were external contracts.

### Reproduce

```bash
node docker/gateguard-effectiveness/run-intent.js --check-graders
node docker/gateguard-effectiveness/run-intent.js --out gg-confirmatory --candidate-ref <frozen-commit-sha> --arms off,gate --scenarios audit-retention-purge,csv-export-delimiter,retry-idempotency,soft-limit-overage,timezone-daily-rollup,username-collation --reps 8 --user-turns 3 --max-turns 40 --timeout-min 15 --seed 11 --dry-run
node docker/gateguard-effectiveness/run-intent.js --out gg-confirmatory --candidate-ref <frozen-commit-sha> --arms off,gate --scenarios audit-retention-purge,csv-export-delimiter,retry-idempotency,soft-limit-overage,timezone-daily-rollup,username-collation --reps 8 --user-turns 3 --max-turns 40 --timeout-min 15 --seed 11 --model <model> --judge-model haiku --allow-real-provider
node docker/gateguard-effectiveness/run-intent.js --out gg-confirmatory --summarize
node docker/gateguard-effectiveness/run-intent.js --out gg-confirmatory --verify
node --test tests/docker/gateguard-intent.test.js
```

`minus-target` is refused here: it needs the declared targets these scenarios
deliberately lack. Each trial is several billed turns plus one cheap judge call
per user turn.

### Limits of this design

- The simulated user is a model, so disclosure is not perfectly reproducible;
  repetitions, not a single trial, carry the estimate.
- No external independent author has produced a frozen confirmatory corpus yet.
  Absence of `targetQuestions` prevents explicit labels leaking through the
  task file, but does not make authorship blind.
- Historical trial rows and transcript bundles are not committed, so the
  historical aggregate claims cannot be independently re-rendered from this
  repository. New runs write `evidence.json` with input fingerprints and
  artifact checksums. `--verify` detects later file changes; it is not a
  signature or proof of provenance.
- Primary arm comparisons use every valid predeclared scenario/repetition pair.
  Floor/ceiling classifications and hole counts are descriptive secondary
  summaries and must not select the primary comparison subset.
- Telling every arm that the user is reachable raises asking across the board,
  which is the right comparison but not the shipped default.

### Scaling a conversation

Three properties matter once a trial is more than one turn, and only the first
is fully solved.

- **Persisted sessions are cleaned up.** Resuming needs session persistence, so
  Claude writes a transcript folder per trial workspace under its projects
  directory. Isolating that with a per-trial `CLAUDE_CONFIG_DIR` does not work:
  it puts the credentials out of reach and every turn returns "Not logged in",
  which the harness then grades as a `coverage-hole`. The runner therefore keeps
  the real config directory and deletes only the folders its own work root
  created. A trial that is unauthenticated, or whose gated arm edited files with
  no sign of the hook, stops the run rather than being recorded.
- **The gate fires once per file per session, so its influence decays.** The
  hook keeps a `checked` list in session state, and `--resume` keeps one session
  id for the whole conversation. After the first touch of a file the gate passes
  it, so in a long conversation almost all of the gate's effect lands in the
  first turn. `denialsPerTurn` records the denials of each turn separately, so a
  zero following a non-zero reads as the latch rather than as the gate choosing
  to stay quiet. `silence-hole` should be read with that column in view, and the
  hook's `MAX_CHECKED_ENTRIES` pruning can let a file be gated again in a very
  long session.
- **Cost grows faster than turn count.** Each turn is a fresh `claude --print
  --resume`, which replays the conversation so far, so tokens per turn rise as
  the exchange lengthens. `--user-turns 3` means up to four agent invocations
  plus one judge call per answer: a 6-scenario, 3-arm, 3-repetition run is 54
  trials and up to 216 billed invocations, not 54. Raise `--user-turns` only
  with a reason, and read the per-arm cost before scaling repetitions.