diff --git a/CHANGELOG.md b/CHANGELOG.md index 43da99709..66e4994d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ - Add opt-in `GATEGUARD_FACT_FORCE_MAX_DENIALS`, a per-session denial cap: once that many first-touch denials have been issued, further new paths pass through (counted in `cap_allows`). Unset keeps the current behaviour; a malformed value is reported once on stderr and leaves the gate uncapped (#2755, thanks @SulimanAbdulrazzaq; closes #2608). - GateGuard loads its target-classification, transcript, search-matching and change-profile modules only on the calls that use them, and parses only the searches that name the target, so shell commands and edits of already checked files stay within a few milliseconds of the previous hook. - Add a development-only harness, `docker/gateguard-effectiveness/`, that runs real Claude sessions on six trapped coding tasks with no gate, the gate, the gate without the question meant to catch each trap, and a placebo gate, and grades the results with hidden graders (see `docs/gateguard/question-effectiveness.md`). +- Report that harness's pilot: 72 sessions in which every arm passed every trial, so it measured the gate's cost (about 45% more turns) and nothing about its effect. Add a second runner, `run-intent.js`, that measures effectiveness instead: the deciding fact exists only in a simulated user's hidden intent, sessions are multi-turn so the gate's question can actually be answered, scenarios are authored blind to the gate's question taxonomy, and each trial is classified into a hole worth fixing rather than checked against a declared target. - Add opt-in `GATEGUARD_METRICS` decision metrics (codes only, never paths, commands or content) and `scripts/gateguard-report.js` to summarise them. - Sensitive targets (`.env*`, keys and certificates, `credentials*`, `secrets.*`, `auth`/`security`/`payment(s)`/`billing`/`migrations` path segments, `.github/workflows/`) and files with more than one hard link get none of these allowances, judged on the path as written and its real location, and are gated in subagents too. `GATEGUARD_EXEMPT_GLOBS` still takes precedence. - Newly denied: a subagent's first touch of a sensitive or hard-linked file, and the first NotebookEdit of each notebook. Every other rule only removes denials and never returns an `allow` decision. See `skills/gateguard/SKILL.md`. diff --git a/docs/gateguard/question-effectiveness.md b/docs/gateguard/question-effectiveness.md index e863b52e4..e7f6c9582 100644 --- a/docs/gateguard/question-effectiveness.md +++ b/docs/gateguard/question-effectiveness.md @@ -153,4 +153,145 @@ folder. `--arms off,gate,main` compares the routing of two commits. ## Pilot results -Not run yet. +The pilot ran: 6 scenarios, 4 arms, 3 repetitions, 72 sessions, sonnet, hook +commit `bdf2dca6`. It returned no information at all. + +| Arm | Passed | Mean score | Median denials | Median turns | +| --- | ---: | ---: | ---: | ---: | +| `off` | 18/18 | 1.000 | 0 | 4.5 | +| `gate` | 18/18 | 1.000 | 1 | 6 | +| `minus-target` | 18/18 | 1.000 | 1 | 5 | +| `placebo` | 18/18 | 1.000 | 1 | 6 | + +Every arm passed every trial on every scenario. Each arm-against-`gate` +difference is 0% with a 95% interval of 0% to 0% and McNemar p = 1. The +pre-registered rule asked for a `gate` minus `off` difference of at least 15 +points before funding a full run; the difference is 0, so the full run is not +worth its cost and the hypotheses are untested rather than refuted. + +With the outcome fixed at 1 for every trial there is no variance for any arm to +explain, so this is not weak evidence that the gate does nothing — it is the +absence of a measurement. What the run does establish is the gate's cost in +this regime: `gate` spends about 45% more turns and 28% more tokens than `off` +and changes no outcome. + +### Why it measured nothing + +1. **Every trap hid its deciding fact in the repository.** `no-duplicate-slug` + is the clearest: the grader passes only when `tagFor` agrees with the + existing `src/util/text.js`, so an agent asked for a URL-safe slug greps, + finds `slugify` and reuses it. The existing test, the CSV's later rows and + the `.env` convention are the same shape. Agents are reliably good at + discovering repository facts, so the gate was measured where it cannot help. +2. **The asking channel was switched off.** Trials ran one-shot with "Work + autonomously in this repository and do not ask clarifying questions". A + fact-forcing gate was evaluated with no user to answer it, so the only + mechanism left was the agent restating facts to itself. +3. **The checks could not fail.** Each scenario declared `targetQuestions`, and + the suite asserted the gate asks one of them for the reference edits. Those + targets were chosen by reading the gate's own question taxonomy, so the + assertion held by construction. The scenario set was likewise built around + the six questions the gate already had, so the design could only confirm + existing coverage and never discover a missing question. + +The first `Limits` bullet anticipated (1) and (3) in part. It understated them: +the problem is not only that the author wrote both sides, but that the pass +criterion referred to the gate's own vocabulary. + +## Hidden-intent evaluation + +`docker/gateguard-effectiveness/scenarios-intent/` and `run-intent.js` answer a +different question, and are built so that a null result cannot be manufactured +by construction. + +**The deciding fact is only in the user's head.** Each scenario carries an +`intent.json` holding an ambiguity the repository cannot settle and the facts +that resolve it, each marked `decisive` or not. `audit-retention-purge` asks for +"our audit retention policy" to be applied; the workspace offers +`config/retention.json` with `sessionDays: 30` and no audit key, while the real +policy is 400 days and a soft mark. Nothing in the tree says so, so an agent +that does not ask can only guess, and the ungated arm cannot sit at the ceiling. + +**A simulated user answers only what is actually asked.** `user-sim.js` runs a +separate cheap model holding the facts, under instructions to answer the +question in front of it, never volunteer, and otherwise repeat a stonewall line. +It reports which fact ids it disclosed, so "asked something" and "asked the +deciding thing" are distinguishable. A judge whose reply cannot be parsed is +recorded as a judge failure rather than silently read as a stonewall. + +**Sessions are multi-turn and symmetric.** `session.js` joins turns with +`--session-id` and `--resume`. Every arm, gated or not, is told the user is +reachable, so the gate's contribution is whether it makes the agent ask the +deciding question — not whether asking was permitted. + +**Scenarios are authored blind.** `task.json` must not declare +`targetQuestions`; the loader rejects it. Which questions a scenario provokes is +observed from the run and reported. + +### Hole classes + +Each trial is classified from behaviour alone — whether the gate fired, whether +the agent asked, whether the deciding facts came out, and whether the outcome +was right. Nothing in the classification reads the taxonomy. + +| Class | Meaning | +| --- | --- | +| `working` | the deciding fact was obtained and the outcome is right | +| `lucky` | right outcome without asking: the scenario does not force the question | +| `follow-through-hole` | the fact was obtained and the outcome is still wrong | +| `targeting-hole` | the agent asked, but not for the deciding fact | +| `silence-hole` | the gate fired and the agent still did not ask | +| `coverage-hole` | a deciding ambiguity the gate never engaged, and the outcome is wrong | + +A scenario is reported as carrying no information when nothing varies across +arms — every trial passing or every trial failing. Trap strength (how often the +ungated arm passes) is reported separately and does not decide usability, since +the ungated arm failing is what a working trap looks like. + +Graders are checked before any session is billed: the start workspace and the +`naive` overlay must both score below 1 and `reference` must score 1, or the run +stops. + +### First hidden-intent run + +Four trials, `off` and `gate`, two repetitions, sonnet under test and haiku as +the user: + +| Arm | working | lucky | follow-through | targeting | silence | coverage | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `gate` | 1 | 0 | 0 | 0 | 1 | 0 | +| `off` | 0 | 0 | 0 | 0 | 0 | 2 | + +`gate` passed 1 of 2 and `off` 0 of 2, so the scenario separates the arms where +the original six did not. Four trials settle nothing about the effect size; what +they show is that the instrument now has a scale. + +The questions the gate actually raised on this scenario were `config-effect`, +`config-reader`, `data-schema`, `importers`, `no-plaintext-secrets`, +`public-api` and `quote-instruction` — seven questions, none of which asks about +a retention window or whether a delete is soft. In the one `silence-hole` trial +the gate fired and the agent proceeded without asking anyway. Both are +improvable gaps, and neither was expressible in the previous design. + +### Reproduce + +```bash +node docker/gateguard-effectiveness/run-intent.js --check-graders +node docker/gateguard-effectiveness/run-intent.js --out gg-intent --arms off,gate --dry-run +node docker/gateguard-effectiveness/run-intent.js --out gg-intent --model --allow-real-provider +node docker/gateguard-effectiveness/run-intent.js --out gg-intent --summarize +node --test tests/docker/gateguard-intent.test.js +``` + +`minus-target` is refused here: it needs the declared targets these scenarios +deliberately lack. Each trial is several billed turns plus one cheap judge call +per user turn. + +### Limits of this design + +- The simulated user is a model, so disclosure is not perfectly reproducible; + repetitions, not a single trial, carry the estimate. +- One scenario is not a corpus. The blind-authoring rule is only as good as the + independence of whoever writes the next ones. +- Telling every arm that the user is reachable raises asking across the board, + which is the right comparison but not the shipped default.