GateGuard evaluation

How to tell whether a change to the fact-forcing gate (scripts/hooks/gateguard-fact-force.js) improves it: ask the right question at the right time, deny less where a denial repeats work already done, and never let a security-critical target through.

scripts/dev/gateguard-eval.js replays a corpus of realistic sessions against the working-tree hook and against the hook at one or more git refs, and compares the decisions. tests/hooks/gateguard-scenarios.test.js replays the same corpus against the working tree only and fails if any step's decision changes.

What is measured

Every step of a scenario is one hook call. The corpus author labels each step with the decision the gate should make and why it matters:

Label	Meaning
`expect`	`deny` or `allow` (allow covers prior-search credit, sibling collapse, trivial edits, read-only shell commands and plain passes)
`mustDeny`	Security-critical: sensitive targets, bypass attempts, mutating or destructive first commands. Any allow is a bypass.
`redundant`	A denial here would repeat investigation the session already did (a scoped search that names the file, a same-turn sibling, a comment-only edit, a read-only first command).
`relevantQuestions`	For code targets, the question ids the change genuinely warrants (see the question table in design-notes.md).

Metrics per hook:

Denials: every denial costs the agent a round trip and context.
Redundant denials: denials on steps labelled redundant.
Must-deny bypasses: allows on steps labelled mustDeny. Must be 0 for the working tree.
Expectation mismatches: decisions that differ from expect. Must be 0 for the working tree; for a baseline they are the behaviour the branch changes.
Irrelevant questions asked: for denials of code targets, asked question ids that are not in relevantQuestions. Questions are read from the numbered list of a full denial, or from the phrases of a condensed one-line denial (issued after the first three denials of a session): the working tree's per-question phrases (condensedQuestionPhrase) and the fixed hint older hooks use. The condensed rows count the ones asked in condensed denials only.
Warranted questions not asked: relevantQuestions ids a code denial did not ask. Asking fewer questions must not drop the ones that matter.
Estimated denial tokens: characters of all denial reasons divided by 4.
Allows with a note: how the working tree avoided a denial.
Hook latency, fresh process: require() of the hook plus run(), timed inside a new Node process per step, which is how Claude Code runs a hook. Node start-up itself is excluded; it is the same for every hook. See Latency.
run() latency, warm: wall time of run() in the scenario's worker, after the hook module is loaded. Modules the hook loads lazily count on the step that first needs them.

The latency rows are the only non-deterministic metrics.

Each scenario runs in its own temp project (files, symlinks and hard links created from the fixture), with its own transcript, GATEGUARD_STATE_DIR, HOME and a clean environment (PATH, CLAUDE_PROJECT_DIR, plus the scenario's env), in a fresh worker thread, so module state and state files never leak between scenarios or hooks. A baseline hook is materialised from git show <ref>:scripts/hooks/gateguard-fact-force.js plus every file it requires relatively at that ref, into a temp tree that mirrors the repo layout.

Results

Working tree = branch gateguard-full; upstream/main = c70874fa; 4b02f669 = the pull request head before this round. Node 22, Linux.

Corpus: 20 scenarios, 184 steps.

Metric	working tree	upstream/main	`4b02f669`
Steps	184	184	184
Denials	132	157	132
Redundant denials	0	38	0
Must-deny bypasses	0	8	0
Expectation mismatches	0	55	0
Irrelevant questions asked	2	187	2
Irrelevant questions in condensed denials	2	111	2
Warranted questions not asked	1	57	1
Estimated denial tokens	25633	27796	25633
Allows with a credit note	12	0	12
Allows with a sibling note	11	0	11
Allows with a trivial-edit note	5	0	5
Hook latency p50, fresh process (ms)	21.18	10.58	25.11
Hook latency p95, fresh process (ms)	30.34	16.67	38.24
run() latency p50, warm (ms)	2.03	1.03	2.01
run() latency p95, warm (ms)	10.56	2.58	7.03

Scenario	Steps	Denials: working tree	Denials: upstream/main	Denials: `4b02f669`
docs-heavy-session	14	6	12	6
scaffold-module	10	3	10	3
bugfix-after-scoped-search	6	1	3	1
cold-writes	5	3	4	3
exported-api-edits	6	6	6	6
internal-only-edits	5	4	4	4
data-handling-edits	5	5	5	5
comment-only-edits	10	6	9	6
sensitive-targets	8	8	8	8
subagent-edits	6	3	1	3
first-shell-commands	20	11	17	11
windows-paths	8	5	7	5
bypass-search-filters	19	15	19	15
bypass-turn-and-batch	7	7	7	7
bypass-siblings	11	11	11	11
cap-with-sensitive	6	4	6	4
bypass-comment-context	22	21	19	21
exported-members	6	6	6	6
notebook-edits	6	4	0	4
hard-linked-targets	4	3	3	3

Reading the results

Against upstream/main:

16% fewer denials (157 to 132) and 8% fewer denial tokens, with every redundant denial in the corpus gone (38 to 0).
All eight must-deny bypasses closed: upstream/main lets a subagent edit src/auth/oauth.js, config/secrets.yaml and a hard-linked file without a question, never gates a MultiEdit call that names its file in tool_input.file_path (the tool's own shape), so .env, config/.env.local from a subagent and a first-touch code file all pass, and never gates NotebookEdit, so notebooks under auth/ and payments/ pass from the parent and from a subagent.
Irrelevant questions drop from 187 to 2 and warranted-but-unasked from 57 to 1: edits without a public-surface line ask for local call sites instead of importers, members of exported interfaces, enums, export lists, dataclasses and pub enums keep the importer questions, the data-schema question is asked only when the change touches data, and condensed denials name the same questions as full ones.
Denials rise only where they should: the subagent's first touch of a sensitive or hard-linked file, MultiEdit calls, NotebookEdit calls, the first mutating shell command after a read-only one (upstream/main spends its once-per-session routine gate on ls), and the first code-changing edit after a comment-only one (the comment edit no longer spends the file's first touch).

Against 4b02f669 (the pull request head before this round): the same decisions, questions and notes on every step; only latency changes (see Latency). bypass-search-filters includes a search whose directory-qualified include (rg -g 'src/**' sweep .) keeps the target in lib/ out of the search; it must be denied, and the control that searches the target's own directory is credited.

Four scenarios come from security reviews of this change. bypass-comment-context holds 21 must-deny steps: comment-looking edits that change code in their file (a comment line after a continued C macro, a line break dropped so the next line joins a comment, lines inside a template literal, a docstring and a heredoc, a snippet that starts inside a string, replace_all reaching a string), directive comments (shebang, encoding cookie, # type:, # nosec, @ts-expect-error, eslint-disable, //go:build, //go:embed, a cgo preamble, a Rust doc test), MultiEdit calls naming their file once, and rg -z as a first shell command; plus one comment edit below closed templates and regexes that should still pass. Before the fixes the branch let 20 of those 21 through (every edit and rg -z as a trivial or read-only pass, and the three MultiEdit calls); upstream/main lets the three MultiEdit calls through. exported-members holds six member edits whose container declaration sits outside the snippet; before the fixes the branch asked for local call sites on the five public ones (5 irrelevant questions, 10 warranted ones not asked). notebook-edits covers a cold NotebookEdit, its retry, a credited test notebook, a comment-only cell and sensitive notebooks in the parent and a subagent; every earlier hook allows all six calls. hard-linked-targets edits a second name of a file after a search that names it, makes a comment-only change to another linked file and edits a linked file from a subagent, with a single-link control; before the fix the branch allowed all three (credit, trivial pass, subagent bypass).

What the working tree still gets wrong, by the corpus's own labels:

Sensitive targets are never profiled, so a sensitive code Write always asks the data-schema question (symlinked-auth-first, symlinked-auth-sibling: the 2 irrelevant questions).
NotebookEdit and hard-linked targets always get the full questions; the corpus labels all four as warranted for them, so this costs nothing in the table, but a notebook cell that handles no data still gets the data-schema question.
The C edit that turns a declaration into a comment continuation (c-comment-continuation) has no data words, so the condensed hint no longer mentions data schemas; the corpus labels that question as warranted (the 1 unasked one).
Latency: a first touch costs more than on upstream/main because it reads the transcript, matches the searches that name the target, reads the target file and profiles the change (fresh-process p50 21.2 ms against 10.6 ms, p95 30.3 ms against 16.7 ms on this corpus, where every step is a first touch or a gate). Shell commands and repeat edits stay within about 1 to 2.5 ms of main; see Latency. Checking a comment-only edit reads and scans the target file, which costs up to about 50 ms for a 1 MiB file (the largest one read).

Reproduce

git fetch upstream main
node scripts/dev/gateguard-eval.js --markdown
node scripts/dev/gateguard-eval.js --markdown --baseline upstream/main --baseline 4b02f669
node scripts/dev/gateguard-eval.js --json > gateguard-eval.json
node scripts/dev/gateguard-eval.js --baseline upstream/main --sarif gateguard-eval.sarif
node tests/hooks/gateguard-scenarios.test.js

--baseline <ref> takes any ref and can be repeated (default upstream/main); pass the pull request head (4b02f669 above) to compare against it. --corpus <dir> points at another scenario directory. The script exits non-zero when the working tree has a mismatch, a must-deny bypass, an explicit allow decision or a thrown error. --sarif <file> also writes those working-tree failures as SARIF 2.1.0 (bypasses, explicit allows and errors as error, other mismatches as warning) with each hook's totals in the run properties. Timing is left out, so the file is identical across reruns of the same tree and corpus.

Latency

The fresh-process pass replays every scenario again, with a new state directory and transcript, and runs each step with node -e <probe> <hook> <payload>. Its decisions must match the worker's; a step decided differently fails the run. --no-cold skips the pass.

Per call type: 25 fresh processes each, p50 of require() plus p50 of run() in ms, Node 22 on Linux, with a 20-search turn and a 50-function target file:

Call	this branch	`4b02f669`	`main`
Shell command, first of session	10.2	19.2	8.7
Shell command, routine gate already passed	12.3	21.3	11.6
Edit of a file already checked	11.1	20.6	8.7
First-touch edit, denied	22.4	29.7	9.6
First-touch edit, comment-only pass	25.7	38.5	9.5

Shell commands and repeat edits, most calls in a session, now load only the modules they use and stay within about 1 to 2.5 ms of main. A first touch reads the transcript, parses the turn's searches that name the target, reads the target file and profiles the change; main does none of these. Each hook call is a new Node process, so the code a call runs is also compiled on that call, and that compilation, not the size of the files, is most of the remaining difference. Node's module compile cache does not reduce it: it caches top-level code, and the gate's functions are compiled on first call.

Corpus format

One JSON file per scenario in tests/fixtures/gateguard-scenarios/:

name, description;
files (project-relative path to content), optional dirs, symlinks (link path to target, relative to the link) and hardlinks (link path to an existing project file, both project-relative), created in a temp project;
optional root (a fixed project path, used for Windows-style paths, where nothing is created on disk) and env (extra environment variables);
steps, in order. Each step has an id, an optional note, the transcript records appended to the session transcript before the call (human prompts with promptId, assistant tool_use records with message.id, tool_result records, compaction boundaries, and the pending call's own tool_use record), the hook payload, and the labels above.

{{root}} and {{transcript}} in any string are replaced with the temp project and transcript paths. Transcripts are the same for every hook: a step's call is assumed to succeed eventually, so the next step's records start with its tool_result.

A scenario with symlinks or hardlinks is skipped (and reported) where the links cannot be created, such as Windows without the symlink privilege or a file system without hard links.

Limits

The corpus is synthetic. Sessions were written to resemble real ones, not sampled from them, so the denial counts show the direction and size of a change, not rates to expect in the field. Opt-in metrics (GATEGUARD_METRICS=1 and scripts/gateguard-report.js) measure real sessions.
The labels are the corpus author's judgement, made with the design in mind. redundant and relevantQuestions in particular are opinions: a different author could call the README or CHANGELOG edits after a docs-wide search redundant, or leave out the data-schema question for c-comment-continuation. Zero redundant denials means the working tree handles the cases this author judged redundant, not that it never repeats a question.
Question relevance is read from denial text. Condensed denials are mapped by phrase, and a hedged phrase ("data schemas if any") counts as asked.
A baseline sees the same transcript as the working tree even where it would have denied an earlier call; retries are not modelled, and a denied call's retry is assumed to pass.
The routine shell gate is once per session, so a baseline that spent it on a read-only command shows an allow on a later mutating command. That counts as a mismatch, not a bypass (mustDeny is set only where the baseline's own gate should also deny).
Windows-style paths run lexically on non-Windows hosts; realpath and symlink behaviour for them is not exercised.
Latency is measured in process for run() only (no Node start-up), on the machine that ran the evaluation.

15 KiB Raw Blame History