montjeffrey
63d812ed97
Four defects found by running the repository's own checks and one more trial against it. ESLint rejected an unused `armSettings` import in intent-eval.js; the caller supplies `armSettingsByName` already. A timeout was lost whenever a later turn came back clean: `a || b ? c : false` evaluates the condition first, so `timedOut` was recomputed from the new turn instead of being kept. Isolating each trial's config directory, to stop persisted sessions piling up under the projects folder, puts the credentials out of reach. Every turn then returns "Not logged in", and the harness graded that as a coverage-hole - an auth failure recorded as a finding. Keep the real config directory and delete only the session folders this run's work root created, and adopt lib.js's own guards: a trial that is unauthenticated, or whose gated arm edited files with no metrics line and no denial, stops the run instead of being recorded. Record denials per turn as well as in total. The hook marks a file checked once per session and --resume keeps one session id for the conversation, so a zero after a non-zero is the latch rather than the gate staying quiet, and silence-hole has to be read with that column in view. Verified: a real trial authenticates, produces a graded silence-hole, and leaves the projects directory at the same count it started with.
GateGuard question effectiveness
Development-only harness that runs real Claude sessions on trapped coding tasks with and without GateGuard's questions. Design, arms, measures and results: docs/gateguard/question-effectiveness.md.
Two runners, measuring different things.
run.js — cost baseline. One-shot autonomous sessions over scenarios/, where
each trap hides its deciding fact in the repository. The pilot returned no
outcome difference between any arm, so treat this as a measure of the gate's
turn and token cost, not of its effect.
node docker/gateguard-effectiveness/run.js --out gg-eff --dry-run
node docker/gateguard-effectiveness/run.js --out gg-eff --model <model> --allow-real-provider
run-intent.js — effectiveness. Multi-turn sessions over scenarios-intent/,
where the deciding fact exists only in a simulated user's hidden intent, so an
agent that does not ask can only guess. Scenarios are authored blind to the
gate's question taxonomy, and the report classifies each trial into a hole worth
fixing.
node docker/gateguard-effectiveness/run-intent.js --check-graders
node docker/gateguard-effectiveness/run-intent.js --out gg-intent --arms off,gate --dry-run
node docker/gateguard-effectiveness/run-intent.js --out gg-intent --model <model> --allow-real-provider
node docker/gateguard-effectiveness/run-intent.js --out gg-intent --summarize