montjeffrey 6a7996aa70 fix(gateguard): make harness graders check content and count only valid trials ...

- csv-export-delimiter's grader adds a fourth check: the data rows are
  the exported values, with an optional header (the hidden intent says a
  header is fine either way). Reference scores 1; both decoys 0.75.
- graderIsSound grades every decoy-* overlay and is sound only when each
  scores below 1; --check-graders and the pre-billing check print the
  decoy scores through formatGraderScores.
- isValidTrial and completedTrialKeys move to lib.js; coverage.js and
  run-intent.js use them from there.
- run.js resumes with completedTrialKeys, so a provider-error or timed-out
  key is retried, and writes one transcript per attempt
  (<key>__<attemptId>.jsonl, recorded on the row) so a retry never
  overwrites the failed attempt's record.
- armSummary computes trials, passes and means over valid rows and counts
  the failed attempts apart as providerErrors.

Docs: question-effectiveness.md states the decoy rule and that a
csv-export-delimiter pass in the reported runs reflects the dialect only;
its Measures section and the harness README describe retries, per-attempt
transcripts and the hookless-trial stop.

2026-10-11 10:00:47 -04:00

..

change-profile.md

fix: preserve opt-in GateGuard contract questions from PR 3497

2026-10-11 02:00:27 -04:00

design-notes.md

fix: preserve opt-in GateGuard contract questions from PR 3497

2026-10-11 02:00:27 -04:00

evaluation.md

docs(gateguard): describe the question effectiveness evaluation

2026-10-04 22:40:42 -04:00

question-effectiveness.md

fix(gateguard): make harness graders check content and count only valid trials

2026-10-11 10:00:47 -04:00