mirror of
https://github.com/affaan-m/ECC.git
synced 2026-10-15 04:35:29 +02:00
montjeffrey
7afde35e49
The existing harness cannot measure what it set out to measure. Every trap hides its deciding fact in the repository, which agents reliably discover, and trials run one-shot under "do not ask clarifying questions", so a fact-forcing gate is evaluated with no user to answer it. Its scenarios also declare the question the gate ought to raise, chosen by reading the gate's own taxonomy, so the suite passes by construction and can never find a missing question. Add a second runner that measures effectiveness instead of coverage: - scenarios-intent/ keeps the deciding fact in intent.json, as a choice between plausible conventions the repository cannot settle, so an agent that does not ask can only guess and the ungated arm cannot sit at the ceiling. - user-sim.js plays that user with a separate cheap model: it answers the question in front of it, never volunteers, and reports which fact ids it disclosed, so asking something and asking the deciding thing are told apart. An unparseable judge is a recorded failure, not a silent stonewall. - session.js runs multi-turn sessions joined by --session-id and --resume, and tells every arm the user is reachable, so the comparison is symmetric and the gate's contribution is whether it makes the agent ask the deciding question. - coverage.js classifies each trial from behaviour alone into working, lucky, or a follow-through, targeting, silence or coverage hole, and reports the questions the gate actually raised rather than asserting declared ones. - run-intent.js refuses minus-target, which needs targets these scenarios deliberately lack, and checks every grader separates the start, the trap and the reference before a session is billed. A scenario counts as carrying no information only when nothing varies across arms. Trap strength is reported separately: the ungated arm failing is what a working trap looks like, so it must not make a scenario unusable.