96 trials, none invalid. The headline reads gate 23/48 against off 13/48, a
21-point paired difference at exact McNemar p = 0.021, and it is an artifact.
The scenarios, prompts, graders and model alias were identical to the previous
run and the off arm has no gate, yet off fell from 18/48 to 13/48 while the
gated arm went down, 25/48 to 23/48. An intervention meant to raise the gated
arm did not raise it, so the p-value reflects a baseline that fell rather than a
gate that improved.
The two scenarios the question cannot reach, having no data for the profile to
detect, serve as a control: they moved by -3 and +3, more than any of the four
it did reach (-1, -1, +1, -1). Noise on eight trials is about plus or minus
three, which is larger than any effect available to the change.
The question is kept, since it asks something the gate never asked and the
corpus says that kind of answer decides the outcome, but no improvement is
claimed. Testing it fairly needs scenarios authored blind to it, which this
corpus no longer is.
96 trials, none invalid, analysed exactly as pre-registered. passed: gate 25/48
against off 18/48, 48 pairs, discordant 15 to 8, exact McNemar p = 0.210. At the
registered alpha of 0.05 the hypothesis is not confirmed, and the bootstrap
interval agrees this time at -15% with a range of -33% to +4%, so there is no
disagreement between the tests to adjudicate. Secondary measures point the same
way and reach nothing: asked 29/48 against 22/48 at p = 0.230, deciding fact
27/48 against 21/48 at p = 0.327.
The effect halved on replication, from +29 points at p = 0.065 to +15 at
p = 0.210, which is the ordinary fate of a near-threshold finding. The runs are
not pooled: the quota leak was closed between them, and combining them after
seeing both would be the post-hoc choice the pre-registration exists to stop.
Two things survive every run. Obtaining the deciding fact is necessary and
nearly sufficient, with no trial in either arm passing without it across 96
trials. And the gate costs 37 to 45% more turns and spend, which is the most
stable number the evaluation has produced.
soft-limit-overage asked for { allowed, overage }. Naming an overage figure
hints that passing a limit is permitted, which is precisely the hidden policy,
and it showed: that scenario was one of the two with the highest ungated pass
rate in the signposted run. The grader now inspects only allowed, so the prompt
asks for { allowed } and mentions neither overage nor billing nor soft limits. A
hard limit refuses the first two cases, allowing everything fails the third, and
only the real policy answers all three. --check-graders still separates start,
trap and reference on all six.
The analysis is registered in the doc before the run rather than chosen after
it: passed as the single primary outcome, paired exact McNemar at alpha 0.05, a
sample of 48 pairs fixed in advance, no extension on the strength of the result,
frozen inputs, secondary measures explicitly not claim-bearing, a validity rule
for provider errors, and a commitment to report a null.
With the ambiguity signposted, the arms were indistinguishable on every measure.
With the prompts rewritten as ordinary tickets and nothing else changed, they
separate: asked the user 16/24 against 10/24, obtained the deciding fact the
same, passed 15/24 against 8/24, exact McNemar p = 0.065 to 0.070, and a paired
pass-rate difference of 29 points in the gate's favour whose bootstrap interval
excludes zero.
The regime contrast carries it. The gated arm passed 15/24 in both runs while
the ungated arm fell from 16/24 to 8/24, which is what removing the only cue
that a convention was missing would do if the gate supplies that cue itself.
Asking stayed necessary throughout: no trial in either arm passed without the
deciding fact.
Reported as suggestive rather than established. All three p-values sit just
above 5%, one more gate-favouring discordant pair would cross it, and extending
this run after seeing that would be optional stopping. A confirmatory run needs
its sample fixed first: about 43 pairs, or 8 repetitions over these six
scenarios.
With one decisive fact per scenario the corpus isolates the mechanism:
obtaining the deciding fact predicts the outcome 14/14 in both arms, and not
obtaining it predicts failure in 9 of 10 and 8 of 10. Against that, the gate
changes nothing measurable. Asked the user 14/24 against 15/24, obtained the
deciding fact 14/24 against 14/24, passed 15/24 against 16/24, every exact
McNemar p = 1.000, and the paired pass-rate difference 5 points in off's favour
with a 95% interval of -25% to +35%. At 20 pairs a 40-point effect is ruled out;
a 15-to-20-point one is not.
The gate does fire - denials in 22 of 24 gated trials - so the failure is in the
conversion. timezone-daily-rollup fired four times out of four and produced no
question at all. Every question the gate raised is about code structure, while
every deciding fact in this corpus is an external contract.
Record the two scenario-design faults the earlier runs exposed, and the bound
the correction imposes: moving shape into the prompt also signposts the
ambiguity, which raises asking in both arms, so the claim is limited to the
signposted regime.
Four defects found by running the repository's own checks and one more trial
against it.
ESLint rejected an unused `armSettings` import in intent-eval.js; the caller
supplies `armSettingsByName` already.
A timeout was lost whenever a later turn came back clean: `a || b ? c : false`
evaluates the condition first, so `timedOut` was recomputed from the new turn
instead of being kept.
Isolating each trial's config directory, to stop persisted sessions piling up
under the projects folder, puts the credentials out of reach. Every turn then
returns "Not logged in", and the harness graded that as a coverage-hole - an
auth failure recorded as a finding. Keep the real config directory and delete
only the session folders this run's work root created, and adopt lib.js's own
guards: a trial that is unauthenticated, or whose gated arm edited files with no
metrics line and no denial, stops the run instead of being recorded.
Record denials per turn as well as in total. The hook marks a file checked once
per session and --resume keeps one session id for the conversation, so a zero
after a non-zero is the latch rather than the gate staying quiet, and
silence-hole has to be read with that column in view.
Verified: a real trial authenticates, produces a graded silence-hole, and leaves
the projects directory at the same count it started with.
The pre-registration said results would be reported whichever way they came
out. They came out empty: 72 sessions in which all four arms passed 18 of 18
with mean score 1.000, every arm-against-gate difference 0% (0% to 0%) and
McNemar p = 1. With the outcome fixed for every trial there is no variance to
explain, so this is the absence of a measurement rather than weak evidence that
the gate does nothing. The run does establish the gate's cost in that regime:
about 45% more turns and 28% more tokens for no outcome change.
Record the three reasons it measured nothing - deciding facts hidden in the
repository, the asking channel switched off, and a pass criterion written in
the gate's own vocabulary - and the hidden-intent design that addresses them,
with its hole classes, its first four-trial run and its own limits.
Brings in the three commits main gained since the branch's merge base: the 2.2.3
release, retiring the Everything Claude Code name for ECC, and the ja-JP sponsor
update. 227 files arrive from main; two conflicted.
CHANGELOG.md: the branch had added an Unreleased section and main a released
2.2.3 section, both at the top. Both are kept verbatim with Unreleased above
2.2.3, since none of the branch's entries shipped in that release.
manifests/context-packs/skill-triggers@1.json: both sides had recomputed
registryDigest from the same earlier value to different results, and neither is
the digest of the merged tree. It is recomputed with loadContextRegistry over the
merged tree and only that field changes; triggers and triggersDigest are
untouched on both sides. validate-context-profiles passes.
All twelve CI validators pass. The full suite reports 44 failures in seven files;
43 of them fail by name and count identically on an unmodified upstream/main, and
the remaining one, plugin-hook-bootstrap, fails only inside run-all and passes in
isolation on both trees. The merge introduces no failing test that main does not
already have.
* chore: retire the Everything Claude Code name, ECC only
Replace the expanded product name and the everything-claude-code slug
with ECC across READMEs, translated docs, harness folders, skills,
commands, scripts and examples. Repo URLs now point at affaan-m/ECC.
Rename repo-internal artifacts to ecc forms (ecc-conventions skill,
ecc-guardrails, ecc-research-playbook, ecc-team-config, ecc-instincts
with ecc-* ids) and update .claude/ecc-tools.json.
Legacy install ids, plugin dirs, package names and MCP prefixes stay
where existing installs resolve against them. The codex global-state
check and harness-audit repo detection accept the new name first and
the legacy one as a fallback. The Discord bot prefers ~/GitHub/ECC and
falls back to the nested legacy checkout path.
Refresh the skill-triggers registryDigest for the changed skill text.
* fix(ecc-tools): drop managed paths to the removed Claude repo skill
.claude/skills/<repo-skill>/SKILL.md was deleted in #2770 but stayed listed in
ecc-tools.json and the team config. Remove those entries and point the Claude
Code adapter at the existing .agents/skills/ecc-conventions/SKILL.md.
pi/core and the bundled blueprint skill now refer to the project as ECC only;
the generated README no longer expands the name. Profile contents, package
name and paths are unchanged (123 skills, 24 commands).
The release workflows poll npm for up to five minutes after publish before
verifying the artifact. In 2.2.2 the check gave up after about 30 seconds while
npm was still processing, failing the run before latest was promoted and the
GitHub Release was created.
Version metadata synced to 2.2.3 with CHANGELOG, translated release headings
and release notes; skill-triggers registryDigest updated for the blueprint text.
Adds a curated, Pi-native, skills+prompts-only profile at pi/core/ for downstream packagers that mirror GitHub Releases.
- manifests/pi-core.json: explicit include lists, per-item exclusion reasons, curation rules, safety allowlists; every root skill and command must be classified.
- scripts/build-pi-core.js regenerates pi/core deterministically (package.json from VERSION, LICENSE, README.md, CURATION.md, skills/, commands/); --check fails CI on drift. council is renamed ecc-council inside pi/core only.
- Safety checks: no callable endpoints outside the allowlist, no npx/curl|sh/pip install, no secrets, no absolute home paths, no symlinks, valid frontmatter, no duplicate names.
- CI: build + drift check and an offline Pi CLI load test of pi/core.
- Release: VERSION, package.json and pi/core/package.json at 2.2.2, CHANGELOG, tag-triggered release verification, and a two-week cadence in CONTRIBUTING.md.
pi/core: 123 of 293 skills and 24 of 94 commands; 35,006 characters of skill description text.
Preserve the original contributor histories and apply the exact independently reviewed repair.
Source-Parent: e41c9241f5
Review-Manifest-SHA256: df9ec6b308f1935e55f13af1845a99b87a0fc921fb30b16e52fbf70d5120ee81
Update four stale skill totals to the293 canonical entries verified by the existing catalog checker.
Reviewed-Parent: 5d4cca811d
Source-Manifest-SHA256: 5281891022c3405cfa0549bb15a15ce4b1df101b7c64780da9cbbd839df3d971
montjeffrey