_parse_classification returned {} for garbage or non-dict output, so a failed classification looked like zero matched steps and graded as non-compliant. Raise ClassifierParseError (a RuntimeError, like the non-zero-rc path) instead.
- windows-test-env.js gains resolvePython: it probes each candidate with
`-c "import sys; sys.exit(0 if sys.version_info[0] == 3 else 1)"`, so the
Store stub is rejected.
- hermes-hooks.test.js launches its four hook spawns through the resolved
interpreter, falling back to python3 when none is found so a machine
without Python still fails loudly instead of skipping.
hermes-hooks goes from 0/14 to 14/14 on Windows 11; the other users of the
helper are unchanged. Product code is untouched: Hermes runs these hooks
itself and the installer writes no interpreter.
tests/hermes-hooks.test.js spawns python3, which on Windows is usually the
Microsoft Store alias stub: it exits 9009 instead of running Python, so all
14 Hermes cases fail with "hook must always exit 0 (got 9009)". python.org
installs provide python and the py launcher, not python3.
Pin resolvePython({ platform, probe }) in the shared windows-test-env
helper: the first candidate that actually runs Python 3 (Windows: python,
py -3, python3; elsewhere: python3, python), or null when none does, plus a
real smoke run of the resolved interpreter.
Against main: 4 of 8 windows-test-env cases fail (no resolvePython).
Snapshot the header and expected rows when the grader loads and pass
exportCsv copies, so an exporter that overwrites its input cannot change
what it is compared with. decoy-mutates now scores 0.75 with the other
two decoys; reference still scores 1.
The CSV grader reads its expected values from the same ROWS objects it
passes to exportCsv, so an exporter that overwrites the input fields and
exports them scores 1 (Greptile on #3519). decoy-mutates does exactly
that; graderIsSound reports the grader unsound on this head
(decoy-mutates=1). The grader-soundness test now prints every decoy
score through formatGraderScores.
normalizeIssueNumber parsed with Number.parseInt, which stops at the
first non-digit ("42oops" -> 42, "42.9" -> 42, "4e2" -> 4) and loses
precision past 2^53 (9007199254740993 became 9007199254740992). The CLI
shares this helper for --issue, --limit, and the positional issue number,
so malformed values silently targeted the wrong issue before any integer
guard downstream could notice.
Validate the entire trimmed decimal token and require
Number.isSafeInteger so malformed or out-of-range values throw
"Invalid issue number" during argument parsing, before the state store
is opened or GitHub is contacted. Whitespace, leading plus, and leading
zero tolerance is preserved. Regression coverage added at both the
library level (tests/lib/github-coordination-gh-api.test.js) and the CLI
argument parser (tests/scripts/github-coordination.test.js).
Fixes#3514
Preserve complete source ancestry for PRs #3512, #3513, #3515, #3516, #3517 and #3518. Repair bounded recovery, alias compatibility and evaluator packaging gaps while keeping registry reuse inside the canonical loader. Verified actual hook, setup, dashboard, evaluator, sandbox and profile workflows plus full npm, Python, lint and supply-chain checks. ECC-043 preparation only; integration remains with the delivery owner.
compileContextProfile now validates its options, loads the registry and
calls compileContextPlan(registry, options), which compiles against a
registry the caller loaded in the same call. resolveTaskContext and
routingEntries load the registry once and compile from it, so the sources
are hashed once per call instead of twice.
On this repository (full@1, Windows 11, median of 5):
resolveTaskContext 1332 ms -> 645 ms, routingEntries 1321 ms -> 721 ms.
The registry never comes from outside the call: nothing caches it or
accepts one from a caller, so a stale or forged registry cannot be passed
in. The two "Registry changed during ..." checks compared the two loads and
are gone with the second load; each call now works from one snapshot, and
skill bytes are still verified against it when read. compileContextProfile
keeps its signature, output and error order.
Split out of #2945, whose prompt hook is superseded by #3453.
resolveTaskContext and routingEntries each compile a plan, which loads and
hashes the whole registry, then load and hash it again for their own use.
Opening one skill's policy file counts the loads: three opens per call on
main (two registry hashes plus the admission read), where one load gives two.
- New: resolution and the routing index hash the registry sources once per
call (resolveTaskContext and routingEntries each open the policy twice).
- The policy-drift test injects its change on the open after the registry
is hashed. With one load that is the second open, not the third; its
assertion that the drift is rejected is unchanged.
Against main: 3 failures in tests/lib/context-selection.test.js.
- csv-export-delimiter's grader adds a fourth check: the data rows are
the exported values, with an optional header (the hidden intent says a
header is fine either way). Reference scores 1; both decoys 0.75.
- graderIsSound grades every decoy-* overlay and is sound only when each
scores below 1; --check-graders and the pre-billing check print the
decoy scores through formatGraderScores.
- isValidTrial and completedTrialKeys move to lib.js; coverage.js and
run-intent.js use them from there.
- run.js resumes with completedTrialKeys, so a provider-error or timed-out
key is retried, and writes one transcript per attempt
(<key>__<attemptId>.jsonl, recorded on the row) so a retry never
overwrites the failed attempt's record.
- armSummary computes trials, passes and means over valid rows and counts
the failed attempts apart as providerErrors.
Docs: question-effectiveness.md states the decoy rule and that a
csv-export-delimiter pass in the reported runs reflects the dialect only;
its Measures section and the harness README describe retries, per-attempt
transcripts and the hookless-trial stop.
Three #3481 review findings still hold on main 8ce348fb:
- csv-export-delimiter's grader checks only the dialect (CRLF, semicolons,
quoting), so a constant '"x";"y"' export or one emitting the column
names scores 1. Add decoy-constant and decoy-keys overlays and pin that
each scores below 1 and that graderIsSound reports them; a scenario whose
decoy passes is not sound.
- The pilot runner (run.js) marks errored keys done, so they are never
retried, and armSummary counts provider-error and timed-out rows in
trials, passes and means. Pin the summary on valid rows only, and drive
run.js end to end: a provider error is retried on resume, keeps its own
row and transcript, and the summary reports it apart.
- Disabling run-intent.js's abort on an unobserved hook left every test
passing. Drive both runners end to end with a gated trial whose hook
never runs: each must stop and record nothing for it.
fake-claude.js gains FAKE_CLAUDE_SKIP_HOOK and FAKE_CLAUDE_PROVIDER_ERROR.
The intent end-to-end setup moves into syntheticIntentRunner, shared with
the existing resume test.
Against main: 2 intent failures (the decoy cases) and 2 pilot failures
(summary, retry). The two abort tests pass on main, since both aborts
exist; each fails when its runner's abort is disabled
(if (false && !row.hookObserved)).
haelyra