test(skills): assert COMMAND-REGISTRY.json is fresh in plan-orchestrate fixtures

- Problem: the registry fixtures validated SKILL.md against the committed docs/COMMAND-REGISTRY.json but nothing in the file asserted the registry itself was fresh, so a standalone run passed against a stale registry (npm test does run command-registry:check first, but this file is also run on its own).
- Fix: rebuild the registry in memory via the exported generateRegistry() and assert deep equality with the committed JSON so drift fails as a regenerate hint with a field diff instead of passing silently; add one sentence to SKILL.md stating that a command retired from commands/ fails closed with the BLOCKED marker like a no-argument command.
- Verification: node tests/skills/plan-orchestrate-registry.test.js passes with the new check 8 (8/8); sabotage with a mutated registry description fails check 8 with the regenerate hint and a field diff.
This commit is contained in:
Santhi Prakash
2026-09-28 17:23:39 +00:00
parent 5bf96bab17
commit 8a6e52c990
2 changed files with 22 additions and 2 deletions
+1 -1
View File
@@ -52,7 +52,7 @@ BLOCKED — /<command> cannot carry plan scope; run manually if wanted: <exact d
Blocked steps appear in the overview and per-step output and are excluded from the Batch execution block. This is deliberate: a bare `/update-docs` pasted inside a five-step batch has no machine-readable link to the step it was planned for. The user can still run the manual command themselves; the router just refuses to present an unscoped command as if the step were orchestrated.
This table is a validated snapshot of the command files, not an independent source of truth. `tests/skills/plan-orchestrate-registry.test.js` validates every command named here against `commands/<name>.md` and `docs/COMMAND-REGISTRY.json` and fails when they disagree. If a command file and this skill ever disagree at runtime, the command file wins.
This table is a validated snapshot of the command files, not an independent source of truth. `tests/skills/plan-orchestrate-registry.test.js` validates every command named here against `commands/<name>.md` and `docs/COMMAND-REGISTRY.json` and fails when they disagree. If a command file and this skill ever disagree at runtime, the command file wins. A command that has since been retired from `commands/` fails closed like a no-argument command — the step is reported `BLOCKED` and the router never emits a name the command surface no longer has.
## Task description (scope-carrying commands only)
+21 -1
View File
@@ -17,7 +17,11 @@
* argument (re-classify the skill if a command grows one);
* 5. the fail-closed rule and the BLOCKED marker are actually stated;
* 6. the agent-chain machinery stays gone — commands own their execution;
* 7. emitted examples carry the [Plan: ...#step-N] scope marker.
* 7. emitted examples carry the [Plan: ...#step-N] scope marker;
* 8. the committed docs/COMMAND-REGISTRY.json is fresh — it equals a
* regeneration from commands/, so a contributor who adds a command and
* forgets to regenerate fails here with a regenerate hint instead of the
* router silently validating against a stale surface.
*/
'use strict';
@@ -219,6 +223,22 @@ function run() {
assert.ok(checked >= 3, `expected at least 3 command examples, checked ${checked}`);
console.log(` \u2713 emitted examples carry the scope marker (${checked} checked)`);
// --- 8. The committed registry itself is fresh -------------------------
// The checks above validate the skill against the committed JSON, which
// makes a stale registry silently pass. Rebuild it in memory from the
// command files and require equality, so drift fails as "regenerate".
// npm test also runs `command-registry:check` before the suites; this keeps
// the file honest when it is run standalone.
const { generateRegistry } = require(path.join(REPO_ROOT, 'scripts', 'ci', 'generate-command-registry.js'));
const committedRegistry = JSON.parse(fs.readFileSync(REGISTRY_PATH, 'utf8'));
assert.deepStrictEqual(
committedRegistry,
generateRegistry(),
'docs/COMMAND-REGISTRY.json is stale — regenerate it with `npm run command-registry:write`'
);
console.log(' \u2713 docs/COMMAND-REGISTRY.json matches a fresh generation from commands/');
console.log(' plan-orchestrate registry fixtures passed');
}