feat: group verified capsules into offline retrospective reports

Independent local Codex review PASS at 472cfa94fb, no P0/P1. Independent retrospective, CLI, envelope and capsule tests: 73 passed. CI run 34664487219 attempt 2 passed all 42 jobs. Read-only offline capsule grouping only; no provenance, scoring, execution or promotion claim. Rollback: revert this squash commit.
This commit is contained in:
Affaan Mustafa
2026-09-12 04:35:38 -04:00
committed by GitHub
parent 1ed03ecf2e
commit ab1f1a09ba
8 changed files with 393 additions and 3 deletions
+7
View File
@@ -73,6 +73,13 @@ capsule recording to the hooks that already log session activity. Then the
next two plan slices: offline retrospective grouping over capsules (no new
rollouts) and forced-compaction tests that prove pinned constraints survive.
Offline code preparation is available as `capsule group` over explicitly
selected, verified local snapshots from one task family. It only groups recorded
counts and digests; it does not run candidates, score outcomes or promote changes.
This utility does not fulfill the executor, hook-recording or stable-taskset
prerequisites for the operational milestone below. See the
[retrospective contract](architecture/eval-harness-frameworks.md#offline-retrospective-preparation).
### Track C: operator skills
The four desk-pattern skills are present in this candidate: operator approval
+63 -2
View File
@@ -29,8 +29,10 @@ The other modules expose local utilities, not a trust decision about code.
| Replay | `replay.js`, `effect-fence.js` | 04 replay-safe branching | Declared determinism and effect class per tool, content-addressed fixtures, `tool.fixture_missing` fail-closed replay, retired child preload refuses execution |
| Receipt | `receipt.js` | 07 verifiable receipts | Offline receipt over capsule root, entry count, artifact digest, and gate receipt; detached signature interface; verification names the failing check |
Epics 05 (offline self-improvement) and 06 (causal triage and compaction
invariance) are not implemented. They consume the records these five produce.
Epic 05 has an offline, report-only capsule grouping utility described below.
Self-improvement, operational retrospective validation and epic 06 (causal
triage and compaction invariance) remain unimplemented. They consume the
records these five frameworks produce.
## Effect classes
@@ -182,6 +184,64 @@ What the chain does not claim: it does not stop an operator from replacing the
whole log. That is the job of a witnessed transparency log, which is a later,
opt-in layer outside this package.
## Offline retrospective preparation
Select 1 to 100 existing capsule directories from one task family:
```sh
node scripts/eval-harness.js capsule group .ecc/capsules/run-41 .ecc/capsules/run-42
```
```js
const { retrospective } = require('./scripts/lib/eval-harness');
const report = retrospective.groupCapsules(['.ecc/capsules/run-41', '.ecc/capsules/run-42']);
```
This read-only utility recomputes each projection from the verified metadata and
journal snapshot using `capsule.project`. It never uses or repairs a saved
`projection.json`. Inputs must be small, quiescent local capsules from the same
task family; a mismatch rejects the entire report. There is no directory
discovery, hook activation, new rollout, fixture replay or candidate execution.
`capsule-retrospective/v1` reports the task family, input count, unique capsule
count, duplicate count, and groups sorted by declared harness version. Each
group contains capsule/entry counts, all five lineage counts, all five declared
effect-class counts, and source digest references. Counts describe recorded
entries, not unique tasks, attempts, successful effects or independently
verified outcomes. Empty journals contribute one capsule and zero entries.
Payload scores, verdicts, costs, durations and pass/fail totals are not used.
The pair `(run_id, capsule_id)` identifies a capsule for deduplication. Repeated
paths or copied snapshots count once when their verified projection hashes
match. Conflicting snapshots of that identity, including different checkpoints,
fail with `retrospective.conflicting_identity`; the utility never picks a winner.
Distinct capsule identities remain distinct even if their event shapes match.
Source references contain the canonical hash of the identity pair, entry count,
root hash, journal digest and projection hash. `report_hash` covers every other
report field; input ordering does not change the result. Repeating an input
changes input/duplicate counts and the report hash, but not the grouped counts.
Reports omit directory arguments, raw run/capsule IDs, journal payloads and
timestamps. **Task-family and harness-version labels are returned verbatim**
and may themselves contain private text or paths. Digest references are not
anonymization: they remain linkable and low-entropy IDs can be guessed. Review
labels and report content before sharing. Neither hashes nor declared labels
authenticate a producer or prove an improvement; `report_only` is always true.
Any invalid, unreadable or mismatched capsule rejects the whole report with
`retrospective.invalid_capsule` and a zero-based input index. Diagnostics omit
underlying reader messages and source paths. Mixed families and invalid input
lists have separate stable codes. CLI success emits JSON to stdout and exits 0;
bad usage exits 2, while verification/refusal exits 1 without partial JSON.
The command accepts no flags and does not write a report file. For a directory
name beginning with `--`, use a relative `./` prefix or an absolute path.
This inherits the existing capsule reader's filesystem and memory limits. The
100-input cap does not bound journal bytes. It does not isolate hostile files,
serialize concurrent writers, validate a signature or establish live provenance.
Executor containment, opt-in hook recording, stable-taskset validation and the
roadmap's operational retrospective milestone remain separate prerequisites.
## Verification gate: unavailable
**Supported candidate execution backends: none, on any OS.** `runGate` and
@@ -321,6 +381,7 @@ and does not validate normal prepack or clear a release.
```sh
node tests/lib/eval-harness/envelope.test.js
node tests/lib/eval-harness/capsule.test.js
node tests/lib/eval-harness/retrospective.test.js
node tests/lib/eval-harness/gate.test.js
node tests/lib/eval-harness/security.test.js
node tests/lib/eval-harness/replay.test.js
+9 -1
View File
@@ -7,6 +7,7 @@
* node scripts/eval-harness.js capsule verify <dir>
* node scripts/eval-harness.js capsule project <dir>
* node scripts/eval-harness.js capsule export <dir> <out-dir>
* node scripts/eval-harness.js capsule group <dir> [<dir> ...]
* node scripts/eval-harness.js gate run <gate.config.json> [--work-dir <dir>] [--capsule <dir>]
* node scripts/eval-harness.js receipt build <capsule-dir> [--artifact <file>] [--gate <gate-receipt.json>] [--out <file>]
* node scripts/eval-harness.js receipt verify <receipt.json> <capsule-dir> [--artifact <file>] [--gate <gate-receipt.json>]
@@ -26,7 +27,7 @@ function usage(message) {
if (message) {
process.stderr.write(`eval-harness: ${message}\n`);
}
const header = fs.readFileSync(__filename, 'utf8').split('\n').slice(3, 15).map((line) => line.replace(/^ \*\s?/, '')).join('\n');
const header = fs.readFileSync(__filename, 'utf8').split('\n').slice(3, 16).map((line) => line.replace(/^ \*\s?/, '')).join('\n');
process.stderr.write(`${header}\n`);
process.exit(2);
}
@@ -64,6 +65,13 @@ function runExample(action) {
function runCapsule(action, rest) {
const dir = rest[0];
if (!dir) usage('capsule commands need a capsule directory');
if (action === 'group') {
if (rest.length > harness.retrospective.MAX_INPUTS || rest.some(arg => !arg.trim() || arg.startsWith('--'))) {
usage(`capsule group needs 1 to ${harness.retrospective.MAX_INPUTS} directory paths and accepts no flags`);
}
print(harness.retrospective.groupCapsules(rest));
return;
}
if (action === 'verify') {
const result = harness.capsule.verify(dir);
print(result);
+2
View File
@@ -5,6 +5,7 @@
*
* envelope capsule-envelope/v1 contract, redaction, secret canaries
* capsule append-only hash-linked journal with five lineages
* retrospective offline report-only grouping of selected capsule snapshots
* gate static inspection and disabled execution gate, syntactic warnings
* replay declared tool effects, fixtures, fail-closed replay, retired effect preload
* receipt offline-verifiable capsule receipts
@@ -16,6 +17,7 @@ module.exports = {
canonical: require('./canonical'),
envelope: require('./envelope'),
capsule: require('./capsule'),
retrospective: require('./retrospective'),
gate: require('./gate'),
replay: require('./replay'),
receipt: require('./receipt'),
+101
View File
@@ -0,0 +1,101 @@
'use strict';
/** Read-only retrospective preparation over explicit, quiescent local capsules.
* Counts are declarations in the records, never scores or promotion evidence.
* Inherits capsule.project's local-reader limits; not hostile-filesystem isolation.
*/
const capsule = require('./capsule');
const { LINEAGES, EFFECT_CLASSES } = require('./envelope');
const { hashValue } = require('./canonical');
const MAX_INPUTS = 100;
const SOURCE_FIELDS = ['entry_count', 'root_hash', 'journal_sha256', 'projection_hash'];
class RetrospectiveError extends Error {
constructor(code, message, inputIndex) {
super(message);
this.name = 'RetrospectiveError';
this.code = code;
if (inputIndex !== undefined) this.input_index = inputIndex;
}
}
function readProjection(dir, index) {
try {
return capsule.project(dir);
} catch {
// Reader errors can contain private paths or journal content. No partial
// report or unchecked diagnostic content crosses the report boundary.
throw new RetrospectiveError('retrospective.invalid_capsule', `capsule at input index ${index} failed verification`, index);
}
}
const compare = (a, b) => a < b ? -1 : a > b ? 1 : 0;
const identity = projection => JSON.stringify([projection.run_id, projection.capsule_id]);
function summarizeGroup(harnessVersion, projections) {
const total = keys => Object.fromEntries(keys.map(key => [key, 0]));
const byLineage = total(LINEAGES);
const byEffect = total(EFFECT_CLASSES);
for (const projection of projections) {
for (const key of LINEAGES) byLineage[key] += projection.by_lineage[key];
for (const key of EFFECT_CLASSES) byEffect[key] += projection.by_effect_class[key];
}
return {
harness_version: harnessVersion,
capsule_count: projections.length,
entry_count: projections.reduce((sum, item) => sum + item.entry_count, 0),
by_lineage: byLineage,
by_effect_class: byEffect,
sources: [...projections].sort((a, b) => compare(identity(a), identity(b)))
.map(item => ({
identity_hash: hashValue([item.run_id, item.capsule_id]),
...Object.fromEntries(SOURCE_FIELDS.map(key => [key, item[key]])),
})),
};
}
/**
* Group up to 100 selected snapshots of one task family by harness_version.
* Duplicate identities count once only if the verified projections match.
* Different checkpoints of the same identity are ambiguous and refused.
* Does not read saved projections, copy payloads, write files or invoke tools.
*/
function groupCapsules(dirs) {
if (!Array.isArray(dirs) || dirs.length < 1 || dirs.length > MAX_INPUTS
|| !Array.from(dirs).every(dir => typeof dir === 'string' && dir.trim() && !dir.includes('\0'))) {
throw new RetrospectiveError('retrospective.invalid_inputs', `supply 1 to ${MAX_INPUTS} capsule directory paths`);
}
const projections = dirs.map(readProjection);
const family = projections[0].task_family;
const unique = new Map();
for (const [index, projection] of projections.entries()) {
if (projection.task_family !== family) {
throw new RetrospectiveError('retrospective.mixed_task_families', 'all capsules must have the same task family', index);
}
const key = identity(projection);
const previous = unique.get(key);
if (previous && previous.projection_hash !== projection.projection_hash) {
throw new RetrospectiveError('retrospective.conflicting_identity', 'conflicting snapshots share a capsule identity', index);
}
unique.set(key, projection);
}
const byHarness = new Map();
for (const projection of unique.values()) {
const version = projection.harness_version;
byHarness.set(version, [...(byHarness.get(version) || []), projection]);
}
const report = {
schema: 'capsule-retrospective/v1',
report_only: true,
task_family: family,
input_count: dirs.length,
capsule_count: unique.size,
duplicate_count: dirs.length - unique.size,
groups: [...byHarness].sort(([a], [b]) => compare(a, b))
.map(([version, items]) => summarizeGroup(version, items)),
};
return { ...report, report_hash: hashValue(report) };
}
module.exports = { groupCapsules, MAX_INPUTS, RetrospectiveError };
+8
View File
@@ -250,6 +250,14 @@ node scripts/eval-harness.js example
closed; SE3 and above are refused in replay. Record mode invokes the registered
implementation, so only register trusted functions.
- Receipt: offline verification of capsule and artifact bytes, with named checks.
- Retrospective preparation: `node scripts/eval-harness.js capsule group <dir> [<dir> ...]`
groups 1 to 100 explicitly selected, verified local capsule snapshots from one
task family by declared harness version. Repeated snapshots count once;
conflicting identities or invalid capsules reject the whole report. This is
read-only record counting, with no new rollouts, scores or promotion. Use small,
quiescent capsules. Payloads, directory arguments and raw run/capsule IDs are
omitted, but task-family/version labels are verbatim and digest references are
linkable; review them before sharing. Operational validation remains pending.
Candidate execution is disabled on every OS because no verified OS containment
backend is implemented. `gate run`, `runGate`, `runVariant`, direct child launch,
+31
View File
@@ -148,4 +148,35 @@ test('capsule CLI projects and exports valid metadata, and rejects forged metada
} finally { cleanup(root); }
});
test('capsule group CLI emits only a read-only report for explicit snapshots', () => {
const dir = tempDir('cli-group');
try {
harness.capsule.Capsule.create(dir, { task_family: 'fixture-family' })
.append('plan', 'start', { note: 'private journal marker' });
const before = fs.readdirSync(dir).map(name => [name, fs.readFileSync(path.join(dir, name))]);
const result = run(['capsule', 'group', dir, dir]);
assert.strictEqual(result.status, 0, result.stderr);
const report = JSON.parse(result.stdout);
assert.strictEqual(report.report_only, true);
assert.strictEqual(report.capsule_count, 1);
assert.strictEqual(report.duplicate_count, 1);
assert.ok(!result.stdout.includes('private journal marker'));
assert.ok(!result.stdout.includes(dir));
assert.deepStrictEqual(fs.readdirSync(dir).map(name => [name, fs.readFileSync(path.join(dir, name))]), before);
} finally { cleanup(dir); }
});
test('capsule group rejects bad usage and content without a partial report', () => {
for (const args of [[], [' '], ['--out', 'missing'], Array(101).fill('missing')]) {
const result = run(['capsule', 'group', ...args]);
assert.strictEqual(result.status, 2, result.stderr);
assert.strictEqual(result.stdout, '');
}
const result = run(['capsule', 'group', '/missing/private-directory-marker']);
assert.strictEqual(result.status, 1);
assert.match(result.stderr, /retrospective.invalid_capsule/);
assert.ok(!result.stderr.includes('private-directory-marker'));
assert.strictEqual(result.stdout, '');
});
finish('cli');
@@ -0,0 +1,172 @@
'use strict';
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const harness = require('../../../scripts/lib/eval-harness');
const { test, tempDir, cleanup, finish, fixedClock } = require('./helpers');
const root = tempDir('retrospective');
let next = 0;
function record(options = {}, events = [['plan', 'start', {}, 'SE0']]) {
const id = `record-${next++}`;
const dir = path.join(root, id);
const capsule = harness.capsule.Capsule.create(dir, {
run_id: id, capsule_id: id, task_family: 'fixture-family', harness_version: 'v1',
clock: fixedClock, ...options,
});
for (const [lineage, kind, payload, effect_class] of events) capsule.append(lineage, kind, payload, { effect_class });
return dir;
}
function group(dirs) { return harness.retrospective.groupCapsules(dirs); }
function rejects(dirs, code) { assert.throws(() => group(dirs), error => error.code === code); }
try {
test('groups one task family by declared harness version without scoring payloads', () => {
const a = record({}, [['plan', 'start', {}, 'SE0'], ['attempt', 'done', { score: 1, verdict: 'PROMOTE' }, 'SE2']]);
const b = record();
const c = record({ harness_version: 'v2' }, [['interaction', 'tool', { note: 'private payload marker' }, 'SE4']]);
const result = group([c, a, b]);
assert.strictEqual(result.schema, 'capsule-retrospective/v1');
assert.strictEqual(result.report_only, true);
assert.strictEqual(result.task_family, 'fixture-family');
assert.strictEqual(result.capsule_count, 3);
assert.strictEqual(result.input_count, 3);
assert.strictEqual(result.duplicate_count, 0);
assert.deepStrictEqual(result.groups.map(g => [g.harness_version, g.capsule_count, g.entry_count]), [['v1', 2, 3], ['v2', 1, 1]]);
assert.deepStrictEqual(result.groups[0].by_lineage, { plan: 2, attempt: 1, interaction: 0, environment: 0, strategy: 0 });
assert.deepStrictEqual(result.groups[0].by_effect_class, { SE0: 2, SE1: 0, SE2: 1, SE3: 0, SE4: 0 });
assert.strictEqual(result.groups[1].by_effect_class.SE4, 1);
const serialized = JSON.stringify(result);
for (const marker of ['private payload marker', 'PROMOTE', root, 'score', 'verdict', 'payload']) assert.ok(!serialized.includes(marker), marker);
});
test('binds every counted snapshot to the existing verified projection digests', () => {
const dir = record();
const projection = harness.capsule.project(dir);
const source = group([dir]).groups[0].sources[0];
assert.deepStrictEqual(source, {
identity_hash: harness.canonical.hashValue([projection.run_id, projection.capsule_id]),
...Object.fromEntries(['entry_count', 'root_hash', 'journal_sha256', 'projection_hash'].map(key => [key, projection[key]])),
});
});
test('raw run and capsule labels are omitted from the report', () => {
const dir = record({ run_id: 'private-run-label', capsule_id: 'private-capsule-label' });
const serialized = JSON.stringify(group([dir]));
assert.ok(!serialized.includes('private-run-label'));
assert.ok(!serialized.includes('private-capsule-label'));
});
test('deduplicates repeated paths and copied snapshots by identity and projection', () => {
const dir = record();
const copy = path.join(root, 'duplicate');
fs.cpSync(dir, copy, { recursive: true });
const result = group([dir, copy, dir]);
assert.strictEqual(result.input_count, 3);
assert.strictEqual(result.capsule_count, 1);
assert.strictEqual(result.duplicate_count, 2);
assert.strictEqual(result.groups[0].entry_count, 1);
});
test('identity uses both run and capsule IDs for distinct evidence', () => {
const result = group([
record({ run_id: 'run-a', capsule_id: 'capsule-a' }),
record({ run_id: 'run-a', capsule_id: 'capsule-b' }),
record({ run_id: 'run-b', capsule_id: 'capsule-a' }),
]);
assert.strictEqual(result.capsule_count, 3);
assert.strictEqual(result.groups[0].sources.length, 3);
assert.strictEqual(result.duplicate_count, 0);
});
test('output bytes and report hash are independent of input ordering', () => {
const dirs = [record({ harness_version: 'z' }), record({ harness_version: 'a' }), record({ harness_version: 'a' })];
const result = group([...dirs, dirs[0]]);
assert.strictEqual(harness.canonical.canonicalJson(result), harness.canonical.canonicalJson(group([dirs[0], ...dirs.reverse()])));
const { report_hash, ...body } = result;
assert.strictEqual(report_hash, harness.canonical.hashValue(body));
});
test('empty journals count as snapshots with zero entries and no inferred outcome', () => {
const result = group([record({}, [])]);
assert.strictEqual(result.capsule_count, 1);
assert.strictEqual(result.groups[0].entry_count, 0);
assert.ok(Object.values(result.groups[0].by_lineage).every(n => n === 0));
assert.strictEqual(result.groups[0].sources[0].root_hash, harness.envelope.GENESIS_HASH);
});
test('prototype-like version and family labels are ordinary data', () => {
const result = group([record({ harness_version: '__proto__', task_family: 'constructor' }), record({ harness_version: 'constructor', task_family: 'constructor' })]);
assert.strictEqual(result.task_family, 'constructor');
assert.deepStrictEqual(result.groups.map(g => g.harness_version), ['__proto__', 'constructor']);
assert.strictEqual(Object.getPrototypeOf(result), Object.prototype);
});
test('mixed task families fail instead of comparing incompatible records', () => {
rejects([record(), record({ task_family: 'other-family' })], 'retrospective.mixed_task_families');
});
test('conflicting snapshots of the same capsule identity fail instead of selecting a winner', () => {
const dir = record();
const copy = path.join(root, 'conflict');
fs.cpSync(dir, copy, { recursive: true });
harness.capsule.Capsule.open(copy).append('attempt', 'later', {});
rejects([dir, copy], 'retrospective.conflicting_identity');
rejects([copy, dir], 'retrospective.conflicting_identity');
});
test('metadata-only conflicts in empty snapshots also fail', () => {
const a = record({ run_id: 'same-run', capsule_id: 'same-capsule' }, []);
const b = record({ run_id: 'same-run', capsule_id: 'same-capsule', harness_version: 'v2' }, []);
rejects([a, b], 'retrospective.conflicting_identity');
});
test('tampered and truncated journals fail without a partial success report', () => {
for (const corrupt of [bytes => bytes.replace('"start"', '"forged"'), bytes => bytes.slice(0, -1)]) {
const dir = record();
const journal = path.join(dir, 'journal.ndjson');
fs.writeFileSync(journal, corrupt(fs.readFileSync(journal, 'utf8')));
rejects([record(), dir], 'retrospective.invalid_capsule');
}
});
test('metadata mismatch and missing sources fail with only input index diagnostics', () => {
const dir = record();
const file = path.join(dir, 'capsule.json');
fs.writeFileSync(file, JSON.stringify({ ...JSON.parse(fs.readFileSync(file)), run_id: 'forged' }));
for (const invalid of [dir, path.join(root, 'private missing marker')]) {
assert.throws(() => group([invalid]), error => {
assert.strictEqual(error.code, 'retrospective.invalid_capsule');
assert.strictEqual(error.input_index, 0);
assert.ok(!error.message.includes(root));
assert.ok(!error.message.includes('private missing marker'));
return true;
});
}
});
test('rejects invalid or excessive input lists before opening any capsule', () => {
for (const value of [null, {}, 'dir', [], [null], [''], [' '], ['x\0y'], Array(2), Array(101).fill('missing')]) {
rejects(value, 'retrospective.invalid_inputs');
}
});
test('accepts the bounded maximum of 100 inputs and deduplicates them', () => {
const result = group(Array(100).fill(record()));
assert.strictEqual(result.capsule_count, 1);
assert.strictEqual(result.duplicate_count, 99);
});
test('does not mutate input lists or capsule files, and ignores stale projections', () => {
const dir = record();
fs.writeFileSync(path.join(dir, 'projection.json'), 'private stale projection marker');
const before = fs.readdirSync(dir).map(name => [name, fs.readFileSync(path.join(dir, name))]);
const inputs = Object.freeze([dir]);
const result = group(inputs);
assert.strictEqual(result.capsule_count, 1);
assert.deepStrictEqual(fs.readdirSync(dir).map(name => [name, fs.readFileSync(path.join(dir, name))]), before);
});
} finally { cleanup(root); }
finish('retrospective');