diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 8a9c91507..9a3273e3c 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -73,6 +73,13 @@ capsule recording to the hooks that already log session activity. Then the next two plan slices: offline retrospective grouping over capsules (no new rollouts) and forced-compaction tests that prove pinned constraints survive. +Offline code preparation is available as `capsule group` over explicitly +selected, verified local snapshots from one task family. It only groups recorded +counts and digests; it does not run candidates, score outcomes or promote changes. +This utility does not fulfill the executor, hook-recording or stable-taskset +prerequisites for the operational milestone below. See the +[retrospective contract](architecture/eval-harness-frameworks.md#offline-retrospective-preparation). + ### Track C: operator skills The four desk-pattern skills are present in this candidate: operator approval diff --git a/docs/architecture/eval-harness-frameworks.md b/docs/architecture/eval-harness-frameworks.md index 9696a13d5..d00e4b4e0 100644 --- a/docs/architecture/eval-harness-frameworks.md +++ b/docs/architecture/eval-harness-frameworks.md @@ -29,8 +29,10 @@ The other modules expose local utilities, not a trust decision about code. | Replay | `replay.js`, `effect-fence.js` | 04 replay-safe branching | Declared determinism and effect class per tool, content-addressed fixtures, `tool.fixture_missing` fail-closed replay, retired child preload refuses execution | | Receipt | `receipt.js` | 07 verifiable receipts | Offline receipt over capsule root, entry count, artifact digest, and gate receipt; detached signature interface; verification names the failing check | -Epics 05 (offline self-improvement) and 06 (causal triage and compaction -invariance) are not implemented. They consume the records these five produce. +Epic 05 has an offline, report-only capsule grouping utility described below. +Self-improvement, operational retrospective validation and epic 06 (causal +triage and compaction invariance) remain unimplemented. They consume the +records these five frameworks produce. ## Effect classes @@ -182,6 +184,64 @@ What the chain does not claim: it does not stop an operator from replacing the whole log. That is the job of a witnessed transparency log, which is a later, opt-in layer outside this package. +## Offline retrospective preparation + +Select 1 to 100 existing capsule directories from one task family: + +```sh +node scripts/eval-harness.js capsule group .ecc/capsules/run-41 .ecc/capsules/run-42 +``` + +```js +const { retrospective } = require('./scripts/lib/eval-harness'); +const report = retrospective.groupCapsules(['.ecc/capsules/run-41', '.ecc/capsules/run-42']); +``` + +This read-only utility recomputes each projection from the verified metadata and +journal snapshot using `capsule.project`. It never uses or repairs a saved +`projection.json`. Inputs must be small, quiescent local capsules from the same +task family; a mismatch rejects the entire report. There is no directory +discovery, hook activation, new rollout, fixture replay or candidate execution. + +`capsule-retrospective/v1` reports the task family, input count, unique capsule +count, duplicate count, and groups sorted by declared harness version. Each +group contains capsule/entry counts, all five lineage counts, all five declared +effect-class counts, and source digest references. Counts describe recorded +entries, not unique tasks, attempts, successful effects or independently +verified outcomes. Empty journals contribute one capsule and zero entries. +Payload scores, verdicts, costs, durations and pass/fail totals are not used. + +The pair `(run_id, capsule_id)` identifies a capsule for deduplication. Repeated +paths or copied snapshots count once when their verified projection hashes +match. Conflicting snapshots of that identity, including different checkpoints, +fail with `retrospective.conflicting_identity`; the utility never picks a winner. +Distinct capsule identities remain distinct even if their event shapes match. +Source references contain the canonical hash of the identity pair, entry count, +root hash, journal digest and projection hash. `report_hash` covers every other +report field; input ordering does not change the result. Repeating an input +changes input/duplicate counts and the report hash, but not the grouped counts. + +Reports omit directory arguments, raw run/capsule IDs, journal payloads and +timestamps. **Task-family and harness-version labels are returned verbatim** +and may themselves contain private text or paths. Digest references are not +anonymization: they remain linkable and low-entropy IDs can be guessed. Review +labels and report content before sharing. Neither hashes nor declared labels +authenticate a producer or prove an improvement; `report_only` is always true. + +Any invalid, unreadable or mismatched capsule rejects the whole report with +`retrospective.invalid_capsule` and a zero-based input index. Diagnostics omit +underlying reader messages and source paths. Mixed families and invalid input +lists have separate stable codes. CLI success emits JSON to stdout and exits 0; +bad usage exits 2, while verification/refusal exits 1 without partial JSON. +The command accepts no flags and does not write a report file. For a directory +name beginning with `--`, use a relative `./` prefix or an absolute path. + +This inherits the existing capsule reader's filesystem and memory limits. The +100-input cap does not bound journal bytes. It does not isolate hostile files, +serialize concurrent writers, validate a signature or establish live provenance. +Executor containment, opt-in hook recording, stable-taskset validation and the +roadmap's operational retrospective milestone remain separate prerequisites. + ## Verification gate: unavailable **Supported candidate execution backends: none, on any OS.** `runGate` and @@ -321,6 +381,7 @@ and does not validate normal prepack or clear a release. ```sh node tests/lib/eval-harness/envelope.test.js node tests/lib/eval-harness/capsule.test.js +node tests/lib/eval-harness/retrospective.test.js node tests/lib/eval-harness/gate.test.js node tests/lib/eval-harness/security.test.js node tests/lib/eval-harness/replay.test.js diff --git a/scripts/eval-harness.js b/scripts/eval-harness.js index 3fa26a743..ccf17b19b 100644 --- a/scripts/eval-harness.js +++ b/scripts/eval-harness.js @@ -7,6 +7,7 @@ * node scripts/eval-harness.js capsule verify * node scripts/eval-harness.js capsule project * node scripts/eval-harness.js capsule export + * node scripts/eval-harness.js capsule group [ ...] * node scripts/eval-harness.js gate run [--work-dir ] [--capsule ] * node scripts/eval-harness.js receipt build [--artifact ] [--gate ] [--out ] * node scripts/eval-harness.js receipt verify [--artifact ] [--gate ] @@ -26,7 +27,7 @@ function usage(message) { if (message) { process.stderr.write(`eval-harness: ${message}\n`); } - const header = fs.readFileSync(__filename, 'utf8').split('\n').slice(3, 15).map((line) => line.replace(/^ \*\s?/, '')).join('\n'); + const header = fs.readFileSync(__filename, 'utf8').split('\n').slice(3, 16).map((line) => line.replace(/^ \*\s?/, '')).join('\n'); process.stderr.write(`${header}\n`); process.exit(2); } @@ -64,6 +65,13 @@ function runExample(action) { function runCapsule(action, rest) { const dir = rest[0]; if (!dir) usage('capsule commands need a capsule directory'); + if (action === 'group') { + if (rest.length > harness.retrospective.MAX_INPUTS || rest.some(arg => !arg.trim() || arg.startsWith('--'))) { + usage(`capsule group needs 1 to ${harness.retrospective.MAX_INPUTS} directory paths and accepts no flags`); + } + print(harness.retrospective.groupCapsules(rest)); + return; + } if (action === 'verify') { const result = harness.capsule.verify(dir); print(result); diff --git a/scripts/lib/eval-harness/index.js b/scripts/lib/eval-harness/index.js index 70de84243..2ffccfbb1 100644 --- a/scripts/lib/eval-harness/index.js +++ b/scripts/lib/eval-harness/index.js @@ -5,6 +5,7 @@ * * envelope capsule-envelope/v1 contract, redaction, secret canaries * capsule append-only hash-linked journal with five lineages + * retrospective offline report-only grouping of selected capsule snapshots * gate static inspection and disabled execution gate, syntactic warnings * replay declared tool effects, fixtures, fail-closed replay, retired effect preload * receipt offline-verifiable capsule receipts @@ -16,6 +17,7 @@ module.exports = { canonical: require('./canonical'), envelope: require('./envelope'), capsule: require('./capsule'), + retrospective: require('./retrospective'), gate: require('./gate'), replay: require('./replay'), receipt: require('./receipt'), diff --git a/scripts/lib/eval-harness/retrospective.js b/scripts/lib/eval-harness/retrospective.js new file mode 100644 index 000000000..b4ae2158d --- /dev/null +++ b/scripts/lib/eval-harness/retrospective.js @@ -0,0 +1,101 @@ +'use strict'; + +/** Read-only retrospective preparation over explicit, quiescent local capsules. + * Counts are declarations in the records, never scores or promotion evidence. + * Inherits capsule.project's local-reader limits; not hostile-filesystem isolation. + */ +const capsule = require('./capsule'); +const { LINEAGES, EFFECT_CLASSES } = require('./envelope'); +const { hashValue } = require('./canonical'); + +const MAX_INPUTS = 100; +const SOURCE_FIELDS = ['entry_count', 'root_hash', 'journal_sha256', 'projection_hash']; + +class RetrospectiveError extends Error { + constructor(code, message, inputIndex) { + super(message); + this.name = 'RetrospectiveError'; + this.code = code; + if (inputIndex !== undefined) this.input_index = inputIndex; + } +} + +function readProjection(dir, index) { + try { + return capsule.project(dir); + } catch { + // Reader errors can contain private paths or journal content. No partial + // report or unchecked diagnostic content crosses the report boundary. + throw new RetrospectiveError('retrospective.invalid_capsule', `capsule at input index ${index} failed verification`, index); + } +} + +const compare = (a, b) => a < b ? -1 : a > b ? 1 : 0; +const identity = projection => JSON.stringify([projection.run_id, projection.capsule_id]); + +function summarizeGroup(harnessVersion, projections) { + const total = keys => Object.fromEntries(keys.map(key => [key, 0])); + const byLineage = total(LINEAGES); + const byEffect = total(EFFECT_CLASSES); + for (const projection of projections) { + for (const key of LINEAGES) byLineage[key] += projection.by_lineage[key]; + for (const key of EFFECT_CLASSES) byEffect[key] += projection.by_effect_class[key]; + } + return { + harness_version: harnessVersion, + capsule_count: projections.length, + entry_count: projections.reduce((sum, item) => sum + item.entry_count, 0), + by_lineage: byLineage, + by_effect_class: byEffect, + sources: [...projections].sort((a, b) => compare(identity(a), identity(b))) + .map(item => ({ + identity_hash: hashValue([item.run_id, item.capsule_id]), + ...Object.fromEntries(SOURCE_FIELDS.map(key => [key, item[key]])), + })), + }; +} + +/** + * Group up to 100 selected snapshots of one task family by harness_version. + * Duplicate identities count once only if the verified projections match. + * Different checkpoints of the same identity are ambiguous and refused. + * Does not read saved projections, copy payloads, write files or invoke tools. + */ +function groupCapsules(dirs) { + if (!Array.isArray(dirs) || dirs.length < 1 || dirs.length > MAX_INPUTS + || !Array.from(dirs).every(dir => typeof dir === 'string' && dir.trim() && !dir.includes('\0'))) { + throw new RetrospectiveError('retrospective.invalid_inputs', `supply 1 to ${MAX_INPUTS} capsule directory paths`); + } + const projections = dirs.map(readProjection); + const family = projections[0].task_family; + const unique = new Map(); + for (const [index, projection] of projections.entries()) { + if (projection.task_family !== family) { + throw new RetrospectiveError('retrospective.mixed_task_families', 'all capsules must have the same task family', index); + } + const key = identity(projection); + const previous = unique.get(key); + if (previous && previous.projection_hash !== projection.projection_hash) { + throw new RetrospectiveError('retrospective.conflicting_identity', 'conflicting snapshots share a capsule identity', index); + } + unique.set(key, projection); + } + const byHarness = new Map(); + for (const projection of unique.values()) { + const version = projection.harness_version; + byHarness.set(version, [...(byHarness.get(version) || []), projection]); + } + const report = { + schema: 'capsule-retrospective/v1', + report_only: true, + task_family: family, + input_count: dirs.length, + capsule_count: unique.size, + duplicate_count: dirs.length - unique.size, + groups: [...byHarness].sort(([a], [b]) => compare(a, b)) + .map(([version, items]) => summarizeGroup(version, items)), + }; + return { ...report, report_hash: hashValue(report) }; +} + +module.exports = { groupCapsules, MAX_INPUTS, RetrospectiveError }; diff --git a/skills/eval-harness/SKILL.md b/skills/eval-harness/SKILL.md index c3f1cf987..6076dd185 100644 --- a/skills/eval-harness/SKILL.md +++ b/skills/eval-harness/SKILL.md @@ -250,6 +250,14 @@ node scripts/eval-harness.js example closed; SE3 and above are refused in replay. Record mode invokes the registered implementation, so only register trusted functions. - Receipt: offline verification of capsule and artifact bytes, with named checks. +- Retrospective preparation: `node scripts/eval-harness.js capsule group [ ...]` + groups 1 to 100 explicitly selected, verified local capsule snapshots from one + task family by declared harness version. Repeated snapshots count once; + conflicting identities or invalid capsules reject the whole report. This is + read-only record counting, with no new rollouts, scores or promotion. Use small, + quiescent capsules. Payloads, directory arguments and raw run/capsule IDs are + omitted, but task-family/version labels are verbatim and digest references are + linkable; review them before sharing. Operational validation remains pending. Candidate execution is disabled on every OS because no verified OS containment backend is implemented. `gate run`, `runGate`, `runVariant`, direct child launch, diff --git a/tests/lib/eval-harness/cli.test.js b/tests/lib/eval-harness/cli.test.js index db3b07515..e552da166 100644 --- a/tests/lib/eval-harness/cli.test.js +++ b/tests/lib/eval-harness/cli.test.js @@ -148,4 +148,35 @@ test('capsule CLI projects and exports valid metadata, and rejects forged metada } finally { cleanup(root); } }); +test('capsule group CLI emits only a read-only report for explicit snapshots', () => { + const dir = tempDir('cli-group'); + try { + harness.capsule.Capsule.create(dir, { task_family: 'fixture-family' }) + .append('plan', 'start', { note: 'private journal marker' }); + const before = fs.readdirSync(dir).map(name => [name, fs.readFileSync(path.join(dir, name))]); + const result = run(['capsule', 'group', dir, dir]); + assert.strictEqual(result.status, 0, result.stderr); + const report = JSON.parse(result.stdout); + assert.strictEqual(report.report_only, true); + assert.strictEqual(report.capsule_count, 1); + assert.strictEqual(report.duplicate_count, 1); + assert.ok(!result.stdout.includes('private journal marker')); + assert.ok(!result.stdout.includes(dir)); + assert.deepStrictEqual(fs.readdirSync(dir).map(name => [name, fs.readFileSync(path.join(dir, name))]), before); + } finally { cleanup(dir); } +}); + +test('capsule group rejects bad usage and content without a partial report', () => { + for (const args of [[], [' '], ['--out', 'missing'], Array(101).fill('missing')]) { + const result = run(['capsule', 'group', ...args]); + assert.strictEqual(result.status, 2, result.stderr); + assert.strictEqual(result.stdout, ''); + } + const result = run(['capsule', 'group', '/missing/private-directory-marker']); + assert.strictEqual(result.status, 1); + assert.match(result.stderr, /retrospective.invalid_capsule/); + assert.ok(!result.stderr.includes('private-directory-marker')); + assert.strictEqual(result.stdout, ''); +}); + finish('cli'); diff --git a/tests/lib/eval-harness/retrospective.test.js b/tests/lib/eval-harness/retrospective.test.js new file mode 100644 index 000000000..aec4d8f10 --- /dev/null +++ b/tests/lib/eval-harness/retrospective.test.js @@ -0,0 +1,172 @@ +'use strict'; + +const assert = require('assert'); +const fs = require('fs'); +const path = require('path'); +const harness = require('../../../scripts/lib/eval-harness'); +const { test, tempDir, cleanup, finish, fixedClock } = require('./helpers'); + +const root = tempDir('retrospective'); +let next = 0; +function record(options = {}, events = [['plan', 'start', {}, 'SE0']]) { + const id = `record-${next++}`; + const dir = path.join(root, id); + const capsule = harness.capsule.Capsule.create(dir, { + run_id: id, capsule_id: id, task_family: 'fixture-family', harness_version: 'v1', + clock: fixedClock, ...options, + }); + for (const [lineage, kind, payload, effect_class] of events) capsule.append(lineage, kind, payload, { effect_class }); + return dir; +} +function group(dirs) { return harness.retrospective.groupCapsules(dirs); } +function rejects(dirs, code) { assert.throws(() => group(dirs), error => error.code === code); } + +try { + test('groups one task family by declared harness version without scoring payloads', () => { + const a = record({}, [['plan', 'start', {}, 'SE0'], ['attempt', 'done', { score: 1, verdict: 'PROMOTE' }, 'SE2']]); + const b = record(); + const c = record({ harness_version: 'v2' }, [['interaction', 'tool', { note: 'private payload marker' }, 'SE4']]); + const result = group([c, a, b]); + assert.strictEqual(result.schema, 'capsule-retrospective/v1'); + assert.strictEqual(result.report_only, true); + assert.strictEqual(result.task_family, 'fixture-family'); + assert.strictEqual(result.capsule_count, 3); + assert.strictEqual(result.input_count, 3); + assert.strictEqual(result.duplicate_count, 0); + assert.deepStrictEqual(result.groups.map(g => [g.harness_version, g.capsule_count, g.entry_count]), [['v1', 2, 3], ['v2', 1, 1]]); + assert.deepStrictEqual(result.groups[0].by_lineage, { plan: 2, attempt: 1, interaction: 0, environment: 0, strategy: 0 }); + assert.deepStrictEqual(result.groups[0].by_effect_class, { SE0: 2, SE1: 0, SE2: 1, SE3: 0, SE4: 0 }); + assert.strictEqual(result.groups[1].by_effect_class.SE4, 1); + const serialized = JSON.stringify(result); + for (const marker of ['private payload marker', 'PROMOTE', root, 'score', 'verdict', 'payload']) assert.ok(!serialized.includes(marker), marker); + }); + + test('binds every counted snapshot to the existing verified projection digests', () => { + const dir = record(); + const projection = harness.capsule.project(dir); + const source = group([dir]).groups[0].sources[0]; + assert.deepStrictEqual(source, { + identity_hash: harness.canonical.hashValue([projection.run_id, projection.capsule_id]), + ...Object.fromEntries(['entry_count', 'root_hash', 'journal_sha256', 'projection_hash'].map(key => [key, projection[key]])), + }); + }); + + test('raw run and capsule labels are omitted from the report', () => { + const dir = record({ run_id: 'private-run-label', capsule_id: 'private-capsule-label' }); + const serialized = JSON.stringify(group([dir])); + assert.ok(!serialized.includes('private-run-label')); + assert.ok(!serialized.includes('private-capsule-label')); + }); + + test('deduplicates repeated paths and copied snapshots by identity and projection', () => { + const dir = record(); + const copy = path.join(root, 'duplicate'); + fs.cpSync(dir, copy, { recursive: true }); + const result = group([dir, copy, dir]); + assert.strictEqual(result.input_count, 3); + assert.strictEqual(result.capsule_count, 1); + assert.strictEqual(result.duplicate_count, 2); + assert.strictEqual(result.groups[0].entry_count, 1); + }); + + test('identity uses both run and capsule IDs for distinct evidence', () => { + const result = group([ + record({ run_id: 'run-a', capsule_id: 'capsule-a' }), + record({ run_id: 'run-a', capsule_id: 'capsule-b' }), + record({ run_id: 'run-b', capsule_id: 'capsule-a' }), + ]); + assert.strictEqual(result.capsule_count, 3); + assert.strictEqual(result.groups[0].sources.length, 3); + assert.strictEqual(result.duplicate_count, 0); + }); + + test('output bytes and report hash are independent of input ordering', () => { + const dirs = [record({ harness_version: 'z' }), record({ harness_version: 'a' }), record({ harness_version: 'a' })]; + const result = group([...dirs, dirs[0]]); + assert.strictEqual(harness.canonical.canonicalJson(result), harness.canonical.canonicalJson(group([dirs[0], ...dirs.reverse()]))); + const { report_hash, ...body } = result; + assert.strictEqual(report_hash, harness.canonical.hashValue(body)); + }); + + test('empty journals count as snapshots with zero entries and no inferred outcome', () => { + const result = group([record({}, [])]); + assert.strictEqual(result.capsule_count, 1); + assert.strictEqual(result.groups[0].entry_count, 0); + assert.ok(Object.values(result.groups[0].by_lineage).every(n => n === 0)); + assert.strictEqual(result.groups[0].sources[0].root_hash, harness.envelope.GENESIS_HASH); + }); + + test('prototype-like version and family labels are ordinary data', () => { + const result = group([record({ harness_version: '__proto__', task_family: 'constructor' }), record({ harness_version: 'constructor', task_family: 'constructor' })]); + assert.strictEqual(result.task_family, 'constructor'); + assert.deepStrictEqual(result.groups.map(g => g.harness_version), ['__proto__', 'constructor']); + assert.strictEqual(Object.getPrototypeOf(result), Object.prototype); + }); + + test('mixed task families fail instead of comparing incompatible records', () => { + rejects([record(), record({ task_family: 'other-family' })], 'retrospective.mixed_task_families'); + }); + + test('conflicting snapshots of the same capsule identity fail instead of selecting a winner', () => { + const dir = record(); + const copy = path.join(root, 'conflict'); + fs.cpSync(dir, copy, { recursive: true }); + harness.capsule.Capsule.open(copy).append('attempt', 'later', {}); + rejects([dir, copy], 'retrospective.conflicting_identity'); + rejects([copy, dir], 'retrospective.conflicting_identity'); + }); + + test('metadata-only conflicts in empty snapshots also fail', () => { + const a = record({ run_id: 'same-run', capsule_id: 'same-capsule' }, []); + const b = record({ run_id: 'same-run', capsule_id: 'same-capsule', harness_version: 'v2' }, []); + rejects([a, b], 'retrospective.conflicting_identity'); + }); + + test('tampered and truncated journals fail without a partial success report', () => { + for (const corrupt of [bytes => bytes.replace('"start"', '"forged"'), bytes => bytes.slice(0, -1)]) { + const dir = record(); + const journal = path.join(dir, 'journal.ndjson'); + fs.writeFileSync(journal, corrupt(fs.readFileSync(journal, 'utf8'))); + rejects([record(), dir], 'retrospective.invalid_capsule'); + } + }); + + test('metadata mismatch and missing sources fail with only input index diagnostics', () => { + const dir = record(); + const file = path.join(dir, 'capsule.json'); + fs.writeFileSync(file, JSON.stringify({ ...JSON.parse(fs.readFileSync(file)), run_id: 'forged' })); + for (const invalid of [dir, path.join(root, 'private missing marker')]) { + assert.throws(() => group([invalid]), error => { + assert.strictEqual(error.code, 'retrospective.invalid_capsule'); + assert.strictEqual(error.input_index, 0); + assert.ok(!error.message.includes(root)); + assert.ok(!error.message.includes('private missing marker')); + return true; + }); + } + }); + + test('rejects invalid or excessive input lists before opening any capsule', () => { + for (const value of [null, {}, 'dir', [], [null], [''], [' '], ['x\0y'], Array(2), Array(101).fill('missing')]) { + rejects(value, 'retrospective.invalid_inputs'); + } + }); + + test('accepts the bounded maximum of 100 inputs and deduplicates them', () => { + const result = group(Array(100).fill(record())); + assert.strictEqual(result.capsule_count, 1); + assert.strictEqual(result.duplicate_count, 99); + }); + + test('does not mutate input lists or capsule files, and ignores stale projections', () => { + const dir = record(); + fs.writeFileSync(path.join(dir, 'projection.json'), 'private stale projection marker'); + const before = fs.readdirSync(dir).map(name => [name, fs.readFileSync(path.join(dir, name))]); + const inputs = Object.freeze([dir]); + const result = group(inputs); + assert.strictEqual(result.capsule_count, 1); + assert.deepStrictEqual(fs.readdirSync(dir).map(name => [name, fs.readFileSync(path.join(dir, name))]), before); + }); +} finally { cleanup(root); } + +finish('retrospective');