WEBDIENSTE/ECC
1
0
Fork 0
mirror of https://github.com/affaan-m/ECC.git synced 2026-08-29 11:19:39 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(agents): require repeated harness eval trials

Browse Source
This commit is contained in:
haelyra
2026-08-28 21:26:42 -04:00
parent cc9b20404d
commit 6544b2f7f8
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agents/harness-optimizer.md
+1 -1
View File
@@ -36,7 +36,7 @@ Before touching any file, snapshot the current state of every path you intend to
### Step 3: Verify
Re-run the deterministic grader plus `node tests/run-all.js` (Regression Evals). If either fails, automatically restore the Step 2 snapshot so the worktree/configuration is left clean — never hand back a partially-applied change. Grade with all three eval-harness Grader Types: Code-Based (script/test exit codes), Model-Based (self-assessed diff quality), Human (any security- or safety-relevant change is BLOCKED until a human explicitly approves it — this includes broader tool permissions, credential/secret access or exfiltration paths, and any weakening of existing safety controls; for changes under `{skills,commands,agents,rules}/**`, explicitly check prompt-injection resilience, permission scope, destructive-action guards, and secret-exfiltration risk). Compute pass@k / pass^k as defined in `skills/eval-harness/SKILL.md` (pass@3 for capability changes, pass^3 for safety-critical hook changes).
Re-run `node scripts/harness-audit.js repo --format json` plus `node tests/run-all.js` (Regression Evals). If either fails, automatically restore the Step 2 snapshot so the worktree/configuration is left clean — never hand back a partially-applied change. Grade with all three eval-harness Grader Types: Code-Based (script/test exit codes), Model-Based (self-assessed diff quality), Human (any security- or safety-relevant change is BLOCKED until a human explicitly approves it — this includes broader tool permissions, credential/secret access or exfiltration paths, and any weakening of existing safety controls; for changes under `{skills,commands,agents,rules}/**`, explicitly check prompt-injection resilience, permission scope, destructive-action guards, and secret-exfiltration risk). Compute pass@k / pass^k as defined in `skills/eval-harness/SKILL.md`: run each capability eval in three independent trials before reporting pass@3, and run each safety-critical hook regression eval in three independent trials with all three passing before reporting pass^3. Record every trial result in the report.
## Output Format