Commit Graph
9 Commits
Author SHA1 Message Date
haelyra 6ef38a2993 fix: validate evaluator files before pipe reads and strengthen grader controls 2026-10-11 17:54:29 -04:00
haelyra 6c9c25158a fix: address evaluator portability and recording review findings 2026-10-11 15:35:17 -04:00
haelyra c4b8325945 fix: prepare six focused ECC reliability repairs
Preserve complete source ancestry for PRs #3512, #3513, #3515, #3516, #3517 and #3518. Repair bounded recovery, alias compatibility and evaluator packaging gaps while keeping registry reuse inside the canonical loader. Verified actual hook, setup, dashboard, evaluator, sandbox and profile workflows plus full npm, Python, lint and supply-chain checks. ECC-043 preparation only; integration remains with the delivery owner.
2026-10-11 11:10:53 -04:00
Eneko Ruiz 95585e4b29 fix: read piped tasks when evaluating output files
RED: the documented --output pipe mode discarded its task. GREEN: 8 tests pass across existing input modes and the real CLI conciseness score. Changed input functions have 100% line coverage.
2026-10-11 11:37:33 +02:00
Hawthorn 8d360fb466 fix: address remaining review nits
- Add top-level hooks wrapper to second JSON example (consistent with hooks.json format)
- Extract hardcoded thresholds as module-level constants (WALL_OF_TEXT_WORDS,
  SUMMARY_CHECK_WORDS, SUMMARY_CHECK_FIRST_N, TASK_OUTPUT_RATIO_HIGH/MEDIUM)

Skipped (not applicable):
- 'Scoring defaults to 5/5' — by design for heuristic fallback; SKILL.md already
  documents pairing with LLM judge for production use
- '--output silently ignored' — already fixed by _read_input refactor (checks
  args.output directly, not elif args.task and args.output)
2026-06-10 18:27:27 +05:30
Hawthorn 7c0a0049a8 fix: address second-round review comments
- Replace httpx.Retry references with correct httpx API usage across all files
  (httpx has no built-in Retry class; use HTTPTransport/Limits instead)
- Fix _check_summary to check first 100 words (not 100 characters)
- Fix template to only show → improvement arrow for non-5 scores
- Clarify hook documentation: hook echoes reminder, does not run evaluator
- Add return type annotation to main()
- Make required parameter keyword-only in _read_file_or_text
2026-06-10 17:59:25 +05:30
Hawthorn 2ea4d779a3 fix: address self-evaluation review comments
- Clarify that agent-evaluator reads skills/agent-self-evaluation/SKILL.md directly
- Standardize on Conciseness terminology, including helper names
- Remove invalid Stop hook matcher and avoid unsupported command-expression matcher examples
- Add explicit hook-integration reference path in SKILL.md
- Add summary and self-check fields to evaluate.py output, template, and agent spec
- Refactor evaluate.py clarity and input parsing helpers
- Remove unused task parameter from check_completeness

Validation:
- python3 -m py_compile skills/agent-self-evaluation/scripts/evaluate.py
- evaluate.py high/low example smoke tests
- node scripts/ci/validate-agents.js
- node scripts/ci/validate-skills.js
- node scripts/ci/validate-hooks.js
- node scripts/ci/validate-no-personal-paths.js
2026-06-10 17:25:24 +05:30
Hawthorn c0f651cf85 fix: align report format across evaluate.py, agent spec, and template
- evaluate.py: add CRITICAL ISSUES (axes ≤ 2) section, VERDICT line
- agent-evaluator.md: match format_report output exactly (title, evidence markers, bar graphs)
- templates/evaluation-report.md: match evaluate.py output format
- All now produce identical AGENT SELF-EVALUATION REPORT structure

Single authoritative format: evaluate.py's format_report() output.
2026-06-10 17:11:44 +05:30
Hawthorn bd45947941 feat(skills,agents): add agent-self-evaluation skill and agent-evaluator persona
Add structured 5-axis self-evaluation framework for agent output quality:
- Accuracy, Completeness, Clarity, Actionability, Conciseness
- Evidence-based scoring with concrete improvement suggestions
- Standalone Python evaluator script with keyword heuristics
- Detailed scoring anchors reference guide
- High-score and low-score annotated examples
- Reusable evaluation report template
- Optional hook integration for session-stop evaluation

Agent persona (agent-evaluator) provides a dedicated subagent
for applying the rubric to agent output with tool-backed verification.

All files tested: Python script runs, examples score correctly
(high 4.2, low 3.4), frontmatter parses clean, 183 lines (under 500).
2026-06-10 16:56:18 +05:30