Commit Graph
6 Commits
Author SHA1 Message Date
haelyra 30c41a9bde fix: close truth and portability review gaps 2026-08-29 14:55:14 -04:00
haelyra d08331f14e fix(skill-comply): harden home path redaction 2026-08-29 14:55:13 -04:00
cyreandhaelyra 2242e4d99d fix(skill-comply): redact operator home path from compliance reports
_parse_stream_json() persisted raw tool_input/tool_response content into
ObservationEvents that grade() scores and generate_report() writes to
results/<skill>.md -- a report meant to be shared and reviewed.

--add-dir restricts the agent's additional accessible directory to the
sandbox (SANDBOX_BASE = /tmp/skill-comply-sandbox), but that doesn't stop
the agent's own tool calls (a Bash command using ~ expansion, a scenario
setup_commands entry referencing a dotfile) from emitting the operator's
home directory into tool_input/tool_response -- which then lands
verbatim, truncated but not sanitized, in the written report.

Adds _redact_home_path(), pure stdlib (Path.home()), applied to both
input_str and output_str before they're stored on the ObservationEvent.
Scoped deliberately to the home directory only -- grade() needs real
tool-call semantics for LLM-based compliance classification, so
truncating/stripping content the way a pure logging hook could isn't an
option here; only the operator-identifying path component needs to go.

New TestParseStreamJsonRedactsHomePath class in
skills/skill-comply/tests/test_runner.py (3 tests) -- full file now
10/10 passing, up from 7/7. Confirmed tests/test_invariant_runner.py (the
sandbox-execution security tests from #2149) still passes clean, 4/4.

Fixes #2730
2026-08-29 14:55:13 -04:00
Affaan MustafaandAffaan Mustafa b39d2244cf docs: salvage focused stale PR contributions
- add Vite and Redis pattern skills from closed stale PRs

- add frontend-slides support assets

- port skill-comply runner fixes and LLM prompt/provider regressions

- harden agent frontmatter validation and sync catalog counts
2026-05-11 05:31:12 -04:00
Affaan Mustafa 0fcb43ea90 fix(grader): handle forward after_step references 2026-04-12 23:36:16 -07:00
a2e465c74d feat(skills): add skill-comply — automated behavioral compliance measurement (#724)
* feat(skills): add skill-comply — automated behavioral compliance measurement

Automated compliance measurement for skills, rules, and agent definitions.
Generates behavioral specs, runs scenarios at 3 strictness levels,
classifies tool calls via LLM, and produces self-contained reports.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(skill-comply): address bot review feedback

- AGENTS.md: fix stale skill count (115 → 117) in project structure
- run.py: replace remaining print() with logger, add zero-division guard,
  create parent dirs for --output path
- runner.py: add returncode check for claude subprocess, clarify
  relative_to path traversal validation
- parser.py: use is_file() instead of exists(), catch KeyError for
  missing trace fields, add file check in parse_spec
- classifier.py: log warnings on malformed classification output,
  guard against non-dict JSON responses
- grader.py: filter negative indices from LLM classification

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 21:51:49 -07:00