Commit Graph
3 Commits
Author SHA1 Message Date
LukasandGitHub 8b951d3bf6 fix(skill-comply): stop a failed step supplying evidence downstream (#3109)
* fix(skill-comply): stop a failed step supplying evidence downstream

`_check_temporal_order` fell back to the raw classifier output whenever
the referenced step was absent from `resolved`. A step only enters
`resolved` once it passes, so "failed" and "not graded yet" were the
same thing to that lookup, and a dependant could pass on an event
belonging to a prerequisite that had failed its own ordering check.

With three steps C, A (before C) and B (after A) and events C@T0,
A@T1, B@T2, A fails and B passed on A's classified event: 2/3 instead
of 1/3. It compounds down a chain, so one failed prerequisite could
leave a five-step workflow reading 4/5.

The grader now tracks which steps have been graded at all. A referenced
step that was graded and is missing from `resolved` failed, and its
events are refused with a reason that says so. A step not graded yet is
a forward reference to a step declared later, and the fallback stays as
it was: that is what makes an out-of-order declaration work, and the
existing regression for it goes red if the fallback is removed instead.

`before_step` deliberately keeps the old fallback. The two fail in
opposite directions: an `after_step` fallback can only turn a failure
into a pass, a `before_step` one can only turn a pass into a failure,
so dropping it would relax a constraint because some other step failed.

* fix(skill-comply): revoke a pass that rested on a later-failing prerequisite

Review of #3109 found the mirror image of the case that PR fixes. `graded`
only catches a prerequisite that had already failed when its dependant was
graded. A step declared *before* its `after_step` is graded against the
classifier's raw events for a step that has not run yet — the fallback that
makes an out-of-order declaration work — and nothing revisited it once that
step went on to fail its own checks.

Add a pass after grading that demotes any detected step whose `after_step`
ended up failing, repeated to a fixed point: one demotion can invalidate
whatever depended on it, in either declaration order. Demotion only removes
passes, so it terminates. `compliance_rate` is computed from the demoted
results.

Also from review: build `graded` and the chain test's step list as new
objects rather than mutating (AGENTS.md immutability rule), and annotate the
injected mocks in the tests this PR owns.
2026-09-21 14:39:20 -04:00
Affaan Mustafa 0fcb43ea90 fix(grader): handle forward after_step references 2026-04-12 23:36:16 -07:00
a2e465c74d feat(skills): add skill-comply — automated behavioral compliance measurement (#724)
* feat(skills): add skill-comply — automated behavioral compliance measurement

Automated compliance measurement for skills, rules, and agent definitions.
Generates behavioral specs, runs scenarios at 3 strictness levels,
classifies tool calls via LLM, and produces self-contained reports.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(skill-comply): address bot review feedback

- AGENTS.md: fix stale skill count (115 → 117) in project structure
- run.py: replace remaining print() with logger, add zero-division guard,
  create parent dirs for --output path
- runner.py: add returncode check for claude subprocess, clarify
  relative_to path traversal validation
- parser.py: use is_file() instead of exists(), catch KeyError for
  missing trace fields, add file check in parse_spec
- classifier.py: log warnings on malformed classification output,
  guard against non-dict JSON responses
- grader.py: filter negative indices from LLM classification

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 21:51:49 -07:00