* fix(skill-comply): stop a failed step supplying evidence downstream
`_check_temporal_order` fell back to the raw classifier output whenever
the referenced step was absent from `resolved`. A step only enters
`resolved` once it passes, so "failed" and "not graded yet" were the
same thing to that lookup, and a dependant could pass on an event
belonging to a prerequisite that had failed its own ordering check.
With three steps C, A (before C) and B (after A) and events C@T0,
A@T1, B@T2, A fails and B passed on A's classified event: 2/3 instead
of 1/3. It compounds down a chain, so one failed prerequisite could
leave a five-step workflow reading 4/5.
The grader now tracks which steps have been graded at all. A referenced
step that was graded and is missing from `resolved` failed, and its
events are refused with a reason that says so. A step not graded yet is
a forward reference to a step declared later, and the fallback stays as
it was: that is what makes an out-of-order declaration work, and the
existing regression for it goes red if the fallback is removed instead.
`before_step` deliberately keeps the old fallback. The two fail in
opposite directions: an `after_step` fallback can only turn a failure
into a pass, a `before_step` one can only turn a pass into a failure,
so dropping it would relax a constraint because some other step failed.
* fix(skill-comply): revoke a pass that rested on a later-failing prerequisite
Review of #3109 found the mirror image of the case that PR fixes. `graded`
only catches a prerequisite that had already failed when its dependant was
graded. A step declared *before* its `after_step` is graded against the
classifier's raw events for a step that has not run yet — the fallback that
makes an out-of-order declaration work — and nothing revisited it once that
step went on to fail its own checks.
Add a pass after grading that demotes any detected step whose `after_step`
ended up failing, repeated to a fixed point: one demotion can invalidate
whatever depended on it, in either declaration order. Demotion only removes
passes, so it terminates. `compliance_rate` is computed from the demoted
results.
Also from review: build `graded` and the chain test's step list as new
objects rather than mutating (AGENTS.md immutability rule), and annotate the
injected mocks in the tests this PR owns.
_parse_stream_json() persisted raw tool_input/tool_response content into
ObservationEvents that grade() scores and generate_report() writes to
results/<skill>.md -- a report meant to be shared and reviewed.
--add-dir restricts the agent's additional accessible directory to the
sandbox (SANDBOX_BASE = /tmp/skill-comply-sandbox), but that doesn't stop
the agent's own tool calls (a Bash command using ~ expansion, a scenario
setup_commands entry referencing a dotfile) from emitting the operator's
home directory into tool_input/tool_response -- which then lands
verbatim, truncated but not sanitized, in the written report.
Adds _redact_home_path(), pure stdlib (Path.home()), applied to both
input_str and output_str before they're stored on the ObservationEvent.
Scoped deliberately to the home directory only -- grade() needs real
tool-call semantics for LLM-based compliance classification, so
truncating/stripping content the way a pure logging hook could isn't an
option here; only the operator-identifying path component needs to go.
New TestParseStreamJsonRedactsHomePath class in
skills/skill-comply/tests/test_runner.py (3 tests) -- full file now
10/10 passing, up from 7/7. Confirmed tests/test_invariant_runner.py (the
sandbox-execution security tests from #2149) still passes clean, 4/4.
Fixes#2730
* fix: V-001 security vulnerability
Automated security fix generated by OrbisAI Security
* fix: sanitize subprocess call in runner.py
The runner
* fix: address PR review comments on V-001 allowlist and test coverage
Remove dangerous interpreters (python, python3, node, curl, wget) from
ALLOWED_SETUP_EXECUTABLES — they can execute arbitrary code via argument
flags and are not needed for sandbox setup. Rewrite test_invariant_runner
to call _setup_sandbox directly instead of spawning runner.py as a
subprocess (which had no __main__ entrypoint and never exercised the fix).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- add Vite and Redis pattern skills from closed stale PRs
- add frontend-slides support assets
- port skill-comply runner fixes and LLM prompt/provider regressions
- harden agent frontmatter validation and sync catalog counts
