ECC/docker/context-profiles/complex-eval/DESIGN.md

# ECC Complex-Task Evaluation (complex-tasks@1)

A reproducible, public benchmark of what ECC's context scoping does for **realistic
agent work** — as opposed to the 30-task repair corpus (`ai-corpus.json`), which
measures small, single-file fixes. This document is the preregistered methodology:
it was written before the first provider call against this corpus, and it is the
reference for anyone who wants to audit or rerun the evaluation.

## Research question

Does ECC's context engineering — the full skill library, manually picked skills
(manual-lean), automatic skill matching (auto-lean), and the ECC-029 changes
themselves — change what a frontier coding agent delivers on multi-step
engineering tasks, and at what cost in tokens, time, and dollars?

## Arms

Five conditions, all launched through the same evaluator with real installs in
isolated config homes, paired per task and repeat:

| Arm | What the agent gets | What it represents |
|---|---|---|
| `full` | Branch skill library installed + ECC context block (catalog/resources) | ECC with scoping machinery present but everything loaded |
| `manual-lean` | lean profile + the maintainer-chosen canonical skill(s) injected | A user who knows exactly which ECC skill applies |
| `auto-lean` | lean profile; ECC's trigger/proposal machinery picks and injects skills | The "auto" experience: no ECC knowledge required |
| `ecc-legacy` | The full skill library **from the pinned pre-ECC-029 commit** (`legacy-source.json`, currently `e482e579` = `origin/main`), bare prompt, no context block | The typical current ECC user experience before the scoping work |
| `baseline` | No ECC install, bare prompt | The provider with no ECC at all (overhead subtraction) |

`ecc-legacy` doubles as a replication control: where its install content matches
`full`, score differences between them isolate the ECC-029 deltas (rewritten
skill descriptions, scoping layer) rather than provider noise.

## The three tasks

Chosen to be the kind of work ECC exists for — multi-step, judgment-heavy,
checkpointable — while deliberately **not** shaped around ECC's current skill
list. Queries are written as a real user would phrase them, with no ECC
vocabulary, no hints about which skill applies, and no instruction to use any
particular methodology. Each task has one clear correct outcome and a
deterministic, dependency-free grader.

1. **`webhook-relay`** (feature build). Finish an asynchronous webhook delivery
   worker: retries with exponential backoff, dead-lettering after 5 attempts,
   status reporting, under load. Graded by 9 in-process behavioral probes
   (delivery after failures, exact attempt counts, backoff timing window,
   dead-lettering, error capture, API preservation, concurrency).
   *Why it belongs here:* everyday backend feature work where test discipline
   and backend patterns genuinely change outcomes; canonical skill:
   `tdd-workflow` (a second skill would exceed the 32 KB selection budget —
   itself a measured constraint of the scoping layer).

2. **`incident-triage`** (debugging / root cause). Finance reports one-cent
   total errors since yesterday's deploy. The repo contains three changelog
   entries (two red herrings), an incident log with concrete amounts, and a
   regression: a "readability" refactor that switched integer-cent math to
   decimal-factor floats, which under-rounds exact half-cent boundaries.
   Graded by 5 boundary-value totals the float path provably gets wrong, one
   regression probe, and 2 deterministic checks on the required `INCIDENT.md`
   (names the right changelog entry, explains the rounding mechanism).
   *Why it belongs here:* evidence-driven diagnosis under uncertainty is the
   highest-leverage agent workflow; guessing is penalized because red herrings
   are plausible; canonical skill: `orch-fix-defect`.

3. **`sentinel-api`** (security review + hardening). A paste service whose
   README documents the secure contract while the code violates it five ways:
   hardcoded admin token, path traversal, reflected XSS, predictable delete
   tokens, no body-size limit. Graded by 10 exploit probes (each vulnerability
   must actually be closed) plus functional regression probes (the documented
   API must still work), including one encoded-traversal variant so partial
   fixes score partially.
   *Why it belongs here:* security review is a canonical agent task with
   objectively checkable outcomes; canonical skill: `security-review`.

### Why these tests are effective

- **Realism over benchmark gaming.** Each task is a small production-shaped
  repo with docs, tests, logs, and changelogs — the inputs a real engineer (or
  a real user of an agent harness) actually has. Nothing references ECC.
- **Correctness is decidable.** Every grader assertion is deterministic:
  behavioral probes against the agent's own running service, exact numeric
  answers on boundary cases, static source checks, exploit probes. No LLM
  judges, no rubrics, no human scoring.
- **Partial credit.** Graders emit `ECC_EVAL_SCORE {"score": 0..1}`, so "found
  4 of 5 vulnerabilities" registers as 0.9-of-task progress instead of a binary
  failure. Pass/fail (score = 1.0) is reported alongside the mean score.
- **Hard to luck into.** Red herrings (incident-triage), timing windows
  (webhook-relay), and exploit-verified fixes (sentinel-api) mean superficial
  plausible work scores low.
- **Fair across arms.** Hidden graders run only after the agent exits, from a
  read-only sandbox; the agent never sees the grader. The same grader scores
  every arm identically. Reference solutions score 1.0 and as-shipped fixtures
  score ≤ 0.3 (`verify-checks.js` proves both before any provider call).

## Measured variables

Per trial (one task × arm × repeat), from the provider's own usage events:

- **Fresh input tokens** (input + cache-creation), **cache-read tokens**,
  **output tokens** — the context-cost story.
- **Provider calls** per trial (1, or 2 when auto-lean needs a routing proposal).
- **Wall-clock time** per provider call and per trial (ms) — time to completion.
- **Score** (0..1) and **pass** (score = 1.0) from the hidden grader.
- **API-equivalent cost**, derived at analysis time at Anthropic Opus list
  prices ($15 / $1.50 / $75 per million fresh-input / cache-read / output
  tokens). This is an accounting convention for comparison, not a billing
  claim; subscription pricing differs.
- **Skill routing** (auto-lean): which skills the trigger/proposal machinery
  selected vs the maintainer-chosen canonical set, reported as the selection
  probe accuracy — the direct measure of "automatic skill matching".

Comparisons are **within-run only**: same provider, model, executable digest,
corpus digest, and source digest, paired by task and repeat. Cross-run and
cross-provider comparisons are invalid by design. This is a descriptive pilot
(3 tasks × 5 arms × 4 repeats = 60 trials): it estimates direction and
magnitude, not population statistics, and the report says so in its gate block.

## Reproducing or auditing

Everything below is committed; there are no hidden inputs.

```bash
# 1. Inspect the tasks: fixtures, queries, graders, and reference solutions.
ls docker/context-profiles/complex-eval/cases/
ls docker/context-profiles/complex-eval/reference/

# 2. Prove the graders: reference solutions must score 1.0, fixtures below 1.0.
node docker/context-profiles/complex-eval/verify-checks.js

# 3. Rebuild the corpus after any fixture edit (digest-pinned at registration).
node docker/context-profiles/complex-eval/build-corpus.js

# 4. Preregister (pins corpus, source, model, executable digests; no provider).
node docker/context-profiles/ai-eval.js --plan \
  --corpus docker/context-profiles/complex-corpus.json --repeats 4 \
  --provider claude --model <model> --executable /absolute/path/to/claude \
  > registration.json

# 5. Run (requires your own Claude subscription login or API key).
node docker/context-profiles/ai-eval.js --allow-real-provider \
  --registration registration.json \
  --corpus docker/context-profiles/complex-corpus.json \
  --provider claude --model <model> --executable /absolute/path/to/claude \
  --repeats 4 --max-calls 400 --deadline-ms 25200000 --call-timeout-ms 600000 \
  --artifact-dir /absolute/path/for/transcripts > report.json
```

The registration digest binds the exact corpus, evaluator source, model, and
executable; the run refuses to start if any of them drift, and aborts if the
tree changes mid-run. `--artifact-dir` retains per-trial session transcripts
for independent inspection (they never enter the report). The `ecc-legacy` arm
is pinned by commit in `legacy-source.json` and exported from git objects at
run time. The Codex provider is unsupported for this corpus (the legacy arm has
no Codex install path); `--provider claude` is required.

## Known limits

- Three tasks is a probe, not a census: treat intervals as descriptive.
- Tasks are Node.js/stdlib by construction (graders must be hermetic); results
  say nothing about other ecosystems directly.
- `webhook-relay` uses wall-clock backoff windows; bounds are wide (250–5000ms)
  but loaded machines could in principle flake a timing probe. The grader
  reports each probe individually so flakes are visible.
- Provider behavior varies week to week; the pinned model/executable digests
  make a rerun comparable only within the same pin.
- Fixture wart observed in the 2026-09-25 run: on Node 24, `node --test test/`
  no longer scans the directory the way Node 22 did, so `npm test` fails as
  shipped. This is identical for every arm (the task says to make `npm test`
  pass, and agents fix the script), so fairness holds, but it adds unplanned
  work per trial. A future corpus revision should ship a portable test script.

## complex-tasks@2 (discriminative revision)

The @1 run saturated: every arm scored 1.000 on every task, so only economics
and routing differed. @2 (`cases2/`, built to `complex-corpus-v2.json`) is
designed to discriminate on the axes users actually pay for — correctness on
traps, solution efficiency, spec thoroughness — with wide partial-credit
spreads. The @1 corpus and its report stay untouched for comparability.

1. **`keccak-selector`** (domain-knowledge trap). Implement Ethereum function
   selectors from scratch, stdlib only. The trap: Node's crypto offers
   SHA3-256, which shares the Keccak-f[1600] permutation but differs in
   padding — the naive one-liner is wrong for every vector (verified: the
   naive control scores 0.25, format checks only). Graded by 9 selector
   vectors including a padding edge case, all cross-validated against Node's
   SHA3-256 on shared-permutation inputs. Canonical skill: `nodejs-keccak256`.
   *Hypothesis:* the skill body carries exactly this knowledge; bare agents
   must rediscover it.

2. **`event-stats-api`** (correctness edges + measured efficiency). A shipped
   implementation that is both wrong on the documented edge semantics
   (interpolated instead of nearest-rank percentiles, zeros instead of nulls,
   unrounded averages, missing 400s) and algorithmically naive (full-log scan
   and sort per query). Graded by 10 independently computed correctness probes
   plus a measured 2,000-query performance budget (threshold 6s; shipped naive
   ~7.7s, reference ~1.5s — calibrated on the grading machine in
   `calibrate-stats.js`). Canonical skill: `backend-patterns`. *Hypothesis:*
   solution *efficiency* separates arms even when correctness doesn't.

3. **`forge-cli`** (spec thoroughness + robustness). Twelve contractual
   behaviors with exact messages, exit codes, sorting, and a never-throw
   guarantee, graded by 26 checks including junk-input fuzzing and static
   hygiene (no leftover TODO/FIXME, no new dependencies). Canonical skill:
   `tdd-workflow`. *Hypothesis:* checklist discipline shows up as breadth of
   completion, and partial credit spreads the distribution.

First @2 run uses `claude-opus-4-8` (cost discipline); the corpus is
provider- and model-pinned per run, so a later Opus 5.5 rerun on the same
digest measures the model difference directly. repeats=2 (30 trials): simple
experimentation, expand later.

## complex-tasks@3 (vagueness and horizon; arms: auto-lean vs baseline)

@2 still saturated on outcomes (30/30) — enumerated specs are within the
model's cold competence. @3 (`cases3/`, built to `complex-corpus-v3.json`)
moves grading to what users actually complain about (see the complaint
taxonomy in this file's discussion: happy-path-only work, unverified
completion, skipped implied work, convention drift, concurrency blindness).
Everything graded is discoverable from repo docs visible to every arm — the
question is whether agents reliably *do* all of it under vague instruction.

1. **`chained-tickets`** (long horizon). Four sequential tickets in one
   accumulating workspace — build a link shortener core, then vague tickets:
   "links need to survive a restart", "we're seeing abuse, deal with it",
   "track redirect hits, consistent with the existing API". 33 hidden probes
   across the four steps grade function, convention compliance (error
   envelope, layering — pinned in a visible CONTRIBUTING.md), and implied
   work (changelog entries, growing tests, accurate README). Stepped trials
   grade each ticket after its call; a failed ticket ends the chain.
2. **`production-ready`** (vague prompt, heavy implication). "This goes to
   production Monday — get it ready." A documented production bar
   (validation envelopes, body limits, /health, structured request logs, env
   config, graceful SIGTERM, nosniff, error-path tests, changelog) graded by
   16 probes against a naive prototype. Fixture scores 0.063.
3. **`idempotent-webhooks`** (the "almost right" trap). A payment receiver
   whose shipped code has a textbook check-then-act race (INC-104). Hidden
   grader fires 50 concurrent identical deliveries plus replay, already-paid,
   mixed-storm, and contract probes. The naive fixture double-applies and
   crashes on unknown orders (0.25). Exactly-once requires claiming events
   synchronously — the discipline skills like `error-handling` encode.

Grader robustness (hard-won, now fixed and unit-tested): a graded server runs
in-process, so a crashing server kills the grader. Graders install
uncaughtException/unhandledRejection handlers, emit their score line via
`process.stdout.write` (immune to the log-capture patching used in probes),
pre-declare their check totals (unreached checks score zero), and the
evaluator itself treats a score-advertising grader that printed nothing as a
zero (`graderDied` guard in `runScoredCheck`). Stepped graders may write to
the workspace (persistence probes); single-step graders stay read-only.

First @3 run: arms `auto-lean` and `baseline` only, repeats=1,
`claude-opus-4-8` — the direct test of "ECC auto-routing vs no harness" on
quality, time, and tokens. Full-arm and Opus 5.5 replications follow if the
spread shows up.

## complex-tasks@4 (learning loops; adds recurring-incident)

@4 (`cases4/`, built to `complex-corpus-v4.json`) keeps the three @3 cases
unchanged and adds a fourth targeting a different ECC value prop: converting
a fix into durable, reusable prevention — and *reusing your own artifacts*
later in the session. Baseline agents can hold this in context; ECC's claim
is that skills/workflows make it systematic.

4. **`recurring-incident`** (learning loop / institutional memory). Three
   chained steps against a dependency-free payments service whose gateway
   records side effects in an append-only JSONL ledger. Step 1: keyless
   refund retries double-refund (INC-201/214/227 "third time this quarter"
   trail in `docs/incidents.md`); the vague ask is "make sure this stops
   being a recurring incident." Probes: functional correctness across a
   module reload (kills in-memory-only fixes) [0.40], regression test wired
   into the suite + mutation probe [0.30], a durable prevention runbook
   [0.20], and the mechanism living in one shared helper module [0.10].
   Step 2: payout retries, "same family of problem" — graded on REUSE of
   the step-1 helper (static import check + no divergent inline
   reimplementation) [0.30] alongside function [0.40], test+mutation [0.20],
   doc update [0.10]. Step 3: "write the handoff note" — graded on
   existence [0.20], every referenced path actually existing on disk [0.30],
   naming the helper + prevention procedure [0.30], and covering both
   incidents [0.20]. Manual skills: `error-handling`, `continuous-learning`.
   *Hypothesis:* learning-loop behavior (abstract once, reuse, document,
   hand off) separates harnessed arms from baseline even when raw bug-fix
   competence doesn't.

Verification: reference 1.000 on all steps of all four cases; naive
recurring-incident scores 0.20 / 0.00 / 0.20 per step; fixtures 0.00–0.25.

First @4 run: arm `auto-lean` only, repeats=1, `claude-opus-5-5` — the
model-difference probe against the @3 opus-4-8 numbers on the shared cases,
plus first signal on the learning-loop case.