feat(profiles): add opt-in Lean/Full and hybrid Auto selection

docker/context-profiles/complex-eval/DESIGN.md

@@ -0,0 +1,288 @@
+# ECC Complex-Task Evaluation (complex-tasks@1)
+
+A reproducible, public benchmark of what ECC's context scoping does for **realistic
+agent work** — as opposed to the 30-task repair corpus (`ai-corpus.json`), which
+measures small, single-file fixes. This document is the preregistered methodology:
+it was written before the first provider call against this corpus, and it is the
+reference for anyone who wants to audit or rerun the evaluation.
+
+## Research question
+
+Does ECC's context engineering — the full skill library, manually picked skills
+(manual-lean), automatic skill matching (auto-lean), and the ECC-029 changes
+themselves — change what a frontier coding agent delivers on multi-step
+engineering tasks, and at what cost in tokens, time, and dollars?
+
+## Arms
+
+Five conditions, all launched through the same evaluator with real installs in
+isolated config homes, paired per task and repeat:
+
+| Arm | What the agent gets | What it represents |
+|---|---|---|
+| `full` | Branch skill library installed + ECC context block (catalog/resources) | ECC with scoping machinery present but everything loaded |
+| `manual-lean` | lean profile + the maintainer-chosen canonical skill(s) injected | A user who knows exactly which ECC skill applies |
+| `auto-lean` | lean profile; ECC's trigger/proposal machinery picks and injects skills | The "auto" experience: no ECC knowledge required |
+| `ecc-legacy` | The full skill library **from the pinned pre-ECC-029 commit** (`legacy-source.json`, currently `e482e579` = `origin/main`), bare prompt, no context block | The typical current ECC user experience before the scoping work |
+| `baseline` | No ECC install, bare prompt | The provider with no ECC at all (overhead subtraction) |
+
+`ecc-legacy` doubles as a replication control: where its install content matches
+`full`, score differences between them isolate the ECC-029 deltas (rewritten
+skill descriptions, scoping layer) rather than provider noise.
+
+## The three tasks
+
+Chosen to be the kind of work ECC exists for — multi-step, judgment-heavy,
+checkpointable — while deliberately **not** shaped around ECC's current skill
+list. Queries are written as a real user would phrase them, with no ECC
+vocabulary, no hints about which skill applies, and no instruction to use any
+particular methodology. Each task has one clear correct outcome and a
+deterministic, dependency-free grader.
+
+1. **`webhook-relay`** (feature build). Finish an asynchronous webhook delivery
+   worker: retries with exponential backoff, dead-lettering after 5 attempts,
+   status reporting, under load. Graded by 9 in-process behavioral probes
+   (delivery after failures, exact attempt counts, backoff timing window,
+   dead-lettering, error capture, API preservation, concurrency).
+   *Why it belongs here:* everyday backend feature work where test discipline
+   and backend patterns genuinely change outcomes; canonical skill:
+   `tdd-workflow` (a second skill would exceed the 32 KB selection budget —
+   itself a measured constraint of the scoping layer).
+
+2. **`incident-triage`** (debugging / root cause). Finance reports one-cent
+   total errors since yesterday's deploy. The repo contains three changelog
+   entries (two red herrings), an incident log with concrete amounts, and a
+   regression: a "readability" refactor that switched integer-cent math to
+   decimal-factor floats, which under-rounds exact half-cent boundaries.
+   Graded by 5 boundary-value totals the float path provably gets wrong, one
+   regression probe, and 2 deterministic checks on the required `INCIDENT.md`
+   (names the right changelog entry, explains the rounding mechanism).
+   *Why it belongs here:* evidence-driven diagnosis under uncertainty is the
+   highest-leverage agent workflow; guessing is penalized because red herrings
+   are plausible; canonical skill: `orch-fix-defect`.
+
+3. **`sentinel-api`** (security review + hardening). A paste service whose
+   README documents the secure contract while the code violates it five ways:
+   hardcoded admin token, path traversal, reflected XSS, predictable delete
+   tokens, no body-size limit. Graded by 10 exploit probes (each vulnerability
+   must actually be closed) plus functional regression probes (the documented
+   API must still work), including one encoded-traversal variant so partial
+   fixes score partially.
+   *Why it belongs here:* security review is a canonical agent task with
+   objectively checkable outcomes; canonical skill: `security-review`.
+
+### Why these tests are effective
+
+- **Realism over benchmark gaming.** Each task is a small production-shaped
+  repo with docs, tests, logs, and changelogs — the inputs a real engineer (or
+  a real user of an agent harness) actually has. Nothing references ECC.
+- **Correctness is decidable.** Every grader assertion is deterministic:
+  behavioral probes against the agent's own running service, exact numeric
+  answers on boundary cases, static source checks, exploit probes. No LLM
+  judges, no rubrics, no human scoring.
+- **Partial credit.** Graders emit `ECC_EVAL_SCORE {"score": 0..1}`, so "found
+  4 of 5 vulnerabilities" registers as 0.9-of-task progress instead of a binary
+  failure. Pass/fail (score = 1.0) is reported alongside the mean score.
+- **Hard to luck into.** Red herrings (incident-triage), timing windows
+  (webhook-relay), and exploit-verified fixes (sentinel-api) mean superficial
+  plausible work scores low.
+- **Fair across arms.** Hidden graders run only after the agent exits, from a
+  read-only sandbox; the agent never sees the grader. The same grader scores
+  every arm identically. Reference solutions score 1.0 and as-shipped fixtures
+  score ≤ 0.3 (`verify-checks.js` proves both before any provider call).
+
+## Measured variables
+
+Per trial (one task × arm × repeat), from the provider's own usage events:
+
+- **Fresh input tokens** (input + cache-creation), **cache-read tokens**,
+  **output tokens** — the context-cost story.
+- **Provider calls** per trial (1, or 2 when auto-lean needs a routing proposal).
+- **Wall-clock time** per provider call and per trial (ms) — time to completion.
+- **Score** (0..1) and **pass** (score = 1.0) from the hidden grader.
+- **API-equivalent cost**, derived at analysis time at Anthropic Opus list
+  prices ($15 / $1.50 / $75 per million fresh-input / cache-read / output
+  tokens). This is an accounting convention for comparison, not a billing
+  claim; subscription pricing differs.
+- **Skill routing** (auto-lean): which skills the trigger/proposal machinery
+  selected vs the maintainer-chosen canonical set, reported as the selection
+  probe accuracy — the direct measure of "automatic skill matching".
+
+Comparisons are **within-run only**: same provider, model, executable digest,
+corpus digest, and source digest, paired by task and repeat. Cross-run and
+cross-provider comparisons are invalid by design. This is a descriptive pilot
+(3 tasks × 5 arms × 4 repeats = 60 trials): it estimates direction and
+magnitude, not population statistics, and the report says so in its gate block.
+
+## Reproducing or auditing
+
+Everything below is committed; there are no hidden inputs.
+
+```bash
+# 1. Inspect the tasks: fixtures, queries, graders, and reference solutions.
+ls docker/context-profiles/complex-eval/cases/
+ls docker/context-profiles/complex-eval/reference/
+
+# 2. Prove the graders: reference solutions must score 1.0, fixtures below 1.0.
+node docker/context-profiles/complex-eval/verify-checks.js
+
+# 3. Rebuild the corpus after any fixture edit (digest-pinned at registration).
+node docker/context-profiles/complex-eval/build-corpus.js
+
+# 4. Preregister (pins corpus, source, model, executable digests; no provider).
+node docker/context-profiles/ai-eval.js --plan \
+  --corpus docker/context-profiles/complex-corpus.json --repeats 4 \
+  --provider claude --model <model> --executable /absolute/path/to/claude \
+  > registration.json
+
+# 5. Run (requires your own Claude subscription login or API key).
+node docker/context-profiles/ai-eval.js --allow-real-provider \
+  --registration registration.json \
+  --corpus docker/context-profiles/complex-corpus.json \
+  --provider claude --model <model> --executable /absolute/path/to/claude \
+  --repeats 4 --max-calls 400 --deadline-ms 25200000 --call-timeout-ms 600000 \
+  --artifact-dir /absolute/path/for/transcripts > report.json
+```
+
+The registration digest binds the exact corpus, evaluator source, model, and
+executable; the run refuses to start if any of them drift, and aborts if the
+tree changes mid-run. `--artifact-dir` retains per-trial session transcripts
+for independent inspection (they never enter the report). The `ecc-legacy` arm
+is pinned by commit in `legacy-source.json` and exported from git objects at
+run time. The Codex provider is unsupported for this corpus (the legacy arm has
+no Codex install path); `--provider claude` is required.
+
+## Known limits
+
+- Three tasks is a probe, not a census: treat intervals as descriptive.
+- Tasks are Node.js/stdlib by construction (graders must be hermetic); results
+  say nothing about other ecosystems directly.
+- `webhook-relay` uses wall-clock backoff windows; bounds are wide (250–5000ms)
+  but loaded machines could in principle flake a timing probe. The grader
+  reports each probe individually so flakes are visible.
+- Provider behavior varies week to week; the pinned model/executable digests
+  make a rerun comparable only within the same pin.
+- Fixture wart observed in the 2026-09-25 run: on Node 24, `node --test test/`
+  no longer scans the directory the way Node 22 did, so `npm test` fails as
+  shipped. This is identical for every arm (the task says to make `npm test`
+  pass, and agents fix the script), so fairness holds, but it adds unplanned
+  work per trial. A future corpus revision should ship a portable test script.
+
+## complex-tasks@2 (discriminative revision)
+
+The @1 run saturated: every arm scored 1.000 on every task, so only economics
+and routing differed. @2 (`cases2/`, built to `complex-corpus-v2.json`) is
+designed to discriminate on the axes users actually pay for — correctness on
+traps, solution efficiency, spec thoroughness — with wide partial-credit
+spreads. The @1 corpus and its report stay untouched for comparability.
+
+1. **`keccak-selector`** (domain-knowledge trap). Implement Ethereum function
+   selectors from scratch, stdlib only. The trap: Node's crypto offers
+   SHA3-256, which shares the Keccak-f[1600] permutation but differs in
+   padding — the naive one-liner is wrong for every vector (verified: the
+   naive control scores 0.25, format checks only). Graded by 9 selector
+   vectors including a padding edge case, all cross-validated against Node's
+   SHA3-256 on shared-permutation inputs. Canonical skill: `nodejs-keccak256`.
+   *Hypothesis:* the skill body carries exactly this knowledge; bare agents
+   must rediscover it.
+
+2. **`event-stats-api`** (correctness edges + measured efficiency). A shipped
+   implementation that is both wrong on the documented edge semantics
+   (interpolated instead of nearest-rank percentiles, zeros instead of nulls,
+   unrounded averages, missing 400s) and algorithmically naive (full-log scan
+   and sort per query). Graded by 10 independently computed correctness probes
+   plus a measured 2,000-query performance budget (threshold 6s; shipped naive
+   ~7.7s, reference ~1.5s — calibrated on the grading machine in
+   `calibrate-stats.js`). Canonical skill: `backend-patterns`. *Hypothesis:*
+   solution *efficiency* separates arms even when correctness doesn't.
+
+3. **`forge-cli`** (spec thoroughness + robustness). Twelve contractual
+   behaviors with exact messages, exit codes, sorting, and a never-throw
+   guarantee, graded by 26 checks including junk-input fuzzing and static
+   hygiene (no leftover TODO/FIXME, no new dependencies). Canonical skill:
+   `tdd-workflow`. *Hypothesis:* checklist discipline shows up as breadth of
+   completion, and partial credit spreads the distribution.
+
+First @2 run uses `claude-opus-4-8` (cost discipline); the corpus is
+provider- and model-pinned per run, so a later Opus 5.5 rerun on the same
+digest measures the model difference directly. repeats=2 (30 trials): simple
+experimentation, expand later.
+
+## complex-tasks@3 (vagueness and horizon; arms: auto-lean vs baseline)
+
+@2 still saturated on outcomes (30/30) — enumerated specs are within the
+model's cold competence. @3 (`cases3/`, built to `complex-corpus-v3.json`)
+moves grading to what users actually complain about (see the complaint
+taxonomy in this file's discussion: happy-path-only work, unverified
+completion, skipped implied work, convention drift, concurrency blindness).
+Everything graded is discoverable from repo docs visible to every arm — the
+question is whether agents reliably *do* all of it under vague instruction.
+
+1. **`chained-tickets`** (long horizon). Four sequential tickets in one
+   accumulating workspace — build a link shortener core, then vague tickets:
+   "links need to survive a restart", "we're seeing abuse, deal with it",
+   "track redirect hits, consistent with the existing API". 33 hidden probes
+   across the four steps grade function, convention compliance (error
+   envelope, layering — pinned in a visible CONTRIBUTING.md), and implied
+   work (changelog entries, growing tests, accurate README). Stepped trials
+   grade each ticket after its call; a failed ticket ends the chain.
+2. **`production-ready`** (vague prompt, heavy implication). "This goes to
+   production Monday — get it ready." A documented production bar
+   (validation envelopes, body limits, /health, structured request logs, env
+   config, graceful SIGTERM, nosniff, error-path tests, changelog) graded by
+   16 probes against a naive prototype. Fixture scores 0.063.
+3. **`idempotent-webhooks`** (the "almost right" trap). A payment receiver
+   whose shipped code has a textbook check-then-act race (INC-104). Hidden
+   grader fires 50 concurrent identical deliveries plus replay, already-paid,
+   mixed-storm, and contract probes. The naive fixture double-applies and
+   crashes on unknown orders (0.25). Exactly-once requires claiming events
+   synchronously — the discipline skills like `error-handling` encode.
+
+Grader robustness (hard-won, now fixed and unit-tested): a graded server runs
+in-process, so a crashing server kills the grader. Graders install
+uncaughtException/unhandledRejection handlers, emit their score line via
+`process.stdout.write` (immune to the log-capture patching used in probes),
+pre-declare their check totals (unreached checks score zero), and the
+evaluator itself treats a score-advertising grader that printed nothing as a
+zero (`graderDied` guard in `runScoredCheck`). Stepped graders may write to
+the workspace (persistence probes); single-step graders stay read-only.
+
+First @3 run: arms `auto-lean` and `baseline` only, repeats=1,
+`claude-opus-4-8` — the direct test of "ECC auto-routing vs no harness" on
+quality, time, and tokens. Full-arm and Opus 5.5 replications follow if the
+spread shows up.
+
+## complex-tasks@4 (learning loops; adds recurring-incident)
+
+@4 (`cases4/`, built to `complex-corpus-v4.json`) keeps the three @3 cases
+unchanged and adds a fourth targeting a different ECC value prop: converting
+a fix into durable, reusable prevention — and *reusing your own artifacts*
+later in the session. Baseline agents can hold this in context; ECC's claim
+is that skills/workflows make it systematic.
+
+4. **`recurring-incident`** (learning loop / institutional memory). Three
+   chained steps against a dependency-free payments service whose gateway
+   records side effects in an append-only JSONL ledger. Step 1: keyless
+   refund retries double-refund (INC-201/214/227 "third time this quarter"
+   trail in `docs/incidents.md`); the vague ask is "make sure this stops
+   being a recurring incident." Probes: functional correctness across a
+   module reload (kills in-memory-only fixes) [0.40], regression test wired
+   into the suite + mutation probe [0.30], a durable prevention runbook
+   [0.20], and the mechanism living in one shared helper module [0.10].
+   Step 2: payout retries, "same family of problem" — graded on REUSE of
+   the step-1 helper (static import check + no divergent inline
+   reimplementation) [0.30] alongside function [0.40], test+mutation [0.20],
+   doc update [0.10]. Step 3: "write the handoff note" — graded on
+   existence [0.20], every referenced path actually existing on disk [0.30],
+   naming the helper + prevention procedure [0.30], and covering both
+   incidents [0.20]. Manual skills: `error-handling`, `continuous-learning`.
+   *Hypothesis:* learning-loop behavior (abstract once, reuse, document,
+   hand off) separates harnessed arms from baseline even when raw bug-fix
+   competence doesn't.
+
+Verification: reference 1.000 on all steps of all four cases; naive
+recurring-incident scores 0.20 / 0.00 / 0.20 per step; fixtures 0.00–0.25.
+
+First @4 run: arm `auto-lean` only, repeats=1, `claude-opus-5-5` — the
+model-difference probe against the @3 opus-4-8 numbers on the shared cases,
+plus first signal on the learning-loop case.