mirror of
https://github.com/affaan-m/ECC.git
synced 2026-10-20 06:51:44 +02:00
feat(profiles): add opt-in Lean/Full and hybrid Auto selection
This commit is contained in:
haelyra
@@ -0,0 +1,288 @@
|
||||
# ECC Complex-Task Evaluation (complex-tasks@1)
|
||||
|
||||
A reproducible, public benchmark of what ECC's context scoping does for **realistic
|
||||
agent work** — as opposed to the 30-task repair corpus (`ai-corpus.json`), which
|
||||
measures small, single-file fixes. This document is the preregistered methodology:
|
||||
it was written before the first provider call against this corpus, and it is the
|
||||
reference for anyone who wants to audit or rerun the evaluation.
|
||||
|
||||
## Research question
|
||||
|
||||
Does ECC's context engineering — the full skill library, manually picked skills
|
||||
(manual-lean), automatic skill matching (auto-lean), and the ECC-029 changes
|
||||
themselves — change what a frontier coding agent delivers on multi-step
|
||||
engineering tasks, and at what cost in tokens, time, and dollars?
|
||||
|
||||
## Arms
|
||||
|
||||
Five conditions, all launched through the same evaluator with real installs in
|
||||
isolated config homes, paired per task and repeat:
|
||||
|
||||
| Arm | What the agent gets | What it represents |
|
||||
|---|---|---|
|
||||
| `full` | Branch skill library installed + ECC context block (catalog/resources) | ECC with scoping machinery present but everything loaded |
|
||||
| `manual-lean` | lean profile + the maintainer-chosen canonical skill(s) injected | A user who knows exactly which ECC skill applies |
|
||||
| `auto-lean` | lean profile; ECC's trigger/proposal machinery picks and injects skills | The "auto" experience: no ECC knowledge required |
|
||||
| `ecc-legacy` | The full skill library **from the pinned pre-ECC-029 commit** (`legacy-source.json`, currently `e482e579` = `origin/main`), bare prompt, no context block | The typical current ECC user experience before the scoping work |
|
||||
| `baseline` | No ECC install, bare prompt | The provider with no ECC at all (overhead subtraction) |
|
||||
|
||||
`ecc-legacy` doubles as a replication control: where its install content matches
|
||||
`full`, score differences between them isolate the ECC-029 deltas (rewritten
|
||||
skill descriptions, scoping layer) rather than provider noise.
|
||||
|
||||
## The three tasks
|
||||
|
||||
Chosen to be the kind of work ECC exists for — multi-step, judgment-heavy,
|
||||
checkpointable — while deliberately **not** shaped around ECC's current skill
|
||||
list. Queries are written as a real user would phrase them, with no ECC
|
||||
vocabulary, no hints about which skill applies, and no instruction to use any
|
||||
particular methodology. Each task has one clear correct outcome and a
|
||||
deterministic, dependency-free grader.
|
||||
|
||||
1. **`webhook-relay`** (feature build). Finish an asynchronous webhook delivery
|
||||
worker: retries with exponential backoff, dead-lettering after 5 attempts,
|
||||
status reporting, under load. Graded by 9 in-process behavioral probes
|
||||
(delivery after failures, exact attempt counts, backoff timing window,
|
||||
dead-lettering, error capture, API preservation, concurrency).
|
||||
*Why it belongs here:* everyday backend feature work where test discipline
|
||||
and backend patterns genuinely change outcomes; canonical skill:
|
||||
`tdd-workflow` (a second skill would exceed the 32 KB selection budget —
|
||||
itself a measured constraint of the scoping layer).
|
||||
|
||||
2. **`incident-triage`** (debugging / root cause). Finance reports one-cent
|
||||
total errors since yesterday's deploy. The repo contains three changelog
|
||||
entries (two red herrings), an incident log with concrete amounts, and a
|
||||
regression: a "readability" refactor that switched integer-cent math to
|
||||
decimal-factor floats, which under-rounds exact half-cent boundaries.
|
||||
Graded by 5 boundary-value totals the float path provably gets wrong, one
|
||||
regression probe, and 2 deterministic checks on the required `INCIDENT.md`
|
||||
(names the right changelog entry, explains the rounding mechanism).
|
||||
*Why it belongs here:* evidence-driven diagnosis under uncertainty is the
|
||||
highest-leverage agent workflow; guessing is penalized because red herrings
|
||||
are plausible; canonical skill: `orch-fix-defect`.
|
||||
|
||||
3. **`sentinel-api`** (security review + hardening). A paste service whose
|
||||
README documents the secure contract while the code violates it five ways:
|
||||
hardcoded admin token, path traversal, reflected XSS, predictable delete
|
||||
tokens, no body-size limit. Graded by 10 exploit probes (each vulnerability
|
||||
must actually be closed) plus functional regression probes (the documented
|
||||
API must still work), including one encoded-traversal variant so partial
|
||||
fixes score partially.
|
||||
*Why it belongs here:* security review is a canonical agent task with
|
||||
objectively checkable outcomes; canonical skill: `security-review`.
|
||||
|
||||
### Why these tests are effective
|
||||
|
||||
- **Realism over benchmark gaming.** Each task is a small production-shaped
|
||||
repo with docs, tests, logs, and changelogs — the inputs a real engineer (or
|
||||
a real user of an agent harness) actually has. Nothing references ECC.
|
||||
- **Correctness is decidable.** Every grader assertion is deterministic:
|
||||
behavioral probes against the agent's own running service, exact numeric
|
||||
answers on boundary cases, static source checks, exploit probes. No LLM
|
||||
judges, no rubrics, no human scoring.
|
||||
- **Partial credit.** Graders emit `ECC_EVAL_SCORE {"score": 0..1}`, so "found
|
||||
4 of 5 vulnerabilities" registers as 0.9-of-task progress instead of a binary
|
||||
failure. Pass/fail (score = 1.0) is reported alongside the mean score.
|
||||
- **Hard to luck into.** Red herrings (incident-triage), timing windows
|
||||
(webhook-relay), and exploit-verified fixes (sentinel-api) mean superficial
|
||||
plausible work scores low.
|
||||
- **Fair across arms.** Hidden graders run only after the agent exits, from a
|
||||
read-only sandbox; the agent never sees the grader. The same grader scores
|
||||
every arm identically. Reference solutions score 1.0 and as-shipped fixtures
|
||||
score ≤ 0.3 (`verify-checks.js` proves both before any provider call).
|
||||
|
||||
## Measured variables
|
||||
|
||||
Per trial (one task × arm × repeat), from the provider's own usage events:
|
||||
|
||||
- **Fresh input tokens** (input + cache-creation), **cache-read tokens**,
|
||||
**output tokens** — the context-cost story.
|
||||
- **Provider calls** per trial (1, or 2 when auto-lean needs a routing proposal).
|
||||
- **Wall-clock time** per provider call and per trial (ms) — time to completion.
|
||||
- **Score** (0..1) and **pass** (score = 1.0) from the hidden grader.
|
||||
- **API-equivalent cost**, derived at analysis time at Anthropic Opus list
|
||||
prices ($15 / $1.50 / $75 per million fresh-input / cache-read / output
|
||||
tokens). This is an accounting convention for comparison, not a billing
|
||||
claim; subscription pricing differs.
|
||||
- **Skill routing** (auto-lean): which skills the trigger/proposal machinery
|
||||
selected vs the maintainer-chosen canonical set, reported as the selection
|
||||
probe accuracy — the direct measure of "automatic skill matching".
|
||||
|
||||
Comparisons are **within-run only**: same provider, model, executable digest,
|
||||
corpus digest, and source digest, paired by task and repeat. Cross-run and
|
||||
cross-provider comparisons are invalid by design. This is a descriptive pilot
|
||||
(3 tasks × 5 arms × 4 repeats = 60 trials): it estimates direction and
|
||||
magnitude, not population statistics, and the report says so in its gate block.
|
||||
|
||||
## Reproducing or auditing
|
||||
|
||||
Everything below is committed; there are no hidden inputs.
|
||||
|
||||
```bash
|
||||
# 1. Inspect the tasks: fixtures, queries, graders, and reference solutions.
|
||||
ls docker/context-profiles/complex-eval/cases/
|
||||
ls docker/context-profiles/complex-eval/reference/
|
||||
|
||||
# 2. Prove the graders: reference solutions must score 1.0, fixtures below 1.0.
|
||||
node docker/context-profiles/complex-eval/verify-checks.js
|
||||
|
||||
# 3. Rebuild the corpus after any fixture edit (digest-pinned at registration).
|
||||
node docker/context-profiles/complex-eval/build-corpus.js
|
||||
|
||||
# 4. Preregister (pins corpus, source, model, executable digests; no provider).
|
||||
node docker/context-profiles/ai-eval.js --plan \
|
||||
--corpus docker/context-profiles/complex-corpus.json --repeats 4 \
|
||||
--provider claude --model <model> --executable /absolute/path/to/claude \
|
||||
> registration.json
|
||||
|
||||
# 5. Run (requires your own Claude subscription login or API key).
|
||||
node docker/context-profiles/ai-eval.js --allow-real-provider \
|
||||
--registration registration.json \
|
||||
--corpus docker/context-profiles/complex-corpus.json \
|
||||
--provider claude --model <model> --executable /absolute/path/to/claude \
|
||||
--repeats 4 --max-calls 400 --deadline-ms 25200000 --call-timeout-ms 600000 \
|
||||
--artifact-dir /absolute/path/for/transcripts > report.json
|
||||
```
|
||||
|
||||
The registration digest binds the exact corpus, evaluator source, model, and
|
||||
executable; the run refuses to start if any of them drift, and aborts if the
|
||||
tree changes mid-run. `--artifact-dir` retains per-trial session transcripts
|
||||
for independent inspection (they never enter the report). The `ecc-legacy` arm
|
||||
is pinned by commit in `legacy-source.json` and exported from git objects at
|
||||
run time. The Codex provider is unsupported for this corpus (the legacy arm has
|
||||
no Codex install path); `--provider claude` is required.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Three tasks is a probe, not a census: treat intervals as descriptive.
|
||||
- Tasks are Node.js/stdlib by construction (graders must be hermetic); results
|
||||
say nothing about other ecosystems directly.
|
||||
- `webhook-relay` uses wall-clock backoff windows; bounds are wide (250–5000ms)
|
||||
but loaded machines could in principle flake a timing probe. The grader
|
||||
reports each probe individually so flakes are visible.
|
||||
- Provider behavior varies week to week; the pinned model/executable digests
|
||||
make a rerun comparable only within the same pin.
|
||||
- Fixture wart observed in the 2026-09-25 run: on Node 24, `node --test test/`
|
||||
no longer scans the directory the way Node 22 did, so `npm test` fails as
|
||||
shipped. This is identical for every arm (the task says to make `npm test`
|
||||
pass, and agents fix the script), so fairness holds, but it adds unplanned
|
||||
work per trial. A future corpus revision should ship a portable test script.
|
||||
|
||||
## complex-tasks@2 (discriminative revision)
|
||||
|
||||
The @1 run saturated: every arm scored 1.000 on every task, so only economics
|
||||
and routing differed. @2 (`cases2/`, built to `complex-corpus-v2.json`) is
|
||||
designed to discriminate on the axes users actually pay for — correctness on
|
||||
traps, solution efficiency, spec thoroughness — with wide partial-credit
|
||||
spreads. The @1 corpus and its report stay untouched for comparability.
|
||||
|
||||
1. **`keccak-selector`** (domain-knowledge trap). Implement Ethereum function
|
||||
selectors from scratch, stdlib only. The trap: Node's crypto offers
|
||||
SHA3-256, which shares the Keccak-f[1600] permutation but differs in
|
||||
padding — the naive one-liner is wrong for every vector (verified: the
|
||||
naive control scores 0.25, format checks only). Graded by 9 selector
|
||||
vectors including a padding edge case, all cross-validated against Node's
|
||||
SHA3-256 on shared-permutation inputs. Canonical skill: `nodejs-keccak256`.
|
||||
*Hypothesis:* the skill body carries exactly this knowledge; bare agents
|
||||
must rediscover it.
|
||||
|
||||
2. **`event-stats-api`** (correctness edges + measured efficiency). A shipped
|
||||
implementation that is both wrong on the documented edge semantics
|
||||
(interpolated instead of nearest-rank percentiles, zeros instead of nulls,
|
||||
unrounded averages, missing 400s) and algorithmically naive (full-log scan
|
||||
and sort per query). Graded by 10 independently computed correctness probes
|
||||
plus a measured 2,000-query performance budget (threshold 6s; shipped naive
|
||||
~7.7s, reference ~1.5s — calibrated on the grading machine in
|
||||
`calibrate-stats.js`). Canonical skill: `backend-patterns`. *Hypothesis:*
|
||||
solution *efficiency* separates arms even when correctness doesn't.
|
||||
|
||||
3. **`forge-cli`** (spec thoroughness + robustness). Twelve contractual
|
||||
behaviors with exact messages, exit codes, sorting, and a never-throw
|
||||
guarantee, graded by 26 checks including junk-input fuzzing and static
|
||||
hygiene (no leftover TODO/FIXME, no new dependencies). Canonical skill:
|
||||
`tdd-workflow`. *Hypothesis:* checklist discipline shows up as breadth of
|
||||
completion, and partial credit spreads the distribution.
|
||||
|
||||
First @2 run uses `claude-opus-4-8` (cost discipline); the corpus is
|
||||
provider- and model-pinned per run, so a later Opus 5.5 rerun on the same
|
||||
digest measures the model difference directly. repeats=2 (30 trials): simple
|
||||
experimentation, expand later.
|
||||
|
||||
## complex-tasks@3 (vagueness and horizon; arms: auto-lean vs baseline)
|
||||
|
||||
@2 still saturated on outcomes (30/30) — enumerated specs are within the
|
||||
model's cold competence. @3 (`cases3/`, built to `complex-corpus-v3.json`)
|
||||
moves grading to what users actually complain about (see the complaint
|
||||
taxonomy in this file's discussion: happy-path-only work, unverified
|
||||
completion, skipped implied work, convention drift, concurrency blindness).
|
||||
Everything graded is discoverable from repo docs visible to every arm — the
|
||||
question is whether agents reliably *do* all of it under vague instruction.
|
||||
|
||||
1. **`chained-tickets`** (long horizon). Four sequential tickets in one
|
||||
accumulating workspace — build a link shortener core, then vague tickets:
|
||||
"links need to survive a restart", "we're seeing abuse, deal with it",
|
||||
"track redirect hits, consistent with the existing API". 33 hidden probes
|
||||
across the four steps grade function, convention compliance (error
|
||||
envelope, layering — pinned in a visible CONTRIBUTING.md), and implied
|
||||
work (changelog entries, growing tests, accurate README). Stepped trials
|
||||
grade each ticket after its call; a failed ticket ends the chain.
|
||||
2. **`production-ready`** (vague prompt, heavy implication). "This goes to
|
||||
production Monday — get it ready." A documented production bar
|
||||
(validation envelopes, body limits, /health, structured request logs, env
|
||||
config, graceful SIGTERM, nosniff, error-path tests, changelog) graded by
|
||||
16 probes against a naive prototype. Fixture scores 0.063.
|
||||
3. **`idempotent-webhooks`** (the "almost right" trap). A payment receiver
|
||||
whose shipped code has a textbook check-then-act race (INC-104). Hidden
|
||||
grader fires 50 concurrent identical deliveries plus replay, already-paid,
|
||||
mixed-storm, and contract probes. The naive fixture double-applies and
|
||||
crashes on unknown orders (0.25). Exactly-once requires claiming events
|
||||
synchronously — the discipline skills like `error-handling` encode.
|
||||
|
||||
Grader robustness (hard-won, now fixed and unit-tested): a graded server runs
|
||||
in-process, so a crashing server kills the grader. Graders install
|
||||
uncaughtException/unhandledRejection handlers, emit their score line via
|
||||
`process.stdout.write` (immune to the log-capture patching used in probes),
|
||||
pre-declare their check totals (unreached checks score zero), and the
|
||||
evaluator itself treats a score-advertising grader that printed nothing as a
|
||||
zero (`graderDied` guard in `runScoredCheck`). Stepped graders may write to
|
||||
the workspace (persistence probes); single-step graders stay read-only.
|
||||
|
||||
First @3 run: arms `auto-lean` and `baseline` only, repeats=1,
|
||||
`claude-opus-4-8` — the direct test of "ECC auto-routing vs no harness" on
|
||||
quality, time, and tokens. Full-arm and Opus 5.5 replications follow if the
|
||||
spread shows up.
|
||||
|
||||
## complex-tasks@4 (learning loops; adds recurring-incident)
|
||||
|
||||
@4 (`cases4/`, built to `complex-corpus-v4.json`) keeps the three @3 cases
|
||||
unchanged and adds a fourth targeting a different ECC value prop: converting
|
||||
a fix into durable, reusable prevention — and *reusing your own artifacts*
|
||||
later in the session. Baseline agents can hold this in context; ECC's claim
|
||||
is that skills/workflows make it systematic.
|
||||
|
||||
4. **`recurring-incident`** (learning loop / institutional memory). Three
|
||||
chained steps against a dependency-free payments service whose gateway
|
||||
records side effects in an append-only JSONL ledger. Step 1: keyless
|
||||
refund retries double-refund (INC-201/214/227 "third time this quarter"
|
||||
trail in `docs/incidents.md`); the vague ask is "make sure this stops
|
||||
being a recurring incident." Probes: functional correctness across a
|
||||
module reload (kills in-memory-only fixes) [0.40], regression test wired
|
||||
into the suite + mutation probe [0.30], a durable prevention runbook
|
||||
[0.20], and the mechanism living in one shared helper module [0.10].
|
||||
Step 2: payout retries, "same family of problem" — graded on REUSE of
|
||||
the step-1 helper (static import check + no divergent inline
|
||||
reimplementation) [0.30] alongside function [0.40], test+mutation [0.20],
|
||||
doc update [0.10]. Step 3: "write the handoff note" — graded on
|
||||
existence [0.20], every referenced path actually existing on disk [0.30],
|
||||
naming the helper + prevention procedure [0.30], and covering both
|
||||
incidents [0.20]. Manual skills: `error-handling`, `continuous-learning`.
|
||||
*Hypothesis:* learning-loop behavior (abstract once, reuse, document,
|
||||
hand off) separates harnessed arms from baseline even when raw bug-fix
|
||||
competence doesn't.
|
||||
|
||||
Verification: reference 1.000 on all steps of all four cases; naive
|
||||
recurring-incident scores 0.20 / 0.00 / 0.20 per step; fixtures 0.00–0.25.
|
||||
|
||||
First @4 run: arm `auto-lean` only, repeats=1, `claude-opus-5-5` — the
|
||||
model-difference probe against the @3 opus-4-8 numbers on the shared cases,
|
||||
plus first signal on the learning-loop case.
|
||||
@@ -0,0 +1,67 @@
|
||||
'use strict';
|
||||
// Development tool: assembles a complex corpus JSON from a reviewed fixture
|
||||
// tree. Usage: node build-corpus.js [casesDir=cases] [outFile=complex-corpus.json] [corpusId=complex-tasks@1]
|
||||
// Run after editing any fixture, query, or grader; commit the tree and the
|
||||
// regenerated corpus together.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const root = __dirname;
|
||||
const casesDir = path.join(root, process.argv[2] || 'cases');
|
||||
const OUT = path.join(root, '..', process.argv[3] || 'complex-corpus.json');
|
||||
const corpusId = process.argv[4] || 'complex-tasks@1';
|
||||
|
||||
function collect(directory, prefix = '') {
|
||||
const files = {};
|
||||
for (const entry of fs.readdirSync(directory, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name))) {
|
||||
const relative = prefix ? `${prefix}/${entry.name}` : entry.name;
|
||||
if (entry.isDirectory()) Object.assign(files, collect(path.join(directory, entry.name), relative));
|
||||
else if (entry.isFile()) files[relative] = fs.readFileSync(path.join(directory, entry.name), 'utf8');
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
const tasks = [];
|
||||
const selection = [];
|
||||
for (const id of fs.readdirSync(casesDir).sort()) {
|
||||
const directory = path.join(casesDir, id);
|
||||
const meta = JSON.parse(fs.readFileSync(path.join(directory, 'meta.json'), 'utf8'));
|
||||
if (meta.id !== id || !/^[a-z][a-z0-9-]{0,63}$/.test(id)) throw new Error(`Invalid task metadata in ${id}`);
|
||||
const files = collect(path.join(directory, 'files'));
|
||||
const stepsDir = path.join(directory, 'steps');
|
||||
let task;
|
||||
if (fs.existsSync(stepsDir)) {
|
||||
const steps = fs.readdirSync(stepsDir).sort().map((name, index) => ({
|
||||
query: fs.readFileSync(path.join(stepsDir, name, 'query.md'), 'utf8').trim(),
|
||||
check: fs.readFileSync(path.join(stepsDir, name, 'check.cjs'), 'utf8'),
|
||||
...(meta.steps?.[index]?.manualIds ? { manualIds: meta.steps[index].manualIds } : {}),
|
||||
...((meta.steps?.[index]?.checkTimeoutMs || meta.checkTimeoutMs)
|
||||
? { checkTimeoutMs: meta.steps?.[index]?.checkTimeoutMs || meta.checkTimeoutMs } : {}),
|
||||
}));
|
||||
task = { id, category: meta.category, manualIds: meta.manualIds || [], files, steps };
|
||||
} else {
|
||||
const query = fs.readFileSync(path.join(directory, 'query.md'), 'utf8').trim();
|
||||
task = { id, category: meta.category, manualIds: meta.manualIds,
|
||||
...(meta.checkTimeoutMs ? { checkTimeoutMs: meta.checkTimeoutMs } : {}),
|
||||
query, files, check: fs.readFileSync(path.join(directory, 'check.cjs'), 'utf8') };
|
||||
}
|
||||
tasks.push(task);
|
||||
selection.push({ id: meta.selection.id, category: meta.selection.category,
|
||||
query: meta.selection.query || task.query || task.steps.map(step => step.query).join(' '),
|
||||
expectedIds: meta.selection.expectedIds });
|
||||
}
|
||||
|
||||
const corpus = {
|
||||
schemaVersion: 'ecc.context-eval-complex-corpus.v1',
|
||||
id: corpusId,
|
||||
sampling: 'Realistic multi-file engineering tasks, fixed before any provider call, with deterministic '
|
||||
+ 'hidden graders scoring partial credit (ECC_EVAL_SCORE). Descriptive pilot: no '
|
||||
+ 'population-representativeness claim. See complex-eval/DESIGN.md for the preregistered methodology.',
|
||||
minimumDistinctTasks: tasks.length,
|
||||
nonInferiorityMargin: 0.05,
|
||||
selection,
|
||||
tasks,
|
||||
};
|
||||
fs.writeFileSync(OUT, `${JSON.stringify(corpus, null, 1)}\n`);
|
||||
console.log(`wrote ${path.basename(OUT)} (${corpusId}): ${tasks.length} tasks, ${selection.length} selection probes, `
|
||||
+ `${tasks.reduce((sum, task) => sum + Object.keys(task.files).length, 0)} fixture files`);
|
||||
@@ -0,0 +1,73 @@
|
||||
'use strict';
|
||||
// Calibration harness (not shipped in the corpus): measures the 2,000-query
|
||||
// workload wall time for the shipped naive app and the reference app, each
|
||||
// staged as a standalone copy (fixture; fixture + reference overlay).
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
|
||||
const root = __dirname;
|
||||
const fixture = path.join(root, 'cases2', 'event-stats-api', 'files');
|
||||
const overlay = path.join(root, 'reference2', 'event-stats-api');
|
||||
|
||||
function stage(withOverlay) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ecc-calib-'));
|
||||
const copy = (from, to) => {
|
||||
for (const entry of fs.readdirSync(from, { withFileTypes: true })) {
|
||||
const target = path.join(to, entry.name);
|
||||
if (entry.isDirectory()) { fs.mkdirSync(target, { recursive: true }); copy(path.join(from, entry.name), target); }
|
||||
else fs.copyFileSync(path.join(from, entry.name), target);
|
||||
}
|
||||
};
|
||||
copy(fixture, dir);
|
||||
if (withOverlay) copy(overlay, dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
function lcg(seed) {
|
||||
let state = seed >>> 0;
|
||||
return () => {
|
||||
state = (Math.imul(state, 1664525) + 1013904223) >>> 0;
|
||||
return state / 2 ** 32;
|
||||
};
|
||||
}
|
||||
|
||||
function workload(types, epoch, span) {
|
||||
const rand = lcg(777);
|
||||
const queries = [];
|
||||
for (let i = 0; i < 2000; i++) {
|
||||
const type = types[Math.floor(rand() * types.length)];
|
||||
const start = epoch + Math.floor(rand() * span * 0.7);
|
||||
queries.push({ type, from: start, to: start + Math.floor(rand() * span * 0.5) });
|
||||
}
|
||||
return queries;
|
||||
}
|
||||
|
||||
async function measure(label, dir) {
|
||||
const { createApp } = require(path.join(dir, 'src', 'app.js'));
|
||||
const { TYPES, EPOCH_MS, SPAN_MS } = require(path.join(dir, 'src', 'data.js'));
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
const queries = workload(TYPES, EPOCH_MS, SPAN_MS);
|
||||
const started = Date.now();
|
||||
for (let i = 0; i < queries.length; i += 20) {
|
||||
await Promise.all(queries.slice(i, i + 20).map(q =>
|
||||
fetch(`http://127.0.0.1:${port}/stats?type=${q.type}&from=${q.from}&to=${q.to}`).then(r => r.json())));
|
||||
}
|
||||
const elapsed = Date.now() - started;
|
||||
app.close();
|
||||
console.log(`${label}: ${elapsed}ms for 2000 queries`);
|
||||
return elapsed;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const naiveDir = stage(false);
|
||||
const refDir = stage(true);
|
||||
await measure('naive 1 ', naiveDir);
|
||||
await measure('naive 2 ', naiveDir);
|
||||
await measure('reference 1 ', refDir);
|
||||
await measure('reference 2 ', refDir);
|
||||
fs.rmSync(naiveDir, { recursive: true, force: true });
|
||||
fs.rmSync(refDir, { recursive: true, force: true });
|
||||
})();
|
||||
@@ -0,0 +1,45 @@
|
||||
'use strict';
|
||||
// Hidden grader for incident-triage: checks exact totals on boundary orders and
|
||||
// the root-cause report. Prints ECC_EVAL_SCORE and always exits 0.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
|
||||
let computeOrderTotal;
|
||||
try { ({ computeOrderTotal } = require(path.join(process.cwd(), 'src', 'totals.js'))); } catch { /* scored below */ }
|
||||
|
||||
// Boundary orders where decimal-factor float math under-rounds by a cent;
|
||||
// expected values follow the README pricing rules (integer cents, half-up per line).
|
||||
const boundary = [
|
||||
{ lines: [{ priceCents: 165, quantity: 1 }], discountPercent: 30, expected: 116 },
|
||||
{ lines: [{ priceCents: 250, quantity: 1 }], discountPercent: 7, expected: 233 },
|
||||
{ lines: [{ priceCents: 325, quantity: 1 }], discountPercent: 30, expected: 228 },
|
||||
{ lines: [{ priceCents: 345, quantity: 1 }], discountPercent: 30, expected: 242 },
|
||||
{ lines: [{ priceCents: 165, quantity: 1 }, { priceCents: 325, quantity: 1 }], discountPercent: 30, expected: 344 },
|
||||
];
|
||||
|
||||
if (typeof computeOrderTotal === 'function') {
|
||||
boundary.forEach((order, index) => {
|
||||
let actual = NaN;
|
||||
try { actual = computeOrderTotal({ lines: order.lines, discountPercent: order.discountPercent }); } catch { /* wrong */ }
|
||||
record(`boundary-total-${index + 1}`, actual === order.expected);
|
||||
});
|
||||
let plain = NaN;
|
||||
try { plain = computeOrderTotal({ lines: [{ priceCents: 1000, quantity: 2 }], discountPercent: 0 }); } catch { /* wrong */ }
|
||||
record('undiscounted-total-unchanged', plain === 2000);
|
||||
} else {
|
||||
for (let index = 0; index < boundary.length; index++) record(`boundary-total-${index + 1}`, false);
|
||||
record('undiscounted-total-unchanged', false);
|
||||
}
|
||||
|
||||
let incident = '';
|
||||
try { incident = fs.readFileSync(path.join(process.cwd(), 'INCIDENT.md'), 'utf8'); } catch { /* missing */ }
|
||||
record('incident-identifies-C-2', /C-2/.test(incident));
|
||||
record('incident-explains-rounding', /round|float|decimal|cent/i.test(incident));
|
||||
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) console.log(`${c.ok ? 'ok' : 'not ok'} - ${c.name}`);
|
||||
console.log(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / checks.length, passed: ok, total: checks.length })}`);
|
||||
process.exit(0);
|
||||
@@ -0,0 +1,11 @@
|
||||
# Changelog
|
||||
|
||||
## 2026-09-23 deploy
|
||||
|
||||
- **C-1**: request logging switched to JSON lines (`src/request-log.js`).
|
||||
Log volume and format only; no request-handling behavior changed.
|
||||
- **C-2**: totals computation refactored for readability (`src/totals.js`).
|
||||
The old cents-as-integers helper was replaced with a direct decimal
|
||||
expression that reviewers found easier to follow. No behavior change intended.
|
||||
- **C-3**: inventory client timeout raised from 2s to 5s (`src/inventory-client.js`).
|
||||
Reduces spurious failures when the inventory service is slow.
|
||||
@@ -0,0 +1,21 @@
|
||||
# order-service
|
||||
|
||||
Computes order totals for the checkout service.
|
||||
|
||||
## Pricing rules
|
||||
|
||||
An order is `{ "lines": [{ "priceCents": number, "quantity": number }], "discountPercent": number }`.
|
||||
|
||||
- All prices are integer cents. There is no such thing as a fraction of a cent
|
||||
in an order total.
|
||||
- The discount applies per line: `lineCents = priceCents * quantity * (100 - discountPercent) / 100`,
|
||||
rounded **half-up** to the nearest cent (0.5 rounds up).
|
||||
- The order total is the sum of the rounded line totals, in integer cents.
|
||||
|
||||
`src/totals.js` is CommonJS and exports `computeOrderTotal(order)` returning the
|
||||
total in integer cents. Run the tests with `npm test`.
|
||||
|
||||
## Operations
|
||||
|
||||
- `CHANGELOG.md` records what shipped in each deploy.
|
||||
- `evidence/incident.txt` holds the finance team's findings for the current incident.
|
||||
@@ -0,0 +1,5 @@
|
||||
2026-09-24T08:57:11Z finance-review order=ORD-2204 note="charged_total_cents=115 expected_total_cents=116 lines=[{priceCents:165,quantity:1}] discountPercent=30"
|
||||
2026-09-24T09:14:02Z finance-review order=ORD-2291 note="charged_total_cents=232 expected_total_cents=233 lines=[{priceCents:250,quantity:1}] discountPercent=7"
|
||||
2026-09-24T09:41:37Z finance-review order=ORD-2310 note="charged_total_cents=227 expected_total_cents=228 lines=[{priceCents:325,quantity:1}] discountPercent=30"
|
||||
2026-09-24T10:05:19Z support-ticket customer="ORDER-2310 looks like it undercharged me by a cent vs the invoice email"
|
||||
2026-09-24T10:22:48Z finance-review summary="12 of 4,813 orders since the 2026-09-23 deploy are off by exactly one cent, always in the store's favor; all pre-deploy orders reconcile"
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "order-service",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/" }
|
||||
}
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
'use strict';
|
||||
|
||||
// Changed 2026-09-23 (C-3): the inventory service has been slow this week;
|
||||
// give it 5s instead of 2s before declaring a failure.
|
||||
const INVENTORY_TIMEOUT_MS = 5000;
|
||||
|
||||
function inventoryClientOptions() {
|
||||
return { timeoutMs: INVENTORY_TIMEOUT_MS, retries: 2 };
|
||||
}
|
||||
|
||||
module.exports = { inventoryClientOptions };
|
||||
@@ -0,0 +1,13 @@
|
||||
'use strict';
|
||||
|
||||
// Changed 2026-09-23 (C-1): emit request logs as JSON lines so the log
|
||||
// pipeline can parse them without regexes.
|
||||
function logRequest(req) {
|
||||
console.log(JSON.stringify({
|
||||
method: req.method,
|
||||
url: req.url,
|
||||
at: new Date().toISOString(),
|
||||
}));
|
||||
}
|
||||
|
||||
module.exports = { logRequest };
|
||||
@@ -0,0 +1,14 @@
|
||||
'use strict';
|
||||
|
||||
// Refactored 2026-09-23 (C-2): express the discount math directly with a
|
||||
// decimal factor instead of the old integer-cents helper, which reviewers
|
||||
// found hard to follow.
|
||||
function computeOrderTotal(order) {
|
||||
let total = 0;
|
||||
for (const line of order.lines) {
|
||||
total += Math.round(line.priceCents * line.quantity * (1 - order.discountPercent / 100));
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
module.exports = { computeOrderTotal };
|
||||
@@ -0,0 +1,16 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { computeOrderTotal } = require('../src/totals');
|
||||
|
||||
test('sums lines without a discount', () => {
|
||||
assert.equal(computeOrderTotal({ lines: [{ priceCents: 1000, quantity: 2 }], discountPercent: 0 }), 2000);
|
||||
});
|
||||
|
||||
test('applies a clean quarter discount', () => {
|
||||
assert.equal(computeOrderTotal({ lines: [{ priceCents: 2000, quantity: 1 }], discountPercent: 25 }), 1500);
|
||||
});
|
||||
|
||||
test('multiplies quantity before discounting', () => {
|
||||
assert.equal(computeOrderTotal({ lines: [{ priceCents: 400, quantity: 3 }], discountPercent: 50 }), 600);
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"id": "incident-triage",
|
||||
"category": "debugging-incident",
|
||||
"manualIds": ["skill:orch-fix-defect"],
|
||||
"checkTimeoutMs": 30000,
|
||||
"selection": {
|
||||
"id": "complex-incident-triage",
|
||||
"category": "complex-debugging-incident",
|
||||
"expectedIds": ["skill:orch-fix-defect"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
Finance flagged that some order totals have been off by a cent since yesterday's deploy — details are in evidence/incident.txt. Three changes shipped yesterday (CHANGELOG.md, entries C-1 to C-3). Find the root cause, fix it so totals are computed exactly per the pricing rules in the README, keep `npm test` green, and write INCIDENT.md at the repo root identifying which changelog entry introduced the regression, with a short explanation of why it produces wrong totals.
|
||||
@@ -0,0 +1,123 @@
|
||||
'use strict';
|
||||
// Hidden grader for sentinel-api: runs exploit probes and functional regression
|
||||
// probes against the agent's service, in-process, plus static source checks.
|
||||
// Prints ECC_EVAL_SCORE and always exits 0.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) console.log(`${c.ok ? 'ok' : 'not ok'} - ${c.name}`);
|
||||
console.log(`ECC_EVAL_SCORE ${JSON.stringify({ score: checks.length ? ok / checks.length : 0, passed: ok, total: checks.length })}`);
|
||||
process.exit(0);
|
||||
}
|
||||
setTimeout(finish, 45000).unref();
|
||||
|
||||
const HARDCODED_TOKEN = 'fixture-token';
|
||||
const GRADER_TOKEN = 'ecc-grader-admin-token-31f7';
|
||||
|
||||
function sourceFiles(directory, found = []) {
|
||||
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
|
||||
const item = path.join(directory, entry.name);
|
||||
if (entry.isDirectory()) sourceFiles(item, found);
|
||||
else if (entry.name.endsWith('.js')) found.push(item);
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
function listen(server) {
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, '127.0.0.1', () => resolve(server.address().port));
|
||||
});
|
||||
}
|
||||
|
||||
async function createPaste(port, content) {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/pastes`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ content }) });
|
||||
return { status: response.status, body: await response.json().catch(() => null) };
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const root = process.cwd();
|
||||
const sources = sourceFiles(path.join(root, 'src'))
|
||||
.map(file => fs.readFileSync(file, 'utf8')).join('\n');
|
||||
|
||||
// Static: the committed admin token must be gone.
|
||||
record('hardcoded-admin-token-removed', !sources.includes(HARDCODED_TOKEN));
|
||||
|
||||
// Static: delete tokens must come from a cryptographic source.
|
||||
record('delete-token-from-crypto', /crypto\.(randomBytes|randomUUID|randomInt)/.test(sources));
|
||||
|
||||
// The service reads ADMIN_TOKEN from the environment at startup.
|
||||
process.env.ADMIN_TOKEN = GRADER_TOKEN;
|
||||
let createApp;
|
||||
try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { finish(); return; }
|
||||
if (typeof createApp !== 'function') { finish(); return; }
|
||||
|
||||
try {
|
||||
const app = createApp();
|
||||
const port = await listen(app);
|
||||
|
||||
// Admin route: rejected without the token, allowed with the environment token.
|
||||
const denied = await fetch(`http://127.0.0.1:${port}/admin/stats`);
|
||||
const wrong = await fetch(`http://127.0.0.1:${port}/admin/stats`, { headers: { 'x-admin-token': HARDCODED_TOKEN } });
|
||||
const allowed = await fetch(`http://127.0.0.1:${port}/admin/stats`, { headers: { 'x-admin-token': GRADER_TOKEN } });
|
||||
record('admin-requires-env-token', denied.status === 401 && wrong.status === 401 && allowed.status === 200);
|
||||
|
||||
// Path traversal: the canary outside data/files must never be served.
|
||||
const traversal = await fetch(`http://127.0.0.1:${port}/files?name=../secret.txt`);
|
||||
const traversalBody = await traversal.text();
|
||||
record('path-traversal-blocked', [400, 401, 403, 404].includes(traversal.status) && !traversalBody.includes('CANARY-7f3c9d'));
|
||||
const nested = await fetch(`http://127.0.0.1:${port}/files?name=${encodeURIComponent('..%2f..%2fsecret.txt')}`);
|
||||
record('encoded-traversal-blocked', !(await nested.text()).includes('CANARY-7f3c9d'));
|
||||
|
||||
// Legitimate files still work.
|
||||
const legit = await fetch(`http://127.0.0.1:${port}/files?name=welcome.txt`);
|
||||
record('public-files-still-served', legit.status === 200 && (await legit.text()).includes('hello from sentinel'));
|
||||
|
||||
// Reflected content is escaped in the HTML view.
|
||||
const xss = `<script>alert('xss')</script>`;
|
||||
const made = await createPaste(port, xss);
|
||||
const view = made.body && made.body.id
|
||||
? await fetch(`http://127.0.0.1:${port}/p/${made.body.id}`) : null;
|
||||
const html = view ? await view.text() : '';
|
||||
record('html-view-escapes-content', Boolean(view) && !html.includes(xss) && html.includes('<script>'));
|
||||
|
||||
// Delete tokens: unique, long, and not derived from the paste id.
|
||||
const first = await createPaste(port, 'one');
|
||||
const second = await createPaste(port, 'two');
|
||||
const t1 = first.body && first.body.deleteToken;
|
||||
const t2 = second.body && second.body.deleteToken;
|
||||
record('delete-token-unpredictable', Boolean(t1 && t2) && t1 !== t2 && t1.length >= 16
|
||||
&& !t1.includes(String(first.body.id)) && t1 !== `tok_${first.body.id}`);
|
||||
|
||||
// Oversized bodies are rejected per the contract.
|
||||
const big = await fetch(`http://127.0.0.1:${port}/pastes`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ content: 'x'.repeat(200 * 1024) }) });
|
||||
record('oversized-body-rejected-413', big.status === 413);
|
||||
|
||||
// Functional regression: the documented flow still works end to end.
|
||||
const flow = await createPaste(port, 'roundtrip content');
|
||||
const readBack = flow.body && flow.body.id
|
||||
? await fetch(`http://127.0.0.1:${port}/pastes/${flow.body.id}`) : null;
|
||||
const readJson = readBack ? await readBack.json().catch(() => null) : null;
|
||||
const deleted = flow.body && flow.body.id
|
||||
? await fetch(`http://127.0.0.1:${port}/pastes/${flow.body.id}`, {
|
||||
method: 'DELETE', headers: { 'x-delete-token': flow.body.deleteToken } }) : null;
|
||||
const afterDelete = flow.body && flow.body.id
|
||||
? await fetch(`http://127.0.0.1:${port}/pastes/${flow.body.id}`) : null;
|
||||
record('documented-api-still-works', Boolean(readJson) && readJson.content === 'roundtrip content'
|
||||
&& Boolean(deleted) && deleted.status === 204 && Boolean(afterDelete) && afterDelete.status === 404);
|
||||
|
||||
app.close();
|
||||
} catch { /* grader-side failure leaves remaining checks unscored */ }
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1,28 @@
|
||||
# sentinel-paste
|
||||
|
||||
Internal paste-sharing service. In-memory store, Node.js standard library only.
|
||||
|
||||
## API contract
|
||||
|
||||
- `POST /pastes` — JSON body `{ "content": string }`. `201` with
|
||||
`{ "id", "deleteToken" }`. Request bodies larger than 64 KB are rejected with
|
||||
`413 Payload Too Large`.
|
||||
- `GET /pastes/:id` — `200` JSON `{ "id", "content" }`, or `404`.
|
||||
- `DELETE /pastes/:id` — requires the `X-Delete-Token` header to match the
|
||||
token returned at creation. `204` on success, `403` otherwise. Delete tokens
|
||||
are shown once and must be unguessable.
|
||||
- `GET /p/:id` — HTML view of a paste. The paste content is rendered as text:
|
||||
it must never be interpreted as markup.
|
||||
- `GET /files?name=<name>` — serves a public file from `data/files/`. Only
|
||||
files inside that directory may ever be served.
|
||||
- `GET /admin/stats` — operational stats. Requires the `X-Admin-Token` header
|
||||
to match the `ADMIN_TOKEN` environment variable; the service reads it at
|
||||
startup and refuses all admin access (`401`) when it is missing or wrong.
|
||||
The token must never be committed to the repository.
|
||||
|
||||
## Module contract
|
||||
|
||||
- `src/app.js` is CommonJS and exports `createApp()`, which returns an
|
||||
`http.Server` that is not yet listening.
|
||||
- `node src/index.js <port>` starts the service.
|
||||
- Run the tests with `npm test`.
|
||||
@@ -0,0 +1 @@
|
||||
hello from sentinel
|
||||
@@ -0,0 +1 @@
|
||||
CANARY-7f3c9d-internal-only
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "sentinel-paste",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/" }
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
'use strict';
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const http = require('node:http');
|
||||
const config = require('./config');
|
||||
const store = require('./store');
|
||||
|
||||
function readBody(req, callback) {
|
||||
let body = '';
|
||||
req.on('data', chunk => { body += chunk; });
|
||||
req.on('end', () => callback(body));
|
||||
}
|
||||
|
||||
function sendJson(res, status, value) {
|
||||
res.writeHead(status, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify(value));
|
||||
}
|
||||
|
||||
function page(paste) {
|
||||
return `<!doctype html><html><head><title>paste ${paste.id}</title></head>`
|
||||
+ `<body><main><pre class="paste">${paste.content}</pre></main></body></html>`;
|
||||
}
|
||||
|
||||
function createApp() {
|
||||
return http.createServer((req, res) => {
|
||||
const url = new URL(req.url, 'http://localhost');
|
||||
|
||||
if (req.method === 'POST' && url.pathname === '/pastes') {
|
||||
readBody(req, body => {
|
||||
let parsed;
|
||||
try { parsed = JSON.parse(body); } catch {
|
||||
sendJson(res, 400, { error: 'invalid JSON body' });
|
||||
return;
|
||||
}
|
||||
if (typeof parsed.content !== 'string') {
|
||||
sendJson(res, 400, { error: 'content must be a string' });
|
||||
return;
|
||||
}
|
||||
const paste = store.create(parsed.content);
|
||||
sendJson(res, 201, { id: paste.id, deleteToken: paste.deleteToken });
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const pasteMatch = /^\/pastes\/([\w-]+)$/.exec(url.pathname);
|
||||
if (pasteMatch && req.method === 'GET') {
|
||||
const paste = store.get(pasteMatch[1]);
|
||||
if (!paste) { sendJson(res, 404, { error: 'not found' }); return; }
|
||||
sendJson(res, 200, { id: paste.id, content: paste.content });
|
||||
return;
|
||||
}
|
||||
if (pasteMatch && req.method === 'DELETE') {
|
||||
const paste = store.get(pasteMatch[1]);
|
||||
if (!paste) { sendJson(res, 404, { error: 'not found' }); return; }
|
||||
if (req.headers['x-delete-token'] !== paste.deleteToken) {
|
||||
sendJson(res, 403, { error: 'bad delete token' });
|
||||
return;
|
||||
}
|
||||
store.remove(paste.id);
|
||||
res.writeHead(204);
|
||||
res.end();
|
||||
return;
|
||||
}
|
||||
|
||||
const pageMatch = /^\/p\/([\w-]+)$/.exec(url.pathname);
|
||||
if (pageMatch && req.method === 'GET') {
|
||||
const paste = store.get(pageMatch[1]);
|
||||
if (!paste) { sendJson(res, 404, { error: 'not found' }); return; }
|
||||
res.writeHead(200, { 'content-type': 'text/html' });
|
||||
res.end(page(paste));
|
||||
return;
|
||||
}
|
||||
|
||||
if (req.method === 'GET' && url.pathname === '/files') {
|
||||
const name = url.searchParams.get('name') || '';
|
||||
try {
|
||||
const content = fs.readFileSync(path.join(config.FILES_DIR, name));
|
||||
res.writeHead(200, { 'content-type': 'text/plain' });
|
||||
res.end(content);
|
||||
} catch {
|
||||
sendJson(res, 404, { error: 'not found' });
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (req.method === 'GET' && url.pathname === '/admin/stats') {
|
||||
if (req.headers['x-admin-token'] !== config.ADMIN_TOKEN) {
|
||||
sendJson(res, 401, { error: 'unauthorized' });
|
||||
return;
|
||||
}
|
||||
sendJson(res, 200, store.stats());
|
||||
return;
|
||||
}
|
||||
|
||||
sendJson(res, 404, { error: 'not found' });
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = { createApp };
|
||||
@@ -0,0 +1,9 @@
|
||||
'use strict';
|
||||
const path = require('node:path');
|
||||
|
||||
module.exports = {
|
||||
// TODO: move this out of the repository before the next audit.
|
||||
ADMIN_TOKEN: 'fixture-token',
|
||||
MAX_BODY_BYTES: 64 * 1024,
|
||||
FILES_DIR: path.join(__dirname, '..', 'data', 'files'),
|
||||
};
|
||||
@@ -0,0 +1,7 @@
|
||||
'use strict';
|
||||
const { createApp } = require('./app');
|
||||
|
||||
const port = Number(process.argv[2] || 8080);
|
||||
createApp().listen(port, () => {
|
||||
console.log(`sentinel-paste listening on ${port}`);
|
||||
});
|
||||
@@ -0,0 +1,26 @@
|
||||
'use strict';
|
||||
|
||||
// In-memory paste store.
|
||||
const pastes = new Map();
|
||||
let nextId = 1;
|
||||
|
||||
function create(content) {
|
||||
const id = `p_${nextId++}`;
|
||||
const paste = { id, content, deleteToken: `tok_${id}` };
|
||||
pastes.set(id, paste);
|
||||
return paste;
|
||||
}
|
||||
|
||||
function get(id) {
|
||||
return pastes.get(id) || null;
|
||||
}
|
||||
|
||||
function remove(id) {
|
||||
return pastes.delete(id);
|
||||
}
|
||||
|
||||
function stats() {
|
||||
return { pastes: pastes.size, created: nextId - 1 };
|
||||
}
|
||||
|
||||
module.exports = { create, get, remove, stats };
|
||||
@@ -0,0 +1,28 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { createApp } = require('../src/app');
|
||||
|
||||
function listen(server) {
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, '127.0.0.1', () => resolve(server.address().port));
|
||||
});
|
||||
}
|
||||
|
||||
test('create and read back a paste', async () => {
|
||||
const server = createApp();
|
||||
const port = await listen(server);
|
||||
try {
|
||||
const created = await fetch(`http://127.0.0.1:${port}/pastes`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ content: 'hello paste' }) });
|
||||
assert.equal(created.status, 201);
|
||||
const { id } = await created.json();
|
||||
const read = await fetch(`http://127.0.0.1:${port}/pastes/${id}`);
|
||||
assert.equal(read.status, 200);
|
||||
assert.equal((await read.json()).content, 'hello paste');
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"id": "sentinel-api",
|
||||
"category": "security-hardening",
|
||||
"manualIds": ["skill:security-review"],
|
||||
"checkTimeoutMs": 60000,
|
||||
"selection": {
|
||||
"id": "complex-sentinel-api",
|
||||
"category": "complex-security-hardening",
|
||||
"expectedIds": ["skill:security-review"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
This internal paste-sharing service failed a security review, but the auditors didn't itemize the findings. Review the implementation against the API contract in the README, find every place the code violates the documented security behavior or is otherwise exploitable, and fix all of them without breaking the documented API. `npm test` must stay green.
|
||||
@@ -0,0 +1,125 @@
|
||||
'use strict';
|
||||
// Hidden grader for webhook-relay: drives the agent's relay in-process against
|
||||
// local target servers and prints ECC_EVAL_SCORE. Always exits 0; the score line
|
||||
// carries the result. Runs under Node's read-only permission model, so it only
|
||||
// reads the workspace and talks to 127.0.0.1.
|
||||
const http = require('node:http');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
let finished = false;
|
||||
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) console.log(`${c.ok ? 'ok' : 'not ok'} - ${c.name}`);
|
||||
console.log(`ECC_EVAL_SCORE ${JSON.stringify({ score: checks.length ? ok / checks.length : 0, passed: ok, total: checks.length })}`);
|
||||
process.exit(0);
|
||||
}
|
||||
setTimeout(finish, 45000).unref();
|
||||
|
||||
function listen(server) {
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, '127.0.0.1', () => resolve(server.address().port));
|
||||
});
|
||||
}
|
||||
|
||||
function postJson(port, urlPath, body) {
|
||||
return fetch(`http://127.0.0.1:${port}${urlPath}`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) })
|
||||
.then(async response => ({ status: response.status, body: await response.json().catch(() => null) }));
|
||||
}
|
||||
|
||||
async function waitForStatus(port, id, wanted, timeoutMs) {
|
||||
const started = Date.now();
|
||||
let last = null;
|
||||
while (Date.now() - started < timeoutMs) {
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/deliveries/${id}`);
|
||||
if (response.status === 200) {
|
||||
last = await response.json();
|
||||
if (last.status === wanted || last.status === 'dead') return { record: last, elapsedMs: Date.now() - started };
|
||||
}
|
||||
} catch { /* relay not ready yet */ }
|
||||
await sleep(25);
|
||||
}
|
||||
return { record: last, elapsedMs: Date.now() - started };
|
||||
}
|
||||
|
||||
(async () => {
|
||||
let createRelay;
|
||||
try { ({ createRelay } = require(path.join(process.cwd(), 'src', 'app.js'))); } catch { finish(); return; }
|
||||
if (typeof createRelay !== 'function') { finish(); return; }
|
||||
|
||||
// Probe group 1: a target that fails 3 times then succeeds.
|
||||
let calls = 0;
|
||||
const flaky = http.createServer((req, res) => {
|
||||
calls++;
|
||||
req.resume();
|
||||
req.on('end', () => { res.writeHead(calls <= 3 ? 500 : 200); res.end('{}'); });
|
||||
});
|
||||
const relay = createRelay();
|
||||
try {
|
||||
const flakyPort = await listen(flaky);
|
||||
const relayPort = await listen(relay);
|
||||
const started = Date.now();
|
||||
const created = await postJson(relayPort, '/deliveries', { url: `http://127.0.0.1:${flakyPort}/hook`, payload: { hello: 'world' } });
|
||||
record('accepts-delivery-202', created.status === 202 && created.body && typeof created.body.id === 'string');
|
||||
if (created.body && created.body.id) {
|
||||
const { record: rec, elapsedMs } = await waitForStatus(relayPort, created.body.id, 'delivered', 8000);
|
||||
record('delivered-after-retries', rec && rec.status === 'delivered' && calls >= 4);
|
||||
record('attempts-counted', rec && rec.attempts === 4);
|
||||
record('backoff-window-respected', rec && rec.status === 'delivered' && elapsedMs >= 250 && elapsedMs <= 5000 && Date.now() - started >= 250);
|
||||
} else {
|
||||
record('delivered-after-retries', false);
|
||||
record('attempts-counted', false);
|
||||
record('backoff-window-respected', false);
|
||||
}
|
||||
|
||||
// Probe group 2: a target that always fails -> dead after exactly 5 attempts.
|
||||
let deadCalls = 0;
|
||||
const deadEnd = http.createServer((req, res) => {
|
||||
deadCalls++;
|
||||
req.resume();
|
||||
req.on('end', () => { res.writeHead(500); res.end('{}'); });
|
||||
});
|
||||
const deadPort = await listen(deadEnd);
|
||||
const doomed = await postJson(relayPort, '/deliveries', { url: `http://127.0.0.1:${deadPort}/hook`, payload: { x: 1 } });
|
||||
if (doomed.body && doomed.body.id) {
|
||||
const { record: rec } = await waitForStatus(relayPort, doomed.body.id, 'dead', 15000);
|
||||
record('dead-after-retries-exhausted', rec && rec.status === 'dead');
|
||||
record('exactly-five-attempts', rec && rec.status === 'dead' && rec.attempts === 5 && deadCalls === 5);
|
||||
record('last-error-recorded', rec && rec.status === 'dead' && typeof rec.lastError === 'string' && rec.lastError.length > 0);
|
||||
} else {
|
||||
record('dead-after-retries-exhausted', false);
|
||||
record('exactly-five-attempts', false);
|
||||
record('last-error-recorded', false);
|
||||
}
|
||||
deadEnd.close();
|
||||
|
||||
// Probe 3: pre-existing API behavior is preserved.
|
||||
const missing = await fetch(`http://127.0.0.1:${relayPort}/deliveries/00000000-0000-0000-0000-000000000000`);
|
||||
record('unknown-id-still-404', missing.status === 404);
|
||||
|
||||
// Probe 4: concurrent deliveries all complete.
|
||||
let goodCalls = 0;
|
||||
const good = http.createServer((req, res) => {
|
||||
goodCalls++;
|
||||
req.resume();
|
||||
req.on('end', () => { res.writeHead(200); res.end('{}'); });
|
||||
});
|
||||
const goodPort = await listen(good);
|
||||
const batch = await Promise.all(Array.from({ length: 10 }, (_, i) =>
|
||||
postJson(relayPort, '/deliveries', { url: `http://127.0.0.1:${goodPort}/hook`, payload: { i } })));
|
||||
const settled = await Promise.all(batch.map(item => item.body && item.body.id
|
||||
? waitForStatus(relayPort, item.body.id, 'delivered', 10000).then(r => r.record && r.record.status === 'delivered')
|
||||
: false));
|
||||
record('concurrent-deliveries-complete', settled.every(Boolean) && goodCalls === 10);
|
||||
good.close();
|
||||
} catch { /* any grader-side failure leaves the missing checks unscored */ }
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1,33 @@
|
||||
# webhook-relay
|
||||
|
||||
In-memory webhook relay. Accepts delivery requests over HTTP and POSTs each
|
||||
payload to its destination URL, retrying failures with exponential backoff.
|
||||
|
||||
## HTTP API
|
||||
|
||||
- `POST /deliveries` — body `{ "url": string, "payload": any }`. Responds
|
||||
`202` with `{ "id" }` and delivers asynchronously. `400` for invalid JSON.
|
||||
- `GET /deliveries/:id` — `200` with
|
||||
`{ "id", "url", "status", "attempts", "lastError" }`, or `404`.
|
||||
`status` is `pending`, `delivered`, or `dead`.
|
||||
|
||||
## Delivery contract
|
||||
|
||||
- The payload is POSTed to `url` with `content-type: application/json`.
|
||||
- Any 2xx response means success: `status` becomes `delivered`.
|
||||
- Any other outcome (non-2xx, connection error, timeout) is a failure and is
|
||||
retried with exponential backoff: the first retry happens after about
|
||||
100ms and the delay doubles each retry. Up to 20% jitter in either
|
||||
direction is fine.
|
||||
- At most 5 attempts are made in total (the initial try plus 4 retries).
|
||||
- After the final failure the delivery becomes `dead` and `lastError`
|
||||
records a short description of the last failure.
|
||||
- `attempts` always reflects how many delivery attempts were made.
|
||||
|
||||
## Module contract
|
||||
|
||||
- `src/app.js` is CommonJS and exports `createRelay()`, which returns an
|
||||
`http.Server` that is not yet listening.
|
||||
- `node src/index.js <port>` starts the service.
|
||||
- No external dependencies; Node.js standard library only.
|
||||
- Run the tests with `npm test`.
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "webhook-relay",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/" }
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
'use strict';
|
||||
const http = require('node:http');
|
||||
const crypto = require('node:crypto');
|
||||
|
||||
// In-memory webhook relay. See README.md for the delivery contract.
|
||||
//
|
||||
// TODO: deliveries are accepted and stored, but the delivery worker was never
|
||||
// finished — nothing ever POSTs to the destination URL, retries never happen,
|
||||
// and records stay "pending" forever.
|
||||
|
||||
function createRelay() {
|
||||
const deliveries = new Map();
|
||||
|
||||
const server = http.createServer((req, res) => {
|
||||
if (req.method === 'POST' && req.url === '/deliveries') {
|
||||
let body = '';
|
||||
req.on('data', chunk => { body += chunk; });
|
||||
req.on('end', () => {
|
||||
let parsed;
|
||||
try { parsed = JSON.parse(body); } catch {
|
||||
res.writeHead(400, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: 'invalid JSON body' }));
|
||||
return;
|
||||
}
|
||||
const id = crypto.randomUUID();
|
||||
deliveries.set(id, { id, url: parsed.url, payload: parsed.payload,
|
||||
status: 'pending', attempts: 0, lastError: null });
|
||||
res.writeHead(202, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ id }));
|
||||
});
|
||||
return;
|
||||
}
|
||||
const match = /^\/deliveries\/([0-9a-f-]+)$/.exec(req.url || '');
|
||||
if (req.method === 'GET' && match) {
|
||||
const record = deliveries.get(match[1]);
|
||||
if (!record) {
|
||||
res.writeHead(404, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: 'not found' }));
|
||||
return;
|
||||
}
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify(record));
|
||||
return;
|
||||
}
|
||||
res.writeHead(404, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: 'not found' }));
|
||||
});
|
||||
return server;
|
||||
}
|
||||
|
||||
module.exports = { createRelay };
|
||||
@@ -0,0 +1,7 @@
|
||||
'use strict';
|
||||
const { createRelay } = require('./app');
|
||||
|
||||
const port = Number(process.argv[2] || 8080);
|
||||
createRelay().listen(port, () => {
|
||||
console.log(`webhook-relay listening on ${port}`);
|
||||
});
|
||||
@@ -0,0 +1,41 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { createRelay } = require('../src/app');
|
||||
|
||||
function listen(server) {
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, '127.0.0.1', () => resolve(server.address().port));
|
||||
});
|
||||
}
|
||||
|
||||
test('accepts a delivery and reports it as pending', async () => {
|
||||
const server = createRelay();
|
||||
const port = await listen(server);
|
||||
try {
|
||||
const created = await fetch(`http://127.0.0.1:${port}/deliveries`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ url: 'http://127.0.0.1:1/hook', payload: { a: 1 } }) });
|
||||
assert.equal(created.status, 202);
|
||||
const { id } = await created.json();
|
||||
const status = await fetch(`http://127.0.0.1:${port}/deliveries/${id}`);
|
||||
assert.equal(status.status, 200);
|
||||
const record = await status.json();
|
||||
assert.equal(record.status, 'pending');
|
||||
assert.equal(record.attempts, 0);
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('unknown delivery id returns 404', async () => {
|
||||
const server = createRelay();
|
||||
const port = await listen(server);
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/deliveries/00000000-0000-0000-0000-000000000000`);
|
||||
assert.equal(response.status, 404);
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"id": "webhook-relay",
|
||||
"category": "feature-build",
|
||||
"manualIds": ["skill:tdd-workflow"],
|
||||
"checkTimeoutMs": 60000,
|
||||
"selection": {
|
||||
"id": "complex-webhook-relay",
|
||||
"category": "complex-feature-build",
|
||||
"expectedIds": ["skill:tdd-workflow"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
The webhook relay in this repo accepts delivery requests but never actually sends them — the delivery worker was never finished, and customers are losing notifications. Implement asynchronous delivery per the README: POST each delivery's JSON payload to its URL, retry failures with exponential backoff starting around 100ms and doubling each time, give up after 5 total attempts and mark the delivery dead. Keep the documented module contract, make `npm test` pass, and extend the test suite to cover the retry and dead-letter behavior.
|
||||
@@ -0,0 +1,149 @@
|
||||
'use strict';
|
||||
// Hidden grader for event-stats-api: independent spec-conformant aggregation
|
||||
// over the deterministic event log, plus a measured 2,000-query performance
|
||||
// probe (threshold calibrated on the grading machine: shipped naive ~7.7s,
|
||||
// reference ~1.5s). Prints ECC_EVAL_SCORE and always exits 0.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) console.log(`${c.ok ? 'ok' : 'not ok'} - ${c.name}`);
|
||||
console.log(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / checks.length, passed: ok, total: checks.length })}`);
|
||||
process.exit(0);
|
||||
}
|
||||
setTimeout(finish, 110000).unref();
|
||||
|
||||
const PERF_THRESHOLD_MS = 6000;
|
||||
const PERF_QUERIES = 2000;
|
||||
|
||||
function lcg(seed) {
|
||||
let state = seed >>> 0;
|
||||
return () => {
|
||||
state = (Math.imul(state, 1664525) + 1013904223) >>> 0;
|
||||
return state / 2 ** 32;
|
||||
};
|
||||
}
|
||||
|
||||
const root = process.cwd();
|
||||
const { events, TYPES, EPOCH_MS, SPAN_MS } = require(path.join(root, 'src', 'data.js'));
|
||||
|
||||
// Independent reference semantics per the README: inclusive bounds,
|
||||
// nearest-rank percentiles, half-up two-decimal average via exact integer math.
|
||||
function expected(type, from, to) {
|
||||
const rows = events
|
||||
.filter(e => e.type === type && (from === null || e.ts >= from) && (to === null || e.ts <= to))
|
||||
.map(e => e.value)
|
||||
.sort((a, b) => a - b);
|
||||
const count = rows.length;
|
||||
if (!count) return { count: 0, sum: 0, avg: null, p50: null, p95: null, p99: null, min: null, max: null };
|
||||
const sum = rows.reduce((a, b) => a + b, 0);
|
||||
const rank = p => rows[Math.ceil((p / 100) * count) - 1];
|
||||
const avgCents = Math.floor((sum * 200 + count) / (count * 2));
|
||||
return { count, sum, avg: avgCents / 100,
|
||||
p50: rank(50), p95: rank(95), p99: rank(99), min: rows[0], max: rows[count - 1] };
|
||||
}
|
||||
|
||||
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
|
||||
|
||||
async function query(port, params) {
|
||||
const qs = Object.entries(params).map(([k, v]) => `${k}=${v}`).join('&');
|
||||
const response = await fetch(`http://127.0.0.1:${port}/stats?${qs}`);
|
||||
return { status: response.status, body: await response.json().catch(() => null) };
|
||||
}
|
||||
|
||||
(async () => {
|
||||
let createApp;
|
||||
try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { finish(); return; }
|
||||
if (typeof createApp !== 'function') { finish(); return; }
|
||||
|
||||
try {
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
|
||||
// 1-2: broad and full-range queries with independently computed expectations.
|
||||
const broadFrom = EPOCH_MS;
|
||||
const broadTo = EPOCH_MS + 30 * 86400000;
|
||||
const broad = await query(port, { type: 'click', from: broadFrom, to: broadTo });
|
||||
record('broad-window-exact', broad.status === 200
|
||||
&& same(broad.body, { type: 'click', from: broadFrom, to: broadTo, ...expected('click', broadFrom, broadTo) }));
|
||||
const full = await query(port, { type: 'purchase' });
|
||||
record('full-range-exact', full.status === 200
|
||||
&& same(full.body, { type: 'purchase', from: null, to: null, ...expected('purchase', null, null) }));
|
||||
|
||||
// 3: nearest-rank vs interpolation is distinguishable on a tiny window.
|
||||
const exportEvents = events.filter(e => e.type === 'export').map(e => e.ts).sort((a, b) => a - b);
|
||||
const pivot = exportEvents[Math.floor(exportEvents.length / 2)];
|
||||
const narrowFrom = pivot - 1;
|
||||
const narrowTo = pivot + 1;
|
||||
const narrow = await query(port, { type: 'export', from: narrowFrom, to: narrowTo });
|
||||
record('narrow-window-nearest-rank', narrow.status === 200
|
||||
&& same(narrow.body, { type: 'export', from: narrowFrom, to: narrowTo, ...expected('export', narrowFrom, narrowTo) }));
|
||||
|
||||
// 4-5: empty range and unknown type return nulls, not zeros or errors.
|
||||
const beyond = await query(port, { type: 'click', from: EPOCH_MS + 200 * 86400000, to: EPOCH_MS + 201 * 86400000 });
|
||||
record('empty-range-nulls', beyond.status === 200 && same(beyond.body,
|
||||
{ type: 'click', from: EPOCH_MS + 200 * 86400000, to: EPOCH_MS + 201 * 86400000, ...expected('click', EPOCH_MS + 200 * 86400000, EPOCH_MS + 201 * 86400000) }));
|
||||
const unknown = await query(port, { type: 'nope' });
|
||||
record('unknown-type-nulls', unknown.status === 200
|
||||
&& same(unknown.body, { type: 'nope', from: null, to: null, ...expected('nope', null, null) }));
|
||||
|
||||
// 6: inclusive bounds — a zero-width window on a real timestamp includes it.
|
||||
const likeTs = events.filter(e => e.type === 'like').map(e => e.ts).sort((a, b) => a - b)[100];
|
||||
const inclusive = await query(port, { type: 'like', from: likeTs, to: likeTs });
|
||||
record('bounds-inclusive', inclusive.status === 200 && inclusive.body.count === expected('like', likeTs, likeTs).count && inclusive.body.count >= 1);
|
||||
|
||||
// 7: average rounding follows half-up two decimals exactly.
|
||||
const rounding = expected('view', EPOCH_MS, EPOCH_MS + 86400000);
|
||||
const rounded = await query(port, { type: 'view', from: EPOCH_MS, to: EPOCH_MS + 86400000 });
|
||||
record('avg-half-up-2dp', rounded.status === 200 && rounded.body.avg === rounding.avg);
|
||||
|
||||
// 8-9: invalid parameters are 400.
|
||||
const inverted = await query(port, { type: 'click', from: 10, to: 5 });
|
||||
record('inverted-bounds-400', inverted.status === 400);
|
||||
const garbage = await query(port, { type: 'click', from: 'abc' });
|
||||
record('non-numeric-bounds-400', garbage.status === 400);
|
||||
|
||||
// 10: performance budget.
|
||||
const rand = lcg(777);
|
||||
const queries = [];
|
||||
for (let i = 0; i < PERF_QUERIES; i++) {
|
||||
const type = TYPES[Math.floor(rand() * TYPES.length)];
|
||||
const start = EPOCH_MS + Math.floor(rand() * SPAN_MS * 0.7);
|
||||
queries.push({ type, from: start, to: start + Math.floor(rand() * SPAN_MS * 0.5) });
|
||||
}
|
||||
const started = Date.now();
|
||||
for (let i = 0; i < queries.length; i += 20) {
|
||||
await Promise.all(queries.slice(i, i + 20).map(q => query(port, q)));
|
||||
}
|
||||
const elapsed = Date.now() - started;
|
||||
console.log(`perf: ${elapsed}ms for ${PERF_QUERIES} queries (threshold ${PERF_THRESHOLD_MS}ms)`);
|
||||
record('performance-budget', elapsed < PERF_THRESHOLD_MS);
|
||||
|
||||
app.close();
|
||||
} catch { /* grader-side failure leaves remaining checks unscored */ }
|
||||
|
||||
// 11: no external dependencies.
|
||||
try {
|
||||
const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));
|
||||
const sources = [];
|
||||
const walk = directory => {
|
||||
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
|
||||
const item = path.join(directory, entry.name);
|
||||
if (entry.isDirectory()) walk(item);
|
||||
else if (entry.name.endsWith('.js')) sources.push(fs.readFileSync(item, 'utf8'));
|
||||
}
|
||||
};
|
||||
walk(path.join(root, 'src'));
|
||||
const bareImport = sources.some(source => /require\(\s*['"](?!node:)[a-z@][^'./]*['"]\s*\)/.test(source));
|
||||
record('no-external-dependencies', !bareImport && !pkg.dependencies && !pkg.devDependencies);
|
||||
} catch { record('no-external-dependencies', false); }
|
||||
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1,41 @@
|
||||
# event-stats
|
||||
|
||||
Analytics endpoint over an in-memory event log (300,000 events, generated
|
||||
deterministically by `src/data.js`).
|
||||
|
||||
## API
|
||||
|
||||
`GET /stats?type=<t>&from=<ms>&to=<ms>` returns JSON:
|
||||
|
||||
```json
|
||||
{ "type": "click", "from": 1754000000000, "to": 1756592000000,
|
||||
"count": 1234, "sum": 56789, "avg": 46.02,
|
||||
"p50": 123, "p95": 456, "p99": 789, "min": 1, "max": 50000 }
|
||||
```
|
||||
|
||||
Semantics (all pinned; follow them exactly):
|
||||
|
||||
- `from`/`to` are millisecond timestamps, **inclusive**, and optional
|
||||
(absent means unbounded). Non-numeric bounds, or `from > to`, are `400`.
|
||||
- Only events of the given `type` within `[from, to]` are included.
|
||||
- `sum` is the exact integer sum of `value`s.
|
||||
- `avg` is `sum / count` rounded **half-up to two decimals**.
|
||||
- Percentiles use the **nearest-rank** method: sort values ascending, take the
|
||||
value at 1-based rank `ceil(p / 100 * count)`. No interpolation.
|
||||
- If no events match (including an unknown `type`), return `200` with
|
||||
`count: 0, sum: 0` and `avg`, `p50`, `p95`, `p99`, `min`, `max` all `null`.
|
||||
- The response echoes the effective `from`/`to` (`null` when unbounded).
|
||||
|
||||
## Performance requirement
|
||||
|
||||
The endpoint must stay fast at this data size: **2,000 mixed queries complete
|
||||
in under 6 seconds** on this machine (the reference does it in ~1.5s).
|
||||
Precompute whatever you need at startup; per-query work must not scan the
|
||||
whole log.
|
||||
|
||||
## Module contract
|
||||
|
||||
- `src/app.js` is CommonJS and exports `createApp()` returning an
|
||||
`http.Server` that is not yet listening.
|
||||
- `node src/index.js <port>` starts the service.
|
||||
- No external dependencies. Run the tests with `npm test`.
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "event-stats",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/" }
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
'use strict';
|
||||
const http = require('node:http');
|
||||
const { events } = require('./data');
|
||||
|
||||
// Current implementation: scan and sort per query. Known slow, and the
|
||||
// analytics team says edge cases don't match the README semantics.
|
||||
function summarize(type, from, to) {
|
||||
const rows = events
|
||||
.filter(e => e.type === type && (from === null || e.ts >= from) && (to === null || e.ts <= to))
|
||||
.map(e => e.value)
|
||||
.sort((a, b) => a - b);
|
||||
const count = rows.length;
|
||||
const sum = rows.reduce((a, b) => a + b, 0);
|
||||
const interpolate = p => {
|
||||
if (!count) return 0;
|
||||
const rank = (p / 100) * (count - 1);
|
||||
const low = Math.floor(rank);
|
||||
const high = Math.ceil(rank);
|
||||
return rows[low] + (rows[high] - rows[low]) * (rank - low);
|
||||
};
|
||||
return { count, sum, avg: count ? sum / count : 0,
|
||||
p50: interpolate(50), p95: interpolate(95), p99: interpolate(99),
|
||||
min: count ? rows[0] : 0, max: count ? rows[count - 1] : 0 };
|
||||
}
|
||||
|
||||
function createApp() {
|
||||
return http.createServer((req, res) => {
|
||||
const url = new URL(req.url, 'http://localhost');
|
||||
if (req.method === 'GET' && url.pathname === '/stats') {
|
||||
const type = url.searchParams.get('type');
|
||||
const from = url.searchParams.has('from') ? Number(url.searchParams.get('from')) : null;
|
||||
const to = url.searchParams.has('to') ? Number(url.searchParams.get('to')) : null;
|
||||
const body = summarize(type, from, to);
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ type, from, to, ...body }));
|
||||
return;
|
||||
}
|
||||
res.writeHead(404, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: 'not found' }));
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = { createApp };
|
||||
@@ -0,0 +1,28 @@
|
||||
'use strict';
|
||||
// Deterministic event log: 300,000 events from a seeded LCG so every run,
|
||||
// grader, and reference sees identical data. Do not change the generator.
|
||||
const TYPES = ['click', 'view', 'signup', 'purchase', 'refund', 'login',
|
||||
'logout', 'share', 'comment', 'like', 'search', 'export'];
|
||||
const DAY_MS = 86400000;
|
||||
const EPOCH_MS = 1754000000000;
|
||||
const SPAN_MS = 90 * DAY_MS;
|
||||
|
||||
function lcg(seed) {
|
||||
let state = seed >>> 0;
|
||||
return () => {
|
||||
state = (Math.imul(state, 1664525) + 1013904223) >>> 0;
|
||||
return state / 2 ** 32;
|
||||
};
|
||||
}
|
||||
|
||||
const rand = lcg(20260925);
|
||||
const events = new Array(300000);
|
||||
for (let i = 0; i < events.length; i++) {
|
||||
events[i] = {
|
||||
type: TYPES[Math.floor(rand() * TYPES.length)],
|
||||
ts: EPOCH_MS + Math.floor(rand() * SPAN_MS),
|
||||
value: Math.floor(rand() * 50000) + 1,
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = { events, TYPES, EPOCH_MS, SPAN_MS };
|
||||
@@ -0,0 +1,7 @@
|
||||
'use strict';
|
||||
const { createApp } = require('./app');
|
||||
|
||||
const port = Number(process.argv[2] || 8080);
|
||||
createApp().listen(port, () => {
|
||||
console.log(`event-stats listening on ${port}`);
|
||||
});
|
||||
@@ -0,0 +1,20 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { createApp } = require('../src/app');
|
||||
const { EPOCH_MS } = require('../src/data');
|
||||
|
||||
test('stats endpoint answers a broad query', async () => {
|
||||
const server = createApp();
|
||||
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
|
||||
try {
|
||||
const port = server.address().port;
|
||||
const response = await fetch(`http://127.0.0.1:${port}/stats?type=click&from=${EPOCH_MS}&to=${EPOCH_MS + 30 * 86400000}`);
|
||||
assert.equal(response.status, 200);
|
||||
const body = await response.json();
|
||||
assert.equal(body.type, 'click');
|
||||
assert.ok(body.count > 0);
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"id": "event-stats-api",
|
||||
"category": "correctness-and-performance",
|
||||
"manualIds": ["skill:backend-patterns"],
|
||||
"checkTimeoutMs": 120000,
|
||||
"selection": {
|
||||
"id": "complex-event-stats-api",
|
||||
"category": "complex-correctness-performance",
|
||||
"expectedIds": ["skill:backend-patterns"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
The /stats endpoint in this repo is wrong on edge cases and too slow — customers on big dashboards are timing out. It currently rescans and resorts the whole 300k-event log on every request, and the analytics team says the numbers don't match the documented semantics (nearest-rank percentiles, half-up two-decimal averages, null fields when nothing matches, proper 400s). Make it correct per the README and fast enough to meet the documented performance budget, without changing the API shape. `npm test` must stay green.
|
||||
@@ -0,0 +1,132 @@
|
||||
'use strict';
|
||||
// Hidden grader for forge-cli: drives run(argv, state) through the twelve
|
||||
// contractual behaviors plus never-throw fuzzing and static hygiene. Prints
|
||||
// ECC_EVAL_SCORE and always exits 0.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
|
||||
const root = process.cwd();
|
||||
let run;
|
||||
try { ({ run } = require(path.join(root, 'src', 'cli.js'))); } catch { /* scored below */ }
|
||||
|
||||
const USAGE = 'usage: snippet <add|get|list|remove|search|export|import>\n';
|
||||
const ADD_USAGE = 'usage: add <name> [--tags t1,t2] <text...>\n';
|
||||
|
||||
if (typeof run !== 'function') {
|
||||
for (let i = 0; i < 26; i++) record(`check-${i + 1}`, false);
|
||||
} else {
|
||||
const call = (argv, state) => {
|
||||
try {
|
||||
const result = run(argv, state);
|
||||
if (!result || typeof result.code !== 'number'
|
||||
|| typeof result.stdout !== 'string' || typeof result.stderr !== 'string') return null;
|
||||
return result;
|
||||
} catch { return null; }
|
||||
};
|
||||
|
||||
// Basic lifecycle.
|
||||
let s = {};
|
||||
let r = call(['add', 'hello', 'hello', 'world'], s);
|
||||
record('add-happy', r && r.code === 0 && r.stdout === 'created hello\n' && r.stderr === '');
|
||||
r = call(['add', 'hello', 'different', 'text'], s);
|
||||
const afterDup = call(['get', 'hello'], s);
|
||||
record('add-duplicate-rejected', r && r.code === 1 && r.stderr === "error: snippet 'hello' already exists\n"
|
||||
&& afterDup && afterDup.stdout === 'hello world\n');
|
||||
const m1 = call(['add'], s);
|
||||
const m2 = call(['add', 'justname'], s);
|
||||
record('add-missing-args-usage', m1 && m1.code === 2 && m1.stderr === ADD_USAGE
|
||||
&& m2 && m2.code === 2 && m2.stderr === ADD_USAGE);
|
||||
r = call(['add', 'Bad_Name', 'text'], s);
|
||||
record('invalid-name-rejected', r && r.code === 2 && r.stderr === "error: invalid snippet name 'Bad_Name'\n");
|
||||
r = call(['get', 'hello'], s);
|
||||
record('get-happy', r && r.code === 0 && r.stdout === 'hello world\n');
|
||||
r = call(['get', 'ghost'], s);
|
||||
record('get-unknown', r && r.code === 2 && r.stderr === "error: no snippet named 'ghost'\n");
|
||||
|
||||
// Listing and tags.
|
||||
s = {};
|
||||
call(['add', 'bravo', 'second'], s);
|
||||
call(['add', 'alpha', '--tags', 'x,y', 'first'], s);
|
||||
call(['add', 'charlie', '--tags', 'y', 'third'], s);
|
||||
r = call(['list'], s);
|
||||
record('list-sorted', r && r.code === 0 && r.stdout === 'alpha\nbravo\ncharlie\n');
|
||||
r = call(['list'], {});
|
||||
record('list-empty', r && r.code === 0 && r.stdout === 'no snippets\n');
|
||||
r = call(['list', '--tag', 'y'], s);
|
||||
record('list-tag-filter', r && r.code === 0 && r.stdout === 'alpha\ncharlie\n');
|
||||
|
||||
// Removal.
|
||||
r = call(['remove', 'bravo'], s);
|
||||
const gone = call(['get', 'bravo'], s);
|
||||
record('remove-happy', r && r.code === 0 && r.stdout === 'removed bravo\n' && gone && gone.code === 2);
|
||||
r = call(['remove', 'bravo'], s);
|
||||
record('remove-unknown', r && r.code === 2 && r.stderr === "error: no snippet named 'bravo'\n");
|
||||
|
||||
// Search over name and text, case-insensitive, sorted.
|
||||
r = call(['search', 'FIRST'], s);
|
||||
record('search-text-case-insensitive', r && r.code === 0 && r.stdout === 'alpha\n');
|
||||
r = call(['search', 'char'], s);
|
||||
record('search-name-match', r && r.code === 0 && r.stdout === 'charlie\n');
|
||||
r = call(['search', 'zzz'], s);
|
||||
record('search-no-matches', r && r.code === 0 && r.stdout === 'no matches\n');
|
||||
|
||||
// Export/import round-trip with stable ordering.
|
||||
r = call(['export'], s);
|
||||
let doc = null;
|
||||
try { doc = r && JSON.parse(r.stdout); } catch { /* wrong */ }
|
||||
record('export-json-sorted', doc && r.code === 0 && sameDoc(doc, {
|
||||
snippets: { alpha: { text: 'first', tags: ['x', 'y'] }, charlie: { text: 'third', tags: ['y'] } } })
|
||||
&& r.stdout.indexOf('alpha') < r.stdout.indexOf('charlie'));
|
||||
const importedState = { snippets: { alpha: { text: 'preexisting', tags: [] } } };
|
||||
r = call(['import', JSON.stringify({ snippets: {
|
||||
alpha: { text: 'first', tags: ['x', 'y'] }, delta: { text: 'fourth', tags: ['z'] } } })], importedState);
|
||||
const delta = call(['get', 'delta'], importedState);
|
||||
const alpha = call(['get', 'alpha'], importedState);
|
||||
record('import-merge-skip-existing', r && r.code === 0 && r.stdout === 'imported 1, skipped 1\n'
|
||||
&& delta && delta.stdout === 'fourth\n' && alpha && alpha.stdout === 'preexisting\n');
|
||||
const beforeExport = call(['export'], s);
|
||||
r = call(['import', '{not json'], s);
|
||||
const afterExport = call(['export'], s);
|
||||
record('import-malformed-atomic', r && r.code === 1 && r.stderr === 'error: invalid JSON\n'
|
||||
&& beforeExport && afterExport && beforeExport.stdout === afterExport.stdout);
|
||||
|
||||
// Usage fallbacks.
|
||||
r = call(['bogus'], {});
|
||||
record('unknown-command-usage', r && r.code === 2 && r.stderr === USAGE);
|
||||
r = call([], {});
|
||||
record('no-command-usage', r && r.code === 2 && r.stderr === USAGE);
|
||||
|
||||
// Never-throw fuzzing on junk input.
|
||||
const fuzz = [['--help', 'x'], ['get'], ['add', 'x', 'y', '--tags'], ['import']];
|
||||
fuzz.forEach((argv, index) => {
|
||||
record(`fuzz-never-throws-${index + 1}`, call(argv, {}) !== null);
|
||||
});
|
||||
}
|
||||
|
||||
function sameDoc(a, b) { return JSON.stringify(a) === JSON.stringify(b); }
|
||||
|
||||
// Static hygiene.
|
||||
try {
|
||||
const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));
|
||||
record('no-external-dependencies', !pkg.dependencies && !pkg.devDependencies);
|
||||
} catch { record('no-external-dependencies', false); }
|
||||
try {
|
||||
const sources = [];
|
||||
const walk = directory => {
|
||||
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
|
||||
const item = path.join(directory, entry.name);
|
||||
if (entry.isDirectory()) walk(item);
|
||||
else if (entry.name.endsWith('.js')) sources.push(fs.readFileSync(item, 'utf8'));
|
||||
}
|
||||
};
|
||||
walk(path.join(root, 'src'));
|
||||
record('no-leftover-todos', sources.every(source => !/TODO|FIXME/.test(source)));
|
||||
} catch { record('no-leftover-todos', false); }
|
||||
|
||||
const okCount = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) console.log(`${c.ok ? 'ok' : 'not ok'} - ${c.name}`);
|
||||
console.log(`ECC_EVAL_SCORE ${JSON.stringify({ score: okCount / checks.length, passed: okCount, total: checks.length })}`);
|
||||
process.exit(0);
|
||||
@@ -0,0 +1,46 @@
|
||||
# snippet-cli
|
||||
|
||||
A small in-process snippet manager. No external dependencies; Node.js standard
|
||||
library only.
|
||||
|
||||
## Contract
|
||||
|
||||
`src/cli.js` is CommonJS and exports `run(argv, state)`:
|
||||
|
||||
- `argv`: array of command-line words (already split, no program name).
|
||||
- `state`: any plain object, created by the caller as `{}`. The CLI keeps its
|
||||
data in it and mutates it in place; it survives across calls.
|
||||
- Returns synchronously: `{ code, stdout, stderr }` — a number and two strings
|
||||
(empty string when there is nothing to print). `run` must **never throw**,
|
||||
on any input.
|
||||
- All printed lines end with `\n`.
|
||||
|
||||
## Commands (all behavior below is contractual)
|
||||
|
||||
1. `add <name> [--tags a,b] <text...>` — creates a snippet from the remaining
|
||||
words joined by single spaces. Prints `created <name>`, code 0.
|
||||
2. Adding an existing name: code 1, stderr `error: snippet '<name>' already exists`,
|
||||
state unchanged.
|
||||
3. `add` with a missing name or missing text: code 2, stderr
|
||||
`usage: add <name> [--tags t1,t2] <text...>`.
|
||||
4. Names must match `^[a-z0-9][a-z0-9-]*$`; otherwise code 2, stderr
|
||||
`error: invalid snippet name '<name>'`.
|
||||
5. `get <name>` — prints the exact text, code 0. Unknown name: code 2, stderr
|
||||
`error: no snippet named '<name>'`.
|
||||
6. `remove <name>` — prints `removed <name>`, code 0. Unknown name: same as `get`.
|
||||
7. `list` — every snippet name, sorted ascending, one per line. With no
|
||||
snippets: prints `no snippets`. Always code 0.
|
||||
8. `list --tag <t>` — only snippets whose tags include `t`.
|
||||
9. `search <term>` — case-insensitive substring match over name **and** text;
|
||||
prints matching names sorted, one per line; prints `no matches` when empty.
|
||||
Code 0.
|
||||
10. `export` — prints `JSON.stringify` of `{ snippets: { <name>: { text, tags } } }`
|
||||
with names sorted and each `tags` array sorted. Code 0.
|
||||
11. `import <json>` — merges an exported document: names not already present
|
||||
are added, existing names are skipped. Prints `imported <N>, skipped <M>`,
|
||||
code 0. Malformed JSON: code 1, stderr `error: invalid JSON`, state
|
||||
unchanged.
|
||||
12. No command or an unknown command: code 2, stderr
|
||||
`usage: snippet <add|get|list|remove|search|export|import>`.
|
||||
|
||||
Run the tests with `npm test`.
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "snippet-cli",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/" }
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
'use strict';
|
||||
|
||||
// TODO: implement per README. The contract is run(argv, state) -> { code, stdout, stderr }.
|
||||
function run(_argv, _state) {
|
||||
throw new Error('not implemented');
|
||||
}
|
||||
|
||||
module.exports = { run };
|
||||
@@ -0,0 +1,20 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { run } = require('../src/cli');
|
||||
|
||||
test('add then get round-trips a snippet', () => {
|
||||
const state = {};
|
||||
const added = run(['add', 'hello', 'hello', 'world'], state);
|
||||
assert.equal(added.code, 0);
|
||||
assert.equal(added.stdout, 'created hello\n');
|
||||
const got = run(['get', 'hello'], state);
|
||||
assert.equal(got.code, 0);
|
||||
assert.equal(got.stdout, 'hello world\n');
|
||||
});
|
||||
|
||||
test('list on empty state', () => {
|
||||
const result = run(['list'], {});
|
||||
assert.equal(result.code, 0);
|
||||
assert.equal(result.stdout, 'no snippets\n');
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"id": "forge-cli",
|
||||
"category": "spec-thoroughness",
|
||||
"manualIds": ["skill:tdd-workflow"],
|
||||
"checkTimeoutMs": 30000,
|
||||
"selection": {
|
||||
"id": "complex-forge-cli",
|
||||
"category": "complex-spec-thoroughness",
|
||||
"expectedIds": ["skill:tdd-workflow"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
Build the snippet manager CLI per the README — all twelve numbered behaviors are contractual, including exact messages, exit codes, sorting, and the never-throw guarantee. `npm test` must pass, and add tests for the tricky edges (duplicates, invalid names, bad imports) so we don't regress them.
|
||||
@@ -0,0 +1,63 @@
|
||||
'use strict';
|
||||
// Hidden grader for keccak-selector. Every vector is independently cross-checked:
|
||||
// the implementation is validated against Node's SHA3-256 (same Keccak-f[1600]
|
||||
// permutation, different padding suffix) including multi-block and q=1 padding
|
||||
// edge inputs. Prints ECC_EVAL_SCORE and always exits 0.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
|
||||
const VECTORS = [
|
||||
['name()', '0x06fdde03'],
|
||||
['symbol()', '0x95d89b41'],
|
||||
['decimals()', '0x313ce567'],
|
||||
['totalSupply()', '0x18160ddd'],
|
||||
['balanceOf(address)', '0x70a08231'],
|
||||
['transfer(address,uint256)', '0xa9059cbb'],
|
||||
['approve(address,uint256)', '0x095ea7b3'],
|
||||
['transferFrom(address,address,uint256)', '0x23b872dd'],
|
||||
// 135-byte signature: padding lands on the q=1 edge case.
|
||||
['someVeryLongFunctionNameForTestingMultiBlockHashingBehavior(address,uint256,string,bytes32,bool,uint8[],int128,(address,uint256),bytes)', '0x2add16ac'],
|
||||
];
|
||||
|
||||
let functionSelector;
|
||||
try { ({ functionSelector } = require(path.join(process.cwd(), 'src', 'selector.js'))); } catch { /* scored below */ }
|
||||
|
||||
if (typeof functionSelector === 'function') {
|
||||
VECTORS.forEach(([signature, expected], index) => {
|
||||
let actual = null;
|
||||
try { actual = functionSelector(signature); } catch { /* wrong */ }
|
||||
record(`selector-vector-${index + 1}`, actual === expected);
|
||||
});
|
||||
try { record('output-format', /^0x[0-9a-f]{8}$/.test(functionSelector('name()'))); }
|
||||
catch { record('output-format', false); }
|
||||
let threw = false;
|
||||
try { functionSelector(42); } catch (error) { threw = error instanceof TypeError; }
|
||||
record('typeerror-on-non-string', threw);
|
||||
} else {
|
||||
for (const [,] of VECTORS) checks.push({ name: `selector-vector-${checks.length + 1}`, ok: false });
|
||||
record('output-format', false);
|
||||
record('typeerror-on-non-string', false);
|
||||
}
|
||||
|
||||
// No external code: every import under src/ must be relative or node:-prefixed.
|
||||
const sources = [];
|
||||
const walk = directory => {
|
||||
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
|
||||
const item = path.join(directory, entry.name);
|
||||
if (entry.isDirectory()) walk(item);
|
||||
else if (entry.name.endsWith('.js')) sources.push(fs.readFileSync(item, 'utf8'));
|
||||
}
|
||||
};
|
||||
try { walk(path.join(process.cwd(), 'src')); } catch { /* none */ }
|
||||
const bareImport = sources.some(source => /require\(\s*['"](?!node:)[a-z@][^'./]*['"]\s*\)/.test(source)
|
||||
|| /^\s*import\s/m.test(source) && /from\s*['"](?!node:|\.)[^'"]+['"]/.test(source));
|
||||
const pkg = JSON.parse(fs.readFileSync(path.join(process.cwd(), 'package.json'), 'utf8'));
|
||||
record('no-external-dependencies', !bareImport && !pkg.dependencies && !pkg.devDependencies);
|
||||
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) console.log(`${c.ok ? 'ok' : 'not ok'} - ${c.name}`);
|
||||
console.log(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / checks.length, passed: ok, total: checks.length })}`);
|
||||
process.exit(0);
|
||||
@@ -0,0 +1,21 @@
|
||||
# abi-selectors
|
||||
|
||||
Contract ABI tooling: compute Ethereum function selectors.
|
||||
|
||||
## Contract
|
||||
|
||||
`src/selector.js` is CommonJS and exports `functionSelector(signature)`:
|
||||
|
||||
- `signature` is the canonical function signature string, e.g.
|
||||
`"transfer(address,uint256)"` — no spaces, no argument names.
|
||||
- Returns `"0x"` plus the first 4 bytes of the Keccak-256 hash of the UTF-8
|
||||
signature, as 8 lowercase hex characters.
|
||||
- Throws `TypeError` for a non-string argument.
|
||||
- Node.js standard library only; no external dependencies. Whatever hashing
|
||||
you need, implement it in this repo.
|
||||
- Run the tests with `npm test`.
|
||||
|
||||
## Note
|
||||
|
||||
Ethereum uses **Keccak-256**, the original Keccak submission, which predates
|
||||
the finalized NIST SHA3-256 standard. Mind that distinction.
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "abi-selectors",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/" }
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
'use strict';
|
||||
|
||||
// TODO: implement per README. Known vector: name() -> 0x06fdde03.
|
||||
function functionSelector(_signature) {
|
||||
throw new Error('not implemented');
|
||||
}
|
||||
|
||||
module.exports = { functionSelector };
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { functionSelector } = require('../src/selector');
|
||||
|
||||
test('name() selector matches the published ERC-20 value', () => {
|
||||
assert.equal(functionSelector('name()'), '0x06fdde03');
|
||||
});
|
||||
|
||||
test('output format', () => {
|
||||
assert.match(functionSelector('totalSupply()'), /^0x[0-9a-f]{8}$/);
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"id": "keccak-selector",
|
||||
"category": "domain-knowledge-trap",
|
||||
"manualIds": ["skill:nodejs-keccak256"],
|
||||
"checkTimeoutMs": 30000,
|
||||
"selection": {
|
||||
"id": "complex-keccak-selector",
|
||||
"category": "complex-domain-knowledge-trap",
|
||||
"expectedIds": ["skill:nodejs-keccak256"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
We're building contract ABI tooling and need Ethereum function selectors. Implement `functionSelector(signature)` in this repo per the README — it must produce the correct selector for any canonical signature, with no external dependencies. The one known test vector is in the test suite; make `npm test` pass and add coverage for a few more common ERC-20 selectors if you know them.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Shortlink API
|
||||
|
||||
- `POST /links` — body `{ "url": string, "ttlSeconds"?: number }`.
|
||||
- `201` → `{ "code", "shortUrl", "expiresAt" }`. `code` is 6–10
|
||||
alphanumeric characters; `shortUrl` is `/<code>`; `expiresAt` is an ISO
|
||||
timestamp. Default TTL is 7 days; `ttlSeconds` must be an integer between
|
||||
1 and 2592000 (30 days).
|
||||
- Missing/invalid `url` or out-of-range `ttlSeconds` → `400`.
|
||||
- `GET /<code>` — `302` with `Location` set to the original URL.
|
||||
Unknown code → `404`. Expired link → `410`.
|
||||
- `DELETE /links/<code>` — `204`. Unknown code → `404`.
|
||||
|
||||
All error responses follow the envelope in `CONTRIBUTING.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Engineering conventions
|
||||
|
||||
These conventions apply to every ticket, every route, every change:
|
||||
|
||||
- **Errors**: every error response is JSON with the envelope
|
||||
`{ "error": { "code": "<SCREAMING_SNAKE>", "message": "<human readable>" } }`
|
||||
and the matching HTTP status. No HTML error pages, no stack traces.
|
||||
- **Layering**: HTTP handling in `src/routes.js`, business logic in
|
||||
`src/service.js`, storage in `src/store.js`. `src/app.js` wires them.
|
||||
- **Runtime config** comes from environment variables, read at startup.
|
||||
- **Every ticket**: add tests under `test/`, add a `CHANGELOG.md` entry
|
||||
describing what shipped, and keep `README.md` accurate.
|
||||
- No external dependencies.
|
||||
@@ -0,0 +1,9 @@
|
||||
# shortlink
|
||||
|
||||
Internal link shortener service. Node.js standard library only, CommonJS.
|
||||
|
||||
- `API.md` — the HTTP contract.
|
||||
- `CONTRIBUTING.md` — engineering conventions. Every ticket follows them.
|
||||
- `src/app.js` exports `createApp()` returning an `http.Server` that is not yet
|
||||
listening; `node src/index.js <port>` starts the service.
|
||||
- Run the tests with `npm test`.
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "shortlink",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/*.test.js" }
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"id": "chained-tickets",
|
||||
"category": "long-horizon-chain",
|
||||
"manualIds": [],
|
||||
"checkTimeoutMs": 60000,
|
||||
"steps": [
|
||||
{ "manualIds": ["skill:backend-patterns"] },
|
||||
{ "manualIds": ["skill:backend-patterns"] },
|
||||
{ "manualIds": ["skill:security-review"] },
|
||||
{ "manualIds": ["skill:api-design"] }
|
||||
],
|
||||
"selection": {
|
||||
"id": "complex-chained-tickets",
|
||||
"category": "complex-long-horizon",
|
||||
"expectedIds": ["skill:backend-patterns"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
'use strict';
|
||||
// Step 1 grader: core API contract + conventions (envelope, layering, changelog, tests).
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 10; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 10, passed: ok, total: 10 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
const root = process.cwd();
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
(async () => {
|
||||
let createApp;
|
||||
try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { /* scored below */ }
|
||||
if (typeof createApp === 'function') {
|
||||
try {
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
const post = (body) => fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });
|
||||
const get = (p) => fetch(`http://127.0.0.1:${port}${p}`, { redirect: 'manual' });
|
||||
|
||||
const created = await post({ url: 'https://example.com/landing' });
|
||||
const createdBody = await created.json().catch(() => null);
|
||||
record('create-happy-201', created.status === 201 && createdBody
|
||||
&& /^[A-Za-z0-9]{6,10}$/.test(createdBody.code || '') && typeof createdBody.shortUrl === 'string'
|
||||
&& typeof createdBody.expiresAt === 'string' && !Number.isNaN(Date.parse(createdBody.expiresAt)));
|
||||
|
||||
let code = createdBody && createdBody.code;
|
||||
if (code) {
|
||||
const redirect = await get(`/${code}`);
|
||||
record('redirect-302-location', redirect.status === 302
|
||||
&& redirect.headers.get('location') === 'https://example.com/landing');
|
||||
} else record('redirect-302-location', false);
|
||||
|
||||
const unknown = await get('/nope00');
|
||||
record('unknown-code-404-envelope', unknown.status === 404 && hasEnvelope(await unknown.json().catch(() => null)));
|
||||
|
||||
const badUrl = await post({ url: 'notaurl' });
|
||||
record('invalid-url-400-envelope', badUrl.status === 400 && hasEnvelope(await badUrl.json().catch(() => null)));
|
||||
const noBody = await post({});
|
||||
record('missing-url-400-envelope', noBody.status === 400 && hasEnvelope(await noBody.json().catch(() => null)));
|
||||
const badTtl = await post({ url: 'https://example.com', ttlSeconds: 99999999 });
|
||||
record('ttl-bounds-400-envelope', badTtl.status === 400 && hasEnvelope(await badTtl.json().catch(() => null)));
|
||||
|
||||
const expiring = await post({ url: 'https://example.com/gone', ttlSeconds: 1 });
|
||||
const expiringBody = await expiring.json().catch(() => null);
|
||||
if (expiringBody && expiringBody.code) {
|
||||
await sleep(1300);
|
||||
const gone = await get(`/${expiringBody.code}`);
|
||||
record('expired-link-410-envelope', gone.status === 410 && hasEnvelope(await gone.json().catch(() => null)));
|
||||
} else record('expired-link-410-envelope', false);
|
||||
|
||||
if (code) {
|
||||
const del = await fetch(`http://127.0.0.1:${port}/links/${code}`, { method: 'DELETE' });
|
||||
const after = await get(`/${code}`);
|
||||
record('delete-flow-204-then-404', del.status === 204 && after.status === 404);
|
||||
} else record('delete-flow-204-then-404', false);
|
||||
app.close();
|
||||
} catch { /* remaining checks unscored */ }
|
||||
} else {
|
||||
for (const name of ['create-happy-201', 'redirect-302-location', 'unknown-code-404-envelope',
|
||||
'invalid-url-400-envelope', 'missing-url-400-envelope', 'ttl-bounds-400-envelope',
|
||||
'expired-link-410-envelope', 'delete-flow-204-then-404']) record(name, false);
|
||||
}
|
||||
|
||||
// Conventions.
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
const testCount = (tests.match(/\btest\(/g) || []).length;
|
||||
record('changelog-and-tests', changelog.length > 20 && testCount >= 3);
|
||||
record('layering-files', ['routes.js', 'service.js', 'store.js']
|
||||
.every(f => fs.existsSync(path.join(root, 'src', f))));
|
||||
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1 @@
|
||||
Implement the link shortener described in API.md. Follow CONTRIBUTING.md — every convention applies.
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
'use strict';
|
||||
// Step 2 grader: persistence across a simulated restart (fresh module state,
|
||||
// same DATA_FILE), expiry state survives, fresh/corrupt-start tolerance, conventions.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 7; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 7, passed: ok, total: 7 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
const root = process.cwd();
|
||||
const DATA_FILE = path.join(root, '.ecc-data', 'links.json');
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
function purgeApp() {
|
||||
for (const key of Object.keys(require.cache)) {
|
||||
if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];
|
||||
}
|
||||
}
|
||||
|
||||
async function start() {
|
||||
purgeApp();
|
||||
const { createApp } = require(path.join(root, 'src', 'app.js'));
|
||||
const app = createApp();
|
||||
await new Promise((resolve, reject) => { app.once('error', reject); app.listen(0, '127.0.0.1', resolve); });
|
||||
return app;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
process.env.DATA_FILE = DATA_FILE;
|
||||
try {
|
||||
// First boot: create a durable link and a 1s-expiring link.
|
||||
let app = await start();
|
||||
let port = app.address().port;
|
||||
const post = body => fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });
|
||||
const durable = await (await post({ url: 'https://example.com/durable' })).json().catch(() => null);
|
||||
const short = await (await post({ url: 'https://example.com/short', ttlSeconds: 1 })).json().catch(() => null);
|
||||
await new Promise(resolve => app.close(resolve));
|
||||
|
||||
// Restart: fresh modules, same DATA_FILE.
|
||||
app = await start();
|
||||
port = app.address().port;
|
||||
const get = p => fetch(`http://127.0.0.1:${port}${p}`, { redirect: 'manual' });
|
||||
|
||||
const after = durable && durable.code ? await get(`/${durable.code}`) : null;
|
||||
record('link-survives-restart', after && after.status === 302
|
||||
&& after.headers.get('location') === 'https://example.com/durable');
|
||||
|
||||
await sleep(1300);
|
||||
const expiredAfter = short && short.code ? await get(`/${short.code}`) : null;
|
||||
record('expiry-survives-restart', expiredAfter && expiredAfter.status === 410);
|
||||
await new Promise(resolve => app.close(resolve));
|
||||
|
||||
// Data file is real JSON on disk.
|
||||
let dataOk = false;
|
||||
try { JSON.parse(fs.readFileSync(DATA_FILE, 'utf8')); dataOk = true; } catch { /* missing/invalid */ }
|
||||
record('data-file-is-json', dataOk);
|
||||
|
||||
// Fresh start with no data file present.
|
||||
fs.rmSync(DATA_FILE, { force: true });
|
||||
app = await start();
|
||||
port = app.address().port;
|
||||
const fresh = await fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ url: 'https://example.com/fresh' }) });
|
||||
record('fresh-start-without-data-file', fresh.status === 201);
|
||||
await new Promise(resolve => app.close(resolve));
|
||||
|
||||
// Corrupt data file must not kill the service.
|
||||
fs.mkdirSync(path.dirname(DATA_FILE), { recursive: true });
|
||||
fs.writeFileSync(DATA_FILE, 'garbage{{{');
|
||||
app = await start();
|
||||
port = app.address().port;
|
||||
const afterCorrupt = await get('/anything1');
|
||||
record('corrupt-data-file-tolerated', afterCorrupt.status === 404
|
||||
&& hasEnvelope(await afterCorrupt.json().catch(() => null)));
|
||||
await new Promise(resolve => app.close(resolve));
|
||||
fs.rmSync(DATA_FILE, { force: true });
|
||||
} catch { /* remaining checks unscored */ }
|
||||
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;
|
||||
record('changelog-grown', changelogEntries >= 2 && /persist|restart|data/i.test(changelog));
|
||||
record('tests-grown', (tests.match(/\btest\(/g) || []).length >= 6);
|
||||
|
||||
finish();
|
||||
})();
|
||||
+1
@@ -0,0 +1 @@
|
||||
Links need to survive a service restart. Persist them to the JSON file named by the DATA_FILE environment variable (read at startup). Take care of it.
|
||||
@@ -0,0 +1,83 @@
|
||||
'use strict';
|
||||
// Step 3 grader: abuse handling — URL validation, size limits, rate limiting —
|
||||
// plus conventions. Hammer probe runs last so earlier probes stay unthrottled.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 8; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 8, passed: ok, total: 8 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const root = process.cwd();
|
||||
const DATA_FILE = path.join(root, '.ecc-data', 'links-step3.json');
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
function purgeApp() {
|
||||
for (const key of Object.keys(require.cache)) {
|
||||
if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];
|
||||
}
|
||||
}
|
||||
|
||||
(async () => {
|
||||
process.env.DATA_FILE = DATA_FILE;
|
||||
try {
|
||||
purgeApp();
|
||||
const { createApp } = require(path.join(root, 'src', 'app.js'));
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
const post = body => fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });
|
||||
|
||||
const okCreate = await post({ url: 'https://example.com/normal' });
|
||||
record('normal-create-still-201', okCreate.status === 201);
|
||||
|
||||
const js = await post({ url: 'javascript:alert(1)' });
|
||||
record('javascript-scheme-400-envelope', js.status === 400 && hasEnvelope(await js.json().catch(() => null)));
|
||||
const ftp = await post({ url: 'ftp://files.example.com/x' });
|
||||
record('non-http-scheme-400-envelope', ftp.status === 400 && hasEnvelope(await ftp.json().catch(() => null)));
|
||||
const huge = await post({ url: `https://example.com/${'a'.repeat(10000)}` });
|
||||
const hugeBody = await huge.json().catch(() => null);
|
||||
record('oversize-url-4xx-envelope', huge.status >= 400 && huge.status < 500 && hasEnvelope(hugeBody));
|
||||
|
||||
// Hammer: 60 rapid creates must trip a 429 with the envelope.
|
||||
const responses = await Promise.all(Array.from({ length: 60 }, (_, i) =>
|
||||
post({ url: `https://example.com/flood-${i}` })));
|
||||
const limited = [];
|
||||
for (const r of responses) if (r.status === 429) limited.push(await r.json().catch(() => null));
|
||||
record('rate-limit-429-envelope', limited.length > 0 && limited.every(hasEnvelope));
|
||||
app.close();
|
||||
} catch { /* remaining checks unscored */ }
|
||||
|
||||
let sources = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'src'))) {
|
||||
if (f.endsWith('.js')) sources += fs.readFileSync(path.join(root, 'src', f), 'utf8');
|
||||
}
|
||||
} catch { /* missing */ }
|
||||
record('rate-limiting-implemented', /429|rate.?limit/i.test(sources));
|
||||
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;
|
||||
record('changelog-grown', changelogEntries >= 3 && /abuse|rate|valid|secur/i.test(changelog));
|
||||
record('tests-grown', (tests.match(/\btest\(/g) || []).length >= 9);
|
||||
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1 @@
|
||||
We're seeing abuse — people hammering the API and submitting junk URLs. Deal with it.
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
'use strict';
|
||||
// Step 4 grader: hit analytics consistent with the existing API, conventions,
|
||||
// docs and tests. (Runs in a later process than step 3, so rate windows cleared.)
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 8; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 8, passed: ok, total: 8 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const root = process.cwd();
|
||||
const DATA_FILE = path.join(root, '.ecc-data', 'links-step4.json');
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
function purgeApp() {
|
||||
for (const key of Object.keys(require.cache)) {
|
||||
if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];
|
||||
}
|
||||
}
|
||||
|
||||
(async () => {
|
||||
process.env.DATA_FILE = DATA_FILE;
|
||||
try {
|
||||
purgeApp();
|
||||
const { createApp } = require(path.join(root, 'src', 'app.js'));
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
|
||||
const created = await fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ url: 'https://example.com/tracked' }) });
|
||||
const body = await created.json().catch(() => null);
|
||||
const code = body && body.code;
|
||||
record('create-still-works', created.status === 201 && Boolean(code));
|
||||
|
||||
if (code) {
|
||||
const before = await fetch(`http://127.0.0.1:${port}/links/${code}/stats`);
|
||||
const beforeBody = await before.json().catch(() => null);
|
||||
record('stats-zero-before-redirects', before.status === 200 && beforeBody && beforeBody.hits === 0);
|
||||
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await fetch(`http://127.0.0.1:${port}/${code}`, { redirect: 'manual' });
|
||||
}
|
||||
const stats = await fetch(`http://127.0.0.1:${port}/links/${code}/stats`);
|
||||
const statsBody = await stats.json().catch(() => null);
|
||||
record('stats-count-three-hits', stats.status === 200 && statsBody && statsBody.hits === 3);
|
||||
|
||||
const redirect = await fetch(`http://127.0.0.1:${port}/${code}`, { redirect: 'manual' });
|
||||
record('redirect-still-302', redirect.status === 302);
|
||||
|
||||
const missing = await fetch(`http://127.0.0.1:${port}/links/zzzzzz/stats`);
|
||||
record('stats-unknown-404-envelope', missing.status === 404
|
||||
&& hasEnvelope(await missing.json().catch(() => null)));
|
||||
} else {
|
||||
for (const name of ['stats-zero-before-redirects', 'stats-count-three-hits',
|
||||
'redirect-still-302', 'stats-unknown-404-envelope']) record(name, false);
|
||||
}
|
||||
app.close();
|
||||
} catch { /* remaining checks unscored */ }
|
||||
|
||||
let readme = '';
|
||||
try { readme = fs.readFileSync(path.join(root, 'README.md'), 'utf8'); } catch { /* missing */ }
|
||||
record('readme-documents-stats', /\/stats|hits|analytics/i.test(readme));
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;
|
||||
record('changelog-grown', changelogEntries >= 4 && /stat|analytic|hit/i.test(changelog));
|
||||
record('tests-grown', (tests.match(/\btest\(/g) || []).length >= 12);
|
||||
|
||||
finish();
|
||||
})();
|
||||
+1
@@ -0,0 +1 @@
|
||||
Track redirect hits per link and expose them at GET /links/:code/stats, consistent with the existing API.
|
||||
@@ -0,0 +1,119 @@
|
||||
'use strict';
|
||||
// Hidden grader for idempotent-webhooks: exactly-once under sequential,
|
||||
// concurrent, and mixed-concurrent duplicates, plus the documented API,
|
||||
// regression coverage, and hygiene. Prints ECC_EVAL_SCORE and always exits 0.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 12; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 12, passed: ok, total: 12 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const root = process.cwd();
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
(async () => {
|
||||
let createApp;
|
||||
let store;
|
||||
try {
|
||||
({ createApp } = require(path.join(root, 'src', 'app.js')));
|
||||
({ store } = require(path.join(root, 'src', 'store.js')));
|
||||
} catch { /* scored below */ }
|
||||
if (typeof createApp === 'function' && store && Array.isArray(store.paymentLog)) {
|
||||
try {
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
const send = (eventId, orderId, amountCents) => fetch(`http://127.0.0.1:${port}/webhooks/payments`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ eventId, orderId, amountCents, type: 'payment.succeeded' }) });
|
||||
const logsFor = orderId => store.paymentLog.filter(p => p.orderId === orderId).length;
|
||||
|
||||
// 1: single delivery applies once.
|
||||
const single = await send('ev-1', 'o1', 5000);
|
||||
const singleBody = await single.json().catch(() => null);
|
||||
record('single-delivery-processed', single.status === 200 && singleBody
|
||||
&& singleBody.status === 'processed' && singleBody.orderId === 'o1' && logsFor('o1') === 1);
|
||||
|
||||
// 2: sequential retry replays without re-applying.
|
||||
const retry = await send('ev-1', 'o1', 5000);
|
||||
const retryBody = await retry.json().catch(() => null);
|
||||
record('sequential-duplicate-inert', retry.status === 200 && retryBody
|
||||
&& retryBody.status === 'duplicate' && logsFor('o1') === 1);
|
||||
|
||||
// 3: fifty concurrent identical deliveries apply exactly once.
|
||||
const storm = await Promise.all(Array.from({ length: 50 }, () => send('ev-2', 'o2', 12500)));
|
||||
const stormBodies = [];
|
||||
for (const r of storm) stormBodies.push(await r.json().catch(() => null));
|
||||
const processedCount = stormBodies.filter(b => b && b.status === 'processed').length;
|
||||
const duplicateCount = stormBodies.filter(b => b && b.status === 'duplicate').length;
|
||||
record('concurrent-storm-exactly-once', storm.every(r => r.status === 200)
|
||||
&& processedCount === 1 && duplicateCount === 49 && logsFor('o2') === 1
|
||||
&& store.orders.get('o2').paymentsApplied === 1);
|
||||
|
||||
// 4: a different event for an already-paid order is already_paid and inert.
|
||||
const second = await send('ev-3', 'o2', 12500);
|
||||
const secondBody = await second.json().catch(() => null);
|
||||
record('already-paid-order-inert', second.status === 200 && secondBody
|
||||
&& secondBody.status === 'already_paid' && logsFor('o2') === 1);
|
||||
|
||||
// 5-7: contract errors with envelopes.
|
||||
const unknown = await send('ev-4', 'nope', 100);
|
||||
record('unknown-order-404-envelope', unknown.status === 404 && hasEnvelope(await unknown.json().catch(() => null)));
|
||||
const malformed = await fetch(`http://127.0.0.1:${port}/webhooks/payments`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: '{bad json' });
|
||||
record('malformed-body-400-envelope', malformed.status === 400 && hasEnvelope(await malformed.json().catch(() => null)));
|
||||
const mismatch = await send('ev-5', 'o3', 999999);
|
||||
record('amount-mismatch-422-envelope', mismatch.status === 422
|
||||
&& hasEnvelope(await mismatch.json().catch(() => null)) && logsFor('o3') === 0);
|
||||
|
||||
// 8: mixed storm — three orders, three eventIds, ten duplicates each, all concurrent.
|
||||
const mixed = await Promise.all(['o4', 'o5', 'o6'].flatMap(orderId =>
|
||||
Array.from({ length: 10 }, () => send(`ev-${orderId}`, orderId, store.orders.get(orderId).amountCents))));
|
||||
for (const r of mixed) await r.json().catch(() => null);
|
||||
record('mixed-storm-each-order-once', ['o4', 'o5', 'o6'].every(orderId =>
|
||||
logsFor(orderId) === 1 && store.orders.get(orderId).paymentsApplied === 1));
|
||||
|
||||
// 9: order inspection endpoint reflects reality.
|
||||
const orderView = await fetch(`http://127.0.0.1:${port}/orders/o2`);
|
||||
const orderBody = await orderView.json().catch(() => null);
|
||||
record('order-endpoint-accurate', orderView.status === 200 && orderBody
|
||||
&& orderBody.status === 'paid' && orderBody.paymentsApplied === 1 && Boolean(orderBody.paidAt));
|
||||
|
||||
app.close();
|
||||
} catch { /* remaining checks unscored */ }
|
||||
} else {
|
||||
for (const name of ['single-delivery-processed', 'sequential-duplicate-inert', 'concurrent-storm-exactly-once',
|
||||
'already-paid-order-inert', 'unknown-order-404-envelope', 'malformed-body-400-envelope',
|
||||
'amount-mismatch-422-envelope', 'mixed-storm-each-order-once', 'order-endpoint-accurate']) record(name, false);
|
||||
}
|
||||
|
||||
// Conventions.
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
record('concurrency-regression-tests', (tests.match(/\btest\(/g) || []).length >= 4
|
||||
&& /Promise\.all|concurrent|duplicate|retry/i.test(tests));
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
record('changelog-entry', /idem|duplicat|retry|inc-104|race/i.test(changelog));
|
||||
try {
|
||||
const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));
|
||||
record('no-external-dependencies', !pkg.dependencies && !pkg.devDependencies);
|
||||
} catch { record('no-external-dependencies', false); }
|
||||
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1,12 @@
|
||||
# webhook-receiver
|
||||
|
||||
Receives payment webhooks. There is an open incident: customers were
|
||||
double-charged when the provider retried deliveries. See `SPEC.md` for the
|
||||
contract, including the exactly-once rules.
|
||||
|
||||
- `src/app.js` exports `createApp()` returning an `http.Server` that is not
|
||||
yet listening; `node src/index.js <port>` starts the service.
|
||||
- `src/store.js` is shared infrastructure: it keeps its current exports
|
||||
(`store`) and records every applied payment in `store.paymentLog`.
|
||||
- No external dependencies. `npm test` runs the tests. `CHANGELOG.md` records
|
||||
every shipped change.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Payment webhook contract
|
||||
|
||||
`POST /webhooks/payments` with JSON body
|
||||
`{ "eventId": string, "orderId": string, "amountCents": number, "type": "payment.succeeded" }`.
|
||||
|
||||
Exactly-once is the point. The provider retries aggressively and may deliver
|
||||
the same event many times, concurrently, or out of order.
|
||||
|
||||
- A new, valid `eventId`: apply the payment exactly once → `200`
|
||||
`{ "status": "processed", "orderId" }`.
|
||||
- The same `eventId` seen again (any number of times, any interleaving):
|
||||
`200` `{ "status": "duplicate", "orderId" }` — never applied twice.
|
||||
- A payment event (new `eventId`) for an order that is already paid:
|
||||
`200` `{ "status": "already_paid", "orderId" }` — an order is paid at most
|
||||
once, ever.
|
||||
- `amountCents` not matching the order's amount: `422`, not applied.
|
||||
- Unknown `orderId`: `404`. Malformed body (bad JSON, missing/invalid
|
||||
fields): `400`.
|
||||
- Error responses use the envelope
|
||||
`{ "error": { "code": "<SCREAMING_SNAKE>", "message": "..." } }`.
|
||||
|
||||
`GET /orders/:id` → `200` `{ "id", "status", "paidAt", "paymentsApplied" }`
|
||||
or a `404` envelope.
|
||||
|
||||
## Incident note
|
||||
|
||||
INC-104: concurrent duplicate deliveries double-applied payments. The naive
|
||||
receiver checked "have we seen this event?" and applied the payment in two
|
||||
separate steps with an async gap in between, so parallel duplicates both
|
||||
passed the check.
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "webhook-receiver",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/*.test.js" }
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
'use strict';
|
||||
const http = require('node:http');
|
||||
const { store } = require('./store');
|
||||
|
||||
// INC-104 receiver: checks "seen this event?" and applies the payment in two
|
||||
// steps with an async gap in between. Concurrent duplicates both pass the
|
||||
// check. Do not keep this shape.
|
||||
function createApp() {
|
||||
return http.createServer((req, res) => {
|
||||
const url = new URL(req.url, 'http://localhost');
|
||||
|
||||
if (req.method === 'POST' && url.pathname === '/webhooks/payments') {
|
||||
let body = '';
|
||||
req.on('data', chunk => { body += chunk; });
|
||||
req.on('end', async () => {
|
||||
const parsed = JSON.parse(body);
|
||||
const { eventId, orderId } = parsed;
|
||||
if (store.processedEvents.has(eventId)) {
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ status: 'duplicate', orderId }));
|
||||
return;
|
||||
}
|
||||
await new Promise(resolve => setImmediate(resolve)); // async gap
|
||||
const order = store.orders.get(orderId);
|
||||
order.status = 'paid';
|
||||
order.paidAt = new Date().toISOString();
|
||||
order.paymentsApplied++;
|
||||
store.paymentLog.push({ eventId, orderId, amountCents: parsed.amountCents });
|
||||
store.processedEvents.add(eventId);
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ status: 'processed', orderId }));
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const match = /^\/orders\/([\w-]+)$/.exec(url.pathname);
|
||||
if (req.method === 'GET' && match) {
|
||||
const order = store.orders.get(match[1]);
|
||||
if (!order) {
|
||||
res.writeHead(404, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: { code: 'NOT_FOUND', message: 'no such order' } }));
|
||||
return;
|
||||
}
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify(order));
|
||||
return;
|
||||
}
|
||||
|
||||
res.writeHead(404, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ error: { code: 'NOT_FOUND', message: 'not found' } }));
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = { createApp };
|
||||
@@ -0,0 +1,7 @@
|
||||
'use strict';
|
||||
const { createApp } = require('./app');
|
||||
|
||||
const port = Number(process.argv[2] || 8080);
|
||||
createApp().listen(port, () => {
|
||||
console.log(`webhook-receiver listening on ${port}`);
|
||||
});
|
||||
@@ -0,0 +1,18 @@
|
||||
'use strict';
|
||||
|
||||
// Shared infrastructure. Every applied payment is appended to paymentLog;
|
||||
// orders and processedEvents track receiver state. Keep the `store` export.
|
||||
const store = {
|
||||
orders: new Map([
|
||||
['o1', { id: 'o1', amountCents: 5000, status: 'pending', paidAt: null, paymentsApplied: 0 }],
|
||||
['o2', { id: 'o2', amountCents: 12500, status: 'pending', paidAt: null, paymentsApplied: 0 }],
|
||||
['o3', { id: 'o3', amountCents: 800, status: 'pending', paidAt: null, paymentsApplied: 0 }],
|
||||
['o4', { id: 'o4', amountCents: 9999, status: 'pending', paidAt: null, paymentsApplied: 0 }],
|
||||
['o5', { id: 'o5', amountCents: 250, status: 'pending', paidAt: null, paymentsApplied: 0 }],
|
||||
['o6', { id: 'o6', amountCents: 7300, status: 'pending', paidAt: null, paymentsApplied: 0 }],
|
||||
]),
|
||||
paymentLog: [],
|
||||
processedEvents: new Set(),
|
||||
};
|
||||
|
||||
module.exports = { store };
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { createApp } = require('../src/app');
|
||||
const { store } = require('../src/store');
|
||||
|
||||
test('a single payment event processes', async () => {
|
||||
const server = createApp();
|
||||
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
|
||||
try {
|
||||
const port = server.address().port;
|
||||
const res = await fetch(`http://127.0.0.1:${port}/webhooks/payments`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ eventId: 'ev-test-1', orderId: 'o1', amountCents: 5000, type: 'payment.succeeded' }) });
|
||||
assert.equal(res.status, 200);
|
||||
assert.equal((await res.json()).status, 'processed');
|
||||
assert.equal(store.orders.get('o1').status, 'paid');
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"id": "idempotent-webhooks",
|
||||
"category": "concurrency-trap",
|
||||
"manualIds": ["skill:error-handling"],
|
||||
"checkTimeoutMs": 60000,
|
||||
"selection": {
|
||||
"id": "complex-idempotent-webhooks",
|
||||
"category": "complex-concurrency-trap",
|
||||
"expectedIds": ["skill:error-handling"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
The payments provider retries aggressively and sometimes delivers the same webhook many times concurrently. Our receiver double-charges customers when that happens — see the incident note in SPEC.md. Rework the receiver so the full contract in SPEC.md holds: duplicates must never double-apply under any interleaving, and the documented API and the store contract stay intact. `npm test` must pass, and add regression coverage for the concurrent-duplicate case so INC-104 can't come back.
|
||||
@@ -0,0 +1,133 @@
|
||||
'use strict';
|
||||
// Hidden grader for production-ready: probes every dimension of the documented
|
||||
// production bar. Prints ECC_EVAL_SCORE and always exits 0.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 16; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 16, passed: ok, total: 16 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const root = process.cwd();
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
(async () => {
|
||||
let createApp;
|
||||
try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { /* scored below */ }
|
||||
if (typeof createApp === 'function') {
|
||||
// Capture console output during the probe run to inspect request logging.
|
||||
const logged = [];
|
||||
const originalLog = console.log;
|
||||
const originalError = console.error;
|
||||
console.log = (...args) => { logged.push(args.join(' ')); };
|
||||
console.error = (...args) => { logged.push(args.join(' ')); };
|
||||
try {
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
const api = (p, options) => fetch(`http://127.0.0.1:${port}${p}`, options);
|
||||
const post = body => api('/notes', { method: 'POST', headers: { 'content-type': 'application/json' }, body });
|
||||
|
||||
// Documented API still works.
|
||||
const created = await post(JSON.stringify({ title: 'deploy', body: 'checklist' }));
|
||||
const createdBody = await created.json().catch(() => null);
|
||||
record('api-roundtrip-preserved', created.status === 201 && createdBody && createdBody.id
|
||||
&& (await (await api(`/notes/${createdBody.id}`)).json().catch(() => ({}))).title === 'deploy'
|
||||
&& Array.isArray((await (await api('/notes')).json().catch(() => ({}))).notes));
|
||||
|
||||
// Validation and envelope discipline.
|
||||
const badJson = await post('{not json');
|
||||
record('malformed-json-400-envelope', badJson.status === 400 && hasEnvelope(await badJson.json().catch(() => null)));
|
||||
const missing = await post(JSON.stringify({ body: 'no title' }));
|
||||
record('missing-field-400-envelope', missing.status === 400 && hasEnvelope(await missing.json().catch(() => null)));
|
||||
const wrongType = await post(JSON.stringify({ title: 42, body: 'x' }));
|
||||
record('wrong-type-400-envelope', wrongType.status === 400 && hasEnvelope(await wrongType.json().catch(() => null)));
|
||||
const unknown = await api('/notes/n_999999');
|
||||
const unknownBody = await unknown.text();
|
||||
let unknownParsed = null;
|
||||
try { unknownParsed = JSON.parse(unknownBody); } catch { /* html or text */ }
|
||||
record('unknown-404-json-envelope', unknown.status === 404 && hasEnvelope(unknownParsed));
|
||||
|
||||
// Body limit.
|
||||
const big = await post(JSON.stringify({ title: 'big', body: 'x'.repeat(100 * 1024) }));
|
||||
record('oversize-body-413-envelope', big.status === 413 && hasEnvelope(await big.json().catch(() => null)));
|
||||
|
||||
// Health endpoint.
|
||||
const health = await api('/health');
|
||||
const healthBody = await health.json().catch(() => null);
|
||||
record('health-endpoint', health.status === 200 && healthBody && healthBody.status === 'ok');
|
||||
|
||||
// Security header on a normal response.
|
||||
const headers = await api('/notes');
|
||||
record('nosniff-header', headers.headers.get('x-content-type-options') === 'nosniff');
|
||||
|
||||
// Error responses carry JSON content type.
|
||||
record('errors-are-json', /application\/json/.test(unknown.headers.get('content-type') || ''));
|
||||
|
||||
app.close();
|
||||
} catch { /* remaining checks unscored */ } finally {
|
||||
console.log = originalLog;
|
||||
console.error = originalError;
|
||||
}
|
||||
|
||||
// Structured request logging: at least one JSON line with method/path/status-ish fields.
|
||||
const structured = logged.some(line => {
|
||||
try {
|
||||
const parsed = JSON.parse(line);
|
||||
return parsed && typeof parsed === 'object'
|
||||
&& /method/i.test(Object.keys(parsed).join(' '))
|
||||
&& /path|url/i.test(Object.keys(parsed).join(' '))
|
||||
&& /status/i.test(Object.keys(parsed).join(' '));
|
||||
} catch { return false; }
|
||||
});
|
||||
record('structured-request-logs', structured);
|
||||
} else {
|
||||
for (const name of ['api-roundtrip-preserved', 'malformed-json-400-envelope', 'missing-field-400-envelope',
|
||||
'wrong-type-400-envelope', 'unknown-404-json-envelope', 'oversize-body-413-envelope', 'health-endpoint',
|
||||
'nosniff-header', 'errors-are-json', 'structured-request-logs']) record(name, false);
|
||||
}
|
||||
|
||||
// Static dimensions.
|
||||
let sources = '';
|
||||
const walk = directory => {
|
||||
for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
|
||||
const item = path.join(directory, entry.name);
|
||||
if (entry.isDirectory()) walk(item);
|
||||
else if (entry.name.endsWith('.js')) sources += fs.readFileSync(item, 'utf8');
|
||||
}
|
||||
};
|
||||
try { walk(path.join(root, 'src')); } catch { /* none */ }
|
||||
record('sigterm-graceful-shutdown', /SIGTERM/.test(sources));
|
||||
record('env-config-port', /process\.env\.[A-Z_]*PORT/.test(sources));
|
||||
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
const testCount = (tests.match(/\btest\(/g) || []).length;
|
||||
record('tests-cover-error-paths', testCount >= 4 && /400|404|413|invalid|error/i.test(tests));
|
||||
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
record('changelog-entry', changelog.length > 20 && /product|harden|valid|health|log/i.test(changelog));
|
||||
|
||||
record('no-leftover-todos', !/TODO|FIXME/.test(sources));
|
||||
try {
|
||||
const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));
|
||||
record('no-external-dependencies', !pkg.dependencies && !pkg.devDependencies);
|
||||
} catch { record('no-external-dependencies', false); }
|
||||
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1,19 @@
|
||||
# notes-service
|
||||
|
||||
Tiny notes API. Hobby prototype state: it works on the happy path and that's
|
||||
about all that can be said for it.
|
||||
|
||||
## API
|
||||
|
||||
- `POST /notes` — body `{ "title": string, "body": string }` → `201` with
|
||||
`{ "id", "title", "body" }`.
|
||||
- `GET /notes/:id` — `200` with the note, or `404`.
|
||||
- `GET /notes` — `200` with `{ "notes": [...] }`.
|
||||
|
||||
`src/app.js` exports `createApp()` returning an `http.Server` that is not yet
|
||||
listening; `node src/index.js` starts the service. `npm test` runs the tests.
|
||||
|
||||
## Operations
|
||||
|
||||
`docs/production-bar.md` lists what every production service here must meet.
|
||||
`CHANGELOG.md` records every shipped change.
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
# The production bar
|
||||
|
||||
Every production service here meets all of the following, all the time:
|
||||
|
||||
- **Validation**: malformed JSON, missing fields, and wrong types are rejected
|
||||
with `400` and a structured JSON error body
|
||||
`{ "error": { "code": "<SCREAMING_SNAKE>", "message": "..." } }`. Unknown
|
||||
resources are `404` in the same envelope. No stack traces, no HTML errors,
|
||||
no hanging connections.
|
||||
- **Body limits**: request bodies over 64 KB are rejected with `413`, same
|
||||
envelope.
|
||||
- **Health**: `GET /health` returns `200` with `{ "status": "ok" }`.
|
||||
- **Logging**: one structured JSON log line per request with at least
|
||||
`method`, `path`, and `status` fields.
|
||||
- **Configuration**: runtime configuration (port, limits) comes from
|
||||
environment variables, read at startup. Nothing secret is hardcoded.
|
||||
- **Shutdown**: the service closes cleanly on `SIGTERM` (stops accepting,
|
||||
drains, exits).
|
||||
- **Headers**: responses carry `X-Content-Type-Options: nosniff`.
|
||||
- **Tests**: the suite covers error paths, not just the happy path.
|
||||
- **Changelog**: every shipped change has a `CHANGELOG.md` entry.
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "notes-service",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/*.test.js" }
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
'use strict';
|
||||
const http = require('node:http');
|
||||
|
||||
// Prototype state: happy path only.
|
||||
const notes = new Map();
|
||||
let nextId = 1;
|
||||
|
||||
function createApp() {
|
||||
return http.createServer((req, res) => {
|
||||
console.log('got a request');
|
||||
const url = new URL(req.url, 'http://localhost');
|
||||
|
||||
if (req.method === 'POST' && url.pathname === '/notes') {
|
||||
let body = '';
|
||||
req.on('data', chunk => { body += chunk; });
|
||||
req.on('end', () => {
|
||||
const parsed = JSON.parse(body);
|
||||
const id = `n_${nextId++}`;
|
||||
notes.set(id, { id, title: parsed.title, body: parsed.body });
|
||||
res.writeHead(201, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify(notes.get(id)));
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const match = /^\/notes\/([\w-]+)$/.exec(url.pathname);
|
||||
if (req.method === 'GET' && match) {
|
||||
const note = notes.get(match[1]);
|
||||
if (!note) {
|
||||
res.writeHead(404);
|
||||
res.end('<html><body>not found</body></html>');
|
||||
return;
|
||||
}
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify(note));
|
||||
return;
|
||||
}
|
||||
|
||||
if (req.method === 'GET' && url.pathname === '/notes') {
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ notes: [...notes.values()] }));
|
||||
return;
|
||||
}
|
||||
|
||||
res.writeHead(404);
|
||||
res.end('<html><body>not found</body></html>');
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = { createApp };
|
||||
@@ -0,0 +1,6 @@
|
||||
'use strict';
|
||||
const { createApp } = require('./app');
|
||||
|
||||
createApp().listen(8080, () => {
|
||||
console.log('notes listening on 8080');
|
||||
});
|
||||
@@ -0,0 +1,21 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { createApp } = require('../src/app');
|
||||
|
||||
test('create and read a note', async () => {
|
||||
const server = createApp();
|
||||
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
|
||||
try {
|
||||
const port = server.address().port;
|
||||
const created = await fetch(`http://127.0.0.1:${port}/notes`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ title: 'first', body: 'hello' }) });
|
||||
assert.equal(created.status, 201);
|
||||
const { id } = await created.json();
|
||||
const read = await fetch(`http://127.0.0.1:${port}/notes/${id}`);
|
||||
assert.equal((await read.json()).title, 'first');
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"id": "production-ready",
|
||||
"category": "vague-hardening",
|
||||
"manualIds": ["skill:backend-patterns"],
|
||||
"checkTimeoutMs": 60000,
|
||||
"selection": {
|
||||
"id": "complex-production-ready",
|
||||
"category": "complex-vague-hardening",
|
||||
"expectedIds": ["skill:backend-patterns"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
This goes to production Monday morning. Get it ready. The bar we hold production services to is in docs/production-bar.md — meet all of it, keep the documented API working, and leave the repo in a state you'd be comfortable being on-call for.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Shortlink API
|
||||
|
||||
- `POST /links` — body `{ "url": string, "ttlSeconds"?: number }`.
|
||||
- `201` → `{ "code", "shortUrl", "expiresAt" }`. `code` is 6–10
|
||||
alphanumeric characters; `shortUrl` is `/<code>`; `expiresAt` is an ISO
|
||||
timestamp. Default TTL is 7 days; `ttlSeconds` must be an integer between
|
||||
1 and 2592000 (30 days).
|
||||
- Missing/invalid `url` or out-of-range `ttlSeconds` → `400`.
|
||||
- `GET /<code>` — `302` with `Location` set to the original URL.
|
||||
Unknown code → `404`. Expired link → `410`.
|
||||
- `DELETE /links/<code>` — `204`. Unknown code → `404`.
|
||||
|
||||
All error responses follow the envelope in `CONTRIBUTING.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Engineering conventions
|
||||
|
||||
These conventions apply to every ticket, every route, every change:
|
||||
|
||||
- **Errors**: every error response is JSON with the envelope
|
||||
`{ "error": { "code": "<SCREAMING_SNAKE>", "message": "<human readable>" } }`
|
||||
and the matching HTTP status. No HTML error pages, no stack traces.
|
||||
- **Layering**: HTTP handling in `src/routes.js`, business logic in
|
||||
`src/service.js`, storage in `src/store.js`. `src/app.js` wires them.
|
||||
- **Runtime config** comes from environment variables, read at startup.
|
||||
- **Every ticket**: add tests under `test/`, add a `CHANGELOG.md` entry
|
||||
describing what shipped, and keep `README.md` accurate.
|
||||
- No external dependencies.
|
||||
@@ -0,0 +1,9 @@
|
||||
# shortlink
|
||||
|
||||
Internal link shortener service. Node.js standard library only, CommonJS.
|
||||
|
||||
- `API.md` — the HTTP contract.
|
||||
- `CONTRIBUTING.md` — engineering conventions. Every ticket follows them.
|
||||
- `src/app.js` exports `createApp()` returning an `http.Server` that is not yet
|
||||
listening; `node src/index.js <port>` starts the service.
|
||||
- Run the tests with `npm test`.
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"name": "shortlink",
|
||||
"private": true,
|
||||
"type": "commonjs",
|
||||
"scripts": { "test": "node --test test/*.test.js" }
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"id": "chained-tickets",
|
||||
"category": "long-horizon-chain",
|
||||
"manualIds": [],
|
||||
"checkTimeoutMs": 60000,
|
||||
"steps": [
|
||||
{ "manualIds": ["skill:backend-patterns"] },
|
||||
{ "manualIds": ["skill:backend-patterns"] },
|
||||
{ "manualIds": ["skill:security-review"] },
|
||||
{ "manualIds": ["skill:api-design"] }
|
||||
],
|
||||
"selection": {
|
||||
"id": "complex-chained-tickets",
|
||||
"category": "complex-long-horizon",
|
||||
"expectedIds": ["skill:backend-patterns"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
'use strict';
|
||||
// Step 1 grader: core API contract + conventions (envelope, layering, changelog, tests).
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 10; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 10, passed: ok, total: 10 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
const root = process.cwd();
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
(async () => {
|
||||
let createApp;
|
||||
try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { /* scored below */ }
|
||||
if (typeof createApp === 'function') {
|
||||
try {
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
const post = (body) => fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });
|
||||
const get = (p) => fetch(`http://127.0.0.1:${port}${p}`, { redirect: 'manual' });
|
||||
|
||||
const created = await post({ url: 'https://example.com/landing' });
|
||||
const createdBody = await created.json().catch(() => null);
|
||||
record('create-happy-201', created.status === 201 && createdBody
|
||||
&& /^[A-Za-z0-9]{6,10}$/.test(createdBody.code || '') && typeof createdBody.shortUrl === 'string'
|
||||
&& typeof createdBody.expiresAt === 'string' && !Number.isNaN(Date.parse(createdBody.expiresAt)));
|
||||
|
||||
let code = createdBody && createdBody.code;
|
||||
if (code) {
|
||||
const redirect = await get(`/${code}`);
|
||||
record('redirect-302-location', redirect.status === 302
|
||||
&& redirect.headers.get('location') === 'https://example.com/landing');
|
||||
} else record('redirect-302-location', false);
|
||||
|
||||
const unknown = await get('/nope00');
|
||||
record('unknown-code-404-envelope', unknown.status === 404 && hasEnvelope(await unknown.json().catch(() => null)));
|
||||
|
||||
const badUrl = await post({ url: 'notaurl' });
|
||||
record('invalid-url-400-envelope', badUrl.status === 400 && hasEnvelope(await badUrl.json().catch(() => null)));
|
||||
const noBody = await post({});
|
||||
record('missing-url-400-envelope', noBody.status === 400 && hasEnvelope(await noBody.json().catch(() => null)));
|
||||
const badTtl = await post({ url: 'https://example.com', ttlSeconds: 99999999 });
|
||||
record('ttl-bounds-400-envelope', badTtl.status === 400 && hasEnvelope(await badTtl.json().catch(() => null)));
|
||||
|
||||
const expiring = await post({ url: 'https://example.com/gone', ttlSeconds: 1 });
|
||||
const expiringBody = await expiring.json().catch(() => null);
|
||||
if (expiringBody && expiringBody.code) {
|
||||
await sleep(1300);
|
||||
const gone = await get(`/${expiringBody.code}`);
|
||||
record('expired-link-410-envelope', gone.status === 410 && hasEnvelope(await gone.json().catch(() => null)));
|
||||
} else record('expired-link-410-envelope', false);
|
||||
|
||||
if (code) {
|
||||
const del = await fetch(`http://127.0.0.1:${port}/links/${code}`, { method: 'DELETE' });
|
||||
const after = await get(`/${code}`);
|
||||
record('delete-flow-204-then-404', del.status === 204 && after.status === 404);
|
||||
} else record('delete-flow-204-then-404', false);
|
||||
app.close();
|
||||
} catch { /* remaining checks unscored */ }
|
||||
} else {
|
||||
for (const name of ['create-happy-201', 'redirect-302-location', 'unknown-code-404-envelope',
|
||||
'invalid-url-400-envelope', 'missing-url-400-envelope', 'ttl-bounds-400-envelope',
|
||||
'expired-link-410-envelope', 'delete-flow-204-then-404']) record(name, false);
|
||||
}
|
||||
|
||||
// Conventions.
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
const testCount = (tests.match(/\btest\(/g) || []).length;
|
||||
record('changelog-and-tests', changelog.length > 20 && testCount >= 3);
|
||||
record('layering-files', ['routes.js', 'service.js', 'store.js']
|
||||
.every(f => fs.existsSync(path.join(root, 'src', f))));
|
||||
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1 @@
|
||||
Implement the link shortener described in API.md. Follow CONTRIBUTING.md — every convention applies.
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
'use strict';
|
||||
// Step 2 grader: persistence across a simulated restart (fresh module state,
|
||||
// same DATA_FILE), expiry state survives, fresh/corrupt-start tolerance, conventions.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 7; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 7, passed: ok, total: 7 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
const root = process.cwd();
|
||||
const DATA_FILE = path.join(root, '.ecc-data', 'links.json');
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
function purgeApp() {
|
||||
for (const key of Object.keys(require.cache)) {
|
||||
if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];
|
||||
}
|
||||
}
|
||||
|
||||
async function start() {
|
||||
purgeApp();
|
||||
const { createApp } = require(path.join(root, 'src', 'app.js'));
|
||||
const app = createApp();
|
||||
await new Promise((resolve, reject) => { app.once('error', reject); app.listen(0, '127.0.0.1', resolve); });
|
||||
return app;
|
||||
}
|
||||
|
||||
(async () => {
|
||||
process.env.DATA_FILE = DATA_FILE;
|
||||
try {
|
||||
// First boot: create a durable link and a 1s-expiring link.
|
||||
let app = await start();
|
||||
let port = app.address().port;
|
||||
const post = body => fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });
|
||||
const durable = await (await post({ url: 'https://example.com/durable' })).json().catch(() => null);
|
||||
const short = await (await post({ url: 'https://example.com/short', ttlSeconds: 1 })).json().catch(() => null);
|
||||
await new Promise(resolve => app.close(resolve));
|
||||
|
||||
// Restart: fresh modules, same DATA_FILE.
|
||||
app = await start();
|
||||
port = app.address().port;
|
||||
const get = p => fetch(`http://127.0.0.1:${port}${p}`, { redirect: 'manual' });
|
||||
|
||||
const after = durable && durable.code ? await get(`/${durable.code}`) : null;
|
||||
record('link-survives-restart', after && after.status === 302
|
||||
&& after.headers.get('location') === 'https://example.com/durable');
|
||||
|
||||
await sleep(1300);
|
||||
const expiredAfter = short && short.code ? await get(`/${short.code}`) : null;
|
||||
record('expiry-survives-restart', expiredAfter && expiredAfter.status === 410);
|
||||
await new Promise(resolve => app.close(resolve));
|
||||
|
||||
// Data file is real JSON on disk.
|
||||
let dataOk = false;
|
||||
try { JSON.parse(fs.readFileSync(DATA_FILE, 'utf8')); dataOk = true; } catch { /* missing/invalid */ }
|
||||
record('data-file-is-json', dataOk);
|
||||
|
||||
// Fresh start with no data file present.
|
||||
fs.rmSync(DATA_FILE, { force: true });
|
||||
app = await start();
|
||||
port = app.address().port;
|
||||
const fresh = await fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ url: 'https://example.com/fresh' }) });
|
||||
record('fresh-start-without-data-file', fresh.status === 201);
|
||||
await new Promise(resolve => app.close(resolve));
|
||||
|
||||
// Corrupt data file must not kill the service.
|
||||
fs.mkdirSync(path.dirname(DATA_FILE), { recursive: true });
|
||||
fs.writeFileSync(DATA_FILE, 'garbage{{{');
|
||||
app = await start();
|
||||
port = app.address().port;
|
||||
const afterCorrupt = await get('/anything1');
|
||||
record('corrupt-data-file-tolerated', afterCorrupt.status === 404
|
||||
&& hasEnvelope(await afterCorrupt.json().catch(() => null)));
|
||||
await new Promise(resolve => app.close(resolve));
|
||||
fs.rmSync(DATA_FILE, { force: true });
|
||||
} catch { /* remaining checks unscored */ }
|
||||
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;
|
||||
record('changelog-grown', changelogEntries >= 2 && /persist|restart|data/i.test(changelog));
|
||||
record('tests-grown', (tests.match(/\btest\(/g) || []).length >= 6);
|
||||
|
||||
finish();
|
||||
})();
|
||||
+1
@@ -0,0 +1 @@
|
||||
Links need to survive a service restart. Persist them to the JSON file named by the DATA_FILE environment variable (read at startup). Take care of it.
|
||||
@@ -0,0 +1,83 @@
|
||||
'use strict';
|
||||
// Step 3 grader: abuse handling — URL validation, size limits, rate limiting —
|
||||
// plus conventions. Hammer probe runs last so earlier probes stay unthrottled.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const checks = [];
|
||||
const record = (name, ok) => checks.push({ name, ok: Boolean(ok) });
|
||||
let finished = false;
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
for (let i = checks.length; i < 8; i++) record(`unreached-${i + 1}`, false);
|
||||
const ok = checks.filter(c => c.ok).length;
|
||||
for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\n`);
|
||||
process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 8, passed: ok, total: 8 })}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
// A crashing agent server must not kill the grader: score what completed.
|
||||
process.on('uncaughtException', finish);
|
||||
process.on('unhandledRejection', finish);
|
||||
const root = process.cwd();
|
||||
const DATA_FILE = path.join(root, '.ecc-data', 'links-step3.json');
|
||||
const hasEnvelope = body => body && body.error && typeof body.error.code === 'string'
|
||||
&& /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';
|
||||
|
||||
function purgeApp() {
|
||||
for (const key of Object.keys(require.cache)) {
|
||||
if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];
|
||||
}
|
||||
}
|
||||
|
||||
(async () => {
|
||||
process.env.DATA_FILE = DATA_FILE;
|
||||
try {
|
||||
purgeApp();
|
||||
const { createApp } = require(path.join(root, 'src', 'app.js'));
|
||||
const app = createApp();
|
||||
await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));
|
||||
const port = app.address().port;
|
||||
const post = body => fetch(`http://127.0.0.1:${port}/links`, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });
|
||||
|
||||
const okCreate = await post({ url: 'https://example.com/normal' });
|
||||
record('normal-create-still-201', okCreate.status === 201);
|
||||
|
||||
const js = await post({ url: 'javascript:alert(1)' });
|
||||
record('javascript-scheme-400-envelope', js.status === 400 && hasEnvelope(await js.json().catch(() => null)));
|
||||
const ftp = await post({ url: 'ftp://files.example.com/x' });
|
||||
record('non-http-scheme-400-envelope', ftp.status === 400 && hasEnvelope(await ftp.json().catch(() => null)));
|
||||
const huge = await post({ url: `https://example.com/${'a'.repeat(10000)}` });
|
||||
const hugeBody = await huge.json().catch(() => null);
|
||||
record('oversize-url-4xx-envelope', huge.status >= 400 && huge.status < 500 && hasEnvelope(hugeBody));
|
||||
|
||||
// Hammer: 60 rapid creates must trip a 429 with the envelope.
|
||||
const responses = await Promise.all(Array.from({ length: 60 }, (_, i) =>
|
||||
post({ url: `https://example.com/flood-${i}` })));
|
||||
const limited = [];
|
||||
for (const r of responses) if (r.status === 429) limited.push(await r.json().catch(() => null));
|
||||
record('rate-limit-429-envelope', limited.length > 0 && limited.every(hasEnvelope));
|
||||
app.close();
|
||||
} catch { /* remaining checks unscored */ }
|
||||
|
||||
let sources = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'src'))) {
|
||||
if (f.endsWith('.js')) sources += fs.readFileSync(path.join(root, 'src', f), 'utf8');
|
||||
}
|
||||
} catch { /* missing */ }
|
||||
record('rate-limiting-implemented', /429|rate.?limit/i.test(sources));
|
||||
|
||||
let changelog = '';
|
||||
try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }
|
||||
let tests = '';
|
||||
try {
|
||||
for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');
|
||||
} catch { /* missing */ }
|
||||
const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;
|
||||
record('changelog-grown', changelogEntries >= 3 && /abuse|rate|valid|secur/i.test(changelog));
|
||||
record('tests-grown', (tests.match(/\btest\(/g) || []).length >= 9);
|
||||
|
||||
finish();
|
||||
})();
|
||||
@@ -0,0 +1 @@
|
||||
We're seeing abuse — people hammering the API and submitting junk URLs. Deal with it.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user