WEBDIENSTE/ECC
1
0
Fork 0
mirror of https://github.com/affaan-m/ECC.git synced 2026-09-28 20:45:11 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
e8cba155a8ebd86a0a988ecc95b8da7ef9c08827
ECC/docker/context-profiles/complex-corpus-v4.json
T

168 lines
81 KiB
JSON
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"schemaVersion": "ecc.context-eval-complex-corpus.v1",
"id": "complex-tasks@4",
"sampling": "Realistic multi-file engineering tasks, fixed before any provider call, with deterministic hidden graders scoring partial credit (ECC_EVAL_SCORE). Descriptive pilot: no population-representativeness claim. See complex-eval/DESIGN.md for the preregistered methodology.",
"minimumDistinctTasks": 4,
"nonInferiorityMargin": 0.05,
"selection": [
{
"id": "complex-chained-tickets",
"category": "complex-long-horizon",
"query": "Implement the link shortener described in API.md. Follow CONTRIBUTING.md — every convention applies. Links need to survive a service restart. Persist them to the JSON file named by the DATA_FILE environment variable (read at startup). Take care of it. We're seeing abuse — people hammering the API and submitting junk URLs. Deal with it. Track redirect hits per link and expose them at GET /links/:code/stats, consistent with the existing API.",
"expectedIds": [
"skill:backend-patterns"
]
},
{
"id": "complex-idempotent-webhooks",
"category": "complex-concurrency-trap",
"query": "The payments provider retries aggressively and sometimes delivers the same webhook many times concurrently. Our receiver double-charges customers when that happens — see the incident note in SPEC.md. Rework the receiver so the full contract in SPEC.md holds: duplicates must never double-apply under any interleaving, and the documented API and the store contract stay intact. `npm test` must pass, and add regression coverage for the concurrent-duplicate case so INC-104 can't come back.",
"expectedIds": [
"skill:error-handling"
]
},
{
"id": "complex-production-ready",
"category": "complex-vague-hardening",
"query": "This goes to production Monday morning. Get it ready. The bar we hold production services to is in docs/production-bar.md — meet all of it, keep the documented API working, and leave the repo in a state you'd be comfortable being on-call for.",
"expectedIds": [
"skill:backend-patterns"
]
},
{
"id": "complex-recurring-incident",
"category": "complex-learning-loop",
"query": "Customers are reporting duplicate refunds again — see docs/incidents.md. When a refund request comes in without an idempotency key and the caller retries, we refund the customer twice. Fix it, and make sure this stops being a recurring incident — it's the third time this quarter. Finance just flagged that their payout batch job is about to start retrying on timeouts, and payout retries can double-pay vendors. Same family of problem as the refunds — handle it. One hard requirement: a retried payout must never pay a vendor twice, even if the service restarts between the attempts. You're rolling off this area. Write the handoff note for whoever picks this up next.",
"expectedIds": [
"skill:continuous-learning"
]
}
],
"tasks": [
{
"id": "chained-tickets",
"category": "long-horizon-chain",
"manualIds": [],
"files": {
"API.md": "# Shortlink API\n\n- `POST /links` — body `{ \"url\": string, \"ttlSeconds\"?: number }`.\n - `201` → `{ \"code\", \"shortUrl\", \"expiresAt\" }`. `code` is 6–10\n alphanumeric characters; `shortUrl` is `/<code>`; `expiresAt` is an ISO\n timestamp. Default TTL is 7 days; `ttlSeconds` must be an integer between\n 1 and 2592000 (30 days).\n - Missing/invalid `url` or out-of-range `ttlSeconds` → `400`.\n- `GET /<code>` — `302` with `Location` set to the original URL.\n Unknown code → `404`. Expired link → `410`.\n- `DELETE /links/<code>` — `204`. Unknown code → `404`.\n\nAll error responses follow the envelope in `CONTRIBUTING.md`.\n",
"CONTRIBUTING.md": "# Engineering conventions\n\nThese conventions apply to every ticket, every route, every change:\n\n- **Errors**: every error response is JSON with the envelope\n `{ \"error\": { \"code\": \"<SCREAMING_SNAKE>\", \"message\": \"<human readable>\" } }`\n and the matching HTTP status. No HTML error pages, no stack traces.\n- **Layering**: HTTP handling in `src/routes.js`, business logic in\n `src/service.js`, storage in `src/store.js`. `src/app.js` wires them.\n- **Runtime config** comes from environment variables, read at startup.\n- **Every ticket**: add tests under `test/`, add a `CHANGELOG.md` entry\n describing what shipped, and keep `README.md` accurate.\n- No external dependencies.\n",
"package.json": "{\n \"name\": \"shortlink\",\n \"private\": true,\n \"type\": \"commonjs\",\n \"scripts\": { \"test\": \"node --test test/*.test.js\" }\n}\n",
"README.md": "# shortlink\n\nInternal link shortener service. Node.js standard library only, CommonJS.\n\n- `API.md` — the HTTP contract.\n- `CONTRIBUTING.md` — engineering conventions. Every ticket follows them.\n- `src/app.js` exports `createApp()` returning an `http.Server` that is not yet\n listening; `node src/index.js <port>` starts the service.\n- Run the tests with `npm test`.\n"
},
"steps": [
{
"query": "Implement the link shortener described in API.md. Follow CONTRIBUTING.md — every convention applies.",
"check": "'use strict';\n// Step 1 grader: core API contract + conventions (envelope, layering, changelog, tests).\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 10; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 10, passed: ok, total: 10 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst sleep = ms => new Promise(resolve => setTimeout(resolve, ms));\nconst root = process.cwd();\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\n(async () => {\n let createApp;\n try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { /* scored below */ }\n if (typeof createApp === 'function') {\n try {\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n const post = (body) => fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });\n const get = (p) => fetch(`http://127.0.0.1:${port}${p}`, { redirect: 'manual' });\n\n const created = await post({ url: 'https://example.com/landing' });\n const createdBody = await created.json().catch(() => null);\n record('create-happy-201', created.status === 201 && createdBody\n && /^[A-Za-z0-9]{6,10}$/.test(createdBody.code || '') && typeof createdBody.shortUrl === 'string'\n && typeof createdBody.expiresAt === 'string' && !Number.isNaN(Date.parse(createdBody.expiresAt)));\n\n let code = createdBody && createdBody.code;\n if (code) {\n const redirect = await get(`/${code}`);\n record('redirect-302-location', redirect.status === 302\n && redirect.headers.get('location') === 'https://example.com/landing');\n } else record('redirect-302-location', false);\n\n const unknown = await get('/nope00');\n record('unknown-code-404-envelope', unknown.status === 404 && hasEnvelope(await unknown.json().catch(() => null)));\n\n const badUrl = await post({ url: 'notaurl' });\n record('invalid-url-400-envelope', badUrl.status === 400 && hasEnvelope(await badUrl.json().catch(() => null)));\n const noBody = await post({});\n record('missing-url-400-envelope', noBody.status === 400 && hasEnvelope(await noBody.json().catch(() => null)));\n const badTtl = await post({ url: 'https://example.com', ttlSeconds: 99999999 });\n record('ttl-bounds-400-envelope', badTtl.status === 400 && hasEnvelope(await badTtl.json().catch(() => null)));\n\n const expiring = await post({ url: 'https://example.com/gone', ttlSeconds: 1 });\n const expiringBody = await expiring.json().catch(() => null);\n if (expiringBody && expiringBody.code) {\n await sleep(1300);\n const gone = await get(`/${expiringBody.code}`);\n record('expired-link-410-envelope', gone.status === 410 && hasEnvelope(await gone.json().catch(() => null)));\n } else record('expired-link-410-envelope', false);\n\n if (code) {\n const del = await fetch(`http://127.0.0.1:${port}/links/${code}`, { method: 'DELETE' });\n const after = await get(`/${code}`);\n record('delete-flow-204-then-404', del.status === 204 && after.status === 404);\n } else record('delete-flow-204-then-404', false);\n app.close();\n } catch { /* remaining checks unscored */ }\n } else {\n for (const name of ['create-happy-201', 'redirect-302-location', 'unknown-code-404-envelope',\n 'invalid-url-400-envelope', 'missing-url-400-envelope', 'ttl-bounds-400-envelope',\n 'expired-link-410-envelope', 'delete-flow-204-then-404']) record(name, false);\n }\n\n // Conventions.\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const testCount = (tests.match(/\\btest\\(/g) || []).length;\n record('changelog-and-tests', changelog.length > 20 && testCount >= 3);\n record('layering-files', ['routes.js', 'service.js', 'store.js']\n .every(f => fs.existsSync(path.join(root, 'src', f))));\n\n finish();\n})();\n",
"manualIds": [
"skill:backend-patterns"
],
"checkTimeoutMs": 60000
},
{
"query": "Links need to survive a service restart. Persist them to the JSON file named by the DATA_FILE environment variable (read at startup). Take care of it.",
"check": "'use strict';\n// Step 2 grader: persistence across a simulated restart (fresh module state,\n// same DATA_FILE), expiry state survives, fresh/corrupt-start tolerance, conventions.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 7; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 7, passed: ok, total: 7 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst sleep = ms => new Promise(resolve => setTimeout(resolve, ms));\nconst root = process.cwd();\nconst DATA_FILE = path.join(root, '.ecc-data', 'links.json');\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\nfunction purgeApp() {\n for (const key of Object.keys(require.cache)) {\n if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];\n }\n}\n\nasync function start() {\n purgeApp();\n const { createApp } = require(path.join(root, 'src', 'app.js'));\n const app = createApp();\n await new Promise((resolve, reject) => { app.once('error', reject); app.listen(0, '127.0.0.1', resolve); });\n return app;\n}\n\n(async () => {\n process.env.DATA_FILE = DATA_FILE;\n try {\n // First boot: create a durable link and a 1s-expiring link.\n let app = await start();\n let port = app.address().port;\n const post = body => fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });\n const durable = await (await post({ url: 'https://example.com/durable' })).json().catch(() => null);\n const short = await (await post({ url: 'https://example.com/short', ttlSeconds: 1 })).json().catch(() => null);\n await new Promise(resolve => app.close(resolve));\n\n // Restart: fresh modules, same DATA_FILE.\n app = await start();\n port = app.address().port;\n const get = p => fetch(`http://127.0.0.1:${port}${p}`, { redirect: 'manual' });\n\n const after = durable && durable.code ? await get(`/${durable.code}`) : null;\n record('link-survives-restart', after && after.status === 302\n && after.headers.get('location') === 'https://example.com/durable');\n\n await sleep(1300);\n const expiredAfter = short && short.code ? await get(`/${short.code}`) : null;\n record('expiry-survives-restart', expiredAfter && expiredAfter.status === 410);\n await new Promise(resolve => app.close(resolve));\n\n // Data file is real JSON on disk.\n let dataOk = false;\n try { JSON.parse(fs.readFileSync(DATA_FILE, 'utf8')); dataOk = true; } catch { /* missing/invalid */ }\n record('data-file-is-json', dataOk);\n\n // Fresh start with no data file present.\n fs.rmSync(DATA_FILE, { force: true });\n app = await start();\n port = app.address().port;\n const fresh = await fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ url: 'https://example.com/fresh' }) });\n record('fresh-start-without-data-file', fresh.status === 201);\n await new Promise(resolve => app.close(resolve));\n\n // Corrupt data file must not kill the service.\n fs.mkdirSync(path.dirname(DATA_FILE), { recursive: true });\n fs.writeFileSync(DATA_FILE, 'garbage{{{');\n app = await start();\n port = app.address().port;\n const afterCorrupt = await get('/anything1');\n record('corrupt-data-file-tolerated', afterCorrupt.status === 404\n && hasEnvelope(await afterCorrupt.json().catch(() => null)));\n await new Promise(resolve => app.close(resolve));\n fs.rmSync(DATA_FILE, { force: true });\n } catch { /* remaining checks unscored */ }\n\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;\n record('changelog-grown', changelogEntries >= 2 && /persist|restart|data/i.test(changelog));\n record('tests-grown', (tests.match(/\\btest\\(/g) || []).length >= 6);\n\n finish();\n})();\n",
"manualIds": [
"skill:backend-patterns"
],
"checkTimeoutMs": 60000
},
{
"query": "We're seeing abuse — people hammering the API and submitting junk URLs. Deal with it.",
"check": "'use strict';\n// Step 3 grader: abuse handling — URL validation, size limits, rate limiting —\n// plus conventions. Hammer probe runs last so earlier probes stay unthrottled.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 8; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 8, passed: ok, total: 8 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst root = process.cwd();\nconst DATA_FILE = path.join(root, '.ecc-data', 'links-step3.json');\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\nfunction purgeApp() {\n for (const key of Object.keys(require.cache)) {\n if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];\n }\n}\n\n(async () => {\n process.env.DATA_FILE = DATA_FILE;\n try {\n purgeApp();\n const { createApp } = require(path.join(root, 'src', 'app.js'));\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n const post = body => fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) });\n\n const okCreate = await post({ url: 'https://example.com/normal' });\n record('normal-create-still-201', okCreate.status === 201);\n\n const js = await post({ url: 'javascript:alert(1)' });\n record('javascript-scheme-400-envelope', js.status === 400 && hasEnvelope(await js.json().catch(() => null)));\n const ftp = await post({ url: 'ftp://files.example.com/x' });\n record('non-http-scheme-400-envelope', ftp.status === 400 && hasEnvelope(await ftp.json().catch(() => null)));\n const huge = await post({ url: `https://example.com/${'a'.repeat(10000)}` });\n const hugeBody = await huge.json().catch(() => null);\n record('oversize-url-4xx-envelope', huge.status >= 400 && huge.status < 500 && hasEnvelope(hugeBody));\n\n // Hammer: 60 rapid creates must trip a 429 with the envelope.\n const responses = await Promise.all(Array.from({ length: 60 }, (_, i) =>\n post({ url: `https://example.com/flood-${i}` })));\n const limited = [];\n for (const r of responses) if (r.status === 429) limited.push(await r.json().catch(() => null));\n record('rate-limit-429-envelope', limited.length > 0 && limited.every(hasEnvelope));\n app.close();\n } catch { /* remaining checks unscored */ }\n\n let sources = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'src'))) {\n if (f.endsWith('.js')) sources += fs.readFileSync(path.join(root, 'src', f), 'utf8');\n }\n } catch { /* missing */ }\n record('rate-limiting-implemented', /429|rate.?limit/i.test(sources));\n\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;\n record('changelog-grown', changelogEntries >= 3 && /abuse|rate|valid|secur/i.test(changelog));\n record('tests-grown', (tests.match(/\\btest\\(/g) || []).length >= 9);\n\n finish();\n})();\n",
"manualIds": [
"skill:security-review"
],
"checkTimeoutMs": 60000
},
{
"query": "Track redirect hits per link and expose them at GET /links/:code/stats, consistent with the existing API.",
"check": "'use strict';\n// Step 4 grader: hit analytics consistent with the existing API, conventions,\n// docs and tests. (Runs in a later process than step 3, so rate windows cleared.)\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 8; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 8, passed: ok, total: 8 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst root = process.cwd();\nconst DATA_FILE = path.join(root, '.ecc-data', 'links-step4.json');\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\nfunction purgeApp() {\n for (const key of Object.keys(require.cache)) {\n if (key.startsWith(path.join(root, 'src') + path.sep)) delete require.cache[key];\n }\n}\n\n(async () => {\n process.env.DATA_FILE = DATA_FILE;\n try {\n purgeApp();\n const { createApp } = require(path.join(root, 'src', 'app.js'));\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n\n const created = await fetch(`http://127.0.0.1:${port}/links`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ url: 'https://example.com/tracked' }) });\n const body = await created.json().catch(() => null);\n const code = body && body.code;\n record('create-still-works', created.status === 201 && Boolean(code));\n\n if (code) {\n const before = await fetch(`http://127.0.0.1:${port}/links/${code}/stats`);\n const beforeBody = await before.json().catch(() => null);\n record('stats-zero-before-redirects', before.status === 200 && beforeBody && beforeBody.hits === 0);\n\n for (let i = 0; i < 3; i++) {\n await fetch(`http://127.0.0.1:${port}/${code}`, { redirect: 'manual' });\n }\n const stats = await fetch(`http://127.0.0.1:${port}/links/${code}/stats`);\n const statsBody = await stats.json().catch(() => null);\n record('stats-count-three-hits', stats.status === 200 && statsBody && statsBody.hits === 3);\n\n const redirect = await fetch(`http://127.0.0.1:${port}/${code}`, { redirect: 'manual' });\n record('redirect-still-302', redirect.status === 302);\n\n const missing = await fetch(`http://127.0.0.1:${port}/links/zzzzzz/stats`);\n record('stats-unknown-404-envelope', missing.status === 404\n && hasEnvelope(await missing.json().catch(() => null)));\n } else {\n for (const name of ['stats-zero-before-redirects', 'stats-count-three-hits',\n 'redirect-still-302', 'stats-unknown-404-envelope']) record(name, false);\n }\n app.close();\n } catch { /* remaining checks unscored */ }\n\n let readme = '';\n try { readme = fs.readFileSync(path.join(root, 'README.md'), 'utf8'); } catch { /* missing */ }\n record('readme-documents-stats', /\\/stats|hits|analytics/i.test(readme));\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const changelogEntries = (changelog.match(/^[-*#]/gm) || []).length;\n record('changelog-grown', changelogEntries >= 4 && /stat|analytic|hit/i.test(changelog));\n record('tests-grown', (tests.match(/\\btest\\(/g) || []).length >= 12);\n\n finish();\n})();\n",
"manualIds": [
"skill:api-design"
],
"checkTimeoutMs": 60000
}
]
},
{
"id": "idempotent-webhooks",
"category": "concurrency-trap",
"manualIds": [
"skill:error-handling"
],
"checkTimeoutMs": 60000,
"query": "The payments provider retries aggressively and sometimes delivers the same webhook many times concurrently. Our receiver double-charges customers when that happens — see the incident note in SPEC.md. Rework the receiver so the full contract in SPEC.md holds: duplicates must never double-apply under any interleaving, and the documented API and the store contract stay intact. `npm test` must pass, and add regression coverage for the concurrent-duplicate case so INC-104 can't come back.",
"files": {
"package.json": "{\n \"name\": \"webhook-receiver\",\n \"private\": true,\n \"type\": \"commonjs\",\n \"scripts\": { \"test\": \"node --test test/*.test.js\" }\n}\n",
"README.md": "# webhook-receiver\n\nReceives payment webhooks. There is an open incident: customers were\ndouble-charged when the provider retried deliveries. See `SPEC.md` for the\ncontract, including the exactly-once rules.\n\n- `src/app.js` exports `createApp()` returning an `http.Server` that is not\n yet listening; `node src/index.js <port>` starts the service.\n- `src/store.js` is shared infrastructure: it keeps its current exports\n (`store`) and records every applied payment in `store.paymentLog`.\n- No external dependencies. `npm test` runs the tests. `CHANGELOG.md` records\n every shipped change.\n",
"SPEC.md": "# Payment webhook contract\n\n`POST /webhooks/payments` with JSON body\n`{ \"eventId\": string, \"orderId\": string, \"amountCents\": number, \"type\": \"payment.succeeded\" }`.\n\nExactly-once is the point. The provider retries aggressively and may deliver\nthe same event many times, concurrently, or out of order.\n\n- A new, valid `eventId`: apply the payment exactly once → `200`\n `{ \"status\": \"processed\", \"orderId\" }`.\n- The same `eventId` seen again (any number of times, any interleaving):\n `200` `{ \"status\": \"duplicate\", \"orderId\" }` — never applied twice.\n- A payment event (new `eventId`) for an order that is already paid:\n `200` `{ \"status\": \"already_paid\", \"orderId\" }` — an order is paid at most\n once, ever.\n- `amountCents` not matching the order's amount: `422`, not applied.\n- Unknown `orderId`: `404`. Malformed body (bad JSON, missing/invalid\n fields): `400`.\n- Error responses use the envelope\n `{ \"error\": { \"code\": \"<SCREAMING_SNAKE>\", \"message\": \"...\" } }`.\n\n`GET /orders/:id` → `200` `{ \"id\", \"status\", \"paidAt\", \"paymentsApplied\" }`\nor a `404` envelope.\n\n## Incident note\n\nINC-104: concurrent duplicate deliveries double-applied payments. The naive\nreceiver checked \"have we seen this event?\" and applied the payment in two\nseparate steps with an async gap in between, so parallel duplicates both\npassed the check.\n",
"src/app.js": "'use strict';\nconst http = require('node:http');\nconst { store } = require('./store');\n\n// INC-104 receiver: checks \"seen this event?\" and applies the payment in two\n// steps with an async gap in between. Concurrent duplicates both pass the\n// check. Do not keep this shape.\nfunction createApp() {\n return http.createServer((req, res) => {\n const url = new URL(req.url, 'http://localhost');\n\n if (req.method === 'POST' && url.pathname === '/webhooks/payments') {\n let body = '';\n req.on('data', chunk => { body += chunk; });\n req.on('end', async () => {\n const parsed = JSON.parse(body);\n const { eventId, orderId } = parsed;\n if (store.processedEvents.has(eventId)) {\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ status: 'duplicate', orderId }));\n return;\n }\n await new Promise(resolve => setImmediate(resolve)); // async gap\n const order = store.orders.get(orderId);\n order.status = 'paid';\n order.paidAt = new Date().toISOString();\n order.paymentsApplied++;\n store.paymentLog.push({ eventId, orderId, amountCents: parsed.amountCents });\n store.processedEvents.add(eventId);\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ status: 'processed', orderId }));\n });\n return;\n }\n\n const match = /^\\/orders\\/([\\w-]+)$/.exec(url.pathname);\n if (req.method === 'GET' && match) {\n const order = store.orders.get(match[1]);\n if (!order) {\n res.writeHead(404, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ error: { code: 'NOT_FOUND', message: 'no such order' } }));\n return;\n }\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify(order));\n return;\n }\n\n res.writeHead(404, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ error: { code: 'NOT_FOUND', message: 'not found' } }));\n });\n}\n\nmodule.exports = { createApp };\n",
"src/index.js": "'use strict';\nconst { createApp } = require('./app');\n\nconst port = Number(process.argv[2] || 8080);\ncreateApp().listen(port, () => {\n console.log(`webhook-receiver listening on ${port}`);\n});\n",
"src/store.js": "'use strict';\n\n// Shared infrastructure. Every applied payment is appended to paymentLog;\n// orders and processedEvents track receiver state. Keep the `store` export.\nconst store = {\n orders: new Map([\n ['o1', { id: 'o1', amountCents: 5000, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o2', { id: 'o2', amountCents: 12500, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o3', { id: 'o3', amountCents: 800, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o4', { id: 'o4', amountCents: 9999, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o5', { id: 'o5', amountCents: 250, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ['o6', { id: 'o6', amountCents: 7300, status: 'pending', paidAt: null, paymentsApplied: 0 }],\n ]),\n paymentLog: [],\n processedEvents: new Set(),\n};\n\nmodule.exports = { store };\n",
"test/webhooks.test.js": "'use strict';\nconst test = require('node:test');\nconst assert = require('node:assert/strict');\nconst { createApp } = require('../src/app');\nconst { store } = require('../src/store');\n\ntest('a single payment event processes', async () => {\n const server = createApp();\n await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));\n try {\n const port = server.address().port;\n const res = await fetch(`http://127.0.0.1:${port}/webhooks/payments`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ eventId: 'ev-test-1', orderId: 'o1', amountCents: 5000, type: 'payment.succeeded' }) });\n assert.equal(res.status, 200);\n assert.equal((await res.json()).status, 'processed');\n assert.equal(store.orders.get('o1').status, 'paid');\n } finally {\n server.close();\n }\n});\n"
},
"check": "'use strict';\n// Hidden grader for idempotent-webhooks: exactly-once under sequential,\n// concurrent, and mixed-concurrent duplicates, plus the documented API,\n// regression coverage, and hygiene. Prints ECC_EVAL_SCORE and always exits 0.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 12; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 12, passed: ok, total: 12 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst root = process.cwd();\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\n(async () => {\n let createApp;\n let store;\n try {\n ({ createApp } = require(path.join(root, 'src', 'app.js')));\n ({ store } = require(path.join(root, 'src', 'store.js')));\n } catch { /* scored below */ }\n if (typeof createApp === 'function' && store && Array.isArray(store.paymentLog)) {\n try {\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n const send = (eventId, orderId, amountCents) => fetch(`http://127.0.0.1:${port}/webhooks/payments`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ eventId, orderId, amountCents, type: 'payment.succeeded' }) });\n const logsFor = orderId => store.paymentLog.filter(p => p.orderId === orderId).length;\n\n // 1: single delivery applies once.\n const single = await send('ev-1', 'o1', 5000);\n const singleBody = await single.json().catch(() => null);\n record('single-delivery-processed', single.status === 200 && singleBody\n && singleBody.status === 'processed' && singleBody.orderId === 'o1' && logsFor('o1') === 1);\n\n // 2: sequential retry replays without re-applying.\n const retry = await send('ev-1', 'o1', 5000);\n const retryBody = await retry.json().catch(() => null);\n record('sequential-duplicate-inert', retry.status === 200 && retryBody\n && retryBody.status === 'duplicate' && logsFor('o1') === 1);\n\n // 3: fifty concurrent identical deliveries apply exactly once.\n const storm = await Promise.all(Array.from({ length: 50 }, () => send('ev-2', 'o2', 12500)));\n const stormBodies = [];\n for (const r of storm) stormBodies.push(await r.json().catch(() => null));\n const processedCount = stormBodies.filter(b => b && b.status === 'processed').length;\n const duplicateCount = stormBodies.filter(b => b && b.status === 'duplicate').length;\n record('concurrent-storm-exactly-once', storm.every(r => r.status === 200)\n && processedCount === 1 && duplicateCount === 49 && logsFor('o2') === 1\n && store.orders.get('o2').paymentsApplied === 1);\n\n // 4: a different event for an already-paid order is already_paid and inert.\n const second = await send('ev-3', 'o2', 12500);\n const secondBody = await second.json().catch(() => null);\n record('already-paid-order-inert', second.status === 200 && secondBody\n && secondBody.status === 'already_paid' && logsFor('o2') === 1);\n\n // 5-7: contract errors with envelopes.\n const unknown = await send('ev-4', 'nope', 100);\n record('unknown-order-404-envelope', unknown.status === 404 && hasEnvelope(await unknown.json().catch(() => null)));\n const malformed = await fetch(`http://127.0.0.1:${port}/webhooks/payments`, {\n method: 'POST', headers: { 'content-type': 'application/json' }, body: '{bad json' });\n record('malformed-body-400-envelope', malformed.status === 400 && hasEnvelope(await malformed.json().catch(() => null)));\n const mismatch = await send('ev-5', 'o3', 999999);\n record('amount-mismatch-422-envelope', mismatch.status === 422\n && hasEnvelope(await mismatch.json().catch(() => null)) && logsFor('o3') === 0);\n\n // 8: mixed storm — three orders, three eventIds, ten duplicates each, all concurrent.\n const mixed = await Promise.all(['o4', 'o5', 'o6'].flatMap(orderId =>\n Array.from({ length: 10 }, () => send(`ev-${orderId}`, orderId, store.orders.get(orderId).amountCents))));\n for (const r of mixed) await r.json().catch(() => null);\n record('mixed-storm-each-order-once', ['o4', 'o5', 'o6'].every(orderId =>\n logsFor(orderId) === 1 && store.orders.get(orderId).paymentsApplied === 1));\n\n // 9: order inspection endpoint reflects reality.\n const orderView = await fetch(`http://127.0.0.1:${port}/orders/o2`);\n const orderBody = await orderView.json().catch(() => null);\n record('order-endpoint-accurate', orderView.status === 200 && orderBody\n && orderBody.status === 'paid' && orderBody.paymentsApplied === 1 && Boolean(orderBody.paidAt));\n\n app.close();\n } catch { /* remaining checks unscored */ }\n } else {\n for (const name of ['single-delivery-processed', 'sequential-duplicate-inert', 'concurrent-storm-exactly-once',\n 'already-paid-order-inert', 'unknown-order-404-envelope', 'malformed-body-400-envelope',\n 'amount-mismatch-422-envelope', 'mixed-storm-each-order-once', 'order-endpoint-accurate']) record(name, false);\n }\n\n // Conventions.\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n record('concurrency-regression-tests', (tests.match(/\\btest\\(/g) || []).length >= 4\n && /Promise\\.all|concurrent|duplicate|retry/i.test(tests));\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n record('changelog-entry', /idem|duplicat|retry|inc-104|race/i.test(changelog));\n try {\n const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));\n record('no-external-dependencies', !pkg.dependencies && !pkg.devDependencies);\n } catch { record('no-external-dependencies', false); }\n\n finish();\n})();\n"
},
{
"id": "production-ready",
"category": "vague-hardening",
"manualIds": [
"skill:backend-patterns"
],
"checkTimeoutMs": 60000,
"query": "This goes to production Monday morning. Get it ready. The bar we hold production services to is in docs/production-bar.md — meet all of it, keep the documented API working, and leave the repo in a state you'd be comfortable being on-call for.",
"files": {
"docs/production-bar.md": "# The production bar\n\nEvery production service here meets all of the following, all the time:\n\n- **Validation**: malformed JSON, missing fields, and wrong types are rejected\n with `400` and a structured JSON error body\n `{ \"error\": { \"code\": \"<SCREAMING_SNAKE>\", \"message\": \"...\" } }`. Unknown\n resources are `404` in the same envelope. No stack traces, no HTML errors,\n no hanging connections.\n- **Body limits**: request bodies over 64 KB are rejected with `413`, same\n envelope.\n- **Health**: `GET /health` returns `200` with `{ \"status\": \"ok\" }`.\n- **Logging**: one structured JSON log line per request with at least\n `method`, `path`, and `status` fields.\n- **Configuration**: runtime configuration (port, limits) comes from\n environment variables, read at startup. Nothing secret is hardcoded.\n- **Shutdown**: the service closes cleanly on `SIGTERM` (stops accepting,\n drains, exits).\n- **Headers**: responses carry `X-Content-Type-Options: nosniff`.\n- **Tests**: the suite covers error paths, not just the happy path.\n- **Changelog**: every shipped change has a `CHANGELOG.md` entry.\n",
"package.json": "{\n \"name\": \"notes-service\",\n \"private\": true,\n \"type\": \"commonjs\",\n \"scripts\": { \"test\": \"node --test test/*.test.js\" }\n}\n",
"README.md": "# notes-service\n\nTiny notes API. Hobby prototype state: it works on the happy path and that's\nabout all that can be said for it.\n\n## API\n\n- `POST /notes` — body `{ \"title\": string, \"body\": string }` → `201` with\n `{ \"id\", \"title\", \"body\" }`.\n- `GET /notes/:id` — `200` with the note, or `404`.\n- `GET /notes` — `200` with `{ \"notes\": [...] }`.\n\n`src/app.js` exports `createApp()` returning an `http.Server` that is not yet\nlistening; `node src/index.js` starts the service. `npm test` runs the tests.\n\n## Operations\n\n`docs/production-bar.md` lists what every production service here must meet.\n`CHANGELOG.md` records every shipped change.\n",
"src/app.js": "'use strict';\nconst http = require('node:http');\n\n// Prototype state: happy path only.\nconst notes = new Map();\nlet nextId = 1;\n\nfunction createApp() {\n return http.createServer((req, res) => {\n console.log('got a request');\n const url = new URL(req.url, 'http://localhost');\n\n if (req.method === 'POST' && url.pathname === '/notes') {\n let body = '';\n req.on('data', chunk => { body += chunk; });\n req.on('end', () => {\n const parsed = JSON.parse(body);\n const id = `n_${nextId++}`;\n notes.set(id, { id, title: parsed.title, body: parsed.body });\n res.writeHead(201, { 'content-type': 'application/json' });\n res.end(JSON.stringify(notes.get(id)));\n });\n return;\n }\n\n const match = /^\\/notes\\/([\\w-]+)$/.exec(url.pathname);\n if (req.method === 'GET' && match) {\n const note = notes.get(match[1]);\n if (!note) {\n res.writeHead(404);\n res.end('<html><body>not found</body></html>');\n return;\n }\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify(note));\n return;\n }\n\n if (req.method === 'GET' && url.pathname === '/notes') {\n res.writeHead(200, { 'content-type': 'application/json' });\n res.end(JSON.stringify({ notes: [...notes.values()] }));\n return;\n }\n\n res.writeHead(404);\n res.end('<html><body>not found</body></html>');\n });\n}\n\nmodule.exports = { createApp };\n",
"src/index.js": "'use strict';\nconst { createApp } = require('./app');\n\ncreateApp().listen(8080, () => {\n console.log('notes listening on 8080');\n});\n",
"test/notes.test.js": "'use strict';\nconst test = require('node:test');\nconst assert = require('node:assert/strict');\nconst { createApp } = require('../src/app');\n\ntest('create and read a note', async () => {\n const server = createApp();\n await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));\n try {\n const port = server.address().port;\n const created = await fetch(`http://127.0.0.1:${port}/notes`, {\n method: 'POST', headers: { 'content-type': 'application/json' },\n body: JSON.stringify({ title: 'first', body: 'hello' }) });\n assert.equal(created.status, 201);\n const { id } = await created.json();\n const read = await fetch(`http://127.0.0.1:${port}/notes/${id}`);\n assert.equal((await read.json()).title, 'first');\n } finally {\n server.close();\n }\n});\n"
},
"check": "'use strict';\n// Hidden grader for production-ready: probes every dimension of the documented\n// production bar. Prints ECC_EVAL_SCORE and always exits 0.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst checks = [];\nconst record = (name, ok) => checks.push({ name, ok: Boolean(ok) });\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n for (let i = checks.length; i < 16; i++) record(`unreached-${i + 1}`, false);\n const ok = checks.filter(c => c.ok).length;\n for (const c of checks) process.stdout.write(`${c.ok ? 'ok' : 'not ok'} - ${c.name}\\n`);\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: ok / 16, passed: ok, total: 16 })}\\n`);\n process.exit(0);\n}\n// A crashing agent server must not kill the grader: score what completed.\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\nconst root = process.cwd();\nconst hasEnvelope = body => body && body.error && typeof body.error.code === 'string'\n && /^[A-Z][A-Z0-9_]+$/.test(body.error.code) && typeof body.error.message === 'string';\n\n(async () => {\n let createApp;\n try { ({ createApp } = require(path.join(root, 'src', 'app.js'))); } catch { /* scored below */ }\n if (typeof createApp === 'function') {\n // Capture console output during the probe run to inspect request logging.\n const logged = [];\n const originalLog = console.log;\n const originalError = console.error;\n const originalStdoutWrite = process.stdout.write.bind(process.stdout);\n const originalStderrWrite = process.stderr.write.bind(process.stderr);\n console.log = (...args) => { logged.push(args.join(' ')); };\n console.error = (...args) => { logged.push(args.join(' ')); };\n // Agents may log through an injectable writer straight to the streams\n // instead of console.*. Capture-then-pass-through: the bytes always reach\n // the stream untouched, so the grader's own ECC_EVAL_SCORE line (emitted\n // via process.stdout.write) can never be swallowed or corrupted.\n const tap = write => (chunk, encoding, callback) => {\n try { logged.push(Buffer.isBuffer(chunk) ? chunk.toString('utf8') : String(chunk)); } catch { /* capture must never break a write */ }\n return write(chunk, encoding, callback);\n };\n process.stdout.write = tap(originalStdoutWrite);\n process.stderr.write = tap(originalStderrWrite);\n try {\n const app = createApp();\n await new Promise(resolve => app.listen(0, '127.0.0.1', resolve));\n const port = app.address().port;\n const api = (p, options) => fetch(`http://127.0.0.1:${port}${p}`, options);\n const post = body => api('/notes', { method: 'POST', headers: { 'content-type': 'application/json' }, body });\n\n // Documented API still works.\n const created = await post(JSON.stringify({ title: 'deploy', body: 'checklist' }));\n const createdBody = await created.json().catch(() => null);\n record('api-roundtrip-preserved', created.status === 201 && createdBody && createdBody.id\n && (await (await api(`/notes/${createdBody.id}`)).json().catch(() => ({}))).title === 'deploy'\n && Array.isArray((await (await api('/notes')).json().catch(() => ({}))).notes));\n\n // Validation and envelope discipline.\n const badJson = await post('{not json');\n record('malformed-json-400-envelope', badJson.status === 400 && hasEnvelope(await badJson.json().catch(() => null)));\n const missing = await post(JSON.stringify({ body: 'no title' }));\n record('missing-field-400-envelope', missing.status === 400 && hasEnvelope(await missing.json().catch(() => null)));\n const wrongType = await post(JSON.stringify({ title: 42, body: 'x' }));\n record('wrong-type-400-envelope', wrongType.status === 400 && hasEnvelope(await wrongType.json().catch(() => null)));\n const unknown = await api('/notes/n_999999');\n const unknownBody = await unknown.text();\n let unknownParsed = null;\n try { unknownParsed = JSON.parse(unknownBody); } catch { /* html or text */ }\n record('unknown-404-json-envelope', unknown.status === 404 && hasEnvelope(unknownParsed));\n\n // Body limit.\n const big = await post(JSON.stringify({ title: 'big', body: 'x'.repeat(100 * 1024) }));\n record('oversize-body-413-envelope', big.status === 413 && hasEnvelope(await big.json().catch(() => null)));\n\n // Health endpoint.\n const health = await api('/health');\n const healthBody = await health.json().catch(() => null);\n record('health-endpoint', health.status === 200 && healthBody && healthBody.status === 'ok');\n\n // Security header on a normal response.\n const headers = await api('/notes');\n record('nosniff-header', headers.headers.get('x-content-type-options') === 'nosniff');\n\n // Error responses carry JSON content type.\n record('errors-are-json', /application\\/json/.test(unknown.headers.get('content-type') || ''));\n\n app.close();\n } catch { /* remaining checks unscored */ } finally {\n console.log = originalLog;\n console.error = originalError;\n process.stdout.write = originalStdoutWrite;\n process.stderr.write = originalStderrWrite;\n }\n\n // Structured request logging: at least one JSON line with method/path/status-ish fields.\n const structured = logged.flatMap(chunk => String(chunk).split('\\n')).some(line => {\n try {\n const parsed = JSON.parse(line);\n return parsed && typeof parsed === 'object'\n && /method/i.test(Object.keys(parsed).join(' '))\n && /path|url/i.test(Object.keys(parsed).join(' '))\n && /status/i.test(Object.keys(parsed).join(' '));\n } catch { return false; }\n });\n record('structured-request-logs', structured);\n } else {\n for (const name of ['api-roundtrip-preserved', 'malformed-json-400-envelope', 'missing-field-400-envelope',\n 'wrong-type-400-envelope', 'unknown-404-json-envelope', 'oversize-body-413-envelope', 'health-endpoint',\n 'nosniff-header', 'errors-are-json', 'structured-request-logs']) record(name, false);\n }\n\n // Static dimensions.\n let sources = '';\n const sourceFiles = [];\n const walk = directory => {\n for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {\n const item = path.join(directory, entry.name);\n if (entry.isDirectory()) walk(item);\n else if (entry.name.endsWith('.js')) {\n const content = fs.readFileSync(item, 'utf8');\n sourceFiles.push(content);\n sources += content;\n }\n }\n };\n try { walk(path.join(root, 'src')); } catch { /* none */ }\n record('sigterm-graceful-shutdown', /SIGTERM/.test(sources));\n // Literal process.env.PORT access, or an injectable-config indirection: a\n // 'PORT' string literal in a file that also reads process.env (for example a\n // loadConfig(env = process.env) + readInt(env, 'PORT', default) module).\n record('env-config-port', sourceFiles.some(content => /process\\.env\\.[A-Z_]*PORT/.test(content)\n || (/(['\"`])PORT\\1/.test(content) && /process\\.env/.test(content))));\n\n let tests = '';\n try {\n for (const f of fs.readdirSync(path.join(root, 'test'))) tests += fs.readFileSync(path.join(root, 'test', f), 'utf8');\n } catch { /* missing */ }\n const testCount = (tests.match(/\\btest\\(/g) || []).length;\n record('tests-cover-error-paths', testCount >= 4 && /400|404|413|invalid|error/i.test(tests));\n\n let changelog = '';\n try { changelog = fs.readFileSync(path.join(root, 'CHANGELOG.md'), 'utf8'); } catch { /* missing */ }\n record('changelog-entry', changelog.length > 20 && /product|harden|valid|health|log/i.test(changelog));\n\n record('no-leftover-todos', !/TODO|FIXME/.test(sources));\n try {\n const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));\n record('no-external-dependencies', !pkg.dependencies && !pkg.devDependencies);\n } catch { record('no-external-dependencies', false); }\n\n finish();\n})();\n"
},
{
"id": "recurring-incident",
"category": "learning-loop-chain",
"manualIds": [],
"files": {
"docs/incidents.md": "# Incident notes\n\n## INC-201 — duplicate refunds (2026-06-14)\n\nCustomers saw two refunds for one order. Traced to the storefront retrying the\nrefund call after a gateway timeout. Asked the storefront team to retry less\naggressively. Closed.\n\n## INC-214 — duplicate refunds, again (2026-07-29)\n\nSame shape as INC-201: a retried refund call landed twice. Reminded the\nstorefront team about backoff. Closed.\n\n## INC-227 — duplicate refunds, third time (2026-09-03)\n\nSame shape as INC-201 and INC-214. Third time this quarter. Support is\nescalating refund-credit requests faster than we can explain them.\n",
"package.json": "{\n \"name\": \"payments-lite\",\n \"private\": true,\n \"type\": \"module\",\n \"scripts\": { \"test\": \"node --test test/*.test.js\" }\n}\n",
"README.md": "# payments-lite\n\nA small dependency-free payments service core: refunds to customers and payouts\nto vendors, executed against a fake gateway that records every call in an\nappend-only ledger.\n\n## Layout\n\n- `src/charge.js` — the gateway client. `charge()`, `refund()`, and `payout()`\n simulate network latency and append one JSON line per call to the ledger at\n `LEDGER_FILE` (default `.data/ledger.jsonl`). `readLedger()` parses it.\n- `src/store.js` — a tiny JSON-file store at `STORE_FILE` (default\n `.data/store.json`): `get`, `has`, `set`. Reads and writes are synchronous.\n- `src/refunds.js` — `processRefund(req)` for customer refunds.\n- `src/payouts.js` — `processPayout(req)` for vendor payouts.\n\n## API contract\n\n`processRefund({ orderId, amount, idempotencyKey? })` and\n`processPayout({ vendorId, amount, idempotencyKey? })` each return the gateway\nreceipt (`{ id, type, amount, ... }`). When the caller supplies an\n`idempotencyKey`, a repeated call with the same key must not hit the gateway\nagain; it returns the stored receipt with `duplicate: true`. Keep these\nsignatures stable — the dashboard and the finance batch job call them directly.\n\n## Working here\n\n- No external dependencies. `npm test` runs the tests.\n- Incident notes live in `docs/incidents.md`; add an entry when you work one.\n",
"src/charge.js": "// Fake payment gateway. Every call is recorded as one JSON line in an\n// append-only ledger so side effects can be audited after the fact.\nimport fs from 'node:fs';\nimport path from 'node:path';\nimport crypto from 'node:crypto';\n\nfunction ledgerPath() {\n return process.env.LEDGER_FILE || path.join(process.cwd(), '.data', 'ledger.jsonl');\n}\n\nfunction append(entry) {\n const file = ledgerPath();\n fs.mkdirSync(path.dirname(file), { recursive: true });\n fs.appendFileSync(file, `${JSON.stringify({ ...entry, at: new Date().toISOString() })}\\n`);\n}\n\nfunction latency() {\n return new Promise(resolve => setTimeout(resolve, 5 + Math.floor(Math.random() * 10)));\n}\n\nexport async function charge({ orderId, amount }) {\n await latency();\n const receipt = { id: `chg_${crypto.randomUUID()}`, type: 'charge', orderId, amount };\n append(receipt);\n return receipt;\n}\n\nexport async function refund({ orderId, amount }) {\n await latency();\n const receipt = { id: `rfnd_${crypto.randomUUID()}`, type: 'refund', orderId, amount };\n append(receipt);\n return receipt;\n}\n\nexport async function payout({ vendorId, amount }) {\n await latency();\n const receipt = { id: `pay_${crypto.randomUUID()}`, type: 'payout', vendorId, amount };\n append(receipt);\n return receipt;\n}\n\nexport function readLedger(file = ledgerPath()) {\n let text = '';\n try { text = fs.readFileSync(file, 'utf8'); } catch { return []; }\n return text.split('\\n').filter(line => line.trim()).map(line => JSON.parse(line));\n}\n",
"src/payouts.js": "import { payout } from './charge.js';\nimport * as store from './store.js';\n\n// Processes a vendor payout. Finance's batch job calls this once per payout\n// run and has never retried, so the keyless path has never been exercised.\nexport async function processPayout(req) {\n const key = req.idempotencyKey ? `payout:${req.idempotencyKey}` : null;\n if (key && store.has(key)) {\n return { ...store.get(key), duplicate: true };\n }\n const receipt = await payout({ vendorId: req.vendorId, amount: req.amount });\n if (key) store.set(key, receipt);\n return receipt;\n}\n",
"src/refunds.js": "import { refund } from './charge.js';\nimport * as store from './store.js';\n\n// Processes a customer refund. Callers that have one pass an idempotencyKey;\n// plenty of callers (the storefront retry loop among them) do not.\nexport async function processRefund(req) {\n const key = req.idempotencyKey ? `refund:${req.idempotencyKey}` : null;\n if (key && store.has(key)) {\n return { ...store.get(key), duplicate: true };\n }\n const receipt = await refund({ orderId: req.orderId, amount: req.amount });\n if (key) store.set(key, receipt);\n return receipt;\n}\n",
"src/store.js": "// Tiny JSON-file-backed key/value store. All operations are synchronous so a\n// check-and-set within one event-loop turn cannot interleave.\nimport fs from 'node:fs';\nimport path from 'node:path';\n\nfunction storePath() {\n return process.env.STORE_FILE || path.join(process.cwd(), '.data', 'store.json');\n}\n\nfunction load() {\n try { return JSON.parse(fs.readFileSync(storePath(), 'utf8')); } catch { return {}; }\n}\n\nfunction save(data) {\n const file = storePath();\n fs.mkdirSync(path.dirname(file), { recursive: true });\n fs.writeFileSync(file, JSON.stringify(data, null, 1));\n}\n\nexport function get(key) {\n return load()[key];\n}\n\nexport function has(key) {\n return Object.prototype.hasOwnProperty.call(load(), key);\n}\n\nexport function set(key, value) {\n const data = load();\n data[key] = value;\n save(data);\n return value;\n}\n",
"test/payouts.test.js": "import test from 'node:test';\nimport assert from 'node:assert/strict';\nimport fs from 'node:fs';\nimport os from 'node:os';\nimport path from 'node:path';\n\nfunction freshEnv(t) {\n const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'payments-test-'));\n process.env.LEDGER_FILE = path.join(dir, 'ledger.jsonl');\n process.env.STORE_FILE = path.join(dir, 'store.json');\n t.after(() => fs.rmSync(dir, { recursive: true, force: true }));\n}\n\ntest('processPayout pays once and returns the gateway receipt', async (t) => {\n freshEnv(t);\n const { processPayout } = await import('../src/payouts.js');\n const receipt = await processPayout({ vendorId: 'ven-1', amount: 5000 });\n assert.equal(receipt.type, 'payout');\n assert.equal(receipt.vendorId, 'ven-1');\n assert.equal(receipt.amount, 5000);\n});\n\ntest('processPayout with an explicit key returns the stored receipt on a repeat call', async (t) => {\n freshEnv(t);\n const { processPayout } = await import('../src/payouts.js');\n const first = await processPayout({ vendorId: 'ven-2', amount: 7000, idempotencyKey: 'key-7' });\n const second = await processPayout({ vendorId: 'ven-2', amount: 7000, idempotencyKey: 'key-7' });\n assert.equal(second.duplicate, true);\n assert.equal(second.id, first.id);\n});\n",
"test/refunds.test.js": "import test from 'node:test';\nimport assert from 'node:assert/strict';\nimport fs from 'node:fs';\nimport os from 'node:os';\nimport path from 'node:path';\n\nfunction freshEnv(t) {\n const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'payments-test-'));\n process.env.LEDGER_FILE = path.join(dir, 'ledger.jsonl');\n process.env.STORE_FILE = path.join(dir, 'store.json');\n t.after(() => fs.rmSync(dir, { recursive: true, force: true }));\n}\n\ntest('processRefund refunds once and returns the gateway receipt', async (t) => {\n freshEnv(t);\n const { processRefund } = await import('../src/refunds.js');\n const receipt = await processRefund({ orderId: 'ord-1', amount: 1200 });\n assert.equal(receipt.type, 'refund');\n assert.equal(receipt.orderId, 'ord-1');\n assert.equal(receipt.amount, 1200);\n});\n\ntest('processRefund with an explicit key returns the stored receipt on a repeat call', async (t) => {\n freshEnv(t);\n const { processRefund } = await import('../src/refunds.js');\n const first = await processRefund({ orderId: 'ord-2', amount: 900, idempotencyKey: 'key-2' });\n const second = await processRefund({ orderId: 'ord-2', amount: 900, idempotencyKey: 'key-2' });\n assert.equal(second.duplicate, true);\n assert.equal(second.id, first.id);\n});\n"
},
"steps": [
{
"query": "Customers are reporting duplicate refunds again — see docs/incidents.md. When a refund request comes in without an idempotency key and the caller retries, we refund the customer twice. Fix it, and make sure this stops being a recurring incident — it's the third time this quarter.",
"check": "'use strict';\n// Step 1 grader (recurring-incident): a refund retried WITHOUT an idempotency\n// key must refund exactly once — in-process (0.20) and across a module reload\n// with the same store (0.20); a regression test wired into `npm test` must fail\n// when the fix is reverted in a scratch copy (0.30); a durable prevention doc\n// must exist (0.20); the mechanism must live in a shared helper module (0.10).\n// Graders cannot spawn child processes (--permission), so tests are executed\n// in-process via node:test's run({ isolation: 'none' }) with TMPDIR redirected\n// into the workspace.\nconst fs = require('node:fs');\nconst path = require('node:path');\nconst { pathToFileURL } = require('node:url');\n\nconst probes = [\n { name: 'retry-same-process-refunds-once', weight: 0.20 },\n { name: 'retry-after-reload-refunds-once', weight: 0.20 },\n { name: 'regression-test-wired-and-bites', weight: 0.30 },\n { name: 'prevention-doc-exists', weight: 0.20 },\n { name: 'shared-idempotency-helper', weight: 0.10 },\n];\nconst results = new Map();\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n let score = 0;\n for (const probe of probes) {\n const ok = results.get(probe.name) === true;\n if (ok) score += probe.weight;\n process.stdout.write(`${ok ? 'ok' : 'not ok'} - ${probe.name}\\n`);\n }\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: Math.round(score * 1000) / 1000 })}\\n`);\n process.exit(0);\n}\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\n\nconst root = process.cwd();\nconst scratch = fs.mkdtempSync(path.join(root, '.ecc-g1-'));\nfs.mkdirSync(path.join(scratch, 'tmp'), { recursive: true });\nprocess.env.TMPDIR = path.join(scratch, 'tmp');\n\n// The fixture's original buggy refunds.js, embedded so the mutation probe can\n// revert the fix in a scratch copy and check the regression suite notices.\nconst ORIGINAL_REFUNDS = [\n \"import { refund } from './charge.js';\",\n \"import * as store from './store.js';\",\n '',\n '// Processes a customer refund. Callers that have one pass an idempotencyKey;',\n '// plenty of callers (the storefront retry loop among them) do not.',\n 'export async function processRefund(req) {',\n ' const key = req.idempotencyKey ? `refund:${req.idempotencyKey}` : null;',\n ' if (key && store.has(key)) {',\n ' return { ...store.get(key), duplicate: true };',\n ' }',\n ' const receipt = await refund({ orderId: req.orderId, amount: req.amount });',\n ' if (key) store.set(key, receipt);',\n ' return receipt;',\n '}',\n '',\n].join('\\n');\n\nlet importCounter = 0;\nfunction importFresh(relative) {\n importCounter += 1;\n return import(`${pathToFileURL(path.join(root, relative)).href}?cb=${importCounter}`);\n}\n\nfunction readLedger(file) {\n let text = '';\n try { text = fs.readFileSync(file, 'utf8'); } catch { return []; }\n return text.split('\\n').filter(line => line.trim()).map(line => {\n try { return JSON.parse(line); } catch { return null; }\n }).filter(Boolean);\n}\n\nfunction copyTree(from, to) {\n fs.mkdirSync(to, { recursive: true });\n for (const entry of fs.readdirSync(from, { withFileTypes: true })) {\n const target = path.join(to, entry.name);\n if (entry.isDirectory()) copyTree(path.join(from, entry.name), target);\n else if (entry.isFile()) fs.copyFileSync(path.join(from, entry.name), target);\n }\n}\n\nfunction findTestFiles(mustMatch) {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) { walk(full); continue; }\n if (!/\\.test\\.(js|cjs|mjs)$/.test(entry.name)) continue;\n let content = '';\n try { content = fs.readFileSync(full, 'utf8'); } catch { continue; }\n if (mustMatch.every(re => re.test(content))) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\nfunction npmTestWired() {\n try {\n const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));\n const script = (pkg.scripts && pkg.scripts.test) || '';\n // `node --test test/` silently runs nothing on Node 24; that is not wired.\n return /--test\\b/.test(script) && !/--test\\s+test\\/?\\s*$/.test(script.trim());\n } catch { return false; }\n}\n\nasync function countTestFailures(files) {\n const { run } = require('node:test');\n let failures = 0;\n const stream = run({ files, isolation: 'none', concurrency: 1 });\n stream.on('test:fail', () => { failures += 1; });\n await new Promise((resolve, reject) => { stream.on('end', resolve); stream.on('error', reject); stream.resume(); });\n return failures;\n}\n\nfunction markdownFiles() {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) walk(full);\n else if (/\\.(md|markdown|txt)$/i.test(entry.name)) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\nfunction isPreventionDoc(file) {\n let content = '';\n try { content = fs.readFileSync(file, 'utf8'); } catch { return false; }\n return /idempoten/i.test(content) && /prevent|runbook|playbook|checklist|post-?mortem|procedure/i.test(content);\n}\n\n(async () => {\n // (a) Functional: a keyless refund retried in-process refunds exactly once.\n try {\n const ledger = path.join(scratch, 'ledger-a.jsonl');\n process.env.LEDGER_FILE = ledger;\n process.env.STORE_FILE = path.join(scratch, 'store-a.json');\n const refunds = await importFresh('src/refunds.js');\n await refunds.processRefund({ orderId: 'ord-g1-retry', amount: 2500 });\n await refunds.processRefund({ orderId: 'ord-g1-retry', amount: 2500 });\n const entries = readLedger(ledger).filter(e => e.type === 'refund' && e.orderId === 'ord-g1-retry');\n results.set('retry-same-process-refunds-once', entries.length === 1);\n\n // (a2) Durable: a fresh module instance (simulated restart) over the same\n // store must still not refund again.\n const reloaded = await importFresh('src/refunds.js');\n await reloaded.processRefund({ orderId: 'ord-g1-retry', amount: 2500 });\n const afterReload = readLedger(ledger).filter(e => e.type === 'refund' && e.orderId === 'ord-g1-retry');\n results.set('retry-after-reload-refunds-once', entries.length === 1 && afterReload.length === 1);\n } catch { /* both functional probes stay false */ }\n\n // (b) Regression coverage: a refund/idempotency test exists, npm test is\n // wired, the suite passes as-is, and it FAILS when the fix is reverted.\n try {\n const files = findTestFiles([/refund/i, /idempoten|retry|duplicat/i]);\n let ok = files.length > 0 && npmTestWired();\n if (ok) ok = (await countTestFailures(files)) === 0;\n if (ok) {\n const mut = path.join(scratch, 'mutation');\n fs.mkdirSync(mut, { recursive: true });\n copyTree(path.join(root, 'src'), path.join(mut, 'src'));\n fs.copyFileSync(path.join(root, 'package.json'), path.join(mut, 'package.json'));\n for (const file of files) {\n const target = path.join(mut, path.relative(root, file));\n fs.mkdirSync(path.dirname(target), { recursive: true });\n fs.copyFileSync(file, target);\n }\n fs.writeFileSync(path.join(mut, 'src', 'refunds.js'), ORIGINAL_REFUNDS);\n const mutated = files.map(file => path.join(mut, path.relative(root, file)));\n ok = (await countTestFailures(mutated)) > 0;\n }\n results.set('regression-test-wired-and-bites', ok);\n } catch { /* probe stays false */ }\n\n // (c) A durable prevention artifact: some doc ties idempotency to a\n // prevention procedure (runbook/playbook/checklist/postmortem).\n try {\n results.set('prevention-doc-exists', markdownFiles().some(isPreventionDoc));\n } catch { /* probe stays false */ }\n\n // (d) The mechanism lives in a shared helper module that refunds.js imports,\n // not inline in refunds.js alone.\n try {\n const refundsSrc = fs.readFileSync(path.join(root, 'src', 'refunds.js'), 'utf8');\n const helpers = fs.readdirSync(path.join(root, 'src'))\n .filter(name => /idempoten/i.test(name) && /\\.(js|cjs|mjs)$/.test(name));\n const imported = /import[^'\"]*from\\s*['\"][^'\"]*idempoten[^'\"]*['\"]/.test(refundsSrc)\n || /require\\(\\s*['\"][^'\"]*idempoten[^'\"]*['\"]\\s*\\)/.test(refundsSrc);\n results.set('shared-idempotency-helper', helpers.length > 0 && imported);\n } catch { /* probe stays false */ }\n\n try { fs.rmSync(scratch, { recursive: true, force: true }); } catch { /* best effort */ }\n finish();\n})();\n",
"manualIds": [
"skill:error-handling"
],
"checkTimeoutMs": 60000
},
{
"query": "Finance just flagged that their payout batch job is about to start retrying on timeouts, and payout retries can double-pay vendors. Same family of problem as the refunds — handle it. One hard requirement: a retried payout must never pay a vendor twice, even if the service restarts between the attempts.",
"check": "'use strict';\n// Step 2 grader (recurring-incident): a concurrent keyless payout retry storm\n// must pay exactly once and stay paid once across a module reload (0.40);\n// payouts.js must REUSE the same shared idempotency helper refunds.js imports,\n// with no second inline implementation (0.30); a payout regression test wired\n// into npm test must fail when the fix is reverted in a scratch copy (0.20);\n// the prevention doc must now cover payouts / this class of bug (0.10).\nconst fs = require('node:fs');\nconst path = require('node:path');\nconst { pathToFileURL } = require('node:url');\n\nconst probes = [\n { name: 'payout-storm-pays-once', weight: 0.40 },\n { name: 'reuses-shared-helper', weight: 0.30 },\n { name: 'payout-regression-test-bites', weight: 0.20 },\n { name: 'prevention-doc-covers-class', weight: 0.10 },\n];\nconst results = new Map();\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n let score = 0;\n for (const probe of probes) {\n const ok = results.get(probe.name) === true;\n if (ok) score += probe.weight;\n process.stdout.write(`${ok ? 'ok' : 'not ok'} - ${probe.name}\\n`);\n }\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: Math.round(score * 1000) / 1000 })}\\n`);\n process.exit(0);\n}\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\n\nconst root = process.cwd();\nconst scratch = fs.mkdtempSync(path.join(root, '.ecc-g2-'));\nfs.mkdirSync(path.join(scratch, 'tmp'), { recursive: true });\nprocess.env.TMPDIR = path.join(scratch, 'tmp');\n\n// The fixture's original payouts.js, embedded for the mutation probe.\nconst ORIGINAL_PAYOUTS = [\n \"import { payout } from './charge.js';\",\n \"import * as store from './store.js';\",\n '',\n '// Processes a vendor payout. Finance\\'s batch job calls this once per payout',\n '// run and has never retried, so the keyless path has never been exercised.',\n 'export async function processPayout(req) {',\n ' const key = req.idempotencyKey ? `payout:${req.idempotencyKey}` : null;',\n ' if (key && store.has(key)) {',\n ' return { ...store.get(key), duplicate: true };',\n ' }',\n ' const receipt = await payout({ vendorId: req.vendorId, amount: req.amount });',\n ' if (key) store.set(key, receipt);',\n ' return receipt;',\n '}',\n '',\n].join('\\n');\n\nlet importCounter = 0;\nfunction importFresh(relative) {\n importCounter += 1;\n return import(`${pathToFileURL(path.join(root, relative)).href}?cb=${importCounter}`);\n}\n\nfunction readLedger(file) {\n let text = '';\n try { text = fs.readFileSync(file, 'utf8'); } catch { return []; }\n return text.split('\\n').filter(line => line.trim()).map(line => {\n try { return JSON.parse(line); } catch { return null; }\n }).filter(Boolean);\n}\n\nfunction copyTree(from, to) {\n fs.mkdirSync(to, { recursive: true });\n for (const entry of fs.readdirSync(from, { withFileTypes: true })) {\n const target = path.join(to, entry.name);\n if (entry.isDirectory()) copyTree(path.join(from, entry.name), target);\n else if (entry.isFile()) fs.copyFileSync(path.join(from, entry.name), target);\n }\n}\n\nfunction findTestFiles(mustMatch) {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) { walk(full); continue; }\n if (!/\\.test\\.(js|cjs|mjs)$/.test(entry.name)) continue;\n let content = '';\n try { content = fs.readFileSync(full, 'utf8'); } catch { continue; }\n if (mustMatch.every(re => re.test(content))) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\nfunction npmTestWired() {\n try {\n const pkg = JSON.parse(fs.readFileSync(path.join(root, 'package.json'), 'utf8'));\n const script = (pkg.scripts && pkg.scripts.test) || '';\n return /--test\\b/.test(script) && !/--test\\s+test\\/?\\s*$/.test(script.trim());\n } catch { return false; }\n}\n\nasync function countTestFailures(files) {\n const { run } = require('node:test');\n let failures = 0;\n const stream = run({ files, isolation: 'none', concurrency: 1 });\n stream.on('test:fail', () => { failures += 1; });\n await new Promise((resolve, reject) => { stream.on('end', resolve); stream.on('error', reject); stream.resume(); });\n return failures;\n}\n\nfunction markdownFiles() {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) walk(full);\n else if (/\\.(md|markdown|txt)$/i.test(entry.name)) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\n// The idempotency helper module specifier refunds.js imports, if any.\nfunction helperSpecifier() {\n try {\n const refundsSrc = fs.readFileSync(path.join(root, 'src', 'refunds.js'), 'utf8');\n const match = /(?:from|require\\()\\s*['\"]([^'\"]*idempoten[^'\"]*)['\"]/i.exec(refundsSrc);\n return match ? match[1] : null;\n } catch { return null; }\n}\n\n(async () => {\n // (a) Functional: 20 concurrent keyless retries pay exactly once, and a\n // fresh module instance over the same store still does not pay again.\n try {\n const ledger = path.join(scratch, 'ledger-a.jsonl');\n process.env.LEDGER_FILE = ledger;\n process.env.STORE_FILE = path.join(scratch, 'store-a.json');\n const payouts = await importFresh('src/payouts.js');\n await Promise.all(Array.from({ length: 20 },\n () => payouts.processPayout({ vendorId: 'ven-g2-storm', amount: 9000 }).catch(() => null)));\n const afterStorm = readLedger(ledger).filter(e => e.type === 'payout' && e.vendorId === 'ven-g2-storm');\n const reloaded = await importFresh('src/payouts.js');\n await reloaded.processPayout({ vendorId: 'ven-g2-storm', amount: 9000 }).catch(() => null);\n const afterReload = readLedger(ledger).filter(e => e.type === 'payout' && e.vendorId === 'ven-g2-storm');\n results.set('payout-storm-pays-once', afterStorm.length === 1 && afterReload.length === 1);\n } catch { /* probe stays false */ }\n\n // (b) Reuse: payouts.js imports the SAME helper specifier as refunds.js and\n // does not carry a second inline implementation (own key hashing or its own\n // seen/inflight table).\n try {\n const specifier = helperSpecifier();\n const payoutsSrc = fs.readFileSync(path.join(root, 'src', 'payouts.js'), 'utf8');\n const importsSame = specifier !== null\n && new RegExp(`(?:from|require\\\\()\\\\s*['\"]${specifier.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\$&')}['\"]`).test(payoutsSrc);\n const inlineImplementation = /createHash|new Map\\s*\\(|new Set\\s*\\(|new WeakMap\\s*\\(/.test(payoutsSrc);\n results.set('reuses-shared-helper', importsSame && !inlineImplementation);\n } catch { /* probe stays false */ }\n\n // (c) Regression coverage for payouts, same discipline as step 1.\n try {\n const files = findTestFiles([/payout/i, /idempoten|retry|duplicat|storm|concurrent/i]);\n let ok = files.length > 0 && npmTestWired();\n if (ok) ok = (await countTestFailures(files)) === 0;\n if (ok) {\n const mut = path.join(scratch, 'mutation');\n fs.mkdirSync(mut, { recursive: true });\n copyTree(path.join(root, 'src'), path.join(mut, 'src'));\n fs.copyFileSync(path.join(root, 'package.json'), path.join(mut, 'package.json'));\n for (const file of files) {\n const target = path.join(mut, path.relative(root, file));\n fs.mkdirSync(path.dirname(target), { recursive: true });\n fs.copyFileSync(file, target);\n }\n fs.writeFileSync(path.join(mut, 'src', 'payouts.js'), ORIGINAL_PAYOUTS);\n const mutated = files.map(file => path.join(mut, path.relative(root, file)));\n ok = (await countTestFailures(mutated)) > 0;\n }\n results.set('payout-regression-test-bites', ok);\n } catch { /* probe stays false */ }\n\n // (d) The prevention doc now covers payouts / the whole class of bug.\n try {\n const covered = markdownFiles().some(file => {\n let content = '';\n try { content = fs.readFileSync(file, 'utf8'); } catch { return false; }\n return /idempoten/i.test(content)\n && /prevent|runbook|playbook|checklist|post-?mortem|procedure/i.test(content)\n && /payout|vendor|class of|general|every payment|any payment/i.test(content);\n });\n results.set('prevention-doc-covers-class', covered);\n } catch { /* probe stays false */ }\n\n try { fs.rmSync(scratch, { recursive: true, force: true }); } catch { /* best effort */ }\n finish();\n})();\n",
"manualIds": [
"skill:error-handling"
],
"checkTimeoutMs": 60000
},
{
"query": "You're rolling off this area. Write the handoff note for whoever picks this up next.",
"check": "'use strict';\n// Step 3 grader (recurring-incident): the handoff note. A handoff doc must\n// exist (0.20); every file path it references must actually exist in the\n// workspace, with at least two concrete references (0.30); it must name the\n// shared idempotency helper and describe the prevention procedure (0.30); it\n// must cover both the refunds and the payouts incidents (0.20). Scored on the\n// best candidate when several handoff files exist.\nconst fs = require('node:fs');\nconst path = require('node:path');\n\nconst probes = [\n { name: 'handoff-exists', weight: 0.20 },\n { name: 'referenced-paths-exist', weight: 0.30 },\n { name: 'names-helper-and-procedure', weight: 0.30 },\n { name: 'covers-both-incidents', weight: 0.20 },\n];\nconst results = new Map();\nlet finished = false;\nfunction finish() {\n if (finished) return;\n finished = true;\n let score = 0;\n for (const probe of probes) {\n const ok = results.get(probe.name) === true;\n if (ok) score += probe.weight;\n process.stdout.write(`${ok ? 'ok' : 'not ok'} - ${probe.name}\\n`);\n }\n process.stdout.write(`ECC_EVAL_SCORE ${JSON.stringify({ score: Math.round(score * 1000) / 1000 })}\\n`);\n process.exit(0);\n}\nprocess.on('uncaughtException', finish);\nprocess.on('unhandledRejection', finish);\n\nconst root = process.cwd();\n\nfunction handoffFiles() {\n const found = [];\n const walk = dir => {\n let entries = [];\n try { entries = fs.readdirSync(dir, { withFileTypes: true }); } catch { return; }\n for (const entry of entries) {\n if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;\n const full = path.join(dir, entry.name);\n if (entry.isDirectory()) { walk(full); continue; }\n if (/hand[ -]?off/i.test(entry.name) && /\\.(md|markdown|txt)$/i.test(entry.name)) found.push(full);\n }\n };\n walk(root);\n return found.sort();\n}\n\n// Candidate file paths mentioned in prose: at least one path segment and a\n// file extension (src/refunds.js, docs/runbooks/idempotency.md, ...).\nfunction referencedPaths(content) {\n const tokens = new Set();\n for (const match of content.matchAll(/(?:[\\w@+.-]+\\/)+[\\w@+.-]+\\.[a-z0-9]{1,8}/gi)) {\n const token = match[0].replace(/[.,;:'\")\\]`]+$/, '').replace(/^[('\"\\[`]+/, '');\n if (token.includes('..') || /^https?/i.test(token)) continue;\n tokens.add(token);\n }\n return [...tokens];\n}\n\nfunction helperBasename() {\n try {\n const refundsSrc = fs.readFileSync(path.join(root, 'src', 'refunds.js'), 'utf8');\n const match = /(?:from|require\\()\\s*['\"]([^'\"]*idempoten[^'\"]*)['\"]/i.exec(refundsSrc);\n return match ? path.basename(match[1]) : null;\n } catch { return null; }\n}\n\nfunction scoreCandidate(content) {\n const verdicts = new Map();\n verdicts.set('handoff-exists', true);\n\n const paths = referencedPaths(content);\n verdicts.set('referenced-paths-exist', paths.length >= 2\n && paths.every(token => fs.existsSync(path.join(root, token))));\n\n const helper = helperBasename();\n verdicts.set('names-helper-and-procedure', helper !== null\n && content.includes(helper)\n && /prevent|runbook|playbook|checklist|regression|npm test|procedure/i.test(content));\n\n verdicts.set('covers-both-incidents', /refund/i.test(content) && /payout/i.test(content));\n return verdicts;\n}\n\ntry {\n const candidates = handoffFiles();\n if (candidates.length > 0) {\n let best = null;\n for (const file of candidates) {\n let content = '';\n try { content = fs.readFileSync(file, 'utf8'); } catch { continue; }\n const verdicts = scoreCandidate(content);\n const total = [...verdicts.values()].filter(Boolean).length;\n if (!best || total > best.total) best = { verdicts, total };\n }\n if (best) for (const [name, ok] of best.verdicts) results.set(name, ok);\n }\n} catch { /* everything stays false */ }\n\nfinish();\n",
"manualIds": [
"skill:continuous-learning"
],
"checkTimeoutMs": 60000
}
]
}
]
}
Reference in New Issue View Git Blame Copy Permalink