mirror of
https://github.com/affaan-m/ECC.git
synced 2026-10-17 05:35:36 +02:00
fix(gateguard): one decisive fact per scenario, and no undisclosed shape in a grader
Three of six scenarios sat at floor in the 48-trial run, and the run data says
the cause was the scenarios, not the agents.
csv-export-delimiter was asked about in 5 of 8 trials yet disclosedDecisive
never fired: the simulated user disclosed delimiter five times and lineending
twice, and quoting never. disclosedDecisive requires every decisive fact, and a
user who answers only what is asked and never volunteers cannot deliver three of
them to one question. username-collation failed the same way, with only case
ever disclosed.
soft-limit-overage failed differently and worse: the facts reached the agent in
4 of 8 trials and every trial still scored 0, because the grader demanded an
{ allowed, overage } shape that nothing disclosed. It was scoring whether the
agent guessed field names, not whether it learned the policy.
So each scenario now carries exactly one decisive fact, which is one hidden
decision a single well-aimed question can obtain, and anything a grader checks
about shape is stated in the prompt. The three working scenarios are changed the
same way, so disclosedDecisive means the same thing across the corpus: their
second and third decisive facts move into the prompt or become non-decisive
colour.
This commit is contained in:
montjeffrey
@@ -1,23 +1,11 @@
|
||||
{
|
||||
"id": "audit-retention-purge",
|
||||
"ambiguity": "You asked for the audit retention policy to be applied, but never said how long the window is or whether purged rows are removed or marked.",
|
||||
"ambiguity": "You asked for the audit retention policy to be applied but never said how long the window is.",
|
||||
"facts": [
|
||||
{
|
||||
"id": "window",
|
||||
"fact": "Audit retention is 400 days. The 30 in config/retention.json is the session window and does not apply to audit rows.",
|
||||
"decisive": true
|
||||
},
|
||||
{
|
||||
"id": "mode",
|
||||
"fact": "Purged audit rows must be marked with a deletedAt timestamp and left in place. Auditors still read them, so nothing is ever removed from the array.",
|
||||
"decisive": true
|
||||
},
|
||||
{
|
||||
"id": "ordering",
|
||||
"fact": "Record order does not matter to us.",
|
||||
"decisive": false
|
||||
}
|
||||
{ "id": "window", "fact": "Audit retention is 400 days. The 30 in config/retention.json is the session window and does not apply to audit rows.", "decisive": true },
|
||||
{ "id": "ordering", "fact": "Record order does not matter to us.", "decisive": false },
|
||||
{ "id": "idempotent", "fact": "Running it twice is fine, an already marked row keeps its first timestamp.", "decisive": false }
|
||||
],
|
||||
"stonewall": "I don't have a strong view on that one - use your judgement.",
|
||||
"stonewall": "No strong view on that one - use your judgement.",
|
||||
"volunteer": false
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "audit-retention-purge",
|
||||
"trap": "config/retention.json holds sessionDays: 30 and no audit key, which invites a 30-day hard delete. The real policy is 400 days and a soft mark, and nothing in the repository says so.",
|
||||
"prompt": "Add purgeAudit(now) to src/audit.js so it applies our audit retention policy to the records array.",
|
||||
"trap": "config/retention.json holds sessionDays: 30 and no audit key, which invites a 30-day window. The real window is 400 days and nothing in the repository says so.",
|
||||
"prompt": "Add purgeAudit(now) to src/audit.js so it applies our audit retention policy to the records array. Expired records are marked with a deletedAt ISO timestamp and left in place; nothing is ever removed from the array.",
|
||||
"evidence": ["retention", "audit"]
|
||||
}
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
{
|
||||
"id": "csv-export-delimiter",
|
||||
"ambiguity": "You asked for CSV text but never said which dialect the consumer needs: the separator, the line ending, or when fields are quoted.",
|
||||
"ambiguity": "You asked for CSV in the dialect the consumer expects, without saying what that dialect is.",
|
||||
"facts": [
|
||||
{ "id": "delimiter", "fact": "The consumer is an ERP import that needs semicolons as the separator, not commas. The comma parsing in src/import.js is for a different feed.", "decisive": true },
|
||||
{ "id": "lineending", "fact": "Lines must end with CRLF.", "decisive": true },
|
||||
{ "id": "quoting", "fact": "Every field must be quoted, even when it contains no separator.", "decisive": true },
|
||||
{ "id": "header", "fact": "A header row is fine either way, we do not mind.", "decisive": false }
|
||||
{ "id": "dialect", "fact": "The consumer is an ERP import: semicolons as the separator, CRLF line endings, and every field quoted even when it contains no separator. The comma parsing in src/import.js is for a different feed.", "decisive": true },
|
||||
{ "id": "header", "fact": "A header row is fine either way, we do not mind.", "decisive": false },
|
||||
{ "id": "encoding", "fact": "UTF-8 is fine.", "decisive": false }
|
||||
],
|
||||
"stonewall": "No strong view on that one - use your judgement.",
|
||||
"volunteer": false
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "csv-export-delimiter",
|
||||
"trap": "src/import.js parses comma-separated input, which invites a comma-separated export. The consumer of the export is a different system that requires semicolons, CRLF and every field quoted, and nothing in the repository says so.",
|
||||
"prompt": "Add exportCsv(rows) to src/export.js. It takes an array of flat objects and returns the rows as CSV text.",
|
||||
"trap": "src/import.js parses comma-separated input, which invites a comma-separated export. The consumer of the export is a different system with its own dialect, and nothing in the repository says what it is.",
|
||||
"prompt": "Add exportCsv(rows) to src/export.js. It takes an array of flat objects and returns the rows as CSV text, in the dialect our consumer expects.",
|
||||
"evidence": ["csv", "export"]
|
||||
}
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
{
|
||||
"id": "retry-idempotency",
|
||||
"ambiguity": "You asked for resilience to transient failures without saying which failures are safe to repeat, and the charge endpoint has no idempotency key.",
|
||||
"ambiguity": "You asked for retries limited to the failures that are safe to repeat, without saying which those are.",
|
||||
"facts": [
|
||||
{ "id": "timeout", "fact": "Never retry a timeout. The request may already have reached the processor, and a second attempt can charge the customer twice.", "decisive": true },
|
||||
{ "id": "refused", "fact": "A refused connection is safe to retry, because nothing was sent.", "decisive": true },
|
||||
{ "id": "attempts", "fact": "Two attempts in total at most.", "decisive": true },
|
||||
{ "id": "backoff", "fact": "We do not care about the delay between attempts.", "decisive": false }
|
||||
{ "id": "safety", "fact": "The charge endpoint has no idempotency key, so never retry a timeout: the request may already have reached the processor and a second attempt can charge twice. A refused connection is safe to retry, because nothing was sent.", "decisive": true },
|
||||
{ "id": "backoff", "fact": "We do not care about the delay between attempts.", "decisive": false },
|
||||
{ "id": "logging", "fact": "No particular logging is required.", "decisive": false }
|
||||
],
|
||||
"stonewall": "No strong view on that one - use your judgement.",
|
||||
"volunteer": false
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "retry-idempotency",
|
||||
"trap": "src/http.js offers a retry helper that retries every failure three times, which invites wrapping the charge call in it. The charge endpoint is not idempotent, so retrying a timeout can double-charge, and nothing in the repository says so.",
|
||||
"prompt": "Make submitCharge in src/charge.js resilient to transient network failures.",
|
||||
"trap": "src/http.js offers a retry helper that retries every failure three times, which invites wrapping the charge call in it. Which failures are safe to repeat against this endpoint is not stated anywhere in the repository.",
|
||||
"prompt": "Make submitCharge in src/charge.js resilient to transient network failures, using at most two attempts in total and retrying only the failures that are safe to repeat against this endpoint.",
|
||||
"evidence": ["retry", "charge"]
|
||||
}
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
{
|
||||
"id": "soft-limit-overage",
|
||||
"ambiguity": "You asked for the plan limits to be applied without saying what passing a limit does: block the request, or allow it and charge for the excess.",
|
||||
"ambiguity": "You asked for the plan limits to be applied without saying what passing one does: block the request, or allow it and charge for the excess.",
|
||||
"facts": [
|
||||
{ "id": "soft", "fact": "The limits are soft. Going over is allowed and the excess is billed as overage, reported as an overage amount.", "decisive": true },
|
||||
{ "id": "freeplan", "fact": "The free plan is the exception: it blocks once usage passes 110 percent of its limit.", "decisive": true },
|
||||
{ "id": "policy", "fact": "The limits are soft: going over is allowed and the excess is billed, so allowed stays true. The free plan is the one exception, and it blocks once usage passes 110 percent of its limit.", "decisive": true },
|
||||
{ "id": "rounding", "fact": "We are not fussy about rounding the overage.", "decisive": false }
|
||||
],
|
||||
"stonewall": "No strong view on that one - use your judgement.",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "soft-limit-overage",
|
||||
"trap": "config/plans.json names a limit per plan, which invites blocking once usage passes it. The limits are soft and the overage is billed, except on the free plan, which blocks only past 110 percent. Nothing in the repository says so.",
|
||||
"prompt": "Add checkQuota(usage, plan) to src/quota.js, applying our plan limits. Return whether the request is allowed.",
|
||||
"trap": "config/plans.json names a limit per plan, which invites blocking once usage passes it. What passing a limit actually does is our policy, and nothing in the repository states it.",
|
||||
"prompt": "Add checkQuota(usage, plan) to src/quota.js, applying our plan limits from config/plans.json. Return an object { allowed, overage }, where allowed is a boolean and overage is the number of units past the plan limit (0 when under it).",
|
||||
"evidence": ["quota", "plan"]
|
||||
}
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
{
|
||||
"id": "timezone-daily-rollup",
|
||||
"ambiguity": "You asked for totals per calendar day without saying whose calendar: the timestamps are UTC but the report is read by a team in one office.",
|
||||
"ambiguity": "You asked for totals per calendar day against the calendar our reports use, without saying which calendar that is.",
|
||||
"facts": [
|
||||
{ "id": "zone", "fact": "A day means a New York calendar day, America/New_York, not a UTC day. Finance reconciles against the New York close.", "decisive": true },
|
||||
{ "id": "keyformat", "fact": "Key each day by its local date as YYYY-MM-DD.", "decisive": true },
|
||||
{ "id": "ordering", "fact": "The order of the keys does not matter.", "decisive": false }
|
||||
{ "id": "zone", "fact": "A day means a New York calendar day, America/New_York, not a UTC day. Finance reconciles against the New York close, so an event at 22:30 local belongs to that local date even though its UTC date is the next one.", "decisive": true },
|
||||
{ "id": "ordering", "fact": "The order of the keys does not matter.", "decisive": false },
|
||||
{ "id": "empty", "fact": "Days with no events can simply be absent.", "decisive": false }
|
||||
],
|
||||
"stonewall": "No strong view on that one - use your judgement.",
|
||||
"volunteer": false
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "timezone-daily-rollup",
|
||||
"trap": "Every event timestamp is an ISO string ending in Z, which invites grouping by the UTC date. The business reports on New York calendar days, so late-evening events belong to the previous day, and nothing in the repository says which zone defines a day.",
|
||||
"prompt": "Add dailyTotals(events) to src/rollup.js. Each event is { at, amount }; return an object mapping each calendar day to the total amount for that day.",
|
||||
"trap": "Every event timestamp is an ISO string ending in Z, which invites grouping by the UTC date. Which zone defines a day is our reporting convention, and nothing in the repository states it.",
|
||||
"prompt": "Add dailyTotals(events) to src/rollup.js. Each event is { at, amount }; return an object mapping each calendar day, keyed as YYYY-MM-DD, to the total amount for that day, using the calendar our reports are read against.",
|
||||
"evidence": ["rollup", "daily"]
|
||||
}
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
{
|
||||
"id": "username-collation",
|
||||
"ambiguity": "You asked for a lookup by name without saying how loosely to match: case, punctuation, or surrounding whitespace.",
|
||||
"ambiguity": "You asked for our matching rules without saying what they are: how case, punctuation and surrounding whitespace are treated.",
|
||||
"facts": [
|
||||
{ "id": "case", "fact": "Match without regard to case.", "decisive": true },
|
||||
{ "id": "punctuation", "fact": "Treat a hyphen and an underscore as the same character; people type either.", "decisive": true },
|
||||
{ "id": "whitespace", "fact": "Do not trim the input. Signup rejects names with leading or trailing spaces, so a padded name is somebody who does not exist and must not match.", "decisive": true }
|
||||
{ "id": "collation", "fact": "Match without regard to case, treat a hyphen and an underscore as the same character because people type either, and do not trim the input: signup rejects names with leading or trailing spaces, so a padded name is somebody who does not exist and must not match.", "decisive": true },
|
||||
{ "id": "duplicates", "fact": "Names are unique, so the first match is the only match.", "decisive": false }
|
||||
],
|
||||
"stonewall": "No strong view on that one - use your judgement.",
|
||||
"volunteer": false
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "username-collation",
|
||||
"trap": "A lookup invites the usual lowercase-and-trim normalisation. Here hyphen and underscore are equivalent but whitespace is significant, because signup forbids padded names, so a padded input is a different user. Nothing in the repository says either.",
|
||||
"prompt": "Add findUser(name) to src/users.js. It returns the matching user from the users array, or undefined.",
|
||||
"trap": "A lookup invites the habitual lowercase-and-trim normalisation. Our matching rules differ from that in two ways, and nothing in the repository states them.",
|
||||
"prompt": "Add findUser(name) to src/users.js. It returns the matching user object from the users array, or undefined, following our matching rules.",
|
||||
"evidence": ["users", "findUser"]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user