openswarm/scripts/ci/GATE_AUDIT.md

# Gate audit: is this a real gate or smoke and mirrors?

A test only counts if breaking the thing it guards turns it RED. This is the
red-team of our own gate: for each check, the claim, the fault we injected, the
result, and an honest note on what it still does NOT cover. Re-run the evidence
with `node scripts/ci/selftest-gate.js` (pure, mutation tests) plus the live
fault-injections noted below.

## Verdict: it has teeth (with one gap found + fixed)

| Check | Claim | Fault injected | Result | Real? |
| --- | --- | --- | --- | --- |
| boot: provenance | the running app is the build at HEAD | built at an older sha, ran against HEAD | **RED** ("provenance sha X != git HEAD Y"), seen live | yes |
| boot: provenance/marks | a good log passes; broken logs don't | missing `[provenance]`, sha mismatch, missing mark, out-of-order, all-zero | **RED on each**, good log green (selftest-gate.js) | yes |
| boot: health | backend actually serves | port not serving | **RED** (health != 200) | yes |
| signature: reject | unsigned bits can't ship | `--require-signed` on the unsigned local build | **RED** (exit 1) | yes |
| signature: recognize | a real signature is seen as valid | `--require-signed` on `node.exe` (OpenJS-signed) | **GREEN, signed=Valid** (not "always unsigned") | yes |
| resilience: locked-port | survives a taken preferred port | held 8324-8333 | app served on **:8334** (behavior changed vs default :8324) | yes |
| resilience: multi-instance | 2nd launch exits, 1st keeps serving | launched a 2nd instance | 2nd **exited code 0**, 1st still 200 | yes |
| network: auth | the bearer is validated, not just present | no-token / **wrong-token** / real-token | **401 / 401 / 200** | yes (after fix) |
| network: 9router | the bundled router is up | TCP probe :20128 | open when up, RED when down | yes |
| agent turn | a real model reply on the user's creds | fresh session, tool-free prompt | **completed, tokens.output > 0** (can't be faked: a fresh session starts at 0) | yes |
| gui hand | a CC instance can drive the real GUI | launched + screenshotted + read log | works; render gate asserts `#root` has children (not a blank window) | yes |
| verify-all | one failure fails the whole gate | bogus app path | **3 sub-checks RED -> exit 1** (not silently green) | yes |

## The gap we found and closed

`verify-network` originally tested only no-token (401) vs real-token (200). A
backend that accepted ANY `Authorization` header would have passed both while auth
was actually broken. Added a **wrong-token probe** that must also get 401; the 200
now means "validated", not "a header was present". Proven live: `401 / 401 / 200`.

## What this gate still does NOT cover (honest residuals)

- **macOS signing path is unverified locally** (no Mac here). The `codesign` +
  `spctl` + staple logic is written but only CI on a Mac runner proves it.
- **Full port-range exhaustion** isn't exercised; we hold the bottom of the range
  (common real case). All-101-taken relies on get-port's own ephemeral fallback.
- **Agent-turn content** isn't asserted (models vary); we assert real output
  tokens were produced, not that the words are correct.
- **Perf marks are emitted by product code**; the gate trusts the app isn't lying
  about its own lifecycle. `first-paint` only exists if the renderer painted, so a
  no-paint boot is still caught.
- **The gate proves boot/serve/resilience/auth, not feature correctness.** That is
  the CC-instance apex layer's job (drive the GUI, judge "does it actually work").