mirror of
https://github.com/affaan-m/ECC.git
synced 2026-10-18 14:07:33 +02:00
montjeffrey
1d66665d39
96 trials, none invalid. The headline reads gate 23/48 against off 13/48, a 21-point paired difference at exact McNemar p = 0.021, and it is an artifact. The scenarios, prompts, graders and model alias were identical to the previous run and the off arm has no gate, yet off fell from 18/48 to 13/48 while the gated arm went down, 25/48 to 23/48. An intervention meant to raise the gated arm did not raise it, so the p-value reflects a baseline that fell rather than a gate that improved. The two scenarios the question cannot reach, having no data for the profile to detect, serve as a control: they moved by -3 and +3, more than any of the four it did reach (-1, -1, +1, -1). Noise on eight trials is about plus or minus three, which is larger than any effect available to the change. The question is kept, since it asks something the gate never asked and the corpus says that kind of answer decides the outcome, but no improvement is claimed. Testing it fairly needs scenarios authored blind to it, which this corpus no longer is.