96 trials, none invalid. The headline reads gate 23/48 against off 13/48, a
21-point paired difference at exact McNemar p = 0.021, and it is an artifact.
The scenarios, prompts, graders and model alias were identical to the previous
run and the off arm has no gate, yet off fell from 18/48 to 13/48 while the
gated arm went down, 25/48 to 23/48. An intervention meant to raise the gated
arm did not raise it, so the p-value reflects a baseline that fell rather than a
gate that improved.
The two scenarios the question cannot reach, having no data for the profile to
detect, serve as a control: they moved by -3 and +3, more than any of the four
it did reach (-1, -1, +1, -1). Noise on eight trials is about plus or minus
three, which is larger than any effect available to the change.
The question is kept, since it asks something the gate never asked and the
corpus says that kind of answer decides the outcome, but no improvement is
claimed. Testing it fairly needs scenarios authored blind to it, which this
corpus no longer is.
montjeffrey
1d66665d39