With the ambiguity signposted, the arms were indistinguishable on every measure.
With the prompts rewritten as ordinary tickets and nothing else changed, they
separate: asked the user 16/24 against 10/24, obtained the deciding fact the
same, passed 15/24 against 8/24, exact McNemar p = 0.065 to 0.070, and a paired
pass-rate difference of 29 points in the gate's favour whose bootstrap interval
excludes zero.
The regime contrast carries it. The gated arm passed 15/24 in both runs while
the ungated arm fell from 16/24 to 8/24, which is what removing the only cue
that a convention was missing would do if the gate supplies that cue itself.
Asking stayed necessary throughout: no trial in either arm passed without the
deciding fact.
Reported as suggestive rather than established. All three p-values sit just
above 5%, one more gate-favouring discordant pair would cross it, and extending
this run after seeing that would be optional stopping. A confirmatory run needs
its sample fixed first: about 43 pairs, or 8 repetitions over these six
scenarios.
montjeffrey
a8fbe44e4f