Simply adding "Repeat the question before answering it." somehow make the models answer the trick question correctly.
Probable explanations: Repeating the question in the model's context, significantly increasing the likelihood of the model detecting any potential "gotchas."
One hypothesis is that maybe it puts the model into more of a completion mo... See more