How Not to Be Wrong

So here’s the procedure for ruling out the null hypothesis, in executive bullet-point form: Run an experiment. Suppose the null hypothesis is true, and let p be the probability (under that hypothesis) of getting results as extreme as those observed. The number p is called the p-value. If it is very small, rejoice; you get to say your results are st

Improbability, as described here, is a relative notion, not an absolute one; when we say an outcome is improbable, we are always saying, explicitly or not, that it is improbable under some set of hypotheses we’ve made about the underlying mechanisms of the world.

That may seem like an unfair prejudice, but without some prejudices we would run the risk of walking around in a constant state of astoundedness. Richard Feynman famously captured this state of mind:

Mathematics is the study of things that come out a certain way because there is no other way they could possibly be.

Blaise Pascal, the seventeenth-century mathematician and philosopher who wrote in his Pensées, “ ‘God is, or He is not.’ But to which side shall we incline? Reason can decide nothing here.”

If I claim I can make the sun come up with my mind, and it does, you shouldn’t be impressed by my powers; but if I claim I can make the sun not come up, and it doesn’t, then I’ve demonstrated an outcome very unlikely under the null hypothesis, and you’d best take notice.

(Let’s face it, famous old maxims attributed to ancient scientists are probably made up, but they’re no less instructive for that.)

(most famously, the Oxford philosopher Nick Bostrom

Nonlinear thinking means which way you should go depends on where you already are.