Everything Is Predictable

What they do is try to establish what the data tells us about a hypothesis. If I give 500 people a Covid vaccine, and 500 people a placebo, and then ten people get Covid in the placebo group and only one in the vaccine group, what does that tell us? How sure can we be that the vaccine works?

But what does that mean? Can I say that bigfoot people are cleverer, or not? What Fisher did was suggest that we should choose some arbitrary level at which we say ‘OK, it’s pretty unlikely that we’d see results this extreme given the null hypothesis, so I’m going to behave as though it’s a real effect.’ Fisher himself said that a p-value of 0.05 –

Bernoulli’s claim was that the more you flipped the coin, the closer, on average, to the ‘true’ probability your results would be. You might reasonably say that this is pretty obvious. And also, so what? You know you’ll see roughly half of the coin-flips come up heads. You don’t need to flip the coin a million times to prove it.

But the point is that when you’re trying to work out how likely something is, what we need to talk about is the number of outcomes – the number of outcomes that result in whatever it is you’re talking about, and the total number of possible outcomes. This was, I think it’s fair to say, the first real formalisation of the idea of ‘probability’.

Jeffreys himself wrote: ‘[E]very scientific advance involves a transition from complete ignorance, through a stage of partial knowledge based on evidence becoming gradually more conclusive, to the stage of practical certainty.’ The uncertain parts of science, he said, are ‘the most interesting part’.96

His insight in probability theory was that probabilities are beliefs; our beliefs, if we act on them, are themselves a kind of bet. As Ramsey put it: ‘All our lives, we are in a sense betting. Whenever we go to the station, we are betting that a train will really run, and if we had not a sufficient degree of belief in this, we should decline the be

‘This test is 99 per cent sensitive and 99 per cent specific.’ What that means is that if you have the disease, there’s a 99 per cent chance that it will, correctly, tell you that you have the disease; if you don’t have the disease, there’s a 99 per cent chance that it will tell you, correctly, that you don’t have the disease.

‘Necessarily,’ wrote David Howie, a historian of statistics, ‘these inferences were tentative. They were advanced not with certainty but with degrees of confidence that were updated or modified to account for new information.’95 That is: they were done in a Bayesian fashion. Each time Jeffreys got new information, he updated his prior confidence in

Again, the most common error was that they assumed a p-value gave the probability that the results were due to chance. This is (as we’ve been discussing for quite some time now) completely backwards. What p-values tell you is how likely you are to see that data, given a hypothesis.