Just make it stop! When will we say that further research isn't needed?

But what does that mean? Can I say that bigfoot people are cleverer, or not? What Fisher did was suggest that we should choose some arbitrary level at which we say ‘OK, it’s pretty unlikely that we’d see results this extreme given the null hypothesis, so I’m going to behave as though it’s a real effect.’ Fisher himself said that a p-value of 0.05 –

What if the null hypothesis is not rejected? This simply means that you did not find any statistically significant differences. That is not the same as stating that there was no difference. Remember, accepting the null hypothesis merely means that the observed differences might have been due simply to random chance, not that they must have been. St

The responses of these investigators reflected the expectation that a valid hypothesis about a population will be represented by a statistically significant result in a sample with little regard for its size.

So here’s the procedure for ruling out the null hypothesis, in executive bullet-point form: Run an experiment. Suppose the null hypothesis is true, and let p be the probability (under that hypothesis) of getting results as extreme as those observed. The number p is called the p-value. If it is very small, rejoice; you get to say your results are st