Hypothesis Testing: An Intuitive Guide for Making Data Driven Decisions

The responses of these investigators reflected the expectation that a valid hypothesis about a population will be represented by a statistically significant result in a sample with little regard for its size.

Once your test reaches statistical significance, you can analyze the data to gather learning and infer the “why?” behind the result.

When companies perform A/B tests, experimenters report how one version changed a particular metric compared to the other version. They also report a statistic called a p-value, which shows the probability that the difference they observed was due to chance.[99] Usually, if p < 0.05 (i.e. there’s a less than 5% chance that the difference was just

The null hypothesis is rejected if your data set is unlikely to have been produced by chance. The significance of the results is described by the confidence level that was defined by the test (as described by the acceptable error “alpha-level”).

The word significant in statistical terms means only that you have high enough confidence in your answer. It does not mean that the effect found in your test is large or important. If you collect a large enough data sample, even tiny differences can be found to be statistically significant.