Evolutionary hypotheses, where one parameter is slightly optimized, have a less profound effect than revolutionary hypotheses, where the approach is fundamentally different. That said, evolutionary hypotheses are more likely to bear fruit.

Z-test – for checking the mean of a normally distributed quantity. Student’s t-test – the same as a z-test, but for small samples (t < 100).

All measurements contain errors. This is a fact, get over it. Errors themselves should be noted and not considered errors as such (I’ll explain how we can monitor this in a later chapter).

In his 1925 monograph Statistical Methods for Research Workers, Ronald Fisher (the founder of hypothesis testing) outlined concepts such as the statistical significance criterion, the rules for testing statistical hypotheses, analysis of variance, and experiment planning. This work defined our current approach to experiment planning.

The go-to alternative for non-normal data is nonparametric tests.

One problem with all of these tests is that they are distribution-specific. For example, the Student’s t-test and the z-test require normally distributed data.

A hypothesis is an idea for how to improve a product.

I believe it is better to use one-sided hypotheses. After testing an idea, we try to improve the metric. Here, we are interested in whether it has improved or not (Hypothesis H1).

In data analysis, survival bias is taking the known into account while neglecting the unknown (which nevertheless exists).