Bootstrapping is one of the most interesting ways to evaluate metrics in A/B tests. It is one of our favoured methods at Retail Rocket for continuous parameters such as average purchase value, average cost of goods and revenue per visitor (RPV).

Evolutionary hypotheses, where one parameter is slightly optimized, have a less profound effect than revolutionary hypotheses, where the approach is fundamentally different. That said, evolutionary hypotheses are more likely to bear fruit.

We start out thinking that hypothesis H0 is true until we find evidence to refute it in favour of hypothesis H1.

Z-test – for checking the mean of a normally distributed quantity. Student’s t-test – the same as a z-test, but for small samples (t < 100).

Classical machine learning can be divided into three types: supervised learning unsupervised learning reinforcement learning.

Worried about losing the tender, the other company fudged the results by moving users who were close to making a purchase (for example, those who had already added an item to their carts) to their own system. They weren’t doing this all the time, only on certain days and at certain times.

The purpose of analytics [10] is to help us formulate hypotheses. The purpose of statistics [10] is to test and confirm these hypotheses.

nine out of ten hypotheses don’t pan out. But you have no idea that a hypothesis will not produce the desired result until you are well into the testing process. I believe that it is best to kill a hypothesis as early as possible – as soon as the first sign that the idea won’t take off presents itself.

Statistical hypothesis testing involves two important concepts: general population and sample.