Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

Using experiments as a guardrail is a difficult cultural change,

this type of review starts being effective in the late Walk or in the Run phases of maturity.

Controlled experiments are especially useful in combination with Agile software development (Martin 2008, K. S. Rubin 2012), Customer Development process (Blank 2005), and MVPs (Minimum Viable Products),

more sensitive variants can be great alternatives, such as revenue indicator-per-user (was there revenue for user: yes/no),

EVI: Expected Value of Information from Douglas Hubbard (2014), which captures how additional information can help you in decision making. The ability to run controlled experiments allows you to significantly reduce uncertainty by trying a Minimum Viable Product (Ries 2011), gathering data, and iterating.

When you have an online business, you will have several key goal and driver metrics, typically measuring user engagement (e.g., active days, sessions-per-user, clicks- per-user) and monetary value (e.g., revenue-per-user). There is usually no simple single metric to optimize for.

organizations and cultures go through stages in learning to experiment as well

In general, for experimentation, you will be choosing the subset of business goal, driver, and organizational guardrail metrics that meet these measurability, computability, sensitivity, and timeliness characteristics.

culture of “test everything” and the limiting factor becomes its ability to convert ideas into code