Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
“Profit” is not a good OEC, as short-term theatrics (e.g., raising prices) can increase short-term profit, but may hurt it in the long run. Customer lifetime value is a strategically powerful OEC (Kohavi, Longbottom et al. 2009).
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
One rough rule of thumb is to try to limit your key metrics to five.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
If you have multiple metrics, one possibility proposed by Roy (2001) is to normalize each metric to a predefined range, say 0–1, and assign each a weight. Your OEC is the weighted sum of the normalized metrics.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
organizations and cultures go through stages in learning to experiment as well
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
when in doubt, measure more, but more importantly: think hard about what you are optimizing for.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
identify metrics that the team can affect today, but which, ultimately, will affect the firm’s long-term goals.”
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
The terms “data-informed” or “data-aware” are sometimes used to avoid the implication that a single source of data (e.g., a controlled experiment) “drives” the decisions
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
always having a portfolio of ideas: most should be investments in attempting to optimize “near” the current location, but a few radical ideas should be tried to see whether those jumps lead to a bigger hill.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
ideally you may want to combine them into an Overall Evaluation Criterion (OEC), which is believed to causally impact long-term objectives.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Controlled experiments are especially useful in combination with Agile software development (Martin 2008, K. S. Rubin 2012), Customer Development process (Blank 2005), and MVPs (Minimum Viable Products),