Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
organizations and cultures go through stages in learning to experiment as well
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
In general, for experimentation, you will be choosing the subset of business goal, driver, and organizational guardrail metrics that meet these measurability, computability, sensitivity, and timeliness characteristics.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
many high-risk/high-reward ideas do not succeed on the first iteration, and learning from failures is critical for the refinement needed to nurture these ideas to success,
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
identify metrics that the team can affect today, but which, ultimately, will affect the firm’s long-term goals.”
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
organization may reject new knowledge that is contradictory per the Semmelweis Reflex
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Controlled experiments are especially useful in combination with Agile software development (Martin 2008, K. S. Rubin 2012), Customer Development process (Blank 2005), and MVPs (Minimum Viable Products),
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
ideally you may want to combine them into an Overall Evaluation Criterion (OEC), which is believed to causally impact long-term objectives.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
One rough rule of thumb is to try to limit your key metrics to five.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
making controlled experiments easy to run also accelerates innovation by decreasing the cost of trying new ideas,
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Having too many metrics may cause cognitive overload and complexity,