Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

experimentation plan required a more holistic approach that considered how hypotheses informed each other, the sites on which they were run, goals and metrics linked to business outcomes, projected sample sizes, implementation steps, and so on. Most of all, a good plan allowed for iterations and led to the exploration and optimization of much bolde

Companies can now base important decisions on real, scientifically valid experiments. In the past, any foray into randomized testing (the random assignment to groups that we mentioned above) meant employing or engaging a PhD in statistics or a “design of experiments” expert.

By combining the power of software and the rigor of controlled experiments, companies can turn themselves into learning organizations—turbocharged! But to unleash this power, you need to build an experimentation organization that masters the science of testing and has a culture, processes, and management system that contradicts what we value today.

One vice president of e-commerce observed that convincing senior leaders of the importance of online testing was the biggest obstacle to scaling experiments.