Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
this type of review starts being effective in the late Walk or in the Run phases of maturity.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
Controlled experiments are especially useful in combination with Agile software development (Martin 2008, K. S. Rubin 2012), Customer Development process (Blank 2005), and MVPs (Minimum Viable Products),
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
making controlled experiments easy to run also accelerates innovation by decreasing the cost of trying new ideas,
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
It is often easier to generate a plan, execute against it, and declare success, with the key metric being: “percent of plan delivered,” ignoring whether the feature has any positive impact to key metrics.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
organization may reject new knowledge that is contradictory per the Semmelweis Reflex
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
more sensitive variants can be great alternatives, such as revenue indicator-per-user (was there revenue for user: yes/no),
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
In general, for experimentation, you will be choosing the subset of business goal, driver, and organizational guardrail metrics that meet these measurability, computability, sensitivity, and timeliness characteristics.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
EVI: Expected Value of Information from Douglas Hubbard (2014), which captures how additional information can help you in decision making. The ability to run controlled experiments allows you to significantly reduce uncertainty by trying a Minimum Viable Product (Ries 2011), gathering data, and iterating.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
many high-risk/high-reward ideas do not succeed on the first iteration, and learning from failures is critical for the refinement needed to nurture these ideas to success,