Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
this type of review starts being effective in the late Walk or in the Run phases of maturity.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
organization may reject new knowledge that is contradictory per the Semmelweis Reflex
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
always having a portfolio of ideas: most should be investments in attempting to optimize “near” the current location, but a few radical ideas should be tried to see whether those jumps lead to a bigger hill.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
more sensitive variants can be great alternatives, such as revenue indicator-per-user (was there revenue for user: yes/no),
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
For k = 5, you have a 23% probability of seeing something statistically significant. For k = 10, that probability rises to 40%.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
When you have an online business, you will have several key goal and driver metrics, typically measuring user engagement (e.g., active days, sessions-per-user, clicks- per-user) and monetary value (e.g., revenue-per-user). There is usually no simple single metric to optimize for.
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
many high-risk/high-reward ideas do not succeed on the first iteration, and learning from failures is critical for the refinement needed to nurture these ideas to success,
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
culture of “test everything” and the limiting factor becomes its ability to convert ideas into code
Ya Xu • Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing
EVI: Expected Value of Information from Douglas Hubbard (2014), which captures how additional information can help you in decision making. The ability to run controlled experiments allows you to significantly reduce uncertainty by trying a Minimum Viable Product (Ries 2011), gathering data, and iterating.