Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

culture of “test everything” and the limiting factor becomes its ability to convert ideas into code

The terms “data-informed” or “data-aware” are sometimes used to avoid the implication that a single source of data (e.g., a controlled experiment) “drives” the decisions

Dan McKinley at Etsy (McKinley 2013) wrote “nearly everything fails” and for features, he wrote “it’s been humbling to realize how rare it is for them to succeed on the first attempt.

identify metrics that the team can affect today, but which, ultimately, will affect the firm’s long-term goals.”

Slack’s Director of Product and Lifecycle tweeted that with all of Slack’s experience, only about 30% of monetization experiments show positive results; “if you are on an experiment-driven team, get used to, at best, 70% of your work being thrown away.

organization may reject new knowledge that is contradictory per the Semmelweis Reflex

If you have multiple metrics, one possibility proposed by Roy (2001) is to normalize each metric to a predefined range, say 0–1, and assign each a weight. Your OEC is the weighted sum of the normalized metrics.

It is often easier to generate a plan, execute against it, and declare success, with the key metric being: “percent of plan delivered,” ignoring whether the feature has any positive impact to key metrics.

many high-risk/high-reward ideas do not succeed on the first iteration, and learning from failures is critical for the refinement needed to nurture these ideas to success,