A common experimentation mistake: discarding a newly launched experience (like an onboarding flow) because it underperforms the old version.
The old version has been optimized over the course of many smaller experiments to a local maximum.
The new version is raw and unoptimized https://t.co/crroZUuEsy
A common experimentation mistake: discarding a newly launched experience (like an onboarding flow) because it underperforms the old version. The old version has been optimized over the course of many smaller experiments to a local maximum. The new version is raw and unoptimized https://t.co/crroZUuEsy
Don’t get paralyzed by testing everything. If you can’t get statistical significance within a month, use pre/post analysis instead. Trust intuition more, especially when:
You have low traffic/conversion volumes
The change is low-risk
You need to move quickly
The impact is obvious
You have low traffic/conversion volumes
The change is low-risk
You need to move quickly
The impact is obvious
The main lesson we’ve learned is that you can almost never optimize directly for a goal—instead, you’re usually optimizing for something that looks like your goal but is slightly different. It’s a proxy.
Because you have to optimize for a proxy, when you optimize too much, you get too good at maximizing your proxy objective—which often takes you far... See more
Because you have to optimize for a proxy, when you optimize too much, you get too good at maximizing your proxy objective—which often takes you far... See more
Dan Shipper • The Optimal Level of Optimization
online marketers argue that complete redesigns deny them so-called “learnings” about which individual elements contributed the most to the improved performance. This is based on the flawed assumption that the individual elements are completely independent of one another. In fact, they are often highly dependent on the context in which they are pres
... See moreMaura Ginty • Landing Page Optimization: The Definitive Guide to Testing and Tuning for Conversions
That the elements of a new product you launch have more gravity than you think. I’ve had dozens of debates about what an MVP should and shouldn’t include. The argument to “just get something out there and start learning” is flawed in two critical ways: (1) You’ll burn early adopters fast if you don’t polish the few things that distinguish your prod... See more
Scott Belsky • What Is “Seeing the Matrix” for a Product Leader?
Companies that view their optimization efforts as a one-time project don’t get the best results. They’re easily discouraged if the first test doesn’t give immediate leaps in their conversion rate and miss learning opportunities that can be gained from each experiment.