A common experimentation mistake: discarding a newly launched experience (like an onboarding flow) because it underperforms the old version.
The old version has been optimized over the course of many smaller experiments to a local maximum.
The new version is raw and unoptimized https://t.co/crroZUuEsy
A common experimentation mistake: discarding a newly launched experience (like an onboarding flow) because it underperforms the old version. The old version has been optimized over the course of many smaller experiments to a local maximum. The new version is raw and unoptimized https://t.co/crroZUuEsy
That the elements of a new product you launch have more gravity than you think. I’ve had dozens of debates about what an MVP should and shouldn’t include. The argument to “just get something out there and start learning” is flawed in two critical ways: (1) You’ll burn early adopters fast if you don’t polish the few things that distinguish your prod... See more
Scott Belsky • What Is “Seeing the Matrix” for a Product Leader?
online marketers argue that complete redesigns deny them so-called “learnings” about which individual elements contributed the most to the improved performance. This is based on the flawed assumption that the individual elements are completely independent of one another. In fact, they are often highly dependent on the context in which they are pres
... See moreMaura Ginty • Landing Page Optimization: The Definitive Guide to Testing and Tuning for Conversions
He began to understand that the initial baseline of usage the team had set wasn’t consistent with how engaged customers were using the product. That doesn’t mean the team shouldn’t have taken a guess. Without that initial line in the sand, they would have had no benchmark for learning, and Kyle might not have picked up the phone. But now he really
... See moreAlistair Croll • Lean Analytics: Use Data to Build a Better Startup Faster (Lean (O'Reilly))
A new era of quant experimentation
In trading, the objective is to place a series of trades to increase the value of a portfolio. This starts with market research. We use indicators—such as company fundamentals, volatility, and momentum—to deepen our understanding. As we learn about the market, we start to identify patterns and develop hypotheses a... See more
In trading, the objective is to place a series of trades to increase the value of a portfolio. This starts with market research. We use indicators—such as company fundamentals, volatility, and momentum—to deepen our understanding. As we learn about the market, we start to identify patterns and develop hypotheses a... See more
Josh Payne • The New Science of Growth Marketing
Trying to rationalize results after the test is a dangerous activity because it may cause you to inappropriately fixate on elements of your design that had nothing to do with the performance improvement.
Maura Ginty • Landing Page Optimization: The Definitive Guide to Testing and Tuning for Conversions
The main lesson we’ve learned is that you can almost never optimize directly for a goal—instead, you’re usually optimizing for something that looks like your goal but is slightly different. It’s a proxy.
Because you have to optimize for a proxy, when you optimize too much, you get too good at maximizing your proxy objective—which often takes you far... See more
Because you have to optimize for a proxy, when you optimize too much, you get too good at maximizing your proxy objective—which often takes you far... See more