Too Much Efficiency Makes Everything Worse: Overfitting and the Strong Version of Goodhart's Law
Jascha Sohl-Dicksteinsohl-dickstein.github.ioSaved by sari and
Too Much Efficiency Makes Everything Worse: Overfitting and the Strong Version of Goodhart's Law
In the high modernist world we live in, we can adapt Drucker’s dictum that “whatever is measured improves” to a more true-to-reality, but less elegant “whatever is legible improves, unless there are second-order consequences which undermine or destroy the original intention.”
taylorpearson.me • The Illegible Margin: Profiting From the Gap Between the Map and the Territory
There’s a lot to love about metrics driven systems: they’re fairly meritocratic, they reward intelligence and hard work, they align (at least in some way) the incentives of producers and consumers, and they accelerate really quickly once they start working.
But, using this framework to understand the systemic failure at the root of a lot of wide sca
... See moreAndrew Kortina • Metrics, Incrementalism, and Local Maxima
The problem of sparsity also has safety implications. If, in some future time, you’re developing a superintelligent AI with vast capabilities, driven by ε-greedy reinforcement learning, and you decide you’ll reward it with one point if it cures cancer and zero otherwise, watch out—because it will have to try a lot of random things before it stumble
... See moreBrian Christian • The Alignment Problem
Unfortunately, metrics have an awfully hard time representing human goals. We humans are messy, fuzzy things, and often our goals cannot be reduced down to one or even a few quantifiable metrics.
DAOs, Tokens, and Goodhart's Forest
Math is good at optimizing a known system; humans are good at finding a new one. Put another way, change favors local maxima; innovation favors global disruption.
Alistair Croll • Lean Analytics: Use Data to Build a Better Startup Faster (Lean (O'Reilly))
