Choosing your metrics, KPIs, etc...
how to measure what matters + philosophical musings on whether the important things can be measured
sari and
Choosing your metrics, KPIs, etc...
Too Much Efficiency Makes Everything Worse: Overfitting and the Strong Version of Goodhart's Law
Jascha Sohl-Dicksteinsohl-dickstein.github.ioFrom Andy Barnes:
The idea that all of the world should be measured in dollars to stockholders is actually a relatively new idea. It used to be that we thought that businesses had their purpose. Your purpose was to be making newspapers or fountain pens or whatever. And now we act as though the only purpose of a... See more
But ambiguity, taste, meaning, value, these remain hard to attain within the hierarchy of verifiability. “Intelligence” races ahead wherever the rewards are liquid and the outcome is easy to check, but it lags at the edges: in murky context, slow feedback, or spaces where the answer is more felt than measured.
While it’s tempting to believe a model... See more
While it’s tempting to believe a model... See more
Market of words
listen for truths that language cannot name
How did people respond? Did I strike an emotional nerve? This goes beyond easy, legible metrics like pageviews or social media likes, or even comments, which are at best hazy approximations of external resonance. It’s about looking for signals that something genuinely MATTERED to one or more humans, and elicited a response that’s out of proportion with the average digital interaction. Lots of likes is an okay-ish signal. Lots of comments is a clearer signal. A small handful of comments or private replies from people saying they’ve never felt so seen or understood by a piece of writing – that’s the kind of thing I’m trying to discern and quantify here.
The concept of efficiency — how much one can accomplish per unit of time (or per dollar, etc.) — requires a quantitative numerator as well as a denominator. It requires a metric. Therefore, it tells us nothing about results we cannot quantify or measure. When we gear our society around efficiency, we produce more and more of the measurable, while... See more
Value capture occurs when an agent’s values are rich and subtle; they enter a social environment that presents simplified — typically quantified — versions of those val-ues; and those simplified articulations come to dominate their practical reasoning. Examples include becoming motivated by FitBit’s step counts, Twitter Likes and Re-tweets,... See more
PhilPapers
excellent work by C. Thi Nguyen
There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we... See more