Metrics, Incrementalism, and Local Maxima

We optimize for proxy metrics, because we don’t have a more accurate way to measure goals.  In education, we seem at best capable of measuring test scores, not actual aptitude. In media, we use time spent as a proxy for customer value.

Tight feedback loops are excellent tools for optimizing efficiency (eg, of understood production processes, operations, or distribution funnels). But if we’re dogmatic in adhering to a fixed feedback loop periodicity, we preclude ourselves from undertaking larger scale projects (the space program that reached the moon does not get fu

Silicon Valley venture capitalists develop the lean startup “release early, release often” mantra as a way to de-risk investments and avoid investing significant capital without gauging market demand, resulting in narrowing / more incremental innovation. Individual creatives might bias towards smaller projects to de-risk the possibility of inv

In K-12 education, it’s nearly impossible to scale metrics of teacher efficacy without very short term measurement of student performance on standardized tests that don’t map very well to the long term value of education.

The basic failure mode I’ve been observing recently is a tendency to get trapped in local maxima because it is easier to identify and measure metrics for short term wins than for long term investments.

There’s a lot to love about metrics driven systems: they’re fairly meritocratic, they reward intelligence and hard work, they align (at least in some way) the incentives of producers and consumers, and they accelerate really quickly once they start working.

But, using this framework to understand the systemic failure at the root of a lot of wide sca