Measuring AI Ability to Complete Long Tasks
metr.org
Measuring AI Ability to Complete Long Tasks
Moravec's paradox in LLM evals
I was reacting to this new benchmark of frontier math where LLMs only solve 2%. It was introduced because LLMs are increasingly crushing existing math benchmarks. The interesting issue is that even though by many accounts (/evals), LLMs are inching well into top expert territory (e.g. in... See more
Andrej Karpathyx.comThe length of tasks (measured by how long they take human professionals) that generalist autonomous frontier model agents can complete with 50% reliability has been doubling approximately every 7 months for the last 6 years. The trend predicts that in under a decade we will see AI agents that can independently complete a large fraction of tasks in... See more
One problem was that tests previously seen as requiring intelligence, such as playing chess or recognizing speech, have now been dismissed as only requiring heuristics, shortcuts, not true intelligence, now that AI has mastered them. So, people asked, what evaluation would we set up ahead of time that everyone would agree on and that we wouldn’t... See more
Saffron Huang • Complex Systems
We are not used to thinking in this way—to asking “how much does being smarter help with this task, and on what timescale?”—but it seems like the right way to conceptualize a world with very powerful AI.