Measuring AI Ability to Complete Long Tasks
metr.org
Measuring AI Ability to Complete Long Tasks
Moravec's paradox in LLM evals
I was reacting to this new benchmark of frontier math where LLMs only solve 2%. It was introduced because LLMs are increasingly crushing existing math benchmarks. The interesting issue is that even though by many accounts (/evals), LLMs are inching well into top expert territory (e.g. in... See more
Andrej Karpathyx.comHow close is AI to replacing product managers?
Closer than I expected.
In collaboration with @hammer_mt (a professional prompt engineer and co-author of the O'Reilly book Prompt Engineering for Generative AI), we've begun to develop a benchmark to track how good AI is at the most common and... See more
Lenny Rachitskyx.comThe length of tasks (measured by how long they take human professionals) that generalist autonomous frontier model agents can complete with 50% reliability has been doubling approximately every 7 months for the last 6 years. The trend predicts that in under a decade we will see AI agents that can independently complete a large fraction of tasks in... See more