AgentBench: Evaluating LLMs as Agents

LLM’s responses can be volatile and unreliable, as slight changes in the prompt can lead to significant variations in the output. Occasionally, the LLM may generate irrelevant or nonsensical information, necessitating ongoing improvements. Additionally, the context window has limited capacity, which can impact our agents’ understanding of complex s... See more

We’re not even on the agency curve. It’s a different curve.

Look, I’m using knowledge and agency a little loosely. LLMs can do more than just regurgitate knowledge; even if you grant them reasoning, though, it still doesn’t feel like humanity emerges from LLM scaleup. They have no curiosity, and don’t ask their own questions.

Dwarkesh Patel Richard Sutton’s “Bitter Lesson” essay64 says that there are two things you can scale: search and learning. LLMs are about the learning aspect. You’ve worked on search throughout your career, where you have an agent interacting with an environment.65 Is that the direction that needs to be explored again, or is that something that nee

... See more