Working with LLMs: A Few Lessons
treat compute as COGS, UX as moat, and rapid iteration as life support.
Rohit Krishnan • Working with LLMs: A Few Lessons
They look like software, but they act like people. And just like people, you can’t just hire someone and pop them on a seat, you have to train them. And create systems around them to make the outputs verifiable.
Which means there’s a pareto frontier of the number of LLM calls you’ll need ot make for verification and the error-rate each LLM introduce... See more
Which means there’s a pareto frontier of the number of LLM calls you’ll need ot make for verification and the error-rate each LLM introduce... See more
Rohit Krishnan • Working with LLMs: A Few Lessons
This means you have to add evaluation frameworks, human-in-the-loop processes, designing for graceful failure, using LLMs for probabilistic guidance rather than deterministic answers, or all of the above, and hope they catch most of what you care about, but know things will still slip through.
Working with LLMs: A Few Lessons
LLMs inherently are probabilistic. This is unlike code that we’re used to running before. That’s why using an LLM can be so cool, because they can do different things.