Do Machine Learning Models Memorize or Generalize?
pair.withgoogle.com
Do Machine Learning Models Memorize or Generalize?
Something I think people continue to have poor intuition for: The space of intelligences is large and animal intelligence (the only kind we've ever known) is only a single point, arising from a very specific kind of optimization that is fundamentally distinct from that of our technology.
Animal intelligence optimization... See more
Andrej Karpathyx.comThe most interesting part for me is where @karpathy describes why LLMs aren't able to learn like humans.
As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits through a straw.”
A single end reward gets broadcast across every token in... See more
Dwarkesh Patelx.com
Differences between AI and human learning, focusing on batch training, lack of continual learning, and catastrophic forgetting in AI.
TRANSCRIPT
So mechanically, there seems to clearly be something different, or at least there's a big community of people in the computational neuroscience community that agree that there's got to be something different going on between these systems.
More abstractly, though, what's fundamentally different, which I think is maybe more practically
... See more