In 2021, researchers made a striking discovery while training a series of tiny models on toy tasks [1]. They found a set of models that suddenly flipped from memorizing their training data to correctly generalizing on unseen inputs after training for much longer. This phenomenon – where generalization seems to happen abruptly and long after fitting... See more
Observing these patterns of influence gives clues about how our models generalize from their training data. For instance, if the models responded to user prompts by splicing together sequences from the training set, then we’d expect the influential sequences for a given model response to include expressions of near-identical thoughts. Conversely,... See more
Memorization vs learning concepts and forming models of the world.
Artificial intelligence is already killing off important parts of the human experience. But one of its most consequential murders—so far—is the demise of a longstanding rite of passage for students worldwide: an attempt to synthesize complex information and condense it into compelling analytical prose. It’s a training ground for the most... See more
The more I use language models, the more monstrous they seem to me. I don’t mean that in a particularly negative sense. Frankenstein’s monster is sad, but also amazing. Godzilla is a monster, and Godzilla rules.
We humans aren't the only ones to experience cognitive decline as we age. Research conducted by neurologists and data scientists reveals that some artificial intelligence models struggle with visual, spatial, and executive tasks as they get older, suggesting that they, too, age away from peak performance. The study highlights an unanticipated... See more