The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
The fact that most individual neurons are uninterpretable presents a serious roadblock to a mechanistic understanding of language models. We demonstrate a method for decomposing groups of neurons into interpretable features with the potential to move past that roadblock.
Powerful AI systems can help us interpret the neurons of weaker AI systems. And those interpretability insights often tell us a bit about how models work. And when they tell us how models work, they often suggest ways that those models could be better or more efficient. —Dario Amodei, Anthropic
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
Nicolay Gerold added
It's a little bit of a conundrum. A model we do not understand explains another model we do not understand.
We currently don't understand how to make sense of the neural activity within language models. Today, we are sharing improved methods for finding a large number of "features"—patterns of activity that we hope are human interpretable.
Hagen Peters added
“testing with concept activation vectors,” or TCAV, which offers a way to use such human concepts to understand the internal workings of the network.
Brian Christian • The Alignment Problem
collective neuroscience involves reckoning with the science of complex systems, where causation is not linear but looping, and social and neuronal structures mesh in unpredictable ways.
Sofia Quaglia • How the brains of social animals synchronise and expand one another
Keely Adler added
IMO @AnthropicAI is very close to making a breakthrough in productizable interpretability.
For ~4 years all we've had to really control LLMs is temperature/top_p and logit bias. We recently got `seed` and constrained structured output, with `interactive=false` on the way.
But now Claude Sonnet can let people clamp up/down 34 million features rang... See more
Nathan Storey added
Generative AI is quite good at certain parts of the value chain of knowledge work, but thinks quite differently from humans. Anthropic, a company that focuses on understanding how AI works, has found that humans working side-by-side with expert AI assistants to perform various tasks produce superior performance compared to either the AI or a human
... See moreNoah Smith • Generative AI: autocomplete for everything
sari added
The size and sophistication of these models make it impossible to fully understand how they make predictions. They have billions of weights that interact in subtle ways to make a final prediction; looking at them is like hoping to understand psychology by examining individual brain cells. Even in academic settings with well-established models, the ... See more
future.com • How Recommendation Algorithms Actually Work | Future - https://future.com/forget-open-source-algorithms-focus-on-experiments-instead
Tom So added
Understanding is beyond GPT-3’s reach because understanding cannot occur in an isolated behavior, no matter how clever. Understanding is not an act but a labor. Labor is entirely irrelevant to a computational model that has no history or trajectory; a tool that endlessly simulates meaning anew from a pool of data untethered to its previous efforts.... See more
dailynous.com • Philosophers on GPT-3 (
Kasper Jordaens added