🧠AI’s $100bn question: The scaling ceiling
In 2020, Jared Kaplan and his collaborators at OpenAI suggested that there was a set of “scaling laws” for neural network models of language; they found that the more data they fed into their neural networks, the better those networks performed.10 The implication was that we could do better and better AI if we gather more data and apply deep learni... See more
Gary Marcus • Deep Learning Is Hitting a Wall
Prashanth Narayan added
How did we get to the doorstep of the next leap in prosperity?
In three words: deep learning worked.
In 15 words: deep learning worked, got predictably better with scale, and we dedicated increasing resources to it.
That’s really it; humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “ru... See more
In three words: deep learning worked.
In 15 words: deep learning worked, got predictably better with scale, and we dedicated increasing resources to it.
That’s really it; humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “ru... See more
samaltman.com • The Intelligence Age
How did we get to the doorstep of the next leap in prosperity?
In three words: deep learning worked.
In 15 words: deep learning worked, got predictably better with scale, and we dedicated increasing resources to it.
That’s really it; humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “rules” that produce any distribution of data). To a shocking degree of precision, the more compute and data available, the better it gets at helping people solve hard problems. I find that no matter how much time I spend thinking about this, I can never really internalize how consequential it is.
Indeed, we may already be running into scaling limits in deep learning, perhaps already approaching a point of diminishing returns. In the last several months, research from DeepMind and elsewhere on models even larger than GPT-3 have shown that scaling starts to falter on some measures, such as toxicity, truthfulness, reasoning, and common sense
Gary Marcus • Deep Learning Is Hitting a Wall
Prashanth Narayan added
Prashanth Narayan and added
The reason experts are so split on whether or not we're on an exponential path to AI is that intelligence is multi-dimensional. You can be both exponentially exploding/scaling in one dimension and stagnating in another. You can be both superhuman and subhuman at the same time. People tend to focus on the dimensions most salient to their inclination... See more
sari added
good take
But just like the internet, someone will show up later and think about something like Uber and cab driving. Someone else showed up and thought, “hey, I wanna check out my friends on Facebook.” Those end up being huge businesses, and it’s not just going to be one model that OpenAI or Databricks or Anthropic or someone builds, and that model will dom... See more
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
Nicolay Gerold added
Not sure about this one. Just an interesting snippet. I figure there are some reinforcing loops in the data, where the models get better with more data, attracting more users, generating more data. At the same time, I believe there are huge advantages in the knowledge on how to train and how to manage inference at scale, which makes a huge difference. I do not see anyone catching up to OpenAI at the moment, especially with their new finetuning offer.
An interesting factor might be figuring out the right data mix for pre-training and using a better screeing to weed out unwanted behavior. Whoever can figure that out at scale might have a huge advantage, if they can keep it a secret.
**Note:** AI as religion - something beyond humans’ comprehension that we’re putting faith in“If the human brain were so simple that we could understand it, we would be so simple that we couldn’t.” This statement makes intuitive sense, but, more importantly, we can point to a concrete example in support of it: the microscopic roundworm C. elegans. ... See more
newyorker.com • Why Computers Won’t Make Themselves Smarter
Alex Wittenberg added