What Builders Talk About When They Talk About AI | Andreessen Horowitz
a lot of the focus today is on the development of foundational large language models (LLMs), the transformer architecture was invented only 6 years ago, and ChatGPT was released less than a year ago. It will likely take years, or even decades, before we have a full tech stack for generative AI and LLMs and a host of transformative applications—thou... See more
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
My basic view is that inference will not get that much more expensive. The basic logic of the scaling laws is that if you increase compute by a factor of n, you need to increase data by a factor of the square root of n and the size of the model by a factor of square root of n. That square root basically means that the model itself does not get that... See more
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
I personally believe when it comes to any creative asset or work automation, the demand is elastic. The more that we make, the more people consume. We’re very much looking forward to a massive expansion in productivity, a lot of new jobs, a lot of new things, just like we saw with the microchip and the internet.
—Martin Casado, a16z
—Martin Casado, a16z
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
Getting models into the hands of users will help us discover new use cases
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
how can you take the knowledge work that someone is doing and use AI to help them be dramatically more productive at doing that particular flavor of cognitive work? In our observation with developers, more than anything else, AI helps keep them in flow state longer than they otherwise would. Rather than hitting a blocker when you’re writing a chunk... See more
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
Powerful AI systems can help us interpret the neurons of weaker AI systems. And those interpretability insights often tell us a bit about how models work. And when they tell us how models work, they often suggest ways that those models could be better or more efficient. —Dario Amodei, Anthropic
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
It's a little bit of a conundrum. A model we do not understand explains another model we do not understand.
I have a Turing test question for AI: if we took AI in 1633 and trained on all the available information at that time, would it predict that the Earth or the sun is the center of the solar system—even though 99.9% of the information is saying the Earth is the center of the solar system? I think 5 years is right at the fringe, but if we were to run ... See more
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
There was a study out there that did this "time-line" training.
But just like the internet, someone will show up later and think about something like Uber and cab driving. Someone else showed up and thought, “hey, I wanna check out my friends on Facebook.” Those end up being huge businesses, and it’s not just going to be one model that OpenAI or Databricks or Anthropic or someone builds, and that model will dom... See more
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
Not sure about this one. Just an interesting snippet. I figure there are some reinforcing loops in the data, where the models get better with more data, attracting more users, generating more data. At the same time, I believe there are huge advantages in the knowledge on how to train and how to manage inference at scale, which makes a huge difference. I do not see anyone catching up to OpenAI at the moment, especially with their new finetuning offer.
An interesting factor might be figuring out the right data mix for pre-training and using a better screeing to weed out unwanted behavior. Whoever can figure that out at scale might have a huge advantage, if they can keep it a secret.
I really think that we could be entering a third epoch of computing. The microchip brought the marginal cost of compute to 0. The internet brought the marginal cost of distribution to 0. These large models actually bring the marginal cost of creation to 0.
Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz
Not just the cost of creation but also the cost of intelligence.