updated 9mo ago
What Builders Talk About When They Talk About AI | Andreessen Horowitz
- Powerful AI systems can help us interpret the neurons of weaker AI systems. And those interpretability insights often tell us a bit about how models work. And when they tell us how models work, they often suggest ways that those models could be better or more efficient. —Dario Amodei, Anthropic
from What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago
It's a little bit of a conundrum. A model we do not understand explains another model we do not understand. - When folks in these communities have access to very powerful tools, they tend to do remarkable things that create economic opportunities for themselves, their families, and their communities. They solve problems that you or I are not going to solve just because we don’t see the whole problem landscape of the world. We don’t have their point of view... See more
from What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago
- We have lots and lots of customers who want to have specialized models that are cheaper, smaller, and have really high accuracy and performance. They’re saying, “Hey, this is what I want to do. I want to classify this particular defect in the manufacturing process from these pictures really well.” And there, the accuracy matters. Every ounce of acc... See more
from What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago
I think ChatGPT will bring a bunch of companies really into AI. They will start out trying to solve things with GPT and then try to figure out how to solve this better with bespoke, smaller model. It could trigger a new AI summer for deep learning. If you made a thousand versions of an LLM, that’s good at a thousand different things, and you have to load each of those into the GPUs and serve them, it becomes very expensive. The big holy grail right now that everybody’s looking for is: are there techniques, where you can just do small modifications where you can get really good results? There
... See morefrom What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago
PEFT in a nutshell.- I have a Turing test question for AI: if we took AI in 1633 and trained on all the available information at that time, would it predict that the Earth or the sun is the center of the solar system—even though 99.9% of the information is saying the Earth is the center of the solar system? I think 5 years is right at the fringe, but if we were to run ... See more
from What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago
- My basic view is that inference will not get that much more expensive. The basic logic of the scaling laws is that if you increase compute by a factor of n, you need to increase data by a factor of the square root of n and the size of the model by a factor of square root of n. That square root basically means that the model itself does not get that... See more
from What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago
- how can you take the knowledge work that someone is doing and use AI to help them be dramatically more productive at doing that particular flavor of cognitive work? In our observation with developers, more than anything else, AI helps keep them in flow state longer than they otherwise would. Rather than hitting a blocker when you’re writing a chunk... See more
from What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago
- But just like the internet, someone will show up later and think about something like Uber and cab driving. Someone else showed up and thought, “hey, I wanna check out my friends on Facebook.” Those end up being huge businesses, and it’s not just going to be one model that OpenAI or Databricks or Anthropic or someone builds, and that model will dom... See more
from What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago
Not sure about this one. Just an interesting snippet. I figure there are some reinforcing loops in the data, where the models get better with more data, attracting more users, generating more data. At the same time, I believe there are huge advantages in the knowledge on how to train and how to manage inference at scale, which makes a huge difference. I do not see anyone catching up to OpenAI at the moment, especially with their new finetuning offer. An interesting factor might be figuring out the right data mix for pre-training and using a better screeing to weed out unwanted behavior. Whoever can figure that out at scale might have a huge advantage, if they can keep it a secret. - Sometimes they just need the model that actually fits for their specific use case and it’s far more economical. We want people to build on top of our models and we want to give them tools to make that easy. We want to give them more and more access and control, so you can bring your data and customize these models. And you can really focus on the l... See more
from What Builders Talk About When They Talk About AI | Andreessen Horowitz by Sarah Wang
Nicolay Gerold added 9mo ago