LLMs

"A key challenge of (LLMs) is that they do not come with a manual! They come with a “Twitter influencer manual” instead, where lots of people online loudly boast about the things they can do with a very low accuracy rate, which is really frustrating..."

Simon Willison, attempting to explain LLM

Meta AI released LLaMA ... and they included a paper which described exactly what it was trained on. It was 5TB of data.

2/3 of it was from Common Crawl. It had content from GitHub, Wikipedia, ArXiv, StackExchange and something called “Books”.

What’s Books? 4.5% of the training data was books. Part of this was Project Gutenberg, which is public

“A more practical answer is that it’s a file. This right here is a large language model, called Vicuna 7B. It’s a 4.2 gigabyte file on my computer. If you open the file, it’s just numbers. These things are giant binary blobs of numbers…”
Simon Willison, attempting to explain LLM