Serim Tarcan

A look back at AlphaGo—the first AI system that beat the world champions at the game of Go, decades before it was thought possible—is useful here as well.

• In step 1, AlphaGo was trained by imitation learning on expert human Go games. This gave it a foundation.


• In step 2, AlphaGo played millions of games against itself. This let it become superhuman a

... See more

In addition to insider bullishness, I think there’s a strong intuitive case for why it should be possible to find ways to train models with much better sample efficiency (algorithmic improvements that let them learn more from limited data). Consider how you or I would learn from a really dense math textbook:

• What a modern LLM does during training i

... See more

You can go somewhat further by repeating data, but academic work on this suggests that repetition only gets you so far, finding that after 16 epochs (a 16-fold repetition), returns diminish extremely fast to nil. At some point, even with more (effective) compute, making your models better can become much tougher because of the data constraint. This... See more

There is a potentially important source of variance for all of this: we’re running out of internet data. That could mean that, very soon, the naive approach to pretraining larger language models on more scraped data could start hitting serious bottlenecks.

Frontier models are already trained on much of the internet. Llama 3, for example, was traine... See more

We can decompose the progress in the four years from GPT-2 to GPT-4 into three categories of scaleups:

1. Compute : We’re using much bigger computers to train these models.


2. Algorithmic efficiencies : There’s a continuous trend of algorithmic progress. Many of these act as “compute multipliers,” and we can put them on a unified scale of growing effecti

... See more

Every act of conscious learning requires the willingness to suffer an injury to one's self-esteem.

That is why young children, before they are aware of their own self-importance, learn so easily; and why older persons, especially if vain or important, cannot learn at all.

Anxiety is an emotional response to risk. When we experience anxiety about an event, we perceive the risk to be real and possible, likely to occur in the near term.

Epistemic Anxiety is the feeling of uneasiness, tension, and concern when you want to know the truth. You worry that your knowledge is incomplete and full of errors, and you may believe ... See more

Where public intellectuals once shaped cultural discourse by challenging ideas, most influential figures today shape culture by modeling a lifestyle.

According to data gathered by Jason Miller, a professor at Michigan State University who specializes in supply-chain management, China produces more than 70 percent of the world’s lithium-ion batteries, air conditioners, and cookware; more than 80 percent of the world’s smartphones, kitchen appliances, and toys; and about 90 percent of the world’s ... See more