Paving the way to efficient architectures: StripedHyena-7B, open source models offering a glimpse into a world beyond Transformers

Phi-1.5

Phi-1.5 is a "small" 1.3 billion parameter LLM with an impressive performance for its size.

Annotated figures from the Textbooks Is All You Need II paper

How does this small model accomplish such a good performance? The secret ingredient seems to be the high-quality data.

The pretraining is based on the Textbooks Is All You Need approach that... See more

1. Nearly every company in the Sequoia network is building language models into their products.

We currently don't understand how to make sense of the neural activity within language models. Today, we are sharing improved methods for finding a large number of "features"—patterns of activity that we hope are human interpretable.

This second Scaling Law led to the creation of Reasoners, which I wrote about in my last post. The new generation of Gen3 models can all operate as Reasoners when needed, so they have two advantages: larger scale in training, and the ability to scale when actually solving a problem.