Why large language models struggle with long contexts

A very common trope is to treat LLMs as if they were intelligent agents going out in the world and doing things. That’s just a category mistake. A much better way of thinking about them is as a technology that allows humans to access information from many other humans and use that information to make decisions. We have been doing this for as long a... See more

I have long been a subscriber to the idea that any intelligence needs to be embodied and situated. The very first problems a baby has to solve are, How do I move my body? How do I move around in the world? Those tasks are entirely missing from existing large language models. And, as many people noted, the phrase “large language model” is a misnomer... See more

Pre-trained language models (LM), by contrast, are capable of directly generating prose that may be responsive to an information need, but at present they are *dilettantes* rather than domain experts – they do not have a true understanding of the world, they are prone to hallucinating, and crucially they are incapable of justifying their utterances... See more

We are entering an era where someone might use a large language model to generate a document out of a bulleted list, and send it to a person who will use a large language model to condense that document into a bulleted list. Can anyone seriously argue that this is an improvement?