Long-Context Retrieval Models with Monarch Mixer
hazyresearch.stanford.edu
Long-Context Retrieval Models with Monarch Mixer
The limited context window is one of the biggest limitations of LLMs today. The reasoning capabilities are great but the relevant knowledge it can reason with is very limited.
Question: nobody understands tokens. Let's talk about a regular page. How many pages of context can it uses?
One of the focus areas at Together Research is new architectures for long context, improved training, and inference performance over the Transformer architecture. Spinning out of a research program from our team and academic collaborators, with roots in signal processing-inspired sequence models, we are excited to introduce the StripedHyena models.... See more
Paving the way to efficient architectures: StripedHyena-7B, open source models offering a glimpse into a world beyond Transformers
As a quick TLDR, there are two key aspects to understanding how REFRAG works:
1. The particular way REFRAG represents context tokens and injects them into LLM decoding, as well as how this speeds up LLM inference. ⚡️
2. The training algorithm used to align the encoder, projection layer,... See more