Long-Context Retrieval Models with Monarch Mixer
hazyresearch.stanford.edu
Long-Context Retrieval Models with Monarch Mixer
in order to have research agents that can run for days, we need context compaction
i used RL to have LLMs naturally learn their own 10x compression! Qwen learned to pack more info per token (ie use Mandarin tokens, prune text)
read the technical blog: https://t.co/3pzRvt4zvr
rajan agarwalx.comAdvances in data processing techniques. You can increase context length in two ways. First, you can train the model with longer context lengths. That’s difficult because it’s much more computationally expensive, and it’s hard to find datasets with long context lengths (most documents in CommonCrawl have fewer than 2,000 tokens).
The second, more... See more
The second, more... See more