Long-Context Retrieval Models with Monarch Mixer
hazyresearch.stanford.edu
Long-Context Retrieval Models with Monarch Mixer
By using this repo https://t.co/HOobVEmInD you can convert any decoder model (such as Llama 3.1 or Gemma 2) into an encoder (such as RoBERTa). Why would you want to do that? Because modern decoders support huge context sizes (128k+ tokens) while the longest context of an encoder is 4K (LongFormer), but encoders excel in such tasks as... See more
BURKOVx.com
Document splitting is common for vector storage / retrieval, but useful context can be lost. @LangChainAI has 3 new "context-aware" text splitters that keep metadata about where each split came from. Works for code (py, js) c/o @cristobal_dev, PDFs c/o @CorranMac, and Markdown .. https://t.co/PaWzq33IoM
Why Chat With PDF Is Hard And How ChatLLM Gets It Right
Chatting on long docs is hard because most LLMs other than Gemini don't have a large context.
However, even with Gemini's 1M context length, in-context learning is hard, and if you stuff the doc in the context, it doesn't do a good job.... See more
Bindu Reddyx.com