GitHub - mit-han-lab/streaming-llm: Efficient Streaming Language Models with Attention Sinks

StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.

In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup.