GitHub - mit-han-lab/streaming-llm: Efficient Streaming Language Models with Attention Sinks

There is often a notable gap between state of the art research and what practitioners can reasonably use. However, I'm glad to say that attention sinks can be added to any pretrained LLM at near to no additional effort.

I have released the attention_sinks Python module, which acts as a drop-in replacement for the transformers API. This Python module... See more