GitHub - mit-han-lab/streaming-llm: Efficient Streaming Lang...

GitHub - mit-han-lab/streaming-llm: Efficient Streaming Language Models with Attention Sinks

mit-han-lab github.com

Saved by Darren LI and

RelatedCollectionsHighlights

Things we learned about LLMs in 2024

Simon Willison simonwillison.net

Scaling: The State of Play in AI

Ethan Mollick oneusefulthing.org

What We Learned From a Year of Building With LLMs

Bryan Bischof oreilly.com

All the Hard Stuff Nobody Talks About When Building Products With LLMs

honeycomb.io honeycomb.io

TL;DR

LLMLingua utilizes a compact, well-trained language model (e.g., GPT2-small, LLaMA-7B) to identify and remove non-essential tokens in prompts. This approach enables efficient inference with large language models (LLMs), achieving up to 20x compression with minimal performance loss.

LLMLingua: Compressing Prompts for Accelerated Inference of La

microsoft • GitHub - microsoft/LLMLingua: To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.