GitHub - turboderp/exllamav2: A fast inference library for running LLMs locally on modern consumer-class GPUs

slowllama

Fine-tune Llama2 and CodeLLama models, including 70B/35B on Apple M1/M2 devices (for example, Macbook Air or Mac Mini) or consumer nVidia GPUs.

slowllama is not using any quantization. Instead, it offloads parts of model to SSD or main memory on both forward/backward passes. In contrast with training large models from scratch (unattainable... See more

2-5x faster 50% less memory local LLM finetuning

• Manual autograd engine - hand derived backprop steps.


• 2x to 5x faster than QLoRA. 50% less memory usage.


• All kernels written in OpenAI's Triton language.


• 0% loss in accuracy - no approximation methods - all exact.


• No change of hardware necessary. Supports NVIDIA GPUs since 2018+. Minimum CUDA Compute Cap

... See more

LLaVA v1.5, a new open-source multimodal model stepping onto the scene as a contender against GPT-4 with multimodal capabilities. It uses a simple projection matrix to connect the pre-trained CLIP ViT-L/14 vision encoder with Vicuna LLM, resulting in a robust model that can handle images and text. The model is trained in two stages: first, updated ... See more