GitHub - cliangyu/Cola: [NeurIPS2023] Official implementation of the paper "Large Language Models are Visual Reasoning Coordinators"

Repository for the paper "The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning", including 1.84M CoT rationales extracted across 1,060 tasks"

Paper Link : https://arxiv.org/abs/2305.14045

LLaVA v1.5, a new open-source multimodal model stepping onto the scene as a contender against GPT-4 with multimodal capabilities. It uses a simple projection matrix to connect the pre-trained CLIP ViT-L/14 vision encoder with Vicuna LLM, resulting in a robust model that can handle images and text. The model is trained in two stages: first, updated ... See more

新的序列建模方法,可以学习大型视觉模型,无需使用任何语言数据。通过定义“视觉句子”格式,将原始图像和视频以及注释数据源表示为序列,然后通过训练模型来预测下一个标记,从而实现模型的训练

Guideline following Large Language Model for Information Extraction

Model Card for GoLLIE 34B

We present GoLLIE, a Large Language Model trained to follow annotation guidelines. GoLLIE outperforms previous approaches on zero-shot Information Extraction and allows the user to perform inferences with annotation schemas defined on the fly. Different from... See more

a couple of the top of my head:

• LLM in the loop with preference optimization


• synthetic data generation


• cross modality "distillation" / dictionary remapping


• constrained decoding

Key unlock: Multimodal models can reason about images, video, or even physical environments without significant tailoring.

Our extensive test over 25 LLMs (including APIs and open-sourced models) shows that, while top commercial LLMs present a strong ability of acting as agents in complex environments, there is a significant disparity in performance between them and open-sourced competitors.