jina-ai/reader: Convert any URL to an LLM-friendly input ... - GitHub

Data-Juicer: A One-Stop Data Processing System for Large Language Models

Data-Juicer is a one-stop data processing system to make data higher-quality, juicier, and more digestible for LLMs. This project is being actively updated and maintained, and we will periodically enhance and add more features and data recipes. We welcome you to join us in pro... See more

Open-Source Pre-Processing Tools for Unstructured Data

The unstructured library provides open-source components for ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, and many more. The use cases of unstructured revolve around streamlining and optimizing the data processing workflow for LLMs. unstructured modular... See more

LLM-PowerHouse: A Curated Guide for Large Language Models with Custom Training and Inferencing

Welcome to LLM-PowerHouse, your ultimate resource for unleashing the full potential of Large Language Models (LLMs) with custom training and inferencing. This GitHub repository is a comprehensive and curated guide designed to empower developers, researche... See more

uniflow provides a unified LLM interface to extract and transform and raw documents.

• Document types: Uniflow enables data extraction from PDFs, HTMLs and TXTs.


• LLM agnostic: Uniflow supports most common-used LLMs for text tranformation, including
  
  
  • OpenAI models (GPT3.5 and GPT4),
  
  
  • Google Gemini models (Gemini 1.5, MultiModal),
  
  
  • AWS BedRock models,
  
  
  • Huggingf

... See more

Original creator : Jesse Zhang (GH: emptycrown, Twitter: @thejessezhang), who courteously donated the repo to LlamaIndex!

This is a simple library of all the data loaders / readers / tools / llama-packs that have been created by the community. The goal is to make it extremely easy to connect large language models to a large variety of knowledge sour... See more