GitHub - towhee-io/towhee: Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.

Koheesio

CI/CD

Package

Meta

Koheesio, named after the Finnish word for cohesion, is a robust Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

The framework is versatile, aiming to support multiple implementations and working sea... See more

DataTrove

DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality.

DataTrove processing pipelines are platform-agnostic, running out of the box locally or on a slurm cluster. Its (relatively) low memory... See more

一款适合需要处理大量非结构化AI数据的工具：datachain，支持多模态API调用和本地AI推理并行处理 可以用它结合大模型一起使用，来完成复杂的数据分析任务，比如做，数据处理和清洗、LLM分析和验证、图像分割等 支持图片、视频、文本、PDF、JSON、CSV、parquet等智能统一管理，自动保存处理记录和版本 可以直接从各种云存储，比如谷歌云、亚马逊云或者本地读取数据，无需手动复制 支持智能搜索和分析，支持超大数据集处理、并行处理 github：https://t.co/bJ9rKKuMBY... See more

uniflow provides a unified LLM interface to extract and transform and raw documents.

• Document types: Uniflow enables data extraction from PDFs, HTMLs and TXTs.


• LLM agnostic: Uniflow supports most common-used LLMs for text tranformation, including
  
  
  • OpenAI models (GPT3.5 and GPT4),
  
  
  • Google Gemini models (Gemini 1.5, MultiModal),
  
  
  • AWS BedRock models,
  
  
  • Huggingf

... See more

Welcome to RAGatouille

Easily use and train state of the art retrieval methods in any RAG pipeline. Designed for modularity and ease-of-use, backed by research.

The main motivation of RAGatouille is simple: bridging the gap between state-of-the-art research and alchemical RAG pipeline practices. RAG is complex, and there are many moving parts. To g... See more

SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, highest GPU availability, and managed execution.

SkyPilot abstracts away cloud infra burdens :

• Launch jobs & clusters on any cloud


• Easy scale-out: queue and run many jobs, automatically managed


• Easy access to object stores (S3, GCS, R2)

SkyPilot... See more