GitHub - NVIDIA/NeMo-Curator: Scalable toolkit for data curation

This begs the question: What might it mean to reimagine the form of these datasets in a world unconstrained by pressures like speed, scale, and universality? By looking to artists like Anna Ridler and others for their rejection of “off-the-shelf” datasets, we can imagine what it would be like to curate datasets with much deeper intention and contex... See more

This begs the question: What might it mean to reimagine the form of these datasets in a world unconstrained by pressures like speed, scale, and universality? By looking to artists like Anna Ridler andothers for their rejection of “off-the-shelf” datasets, we can imagine what it would be like to curate datasets with much deeper intention and context... See more

Clean & curate your data with LLMs

databonsai is a Python library that uses LLMs to perform data cleaning tasks.

Features

• Suite of tools for data processing using LLMs including categorization, transformation, and extraction


• Validation of LLM outputs


• Batch processing for token savings


• Retry logic with exponential backoff for handling rate limits an

... See more