Data Processing

DataTrove

DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality.

DataTrove processing pipelines are platform-agnostic, running out of the box locally or on a slurm cluster. Its (relatively) low memory... See more

What's a Data Diff?

A data diff is the value-level comparison between two tables—used to identify critical changes to your data and guarantee data quality.

There is a lot you can do with data-diff: you can test SQL code by comparing development or staging environment data to production, or compare source and target data to identify discrepancies whe... See more

Optimizing Further

Creating so many indices and aggregating so many tables is sub-optimal. To optimize this, we employ materialized views, which create a separate disk-based entity and hence support indexing. The only downside is that we have to keep it updated.

CREATE MATERIALIZED VIEW search_view AS

ᅠᅠSELECT c.name FROM company c UNION

ᅠᅠSELECT c.na... See more

Clean & curate your data with LLMs

databonsai is a Python library that uses LLMs to perform data cleaning tasks.

Features

• Suite of tools for data processing using LLMs including categorization, transformation, and extraction


• Validation of LLM outputs


• Batch processing for token savings


• Retry logic with exponential backoff for handling rate limits an

... See more

Programmable platform for data in motion

An open-source data streaming platform with in-line computation capabilities. Apply your custom programs to aggregate, correlate, and transform data records in real-time as they move over the network.

This job copied 12m rows from Clickhouse to Snowflake in 16 minutes using:

• 5 CPUs : at $0.192 / CPU hour that comes out to $0.26


• 4.4 GiB of memory: at $0.024 / GiB per hour that comes out to $0.03

Even if Fivetran had a ClickHouse connector (it doesn’t at the time of this writing), syncing 12m rows would cost ~$3300. The total cost of this Modal job... See more

(1) The separation between storage and compute , as encouraged by data lake architectures (e.g. the implementation of P would look different in a traditional database like PostgreSQL, or a cloud warehouse like Snowflake). This architecture is the focus of the current system, and it is prevalent in most mid-to-large enterprises (its benefits that be... See more

Overview¶

Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow.

Ballista has a scheduler and an executor process that are standard Rust executables and can be executed directly, but Dockerfiles are provided to build images for use in containerized environments, such as Docker, Docker Compose, and Kube... See more

1️⃣ RudderStack provides data pipelines to collect data from applications, websites and SaaS platforms.

2️⃣ Its API architecture and SDKs ensure developers can gather data from different sources and leverage them into their applications without disruptions.

3️⃣ RudderStack is highly versatile and integrates with over 90+ tools and data warehouse dest... See more

data load tool (dlt) — the open-source Python library for data loading

Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,

or a GPT-4 assisted development playground— dlt can be dropped in anywhere.