Reproducible data science over data lakes: replayable data p...

Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie.

Reproducible data science is enabled through Bauplan and Nessie, providing time-travel and branching semantics on data lakes, decoupling compute from data management.

arxiv.org

RelatedHighlights

The data architecture encompasses data structures, databases (source and target), data flows, and data integration. It also includes the data and meta-data requirements for the BI/DW environment.

Steve Williams • The Profit Impact of Business Intelligence

So what abstractions do we have as of today? For example, let’s take the resource abstraction (Dagster, Prefect, referred to as an operator in Airflow). You abstract complex environments and connections away with a simple construct like that. You have the immediate benefits of defining that once and using it in every task or pipeline with context.r... See more

Data Engineering • Data Orchestration Trends: The Shift From Data Pipelines to Data Products

Expose Delta Tables via REST APIs

Git repo to test 3 architectures to expose delta tables via REST APIs. See also my blogpost here. Architectures can be described as follows:

Architecture A: Direct, Web App with DuckDB. In this architecture, APIs are directly connecting to the delta table and there is no layer in between. This implies that all data

GitHub - rebremer/expose-deltatable-via-restapi

The ability to implement custom Polars plugins in Rust is invaluable. Since we process a lot of textual data for our NLP applications, we can create optimized functions to clean text or detect a language, with data being processed efficiently in batches. This level of customization is rarely seen in other typical processing engines and is even impo... See more

Polars — Processing hundreds of GBs of textual data on a daily basis at MDPI

Continuous Architecture in Practice: Software Architecture in the Age of Agility and DevOps (Addison-Wesley Signature Series (Vernon))

Pierre Pureur

amazon.com

Cover of Continuous Architecture in Practice: Software Architecture in the Age of Agility and DevOps (Addison-Wesley Signature Series (Vernon))

The language is always only as good as its community. Let’s look at some of the existing open-source tools and frameworks built in and around Rust:

DataFusion based on Apache Arrow: Apache Arrow DataFusion SQL Query Engine similar to Spark

Polars: It’s a faster Pandas. Probably going to compete with DuckDB (?)

Delta Lake Rust: A native Rust library fo