Reproducible data science over data lakes: replayable data p...

Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie.

Reproducible data science is enabled through Bauplan and Nessie, providing time-travel and branching semantics on data lakes, decoupling compute from data management.

arxiv.org

RelatedHighlights

The commodity modeling concept discussed in Chapter 6 is a perfect example of a type of analysis that would not be feasible without an enterprise analytic data set to support it.

Bill Franks • Taming The Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics (Wiley and SAS Business Series)

it's hard, as Salesforce, to re-architect your entire platform to be built on top of this notion of a cloud data warehouse, especially when all these other vendors have already built tools on top of your own platform. I'm not sure how you go from point A to point B there.

Jan-Erik Asplund • Earl Lee, co-founder and CEO of HeadsUp, on the modern data stack value chain

ETL

The part of the system I'm most proud of, and on which I spent the most effort, is the ETL process.

We had a series of shell scripts for each data source we ingested (there were many), which would pull the data and put it in an s3 bucket.

Then, early in the morning, a cron job would spin up an EC2 instance, which would pull in the latest ETL code... See more

Bill Mill • notes.billmill.org

If analytic professionals are experimenting with some new data to figure out what it offers, a cloud is a terrific environment to consider. A lot of times in exploratory analytical development work, performance isn’t an issue.

Bill Franks • Taming The Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics (Wiley and SAS Business Series)

Data bases have gotten so good at this, that the term is almost misleading now. “Base” suggests something rigid, without which the data would slip away. But the data is always there, just bits on a nameless hard disk. The structure and the accessibility that a modern database provides exist completely independently from that hard disk. That’s right... See more

DuckDB Doesn’t Need Data To Be a Database

changes are attempts to eliminate the tedious ETL step before data can be assessed and analyzed. This objective is being addressed through real-time messaging and computation tools such as Apache Kafka and Storm.

Bill Franks • Taming The Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics (Wiley and SAS Business Series)

Jan-Erik Asplund • Earl Lee, co-founder and CEO of HeadsUp, on the modern data stack value chain

Bill Mill • notes.billmill.org

Bill Franks • Taming The Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics (Wiley and SAS Business Series)

DuckDB Doesn’t Need Data To Be a Database

Thomas H. Davenport • Big Data at Work: Dispelling the Myths, Uncovering the Opportunities