The Open Data Stack Distilled into Four Core Tools

RelatedHighlights

In today’s data world, there are so many options for an EL tool to avoid you developing your own extracting script and to help you gain a LOT of time.

Fivetran, Mage, and Airbyte to mention a few.

You don’t have to maintain custom scripts, these tools come with +300 connectors, basic scheduling, and error handling.

Jeremy • What I Learned After One Year of Building a Data Platform From Scratch

Let us briefly touch on the concepts of data science and data engineering. If we go back to the DIKW triangle, we can say that data science focuses on extracting knowledge and wisdom from the information we have. Data scientists combine tools from mathematics and statistics to analyze information to arrive at insights. Exponentially increasing amou

Murat Erder • Continuous Architecture in Practice: Software Architecture in the Age of Agility and DevOps (Addison-Wesley Signature Series (Vernon))

Most commonly, ETL means moving data from some source system (e.g. a production database, Slack API) into an analytical data warehouse (e.g. Snowflake) where the data is easier to combine and analyze. Most data teams use a vendor like Fivetran or an orchestration platform like Airflow to do this.

Modal is a great solution for ETL if you are primaril... See more

Why you should move your ETL stack to Modal

Earl Lee, co-founder and CEO of HeadsUp, on the modern data stack value chain

Sacra sacra.com

(1) The separation between storage and compute , as encouraged by data lake architectures (e.g. the implementation of P would look different in a traditional database like PostgreSQL, or a cloud warehouse like Snowflake). This architecture is the focus of the current system, and it is prevalent in most mid-to-large enterprises (its benefits that be... See more

Jacopo Tagliabue • Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie.