So I wrote a 5400-word lecture note on the basics of data engineering for my students, covering: * data formats (row- vs. column-based, text vs. binary) * ETL * batch processing vs. stream processing * training datasets WIP. Feedback much... See more

So what abstractions do we have as of today? For example, let’s take the resource abstraction (Dagster, Prefect, referred to as an operator in Airflow). You abstract complex environments and connections away with a simple construct like that. You have the immediate benefits of defining that once and using it in every task or pipeline with... See more

The last core data stack tool is the orchestrator. It’s used quickly as a data orchestrator to model dependencies between tasks in complex heterogeneous cloud environments end-to-end. It is integrated with above-mentioned open data stack tools. They are especially effective if you have some glue code that needs to be run on a certain cadence,... See more