Why you should move your ETL stack to Modal

This basic stack is so accepted (and frankly so much easier now with AWS etc.) that we rarely question it. But it's ridiculously inefficient. Why should every potential business need to build all 3 parts of this stack when their core innovation or value add comes primarily from one or two?

On many occasions, MapReduce is used similarly to an extract, load, and transform (ETL) tool. ETL tools read a set of source data, perform a set of formatting or reorganization steps, and then load the results into a destination data source. To support analysis, ETL tools take data from operational systems and get it loaded into a relational databa

... See more

ETL

The part of the system I'm most proud of, and on which I spent the most effort, is the ETL process.

We had a series of shell scripts for each data source we ingested (there were many), which would pull the data and put it in an s3 bucket.

Then, early in the morning, a cron job would spin up an EC2 instance, which would pull in the latest ETL code... See more

You need to be able to draw insight and to structure that information such that people can then act on it. What is doing that? Today, transformation tools like dbt are doing that, if you take the lens of the data team really owning everything end-to-end, but I think also applications that are able to plug into the data warehouse, consume this raw i... See more

A common argument against moldability is the increased cost of maintaining integrations or modifications that exist on top of a perpetually changing piece of software. This is a real issue, as evidenced by the fragility of plugin ecosystems such as those of the free content management system, Wordpress. However, this points to the fact that there a... See more