r/dataengineering - Reddit

Traditional ETL solutions are still quite powerful when it comes to:

• Common connectors with small-medium data volumes : we still have a lot of respect for companies like Fivetran, who have really nailed the user experience for the most common ETL use cases, like syncing Zendesk tickets or a production Postgres read replica into Snowflake. The only

... See more

Data bases have gotten so good at this, that the term is almost misleading now. “Base” suggests something rigid, without which the data would slip away. But the data is always there, just bits on a nameless hard disk. The structure and the accessibility that a modern database provides exist completely independently from that hard disk. That’s right... See more

The usage of dbt+Snowflake in spilling to disk in running dbt models at Instacart is bananas. 🍌🍌 *This is very expensive, and unnecessary **Instacart is throwing cash at the problem, which they can do because the wheelbarrow of VC money, but this will get you fired in a… Show more

The next step is data transformation. Data transformation is the process of converting data from one format to another. Reasons for doing this could be to optimize the data for a different use case than it was originally intended or to meet the requirements for storing data in a different system. Data transformation may involve steps such as cleans... See more

• Bob Muglia has the best definition. Others simply miss essential parts. The MDS isn’t just about open source or dbt; it is about SaaS, Cloud, Snowflake, and more. It is the wrapper around the progress in analytics over the last years.


• You should try to go for a 100% SaaS MDS . But try not to build up too many dependencies (yes, that’s possible; yo

... See more