Sublime

So I wrote a 5400-word lecture note on the basics of data engineering for my students, covering: * data formats (row- vs. column-based, text vs. binary) * ETL * batch processing vs. stream processing * training datasets WIP. Feedback much... See more

ETL

The part of the system I'm most proud of, and on which I spent the most effort, is the ETL process.

We had a series of shell scripts for each data source we ingested (there were many), which would pull the data and put it in an s3 bucket.

Then, early in the morning, a cron job would spin up an EC2 instance, which would pull in the latest ETL code... See more

ETL

The part of the system I'm most proud of, and on which I spent the most effort, is the ETL process.

We had a series of shell scripts for each data source we ingested (there were many), which would pull the data and put it in an s3 bucket.

Then, early in the morning, a cron job would spin up an EC2 instance, which would pull in the latest ETL code... See more

🚨BREAKING: New Python library for agentic data processing and ETL with AI Introducing DocETL. Here's what you need to know: https://t.co/94glNVRQfX

Bruin is a data pipeline tool for ingesting, transforming, and quality-checkingdata using SQL, Python, and various platforms, with support for local machines,EC2 instances, and GitHub Actions https://t.co/rfsLShLbIA

"Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data" https://t.co/r9sai5bvQd