Saved by Ted Glasnow
Thomas Schiavone, co-founder and CEO of Calixa, on the PLG data pipeline
Focus
joincolossus.com
makes a lot more sense to get the other 5 to 10 percent into the EDW and analyze all of the data there. In other words, move the analysis to the data instead of moving the data to the analysis. This is the concept of in-database analytics.
Bill Franks • Taming The Big Data Tidal Wave: Finding Opportunities in Huge Data Streams with Advanced Analytics (Wiley and SAS Business Series)
ETL
The part of the system I'm most proud of, and on which I spent the most effort, is the ETL process.
We had a series of shell scripts for each data source we ingested (there were many), which would pull the data and put it in an s3 bucket.
Then, early in the morning, a cron job would spin up an EC2 instance, which would pull in the latest ETL code... See more
The part of the system I'm most proud of, and on which I spent the most effort, is the ETL process.
We had a series of shell scripts for each data source we ingested (there were many), which would pull the data and put it in an s3 bucket.
Then, early in the morning, a cron job would spin up an EC2 instance, which would pull in the latest ETL code... See more