Data Orchestration Trends: The Shift From Data Pipelines to Data Products
There are big data activities taking place across the company, including in the Data Engineering team, the Data Products team, the Business Analytics team, and the Product Data Science
Thomas H. Davenport • Big Data at Work: Dispelling the Myths, Uncovering the Opportunities
changes are attempts to eliminate the tedious ETL step before data can be assessed and analyzed. This objective is being addressed through real-time messaging and computation tools such as Apache Kafka and Storm.
Thomas H. Davenport • Big Data at Work: Dispelling the Myths, Uncovering the Opportunities
(1) The separation between storage and compute , as encouraged by data lake architectures (e.g. the implementation of P would look different in a traditional database like PostgreSQL, or a cloud warehouse like Snowflake). This architecture is the focus of the current system, and it is prevalent in most mid-to-large enterprises (its benefits that be... See more
Jacopo Tagliabue • Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie.
The next step is data transformation. Data transformation is the process of converting data from one format to another. Reasons for doing this could be to optimize the data for a different use case than it was originally intended or to meet the requirements for storing data in a different system. Data transformation may involve steps such as cleans... See more
Data Engineering • The Open Data Stack Distilled into Four Core Tools
The important point to consider here is that the choice of data architecture influences what tasks are undertaken and the methods for performing those tasks.
Steve Williams • The Profit Impact of Business Intelligence
Because strategic and tactical decision making requires historical views and analyzing trends, as well as seeing views of the business that crossed functional areas or lines of business, before data warehousing this information was often unavailable. As a result, managers learned to make do with