Introduction

Data lineage uncovers the life cycle of data, it aims to show the complete data flow, from start to finish. Data lineage is the process of understanding, recording, and visualizing data as it flows from data sources to consumption. This includes all transformations the data underwent along the way, how the data was transformed, what changed, and why. Both data warehouse and data lake administrators need to be concerned about tracking data provenance and data lineage. Understanding when and where data originated, who touched it, and how data was modified are critical aspects of metadata management.

Why Does Data Lineage Matter

Understanding the provenance and lineage of data sources is valuable for several reasons:

• Evaluating the trustworthiness of data based on its provenance
• Understanding and correcting sources of error
• Identifying incorrect assumptions about data that may skew analysis
• Providing audit trails for data governance and regulatory purposes
• Ensuring data flows are protected and not subject to tampering
• Identifying and avoiding data duplication to simplify operations and reduce cost
• Organizations need visibility to how information moves through various workflow steps to ensure the quality of query results, business reports, business intelligence (BI) dashboards, and training sets. Data quality is enhanced when data engineers can track who made a change and why, how something was updated, and which process was used.