Data lineage is the documented history of a data asset: where it originated, which systems and processes changed it, who or what acted on it, and where the resulting data was delivered. The World Wide Web Consortium formalised that record in 2013 as PROV-DM, a provenance data model that lineage tools still follow today.
Your quarterly revenue figure has moved. Finance pulled one number from the reporting layer, the board pack shows another, and the analyst who built both is on leave. Nobody in the room can say which figure is correct, so the next two days disappear into extract logs, scheduled jobs and half-remembered SQL.
Lineage answers that question in minutes rather than days. It records how a number reached the screen: the source table it started in, the jobs that touched it, the rules that reshaped it and the report it landed in.
Table of Contents
- Data Lineage Explained with a Simple Analogy
- How a Data Record Travels from Source to Report
- Logical and Physical Lineage Compared
- Why Lineage Matters for Governance, Compliance and AI
- Capturing Lineage and Avoiding Stale Maps
- Lineage in Microsoft, SAP and Databricks Estates
- Start Mapping Lineage for Your Highest-Risk Data
Data Lineage Explained with a Simple Analogy
Think of lineage as a family tree for a figure. Every value has parents, the records and files it was built from, and every transformation adds a generation, the step where a value was cleaned, joined or aggregated. Ask about a person and the tree shows their line. Ask about a revenue figure and lineage shows its line.
The W3C PROV-DM specification makes the idea precise. It defines provenance as a record of the people, institutions, entities and activities involved in producing, influencing or delivering data, which is why lineage is more than a source-to-target map. It carries entities, activities, responsible agents, derivations, collections and annotations, and it exists to support judgements about quality, reliability and trustworthiness.
Lineage also sits inside the broader idea of data provenance, the question of where data came from and why anyone should trust it. Provenance tells you a record's origin. Lineage follows the record across its whole journey, including the who and the when at each stop.
A sales order created in SAP gets extracted into a cloud lake, cleaned by a transformation job and published as revenue in a Power BI report. Four steps, at least three teams, and one trail that should connect them. For an SAP- and Microsoft-centred organisation, that trail is the practical answer to what is data lineage: a traceable chain from operational records and transformations to governed analytical outputs.
How a Data Record Travels from Source to Report
A governed lineage record connects an originating table and its extraction timestamp to the ingestion job, the transformation logic, the target model and the downstream report or AI feature. The PROV data model documentation formalises this as relationships among entities (the datasets or records), activities (ETL jobs, transformations, validations, queries) and agents (the people, organisations or software responsible for each). It also represents derivation, temporal events, collections and links between identifiers that refer to the same underlying object.
Why a pipeline diagram isn't enough
A diagram shows the route data could take. It doesn't show the run that produced this morning's number. That gap matters most when something breaks, because a route without execution evidence can't tell you which version of the logic ran, which rows were dropped, or whether the extraction that fed today's report was the scheduled one or a manual rerun.
Four details turn a picture into evidence: the run identifier for each job, the timestamp of the load, the version of the transformation code that executed, and the count of rows the job rejected. Without them, a graph can show that a dashboard depends on a curated table but can't explain whether that table came from the correct SAP extraction.
Practical rule: A lineage graph without run evidence tells you where a number could have come from. It doesn't tell you where today's number did come from.
Say a dashboard total drops by 4 percent overnight. Backward tracing points to a transformation changed the previous Tuesday, and the run log shows 312 rejected rows entered the pipeline at that step. That's a five-minute explanation instead of an afternoon of guesswork.
Lineage works in both directions. Backward tracing finds the cause of a change. Forward impact analysis tells you which datasets and consumers break if a source field, rule or pipeline owner changes, before deployment rather than after.
Logical and Physical Lineage Compared
Two families of lineage answer different questions for different people. Business lineage (sometimes called logical lineage) describes terms, definitions and rules, such as how net revenue is defined and which source fields it depends on. Technical lineage (physical lineage) describes the actual tables, jobs, columns and pipeline runs that move the data.
Granularity cuts across both. Table-level lineage shows dataset-to-dataset dependencies, which is enough to plan a migration. Column-level lineage shows which source fields feed each output field, which is what you need for personal data mapping, regulatory reporting and any calculation a regulator might question.
| Lineage type | What it shows | Main users | Example |
|---|---|---|---|
| Business (logical) lineage | Business terms, definitions and rules behind a metric | Business users, data stewards | Net revenue defined from SAP sales order fields |
| Technical (physical) lineage | Actual tables, jobs and pipeline runs that move data | Data engineers, architects | ETL job moving an SAP table into a Fabric lakehouse |
| Table-level lineage | Dataset-to-dataset dependencies | Platform owners | Curated sales table feeding three Power BI reports |
| Column-level lineage | Which source columns feed each output field | Engineers, auditors | Net amount traced back to price and discount fields |
The rule of thumb is simple. Business users need logical lineage to trust a metric. Engineers need physical, column-level lineage to debug one. Governance teams need both, joined together, so a business definition can be followed straight into the code that implements it.
A business glossary without technical lineage is a dictionary with no page numbers.
Why Lineage Matters for Governance, Compliance and AI
Governance runs on impact analysis. When someone proposes renaming an SAP field, retiring a pipeline or changing a currency rule, lineage shows what breaks before the change ships. A renamed field that breaks a dozen reports is a governance failure with a technical cause, and lineage is how you catch it in the change request rather than in the month-end close.
Compliance asks a harder question: prove where this figure came from and who changed it. Auditors don't accept a reconstructed story. They accept a traceable chain, which is why lineage belongs inside a wider enterprise data governance framework rather than sitting in a catalogue nobody outside the data team opens. Regulated finance, healthcare and public-sector teams need that chain even when data crosses SAP, Azure, Fabric and third-party services.
What AI governance adds to the evidence trail
Traditional lineage stops at a dashboard. AI systems push the requirement further. Recent AI governance coverage describes lineage as an evidence trail for training, retrieval, evaluation and production decisions, with documentation expected for data origin, collection methods, preparation steps, representativeness, quality limitations and transformations. For high-risk AI, that documentation and oversight extend across the whole system lifecycle.
In practice, the chain now runs from source records to curated datasets, feature or embedding generation, model versions, prompts and retrieval indexes, outputs, human approvals and later corrections. Three terms get used loosely and shouldn't be:
- Data lineage tracks how data moved and changed.
- Model lineage tracks which datasets, features and code produced which model version.
- AI decision provenance tracks what a specific system did at the moment it produced a specific output.
The limit is worth stating plainly. Lineage proves traceability. It does not prove accuracy, fairness or lawfulness. Quality controls, access controls, testing and monitoring provide the assurance, and lineage is what lets an auditor walk from a decision back to the evidence those controls produced.
A credit-scoring model starts showing an unexpected decline pattern. Lineage reveals that its feature table was refreshed from a changed source extract three weeks earlier, and the reviewer now has a starting point instead of a mystery.
Capturing Lineage and Avoiding Stale Maps
Lineage comes from four places, and most enterprises need at least three of them.
| Capture method | How lineage is captured | Strength | Limitation |
|---|---|---|---|
| Manual documentation | Stewards record flows in spreadsheets or wikis | Captures business context | Goes stale after the first schema change |
| Metadata scanning | Tool parses SQL, ETL code and catalogues | Broad coverage with low effort | Misses logic in custom code and scripts |
| Runtime capture | Pipelines emit run IDs, versions and inputs as they execute | Precise and current | Needs engineering work in every pipeline |
| Inferred lineage | Tool guesses links from names or usage logs | Fast first view | Confidence varies and needs validation |
Use scanning for breadth across warehouses, BI tools and catalogues. Use runtime capture for the datasets that carry financial or regulatory weight. Use manual capture for business definitions no parser can infer. Use inferred lineage to bootstrap a first view, and label those links until a steward validates them. Scanning alone misses logic buried in custom code, and in an SAP estate that's exactly where the risk sits, in bespoke extraction routines and hand-written transformation scripts.
What a trustworthy trace records
The NIST Special Publication on provenance frames provenance as a historical, attributed, documented record covering where, when, how and by whom data was generated, acquired, processed or altered. Its big-data guidance goes further than most catalogues do, naming calibration information, errors, missing values, timestamps, locations, equipment identifiers, transaction numbers and authority as potentially necessary to establish trustworthy provenance.
The architectural consequence is that lineage should hold structural edges and execution evidence together, and every pipeline run should emit immutable metadata linked to the inputs that produced each output version. Queryable lineage then serves backward tracing, forward impact analysis and migration validation from the same store.
Why complete-looking maps mislead
Manually documented lineage falls behind first, because schemas, pipelines, APIs, business rules and cloud services keep changing while the spreadsheet doesn't. A recent data governance survey identifies significant organisational dissatisfaction with lineage, data-quality tools and data dictionaries, which points at maintenance rather than comprehension. The same survey notes movement toward regulatory-aligned, end-to-end lineage alongside persistent integration difficulty across distributed source systems.
More diagrams don't produce more trust. An outdated map creates false confidence, and false confidence makes impact analysis worse than an openly documented coverage gap.
Practical fixes, in the order that pays back fastest:
- Label every link as automated, manual or inferred, so a reader knows what they're looking at.
- Score each path on freshness, completeness and confidence rather than treating the graph as binary.
- Log exceptions where metadata can't be collected, instead of leaving a silent hole.
- Review lineage inside the change request for every schema change, so the map updates when the system does.
- Keep a coverage gap register naming what's still unmapped and who owns closing it.
Pairing that discipline with established data observability best practices closes the loop, because freshness and completeness scores only earn their keep when something acts on them.
Before selecting a tool, ask three questions. Does it read our SQL and ETL code? Does it capture run-level evidence, not just relationships? Does it expose lineage through an API so pipelines, catalogues and AI systems can consume it? A tool that answers all three is a control. One that answers only the first is a diagram with a licence fee.
Lineage in Microsoft, SAP and Databricks Estates
| Platform | Where lineage comes from | Granularity to verify | What to check first |
|---|---|---|---|
| SAP landscape (ECC, S/4HANA) | Extraction tables, function modules and custom ABAP logic | Table and field, where documented | Whether custom logic and RFC-based extractions are recorded |
| Microsoft Fabric and Power BI | Items, dataflows, lakehouse tables and reports | Item and table level, column where supported | Whether catalogue scanning covers every workspace |
| Databricks (Unity Catalog) | Table and query activity from Spark and SQL workloads | Table and column, where supported | Whether feeds from non-Databricks tools are captured |
| Cross-platform flow (SAP to Azure to Databricks to Power BI) | Lineage stitched across each handoff | Table and column, per hop and end to end | Whether lineage links survive each handoff |
In an SAP environment, lineage starts at extraction tables, function modules and custom ABAP logic. The usual gap is bespoke code: a custom routine or an RFC-based extraction that no parser reads, so the graph stops at the table boundary.
In Microsoft Fabric and Power BI, lineage spans workspace items, dataflows, lakehouse tables, semantic models and reports. The first check is coverage: whether catalogue scanning reaches every workspace, including the ones business teams created themselves.
In Databricks, Unity Catalog captures table and query activity from Spark and SQL workloads, and column-level detail is available where supported. Feeds arriving from outside Databricks are where coverage thins, and those are usually the ones that matter. Teams standardising on the platform should review Unity Catalog governance practices before assuming the default view is complete.
The cross-platform case is the hardest. A dataset moving from SAP to Azure to Databricks to Power BI breaks lineage at every handoff unless each hop is captured deliberately. Those handoffs are where migrations get delayed, because nobody can prove the new pipeline reproduces the old result. Kagool supports SAP data management with efficient tools, methodologies and a mix of business and data specialists to deliver a low-risk data migration process, which is largely a lineage discipline in practice. Check coverage at every handoff, not only inside the platform your own team runs, and record the gaps that remain.
Start Mapping Lineage for Your Highest-Risk Data
Don't start with the whole estate. Start with exposure.
- List the three to five reports or AI features carrying the highest regulatory or financial exposure.
- Trace each one backward to its source systems, marking every link as automated, manual or missing.
- Score each path on freshness, completeness and confidence, so weak spots are visible before an auditor finds them.
- Fix the highest-risk gaps first, beginning with SAP extraction routines and custom transformation logic, where coverage is thinnest.
- Set a review rule that lineage updates ride along with every schema change, in the same change request.
That sequence produces something repeatable rather than a one-off mapping exercise. Kagool's data governance and master data service improves the quality, consistency and governance of enterprise data for regulatory compliance and AI readiness, and the first step in our data engineering as a service roadmap is a detailed data maturity model assessment, so priorities are set against your estate rather than a template. The aim is repeatable foundations that move validated use cases into production with less rework and clearer accountability.
Lineage stops being a documentation project the moment you treat it as a control that has to stay current, and that shift is where most of the value sits. Kagool helps enterprises build lineage and governance foundations across SAP, Microsoft Fabric, Power BI and Databricks, connecting source systems to the reports and AI features that depend on them. If you want a practical starting point, visit Kagool and we'll help you trace your highest-risk metric end to end.
