Guide · Lineage

Data lineage: what it is, and what it needs from you first

Lineage is the most demonstrable feature in any catalogue tool. A diagram appears, boxes connect, and everyone in the room feels the problem receding. It is also the feature most likely to be bought before the two questions it is meant to answer have been written down.

7 minute read For: the evaluator, the analyst Reviewed September 2026

The short answer

Data lineage is the record of where a dataset came from, what happened to it in transit, and which reports depend on it. Tools derive it automatically by parsing the code they can read — SQL, pipeline definitions, notebook jobs — and lose the trail at any step they cannot read, which in most mid-market estates means the moment a number passes through a spreadsheet, a manual upload or a hand-edited report.

A left-to-right chain of five stages — source, pipeline, warehouse, spreadsheet, report. The first two arrows are solid grey; the two either side of the spreadsheet are broken red dashes, marking where the automated trail stops. — The break is normal, not exceptional. Every mid-market estate has at least one spreadsheet in the middle of a chain that matters. Illustration

The two questions it exists to answer

Strip away the diagram and lineage is bought to answer two things.

Upstream, after a failure: this number is wrong — where did it come from and what else is affected? You are working backwards from a symptom.

Downstream, before a change: we want to change this column, retire this system, or fix this definition — what breaks? You are working forwards from a proposal.

The second is worth more and is asked less, because nobody notices the cost of the change they were too nervous to make. An organisation that cannot answer it does not have outages; it has a slow accumulation of things nobody dares touch.

What automation actually gives you

Modern tools — Microsoft Purview, the catalogue inside Fabric, AWS Glue's catalogue, the various independents — derive lineage by parsing artefacts they can read: SQL queries, pipeline and job definitions, notebook code, the semantic model in a BI tool. Where the estate is code, the coverage is genuinely good and genuinely automatic.

Where it stops:

Spreadsheets. A CSV exported, adjusted, and re-uploaded is a hole. Nothing recovers the transformation in between.
Manual steps. The month-end adjustment applied by hand. The "correction" tab.
Systems outside the platform. A finance package writing to its own database that the pipeline reads from is where the trail typically starts, not the actual origin of the number.
Meaning. This is the big one. Lineage tells you that column A became column B. It cannot tell you that the definition changed on the way — that revenue upstream is gross and revenue downstream is net of returns. The graph is complete and the reader is misled.

That last limitation is why lineage does not replace a definition. It complements one.

Column-level, table-level, and what you need

Table-level lineage says this table feeds that table. Column-level says this field feeds that field. Column-level is the one demonstrated in sales meetings and it costs materially more, in licence and in the effort of keeping parsing coverage high.

For a business of 50 to 1,000 people, table-level answers most of the real questions. The exception is a regulated reporting line where you have to evidence the derivation of a specific figure. If you have one of those, buy column-level for that lineage and accept table-level everywhere else.

What lineage needs from you before it is useful

A complete graph of a system nobody has named is a picture, not an answer. Three things have to exist first.

A register with owners. Tracing a wrong number to its source is only progress if there is somebody to tell. Without owners, the graph ends at a table and the conversation ends there too.
Written definitions for the measures that matter. Twenty of them is enough. This is what closes the gap between "A became B" and "A meant something different from B".
An honest map of the manual steps. Ask the four or five people who build reports where they export to Excel and what they do there. This takes an hour, produces a list of six to twelve places, and is more valuable than any automated graph — because those are exactly the points automation cannot see, and they are where the errors are.

The cheap version

Before any tool, you can get most of the operational value with a spreadsheet and a morning. For each of your twenty most important reports, record: the systems it draws from, the intermediate steps including manual ones, the owner, and the definition of its headline measure.

Twenty rows. It will not render a diagram, and it answers "what breaks if we change this?" for the things that actually matter. When you later buy a tool, this list is what you check its coverage against — and the honest answer will be that it found fifteen of your twenty chains and missed the five with a spreadsheet in the middle.

A reasonable sequence

Register and owners first. Definitions for the top measures second. The manual-step map third. Then, if a tool is on the table, evaluate it against those twenty known chains rather than against a demo, and price it against the two questions rather than the diagram.

Lineage bought first tends to produce a beautiful graph, a quarterly licence, and the same argument in the same meeting about which revenue number is right.

Common questions

What is data lineage?

The record of where a dataset came from, what transformations happened to it in transit, and which reports and dashboards depend on it. It is used to trace a wrong number back to its source, and to work out what breaks before changing a column, retiring a system or altering a definition.

Can data lineage be captured automatically?

Partly. Tools parse the artefacts they can read — SQL, pipeline definitions, notebook jobs, BI semantic models — and coverage is good where the estate is code. The trail stops at anything they cannot read: spreadsheets, manual uploads, hand-applied adjustments, and systems outside the platform. Most mid-market estates have at least one such break in a chain that matters.

Do we need column-level lineage?

Usually not, at this size. Table-level lineage answers most operational questions and costs materially less. Column-level earns its price where you must evidence the derivation of a specific figure for a regulator — in which case buy it for that reporting line and accept table-level elsewhere.

What should we do before buying a lineage tool?

Three things: get a register with named owners, so a traced problem has somebody to go to; write down the definitions of the twenty measures that appear in board papers, because lineage shows that a column moved and not that its meaning changed; and spend an hour asking report builders where they export to Excel, which maps the gaps automation will never see.

Where the product comes in

Lineage is one tab on the dataset, next to the owner

Lake On Rails records lineage on the dataset record alongside the owner, the columns, the quality checks, the access and the history — so a traced problem lands on a named person rather than on a table. It reads platform capability from Microsoft Fabric, Purview and AWS Glue connections and shows which capabilities those platforms cover natively and which still need a human decision recorded.

It is not a pipeline-parsing lineage engine and does not claim to replace one.

Request a demo See what you get

Forty-five minutes with whoever runs your reporting

We tell you honestly whether this is worth doing at all, and roughly what it would take. If the answer is not yet, you will hear that. "Not for us" is a fine outcome, and a better one than a slow maybe.

Request a demo Is it for us?