Lineage Is a Graph Problem

Lineage is one of those things every data team agrees they should have and almost nobody has properly. The vendor demos look great. The actual production lineage in most organisations is either non-existent, partial, or wrong in ways worse than non-existent because they create false confidence. Worth understanding why, because the problem is harder than the marketing suggests.

Two flavours of lineage

Lineage comes at two levels of granularity, and they’re very different problems.

Table-level lineage. Table A is built from tables B and C, which are built from tables D, E, and F. A directed graph where the nodes are tables and the edges are “is built from.” This is what most lineage tools actually deliver. It’s relatively easy to extract from SQL parse trees or dbt’s manifest, and it’s genuinely useful for impact analysis.

Column-level lineage. Column A.x is derived from columns B.y and C.z. Now the graph nodes are columns, not tables, and the edges represent specific derivations. This is what people actually want when they ask “where does this number come from?” It’s vastly harder to compute and most tools that claim to do it are doing a partial job.

Why column-level is hard

A SQL query like SELECT a.x + b.y AS z FROM a JOIN b ON a.id = b.a_id is trivially analysable: z depends on a.x and b.y. But real SQL doesn’t look like that. Real SQL has subqueries, CTEs, window functions, lateral joins, dynamic pivots, CASE expressions, aggregations, recursive structures, vendor-specific extensions. A column produced by a CASE expression depends on every column referenced in the conditions and every column in the WHEN branches. A column produced by an aggregation depends on the aggregated column and the GROUP BY columns. A column produced by a UDF depends on whatever the UDF reads, which the parser usually can’t see.

And that’s just SQL. Add Python transformations (Spark, pandas, Polars) and most lineage tools treat them as black boxes: “table A goes in, table B comes out, dependencies unknown.” The column-level lineage stops at the Python boundary and resumes on the SQL side. The graph has holes.

The honest position is that perfect column-level lineage is a research problem and most teams need to accept partial lineage with explicit gaps rather than complete lineage that turns out to be wrong.

Where lineage comes from

Three sources, each with limits.

Static SQL parsing. The query text gets analysed by a SQL parser that produces a lineage graph. Works well for simple queries; works less well for vendor-specific SQL extensions, dynamic queries, or templated SQL. dbt-style projects are the best case because the SQL is in version control and structured.

Runtime tracing. The query engine instruments execution and emits lineage events. OpenLineage is the standard. It works across SQL engines and into Spark. The advantage over static parsing is that it captures runtime behaviour, including queries built dynamically. The disadvantage is that it requires instrumentation, which most legacy engines don’t have.

Manual annotation. Engineers describe lineage explicitly in metadata. Reliable when it’s done; almost never complete; goes stale within months.

The serious lineage tools (Marquez, DataHub, OpenMetadata, Atlan, Collibra) combine these sources. dbt’s built-in lineage is static-parsed from project manifests. Snowflake’s Access History gives you runtime lineage at the table level. Databricks Unity Catalog has table-level lineage; column-level depends on the engine. Stitching across the stack is something you do, not something you buy.

OpenLineage

OpenLineage is the standard worth knowing about. It defines an open spec for what a lineage event looks like – runs, jobs, datasets, inputs, outputs, facets. Producers (Spark, Airflow, dbt, Flink) emit events; consumers (lineage UIs, catalogs) ingest them. The advantage is that you can integrate multiple sources into one lineage view without each tool having to natively understand the others.

The reality of OpenLineage in 2026 is that the spec is solid, the producers are improving, but the consumers are still maturing. You can run an OpenLineage stack today and get useful lineage. You won’t get a complete or perfectly accurate one without effort.

What lineage doesn’t solve

• It doesn’t tell you what a column means. Lineage tells you where it’s computed from. Semantics still need to be documented separately.
• It doesn’t tell you whether the value is correct. A value can be computed correctly from the wrong inputs, or incorrectly from the right inputs.
• It doesn’t tell you who consumes a dataset. That requires usage tracking, which is a different problem.
• It doesn’t replace data quality testing. Lineage is structural; quality is content. Both are needed.

The realistic ambition

A pragmatic lineage strategy for 2026:

1. Table-level lineage everywhere, computed automatically, no human curation required.
2. Column-level lineage within dbt or equivalent where the SQL is in version control and the parser can do its job.
3. Acknowledged gaps wherever the lineage hits Python, UDFs, dynamic SQL, or anything the parser can’t handle. Don’t pretend you have lineage you don’t have.
4. Usage tracking from query logs to identify consumers, separately from lineage.
5. Integration with catalog and semantics so the graph is browsable in business terms, not just technical ones.

Anything more is aspirational. Lineage tools that promise more are mostly selling you the appearance of completeness. Worth checking by picking a transformation that involves a Python step, asking the tool to show column-level lineage, and seeing what it does. The honest tools tell you they can’t. The optimistic ones make something up.

Next week – catalogs. Unity, Polaris, Gravitino, Nessie. The wars that nobody outside the industry knows about.