Lineage

Overview

Lineage records which jobs read which datasets and wrote which others. Gravitino collects it as OpenLineage run events: an engine sends an event as a job runs, Gravitino receives it and forwards it to a lineage store, and the UI shows the datasets and jobs recorded there, with column-level lineage where the engine reports it.

Datasets are named with Gravitino identifiers. The namespace is the metalake and the name is catalog.schema.table, so a dataset in the lineage view is the same object that carries tags, policies, and ownership elsewhere in Gravitino.

Quick start

1. Add the lineage plugin to Spark. See Spark Lineage.

2. Run a job that reads and writes tables. Each run sends events as it starts and finishes.

3. Open the lineage view and choose the metalake's namespace. The job's input and output datasets appear, and selecting one shows the jobs upstream and downstream of it.

How events flow

Events arrive at the main server at /api/lineage, pass through a processor, and go to one or more sinks. When lineage is enabled at deployment, the sink is an HTTP sink pointed at the lineage store bundled with Enterprise, which is what the lineage view reads. See Server Lineage for the configuration.

Sources of lineage

Spark is the supported source. The Gravitino OpenLineage plugin rewrites dataset identifiers into Gravitino form, including for tables reached through the Gravitino Spark connector and for filesets reached through GVFS, and reports column-level lineage.

Any other OpenLineage producer can send events to the same endpoint. Its datasets are recorded under the names it sends, so they line up with Gravitino objects only if the producer uses the same namespace and naming.