Spark Lineage

Overview

The Gravitino OpenLineage plugin for Spark builds on the OpenLineage Spark integration and rewrites each dataset identifier into Gravitino form, so the lineage Spark reports lines up with the objects Gravitino governs. It reports column-level lineage, works with and without the Gravitino Spark connector, and resolves GVFS paths to the filesets behind them.

Quick Start

1. Put the plugin on Spark's classpath. Download the Gravitino OpenLineage plugin and add it with the Gravitino Spark connector runtime.

2. Point OpenLineage at Gravitino. Events go to the main server's /api/lineage endpoint, using the metalake as the namespace.

./bin/spark-sql \
  --jars /{path}/openlineage-spark_2.12-{plugin_version}.jar,/{path}/gravitino-spark-connector-runtime-3.5_2.12-{connector_version}.jar \
  --conf spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin \
  --conf spark.sql.gravitino.uri=https://{gravitino_host} \
  --conf spark.sql.gravitino.metalake={metalake} \
  --conf spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener \
  --conf spark.openlineage.transport.type=http \
  --conf spark.openlineage.transport.url=https://{gravitino_host} \
  --conf spark.openlineage.transport.endpoint=/api/lineage \
  --conf spark.openlineage.transport.auth.type=api_key \
  --conf spark.openlineage.transport.auth.apiKey={token} \
  --conf spark.openlineage.namespace={metalake} \
  --conf spark.openlineage.appName={app_name} \
  --conf spark.openlineage.columnLineage.datasetLineageEnabled=true

The api_key transport sends the token as a bearer token, which the endpoint requires.

3. Run a job. Its datasets appear in the lineage view under the metalake's namespace.

Dataset Names

The namespace is always the metalake. The name depends on how Spark reached the dataset.

How Spark Reached It	Name	Example
A table through the Gravitino Spark connector	{catalog}.{schema}.{table}	iceberg_catalog.db.score
A Hive table outside Gravitino	spark_catalog.{schema}.{table}	spark_catalog.db.table
An Iceberg or JDBC v2 table outside Gravitino	{catalog}.{schema}.{table}	iceberg_catalog.db.table
A JDBC v1 table outside Gravitino	spark_catalog.{schema}.{table}	spark_catalog.postgres.public.table
A path under GVFS	{catalog}.{schema}.{fileset}	fileset_catalog.schema.fileset_a
Any other path	The path itself	hdfs://127.0.0.1:9000/tmp/a/student

Datasets reached through GVFS also carry a fileset-location facet with the virtual path.

Configuration

These add to the OpenLineage Spark settings.

Configuration Item	Description	Default Value
spark.sql.gravitino.useGravitinoIdentifier	Use Gravitino identifiers for datasets outside Gravitino. When false, the original OpenLineage identifiers are kept, such as an HDFS namespace and path	true
spark.sql.gravitino.catalogMappings	Maps Spark catalog names to Gravitino catalog names for datasets outside Gravitino, as in spark_catalog:catalog1,iceberg_catalog:catalog2	(none)