Collect lineage from Spark

The Gravitino OpenLineage plugin for Spark builds on the OpenLineage Spark integration and rewrites each dataset identifier into Gravitino form, so the lineage Spark reports lines up with the objects Gravitino governs. It reports column-level lineage, works with and without the Gravitino Spark connector, and resolves GVFS paths to the filesets behind them.

Quick start

1. Put the plugin on Spark's classpath. Download the Gravitino OpenLineage plugin and add it alongside the Gravitino Spark connector jars.

2. Point OpenLineage at Gravitino. Events go to the main server's /api/lineage endpoint, using the metalake as the namespace.

   Start Spark SQL with OpenLineage

   ./bin/spark-sql \
     --jars /{path}/openlineage-spark_2.12-{plugin_version}.jar \
     --packages org.apache.gravitino:gravitino-spark-connector-runtime-3.5_2.12:1.3.1,com.datastrato:datastrato-spark-connector-ext-3.5_2.12:1.3.0 \
     --conf spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin \
     --conf spark.sql.gravitino.uri=https://{gravitino_host} \
     --conf spark.sql.gravitino.metalake={metalake} \
     --conf spark.sql.gravitino.authType=basic \
     --conf spark.sql.gravitino.basic.username={username} \
     --conf spark.sql.gravitino.basic.password={password} \
     --conf spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener \
     --conf spark.openlineage.transport.type=http \
     --conf spark.openlineage.transport.url=https://{gravitino_host} \
     --conf spark.openlineage.transport.endpoint=/api/lineage \
     --conf spark.openlineage.transport.auth.type=api_key \
     --conf spark.openlineage.transport.auth.apiKey=$TOKEN \
     --conf spark.openlineage.namespace={metalake} \
     --conf spark.openlineage.appName={app_name} \
     --conf spark.openlineage.columnLineage.datasetLineageEnabled=true

   The api_key transport sends $TOKEN, an access token for the server, as a bearer token, which the endpoint requires. The spark.sql.gravitino.* settings sign the connector in separately; see Spark authentication for OAuth.

3. Run a job. Its datasets appear in the lineage view under the metalake's namespace.

Dataset names

The namespace is always the metalake. The name depends on how Spark reached the dataset.

How Spark reached it	Name	Example
A table through the Gravitino Spark connector	{catalog}.{schema}.{table}	iceberg_catalog.db.score
A Hive table outside Gravitino	spark_catalog.{schema}.{table}	spark_catalog.db.table
An Iceberg or JDBC v2 table outside Gravitino	{catalog}.{schema}.{table}	iceberg_catalog.db.table
A JDBC v1 table outside Gravitino	spark_catalog.{schema}.{table}	spark_catalog.postgres.public.table
A path under GVFS	{catalog}.{schema}.{fileset}	fileset_catalog.schema.fileset_a
Any other path	The path itself	hdfs://127.0.0.1:9000/tmp/a/student

Datasets reached through GVFS also carry a fileset-location facet with the virtual path.

Configuration

These add to the OpenLineage Spark settings.

Setting	Description	Default
spark.sql.gravitino.useGravitinoIdentifier	Use Gravitino identifiers for datasets outside Gravitino. When false, the original OpenLineage identifiers are kept, such as an HDFS namespace and path	true
spark.sql.gravitino.catalogMappings	Maps Spark catalog names to Gravitino catalog names for datasets outside Gravitino, as in spark_catalog:catalog1,iceberg_catalog:catalog2	(none)

Related

• Lineage
• Lineage service configuration
• Spark connector