Skip to main content

Lineage service configuration

The server receives OpenLineage run events, passes each through a processor, and writes it to one or more sinks. Each stage is pluggable, so events can be transformed or sent to systems beyond the bundled lineage store.

Quick start​

  1. Send an event. The HTTP source accepts events that follow the OpenLineage API specification.

    Send a START run event
    curl -sS -X POST "https://{gravitino_host}/api/lineage" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $TOKEN" \
      -d '{
      "eventType": "START",
      "eventTime": "2026-01-01T00:00:00.000Z",
      "run": {"runId": "0176a8c2-fe01-7439-87e6-56a1a1b4029f"},
      "job": {"namespace": "{metalake}", "name": "{job_name}"},
      "inputs": [{"namespace": "{metalake}", "name": "{catalog}.{schema}.{table}"}],
      "producer": "https://github.com/OpenLineage/OpenLineage/tree/1.0.0/client",
      "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
    }'

  2. Confirm it arrived. The dataset appears in the lineage view under the metalake's namespace.

Configuration​

The chart manages these properties from the values under gravitino.lineage, and turns lineage on with the bundled Marquez sink by default: source, processorClass, sinks, and marquez.sinkClass and marquez.url. Set any other property, such as gravitino.lineage.queueCapacity, under gravitino.additionalConfigItems; see Set other server properties.

SettingDescriptionDefault
gravitino.lineage.sourceName of the event sourcehttp
gravitino.lineage.{source}.sourceClassClass implementing org.apache.gravitino.lineage.source.LineageSource(none)
gravitino.lineage.processorClassClass implementing org.apache.gravitino.lineage.processor.LineageProcessororg.apache.gravitino.lineage.processor.NoopProcessor
gravitino.lineage.sinksComma-separated sink nameslog
gravitino.lineage.{sink}.sinkClassClass implementing org.apache.gravitino.lineage.sink.LineageSink(none)
gravitino.lineage.queueCapacityTotal queue capacity, divided evenly across sinks since each has its own queue10000

The default processor passes events through unchanged.

Sinks​

Log sink​

Writes each event to gravitino_lineage.log, whose location and rotation are set in the server's Log4j2 configuration. The log is the place to check that events arrive.

Lineage store sink​

The chart enables the sinks log,marquez. The marquez sink is the one that feeds the bundled lineage store, and the chart points it at the in-cluster Marquez API.

SettingDescriptionDefault
gravitino.lineage.marquez.sinkClasscom.datastrato.gravitino.lineage.sink.HTTPLineageSink(none)
gravitino.lineage.marquez.urlBase URL of the lineage store's API(none)
gravitino.lineage.marquez.timeoutInMsHow long to wait for the lineage store, in milliseconds10000

HTTP sink​

Sends each event to any other server that follows the OpenLineage REST specification. The sink's name is your choice; the examples use http.

SettingDescriptionDefault
gravitino.lineage.http.sinkClassorg.apache.gravitino.lineage.sink.LineageHttpSink(none)
gravitino.lineage.http.urlBase URL of the receiving serverhttp://localhost:5000/
gravitino.lineage.http.authTypeapiKey or nonenone
gravitino.lineage.http.apiKeyAPI key, when authType is apiKey(none)

Add the sink's name to the chart value gravitino.lineage.sinks, after the default log,marquez, and put the sink's own properties under gravitino.additionalConfigItems.

my-values.yaml
gravitino:
  lineage:
    sinks: log,marquez,http
  additionalConfigItems:
    gravitino.lineage.http.sinkClass: org.apache.gravitino.lineage.sink.LineageHttpSink
    gravitino.lineage.http.url: https://{openlineage_host}/

When sinks fall behind​

Each sink reads from a queue. When a queue passes 90 percent of its capacity, the server stops accepting events and the HTTP source answers 429 Too Many Requests, so a producer can back off and retry rather than lose events silently. A producer that does not retry on 429 drops those events.

Last updated on