Skip to main content

Connectors on Kubernetes

The Helm chart installs the Datastrato Enterprise server and nothing into Trino, Spark, or Flink. Each engine needs the Datastrato Enterprise connector in its own pods, and every connector is published to Maven Central under the group com.datastrato. The Datastrato artifacts add Oracle and SQL Server catalog support to the open source Gravitino connectors.

The connectors are for the Gravitino REST interface. An engine that only reads Iceberg tables through the Iceberg REST service needs none of these artifacts; it uses its own Iceberg REST client, as described in Iceberg REST clients.

The sections below cover which artifacts each engine takes, how to get them into an engine running on Kubernetes, and how to point the engine at the server.

Quick start​

  1. Find the server's in-cluster address. The chart publishes the server as the service {release}-gravitino on port 8090, enterprise-gravitino with the release name used in Install. Confirm the name in the service list.

    List services in the namespace
    kubectl get svc -n {namespace}
    Server address
    http://{release}-gravitino.{namespace}.svc.cluster.local:8090

    An engine outside the cluster uses the published REST API host instead, as described in Client access.

  2. Pick the artifacts for your engine version from Artifacts.

  3. Get them into the engine and point it at the server, as described for Spark, Flink, or Trino. The examples sign in with a user from the local user store; for OAuth, see the authentication page of each connector.

Artifacts​

Spark and Flink each take two jars on the classpath: the open source Gravitino connector runtime jar and the Datastrato extension jar that sits beside it. Trino takes one zip per Trino version range, which already holds the whole plugin, open source connector included.

Datastrato Enterprise 1.3.0 is built on Apache Gravitino 1.3.1, so the open source jars are version 1.3.1 and the Datastrato artifacts are version 1.3.0.

EngineOpen source runtime jar (org.apache.gravitino)Datastrato artifact (com.datastrato)
Sparkgravitino-spark-connector-runtime-{spark}_{scala}datastrato-spark-connector-ext-{spark}_{scala} jar
Flinkgravitino-flink-connector-runtime-{flink}_2.12datastrato-flink-connector-ext-{flink}_2.12 jar
TrinoIncluded in the zipdatastrato-trino-connector-ext-{range} zip
EngineVersions
Spark3.3 with Scala 2.12; 3.4 and 3.5 with Scala 2.12 or 2.13
Flink1.18, 1.19, and 1.20, Scala 2.12 only
TrinoRanges 435-439, 440-445, 446-451, 452-468, 469-472, and 473-478; pick the one containing your Trino version

Use the open source jar and the Datastrato jar for the same Spark or Flink version and Scala version, and do not mix jars for different engine versions in one deployment.

Spark​

Spark resolves both jars from Maven Central at submit time, so the Spark image does not change. Add the coordinates to the submitted application.

Spark application configuration
spark.jars.packages: org.apache.gravitino:gravitino-spark-connector-runtime-3.5_2.12:1.3.1,com.datastrato:datastrato-spark-connector-ext-3.5_2.12:1.3.0
spark.plugins: org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri: http://{release}-gravitino.{namespace}.svc.cluster.local:8090
spark.sql.gravitino.metalake: {metalake}
spark.sql.gravitino.authType: basic
spark.sql.gravitino.basic.username: {username}
spark.sql.gravitino.basic.password: {password}

A cluster that cannot reach Maven Central resolves the same coordinates from an internal mirror set through Spark's repository settings, or carries both jars in the Spark image.

Iceberg catalogs also need the Iceberg Spark runtime jar on the classpath; see the Spark connector.

Flink loads connectors from its lib directory when the job manager starts, so both jars have to be there before startup. A derived image is the simplest way.

Dockerfile
FROM flink:1.20
ADD https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-flink-connector-runtime-1.20_2.12/1.3.1/gravitino-flink-connector-runtime-1.20_2.12-1.3.1.jar /opt/flink/lib/
ADD https://repo1.maven.org/maven2/com/datastrato/datastrato-flink-connector-ext-1.20_2.12/1.3.0/datastrato-flink-connector-ext-1.20_2.12-1.3.0.jar /opt/flink/lib/

An init container that downloads both jars into a volume mounted at /opt/flink/lib works too. It avoids rebuilding the image for a version change, at the cost of a download on every pod start.

Point the connector at the server through the Flink configuration.

Flink configuration
table.catalog-store.kind: gravitino
table.catalog-store.gravitino.gravitino.uri: http://{release}-gravitino.{namespace}.svc.cluster.local:8090
table.catalog-store.gravitino.gravitino.metalake: {metalake}
table.catalog-store.gravitino.gravitino.client.auth.type: basic
table.catalog-store.gravitino.gravitino.client.basic.username: {username}
table.catalog-store.gravitino.gravitino.client.basic.password: {password}

Trino​

The Trino zip unpacks to one directory holding the complete plugin: the open source connector, its dependencies, and the Datastrato extension. Trino loads every directory under its plugin path, so the unpacked directory goes there under a name of your choosing; the examples use gravitino. The directory has to be in place before Trino starts.

Build an image​

An image survives restarts, rescheduling, and scaling with no further mechanism, and is the approach for production.

Dockerfile
FROM alpine:3.20 AS plugin
ARG RANGE=469-472
ARG VERSION=1.3.0
RUN wget -q -O /tmp/connector.zip \
      https://repo1.maven.org/maven2/com/datastrato/datastrato-trino-connector-ext-${RANGE}/${VERSION}/datastrato-trino-connector-ext-${RANGE}-${VERSION}.zip \
    && unzip -q /tmp/connector.zip -d /tmp \
    && mv /tmp/datastrato-trino-connector-ext-${RANGE}-${VERSION} /gravitino

FROM trinodb/trino:469
COPY --from=plugin /gravitino /usr/lib/trino/plugin/gravitino

The first stage downloads and unpacks the zip, so the Trino image needs no extra tools. The plugin path differs between Trino images and distributions; confirm it for yours.

Use an init container​

An init container that downloads and unpacks the zip into a volume shared with the Trino container avoids maintaining an image, at the cost of a download on every pod start.

Init container that installs the plugin
initContainers:
  - name: gravitino-plugin
    image: alpine:3.20
    command:
      - sh
      - -c
      - |
        wget -q -O /tmp/connector.zip \
          https://repo1.maven.org/maven2/com/datastrato/datastrato-trino-connector-ext-469-472/1.3.0/datastrato-trino-connector-ext-469-472-1.3.0.zip
        unzip -q /tmp/connector.zip -d /tmp
        mkdir -p /plugins/gravitino
        cp /tmp/datastrato-trino-connector-ext-469-472-1.3.0/* /plugins/gravitino/
    volumeMounts:
      - name: gravitino-plugin
        mountPath: /plugins

Mount the same volume into the Trino container so gravitino lands under its plugin path. Whether your Trino chart exposes init containers and extra volumes depends on the chart; check its values first.

Configure Trino​

The coordinator needs dynamic catalog management, or the catalogs from the metalake never appear.

etc/config.properties
catalog.management=dynamic

The catalog file names the connector and the server.

etc/catalog/gravitino.properties
connector.name=gravitino
gravitino.uri=http://{release}-gravitino.{namespace}.svc.cluster.local:8090
gravitino.metalake={metalake}
gravitino.client.authType=basic
gravitino.client.basic.username={username}
gravitino.client.basic.password={password}

Most Trino charts render both files from values rather than a mounted file.

Verify​

List the catalogs and confirm that gravitino appears with the catalogs your metalake exposes.

List catalogs in Trino
kubectl exec -it {trino_pod} -n {namespace} -- trino --execute "SHOW CATALOGS"

When gravitino is missing, the plugin directory is absent or in the wrong place. When gravitino is present but the metalake's catalogs are not, dynamic catalog management is off or the pod cannot reach the server.

A connector copied into a running container works until the pod restarts, then disappears, and a rescheduled or scaled pod starts without it. Use an image or an init container.

Last updated on