Supported catalogs
The sources Spark supports through the Gravitino Spark connector are listed in Supported sources. Each catalog is backed by the corresponding Spark connector and behaves the way that connector behaves. The difference is that catalogs come from Gravitino rather than from Spark configuration.
Setup
| Catalog | Configuration | Additional JARs |
|---|---|---|
| Hive | None | None |
| Iceberg | spark.sql.gravitino.enableIcebergSupport=true | Iceberg Spark runtime, version-matched |
| JDBC | None | The database's JDBC driver |
| Paimon | spark.sql.gravitino.enablePaimonSupport=true | Paimon Spark runtime |
| Glue | None | Patched Hive and Glue client JARs |
Catalogs are selected by their Gravitino name with USE. SHOW CATALOGS lists only
spark_catalog until a catalog has been used, which is a Spark catalog manager limitation.
Any Gravitino catalog property prefixed with spark.bypass. passes through to the Spark connector
with the prefix stripped.
Catalogs
Each catalog type below has its own requirements, supported statements, and property mapping.
Hive
Requires Hive metastore 2.x and HDFS 2.x or 3.x.
Most DDL and DML work. Not supported: function operations (use
Gravitino UDFs), partition operations, view DDL, LOAD,
CREATE TABLE LIKE, and TRUNCATE TABLE. Tables using OpenCSVSerde can be neither read nor
written.
Existing views can be read even though view DDL is unsupported. The connector resolves the view's
SQL from Gravitino and materializes the entire result on the driver using LocalScan, so a large
view will exhaust driver memory.
| Gravitino property | Spark Hive configuration | Description |
|---|---|---|
metastore.uris | hive.metastore.uris | Hive metastore address |
With the spark-sql shell, spark.bypass.spark.sql.hive.metastore.jars must be set explicitly.
The default builtin does not work.
For S3, configure the catalog per
Hive catalog with cloud storage, set
spark.sql.catalog.{catalog}.fs.s3a.access.key and the matching secret key, and add
hadoop-aws and
aws-java-sdk-bundle to the
classpath.
Iceberg
Spark clients run a different Iceberg version from the Gravitino server, which is on 1.11.0. Take both JARs from the same row, and do not mix Iceberg versions on one classpath.
| Spark | Iceberg | Iceberg runtime artifact | Gravitino connector artifact |
|---|---|---|---|
| 3.3 | 1.8.1 | iceberg-spark-runtime-3.3_{scala}-1.8.1.jar | gravitino-spark-connector-runtime-3.3_{scala}-{gravitino_version}.jar |
| 3.4 | 1.11.0 | iceberg-spark-runtime-3.4_{scala}-1.11.0.jar | gravitino-spark-connector-runtime-3.4_{scala}-{gravitino_version}.jar |
| 3.5 | 1.11.0 | iceberg-spark-runtime-3.5_{scala}-1.11.0.jar | gravitino-spark-connector-runtime-3.5_{scala}-{gravitino_version}.jar |
Scala is 2.12 or 2.13.
A catalog whose catalog-backend is hive or jdbc is routed through the Gravitino
Iceberg REST service by default, with the Gravitino catalog name as
warehouse and prefix, and storage credentials vended per table. Catalogs with the rest or custom
backend keep the translation in the table below.
- Credential vending is required for object storage. A routed catalog whose warehouse is on S3,
GCS, or ADLS fails at load unless the catalog sets
credential-providers, because routing replaces static storage keys with vended ones. See Credential vending. - The service address is discovered from the Gravitino server. Spark outside the cluster sets
spark.sql.gravitino.iceberg.rest-urito the published address,https://{iceberg_host}/iceberg. - OAuth is reused. With
spark.sql.gravitino.authType=oauth2, Iceberg signs in to the service with the same client credentials. Basic credentials are not reused; pass them to the Iceberg client underspark.sql.gravitino.iceberg.rest., which reaches Iceberg with that prefix removed.
--conf spark.sql.gravitino.iceberg.rest.rest.auth.type=basic \
--conf spark.sql.gravitino.iceberg.rest.rest.auth.basic.username={username} \
--conf spark.sql.gravitino.iceberg.rest.rest.auth.basic.password={password}
spark.sql.gravitino.iceberg.rest-routing-enabled=false restores the translation below for every
backend, with static storage keys.
Table DDL and DML work, including MERGE INTO, DELETE FROM, UPDATE, CALL, and time travel.
CREATE TABLE does not accept distribution or sort orders. Not supported: views, metadata tables,
Iceberg extension SQL such as ADD PARTITION FIELD, WRITE ORDERED BY, CREATE BRANCH, and
CREATE TAG, and the atomic create-or-replace-as-select variants.
| Gravitino property | Spark Iceberg configuration | Description |
|---|---|---|
catalog-backend | type | hive, jdbc, rest, or custom |
catalog-backend-impl | catalog-impl | Custom catalog class, for the custom backend |
uri | uri | Catalog backend address |
warehouse | warehouse | Catalog backend warehouse |
jdbc-user | jdbc.user | JDBC user, for the JDBC backend |
jdbc-password | jdbc.password | JDBC password, for the JDBC backend |
io-impl | io-impl | The FileIO implementation |
s3-endpoint | s3.endpoint | Alternative or private S3 endpoint |
s3-region | client.region | S3 region |
s3-access-key-id | s3.access-key-id | S3 access key |
s3-secret-access-key | s3.secret-access-key | S3 secret key |
s3-path-style-access | s3.path-style-access | Whether to use path style access |
oss-endpoint | oss.endpoint | Aliyun OSS endpoint |
oss-access-key-id | client.access-key-id | OSS access key |
oss-secret-access-key | client.access-key-secret | OSS secret key |
azure-storage-account-name | adls.auth.shared-key.account.name | ADLS storage account name |
azure-storage-account-key | adls.auth.shared-key.account.key | ADLS storage account key |
cache-enabled is forced to false and cannot be changed.
Storage properties convert automatically, so S3, ADLS, OSS, and GCS need no extra Spark
configuration beyond the bundle on the classpath:
iceberg-aws-bundle,
iceberg-azure-bundle,
the Aliyun OSS SDK, or
iceberg-gcp-bundle with
a credential file reachable through GOOGLE_APPLICATION_CREDENTIALS. Other storage takes
spark.sql.catalog.{catalog}.{key} plus a FileIO implementation.
JDBC
The supported JDBC sources are listed in Supported sources. Oracle and SQL Server come from the Datastrato extension jar.
CREATE TABLE, DROP TABLE, ALTER TABLE, SELECT, and INSERT work. UPDATE, DELETE, and
TRUNCATE do not.
Writes are not distributed transactions. Each Spark task commits independently, so a partial failure leaves partial data rather than rolling back.
| Gravitino property | Spark JDBC configuration | Description |
|---|---|---|
jdbc-url | url | Connection URL, e.g. jdbc:mysql://{host}:3306 |
jdbc-user | jdbc.user | JDBC user |
jdbc-password | jdbc.password | JDBC password |
jdbc-driver | driver | Driver class |
Paimon
The connector passes the catalog backend and the S3 and OSS credentials through to Paimon, using
the same mapping as the Flink connector. The connector's tests cover only the filesystem backend
on HDFS. The hive and jdbc backends and a warehouse on S3 or OSS are passed through but not
tested with Spark; object storage also needs Paimon's file IO jar (paimon-s3 or paimon-oss) on
the Spark classpath.
Namespace and table DDL work, along with SELECT, INSERT INTO, INSERT OVERWRITE, schema
evolution, and partition management. LOAD NAMESPACE does not return user-specified configs, and
CREATE TABLE does not accept distribution or sort orders. Not supported: ALTER NAMESPACE, which
Paimon itself lacks, row-level operations such as MERGE INTO, DELETE, UPDATE, and TRUNCATE,
metadata tables, Paimon extension SQL such as tags, CALL statements, views, and time travel.
| Gravitino property | Spark Paimon configuration | Description |
|---|---|---|
catalog-backend | metastore | Catalog backend type |
uri | uri | Catalog backend address |
warehouse | warehouse | Catalog backend warehouse |
AWS Glue
Requires network access to the Glue API and S3, IAM credentials with Glue and S3 permissions as described on the Glue catalog page, and Spark 3.3, 3.4, or 3.5.
Capabilities match Hive: most DDL and DML, without function operations, partition operations, view
operations, LOAD, CREATE TABLE LIKE, or TRUNCATE TABLE.
A single database can hold both table formats. Routing is by the table-format property in the
Glue table parameters, so table-format=ICEBERG goes to Iceberg's SparkCatalog and everything
else to the Hive table catalog. Iceberg tables additionally need iceberg-spark-runtime and
iceberg-aws-bundle on the classpath.
Spark's bundled Hive 2.3.9 lacks the HiveMetaStoreClientFactory interface from
HIVE-12679 that the Glue client requires, so
the bundled Hive JARs must be replaced with patched versions. On Amazon EMR this is already done
and the Glue client is preinstalled, so no JAR work is needed there.
Elsewhere, take the patched Hive 2.3.10 and Glue client JARs from the spark3/glue-3.4.0 directory
of spark-hive-glue-libs. The directory name is
the Glue client version, and the JARs work with Spark 3.3, 3.4, and 3.5. Point Spark's Hive
metastore client at them so the bundled Hive JARs are not used.
spark-submit \
--conf spark.sql.hive.metastore.version=2.3.10 \
--conf spark.sql.hive.metastore.jars=path \
--conf spark.sql.hive.metastore.jars.path=/opt/glue-hive-jars/* \
--conf spark.sql.hive.metastore.sharedPrefixes=com.amazonaws,org.apache.thrift,org.slf4j,com.google.common
The AWS SDK JARs in that directory load in Spark's isolated Hive client loader, which keeps them
from conflicting with hadoop-aws.
| Gravitino property | Spark Hive configuration | Iceberg GlueCatalog property |
|---|---|---|
aws-region | aws.region | client.region |
aws-glue-catalog-id | aws.glue.catalog.id | glue.id |
aws-glue-endpoint | aws.glue.endpoint | glue.endpoint |
warehouse | (Iceberg only) | warehouse |
aws-access-key-id | client.credentials-provider.* | |
aws-secret-access-key | client.credentials-provider.* |
For S3, both fs.s3a.* and fs.s3.* must be mapped to S3AFileSystem. The Glue Hive client
records table locations with the s3:// scheme while Spark reads through s3a://, so mapping only
one leaves Hive-format tables unreadable.
spark-submit \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.access.key={access_key} \
--conf spark.hadoop.fs.s3a.secret.key={secret_key} \
--conf spark.hadoop.fs.s3a.endpoint.region={region} \
--conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3.access.key={access_key} \
--conf spark.hadoop.fs.s3.secret.key={secret_key}
The same values can be set as spark.bypass.fs.s3a.* catalog properties instead. Either way the
AWS Java SDK and Hadoop AWS JARs must be on the classpath.