Supported catalogs
The Gravitino Spark connector exposes Hive, Iceberg, JDBC, Paimon, and AWS Glue catalogs to Spark. Each is backed by the corresponding Spark connector and behaves the way that connector behaves. The difference is that catalogs come from Gravitino rather than from Spark configuration.
Setup
| Catalog | Configuration | Additional JARs |
|---|---|---|
| Hive | None | None |
| Iceberg | spark.sql.gravitino.enableIcebergSupport=true | Iceberg Spark runtime, version-matched |
| JDBC | None | The database's JDBC driver |
| Paimon | spark.sql.gravitino.enablePaimonSupport=true | Paimon Spark runtime |
| Glue | None | Patched Hive and Glue client JARs |
Catalogs are selected by their Gravitino name with USE. SHOW CATALOGS lists only
spark_catalog until a catalog has been used, which is a Spark catalog manager limitation.
Any Gravitino catalog property prefixed with spark.bypass. passes through to the Spark connector
with the prefix stripped.
Hive
Requires Hive metastore 2.x and HDFS 2.x or 3.x.
Most DDL and DML work. Not supported: function operations (use
Gravitino UDFs), partition operations, view DDL, LOAD,
CREATE TABLE LIKE, and TRUNCATE TABLE. Tables using OpenCSVSerde can be neither read nor
written.
Existing views can be read even though view DDL is unsupported. The connector resolves the view's
SQL from Gravitino and materializes the entire result on the driver using LocalScan, so a large
view will exhaust driver memory.
| Gravitino property | Spark Hive configuration | Description |
|---|---|---|
metastore.uris | hive.metastore.uris | Hive metastore address |
With the spark-sql shell, spark.bypass.spark.sql.hive.metastore.jars must be set explicitly.
The default builtin does not work.
For S3, configure the catalog per
Hive catalog with cloud storage, set
spark.sql.catalog.{catalog}.fs.s3a.access.key and the matching secret key, and add
hadoop-aws and
aws-java-sdk-bundle to the
classpath.
Iceberg
Spark clients run a different Iceberg version from the Gravitino server, which is on 1.11.0. Take both JARs from the same row, and do not mix Iceberg versions on one classpath.
| Spark | Iceberg | Iceberg runtime artifact | Gravitino connector artifact |
|---|---|---|---|
| 3.3 | 1.8.1 | iceberg-spark-runtime-3.3_{scala}-1.8.1.jar | gravitino-spark-connector-runtime-3.3_{scala}-{gravitino_version}.jar |
| 3.4 | 1.11.0 | iceberg-spark-runtime-3.4_{scala}-1.11.0.jar | gravitino-spark-connector-runtime-3.4_{scala}-{gravitino_version}.jar |
| 3.5 | 1.11.0 | iceberg-spark-runtime-3.5_{scala}-1.11.0.jar | gravitino-spark-connector-runtime-3.5_{scala}-{gravitino_version}.jar |
Scala is 2.12 or 2.13.
Table DDL and DML work, including MERGE INTO, DELETE FROM, UPDATE, CALL, and time travel.
CREATE TABLE does not accept distribution or sort orders. Not supported: views, metadata tables,
Iceberg extension SQL such as ADD PARTITION FIELD, WRITE ORDERED BY, CREATE BRANCH, and
CREATE TAG, and the atomic create-or-replace-as-select variants.
| Gravitino property | Spark Iceberg configuration | Description |
|---|---|---|
catalog-backend | type | hive, jdbc, rest, or custom |
catalog-backend-impl | catalog-impl | Custom catalog class, for the custom backend |
uri | uri | Catalog backend address |
warehouse | warehouse | Catalog backend warehouse |
jdbc-user | jdbc.user | JDBC user, for the JDBC backend |
jdbc-password | jdbc.password | JDBC password, for the JDBC backend |
io-impl | io-impl | The FileIO implementation |
s3-endpoint | s3.endpoint | Alternative or private S3 endpoint |
s3-region | client.region | S3 region |
s3-access-key-id | s3.access-key-id | S3 access key |
s3-secret-access-key | s3.secret-access-key | S3 secret key |
s3-path-style-access | s3.path-style-access | Whether to use path style access |
oss-endpoint | oss.endpoint | Aliyun OSS endpoint |
oss-access-key-id | client.access-key-id | OSS access key |
oss-secret-access-key | client.access-key-secret | OSS secret key |
azure-storage-account-name | adls.auth.shared-key.account.name | ADLS storage account name |
azure-storage-account-key | adls.auth.shared-key.account.key | ADLS storage account key |
cache-enabled is forced to false and cannot be changed.
Storage properties convert automatically, so S3, ADLS, OSS, and GCS need no extra Spark
configuration beyond the bundle on the classpath:
iceberg-aws-bundle,
iceberg-azure-bundle,
the Aliyun OSS SDK, or
iceberg-gcp-bundle with
a credential file reachable through GOOGLE_APPLICATION_CREDENTIALS. Other storage takes
spark.sql.catalog.{catalog}.{key} plus a FileIO implementation.
JDBC
MySQL and PostgreSQL are supported. OceanBase works through the MySQL driver. Doris does not, since it lacks MySQL dialect support.
CREATE TABLE, DROP TABLE, ALTER TABLE, SELECT, and INSERT work. UPDATE, DELETE, and
TRUNCATE do not.
Writes are not distributed transactions. Each Spark task commits independently, so a partial failure leaves partial data rather than rolling back.
| Gravitino property | Spark JDBC configuration | Description |
|---|---|---|
jdbc-url | url | Connection URL, e.g. jdbc:mysql://{host}:3306 |
jdbc-user | jdbc.user | JDBC user |
jdbc-password | jdbc.password | JDBC password |
jdbc-driver | driver | Driver class |
Paimon
Only the FilesystemCatalog on HDFS is supported. Hive and JDBC backends are not, and neither is object storage for FilesystemCatalog.
Namespace and table DDL work, along with SELECT, INSERT INTO, INSERT OVERWRITE, schema
evolution, and partition management. LOAD NAMESPACE does not return user-specified configs, and
CREATE TABLE does not accept distribution or sort orders. Not supported: ALTER NAMESPACE, which
Paimon itself lacks, row-level operations such as MERGE INTO, DELETE, UPDATE, and TRUNCATE,
metadata tables, Paimon extension SQL such as tags, CALL statements, views, and time travel.
| Gravitino property | Spark Paimon configuration | Description |
|---|---|---|
catalog-backend | metastore | Catalog backend type |
uri | uri | Catalog backend address |
warehouse | warehouse | Catalog backend warehouse |
AWS Glue
Requires network access to the Glue API and S3, IAM credentials with Glue and S3 permissions as described on the Glue catalog page, and Spark 3.3, 3.4, or 3.5.
Capabilities match Hive: most DDL and DML, without function operations, partition operations, view
operations, LOAD, CREATE TABLE LIKE, or TRUNCATE TABLE.
A single database can hold both table formats. Routing is by the table-format property in the
Glue table parameters, so table-format=ICEBERG goes to Iceberg's SparkCatalog and everything
else to the Hive table catalog. Iceberg tables additionally need iceberg-spark-runtime and
iceberg-aws-bundle on the classpath.
Spark's bundled Hive 2.3.9 lacks the HiveMetaStoreClientFactory interface from
HIVE-12679 that the Glue client requires, so
the bundled Hive JARs must be replaced with patched versions. On Amazon EMR this is already done
and the Glue client is preinstalled, so no JAR work is needed there.
| Gravitino property | Spark Hive configuration | Iceberg GlueCatalog property |
|---|---|---|
aws-region | aws.region | client.region |
aws-glue-catalog-id | aws.glue.catalog.id | glue.id |
aws-glue-endpoint | aws.glue.endpoint | glue.endpoint |
warehouse | (Iceberg only) | warehouse |
aws-access-key-id | client.credentials-provider.* | |
aws-secret-access-key | client.credentials-provider.* |
For S3, both fs.s3a.* and fs.s3.* must be mapped to S3AFileSystem. The Glue Hive client
records table locations with the s3:// scheme while Spark reads through s3a://, so mapping only
one leaves Hive-format tables unreadable.
spark-submit \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.access.key={access_key} \
--conf spark.hadoop.fs.s3a.secret.key={secret_key} \
--conf spark.hadoop.fs.s3a.endpoint.region={region} \
--conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3.access.key={access_key} \
--conf spark.hadoop.fs.s3.secret.key={secret_key}
The same values can be set as spark.bypass.fs.s3a.* catalog properties instead. Either way the
AWS Java SDK and Hadoop AWS JARs must be on the classpath.