Supported catalogs
The sources Spark supports through the Gravitino Spark connector are listed in Supported sources. Each catalog is backed by the corresponding Spark connector and behaves the way that connector behaves. The difference is that catalogs come from Gravitino rather than from Spark configuration.
Setup
| Catalog | Configuration | Additional JARs |
|---|---|---|
| Hive | None | None |
| Iceberg | spark.sql.gravitino.enableIcebergSupport=true | Iceberg Spark runtime, version-matched |
| JDBC | None | The database's JDBC driver |
| Paimon | spark.sql.gravitino.enablePaimonSupport=true | Paimon Spark runtime |
| Glue | None | Patched Hive and Glue client JARs |
Catalogs are selected by their Gravitino name with USE. SHOW CATALOGS lists only
spark_catalog until a catalog has been used, which is a Spark catalog manager limitation.
Any Gravitino catalog property prefixed with spark.bypass. passes through to the Spark connector
with the prefix stripped.
Catalogs
Each catalog type below has its own requirements, supported statements, and property mapping.
Hive
Requires Hive metastore 2.x and HDFS 2.x or 3.x.
Most DDL and DML work. Not supported: function operations (use
Gravitino UDFs), partition operations, view DDL, LOAD,
CREATE TABLE LIKE, and TRUNCATE TABLE. Tables using OpenCSVSerde can be neither read nor
written.
Existing views can be read even though view DDL is unsupported. The connector resolves the view's
SQL from Gravitino and materializes the entire result on the driver using LocalScan, so a large
view will exhaust driver memory.
| Gravitino property | Spark Hive configuration | Description |
|---|---|---|
metastore.uris | hive.metastore.uris | Hive metastore address |
With the spark-sql shell, spark.bypass.spark.sql.hive.metastore.jars must be set explicitly.
The default builtin does not work.
For S3, configure the catalog per
Hive catalog with cloud storage, set
spark.sql.catalog.{catalog}.fs.s3a.access.key and the matching secret key, and add
hadoop-aws and
aws-java-sdk-bundle to the
classpath.
Iceberg
Spark clients run a different Iceberg version from the Gravitino server, which is on 1.11.0. Take both JARs from the same row, and do not mix Iceberg versions on one classpath.
| Spark | Iceberg | Iceberg runtime artifact | Gravitino connector artifact |
|---|---|---|---|
| 3.3 | 1.8.1 | iceberg-spark-runtime-3.3_{scala}-1.8.1.jar | gravitino-spark-connector-runtime-3.3_{scala}-{gravitino_version}.jar |
| 3.4 | 1.11.0 | iceberg-spark-runtime-3.4_{scala}-1.11.0.jar | gravitino-spark-connector-runtime-3.4_{scala}-{gravitino_version}.jar |
| 3.5 | 1.11.0 | iceberg-spark-runtime-3.5_{scala}-1.11.0.jar | gravitino-spark-connector-runtime-3.5_{scala}-{gravitino_version}.jar |
Scala is 2.12 or 2.13.
A catalog whose catalog-backend is hive or jdbc is routed through the Gravitino
Iceberg REST service by default, with the Gravitino catalog name as
warehouse and prefix, and storage credentials vended per table. Catalogs with the rest or custom
backend keep the translation in the table below.
- Credential vending is required for object storage. A routed catalog whose warehouse is on S3,
GCS, or ADLS fails at load unless the catalog sets
credential-providers, because routing replaces static storage keys with vended ones. See Credential vending. - The service address is discovered from the Gravitino server. Spark outside the cluster sets
spark.sql.gravitino.iceberg.rest-urito the published address,https://{iceberg_host}/iceberg. - OAuth is reused. With
spark.sql.gravitino.authType=oauth2, Iceberg signs in to the service with the same client credentials. Basic credentials are not reused; pass them to the Iceberg client underspark.sql.gravitino.iceberg.rest., which reaches Iceberg with that prefix removed.
--conf spark.sql.gravitino.iceberg.rest.rest.auth.type=basic \
--conf spark.sql.gravitino.iceberg.rest.rest.auth.basic.username={username} \
--conf spark.sql.gravitino.iceberg.rest.rest.auth.basic.password={password}
spark.sql.gravitino.iceberg.rest-routing-enabled=false restores the translation below for every
backend, with static storage keys.
Table DDL and DML work, including MERGE INTO, DELETE FROM, UPDATE, CALL, and time travel.
CREATE TABLE does not accept distribution or sort orders. Not supported: views, metadata tables,
Iceberg extension SQL such as ADD PARTITION FIELD, WRITE ORDERED BY, CREATE BRANCH, and
CREATE TAG, and the atomic create-or-replace-as-select variants.
| Gravitino property | Spark Iceberg configuration | Description |
|---|---|---|
catalog-backend | type | hive, jdbc, rest, or custom |
catalog-backend-impl | catalog-impl | Custom catalog class, for the custom backend |
uri | uri | Catalog backend address |
warehouse | warehouse | Catalog backend warehouse |
jdbc-user | jdbc.user | JDBC user, for the JDBC backend |
jdbc-password | jdbc.password | JDBC password, for the JDBC backend |
io-impl | io-impl | The FileIO implementation |
s3-endpoint | s3.endpoint | Alternative or private S3 endpoint |
s3-region | client.region | S3 region |
s3-access-key-id | s3.access-key-id | S3 access key |
s3-secret-access-key | s3.secret-access-key | S3 secret key |
s3-path-style-access | s3.path-style-access | Whether to use path style access |
oss-endpoint | oss.endpoint | Aliyun OSS endpoint |
oss-access-key-id | client.access-key-id | OSS access key |
oss-secret-access-key | client.access-key-secret | OSS secret key |
azure-storage-account-name | adls.auth.shared-key.account.name | ADLS storage account name |
azure-storage-account-key | adls.auth.shared-key.account.key | ADLS storage account key |
cache-enabled is forced to false and cannot be changed.
Storage properties convert automatically, so S3, ADLS, OSS, and GCS need no extra Spark
configuration beyond the bundle on the classpath:
iceberg-aws-bundle,
iceberg-azure-bundle,
the Aliyun OSS SDK, or
iceberg-gcp-bundle with
a credential file reachable through GOOGLE_APPLICATION_CREDENTIALS. Other storage takes
spark.sql.catalog.{catalog}.{key} plus a FileIO implementation.
JDBC
The supported JDBC sources are listed in Supported sources. Oracle and SQL Server come from the Datastrato extension jar.
CREATE TABLE, DROP TABLE, ALTER TABLE, SELECT, and INSERT work. UPDATE, DELETE, and
TRUNCATE do not.
Writes are not distributed transactions. Each Spark task commits independently, so a partial failure leaves partial data rather than rolling back.
Spark pushes filters, aggregates, Top-N, limits and offsets down to the source when its pushDown*
options allow it, except where the source could return a different result than Spark. These stay in
Spark:
- Expressions on string, char, varchar, binary, UUID or external-type columns, including inside
arrays, because the source can compare strings with a different collation. On PostgreSQL and MySQL,
a simple string equality or
INfilter is also sent to the source as a prefilter to cut the rows transferred, and Spark rechecks it. - Expressions with a string or binary literal or cast, such as
cast(id AS string) > '9'. - Casts to a timestamp type, and casts or date parts of a Spark
TIMESTAMPcolumn read from a source type without time zone, such as MySQLDATETIME, because Spark applies the session time zone and the source its own. - Date parts other than year, quarter, month, day, hour and minute, such as
second()orweekofyear(), because the source can compute them differently. IS NULLandcount(column)on a column of an external type.
Everything else is pushed down and runs with the source's semantics: a source avg() can return
fewer decimal places than Spark. When a filter stays in Spark, the query's limit and aggregate stay
in Spark too.
| Gravitino property | Spark JDBC configuration | Description |
|---|---|---|
jdbc-url | url | Connection URL, e.g. jdbc:mysql://{host}:3306 |
jdbc-user | jdbc.user | JDBC user |
jdbc-password | jdbc.password | JDBC password |
jdbc-driver | driver | Driver class |
Spark receives jdbc-user and jdbc-password from the catalog's credentials endpoint and connects to
the database with them from the user's own job. Loading a JDBC catalog in Spark therefore takes
USE_SECRETS and INCLUDE_CREDENTIAL_SECRETS on the catalog, or ownership of the metalake, and
whoever holds them receives the catalog's database account in plaintext. See
Catalog credentials before granting them.
PostgreSQL timestamps without time zone
A PostgreSQL timestamp (without time zone) maps to Spark TIMESTAMP by default, so the value read
shifts with the Spark session time zone. To read the stored wall-clock value on Spark 3.5, set
spark.sql.catalog.{catalog}.preferTimestampNTZ=true, or the Gravitino catalog property
spark.bypass.preferTimestampNTZ=true; the Spark setting wins when both are set. The option is set
per catalog, applies to PostgreSQL catalogs only, and is refused on Spark 3.3 and 3.4. With it on:
timestampmaps to SparkTIMESTAMP_NTZ, andtimestamptzstill maps toTIMESTAMP.- Writes take the value in the Spark session time zone rather than the JVM time zone, so a job whose two time zones differ writes different values than before.
- A timestamp column cannot be the
partitionColumn. - Reading
infinityor-infinityfails withlong overflow. - Timestamp array elements still map to
TIMESTAMPon Spark 3.5.
Column defaults from Spark
A string default on a timestamp column is refused on Spark 3.4 and 3.5; use a TIMESTAMP or
TIMESTAMP_NTZ literal instead. ALTER COLUMN ... DROP DEFAULT is refused, while
SET DEFAULT NULL and SET DEFAULT '' set those literal values. One ALTER TABLE cannot combine a
timestamp, date, or string default with adding, dropping, renaming, or retyping a column.
Paimon
The connector passes the catalog backend and the S3 and OSS credentials through to Paimon, using
the same mapping as the Flink connector. The connector's tests cover only the filesystem backend
on HDFS. The hive and jdbc backends and a warehouse on S3 or OSS are passed through but not
tested with Spark; object storage also needs Paimon's file IO jar (paimon-s3 or paimon-oss) on
the Spark classpath.
Namespace and table DDL work, along with SELECT, INSERT INTO, INSERT OVERWRITE, schema
evolution, and partition management. LOAD NAMESPACE does not return user-specified configs, and
CREATE TABLE does not accept distribution or sort orders. Not supported: ALTER NAMESPACE, which
Paimon itself lacks, row-level operations such as MERGE INTO, DELETE, UPDATE, and TRUNCATE,
metadata tables, Paimon extension SQL such as tags, CALL statements, views, and time travel.
| Gravitino property | Spark Paimon configuration | Description |
|---|---|---|
catalog-backend | metastore | Catalog backend type |
uri | uri | Catalog backend address |
warehouse | warehouse | Catalog backend warehouse |
AWS Glue
Requires network access to the Glue API and S3, IAM credentials with Glue and S3 permissions as described on the Glue catalog page, and Spark 3.3, 3.4, or 3.5.
Capabilities match Hive: most DDL and DML, without function operations, partition operations, view
operations, LOAD, CREATE TABLE LIKE, or TRUNCATE TABLE.
A single database can hold both table formats. Routing is by the table-format property in the
Glue table parameters, so table-format=ICEBERG goes to Iceberg's SparkCatalog and everything
else to the Hive table catalog. Iceberg tables additionally need iceberg-spark-runtime and
iceberg-aws-bundle on the classpath.
Iceberg tables follow the Iceberg type mapping: a Spark TIMESTAMP column becomes an Iceberg
timestamptz, and TIMESTAMP_NTZ becomes timestamp. Hive-format tables store TIMESTAMP as a
Hive timestamp and do not support TIMESTAMP_NTZ.
Spark's bundled Hive 2.3.9 lacks the HiveMetaStoreClientFactory interface from
HIVE-12679 that the Glue client requires, so
the bundled Hive JARs must be replaced with patched versions. On Amazon EMR this is already done
and the Glue client is preinstalled, so no JAR work is needed there.
Elsewhere, take the patched Hive 2.3.10 and Glue client JARs from the spark3/glue-3.4.0 directory
of spark-hive-glue-libs. The directory name is
the Glue client version, and the JARs work with Spark 3.3, 3.4, and 3.5. Point Spark's Hive
metastore client at them so the bundled Hive JARs are not used.
spark-submit \
--conf spark.sql.hive.metastore.version=2.3.10 \
--conf spark.sql.hive.metastore.jars=path \
--conf spark.sql.hive.metastore.jars.path=/opt/glue-hive-jars/* \
--conf spark.sql.hive.metastore.sharedPrefixes=com.amazonaws,org.apache.thrift,org.slf4j,com.google.common
The AWS SDK JARs in that directory load in Spark's isolated Hive client loader, which keeps them
from conflicting with hadoop-aws.
| Gravitino property | Spark Hive configuration | Iceberg GlueCatalog property |
|---|---|---|
aws-region | aws.region | client.region |
aws-glue-catalog-id | aws.glue.catalog.id | glue.id |
aws-glue-endpoint | aws.glue.endpoint | glue.endpoint |
warehouse | (Iceberg only) | warehouse |
aws-access-key-id | client.credentials-provider.* | |
aws-secret-access-key | client.credentials-provider.* |
For S3, both fs.s3a.* and fs.s3.* must be mapped to S3AFileSystem. The Glue Hive client
records table locations with the s3:// scheme while Spark reads through s3a://, so mapping only
one leaves Hive-format tables unreadable.
spark-submit \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.access.key={access_key} \
--conf spark.hadoop.fs.s3a.secret.key={secret_key} \
--conf spark.hadoop.fs.s3a.endpoint.region={region} \
--conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3.access.key={access_key} \
--conf spark.hadoop.fs.s3.secret.key={secret_key}
The same values can be set as spark.bypass.fs.s3a.* catalog properties instead. Either way the
AWS Java SDK and Hadoop AWS JARs must be on the classpath.