Skip to main content

Supported catalogs

The sources Spark supports through the Gravitino Spark connector are listed in Supported sources. Each catalog is backed by the corresponding Spark connector and behaves the way that connector behaves. The difference is that catalogs come from Gravitino rather than from Spark configuration.

Setup​

CatalogConfigurationAdditional JARs
HiveNoneNone
Icebergspark.sql.gravitino.enableIcebergSupport=trueIceberg Spark runtime, version-matched
JDBCNoneThe database's JDBC driver
Paimonspark.sql.gravitino.enablePaimonSupport=truePaimon Spark runtime
GlueNonePatched Hive and Glue client JARs

Catalogs are selected by their Gravitino name with USE. SHOW CATALOGS lists only spark_catalog until a catalog has been used, which is a Spark catalog manager limitation.

Any Gravitino catalog property prefixed with spark.bypass. passes through to the Spark connector with the prefix stripped.

Catalogs​

Each catalog type below has its own requirements, supported statements, and property mapping.

Hive​

Requires Hive metastore 2.x and HDFS 2.x or 3.x.

Most DDL and DML work. Not supported: function operations (use Gravitino UDFs), partition operations, view DDL, LOAD, CREATE TABLE LIKE, and TRUNCATE TABLE. Tables using OpenCSVSerde can be neither read nor written.

Existing views can be read even though view DDL is unsupported. The connector resolves the view's SQL from Gravitino and materializes the entire result on the driver using LocalScan, so a large view will exhaust driver memory.

Gravitino propertySpark Hive configurationDescription
metastore.urishive.metastore.urisHive metastore address

With the spark-sql shell, spark.bypass.spark.sql.hive.metastore.jars must be set explicitly. The default builtin does not work.

For S3, configure the catalog per Hive catalog with cloud storage, set spark.sql.catalog.{catalog}.fs.s3a.access.key and the matching secret key, and add hadoop-aws and aws-java-sdk-bundle to the classpath.

Iceberg​

Spark clients run a different Iceberg version from the Gravitino server, which is on 1.11.0. Take both JARs from the same row, and do not mix Iceberg versions on one classpath.

SparkIcebergIceberg runtime artifactGravitino connector artifact
3.31.8.1iceberg-spark-runtime-3.3_{scala}-1.8.1.jargravitino-spark-connector-runtime-3.3_{scala}-{gravitino_version}.jar
3.41.11.0iceberg-spark-runtime-3.4_{scala}-1.11.0.jargravitino-spark-connector-runtime-3.4_{scala}-{gravitino_version}.jar
3.51.11.0iceberg-spark-runtime-3.5_{scala}-1.11.0.jargravitino-spark-connector-runtime-3.5_{scala}-{gravitino_version}.jar

Scala is 2.12 or 2.13.

A catalog whose catalog-backend is hive or jdbc is routed through the Gravitino Iceberg REST service by default, with the Gravitino catalog name as warehouse and prefix, and storage credentials vended per table. Catalogs with the rest or custom backend keep the translation in the table below.

  • Credential vending is required for object storage. A routed catalog whose warehouse is on S3, GCS, or ADLS fails at load unless the catalog sets credential-providers, because routing replaces static storage keys with vended ones. See Credential vending.
  • The service address is discovered from the Gravitino server. Spark outside the cluster sets spark.sql.gravitino.iceberg.rest-uri to the published address, https://{iceberg_host}/iceberg.
  • OAuth is reused. With spark.sql.gravitino.authType=oauth2, Iceberg signs in to the service with the same client credentials. Basic credentials are not reused; pass them to the Iceberg client under spark.sql.gravitino.iceberg.rest., which reaches Iceberg with that prefix removed.
Iceberg REST authentication for a Basic-auth session
--conf spark.sql.gravitino.iceberg.rest.rest.auth.type=basic \
--conf spark.sql.gravitino.iceberg.rest.rest.auth.basic.username={username} \
--conf spark.sql.gravitino.iceberg.rest.rest.auth.basic.password={password}

spark.sql.gravitino.iceberg.rest-routing-enabled=false restores the translation below for every backend, with static storage keys.

Table DDL and DML work, including MERGE INTO, DELETE FROM, UPDATE, CALL, and time travel. CREATE TABLE does not accept distribution or sort orders. Not supported: views, metadata tables, Iceberg extension SQL such as ADD PARTITION FIELD, WRITE ORDERED BY, CREATE BRANCH, and CREATE TAG, and the atomic create-or-replace-as-select variants.

Gravitino propertySpark Iceberg configurationDescription
catalog-backendtypehive, jdbc, rest, or custom
catalog-backend-implcatalog-implCustom catalog class, for the custom backend
uriuriCatalog backend address
warehousewarehouseCatalog backend warehouse
jdbc-userjdbc.userJDBC user, for the JDBC backend
jdbc-passwordjdbc.passwordJDBC password, for the JDBC backend
io-implio-implThe FileIO implementation
s3-endpoints3.endpointAlternative or private S3 endpoint
s3-regionclient.regionS3 region
s3-access-key-ids3.access-key-idS3 access key
s3-secret-access-keys3.secret-access-keyS3 secret key
s3-path-style-accesss3.path-style-accessWhether to use path style access
oss-endpointoss.endpointAliyun OSS endpoint
oss-access-key-idclient.access-key-idOSS access key
oss-secret-access-keyclient.access-key-secretOSS secret key
azure-storage-account-nameadls.auth.shared-key.account.nameADLS storage account name
azure-storage-account-keyadls.auth.shared-key.account.keyADLS storage account key

cache-enabled is forced to false and cannot be changed.

Storage properties convert automatically, so S3, ADLS, OSS, and GCS need no extra Spark configuration beyond the bundle on the classpath: iceberg-aws-bundle, iceberg-azure-bundle, the Aliyun OSS SDK, or iceberg-gcp-bundle with a credential file reachable through GOOGLE_APPLICATION_CREDENTIALS. Other storage takes spark.sql.catalog.{catalog}.{key} plus a FileIO implementation.

JDBC​

The supported JDBC sources are listed in Supported sources. Oracle and SQL Server come from the Datastrato extension jar.

CREATE TABLE, DROP TABLE, ALTER TABLE, SELECT, and INSERT work. UPDATE, DELETE, and TRUNCATE do not.

Writes are not distributed transactions. Each Spark task commits independently, so a partial failure leaves partial data rather than rolling back.

Spark pushes filters, aggregates, Top-N, limits and offsets down to the source when its pushDown* options allow it, except where the source could return a different result than Spark. These stay in Spark:

  • Expressions on string, char, varchar, binary, UUID or external-type columns, including inside arrays, because the source can compare strings with a different collation. On PostgreSQL and MySQL, a simple string equality or IN filter is also sent to the source as a prefilter to cut the rows transferred, and Spark rechecks it.
  • Expressions with a string or binary literal or cast, such as cast(id AS string) > '9'.
  • Casts to a timestamp type, and casts or date parts of a Spark TIMESTAMP column read from a source type without time zone, such as MySQL DATETIME, because Spark applies the session time zone and the source its own.
  • Date parts other than year, quarter, month, day, hour and minute, such as second() or weekofyear(), because the source can compute them differently.
  • IS NULL and count(column) on a column of an external type.

Everything else is pushed down and runs with the source's semantics: a source avg() can return fewer decimal places than Spark. When a filter stays in Spark, the query's limit and aggregate stay in Spark too.

Gravitino propertySpark JDBC configurationDescription
jdbc-urlurlConnection URL, e.g. jdbc:mysql://{host}:3306
jdbc-userjdbc.userJDBC user
jdbc-passwordjdbc.passwordJDBC password
jdbc-driverdriverDriver class

Spark receives jdbc-user and jdbc-password from the catalog's credentials endpoint and connects to the database with them from the user's own job. Loading a JDBC catalog in Spark therefore takes USE_SECRETS and INCLUDE_CREDENTIAL_SECRETS on the catalog, or ownership of the metalake, and whoever holds them receives the catalog's database account in plaintext. See Catalog credentials before granting them.

PostgreSQL timestamps without time zone​

A PostgreSQL timestamp (without time zone) maps to Spark TIMESTAMP by default, so the value read shifts with the Spark session time zone. To read the stored wall-clock value on Spark 3.5, set spark.sql.catalog.{catalog}.preferTimestampNTZ=true, or the Gravitino catalog property spark.bypass.preferTimestampNTZ=true; the Spark setting wins when both are set. The option is set per catalog, applies to PostgreSQL catalogs only, and is refused on Spark 3.3 and 3.4. With it on:

  • timestamp maps to Spark TIMESTAMP_NTZ, and timestamptz still maps to TIMESTAMP.
  • Writes take the value in the Spark session time zone rather than the JVM time zone, so a job whose two time zones differ writes different values than before.
  • A timestamp column cannot be the partitionColumn.
  • Reading infinity or -infinity fails with long overflow.
  • Timestamp array elements still map to TIMESTAMP on Spark 3.5.

Column defaults from Spark​

A string default on a timestamp column is refused on Spark 3.4 and 3.5; use a TIMESTAMP or TIMESTAMP_NTZ literal instead. ALTER COLUMN ... DROP DEFAULT is refused, while SET DEFAULT NULL and SET DEFAULT '' set those literal values. One ALTER TABLE cannot combine a timestamp, date, or string default with adding, dropping, renaming, or retyping a column.

Paimon​

The connector passes the catalog backend and the S3 and OSS credentials through to Paimon, using the same mapping as the Flink connector. The connector's tests cover only the filesystem backend on HDFS. The hive and jdbc backends and a warehouse on S3 or OSS are passed through but not tested with Spark; object storage also needs Paimon's file IO jar (paimon-s3 or paimon-oss) on the Spark classpath.

Namespace and table DDL work, along with SELECT, INSERT INTO, INSERT OVERWRITE, schema evolution, and partition management. LOAD NAMESPACE does not return user-specified configs, and CREATE TABLE does not accept distribution or sort orders. Not supported: ALTER NAMESPACE, which Paimon itself lacks, row-level operations such as MERGE INTO, DELETE, UPDATE, and TRUNCATE, metadata tables, Paimon extension SQL such as tags, CALL statements, views, and time travel.

Gravitino propertySpark Paimon configurationDescription
catalog-backendmetastoreCatalog backend type
uriuriCatalog backend address
warehousewarehouseCatalog backend warehouse

AWS Glue​

Requires network access to the Glue API and S3, IAM credentials with Glue and S3 permissions as described on the Glue catalog page, and Spark 3.3, 3.4, or 3.5.

Capabilities match Hive: most DDL and DML, without function operations, partition operations, view operations, LOAD, CREATE TABLE LIKE, or TRUNCATE TABLE.

A single database can hold both table formats. Routing is by the table-format property in the Glue table parameters, so table-format=ICEBERG goes to Iceberg's SparkCatalog and everything else to the Hive table catalog. Iceberg tables additionally need iceberg-spark-runtime and iceberg-aws-bundle on the classpath.

Iceberg tables follow the Iceberg type mapping: a Spark TIMESTAMP column becomes an Iceberg timestamptz, and TIMESTAMP_NTZ becomes timestamp. Hive-format tables store TIMESTAMP as a Hive timestamp and do not support TIMESTAMP_NTZ.

Spark's bundled Hive 2.3.9 lacks the HiveMetaStoreClientFactory interface from HIVE-12679 that the Glue client requires, so the bundled Hive JARs must be replaced with patched versions. On Amazon EMR this is already done and the Glue client is preinstalled, so no JAR work is needed there.

Elsewhere, take the patched Hive 2.3.10 and Glue client JARs from the spark3/glue-3.4.0 directory of spark-hive-glue-libs. The directory name is the Glue client version, and the JARs work with Spark 3.3, 3.4, and 3.5. Point Spark's Hive metastore client at them so the bundled Hive JARs are not used.

Load the patched Hive and Glue client
spark-submit \
--conf spark.sql.hive.metastore.version=2.3.10 \
--conf spark.sql.hive.metastore.jars=path \
--conf spark.sql.hive.metastore.jars.path=/opt/glue-hive-jars/* \
--conf spark.sql.hive.metastore.sharedPrefixes=com.amazonaws,org.apache.thrift,org.slf4j,com.google.common

The AWS SDK JARs in that directory load in Spark's isolated Hive client loader, which keeps them from conflicting with hadoop-aws.

Gravitino propertySpark Hive configurationIceberg GlueCatalog property
aws-regionaws.regionclient.region
aws-glue-catalog-idaws.glue.catalog.idglue.id
aws-glue-endpointaws.glue.endpointglue.endpoint
warehouse(Iceberg only)warehouse
aws-access-key-idclient.credentials-provider.*
aws-secret-access-keyclient.credentials-provider.*

For S3, both fs.s3a.* and fs.s3.* must be mapped to S3AFileSystem. The Glue Hive client records table locations with the s3:// scheme while Spark reads through s3a://, so mapping only one leaves Hive-format tables unreadable.

Map both S3 schemes to S3AFileSystem
spark-submit \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.access.key={access_key} \
--conf spark.hadoop.fs.s3a.secret.key={secret_key} \
--conf spark.hadoop.fs.s3a.endpoint.region={region} \
--conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3.access.key={access_key} \
--conf spark.hadoop.fs.s3.secret.key={secret_key}

The same values can be set as spark.bypass.fs.s3a.* catalog properties instead. Either way the AWS Java SDK and Hadoop AWS JARs must be on the classpath.