Connect Spark to Iceberg REST

Introduction

Apache Gravitino exposes an Iceberg REST catalog endpoint that any Iceberg-compatible engine can connect to directly, without installing a Gravitino-specific connector plugin. This page describes how to configure Apache Spark to use Gravitino's Iceberg REST (IRC) endpoint.

This integration uses the standard Apache Iceberg REST catalog specification. Gravitino enforces its full access-control model on all IRC requests. Per-user identity propagation from the engine is planned for a future release; current requests are authorized using the credentials supplied in the Spark configuration.

Prerequisites

• Apache Gravitino running with the Iceberg REST service enabled. See Iceberg REST catalog service for setup instructions.
• The Gravitino IRC endpoint is accessible from your Spark environment. The default port is 9001.
• The following JAR files available in your Spark environment:
  • iceberg-spark-runtime-3.5_2.12-1.7.1.jar
  • hadoop-aws-3.3.4.jar
  • aws-bundle-2.29.38.jar

This page uses Spark 3.5.3 with Iceberg 1.7.1. For other versions, ensure compatibility between Spark, Scala, and Iceberg runtime versions.

Configuration

spark-defaults.conf is Spark's persistent configuration file. Properties set here are automatically applied to every Spark session, with no command-line flags needed.

spark-defaults.conf location

$SPARK_HOME/conf/spark-defaults.conf

If the file doesn't exist yet, copy the template:

Create spark-defaults.conf from the template

cp $SPARK_HOME/conf/spark-defaults.conf.template $SPARK_HOME/conf/spark-defaults.conf

Simple authentication

spark-defaults.conf

# Iceberg extensions
spark.sql.extensions                                    org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

# Gravitino IRC catalog
spark.sql.catalog.gravitino_irc                         org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.gravitino_irc.type                    rest
spark.sql.catalog.gravitino_irc.uri                     http://{gravitino_host}:9001/iceberg

# S3 FileIO
spark.sql.catalog.gravitino_irc.io-impl                 org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.gravitino_irc.s3.region               us-east-1
spark.sql.catalog.gravitino_irc.s3.access-key-id        {access_key}
spark.sql.catalog.gravitino_irc.s3.secret-access-key    {secret_key}

# Hadoop S3A (for s3a:// paths)
spark.hadoop.fs.s3a.impl                                org.apache.hadoop.fs.s3a.S3AFileSystem

# Set as default catalog (optional)
spark.sql.defaultCatalog                                gravitino_irc

gravitino_irc is the catalog identifier used within Spark. It maps to the Gravitino IRC endpoint via the uri property. You may use any identifier you prefer. S3 credentials can alternatively be supplied via environment variables (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) or an IAM instance profile, in which case the explicit credential lines can be omitted.

Basic authentication

If Gravitino uses local users and groups for Basic authentication, add the auth properties.

spark-defaults.conf

# Iceberg extensions
spark.sql.extensions                                      org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

# Gravitino IRC catalog
spark.sql.catalog.gravitino_irc                           org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.gravitino_irc.type                      rest
spark.sql.catalog.gravitino_irc.uri                       http://{gravitino_host}:9001/iceberg

# Basic authentication
spark.sql.catalog.gravitino_irc.rest.auth.type            basic
spark.sql.catalog.gravitino_irc.rest.auth.basic.username  {username}
spark.sql.catalog.gravitino_irc.rest.auth.basic.password  {password}

# S3 FileIO
spark.sql.catalog.gravitino_irc.io-impl                   org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.gravitino_irc.s3.region                 us-east-1
spark.sql.catalog.gravitino_irc.s3.access-key-id          {access_key}
spark.sql.catalog.gravitino_irc.s3.secret-access-key      {secret_key}

# Hadoop S3A (for s3a:// paths)
spark.hadoop.fs.s3a.impl                                  org.apache.hadoop.fs.s3a.S3AFileSystem

# Set as default catalog (optional)
spark.sql.defaultCatalog                                  gravitino_irc

OAuth authentication

If Gravitino is configured with OAuth, add the auth properties to the same file.

spark-defaults.conf

# Iceberg extensions
spark.sql.extensions                                    org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

# Gravitino IRC catalog
spark.sql.catalog.gravitino_irc                         org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.gravitino_irc.type                    rest
spark.sql.catalog.gravitino_irc.uri                     http://{gravitino_host}:9001/iceberg

# OAuth2 authentication
spark.sql.catalog.gravitino_irc.rest.auth.type          oauth2
spark.sql.catalog.gravitino_irc.rest.auth.oauth2.token  {token}

# S3 FileIO
spark.sql.catalog.gravitino_irc.io-impl                 org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.gravitino_irc.s3.region               us-east-1
spark.sql.catalog.gravitino_irc.s3.access-key-id        {access_key}
spark.sql.catalog.gravitino_irc.s3.secret-access-key    {secret_key}

# Hadoop S3A (for s3a:// paths)
spark.hadoop.fs.s3a.impl                                org.apache.hadoop.fs.s3a.S3AFileSystem

# Set as default catalog (optional)
spark.sql.defaultCatalog                                gravitino_irc

See How to authenticate for Gravitino authentication configuration options.

Local development.

For local development, MinIO can be used as an S3-compatible storage backend. Replace the S3 FileIO section with:

S3 FileIO properties for MinIO

spark.sql.catalog.gravitino_irc.io-impl                 org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.gravitino_irc.s3.endpoint             http://{minio_host}:9000
spark.sql.catalog.gravitino_irc.s3.path-style-access    true
spark.sql.catalog.gravitino_irc.s3.access-key-id        {minio_access_key}
spark.sql.catalog.gravitino_irc.s3.secret-access-key    {minio_secret_key}
spark.hadoop.fs.s3a.impl                                org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.endpoint                            http://{minio_host}:9000
spark.hadoop.fs.s3a.path.style.access                   true
spark.hadoop.fs.s3a.connection.ssl.enabled              false

See gravitino-irc-quickstart for a complete local development environment using MinIO.

Credential vending

If Gravitino is configured with credential vending, add the following to enable it on the client side:

Request vended credentials

spark.sql.catalog.gravitino_irc.header.X-Iceberg-Access-Delegation    vended-credentials

See Credential vending for server-side configuration.

For storage not managed by Gravitino, properties are not automatically transferred from the server to the client. Pass custom properties to initialize FileIO explicitly:

Pass a custom FileIO property

spark.sql.catalog.gravitino_irc.{configuration_key}    {property_value}

Start Spark

Once spark-defaults.conf is in place, start your Spark session normally. The Gravitino IRC catalog is available immediately without any additional flags.

Spark shell (Scala)

Start the Spark shell

$SPARK_HOME/bin/spark-shell

Spark SQL

Start Spark SQL

$SPARK_HOME/bin/spark-sql

PySpark

Start PySpark

$SPARK_HOME/bin/pyspark

Examples

List namespaces

List namespaces in the catalog

SHOW NAMESPACES IN gravitino_irc;

List tables

List tables in a namespace

SHOW TABLES IN gravitino_irc.{namespace};

Query a table

Query a table

SELECT * FROM gravitino_irc.{namespace}.{table} LIMIT 10;

Create a table

Create a table

CREATE TABLE gravitino_irc.{namespace}.new_table (
  id INT,
  name STRING,
  created_at TIMESTAMP
) USING iceberg;

Insert data

Insert a row

INSERT INTO gravitino_irc.{namespace}.new_table VALUES (1, 'example', current_timestamp());

Gravitino connector vs. Iceberg REST

Feature	Gravitino Engine Connector	Iceberg REST
Engine plugin required	Yes	No
Gravitino access control	Yes	Yes
Supported engines	Trino, Spark, Flink, Daft	Any Iceberg-compatible engine
Credential vending	Varies	Yes (S3, GCS, OSS, ADLS)

Related

• Iceberg REST catalog service
• Connect Trino via Iceberg REST
• Connect Flink via Iceberg REST
• Spark Gravitino connector