Connect Spark to Iceberg REST

Apache Gravitino exposes an Iceberg REST catalog (IRC) endpoint that any Iceberg-compatible engine can connect to directly, without installing a Gravitino-specific connector plugin. Spark connects to it with the standard Iceberg Spark runtime.

Gravitino enforces its full access-control model on every IRC request, and authorizes each request as the identity in the Spark catalog configuration. Whether that identity is one person or an account shared by many depends on whose credentials the configuration holds; see Authentication.

Quick start

The example signs you in with your own token from the identity provider, connects to the Gravitino catalog sales_lake as the Spark catalog sales, and reads storage through vended credentials. Gravitino authorizes every request as you. Identity provider tokens has scripts that get a token from Microsoft Entra ID, Okta, Keycloak, and Auth0.

Start Spark SQL against the Iceberg REST service

$SPARK_HOME/bin/spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.sales=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.sales.type=rest \
  --conf spark.sql.catalog.sales.uri=https://{iceberg_host}/iceberg \
  --conf spark.sql.catalog.sales.warehouse=sales_lake \
  --conf spark.sql.catalog.sales.token=$TOKEN \
  --conf spark.sql.catalog.sales.header.X-Iceberg-Access-Delegation=vended-credentials

List schemas and query a table

SHOW NAMESPACES IN sales;
SELECT * FROM sales.q4.leads LIMIT 10;

The catalog must vend credentials for the read to reach storage; see Credential vending. For jobs that run as a service account rather than as a person, see Authentication.

Prerequisites

• Gravitino running with the Iceberg REST service enabled. See Iceberg REST catalog service.
• The Iceberg REST service published and reachable from your Spark environment. See Client access.
• The Iceberg Spark runtime and the Iceberg AWS bundle on the Spark classpath, either with --packages as in the Quick start or as these JAR files:
  • iceberg-spark-runtime-3.5_2.12-1.11.0.jar
  • iceberg-aws-bundle-1.11.0.jar

The examples use Spark 3.5 with Iceberg 1.11.0, the Iceberg version the server runs. An older client still creates, reads and writes tables through the IRC, but Basic authentication needs Iceberg 1.8 or later, and server-side scan planning, where the IRC plans a scan and returns its file tasks, needs Iceberg 1.11 or later. Use an Iceberg runtime built for your Spark and Scala versions.

Connection properties

spark-defaults.conf is Spark's persistent configuration file. Properties set there apply to every Spark session on that installation, with no command-line flags needed.

spark-defaults.conf location

$SPARK_HOME/conf/spark-defaults.conf

If the file doesn't exist yet, copy the template:

Create spark-defaults.conf from the template

cp $SPARK_HOME/conf/spark-defaults.conf.template $SPARK_HOME/conf/spark-defaults.conf

The file below connects to the IRC and reads storage through vended credentials. It holds no credentials, because every session on the installation reads it; each person supplies their own when starting a session, as described in Authentication.

spark-defaults.conf

# Iceberg extensions
spark.sql.extensions                                        org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

# Gravitino IRC catalog
spark.sql.catalog.gravitino_irc                             org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.gravitino_irc.type                        rest
spark.sql.catalog.gravitino_irc.uri                         https://{iceberg_host}/iceberg
spark.sql.catalog.gravitino_irc.warehouse                   {catalog}

# Storage access
spark.sql.catalog.gravitino_irc.header.X-Iceberg-Access-Delegation    vended-credentials

# Set as default catalog (optional)
spark.sql.defaultCatalog                                    gravitino_irc

gravitino_irc is the catalog's name within Spark; any name works. warehouse is the name of the Gravitino catalog to connect to. The client sends it to the service, receives that catalog's prefix, and sends every later request to that catalog. Without it, the service serves its default catalog, if one is set; see Select a catalog.

Authentication

Every request to the IRC needs an identity; a request with none is rejected with 401. See How to authenticate for the Gravitino side.

Use OAuth. Each Spark session signs in with exactly one identity and sends it on every request: either a shared identity, from a client ID and secret or a pre-issued token, or a person's own token. Unlike Trino, Spark has no identity of its own besides the one configured for the session, so a session with a person's token needs no client ID or secret. Basic authentication signs in as one Gravitino local account.

Where the credentials sit decides who shares them. Credentials placed in spark-defaults.conf identify everyone who runs Spark from that installation as the same account. A shared Spark service, such as the Spark Thrift Server, loads the catalog with the credentials in its own configuration and uses them for every user it serves. To have Gravitino authorize each person as themselves, keep credentials out of shared configuration and pass them when each person starts a session.

Shared identity for Spark

Use a shared identity for jobs and pipelines that run as a service account. Every session that signs in with it is authorized as the same account, so grant that account in Gravitino only what those jobs need.

Method	The shared identity	Renewed
Client credentials	A client registered at the identity provider, for every session that uses its client ID and secret	Yes, by Spark
Pre-issued token	A token issued to a service account, for every session that uses it	No

Client ID and client secret

Spark holds the client ID and secret of a client registered at the identity provider, requests a token from the token endpoint, and requests a new one when it expires. Pass them when starting the job:

Start a job with client credentials

$SPARK_HOME/bin/spark-submit \
  --conf spark.sql.catalog.gravitino_irc.credential={client_id}:{client_secret} \
  --conf spark.sql.catalog.gravitino_irc.oauth2-server-uri={token_endpoint_uri} \
  --conf spark.sql.catalog.gravitino_irc.scope={scope} \
  {job_file}

oauth2-server-uri is the token endpoint, not the issuer.

Pre-issued token

A token issued to a service account, passed the same way as a personal token. Nothing renews it, so the job fails once it expires. Use it only when Spark cannot reach the identity provider.

Token exchange settings

Add this to the client credentials configuration:

Turn off token exchange for renewals

--conf spark.sql.catalog.gravitino_irc.token-exchange-enabled=false

The setting controls how Spark renews the client's token. With the default, true, Spark renews it by exchanging the old token at the identity provider; an identity provider that does not support token exchange, such as a default Keycloak realm, refuses with unsupported_grant_type and Spark retries repeatedly. With false, Spark signs in again with the client ID and secret.

Per-user identity

A person gets a token from the identity provider, outside Spark, and passes it when starting the session. Spark sends it as the bearer token on every request, and Gravitino authorizes each request as that person. No client ID or secret is needed.

Start Spark SQL with your own token

$SPARK_HOME/bin/spark-sql \
  --conf spark.sql.catalog.gravitino_irc.token=$TOKEN

The token must carry the audience configured for Gravitino and a principal claim naming a member of the metalake. Nothing renews it, so requests fail once it expires, and a long-running session needs a new token and a restart.

Verified against Gravitino 1.3.1, Spark 3.5.5 and Iceberg 1.10.0: with a personal token, a user with a grant on one table read it and was refused a second table by name, and the same query under client credentials with no table grant was refused as the client.

Basic authentication

A local user signs in with Basic credentials:

Start Spark SQL with a local user

$SPARK_HOME/bin/spark-sql \
  --conf spark.sql.catalog.gravitino_irc.rest.auth.type=basic \
  --conf spark.sql.catalog.gravitino_irc.rest.auth.basic.username={username} \
  --conf spark.sql.catalog.gravitino_irc.rest.auth.basic.password={password}

Storage access

Vended credentials

With the X-Iceberg-Access-Delegation header, as in the file above, the IRC returns short-lived storage credentials scoped to each table it loads, together with the storage endpoint and region, so Spark holds no storage keys. The catalog must have a credential provider; see Credential vending.

Static credentials

To use storage keys held by Spark instead, remove the header.X-Iceberg-Access-Delegation line and set the keys:

spark-defaults.conf

spark.sql.catalog.gravitino_irc.s3.access-key-id            {access_key}
spark.sql.catalog.gravitino_irc.s3.secret-access-key        {secret_key}

The keys can also come from the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables or an instance profile, in which case the two lines can be omitted.

For S3-compatible storage such as MinIO, also set the endpoint and path-style access when the catalog does not already send them:

S3 properties for MinIO

spark.sql.catalog.gravitino_irc.s3.endpoint                 https://{minio_host}:9000
spark.sql.catalog.gravitino_irc.s3.path-style-access        true

For storage that Gravitino does not manage, properties are not passed from the server to the client. Pass them to the client's FileIO explicitly:

Pass a custom FileIO property

spark.sql.catalog.gravitino_irc.{configuration_key}         {property_value}

Start Spark

Once spark-defaults.conf is in place, start a session with your credentials, as shown in Authentication. Add those flags to whichever command below you use.

Spark shell (Scala)

Start the Spark shell

$SPARK_HOME/bin/spark-shell

Spark SQL

Start Spark SQL

$SPARK_HOME/bin/spark-sql

PySpark

Start PySpark

$SPARK_HOME/bin/pyspark

Examples

List namespaces

List namespaces in the catalog

SHOW NAMESPACES IN gravitino_irc;

List tables

List tables in a namespace

SHOW TABLES IN gravitino_irc.{namespace};

Query a table

Query a table

SELECT * FROM gravitino_irc.{namespace}.{table} LIMIT 10;

Create a table

Create a table

CREATE TABLE gravitino_irc.{namespace}.new_table (
  id INT,
  name STRING,
  created_at TIMESTAMP
) USING iceberg;

Insert data

Insert a row

INSERT INTO gravitino_irc.{namespace}.new_table VALUES (1, 'example', current_timestamp());

Gravitino connector vs. the IRC

Feature	Gravitino engine connector	Iceberg REST
Engine plugin required	Yes	No
Gravitino access control	Yes	Yes
Supported engines	Trino, Spark, Flink, Daft	Any Iceberg-compatible engine
Credential vending	Varies	Yes (S3, GCS, OSS, ADLS)

Related

• Iceberg REST catalog service
• Connect Trino via Iceberg REST
• Connect Flink via Iceberg REST
• Spark Gravitino connector