Skip to main content

Iceberg REST catalog service

Gravitino serves an Iceberg REST catalog (IRC) that implements the Apache Iceberg REST catalog specification; the Iceberg REST API reference lists the operations it serves. Any engine with an Iceberg REST client, including Spark, Trino, Flink, and PyIceberg, reads and writes Iceberg tables through it with no Gravitino connector installed.

The IRC serves the Iceberg catalogs of a metalake. A catalog created in Gravitino with the lakehouse-iceberg provider is available through the IRC as soon as it exists, under its own name, and every request is authenticated and authorized by Gravitino against the same privileges that govern the Gravitino API. A catalog that federates a remote Iceberg REST catalog is the exception: by default the remote catalog makes the authorization decision; see Iceberg federation.

ItemIceberg REST catalogGravitino API
ProtocolThe Apache Iceberg REST specificationThe Gravitino REST API
ServesIceberg tables, namespaces, and viewsEvery catalog type Gravitino connects
Used byEngines with an Iceberg REST clientThe UI, Gravitino connectors, and client libraries

Quick start​

  1. Create an Iceberg catalog in the metalake. The catalog keeps its table pointers in a JDBC database or a Hive metastore of its own, not in the metadata database. A JDBC database can sit on the same server as the metadata database; see JDBC backend. See Iceberg catalog for the other backend and storage properties.

  2. Point an engine at the IRC. The catalog name is the warehouse, and {iceberg_host} is the address the IRC is published at; see Client access.

    Connect Spark with a token and vended credentials
    ./bin/spark-sql \
      --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
      --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
      --conf spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog \
      --conf spark.sql.catalog.lake.type=rest \
      --conf spark.sql.catalog.lake.uri=https://{iceberg_host}/iceberg \
      --conf spark.sql.catalog.lake.warehouse={catalog} \
      --conf spark.sql.catalog.lake.token=$TOKEN \
      --conf spark.sql.catalog.lake.header.X-Iceberg-Access-Delegation=vended-credentials

  3. Grant access. Reading a table takes USE_CATALOG, USE_SCHEMA, and SELECT_TABLE; writing takes MODIFY_TABLE in place of SELECT_TABLE. See Roles.

Deployment​

The IRC runs inside the Gravitino server as an auxiliary service, on its own port, 9001 by default, under the path /iceberg/. It is enabled when the server is deployed. Engines outside the cluster reach it at the address it is published at, https://{iceberg_host}/iceberg, as described in Client access; the UI shows that address alongside the other client endpoints, or shows the service as not enabled.

The service reads the metalake named by gravitino.iceberg-rest.gravitino-metalake and serves that metalake's Iceberg catalogs. One deployment serves one metalake through the IRC.

Every server replica runs the IRC, so running more than one server spreads engine traffic across them. Loads, scan planning and table commits are safe on any replica.

Select a catalog​

One IRC serves every Iceberg catalog in its metalake, and a metalake can hold as many as you need, each with its own backend, storage and access grants. An engine attaches one or more of them.

The Iceberg REST specification carries the catalog in the request path as a prefix, as in /iceberg/v1/{catalog}/namespaces. Clients do not build that path themselves: a client names the catalog as its warehouse, calls /iceberg/v1/config with it, and receives the prefix to use for every later request.

A client that names no warehouse gets the catalog set in gravitino.iceberg-rest.default-catalog-name. Naming the warehouse explicitly is the safer habit.

Catalogs on more than one cloud​

Each catalog has its own warehouse, storage properties and credential provider, so one IRC can serve a catalog on Amazon S3, another on Azure Data Lake Storage and a third on Google Cloud Storage side by side. An engine attaches each one as a separate catalog, naming it as the warehouse, and receives credentials for that catalog's storage when it loads a table. The engine needs the Iceberg storage library for each cloud it reads, such as iceberg-aws-bundle, iceberg-azure-bundle and iceberg-gcp-bundle for Spark. See Object storage for each cloud's catalog properties.

Authentication and access control​

Every request needs a credential. A request with none returns 401, and one with an expired token returns 419, a status some Iceberg clients do not recognize as an expiry. Bearer tokens from the identity provider and local accounts with Basic credentials are both accepted.

Authorization is the same as for the Gravitino API. The IRC passes the caller, the object, and the operation to Gravitino, and the request proceeds only if the caller holds the privileges for it. A denied request returns 403.

OperationPrivileges
List namespacesUSE_CATALOG
List tables in a namespaceUSE_CATALOG and USE_SCHEMA
Load a tableUSE_CATALOG, USE_SCHEMA, and SELECT_TABLE or MODIFY_TABLE
Create a tableUSE_CATALOG, USE_SCHEMA, and CREATE_TABLE
Commit to a tableUSE_CATALOG, USE_SCHEMA, and MODIFY_TABLE

Owning an object satisfies these checks as described in Required privileges.

Token refresh​

Some engines try to exchange or refresh a token in ways an identity provider does not support, which surfaces as authentication failures partway through a long job. Turning off token exchange avoids it:

Turn off token exchange
# Spark
spark.sql.catalog.{catalog}.token-exchange-enabled=false

# Trino
iceberg.rest-catalog.oauth2.token-exchange-enabled=false

Credential vending​

An engine that sends X-Iceberg-Access-Delegation: vended-credentials receives short-lived storage credentials with each table it loads, so it never holds long-lived storage keys of its own.

On S3 with the s3-token provider, the credential is scoped by a session policy to the table's own location: it can read and write that table and nothing else in the bucket, even when the role behind it can reach the whole account. The session is named gravitino_{principal}, so storage access logs attribute each read and write to the Gravitino user who made it. A caller with MODIFY_TABLE receives a credential that can write; a caller with only SELECT_TABLE receives one that can only read.

See Credential vending for providers and their properties.

Scan planning​

By default an engine plans each scan itself: it reads the table's manifests from storage and works out which data and delete files the query needs. With server-side scan planning, the engine sends the query's filter and column selection to the IRC instead, and the server reads the manifests and returns the list of files to read. The engine then reads those files directly, as before.

A client turns it on with the catalog property scan-planning-mode set to server; the default is client. The property needs the Iceberg Java library 1.11 or later, or PyIceberg 0.12 or later. For Spark, add one line to the command in the Quick start:

Turn on server-side scan planning in Spark
  --conf spark.sql.catalog.lake.scan-planning-mode=server

Planning on the server also means a question about which files match needs no engine at all. A short PyIceberg script can check whether a day's data has arrived, and how many rows it holds, without starting Spark or Trino. The row counts come from the file metadata in the plan, so the script reads no data files.

check-arrival.py
import os
from pyiceberg.catalog import load_catalog

catalog = load_catalog("lake", **{
    "type": "rest",
    "uri": "https://{iceberg_host}/iceberg",
    "warehouse": "{catalog}",
    "auth": {"type": "basic", "basic": {"username": "{user}", "password": os.environ["PASSWORD"]}},
    "scan-planning-mode": "server",
})

table = catalog.load_table("{schema}.{table}")
files = list(table.scan(row_filter="event_date >= '2026-10-19'").plan_files())
print(f"{len(files)} files, {sum(f.file.record_count for f in files)} rows")

The server plans a scan of the current snapshot, of a snapshot the engine names, or of the rows appended between two snapshots. It returns the whole plan in one response, so for a table with many files the response lists every file the query touches. Planning a scan needs the same access as loading the table. Plans for repeated identical queries can be cached on the server with the scan-plan-cache-* catalog properties in Configuration.

Metadata cache​

The IRC keeps the parsed metadata of recently loaded tables in memory, so a repeated load of an unchanged table is served without reading its metadata file from object storage. Before serving a cached copy, the server checks the table's current metadata location in the catalog's backend. If the table has changed since, it reads the new metadata file and replaces the copy, so the cache never serves an out-of-date table. The cache is on by default, and its size and expiry are set per catalog with the table-metadata-cache-* properties in Configuration.

Each table load also returns an ETag. A client that sends it back in If-None-Match on its next load of the table receives 304 Not Modified when the table has not changed, and skips the transfer.

Capabilities and limits​

The IRC supports namespace, table, and view operations, hierarchical namespaces, server-side scan planning, and optimistic concurrency on commits, so a commit made against a stale table state is rejected with 409 rather than overwriting a concurrent change.

Not supportedEffect
Multi-table transactionsEach commit applies to one table
View registrationViews are created through the IRC; existing views cannot be registered

With the JDBC catalog backend, views need its schema at version V1, which is the default.

Configuration​

The service itself has a few server settings:

Configuration itemDescriptionDefault value
gravitino.iceberg-rest.gravitino-metalakeThe metalake whose Iceberg catalogs the IRC serves(none)
gravitino.iceberg-rest.default-catalog-nameThe catalog served to a client that names no warehouse(none)
gravitino.iceberg-rest.advertised-uriThe address the server gives the Gravitino connectors for this service, such as https://{iceberg_host}/iceberg. Set it to the published host when the service is publishedhttp://{host}:9001/iceberg
gravitino.iceberg-rest.catalog-cache-eviction-interval-msHow long a catalog's configuration is cached before being reloaded3600000

Set gravitino-metalake and default-catalog-name through their dedicated chart values, gravitino.icebergRest.dynamicConfigProvider.metalake and gravitino.icebergRest.dynamicConfigProvider.defaultCatalogName. Set the others under gravitino.additionalConfigItems in my-values.yaml as quoted strings; see Set other server properties.

Each catalog is configured from its own properties, so the caches and the JDBC schema version are set on the Iceberg catalog, not on the server. The same names under gravitino.iceberg-rest. in the server configuration have no effect on the catalogs the IRC serves.

Catalog propertyDescriptionDefault value
table-metadata-cache-implTable metadata cache implementation; an empty value turns the cache offorg.apache.gravitino.iceberg.common.cache.LocalTableMetadataCache
table-metadata-cache-capacityTables whose metadata is cached1000
table-metadata-cache-expire-minutesHow long cached table metadata is kept60
scan-plan-cache-implScan plan cache implementation, org.apache.gravitino.iceberg.service.cache.LocalScanPlanCache. Unset, scan plans are not cached(none)
scan-plan-cache-capacityScan plans cached for repeated identical queries, when the cache is on200
scan-plan-cache-expire-minutesHow long a cached scan plan is kept60
jdbc-schema-versionSchema version of the JDBC backend; V1 enables viewsV1

A change to a catalog's properties in Gravitino reaches the IRC when its cached configuration is next reloaded, up to the eviction interval later. Cached scan plans are keyed by snapshot, so a plan never outlives the table state it was computed for.

Health​

The IRC answers its own health checks, which need no credential.

EndpointReturns
GET /iceberg/health/live200 whenever the service can answer
GET /iceberg/health/ready200 once its catalogs are initialized, 503 before that
GET /iceberg/health200 when both pass, 503 otherwise

See Health and readiness for the main server's checks.

Last updated on