Connect Ray to Iceberg REST

Apache Gravitino exposes an Iceberg REST catalog (IRC) endpoint that any Iceberg-compatible client can connect to directly. Ray Data reaches it through PyIceberg. No Gravitino engine connector is needed; the engine connectors are only for the Gravitino REST interface.

Ray Data reads from and writes to existing Iceberg tables only. It does not create, drop, or alter tables, schemas, or catalogs; use Spark or PyIceberg for that.

Quick start

The example signs in with OAuth client credentials, connects to the Gravitino catalog sales_lake, and reads the existing table q4.leads through vended credentials.

Read an Iceberg table with Ray Data
import ray

catalog_kwargs = {
    "name": "sales",
    "type": "rest",
    "uri": "https://{iceberg_host}/iceberg",
    "warehouse": "sales_lake",
    "credential": "{client_id}:{client_secret}",
    "oauth2-server-uri": "{token_endpoint_uri}",
    "scope": "{scope}",
    "header.X-Iceberg-Access-Delegation": "vended-credentials",
}

ds = ray.data.read_iceberg(
    table_identifier="q4.leads",
    catalog_kwargs=catalog_kwargs
)
ds.show(limit=10)

The catalog must vend credentials for the read to reach storage; see Credential vending.

Prerequisites

Apache Gravitino running with the Iceberg REST service enabled. See Iceberg REST catalog service for setup instructions.
The Iceberg REST service published and reachable from your Python environment. See Client access.
Ray Data and PyIceberg installed: pip install "ray[data]" "pyiceberg[s3fs,pyarrow]"

Configuration

Ray Data connects to the IRC through catalog_kwargs, passed to each read or write function. No separate catalog registration is required. The settings are PyIceberg catalog properties, so PyIceberg describes each of them.

warehouse is the name of the Gravitino catalog to connect to. Ray, through PyIceberg, sends it to the service, receives that catalog's prefix, and sends every later request to that catalog. Without it, the service serves its default catalog, if one is set; see Select a catalog.

Every request to the IRC needs an identity; a request with none is rejected with 401. The Quick start signs in with OAuth client credentials. A local user signs in with Basic credentials instead:

Catalog settings with basic auth and vended credentials
catalog_kwargs = {
    "name": "default",
    "type": "rest",
    "uri": "https://{iceberg_host}/iceberg",
    "warehouse": "{catalog}",
    "header.X-Iceberg-Access-Delegation": "vended-credentials",
    "auth": {
        "type": "basic",
        "basic": {"username": "{user}", "password": "{password}"}
    }
}

See How to authenticate for the Gravitino side.

Examples

Write to an Iceberg table

Write a dataset to an Iceberg table
import ray
import pandas as pd

docs = [{"id": i, "data": f"Doc {i}"} for i in range(4)]
ds = ray.data.from_pandas(pd.DataFrame(docs))
ds.write_iceberg(
    table_identifier="{namespace}.sample",
    catalog_kwargs=catalog_kwargs
)

Read from an Iceberg table

Read an Iceberg table
import ray

ds = ray.data.read_iceberg(
    table_identifier="{namespace}.sample",
    catalog_kwargs=catalog_kwargs
)
ds.show(limit=1)

Gravitino connector vs. Iceberg REST

Feature	Gravitino engine connector	Iceberg REST
Engine plugin required	Yes	No
Gravitino access control	Yes	Yes
Supported engines	Trino, Spark, Flink, Daft	Any Iceberg-compatible engine
Credential vending	Varies	Yes (S3, GCS, OSS, ADLS)

Quick start​

Prerequisites​

Configuration​

Examples​

Write to an Iceberg table​

Read from an Iceberg table​

Gravitino connector vs. Iceberg REST​

Related​