Connect PyIceberg to Iceberg REST

Apache Gravitino exposes an Iceberg REST catalog (IRC) endpoint that any Iceberg-compatible client can connect to directly. PyIceberg, the Python Iceberg library, connects to it with its REST catalog. No Gravitino engine connector is needed; the engine connectors are only for the Gravitino REST interface.

Quick start

The example signs in with OAuth client credentials, connects to the Gravitino catalog sales_lake, and reads storage through vended credentials.

Connect and read a table
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "sales",
    **{
        "type":                               "rest",
        "uri":                                "https://{iceberg_host}/iceberg",
        "warehouse":                          "sales_lake",
        "credential":                         "{client_id}:{client_secret}",
        "oauth2-server-uri":                  "{token_endpoint_uri}",
        "scope":                              "{scope}",
        "header.X-Iceberg-Access-Delegation": "vended-credentials",
    }
)

print(catalog.list_namespaces())
print(catalog.load_table("q4.leads").scan(limit=10).to_arrow())

The catalog must vend credentials for the read to reach storage; see Credential vending.

Prerequisites

Apache Gravitino running with the Iceberg REST service enabled. See Iceberg REST catalog service for setup instructions.
The Iceberg REST service published and reachable from your Python environment. See Client access.
PyIceberg with S3 support installed: pip install "pyiceberg[s3fs,pyarrow]"

Configuration

Every request to the IRC needs an identity; a request with none is rejected with 401. The configuration below signs in with OAuth client credentials and asks for vended credentials. PyIceberg requests a token from the identity provider, sending the client ID and secret in the body of the token request, and sends the token on every request.

Connect with OAuth client credentials
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "gravitino_irc",
    **{
        "type":                               "rest",
        "uri":                                "https://{iceberg_host}/iceberg",
        "warehouse":                          "{catalog}",
        "credential":                         "{client_id}:{client_secret}",
        "oauth2-server-uri":                  "{token_endpoint_uri}",
        "scope":                              "{scope}",
        "header.X-Iceberg-Access-Delegation": "vended-credentials",
    }
)

warehouse is the name of the Gravitino catalog to connect to. PyIceberg sends it to the service, receives that catalog's prefix, and sends every later request to that catalog. Without it, the service serves its default catalog, if one is set; see Select a catalog.

oauth2-server-uri is the identity provider's token endpoint, not its issuer. See How to authenticate for the Gravitino side.

Token authentication

With a token you already hold, replace credential, oauth2-server-uri, and scope with token. Identity provider tokens has scripts that get one from Microsoft Entra ID, Okta, Keycloak, and Auth0. Nothing renews it, so requests fail once it expires.

Connect with an OAuth token
catalog = load_catalog(
    "gravitino_irc",
    **{
        "type":                               "rest",
        "uri":                                "https://{iceberg_host}/iceberg",
        "warehouse":                          "{catalog}",
        "token":                              "{token}",
        "header.X-Iceberg-Access-Delegation": "vended-credentials",
    }
)

A local user signs in with a user name and password through PyIceberg's basic sign-in, in place of credential, oauth2-server-uri, and scope. Setting an Authorization header by hand does not work, because PyIceberg replaces it with its own.

Connect as a local user
import os

catalog = load_catalog(
    "gravitino_irc",
    **{
        "type":                               "rest",
        "uri":                                "https://{iceberg_host}/iceberg",
        "warehouse":                          "{catalog}",
        "auth":                               {"type": "basic", "basic": {"username": "{user}", "password": os.environ["PASSWORD"]}},
        "header.X-Iceberg-Access-Delegation": "vended-credentials",
    }
)

Storage access

With header.X-Iceberg-Access-Delegation, the IRC returns short-lived storage credentials scoped to each table, together with the storage endpoint and region. The catalog must have a credential provider; see Credential vending. Without the header, PyIceberg needs its own storage keys, such as s3.access-key-id and s3.secret-access-key.

Examples

List namespaces

List namespaces
catalog.list_namespaces()

Load a table

Load a table
table = catalog.load_table("{namespace}.{table}")
print(table.schema())

Scan a table

Scan a table
df = table.scan().to_arrow()
print(df)

Create a namespace and table

Create a namespace and table
catalog.create_namespace("{namespace}")

from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, LongType, StringType

schema = Schema(
    NestedField(1, "id",   LongType(),   required=True),
    NestedField(2, "name", StringType(), required=False),
)
catalog.create_table("{namespace}.new_table", schema=schema)

Gravitino connector vs. Iceberg REST

Feature	Gravitino engine connector	Iceberg REST
Engine plugin required	Yes	No
Gravitino access control	Yes	Yes
Supported engines	Trino, Spark, Flink, Daft	Any Iceberg-compatible engine
Credential vending	Varies	Yes (S3, GCS, OSS, ADLS)

Quick start​

Prerequisites​

Configuration​

Token authentication​

Local user sign-in​

Storage access​

Examples​

List namespaces​

Load a table​

Scan a table​

Create a namespace and table​

Gravitino connector vs. Iceberg REST​

Related​