Connect PyIceberg to Iceberg REST

Introduction

Apache Gravitino exposes an Iceberg REST catalog endpoint that any Iceberg-compatible client can connect to directly. This page describes how to use PyIceberg with Gravitino's Iceberg REST (IRC) endpoint.

Prerequisites

Apache Gravitino running with the Iceberg REST service enabled. See Iceberg REST catalog service for setup instructions.
The Gravitino IRC endpoint is accessible from your Python environment. The default port is 9001.
PyIceberg installed: pip install pyiceberg

Configuration

Connect to the Iceberg REST endpoint
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "gravitino_irc",
    **{
        "type": "rest",
        "uri":  "http://{gravitino_host}:9001/iceberg",
    }
)

Credential vending

Request vended credentials
catalog = load_catalog(
    "gravitino_irc",
    **{
        "type":                            "rest",
        "uri":                             "http://{gravitino_host}:9001/iceberg",
        "header.X-Iceberg-Access-Delegation": "vended-credentials",
    }
)

OAuth authentication

Connect with an OAuth token
catalog = load_catalog(
    "gravitino_irc",
    **{
        "type":  "rest",
        "uri":   "http://{gravitino_host}:9001/iceberg",
        "token": "{token}",
    }
)

See How to authenticate for Gravitino authentication configuration options.

Examples

List namespaces

List namespaces
catalog.list_namespaces()

Load a table

Load a table
table = catalog.load_table("db.table")
print(table.schema())

Scan a table

Scan a table
df = table.scan().to_arrow()
print(df)

Create a namespace and table

Create a namespace and table
catalog.create_namespace("db")

from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, LongType, StringType

schema = Schema(
    NestedField(1, "id",   LongType(),   required=True),
    NestedField(2, "name", StringType(), required=False),
)
catalog.create_table("db.new_table", schema=schema)

Gravitino connector vs. Iceberg REST

Feature	Gravitino Engine Connector	Iceberg REST
Engine plugin required	Yes	No
Gravitino access control	Yes	Yes
Supported engines	Trino, Spark, Flink, Daft	Any Iceberg-compatible engine
Credential vending	Varies	Yes (S3, GCS, OSS, ADLS)

Introduction​

Prerequisites​

Configuration​

Credential vending​

OAuth authentication​

Examples​

List namespaces​

Load a table​

Scan a table​

Create a namespace and table​

Gravitino connector vs. Iceberg REST​

Related​