Skip to main content

Iceberg catalog

An Iceberg catalog connects Gravitino to the store that holds an Apache Iceberg catalog's table pointers: a Hive metastore, a JDBC database, or another Iceberg REST catalog. Gravitino manages the catalog's schemas, tables, and views through that backend, and engines reach the same tables through the Iceberg REST catalog service, as described in Iceberg REST clients, or through the Gravitino connectors. The pointer is the top of each table's tree of files; see How Iceberg works.

Quick start​

The example connects the catalog iceberg_catalog, which keeps its table pointers in a PostgreSQL database and its warehouse in S3, then creates the schema sales and the table orders. Creating the table writes its first metadata file to the bucket, so it is the first step that tests the storage settings. To vend temporary credentials to engines rather than share a key, see Amazon S3.

  1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter iceberg_catalog as the Catalog name, keep Relational as the Catalog type, and choose Apache Iceberg in Select provider.

  2. Fill in the connection.

    FieldValue
    Catalog backendJDBC
    URIjdbc:postgresql://{postgres_host}:5432/{database}
    User, PasswordThe backend database's user and password
    StorageS3
    Warehouses3://{bucket}/warehouse
    AWS regionThe bucket's region, such as us-west-2
    Access key ID, AWS secret keyAn access key that can read and write the bucket

  3. Test and create it. Click Test connection, which checks the backend database but not the bucket, then Create catalog.

  4. Create the schema. In Explore, open iceberg_catalog, click + Schema, and create sales. Tables are created through engines and the API rather than the UI; see Iceberg tables to create one from Spark.

Requirements and limitations​

The server is built with Apache Iceberg 1.11.0. New tables use Iceberg table format version 2 unless format-version is set.

Flink and Spark clients may use a different Iceberg version than the server; the client JAR requirements are on these pages:

Do not mix Iceberg JARs of different versions on one client classpath; the result fails at runtime.

Catalog​

Capabilities​

  • Works as a catalog proxy over a Hive, JDBC, or REST catalog backend.
  • Creates, alters, and drops Iceberg schemas and tables.
  • Does not manage snapshots; use an engine for snapshot operations such as rollback and expiry.
  • Holds tables in several storage systems at once, including S3, GCS, ADLS, OSS, and HDFS.
  • Reaches an HDFS warehouse with Kerberos or Hadoop simple authentication, for the Hive and JDBC backends.
  • Caches table metadata; see Table metadata cache.

Properties​

Property nameDescriptionDefault valueRequired
catalog-backendCatalog backend of Gravitino Iceberg catalog. Supports hive, jdbc, or rest.(none)Yes
uriThe address of the catalog backend, such as thrift://{hms_host}:9083, jdbc:postgresql://{postgres_host}:5432/{database}, jdbc:mysql://{mysql_host}:3306/{database}, or https://{rest_catalog_host}/iceberg.(none)Yes
warehouseWarehouse location of catalog. Use a physical S3 or HDFS location for hive or jdbc catalog backend, use catalog name for REST catalog backend.(none)Yes for hive and jdbc catalog backend
catalog-backend-nameThe catalog name passed to underlying Iceberg catalog backend. Catalog name in JDBC backend is used to isolate namespace and tables.The property value of catalog-backend, like jdbc for JDBC catalog backend.No

A property with the gravitino.bypass. prefix that Gravitino does not define is passed, without the prefix, to the Iceberg catalog properties and the HDFS configuration. For example, gravitino.bypass.list-all-tables passes list-all-tables to the Iceberg catalog.

The trino.bypass. prefix passes a setting to the Trino Iceberg connector when Trino loads the catalog through the Gravitino connector. For example, trino.bypass.iceberg.table-statistics-enabled sets iceberg.table-statistics-enabled.

The spark.bypass. prefix does the same for the Spark Iceberg connector. For example, spark.bypass.io-impl sets io-impl in Spark.

JDBC backend​

The JDBC backend needs jdbc-user, jdbc-password, and jdbc-driver for the connection to the database that holds the table pointers. When the warehouse is on Kerberos-secured HDFS, set authentication.type to kerberos and configure the Kerberos properties in Catalog backend security.

The database that holds the table pointers is not the metadata database. The metadata database holds Gravitino's own objects, such as catalogs, schemas, tags, roles, and grants, but not the location of each Iceberg table's current metadata file, which is what Iceberg reads and swaps on every commit. The catalog's own database holds that. Both can live on one PostgreSQL or MySQL server: give the catalog a database and user of its own there, so that its backups and permissions are managed apart from the metadata database. Several Iceberg catalogs can share that database by setting a different catalog-backend-name on each.

Create the catalog's PostgreSQL database and user
CREATE USER {jdbc_user} WITH PASSWORD '{jdbc_password}';
CREATE DATABASE {database} OWNER {jdbc_user};

On MySQL, use the statements in Use MySQL with the catalog's database and user. With jdbc-initialize left at true, Gravitino creates the tables Iceberg needs in the database when the catalog is created.

Property nameDescriptionDefault valueRequired
jdbc-userJDBC user name(none)Yes
jdbc-passwordJDBC password(none)Yes
jdbc-drivercom.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver for MySQL, org.postgresql.Driver for PostgreSQL(none)Yes
jdbc-initializeWhether to create the Iceberg tables in the database when the catalog is createdtrueNo

To connect an existing JDBC Iceberg catalog, set catalog-backend-name to its catalog name so that Gravitino sees its existing namespaces and tables.

The PostgreSQL driver is in the server image. MySQL Connector/J is not, so a MySQL backend needs the chart to supply it; see JDBC driver.

With several JDBC catalog backends in one MySQL database, jdbc-initialize may not create the Iceberg tables; create them in the database before creating the catalog.

REST catalog backend​

For the REST catalog backend, warehouse identifies the catalog in the Iceberg REST spec. In the Gravitino Iceberg REST server, warehouse maps to the catalog name. An empty value means the default catalog.

The following properties tune REST backend behavior:

Property nameDescriptionDefault valueRequired
data-accessData access mode for REST catalog backend. Supported values are vended-credentials and remote-signing.(none)No
rest-client-connection-timeout-msThe HTTP connection timeout in milliseconds for requests to the REST catalog backend.10000No
rest-client-socket-timeout-msThe HTTP socket timeout in milliseconds for requests to the REST catalog backend.60000No
  • vended-credentials: request credential vending from the Iceberg REST server.
  • remote-signing: Gravitino doesn't support this mode yet.

Without warehouse, requests go to the remote service's default catalog, at paths such as https://{rest_catalog_host}/iceberg/v1/namespaces/{namespace}/tables/{table}.

Create an Iceberg catalog with the REST backend
curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "iceberg_rest",
    "type": "RELATIONAL",
    "comment": "Iceberg REST catalog",
    "provider": "lakehouse-iceberg",
    "properties": {
      "catalog-backend": "rest",
      "uri": "https://{rest_catalog_host}/iceberg",
      "rest-client-connection-timeout-ms": "10000",
      "rest-client-socket-timeout-ms": "60000",
      "data-access": "vended-credentials"
    }
  }'

To reach a non-default catalog, set warehouse to its name; requests then go to paths such as https://{rest_catalog_host}/iceberg/v1/{catalog}/namespaces/{namespace}/tables/{table}. See Selecting a catalog.

Object storage​

If io-impl is not set, the catalog uses org.apache.iceberg.io.ResolvingFileIO, which picks the FileIO for each file from its URI scheme. The server image includes the storage libraries for every scheme below, so nothing has to be downloaded.

StorageSchemesWarehouseSetup
Amazon S3s3, s3a, s3ns3://{bucket}/{path}; s3a:// with the Hive backendAmazon S3
S3-compatible, such as R2s3s3://{bucket}/{path}, with s3-endpoint setS3-compatible storage
Azure Data Lake Storage Gen2abfs, abfss, wasb, wasbsabfss://{container}@{account}.dfs.core.windows.net/{path}Azure Data Lake Storage
Google Cloud Storagegs, gcsgs://{bucket}/{path}Google Cloud Storage
Alibaba Cloud OSSossoss://{bucket}/{path}OSS, below

Each storage page lists the catalog properties the server uses to reach the storage, including the ADLS service principal (azure-tenant-id, azure-client-id, azure-client-secret) and the GCS key file (gcs-service-account-file), and the credential providers that vend access to engines.

To pass an Iceberg FileIO property that Gravitino does not manage, such as s3.sse.type, gcs.project-id or adls.read.block-size-bytes, prefix it with gravitino.bypass. on the catalog: gravitino.bypass.s3.sse.type. A bypass property is not declared as hidden, so it can come back when the catalog is loaded; do not pass credentials this way. See Property behavior.

OSS​

Configuration itemDescriptionDefault valueRequired
io-implThe IO implementation for FileIO in Iceberg. Set it to org.apache.iceberg.aliyun.oss.OSSFileIO to explicitly use OSSFileIO.org.apache.iceberg.io.ResolvingFileIONo
oss-access-key-idThe static access key ID used to access OSS data.(none)No
oss-secret-access-keyThe static secret access key used to access OSS data.(none)No
oss-endpointThe endpoint of Aliyun OSS service.(none)No

Set warehouse to oss://{bucket_name}/{prefix_name}. Credential providers for OSS are described in Credential vending.

Other storage​

Storage that Gravitino does not support directly can be reached through a custom FileIO.

Configuration itemDescriptionDefault valueRequired
io-implThe IO implementation for FileIO in Iceberg. Use the fully qualified class name to override the default implementation.org.apache.iceberg.io.ResolvingFileIONo

To pass a property such as security-token to a custom FileIO, set gravitino.bypass.security-token; the FileIO receives security-token when it is initialized. A bypass property is not declared as hidden, so a credential passed this way can come back when the catalog is loaded; see Property behavior.

Set warehouse to {storage_scheme}://{bucket}/{prefix}. The custom FileIO and its dependencies must be on the server's classpath.

Catalog backend security​

The following properties secure access to the warehouse storage. For example, if you are using a Hive or JDBC catalog backend with a Kerberos-secured HDFS warehouse, set authentication.type to kerberos and provide authentication.kerberos.principal and authentication.kerberos.keytab-uri. For JDBC backend, JDBC username/password authentication for the metadata store is configured separately via jdbc-user and jdbc-password.

Property nameDescriptionDefault valueRequired
authentication.typeThe authentication type for HDFS warehouse access. Supports kerberos and simple for Hive and JDBC catalog backends.simpleNo
authentication.impersonation-enableWhether to enable impersonation for the Iceberg catalog.falseNo
hive.metastore.sasl.enabledWhether to enable SASL when connecting to a Kerberos Hive Metastore, passed to Hive unchanged.falseNo. Set to true in most cases when authentication.type is kerberos (some deployments use SSL instead, but that is rare).
authentication.kerberos.principalThe principal for Kerberos authentication.(none)Yes, if authentication.type is kerberos.
authentication.kerberos.keytab-uriThe URI of the keytab for Kerberos authentication.(none)Yes, if authentication.type is kerberos.
authentication.kerberos.check-interval-secThe check interval of Kerberos credential for the Iceberg catalog.60No
authentication.kerberos.keytab-fetch-timeout-secThe fetch timeout for retrieving Kerberos keytab from authentication.kerberos.keytab-uri.60No

Table metadata cache​

Gravitino caches table metadata to avoid rereading metadata files. Before serving a cached entry, it checks the table's current metadata location against the catalog backend, so a table changed by another writer is reread.

Configuration itemDescriptionDefault valueRequired
table-metadata-cache-implThe implementation of the table metadata cache. An empty string ("") turns the cache off. A backend that cannot report a table's metadata location, such as rest, runs without the cache and logs a warning until the property is set to "".org.apache.gravitino.iceberg.common.cache.LocalTableMetadataCacheNo
table-metadata-cache-capacityThe capacity of the table metadata cache.1000No
table-metadata-cache-expire-minutesThe expiration time (in minutes) of the table metadata cache.60No

The built-in LocalTableMetadataCache keeps the cache in server memory. A custom cache implements org.apache.gravitino.iceberg.common.cache.TableMetadataCache.

Operations​

Refer to Catalogs and schemas for more details.

Sensitive catalog properties such as s3-access-key-id, s3-secret-access-key, oss-access-key-id, and oss-secret-access-key are hidden from the load catalog response. Engines that need storage access get short-lived credentials through credential vending rather than the stored keys.

Schema​

Capabilities​

  • Does not support cascading schema drops; drop the tables first.
  • Supports hierarchical (multi-level) schemas, mapping each level to an Iceberg namespace level. See Hierarchical schema.

Properties​

Any property can be set except comment, which is a field of the schema itself.

Operations​

Refer to Catalogs and schemas for more details.

Hierarchical schema​

The Iceberg catalog supports a hierarchical (multi-level) schema, where a schema can be nested under another schema, mapping each level to an Iceberg multi-level namespace.

A hierarchical schema name is a path whose levels are joined by the configured separator gravitino.schema.separator (default :, see Server settings). For example, with the default separator the name a:b:c denotes a schema c nested under a:b, which in turn is nested under a. The separator is only used at the API boundary; Gravitino stores the name internally using a physical separator that never collides with user input.

To create a hierarchical schema, supply its full hierarchical name. Any missing ancestor schemas are created automatically, so creating a:b:c also creates a and a:b if they don't already exist. The following example creates the schema a:b:c:

Create a hierarchical schema
curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/iceberg_catalog/schemas" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "a:b:c",
    "comment": "a hierarchical schema",
    "properties": {}
  }'

To list the schemas directly under a parent schema, pass the parent schema name. Over REST this is the optional parentSchema query parameter; in the clients it is an argument to the list-schemas method. Given the schemas a, a:b and a:b:c, listing the children of a:b returns [a:b:c]. When the parent is omitted, only the top-level schemas, such as a, are returned.

List child schemas
curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/iceberg_catalog/schemas?parentSchema=a:b" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Authorization: Bearer $TOKEN"

Table​

Column types, format versions, partitioning, sort orders, table properties and the changes a table accepts are covered in Iceberg tables. How the tables work, from snapshots and commits to time travel and maintenance, is covered in How Iceberg works.

View​

Capabilities​

  • Supports list, create, load, alter, and drop for views managed by the underlying Iceberg backend.
  • Accepts any dialect name, such as trino, spark, flink, or hive.
  • Can preserve multiple SQL representations for the same logical view; the full set of representations round-trips through Gravitino.
  • defaultCatalog and defaultSchema are stored and returned as-is by the backend.
  • View support depends on whether the Iceberg catalog backend supports views; the REST and Hive backends do.

A rename cannot be combined with other changes in one alterView call; send it as its own request.

Operations​

Refer to Tables and views for more details.

HDFS configuration​

The server is built with Hadoop 3.3.6. HDFS client settings, such as the name service settings an HA cluster needs, are passed as catalog properties with the gravitino.bypass. prefix, which the catalog adds to its HDFS configuration. For example, gravitino.bypass.dfs.nameservices sets dfs.nameservices.

The server writes to HDFS as a single HDFS user and does not impersonate other users. See Apache Hadoop access to set that user.

Last updated on