Skip to main content

Hive metastore as an Iceberg catalog

Iceberg tables registered in a Hive metastore are served by an Iceberg catalog whose backend is that metastore. Connect the metastore as an Iceberg catalog, and Gravitino serves its Iceberg tables through the Iceberg REST catalog service and the Gravitino connectors, with credential vending, scan planning and the metadata cache. Nothing is moved or rewritten.

The same metastore can stay connected as a Hive catalog for its Hive tables and views. Each catalog shows only its own kind of table:

Catalog over the metastoreShows
Hive catalogHive tables and views; Iceberg tables hidden
Iceberg catalogIceberg tables only

Quick start​

The example connects a metastore that holds Iceberg tables on S3 as the catalog hms_iceberg, lists the tables in its database sales, and points an engine at them through the Iceberg REST catalog service.

  1. Create the catalog. Set catalog-backend to hive and uri to the metastore's Thrift address. warehouse is where new tables are created; existing tables keep their own locations.

    Connect the metastore as an Iceberg catalog
    curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
      -H "Accept: application/vnd.gravitino.v1+json" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $TOKEN" \
      -d '{
        "name": "hms_iceberg",
        "type": "RELATIONAL",
        "provider": "lakehouse-iceberg",
        "properties": {
          "catalog-backend": "hive",
          "uri": "thrift://{metastore_host}:9083",
          "warehouse": "s3://{bucket}/warehouse",
          "s3-region": "{region}",
          "s3-access-key-id": "{access_key_id}",
          "s3-secret-access-key": "{secret_access_key}"
        }
      }'

  2. List the tables. The metastore's databases appear as schemas, and their Iceberg tables as tables.

    List the Iceberg tables in a database
    curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/hms_iceberg/schemas/sales/tables" \
      -H "Accept: application/vnd.gravitino.v1+json" \
      -H "Authorization: Bearer $TOKEN"

  3. Point an engine at the tables. Use the catalog name, hms_iceberg, as the warehouse in the engine's Iceberg REST settings, as in the Iceberg REST catalog service Quick start.

Properties​

The catalog takes the properties of any Iceberg catalog, with catalog-backend set to hive. Storage properties for S3, ADLS and GCS, and the credential providers that vend access to engines, are in Object storage. A metastore secured with Kerberos is configured as in Catalog backend security.

Access control​

Grants are made on each catalog's own objects. A grant on a schema in the Hive catalog does not cover the Iceberg tables in the same metastore database; grant on the Iceberg catalog's schema or tables for those. See Privileges.

Moving off the metastore​

The metastore can stay the system of record for as long as you like. To retire it later, register each table's current metadata file in an Iceberg catalog with a JDBC backend, through the Iceberg REST catalog service's register table operation, then drop the table from this catalog without purging its files.

Last updated on