Hive metastore as a Hudi catalog

A Hudi catalog connects the Hive metastore where Apache Hudi registers its tables, so the Hudi databases and tables appear in Gravitino as schemas and tables. The catalog is read-only: it lists and loads, and tables are created and changed with Hudi itself. Only tables stored in Hudi formats are listed; other tables in the same metastore are skipped. The catalog is tested with Apache Hudi 0.15.0.

Quick start

The example connects the Hive metastore that holds your Hudi tables as the catalog hudi_catalog and lists what it finds. The catalog is read-only, so tables are created with Hudi itself and appear in Gravitino through the metastore.

UI

1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter hudi_catalog as the Catalog name.

2. Choose the provider. Keep Relational as the Catalog type and choose Apache Hudi under Lakehouse formats in Select provider. Under Connection, the Catalog backend is already hms. Enter thrift://{metastore_host}:9083 as the URI and keep None as the Authentication.

3. Create the catalog. Click Test connection, then Create catalog.

4. Browse the catalog. Open Explore and open hudi_catalog to list its schemas, which are the metastore databases. Open a schema to list its Hudi tables. The catalog page has no + Schema button, because the catalog is read-only.

REST

1. Create the catalog. hms is the only backend.

   Create a Hudi catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "hudi_catalog",
       "type": "RELATIONAL",
       "provider": "lakehouse-hudi",
       "comment": "Hudi catalog",
       "properties": {
         "catalog-backend": "hms",
         "uri": "thrift://{metastore_host}:9083"
       }
     }'

2. List the schemas. They are the metastore databases.

   List the schemas in the Hudi catalog

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/hudi_catalog/schemas" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

3. List the tables in a schema. Only Hudi tables are listed.

   List the Hudi tables in a schema

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/hudi_catalog/schemas/{schema}/tables" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

Catalog properties

Property	Description	Default	Required
catalog-backend	Where Hudi tables are registered. Only hms, the Hive metastore, is supported.	(none)	Yes
uri	Address of the Hive metastore, such as thrift://{hms_host}:9083.	(none)	Yes
client.pool-size	Maximum number of Hive metastore clients the catalog keeps open.	1	No
client.pool-cache.eviction-interval-ms	Interval at which idle metastore clients are evicted, in milliseconds.	300000	No
default.catalog	Hive catalog to read in a Hive 3 metastore. Ignored by a Hive 2 metastore.	hive	No
gravitino.bypass.*	Passed to the metastore client with the prefix removed; for example, gravitino.bypass.hive.metastore.failure.retries sets hive.metastore.failure.retries.	(none)	No

Catalog backend security

For a Kerberized metastore, set authentication.type to kerberos and give the principal and keytab.

Property	Description	Default	Required
authentication.type	How the catalog authenticates to the metastore: simple or kerberos.	simple	No
authentication.impersonation-enable	Act as the calling user when talking to the metastore.	false	No
authentication.kerberos.principal	Kerberos principal.	(none)	With Kerberos
authentication.kerberos.keytab-uri	URI of the keytab for the principal.	(none)	With Kerberos
authentication.kerberos.check-interval-sec	Seconds between checks of the Kerberos credential.	60	No
authentication.kerberos.keytab-fetch-timeout-sec	Seconds allowed to fetch the keytab.	60	No

Other metastore client settings go through gravitino.bypass.*, such as gravitino.bypass.hive.metastore.sasl.enabled=true or gravitino.bypass.hive.metastore.kerberos.principal={metastore_principal}.

Schemas and tables

Schemas and tables can be listed and loaded; the catalog creates, alters, and drops nothing. A schema shows its storage path in the location property. A table carries every parameter the metastore holds for it as table properties. Partitioned Hudi tables load with identity partitions, the only kind Hudi has. Sort orders, distributions, indexes, and Hudi timeline operations are not exposed. See Catalogs and schemas and Tables and views.

Column types

Gravitino types map to Apache Hudi column types as follows:

Gravitino Type	Apache Hudi Type
boolean	boolean
integer	int
long	long
date	date
timestamp	timestamp
float	float
double	double
string	string
decimal	decimal
binary	bytes
array	array
map	map
struct	struct

Related

• Catalogs and schemas
• Tables and views
• Hive catalog