Generic lakehouse catalog

A generic lakehouse catalog manages tables whose data sits in files on a file system or object store, such as HDFS, Amazon S3, Google Cloud Storage, or Azure storage, and records their metadata in Gravitino. Each table names its format in its format property, and the catalog supports two formats: Lance and Delta Lake. Tables in the catalog take part in governance, access control, and lineage like any other Gravitino table, and existing tables can be registered without moving their data. How Lance works explains the Lance format and walks through filling a Lance table with embeddings from a folder of files.

Quick start

The example creates the catalog lakehouse_catalog with its root on HDFS and the schema sales in it. Tables in sales land under hdfs://{namenode_host}:8020/lakehouse/sales/ unless a table sets its own location.

UI

1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter lakehouse_catalog as the Catalog name.

2. Choose the provider. Keep Relational as the Catalog type and choose Lance and Delta under Lakehouse formats in Select provider. Under Connection, enter hdfs://{namenode_host}:8020/lakehouse as the Root path.

3. Create the catalog. Click Create catalog. The form offers no Test connection for this provider, because the catalog connects to no external service.

4. Create a schema. Open Explore, open lakehouse_catalog, click + Schema, enter sales as the Schema name, and click Create schema.

5. Add tables. Every table names its format in the format property. Continue with Lance tables or Delta Lake tables.

REST

1. Create the catalog. location is optional, but without it every schema or table has to set its own.

   Create a generic lakehouse catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "lakehouse_catalog",
       "type": "RELATIONAL",
       "provider": "lakehouse-generic",
       "comment": "Generic lakehouse catalog",
       "properties": {
         "location": "hdfs://{namenode_host}:8020/lakehouse"
       }
     }'

2. Create a schema.

   Create a schema in the generic lakehouse catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/lakehouse_catalog/schemas" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{"name": "sales", "comment": "Sales data", "properties": {}}'

3. Add tables. Every table names its format in the format property. Continue with Lance tables or Delta Lake tables.

Catalog properties

Property	Description	Default	Required
location	Root storage path for every schema and table in the catalog. See Storage location.	(none)	No
lance.storage.*	Storage options for Lance tables, such as lance.storage.access_key_id or lance.storage.region, passed to Lance when it reads or writes table data. See Lance tables.	(none)	No
lance.schema-refresh-mode	When Lance table schemas are refreshed from the dataset. DECLARED_AND_EMPTY refreshes declared tables and tables with no stored columns. VERSION_CHECK also refreshes whenever the Lance dataset version changes.	DECLARED_AND_EMPTY	No

The provider name for a generic lakehouse catalog is lakehouse-generic. Catalog operations other than creation, such as listing, altering, and dropping, work as they do for every catalog; see Catalogs and schemas.

Storage location

Every table resolves its storage location from the most specific level that sets one:

1. The table's own location.
2. The schema's location, giving {schema_location}/{table_name}.
3. The catalog's location, giving {catalog_location}/{schema_name}/{table_name}.

Example location resolution

Only the catalog location is set
  Catalog location: hdfs://{namenode_host}:8020/lakehouse
  Table sales.orders    -> hdfs://{namenode_host}:8020/lakehouse/sales/orders

The schema location is set
  Schema sales location: s3://sales-bucket/data
  Table sales.orders    -> s3://sales-bucket/data/orders

The table location is set
  Table sales.orders location: s3://sales-bucket/my_orders
  Table sales.orders    -> s3://sales-bucket/my_orders

When none of the three is set, creating the table fails.

Schema properties

Property	Description	Default	Required
location	Root storage path for every table in the schema, overriding the catalog location.	(none)	No

Schema operations work as they do for every catalog; see Catalogs and schemas.

Tables

Table operations and table properties depend on the format:

• Lance tables
• Delta Lake tables

Related

• How Lance works

• Catalogs and schemas

• Lance REST service

• Lance integration