Catalogs and schemas

A catalog is a connection to a system that holds data or metadata, and a schema is a namespace inside it. Together they are the two levels between a metalake and the objects you actually work with.

A catalog is where federation happens. Registering one tells Gravitino how to reach a Hive metastore, an Iceberg REST catalog, a MySQL database, a Kafka cluster, or object storage, and from that point the objects inside appear alongside everything else in the metalake. Gravitino does not copy them. Listing a schema asks the source system, so what you see is what is there now.

A schema is the middle level, and what it means depends on the system underneath. In a relational catalog it is a database. In a fileset catalog it groups filesets under a location. In a model catalog it is a namespace for models with no physical counterpart at all. The schema operations themselves are the same for every catalog type.

Catalogs and schemas both carry tags and policies, and both are the level to attach them at when something should apply broadly. A tag on a catalog reaches every object beneath it.

Quick start

The example connects a Hive metastore as the catalog sales in the metalake example, and creates the schema public in it. Connecting to an existing system surfaces the schemas already there, and creating a schema through Gravitino creates it in the source system too, where the source supports that. Tables, filesets, topics, models, and functions all live under a schema and are reached the same way regardless of which system stores them.

UI

1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter sales as the Catalog name. A Comment and Tags are optional.

2. Choose the type and provider. Choose Relational as the Catalog type; the other types are Messaging, Fileset, and Model. Choose Apache Hive in Select provider and enter thrift://localhost:9083 as the Metastore URI under Connection. The connection fields depend on the provider; for Apache Iceberg they are the Catalog backend, URI, Warehouse, and Authentication. Advanced holds the cloud region and any other catalog properties.

3. Create the catalog. Click Test connection, then Create catalog.

4. Create the schema. Open Explore, which shows the catalogs of the current metalake as a tree, grouped by type. Open sales to list its schemas, click + Schema on the catalog's page, and create public.

The Catalogs list shows each catalog's provider, endpoint, and connection status, with controls on each row to test the connection again and to delete the catalog. Opening a catalog from the list edits its name, description, properties, and storage credentials.

The header of each catalog and schema in Explore shows its grants, tags, and policies, with controls to change them. To tag or attach a policy to several catalogs or schemas at once, select their rows and click + Tags or + Policies.

REST

1. Create the catalog. Properties carry the connection details, and other providers take different ones.

   Create a Hive catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/example/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
     "name": "sales",
     "type": "RELATIONAL",
     "provider": "hive",
     "comment": "Sales estate",
     "properties": {"metastore.uris": "thrift://localhost:9083"}
   }'

2. Create the schema.

   Create a schema

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/example/catalogs/sales/schemas" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
     "name": "public",
     "comment": "Shared datasets",
     "properties": {}
   }'

3. List the schemas. The list includes the schemas that already existed in the metastore.

   List schemas

   curl -sS "https://{gravitino_host}/api/metalakes/example/catalogs/sales/schemas" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

The catalog model

Catalog types and providers

Every catalog has a type, which decides what kind of object lives inside it.

Type	Contains
RELATIONAL	Tables and views
FILESET	Filesets
MESSAGING	Topics
MODEL	Models

Most types also need a provider, which names the system being connected: hive, lakehouse-iceberg, jdbc-mysql, kafka, and so on. Fileset and model catalogs are the exception. Gravitino manages their metadata itself, so the type is enough and the provider is inferred.

Each provider has its own connection properties and its own page describing them.

Names and properties

A catalog name is unique within its metalake, and a schema name is unique within its catalog.

Properties carry the connection details, and which ones apply depends entirely on the provider. Two reserved keys behave the same everywhere. in-use records whether the catalog is available and defaults to true. location sets a base path where the catalog type uses one.

A schema accepts property changes only. It cannot be renamed, and its comment cannot be changed after creation.

In use and not in use

A catalog that is not in use can only be listed, loaded, enabled, or dropped. Every other operation on it or on anything inside it fails, which makes disabling a way to take a connection out of service without removing its metadata.

Disabling a catalog does not touch the source system. The tables and files it points at stay exactly where they are.

What Gravitino stores

Gravitino stores the catalog registration, the schemas and objects created through it, and anything attached to those objects such as tags, policies, and ownership. It does not store a copy of the source system's contents.

Listing tables in a schema reaches the source system at request time, so a table created directly in Hive appears the next time Gravitino is asked. Until something lists or loads such a table, Gravitino holds no record of it.

Delete catalogs and schemas

Deleting is guarded. A catalog is dropped with or without force, and a schema with or without cascade.

Without force, a catalog must have no schemas and must not be in use. Either condition fails the request rather than removing anything. With force, Gravitino removes the registration and everything it holds about the contents. For managed objects, such as a managed fileset or a model, that removes the data too. For federated objects, the Hive tables or S3 files stay where they are and only Gravitino's record of them goes.

A schema is dropped in the source system as well as in Gravitino. Without cascade, it must have no objects, or the request fails. With cascade, the objects inside it are dropped with it.

Permissions

Creating a catalog requires CREATE_CATALOG on the metalake, and reaching a catalog and its contents requires USE_CATALOG. Creating a schema requires CREATE_SCHEMA, and reaching a schema and its contents requires USE_SCHEMA.

Granting at a wider scope covers everything beneath it, so USE_CATALOG on the metalake reaches every catalog in it.

Altering and dropping a catalog or schema are reserved for the metalake owner and the object owner. Ownership resolves down the hierarchy, so the owner of a catalog has the owner path to everything beneath it.

See Privileges for a comprehensive list of privilege grants.

Use the API

Catalogs and schemas can be created, listed, altered, and dropped over REST and through the Java and Python clients. The REST reference documents each call for catalogs and schemas, with the client code under the Java and Python tabs.