Manage catalogs and schemas

Introduction

This page covers the Gravitino API for catalogs and schemas. For what a catalog and a schema are, the catalog types, what Gravitino stores, permissions, and how to work with them in the UI, see Catalogs and Schemas.

Connection properties differ by provider and are documented on each catalog type's own page.

Catalog operations

Create

A catalog needs a name, a type, and for most types a provider. Properties carry the connection details. The example below registers a Hive metastore; other providers take different properties.

REST

Create a Hive catalog

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "sales",
  "type": "RELATIONAL",
  "provider": "hive",
  "comment": "Sales estate",
  "properties": {"metastore.uris": "thrift://localhost:9083"}
}' http://localhost:8090/api/metalakes/example/catalogs

Java

Create a Hive catalog

GravitinoClient client = GravitinoClient
    .builder("http://localhost:8090")
    .withMetalake("example")
    .build();

Catalog catalog = client.createCatalog(
    "sales",
    Catalog.Type.RELATIONAL,
    "hive",
    "Sales estate",
    ImmutableMap.of("metastore.uris", "thrift://localhost:9083"));

Python

Create a Hive catalog

catalog = client.create_catalog(
    name="sales",
    catalog_type=Catalog.Type.RELATIONAL,
    provider="hive",
    comment="Sales estate",
    properties={"metastore.uris": "thrift://localhost:9083"})

Fileset and model catalogs are managed by Gravitino rather than federated, so they take no provider.

REST

Create a fileset catalog

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "landing",
  "type": "FILESET",
  "comment": "Landing zone",
  "properties": {"location": "s3a://example-bucket/landing"}
}' http://localhost:8090/api/metalakes/example/catalogs

Load

REST

Load a catalog

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/sales

Java

Load a catalog

Catalog catalog = client.loadCatalog("sales");

Python

Load a catalog

catalog = client.load_catalog("sales")

Alter

Changes are applied as a list in one request.

Change	JSON	Java	Python
Rename	{"@type":"rename","newName":"sales_v2"}	CatalogChange.rename("sales_v2")	CatalogChange.rename("sales_v2")
Update the comment	{"@type":"updateComment","newComment":"new_comment"}	CatalogChange.updateComment("new_comment")	CatalogChange.update_comment("new_comment")
Set a property	{"@type":"setProperty","property":"key1","value":"value1"}	CatalogChange.setProperty("key1", "value1")	CatalogChange.set_property("key1", "value1")
Remove a property	{"@type":"removeProperty","property":"key1"}	CatalogChange.removeProperty("key1")	CatalogChange.remove_property("key1")

REST

Update a catalog comment

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "updates": [
    {"@type": "updateComment", "newComment": "Sales estate, production"}
  ]
}' http://localhost:8090/api/metalakes/example/catalogs/sales

Java

Update a catalog comment

Catalog catalog = client.alterCatalog(
    "sales", CatalogChange.updateComment("Sales estate, production"));

Python

Update a catalog comment

catalog = client.alter_catalog(
    "sales", CatalogChange.update_comment("Sales estate, production"))

Enable or disable

A catalog that is not in use can only be listed, loaded, enabled, or dropped.

REST

Disable or enable a catalog

curl -X PATCH -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{"inUse": false}' \
  http://localhost:8090/api/metalakes/example/catalogs/sales

curl -X PATCH -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{"inUse": true}' \
  http://localhost:8090/api/metalakes/example/catalogs/sales

Java

Disable or enable a catalog

client.disableCatalog("sales");
client.enableCatalog("sales");

Python

Disable or enable a catalog

client.disable_catalog("sales")
client.enable_catalog("sales")

Drop

Without force, the catalog must have no schemas and must not be in use. With force, Gravitino removes the registration and everything it holds about the contents.

REST

Drop a catalog

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
  "http://localhost:8090/api/metalakes/example/catalogs/sales?force=false"

Java

Drop a catalog

boolean dropped = client.dropCatalog("sales", false);

Python

Drop a catalog

dropped = client.drop_catalog("sales", force=False)

List

REST

List catalogs

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  "http://localhost:8090/api/metalakes/example/catalogs?details=true"

Java

List catalogs

String[] catalogNames = client.listCatalogs();
Catalog[] catalogs = client.listCatalogsInfo();

Python

List catalogs

catalog_names = client.list_catalogs()
catalogs = client.list_catalogs_info()

Schema operations

Schema operations are the same for every catalog type. Creating a schema through Gravitino creates it in the source system too, where the source supports that.

Create

REST

Create a schema

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "public",
  "comment": "Shared datasets",
  "properties": {}
}' http://localhost:8090/api/metalakes/example/catalogs/sales/schemas

Java

Create a schema

Catalog catalog = client.loadCatalog("sales");
Schema schema = catalog.asSchemas().createSchema(
    "public", "Shared datasets", Collections.emptyMap());

Python

Create a schema

catalog = client.load_catalog("sales")
schema = catalog.as_schemas().create_schema(
    schema_name="public", comment="Shared datasets", properties={})

Load

REST

Load a schema

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/sales/schemas/public

Java

Load a schema

Schema schema = catalog.asSchemas().loadSchema("public");

Python

Load a schema

schema = catalog.as_schemas().load_schema("public")

Alter

A schema takes property changes only. It cannot be renamed, and its comment cannot be changed.

Change	JSON	Java	Python
Set a property	{"@type":"setProperty","property":"key1","value":"value1"}	SchemaChange.setProperty("key1", "value1")	SchemaChange.set_property("key1", "value1")
Remove a property	{"@type":"removeProperty","property":"key1"}	SchemaChange.removeProperty("key1")	SchemaChange.remove_property("key1")

REST

Set a schema property

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "updates": [
    {"@type": "setProperty", "property": "owner", "value": "sales-eng"}
  ]
}' http://localhost:8090/api/metalakes/example/catalogs/sales/schemas/public

Java

Set a schema property

Schema schema = catalog.asSchemas().alterSchema(
    "public", SchemaChange.setProperty("owner", "sales-eng"));

Python

Set a schema property

schema = catalog.as_schemas().alter_schema(
    "public", SchemaChange.set_property("owner", "sales-eng"))

Drop

Without cascade, the schema must be empty. With cascade, everything inside it goes as well.

REST

Drop a schema

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
  "http://localhost:8090/api/metalakes/example/catalogs/sales/schemas/public?cascade=false"

Java

Drop a schema

boolean dropped = catalog.asSchemas().dropSchema("public", false);

Python

Drop a schema

dropped = catalog.as_schemas().drop_schema("public", cascade=False)

List

REST

List schemas

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/sales/schemas

Java

List schemas

String[] schemaNames = catalog.asSchemas().listSchemas();

Python

List schemas

schema_names = catalog.as_schemas().list_schemas()