Manage metadata using Gravitino
This page introduces how to manage metadata by Gravitino. Through Gravitino, you can create, edit, and delete metadata like metalakes, catalogs, schemas, and tables. This page includes the following contents:
In this document, Gravitino uses Apache Hive catalog as an example to show how to manage metadata by Gravitino. Other catalogs are similar to Hive catalog, but they may have some differences, especially in catalog property, table property, and column type. For more details, please refer to the related doc.
Assuming Gravitino has just started, and the host and port is http://localhost:8090.
Metalake operations
Create a metalake
You can create a metalake by sending a POST
request to the /api/metalakes
endpoint or just use the Gravitino Java client.
The following is an example of creating a metalake:
- Shell
- Java
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{"name":"metalake","comment":"comment","properties":{}}' \
http://localhost:8090/api/metalakes
GravitinoClient gravitinoClient = GravitinoClient
.builder("http://127.0.0.1:8090")
.build();
GravitinoMetaLake newMetalake = gravitinoClient.createMetalake(
NameIdentifier.of("metalake"),
"This is a new metalake",
new HashMap<>());
// ...
Load a metalake
You can create a metalake by sending a GET
request to the /api/metalakes/{metalake_name}
endpoint or just use the Gravitino Java client. The following is an example of loading a metalake:
- Shell
- Java
curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/metalake
// ...
GravitinoMetaLake loaded = gravitinoClient.loadMetalake(
NameIdentifier.of("metalake"));
// ...
Alter a metalake
You can modify a metalake by sending a PUT
request to the /api/metalakes/{metalake_name}
endpoint or just use the Gravitino Java client. The following is an example of altering a metalake:
- Shell
- Java
curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"updates": [
{
"@type": "rename",
"newName": "metalake"
},
{
"@type": "setProperty",
"property": "key2",
"value": "value2"
}
]
}' http://localhost:8090/api/metalakes/new_metalake
// ...
GravitinoMetaLake renamed = gravitinoClient.alterMetalake(
NameIdentifier.of("new_metalake"),
MetalakeChange.rename("new_metalake_renamed")
);
// ...
Currently, Gravitino supports the following changes to a metalake:
Supported modification | JSON | Java |
---|---|---|
Rename metalake | {"@type":"rename","newName":"metalake_renamed"} | MetalakeChange.rename("metalake_renamed") |
Update comment | {"@type":"updateComment","newComment":"new_comment"} | MetalakeChange.updateComment("new_comment") |
Set a property | {"@type":"setProperty","property":"key1","value":"value1"} | MetalakeChange.setProperty("key1", "value1") |
Remove a property | {"@type":"removeProperty","property":"key1"} | MetalakeChange.removeProperty("key1") |
Drop a metalake
You can remove a metalake by sending a DELETE
request to the /api/metalakes/{metalake_name}
endpoint or just use the Gravitino Java client. The following is an example of dropping a metalake:
- Shell
- Java
curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/metalake
// ...
boolean success = gravitinoClient.dropMetalake(
NameIdentifier.of("metalake")
);
// ...
Dropping a metalake only removes metadata about the metalake and catalogs, schemas, tables under the metalake in Gravitino, It doesn't remove the real schema and table data in Apache Hive.
List all metalakes
You can list metalakes by sending a GET
request to the /api/metalakes
endpoint or just use the Gravitino Java client. The following is an example of listing all metalake names:
- Shell
- Java
curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes
// ...
GravitinoMetaLake[] allMetalakes = gravitinoClient.listMetalakes();
// ...
Catalog operations
Create a catalog
Users should create a metalake before creating a catalog.
The code below is an example of creating a Hive catalog. For other catalogs, the code is similar, but the catalog type, provider, and properties may be different. For more details, please refer to the related doc.
You can create a catalog by sending a POST
request to the /api/metalakes/{metalake_name}/catalogs
endpoint or just use the Gravitino Java client. The following is an example of creating a catalog:
- Shell
- Java
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "catalog",
"type": "RELATIONAL",
"comment": "comment",
"provider": "hive",
"properties": {
"metastore.uris": "thrift://localhost:9083"
}
}' http://localhost:8090/api/metalakes/metalake/catalogs
GravitinoClient gravitinoClient = GravitinoClient
.builder("http://127.0.0.1:8090")
.build();
// Assuming you have just created a metalake named `metalake`
GravitinoMetaLake gravitinoMetaLake =
gravitinoClient.loadMetalake(NameIdentifier.of("metalake"));
Map<String, String> hiveProperties = ImmutableMap.<String, String>builder()
// You should replace the following with your own hive metastore uris that Gravitino can access
.put("metastore.uris", "thrift://localhost:9083")
.build();
Catalog catalog = gravitinoMetaLake.createCatalog(
NameIdentifier.of("metalake", "catalog"),
Type.RELATIONAL,
"hive", // provider, We support hive, jdbc-mysql, jdbc-postgresql, lakehouse-iceberg, etc.
"This is a hive catalog",
hiveProperties); // Please change the properties according to the value of the provider.
// ...
Currently, Gravitino supports the following catalog providers:
Catalog provider | Catalog property |
---|---|
hive | Hive catalog property |
lakehouse-iceberg | Iceberg catalog property |
jdbc-mysql | MySQL catalog property |
jdbc-postgresql | PostgreSQL catalog property |
Load a catalog
You can load a catalog by sending a GET
request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}
endpoint or just use the Gravitino Java client. The following is an example of loading a catalog:
- Shell
- Java
curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/metalake/catalogs/catalog
// ...
// Assuming you have just created a metalake named `metalake`
GravitinoMetaLake gravitinoMetaLake =
gravitinoClient.loadMetalake(NameIdentifier.of("metalake"));
Catalog catalog = gravitinoMetaLake.loadCatalog(NameIdentifier.of("metalake", "catalog"));
// ...
Alter a catalog
You can modify a catalog by sending a PUT
request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}
endpoint or just use the Gravitino Java client. The following is an example of altering a catalog:
- Shell
- Java
curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"updates": [
{
"@type": "rename",
"newName": "alter_catalog"
},
{
"@type": "setProperty",
"property": "key3",
"value": "value3"
}
]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog
// ...
// Assuming you have just created a metalake named `metalake`
GravitinoMetaLake gravitinoMetaLake =
gravitinoClient.loadMetalake(NameIdentifier.of("metalake"));
Catalog catalog = gravitinoMetaLake.alterCatalog(NameIdentifier.of("metalake", "catalog"),
CatalogChange.rename("alter_catalog"), CatalogChange.updateComment("new comment"));
// ...
Currently, Gravitino supports the following changes to a catalog:
Supported modification | JSON | Java |
---|---|---|
Rename metalake | {"@type":"rename","newName":"metalake_renamed"} | CatalogChange.rename("catalog_renamed") |
Update comment | {"@type":"updateComment","newComment":"new_comment"} | CatalogChange.updateComment("new_comment") |
Set a property | {"@type":"setProperty","property":"key1","value":"value1"} | CatalogChange.setProperty("key1", "value1") |
Remove a property | {"@type":"removeProperty","property":"key1"} | CatalogChange.removeProperty("key1") |
Drop a catalog
You can remove a catalog by sending a DELETE
request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}
endpoint or just use the Gravitino Java client. The following is an example of dropping a catalog:
- Shell
- Java
curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog
// ...
// Assuming you have just created a metalake named `metalake`
GravitinoMetaLake gravitinoMetaLake =
gravitinoClient.loadMetalake(NameIdentifier.of("metalake"));
gravitinoMetaLake.dropCatalog(NameIdentifier.of("metalake", "catalog"));
// ...
// ...
Dropping a catalog only removes metadata about the catalog, schemas, and tables under the catalog in Gravitino, It doesn't remove the real data (table and schema) in Apache Hive.
List all catalogs in a metalake
You can list all catalogs under a metalake by sending a GET
request to the /api/metalakes/{metalake_name}/catalogs
endpoint or just use the Gravitino Java client. The following is an example of listing all catalogs in
a metalake:
- Shell
- Java
curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs
// ...
// Assuming you have just created a metalake named `metalake`
GravitinoMetaLake gravitinoMetaLake =
gravitinoClient.loadMetalake(NameIdentifier.of("metalake"));
NameIdentifier[] catalogsIdents = gravitinoMetaLake.listCatalogs(Namespace.ofCatalog("metalake"));
// ...
Schema operations
Users should create a metalake and a catalog before creating a schema.