Skip to main content
Version: 0.5.1

Manage massaging metadata using Gravitino

This page introduces how to manage messaging metadata using Gravitino. Messaging metadata refers to the topic metadata of the messaging system such as Apache Kafka, Apache Pulsar, Apache RocketMQ, etc. Through Gravitino, you can create, update, delete, and list topics via unified RESTful APIs or Java client.

To use messaging catalog, please make sure that:

  • Gravitino server has started, and the host and port is http://localhost:8090.
  • A metalake has been created.

Catalog operations

Create a catalog

tip

For a messaging catalog, you must specify the type as messaging when creating a catalog.

You can create a catalog by sending a POST request to the /api/metalakes/{metalake_name}/catalogs endpoint or just use the Gravitino Java client. The following is an example of creating a messaging catalog:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "catalog",
"type": "MESSAGING",
"comment": "comment",
"provider": "kafka",
"properties": {
"bootstrap.servers": "localhost:9092",
}
}' http://localhost:8090/api/metalakes/metalake/catalogs

Currently, Gravitino supports the following catalog providers:

Catalog providerCatalog property
kafkaKafka catalog property

Load a catalog

Refer to Load a catalog in relational catalog for more details. For a messaging catalog, the load operation is the same.

Alter a catalog

Refer to Alter a catalog in relational catalog for more details. For a messaging catalog, the alter operation is the same.

Drop a catalog

Refer to Drop a catalog in relational catalog for more details. For a messaging catalog, the drop operation is the same.

List all catalogs in a metalake

Please refer to List all catalogs in a metalake in relational catalog for more details. For a messaging catalog, the list operation is the same.

List all catalogs' information in a metalake

Please refer to List all catalogs' information in a metalake in relational catalog for more details. For a messaging catalog, the list operation is the same.

Schema operations

Schema is a logical grouping of topics in a messaging catalog, if the messaging system does not support topics grouping, schema operations are not supported but a "default" schema will be automatically created to include all topics

note

Gravitino currently only supports the Kafka catalog. Since Kafka does not support topic grouping, only list and load operations are supported for schema.

Create a schema

Please refer to Create a schema in relational catalog for more details. For a messaging catalog, the create operation is the same.

Load a schema

Please refer to Load a schema in relational catalog for more details. For a messaging catalog, the load operation is the same.

Alter a schema

Please refer to Alter a schema in relational catalog for more details. For a messaging catalog, the alter operation is the same.

Drop a schema

Please refer to Drop a schema in relational catalog for more details. For a messaging catalog, the drop operation is the same.

List all schemas under a catalog

Please refer to List all schemas under a catalog in relational catalog for more details. For a messaging catalog, the list operation is the same.

Topic operations

tip

Users should create a metalake, a catalog and a schema before creating a table.

Create a topic

You can create a topic by sending a POST request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/topics endpoint or just use the Gravitino Java client. The following is an example of creating a topic:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "example_topic",
"comment": "This is an example topic",
"properties": {
"partition-count": "3",
"replication-factor": 1
}
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/default/topics

Alter a topic

You can modify a topic by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/topics/{topic_name} endpoint or just use the Gravitino Java client. The following is an example of altering a topic:

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"updates": [
{
"@type": "removeProperty",
"property": "key2"
}, {
"@type": "setProperty",
"property": "key3",
"value": "value3"
}
]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/default/topics/topic

Currently, Gravitino supports the following changes to a topic:

Supported modificationJSONJava
Update a comment{"@type":"updateComment","newComment":"new_comment"}TopicChange.updateComment("new_comment")
Set a topic property{"@type":"setProperty","property":"key1","value":"value1"}TopicChange.setProperty("key1", "value1")
Remove a topic property{"@type":"removeProperty","property":"key1"}TopicChange.removeProperty("key1")

Drop a topic

You can remove a topic by sending a DELETE request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/topics/{topic_name} endpoint or by using the Gravitino Java client. The following is an example of dropping a topic:

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/default/topics/topic

List all topics under a schema

You can list all topics in a schema by sending a GET request to the /api/metalakes/ {metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/topics endpoint or by using the Gravitino Java client. The following is an example of listing all the topics in a schema:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/topics