Kafka catalog

A Kafka catalog connects one Apache Kafka cluster, so its topics appear in Gravitino where they can be created, altered, dropped, tagged, and governed like any other object. A Kafka catalog is a messaging catalog, the MESSAGING catalog type.

The catalog uses the Kafka admin API, so services compatible with Kafka, such as Redpanda and Amazon MSK, also work for the topic operations they implement. Apache Kafka is the tested broker.

Quick start

The example connects a Kafka cluster as the messaging catalog streams and lists its topics. Topics already in the cluster appear as soon as the catalog is connected. To create and tag a topic, see the Topics Quick start.

UI

1. Connect the cluster. Open Connect → Catalogs, click + Catalog, enter streams as the Catalog name, and choose Messaging as the Catalog type. Apache Kafka is the provider. Enter {broker_1}:9092,{broker_2}:9092 as the Bootstrap servers, click Test connection, then Create catalog.

2. List the topics. In Explore, open streams and then its one schema, default. Every topic in the cluster is listed there.

REST

1. Connect the cluster.

   Create a Kafka catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "streams",
       "type": "MESSAGING",
       "provider": "kafka",
       "comment": "Kafka cluster",
       "properties": {"bootstrap.servers": "{broker_1}:9092,{broker_2}:9092"}
     }'

2. List the topics. A Kafka catalog has one schema, default, which holds every topic.

   List the topics in a Kafka catalog

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/streams/schemas/default/topics" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

Catalog properties

Besides the common catalog properties, the Kafka catalog takes these:

Property	Description	Default	Required
bootstrap.servers	Kafka brokers to connect to, separated by commas, such as {broker_1}:9092,{broker_2}:9092.	(none)	Yes
gravitino.bypass.*	Passed to the Kafka admin client with the prefix removed, such as gravitino.bypass.security.protocol. See the Kafka admin configs.	(none)	No

A secured cluster takes its client security settings, such as security.protocol and the SASL settings, through gravitino.bypass.*. Other catalog operations are covered in Catalogs and schemas.

Schema

The catalog creates one schema, default, which holds every topic in the cluster. It can be listed and loaded but not created, altered, or dropped, and it has no properties.

Topics

Topics can be created, altered, dropped, and listed. Creating a topic in Gravitino creates it in Kafka, and topics created in Kafka appear in Gravitino.

Property	Description	Default	Required
partition-count	Number of partitions of the topic.	The broker's num.partitions	No
replication-factor	Number of replicas of each partition.	The broker's default.replication.factor	No

Any other topic configuration, such as retention.ms, can be passed as a topic property.

Altering a topic can change its comment, set or remove topic configurations, and raise partition-count. Kafka cannot remove partitions, so lowering partition-count or removing it is rejected, and replication-factor cannot change after the topic is created. Creating, tagging, and altering topics is covered in Topics.

Related

• Topics
• Catalogs and schemas