Skip to main content

Topics

A topic is a stream of messages, and in Gravitino it is a metadata object like any other. Registering a Kafka cluster as a messaging catalog puts its topics in the same hierarchy as tables and filesets, so a stream can be found, classified, and governed alongside the data it feeds.

Gravitino holds the reference rather than the messages. Listing topics asks the cluster at request time, so a topic created directly in Kafka appears the next time Gravitino is asked, and deleting a topic through Gravitino deletes it in the cluster.

The value is coverage rather than new capability. Without it, streams sit outside the catalog and outside whatever classification and access rules apply to everything else.

Quick start​

The example connects a Kafka cluster as the messaging catalog streams, creates the topic clicks in it with three partitions and a seven-day retention, and tags the topic pii. The tag must already exist; see Tags. Topics already in the cluster appear in the catalog as soon as it is connected, and creating a topic through Gravitino creates it in the cluster.

  1. Connect the cluster. Open Connect → Catalogs, click + Catalog, enter streams as the Catalog name, and choose Messaging as the Catalog type. Apache Kafka is the provider. Enter kafka-1:9092,kafka-2:9092 as the Bootstrap servers, click Test connection, then Create catalog. See Apache Kafka catalog for the other connection properties.

  2. Open the topics. In Explore, open streams and then its one schema, default. Every topic in the cluster is listed there with its tags and policies.

  3. Create the topic. Click + Topic and enter clicks as the Topic name. Under Delivery, enter 3 as the Partitions and leave Replication factor empty to take the broker default. Retention starts at 7 days, which the example keeps; choose Broker default to leave it to the broker. Other Kafka topic settings go under Properties. Click Create topic.

  4. Tag it. Select the clicks row, click + Tags, choose pii under Additional tags, and click Update tags.

Opening a topic shows its description, owner, and properties on Overview, including every Kafka setting the topic carries. The description, owner, and properties can be edited and saved there. The header shows its grants, tags, and policies, with a control for each.

The topic model​

Messaging systems​

Kafka is the only messaging system with a catalog today, connected with the kafka provider and the cluster's bootstrap.servers. Anything speaking the Kafka protocol works through the same connector. Other messaging systems have no catalog.

The default schema​

A messaging catalog presents one schema, named default, holding every topic in the cluster. Kafka has no namespace of its own to map onto, so creating or dropping a schema in a messaging catalog is rejected rather than silently ignored.

The consequence for a large cluster is that topics are not grouped in the catalog the way tables are grouped by database. Tags are the way to organize them.

Names and properties​

A topic name is unique within its schema, and matches the topic name in the cluster.

partition-count sets the number of partitions and can be increased afterward but not reduced. replication-factor sets the replication and is immutable once the topic exists. Leaving either unset takes the broker's own default, from num.partitions and default.replication.factor respectively. Any other property given when creating a topic is passed to Kafka as a topic setting, such as retention.ms or cleanup.policy, and loading a topic returns every setting it carries.

What Gravitino stores and what it does not​

Gravitino stores the topic's place in the hierarchy and anything attached to it, including tags, policies, and ownership. Message content, offsets, consumer groups, and lag stay entirely in the cluster. Dropping a topic through Gravitino deletes it in the cluster, along with its messages.

Message schemas are also outside the catalog. Gravitino does not integrate with a schema registry, so the structure of the messages in a topic is not described here and cannot be classified per field the way table columns can. Tags and policies attach to the topic as a whole.

Permissions​

Creating a topic requires CREATE_TOPIC on its schema or above. Reading from a topic requires CONSUME_TOPIC, and writing to or altering a topic requires PRODUCE_TOPIC, which also allows reading.

Granting at a wider scope covers everything beneath it. Dropping a topic is reserved for the metalake owner and the object owner.

See Privileges for a comprehensive list of privilege grants.

Use the API​

Topics can be created, listed, altered, and dropped over REST and through the Java client. The Python client does not cover topics. The REST reference documents each call, with the client code under the Java tab.

Last updated on