August 2026
How many data sources a single catalog can connect to used to be an easy way to judge a unified metadata platform. Apache Gravitino, an open-source federated metadata catalog and Apache Top-Level Project, spent its early life winning exactly that comparison: Hive, Iceberg, relational databases, and more, all through one API.
But "it connects" is the Hello World of catalog problems. Production asks harder questions.
What happens when Iceberg tables are spread across multiple clouds? If each region runs its own catalog, does every client have to configure them one by one? When a permission is revoked, does every node find out right away? And if you just want to run a quick POC, do you really need to stand up Keycloak first?
These are Day 2 problems, and they are where Gravitino's recent work has gone. Multi-cloud IRC, federated IRC, AWS Glue support, unified views, local identity, group-based authorization, audit logging, and cache consistency look like separate features, but they point at the same shift:
Gravitino is moving beyond unified catalog access to an enterprise-grade metadata control plane.
Apache Gravitino PMC Chair and Datastrato CTO Jerry Shao walked through these capabilities in a recent community session, and PMC member Rory Qi looked further ahead: table optimization, the Iceberg REST Catalog specification, and permission control for agent scenarios. If you are using or evaluating Gravitino, this article covers what those capabilities mean for production, and where the project is heading next.
The focus has shifted from bringing catalogs in to making them run
Gravitino set out to solve a simple, widespread problem: enterprises had too many catalogs, all isolated from one another.
Your Hive deployment stays where it is, Iceberg tables stay in object storage, and relational databases don't migrate. Gravitino brings the metadata from those systems into one place and provides a consistent metadata management layer for discovery, access, and governance.
For a long time, the main direction was expanding catalog coverage. That work continues, but the newer capabilities answer a different question: how should a catalog operate once it's in production, with multiple clouds, multiple regions, and multiple nodes?
That question is the thread that ties everything below together.
How does one data catalog span multiple clouds and regions?
Two Iceberg REST Catalog (IRC) capabilities come first: multi-cloud IRC and federated IRC.
The names sound similar, but they solve different problems. Multi-cloud IRC lets a single catalog manage tables stored across different clouds. Federated IRC lets a local catalog proxy catalog services running in other regions.
Multi-cloud Iceberg REST Catalog: one catalog for tables in S3, GCS, and ADLS
Traditionally, an Iceberg REST Catalog is configured with a specific FileIO. AWS S3 uses S3FileIO, while Google Cloud Storage and Azure Data Lake Storage each have their own implementations.
That works in a single-cloud environment. But in many enterprises, multi-cloud data architecture is already the reality, not the exception. Some Iceberg tables live in S3, others in GCS, and still others in MinIO, Pure Storage, or S3-compatible private object stores.
If catalogs are split by storage system, the math is ugly: the more clouds you have, the more catalogs you have. Compute engines maintain multiple catalog connections, and before users can touch a table, they first need to know which catalog it belongs to.
Gravitino addresses this with Resolving FileIO, a routing layer above the individual FileIO implementations. The system reads the table location, determines the object store from the URI scheme, and routes the request to the right FileIO.
Suppose a catalog's default warehouse is in GCS. A user who creates a table without specifying a location gets a table in GCS. A user who explicitly specifies an S3 location gets a table in S3. The same catalog can hold tables in GCS, S3, ADLS, and private enterprise object stores at once.
Path routing is only half of it, though. Different clouds require different credentials, and if users still manage those themselves, the complexity hasn't gone anywhere. So Gravitino obtains vended credentials based on the table location and injects the temporary credential into the actual read/write flow.
From the user's point of view, there is one catalog. Which cloud stores the table, which FileIO applies, and which credentials to fetch are all handled underneath.
Federated Iceberg REST Catalog: a local catalog that proxies remote regions
Multi-cloud IRC solves storage spread inside one catalog. But enterprises commonly run another architecture on top: different regions and data centers each operate their own independent Iceberg REST Catalog. North America, Europe, and Asia might each deploy an IRC with its own users, permissions, storage, and audit system.
Traditionally, upper-layer engines configure each IRC endpoint separately, and users accessing data in another region have to explicitly select a different catalog.
Federated IRC takes a different approach: a local IRC can proxy access to a remote IRC. The compute engine connects only to the local IRC. When a query touches a table in a remote catalog, the local IRC forwards the request, and the remote IRC handles the actual authentication, authorization, metadata read, and credential vending.
There's an important boundary here. Federation does not mean merging every catalog into one system. Each IRC keeps its own user system, permission rules, audit logs, storage connections, and credential vending. The local IRC provides a unified entry point, while each remote IRC continues to control the data and governance boundary for its own region.
That combination matters for multi-region, multi-data-center, and Data Mesh setups: regions stay autonomous, and clients stop maintaining a separate catalog configuration for every region.
Put the two side by side and the distinction is clean: Multi-cloud IRC solves storage distribution, Federated IRC solves service distribution.
A catalog can manage more than one format, and more than tables
Beyond multi-cloud and federation, Gravitino keeps expanding the systems and metadata objects it can manage. Two examples: AWS Glue and unified views.
AWS Glue: no need to split catalogs just because table formats differ
Traditionally, one catalog maps to one table format. A Hive Catalog manages Hive tables, and if the same metastore also contains Iceberg tables, the upper layer creates a separate Iceberg Catalog. The metadata comes from one underlying system, but it gets split again because the formats differ.
Gravitino's Glue Catalog changes that model: a single Glue Catalog can manage both Hive and Iceberg tables. Users no longer create two Gravitino catalogs because of format, and Spark-side support lets compute engines reach that Glue metadata through one entry point.
Behind this is an adjustment to what a catalog is: it doesn't have to map strictly to a single table format. It can map to the metadata system that actually exists in the enterprise.
For users, that removes a layer of duplicate configuration. For the platform, it reflects how the underlying data environment is actually organized.
Views: unified management beyond tables
Gravitino has long managed catalogs, schemas, and tables, with partial Iceberg view support arriving earlier. What was missing was a complete, unified view model at the Gravitino layer.
The hard part is that compute engines don't represent views the same way. The same business logic might be Trino SQL in Trino, a different SQL dialect in Spark, and in some cases a Python representation.
Gravitino doesn't try to invent a universal SQL dialect every engine can execute. Instead, one view stores multiple representations: a Trino representation, a Spark representation, a Python representation. When an engine accesses the view, it retrieves the one that fits. The goal is not to erase differences between engines, but to bring those differences into one metadata object and manage them consistently.
Gravitino can also convert views from underlying systems such as Iceberg and Hive into a unified definition, with existing database views in JDBC sources on the adaptation list.
Moving from tables to views looks like adding one more object type, but it opens the next question: if logical views can be managed in a unified way, can metrics, dimensions, and business semantics enter the same metadata system? That's where Metric View comes in, and it's a direction the community is actively working on.
The Python client is catching up with Java
As the set of objects under unified management grows, another question matters: who can conveniently consume this metadata?
The Python client now covers table operations, permission operations, and statistics operations, and is largely aligned with the Java client. The community continues to close the remaining gaps.
This is less visible than multi-cloud or federation, but it's practical for data analysis, automation scripts, and the growing number of AI agents that run on Python. Unified metadata isn't only something to manage. New applications need to actually use it.
What makes a data catalog production-ready? Identity, audit, and consistency
Multi-cloud, federation, Glue, and views are easy to notice. But whether a system can run in an enterprise production environment often depends on less visible capabilities. Where does identity come from? How do you manage permissions in bulk? Can operations be traced afterward? When the service runs on multiple nodes, do all nodes see the same result?
Handle any one of these poorly and it can block production adoption entirely.
You should not need Keycloak just to run a POC
Gravitino has supported external Identity Providers such as Keycloak, Okta, and Azure AD for a while. That's natural for enterprises with a mature identity system. It's less natural when you're just trying the product: to create a few test users and verify one permission flow, you'd first have to deploy and configure an entire external IdP.
Gravitino now ships a local IdP. Users and user groups can be created directly in Gravitino, with basic username-and-password authentication. You can run the complete authentication, authorization, and data-access flow without deploying a separate identity system.
The local IdP can also complement an external one: if network problems make the external identity system temporarily unavailable, local identity information provides a degree of fallback.
Once the system enters production, user synchronization, password security, and identity switching still need to follow the enterprise's own security policies. But there's now an out-of-the-box local identity foundation to start from.
Permissions no longer need to be configured user by user
Once identity is in place, authorization comes next: Gravitino supports group-based permission management.
Enterprise users are already organized by department, project, business line, or job role. Assign permissions one user at a time and the number of rules explodes. With groups, permissions attach to a user group and are inherited by its members.
That reduces authorization configuration, and it aligns with the enterprise's existing organizational structure and external IdP. The value isn't that the system can create a role. It's that the permission model fits the identity and org structure the enterprise already has.
Audit logs that plug into the enterprise stack
Audit information used to be emitted mainly as plain text. Gravitino now adds JSON output and more contextual fields, including user, IP address, and millisecond-level timestamps. The local audit log supports rotation by file size and by time.
Structured logs are far easier to integrate with existing log collection, observability, and security analytics platforms.
A permission system has to answer "can this person access the resource?" It also has to answer, afterward, "who did what, when, from where, and to which object?" The audit log fills in that second half.
Caching is for performance, but nodes cannot disagree
One capability that's easy to underestimate: cache consistency.
Start with the metadata cache in the Iceberg REST Catalog. Iceberg table metadata usually lives in object storage, and rereading and parsing the full metadata file on every table load costs network I/O and parsing time. Gravitino keeps table metadata in memory, and this cache is now GA and enabled by default.
The hard part of caching was never putting something in memory. It's knowing whether the cached data has expired. Gravitino checks version information in the database: if the version hasn't changed, use the cache; if it has, invalidate and reload. ETag support lets clients determine whether table metadata has changed, so unchanged results don't need to be transferred again.
The same issue exists in the authorization cache. Previously, if a role or privilege changed on one node, other nodes might not notice immediately. One node believes the user still has permission while another believes it's been revoked, and the same request gets different answers depending on which node handles it.
The refactored authorization cache uses the same version-check approach: when permissions change, the old cache is invalidated; when nothing has changed, the cached result stands.
The two caches solve different problems. The metadata cache reduces repeated reads of table metadata. The authorization cache keeps permission decisions consistent across nodes. The principle behind both is the same: caching can trade memory for performance, but it cannot trade away correctness.
One clarification: this consistency work currently covers the IRC metadata cache and the authorization cache. Consistency for other entities is still being improved. Enterprise-grade high availability isn't finished in a single pass; it's a long-term capability that needs continuous reinforcement.
Where is Apache Gravitino heading next?
Rory Qi's portion of the session looked at the directions Gravitino will continue to pursue. Three are worth watching: turning table optimization into a service, continued evolution of the Iceberg REST Catalog specification, and more fine-grained security including agent permissions.
Table optimization: from CLI commands to a long-running service
Iceberg tables accumulate maintenance needs as they run. Historical snapshots need cleanup, orphan files no longer referenced by metadata need to be reclaimed, and data files may need reordering based on query patterns.
Today, some of this runs through CLI commands. The plan is to move toward snapshot cleanup policies, orphan-file cleanup policies, sorting optimization policies, and table optimization as a service.
Moving from CLI to a service isn't about typing fewer commands. It means these operations become long-running, centrally scheduled system capabilities.
A traditional catalog is passive: a client asks a question, it returns metadata. With table optimization, the catalog starts acting proactively based on metadata: which snapshots should expire, which files should be reclaimed, which tables should be reorganized. That's one very concrete difference between a control plane and a directory.
Iceberg keeps evolving, and so does the REST specification
Gravitino supports a core set of Iceberg REST Catalog interfaces, while the Iceberg community keeps adding new types, transactions, and server-side capabilities. Catalog implementations have to keep up.
The areas ahead include new types in Iceberg V3, Register View, multi-table transactions, new planning-related APIs, remote signing and related authentication support, and whatever else the upstream community adds.
Supporting Iceberg REST doesn't mean implementing Create Table and Load Table and calling it done. As Iceberg evolves, catalogs must keep adapting to new table types, transaction models, view capabilities, and the shifting responsibility boundary between client and server. How quickly users get these capabilities depends on how quickly and consistently the project follows upstream. That's why Gravitino treats REST specification alignment as a long-term direction rather than a one-time task.
AI agents and least-privilege data access
The third direction is security. The roadmap includes sensitive-information encryption, data encryption and decryption, secure storage, and mTLS. With ordinary HTTPS, the client verifies the server's identity; with mTLS, the server also verifies the client certificate. Mutual authentication matters more and more for cross-region, cross-cluster, and high-security environments.
The more interesting issue is how roles are used.
Today, if a user has multiple roles, authorization typically considers them all together. Some enterprises want users to specify, for a given task, that only certain roles apply. A user with both a developer role and an administrator role who is just querying a business table has no reason to activate administrator privileges.
This matters even more for agents. An agent may act on behalf of a user, but that doesn't mean it should inherit all of that user's permissions. An agent generating a sales report should receive only the roles required to access sales data.
The question the permission model must answer is changing. It used to be "what permissions does this account have?" It's becoming "which permissions should this particular task be allowed to use?" As agents start operating inside enterprise data systems, moving from user-level permissions toward task-level least privilege becomes essential.
One catalog spanning multiple clouds is only the most visible step
Gravitino set out to solve the problem of too many isolated catalogs. Over time, the problem has expanded well past that.
When data is distributed across clouds, catalogs are distributed across regions, and identity and permissions must stay consistent across service nodes, a unified entry point is table stakes. The hard part is keeping all of those distributed pieces running safely, consistently, and over the long term.
Looking ahead, table optimization, Iceberg REST server-side capabilities, and least-privilege access for agents push the catalog toward a more active role. It no longer only answers "where is this table?" It increasingly participates in how data is accessed, how credentials are vended, how permissions take effect, how operations are traced, and which maintenance actions should happen.
Metadata is no longer just a directory that gets queried. It's becoming a control plane that connects data, identity, permissions, audit, and operational actions.
A single catalog spanning multiple clouds is simply the most visible part of that shift. What's worth watching is how far the boundary of the catalog keeps expanding, and that's something we invite you to explore and shape with the Gravitino community.
Want to go deeper? The code is at github.com/apache/gravitino, the docs at gravitino.apache.org/docs/latest, and Datastrato offers guided walkthroughs, including Enterprise.
