August 2026
Your query engine shouldn't just read Iceberg tables. Not directly. Before a single data file gets scanned, the engine asks a catalog one question: "What's the current state of this table?" ACID transactions, time travel, safe concurrent writes: everything Iceberg promises hangs on that question getting a consistent answer.
The Iceberg REST Catalog (IRC) is the industry's agreement on how to ask that question. It's the reason you can point Spark, Trino, Flink, DuckDB, and a Python script at the same table and trust what comes back. And it's quietly become the most important interface in the lakehouse stack.
Here's what it actually standardizes, why every engine adopted it, and what it deliberately leaves unsolved.
Why Iceberg needs a catalog at all
An Iceberg table is a tree of files: data files at the bottom, manifests above them, and a single metadata file at the top that describes the table's current state, its schema, partitioning, and snapshot history.
The catalog's job is deceptively small: it stores the pointer to that current metadata file, and it swaps that pointer atomically when the table changes.
That atomic swap is where Iceberg's ACID guarantees actually live. Two writers commit at the same time? The catalog accepts one, rejects the other, and the loser retries against the new state. No catalog, no atomicity. Your "table" is just a pile of Parquet files with opinions.
So the catalog was always load-bearing. The problem was how engines talked to it.
The problem IRC solved: the N×M integration matrix
Before the REST spec, every catalog implementation needed a native client inside every engine. Hive Metastore spoke Thrift. JDBC catalogs needed database drivers. Glue needed the AWS SDK. Nessie had its own client. Each of those clients had to be independently embedded in Spark, Trino, Flink, and everything else, at compatible versions, with compatible JARs.
That's an N×M compatibility matrix, and anyone who has debugged a NoSuchMethodError at 5pm on a Friday knows exactly what living inside that matrix feels like. Worse, it locked catalog logic into JVM land. If your language couldn't load the client library, your tool couldn't safely touch the table.
The REST spec collapses the matrix. The catalog becomes a service with a standard HTTP API. Engines ship one thin REST client, and any catalog that implements the spec works with any engine that speaks it. N×M becomes N+M.
That's also why the long tail of tools showed up. PyIceberg, DuckDB, and Rust-based engines didn't have to reimplement catalog clients. They just had to speak HTTP.
What the spec actually standardizes
"REST catalog" undersells it. The spec defines four things that matter operationally.
1. The catalog API itself
Namespaces, tables, views: create, load, list, drop, rename, all as HTTP endpoints with defined request and response shapes. Boring. That's the point.
Boring is what makes it universal.
2. The commit protocol
This is the clever part. A client never overwrites table metadata directly. Instead, a commit is a list of requirements ("I expect the current snapshot to be X") plus a list of updates ("add this snapshot, set this property"). The server checks the requirements against reality and applies the updates atomically, or rejects the commit so the client can retry.
Moving that arbitration server-side is what makes multi-engine writes safe. Spark and Flink don't need to coordinate with each other. They both negotiate with the catalog, and the catalog is the single referee. The spec also covers multi-table transactions, so a commit spanning tables can succeed or fail as a unit.
3. Credential vending
Classically, every engine that touched a table needed storage credentials broad enough to read the whole bucket. Security teams love that (said no one).
With credential vending, the engine authenticates to the catalog, and the catalog hands back short-lived, table-scoped storage credentials, just enough to read or write the files for that specific table. Access control moves from "who has bucket keys" to "what did the catalog authorize," which is a policy decision you can actually audit. Implementations vary in how completely they support this, so it's worth verifying for your specific catalog and cloud.
4. Server-side scan planning
The newest piece, and arguably the most consequential. Instead of the engine fetching all manifests and planning its own scan, the catalog can plan the scan server-side and return a filtered list of files. That opens the door to enforcing row filters and column masks at planning time, consistently, for every engine, because the enforcement point is the catalog rather than each engine's goodwill. Engine support is still rolling out across the ecosystem, but the direction is clear: the catalog is becoming the governance chokepoint, in the good sense.
Why adoption tipped
Standards win when the incentives line up on all sides, and here they did.
Engines wanted out of the client-maintenance business: one REST client replaces five native ones. Catalog builders wanted a stable target: implement one spec, inherit every engine. Platform teams wanted decoupling: you can swap or upgrade the catalog service without touching engine deployments. And security teams got the two features they'd been asking for all along, credential vending and a central enforcement point.
The result: essentially every catalog that matters now speaks REST, whether open source (Gravitino, Polaris, Lakekeeper) or managed (S3 Tables, Snowflake Open Catalog, BigLake, and friends). The spec is no longer a bet. It's table stakes, pun fully intended.
What IRC deliberately doesn't solve
Governance doesn't travel. The spec standardizes the interface, not the policy model. Your access rules, ownership metadata, and tags live inside whichever catalog you chose, and there's no standard for moving them between catalogs. The interface is portable; the governance isn't. Choose your catalog like it will outlive your engines, because it will. (That decision framework is pillar #3 of this series.)
Your estate isn't only Iceberg. IRC speaks Iceberg tables, full stop. The Kafka topics, the Postgres tables, the fileset full of training data, the ML models: none of that is addressable through an Iceberg REST endpoint. If your metadata problem spans formats, you need a layer above any single-format catalog. (That's pillar #6.)
Neither of these is a flaw. A spec that tried to standardize governance and every data format would have shipped never. But you should know where the edges are before you architect against it.
A working example: standing up an IRC endpoint with Gravitino
To make this concrete, here's the pattern with Apache Gravitino, an Apache Top-Level Project that includes a spec-compliant Iceberg REST service. (Any spec-compliant catalog follows the same shape; that's the whole point of the spec.)
Gravitino's REST service fronts a backend of your choice (JDBC, Hive Metastore, and others), so you can expose an existing metastore through the standard interface without migrating anything:
# gravitino-iceberg-rest-server config (excerpt)
gravitino.iceberg-rest.catalog-backend = jdbc
gravitino.iceberg-rest.uri = jdbc:postgresql://db:5432/iceberg
gravitino.iceberg-rest.warehouse = s3://lakehouse/warehouse
gravitino.iceberg-rest.credential-providers = s3-token # vend short-lived creds
Then every engine connects the same way, with the same few lines:
# Spark
spark.sql.catalog.lake = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lake.type = rest
spark.sql.catalog.lake.uri = http://gravitino:9001/iceberg/
# PyIceberg: same table, no JVM in sight
from pyiceberg.catalog import load_catalog
catalog = load_catalog("lake", uri="http://gravitino:9001/iceberg/")
table = catalog.load_table("analytics.events")
Trino and Flink are the same story: type = rest, one URI. The same service arbitrates commits from all of them, and with credential vending on, none of them hold long-lived bucket keys.
That's the IRC promise in practice: one table, many engines, one referee.
The takeaway
Three things to remember, and one to do:
- The catalog is where Iceberg's guarantees actually live
- So treat it as tier-one infrastructure, not a config afterthought.
- The REST spec turned catalog access from an N×M JAR-matching exercise into one HTTP contract
- Which is why the whole ecosystem converged on it.
- The spec's edges: policy portability, non-Iceberg estates
- Exactly where your architecture decisions still matter.
The thing to do: check what your current catalog setup actually is. If engines are still talking Thrift to a metastore because that's how it's always been, you now know what the migration path looks like, and why it's worth it.
This is pillar #1 of Iceberg in Practice, an educational series on running Apache Iceberg well.
Iceberg in Practice is an educational series for the people actually running Apache Iceberg: data engineers and architects who have tables in production, or are about to.
Plenty has been written about what Iceberg is. Much less about how to run it well: what the REST catalog actually standardizes, how to point four engines at one table without breaking anything, which catalog decision you'll still be living with in five years, and the unglamorous maintenance work that bites at scale.
That's this series. Each piece stands alone, teaches something you can use this week, and builds toward a complete picture of a well-run open lakehouse: real configs, real tradeoffs, real production lessons.
Next up: one table, many engines, a demo-heavy look at Trino, Spark, Flink, and DuckDB sharing the same tables without config sprawl.
Apache and the names of Apache projects referenced here are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other trademarks, product names, and company names are the property of their respective owners.