One Table, Many Engines | Iceberg in Practice #2

August 2026

One of the most appealing benefits of Apache Iceberg is engine freedom. Your tables stop living inside a proprietary black box with a renewal date and become an open format any engine can work with. You choose the technology to fit the use case, instead of bending every use case to fit whatever you're licensed for.

Nobody needs convincing on multi-engine; they need to reduce the complexity. Every additional engine used to mean another catalog integration, another set of JARs, another config file that drifts out of sync, and another way for two writers to corrupt each other's commits.

That tax is gone now. Here's a working setup where Spark, Trino, Flink, and DuckDB all operate on the same Iceberg tables, safely, with a few lines of config each.

Why you'd actually want four engines

This isn't tool collecting. Each engine earns its slot:

Spark is still the workhorse for heavy batch ETL and backfills. When you need to rewrite a year of data, you want Spark's shuffle machinery.

Trino is what your analysts and BI tools should be hitting: low-latency, high-concurrency interactive SQL that doesn't spin up a job to answer a question.

Flink owns the streaming lane. CDC ingestion and continuous upserts into Iceberg tables, so your lakehouse stops being a day behind your OLTP databases.

DuckDB is the one people keep sleeping on. It runs on your laptop, in CI, even in the browser via WASM. For development, debugging, and "let me just look at this table real quick," nothing beats a local engine that can attach straight to your catalog. No cluster, no wait.

One table, four access patterns. The question is what makes that safe.

The prerequisite: one referee

Quick recap from part 1: an Iceberg catalog stores the pointer to each table's current metadata and swaps it atomically on commit. That atomic swap is the transaction. The Iceberg REST Catalog (IRC) spec turns that into a standard HTTP contract, and its commit protocol (requirements + updates, validated server-side) is what lets writers that know nothing about each other share tables without coordination.

So the architecture is one catalog service in the middle, four engines around it:

One Iceberg REST Catalog endpoint referees commits from Spark, Flink, Trino, and DuckDB: batch, streaming, interactive, and local access patterns against the same table.

Standing up the catalog

For this walkthrough, the IRC endpoint is Apache Gravitino's Iceberg REST service, running in what Gravitino calls dynamic mode: the REST service runs inside the main Gravitino server and serves the Iceberg catalogs Gravitino manages. That last part matters. If the catalog is going to be your commit referee and your policy enforcement point, you want the mode where the referee actually knows about all of your catalogs.

Enable the service in gravitino.conf:

# gravitino.conf (excerpt)
gravitino.auxService.names = iceberg-rest
gravitino.iceberg-rest.classpath = iceberg-rest-server/libs, iceberg-rest-server/conf
gravitino.iceberg-rest.host = 0.0.0.0
gravitino.iceberg-rest.httpPort = 9001
gravitino.iceberg-rest.catalog-config-provider = dynamic-config-provider
gravitino.iceberg-rest.gravitino-metalake = demo_metalake
gravitino.iceberg-rest.default-catalog-name = lake

Then create a metalake and the catalog through Gravitino's API. The IRC endpoint picks the catalog up immediately, no restart:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" -H "Content-Type: application/json" \
  -d '{"name": "demo_metalake", "comment": "multi-engine demo"}' \
  http://gravitino:8090/api/metalakes

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" -H "Content-Type: application/json" \
  -d '{
    "name": "lake",
    "type": "RELATIONAL",
    "provider": "lakehouse-iceberg",
    "properties": {
      "catalog-backend": "jdbc",
      "uri": "jdbc:postgresql://postgres:5432/iceberg",
      "warehouse": "s3://lakehouse/warehouse",
      "jdbc-driver": "org.postgresql.Driver",
      "jdbc-user": "iceberg",
      "jdbc-password": "iceberg",
      "jdbc-initialize": "true",
      "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
      "s3-endpoint": "http://minio:9000",
      "s3-access-key-id": "minioadmin",
      "s3-secret-access-key": "minioadmin",
      "s3-region": "us-east-1",
      "s3-path-style-access": "true",
      "credential-providers": "s3-token",
      "s3-role-arn": "arn:aws:iam::000000000000:role/gravitino",
      "s3-token-service-endpoint": "http://minio:9000"
    }
  }' \
  http://gravitino:8090/api/metalakes/demo_metalake/catalogs

Four details will save you a bad afternoon: two in the POST, two in the deployment.

• "type" is RELATIONAL, not ICEBERG. The type field is Gravitino's entity kind. What actually selects Iceberg is "provider": "lakehouse-iceberg".
• Always set credential-providers explicitly. A catalog holding both JDBC and S3 credentials without a declared provider works fine — right up until the first client requests vended credentials, then fails with a 406 ambiguity error. Declare it at creation and that failure mode never exists.

Two deployment footnotes from our testing:

• The main server's entity store needs its schema applied once before first boot. The image ships scripts/postgresql/schema-1.3.0-postgresql.sql; there's no jdbc-initialize equivalent for it. Miss it, and the server crashes silently. The real error only shows up in its internal log file.
• If you bind-mount the conf file, set SKIP_CONFIG_REWRITE=true, or the entrypoint's config rewrite will fight the mount.

The demo: four engines, one table

Everything in this walkthrough is runnable. The full stack, compose file, configs, and per-engine scripts are in the series repo at github.com/datastrato/blog-iceberg-in-practice, and every number in this post came out of it.

The IRC endpoint lives at http://catalog:9001/iceberg/. One convention to know before the configs make sense: against Gravitino's IRC, the client-side warehouse value is the catalog's name, not a storage path. The storage location was configured in Gravitino when the catalog was created.

If your muscle memory (or Iceberg's own docs) tells you to put an s3:// path in the client's warehouse property, you'll get a confusing NoSuchCatalogException 404, because the server treats that string as a catalog-name lookup. Set it to lake, and everything lines up.

Spark: create and load

spark.sql.catalog.lake                      = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lake.type                 = rest
spark.sql.catalog.lake.uri                  = http://catalog:9001/iceberg/
spark.sql.catalog.lake.warehouse            = lake
spark.sql.catalog.lake.io-impl              = org.apache.iceberg.aws.s3.S3FileIO
# MinIO-specific (skip on real S3):
spark.sql.catalog.lake.s3.endpoint          = http://minio:9000
spark.sql.catalog.lake.s3.path-style-access = true

CREATE NAMESPACE lake.demo;

CREATE TABLE lake.demo.orders (
  order_id BIGINT, customer_id BIGINT,
  amount DECIMAL(10,2), order_ts TIMESTAMP
) USING iceberg;

INSERT INTO lake.demo.orders SELECT * FROM staging_orders;  -- 5,000 rows

(Gravitino 1.3 runs the REST spec's strict mode by default, so create the namespace before the table; operations against a namespace that doesn't exist return 404 rather than silently succeeding.)

Flink: stream into the same table

CREATE CATALOG lake WITH (
  'type'                 = 'iceberg',
  'catalog-type'         = 'rest',
  'uri'                  = 'http://catalog:9001/iceberg/',
  'warehouse'            = 'lake',
  'io-impl'              = 'org.apache.iceberg.aws.s3.S3FileIO',
  -- MinIO-specific (skip on real S3):
  's3.endpoint'          = 'http://minio:9000',
  's3.path-style-access' = 'true'
);

INSERT INTO lake.demo.orders
SELECT order_id, customer_id, amount, order_ts
FROM kafka_orders_cdc;   -- continuous

One classpath note that isn't in any three-line snippet: Iceberg's Flink connector needs Hadoop classes even for a pure S3/REST setup (it builds a Hadoop Configuration for every catalog loader). The two official shaded jars, hadoop-client-api and hadoop-client-runtime, dropped flat into Flink's lib/, cover it.

Flink is now committing small snapshots every checkpoint interval while Spark's batch job runs. Nobody coordinates. The catalog referees. In our test runs, overlapping Spark and Flink writes produced an unbroken four-snapshot chain with zero lost commits.

Trino: query it live

connector.name                 = iceberg
iceberg.catalog.type           = rest
iceberg.rest-catalog.uri       = http://catalog:9001/iceberg/
iceberg.rest-catalog.warehouse = lake
# Native S3 must be enabled explicitly; legacy hive.s3.* keys no longer
# exist, and without this Trino has no handler for s3:// paths at all.
fs.native-s3.enabled           = true
s3.region                      = us-east-1
# MinIO-specific (skip on real S3):
s3.endpoint                    = http://minio:9000
s3.path-style-access           = true
s3.aws-access-key              = minioadmin
s3.aws-secret-key              = minioadmin

SELECT date_trunc('hour', order_ts) AS h, sum(amount)
FROM lake.demo.orders
WHERE order_ts > current_timestamp - INTERVAL '1' DAY
GROUP BY 1 ORDER BY 1 DESC;

Every query reads a consistent snapshot. Flink can commit mid-query and Trino doesn't care; it's still reading the snapshot it started with. In the test runs, Trino read back the combined total from all four writers, exactly once each.

DuckDB: the same table, from your laptop

This is the one that changes daily workflows. DuckDB's iceberg extension attaches an entire REST catalog like it's a local database:

INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs;

ATTACH 'lake' AS lake (
    TYPE iceberg,
    ENDPOINT 'http://catalog:9001/iceberg/',
    AUTHORIZATION_TYPE 'none',
    ACCESS_DELEGATION_MODE 'vended_credentials'
);

SELECT count(*), max(order_ts) FROM lake.demo.orders;

Two of those settings matter more than they look.

The first, AUTHORIZATION_TYPE 'none', has to be spelled out. DuckDB assumes an Iceberg catalog uses OAuth2 and will refuse to connect without credentials, even when the catalog has no auth at all. Setting it to 'none' tells DuckDB to just connect.

The second, ACCESS_DELEGATION_MODE 'vended_credentials' (note the underscore), is the interesting one. With it, DuckDB never needs S3 credentials of its own. When a query runs, the catalog hands it short-lived credentials scoped to just the tables being read. In our test, a completely fresh DuckDB process with zero secrets configured read all 9,000 rows. No cluster, no JVM, no keys.

Two footnotes. The extension is still marked experimental, and write support is newer than read support, so whether laptops should write to production tables is a policy question, not just a technical one. And since DuckDB now runs Iceberg in the browser via WASM, this same pattern works in a web page. We're experimenting with an in-browser version of this exact demo.

What "without config sprawl" actually means

Count what we just did NOT do: no Hive Metastore thrift URIs in four places, no engine-specific catalog implementations, no JAR version matrix, and no storage credentials sprayed across engine configs. Each engine got the same three facts: the catalog speaks REST, here's the URI, here's how I authenticate.

When you rotate storage credentials or move the warehouse, you change it in one place: on the catalog.

The honest caveats

Multi-engine is worth it, but it also changes how you operate. Three things are worth planning for before you commit.

Feature support varies by engine. Iceberg v3 features (deletion vectors, row lineage) reached engines at different speeds. Before you adopt a table feature, check that every engine touching that table supports it. The lowest common denominator is a real constraint.

Concurrent writers create maintenance pressure. Flink committing every checkpoint means lots of small files and lots of snapshots. That's normal, and it's exactly why table maintenance (compaction, snapshot expiry) stops being optional in a multi-engine world. That's pillar #4 of this series, and it deserves its own post.

The catalog is now your single point of truth, and of failure. Run it like tier-one infrastructure, because it is. And choose it carefully, because your policies and metadata live in it and don't port between catalogs easily. That decision framework is pillar #3.

The takeaway

Three things to remember, and one to do:

Engine freedom is now a config change, not an integration project. Four engines joined one table in this post, each with a handful of lines pointed at the same REST endpoint. The N×M matrix from part 1 is genuinely gone.

The catalog referees everything, so run it accordingly. Concurrent writers, snapshot isolation, and credential vending all hang on that one service. Deploy it in the mode where governance actually applies, and treat it like tier-one infrastructure.

Multi-engine amplifies maintenance. More writers means more snapshots and smaller files. Budget for compaction and snapshot expiry from day one, not after the first slow query.

The thing to do: clone the series repo and run this stack, or point DuckDB at your existing catalog, read-only, and see how much faster your team's "quick look at the data" loop gets. That's the gateway drug.

This is pillar #2 of Iceberg in Practice, an educational series on running Apache Iceberg well: catalogs, engines, maintenance, federation, and AI access. Pillar #1 covered the Iceberg REST Catalog, actually explained. Next up: catalog choice, the decision that outlives your engines.

Apache and the names of Apache projects referenced here are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other trademarks, product names, and company names are the property of their respective owners.