Iceberg tables
Apache Iceberg is an open table format: a table is a set of data files plus metadata files that record its schema, partitioning and snapshots. Datastrato Enterprise is the catalog for Iceberg tables. It does not read or write table data itself; engines do, against the same files, through the Iceberg REST service or the Gravitino connectors.
What the catalog does
An Iceberg table has three layers, and the catalog owns the top one:
| Layer | Holds | Lives in |
|---|---|---|
| Catalog | The table's name and a pointer to its current metadata file | The catalog's backend, such as a JDBC database |
| Metadata | Schema, partition spec, snapshots and the manifests of each | Metadata files in object storage |
| Data | The rows | Data files, usually Parquet, in object storage |
On top of the pointer, Datastrato Enterprise decides who may reach each table, vends storage credentials scoped to it, applies tags and policies, and compacts it on request. Everything about the table's contents, from time travel to row-level deletes, is the format's own and works the same in every engine that reads it.
Commits
Every change to a table, from an insert to a schema change, ends in a commit: the engine writes new
metadata and asks the catalog to move the pointer to it. The catalog accepts the commit only if the
table is still in the state the engine started from. When another commit got there first, the
Iceberg REST service rejects the late one with 409, and the engine retries against the new state.
Concurrent writers therefore never overwrite each other.
Each commit applies to one table. Changes to several tables cannot be committed together.
Format versions
| Version | Status in this release |
|---|---|
1 | Supported |
2 | Supported. The default for new tables |
3 | Supported. Needed for v3 types such as variant |
4 | Can be created, but the Iceberg spec for it is not final |
The version is set with the format-version table property when the table is created; see
Iceberg catalog. An engine can only read a table
whose version its own Iceberg library supports.
What runs in engines
These operations belong to the engine, not the catalog. Run them in Spark, Trino or another engine connected to the table:
- Time travel and reading a past snapshot.
- Rolling a table back to an earlier snapshot.
- Expiring old snapshots and removing orphan files.
- Querying metadata tables such as
snapshots,historyandfiles.
Datastrato Enterprise compacts small files and delete files when you run table maintenance; see Table maintenance. It does not expire snapshots or remove orphan files, so schedule those in an engine.
Dropping and purging
Dropping a table removes it from the catalog and leaves its files in storage. Purging removes the files as well. See Tables and views.