Object storage
An Iceberg catalog keeps its tables' data and metadata files in object storage. The catalog's
warehouse names where, and the catalog's storage properties tell the server how to reach it. The
Iceberg REST service then gives engines what they need to read and
write the same files: the endpoint and region, and, when they ask for them, credentials scoped to
each table.
| Storage | Warehouse URI | Page |
|---|---|---|
| Amazon S3 | s3://{bucket}/{path} | Amazon S3 |
| Azure Data Lake Storage Gen2 | abfss://{container}@{account}.dfs.core.windows.net/{path} | Azure Data Lake Storage |
| Google Cloud Storage | gs://{bucket}/{path} | Google Cloud Storage |
| Cloudflare R2, MinIO, and other S3-compatible | s3://{bucket}/{path}, with an endpoint | S3-compatible storage |
| Alibaba Cloud OSS | oss://{bucket}/{path} | Credential vending |
The server image includes the storage libraries for all of these, so nothing has to be downloaded or added to the image.
Two sets of credentials
Every catalog on object storage involves two identities:
- The server's. The server writes table metadata itself when engines create and commit tables through the Iceberg REST service. It uses the catalog's storage properties, or the cloud identity of the server pod when those are not set.
- The engine's. Engines read and write data files directly. They either bring their own credentials or ask the Iceberg REST service to vend them, as described in Credential vending.
Vended credentials are the better choice: engines hold no long-lived storage key, and a caller who may only read a table gets credentials that can only read it. Each storage page lists the providers it supports.
How engines reach the storage
The Iceberg REST service tells engines how to reach a catalog's storage in two places:
- In its configuration response, the endpoint, region and path-style setting of the catalog, when they are set. Engines that bring their own keys use these.
- In each table it loads for an engine that sends
X-Iceberg-Access-Delegation: vended-credentials, the same settings plus the vended credentials for that table.
An engine that asks for vended credentials from a catalog with no provider for the table's storage gets an error, HTTP 400. Engines that do not ask are unaffected.
Checking a catalog
Test connection on the Create catalog form checks only that the server can reach the catalog's backend. It does not touch the bucket. To check storage, create a table from an engine, insert a row, and read it back. When you add a credential provider on the catalog's Storage credentials tab, the Test connection button in that drawer checks that the provider can issue a credential for the warehouse; it does not read or write an object either.