S3-compatible storage

Storage that speaks the S3 API, such as Cloudflare R2 and MinIO, is set up as S3 with three differences: the catalog names the service's endpoint, the region is whatever the service expects, and the choice of credential providers depends on whether the service offers AWS-style STS. Everything else on Amazon S3 applies.

Quick start: Cloudflare R2

The example creates the catalog iceberg_r2 with its warehouse in the R2 bucket {bucket} and a PostgreSQL backend.

Before you start, create an R2 API token in the Cloudflare dashboard with Object Read & Write permission, limited to the bucket. Note its Access Key ID and Secret Access Key, and your account ID, which is part of the endpoint https://{account_id}.r2.cloudflarestorage.com.

UI

1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter iceberg_r2 as the Catalog name, choose Relational as the Catalog type, and Apache Iceberg as the provider.

2. Fill in the connection. There is no separate choice for S3-compatible storage; choose S3 and fill in the endpoint.

   Field	Value
   Catalog backend	JDBC
   URI	jdbc:postgresql://{postgres_host}:5432/{database}
   User, Password	The backend database's user and password
   Storage	S3
   Warehouse	s3://{bucket}/warehouse
   S3 endpoint	https://{account_id}.r2.cloudflarestorage.com
   AWS region	auto
   Access key ID, AWS secret key	The R2 token's keys
   Path-style access	On

3. Create it. Click Test connection, which checks the backend, then Create catalog. With the keys set, the catalog vends them to engines that ask, through s3-secret-key.

REST

1. Create the catalog.

   Create an Iceberg catalog on Cloudflare R2

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "iceberg_r2",
       "type": "RELATIONAL",
       "provider": "lakehouse-iceberg",
       "properties": {
         "catalog-backend": "jdbc",
         "uri": "jdbc:postgresql://{postgres_host}:5432/{database}",
         "jdbc-driver": "org.postgresql.Driver",
         "jdbc-user": "{jdbc_user}",
         "jdbc-password": "{jdbc_password}",
         "warehouse": "s3://{bucket}/warehouse",
         "s3-endpoint": "https://{account_id}.r2.cloudflarestorage.com",
         "s3-region": "auto",
         "s3-path-style-access": "true",
         "s3-access-key-id": "{r2_access_key_id}",
         "s3-secret-access-key": "{r2_secret_access_key}",
         "credential-providers": "s3-secret-key"
       }
     }'

   Over this API, jdbc-driver has to be set; the UI fills it in from the URI.

Then connect an engine and check the storage:

• Connect an engine. Point it at the Iceberg REST service with iceberg_r2 as the warehouse. The service passes the endpoint, region and path-style setting to engines, so they need no R2 settings of their own. See Spark or Trino.
• Check it. Create a table, insert a row, and read it back. Test connection never touches the bucket.

Catalog properties

Property	Meaning
warehouse	s3://{bucket}/{path}. S3-compatible storage always uses the s3 scheme
s3-endpoint	The service's S3 endpoint
s3-region	The region the service expects: auto for R2, the configured region for MinIO, often us-east-1
s3-path-style-access	true for MinIO and most self-hosted services. R2 accepts either
s3-access-key-id, s3-secret-access-key	The service's access key. Hidden

The Iceberg REST service sends s3-endpoint, s3-region and s3-path-style-access to engines in its configuration and with every table it vends credentials for.

Credentials for engines

Service	r2-token	s3-token	aws-irsa	s3-secret-key
Cloudflare R2	Yes	No	No	Yes
MinIO	No	Untested, see below	Untested, see below	Yes

Cloudflare R2. R2 has no STS, so s3-token cannot be used. Use r2-token instead. It issues R2 temporary credentials derived from the catalog's R2 API token, scoped to the table's location, read-only unless the caller holds a write privilege on the table, and valid for s3-token-expire-in-secs seconds (3600 by default). It reads these catalog properties:

Property	Meaning
s3-access-key-id	Access key ID of the parent R2 API token, which needs Object Read & Write on the bucket
s3-secret-access-key	Secret access key of the same token
s3-endpoint	The account's R2 endpoint, https://{account_id}.r2.cloudflarestorage.com, or a jurisdiction endpoint such as https://{account_id}.eu.r2.cloudflarestorage.com
s3-token-expire-in-secs	Lifetime of each issued credential, in seconds. Default 3600

Set credential-providers to r2-token. In the UI, open the catalog's Storage credentials tab, click + Cloud, and choose r2-token. The provider has no Test connection button.

s3-secret-key still works, but it hands engines the catalog's own key, which is long-lived and covers the whole bucket. If you use it, limit that key to the warehouse bucket.

Engines can bring their own R2 keys instead: they leave out the vended-credentials request and use the endpoint, region and path-style setting the service sends. The catalog still needs its own keys for the server, and while they are set, an engine that does ask for vended credentials receives them.

MinIO. MinIO implements STS, so two providers can in principle vend temporary credentials from it. Neither has been tested against MinIO for this release:

• s3-token, with s3-token-service-endpoint set to the MinIO URL. s3-role-arn must still be set.
• aws-irsa, which accepts a MinIO role ARN, arn:minio:..., when s3-token-service-endpoint is set.

s3-secret-key works as it does for R2.

Troubleshooting

Engines cannot resolve a host name that starts with the bucket name: the engine is addressing the bucket by host name. Set s3-path-style-access to true.

Requests fail with a signature or region error: s3-region does not match what the service expects. Use auto for R2.

HTTP 400 "No credential provider found for path": the engine asked for vended credentials and the catalog has no S3 provider. On R2, set credential-providers to s3-secret-key, or stop the engine asking.

Related

• Amazon S3
• Object storage
• Credential vending