S3-compatible storage

Storage that speaks the S3 API, such as Cloudflare R2 and MinIO, is set up as S3 with three differences: the catalog names the service's endpoint, the region is whatever the service expects, and the choice of credential providers depends on whether the service offers AWS-style STS. Everything else on Amazon S3 applies.

Quick start: Cloudflare R2

The example creates the catalog iceberg_r2 with its warehouse in the R2 bucket {bucket} and a PostgreSQL backend.

1. Create an R2 API token. In the Cloudflare dashboard, create an R2 API token with Object Read & Write permission, limited to the bucket. Note its Access Key ID and Secret Access Key, and your account ID, which is part of the endpoint https://{account_id}.r2.cloudflarestorage.com.

UI

2. Start the catalog. Open Connect → Catalogs and click Catalog. Enter iceberg_r2 as the Catalog name, choose Relational as the Catalog type, and Apache Iceberg as the provider.

3. Fill in the connection. There is no separate choice for S3-compatible storage; choose S3 and fill in the endpoint.

   Field	Value
   Catalog backend	jdbc
   URI	jdbc:postgresql://{postgres_host}:5432/{database}
   User, Password	The backend database's user and password
   Storage	S3
   Warehouse	s3://{bucket}/warehouse
   S3 endpoint	https://{account_id}.r2.cloudflarestorage.com
   AWS region	auto
   Access key ID, AWS secret key	The R2 token's keys
   Path-style access	On

4. Create it. Click Test connection, which checks the backend, then Create catalog. With the keys set, the catalog vends them to engines that ask, through s3-secret-key.

REST

2. Create the catalog.

   Create an Iceberg catalog on Cloudflare R2

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "iceberg_r2",
       "type": "RELATIONAL",
       "provider": "lakehouse-iceberg",
       "properties": {
         "catalog-backend": "jdbc",
         "uri": "jdbc:postgresql://{postgres_host}:5432/{database}",
         "jdbc-driver": "org.postgresql.Driver",
         "jdbc-user": "{jdbc_user}",
         "jdbc-password": "{jdbc_password}",
         "warehouse": "s3://{bucket}/warehouse",
         "s3-endpoint": "https://{account_id}.r2.cloudflarestorage.com",
         "s3-region": "auto",
         "s3-path-style-access": "true",
         "s3-access-key-id": "{r2_access_key_id}",
         "s3-secret-access-key": "{r2_secret_access_key}",
         "credential-providers": "s3-secret-key"
       }
     }'

   Over this API, jdbc-driver has to be set; the UI fills it in from the URI.

5. Connect an engine. Point it at the Iceberg REST service with iceberg_r2 as the warehouse. The service passes the endpoint, region and path-style setting to engines, so they need no R2 settings of their own. See Spark or Trino.

6. Check it. Create a table, insert a row, and read it back. Test connection never touches the bucket.

Catalog properties

Property	Meaning
warehouse	s3://{bucket}/{path}. S3-compatible storage always uses the s3 scheme
s3-endpoint	The service's S3 endpoint
s3-region	The region the service expects: auto for R2, the configured region for MinIO, often us-east-1
s3-path-style-access	true for MinIO and most self-hosted services. R2 accepts either
s3-access-key-id, s3-secret-access-key	The service's access key. Hidden

The Iceberg REST service sends s3-endpoint, s3-region and s3-path-style-access to engines in its configuration and with every table it vends credentials for.

Credentials for engines

Service	s3-token	aws-irsa	s3-secret-key
Cloudflare R2	No	No	Yes
MinIO	Untested, see below	Untested, see below	Yes

Cloudflare R2. s3-token calls AWS-style STS AssumeRole, which R2 does not provide, so the only way to vend R2 credentials is s3-secret-key, which hands engines the catalog's key. Limit that key to the warehouse bucket, as the R2 API token above is.

Engines can bring their own R2 keys instead: they leave out the vended-credentials request and use the endpoint, region and path-style setting the service sends. The catalog still needs its own keys for the server, and while they are set, an engine that does ask for vended credentials receives them.

MinIO. MinIO implements STS, so two providers can in principle vend temporary credentials from it. Neither has been tested against MinIO for this release:

• s3-token, with s3-token-service-endpoint set to the MinIO URL. s3-role-arn must still be set.
• aws-irsa, which accepts a MinIO role ARN, arn:minio:..., when s3-token-service-endpoint is set.

s3-secret-key works as it does for R2.

Troubleshooting

Engines cannot resolve a host name that starts with the bucket name: the engine is addressing the bucket by host name. Set s3-path-style-access to true.

Requests fail with a signature or region error: s3-region does not match what the service expects. Use auto for R2.

HTTP 400 "No credential provider found for path": the engine asked for vended credentials and the catalog has no S3 provider. On R2, set credential-providers to s3-secret-key, or stop the engine asking.

Related

• Amazon S3
• Object storage
• Credential vending