Amazon S3

An Iceberg catalog on Amazon S3 keeps its tables under an s3:// warehouse. The server reaches the bucket with an access key or the pod's AWS identity, and the Iceberg REST service vends engines temporary credentials scoped to each table.

Quick start

The example creates the catalog iceberg_s3 with its warehouse at s3://{bucket}/warehouse, a PostgreSQL backend, and temporary credentials vended through s3-token. It needs an IAM user whose access key the server uses, and a role that user can assume; see s3-token for the two policies on the role.

UI

1. Start the catalog. Open Connect → Catalogs and click Catalog. Enter iceberg_s3 as the Catalog name, choose Relational as the Catalog type, and Apache Iceberg as the provider.

2. Fill in the connection.

   Field	Value
   Catalog backend	jdbc
   URI	jdbc:postgresql://{postgres_host}:5432/{database}
   User, Password	The backend database's user and password
   Storage	S3
   Warehouse	s3://{bucket}/warehouse
   AWS region	The bucket's region, such as us-west-2
   Access key ID, AWS secret key	The IAM user's access key

3. Create it. Click Test connection, which checks the backend, then Create catalog.

4. Vend temporary credentials. Open the catalog, choose the Storage credentials tab, and click Cloud. Choose s3-token as the Provider type, and enter the same Access key ID and Secret access key, the Role ARN of the role to assume, and the Region. Save.

REST

1. Create the catalog.

   Create an Iceberg catalog on S3

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "iceberg_s3",
       "type": "RELATIONAL",
       "provider": "lakehouse-iceberg",
       "properties": {
         "catalog-backend": "jdbc",
         "uri": "jdbc:postgresql://{postgres_host}:5432/{database}",
         "jdbc-driver": "org.postgresql.Driver",
         "jdbc-user": "{jdbc_user}",
         "jdbc-password": "{jdbc_password}",
         "warehouse": "s3://{bucket}/warehouse",
         "s3-region": "{region}",
         "s3-access-key-id": "{access_key_id}",
         "s3-secret-access-key": "{secret_access_key}",
         "s3-role-arn": "arn:aws:iam::{account_id}:role/{role_name}",
         "credential-providers": "s3-token"
       }
     }'

   Over this API, jdbc-driver has to be set; the UI fills it in from the URI.

5. Connect an engine. Point it at the Iceberg REST service with iceberg_s3 as the warehouse, and have it ask for vended credentials. See Spark or Trino.

6. Check it. Create a table, insert a row, and read it back. Test connection never touches the bucket, so this is the first real test of the storage settings.

Catalog properties

Property	Meaning	Default
warehouse	s3://{bucket}/{path}	(none)
s3-region	The bucket's region. Also sent to engines as client.region	(none)
s3-access-key-id	Access key the server uses for the bucket, and for STS with s3-token. Hidden	The pod's AWS identity
s3-secret-access-key	Its secret. Hidden	The pod's AWS identity
s3-endpoint	Leave unset for AWS. For other S3 services, see S3-compatible storage	AWS
s3-path-style-access	true to address buckets by path rather than host name	false

Without an access key, the server uses the AWS identity of its pod, such as an IAM role for the service account on EKS. The chart attaches an IAM role through gravitino.serviceAccount.annotations.

The server's identity needs read, write, delete and list access to the warehouse prefix, because it writes table metadata when engines create and commit tables.

Credentials for engines

Provider	Engines receive	Needs
s3-token	Temporary credentials from STS, scoped to the table	An access key and a role it can assume
aws-irsa	Temporary credentials, scoped to the table, with no access key anywhere	The server on EKS with IAM roles for service accounts
s3-secret-key	The catalog's own access key	An access key
(none)	Nothing; engines bring their own keys

Set one S3 provider per catalog. Naming two makes credential requests fail.

A catalog with an access key and no credential-providers vends that key through s3-secret-key without being asked to. Setting credential-providers, which adding a provider on the Storage credentials tab does, turns that off.

s3-token

The server calls STS AssumeRole with the catalog's access key and returns temporary credentials. It attaches a session policy that narrows the role to the table's location: read and list for a caller who may only read the table, and put and delete as well for one who may modify it.

Property	Meaning	Default
s3-access-key-id, s3-secret-access-key	The key that calls AssumeRole. Required	(none)
s3-role-arn	The role to assume, arn:aws:iam::{account_id}:role/{role_name}. Required	(none)
s3-region	Region for STS	(none)
s3-external-id	External ID, for trust policies that require one	(none)
s3-token-expire-in-secs	Lifetime of the vended credentials, up to the role's maximum session duration	3600
s3-token-service-endpoint	A different STS endpoint	AWS

Engines receive an access key, secret and session token, with their expiry, and an endpoint to refresh them from. The role needs two policies.

The trust policy lets the access key's user assume the role:

Trust policy on the role

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::{account_id}:user/{gravitino_user}" },
    "Action": "sts:AssumeRole"
  }]
}

The permission policy grants access to the warehouse. Vended credentials never exceed it:

Permission policy on the role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::{bucket}/warehouse/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::{bucket}"
    }
  ]
}

aws-irsa

For the server on EKS with IAM roles for service accounts. The server calls STS AssumeRoleWithWebIdentity on s3-role-arn with the pod's web identity token, and attaches the same session policy as s3-token. No access key is involved.

Property	Meaning	Default
s3-role-arn	The role to assume. Required	(none)
s3-region	Region for STS	(none)
s3-token-expire-in-secs	Lifetime of the vended credentials	3600
s3-token-service-endpoint	A different STS endpoint	AWS

The role's trust policy names the cluster's OIDC provider and allows sts:AssumeRoleWithWebIdentity for the server's service account. Its permission policy is the one shown for s3-token.

Attach the identity to the server pod with gravitino.serviceAccount.annotations; EKS then mounts the token and sets AWS_WEB_IDENTITY_TOKEN_FILE in the pod. aws-irsa cannot be added from the Storage credentials tab; set it on the catalog's Advanced tab or over the API.

s3-secret-key

Returns the catalog's access key to engines unchanged. The key is long-lived, carries every permission its user has, and is not limited to the table. Use it to confirm the vending path works before setting up a role.

Troubleshooting

HTTP 400 "No credential provider found for path": the engine asked for vended credentials and the catalog has no S3 provider. Add one, or stop the engine asking.

AccessDenied on AssumeRole: the role's trust policy does not name the access key's user, or, for aws-irsa, the cluster's OIDC provider and service account.

Credentials are vended but reads fail with 403: the role's permission policy does not cover the warehouse prefix.

Writes fail for some users only: those users may read but not modify the table, so they receive read-only credentials. Grant MODIFY_TABLE; see Privileges.

Related

• Object storage
• S3-compatible storage
• Credential vending
• Iceberg catalog