Credential vending
Credential vending gives an engine a short-lived or static credential for the data behind a table, at the moment it loads the table. Storage access is then governed in one place, by Gravitino privileges, across every storage system and engine.
The engine never holds a long-lived storage credential of its own. It asks Gravitino for the table, Gravitino checks the caller's privileges, and the credential provider returns a credential the engine uses to reach the files directly.
Step 2 also decides the scope: a caller entitled to modify the table receives a write credential,
and any other caller a read credential. Static providers such as s3-secret-key skip steps 3 and 4
and return the configured key, which is long-lived and not limited to the table path.
Quick start
The example vends S3 credentials scoped to each table to Spark through the Iceberg REST service (IRC). Create the catalog through the Gravitino REST API:
curl -sS -X POST https://{gravitino_host}/api/metalakes/{metalake}/catalogs \
-H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "iceberg_catalog",
"type": "RELATIONAL",
"provider": "lakehouse-iceberg",
"properties": {
"catalog-backend": "jdbc",
"uri": "jdbc:postgresql://{postgres_host}:5432/{database}",
"jdbc-driver": "org.postgresql.Driver",
"jdbc-user": "{jdbc_user}",
"jdbc-password": "{jdbc_password}",
"jdbc-initialize": "true",
"warehouse": "s3://{bucket_name}/{warehouse_path}",
"io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
"credential-providers": "s3-token",
"s3-access-key-id": "{access_key_id}",
"s3-secret-access-key": "{secret_access_key}",
"s3-region": "{region_name}",
"s3-role-arn": "{role_arn}"
}
}'
Point Spark at the IRC and request vended credentials with the delegation header:
./bin/spark-sql -v \
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.11.0,org.apache.iceberg:iceberg-aws-bundle:1.11.0 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.rest.type=rest \
--conf spark.sql.catalog.rest.uri=https://{iceberg_host}/iceberg/ \
--conf spark.sql.catalog.rest.warehouse=iceberg_catalog \
--conf spark.sql.catalog.rest.rest.auth.type=basic \
--conf spark.sql.catalog.rest.rest.auth.basic.username={user} \
--conf spark.sql.catalog.rest.rest.auth.basic.password={password} \
--conf spark.sql.catalog.rest.header.X-Iceberg-Access-Delegation=vended-credentials
The role in s3-role-arn needs a trust policy and S3 permissions before this works. See s3-token.
For Trino instead of Spark:
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://{iceberg_host}/iceberg/
iceberg.rest-catalog.warehouse=iceberg_catalog
iceberg.rest-catalog.vended-credentials-enabled=true
fs.native-s3.enabled=true
s3.region={region_name}
{iceberg_host} is the IRC's ingress host, set by ingress.iceberg.host, and the warehouse is the Gravitino catalog name. Trino also needs the IRC's authentication settings; for the full Trino setup, see Connect Trino to the IRC.
Supported catalogs
| Catalog type | Vends |
|---|---|
| Fileset | S3, OSS, GCS, ADLS |
| Hive | S3, OSS, GCS, ADLS |
| Iceberg | S3, OSS, GCS, ADLS |
| Glue | S3, static AWS key |
| JDBC | JDBC user and password |
| Paimon | S3, OSS, JDBC user and password, Aliyun DLF key |
S3 is Amazon S3, OSS is Alibaba Cloud OSS, GCS is Google Cloud Storage, and ADLS is Azure Data Lake Storage. The Gravitino Spark, Flink, and Trino connectors consume vended credentials automatically for these catalogs.
Set properties
Credential vending properties go in the catalog's properties map when you create it through the Gravitino REST catalog API, alongside the warehouse location and the other catalog settings.
The Gravitino Iceberg REST Catalog (IRC) is the exception. It can also read a catalog straight from the server configuration, using the same property names prefixed with gravitino.iceberg-rest.. The chart sets the common ones from values under gravitino.icebergRest; the catalog property s3-role-arn, for example, is gravitino.icebergRest.s3.roleArn. Set a key with no chart value under gravitino.additionalConfigItems; see Set other server properties.
gravitino:
icebergRest:
s3:
roleArn: "{role_arn}"
Catalogs defined in the server configuration are not registered in a metalake, so Gravitino access control does not apply to them. Privileges are granted on catalogs in a metalake, and there is nothing to grant them on.
General configurations
| Property | Description | Default value | Required |
|---|---|---|---|
credential-providers | The credential provider types, separated by comma. If omitted, Gravitino infers some providers from the other properties present. | (none) | Yes, unless inferred |
credential-cache-expire-ratio | Ratio of the credential's expiration time when Gravitino removes the credential from the cache. | 0.15 | No |
credential-cache-max-size | Maximum number of credentials cached. | 10000 | No |
Provider values
The values credential-providers accepts:
| Value | Storage | Vends |
|---|---|---|
s3-token | S3 | Temporary STS credentials, scoped to the table path |
aws-irsa | S3 | Credentials from an IAM role for service accounts, for EKS |
s3-secret-key | S3 | The configured static access key and secret |
oss-token | OSS | Temporary STS credentials, scoped to the table path |
oss-secret-key | OSS | The configured static access key and secret |
adls-token | ADLS | A user delegation SAS token |
azure-account-key | ADLS | The configured static storage account key |
gcs-token | GCS | A downscoped access token |
jdbc-user-password | JDBC | The configured JDBC username and password |
aws-secret-key | Glue | The catalog's static aws-access-key-id and secret |
dlf-secret-key | Paimon | The catalog's static Aliyun DLF access key and secret |
A Glue catalog with aws-access-key-id and aws-secret-access-key adds aws-secret-key to its
providers on its own, and a Paimon catalog with DLF keys adds dlf-secret-key. Each value has its own properties, listed in the sections below. To vend for more than one storage type on a catalog, separate values with a comma. Custom providers can be added by implementing CredentialProvider, described under Custom credentials.
When no provider is set
If a catalog does not set credential-providers, Gravitino infers providers from the credential properties present:
| Properties present | Provider enabled |
|---|---|
s3-access-key-id and s3-secret-access-key | s3-secret-key |
oss-access-key-id and oss-secret-access-key | oss-secret-key |
azure-storage-account-name and azure-storage-account-key | azure-account-key |
gcs-service-account-file | gcs-token |
JDBC catalogs additionally infer jdbc-user-password from jdbc-user and jdbc-password.
Four providers have no inference rule and must always be set explicitly: s3-token, oss-token, adls-token, and aws-irsa. In particular, setting s3-role-arn without credential-providers does not enable s3-token. The catalog falls back to s3-secret-key and vends the static access key instead, which is long-lived and not scoped to the table path. Set credential-providers explicitly whenever you want token-based vending.
Providers by storage
The properties, policies and roles for each provider are on the page for its storage:
| Storage | Providers | Page |
|---|---|---|
| Amazon S3 | s3-token, aws-irsa, s3-secret-key | Amazon S3 |
| S3-compatible, such as R2 | s3-secret-key, and on MinIO s3-token | S3-compatible storage |
| Azure Data Lake Storage | adls-token, azure-account-key | Azure Data Lake Storage |
| Google Cloud Storage | gcs-token | Google Cloud Storage |
| Alibaba Cloud OSS | oss-token, oss-secret-key | OSS, below |
OSS
OSS temporary credentials
With the oss-token provider, Gravitino calls Alibaba Cloud STS AssumeRole and returns temporary credentials scoped to the table path.
Also set oss-access-key-id and oss-secret-access-key. Gravitino uses them to call AssumeRole, not to reach data, and they are never sent to the engine.
| Property | Description | Default value | Required |
|---|---|---|---|
oss-access-key-id | The static access key ID used to access OSS data. | (none) | Yes |
oss-secret-access-key | The static secret access key used to access OSS data. | (none) | Yes |
oss-role-arn | The ARN of the role to access the OSS data. | (none) | Yes |
oss-region | The region of the OSS service, like oss-cn-hangzhou, only used when credential-providers is oss-token. | (none) | No |
oss-external-id | The external ID passed to STS when assuming the role. | (none) | No |
oss-token-expire-in-secs | Lifetime of the OSS security token, in seconds. | 3600 | No |
Trust policy on the RAM role
The role in oss-role-arn must allow the oss-access-key-id principal to assume it.
{
"Version": "1",
"Statement": [{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": { "RAM": ["acs:ram::{account_id}:user/{gravitino_user}"] }
}]
}
Permission policy on the RAM role
The vended credentials inherit this policy, narrowed to the table path.
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": ["oss:GetObject", "oss:PutObject", "oss:DeleteObject"],
"Resource": "acs:oss:*:*:{bucket_name}/{warehouse_path}/*"
},
{
"Effect": "Allow",
"Action": ["oss:ListObjects", "oss:GetBucketInfo"],
"Resource": "acs:oss:*:*:{bucket_name}"
}
]
}
OSS static keys
The oss-secret-key provider returns the catalog's configured access key and secret to the client, unchanged.
The key is long-lived, carries whatever permissions its RAM user has, and is not scoped to the table path. Any client that can load a table receives it, and it stays valid after the query finishes. Prefer oss-token. Use oss-secret-key to confirm the vending path works before configuring a role.
| Property | Description | Default value | Required |
|---|---|---|---|
oss-access-key-id | The static access key ID used to access OSS data. | (none) | Yes |
oss-secret-access-key | The static secret access key used to access OSS data. | (none) | Yes |
Request vended credentials
How a client asks depends on which interface it uses.
Over the IRC
Credentials are vended only when the client asks for them. Spark, Flink, and other IRC clients ask with a header:
X-Iceberg-Access-Delegation: vended-credentials
In Spark, set it as a catalog config key:
spark.sql.catalog.{name}.header.X-Iceberg-Access-Delegation=vended-credentials
Trino asks with a catalog property instead, and sends the header for you:
iceberg.rest-catalog.vended-credentials-enabled=true
Over the Gravitino REST catalog API
Hive, Glue, JDBC, Paimon, and Fileset catalogs are reached through the Gravitino REST catalog API, which has no delegation header. Credential properties are hidden from the catalog GET response, so clients fetch them from the Gravitino credential endpoint instead. It works for any metadata object:
GET /api/metalakes/{metalake}/objects/{type}/{full_name}/credentials
For a catalog, {type} is catalog and {full_name} is the catalog name:
GET /api/metalakes/{metalake}/objects/catalog/{catalog}/credentials
The Gravitino Spark and Flink connectors call this for you and inject the returned credentials, so no client configuration is needed.
Read or write scope
Over the IRC, a credential is vended for writing when the caller is entitled to modify the table, and for reading otherwise.
Custom credentials
A custom provider implements org.apache.gravitino.credential.CredentialProvider. The server image
already includes the providers for S3, OSS, Azure and Google Cloud for every catalog type, so only a
custom provider needs a jar added. The chart has no value that puts a jar on the server classpath, so
a custom provider needs a custom server image that contains its jar.
Upgrade from a release earlier than 1.3.0
Sensitive catalog properties such as s3-access-key-id, s3-secret-access-key, jdbc-user, and jdbc-password are excluded from GET /api/metalakes/{metalake}/catalogs/{catalog}. Clients written against earlier releases that read those properties directly lose access to them.
For a zero-downtime migration, set the following in my-values.yaml and re-run the helm upgrade --install command. The property has no dedicated chart value, so it goes under gravitino.additionalConfigItems as a quoted string; see Set other server properties.
gravitino:
additionalConfigItems:
gravitino.catalog.credential.backfillToProperties: "true"
The Gravitino server then re-includes the hidden properties in its catalog GET responses. Turn it off once all clients use the Gravitino credential endpoint, since it exposes credentials in plaintext.