Azure Data Lake Storage

An Iceberg catalog on Azure Data Lake Storage Gen2 keeps its tables under an abfss:// warehouse. The server reaches the storage account with a Microsoft Entra ID service principal or the account key, and the Iceberg REST service vends engines SAS tokens scoped to each table.

Quick start

The example creates the catalog iceberg_adls with its warehouse in the container {container} of the storage account {account}, a PostgreSQL backend, and SAS tokens vended through adls-token. It needs a service principal with the two roles described under adls-token.

UI

1. Start the catalog. Open Connect → Catalogs and click Catalog. Enter iceberg_adls as the Catalog name, choose Relational as the Catalog type, and Apache Iceberg as the provider.

2. Fill in the connection.

   Field	Value
   Catalog backend	jdbc
   URI	jdbc:postgresql://{postgres_host}:5432/{database}
   User, Password	The backend database's user and password
   Storage	ADLS
   Warehouse	abfss://{container}@{account}.dfs.core.windows.net/warehouse
   Storage account name	{account}

   Leave Storage account key empty. A key entered here is vended to engines unchanged, and it grants access to the whole storage account.

3. Create it. Click Test connection, which checks the backend, then Create catalog.

4. Vend SAS tokens. Open the catalog, choose the Storage credentials tab, and click Cloud. Choose adls-token as the Provider type, and enter the Storage account name, Tenant ID, Client ID and Client secret of the service principal. Save.

REST

1. Create the catalog.

   Create an Iceberg catalog on ADLS

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "iceberg_adls",
       "type": "RELATIONAL",
       "provider": "lakehouse-iceberg",
       "properties": {
         "catalog-backend": "jdbc",
         "uri": "jdbc:postgresql://{postgres_host}:5432/{database}",
         "jdbc-driver": "org.postgresql.Driver",
         "jdbc-user": "{jdbc_user}",
         "jdbc-password": "{jdbc_password}",
         "warehouse": "abfss://{container}@{account}.dfs.core.windows.net/warehouse",
         "azure-storage-account-name": "{account}",
         "azure-tenant-id": "{tenant_id}",
         "azure-client-id": "{client_id}",
         "azure-client-secret": "{client_secret}",
         "credential-providers": "adls-token"
       }
     }'

   Over this API, jdbc-driver has to be set; the UI fills it in from the URI.

5. Connect an engine. Point it at the Iceberg REST service with iceberg_adls as the warehouse, and have it ask for vended credentials. See Spark or Trino.

6. Check it. Create a table, insert a row, and read it back. Test connection never touches the storage account.

Catalog properties

Property	Meaning	Default
warehouse	abfss://{container}@{account}.dfs.core.windows.net/{path}	(none)
azure-storage-account-name	The storage account	(none)
azure-storage-account-key	The account key, for the server and for azure-account-key. Hidden	(none)
azure-tenant-id	The service principal's tenant	(none)
azure-client-id	The service principal's client ID	(none)
azure-client-secret	The service principal's secret. Hidden	(none)

The server reaches the storage account with the account key when one is set, and otherwise with the service principal. With neither, it uses the Azure identity of its pod. The abfs, wasb and wasbs schemes work as well as abfss.

Credentials for engines

Provider	Engines receive	Needs
adls-token	A user delegation SAS, scoped to the table	A service principal with two roles
azure-account-key	The storage account key	The account name and key
(none)	Nothing; engines bring their own credentials

A catalog with an account name and key and no credential-providers vends the key through azure-account-key without being asked to. Setting credential-providers, which adding a provider on the Storage credentials tab does, turns that off.

adls-token

The server requests a user delegation key from the storage account with the service principal, and signs a user delegation SAS for the table's directory: read and list for a caller who may only read the table, and write, create and delete as well for one who may modify it.

Property	Meaning	Default
azure-storage-account-name	The storage account. Required	(none)
azure-tenant-id	Required	(none)
azure-client-id	Required	(none)
azure-client-secret	Required	(none)
adls-token-expire-in-secs	Lifetime of the SAS	3600

The service principal needs two role assignments:

• Storage Blob Delegator on the storage account, to request the user delegation key.
• Storage Blob Data Contributor on the container or the warehouse path, which sets what the SAS can do. Without it the SAS is issued but grants nothing.

In this release adls-token cannot vend for a table that sets write.data.path or write.metadata.path, even to a directory inside the table's location; the request fails with HTTP 400. This is a defect in our implementation, tracked as gravitino-enterprise #2623.

azure-account-key

Returns the storage account key to engines unchanged. The key opens every container in the account and does not expire. Use it to confirm the vending path works before setting up a service principal. It cannot be added from the Storage credentials tab; it is enabled by entering the key when creating the catalog, or set on the Advanced tab or over the API.

Troubleshooting

HTTP 400 "No credential provider found for path": the engine asked for vended credentials and the catalog has no ADLS provider.

Requesting the SAS fails with 403: the service principal lacks Storage Blob Delegator on the storage account.

The SAS is issued but reads fail with 403: the service principal lacks a data role on the container.

Related

• Object storage
• Credential vending
• Iceberg catalog