Skip to main content

Connect Snowflake to Iceberg REST

Snowflake reads and writes Iceberg tables managed by Datastrato Enterprise through a Snowflake catalog integration pointed at the Iceberg REST service. Gravitino authorizes every request Snowflake makes and vends the storage credentials Snowflake uses to reach the data files, so no Snowflake external volume is needed unless the tables are shared onward to other engines. No Gravitino engine connector is involved; the engine connectors are only for the Gravitino REST interface.

A catalog integration exposes tables to Snowflake in one of two ways:

  • A catalog-linked database (CLD) mirrors a whole Gravitino catalog. Snowflake discovers its schemas and tables, keeps them in sync, and can write to them.
  • Individually registered tables expose one table at a time, read-only.

Quick start​

The example signs in with OAuth client credentials, links the Gravitino catalog sales_lake as the Snowflake database sales_lake_db, and reads its table q4.leads. The token endpoint must be https://.

Link a Gravitino catalog and query a table
CREATE OR REPLACE CATALOG INTEGRATION gravitino_iceberg
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
REST_CONFIG = (
CATALOG_URI = 'https://{iceberg_host}/iceberg'
CATALOG_NAME = 'sales_lake'
ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
)
REST_AUTHENTICATION = (
TYPE = OAUTH
OAUTH_TOKEN_URI = '{token_endpoint_uri}'
OAUTH_CLIENT_ID = '{client_id}'
OAUTH_CLIENT_SECRET = '{client_secret}'
OAUTH_ALLOWED_SCOPES = ('{scope}')
)
ENABLED = TRUE;

SELECT SYSTEM$VERIFY_CATALOG_INTEGRATION('gravitino_iceberg');

CREATE DATABASE sales_lake_db
LINKED_CATALOG = ( CATALOG = 'gravitino_iceberg' );

SELECT SYSTEM$CATALOG_LINK_STATUS('sales_lake_db');

SELECT * FROM sales_lake_db."q4"."leads" LIMIT 10;

The catalog must vend credentials for the read to reach storage; see Credential vending.

Prerequisites​

  • Iceberg REST service reachable from Snowflake over HTTPS. Snowflake connects from the public internet, so the service needs a public hostname and a certificate from a public authority. With the Helm chart, publish it through ingress.iceberg with TLS enabled, as described in Client access.
  • Table data in Amazon S3, Google Cloud Storage or Azure Storage. Snowflake does not accept vended credentials for S3-compatible stores.
  • Credential vending on the catalog. The Gravitino catalog Snowflake reads must vend credentials, for example credential-providers = s3-token with s3-role-arn set. See Credential vending.
  • An OAuth identity provider with an HTTPS token endpoint. Snowflake accepts only https:// token URIs. The Gravitino server must accept tokens from the same provider through the oauth authenticator, so the token's audience must match gravitino.authenticator.oauth.serviceAudience.
  • A Gravitino user for the Snowflake service account, with privileges. Gravitino maps the token to a user through gravitino.authenticator.oauth.principalFields. The user appears under Authorize after Snowflake's first request, which Gravitino refuses until a role grants access. Add the user to a role with USE_CATALOG, USE_SCHEMA and SELECT_TABLE on what Snowflake reads, plus MODIFY_TABLE on what it writes.
  • A Snowflake role that can create integrations and databases (ACCOUNTADMIN by default) and a running warehouse.

For best read performance, run the Snowflake account in the same cloud and region as the table data.

Create the catalog integration​

The CREATE CATALOG INTEGRATION statement in the Quick start is the complete form. Its settings:

SettingValue
CATALOG_URIThe Iceberg REST service base URL, ending in /iceberg. Snowflake appends /v1/config itself.
CATALOG_NAMEThe name of the Gravitino catalog. Gravitino returns it as the prefix for every later request.
CATALOG_NAMESPACEOptional. The default schema for tables registered individually through this integration.
ACCESS_DELEGATION_MODEVENDED_CREDENTIALS, so Snowflake reads data with the scoped credentials Gravitino returns for each table.
OAUTH_ALLOWED_SCOPESThe scopes your identity provider requires for a client-credentials token.
REFRESH_INTERVAL_SECONDSOptional. How often Snowflake polls Gravitino for table changes, 30 seconds by default.

REST_AUTHENTICATION also accepts TYPE = BEARER with BEARER_TOKEN set to a token from your identity provider. Gravitino validates a bearer token exactly as it validates one Snowflake obtains through OAuth, but the integration stops working when the token expires. Replace it before then:

Replace an expiring bearer token
ALTER CATALOG INTEGRATION gravitino_iceberg SET REST_AUTHENTICATION = (
TYPE = BEARER
BEARER_TOKEN = '{new_token}'
);
Check the connection
SELECT SYSTEM$VERIFY_CATALOG_INTEGRATION('gravitino_iceberg');

A catalog-linked database turns each namespace of the Gravitino catalog into a Snowflake schema and each Iceberg table into a Snowflake Iceberg table, and picks up tables created later without any further statements.

Create and check a catalog-linked database
CREATE DATABASE {database}
LINKED_CATALOG = ( CATALOG = 'gravitino_iceberg' );

SELECT SYSTEM$CATALOG_LINK_STATUS('{database}');
SHOW SCHEMAS IN DATABASE {database};
SHOW ICEBERG TABLES IN DATABASE {database};

Snowflake folds unquoted identifiers to uppercase, so quote the lowercase schema, table and column names that come from Gravitino:

Query a linked table
SELECT * FROM {database}."{schema}"."{table}" WHERE "{column}" = 4;

Snowflake discovers new schemas and tables on the database's SYNC_INTERVAL_SECONDS and picks up new snapshots of tables it already knows on the integration's REFRESH_INTERVAL_SECONDS. Set either with ALTER DATABASE ... UPDATE LINKED_CATALOG or ALTER CATALOG INTEGRATION.

Writes​

Writes through a catalog-linked database commit to Gravitino, so other engines see them on their next read:

Write through a linked table
INSERT INTO {database}."{schema}"."{table}" VALUES (4, 'fresh');

The credentials Gravitino vends follow the Snowflake service account's privileges. Without MODIFY_TABLE on the table, Gravitino vends read-only storage credentials and the write fails when Snowflake writes the data file:

Error when the Snowflake service account lacks MODIFY_TABLE
Failed to access remote file: access denied. Please check your credentials

A grant in Gravitino applies to the next statement Snowflake runs.

Register and query a single table​

To expose one table without linking the catalog, register it as a Snowflake Iceberg table that names the integration and the table's name in the Gravitino catalog. Set CATALOG_NAMESPACE on the integration to the table's schema.

Register and query a table
CREATE OR REPLACE ICEBERG TABLE {database}.public.{table}
CATALOG = 'gravitino_iceberg'
CATALOG_TABLE_NAME = '{table}';

SELECT * FROM {database}.public.{table};

Access control​

Snowflake does not pass its users' identities to Gravitino. Every request from the integration reaches Gravitino as the one Snowflake service account, so access control has two layers:

LayerGrant toControls
GravitinoThe Snowflake service accountEverything Snowflake can reach in the catalog, for all of its users together
SnowflakeIndividual Snowflake rolesWhich Snowflake users see which schemas and tables of the linked database

Grant the service account privileges in Gravitino under Authorize, covering everything any Snowflake user needs, and grant individual access with Snowflake grants on the linked database. Gravitino's privileges for individual people do not apply to queries that come through Snowflake.

When Gravitino denies the service account access to a table, including a table Snowflake has already registered, the query fails with Gravitino's message, for example:

Error when Gravitino denies access
Resource on the REST endpoint of catalog integration GRAVITINO_ICEBERG is forbidden due to error:
Forbidden: User 'svc-snowflake' is not authorized to load table 'acme.lakehouse.sales.customers'.

Access returns as soon as the grant is restored.

Share linked tables with other engines through Snowflake​

Snowflake Horizon Catalog serves the tables of a catalog-linked database to outside engines, and BigQuery reaches Gravitino tables that way. Horizon passes those engines storage credentials only from a Snowflake external volume, never credentials Gravitino vended, so a catalog-linked database meant for other engines uses an integration with ACCESS_DELEGATION_MODE = EXTERNAL_VOLUME_CREDENTIALS and a database created with EXTERNAL_VOLUME. The full setup is in Connect BigQuery through Snowflake.

Limitations​

  • Iceberg views are not synced. A catalog-linked database exposes the catalog's tables only, even for views that carry a Snowflake SQL representation. Define the views in Snowflake over the linked tables instead.
  • Individually registered tables are read-only. Use a catalog-linked database to write.
  • Iceberg catalogs only. Catalogs of other types are not served by the Iceberg REST service.
  • The OAuth token endpoint must be https://.