Skip to main content

Connect DuckDB to the Iceberg REST catalog

DuckDB connects to the Iceberg REST service through its iceberg extension, with nothing to install on the Gravitino side. One ATTACH makes a Gravitino catalog appear as a DuckDB database, and DuckDB can then read, create, insert into, update and delete from its tables. Storage access comes from vended credentials, so DuckDB needs no keys of its own for the catalog's bucket.

Quick start​

The example signs in with OAuth client credentials and attaches the Gravitino catalog sales_lake.

Attach a Gravitino catalog in DuckDB
INSTALL iceberg;
LOAD iceberg;
INSTALL httpfs;
LOAD httpfs;

CREATE SECRET gravitino (
    TYPE iceberg,
    CLIENT_ID '{client_id}',
    CLIENT_SECRET '{client_secret}',
    OAUTH2_SERVER_URI '{token_endpoint_uri}',
    OAUTH2_SCOPE '{scope}'
);

ATTACH 'sales_lake' AS sales (
    TYPE iceberg,
    ENDPOINT 'https://{iceberg_rest_host}/iceberg',
    SECRET gravitino
);

SHOW ALL TABLES;
SELECT * FROM sales.q4.leads LIMIT 10;

The schemas of the catalog appear as schemas of the sales database, so a table is sales.{schema}.{table}.

Prerequisites​

  • DuckDB 1.5 or later, with the iceberg and httpfs extensions. The steps on this page were run with DuckDB 1.5.
  • The Iceberg REST service reachable from where DuckDB runs. On an install made with the Helm chart, that is the hostname set in ingress.iceberg.host.
  • An identity provider configured for the server, since DuckDB authenticates with OAuth tokens only; see Authentication.
  • For storage access, credential vending configured on the catalog; see Credential vending.

Select the catalog​

The first argument to ATTACH is the name of the Gravitino catalog, and the AS name is what it is called in DuckDB. DuckDB sends the catalog name to the service as the warehouse and receives the catalog's prefix in return, so every later request goes to that catalog. Attach several catalogs under different AS names to work across them in one session.

Attach two catalogs
ATTACH 'sales_lake' AS sales (TYPE iceberg, ENDPOINT 'https://{iceberg_rest_host}/iceberg', SECRET gravitino);
ATTACH 'finance_lake' AS finance (TYPE iceberg, ENDPOINT 'https://{iceberg_rest_host}/iceberg', SECRET gravitino);

An empty first argument, ATTACH '', names no warehouse, and the service then serves its default catalog, if one is set. Naming the catalog is the safer habit.

Authentication​

DuckDB sends a bearer token on every request. It supports two ways to get one.

OAuth client credentials, as in the quick start. DuckDB fetches a token from the identity provider's token endpoint and sends it to the service. It sends the client ID and secret to the token endpoint as an HTTP Authorization: Basic header rather than in the request body, so the client in the identity provider must accept client authentication by Basic header. Keycloak accepts it for any confidential client.

A token you already hold, for example one issued to a user:

Use a pre-issued token
CREATE SECRET gravitino (TYPE iceberg, TOKEN '{access_token}');

DuckDB has no option for Basic credentials on the catalog connection, so a local user cannot connect DuckDB directly. Users and service accounts that use DuckDB need an identity in the identity provider. Every request is authorized as that identity, with the privileges listed under Authentication and access control.

Storage access​

DuckDB asks the service for vended credentials when it loads a table, and the service returns credentials scoped to that table along with the storage settings DuckDB needs to use them, such as the S3 endpoint, region and path-style access for an S3-compatible service. No CREATE SECRET for S3, GCS or Azure is needed for tables in an attached catalog.

DuckDB requests vended credentials by default. Add ACCESS_DELEGATION_MODE 'vended_credentials' to the ATTACH options to state it explicitly. If the catalog has no credential provider, DuckDB receives no credentials, and reads of table data fail unless DuckDB has storage credentials of its own; see Credential vending.

Examples​

Create, write, change and read a table
CREATE SCHEMA sales.q4;
CREATE TABLE sales.q4.orders (id BIGINT, amount DECIMAL(10, 2));
INSERT INTO sales.q4.orders VALUES (1, 9.99), (2, 20.00);
UPDATE sales.q4.orders SET amount = 21.00 WHERE id = 2;
DELETE FROM sales.q4.orders WHERE id = 1;
SELECT * FROM sales.q4.orders;

Each statement that changes data commits a new snapshot through the service, so the change is visible to every other engine on the catalog as soon as the statement returns.

Known issues​

iceberg_snapshots() and iceberg_metadata() fail on an attached table. These table functions read the table's metadata files with DuckDB's own storage settings rather than the vended credentials, so they fail with a 404 or 403 from the storage service. Querying the table itself is unaffected. The behavior is in DuckDB's iceberg extension, not in the Iceberg REST service.

Last updated on