Skip to main content

Connect DuckDB to the Iceberg REST catalog

DuckDB connects to the Iceberg REST service through its iceberg extension, with no Gravitino engine connector to install. One ATTACH makes a Gravitino catalog appear as a DuckDB database, and DuckDB can then read, create, insert into, update and delete from its tables. Storage access comes from vended credentials, so DuckDB needs no keys of its own for the catalog's bucket.

Quick start​

The example signs you in with your own token from the identity provider and attaches the Gravitino catalog sales_lake. Gravitino authorizes every request as you.

Attach a Gravitino catalog in DuckDB
INSTALL iceberg;
LOAD iceberg;
INSTALL httpfs;
LOAD httpfs;

CREATE SECRET gravitino (TYPE iceberg, TOKEN '{token}');

ATTACH 'sales_lake' AS sales (
    TYPE iceberg,
    ENDPOINT 'https://{iceberg_host}/iceberg',
    SECRET gravitino
);

SHOW ALL TABLES;
SELECT * FROM sales.q4.leads LIMIT 10;

The schemas of the catalog appear as schemas of the sales database, so a table is sales.{schema}.{table}.

Prerequisites​

  • DuckDB 1.5 or later, with the iceberg and httpfs extensions. The steps on this page were run with DuckDB 1.5.
  • The Iceberg REST service published and reachable from where DuckDB runs, at the hostname set in ingress.iceberg.host. See Client access.
  • An identity provider configured for the server, since DuckDB authenticates with OAuth tokens only; see Authentication.
  • For storage access, credential vending configured on the catalog; see Credential vending.

Select the catalog​

The first argument to ATTACH is the name of the Gravitino catalog, and the AS name is what it is called in DuckDB. DuckDB sends the catalog name to the service as the warehouse and receives the catalog's prefix in return, so every later request goes to that catalog. Attach several catalogs under different AS names to work across them in one session.

Attach two catalogs
ATTACH 'sales_lake' AS sales (TYPE iceberg, ENDPOINT 'https://{iceberg_host}/iceberg', SECRET gravitino);
ATTACH 'finance_lake' AS finance (TYPE iceberg, ENDPOINT 'https://{iceberg_host}/iceberg', SECRET gravitino);

An empty first argument, ATTACH '', names no warehouse, and the service then serves its default catalog, if one is set. Naming the catalog is the safer habit.

Authentication​

DuckDB authenticates with OAuth only. It sends one bearer token on every request, taken from the iceberg secret the catalog is attached with: either a shared identity, from a client ID and secret or a pre-issued token, or a person's own token. A secret with a person's token needs no client ID or secret. See How to authenticate for the Gravitino side.

DuckDB has no option for Basic credentials on the catalog connection, so a local user cannot connect DuckDB directly. Everyone who uses DuckDB needs an identity in the identity provider. Every request is authorized as that identity, with the privileges listed under Authentication and access control.

Shared identity for DuckDB​

Use a shared identity for scripts and jobs that run as a service account. Every session that signs in with it is authorized as the same account, so grant that account in Gravitino only what those jobs need.

Shared identity for DuckDB

Client ID and client secret​

DuckDB fetches a token from the identity provider's token endpoint and sends it to the service.

Sign in with client credentials
CREATE SECRET gravitino (
    TYPE iceberg,
    CLIENT_ID '{client_id}',
    CLIENT_SECRET '{client_secret}',
    OAUTH2_SERVER_URI '{token_endpoint_uri}',
    OAUTH2_SCOPE '{scope}'
);

DuckDB sends the client ID and secret to the token endpoint as an HTTP Authorization: Basic header rather than in the request body, so the client in the identity provider must accept client authentication by Basic header. Keycloak accepts it for any confidential client.

Pre-issued token​

A token issued to a service account, set in the secret's TOKEN option the same way as a personal token. Nothing renews it, so requests fail once it expires.

Per-user identity​

A person gets a token from the identity provider, outside DuckDB, and puts it in the secret, as in the Quick start. DuckDB sends it as the bearer token on every request, and Gravitino authorizes each request as that person.

Personal token through DuckDB

Sign in with your own token
CREATE SECRET gravitino (TYPE iceberg, TOKEN '{token}');

The token must carry the audience configured for Gravitino and a principal claim naming a member of the metalake. Nothing renews it, so requests fail once it expires; create the secret again with a new token.

Storage access​

DuckDB asks the service for vended credentials when it loads a table, and the service returns credentials scoped to that table along with the storage settings DuckDB needs to use them, such as the S3 endpoint, region and path-style access for an S3-compatible service. No CREATE SECRET for S3, GCS or Azure is needed for tables in an attached catalog.

DuckDB requests vended credentials by default. Add ACCESS_DELEGATION_MODE 'vended_credentials' to the ATTACH options to state it explicitly. If the catalog has no credential provider, DuckDB receives no credentials, and reads of table data fail unless DuckDB has storage credentials of its own; see Credential vending.

Examples​

Create, write, change and read a table
CREATE SCHEMA sales.q4;
CREATE TABLE sales.q4.orders (id BIGINT, amount DECIMAL(10, 2));
INSERT INTO sales.q4.orders VALUES (1, 9.99), (2, 20.00);
UPDATE sales.q4.orders SET amount = 21.00 WHERE id = 2;
DELETE FROM sales.q4.orders WHERE id = 1;
SELECT * FROM sales.q4.orders;

Each statement that changes data commits a new snapshot through the service, so the change is visible to every other engine on the catalog as soon as the statement returns.

Known issues​

iceberg_snapshots() and iceberg_metadata() fail on an attached table. These table functions read the table's metadata files with DuckDB's own storage settings rather than the vended credentials, so they fail with a 404 or 403 from the storage service. Querying the table itself is unaffected. The behavior is in DuckDB's iceberg extension, not in the Iceberg REST service.

Last updated on