Multiple Storage Systems in One Catalog

Overview

A single Gravitino catalog can hold data that lives in several storage systems at once. Tables in Amazon S3, filesets in Google Cloud Storage, and directories on an on-premises HDFS cluster can all sit in the same catalog, and a caller reaches them through one endpoint without knowing where any of them are stored.

Storage systems are not configured one per catalog. A catalog carries a list of credential providers, and Gravitino selects among them for each object by looking at the URI scheme of that object's location. A table at s3://{bucket}/orders is served with S3 credentials and a table at gs://{bucket}/events in the same catalog is served with Google Cloud Storage credentials, decided at request time rather than at catalog creation.

This is what makes a catalog a governance boundary rather than a storage boundary. Tags, policies, and access control apply to everything in the catalog, and the storage system each object happens to sit in is a property of the object.

Quick Start

1. Create a catalog listing every storage system it needs. Set credential-providers to a comma-separated list, then add the properties each provider requires.

export GRAVITINO_URI=http://{gravitino_host}:8090
export METALAKE={metalake_name}

curl -X POST -H "Content-Type: application/json" \
  -d '{
        "name": "mixed_storage",
        "type": "RELATIONAL",
        "provider": "lakehouse-iceberg",
        "properties": {
          "catalog-backend": "jdbc",
          "uri": "jdbc:postgresql://{host}:5432/{database}",
          "warehouse": "s3://{bucket}/{prefix}",
          "credential-providers": "s3-token,gcs-token",
          "s3-role-arn": "arn:aws:iam::{account}:role/{role_name}",
          "s3-region": "{region}",
          "gcs-service-account-file": "/{path}/{service_account}.json"
        }
      }' \
  "${GRAVITINO_URI}/api/metalakes/${METALAKE}/catalogs"

2. Create objects in whichever storage system each one belongs in. No further configuration is needed, because the location determines the credential.

3. Read them through one endpoint. A client that requests credential vending receives credentials for the storage system holding the object it asked for.

Choosing a Credential

Gravitino takes the location of the object being accessed, extracts the URI scheme, and asks each configured provider whether it supports that scheme. The provider that does is the one used.

Because the match is on scheme rather than on catalog configuration, adding a storage system to an existing catalog means adding one provider to credential-providers and its properties. Existing objects are unaffected, since nothing about their resolution changes.

Storage System	Schemes	Credential Providers
Amazon S3 and S3-compatible storage	s3, s3a, s3n	s3-token, s3-secret-key, aws-irsa
Google Cloud Storage	gs, gcs	gcs-token
Azure Data Lake Storage	abfs, abfss, wasb, wasbs	adls-token, azure-account-key
Alibaba Cloud OSS	oss	oss-token, oss-secret-key
HDFS and local file systems	hdfs, file	None, see On-Premises Storage below

Only one provider may claim a given scheme on a catalog. Listing two providers for the same storage system, such as s3-token together with s3-secret-key, fails at request time with an error naming both, so choose the one credential style you want per storage system.

Filesets and Models

A fileset or a model version can span more than one storage system, and Gravitino returns a credential for each system involved. Every configured provider is offered the object's locations, keeps the ones whose scheme it supports, and contributes a credential for those. A fileset with one location in S3 and another in Azure Data Lake Storage therefore comes back with two credentials in a single response.

Locations that no provider claims are dropped rather than reported, so a fileset in an unconfigured storage system returns fewer credentials than it has locations rather than returning an error.

Iceberg Tables

An Iceberg table has a single location, so exactly one credential applies. Gravitino resolves the table location, the write data location, and the write metadata location together, and returns one credential scoped to them.

Requesting vended credentials for a table whose scheme no provider claims fails with a service unavailable error rather than returning the table without credentials.

On-Premises Storage

On-premises data belongs in the same catalog as cloud data and is reached the same way, but it is not served with vended credentials. Tables on HDFS and on local file systems are recognized by their scheme and excluded from vending, because there is no short-lived credential to hand out for them. A location with no scheme at all is treated as a local path.

Access to those tables is authenticated by the engine's own configuration, which for a secured cluster means Kerberos. Configure it on the catalog with the Kerberos properties described in How to Use Kerberos, and place the cluster's core-site.xml and hdfs-site.xml where the catalog can load them.

The practical consequence is that a mixed catalog has two access paths rather than one. Cloud objects are reached with credentials Gravitino vends per request, and on-premises objects are reached with the credentials the engine already holds. Governance is unaffected, since both paths go through the same catalog and the same access control.

Limits

The scheme is the only input to the decision, so two accounts in the same cloud cannot be separated within one catalog. A catalog reaching two different S3 accounts needs a credential provider that can serve both, such as a role that can be assumed across them, or the buckets need to be split across catalogs.

Providers that do not declare which schemes they support match every scheme. Combining one of those with a storage provider on the same catalog produces the multiple-match error, so keep the storage providers as the only entries in credential-providers on a catalog that vends for storage.

Credentials are cached per provider and per request context, and the cache expiry follows the credential's own lifetime rather than a fixed interval. See Credential Vending for the cache settings and the properties each provider takes.

Using the API

Credential providers are ordinary catalog properties, so they are set at catalog creation and changed with a catalog alter.

Operation	Method	Path
Create catalog	POST	/api/metalakes/{metalake}/catalogs
Alter catalog properties	PUT	/api/metalakes/{metalake}/catalogs/{catalog}
Get fileset credentials	GET	/api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/filesets/{fileset}/credentials

See Manage Catalogs and Schemas for the full payloads.