Skip to main content

Catalog connections

A catalog connects Gravitino to a system that already holds metadata, such as a lakehouse, a metastore, or a relational database. Gravitino does not copy that metadata. It reads and writes through to the source, so the source stays authoritative and a change made outside Gravitino is visible the next time the catalog is read.

Each kind of source has its own page with the properties it needs; Source types lists them. The sections below cover what every catalog has in common: how a connection is validated, how properties behave, and what happens when a catalog cannot be reached.

Quick start​

The example connects a PostgreSQL database as the catalog pg_sales, confirms the connection, and lists the schemas it finds. Other providers follow the same steps with their own connection details, listed on each provider's page.

  1. Start the catalog. Open Connect → Catalogs, click + Catalog, enter pg_sales as the Catalog name, and choose Relational as the Catalog type and PostgreSQL in Select provider.

  2. Enter the connection. Under Connection, enter jdbc:postgresql://pg.example.com:5432/sales as the URL, sales as the Database, and the User and Password Gravitino connects as. Advanced holds the provider's other properties.

  3. Test and create it. Click Test connection, then Create catalog. Gravitino tests the connection again before it saves the catalog, so a catalog that is created has reached its source.

  4. List the schemas. In Explore, open pg_sales. The schemas listed are read from the database, so one created there directly appears without any step in Gravitino.

The Catalogs list shows each catalog's provider, endpoint, and connection status, with controls on each row to test the connection again and to delete the catalog.

Source types​

SourceProviderPage
Apache Iceberglakehouse-icebergIceberg
Apache Paimonlakehouse-paimonPaimon
Apache Hudilakehouse-hudiHudi
Lance and Delta tableslakehouse-genericLance and Delta
Apache Hive metastorehiveHive
AWS GlueglueGlue
Relational databases over JDBCjdbc-mysql, jdbc-postgresql, and eight othersJDBC catalogs
Files and directoriesNone; catalog type FILESETFileset
Apache KafkakafkaKafka
ML modelsNone; catalog type MODELML models

A catalog's provider is fixed when the catalog is created. Fileset and model catalogs take no provider, because Gravitino manages what they hold. Properties shared by every JDBC connector, such as jdbc-url and the connection pool settings, are on JDBC catalogs.

Connection test at creation​

Gravitino tests the connection before it commits the catalog. It builds a throwaway catalog instance from the properties you supplied, asks the connector to test the connection, and closes it again. Only then is the catalog entity written.

A wrong host, a bad credential, or a missing driver therefore surfaces immediately as a failed create rather than as a working catalog that fails on first use. The error comes from the connector and names the underlying cause.

Property behavior​

Properties fall into four categories, and a connector's page states which of its own properties fall where.

Required properties must be supplied at creation. Optional properties have defaults that apply when omitted. Immutable properties are fixed at creation and cannot be changed by an alter. Reserved properties are set by Gravitino rather than by you, and are rejected if you try to supply them.

A property may also be hidden, which affects reading rather than writing. A hidden property is accepted at creation and used by the connector, but is not returned when the catalog is loaded. Credentials are hidden for this reason, so a password you set is never echoed back.

Beyond a connector's own properties, every catalog accepts a small set of common ones covering the classpath package to load the connector from, an override for the catalog operations implementation, an authorization provider, and the cloud and region the catalog runs in. Gravitino also maintains reserved properties recording whether the catalog and its metalake are in use.

Catalog lifetime and cache​

A connected catalog is cached in memory along with its connection pool and its isolated classloader. The cache evicts on an access interval rather than on a fixed lifetime, so a catalog nobody has touched is closed and its connections released, and the next request reopens it.

The practical consequence is that the first request to an idle catalog pays reconnection cost. A source that is slow to accept connections shows this as intermittent latency on otherwise identical requests.

Each catalog loads its connector in an isolated classloader, which is why two catalogs can use different, incompatible driver versions of the same database in one server.

When a catalog cannot be reached​

A catalog whose source has become unreachable stays defined. Gravitino holds the catalog entity in its own store, so the catalog continues to appear in listings and its properties remain readable while operations against it fail.

The failure surfaces per operation and carries the connector's own error. Listing schemas on an unreachable catalog fails; listing catalogs in the metalake does not.

A catalog can also be disabled deliberately, which is distinct from being unreachable. A disabled catalog rejects operations because Gravitino refuses them, not because the source is down, and disabling a metalake disables all its catalogs.

Multiple storage systems​

A single catalog can hold data in several storage systems at once. An Iceberg catalog can keep tables in Amazon S3, Google Cloud Storage and an on-premises HDFS cluster, and a fileset catalog can do the same for files. Callers reach all of them through one endpoint, and tags, policies and access control apply to everything in the catalog wherever each object is stored.

Storage is not configured once per catalog. The catalog lists credential providers in credential-providers, and Gravitino picks one for each object from the URI scheme of that object's location when the object is accessed. A table at s3://{bucket}/orders gets S3 credentials and a table at gs://{bucket}/events in the same catalog gets Google Cloud Storage credentials.

The catalog mixed_storage lists the s3-token and gcs-token providers; its table at s3://bucket/orders is served by s3-token with Amazon S3 credentials and its table at gs://bucket/events by gcs-token with Google Cloud Storage credentials, chosen when the table is loaded

Create a catalog with S3 and GCS
curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
        "name": "mixed_storage",
        "type": "RELATIONAL",
        "provider": "lakehouse-iceberg",
        "properties": {
          "catalog-backend": "jdbc",
          "uri": "jdbc:postgresql://{host}:5432/{database}",
          "jdbc-driver": "org.postgresql.Driver",
          "jdbc-user": "{jdbc_user}",
          "jdbc-password": "{jdbc_password}",
          "warehouse": "s3://{bucket}/{prefix}",
          "credential-providers": "s3-token,gcs-token",
          "s3-access-key-id": "{access_key_id}",
          "s3-secret-access-key": "{secret_access_key}",
          "s3-role-arn": "arn:aws:iam::{account}:role/{role_name}",
          "s3-region": "{region}",
          "gcs-service-account-file": "/etc/gcs/key.json"
        }
      }'

Objects are then created in whichever storage system each belongs in, with no further configuration. Each provider's properties are on its object storage page. Per-object selection applies to Iceberg tables loaded through the Iceberg REST service and to filesets read through GVFS, where each object has its own location.

Choose a credential​

Gravitino extracts the scheme from the object's location and uses the provider that supports it. Adding a storage system to an existing catalog means adding one provider and its properties; existing objects are unaffected.

Storage systemSchemesCredential providers
Amazon S3 and S3-compatible storages3, s3a, s3ns3-token, s3-secret-key, aws-irsa
Google Cloud Storagegs, gcsgcs-token
Azure Data Lake Storageabfs, abfss, wasb, wasbsadls-token, azure-account-key
Alibaba Cloud OSSossoss-token, oss-secret-key
HDFS and local file systemshdfs, fileNone; see On-premises storage

Only one provider may claim a scheme on a catalog. Listing two for the same storage system, such as s3-token with s3-secret-key, fails at request time with an error naming both.

A fileset with locations in several storage systems gets one credential per system in a single response; locations no provider claims are left out. An Iceberg table has one location and gets one credential; requesting vended credentials for a table whose scheme no provider claims fails with HTTP 400.

On-premises storage​

Tables and files on HDFS and local file systems belong in the same catalog but are not served with vended credentials, because there is no short-lived credential to hand out. A location without a scheme is treated as a local path. Engines reach that data with their own credentials, which for a secured cluster means Kerberos; configure the catalog as described in Catalog backend security and pass HDFS client settings as gravitino.bypass. properties, as in HDFS configuration.

Limits​

  • The scheme is the only input, so two accounts in the same cloud cannot be separated in one catalog. Use a credential that can serve both, such as a role assumable across them, or split the buckets across catalogs.
  • A provider that declares no schemes matches every scheme, and combining it with a storage provider produces the multiple-match error. Keep storage providers as the only entries in credential-providers on a catalog that vends for storage.
  • Credentials are cached per provider and request context until they expire. See Credential vending for the cache settings and each provider's properties.
Last updated on