Paimon catalog

A Paimon catalog connects an Apache Paimon warehouse to Gravitino, so its databases appear as schemas and its tables and views can be listed, created, altered, and governed. Paimon keeps its own catalog metadata in a backend: the warehouse file system itself, a Hive metastore, a JDBC database, or a Paimon REST catalog service such as Aliyun DLF. The catalog uses Apache Paimon 1.2.

Quick start

The example connects a Paimon warehouse on HDFS as the catalog paimon_catalog with the filesystem backend, which needs no metastore, and creates the schema sales in it.

UI

1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter paimon_catalog as the Catalog name.

2. Choose the provider and backend. Keep Relational as the Catalog type and choose Apache Paimon under Lakehouse formats in Select provider. Under Connection, keep filesystem as the Catalog backend, choose HDFS as the Storage, and enter hdfs://{namenode_host}:8020/paimon as the Warehouse.

3. Create the catalog. Click Test connection, then Create catalog.

4. Create a schema. Open Explore, open paimon_catalog, click + Schema, enter sales as the Schema name, and click Create schema.

REST

1. Create the catalog. The filesystem backend needs only the warehouse.

   Create a Paimon catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "paimon_catalog",
       "type": "RELATIONAL",
       "provider": "lakehouse-paimon",
       "comment": "Paimon catalog",
       "properties": {
         "catalog-backend": "filesystem",
         "warehouse": "hdfs://{namenode_host}:8020/paimon"
       }
     }'

2. Create a schema. The schema is a Paimon database under the warehouse.

   Create a schema in the Paimon catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/paimon_catalog/schemas" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{"name": "sales", "comment": "Sales data", "properties": {}}'

To use a Hive metastore or a JDBC database as the backend instead, set catalog-backend to hive or jdbc and add uri, plus the JDBC backend properties for jdbc.

Catalog properties

Property	Description	Default	Required
catalog-backend	Where Paimon keeps its catalog metadata: filesystem, hive, jdbc, or rest.	(none)	Yes
uri	Address of the backend, such as thrift://{hms_host}:9083 for hive or jdbc:postgresql://{db_host}:5432/{database} for jdbc.	(none)	Unless catalog-backend is filesystem
warehouse	Root directory of the warehouse, such as hdfs://{namenode_host}:8020/paimon, s3://{bucket}/paimon, or oss://{bucket}/paimon.	(none)	Yes
catalog-backend-name	Catalog name passed to the Paimon backend.	Value of catalog-backend	No
authentication.type	How the catalog authenticates to HDFS and the metastore: simple or kerberos.	simple	No
authentication.kerberos.principal	Kerberos principal.	(none)	With Kerberos
authentication.kerberos.keytab-uri	URI of the keytab for the principal.	(none)	With Kerberos
authentication.kerberos.check-interval-sec	Seconds between checks of the Kerberos credential.	60	No
authentication.kerberos.keytab-fetch-timeout-sec	Seconds allowed to fetch the keytab from authentication.kerberos.keytab-uri.	60	No
gravitino.bypass.hive.metastore.sasl.enabled	Use SASL when connecting to a Kerberized Hive metastore. Usually true with Kerberos. The bare key without the prefix is ignored.	false	No
s3-endpoint	S3 endpoint.	(none)	For an S3 warehouse
s3-access-key-id	S3 access key ID.	(none)	For an S3 warehouse
s3-secret-access-key	S3 secret access key.	(none)	For an S3 warehouse
oss-endpoint	Aliyun OSS endpoint.	(none)	For an OSS warehouse
oss-access-key-id	Aliyun OSS access key ID.	(none)	For an OSS warehouse
oss-secret-access-key	Aliyun OSS secret access key.	(none)	For an OSS warehouse
token-provider	How the rest backend authenticates: bearer or dlf.	(none)	With the rest backend
token	Bearer token for the Paimon REST catalog.	(none)	With token-provider set to bearer
dlf-access-key-id	Aliyun DLF access key ID.	(none)	With token-provider set to dlf
dlf-access-key-secret	Aliyun DLF access key secret.	(none)	With token-provider set to dlf
dlf-security-token	Aliyun DLF security token.	(none)	No
dlf-token-path	Aliyun DLF token path.	(none)	No
dlf-token-loader	Aliyun DLF token loader.	(none)	No

Kerberos works with the filesystem and jdbc backends, not with hive.

Properties that start with gravitino.bypass. are passed to Paimon with the prefix removed; for example, gravitino.bypass.table.type reaches Paimon as table.type.

The server image includes the Paimon S3 and OSS file systems, so an S3 or OSS warehouse needs only the properties above. Secret properties such as s3-secret-access-key and jdbc-password are hidden when the catalog is loaded; engines get storage access through credential vending.

JDBC backend

The jdbc backend keeps Paimon's catalog metadata in a MySQL or PostgreSQL database.

Property	Description	Default	Required
jdbc-user	Database user.	(none)	With the jdbc backend
jdbc-password	Database password.	(none)	With the jdbc backend
jdbc-driver	org.postgresql.Driver for PostgreSQL, com.mysql.cj.jdbc.Driver for MySQL.	(none)	With the jdbc backend

The PostgreSQL driver ships in the server image. The MySQL driver does not; supply it with gravitino.jdbcDrivers as described in the MySQL catalog page.

REST backend and Aliyun DLF

The rest backend connects to a Paimon REST catalog service. For Aliyun DLF, set token-provider to dlf and give the DLF keys.

Create a Paimon catalog on Aliyun DLF

curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "dlf_paimon",
    "type": "RELATIONAL",
    "provider": "lakehouse-paimon",
    "properties": {
      "catalog-backend": "rest",
      "uri": "{dlf_endpoint}",
      "warehouse": "{dlf_catalog}",
      "token-provider": "dlf",
      "dlf-access-key-id": "{access_key_id}",
      "dlf-access-key-secret": "{access_key_secret}"
    }
  }'

See the Paimon REST catalog documentation for the service itself.

Schemas

Schemas can be created, listed, loaded, and dropped, including a cascading drop. Altering a schema is not supported. With the filesystem backend, a schema keeps no properties and no comment, and cannot take a location. Other operations are covered in Catalogs and schemas.

Tables

Tables can be created, listed, loaded, altered, and purged. Dropping is not supported, because Paimon's drop deletes the table data without passing through the trash, so use purge to remove a table. Other operations are covered in Tables and views.

• Changes: rename the table; add, delete, or rename a column; change a column's comment, nullability, position, or type; change the table comment; set or remove a property.
• Partitions: identity partitions only. Do not make the primary key the same as the partition columns, or each partition holds a single row.
• Indexes: one primary key, which can span several columns.
• Buckets: set through the table distribution with the HASH strategy, which fills Paimon's bucket and bucket-key options.
• Not supported: sort orders, auto-increment columns, and the Paimon MultisetType.
• Default values: set a column default with the table property fields.{column_name}.default-value, not a column expression.

Column types

Gravitino Type	Apache Paimon Type
Struct	Row
Map	Map
List	Array
Boolean	Boolean
Byte	TinyInt
Short	SmallInt
Integer	Int
Long	BigInt
Float	Float
Double	Double
Decimal	Decimal
String	VarChar(Integer.MAX_VALUE)
VarChar	VarChar
FixedChar	Char
Date	Date
Time(p)	Time(p)
Timestamp(p)	Timestamp(p)
Timestamp_tz(p)	LocalZonedTimestamp(p)
Fixed	Binary
Binary	VarBinary

Table properties

Any Paimon table option can be passed as a table property, except the reserved ones below, which Gravitino sets itself. Immutable properties cannot change after the table is created.

Property	Description	Reserved	Immutable
merge-engine	Merge engine of a primary key table.	No	Yes
sequence.field	Field that orders records with the same key.	No	Yes
rowkind.field	Field that carries the row kind.	No	Yes
comment	Table comment, from the table definition.	Yes	No
owner	Table owner.	Yes	No
bucket-key	Bucket key, from the distribution.	Yes	No
bucket	Bucket count, from the distribution.	Yes	No
primary-key	Primary key, from the index.	Yes	No
partition	Partition columns, from the partitioning.	Yes	No

Views

Views can be listed, created, loaded, altered, and dropped when the Paimon backend supports views. Each view needs exactly one SQL representation with the dialect query, which is the canonical definition, and may add one representation per other dialect, such as flink or spark. The view's defaultCatalog and defaultSchema resolve unqualified names in the SQL. A rename must be sent on its own, not combined with other changes in one request. See Tables and views.

HDFS configuration

The catalog loads core-site.xml and hdfs-site.xml from /opt/gravitino/catalogs/lakehouse-paimon/conf in the server container. To supply them, put the files in a ConfigMap and mount each one at that path with a subPath, using gravitino.extraVolumes and gravitino.extraVolumeMounts as described in Customize the Helm chart.

With Kerberos, the server reads and writes HDFS as the configured principal only; it cannot act as other Kerberos users.

Related

• Catalogs and schemas
• Tables and views
• Credential vending