Skip to main content

Supported catalogs

The connector exposes Hive, Iceberg, MySQL, PostgreSQL, and AWS Glue catalogs to Trino. Each is backed by the corresponding Trino connector and behaves the way that connector behaves. The difference is that catalogs come from Gravitino rather than from a properties file.

Managing Catalogs​

Three stored procedures live under the gravitino connector in the system schema. Both positional and named arguments work.

create_catalog(CATALOG varchar, PROVIDER varchar, PROPERTIES MAP(VARCHAR, VARCHAR), IGNORE_EXIST boolean);
drop_catalog(CATALOG varchar, IGNORE_NOT_EXIST boolean);
alter_catalog(CATALOG varchar, SET_PROPERTIES MAP(VARCHAR, VARCHAR), REMOVE_PROPERTIES ARRAY[VARCHAR]);

PROVIDER is hive, lakehouse-iceberg, jdbc-mysql, jdbc-postgresql, or glue. The IGNORE_* flags are optional and default to false.

CALL gravitino.system.create_catalog(
    'mysql_catalog',
    'jdbc-mysql',
    map(
        array['jdbc-url', 'jdbc-user', 'jdbc-password', 'jdbc-driver'],
        array['jdbc:mysql://{host}:3306?useSSL=false', '{user}', '{password}', 'com.mysql.cj.jdbc.Driver']
    )
);

Catalog properties are documented with the catalog itself, on the Hive, Iceberg, MySQL, PostgreSQL, and Glue pages. SELECT * FROM gravitino.system.catalog lists what is registered.

Trino connector settings pass through with a trino.bypass. prefix on the property key. Settings the Trino connector inherits from the Gravitino catalog cannot be overridden this way: the MySQL connector takes connection-url, connection-user, and connection-password from the catalog's jdbc-url, jdbc-user, and jdbc-password, so a trino.bypass.connection-url is silently ignored.

Type Mapping​

Gravitino typeTrino type
BooleanBOOLEAN
ByteTINYINT
ShortSMALLINT
IntegerINTEGER
LongBIGINT
FloatREAL
DoubleDOUBLE
DecimalDECIMAL
String, VarcharVARCHAR
FixedCharCHAR
BinaryVARBINARY
DateDATE
TimeTIME
TimestampTIMESTAMP
TimestampWithTimezoneTIMESTAMP WITH TIME ZONE
ListARRAY
MapMAP
StructROW

A given provider may not support every row. Hive has no TIME type, for example.

Hive​

Requires a Hive metastore or a compatible implementation, reachable from the coordinator and every worker, with Thrift on port 9083 by default. HDFS 2.x is supported, along with S3 and S3-compatible systems, GCS, Azure Storage, and IBM Cloud Object Storage. Files must be ORC, Parquet, Avro, RCFile, SequenceFile, JSON, CSV, or text.

Schemas take a location property. Tables take format (default TEXTFILE), location, input_format, output_format, serde_lib, serde_name, partitioned_by, bucketed_by, bucket_count, and sorted_by. Four more are set by Gravitino and readable but not settable: total_size, num_files, external, and table_type.

The HDFS client is configured from catalog properties, with hdfs-site.xml and core-site.xml supplied through trino.bypass.hive.config.resources. Before any INSERT, confirm the user Trino runs as can write to the warehouse directory, overriding it with -DHADOOP_USER_NAME={user} in the Trino JVM configuration if needed.

S3 credentials pass through as trino.bypass.hive.s3.aws-access-key, trino.bypass.hive.s3.aws-secret-key, and trino.bypass.hive.s3.region. The metastore behind the catalog has to support S3 itself.

Iceberg​

Requires network access to the object store plus a Hive metastore, Glue catalog, JDBC catalog, REST catalog, or Nessie server. Files must be ORC or Parquet, Parquet by default.

Schemas take no properties. Tables take partitioning and sorted_by.

Maintenance is delegated to the Iceberg connector through ALTER TABLE ... EXECUTE: expire_snapshots, remove_orphan_files, optimize, and rewrite_manifests. Expiring below the default retention threshold requires the minimum retention override to be set at or below the requested value. Parameters are in the Trino Iceberg documentation.

S3 needs two credential sets, which is the part that catches people out. The trino.bypass.hive.s3.* properties configure the Trino side and the s3-* properties configure the Gravitino Iceberg catalog. Both are required, with the same values.

CALL gravitino.system.create_catalog(
    'iceberg_catalog',
    'lakehouse-iceberg',
    map(
        array['uri', 'catalog-backend', 'warehouse',
          'trino.bypass.hive.s3.aws-access-key', 'trino.bypass.hive.s3.aws-secret-key', 'trino.bypass.hive.s3.region',
          's3-access-key-id', 's3-secret-access-key', 's3-region', 'io-impl'
        ],
        array['thrift://{hms_host}:9083', 'hive', 's3a://{bucket}/{path}',
        '{access_key}', '{secret_key}', '{region}',
        '{access_key}', '{secret_key}', '{region}', 'org.apache.iceberg.aws.s3.S3FileIO']
    )
);

The warehouse path must exist on S3 already. See Iceberg catalog for the Gravitino side. HADOOP_USER_NAME applies here as it does for Hive.

MySQL​

MySQL 5.7 or 8.0 and later, port 3306 by default. Schemas take no properties.

Tables take engine (default InnoDB), auto_increment_offset, primary_key whose columns must all be NOT NULL, and unique_key, each written as keyName:col1,col2. Columns take auto_increment and default.

Only constant defaults are supported, not expressions, and SHOW CREATE TABLE renders only constants. Defaults are accepted on TINYINT, SMALLINT, INT, BIGINT, REAL, DOUBLE, DECIMAL, VARCHAR, CHAR, DATE, TIME, and TIMESTAMP.

PostgreSQL​

PostgreSQL 10.x or later, port 5432 by default. Tables and schemas take no properties.

PostgreSQL folds unquoted identifiers to lowercase, so MyTable and MYTABLE both resolve to mytable, while an identifier created with quotes keeps its case and must be quoted thereafter. Use unquoted identifiers with this connector; names containing uppercase letters may not be found otherwise.

AWS Glue​

Requires network access to the Glue API and S3, IAM credentials with Glue and S3 permissions as described on the Glue catalog page, and data on S3.

Tables default to Hive format. Setting type = 'ICEBERG' creates an Iceberg table and unlocks Iceberg's partition transforms in partitioned_by: identity, year, month, day, hour, bucket(column, N), and truncate(column, W).

Schemas take location. Tables take type (HIVE or ICEBERG), format, location which defaults to a path derived from the catalog warehouse, partitioned_by, and for Hive format bucketed_by, bucket_count, and sorted_by.

CREATE OR REPLACE TABLE AS SELECT is not supported. Drop and recreate instead.

Last updated on