Hive catalog

A Hive catalog connects an Apache Hive metastore, so its databases appear in Gravitino as schemas and its tables and views can be listed, created, altered, and governed. Every change Gravitino makes is written to the metastore, where Hive, Spark, Trino, and other engines see it.

For which content in a metastore the Hive catalog serves, and which tables need an Iceberg, Hudi or Paimon catalog instead, see Hive metastore.

Quick start

The example connects the Hive metastore at thrift://{metastore_host}:9083 as the catalog hive_catalog, creates the schema sales in it, and lists the schemas. The list includes the databases that already existed in the metastore, and sales is created there as a Hive database.

UI

1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter hive_catalog as the Catalog name, keep Relational as the Catalog type, and choose Apache Hive in Select provider.

2. Enter the connection. Under Connection, enter thrift://{metastore_host}:9083 as the Metastore URI.

3. Test and create it. Click Test connection, then Create catalog.

4. Create the schema. In Explore, open hive_catalog to list its schemas, click + Schema, and create sales.

REST

1. Create the catalog.

   Create a Hive catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "hive_catalog",
       "type": "RELATIONAL",
       "provider": "hive",
       "comment": "Hive metastore",
       "properties": {"metastore.uris": "thrift://{metastore_host}:9083"}
     }'

2. Create the schema. With no location, the metastore places it under its hive.metastore.warehouse.dir.

   Create a schema

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/hive_catalog/schemas" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{"name": "sales", "comment": "Sales data"}'

3. List the schemas.

   List the schemas in the catalog

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/hive_catalog/schemas" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

Requirements

• A Hive metastore (HMS), version 2.x or 3.x. The catalog detects the version itself. For AWS Glue, use the Glue catalog.
• Network access from the Gravitino server pod to the metastore's Thrift port, 9083 by default.

Catalog properties

Besides the common catalog properties, the Hive catalog takes these:

Property Name	Description	Default Value	Required
metastore.uris	Metastore addresses, separated by commas, such as thrift://{metastore_host}:9083.	(none)	Yes
client.pool-size	Maximum number of metastore clients the catalog keeps open.	1	No
gravitino.bypass.	Passed to the metastore client with the prefix removed; for example, gravitino.bypass.hive.metastore.failure.retries sets hive.metastore.failure.retries.	(none)	No
client.pool-cache.eviction-interval-ms	Interval at which idle metastore clients are evicted, in milliseconds.	300000	No
impersonation-enable	Act as the calling user when talking to the metastore and HDFS.	false	No
kerberos.principal	Kerberos principal of the catalog. With Kerberos, also set gravitino.bypass.hadoop.security.authentication, gravitino.bypass.hive.metastore.kerberos.principal, and gravitino.bypass.hive.metastore.sasl.enabled.	(none)	required if you use Kerberos
kerberos.keytab-uri	URI of the keytab, over https, http, ftp, or file.	(none)	required if you use Kerberos
kerberos.check-interval-sec	Seconds between checks of the Kerberos credential.	60	No
kerberos.keytab-fetch-timeout-sec	Seconds allowed to fetch the keytab.	60	No
list-all-tables	Whether to list all tables in a database, including non-Hive tables such as Iceberg, Paimon, and Hudi. When false, non-Hive tables are filtered out on a best-effort basis; see the note below for known limitations.	false	No
default.catalog	Hive catalog to use in a Hive 3 metastore. Ignored by a Hive 2 metastore.	hive	No

With list-all-tables left at false, the metastore does the filtering and can match only property keys without dots. A Hudi table written by Spark, for example with saveAsTable, usually sets only spark.sql.sources.provider=hudi, so it still appears. Give such a table the property provider=hudi so the filter can match it:

Set the provider on an existing table

ALTER TABLE {db}.{table} SET TBLPROPERTIES ('provider'='hudi');

Engine settings

Properties with the trino.bypass. prefix reach the Hive connector in Trino with the prefix removed; for example, trino.bypass.hive.config.resources sets hive.config.resources. Properties with the spark.bypass. prefix reach the Spark Hive connector the same way, such as spark.bypass.hive.exec.dynamic.partition.mode.

To push Gravitino grants down to Apache Ranger, add the properties in Authorization pushdown.

Secret properties such as s3-secret-access-key and oss-secret-access-key are hidden when the catalog is loaded; engines get storage access through credential vending. Other catalog operations are covered in Catalogs and schemas.

Schemas

A schema is a Hive database, and the catalog creates, alters, and drops databases in the metastore. Besides location, any property you set is stored on the database.

Property name	Description	Default value	Required
location	Directory of the database, such as hdfs://{namenode_host}:8020/user/hive/warehouse/sales.db.	The metastore's hive.metastore.warehouse.dir	No

To keep a schema's tables in Amazon S3, Azure Data Lake Storage or Google Cloud Storage, give it an object storage location; the metastore must be able to reach the store. See Hive tables in object storage.

Other schema operations are covered in Catalogs and schemas.

Tables

The catalog creates, alters, and drops tables in the metastore. Column default values and indexes are not supported.

Tables written by Spark

Spark records a table created with USING as a data source table. When the format is one Hive can read, such as Parquet or ORC, Spark also writes the real columns to the metastore and the table loads in Gravitino with its columns as usual. For other formats, such as CSV with USING csv, Spark keeps the real schema only in its own table properties and gives the metastore a single placeholder column, col array<string>. Gravitino reads the columns from the metastore, so it shows only that placeholder, and the table cannot be used through Gravitino. Convert such a table to Parquet, ORC or Iceberg, for example with CREATE TABLE ... USING parquet AS SELECT * FROM ... in Spark, and create new Spark tables in Parquet or ORC.

Partitioning

Hive partitions by column values, so only identity partitioning is supported, and each partition field must be a column of the table. Partitions can then be listed, added, and dropped; see Partitions.

Sort orders and distributions

Hive bucketed sorted tables are created with a HASH distribution, the only kind Hive has, and sort orders. Every field in them must be a column of the table.

Column types

Hive data types map to Gravitino types as follows.

Hive Data Type	Gravitino Data Type
boolean	boolean
tinyint	byte
smallint	short
int/integer	integer
bigint	long
float	float
double/double precision	double
decimal	decimal
string	string
char	char
varchar	varchar
timestamp	timestamp
date	date
interval_year_month	interval_year
interval_day_time	interval_day
binary	binary
array	list
map	map
struct	struct
uniontype	union

Any other Hive type appears as a Gravitino external type. A struct with field comments is rejected, because Hive cannot store it (HIVE-26593).

Properties

Besides the properties below, any property you set is stored on the Hive table. Reserved properties are set by Hive or Gravitino and cannot be set by you; immutable ones cannot change after the table is created.

Property Name	Description	Default Value	Required	Reserved	Immutable
location	The location for table storage, such as /user/hive/warehouse/test_table.	HMS uses the database location as the parent directory by default.	No	No	Yes
table-type	Type of the table. Valid values include MANAGED_TABLE and EXTERNAL_TABLE.	MANAGED_TABLE	No	No	Yes
format	The table file format. Valid values include TEXTFILE, SEQUENCEFILE, RCFILE, ORC, PARQUET, AVRO, JSON, CSV, and REGEX.	TEXTFILE	No	No	Yes
input-format	The input format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.	The property format sets the default value org.apache.hadoop.mapred.TextInputFormat and can change it to a different default.	No	No	Yes
output-format	The output format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat.	The property format sets the default value org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat and can change it to a different default.	No	No	Yes
serde-lib	The serde library class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcSerde.	The property format sets the default value org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe and can change it to a different default.	No	No	Yes
serde.parameter.	The prefix of the serde parameter, such as "serde.parameter.orc.create.index" = "true", indicating ORC serde lib to create row indexes	(none)	No	No	Yes
serde-name	The name of the serde	Table name by default.	No	No	Yes
comment	Used to store a table comment.	(none)	No	Yes	No
numFiles	Used to store the number of files in the table.	(none)	No	Yes	No
totalSize	Used to store the total size of the table.	(none)	No	Yes	No
EXTERNAL	Indicates whether the table is external.	(none)	No	Yes	No
transient_lastDdlTime	Used to store the last DDL time of the table.	(none)	No	Yes	No

Alter operations

Most Hive ALTER operations have a matching table update request:

Alter table

Hive Alter Operation	Gravitino Table Update Request
Rename Table	Rename table
Alter Table Properties	Set a table property
Alter Table Comment	Update comment
Alter SerDe Properties	Set a table property
Remove SerDe Properties	Remove a table property
Alter Table Storage Properties	Unsupported
Alter Table Skewed or Stored as Directories	Unsupported
Alter Table Constraints	Unsupported

The table comment is changed with the update comment request, not as the comment property, which is reserved.

Alter column

Hive Alter Operation	Gravitino Table Update Request
Change Column Name	Rename a column
Change Column Type	Update the type of a column
Change Column Position	Update the position of a column
Change Column Comment	Update the column comment
Add Columns	Add a column
Replace Columns	Delete a column

A partition column can only have its comment changed; renaming, retyping, moving, or deleting it is rejected. A column cannot be added or moved after a partition column; a new column without a position goes after the last non-partition column.

Partitions are added and dropped with their own requests; see Partitions. Other table operations are covered in Tables and views.

Views

Views are stored in the metastore as VIRTUAL_VIEW and can be listed, created, loaded, altered, and dropped. See Tables and views.

• Each view has exactly one SQL representation, in the hive, flink, or spark dialect. When an existing view is loaded, Gravitino detects its dialect from the view's properties and text.
• Views created by Trino cannot be loaded or changed through Gravitino. Loading one returns an error naming the trino dialect.
• For hive and flink, defaultCatalog and defaultSchema must be empty.
• A flink view needs at least one view property starting with flink.. The Flink connector sets flink.schema.num-columns itself; through the REST API, set one yourself.
• A spark view needs the property spark.sql.create.version; without it the view reloads as hive.

Related

• Hive tables in object storage
• Authorization pushdown
• Catalogs and schemas
• Tables and views