AWS Glue catalog
A Glue catalog connects an AWS Glue Data Catalog, so its databases appear in Gravitino as schemas and its Hive, Iceberg, Delta, and Parquet tables can be listed, created, altered, and governed. Changes are written to Glue, where Athena, EMR, Spark, and Trino see them.
Quick start
The example connects the Glue Data Catalog in us-east-1 as the catalog glue_catalog and lists its
databases as schemas, then the tables in one of them. Without an access key in the request, the
server reaches Glue with the default AWS credential chain, such as the pod's IAM role.
- UI
- REST
-
Start the catalog. Open Connect → Catalogs and click + Catalog. Enter
glue_catalogas the Catalog name. -
Choose the provider. Keep Relational as the Catalog type and choose AWS Glue under Metastores and catalog services in Select provider. Under Connection, enter
us-east-1as the AWS region ands3://{bucket}/warehouseas the Warehouse. Leave Access key ID and Secret access key blank to use the server's default AWS credentials. -
Create the catalog. Click Test connection, then Create catalog.
-
Browse the catalog. Open Explore and open
glue_catalogto list its Glue databases as schemas. Open a schema to list its tables.
-
Create the catalog.
warehouseis where tables created through Gravitino go when neither the table nor its database names a location.Create a Glue catalogcurl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
-H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "glue_catalog",
"type": "RELATIONAL",
"provider": "glue",
"comment": "AWS Glue Data Catalog",
"properties": {
"aws-region": "us-east-1",
"warehouse": "s3://{bucket}/warehouse"
}
}'To use a static key instead, add
aws-access-key-idandaws-secret-access-key. -
List the schemas. They are the Glue databases, with names in lowercase.
List the schemas in the catalogcurl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/glue_catalog/schemas" \
-H "Accept: application/vnd.gravitino.v1+json" \
-H "Authorization: Bearer $TOKEN" -
List the tables in a database. Hive, Iceberg, Delta, and Parquet tables are all listed, unless
table-format-filternarrows them.List the tables in a schemacurl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/glue_catalog/schemas/{database}/tables" \
-H "Accept: application/vnd.gravitino.v1+json" \
-H "Authorization: Bearer $TOKEN"
Requirements
- Network access from the Gravitino server pod to the AWS Glue API, directly or through a VPC
endpoint set in
aws-glue-endpoint. - AWS credentials with the IAM permissions below: either an access key in the
catalog properties, or the default AWS credential chain of the server pod. On EKS, the default
chain picks up an IAM role given to the server's service account through
gravitino.serviceAccount.annotations; see Customize the Helm chart.
Schema and table names are case-insensitive, because Glue stores database and table names in lowercase. Column names of Hive-format tables are stored in lowercase too. Iceberg tables keep the case of their column names in the Iceberg metadata, but the copy of the columns in Glue is lowercase, so some engines cannot read mixed-case Iceberg columns.
Manage each Glue Data Catalog (one AWS account, region and aws-glue-catalog-id) from only one
Gravitino catalog, in the same way that two Hive catalogs must not share one Hive Metastore. Two
Gravitino catalogs on the same Glue Data Catalog are not supported.
Catalog
Capabilities
The Glue catalog supports creating, updating, and deleting databases and tables in the AWS Glue Data Catalog.
- Supports all table types stored in Glue (Hive, Iceberg, Delta, Parquet, and others) by default.
- Supports Hive-format table partitioning, bucketing, and sort orders.
- Does not support views. Glue views (tables with
TableType=VIRTUAL_VIEW) are filtered out.
Properties
Besides the common catalog properties, the Glue catalog has the following properties:
| Property Name | Description | Default Value | Required | Immutable |
|---|---|---|---|---|
aws-region | AWS region for the Glue Data Catalog, as a known AWS region identifier in lower case (e.g. us-east-1). | (none) | Yes | Yes |
aws-glue-catalog-id | The 12-digit AWS account ID that owns the Glue catalog. When omitted, defaults to the caller's AWS account ID. | (none) | No | Yes |
aws-access-key-id | AWS access key ID for static credential authentication. When omitted, the default credential chain is used. Temporary keys (prefix ASIA) are refused, because they need a session token. | (none) | No | No |
aws-secret-access-key | AWS secret access key paired with aws-access-key-id. When omitted, the default credential chain is used. | (none) | No | No |
aws-glue-endpoint | Glue endpoint to call instead of the regional default, such as a VPC endpoint URL. Must be an http or https URL. | (none) | No | No |
aws-glue-api-call-timeout-ms | Upper bound in milliseconds for one Glue API call, including its retries. Applies to the Glue calls of Iceberg tables too, but not to S3 reads and writes. | 30000 | No | No |
aws-glue-api-call-attempt-timeout-ms | Upper bound in milliseconds for a single attempt of a Glue API call. Also the connection and socket timeout of the S3 client used for Iceberg tables. | 10000 | No | No |
aws-glue-max-attempts | Maximum number of attempts of a Glue API call, including the first. Throttling and transient failures are retried. Applies to the Glue calls of Iceberg tables too, but not to S3 I/O. | 4 | No | No |
warehouse | Base storage path used when a table is created without a location and its Glue database declares no LocationUri (e.g. s3://my-bucket/warehouse). The table location is then warehouse/database/table. Must be a URI with a scheme; Iceberg tables need s3://, s3a:// or s3n://. | (none) | Yes | No |
default-table-format | Default format for tables created via Gravitino's createTable() API. Accepted values: iceberg, hive. | hive | No | No |
table-format-filter | Comma-separated list of table formats the catalog exposes. Accepted values: all, hive, iceberg, delta, parquet. A table is iceberg when it has table_type=ICEBERG and hive otherwise, including Delta tables registered by other engines, so delta and parquet match no table. Tables of other formats are absent from list, load, alter and drop, and cannot be created. | all | No | No |
An access key in aws-access-key-id and aws-secret-access-key takes precedence over the default credential chain of environment variables, the service account's IAM role, and the instance profile.
Operations
Refer to Catalogs and schemas for more details.
aws-access-key-id and aws-secret-access-key are hidden when the catalog is loaded; engines get storage access through credential vending.
Schema
Capabilities
The Glue catalog supports creating, updating, and deleting databases in the AWS Glue Data Catalog.
Properties
A schema has no properties of its own beyond comment. Any property you set is stored on the Glue database.
Operations
See Catalogs and schemas.
Table
Capabilities
- The Glue catalog supports creating, updating, and deleting tables in the AWS Glue Data Catalog.
- All entries in the Glue
Table.parameters()pass through Gravitino intact, so downstream tools can correctly identify the table format. - Does not support column default value.
- Does not support NOT NULL constraints on columns of Hive-format tables, including on nested fields (list elements, map values and struct fields). Iceberg tables support them.
charandvarcharcolumns of Hive-format tables follow the Hive length limits:chartakes 1 to 255 andvarchar1 to 65535. Other lengths are refused.- Does not support table indexes.
Partitioning
The Glue catalog supports partitioned tables. Create partitioned tables in the Glue catalog by specifying the partitioning attribute.
The supported partitioning strategies depend on the table format:
- Hive-format tables: Only
Identitypartitioning is supported, because the native Glue partition model is Hive-style key=value. - Iceberg-format tables: All Iceberg partition transforms are supported:
identity,year,month,day,hour,bucket, andtruncate.
The fieldName specified in the partitioning attribute must be the name of a column defined in the table.
For Hive-format tables, the partition columns must be the last columns of the table, in partitioning
order, and must have a primitive type.
Sort orders and distributions
The Glue catalog supports bucketed sorted tables. Create bucketed sorted tables by setting the distribution and sortOrders attributes.
Although Gravitino supports several distribution strategies, AWS Glue inherently only supports a single distribution strategy (clustered by column). Therefore, the Glue catalog only supports Hash distribution.
The fieldName specified in the distribution and sortOrders attribute must be the name of a column defined in the table.
Only plain column references are supported, and the number of buckets must be at least 1. A sort order
must use the default null ordering of its direction (nulls_first ascending, nulls_last descending),
because Glue stores only the direction.
Column types
The Glue catalog supports all data types defined in the Hive Language Manual. The following table lists the data types mapped from the Glue catalog to Gravitino.
| Glue Data Type | Gravitino Data Type |
|---|---|
boolean | boolean |
tinyint | byte |
smallint | short |
int / integer | integer |
bigint | long |
float | float |
double | double |
double precision | double |
decimal | decimal |
decimal (no args) | decimal(10,0) |
string | string |
char | char |
varchar | varchar |
timestamp | timestamp |
date | date |
interval_year_month | interval_year |
interval_day_time | interval_day |
binary | binary |
array | list |
map | map |
struct | struct |
uniontype | union |
Data types not listed above map to Gravitino External Type, which represents an unresolvable data type from the Glue catalog.
Unsigned integer types are refused, because Glue and Hive have signed integers only. Struct field
names must not contain :, ,, < or >, because Glue type strings cannot quote them.
Properties
Besides the properties below, any property you set is stored as a Glue table parameter. Reserved properties cannot be set by you; immutable ones cannot change after the table is created.
| Property Name | Description | Default Value | Required | Reserved | Immutable |
|---|---|---|---|---|---|
location | The location for table storage, such as s3://bucket/prefix/test_table. When not set, it is the Glue database's LocationUri followed by the table name, or warehouse/database/table when the database declares no location. | (derived from database location or warehouse) | No | No | No |
format | The table file format (parquet, orc, textfile, etc.). When set, input-format, output-format, and serde-lib are derived from it, and any of them also set must match. Without format, set all three classes or none; with none, the table is a text table. | (none) | No | No | Yes |
input-format | The input format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcInputFormat. | org.apache.hadoop.mapred.TextInputFormat | No | No | Yes |
output-format | The output format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat. | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | No | No | Yes |
serde-lib | The serde library class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcSerde. | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | No | No | Yes |
serde-name | The name of the serde. | (none) | No | No | No |
serde.parameter. | The prefix of the serde parameter, such as "serde.parameter.orc.create.index" = "true", indicating ORC serde lib to create row indexes. | (none) | No | No | No |
table-format | Format the table is created in: HIVE or ICEBERG. It is not stored; a loaded table reports ICEBERG when it has table_type=ICEBERG, and HIVE otherwise. | (catalog default-table-format) | No | No | Yes |
metadata_location | Iceberg table metadata file location stored in Glue Table.parameters(). When set during createTable(), registers an existing Iceberg table rather than creating a new one; see Register an existing table. | (none) | No | No | No |
comment | Used to store a table comment. | (none) | No | Yes | No |
Every Glue table parameter passes through Gravitino unchanged, so format markers such as table_type=ICEBERG, metadata_location, and spark.sql.sources.provider=delta reach engines intact.
Operations
Refer to Tables and views for more details.
Alter operations
Gravitino defines a unified set of metadata operation interfaces. The following table maps Glue alter operations to Gravitino table update requests.
Alter table
| Glue Alter Operation | Gravitino Table Update Request |
|---|---|
Alter Table Properties | Set a table property |
Alter Table Comment | Update comment |
Remove Properties | Remove a table property |
Hive-format table rename is not supported. AWS Glue does not provide a native rename API for tables; renaming would require recreating the table. Iceberg-format table rename is supported.
Alter column
| Glue Alter Operation | Gravitino Table Update Request |
|---|---|
Change Column Name | Rename a column |
Change Column Type | Update the type of a column |
Change Column Position | Update the position of a column |
Change Column Comment | Update the column comment |
For Hive-format tables, changes to nested struct fields are not supported; change the whole column instead. Partition columns accept only comment changes. Making a column nullable is accepted and changes nothing, since every column is nullable already, and making a column NOT NULL is refused. Bucket and sort columns can be renamed but not deleted.
Alter partition
The Glue catalog supports partition operations via SupportsPartitions for Hive-format identity-partitioned tables:
listPartitions()/listPartitionNames()getPartition(partitionName)addPartition(partition)dropPartition(partitionName)
Only IdentityPartition is supported because the Glue partition model is Hive-style key=value.
Partition names escape special characters in values as Hive does, for example / becomes %2F. A
new partition takes its columns, formats and SerDe from the table, and its location is the table
location followed by the partition name.
Iceberg tables
The Glue catalog supports creating and managing Iceberg-format tables through the Apache Iceberg SDK's GlueCatalog. When an Iceberg table is created, Gravitino writes the metadata.json file to S3 and registers the table in Glue with the correct metadata_location parameter, making it usable by Trino (Lakehouse connector), Spark, and other Iceberg-native query engines.
Create
Set table-format=ICEBERG in the table properties, or configure default-table-format=iceberg on the catalog to make all tables Iceberg by default.
The warehouse catalog property must be configured. The table location is derived as warehouse/database/table when no explicit location is specified.
Register an existing table
To register an Iceberg table that already exists in S3, set metadata_location to the path of the
existing metadata.json file during createTable(). Gravitino reads the file before it creates the
Glue table and refuses the request if the file is missing or is not Iceberg table metadata. No new
metadata file is written.
The columns, location, partitioning and sort order of a registered table come from the metadata file.
Send "columns": [] to take the columns from the file; columns you do send must match it by name. Do
not set location, partitioning or sort order, and do not give a property of the metadata file a
different value.
A registered table shares its metadata and data files with the table it came from. Gravitino marks it
with the Glue parameter gravitino.iceberg.registered=true and afterwards changes only its Glue
parameters: property changes are stored in Glue, and column, partition and other schema changes are
refused. Properties of the metadata file cannot be set or removed, because Iceberg engines read them
from that file. A registered table cannot be purged.
A table registered by Gravitino before 1.3.2, or by a registration that stopped before the parameter
was written, has no marker. Gravitino then treats it as the owner of its metadata, so an alter writes
new metadata files and forks the history it shares with the source table. Gravitino cannot detect such
tables or set the parameter. To protect one, add gravitino.iceberg.registered=true to the table
directly in AWS Glue.
Column types
Iceberg tables use the Iceberg type system, which differs from Hive types. The following table lists the Gravitino types supported for Iceberg tables and how they map to Iceberg types:
| Gravitino Data Type | Iceberg Data Type | Notes |
|---|---|---|
boolean | boolean | |
integer | int | |
long | long | |
float | float | |
double | double | |
decimal(p, s) | decimal(p, s) | |
string | string | |
varchar | string | Iceberg has no variable-length char types |
char | string | Iceberg has no variable-length char types |
date | date | |
time(6) | time | Only microsecond precision (6) is supported |
timestamp(6) | timestamp | |
timestamptz(6) | timestamptz | |
timestamp(9) | timestamp_ns | Needs format version 3 |
timestamptz(9) | timestamptz_ns | Needs format version 3 |
binary | binary | |
fixed(n) | fixed(n) | |
uuid | uuid | |
null | unknown | Needs format version 3 |
list | list | |
map | map | |
struct | struct |
byte, short and unsigned types are refused for Iceberg tables; use integer or long.
Alter operations
For Iceberg tables, the following alter operations are supported:
| Operation | Gravitino Table Update Request |
|---|---|
| Add column | Add a column |
| Delete column | Delete a column |
| Rename column | Rename a column |
| Update column type | Update the type of a column |
| Update column position | Update the position of a column |
| Update column comment | Update the column comment |
| Update column nullability | Update column nullability |
| Set table property | Set a table property |
| Remove table property | Remove a table property |
Iceberg allows a column to change only from required to optional. Making an optional column required, or adding a required column, is refused as an incompatible schema change.
The schema and property changes of one request are committed in one Iceberg transaction, so they are applied together or not at all. A request that reports an error may still have been applied, for example when the commit state is unknown or the reload after the commit fails, so check the table before retrying. Property changes of a registered table are stored in Glue, outside any Iceberg transaction.
Nested column operations, auto-increment and column default values are not supported for Iceberg tables.
Drop and purge
Dropping an Iceberg table removes only its Glue entry; the metadata and data files stay in S3. Purging an Iceberg table also deletes the data and metadata files its metadata references. Hive-format tables and registered Iceberg tables cannot be purged.
Partitions
Iceberg tables keep their partitions in Iceberg metadata, not in Glue, so partition operations are
not supported for them. The table itself reports its partitioning. Fields with the void transform,
which Iceberg v1 tables keep for dropped partition fields, are left out.
Security
AWS IAM permissions
The IAM policy attached to the credential used by the Glue catalog must cover both Glue metadata access and S3 data access:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GlueMetadataAccess",
"Effect": "Allow",
"Action": [
"glue:GetCatalog",
"glue:GetDatabase", "glue:GetDatabases",
"glue:CreateDatabase", "glue:UpdateDatabase", "glue:DeleteDatabase",
"glue:GetTable", "glue:GetTables",
"glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable",
"glue:GetPartition", "glue:GetPartitions",
"glue:CreatePartition", "glue:DeletePartition"
],
"Resource": [
"arn:aws:glue:{region}:{account_id}:catalog",
"arn:aws:glue:{region}:{account_id}:database/*",
"arn:aws:glue:{region}:{account_id}:table/*/*"
]
},
{
"Sid": "S3DataAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::{warehouse_bucket}",
"arn:aws:s3:::{warehouse_bucket}/*"
]
}
]
}